I'm interested in both:
1. Basic (must have) metrics
2. Advanced (good to have) metrics
It depends (tm BobeV) on what you want to manage and report on!
Recording any data just for the sake of it is somewhat pointless and reporting on what is "right" for me might be of no use to you..
So define what metrics are useful to you and then record and report on them....
Obviously any tool you use will have a fair few default/standard metadata attributes which will probably give you a place to start.
To expand on Drew's 'it depends', some examples of common situations I've encountered include:
Build success rates: measuring build failures and root cause of build failures is interesting when evaluating your environment over time. An increase in build failures could indicate issues with your hardware, changes in process that haven't been reflected in build process, or code churn. Being able to report on these numbers demonstrates you have an understanding of the health of your build environment.
Build times: measuring the time it takes to complete sections of a build can be useful for understanding how to improve build performance - where you can get the most improvement for the least investment. Or it can be a measure of the health of your environment - are builds starting to take longer? You can start investigating what has changed - maybe the code base has simply increased, but maybe a machine is developing a hardware issue or, worse, a virus or malware.
Metrics for the sake of metrics are a waste of everyone's time, so define what you want to understand and build from there. Also important, though, is to review what you're reporting on periodically - make sure it is still reporting accurate information and that the reports are still relevant.
nsheth,
Can't add much to what has already been said, but I would ask you this? Who is driving this? Is this something you want for your own sake or is this being requested by a higher up? If the latter, ask them one question, What are you trying to gather from these Metrics? Metrics without interpretation are useless, numbers can be skewed or used against you or others unless context in which the Metrics were gathered.
I had one simple Metric I used for Version Control at a site once. It was really telling of the situation. We had a group that said they where a week away from releasing code into the test environment. So I created a baseline and then ran a report to show what had changed in the last week. They would consistently have 3000 to 3500 files checked in since the baseline was set. Though not all of those files where changes, people could check in unchaged files, if even 10% where changes, I would say that is way to high for a product about to be released into a UAT environment.
But again, Metrics can be used incorrectly and with malice and that's dangerous, so be careful before you release results or start gathering them to make sure of the reason why they are needed.
Reagrds,
Joe
Drew, Veroamore and Joe Thank you for your comments.
The metrics that I want to create are for
1) Project Manager - as an SCM engineer I want to show project status to the PM on a weekly basis from week one of the project. This mean I want to be able to give status to the PM, even before the code/baseline has been give to SCM for the first build. ( I hope it make senses, for example, in a 4 month project with tight deadline and QA date starting off in 3rd month, I want to be able to give weekly project status to the PM from the SCM point of view from week one)
2) SCM Management - We already collect certain metrics, like Build success rates, Build times & Line of code changed between each build, however is there additional metrics that I can offer either the PM or Testing or SCM ?
Aha! Identification of "who wants it" and "what do they want" is a big help.
nsheth,
1. You might track Test Problem Reports (TPR) over time-increments or TPR per module over increments. The premise is that "as the module matures, fewer defects will occur." Be careful here because the PM must understand that the desired outcome is a diminishing number of TPR throughout the period (down to zero or near-zero for the release).
2. You might also track TPR by type of defect. (For example: "Typos" "Unclosed Loops" "Build Failures", etc) In other words, are we being careless (typos), do we need a code-check in the process, do we need training on the build engine, etc?
3. For short increments, say 1-week, track "New change reports" (anything, from requirements changes to testing defects, those that require a change to the code), "Close change reports." Using these two measurements, you can compute "Currently Open." Two more measures can give a more complete picture to the PM, "(Total) Modules (or components/pieces) to create" and "Modules Started."
The combination of “Modules to Create” and “Modules Started” lets the PM know how much of the project is being worked on and will also help to explain the defect report numbers. (It is to be expected that the number of defect reports will be higher if more components are being worked on.)
For example:
Suppose you have a product of 20 modules.
START
Modules to create - 20
Modules started - 0
New defects reported - 0
Closed - 0
Now Open - 0
END OF WEEK 1
Modules to create - 20
Modules started - 3
New defects reported - 5 (This is for the 1-week period)
Defects closed - 3 (This is the total "closed" to date)
Now Open - 2 (This is as of the end of the week)
END OF WEEK 2
Modules to create - 20
Modules started - 5
New defects reported - 8 (since the last report from Week 1)
Defects closed - 5 (three from last week and two this week)
Now Open - 8 (had 2 open from Week 1, plus 8 new ones this week, minus the 5 that were closed this week).
If you graph these on a single chart and present it to the PM on a weekly basis, he/she can have a picture of what's going on, PROVIDED THAT he/she understands how to interpret the data. What I mean is this --
Each new module will present a spike in defect reports and this spike will diminish as the module reaches maturity;
It should be expected that each additional module/component under development will cause a rise in "new defects" overall; Maturity of each module should mean fewer defects overall;
All defects cannot be closed in a single week.
Thank You for your response.
CMCrossroads is a TechWell community.
Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.