 SCM people have for years supplied metrics
data to the rest of the software organization because we sit on the repository,
which is a literal gold mine of data waiting to be mined. Various parts of the
organization have used these metrics to measure their efficiency and the effect
of changes, to plan and schedule their resources, and to monitor their
processes.
We believe it is about time SCM started
using this gold mine of data themselves to pull out metrics that can provide
the same benefits for the people, processes and tools that SCM is responsible
for.
Since we did not want to reinvent the
wheel, we started by looking around for previous work and inspiration - the
result of which we report on in this first part.
Metrics and SCM
We have several times
found ourselves in situations where we have had to convince other people of
something. It could be to convince management that we need a new tool, or it
could be to convince students that they should work in a different - and better
- way on their projects. In the first case, the bean counters would often ask
for the economic return - in the latter case, the students would usually say
"yeah, right" and continue the same way they had always done (to be true, some
students would ask "why").
In many cases it turned
out that we had little more than "good reasoning" and "gut feeling" to argue
our case. Gradually we realized that a couple of appropriate numbers in many
cases could terminate endless discussions - or get attention from students or
developers. So we started looking into metrics for SCM.
In general, metrics can be used for:
- funding change: first they
buy-in, then we verify that
savings are as promised
- planning: estimating and managing resources
- insurance: early indicators (trends) that we are (will be)
going off track
- fine-tuning: comparison with benchmark values
There are many examples
of metrics around and much of the data for metrics is provided by the SCM team,
because we sit on top of data and meta-data in our repository. In the beginning the "funding" part was most dominant for us, but
gradually we discovered the advantages of the others.
However, we were not interested in metrics in general, but only in metrics that
could serve our specific interests as responsible for SCM tasks, tools and
processes. Your situation as an SCM responsible might be different from ours,
so some of the metrics we find valuable might not interest you - and the other
way around.
Since our interest for SCM specific metrics
started quite some time ago, most of our work was done prior to March 2007. We
had wished for the special issue of the CM Journal to have happened two years
earlier - that would have made our life easier. However, still find that it is
worth reporting our experience to people that are in the same situation we were
in back then. In this first part, we give an overview of existing literature
related to SCM specific metrics.
CM Body of Knowledge
The first place one
thinks to go for information - after Google - is to the Configuration
Management Body of Knowledge [CMBoK]. Especially for a topic that sounds as
exotic as metrics for SCM processes and tools.
The duality of the
relation between SCM and metrics is reflected in their description of the two
driving forces for the collection of data for metrics: information the project
manager requires (which is what we consider to be the metrics SCM provides),
and information the SCM team requires in order to monitor their own efforts
(which is exactly what we are interested in). The CMBoK does not list many
examples, but we would like to mention: cycle promotions as an indicator of
workload, and average process time to monitor workflow effectiveness.
CM Books
In the context of books
on configuration management, SCM metrics (if mentioned at all) are taken to
mean that SCM through the status accounting activity provides the rest of the
organization with numbers through regularly producing status accounting
reports. So if one really wants to get any SCM specific metrics out of that,
you have to go into the descriptions and dig out the parts (if any) of the
status accounting reports that are targeted towards the SCM team itself.
The first explicit
mentioning of metrics and SCM, that we have found, is Fletcher Buckley
[Buckley, 1993], who in his book on configuration management has a section on
metrics - sub-divided in configuration management metrics and project metrics.
He primarily describes metrics as a bonus or by-product that project management
can use in its work, but also hints at the fact that they can provide
indicators of where improvements can be made to configuration management. To
us, some of his examples of metrics seem to belong to the former category
(lines of code, number of revisions to files); others are more in the spirit of
our understanding (number of changes processed, number of open and closed
problem reports).
More recent books on
configuration management tend to include explicit mentioning of metrics, though
there is still a bias towards SCM as a supplier of metrics and not so much as a
consumer. Alexis Leon [Leon,
2005] mentions metrics in passing and list a number of examples, most of which
look at as meant for others than the SCM team itself. However, there are some
that we find interesting (average time to resolve a change request, percentage
of change requests that conform, difference between estimated and actual value
for activities).
Mario Moreira [Moreira,
2004] splits up his treatment of configuration management into three levels
(organizational, application and project). For two of these levels he mentions
how metrics can and should be used. At the organizational level there should be
established measures of success for each of the SCM objectives (implicitly
stating that these objectives should be made explicit), and he mentions the
percentage of configuration items in the repository (which should ideally be
100%) as one such measure. At the application level he mentions examples that
we tend to consider as provisions for others (percentage of files changed), but
also some that we can relate to in our day-to-day business in the SCM team
(build times and success rate). Moreira does not explicitly treat SCM metrics
for the project level. It could be because he finds that they follow implicitly
from the -sometimes rather detailed - SCM tasks he establishes for the project
level.
A particular problem in
trying to identify SCM specific metrics, as pointed out by Jessica Keyes
[Keyes, 2004], is that many SCM activities are performed in interaction with
(or even only performed by) other people. She believes that it is not the
efficiency of the SCM activities, per se, that is important, but rather their result
in contributing to the overall project objectives. We believe that she is
right, but would still very much like to pursue our quest for metrics that are
specific for SCM activities and tasks, and leave it to others to measure the
compound effects.
CM Journal
The March issue of the
CM Journal was dedicated to SCM and metrics. We have reviewed these papers
[Appleton et al, 2007], [Wagner, 2007], [Kock, 2007], [Weatherall, 2007] and
[Farah, 2007] in a previous paper [Bendix et al, 2007], so here we will only
repeat that if you are interested in SCM and metrics, whether it is metrics for
SCM or SCM providing metrics, then these papers are very inspiring sources of
information.
In a previous paper
[Farah, 2004], Joe Farah explains that metrics are very useful for tuning of
processes and forecasting, and he gives some general advice on how to make
metrics work. He gives an extensive list of his favourite SCM metrics, many of
which are really interesting from the point of view of an SCM team. His
examples are not all SCM metrics in our strict sense, but we would like to
mention just a few of those we found interesting: SCM repository growth rate,
revisions/branches per file, times to query history / retrieve files / create
baselines and so on, and build preparation time. In his paper you will find
many more, plus an explanation of what he uses each metric for.
Whereas, Farah has many
"technical" metrics, Henrik Jönsson [Jönsson, 2004] tends to have a more
"administrative" bias in his approach to metrics for SCM. He focuses on the
processes for handling change requests and shows both a number of specific
metrics and how they can be visualized as graphs for easy understanding.
Together with each metric it is explained how to use it, how it can be
visualized and in some cases also how it can be combined with other metrics to
give a more complete picture.
Other sources
Searching for SCM
specific metrics information on Google gives very little. There are many hits,
but when you have filtered out hits that are about Supply Chain Management, you
get very little.
A potential source for
information could be white papers from tool vendors. It seems like there is no
limitations to what they promise you, if you just buy their tool. In our
opinion their figures have little credibility, also because there is no
explicit mentioning of what exactly they compare with. Also many tool vendor
white papers (none mentioned, none "forgotten") deal with return-on-investment,
which is not really what we are looking for. However, we could still use bits
and pieces from their calculations for inspiration for single metrics that
could be valuable, like number of merged files, cost of browsing version
history, and time to build.
Similarly, the work of
Borracci [Borracci, 2005] is focused on a
return-on-investment model and as such is of no direct interest to us. However,
even though he has difficulties with his model because he cannot put financial
value to some of his parameters, we can still use the parameters as metrics
because we can actually measure them (like time-to-market). On the other hand,
the study reported by Larsen and Roald [Larsen
et al, 1998] seems more difficult to use for metrics as their parameters mostly
measure compound effects, which makes it difficult to say what is due to
improvement of SCM processes.
Summary
Navigating the
information highway wilderness, this is what we found. If you need to take a
similar road, we can suggest you a map to start from. If someone knows of other
sources of information, please let us/the community know.
As we stated in the
beginning of this paper, your mileage may vary. You may be in an SCM context
that is different from ours and have different needs. Therefore, we encourage
interested people to read the referenced literature for more details on metrics
and their relation to and use for SCM.
Even though literature
is not abundant with examples of SCM metrics, we did find a few metrics that we
were interested in trying out - and some more that served as good inspiration
for coming up with others. More about this in part two, where we will describe
a couple of case studies that led to some metrics for SCM processes that other
people may/may not be able to use.
Lars Bendix, Lund Institute of Technology, Sweden.
Dag Ehnbom, Ulf Steen, ABB, Malmö, Sweden.
References
[Appleton et al, 2007]:
Brad Appleton, Robert Cowham, Steve Berczuk: Lean-based Metrics for Agile CM Environments, CM Journal, March
2007.
[Bendix et al, 2007]:
Lars Bendix, Dag Ehnbom, Ulf Steen: SCM
metrics - a response, CM Journal, May 2007.
[Borracci, 2005]: Lorenzo Borracci: A Return on Investment Model for Software Configuration
Management, Master's Thesis, Department of Computer Science, Lund Institute
of Technology, Sweden,
May 2005.
[Buckley, 1993]: Fletcher
Buckley: Implementing Configuration
Management - Hardware, Software, and Firmware, IEEE Computer Society Press,
1993.
[CMBoK]: Configuration Management Body of Knowledge,
http://www.cmbok.com
[Farah, 2004]: Joe Farah:
Metrics and Process Maturity, CM
Journal, December 2004.
[Farah, 2007]: Joe
Farah: CM - The Next Generation of
Metrics, CM Journal, March 2007.
[Jönsson, 2004]: Henrik
Jönsson: Graphs for Change Requests,
CM Journal, December 2004.
[Keyes, 2004]: Jessica
Keyes: Software Configuration Management,
Auerbach Publications, 2004.
[Koch, 2007]: Alan S. Koch: Is
Your Metrics Database Write-Only?, CM Journal, March 2007.
[Larsen et al, 1998]: Jens-Otto Larsen,
Helge M. Roald: Introducing ClearCase as
a Process Improvement Experiment, in proceedings of the SCM-8 Symposium, Brussels, Belgium,
1998.
[Leon, 2005]: Alexis
Leon: Software Configuration Management
Handbook, second edition, Artech House, 2005.
[Moreira, 2004]: Mario
Moreira: Software Configuration
Management Implementation Roadmap, John Wiley & Sons, 2004.
[Wagner, 2007]: Randy Wagner: Metric
Mania, CM Journal, March 2007.
[Weatherall, 2007]: Ben Weatherall: Metrics
for Fun and Profit, CM Journal, March 2007.
Trackback(0)
Comments 
Write comment
 |
CM Journal Articles 
TrackBack URI for this entry
Subscribe to this comment's feed