CM Metrics

Below we list a few simple CM metrics that we have used to support the CM audit process. However, be careful, the real power of metrics comes from the ability to compare them with historical data i.e. trend analysis. Hence the longer you collect the CM data the more useful, valuable and accurate your evaluation becomes.

Formula Metrics Purpose

• TE-SYS= SI/(SI+AI+PI) Testing Efficiency - Ratio of defects caught in System Test
• TE-UAT=AI/(AI+PI) Testing Efficiency - Ratio of defects caught in User Acceptance Test
• PR-SYS=UD/SD Determines Ratio of kits promoted from System Test
• PR-UAT=PD/UD Deternines Ratio of kits promoted from User Acceptance Test
• BR=SB/(SB+FB) Determines Successful Build Ratio
• BD=SB/TBD Determines Build Duration Mean
• RR=SR/(SR+FR) Determines Successful Release Ratio
• RF=(SR+FR)/X Determines Release Frequency
• RD=SR/TRD Determines Release Duration Mean

Key:

• SI = Number of Incident raised in System Testing
• AI = Number of Incidents raised in Acceptance Testing
• PI = Number of Incident raised in a period (e.g. 1 month) in production
• SD = Number of System Test releases
• UD = Number of UAT releases (promotions from System Test)
• PD = Number of Prod releases in period (e.g. 1 month) after original releases
• SB = Total Successful Builds
• FB = Total Failed Builds
• SR = Total Successful Releases
• FR = Total Failed Releases
• TBD = Total Build Duration
• TRD = Total Release Duration
• X = Period (eg 14 days)

Critical Process Assessment Tool -- Configuration Management

Once the necessary level of CM requirements have been decided upon, it may be highly effective to ensure that meaningful metrics are identified for maintenance by the contractor. Metrics should be the key for management visibility and control. Metrics that you identify should be able to provide detailed insight into the CM program, regardless of the level of control to be employed. Typical examples of metrics are:

• Number of on-schedule configuration documentation releases
• Average number of engineering changes per document (by product, by classification, by phase)
• Average engineering change cycle time (by product, by classification)
• Average revisions per engineering change (in-house, after submittal to customer)
• Number of changes (by reason for change)
• Number of variances (by product, by classification, by phase)
• Average variances per delivered unit
• Number of action items per configuration audit (categorized by magnitude)
• Average number of un-incorporated changes per engineering drawing

A good approach to adhere to is to identify metrics expectations as exiting or transition criteria. By ensuring that these metrics are in place, prior to first occurrence of need, will assist in more effective management oversight. Remember, trying to reconstruct what you should have done (or known) initially usually increases program cost substantially.

I recommend that you keep the metrics simple and manageable. As the saying goes, "You cannot manage what is not measured," (or words to that effect).

Start by measuring the status of CM activities and work products (e.g., CCB activities, records required by management, reports prepared for project management, CM audits and reviews, CM plan and revisions, baselines, proposed changes, approved changes, implemented changes, problem/trouble reports, product releases, etc.), by development phase (e.g., initiation or requirements, design, construction or manufacturing, testing, deployment or delivery, whatever).

Check the ACDM Website for CM references. Follow the 'Resources' link to 'Recommended Reading'. Many of these references contain some good ideas for measuring CM activities and work products.

One way I use to measure the cost savings of CM is to measure the average of changes per document per program. A standard cost estimate per change is (US)$1500.00.

This $1500.00 figure came from a Department of Defense guide for standard costs. Phillip Crosby was a Quality guru. A good book of his is "Quality without Tears". You can also review a description of this book in the CM Crossroads Bookstore.

• From Helene Frank (EAB) mailto:Helene.Frank@emw.ericsson.se

What about CMCostEffectiveness?

-- PatrickEgan? - 14 Dec 2002