Did you forget your: Username / Password ?

Microsoft

TechWell

Home

CM Journal Articles

Behaviorally Speaking - Change Control or Control of Changes

Written by Bob Aiello

Monday, 22 June 2009 10:09

Change Control is a critical CM function that is often misunderstood and many technology professionals do not fully comprehend when to use Change Control while even fewer really grasp how to implement an effective and efficient Change Control function. Done right, Change Control can be the basis for all of your Software (and Systems) Configuration and Release Management practices. However, the task of determining exactly "what" is going to be controlled and "how" is not easy to accomplish. Read on if you would like to know how to stand-up an effective and efficient Change Control function - to support all of your Configuration Management requirements.

I have implemented Change Control in many organizations. In one large bank, I mentored a Change Control Board to coordinate modifications to a codebase that was shared across multiple applications (on multiple continents). In other organizations we setup Change Control to manage the promotion of systems from QA to Production. In other words, Change Control can be used to manage the future modifications to a system (known as "a priori change control") or Change Control can simply be the gatekeeper to coordinate when a release will be promoted to Production.

Right-sizing - Light or Heavy?
I have seen financial services firms that allowed their Project Managers and Development Leads to handle the coordination of changes to an application (with a very light "implied" process) and I have seen defense contractors that had a hierarchy of Change Control Boards to manage every aspect of change to the application. The defense contractor was working on a system that could potentially cause planes to literally fall from the sky so the extra control of changes was very much appropriate and necessary. Change Control can be your most critical CM function or an amazing waste of time and resources. In all cases you need to "right-size" your Change Control to your requirements.

Requirements for Change Control
The first step is to determine what you need to Control through your Change Control function. Controlling the release of code to QA or Production is often the first step. The most basic Change Control function will establish a group (usually called the Change Control Board or CCB) to manage the proposed changes that might impact other applications and resources. But Change Control can do a lot more. You can use Change Control to review and manage all of the CM related functions that can impact your project.

The Change Control Board should also ask to review CM plans, test plans, interface dependencies and especially the steps that should be taken proactively to manage risks that are implicit in any real world Change Control effort. The CCB also confirms that all of the stakeholders have been made aware of the pending changes including potential impact to other systems and accepted inherent risk. An example of this is when there is no alternative, but to test the pending changes in the Production Environment.

Testing in Production
It's pretty common for the QA environment to be less than adequate for testing all of the scenarios that may occur in many critical systems. I have often seen situations where changes just simply had to be tested in production. This may sound like "bad" process - but the truth is that this is often just a matter of making the most pragmatic business choice because creating a QA environment that really matches production is often impossible or at least impractical at best. The change control function should uncover this "risk" and then determine the proper "mitigating controls" to be established, in response. In my world that means that you carefully establish solid practices to promote a release and fallback if necessary. Then you determine the time needed for testing and that gives you the required window for your release.

1 Hour to Promote the Release to Production
2 Hours to test in Production
1 Hour to fallback - if necessary
____________________________________
Total window in Production = 4 hours

This is a good example of how a Change Control Board can ascertain and deal with an inherent risk that occurs all the time in real world scenarios. The CCB is responsible for communicating this risk to all of the stakeholders (or at least confirming that the communication has been done) and then overseeing the execution of this Release.

A priori Change Control
Too often developers are allowed to make changes at will and there is little or no tracking of whether or not their changes are warranted and an efficient use of time and resources. In "a priori" Change Control, all changes are reviewed before they are made and then authorized or denied. In addition the work to make the change is assigned to a specific resource. I have worked in many organizations where there was essentially no "a priori" change control and the Project Manager loosely coordinated the work to be done and assigned to specific resources. I have also seen a lot mistakes made because there was a lack of coordination. The Change Control Board also does a lot of other work as well.

Are we ready?
The Change Control Board needs to ascertain whether or not the Release is actually ready to be promoted to Production (or QA for that matter). This means that the CCB should review the CM Plans, Test and QA plans, Review Risks, Interface Dependencies, and especially confirm that all of the stakeholders are aware of the pending changes. The CCB needs to make sure that the release is ready for promotion to Production (or QA for that matter).

CM Planning
CM Plans should be reviewed and any gaps or deficiencies identified. The CCB can reject the release if all entry/exit criteria have not been met or they approve the release with the agreement that future releases will managed better and improved.

Automating the Release and Deployment
The CCB should understand exactly how the Release will be packaged and deployed to production. This review might uncover steps that are being done in a manual (and perhaps error-prone) way. The CCB should require that the Release Packaging and Deployment be improved to avoid mistakes and improve productivity.

Environment Dependencies
All too often the environment dependencies between Development, QA and Production are not well understood. The CCB needs to recognize this as a Risk and plan identifying and controlling all environment dependencies.

Compliance with SOX and/or SAS 70
I have seen organizations fail their SOX and SAS-70 audits because they lacked the controls to trace back exactly when a release was done and by whom. Change Control can help to manage all of the requirements to meet SOX, SAS-70 or other Compliance Requirements as needed.

Plan for Emergencies
There is always an exception to every rule. It's important to be prepared for Emergency Releases that just cannot go through the normal channels. I suggest that you require a very senior manager (e.g. CTO) to sign-off on all emergency releases as a deterrent to using the Emergency Release process to simply avoid participating in Change Control. There should also be an "after-incident" meeting to confirm that the Emergency Change was indeed required (and not simply an attempt to circumvent the Change Control process).

Improving the process
Even the best processes can be improved. It is always a great idea to plan for reviewing and improving your Change Control processes on a regular basis. I usually suggest that development managers accept that there will be one mistake - however the same mistake should never occur twice. A mistake is usually your best friend for determining how to improve your Change Control process in a practical and reasonable way (assuming of course that a mistake does not aim a missile at anyone's bedroom window).

Conclusion
Change Control can drive your entire CM process from start to finish. You should keep your processes as Lean as possible and not introduce extra steps - unless they are absolutely necessary. Change Control should be used to review all of the entry/exit criteria for your release and lead your organization to Excellent CM Practices!

Bob Aiello is the Editor-in-Chief for CM Crossroads and a Software Engineer specializing in Software Process Improvement including Software Configuration and Release Management. Mr. Aiello has over 25 years experience as a technical manager in several top NYC Financial Services firms where he had had company-wide responsibility for CM, often providing hands-on technical support for enterprise Source Code Management tools, SOX/Cobit compliance, build engineering, continuous integration and automated application deployment. Bob is the Vice Chair of the IEEE 828 Standards working group (CM Planning) and is a member of the IEEE Software and Systems Engineering Standards Committee (S2ESC) Management Board. He is a long standing member of the Steering Committee of the NYC Software Process Improvement Network (CitySPIN), where he serves as the chair of the CM SIG. Mr. Aiello holds a Masters in Industrial Psychology from NYU and a B.S. in Computer Science and Math from Hofstra University. You may contact Mr. Aiello at raiello@acm.org or link with him at http://www.linkedin.com/in/bobaiello

Set as favorite

Bookmark

Email this