Amid all the excitement of DevOps, continuous delivery, and the magic of single-push-button deploys, some folks have forgotten the prerequisites. You cannot implement continuous anything without effective configuration management. This article will help you reassess where you are and ensure that you have the basic building blocks in place to ensure success.
Amid all the excitement of DevOps, continuous delivery, and the magic of single-push-button deploys, it seems that some folks have forgotten the prerequisites. The fact is that you cannot implement continuous anything without having effective configuration management.
This article will help you reassess where you are and ensure that you have the basic building blocks in place to ensure your success.
Effective configuration management starts with source code management. This is an absolute requirement for success, but sadly, I have seen many teams struggle with unreliable tools and even losing code. Version-control systems (VCS) are a must-have, and you need to know how to use them effectively. This includes being able to label code baselines and create branches to manage variants in the code, such as bug fixes. Regardless of which tool you are using, the most important consideration is ensuring that you provide training to your team that explains how to use your VCS, as well as essential procedures such as tagging and branching.
Build engineering must be fully automated and is also an absolute prerequisite to continuous integration, delivery, and deployment. You should always be building code from an identified baseline and embedding an immutable version ID into each configuration item. Having a repeatable process to build your code enables your team to support bug fixes without running the risk of code regressing due to the wrong version of some compile or runtime dependency.
This step is where you verify that the code baseline can be built using your existing procedures. Many companies fail to implement effective build engineering. I find that the key to success is ensuring that someone else besides the developer can successfully build the code, usually a build engineer.
Environment management includes understanding and monitoring all environment runtime dependencies. Many companies struggle with environment management, and it is a very common cause of system outages. Too often developers do not communicate runtime dependencies, and this information is lost as they move on to their next project. You need to ascertain and manage environment dependencies from the very beginning of the lifecycle, or else you will be discovering them one by one when your system goes down.
Release engineering helps provide a framework for packaging releases. You should have a manifest in your release package listing all its contents. This is often called a bill of materials (BOM). Your release container should also be signed using asymmetric cryptography with a public/private key pair, which means you can be certain that the package has come from a trusted source.
I always include a script to recalculate the BOM directly from the runtime configuration items. I call this my release map, and it has saved me many times by giving me a way to compare what I shipped to what is actually found in the production environment. This is how I discover changes made by systems administrators who may mean well but whose actions could cause the system to crash.
Deployment engineering focuses on automating the promotion of the release package to the target environment. Procedures to fully verify that the deployment was successful are known as a physical configuration audit. The most common problems I see when assessing CM best practices is that most companies cannot prove they have the right code in production, and they also cannot detect unauthorized changes, which creates a tremendous amount of risk. Your procedures must be completely reliable, and you must have both a way to verify that the correct code was deployed and an automated procedure to detect unauthorized changes due to either malicious intent or human error. You also must have procedures to back out a deployment if necessary.
All these practices should be driven by an effective change management process that helps identify technical risk. Risk is not inherently bad, but you must be able to understand and mitigate any potential challenges that might occur. Don’t forget to manage changes to your process, as well—this is usually handled by the software engineering process group (SEPG).
Getting started with CM best practices should always begin with an assessment of your current practices and a pragmatic plan for process improvement. I tell folks to start with a couple of easy improvements to help the team realize that change is very doable, effectively improving their culture and sense of effectiveness. The SEPG takes responsibility for communicating all process improvement initiatives, including establishing IT controls to meet audit and regulatory requirements.
DevOps can certainly help establish good CM by improving effective communication and collaboration. Continuous delivery and deployment cannot happen without configuration management. When we are asked to help teams establish continuous delivery, our assessment often finds major gaps in configuration management best practices that must be addressed before the magic bells and whistles get implemented. DevOps and continuous delivery may be the flavor of the day, but you still must have the ingredients in place to bake the pie.
Good stuff Bob, thanks. I agree that CM must be executed across the entire SDLC :-). I teach all my students that with full coverge of the CM function including the history, 4 concepts (ID, Control, Status, V&V), and how to work with other functional areas.
Bob, I am just in the middle of reading your CM book and I find it really great value! Thanks!
I noticed you are using the term "Environment Management" in this post, but I think you were actually meaning "Environment Configuration" (as in your book).
Thought the two functional areas may actually work on the same concepts I think the objectives are different (i.e. configuration of an app in different environments VS management of specific envs).
Please let me know your ideas about this.
thank you for your very perceptive comment. There are a few places in the book, where I am starting to rethink what I wrote based upon the awesome feedback that I have received from my colleagues like you. (I believe that writing is a team sport). Release Management vs coordination, Change Management vs control and Environment Management vs configuration are three areas where I think that I need to do a better job of distinguishing between the management function and the in the trenches control activities. Environment configuration is the activity where we modify environment dependencies in a controlled and traceable way. Environment management is broader and includes monitoring and related communication. In this post, I meant the broader function, but your point is well taken nonetheless. Part of the challenge is that I want to teach classic terminology used in IEEE/ISO/ANSI standards while also explaining the modern methods in use today. Thanks again for your comment. Hey when are you going to submit an article and share your best practices?? (Hint - I will help you :-)
Bob, thanks for the explanation!
SCM as an established and well defined discipline is quite new to me (3 weeks!) ... by now I can only share plenty of unbelievably bad practices, but no best one yet :-)
Just to mention ... most common "unbelievably" bad practice from the early 2000: not having a process in place, ignoring it, and supposedly going live tomorrow!