This month we would like to address briefly the term DevOps which is steadily gaining traction and currency, particularly in the world of web apps. We want to give an introduction and some pointers to resources and further reading.

Although a relatively recent term, many of the principles have been around for a long time. This is similar to Agile methods – in some ways a repackaging of existing principles. Indeed Robert’s first article for this column, back in 2004,  addressed some relevant aspects.

Until it is released (in the hands of end-users), new developments are “inventory”, to use the lean software term – you have invested time, effort and resources in making changes, but no value is realised for the organisation until any development is actually released and being used by the end-users.

So What is DevOps?
The term DevOps suggests the combining of development and operations together and applying some of the skills of development to the issues of deployment. It is also about looking to break down many of the barriers that have grown up between these two parts of many organisations.

Is it New?
As mentioned at the start, we believe that the principles have been around in various forms for many years.

For those of us who do internal software tool development as part of their job (either from scratch, or to customize and configure), DevOps has always been a part of the reality that we have lived with day-in and day-out for at least a decade or two. Just as Agile Development calls for generalizing specialists across functions, we have had to do the same across the enterprise:

• We’ve had to be experts not merely (nor even primarily) with tool administration/usage but also with the organizations' business-processes and most (if not all) of the enteprise application architecture
• We’ve had to work across not merely the software development lifecycle of these tools, but also the deployment lifecycle for installing, upgrading and supporting the tools
• We’ve had to do far more than mere administrative support too, including new development to any existing internally developed customizations and their configurations.

We've had to be process experts, application development experts, application deployment exports, tool support and administration experts, etc., for most all the business-activities and their processes surrounding those tools.

Is it Elitist?
One of the dangers is that DevOps is perceived as elitist and as relevant to only a few organisations. See Stephen Nelson-Smith’s article which addresses some of these concerns. Does it mean that all sysadmins must be able to program (or at least script actions)? To some people the answer is yes – a new breed of sysadmin-coders will sweep all before them. I believe that is not strictly necessary – although it can be an advantage. The key principle for me is that the operations team needs to consider how to improve the automation of their tasks, and if they can't automate it easily themselves, to seek help in achieving this automation (and the business should allocate necessary resources to doing this).

It is Just a Fad?
Another danger is that of being perceived as a trendy fad – and thus something that can safely be ignored by the vast majority!

The underlying principles being espoused under the term DevOps have been around for a long time and successful organisations have been implementing them well for some time.

The fact that they are wrapped up in a catchy title is good marketing 

Let’s look at some of the issues and principles involved.

Global vs Local Optimizations
Too many companies still have local optimisations rather than optimizations across the complete lifecycle – their development may utilize Agile methods such as TDD, and seem dramatically improved from previous efforts, and yet releases happen infrequently and are not always successful. It doesn’t matter how good your development is if there is a large stumbling block later in the life cycle.

Improving Communication
There is often a lack of communication between development and operations. Each department is often incentivised very differently – development gets rewarded for making changes, operations gets rewarded for lack of outages (which often means resisting change). It is important to break down the “Great Walls of Ire” as Brad memorably put it!

This means improving global communication, and thus co-ordination. Make sure that new releases are not a surprise to operations – sprung at the last minute.

As suggested in a previous column (CM as Communication and Coordination Enabler) some low tech approaches can have very good results.

What are the Constraints for Dealing with Releases
Even if the software team doing the development and support of the tool(s) has the ability to release & deliver frequently, the organization (enterprise) may not have the capability to assimilate release as frequently:

• Some deployments/upgrades may require downtime of business-critical systems or unavailability of business-critical repositories and databases, and the business simply cant tolerate that kind of disruption as frequently or during crunch-times of any particular business-product that uses the tools
• Some software changes go hand-in-hand with some corresponding process-change and/or organizational change to the business, and the customer organizations cant begin utilizing and realizing the value of your latest software/tool changes until they have made the necessary organization & process changes (which means many of us also have to become organizational change agent/experts in addition to process & tool experts)
• Any one of many other existing business-critical systems or IT assets which are needed for the deployment of your tool (or its users) may have critical dependencies or "freeze periods" making it difficult to deploy new releases of your own tools as frequently as desired.

By improving your communications you become aware of and deal with these issues.

Reducing Risk
Big bang releases are typically much riskier than smaller more frequent releases. If too much has changed as part of a release then the cause of any problem can be difficult to determine. In such situations you may have to roll-back the complete release, including lots of totally innocent and very useful changes!

Don’t Forget Testing and Environment Management
Lurking between development and operations is usually a testing and integration phase. Changes are tested in pre-production environments which supposedly are a copy of production, but in many cases are subtly or no-so-subtly wrong.

I have been working with one client recently who recognise the problems they have in this area. Testing is a large bottleneck for them, and this is mainly due to poor deployment and environment management. It typically takes hours and sometimes days to get an environment into a situation in which they are actually able to test the new change reliably. With multiple testers being involved, conservative cost calculations showed 6 figure sums being wasted yearly.

In this case we have started with an environment audit tool to be able to show what is in the current environment and how it differs from what is in production. It is then a relatively small step to being able to automatically deploy a new environment directly from their repository.

Designing for Release
You need to consider the requirements for release at the start of your development: the use of some new piece of technology may ease development significantly but can cause major problems to the new functionality being deployed. These days this is a lot easier for many companies whose business involves customer facing web sites. The servers and technologies involved are rather more homogenous and subject to central control – you only have to update your server(s) to have all customers using the new version. Life can be considerably harder where you are releasing products, coping with the deployment of fat clients and multiple client operating systems, and also having to support multiple versions of your system at the same time.

Increasing Automation
This means automation throughout the lifecycle. Agile practices such as continuous integration are now commonplace, but continuous deployment is much less so.

However, some companies are achieving multiple successful deployments of new functionality per day, e.g. John Allspaw’s presentation: "10+ Deploys Per Day: Dev and Ops Cooperation at Flickr".

The key here is having the right mind set to seek out opportunities for automation and then to actually implement them – whether through acquiring tools, or by developing scripts or local tools.

What About ITIL?
Many organisations have reaped large benefits from implementing ITIL practices within their service and support teams. However, the development side of the organisation has often not been included in such initiatives. Thus there is a still often a “wall” over which new releases are thrown. Once things are in production, the operations team may very effective at supporting it – but there is still that hurdle to get over.

There is nothing in ITIL which says this is the way it must be. Indeed, ITIL V3 takes a more holistic view of the whole lifecycle starting with service design. The concept of an application service then allows organisations to wrap up application development as part of the overall process – adding some useful checks and balances and improving communications and co-ordination. The Service Transition book covers the transition into production - but while looking at how to control it does not mean that it has to be slow or a burden - just well managed!

By taking a DevOps mindset, we are not asking operations to give up control – we are working with them to improve the overall process, and typically achieve smoother and faster releases with much less friction. The underlying principles are still the same.

How to Implement DevOps
If you want to implement this, how do you go about it? Can you hire a “DevOp” and expect all your problems to go away?!

The key starting point has to be around communication – ensuring that your development and operations departments are working together closely on a daily basis.

Look at what is causing deployment problems and releases to fail. Ensure that deployment considerations are factored in very early in the development cycle. Look at automation possibilities. Make sure you have a well managed common set of tools including your source repository.

Consider the throughput rate you are achieving – the rate at which changes are actually being deployed – and how long on average that is. Measure “inventory” – work-in-progress that has not been deployed – and seek to reduce it.

If your software architecture makes frequent deployment difficult, then seek to change it as a high priority (treat this as a major “code smell”) or item of technical debt. This can be a major issue with legacy code and tools, and may take significant time to fix.

Vendors and Tools
There are a number of companies and also open source communities which provide tools and methods :

• Puppet Labs’ Puppet
• Opscode’s Chef – open source “recipes”

Commercial vendors include:

• XebiaLabs
• Nolio (also included as part of Serena’s offering)

One of the advantages of the open source offerings is the way it encourages thinking and sharing within the wider community. People can get actively involved and engaged which is a major benefit for all (not to say that the commercial tools are bad!).

It is also important to realise that a tool will not solve the problems, or necessarily make a huge difference by itself. It is the mindset and practices around the use of that tool that make the most difference.

Summary
This has been perhaps a whistle stop tour of some of the issues involved – many points have just briefly been touched upon above! (Leaving food for further columns perhaps...)

DevOps takes the agile notion of cross-functional self-organizing teams of generalizing specialists across disciplines and lifecycles and applies that directly to the IT and process architecture of the enterprise itself, forcing those of us who are CM/ALM professionals be experts in not only the processes and tools/technology, but in enterprise application development & deployment, and in organizational and process change,

We need to learn and become adept at not only applying CM/ALM to agile projects, but also at applying Lean/Agile principles and techniques to CM & ALM. That includes figuring out what it means to apply TDD, refactoring and continuous integration, or identification of "smells" and "technical debt" for processes and documentation (and even organizations) as well as to the development and deployment of the tools & processes we develop, deploy and support.

The rise of DevOps has many things to commend it – and many organisations would do well to take the ideas on board. It isn’t just a fad, and it is here to stay. Like most things it is not a magic bullet but if you embrace the key ideas in a pragmatic and sensible manner, you will get real value from them.

So don’t oversell it within your organisation, but instead use it as part of your arsenal for improving your process and making your own job more satisfying and effective.

Resources and Further Reading
As well as the links referenced above, see:

• Damon Edwards - What is DevOps? - http://dev2ops.org/blog/2010/2/22/what-is-devops.html
  
• Anti-pattern: The release vehicle - http://www.rendell.org/pebble/software/2010/02/09/1265756844985.html
  
• Deployment is just a part of dev/ops cooperation, not the whole thing - http://www.kitchensoap.com/2009/12/12/devops-cooperation-doesnt-just-happen-with-deployment/
  
• 20 DevOps guys you should follow by Matthias Marschall -  http://www.agileweboperations.com/20-devops-guys/
  
• Conference: http://www.devopsdays.org/
  
• There is also a recent book "Continuous Delivery" and its corresponding website http://continuousdelivery.com/
  

Trackback(0)

TrackBack URI for this entry

Comments (1)

Subscribe to this comment's feed

Eric Minick said:

... Thanks for this article. This is one of the more coherent explanations of DevOps and where it fits that I've seen. In your title, you allude to Agile, which I think is actually a pretty key driver. To a large degree, it has been the role of Agile (and lean) that's driven the rapid rate of releases that has put stress on the IT / Ops side of the house to release faster. A painful, manual process works fine if you release once or twice a year. It works less well as the release rate increases. At Urbancode, we've been convinced of the benefits of co-operation between the development, test and release teams for quite a while, and built AnthillPro to facilitate automation from software build through production release. Deployments to the early test environments should facilitate good testing and rapid feedback, but they should also be a test of the deployment process itself to reduce deployment risk. Prior to the term "DevOps" catching on, we called this end to end cooperation "Enterprise Continuous Integration" as it took the automation principals common in Dev and applied them across the enterprise into test and operations. Our cheat sheet to review your ECI / DevOps maturity is up on the website: http://www.anthillpro.com/html...se-ci.html as is a matching white paper.

report abuse vote down vote up

October 21, 2010
Votes: +2

Write comment