| 
The Unchangeable Rules of Software Change (and what to do about it) PDF Print E-mail
Tuesday, 14 February 2006

This month we examine the fundamental truths about project and requirements changes: What can we control? What is beyond our control? What are some of the common perils and pitfalls of change-control and iterative development? We discuss how to avoid many of these common pitfalls without creating new ones along the way, and provide a wealth of resources for first-timers to iterative development.


Pitfalls of Controlling Change

I often come across teams in the early stages of learning to deal with changing requirements. They typically run into two pitfalls in the following order: 

Pitfall #1: No scope change-management

The very first pitfall they fall into is not having any kind of change-management for the project/requirements. They allow any and all changes in a very reactive fashion without thinking about trying to renegotiate the scope, schedule, or resources for the project/release.

Pitfall #2: Preventing scope changes

The very next time around, they run into the second pitfall: they overcompensate for being burned/bitten by the first pitfall by trying to prevent any and all changes to the scope of their next product release.

They keep insisting that the fundamental problem is they don't have completely stable detailed requirements. If the requirements were detailed enough up-front, they think their estimates would be more accurate; and if only the requirements would stop changing before a single line of code is written, then the schedule wouldn't keep fluctuating so much. It's those darn users/customers/analysts that don't know what exactly they want at the outset. It's all their fault!

This leads to another pitfall:

Pitfall #3: Requirements Analysis Paralysis

They spend inordinately more and more effort analyzing and engineering requirements and specifying them in far too much detail before attempting to validate their results with tangible feedback.

The irony is that the more they try to get stable detailed requirements up-front, the more the schedule becomes protracted: first to get all the gory-details analyzed and specified up-front; and second because so many of the gory-details were either incorrect or still not detailed enough. It becomes this vicious cycle of trying to prevent change with more up-front detail, and yet things keep getting worse instead of better. If the cycle goes unrecognized and unchecked, it can turn into a "holy grail" crusade in search of the perfect and unchanging requirements.

Biology, Stability, Humanity, Uncertainty, and Inevitability

When I encounter a group in the midst of these pitfalls, the first thing I commonly do here is explain the following:

There is a very fancy technical term that biologists use to describe completely stable systems. This highly sophisticated technical term is the word "DEAD!"

I then try to explain that we meager humans (including ourselves and our customers) are imperfect, and we have imperfect and incomplete knowledge: We don't know things, and we don't know that we don't know things, and we don't know how to find out many of those things earlier.

At this point I deem it appropriate to introduce them to Phil Armour's wonderfully insightful work, particularly his description of the Five Orders of Ignorance and how Software is not a Product, and that software development is a therefore a knowledge creation activity which involves reducing our ignorance over time though learning and discovery about the domain (our requirements) and ourselves (our process, culture, and skill/capabilities).

Then I wax philosophically and authoritatively explain the unalterable truth about the fundamental nature of software development and changing requirements. I call it my "tried and true, battle-proven and industry-acknowledged, unchangeable rules of software change ...

The Unchangeable Rules of Software Change

Rule #0: Change is Inevitable!

The Requirements/Plans ARE going to change!

Rule #1: Resistance is Futile!

There isn't a darn thing you can do to prevent Rule #0.

Rule #2: Change is like Quicksand -- Fighting it only makes it worse!

The more you try to deny and defy rule #1 by attempting to prevent rule #0, the worse things will get.

Rule #3: Change is like Quicksilver -- Tightening your grip makes it slip out of control

The more you try to deny and defy rule #2 by attempting to precisely predict, or rigidly control change, the more erratic and onerous the result will be.

Rule #4: Embrace change to control change

The more flexible and adaptive you are at accommodating change, the more control you will have over your outcomes.


These rules are very hard to accept! Often, they are accepted only one at a time, after much frustration and over a period of hard learning and reflection. The stages one passes through while learning these rules bears more than a passing resemblance to "the five stages of grief."

Traditional Change-Control

When faced with a requested or impending change that impacts the scope of the project, generally accepted best-practice is to utilize a Change-Control Board (CCB) consisting of your project's key stakeholders:

  • Assess the impacts of the change make those known to your stakeholders
  • Present the viable options the risks associated with each one (possibly accompanied by a recommendation)
  • Let them decide which option to "exercise" based on informed consent, making sure they are fully cognizant of the benefits, consequences, and risks of the chosen option.

Still, the idea of giving up control to another group every time a change arises isn't always palatable. And of course there is also the fear that every a scope change that becomes visible will make them look bad, as if they don't really know what they are doing.

Recently I was talking to a group that was struggling with rule #2. They thought if they could only do even more detailed specification up-front (they already do a reasonable amount of up-front detail), that it would somehow eliminate problems with estimation accuracy, which in turn would alleviate problems with "conformance to plan" and prevent the details from being discovered later (because they would instead get them all "right" up-front).

Despite having plenty of historical evidence/data in this particular product to support the "inescapable truth" laid out by these rules, there still seemed to be that desire to cling to the illusion of control that we can somehow prevent such changes if only we spend more time & effort getting a more complete, perfect & detailed specification up-front near the very beginning.

Software Evolution & Feedback

As it so happens, I don't always come across as authoritatively sage as I might hope. So I looked for some materials from other folks more sagacious and experienced than myself. The first that came to mind was Lehman and Belady's Laws of Software Evolution (ca.1971) and the subsequent work on the FEAST Projects. Those are primarily about software maintenance however, and refer to the inevitability of needing to maintain and evolve an already released piece of software. They don't speak quite as strongly to the inevitability of changes via feedback, discovery and learning during the course of any non-trivial software development effort.

So I searched a bit more and came across a handful of particularly interesting (to me) references that are quite recent:

The fundamentally difficult hurdle managers need to come across is that when you do not know enough to produce an accurate estimate, it will not help to do a more fine-granular plan. Rather, you need to do a list of what are the unknowns and risks preventing you from doing a more detailed estimate, and then rapidly do the analysis, design, implementation, testing, or other risk mitigation turning unknowns into knowns. This sounds as something that should not be very difficult to understand, but it really is a big mountain to climb for many people...

Iterative Development to the Rescue?

One of the things the particular team finally figured out it needed was to try and use iterative development as a means of managing change and being more responsive to customer feedback while still controlling scope:

  • One of the ways in which iterative development is most effective at controlling change is that it shortens the time-window of opportunity during which changes can occur!
  • The other is that it shortens the cycle-time of the feedback-loop for obtaining validation of our end-result. The sooner we can see tangible executable and obtain stakeholder feedback, the more we reduce the risk of unmet expectations and the associated cost of rework!

Sounds simple - right? The answer our prayers is clearly iterative development! Of course! (It's all so obvious now!) All we need to do is iterate! That's what so many of these groups often do next. Well, as you might have guessed, their first attempt at iterative development doesn't quite work as they expected. They often run into three more pitfalls: 

Pitfall #4: Insufficient Requirements Analysis

Thinking iterative development the "silver bullet" to their requirements problems, they abandon all or most of the requirements analysis they had been doing, and don't understand the requirements as well as they should before starting to code.

Pitfall #5: No iteration planning

They jump into iterative development in an overly ad-hoc fashion, without a suitable iteration plan for each iteration. They naively assumed "going iterative" would eliminate the need to define specific goals and milestones for each iteration at the inch/pebble level.

Pitfall #6: Too much iteration planning

Once again, on the heels of their previous failure, they go too far to the other extreme and try to apply all the formality and rigor of planning a large project to the planning of each iteration. They burden themselves with an amount of project management and tracking effort that was intended to take place over many months and try to squeeze all of it into a period of a few weeks.

Walking the Tightrope and Avoiding the Pitfalls

I've seen teams that ran into all of these pitfalls as well. The remedy for these three latest pitfalls is not necessarily to fall back into trying to overanalyze and overspecify detailed plans and requirements on a per-iteration basis. Some initial amount of planning and requirements is necessary to know where you are trying to go and what to estimate. The remedy is to stop bouncing back and forth between extremes, and instead of doing all or nothing "up front," do a reasonable initial amount and then incrementally and iteratively flesh out each individual piece, instead of trying to do scores or hundreds of partially completed pieces all at the same time.

Good change and configuration management is supposed to be about facilitating change, not about preventing it. That's what the Agile Manifesto really means when it talks about "Responding to Change over Following a Plan." The point isn't to discard or dismiss plans or planning, it is to deal with and adjust to reality.

As Winston Churchill once said "Plans are of little importance, but planning is essential" -- Dwight Eisenhower said it much the same way: Plans are nothing; planning is everything!" But conforming to reality must takes precedence over conformance to "the plan" (trust the terrain, not the map!). More recently, James Highsmith wrote: "Agile projects are not controlled by conformance to plan but by conformance to business value.... If we accept the notion of constant change and turbulence, then plans are still useful as guides, but not as control mechanisms -- because they tend to punish correct actions."

It may seem counter-intuitive for some, but in the current era of business software development, "Change" is not the enemy -- stagnation is! Mary Poppendieck notes that: "Requirements changes late in the lifecycle are competitive advantage, IF you can act on them!"

How does all this relate to traditional CCBs and how does it change "change control" for agile and iterative development? It means that:

  • Plans and requirements are still necessary to identify, but should be elaborated iteratively in progressively more detail, usually a feature at a time.
  • Regular (e.g., at least weekly) milestones are still important to set and track -- don't forget to do that just because you're working in short, fast increments
  • Incremental integration needs to be even more frequent (and automated) when iterations are short
  • Manage stakeholder expectations with close communication and simple boundaries (short, frequent iterations)
  • If iterations are very short, there is little opportunity to work on anything but the agreed upon features for the current iteration. In this case, CCBs are really just iteration planning meetings with your stakeholders at the beginning of each iteration.
  • CCB's and related decision-makers must be more dynamic, comprised of stakeholders that are readily available and accessible to give rapid response to issues and questions soon after they arise.
  • At the start of each iteration, expectations and priorities are (re)set and (re)calibrated
Iterative Development Resources:

Of course, saying "do iterative development" is one thing. Figuring out how to actually do it for a group in an organization that isn't accustomed to it is another thing entirely. So here is a list of resources on the subject of adopting, planning/managing, and doing iterative software development -- particularly for those coming from a background of phased-sequential (waterfall or "V") model of planning:
Acknowledgements:

Many thanks to Frank Schophuizen, Craig Larman, Ron Jeffries, Stephen Gordon, Scott Ambler, Mary & Tom Poppendieck, Jeff Sutherland, Paul Oldfield, Geoffrey Slinker, and Scott Preece provided lots of useful feedback, suggestions, and resources (as did numerous other contributors on the various "agile-related" YahooGroups forums).


Steve Berczuk develops software applications at Fast Search and Transfer in Boston, MA. He has been developing object-oriented software applications since 1989, often as part of geographically distributed teams. In addition to developing software he helps teams use Software Configuration Management effectively in their development process. Steve is co-author of the book Software Configuration Management Patterns: Effective Teamwork, Practical Integration. He has an M.S. in Operations Research from Stanford University and an S.B. in Electrical Engineering from MIT. You can contact him at steve@berczuk.com.

Brad Appleton is an enterprise SCM/ALM solution architect for a Fortune 100 technology company. He is co-author of Software Configuration Management Patterns: Effective Teamwork, Practical Integration, the "Agile SCM" column in CMCrossroads.com's CM Journal, and a former section editor for The C++ Report. Since 1987, Brad has extensive experience using, developing, and supporting SCM environments for teams of all shapes and sizes. He holds an M.S. in Software Engineering and a B.S. in Computer Science and Mathematics. You can reach Brad by email at brad@bradapp.net

Robert Cowham has been in software development for over 20 years in roles ranging from programming to project management. He is one of that strange breed of developers who enjoys SCM and the whole subject of improving software development processes (a sucker for the quick feedback wins of making a software development team more effective). He is the Chair of the CM Specialist Group of the British Computer Society, has a BSc in Computer Science from Edinburgh University and is a Chartered Engineer (CEng MBCS CITP). You can contact him at rc@vaccaperna.co.uk
Trackback(0)
Comments (2)add comment
feedSubscribe to this comment's feed
6340
Brad Appleton: ...
Responding to Edgars P., I would say plenty has actually been said and that Edgar P's complaint is that nothing new has actually been said.

While that may (or may not) be true, something need not be new or novel to be of value. The entire body of software patterns is a testimony to this. Patterns dont describe new or uncommon problems and solutions, that describe uncommon insight about known recurring problems and solutions. And such insight is sorely needed. There are too many people out there needing to conquer the same common problems before they can even start trying to utilize much of the new, uncommon ones.

Saying that everything can be reduced to PM experience anyway is an obvious overgeneralization that excludes the other key roles in software & systems development who also have these experience in common. This problem and its insights arent exclusive to nor even primarily centered on PM, they are much broader and much more pervasive.

I would even say that the "unchangeable rules of change" listed in the article are far older than software development itself and apply to most forms of behavior/culture change. And yet the feedback thus far from emails and in the blogosphere has overwhelmingly indicated that those rules and this article have deeply resonated with many people's experience and has provided useful guidelines, resources and pitfalls to avoid when attempting iterative development.

Of course your own mileage may vary (as Edgar P's obviously did).
1

April 25, 2007
Votes: +0
0
Edgars P.: ...
Nothing has actually been said. Seems like we everything can be reduced to PM experience anyway.
2

April 24, 2007
Votes: +0

Write comment
smaller | bigger

security image
Write the displayed characters


busy
Last Updated ( Wednesday, 17 October 2007 )
 
< Prev   Next >
[ Back ]
If you already have an account on CM Crossroads, Login Now. If you do not, register using the link below...

NOTE: Once you register you will need to activate your account by clicking the link sent to you by email.
Accurev

Sponsored Links