Rapid iteration in software development means performing short, repeated development cycles based on a set pattern. A two-week development cycle is an example of a rapid iteration. Using iterative software development cycles that incrementally build the system is one of the primary principles of agile software development.

In agile, the primary benefit of rapid iteration is the ability to review and validate user requirements. By using short development cycles, you add the ability to become “agile” and change those requirements in future iterations based on ongoing customer input.

The short development cycle naturally presents a challenge to the quality assurance side of building and maintaining software, as there simply isn't enough time to test everything. Another challenge is that agile teams might not include full-time testers to carry out a comprehensive regimen of tests. Such problems can be resolved with a mix of good design practices and agile testing techniques, which, when combined with a short development cycle, will result in increased product quality, early defect detection, and reduced cost. Various case studies have shown a substantial improvement in quality and customer satisfaction.

In this article, I will explore how rapid iteration can improve software quality through reduced complexity and early defect detection.

Iterative, Incremental Development Challenges
The two terms “iteration” and “increment” are often used together, as in this quote from Ken Schwaber's The Enterprise and Scrum: “Scrum makes workload management possible through iterative, incremental development.” “Iterative” in this context refers to following a pattern of steps in repeated cycles for fixed durations, while “incremental” refers to the actual building of the software application—or, in a broader sense, the entire project, since you can also be building tests, documentation, and so on.

Incremental development is challenging if each programmer is left to his own devices with no set structure, pattern, or common development practice, and the project becomes a series of ad-hoc pieces meshed together only through the heroic, often last-minute actions of the programmers. Complexity greatly increases, and the code becomes harder to decipher as the architecture of the project varies widely based on too many random variables. For example, even simple things like documentation and comments can become difficult (or impossible) to decipher if there is no standard practice for them.

Incremental design therefore has the ability to become disasterous. Change is required if the developers want to keep their jobs. So, how can change be introduced frequently through the rapid, iterative approach without one change trashing another?

The first solution to this problem is for agile developers to use incremental—or, as Martin Fowler prefers, “evolutionary”—design.

Good Design
Agile software methodologies describe incremental design, such as this example from Tips from Scrum and XP from the Trenches: “This means keeping the design simple from the start and continuously improving it, rather than trying to get it all right from the start and then freezing it.”

“Incremental design” simply means designing as you go. This approach upholds agile principles, since valuable time and resources are not wasted up front documenting future changes that may never be implemented anyway or may be unwanted once they are implemented. I recommend Martin Fowler's “Is Design Dead.” Here are some bullet points:

• Using patterns will allow the software to evolve in a common way.
• Using frameworks and resuable components will yield simplicity and flexibility.
• Enable good testing practices.

A rapidly iterative and incremental workflow provides the proper framework for this type of development strategy, as continuous improvement is built into the workflow in a couple different ways— improving not just through coding but through code review/rewrite, process improvements (through introspections), and testing.

Some of the tools in the agile design toolbox include:

• Code Review—Instantaneous in some agile practices, retroactive in others; always a good idea (inherent in some practices like XP)
• Refactoring—Set of techniques designed to ensure the code base evolves in a way that promotes simplicity, reusability, readability, and more
• Modeling—Not used by all agile practitioners but can help communicate design using simple diagrams

It is not necessary for a developer to know every part of a million-line code base if there is a common pattern and framework that every developer uses and new classes, methods, objects, etc. are all created in the same vein. Ensuring that the patterns are continuously followed by everyone is a benefit of code reviews.

Good development and design practices reduces complexity. Again, I refer you to Fowler on the details. In a rapid, iterative cycle, these practices become even more important, since rapid change can lead to increased complexity and reduced quality without good coding and testing practices. But, as I will soon demonstrate, using a rapid iteration development cycle provides an opportunity to increase quality and reduce cost significantly.

Defect-cost Analysis
There is a well-known 2002 study from NIST that claims a $60 billion cost annually for software defects in the US alone. The conclusion of the study is “The path to higher software quality is significantly improved software testing.”

The cost of a software error or defect is different depending on where in the development cycle it appears. A defect discovered by a programmer coding at his desk is far cheaper to correct then a defect that has shipped out the door and is in production. Studies show the relative cost of a defect based on when in the development cycle it is discovered (e.g., it is ten times more expensive to fix a bug in production then during development). It is vital to reduce the overall number of bugs through better testing and to discover any bugs as early as possible in the software development process.

Rapid iteration solves this problem nicely. Using rapid iterations—where testing is an integral part of the iteration itself, a repeated step in each cycle—can lead to discovering defects earlier in the process. Agile testing will then lead to less defects overall as shown in the next two tables.

This table is from How Agile Projects Measure Up and What This Means to You by Michael Mah The project shows a team using Collocated XP methodology with rapid iterations compared to the industry standard for comparable project sizes.

Follett Software Collocated XP vs. Industry Average

*Using an average project size of 500,000 lines of new and modified code

The next table is from the same study only using Scrum.

BMC Software Distributed Scrum vs. Industry Average

*Using average project size of 700,000 lines of new and modified code

Both tables show a clear and rather significant reduction in defects, and there is a direct correlation between cost (“efficiency” in the first table) and defects. Agile practices that rely on rapid iteration, such as Scrum (first study) and XP (second study), can lead to measurable cost savings. It’s important that these two studies both include agile design and testing practices as part of the iteration.

But, remember that rapid iteration doesn’t leave much time for testing. So, how can rapid iterations lead to catching defects sooner and reducing defect incidence overall?

Agile Testing
There is a whole lot to tell about agile testing. Entire agile practices are centered on testing (TDD, for example), but many incorporate testing into the iteration—a task just like a bug fix, code review, or new feature.

In a short development cycle, there isn’t a whole lot of time to run a full coverage suite of tests. Some companies even have regression tests that take days to complete. Regression tests of that sort are great in a sense, but they can only discover defects later in the development process. Remember, the longer you wait to catch a bug, the more it costs to fix it.

If it takes five days for a regression test to complete and catch a bug, how many more changes have been introduced to the code since then, how easy will it be to fix now without disrupting something else, and what happens after you wait another five days to confirm the fix? It’s not a very agile approach, but automation is important to agile testing with some manual testing aspects thrown in for good measure.

Popular examples of agile testing techniques include:

• Continuous Integration—Builds occur right after a developer commits a change to catch errors right away and ensure new changes play nicely with the rest of the system, including recent changes from other developers.
• Unit Testing—This is an example of testing that can be automated at the code level (“white box”) and checks for logic errors. For example, if X is set to 7, do we get the expected result for Y?
• Functional and Smoke Testing—This is a manual or automated way to test features—usually newly created elements—to ensure they are working. In a rapid iteration, there will be minimal manual testing coverage, but it can be greatly expanded with automation.
• Regression Testing—Verify that old features and functionality still work and have not been broken by new changes, can be automated, and can include functional and unit tests.

As Martin and Schwaber write, “The short cycles of Scrum require continuous testing, continuous integration, and continuous improvement of the code.” We’ve already looked at good design, and continuous improvement of the code is part of that, so lets look at continuous testing and integration.

Continuous Integration
Continuous integration is one of the best testing practices that every development team—not just agile practitioners—can benefit from. The main principle is that as every change is introduced into the code repository, a new build is kicked off, usually automatically. If the build passes, it means introducing new changes doesn’t immediately break old changes. If the build fails, someone is notified immediately by the system (usually through email).

It is far easier to fix a broken build that you coded five minutes ago than one you coded five days ago, because you may have forgotten why you made a certain change. Or, you may have added more changes that work but depend on the broken section, so five days of work must be rewritten or tossed.Continuous integration also identifies who originally introduced the error, allowing the same person who made the change to fix it, rather than having someone else do it.

Continuous integration is ideal for agile teams because it occurs so quickly. With agile methodolgoies, your goal is to have functioning software to demo to your customers and stakeholders without much time in the rapid iteration cycle. So, knowing of build errors right away is a huge benefit, because you can reduce cost through discovering the root cause of the build error ,as shown in figure 1.

Unit Testing
Unit testing has fast become one of the most popular ways to automate testing. Many people merge the unit test suite with the continuous integration build so that all the unit tests must pass before each build even starts. Since unit tests are working at the code level, it can be very fast, as opposed to manual or even automated functional testing. And, a large number of tests can be run in a short period of time. This is an ideal way to shorten the length of time it would take to run a suite of regression tests, however it only tests the individual “units” of an application and not necessarily the end-user experience. So, for example, your finance application might pass a unit test where the labels for last name and first name are accidentally swapped. Although you cannot survive on unit tests alone (if you have a GUI, that is), they are still very useful.

You can create unit tests as a form of documentation. A developer can look at the test when he looks at code and wants to know how it is supposed to behave This can be more useful than maintaining detailed documentation, which can be time consuming to create and can quickly become obsolete if not continuously updated.

Functional and Smoke Testing
The five-day regression tests scenario I mentioned earlier came from a company I once consulted for. The test suite was a combination of automated and manual tests. Testing a GUI application—even with automated tools—e takes longer than a unit test, since the GUI or web page has to load or render before the test can begin. And, if the playback of the test goes too fast, it might skip over some part before it has chance to load and fail.

But, automated tests are good, and we don’t have time for manual testing, right? So, what’s the solution?

For automated regression tests, you can limit your focus to the most important features of the application, and run a much-reduced suite that ensures the most important functionality always works. Developers can also smoke test new changes that effect the GUI as they are completed—meaning that if you add a new button to your application, you fire it up, click on the button to make sure it works, and, if it does, commit your code.

Unit tests and continuous builds will look for any error introduced after each change is made. Smoke testing by developers will prevent obvious flaws. An automated functional regression suite is the last line of defense to ensure the most important features always work before the product ships out the door.

Often, agile teams don’t have a dedicted tester as defined in Scrum And XP From the Trenches: “What I mean by ‘tester’in this case is ‘a guy whose primary skill is testing,’ rather than ‘a guy whose role is to do only testing.’” Building and maintaining an automated test suite is time consuming but worth the cost. It is something that every member on the team participates in (and, in some cases like TDD, it is inherent to the process), and team compositon can be adjusted to include members with testing skills.

There are formidable challenges with instituting automated testing. It is a time consuming and a continuous process, as new tests must be created all the time to cover new changes, and new changes may cause old tests to fail while maintenance must be performed to update them. It is a worthwhile activity that can lead to a great increase in software quality and reduction in cost.

How Rapid Should My Iteration Be?
Velocity in agile methodologies is the amount of work you can get done during the iteration. In Scrum, you estimate your velocity before each sprint (the number of backlogs you can complete) and you revise your estimate based on previous experience. How many testing activities you work into the iteration may affect your velocity, and this has to be taken into consideration when you decide how long your iteration should be. A longer iteration may support additional activities like more testing, while shorter iterations promote agility and the code doesn’t have time to change much between reviews, ensuring proper design.

So, which is the best? That’s up to each team to decide, and factors like testing, team composition, and more will determine the answer. “The Ideal Sprint Length” is a good overview by Dominique Stender that goes into greater detail on this question.

Conclusion
A rapid, iterative development cycle provides an ideal opportunity to inject proper testing practices—both manual and automated—such as code reviews, continuous integration, and automated unit and regression testing. Teams working this way can test quality up front and therefore correct errors and defects early in the development cycle, when it costs the least. When working in tandem with proper design practices that encourage code reviews, refactoring, and simplicity, the number of defects that occur can be significantly reduced. The marriage of the agile principle of rapid iteration, proper agile testing, and design practices will help yield higher quality software applications and sustain that quality over time.

References
Schwaber, Ken, 2007. The Enterprise and Scrum. Redmond, Washington: Microsoft Press

Schwaber, Ken and Martin, Robert, 2004. Best Practices in Scrum Project Management and XP Agile Software Development. Object Mentor, Inc. and Advanced Development Methods.

Kniberg, Henrik, 2006. Scrum and XP from the Trenches

Mah, Michael, 2008. How Agile Projects Measure Up, and What This Means to You. Arlington, MA: Cutter Consortium: http://www.cutter.com/offers/measureup.html

Fowler, Martin, 2004. Is Design Dead. http://www.martinfowler.com/articles/designDead.html
 

TrackBack URI for this entry

Comments (1)

Subscribe to this comment's feed

Sankar Santhanaraman said:

... I concur with the author on the views presented here about agile testing. Based on my experience, I have seen that the success of a project following agile testing depends on couple of important things - (a) Effective unit testing: Unlike a conventional testing approach where multiple cycles of testing is performed, no such liberty is available to the agile testers. In such a scenario, it is very important to have the upstream testing phases like unit testing being very effective, thereby ensuring minimal defect leakage. As mentioned by the author, there are many tools available in the marketplace to perform unit testing and the latest ones are compatible with state of the art architectures like SOA. (b) Progressive automation: An efficient way to test agile projects is to have a progressive test automation to manage the growing user stories across the lifecycle. Consider scenarios where in the initial sprint there are manageable number of user stories. But as the sprint cycles progress the size of the functionality implemented becomes reasonably huge, which will demand a huge test effort through manual validation. This is where progressive test automation techniques come to the fore, where in the user stories implemented are automated as the application build gets stable and thereby ensuring higher amounts of automated testing in every test cycle. This gives the double advantage of validating the existing user stories using automated test scripts within a short time span, while focusing on validating the new user stories manually. Test teams having such an approach have had greater success in delivering agile projects. To know more about Infosys’ Validation and testing Services go to http://www.infosysblogs.com/testing-services/

report abuse vote down vote up

August 23, 2011
Votes: +0