Is Test Automation a “Project”?

Recently I read a post on a mailing list about “failed test automation projects.” This expression made me stop and think. Automating different types of tests is an integral part of software development, and that’s not only on agile teams. I worked in a waterfall shop back in the ‘90s where the programmers automated all their unit tests and ran them in a continuous build. My test team, which was involved in every project from start to finish, automated GUI-level functional tests as well as performance and load tests. Our product had no critical bugs in production.

I’m dismayed that many people still think test automation is something to be done by a team of test automation specialists and that QA should be a separate team, even in agile projects. I think every software team would benefit if they stopped thinking in terms of “test automation projects.”

Why Do We Automate Tests?
We automate tests to help our team work at a sustainable pace with a manageable level of technical debt. We automate regression tests in order to shorten the feedback loop between check-ins of new code and discovery of regression failures. We automate load and performance tests because, well, there is really no other way to do those types of tests. We automate repetitive steps to save time in setting up scenarios for exploratory testing. We automate the generation of test data for various purposes.

These are not optional activities. They’re all necessary in order to deliver high-quality software that provides business value in a timely manner without killing the team through overwork or sapping them of the will to live by having them do the same tedious keystrokes over and over.

The Problem with “Projects”
Because the term “test automation” contains the word “test,” managers think testers should do the test automation. They purchase a GUI test tool, usually one with a “record and playback” feature, and start a project for the QA department to automate all the manual regression tests. Perhaps they hope that once these tests are automated, they won’t need that pesky QA department anymore!

Some companies form a team dedicated to the test automation project but staff it with testers. After all, they’ve purchased a test tool that they believe is so easy to use that anyone can create the automated scripts.

In these scenarios, if the teams actually automate any tests, they’re likely to do so at the GUI level. The testers who create the scripts most likely have little or no programming experience. They aren’t familiar with code design patterns such as “Don’t Repeat Yourself,” so the test code is full of duplication. The test tool may not enable good code design for the test scripts. The scripts are brittle and hard to maintain.

I’ve heard of test teams that had several hundred thousand lines of test scripts that no longer worked. I’ve met testers on other teams who spend 100 percent of their time updating tests to keep them passing in the continuous integration process. This is a bad investment that adds to technical debt and slows the development process down even more.

Some companies are wise enough to staff their test automation department with programmers. These have a better track record. However, if they don’t leverage tests to help guide development, as in test-driven development and specification by example (also known as acceptance-test-driven development), then they are missing out on significant value that tests could provide. And, if they don’t collaborate closely with the programmers who write production code, then they may have to work with code that isn’t designed for ease of test automation.

Leveraging Different Skills
Let’s contrast the “test automation project done by test teams” approach with a whole-team approach where automation is an integral part of software development.

Learning to automate tests of any type is hard. At first, it’s a lot of extra work with no reward. You only realize benefits if you do it properly and stick with it.

Over the past several years, I have asked many people whose teams successfully automate tests to share the key factors that enable their achievement. In every case, they credit a whole-team approach to overcoming impediments to automation. Programmers who write production code are, hopefully, skilled at applying good design patterns. Writing automated test code is generally easy for them.

Testers are skilled at knowing which tests to write and to automate. When a programmer and tester pair to automate a test, not only is it much faster than a tester’s struggling to automate alone, but also it enhances collaboration and communication that benefits the whole product. I personally enjoy automating tests, but I’ve had to admit that the tests I design on my own are not nearly as clear and maintainable as the ones I do in collaboration with my programmer teammates.

Database experts on the team help provide test data and procedures such as database refreshes. System administrators help integrate automated tests into continuous build processes. The diversity of skills on a software development team helps overcome any impediment to automating any given test.

Automation with Good ROI
With the whole-team approach, everyone works together to find the appropriate approach, format, and style for their tests. Then they look for a driver, framework, or other tool that does what they need, or they build their own. The whole team experiments until they find what works best for them.

Automating tests is a huge investment. Done correctly, it pays off over the long term by keeping technical debt low and keeping the development feedback loop short. Automating all regression tests frees testers to engage in more valuable activities to help deliver the best possible software product.

Here’s an Example
In 2003, my team made a commitment to deliver the best quality software we possibly could. To achieve that goal, we adopted test-driven development, specification by example, and a whole-team approach to automating all types of tests at all levels. The first year was hard, but we’ve continually improved our automation over the years. This allows us to implement new features confidently and quickly while keeping technical debt low.

Recently, we wanted to use Dojo code in our web application, but our existing GUI test tools didn’t support it. As a team, we brainstormed ideas to solve this problem. In the end, our system administrator spiked a solution with an open-source driver and a homegrown framework. He turned this over to our lead architect, who spiked a solution with the same driver and an open-source framework. The whole team considered the results and chose a way to proceed. We treated this as a user story, budgeting time to experiment. However, it was a user story on which the whole team worked.

Don’t Make Test Automation a “Project”
We’ll never run out of test automation challenges, but if we stop treating test automation as a separate “project”—something “nice to have” that may or may not get done—then we can successfully overcome those challenges. Instead of focusing on test automation, let’s focus on delivering the best quality code possible and do that frequently, at a sustainable pace.

Test automation isn’t an end in itself; it is a means to manageable technical debt, high software quality, and enjoyment through doing our best work. For that matter, testing isn’t a separate phase; it’s an integral part of software development. More and more teams understand this. Consider viewing testing and coding as two parts of one whole, and see where that leads your team.