6 Steps for Succeeding with Test Automation in Agile

Lots of test automation efforts in agile software development fail, or at least do not maximize their potential. This article looks at two main reasons test automation may not live up to the expectations that testers and other stakeholders in the agile development process have, then outlines six steps to avoid falling into these traps. Here's how to succeed with test automation in an agile environment.

In order to keep up with the ever-shorter release cycles that come with the adoption of agile software development, many development teams are embracing test automation as a means to continuously ensure that every software release conforms to the desired level of quality.

This is a significant shift from traditional software development practices, where testing was often stuck on at the end of the development process and seen as a process burden rather than a benefit. Testers working in an organization that has adopted agile software development, moved to a DevOps culture, and embraced continuous integration and continuous delivery are therefore required to have at least a basic level of understanding of how to effectively implement test automation as part of their daily activities.

Unfortunately, lots of test automation efforts in agile software development fail, or at least do not maximize their potential.

I'd like to explore what I think are the two most important reasons for test automation not living up to the expectations that testers and other stakeholders in the agile software development process have. Then, let’s look at strategies and tactics for how to avoid falling into these traps in order to succeed with test automation in an agile environment.

Unreasonable Expectations

The first of the two main reasons I see so many automation efforts fail to reach their potential is because of the unreasonable expectations that precede its implementation. Too many team leads, development and project managers, and C-level executives (although other roles are not innocent, either) see test automation as the one-stop solution to all of their testing bottlenecks.

However, reality has shown over and over again that:

  • Implementing test automation takes time, effort, and specific skills
  • Automation is an activity that supports testers, not replaces them
  • Far from every activity related to testing can be automated

Yet there's still a widespread belief that test automation somehow, magically and with the press of a button, will perform all of the required testing for you. When, after a couple of months of diligently building and running tests, this notion turns out to be a fantasy, the people who have been working hard on getting the automation to work are often scapegoated, and sometimes even laid off.

In my opinion, the best way testers and automation engineers can deal with this problem is to think and communicate before acting. Make sure that all stakeholders are on the same level with regards to what you can reasonably expect from automation. Take a look at previous efforts, both within your organization and in the wider software testing and development community, and learn from those experiences.

What should work? What will likely not? Don't expect automation to be the magic bullet that will solve all of your testing problems.

Lack of Attention Dedicated to Automation

The other main cause of failing automation efforts is the lack of time that development teams (and, on a larger scale, whole organizations) allow for creating useful, maintainable, and effective automation solutions. Even though it should not be a surprise that implementing automation takes time and effort, automation is still too often one of the first things to suffer when time becomes sparse.

This applies to projects, but it also applies to teams working in an agile environment. Even though automation is high on the wish list of many a software development team, when the end of a sprint comes close, delivering features almost always takes precedence over automation.

Note that I do not think that this is necessarily a bad thing. In the end, it's the features that deliver value to the end-user, not the automated tests that help safeguard them working correctly. In the long term, however, teams that allow the releasing of features to take precedence over everything might just run out of breath while trying to make deadline after deadline. It's almost as if they forget that adopting an agile way of working is about creating software at a sustainable pace while receiving feedback early and often, not about delivering features at warp speed.

Not allowing for enough time to create a solid automation solution also has an unwanted side effect: If you don't award automation the priority it deserves (and that may not be the highest priority), it is unlikely that your team members will have enough time to become skilled automation engineers. I see automation as a craft, and like any other craft, it requires continuous learning and honing of your skills.

Making Automation Part of Your Agile Development Process

Now that we’ve detailed two main reasons for failing automation efforts, I'd like to propose a step-by-step guide to help you avoid these pitfalls and successfully implement test automation as part of your agile software development activities. This is by no means a definitive guide, and not all steps might apply equally to your situation. But following them might just help you to be more successful in your agile automation efforts. 

1. Set reasonable expectations

As I said before, the success of any automation efforts starts with reasonable expectations. I find that asking and agreeing on the “why” is a good way to set these expectations. Why are we automating in the first place? Why do we think we need test automation at all?

Note that in my opinion, there are good and, well, less good answers to this question. “Because we want to be able to get fast feedback on every commit done by a developer” is a good reason, whereas “Because we don't want to have manual testing at all” is a prime example of a source of unreasonable expectations.

2. Treat test automation as software development

Make sure that all parties involved are aware that the introduction of test automation is essentially equal to introducing a software development project within your software development project.

This applies to both the project planning aspect (you should assign resources to it and allow for time spent on developing and maintaining the automation, etc.) as well as the technical implementation of it (you're writing code, so be sure to apply good development patterns and practices and respect the fact that test automation is a craft that requires specific skills).

3. Assign dedicated resources to automation

In order to be successful with test automation in your agile software development efforts, you'll need to make sure that the people responsible for creating and maintaining the automation both have the right skill set and get enough time to do so.

The number of people who will dedicate their time toward automation depends on a number of factors, including their skills, the type of automation that needs to be created, and the complexity of and risk associated with the application to be developed. If your organization currently does not employ enough people to fulfill your automation desires, or the people lack the required experience, temporarily hiring external experts might be an option worth considering to get you started.

4. Pick a starting point

Just like with any significantly large project, it might look like a daunting task to decide where to start with test automation. I have two pieces of advice here:

  • Start either with some low-hanging fruit (this helps show stakeholders the added value of test automation quickly) or with a part of your application associated with the highest risk or the highest impact of defects.
  • Try to avoid starting with end-to-end test automation, like using Selenium. While this might seem like the obvious choice when you're looking to write automated regression tests, this type of test is the hardest to write, the slowest to execute, and the most prone to fail, either due to changes in the application under test or to false positives caused by synchronization or environment (e.g., test data) issues. Instead, see if you can create a solid set of meaningful unit tests, or leverage API-level testing to verify critical business logic.

5. Make automation part of your definition of done

When you're working in an agile setting, it makes sense to make test automation an integral part of your definition of done for a given feature. Try to avoid these two traps, though:

  • Including a statement like “All tests should be automated” or “We should have automation in place for every feature delivered”—sometimes automation doesn't make sense, is cumbersome, or is even downright impossible to create. Instead, opt for statements like “Existing automation is updated to reflect changes brought by this feature” or “Additional automation has been created where deemed relevant and useful by the development team.”
  • Relying on percentages—“100 percent code coverage” is an empty statement. It says nothing about the quality or relevance of the tests. Likewise, “80 percent of all tests done are automated” does not make sense, either. For one, this relies on a one-to-one translation of tests executed to automated tests, an approach that has time and again proven to be ineffective. But more importantly, how do you define that 80 percent in the first place? Is that 80 percent of everything that can be automated or of all testing performed? I think you catch my drift here.

6. Learn and adjust

This should not come as a surprise anymore: Test automation is a software development activity, and when you're doing this in an agile way of working, it makes perfect sense to apply the agile principle of fast feedback, quick evaluation, and learning on the go.

You don't have to get it right from the start! Just like with your application under test, take the time to experiment, evaluate early and often, learn from any mistakes, and stick with what works. Over time, and with the right amount of nurturing, this should lead to an automation approach that fits hand-in-glove with your software development efforts.

Please note that every situation is different, and what works for one organization might not work as well somewhere else. Having said that, I sincerely believe that the information above will be helpful to most organizations that are struggling with effective testing and are therefore looking toward automation as a means of improving their agile testing efforts.

User Comments

Mark Bentsen's picture

Thanks for this article. Great points all around. There are some assumptions here that may be different from one shop to the next. Our expectation is that a lot of the automation is unit tests and written and maintained by dev. Middle tier tests are defined by QA, but dev writes most of these and QA owns data integrity. In other words, dev creates a test with soapUI pro, but then QA provides the data so it is run many, many times by changing up the data and expected results. When it comes to UI automation, we treat this very sparsely and try to not make tests we fall in love with as the cost of maintaining them is steep. In many cases we look to a flexible framework and know tests for reliability coverage will need to be rewritten a lot. Because of this, we work to make it easy to write them. Having one end to end test may require 2-3 hours of work and we rewrite it when we need to test load.

May 20, 2019 - 4:52pm
Bas Dijkstra's picture

Hey Mark, thanks to you too for the comment. You are absolutely right when you say that a lot of this is dependent on the specific context of a project, team or organization. My aim was to provide some general tips and tricks, I wasn't expecting to lay out a blueprint that should be followed by all :)

You're absolutely right on the approach with regards to E2E UI automation too, they are wonderful but should be created carefully. 

May 21, 2019 - 3:50am
Suhas Koorse's picture

I have to test a 4GB file with 127kbytes on each line. 

Can you tell me how should I approach this testing? It is in JSON format


July 1, 2019 - 9:33am
Jesse Phillips's picture

You'll possibly want to use a memory mapped file, read data line by line. Confirm the expected line separator. Then assert the byte count, check with your language to avoid character counting. 

What you may want to do is go back to the business and ask why they requirements on this file which specify sizes. It does not sound relevant to end user experience. 

May 15, 2020 - 12:45am
Alisha Henderson's picture

Thank you for sharing this valuable article.
Test Automation is an interesting and challenging practice for many enterprises. The abundant benefits even influence many of them to choose automation for their projects. But, an improper understanding to perform test automation can make many teams face several challenges and even fail the test automation projects. you can read this automation in testing challenges here also. I just read this blog and found helpful. 

April 17, 2020 - 7:29am
Kamal Singh's picture

Thanks a lot, It's really so informative blog to us.

August 7, 2020 - 2:20pm
kirankumar paita's picture

Thanks for sharing such a nice information about steps for succeeding with test automation in agile.

nice content.

please keep sharing.


September 18, 2020 - 5:47am
kirankumar paita's picture

Thanks for sharing such a nice article about how to succeding with test automation in Agile.

very informative.

please keep sharing.

September 22, 2020 - 5:56am
Mike Kramer's picture

Thank you. Very informative article. I think every software development company will find it useful to familiarize yourself with it.

December 22, 2020 - 7:51am
fleekit solutions's picture

Great share! Thanks for the information. Keep sharing!

July 11, 2022 - 2:27am

About the author

CMCrossroads is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.