Testing in production gives more realistic opportunities to test, increases application transparency between the core product team and users, and supports the idea of continuous development through continuous testing. It's a good technique to embrace in your testing process—but it should not be entered into unprepared. Learn the advantages and pitfalls here.
In the earlier days when most companies used a waterfall style of development, Google was joked about in the industry that it had its products in beta forever. In fact, Google has been a pioneer in building a case for testing in production. Traditionally, a tester has been responsible for testing all scenarios, both defined and extempore, in a test or staging environment before the build could go live. But today, this premise is changing on several fronts.
For instance, the tester is no longer alone in testing. Developers, designers, build engineers, other stakeholders, and end-users, both within and outside the product team, are testing the application to provide feedback.
The test environment, product under test, underlying technologies, test combinations (devices, platforms, browsers, etc.), are all so complex today. The services mindset is very high; gone are the days of a local setup to test for the product. Cloud environments offer the scale and ease needed in testing complex interfaces, and the production environment offers unique testing opportunities that cannot be fully implemented before release.
And the constraints the test team works within, including time, cost, and availability of niche testers, are more prominent than ever before.
With all these factors at play, testing in production is inevitable. However, instead of looking at testing in production as an option that is being thrust on the teams, if one closely looks at the inherent benefits it holds, this exercise can greatly help in beefing up the quality of the product.
So, what does it really mean to test in production, what are some of the ways in which it can be done, and how does it help?
What It Means to Test in Production
Testing in production is an exercise where the quality function of validating and verifying an application is taken up in the live environment, after release, either by the tester himself or by end-users. There could also be others such as business people, marketing teams, or analysts who share feedback with the product team. Testing in production gives more realistic opportunities to test, increases application transparency between the core product team and users, and supports the idea of continuous development through continuous testing. In this mobile-first world, testing in production is a core technique to embrace in your testing process.
Of course there is a negative connotation to issues reported from the field: This basically meant the tester did not effectively and comprehensively test the product. An issue reported from the field is handled with very high priority and fixed as soon as possible. While issues that show up in production can still leave a black mark on the tester’s effort, a test in production initiative nowadays has an expanded scope. It is not just reactive actions in response to user-reported issues, but also proactive planned test efforts that are taken up before the product is officially rolled out.
Techniques for Testing in Production
The core techniques for testing in production encompass active and passive monitoring (involving real data and synthetic transactions), experimentation (both controlled and uncontrolled) with real users, and stress tests to monitor system response. While these may sound simple, one has to be extremely sensitive in all of these tasks, as live users’ data is often involved and has to be protected (the sensitivity is not just with any data loss, but also with data privacy and security). Also, the volume of transactions in live environments are so high that any test effort here has to be adequately monitored and followed up on.