Shifting Right Offers New Possibilities for Agile and DevOps Teams

The shift-right concept originates from testing. But agile and DevOps teams also can use it to improve their systems and service to the client. However, there is a complicating factor: Different people have different explanations for what shifting right is.

Let’s look at the different forms of shifting right, what the potential benefits are, and who should ideally be involved in your shift-right process.

The Different Forms of Shifting Right

The idea of shifting left, or testing earlier in the development lifecycle, has been around for a while. Analogous to this is shifting right, or performing test-like activities later in the development lifecycle and beyond. This includes getting feedback from the users.

We have identified three different forms of shifting right.

The first is testing in production. Some people, especially testers, start to panic when they hear that phrase—the client could be confronted with bugs! But there are definitely situations in which testing in production can be a good option.

First of all, testing in production is mostly on top of unit tests, functional tests, and nonfunctional tests, so we already know the product is working. But do the users use it in the right way? Can they find the new feature and understand it? And is it working on all devices used by our customers? These are the questions that can best be answered by testing in production. At the same time, we can monitor the services we deliver in order to make sure we deliver them correctly.

A lot of organizations already do a relatively limited form of testing in production. After a deployment, or when the organization goes into production several times a day, the testers make sure the system is available, the interfaces are up, and the functionality is working. Some organizations “follow” the first transactions though the system, sometimes even continuously.

We sometimes get questions about whether this should be called testing or monitoring, but we think this discussion is irrelevant. One way or the other, the team is responsible for these activities, so they should be done if they add value. The main goal is to be sure the system works properly, and it’s undeniable that testing skills are useful for activities like this.

Case 1
A big online web shop scheduled a course to be delivered. The course was planned for the day before a new release was to be deployed, and the team members were supposed to have time. But the course day was not a success.

The new functionality was a shop-in-shop on the main web shop, the first time the organization deployed something like this. Because of the complexity and the inexperience with this kind of functionality, the DevOps team was more focused on the deployment. On big screens the team members could see the first visitors going to this new part of the website and ordering products. The deployment was not fault-free, and the team had to correct some bugs and deploy new versions immediately.

At the end of the day some minor things were disabled in production, but the main process worked well. We had a drink and rescheduled the course.

Another form of shifting right can be pretty close to testing in production but is not necessarily: A/B testing and canary testing. With A/B testing, the team develops two solutions for a feature and presents both to the users, who then give their opinions on the best option. This can be a group of user representatives, such as user acceptance testers, or it can be real users in production. Canary testing is making a new feature available to a small number of users, with the team monitoring how they use it. Both A/B testing and canary testing could be done in production, but they don’t necessarily have to be.

Drawbacks of A/B testing and canary testing in production is that it is sometimes difficult to ask the user what they liked more, simply because you sometimes don’t know who the user is. And in the case of A/B testing in production, every user sees only one solution, so they can’t compare it to anything. Sometimes you can measure things like conversion rate or the number of times a process is completed, but this is not always possible.

For this reason A/B testing and canary testing are sometimes done in a user acceptance test, where the testers see both solutions (A/B testing) or see the old and the new feature (Canary testing).

Case 2
An A/B test was performed with a controlled group of users. This was not done in production, but in a user acceptance test environment. We gave half the users solution A and half the users solution B. After they tested the new features (which included much more than this A/B test), we asked the users how they liked the solution by means of some questions they had to rate on a scale of one to five.

The two solutions were almost equally rated, and the general idea was, who cares, they are both good. So it didn’t really matter which solution we chose. But we spent a lot of time designing, developing, and testing the two solutions that could have been spent on building another feature. What we learned is that A/B testing only adds value when you have two fundamentally different solutions. Otherwise your time could be better spent flipping a coin.

The third form of shifting right has to do with measuring the effect of a change—not only directly after deployment, but over a longer period of time. Is the conversion rate higher? Do more users stay with the process until the end? Do more users use the feature after the initial launch? Is the intended value delivered with the system or as an adjustment to an existing system?

This form of shifting right is done in the operations phase and is about measuring relevant data and translating it into valuable information for the team and other stakeholders. The development of a feature costs money, but some organizations don’t bother to measure whether benefits are realized from the feature. This is what this third form of testing in production is all about.

Some people say that looking back on whether benefits are reached is not necessary because the feature is already in production, but we don’t agree with this. DevOps and agile are about learning in order to do things better in the future, and the information we get out of measuring the effect of a change can help us to estimate and prioritize changes better. System development is not a one-time project anymore, but a continuous activity.

Case 3
A development team was responsible for realizing a web application for a marketplace for specific products. There already were some comparable marketplaces available online, but this new one had some unique selling points. Before development was started, the team defined the minimum viable product (MVP), along with measurable benefits in terms of market share, minimum number of visitors, minimum number of transaction,s and minimum number of providers.

This set of measurable benefits and the definition of the MVP was important guidance for both the product owner and the team. Benefits were measured after implementation, and about three months after the deployment of the MVP, the goals were met. This generated enough funds to develop the marketplace further.

It’s All about Value

Regardless of the form your shifting right taks, the aim should be measuring the value of the system or the change for the customer. Finding defects is not the main goal of shift right; measuring the extent to which the intended value is realized is.

Customer value is not always easy to predict up front, and in order to evaluate it, we need feedback—feedback from real customers, maybe in production, maybe before we go to production, or maybe after a longer period of time. In this way shifting right is essential in the inspect-and-adapt cycle.

The cycle of inspecting and adapting is a useful practice in DevOps and agile. In the Scaled Agile Framework (SAFe) in particular, inspecting and adapting is a significant event, held at the end of each program increment, where the current state of the solution is demonstrated and evaluated by the train. According to this definition, inspecting and adapting is limited to a product increment, but in our opinion, shifting right broadens the use of inspecting and adapting to user acceptance testing and production.

The inspect-and-adapt cycle focuses on the product, process, and performance of the team as well as on adjusting the way teams work to get more value. Seen this way, inspecting and adapting is a team activity, but shifting right adds value to the inspect-and-adapt practice from a customer perspective. And customer value is what it’s all about.

Case 4
A development team had the task of migrating a big, monolithic system to a new, multilayer architecture. The maintenance costs of the old system were high and the system was hard to test, which led to a long time to market. Due to the amount of time needed for this migration, it was financially not possible to do the migration in one release.

The solution was to migrate in small, incremental steps. In order to decide which parts to migrate first, the team installed a logging system that gave information about which functional parts were used most by the users—that was the inspect part. Based on the analysis of the user data, which was gathered for a few months, it was easy to determine what the MVP was and to make a user-driven roadmap for the migration—this was the adapt part.

The main lesson learned here was that gathering this kind of information is easy to realize and essential for making well-founded, user-driven decisions.

The Tester in Shifting Right

Before looking at the question of whether the tester could have a role in shifting right, let’s look at another question first: Can all testing be done by means of shifting right?

The answer to that is no. Testing is also finding bugs, investigating the system, and determining how well the system is working. Shifting right will always be on top of unit testing, functional testing, and nonfunctional testing.

The second question is whether the tester should execute or organize the shift-right approach. The theoretical correct answer would be no, because there is no tester in agile or DevOps. But we all know that in practice most teams have a team member with testing skills, at least when a complex product is being built or the quality of the product is important to the customer.

In our opinion, every team member could have a hand in executing or organizing shift-right activities. But at the same time, we often see that testers have the right skills and focus to do these activities. Good testers have analytical skills, and they know how to investigate things. The reporting about the benefits of a feature has similarities to modern test reports. The transition for the tester when it comes to shifting right involves measuring and evaluating the value after the usual tests, or even after the system is in production to their work. If possible, pairing a tester with an operations or DevOps team member will boost the speed in which the team can shift to the right.

Even if testers are very involved in the shift-right activities, they can’t do it alone. The tester will have to involve different people from inside and outside the team, including the product owner or business owner, other team members, and clients. Ideally, shifting right starts with a stakeholder analysis in order to decide who should be involved, whether shifting right should be used in the inspect-and-adapt cycle, and how the outcome will be reported. This way the right people will be involved in the right way.

Shift Right to Serve Your Customers

Shifting right as part of the inspect-and-adapt cycle offers agile and DevOps teams techniques to measure the value of the system. Teams should consider each of the possible ways to shift right and their applicability to the team’s specific context, then proactively take the initiative and make the shift-right approach a part of their job. When the right stakeholders are involved, shifting right can help teams serve their customers better.