Risk-Based Testing: Test Only What Matters

In today’s era, the IT industry is running on “fast mode” and the majority of stakeholders are looking for quicker solutions. The project management office is always hurrying to ship products out of the door. While there is nothing wrong in doing things this way, it does become a challenge for test teams who have to accept the fast-mode schedule and promote the product to production.

My recent experience with a retail client and its test team has shown me how desperately test teams want to meet deadlines while dealing with ongoing changes. In this instance, we had a big challenge in that the business had not prioritized its need.

In a situation like this all responsibility falls on the test team, which could easily become a scapegoat for shipping slipped defects to production. A delay in the upstream software development lifecycle (SDLC) process like requirements and development can make things worse. For my team, the stakeholder’s constant demand to make the product reachable to the customer was adding fuel to our fire; the test team was under tremendous pressure.

Being a solution provider, a test team must have a logical and scientific answer to the above challenge. Using a risk-basked testing (RBT) approach worked really well as the test team could deliver on the promise. While there could be many more factors when you assess the risk, I just chose two factors to explore in this piece for the simplicity of implementation and it’s easier to understand. With the approach we chose, our test team saved a big portion on the test execution effort; it allowed for great savings yet it delivered all that was important to the business.

All of us working in the IT industry are continuously looking for cost-effective solutions. Obviously, this means implementing the most logical solution that can bring lots of value with less effort. In my opinion, this solution is for all kinds and sizes of projects. Depending upon the industry, the risk factor may change but the concept remains the same; except, of course, if you are in the pace-maker industry and you have zero risk-taking ability.

Depending on the industry, we must always maintain medium-to-low open defects during production. For example, a company making pacemakers will have lower risk-taking ability than the financial industry, which has a lower risk-taking ability than the education industry and so on. One’s risk-taking ability depends upon the industry and then the application being designed within that company.

Some industry leaders are willing to take on known risks than delay time-to-market. I believe that in the IT industry, the QA team should find a way to support the thought of a faster time-to-market concept.

In order to achieve the goal of testing only that matters and achieving a faster time-to-market, we need to rethink testing scope. You might have some questions about doing so, which could include:

• Do we need to test everything?
• How do we reduce the testing effort?
• When do we stop testing?
• What is the best way to have a shorter test-phase cycle duration?
• How do we be successful with more test cycles?
• How do we get buy-in from management when we do not test certain test cases?
• What metrics do we produce?

The answer to all of the above can be found in using a risk-based testing framework.

Risk-based testing is an approach that takes a scientific approach when accounting for risk. It is mainly based on the factors of the business impact and the likelihood of failure, although there could be more.

Using this solution will help you in many ways. For example, risk-based testing will help when expediting through the software quality assurance gate, which results in faster time-to-market. It also helps reduce schedule slippage, communicates to IT management on what exactly is at risk, helps define the severity of defects, and is a great way to flag test cases for regression testing. It’s a great tool for prioritizing test execution and selecting candidates for automation. Risk-based testing helps define the severity of defects and it can be suggestive of “what to test” when we do not have enough time for testing. When we run out of money in our budget, we can still ship the product with known risk. 

What Is Risk
We must understand how much the functionality has impacted on business. This is the soul of RBT. The definition of impact and the impact on business needs to be explained and understood correctly by all parties.

There could be many factors that need to be considered while calculating risk. Here, I have taken two main factors as mentioned below:

Risk = Impact * Probability of Failure

Categorizing impact will help you understand the impact on the business process if any functionality failed. In testing, it’s possible that any category (C, H, M, and Low) of script can fail to execute. For example, test scripts categorized as high that fail to execute will have a big impact on the business process. Conversely, a low category of test scripts will cause a low impact. Hence, it is very important that you get the appropriate categorization of requirements from the business team.

Why We Need to Define Impact
When creating a test scenario, adding an impact column and a test case that is derived from the business requirement document has the following benefits that will help you conduct RBT analysis:

• This will provide a method of identifying the high risk area within the application
• We will have a controlled risk-taking approach
• It will provide data points to identify regression test scripts
• It will be a great tool for identifying the candidate for automation
• It will get us the tool for risk-based testing approach
• This will be the only tool that reduces the scope of retesting and re-executing test scripts
• This will help define the severity of the defects during the execution phase

Risk quantification will assist you when you focus on the overall scale of impact of each risk; it’s also referred to as the level of damage that is potentially caused by the change. The following is an estimate of the overall scale of impact that will occur if a defect related to the change was found during production.

• Critical impact—5: The problem caused all major and critical functioning of the application to fail. This resulted in a large loss of revenue. A related defect impacted the ability of the business unit or the end user to receive services. 
• High impact—4: An application or transaction caused a significant disruption to production; customers are not directly affected, but the backend system was not performing.
• Medium impact—3: The progress was disrupted with large extensions to project schedule and cost; customers were not directly affected, but a change was made that affected other code. Time and research is needed to resolve the problem.
• Moderate impact—2: The progress was disrupted with manageable extensions to the short-term schedule and cost; customers were not affected. An example of this would be a field added for the business that does not work as designed.
• Marginal impact—1: A cosmetic error.

What is Probability of Failure?
This section is primarily owned by IT. It suggests a failure of coded functionality due to various reasons, including code complexity, business complexity, system architectural design, and the impact of peripheral systems. The probability of failure is also defined in five categories:

• Critical—5: This suggests extreme complex code that has been written on top of on extremely complicated architectural design. The likelihood of failure as at the top of this scale and the impact on the peripheral system is almost possible.
• High—4: There is a large amount of complex code written on a highly complicated system architectural design. The likelihood of failure is high and the impact to the peripheral system is high as well.
• Medium—3:  Medium complex codes written on medium complicated system architectural design. Likelihood of failure is medium and impact of peripheral system is medium.
• Moderate—2: Moderate complex code written on moderate complicated system architectural design. Likelihood of failure is moderate  and impact of peripheral system is moderate
• Marginal—1: Low complex code written on low complicated system architectural design. Likelihood of failure is low  and impact of peripheral system is low

Once Again 
Risk = Impact *Probability of Failure

Entering the above defined value in the graph below, we get test cases falling in four quadrants.

P1 – Critical Impact: Must be tested; ideal candidate for automation (candidate for automated smoke testing)

P2 – High Impact: Should be tested; ideal candidate for automation

P3 – Medium Impact: Can be tested if budget and schedule permits; may be automated

P4 – Low Impact: May not be tested; no impact on application and no need of automation

The business impact often gets confused with the priority of test execution. Test team will identify the test cases with the help from team and execute them in order of their priority, from critical to low.

Once the above RBT analysis is done, it’s easy for the QA team to identify the risk area within the application. In every project there are delays and schedule shrinkage, and the responsibility is on QA to complete the testing. In that scenario, RBT can be used as a handy tool for management to make informed decisions.

It is important here that the business clearly defines what is the impact. It’s often that that the business side confuses impact with prioritization. IT, business analysts (BA), and operation staff should clearly understand it and provide input to QA team.

To summarize, the QA team should consider value based testing. Testing can be a never-ending process in which there are frequent changes to the environment, application patches, and the fact that OS patches frequently happen. It is important for the QA team to define when to stop testing. Remember, the cost of quality and the cost of defect needs to be justified.

In Conclusion
Organizations should consider using an RBT methodology when working on their projects. While some of organizations are more mature than others, it should be the mature IT organizations that practice RBT at an enterprise level on all projects. A procedural teaching of this methodology to IT management will help them understand its benefit. It may take little effort to implement, but it’s worth the effort because of the great results you will see