100 Percent Unit Test Coverage Is Not Enough

[article]
Summary:
Many people equate 100 percent unit test coverage with high code quality, but that is not enough. Code coverage tools only measure whether the tests execute the code; they make no judgment on the effectiveness of the tests. Testers should review unit tests, even if they have high coverage levels, and either help improve the tests or supplement them with extra tests where necessary.

Why do we need testers when our code is 100 percent covered by unit tests run by developers? Why should testers review unit tests when creating their tests? Because 100 percent unit test code coverage is not enough. 

Code coverage tools trace the execution of your code and provide metrics about that execution. One of the most common measures is statement coverage. Statement coverage gives a percentage of the statements executed over the total number of statements in the program. Many organizations set goals for unit test coverage, with a common target being 80 percent statement coverage.

Developers pride themselves on getting to 100 percent unit test coverage, and people on the project teams associate this with having high-quality code. But even though the tests execute every line of code and we then call the code “fully tested,” this can be misleading.

One hundred percent unit test coverage does not mean we had good tests, or even that the tests are complete. The tests could be missing important data and only testing with data that succeeds, failing to test data that causes failures. One hundred percent unit test coverage doesn’t say anything about missing code, missing error handling, or missing requirements.

Tests also might not actually check the functionality of the code. Merely executing the code without checking its functionality still counts in the coverage metrics.

The following examples illustrate what can go wrong with relying on unit test coverage alone. The code is Python, and tests are written with the unittest module. Hopefully these examples inspire testers to review unit tests, improve these tests, or supplement them with functional tests to fill in the gaps—even if test coverage is 100 percent.

Missing Test Cases

Consider the following function (only_correct_data), which takes several parameters, performs some math, and returns the result. It would only take one unit test to achieve 100 percent code coverage, because this function only has one line of code:

def only_correct_data (a, b, c) :

    return (a / (b - c))

def test_only_correct_data(self):
   #only tests with data that leads to correct results

    self.assertEqual( only_correct_data(1,2,3) , -1)
    self.assertEqual( only_correct_data(2,3,1) , 1)
    self.assertEqual( only_correct_data(0,2,3) , 0)

The test coverage is 100 percent, with three test cases that cover a negative result, a positive result, and a zero result. What is missing is the test case that would create a divide by zero error. Using the call only_correct_data(2, 2, 2) would result in the divide by zero exception. Finding missing tests cases for the developer is a great way to improve the unit tests.

Another benefit of reviewing these unit tests and making sure they are thorough is that the tester can concentrate on the overall system quality, the user experience, and acceptability to the customer. Knowing that the math is correct and verified by unit tests allows the tester to concentrate on these other factors related to quality.

Missing Functionality 

The unit tests typically verify that the developer’s intent is implemented correctly, but even 100 percent code coverage does not mean the requirements are fully met. 

In this example, there is a class, called “Dog,” that returns a “bark” based on the size of the dog. 

class Dog(object):

    def __init__(self, size):
       if size == "Large" :
           self.bark = "Woof!"
       if size == "Small" :
           self.bark = "Bow Wow"

def test_missing(self):
   #missing a test for medium dogs    

    l = Dog("Large")    
    self.assertEqual( l.bark, "Woof!" )    
    s = Dog("Small")
   self.assertEqual( s.bark, "Bow Wow" )

The Dog class is missing the medium-sized dog that barks “Ruff.” The code is missing a requirement, and thanks to additional testing at this stage, the bug was found well before acceptance testing, when it takes much less effort to fix. 

This example shows that having 100 percent code coverage doesn’t tell you anything about code that is missing. The unit tests will check that the code is working as the developer intended the code to work, but not necessarily that the code meets customer requirements.

Bad or Incorrect Tests

Having the code 100 percent executed during testing also does not mean the tests are actually good tests, or even test anything at all.

def rev_string( in_string) :
   return in_string 

def test_bad_test(self):
   #does not actually check if code is correct

    self.assertIsNotNone(rev_string("Test String"))

In this example, the function should reverse the string, but it doesn’t. Even though the unit test has 100 percent coverage, it never actually checks if the string is reversed. It’s only checking that the return value is not “None.” The test case could just execute the code without actually testing anything and still achieve 100 percent code coverage. The tester should double-check the quality of these tests or create tests that actually verify the functionality.

    do_nothing = rev_string("Test String")

This line will execute rev_string but not actually test it, still achieving 100 percent coverage. In these cases the tester could help the developer think through the proper ways to test the functionality to fill in the testing gap.

Exception Handling

This example shows a very common situation where the code has some error checking but is not fully tested. The code coverage for all the code created for this article is 98 percent, but the one missing test here is an important one.

def missing_important( a, b) :

    if a is None :
       #handle exception
       a = 1 / 0  #intention bug
   return (a+b)

def test_incomplete(self):
   # exception handling not tested        
   self.assertEqual (missing_important(1,2), 3)

Unit testing is a fantastic way to test exception handling because the developer has full control of the context and can inject the conditions necessary to test the errors. The tester, working at system level or through an API, might not be able to create the condition where, in this example, the variable “a” has the value of “None.”

Next Steps for Testing 

These examples are simple in order to demonstrate these points, and of course in practice, unit tests are valuable. But a high percentage of unit test coverage does not automatically equal high code quality; the tests need to be effective, as well.

Testers are still needed to review the unit tests to make sure they are the best tests possible, to improve or supplement unit tests with tests of their own, to keep the customer requirements perspective in mind when reviewing the unit tests, and to identify areas that are covered by unit testing that can help optimize the integration and system tests.

If you have other suggestions, please mention them in the comments section below.

User Comments

8 comments
Nikhil Bhandari's picture

Good article John Ruberto. Thanks for sharing your thoughts.

 

There few more tips that can be useful in this context

  • Do mutation testing, ensure you have validated your tests
  • Validate your assertions, many a times assertions are missing in un Unit tests
  • Along with Unit Testing, have your Integration and Functional testing measure code coverage
  • Use Mocks, best way to cover all your negative scenarios
  • Have a seperate set of unit tests for UI and measure code coverage

 

 

July 11, 2017 - 12:58am
John Ruberto's picture

Thanks Nikhil!  Great insignts.  

July 17, 2017 - 6:36pm
Gene Gotimer's picture

I really enjoy mutation testing for myself when checking for missing tests and asserts. I use pitest for Java, but I've heard good things about Ninja Turtles for .Net.

July 11, 2017 - 2:40pm
John Ruberto's picture

Thanks Gene, I agree test the tests...

July 17, 2017 - 6:37pm
Kathy Iberle's picture

John, do you find that using TDD causes people to  write those missing unit tests right away?   I notice that most of these cases involve failing to handle possible input values, which I would expect to be fairly obvious in TDD.  I don't think TDD would trigger everything - for instance, failing the handle an exception thrown by an API call internally probably wouldn't occur to the developer right off - but seems TDD would help make the tests more thorough from the start.  What is your experience? . Kathy Iberle

July 17, 2017 - 5:51pm
John Ruberto's picture

Great question.  I definitely see a strong correlation between coverage level & those that use TDD, but I don't think I have a good data set to show that TDD leads to better tests (or not).  I have to believe that the "missing requirements" examples might be better covered with TDD (and for sure BDD).  Also, the focus on creating a failing test that eventually passes must put some good thinking behind the assertions - leading to better outcomes than my example here.  I believe, but don't have the data....

July 17, 2017 - 6:36pm
Kathy Iberle's picture

The old "defensive programming" practices required the programmer to pay attention to the parameters & purpose of the module.  I would think that TDD would do the same, and requires creation of the unit tests too, which is a bonus.

July 25, 2017 - 8:21pm
Bas Dijkstra's picture

Hi John,

great article. I wholeheartedly agree that testers and automation engineers should at the very least be aware of what's happening in unit testing, so as to prevent either untested code or things that are tested twice and make the overall automation efforts as efficient as possible.

I recently wrote a blog post on pretty much the same topic, but I wasn't aware of this article. I'll update the post and include a link!

July 28, 2017 - 3:38am

About the author

CMCrossroads is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.