Code Coverage vs Code Quality: What the Numbers Don’t Tell You

Development teams place great emphasis on metrics related to code coverage. Often times arbitrary thresholds are set for code coverage, such as 80% or 90% code coverage, which in spirit are the threshold of quality gate. When code coverage metrics rise, managers feel a sense of celebration when this happens and developers race to hit coverage targets before each release.

Unfortunately. high percentages of code coverage rarely mean that the quality of the software built has improved, creating a false security that can be more dangerous than not having any metrics at all. 

The Myth of Comprehensive Testing

Code coverage refers to the number of lines of the code that run during test execution; however, simply executing the code does not require or imply veracity to anything. In other words, a test could execute every line in a particular method without asserting anything meaningful, so the code would return with 100% code coverage and actually validate nothing in a traditional sense. 

Think about a method that processes payments and contains numerous types of business logic. The tests would likely execute all of the possible types of flows that exist in the payment process (successful payment, card declined, network timeout, or fraud check), which would also give the tests covering all the lines of code in the method.

However, if the tests do not verify the appropriate amount is sent, status is updated correctly, or if the proper error message is displayed, it does not demonstrate that the coverage metric is very misleading to all when it comes to the quality of testing.

This distinction between code coverage and quality becomes more important around code review and refactoring. The "well-tested" code with good coverage percentage is often inevitably found to have a significant amount of defects, and this is not discovered until too late.

When High Coverage Ends Up Meaning Low Quality

In an ironic turn of events, extremely high code coverage occasionally indicates testing flaws instead of testing excellence. Goal coverage of 95% or better for an entire codebase often means teams are testing trivial code that requires no testing while potentially ignoring complex code logic that should have been validated.

Tests that cover getters, setters, or other simple transformations of data inflate coverage statistics with no additional safety net. In contrast, complex algorithms, edge cases in business logic, or never-executed error paths might receive a very shallow level of testing just to obtain coverage rather than validate behavior.

The goal of code coverage percentages could create poor design decisions. For example, a developer might be tempted to keep complex logic consolidated into a single module rather than abstracting its logic out, thus keeping down the goals of overall coverage. While it is powerful to have metrics, code coverage driven development flips the priorities that define a solid architecture. In other words, metrics should serve to inform quality, not define it.

What You can Actually Learn from Code Coverage

Code coverage, despite its shortcomings, can still be informative if used appropriately. It is particularly useful when it indicates code that has not been tested at all—areas where coverage is zero or near zero indicate risk that should be reviewed.

For instance, if mission critical business logic shows 15% coverage code – that indicates it is definitely a problem worth investigating. Coverage measurements can also help highlight code that has been forgotten, orphaned utility functions, or features added without tests being written. This negative signal—highlighting what is not tested—can be much more reliable than the positive signal of well-tested code.

Coverage trends over time also have limited utility. If code coverage is going down, sprint after sprint, new code is simply outpacing test code, and that is certainly going to pile on more technical debt that will need to be dealt with eventually in the future. Stable measurements or increasing coverage over time suggests testing is at a proper level compared to features being released to master.

Quality Metrics that are Worth Mentioning

Going beyond code coverage and thinking about testing quality takes a little more time and consideration, as you will not only want to think about test quality in a single dimension. Mutation testing is expensive, but it determines if tests have real defect detection capabilities. By making changes to the code (or creating mutations), and observing whether the tests fail, mutation testing takes a more realistic measure of test quality instead of just test execution.

Assertion density (the number of assertions per number of lines of test code) is a second quality indicator to consider. Instances of tests having very few assertions compared to their total number of lines of test code may simply result in a lot of code executing without properly verifying the behavior of that code.

For each test situation there is not a magic formula or answer, but in tests that require many setup lines compared to the number of assertions, are often indicative of a validity problem.

Bug escape rate is where testing effectiveness is revealed. Even if the tests successfully run, how many defects actually get into production? A test suite of a high bug escape rate, even if high in coverage, means the tests are not effective. In cases of lower coverage but few bug escapes, test cases are likely both descriptive and much more practical.

Creating Tests for Quality, Not Coverage

When creating a testing strategy, the behaviors not the lines of code executed should be the focus. A test should formally state and enforce what a piece of code is intended to do, not just verify it executed some code paths.

Begin testing with high priority user journeys and business requirements. In your opinion, what would be the most harmful functionality to the business if it broke? What edge cases have caused problems in the past? Start with those scenarios and let coverage occur naturally by writing meaningful tests.

Quarterback your focus for testing effort on the code you wrote because of its complexity, the integration points, or frequent changes (your high risk areas). Simple, stable code needs much less validation than complicated algorithms or modules you are frequently changing. Risk-based testing is an allocation methodology that simply uses risk as a guide for allocating testing activities where they will provide maximum value.

Finding the balance between coverage and pragmatism

There's some code that really only requires limited testing. For example, construct framework boilerplate, trivial data classes, or simple configuration files generally do not warrant extensive coverage - if any testing is even performed. The goal of coverage testing is not to "hitting the numbers". Don't create unnecessary tests to cover the gaps. Just don't count those areas with no actual coverage in the coverage statistics. That's the ideal situation - we don't need to test everything.

Set coverage targets for the right levels of granularity. Instead of an organization-wide target, set your expectations based at the module or component level - in accordance of criticality. A payment processing module may require 90%+ coverage. But internal admin utilities - consider half that much, say, 50% coverage. Consider standards that apply to your context as more effective than a high-level solution.

Finally, coverage is one data point of potentially many. Combine it with quality of code review, defect metrics, customer significants/lessons learned, velocity, etc to create a picture of quality. Quality represents multiple metrics - one metric does not tell the entire story.

Conclusion

Code coverage is a valuable resource for identifying untested code; however, it does not gauge test quality or predict software reliability. High percentages of code coverage can conceal weak assertions or trivial tests, as well as the absence of meaningful verification. Teams focus on coverage metrics are susceptible to developing a false sense of confidence and missing quality defects. 

Ultimately, organizations can create truly reliable software - as opposed to merely hitting impressive code coverage numbers that have not complete stories - if they understand the role of coverage and utilize testing techniques based on behavioral verification, defect identification, and risk.