Code Coverage as a Quality Gate

Photo by vadim kaipov on Unsplash

In software development, code coverage is one of those terms that really seems important. As software developers, we certainly make it important from time to time. What code coverage means to you, how you achieve it, and the value placed upon that number likely says more about you than the number itself.

The conversation around code coverage should always be based on the outcome needed (i.e. higher quality code) and not the number itself. Code coverage should be a signal of the system and never a hard line.

What is code coverage?

"is a percentage measure of the degree to which the source code of a program is executed when a particular test suite is run."

 - Wikipedia

Meaning, the code was executed, not that it was correct.

Teams with emphasis on code coverage value itself may ship high quality code but how they arrive at that number is not generally sustainable and will be based more on ceremony than striving for the outcome of quality code.

Teams treat code coverage as a signal, and not the value, will likely have a higher and more sustainable approach to code quality based on the practices they employ.

The difference is the team culture.

Examples

A team that has a declared a code quality gate (e.g. 80%) generally follows this downward spiral

• It's a new project, and they want to do it "right"
• PR branches will have tests, however, branches that pass code coverage may still bring the main branch down just enough to fail the build for quality, even when the new code has near 100% coverage
• The side effect of this is that it introduces chaos, and suspicion, as the new changes worked fine in isolation but fail as part of the whole, requiring additional effort, often orthogonal to the task at hand, in order to bring the code quality number for the main branch as a whole back up just enough to pass
• Delivery pressure dips code quality below the threshold of 80%, so a new, lower threshold is set
• Work is paused to focus on getting code coverage number back above the threshold - often not even valuable work
• Rinse and repeat

This is not a healthy team cycle and is not sustainable as the code base complexity grows as it leads to an "exception based" or "yeah, but" style of delivery.

A team that understands quality is more than a number will generally

• Write test for the important things 
• Treats tests as production code
• PR branches will have tests 
• Observes signals in the system, including code coverage percentage. Other signals might be
• bug backlog
• number of incidents 
• quality of the tests, i.e. ease of change
• Observes the system as a whole to see where improvements needed, pausing work to improve overall throughput, and quality of the system
• Rinse and repeat

This is a sustainable team environment that takes a more end-to-end view of quality and not just a number. The difference is the team culture and how they understand the why of it all.