Code coverage

Code coverage measures how much of the code was executed by a test run. A tool instruments the code (JaCoCo for Java, Istanbul/nyc or V8 coverage for JavaScript, coverage.py for Python), runs the tests, and reports which lines and branches were hit. It is a white-box metric and is most often collected for unit and integration tests.

Types of coverage

Consider:

int shippingCost(double total, boolean isMember) {
    if (total >= 100 || isMember) {
        return 0;
    }
    return 5;
}

• Statement (line) coverage: were both return statements executed? Two tests, (150, false) and (20, false), give 100%.
• Branch (decision) coverage: was the if both true and false? The same two tests also give 100%.
• Condition coverage: did each boolean sub-condition evaluate to both true and false? The tests above never made isMember true, so this is not complete. Adding (20, true) fixes that.
• Function coverage: was each function called at least once.

Branch coverage is generally a more useful target than line coverage, because one line can hide several paths.

What coverage does not tell you

Coverage proves code ran, not that it was checked. This test gives 100% line and branch coverage of the method above and verifies nothing:

@Test
void runsShippingCost() {
    shippingCost(150, false);
    shippingCost(20, false);
}

Coverage also cannot reveal missing code. If the requirement says members in a certain region pay shipping and nobody implemented it, there are no lines to be uncovered. Mutation testing is the usual way to measure whether tests actually detect faults.

Using it well

• Look at the uncovered lines, not just the number. An untested error handler on the payment path matters more than an untested toString().
• Gate on coverage of changed code, for example “new code must have at least 80% branch coverage”, rather than a global percentage that pushes people to test trivial getters.
• Never treat 100% as the goal. When a metric becomes a target, assertion-free tests appear.