Home > Development > 100% code coverage!

100% code coverage!

The basis of this article is a sequences of tweets betwen me and M. Robert Martin, on April 8th 2011:

  1. If you have 100% coverage you don’t know if your system works, but you _do_ know that every line you wrote does what you thought it should.
  2. @unclebobmartin 100% code coverage doesn’t achieve anything, save making you safer while nothing could be further from the truth.
  3. @nicolas_frankel 100% code coverage isn’t an achievement, it’s a minimum requirement. If you write a line of code, you’d better test it.
  4. @unclebobmartin I can get 100% code coverage and test nothing because if have no assert. Stop making coverage a goal, it’s only a way!

This left me a little speechless, even more coming from someone as respected as M. Martin. Since the size of my arguments is well beyond the 140-characters limit, a full-fledged article is in order.

Code coverage doesn’t test anything

Let’s begin with the base assertion of M. Martin, namely that 100% code coverage ensures that every code line behaves as intended. Nothing could be further from the truth!

Code coverage is simply a way to trace where the flow passed during the execution of a test. Since coverage is achieved through instrumentation, code coverage could also be measured during the execution of standard applications.

Testing, however, is to execute some code by providing input and verifying the output is the desired one. In Java applications, this is done at the unit level with the help of frameworks like JUnit and TestNG. Both use the assert() method to check output.

Thus, code coverage and test are completely different things. Taken to the extreme, we could code-cover the entire application, that is achieve 100%, and not test a single line because we have no assertion!

100% coverage means test everything

Not all of our code is neither critical nor complex. In fact, some can even seen as downright trivial. Any doubt? Think about getters and setter then. Should they be tested, even though any IDE worth his salt can generate them faithfully for us? If your answer is yes, you should also consider testing the frameworks you use, because they are a bigger source of bugs than generated getters and setters.

Confusion between the goal and the way to achieve it

The second assertion of M. Martin is that 100% of code coverage is a requirement. I beg to differ, but I use the word ‘requirement’ with a different meaning. Business analysts translate business needs into functional requirements whereas architects translate them into non-functional requirements.

Code coverage and unit testing are just a way to decrease the likehood of bugs, they are not requirements. In fact, users couldn’t care less about code average, as long as the software meet their needs. Only IT, and only in the case of long lived applications, may have an interest in high code coverage, and even then not 100%.

Cost effectiveness

I understand that in certain particular contexts, software cannot fail: in chirurgy or aeronautics, lives are at stake whereas in financial applications, a single failure can cost millions or even billions.

However, what my meager experience taught me so far, it’s that money reigns, whether we want or not. The equation is very simple: what’s the cost of the test, what’s the cost of the bug and what’s the likehood of it. If the cost of the bug is a human life and the likehood is high, the application better be tested like hell. On the contrary, if the cost of the bug is half a man-day and the likehood low, why should we take 1 man-day to correct it in advance? The technical debt point of view help us to answer this question. Moreover, it’s a decision managers have to make, with the help of IT.

Point is, achieving 100% testing (and not 100% coverage) is overkill in most software.

For the sake of it

Last but not least, I’ve seen in my line of work some interesting deviant behaviours.

The first is being, quality for quality’s sake. Me being a technical person and all, I must admit I fell for it: “Wow, if I just test this, I can gain 1% code coverage!” Was it necessary? More importantly, did it increase maintainability or decrease the likehood of bugs? In all cases, you should ask yourself these questions. If both answers are no, forget about it.

A slightly different case is the challenge between teams. Whereas challenges are good in that they create an emulation and can make everyone better, the object of the challenge should bring some added-value, something the raw code coverage percentage doesn’t.

Finally, I already rambled about it, but some egos there just do things to write on their CV, 100% coverage is just one of them. This is bad and if you have any influence over such individuals, you should strive to orient them toward more project-oriented goal (such as 100% OK acceptance tests).

Conclusion

With the above arguments, I proved beyond doubt that assertions like ‘we must achieve 100% code coverage’ or ‘it’s a requirement’ cannot be taken as a general rule and is utter nonsense without some proper context.

As for myself, before beginning a project, one of the thing on my todo list is to agree on a certain level of code coverage with all concerned team members (mainly QA, PM and developers). Then, I take great care to explain that this metrics is not a hard-and-fast one and I list getters/setters and no assert as ways of artificially increasing it. The more critical and the more complex the code, the higher the coverage it should have. If offered a choice, I prefer not reaching the agreed upon number and yet having both firmly tested.

For teams to embrace quality is a worthy goal. It’s even more attainable if we set SMART objectives: 100% code coverage is only Measurable, which is why it’s better to forget it, the sooner the better, and focus on more business-oriented targets.

email
Send to Kindle
Categories: Development Tags: ,
  1. Pablo
    April 18th, 2011 at 10:38 | #1

    Mutation testing would solve the “no assert” problem, but all the mutations testing tools for Java seems very difficult to use.

  2. April 19th, 2011 at 06:31 | #2

    I tend to agree, achieving 100% coverage for the sake of it is pointless. Also, just because you achieve 100% coverage doesn’t mean that your system is bug free or perfect, it merely means you have some tests for every single method. And I think we all agree that writing test cases is hard and sometimes border conditions are hard to spot.

    But apart from that a high coverage is definitely something to strive for. The obvious reason is, of course, that all important (or more) parts are tested. But then there is also the fact that it makes refactoring easier. With a high coverage I can confidently refactor large portions of the code and if I happen to change important behaviour, the unit tests should flag that.

  3. Thibault Delor
    April 19th, 2011 at 09:07 | #3

    100% in code coverage is easy. 100% in test is impossible. There’s some thesis on the subject. The most rigorous way to test the code is to use form

  4. Thibault Delor
    April 19th, 2011 at 09:21 | #4

    100% in code coverage is easy. 100% in test is impossible, there’s some thesis on the subject. Some methods allow to limits the risks, like formal methods, which are often expensive and so not always relevant.
    To conclude I think that code coverage is just a measure that means nothing, As R.Martin say, it also depends on the quality of the test. Even more I really prefer a 15% code coverage, only on critical parts of the code, with the asserts and the relevant methods input…

  5. Alban Auzeill
    April 19th, 2011 at 10:16 | #5

    I can’t apply TDD and design a new solution at the same time. My brain is not wired for that. So I’m writing some test only if I’m satisfied by the design skeleton. And in this case, the only way I’ve found to ensure that all important things are tested, is a 100% test coverage. Because I can’t remember all important methods and all critical paths in my code. And yes, I do test getters and setters because I don’t know how to configure the coverage tool of sonar to ignore getters or setters that contains only one simple statement with the excepted field name. If someone knows how to configure the sonar coverage tool I would appreciate.
    I would like that the coverage tool can recognize some pattern. Then if a pattern matches exactly, the block of code can be ignored by the coverage tool. For example, ignore this:
    public \w+ get(\w+) \(\) {
                    return $1;
    }
    The main advantage of this behavior can be with the exceptions handling. For me it’s sometime a waste of time to test all the error paths. Especially if the exception handling code just used a proven and simple log statement, used everywhere.
    So with a provided list of well known pattern, a coverage tool can become enough intelligent to ask you to test only your new alien patterns and not the proven one.

  6. Marcus K.
    April 19th, 2011 at 22:16 | #6

    I think you just think in different spaces. Non-TDD programmers cannot see any benefit in the “100% code coverage” requirement. But real TDD developers like Robert C. Martin get 100% code coverage for free – they only write code to make their failing test pass. So Twitter is obviously not the right place for such controveral stuff ;-)

  7. April 19th, 2011 at 22:25 | #7

    @Marcus K.
    I’m sorry but even TDD developers do not test getters and setters: I’m sure of that!

    As for me, I wouldn’t qualify M. Martin as a developer, but as a theorician (this doesn’t mean he has no experience as a developer!). We need both… but the former category is here to deliver, whereas the latter has no such constraints.

  8. April 19th, 2011 at 23:30 | #8

    @Nicolas Frankel
    TDD developers might not *directly* test getters and setters but
    – they probably get 100% code coverage on getters and setters through the test of other methods
    – a change in the behavior of any getter/setter would probably imply a failing test somewhere
    Why would someone write a getter/setter if it was not to use it?

  9. Marcus K.
    April 19th, 2011 at 23:33 | #9

    @Nicolas Frankel
    Real TDD devs write getters (and setters) only when they really need them, so why not write a simple stupid test for that? It costs almost nothing and hopefully catches some clever dude that “optimizes” the getter “in the name of performance”.

    I understand your position. I wouldn’t call me a real TDD programmer (although I use this technique more and more). For me the important question is: What do you want the tests to do for you? If you just want to proof that in the end of development your code is correct, automated tests could be just waste. But if you want them to make your code more robust against too clever code refactorings, then they become valuable. Acceptance testing is something that is done at the end, usually and hopefully done once. Tests created while using real TDD enforce the resulting production code and thus imply 100% code coverage. I think that this is what Robert wants to tell – at least I hope so…

    Regarding your last statement: Delivery IS important, but it only makes sense for WORKING software. Code coverage alone is not the holy grail and can be simply cheated, but
    a) I would kick out every developer who tries to cheat, and
    b) the uncovered code parts are at least nice candidates for peer code reviews.
    For me the interesting thing is: If I use TDD, and Cobertura / Emma /whatever tell meh that I have untested code, then I must ask myself where I coded too much in advance. And that is also nice feedback :-)

  10. May 21st, 2011 at 10:29 | #10

    @Pablo

    I’ve been working on a new tool that aims to make mutation testing practical for real projects by being fast, scalable and user friendly.

    It’s available at

    http://pitest.org

    Any feedback welcomed.

  1. No trackbacks yet.