Archive

Posts Tagged ‘quality’

Your code coverage metric is not meaningful

October 5th, 2014 1 comment

Last week, I had a heated but interesting Twitter debate about Code Coverage with my long-time friend (and sometimes squash partner) Freddy Mallet.

The essence of my point is the following: the Code Coverage metric that most quality-conscious software engineers cherish doesn’t guarantee anything. Thus, achieving 80% (or 100%) Code Coverage and bragging about it is just as useful as blowing in the wind. For sure, it’s quite hard to have a fact-based debate over Twitter, as 140 chars put a hard limit on any argument. This article is an attempt at writing down my arguments in a limitless space.

The uselessness of raw Code Coverage can be proved quite easily. Let’s have a simple example with the following to-be-tested class:

public class PassFilter {

    private int limit;

    public PassFilter(int limit) {
        this.limit = limit;
    }

    public boolean filter(int i) {
        return i < limit;
    }
}

This class is quite straightforward, there’s no need to comment. A possible test would be the following:

public class PassFilterTest {

    private PassFilter passFilterFive;

    @BeforeMethod
    protected void setUp() {
        passFilterFive = new PassFilter(5);
    }

    @Test
    public void should_pass_when_filtering_one() {
        boolean result = passFilterFive.filter(1);
    }

    @Test
    public void should_not_pass_when_filtering_ten() {
        boolean result = passFilterFive.filter(10);
    }
}

This test class will happily return 100% code coverage as well as 100% line coverage: executing the test will go through all the code’s lines and on both sides of its single branch. Isn’t life sweet? Too bad there are no assertions; they could have been “forgotten” on purpose by a contractor who couldn’t achieve the previously agreed-on code coverage metric. Let’s give the contractor the benefit of the doubt, and assume programmers are of good faith – and put assertions:

public class PassFilterTest {

    private PassFilter passFilterFive;

    @BeforeMethod
    protected void setUp() {
        passFilterFive = new PassFilter(5);
    }

    @Test
    public void should_pass_when_filtering_one() {
        boolean result = passFilterFive.filter(1);
        Assert.assertTrue(result);
    }

    @Test
    public void should_not_pass_when_filtering_ten() {
        boolean result = passFilterFive.filter(10);
        Assert.assertFalse(result);
    }
}

Still 100% code coverage and 100% line coverage – and this time, “real” assertions! But it still is no use… It is a well-know fact that developers tend to test for passing cases. In this case, the two cases use parameters of 1 and 10, while the potential bug is at the exact threshold of 5 (should the filter let it pass or not with this value?).

In conclusion, the raw Code Coverage only guarantees the maximum possible Code Coverage. If it’s 0%, of course, it will be 0%; however, with 80%, it gives you nothing… just the insurance that at most, your code is covered at 80%. But it can also be anything in between, 60%, 40%… or even 0%. What good is a metric that only hints at the maximum? In fact, in this light, Code Coverage is a Bloom Filter. IMHO, the only way to guarantee that test cases and the associated test coverage are really meaningful is to use Mutation Testing. For a basic introduction to Mutation Testing, please check my Introduction to Mutation Testing talk at JavaZone (10 minutes).

The good thing is that Mutation Testing is not some kind of academic paper only known by nerdy scientists, there tools in Java to start using it right now. Configuring PIT with the previous test will yield the following result:

This report pinpoints the remaining mutant to be killed and its associated line (line 12), and it’s easy to add the missing test case:

@Test
public void should_not_pass_when_filtering_five() {
    boolean result = passFilterFive.filter(5);
    Assert.assertFalse(result);
}

Now that we’ve demonstrated without doubt the value of Mutation Testing, what do we do? Some might tried a couple of arguments against Mutation Testing. Let’s have a review of each of them:

  • Mutation Testing takes a long time to be executed. For sure, the combination of all possible mutations takes much more longer than standard Unit Testing – which has to be be under 10 minutes. However, this is not a problem as Code Coverage is not a metric that has to be checked at each build: a nightly build is more than enough.
  • Mutation Testing takes a long time to analyze results. Also right… but what of the time to analyze Code Coverage results that, as seen before, only hint at the maximum possible Code Coverage?

On one hand, the raw Code Coverage metric is only relevant when too low – and requires further analysis when high. On the other hand, Mutation Testing lets you have confidence in the Code Coverage metric at no additional cost. The rest is up to you…

Send to Kindle
Categories: Development Tags: ,

Throwing a NullPointerException… or not

September 28th, 2014 7 comments

This week, I’ve lived again an experience from a few years ago, but in the opposite seat.

As a software architect/team leader/technical lead (select the term you’re more comfortable with), I was doing code reviews on an project we were working on and  I stumbled upon a code that looked like that:

public void someMethod(Type parameter) {
    if (parameter == null) {
        throw new NullPointerException("Parameter Type cannot be null");
    }
    // Rest of the method
}

I was horribly shocked! An applicative code throwing a NullPointerException, that was a big coding mistake. So I gently pointed to the developer that it was a bad idea and that I’d like him to throw an IllegalArgumentException instead, which exactly the exception type matching the use-case. This was very clear in my head, until the dev pointed me the NullPointerException‘s Javadoc. For simplicity’s sake, here’s the juicy part:

Thrown when an application attempts to use null in a case where an object is required. These include:

  • Calling the instance method of a null object.
  • Accessing or modifying the field of a null object.
  • Taking the length of null as if it were an array.
  • Accessing or modifying the slots of null as if it were an array.
  • Throwing null as if it were a Throwable value.

Applications should throw instances of this class to indicate other illegal uses of the null object.

Read the last sentence again: “Applications should throw instances of this class to indicate other illegal uses of the null object“. It seems to be legal for an application to throw an NPE, even recommended by the Javadocs.

In my previous case, I read it again and again… And ruled that yeah, it was OK for an application to throw an NPE. Back to today: a dev reviewed the pull request from another one, found out it threw an NPE and ruled it was bad practice. I took the side of the former dev and said it was OK.

What would be your ruling and more importantly, why?

Note: whatever the decision, this is not a big deal and probably not worth the time you spend bickering about it ;-)

Send to Kindle

Cost of correcting rule breaks

March 7th, 2011 3 comments

Last week, Simon Brandhof and Freddy Mallet of Sonar team and me were discussing Vaadin uses. At the end of our meeting, and although Vaadin finally did not fit their particular needs regarding Sonar, the talk disgressed toward quality.

As regular Sonar users know, the former includes a very interesting plugin aptly named Technical Debt plugin. As a reminder:

Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite… The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise

Ward Cunningham

So, this nice plugin monetarizes costs associated to a rule break in a project. It’s a long way from nothing, but still is only a one-sided argument. Let me explain: we, whether as architects or developers are technical-oriented people. The problem lies in that correcting rules breaks are not a decision made by us technicians. They are made by managers that tend to have a different mindset. I’m apologize to managers, but I will speak in their collective name: my observations so far led me to believe that managers think in terms of costs and risks (correct me if I’m wrong).

What’s good is that the aforementioned plugin assign a reimbursement cost to each rule break: it speaks to managers. They know what it will cost to correct the break. Cost is calculated from a simple formula: time to correct (including locating the right place, checking out, correcting the code itself, checking in) times salary cost. On the opposite side, and in order to take the right decision, managers have to know the costs associated with not correcting. There lies the rub: how can we provide that?

Now, us technical people, we frown at the breaking of a rule, either because it’s genetic, or from hard won experience. As for me, I made my part of crap when developing, but I learnt from it: for example, I can now understand why cycle dependencies is bad because I coded it and had to deal with its associated problems (uh, what do I compile first?). Nowadays, I can tell it’s bad, why it’s bad and do all I can to avoid it. I could even assign a cost to some specific cycles, but it will only be gut-feeling and likely to be challenged by any project manager who is evaluated only on planning and costs (and who don’t give a damn about maintenance).

Do you see the whole picture now? On one hand, we quantify the cost of quality with the technical debt concept and Sonar plugin, but on the other hand, it’s not possible to do the same for “won’t fix”. We can evangelize all we want about quality, there will be only agonizing slow improvement regarding quality as long as we cannot monetarize both costs.

Solutions? It would take some time, but we could create metrics, that will be refined with each project, and that could lead to some average cost for each rule break. Unluckily, that requires an already present quality-oriented mindset that is our target. Who said chicken and egg?

Send to Kindle
Categories: Development Tags: ,

Manage the quality of your projects with Sonar

January 10th, 2009 1 comment

Sonar is a free OpenSource product that provides you with a general dashboard displaying information about the codebase of each configured project, such as:

  • Number of failed tests,
  • % of code coverage,
  • % of rules compliance (more later),
  • % of duplicated lines,
  • and much more.

Read more…

Send to Kindle
Categories: Java Tags: , , ,