A Java geek

The basis of this article is a sequences of tweets betwen me and M. Robert Martin, on April 8th 2011:

1. If you have 100% coverage you don’t know if your system works, but you _do_ know that every line you wrote does what you thought it should.
2. @unclebobmartin 100% code coverage doesn’t achieve anything, save making you safer while nothing could be further from the truth.
3. @nicolas_frankel 100% code coverage isn’t an achievement, it’s a minimum requirement. If you write a line of code, you’d better test it.
4. @unclebobmartin I can get 100% code coverage and test nothing because if have no assert. Stop making coverage a goal, it’s only a way!

This left me a little speechless, even more coming from someone as respected as M. Martin. Since the size of my arguments is well beyond the 140-characters limit, a full-fledged article is in order.

Code coverage doesn’t test anything

Let’s begin with the base assertion of M. Martin, namely that 100% code coverage ensures that every code line behaves as intended. Nothing could be further from the truth!

Code coverage is simply a way to trace where the flow passed during the execution of a test. Since coverage is achieved through instrumentation, code coverage could also be measured during the execution of standard applications.

Testing, however, is to execute some code by providing input and verifying the output is the desired one. In Java applications, this is done at the unit level with the help of frameworks like JUnit and TestNG. Both use the assert() method to check output.

Thus, code coverage and test are completely different things. Taken to the extreme, we could code-cover the entire application, that is achieve 100%, and not test a single line because we have no assertion!

100% coverage means test everything

Not all of our code is neither critical nor complex. In fact, some can even seen as downright trivial. Any doubt? Think about getters and setter then. Should they be tested, even though any IDE worth his salt can generate them faithfully for us? If your answer is yes, you should also consider testing the frameworks you use, because they are a bigger source of bugs than generated getters and setters.

Confusion between the goal and the way to achieve it

The second assertion of M. Martin is that 100% of code coverage is a requirement. I beg to differ, but I use the word ‘requirement’ with a different meaning. Business analysts translate business needs into functional requirements whereas architects translate them into non-functional requirements.

Code coverage and unit testing are just a way to decrease the likehood of bugs, they are not requirements. In fact, users couldn’t care less about code average, as long as the software meet their needs. Only IT, and only in the case of long lived applications, may have an interest in high code coverage, and even then not 100%.

Cost effectiveness

I understand that in certain particular contexts, software cannot fail: in chirurgy or aeronautics, lives are at stake whereas in financial applications, a single failure can cost millions or even billions.

However, what my meager experience taught me so far, it’s that money reigns, whether we want or not. The equation is very simple: what’s the cost of the test, what’s the cost of the bug and what’s the likehood of it. If the cost of the bug is a human life and the likehood is high, the application better be tested like hell. On the contrary, if the cost of the bug is half a man-day and the likehood low, why should we take 1 man-day to correct it in advance? The technical debt point of view help us to answer this question. Moreover, it’s a decision managers have to make, with the help of IT.

Point is, achieving 100% testing (and not 100% coverage) is overkill in most software.

For the sake of it

Last but not least, I’ve seen in my line of work some interesting deviant behaviours.

The first is being, quality for quality’s sake. Me being a technical person and all, I must admit I fell for it: “Wow, if I just test this, I can gain 1% code coverage!” Was it necessary? More importantly, did it increase maintainability or decrease the likehood of bugs? In all cases, you should ask yourself these questions. If both answers are no, forget about it.

A slightly different case is the challenge between teams. Whereas challenges are good in that they create an emulation and can make everyone better, the object of the challenge should bring some added-value, something the raw code coverage percentage doesn’t.

Finally, I already rambled about it, but some egos there just do things to write on their CV, 100% coverage is just one of them. This is bad and if you have any influence over such individuals, you should strive to orient them toward more project-oriented goal (such as 100% OK acceptance tests).

Conclusion

With the above arguments, I proved beyond doubt that assertions like ‘we must achieve 100% code coverage’ or ‘it’s a requirement’ cannot be taken as a general rule and is utter nonsense without some proper context.

As for myself, before beginning a project, one of the thing on my todo list is to agree on a certain level of code coverage with all concerned team members (mainly QA, PM and developers). Then, I take great care to explain that this metrics is not a hard-and-fast one and I list getters/setters and no assert as ways of artificially increasing it. The more critical and the more complex the code, the higher the coverage it should have. If offered a choice, I prefer not reaching the agreed upon number and yet having both firmly tested.

For teams to embrace quality is a worthy goal. It’s even more attainable if we set SMART objectives: 100% code coverage is only Measurable, which is why it’s better to forget it, the sooner the better, and focus on more business-oriented targets.

Ten years ago, the only tests we knew of were users acceptance tests. The last decade saw a gigantic leap forward: it brought unit testing. Unit testing was made popular with JUnit. In turn, TestNG added annotations to the test classes, making them ever easier. Then, EasyMock provided the means to mock our class dependencies in a test context while Mockito streamlined the process of doing so. One could think everything has been said and done for unit testing and we should move forward to a more worthy goal of our workforce.

I think nothing could be further from the truth. I do think that the next step will be making tests more maintainable. During my career, I had to maintain software that had no unit tests and that was bad. However, when I had to work on software with tests I had difficulties understanding, I cannot say for sure if it was better. Taking time making sure there’s no regression nor side-effect is one thing, taking time to debug test intricacies is another, and hardly expected by the management at that. One particular obstacle is in the assertion statement. I understand it may seem strange at first. How in the world could the following assertion be undecipherable?

Assert.assertFalse(myList.isEmpty());

Well, I infer that some people must have had enough of always writing the same assertions and provided some common API to test them. FEST and Hamcrest are example of such testing commodities that let us boost our productivity through the use of a DSL where the domain is assertion. Left it at that, I couldn’t agree more with their use and even promote it in my teams. However, Hamcrest also provides an API to create our own custom matcher: while it’s true that creating such matcher may be a help at first glance, I’ll try to raise some points against it.

First, what about maintainability? If a test fails after I made updates, is it because of my code, the matcher or a combination of both? I’ll have to either ask the original coder, but most of the times, I’ll be on my own to decide such things. This will likely involve sweat, toil and much debugging. Taken individually, the cost is not much, but it’s exponentially proportional to the number of matchers.

Then, in order to decrease the probability of a bug in a matcher, it will have to be tested on its own! Will their respective tests need assertion matchers? It seems like it will be turtles down to the bottom. Again, costs are bound to rise.

Moreover, we are taught in school that code duplication is bad (many tools provide this metric) and should be avoided at all costs. I can only support the first axiom, just as I have to condemn the second, and more particularly the ‘at all costs’ part. This isn’t true! Duplication is the first level of reuse, and I hope the previous arguments made my point regarding the cost of writing and maintaining custom matchers. For example, if we duplicate two lines three times, that’s not enough in my eyes to justify the added cost of a matcher!

Finally, it may seem strange but testing is not the goal of software, only a mean to an end (something I will have to write about in a next post). We’re not tasked to make our CV shine brighter, nor to partake in the highest code coverage challenge. We’re just paid to deliver software to our customers, while respecting the holy trinity of costs, delay and quality. If writing custom matchers can help us reach this goal, fine but whether it’s effectively the case is very contextual.

At the present, it seems advantageous to me to write custom matchers under the following circumstances:

• There’s a real advantage in doing so. Search how many duplicated lines exist and whether it’s cost effective to factorize them in a matcher.
• It won’t impact delivery. What happens if we’re in the middle of writing the best matcher ever and we’re ordered to ship the product? The customer doesn’t care about matchers.
• Matchers shouldn’t ever be owned by a single developer but by the whole team. This prevents, or at least decreases the risk of having to debug a matcher to understand what its goal was.

In conclusion, I hope this article made you think about the whole custom matchers stuff. Next time you are about to create a matcher, think about it: it’s perhaps a good thing to do, but it’s not necessarily the case.

Software testing traditionally has been separated between unit testing – testing classes in isolation – and integration testing – testing across all layers. Whereas unit testing is the domain of developers, integration testing is the domain of analysts. Moreover, most of the time, those tests are not automated and are painfully reexecuted by hand each time they are needed. This means that your build process has a safeguard harness on the unit level but nothing on the feature level (at least nothing remotely automated).

In order to further streamline the build process, it would be order to also automate integration tests. I already have some successful experience with integration testing with batches. Batches are very simple to test in this way since they lack a very hard-to-test feature, namely GUI. I developed those tests with the TestNG framework, mainly because TestNG provides the way to order test methods.

Yet, those tests I was pleased with lack one essential feature: the analyst point-of-view. It’s me, as developer, that created them and as such, they may not adress the uses cases deemed critical to the business.

What’s essential is what lies between the description of the tested use cases in plain text and their implementations in Java. My TestNG tests were developed by the developer, for the developer, in Java and couldn’t be read by a typical business analyst. At the time, it was the best I could do but a thread inside my head longed to bridge the gap between developer and analyst points of view. Luckily, I have found (at least I think) the right tool in order to solve this.

This tool is JBehave, and as you can guess from its name, it advocates for Behaviour Driven Development. Its strongest point is that the business analyst is a the root of the process. Given a few simple rules, he can describe use cases in plain language. Then, the developer ties these stories, as they are called in JBehave, into the code. There are only a few rules for these descriptions:

• describe the initial situation with the Given keyword
• describe events with the When keyword
• describe expectations with the Then keyword

Let’s take a simple example, as it would be written by a business analyst :

Given a student
Given a course
When the student registers in the course
Then the course must have the student as one of its student

Now, for each line, the developer creates a method and adds the right annotation to it:

public class AddCourseSteps {

    private Student student;

    private Course course;

    @Given("a student")
    public void aStudent() {

        this.student = new Student("dummy");
    }

    @Given("a course")
    public void aCourse(String label) {

        this.course = new Course("anotherDummy");
    }

    @When("the student registers in the course")
    public void heRegistersInACourse() {

        student.addCourse(course);
    }

    @Then("the course must have the student as one of its student")
    public void courseMustHaveStudent() {

        assertTrue(course.getStudents().contains(student));
    }
}

Notice how the “business” semantics is coded in the Java language. This example is of course trivial but higher-level stories would be treated just the same.

Running the story is just coding the test class in itself, which will tie the plain text description and the steps class and make the test class runnable in your favorite build tool and/or IDE. This wil produce the following trace:

Given a student
Given a course
When the student registers in the course
Then the course must have the student as one of its student (FAILED)

The FAILED part is of course only there is the test fails. You can object this is much ado about nothing, but only until now, because JBehave also let you parameterize your tests from the textual description, letting you write this:

Scenario: students are ranked according to marks

Given a course
Given a student named John Doe
Given a student named Jane Doe
When the first student obtains a A
When the second student obtains a B
Then first student is ranked 1 in the course
Then second student is ranked 2 in the course

Scenario: students are ranked according to marks or not ranked if they have no marks

Given a course
Given a student named John Doe
Given a student named Jane Doe
When the first student obtains a A
When the first student obtains a not_present
Then first student is ranked 1 in the course
Then second student is ranked not_ranked in the course

This can easily be written by business analysts.

To be frank, I didn’t use JBehave in any of my projects yet, but I intend to in the near future. I would rather be interested if anyone could give me feedback, mainly on v3.0. Otherwise, I encourage you to have a look at the JBehave site.

You can find the sources for the first example here in Maven/Eclipse format. I’ll let the second example as an exercise for the curious reader.

Whether you choose to choose to adopt TDD or a more traditional approach, you will test your classes in isolation. This usually mean you code with interfaces and inject your dependencies in your classes.

Last year, I had to test a servlet. Servlets are the showcase to demonstrate out-of-container testing because of their dependencies to said container in their doXxx() method. Every one of these methods has a dependency on HttpServletRequest and HttpServletResponse. Both are interfaces that have no concrete implementations in the Servlet API. So, basically, in your tests, you’re stuck with three options:

1. Use your container implementations (yuck!),
2. Create your own implementations (rather time-consuming),
3. Use already available implementations that have no dependencies.

Choosing option the 3rd option, I found this little jewel, MockRunner. MockRunner put at your dispositions mock implementations for the following API and frameworks: Read more…