Posts Tagged ‘test’

Creative Rx usage in testing setup

August 16th, 2015 No comments

I don’t know if events became part of software engineering since Graphical-User Interface interactions, but for sure they are a very convenient way of modeling them. With more and more interconnected systems, asynchronous event management has become an important issue to tackle. With Functional-Programming also on the raise, this gave birth to libraries such as RxJava. However, modeling a problem as the handling of a stream of events shouldn’t be restricted to system events handling. It can also be used in testing in many different ways.

One common use-case for testing setup is to launch a program, for example an external dependency such as a mock server. In this case, we need to wait until the program has been successfully launched. On the contrary, the test should stop as soon as the external program launch fails. If the program has a Java API, that’s easy. However, this is rarely the case and the more basics API are generally used, such as ProcessBuilder or Runtime.getRuntime().exec():

ProcessBuilder builder;

protected void setUp() throws IOException {
    builder = new ProcessBuilder().command("");
    process = builder.start();

protected void tearDown() {

The traditional way to handle this problem was to put a big Thread.sleep() just after the launch. Not only was it system dependent as the launch time changed from system to system, it didn’t tackle the case where the launch failed. In this later case, precious computing time as well as manual relaunch time were lost. Better solutions exist, but they involve a lot of lines of code as much at some (or a high) degree of complexity. Wouldn’t it be nice if we could have a simple and reliable way to start the program and depending on the output, either continue the setup or fail the test? Rx to the rescue!

The first step is to create an Observable around the launched process’s input stream:

Observable<String> observable = Observable.create(subscriber -> {
    InputStream stream = process.getInputStream();
    try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
        String line;
        while ((line = reader.readLine()) != null) {
    } catch (Exception e) {

In the above snippet:

  • Each time the process writes on the output, a next event is sent
  • When there is no more output, a complete event is sent
  • Finally, if an exception happens, it’s an error event

By Rx definition, any of the last two events mark the end of the sequence.

Once the observable has been created, it just needs to be observed for events. A simple script that emits a single event can be listened to with the following snippet:

BlockingObservable<String> blocking = observable.toBlocking();

The thing of interest here is the wrapping of the Observable instance in a BlockingObservable. While the former can be combined together, the later adds methods to manage events. At this point, the first() method will listen to the first (and single) event.

For a more complex script that emits a random number of regular events terminated by a single end event, a code snippet could be:

BlockingObservable<String> blocking = observable

In this case, whatever the number of regular events, the filter() method provides the way to listen to the only event we are interested in.

Previous cases do not reflect the reality, however. Most of the time, setup scripts should start before and run in parallel to the tests i.e. the end event is never sent – at least until after tests have finished. In this case, there are some threading involved. Rx let it handle that quite easily:

BlockingObservable<String> blocking = observable

There is a simple difference there: the subscriber will listen on a new thread thanks to the subscribeOn() method. Alternatively, events could have been emitted on another thread with the observeOn() method. Note I replaced the filter() method with take() to pretend to be interested only in the first 5 events.

At this point, the test setup is finished. Happy testing!

Sources for this article can be found here in IDEA/Maven format.

Send to Kindle
Categories: Java Tags: , ,

Improve your tests with Mockito’s capture

July 5th, 2015 No comments

Unit Testing mandates to test the unit in isolation. In order to achieve that, the general consensus is to design our classes in a decoupled way using DI. In this paradigm, whether using a framework or not, whether using compile-time or runtime compilation, object instantiation is the responsibility of dedicated factories. In particular, this means the new keyword should be used only in those factories.

Sometimes, however, having a dedicated factory just doesn’t fit. This is the case when injecting an narrow-scope instance into a wider scope instance. A use-case I stumbled upon recently concerns event bus, code like this one:

 public class Sample {

    private EventBus eventBus;

    public Sample(EventBus eventBus) {
        this.eventBus = eventBus;

    public void done() {
        Result result = computeResult() DoneEvent(result));

    private Result computeResult() {

With a runtime DI framework – such as the Spring framework, and if the DoneEvent had no argument, this could be changed to a lookup method pattern.

public void done() {;

public abstract DoneEvent getDoneEvent();

Unfortunately, the argument just prevents us to use this nifty trick. And it cannot be done with runtime injection anyway. It doesn’t mean the done() method shouldn’t be tested, though. The problem is not only how to assert that when the method is called, a new DoneEvent is posted in the bus, but also check the wrapped result.

Experienced software engineers probably know about the Mockito.any(Class) method. This could be used like this:

public void doneShouldPostDoneEvent() {
    EventBus eventBus = Mockito.mock(EventBus.class);
    Sample sample = new Sample(eventBus);

In this case, we make sure an event of the right kind has been posted to the queue, but we are not sure what the result was. And if the result cannot be asserted, the confidence in the code decreases. Mockito to the rescue. Mockito provides captures, that act like placeholders for parameters. The above code can be changed like this:

public void doneShouldPostDoneEventWithExpectedResult() {
    ArgumentCaptor<DoneEvent> captor = ArgumentCaptor.forClass(DoneEvent.class);
    EventBus eventBus = Mockito.mock(EventBus.class);
    Sample sample = new Sample(eventBus);
    DoneEvent event = captor.getCapture();
    assertThat(event.getResult(), is(expectedResult));

At line 2, we create a new ArgumentCaptor. At line 6, We replace any() usage with captor.capture() and the trick is done. The result is then captured by Mockito and available through captor.getCapture() at line 7. The final line – using Hamcrest, makes sure the result is the expected one.

Send to Kindle
Categories: Java Tags: ,

Final release of Integration Testing from the Trenches

March 1st, 2015 3 comments
Writing a book is a journey. At the beginning of the journey, you mostly know where you want to go, but have only vague notion of the way to get there and the time it will take. I’ve finally released the paperback version of on Amazon and that means this specific journey is at end.

The book starts by a very generic discussion about testing and continues by defining Integration Testing in comparison to Unit Testing. The next chapter compares the respective merits of Junit and TestNG. It is followed by complete description on how to make a design testable: what works for Unit Testing works also for Integration Testing. Testing in software relies on automation, so that specific usage of the Maven build tool is described in regard to Integration Testing – as well as Gradle. Dependencies on external resources make integration tests more fragile so faking those make them more robust. Those resources include: databases, the file system, SOAP and REST web services, etc. The most important dependency in any application is the container. The last chapters are dedicated to the Spring framework, including Spring MVC and Java EE.

In this journey, I also dared ask Josh Long of Spring fame and Aslak Knutsen, team lead of the Arquillian project to write a foreword to the book – and I’ve been delighted to have them both answer positively. Thank you guys!

I’ve also talked on the subject at some JUG and European conferences: JavaDay Kiev, Joker, Agile Tour London, and JUG Lyon and will again at JavaLand, DevIt, TopConf Romania and GeeCon. I hope that by doing so, Integration Testing will be used more effectively on projects and with bigger ROI.

Should you want to go further, the book is available in multiple formats:

  1. A paperback version on for $49.99
  2. Electronic versions for Mac, Kindle and plain old PDF on . The pricing here is more open, starting from $21.10 with a suggested price of $31.65. Note you can get it in all formats to read on all your devices.

If you’re already a reader and you like it, please feel free to recommend it. If you don’t, I welcome your feedback in the comments section. Of course, if neither – I encourage you to get a book and see for yourself!

Send to Kindle

Introduction to Mutation Testing

April 20th, 2014 2 comments

Last week, I took some days off to attend Devoxx France 2014 3rd edition. As for oysters, the largest talks do not necessarily contain the prettiest pearls. During this year’s edition, my revelation came from a 15 minutes talk by my friend Alexandre Victoor, who introduced me to the wonders of Mutation Testing. Since I’m currently writing about Integration Testing, I’m very much interested in Testing flavors I don’t know about.

Experience software developers know not to put too much faith in code coverage metrics. Reasons include:

  • Some asserts may have been forgotten (purposely or not)
  • Code with no value, such as getters and setters, may have been tested
  • And so on…

Mutation Testing tries to go beyond code coverage metrics to increase one’s faith in tests. Here’s is how this is achieved: random code changes called mutations are introduced in the tested code. If a test still succeed despite a code change, something is definitely fishy as the test is worth nothing. As an example is worth a thousand words, here is a snippet that needs to be tested:

public class DiscountEngine {

    public Double apply(Double discount, Double price) {

        return (1 - discount.doubleValue()) * price.doubleValue();

The testing code would be akin to:

public class DiscountEngineTest {

    private DiscountEngine discounter;

    protected void setUp() {

        discounter = new DiscountEngine();

    public void should_apply_discount() {

        Double price = discounter.apply(new Double(0.5), new Double(10));

        assertEquals(price, new Double(5));

Now, imagine line 16 was forgotten: results from DiscountEngineTest will still pass. In this case, however, wrong code updates in the DiscountEngine would not be detected. That’s were mutation testing enters the arena. By changing DiscountEngine, DiscountEngineTest will still pass and that would mean nothing is tested.

PIT is a Java tool offering mutation testing. In order to achieve this, PIT creates a number of alternate classes called mutants, where the un-mutated class is the initial source class. Those mutants will be tested against existing tests targeting the original class. If the test still pass, well, there’s a problem and the mutant is considered to have survived; if not, everything is fine as the mutant has been killed. For a single un-mutated class, this goes until the mutant gets killed or all tests targeting the class have been executed and it is still surviving.

Mutation testing in general and PIT in particular has a big disadvantage: the higher the number of mutants for a class, the higher the confidence in the results, but the higher the time required to execute tests. Therefore, it is advised to run Mutating Testing only on nightly builds. However, this cost is nothing in comparison to having trust in your tests again…

Out-of-the-box, PIT offers:

  • Maven integration
  • Ant integration
  • Command-line

Also, Alexandre has written a dedicated plugin for Sonar.

Source code for this article can be found in IntelliJ/Maven format there.

Send to Kindle
Categories: Java Tags: ,

Integration tests from the trenches

April 28th, 2013 No comments

This post is the written form of one of my submission for Devoxx France 2013. As it was only chosen as backup, I lacked the necessary motivation to prepare it. The subject is important though, so I finally decided to write it down.

In 2013, if you’re a standard developer, it is practically a given that you test your code. Whether you’re a practicioner of TDD or just create them afterwards, most realize a robust automated test harness is not optional but mandatory.

Unit Test vs Integration Test

There’s some implicit behind the notion of ‘test’ though. Depending on the person you ask, there may be some differences of what a test is. Let us provide two definitions to prevent potential future misunderstanding.

Unit test
A unit test tests a method in isolation. In this case, the unit is the method. Outside dependencies should be mocked to provide known input(s). This input can be set different values, including bordercase values, to check all code branches.
Integration test
An integration test tests the collaboration of two (or more) units. In this case, the unit is the class. This means that as soon as you want to check the execution result of more than one class, it is an integration test.

On one hand, achieving 100% UT sucess with 100% code coverage is not enough to guarantee a bug-free application. On the other hand, a complete IT harness is not feasible in the context of real-world project since it would cost too much. The consequence is that both are complementary.

Knowing the good IT & UT ratio is part of being a successful developer. It depends on many factors, including some outside the pure technical, such as the team skills and experience, application lifetime and so on.

Fail-fast tests

Most IT are usually inherently slower than UT, as they require some kind of initialization. For example, in modern-day applications, this directly translates to some DI framework bootstrap.

It is highly unlikely that IT be successful while UT fail. Therefore, in order to have the shortest feedback loop in case of a failure, it is a good idea to execute the fastest tests first, check if they fail, and only execute slowest ones then.

Maven provides the way to achieve this with the following plugins combination:

  1. maven-surefire-plugin to execute unit tests; convention is that their names end with Test
  2. maven-failsafe-plugin to execute integration tests; names end with IT

By default, the former is executed during the test phase of the Maven build lifecycle. Unfortunately, configuration of the latter is mandatory:


Note the Maven mojo is already bound to the integration-test phase, so that it is not necessary to configure it explicitly.

This way, UT – whose name pattern is *Test, are executed during the test phase while IT – whose name patter is *IT, are executed in the integration-test phase.

System under test boundaries

The first step before writing IT is to define the SUT boundaries. Inside lies everything that we can manage, outside the rest. For example, the database clearly belongs to the SUT because we can easily provide test data, whereas a web service does not.

Inside the SUT, mocking or initializing subsystems depend on the needed/wanted integration level:

  1. To UT a use-case, one could mock DAOs in the service class
  2. Alternatively, one could initialize a test database and use real world data. DBUnit is the framework to use in this case, providing ways to initialize data during set up and check results afterwards.

Doing both would let us test corner cases in the mocking approach (faster) and the standard case in the initialization one (slower).

Outside the SUT, there is an important decision to make regarding IT. If external dependencies are not up and stable most of the time, the build would break too often and throws too many red herrings. Those IT will probably we deactivated very quickly.

In order to still benefit from those IT but so that they do not get ignored (or even worse, removed), they should be put in a dedicated module outside the standard build. When using a continuous integration server, you should set two jobs: one for the standard build and one for those dependent IT.

In Maven, it is easily achieved by creating a module at the parent root and not referencing it in the parent POM. This way, running mvn integration-test will only launch tests that are in the standard modules, those that are reliable enough.

Fragile tests

As seen above, fragility in IT comes from dependency on external systems. Those fragile tests are best handled as a separate module, outside the standard build process.

However, another cause of fragility is unstability. And in any application, there’s a layer that is fundamentally unstable and that is the presentation layer. Whatever the chosen technology, this layer is exposed to end-users and you can be sure there will be many changes there. Whereas your service API are your own, GUI is subject to users whims, period.

IT that uses GUI, whether dedicated GUI tests or end-to-end tests should thus also be considered fragile and isolated in a dedicated module, as above.

My personal experience, coupled by some writings by Gojko Adzic, taught me to bypass the GUI layer and start my tests at the servlet layer. Whether you use  Spring Test provides a bunch of Fakes regarding the Servlet API.

Tests ordering

Developers unfamiliar with IT should probably be more than astounded by this section title. In fact, as UT should be context-independent and not be ordered in any case.

For IT, things are a little different: I like to define some UT as user stories. For example, user logs in then performs some action and then another. Those steps can of course be defined in the same test method.

The negative side of this, is that if the test fails, we have no way of easily knowing what went wrong and during which step. We could remedy to that by isolating each step in a single method and ordering those methods.

TestNG – a JUnit offshoot fully integrated in Maven and Spring, let us do that easily:

public class MyScenarioIT {

    public void userLogsIn() {
    @Test(dependsOnMethod = "userLogsIn")
    public void someAction() {
    @Test(dependsOnMethod = "someAction")
    public void anotherAction() {

This way, when an error occurs or an assert fails, log displays the faulty method: we just need to have adequate name for each method.

Framework specifics UT

Java EE in-container UT

Up until recently, automated UT were executed independently of any container. For Spring applications, this was no big deal but for Java EE intensive applications, you had to fake and mock dependencies. In the end, there was no guarantee really running the application inside the container would produce expected results.

Arquillian brought a paradigm shift, with the ability to produce UT for applications using Java EE. If you do not use it already, know that Arquillian let you automate creation of the desired archive and its deployment on one or more configured application servers.

Those can be either existing or downloaded, extracted and configured during UT execution. The former category would need to be segregated in a separate module as to prevent breaking the build while the latter can safely be assimilated to regular UT.

Spring UT

Spring Test provide a feature to assemble different Spring configuration fragments (either XML or Java classes). This means that by designing our application with enough modularity, we can use desired production-ready and tests-only configuration fragments.

As an example, by separating our data source and DAO in two different fragments, we can reuse the regular DAO fragment in UT and use a test fragment declaring an in-memory database. With Maven, this means having the former in src/main/resources and the latter in src/test/resources.

@ContextConfiguration(locations = {"classpath:datasource.xml", "classpath:dao.xml"})
public class MyCustomUT extends AbstractTestNGSpringContextTests {



This is the big picture regarding my personal experience regarding UT. As always, they shouldn’t be seen as hard and fast rules but be adapted to your specific project context.

However, tools listed in the article should be a real asset in all cases.

Send to Kindle

Re-use your test classes across different projects

August 25th, 2012 1 comment

Sometimes, you need to reuse your test classes across different projects. These are two use-cases that I know of:

  • Utility classes that create relevant domain objects used in different modules
  • Database test classes (ans resources) that need to be run in the persistence project as well as the integration test project

Since I’ve seen more than my share of misuses, this article aim to provide an elegant solution once and for all.

Creating the test artifact

First, we have to use Maven: I know that not everyone is a Maven fanboy, but it get the work done – and in our case, it does it easily. Then, we configure the JAR plugin to attach tests. This will compile test classes and copy test resources, and package them in an attached test artifact.


The test artifact is stored side-by-side with the main artifact once deployed in the repository. Note that the configured test-jar is bound to the install goal.

Using the test artifact

The newly-created test artifact can be expressed as a dependency of a projet with the following snippet:

The type has to be test-jar instead of simply jar in order to Maven to pick the attached artifact and not the main one. Also, note although you could configure the dependency with a classifier instead of a type, the current documentation warns about possible bugs and favor the type configuration.

To go further:

Send to Kindle
Categories: Java Tags: ,

Database unit testing with DBUnit, Spring and TestNG

June 3rd, 2012 4 comments

I really like Spring, so I tend to use its features to the fullest. However, in some dark corners of its philosophy, I tend to disagree with some of its assumptions. One such assumption is the way database testing should work. In this article, I will explain how to configure your projects to make Spring Test and DBUnit play nice together in a multi-developers environment.


My basic need is to be able to test some complex queries: before integration tests, I’ve to validate those queries get me the right results. These are not unit tests per se but let’s assilimate them as such. In order to achieve this, I use since a while a framework named DBUnit. Although not maintained since late 2010, I haven’t found yet a replacement (be my guest for proposals).

I also have some constraints:

  • I want to use TestNG for all my test classes, so that new developers wouldn’t think about which test framework to use
  • I want to be able to use Spring Test, so that I can inject my test dependencies directly into the test class
  • I want to be able to see for myself the database state at the end of any of my test, so that if something goes wrong, I can execute my own queries to discover why
  • I want every developer to have its own isolated database instance/schema

Considering the last point, our organization let us benefit from a single Oracle schema per developer for those “unit-tests”.

Basic set up

Spring provides the AbstractTestNGSpringContextTests class out-of-the-box. In turn, this means we can apply TestNG annotations as well as @Autowired on children classes. It also means we have access to the underlying applicationContext, but I prefer not to (and don’t need to in any case).

The structure of such a test would look like this:

@ContextConfiguration(location = "classpath:persistence-beans.xml")
public class MyDaoTest extends AbstractTestNGSpringContextTests {

    private MyDao myDao;

    public void whenXYZThenTUV() {

Readers familiar with Spring and TestNG shouldn’t be surprised here.

Bringing in DBunit

DbUnit is a JUnit extension targeted at database-driven projects that, among other things, puts your database into a known state between test runs. […] DbUnit has the ability to export and import your database data to and from XML datasets. Since version 2.0, DbUnit can also work with very large datasets when used in streaming mode. DbUnit can also help you to verify that your database data match an expected set of values.

DBunit being a JUnit extension, it’s expected to extend the provided parent class org.dbunit.DBTestCase. In my context, I have to redefine some setup and teardown operation to use Spring inheritance hierarchy. Luckily, DBUnit developers thought about that and offer relevant documentation.

Among the different strategies available, my tastes tend toward the CLEAN_INSERT and NONE operations respectively on setup and teardown. This way, I can check the database state directly if my test fails. This updates my test class like so:

@ContextConfiguration(locations = {"classpath:persistence-beans.xml", "classpath:test-beans.xml"})
public class MyDaoTest extends AbstractTestNGSpringContextTests {

    private MyDao myDao;

    private IDatabaseTester databaseTester;

        protected void setUp() throws Exception {

            // Get the XML and set it on the databaseTester
            // Optional: get the DTD and set it on the databaseTester


        public void whenXYZThenTUV() {

Per-user configuration with Spring

Of course, we need to have a specific Spring configuration file to inject the databaseTester. As an example, here is one:


However, there’s more than meets the eye. Notice the databaseTester has to be fed a datasource. Since a requirement is to have a database per developer, there are basically two options: either use a in-memory database or use the same database as in production and provide one such database schema per developer. I tend toward the latter solution (when possible) since it tends to decrease differences between the testing environment and the production environment.

Thus, in order for each developer to use its own schema, I use Spring’s ability to replace Java system properties at runtime: each developer is characterized by a different Then, I configure a PlaceholderConfigurer that looks for {} file, that will look like so:


This let me achieve my goal of each developer using its own instance of Oracle. If you want to use this strategy, do not forget to provide a specific for the Continuous Integration server.

Huh oh?

Finally, the whole testing chain is configured up to the database tier. Yet, when the previous test is run, everything is fine (or not), but when checking the database, it looks untouched. Strangely enough, if you did load some XML dataset and assert it during the test, it does behaves accordingly: this bears all symptoms of a transaction issue. In fact, when you closely look at Spring’s documentation, everything becomes clear. Spring’s vision is that the database should be left untouched by running tests, in complete contradiction to DBUnit’s. It’s achieved by simply rollbacking all changes at the end of the test by default.

In order to change this behavior, the only thing to do is annotate the test class with @TransactionConfiguration(defaultRollback=false). Note this doesn’t prevent us from specifying specific methods that shouldn’t affect the database state on a case-by-case basis with the @Rollback annotation.

The test class becomes:

@ContextConfiguration(locations = {classpath:persistence-beans.xml", "classpath:test-beans.xml"})
public class MyDaoTest extends AbstractTestNGSpringContextTests {
private MyDao myDao;
private IDatabaseTester databaseTester;
protected void setUp() throws Exception {
// Get the XML and set it on the databaseTester
// Optional: get the DTD and set it on the databaseTester
public void whenXYZThenTUV() {


Though Spring and DBUnit views on database testing are opposed, Spring’s configuration versatility let us make it fit our needs (and benefits from DI). Of course, other improvements are possible: pushing up common code in a parent test class, etc.

To go further:

Send to Kindle
Categories: Java Tags: , , ,

Specification by Example review

This review is about Specifications by Example by Gojko Adzic from Manning.


  • 18 chapters, 254 pages, $29.21
  • This book covers Specifications by Example (you could have guessed it from the title). In effect, SBE are a way to build the right software (for the customers), as opposed to build the software right (which is our trade as engineers).

Specification by Example is a set of process patterns that facilitate change in software products to ensure that the right product is delivered efficiently.


  • Not all methods are adequate for each context. Each hint described is made contextual.
  • Do’s are detailed all right, with their respective benefits, but dont’s are also characterized along with the reasons why you shouldn’t go down their path.
  • Full of real-life examples, with people and teams talking about what they did and why.
  • 6 case studies


  • Missing a tooling section to explain how to put these ideas in action.


As software engineers, we are on the lower part in the software creation chain and it shows. We focus on building the software right… Yet, we fail to address problems that find their origin in the upper part of the chain; agile methodologies are meant to prevent this. This book comes from a simple idea: in most projects, specifications, tests and software are all but synchronized. It lists a collection of agile recipes to remedy that.

Whether you’re a business analyst, a project manager or just a technical guy who wants to build better software, this book is for you. In any cases, you’ll want to keep it under arm’s reach.

Send to Kindle
Categories: Book review Tags: ,

100% code coverage!

April 18th, 2011 10 comments

The basis of this article is a sequences of tweets betwen me and M. Robert Martin, on April 8th 2011:

  1. If you have 100% coverage you don’t know if your system works, but you _do_ know that every line you wrote does what you thought it should.
  2. @unclebobmartin 100% code coverage doesn’t achieve anything, save making you safer while nothing could be further from the truth.
  3. @nicolas_frankel 100% code coverage isn’t an achievement, it’s a minimum requirement. If you write a line of code, you’d better test it.
  4. @unclebobmartin I can get 100% code coverage and test nothing because if have no assert. Stop making coverage a goal, it’s only a way!

This left me a little speechless, even more coming from someone as respected as M. Martin. Since the size of my arguments is well beyond the 140-characters limit, a full-fledged article is in order.

Code coverage doesn’t test anything

Let’s begin with the base assertion of M. Martin, namely that 100% code coverage ensures that every code line behaves as intended. Nothing could be further from the truth!

Code coverage is simply a way to trace where the flow passed during the execution of a test. Since coverage is achieved through instrumentation, code coverage could also be measured during the execution of standard applications.

Testing, however, is to execute some code by providing input and verifying the output is the desired one. In Java applications, this is done at the unit level with the help of frameworks like JUnit and TestNG. Both use the assert() method to check output.

Thus, code coverage and test are completely different things. Taken to the extreme, we could code-cover the entire application, that is achieve 100%, and not test a single line because we have no assertion!

100% coverage means test everything

Not all of our code is neither critical nor complex. In fact, some can even seen as downright trivial. Any doubt? Think about getters and setter then. Should they be tested, even though any IDE worth his salt can generate them faithfully for us? If your answer is yes, you should also consider testing the frameworks you use, because they are a bigger source of bugs than generated getters and setters.

Confusion between the goal and the way to achieve it

The second assertion of M. Martin is that 100% of code coverage is a requirement. I beg to differ, but I use the word ‘requirement’ with a different meaning. Business analysts translate business needs into functional requirements whereas architects translate them into non-functional requirements.

Code coverage and unit testing are just a way to decrease the likehood of bugs, they are not requirements. In fact, users couldn’t care less about code average, as long as the software meet their needs. Only IT, and only in the case of long lived applications, may have an interest in high code coverage, and even then not 100%.

Cost effectiveness

I understand that in certain particular contexts, software cannot fail: in chirurgy or aeronautics, lives are at stake whereas in financial applications, a single failure can cost millions or even billions.

However, what my meager experience taught me so far, it’s that money reigns, whether we want or not. The equation is very simple: what’s the cost of the test, what’s the cost of the bug and what’s the likehood of it. If the cost of the bug is a human life and the likehood is high, the application better be tested like hell. On the contrary, if the cost of the bug is half a man-day and the likehood low, why should we take 1 man-day to correct it in advance? The technical debt point of view help us to answer this question. Moreover, it’s a decision managers have to make, with the help of IT.

Point is, achieving 100% testing (and not 100% coverage) is overkill in most software.

For the sake of it

Last but not least, I’ve seen in my line of work some interesting deviant behaviours.

The first is being, quality for quality’s sake. Me being a technical person and all, I must admit I fell for it: “Wow, if I just test this, I can gain 1% code coverage!” Was it necessary? More importantly, did it increase maintainability or decrease the likehood of bugs? In all cases, you should ask yourself these questions. If both answers are no, forget about it.

A slightly different case is the challenge between teams. Whereas challenges are good in that they create an emulation and can make everyone better, the object of the challenge should bring some added-value, something the raw code coverage percentage doesn’t.

Finally, I already rambled about it, but some egos there just do things to write on their CV, 100% coverage is just one of them. This is bad and if you have any influence over such individuals, you should strive to orient them toward more project-oriented goal (such as 100% OK acceptance tests).


With the above arguments, I proved beyond doubt that assertions like ‘we must achieve 100% code coverage’ or ‘it’s a requirement’ cannot be taken as a general rule and is utter nonsense without some proper context.

As for myself, before beginning a project, one of the thing on my todo list is to agree on a certain level of code coverage with all concerned team members (mainly QA, PM and developers). Then, I take great care to explain that this metrics is not a hard-and-fast one and I list getters/setters and no assert as ways of artificially increasing it. The more critical and the more complex the code, the higher the coverage it should have. If offered a choice, I prefer not reaching the agreed upon number and yet having both firmly tested.

For teams to embrace quality is a worthy goal. It’s even more attainable if we set SMART objectives: 100% code coverage is only Measurable, which is why it’s better to forget it, the sooner the better, and focus on more business-oriented targets.

Send to Kindle
Categories: Development Tags: ,

Cons of custom assertion matchers

April 11th, 2011 2 comments

Ten years ago, the only tests we knew of were users acceptance tests. The last decade saw a gigantic leap forward: it brought unit testing. Unit testing was made popular with JUnit. In turn, TestNG added annotations to the test classes, making them ever easier. Then, EasyMock provided the means to mock our class dependencies in a test context while Mockito streamlined the process of doing so. One could think everything has been said and done for unit testing and we should move forward to a more worthy goal of our workforce.

I think nothing could be further from the truth. I do think that the next step will be making tests more maintainable. During my career, I had to maintain software that had no unit tests and that was bad. However, when I had to work on software with tests I had difficulties understanding, I cannot say for sure if it was better. Taking time making sure there’s no regression nor side-effect is one thing, taking time to debug test intricacies is another, and hardly expected by the management at that. One particular obstacle is in the assertion statement. I understand it may seem strange at first. How in the world could the following assertion be undecipherable?


Well, I infer that some people must have had enough of always writing the same assertions and provided some common API to test them. FEST and Hamcrest are example of such testing commodities that let us boost our productivity through the use of a DSL where the domain is assertion. Left it at that, I couldn’t agree more with their use and even promote it in my teams. However, Hamcrest also provides an API to create our own custom matcher: while it’s true that creating such matcher may be a help at first glance, I’ll try to raise some points against it.

First, what about maintainability? If a test fails after I made updates, is it because of my code, the matcher or a combination of both? I’ll have to either ask the original coder, but most of the times, I’ll be on my own to decide such things. This will likely involve sweat, toil and much debugging. Taken individually, the cost is not much, but it’s exponentially proportional to the number of matchers.

Then, in order to decrease the probability of a bug in a matcher, it will have to be tested on its own! Will their respective tests need assertion matchers? It seems like it will be turtles down to the bottom. Again, costs are bound to rise.

Moreover, we are taught in school that code duplication is bad (many tools provide this metric) and should be avoided at all costs. I can only support the first axiom, just as I have to condemn the second, and more particularly the ‘at all costs’ part. This isn’t true! Duplication is the first level of reuse, and I hope the previous arguments made my point regarding the cost of writing and maintaining custom matchers. For example, if we duplicate two lines three times, that’s not enough in my eyes to justify the added cost of a matcher!

Finally, it may seem strange but testing is not the goal of software, only a mean to an end (something I will have to write about in a next post). We’re not tasked to make our CV shine brighter, nor to partake in the highest code coverage challenge. We’re just paid to deliver software to our customers, while respecting the holy trinity of costs, delay and quality. If writing custom matchers can help us reach this goal, fine but whether it’s effectively the case is very contextual.

At the present, it seems advantageous to me to write custom matchers under the following circumstances:

  • There’s a real advantage in doing so. Search how many duplicated lines exist and whether it’s cost effective to factorize them in a matcher.
  • It won’t impact delivery. What happens if we’re in the middle of writing the best matcher ever and we’re ordered to ship the product? The customer doesn’t care about matchers.
  • Matchers shouldn’t ever be owned by a single developer but by the whole team. This prevents, or at least decreases the risk of having to debug a matcher to understand what its goal was.

In conclusion, I hope this article made you think about the whole custom matchers stuff. Next time you are about to create a matcher, think about it: it’s perhaps a good thing to do, but it’s not necessarily the case.

Send to Kindle
Categories: Java Tags: