Archive

Posts Tagged ‘test’

Introduction to Mutation Testing

April 20th, 2014 2 comments

Last week, I took some days off to attend Devoxx France 2014 3rd edition. As for oysters, the largest talks do not necessarily contain the prettiest pearls. During this year’s edition, my revelation came from a 15 minutes talk by my friend Alexandre Victoor, who introduced me to the wonders of Mutation Testing. Since I’m currently writing about Integration Testing, I’m very much interested in Testing flavors I don’t know about.

Experience software developers know not to put too much faith in code coverage metrics. Reasons include:

  • Some asserts may have been forgotten (purposely or not)
  • Code with no value, such as getters and setters, may have been tested
  • And so on…

Mutation Testing tries to go beyond code coverage metrics to increase one’s faith in tests. Here’s is how this is achieved: random code changes called mutations are introduced in the tested code. If a test still succeed despite a code change, something is definitely fishy as the test is worth nothing. As an example is worth a thousand words, here is a snippet that needs to be tested:

public class DiscountEngine {

    public Double apply(Double discount, Double price) {

        return (1 - discount.doubleValue()) * price.doubleValue();
    }
}

The testing code would be akin to:

public class DiscountEngineTest {

    private DiscountEngine discounter;

    @BeforeMethod
    protected void setUp() {

        discounter = new DiscountEngine();
    }

    @Test
    public void should_apply_discount() {

        Double price = discounter.apply(new Double(0.5), new Double(10));

        assertEquals(price, new Double(5));
    }
}

Now, imagine line 16 was forgotten: results from DiscountEngineTest will still pass. In this case, however, wrong code updates in the DiscountEngine would not be detected. That’s were mutation testing enters the arena. By changing DiscountEngine, DiscountEngineTest will still pass and that would mean nothing is tested.

PIT is a Java tool offering mutation testing. In order to achieve this, PIT creates a number of alternate classes called mutants, where the un-mutated class is the initial source class. Those mutants will be tested against existing tests targeting the original class. If the test still pass, well, there’s a problem and the mutant is considered to have survived; if not, everything is fine as the mutant has been killed. For a single un-mutated class, this goes until the mutant gets killed or all tests targeting the class have been executed and it is still surviving.

Mutation testing in general and PIT in particular has a big disadvantage: the higher the number of mutants for a class, the higher the confidence in the results, but the higher the time required to execute tests. Therefore, it is advised to run Mutating Testing only on nightly builds. However, this cost is nothing in comparison to having trust in your tests again…

Out-of-the-box, PIT offers:

  • Maven integration
  • Ant integration
  • Command-line

Also, Alexandre has written a dedicated plugin for Sonar.

Source code for this article can be found in IntelliJ/Maven format there.

Send to Kindle
Categories: Java Tags: ,

Integration tests from the trenches

April 28th, 2013 No comments

This post is the written form of one of my submission for Devoxx France 2013. As it was only chosen as backup, I lacked the necessary motivation to prepare it. The subject is important though, so I finally decided to write it down.

In 2013, if you’re a standard developer, it is practically a given that you test your code. Whether you’re a practicioner of TDD or just create them afterwards, most realize a robust automated test harness is not optional but mandatory.

Unit Test vs Integration Test

There’s some implicit behind the notion of ‘test’ though. Depending on the person you ask, there may be some differences of what a test is. Let us provide two definitions to prevent potential future misunderstanding.

Unit test
A unit test tests a method in isolation. In this case, the unit is the method. Outside dependencies should be mocked to provide known input(s). This input can be set different values, including bordercase values, to check all code branches.
Integration test
An integration test tests the collaboration of two (or more) units. In this case, the unit is the class. This means that as soon as you want to check the execution result of more than one class, it is an integration test.

On one hand, achieving 100% UT sucess with 100% code coverage is not enough to guarantee a bug-free application. On the other hand, a complete IT harness is not feasible in the context of real-world project since it would cost too much. The consequence is that both are complementary.

Knowing the good IT & UT ratio is part of being a successful developer. It depends on many factors, including some outside the pure technical, such as the team skills and experience, application lifetime and so on.

Fail-fast tests

Most IT are usually inherently slower than UT, as they require some kind of initialization. For example, in modern-day applications, this directly translates to some DI framework bootstrap.

It is highly unlikely that IT be successful while UT fail. Therefore, in order to have the shortest feedback loop in case of a failure, it is a good idea to execute the fastest tests first, check if they fail, and only execute slowest ones then.

Maven provides the way to achieve this with the following plugins combination:

  1. maven-surefire-plugin to execute unit tests; convention is that their names end with Test
  2. maven-failsafe-plugin to execute integration tests; names end with IT

By default, the former is executed during the test phase of the Maven build lifecycle. Unfortunately, configuration of the latter is mandatory:


  ...
  
    
      
        maven-failsafe-plugin
        2.14.1
        
          
            
              integration-test
            
          
        
      
    
  

Note the Maven mojo is already bound to the integration-test phase, so that it is not necessary to configure it explicitly.

This way, UT – whose name pattern is *Test, are executed during the test phase while IT – whose name patter is *IT, are executed in the integration-test phase.

System under test boundaries

The first step before writing IT is to define the SUT boundaries. Inside lies everything that we can manage, outside the rest. For example, the database clearly belongs to the SUT because we can easily provide test data, whereas a web service does not.

Inside the SUT, mocking or initializing subsystems depend on the needed/wanted integration level:

  1. To UT a use-case, one could mock DAOs in the service class
  2. Alternatively, one could initialize a test database and use real world data. DBUnit is the framework to use in this case, providing ways to initialize data during set up and check results afterwards.

Doing both would let us test corner cases in the mocking approach (faster) and the standard case in the initialization one (slower).

Outside the SUT, there is an important decision to make regarding IT. If external dependencies are not up and stable most of the time, the build would break too often and throws too many red herrings. Those IT will probably we deactivated very quickly.

In order to still benefit from those IT but so that they do not get ignored (or even worse, removed), they should be put in a dedicated module outside the standard build. When using a continuous integration server, you should set two jobs: one for the standard build and one for those dependent IT.

In Maven, it is easily achieved by creating a module at the parent root and not referencing it in the parent POM. This way, running mvn integration-test will only launch tests that are in the standard modules, those that are reliable enough.

Fragile tests

As seen above, fragility in IT comes from dependency on external systems. Those fragile tests are best handled as a separate module, outside the standard build process.

However, another cause of fragility is unstability. And in any application, there’s a layer that is fundamentally unstable and that is the presentation layer. Whatever the chosen technology, this layer is exposed to end-users and you can be sure there will be many changes there. Whereas your service API are your own, GUI is subject to users whims, period.

IT that uses GUI, whether dedicated GUI tests or end-to-end tests should thus also be considered fragile and isolated in a dedicated module, as above.

My personal experience, coupled by some writings by Gojko Adzic, taught me to bypass the GUI layer and start my tests at the servlet layer. Whether you use  Spring Test provides a bunch of Fakes regarding the Servlet API.

Tests ordering

Developers unfamiliar with IT should probably be more than astounded by this section title. In fact, as UT should be context-independent and not be ordered in any case.

For IT, things are a little different: I like to define some UT as user stories. For example, user logs in then performs some action and then another. Those steps can of course be defined in the same test method.

The negative side of this, is that if the test fails, we have no way of easily knowing what went wrong and during which step. We could remedy to that by isolating each step in a single method and ordering those methods.

TestNG – a JUnit offshoot fully integrated in Maven and Spring, let us do that easily:

public class MyScenarioIT {

    @Test
    public void userLogsIn() {
        ...
    }
    @Test(dependsOnMethod = "userLogsIn")
    public void someAction() {
        ...
    }
    @Test(dependsOnMethod = "someAction")
    public void anotherAction() {
        ...
    }
}

This way, when an error occurs or an assert fails, log displays the faulty method: we just need to have adequate name for each method.

Framework specifics UT

Java EE in-container UT

Up until recently, automated UT were executed independently of any container. For Spring applications, this was no big deal but for Java EE intensive applications, you had to fake and mock dependencies. In the end, there was no guarantee really running the application inside the container would produce expected results.

Arquillian brought a paradigm shift, with the ability to produce UT for applications using Java EE. If you do not use it already, know that Arquillian let you automate creation of the desired archive and its deployment on one or more configured application servers.

Those can be either existing or downloaded, extracted and configured during UT execution. The former category would need to be segregated in a separate module as to prevent breaking the build while the latter can safely be assimilated to regular UT.

Spring UT

Spring Test provide a feature to assemble different Spring configuration fragments (either XML or Java classes). This means that by designing our application with enough modularity, we can use desired production-ready and tests-only configuration fragments.

As an example, by separating our data source and DAO in two different fragments, we can reuse the regular DAO fragment in UT and use a test fragment declaring an in-memory database. With Maven, this means having the former in src/main/resources and the latter in src/test/resources.

@ContextConfiguration(locations = {"classpath:datasource.xml", "classpath:dao.xml"})
public class MyCustomUT extends AbstractTestNGSpringContextTests {

    ...
}

Conclusion

This is the big picture regarding my personal experience regarding UT. As always, they shouldn’t be seen as hard and fast rules but be adapted to your specific project context.

However, tools listed in the article should be a real asset in all cases.

Send to Kindle

Re-use your test classes across different projects

August 25th, 2012 1 comment

Sometimes, you need to reuse your test classes across different projects. These are two use-cases that I know of:

  • Utility classes that create relevant domain objects used in different modules
  • Database test classes (ans resources) that need to be run in the persistence project as well as the integration test project

Since I’ve seen more than my share of misuses, this article aim to provide an elegant solution once and for all.

Creating the test artifact

First, we have to use Maven: I know that not everyone is a Maven fanboy, but it get the work done – and in our case, it does it easily. Then, we configure the JAR plugin to attach tests. This will compile test classes and copy test resources, and package them in an attached test artifact.


  
    
     
       org.apache.maven.plugins
       maven-jar-plugin
       2.2
       
         
           
             test-jar
           
         
       
     
    
  

The test artifact is stored side-by-side with the main artifact once deployed in the repository. Note that the configured test-jar is bound to the install goal.

Using the test artifact

The newly-created test artifact can be expressed as a dependency of a projet with the following snippet:


  ch.frankel.blog.foo
  foo
  1.0.0
  test-jar
  test

The type has to be test-jar instead of simply jar in order to Maven to pick the attached artifact and not the main one. Also, note although you could configure the dependency with a classifier instead of a type, the current documentation warns about possible bugs and favor the type configuration.

To go further:

Send to Kindle
Categories: Java Tags: ,

Database unit testing with DBUnit, Spring and TestNG

June 3rd, 2012 4 comments

I really like Spring, so I tend to use its features to the fullest. However, in some dark corners of its philosophy, I tend to disagree with some of its assumptions. One such assumption is the way database testing should work. In this article, I will explain how to configure your projects to make Spring Test and DBUnit play nice together in a multi-developers environment.

Context

My basic need is to be able to test some complex queries: before integration tests, I’ve to validate those queries get me the right results. These are not unit tests per se but let’s assilimate them as such. In order to achieve this, I use since a while a framework named DBUnit. Although not maintained since late 2010, I haven’t found yet a replacement (be my guest for proposals).

I also have some constraints:

  • I want to use TestNG for all my test classes, so that new developers wouldn’t think about which test framework to use
  • I want to be able to use Spring Test, so that I can inject my test dependencies directly into the test class
  • I want to be able to see for myself the database state at the end of any of my test, so that if something goes wrong, I can execute my own queries to discover why
  • I want every developer to have its own isolated database instance/schema

Considering the last point, our organization let us benefit from a single Oracle schema per developer for those “unit-tests”.

Basic set up

Spring provides the AbstractTestNGSpringContextTests class out-of-the-box. In turn, this means we can apply TestNG annotations as well as @Autowired on children classes. It also means we have access to the underlying applicationContext, but I prefer not to (and don’t need to in any case).

The structure of such a test would look like this:

@ContextConfiguration(location = "classpath:persistence-beans.xml")
public class MyDaoTest extends AbstractTestNGSpringContextTests {

    @Autowired
    private MyDao myDao;

    @Test
    public void whenXYZThenTUV() {
        ...
    }
}

Readers familiar with Spring and TestNG shouldn’t be surprised here.

Bringing in DBunit

DbUnit is a JUnit extension targeted at database-driven projects that, among other things, puts your database into a known state between test runs. […] DbUnit has the ability to export and import your database data to and from XML datasets. Since version 2.0, DbUnit can also work with very large datasets when used in streaming mode. DbUnit can also help you to verify that your database data match an expected set of values.

DBunit being a JUnit extension, it’s expected to extend the provided parent class org.dbunit.DBTestCase. In my context, I have to redefine some setup and teardown operation to use Spring inheritance hierarchy. Luckily, DBUnit developers thought about that and offer relevant documentation.

Among the different strategies available, my tastes tend toward the CLEAN_INSERT and NONE operations respectively on setup and teardown. This way, I can check the database state directly if my test fails. This updates my test class like so:

@ContextConfiguration(locations = {"classpath:persistence-beans.xml", "classpath:test-beans.xml"})
public class MyDaoTest extends AbstractTestNGSpringContextTests {

    @Autowired
    private MyDao myDao;

    @Autowired
    private IDatabaseTester databaseTester;

        @BeforeMethod
        protected void setUp() throws Exception {

            // Get the XML and set it on the databaseTester
            // Optional: get the DTD and set it on the databaseTester

            databaseTester.setSetUpOperation(DatabaseOperation.CLEAN_INSERT);
            databaseTester.setTearDownOperation(DatabaseOperation.NONE);
            databaseTester.onSetup();
        }

        @Test
        public void whenXYZThenTUV() {
            ...
    }
}

Per-user configuration with Spring

Of course, we need to have a specific Spring configuration file to inject the databaseTester. As an example, here is one:




        
            
        

        
             
             
             
             
        

        
            
        

However, there’s more than meets the eye. Notice the databaseTester has to be fed a datasource. Since a requirement is to have a database per developer, there are basically two options: either use a in-memory database or use the same database as in production and provide one such database schema per developer. I tend toward the latter solution (when possible) since it tends to decrease differences between the testing environment and the production environment.

Thus, in order for each developer to use its own schema, I use Spring’s ability to replace Java system properties at runtime: each developer is characterized by a different user.name. Then, I configure a PlaceholderConfigurer that looks for {user.name}.database.properties file, that will look like so:

db.username=myusername1
db.password=mypassword1
db.schema=myschema1

This let me achieve my goal of each developer using its own instance of Oracle. If you want to use this strategy, do not forget to provide a specific database.properties for the Continuous Integration server.

Huh oh?

Finally, the whole testing chain is configured up to the database tier. Yet, when the previous test is run, everything is fine (or not), but when checking the database, it looks untouched. Strangely enough, if you did load some XML dataset and assert it during the test, it does behaves accordingly: this bears all symptoms of a transaction issue. In fact, when you closely look at Spring’s documentation, everything becomes clear. Spring’s vision is that the database should be left untouched by running tests, in complete contradiction to DBUnit’s. It’s achieved by simply rollbacking all changes at the end of the test by default.

In order to change this behavior, the only thing to do is annotate the test class with @TransactionConfiguration(defaultRollback=false). Note this doesn’t prevent us from specifying specific methods that shouldn’t affect the database state on a case-by-case basis with the @Rollback annotation.

The test class becomes:

@ContextConfiguration(locations = {classpath:persistence-beans.xml", "classpath:test-beans.xml"})
@TransactionConfiguration(defaultRollback=false)
public class MyDaoTest extends AbstractTestNGSpringContextTests {

    @Autowired
    private MyDao myDao;

    @Autowired
    private IDatabaseTester databaseTester;

	@BeforeMethod
	protected void setUp() throws Exception {

		// Get the XML and set it on the databaseTester
		// Optional: get the DTD and set it on the databaseTester

        databaseTester.setSetUpOperation(DatabaseOperation.CLEAN_INSERT);
        databaseTester.setTearDownOperation(DatabaseOperation.NONE);
        databaseTester.onSetup();
    }

	@Test
	public void whenXYZThenTUV() {
		...
	}
}

Conclusion

Though Spring and DBUnit views on database testing are opposed, Spring’s configuration versatility let us make it fit our needs (and benefits from DI). Of course, other improvements are possible: pushing up common code in a parent test class, etc.

To go further:

Send to Kindle
Categories: Java Tags: , , ,

Specification by Example review

This review is about Specifications by Example by Gojko Adzic from Manning.

Facts

  • 18 chapters, 254 pages, $29.21
  • This book covers Specifications by Example (you could have guessed it from the title). In effect, SBE are a way to build the right software (for the customers), as opposed to build the software right (which is our trade as engineers).

Specification by Example is a set of process patterns that facilitate change in software products to ensure that the right product is delivered efficiently.

Pros

  • Not all methods are adequate for each context. Each hint described is made contextual.
  • Do’s are detailed all right, with their respective benefits, but dont’s are also characterized along with the reasons why you shouldn’t go down their path.
  • Full of real-life examples, with people and teams talking about what they did and why.
  • 6 case studies

Cons

  • Missing a tooling section to explain how to put these ideas in action.

Conclusion

As software engineers, we are on the lower part in the software creation chain and it shows. We focus on building the software right… Yet, we fail to address problems that find their origin in the upper part of the chain; agile methodologies are meant to prevent this. This book comes from a simple idea: in most projects, specifications, tests and software are all but synchronized. It lists a collection of agile recipes to remedy that.

Whether you’re a business analyst, a project manager or just a technical guy who wants to build better software, this book is for you. In any cases, you’ll want to keep it under arm’s reach.

Send to Kindle
Categories: Book review Tags: ,

100% code coverage!

April 18th, 2011 10 comments

The basis of this article is a sequences of tweets betwen me and M. Robert Martin, on April 8th 2011:

  1. If you have 100% coverage you don’t know if your system works, but you _do_ know that every line you wrote does what you thought it should.
  2. @unclebobmartin 100% code coverage doesn’t achieve anything, save making you safer while nothing could be further from the truth.
  3. @nicolas_frankel 100% code coverage isn’t an achievement, it’s a minimum requirement. If you write a line of code, you’d better test it.
  4. @unclebobmartin I can get 100% code coverage and test nothing because if have no assert. Stop making coverage a goal, it’s only a way!

This left me a little speechless, even more coming from someone as respected as M. Martin. Since the size of my arguments is well beyond the 140-characters limit, a full-fledged article is in order.

Code coverage doesn’t test anything

Let’s begin with the base assertion of M. Martin, namely that 100% code coverage ensures that every code line behaves as intended. Nothing could be further from the truth!

Code coverage is simply a way to trace where the flow passed during the execution of a test. Since coverage is achieved through instrumentation, code coverage could also be measured during the execution of standard applications.

Testing, however, is to execute some code by providing input and verifying the output is the desired one. In Java applications, this is done at the unit level with the help of frameworks like JUnit and TestNG. Both use the assert() method to check output.

Thus, code coverage and test are completely different things. Taken to the extreme, we could code-cover the entire application, that is achieve 100%, and not test a single line because we have no assertion!

100% coverage means test everything

Not all of our code is neither critical nor complex. In fact, some can even seen as downright trivial. Any doubt? Think about getters and setter then. Should they be tested, even though any IDE worth his salt can generate them faithfully for us? If your answer is yes, you should also consider testing the frameworks you use, because they are a bigger source of bugs than generated getters and setters.

Confusion between the goal and the way to achieve it

The second assertion of M. Martin is that 100% of code coverage is a requirement. I beg to differ, but I use the word ‘requirement’ with a different meaning. Business analysts translate business needs into functional requirements whereas architects translate them into non-functional requirements.

Code coverage and unit testing are just a way to decrease the likehood of bugs, they are not requirements. In fact, users couldn’t care less about code average, as long as the software meet their needs. Only IT, and only in the case of long lived applications, may have an interest in high code coverage, and even then not 100%.

Cost effectiveness

I understand that in certain particular contexts, software cannot fail: in chirurgy or aeronautics, lives are at stake whereas in financial applications, a single failure can cost millions or even billions.

However, what my meager experience taught me so far, it’s that money reigns, whether we want or not. The equation is very simple: what’s the cost of the test, what’s the cost of the bug and what’s the likehood of it. If the cost of the bug is a human life and the likehood is high, the application better be tested like hell. On the contrary, if the cost of the bug is half a man-day and the likehood low, why should we take 1 man-day to correct it in advance? The technical debt point of view help us to answer this question. Moreover, it’s a decision managers have to make, with the help of IT.

Point is, achieving 100% testing (and not 100% coverage) is overkill in most software.

For the sake of it

Last but not least, I’ve seen in my line of work some interesting deviant behaviours.

The first is being, quality for quality’s sake. Me being a technical person and all, I must admit I fell for it: “Wow, if I just test this, I can gain 1% code coverage!” Was it necessary? More importantly, did it increase maintainability or decrease the likehood of bugs? In all cases, you should ask yourself these questions. If both answers are no, forget about it.

A slightly different case is the challenge between teams. Whereas challenges are good in that they create an emulation and can make everyone better, the object of the challenge should bring some added-value, something the raw code coverage percentage doesn’t.

Finally, I already rambled about it, but some egos there just do things to write on their CV, 100% coverage is just one of them. This is bad and if you have any influence over such individuals, you should strive to orient them toward more project-oriented goal (such as 100% OK acceptance tests).

Conclusion

With the above arguments, I proved beyond doubt that assertions like ‘we must achieve 100% code coverage’ or ‘it’s a requirement’ cannot be taken as a general rule and is utter nonsense without some proper context.

As for myself, before beginning a project, one of the thing on my todo list is to agree on a certain level of code coverage with all concerned team members (mainly QA, PM and developers). Then, I take great care to explain that this metrics is not a hard-and-fast one and I list getters/setters and no assert as ways of artificially increasing it. The more critical and the more complex the code, the higher the coverage it should have. If offered a choice, I prefer not reaching the agreed upon number and yet having both firmly tested.

For teams to embrace quality is a worthy goal. It’s even more attainable if we set SMART objectives: 100% code coverage is only Measurable, which is why it’s better to forget it, the sooner the better, and focus on more business-oriented targets.

Send to Kindle
Categories: Development Tags: ,

Cons of custom assertion matchers

April 11th, 2011 2 comments

Ten years ago, the only tests we knew of were users acceptance tests. The last decade saw a gigantic leap forward: it brought unit testing. Unit testing was made popular with JUnit. In turn, TestNG added annotations to the test classes, making them ever easier. Then, EasyMock provided the means to mock our class dependencies in a test context while Mockito streamlined the process of doing so. One could think everything has been said and done for unit testing and we should move forward to a more worthy goal of our workforce.

I think nothing could be further from the truth. I do think that the next step will be making tests more maintainable. During my career, I had to maintain software that had no unit tests and that was bad. However, when I had to work on software with tests I had difficulties understanding, I cannot say for sure if it was better. Taking time making sure there’s no regression nor side-effect is one thing, taking time to debug test intricacies is another, and hardly expected by the management at that. One particular obstacle is in the assertion statement. I understand it may seem strange at first. How in the world could the following assertion be undecipherable?

Assert.assertFalse(myList.isEmpty());

Well, I infer that some people must have had enough of always writing the same assertions and provided some common API to test them. FEST and Hamcrest are example of such testing commodities that let us boost our productivity through the use of a DSL where the domain is assertion. Left it at that, I couldn’t agree more with their use and even promote it in my teams. However, Hamcrest also provides an API to create our own custom matcher: while it’s true that creating such matcher may be a help at first glance, I’ll try to raise some points against it.

First, what about maintainability? If a test fails after I made updates, is it because of my code, the matcher or a combination of both? I’ll have to either ask the original coder, but most of the times, I’ll be on my own to decide such things. This will likely involve sweat, toil and much debugging. Taken individually, the cost is not much, but it’s exponentially proportional to the number of matchers.

Then, in order to decrease the probability of a bug in a matcher, it will have to be tested on its own! Will their respective tests need assertion matchers? It seems like it will be turtles down to the bottom. Again, costs are bound to rise.

Moreover, we are taught in school that code duplication is bad (many tools provide this metric) and should be avoided at all costs. I can only support the first axiom, just as I have to condemn the second, and more particularly the ‘at all costs’ part. This isn’t true! Duplication is the first level of reuse, and I hope the previous arguments made my point regarding the cost of writing and maintaining custom matchers. For example, if we duplicate two lines three times, that’s not enough in my eyes to justify the added cost of a matcher!

Finally, it may seem strange but testing is not the goal of software, only a mean to an end (something I will have to write about in a next post). We’re not tasked to make our CV shine brighter, nor to partake in the highest code coverage challenge. We’re just paid to deliver software to our customers, while respecting the holy trinity of costs, delay and quality. If writing custom matchers can help us reach this goal, fine but whether it’s effectively the case is very contextual.

At the present, it seems advantageous to me to write custom matchers under the following circumstances:

  • There’s a real advantage in doing so. Search how many duplicated lines exist and whether it’s cost effective to factorize them in a matcher.
  • It won’t impact delivery. What happens if we’re in the middle of writing the best matcher ever and we’re ordered to ship the product? The customer doesn’t care about matchers.
  • Matchers shouldn’t ever be owned by a single developer but by the whole team. This prevents, or at least decreases the risk of having to debug a matcher to understand what its goal was.

In conclusion, I hope this article made you think about the whole custom matchers stuff. Next time you are about to create a matcher, think about it: it’s perhaps a good thing to do, but it’s not necessarily the case.

Send to Kindle
Categories: Java Tags:

Automate your integration tests

September 20th, 2010 3 comments

Software testing traditionally has been separated between unit testing – testing classes in isolation – and integration testing – testing across all layers. Whereas unit testing is the domain of developers, integration testing is the domain of analysts. Moreover, most of the time, those tests are not automated and are painfully reexecuted by hand each time they are needed. This means that your build process has a safeguard harness on the unit level but nothing on the feature level (at least nothing remotely automated).

In order to further streamline the build process, it would be order to also automate integration tests. I already have some successful experience with integration testing with batches. Batches are very simple to test in this way since they lack a very hard-to-test feature, namely GUI. I developed those tests with the TestNG framework, mainly because TestNG provides the way to order test methods.

Yet, those tests I was pleased with lack one essential feature: the analyst point-of-view. It’s me, as developer, that created them and as such, they may not adress the uses cases deemed critical to the business.

What’s essential is what lies between the description of the tested use cases in plain text and their implementations in Java. My TestNG tests were developed by the developer, for the developer, in Java and couldn’t be read by a typical business analyst. At the time, it was the best I could do but a thread inside my head longed to bridge the gap between developer and analyst points of view. Luckily, I have found (at least I think) the right tool in order to solve this.

This tool is JBehave, and as you can guess from its name, it advocates for Behaviour Driven Development. Its strongest point is that the business analyst is a the root of the process. Given a few simple rules, he can describe use cases in plain language. Then, the developer ties these stories, as they are called in JBehave, into the code. There are only a few rules for these descriptions:

  • describe the initial situation with the Given keyword
  • describe events with the When keyword
  • describe expectations with the Then keyword

Let’s take a simple example, as it would be written by a business analyst :

Given a student
Given a course
When the student registers in the course
Then the course must have the student as one of its student

Now, for each line, the developer creates a method and adds the right annotation to it:

public class AddCourseSteps {

    private Student student;

    private Course course;

    @Given("a student")
    public void aStudent() {

        this.student = new Student("dummy");
    }

    @Given("a course")
    public void aCourse(String label) {

        this.course = new Course("anotherDummy");
    }

    @When("the student registers in the course")
    public void heRegistersInACourse() {

        student.addCourse(course);
    }

    @Then("the course must have the student as one of its student")
    public void courseMustHaveStudent() {

        assertTrue(course.getStudents().contains(student));
    }
}

Notice how the “business” semantics is coded in the Java language. This example is of course trivial but higher-level stories would be treated just the same.

Running the story is just coding the test class in itself, which will tie the plain text description and the steps class and make the test class runnable in your favorite build tool and/or IDE. This wil produce the following trace:

Given a student
Given a course
When the student registers in the course
Then the course must have the student as one of its student (FAILED)

The FAILED part is of course only there is the test fails. You can object this is much ado about nothing, but only until now, because JBehave also let you parameterize your tests from the textual description, letting you write this:

Scenario: students are ranked according to marks

Given a course
Given a student named John Doe
Given a student named Jane Doe
When the first student obtains a A
When the second student obtains a B
Then first student is ranked 1 in the course
Then second student is ranked 2 in the course

Scenario: students are ranked according to marks or not ranked if they have no marks

Given a course
Given a student named John Doe
Given a student named Jane Doe
When the first student obtains a A
When the first student obtains a not_present
Then first student is ranked 1 in the course
Then second student is ranked not_ranked in the course

This can easily be written by business analysts.

To be frank, I didn’t use JBehave in any of my projects yet, but I intend to in the near future. I would rather be interested if anyone could give me feedback, mainly on v3.0. Otherwise, I encourage you to have a look at the JBehave site.

You can find the sources for the first example here in Maven/Eclipse format. I’ll let the second example as an exercise for the curious reader.

Send to Kindle
Categories: Java Tags: ,

Two different mocking approaches

April 22nd, 2009 2 comments

Whether you choose to choose to adopt TDD or a more traditional approach, you will test your classes in isolation. This usually mean you code with interfaces and inject your dependencies in your classes.

Last year, I had to test a servlet. Servlets are the showcase to demonstrate out-of-container testing because of their dependencies to said container in their doXxx() method. Every one of these methods has a dependency on HttpServletRequest and HttpServletResponse. Both are interfaces that have no concrete implementations in the Servlet API. So, basically, in your tests, you’re stuck with three options:

  1. Use your container implementations (yuck!),
  2. Create your own implementations (rather time-consuming),
  3. Use already available implementations that have no dependencies.

Choosing option the 3rd option, I found this little jewel, MockRunner. MockRunner put at your dispositions mock implementations for the following API and frameworks: Read more…

Send to Kindle
Categories: Java Tags: , , , ,