Friends of OpenJDK Today

Reassessing TestNG vs. Junit

October 19, 2021

Author(s)

  • Nicolas Frankel

    Nicolas is a developer advocate with 15+ years experience consulting for many different customers, in a wide range of contexts (such as telecoms, banking, insurances, large retail and public sector). ... Learn more

In a recent article on Spring, I advised reassessing one's opinion now and then as the IT world changes so fast. What was true a couple of years ago could be dead wrong nowadays, and you probably don't want to base your decisions on outdated data. This week, I'd like to follow my advice.

One of my first posts was advocating for TestNG vs. JUnit. In the post, I mentioned several features that JUnit lacked:

  • No parameterization
  • No grouping
  • No test method ordering

Since JUnit 5 has been out for some time already, let's check if it fixed those issues.

Parameterization

I wrote the initial post in 2008, and I think JUnit was available in version 3 at the time. Let's skip directly to version 4: JUnit did indeed offer parameterization. Here's a snippet from their wiki:

@RunWith(Parameterized.class)                               // 1
public class FibonacciTest {

    private int fInput;                                     // 2
    private int fExpected;                                  // 2

    public FibonacciTest(int input, int expected) {         // 3
        this.fInput = input;
        this.fExpected = expected;
    }

    @Parameters                                             // 4
    public static Collection<Object[]> data() {             // 5
        return Arrays.asList(new Object[][] {
            { 0, 0 }, { 1, 1 }, { 2, 1 }, { 3, 2 }, { 4, 3 }, { 5, 5 }, { 6, 8 }
        });
    }

    @Test
    public void test() {
        assertEquals(fExpected, Fibonacci.compute(fInput));
    }
}
  1. Set the runner
  2. Define an attribute for each test parameter
  3. Define a constructor with parameters for each parameter
  4. Annotate the parameters method
  5. Implement the parameters method. It must be static and return a Collection

Here's how you'd achieve the same with TestNG:

public class FibonacciTest {

    @DataProvider                                             // 1
    public Object[][] data() {                                // 2
        return new Object[][] {
            { 0, 0 }, { 1, 1 }, { 2, 1 }, { 3, 2 }, { 4, 3 }, { 5, 5 }, { 6, 8 }
        };
    }

    @Test(dataProvider = "data")                              // 3
    public void test(String input, String expected) {
        assertEquals(input, expected);
    }
}
  1. Annotate with @DataProvider
  2. Must return a Object[][], no need to be static
  3. @Test point to the data providing method - data

With version 5, JUnit offers the @ParamerizedTest annotation. Parameterized tests expect at least one parameter source, also specified by an annotation. Multiple sources are available:

Annotation Data
@ValueSource Primitives, String and Class
@ValueSource A dedicated enum
@MethodSource A specific method. The method needs to be static and return a Stream
@MethodSource Multi-valued data formatted as CSV
@CsvFileSource External third-party file
@ArgumentsSource Dedicated class that implements ArgumentsProvider

While TestNG's approach can address all use cases, JUnit 5 multiple configuration capabilities are more custom-tailored.

Grouping

Again, the initial post mentions that with JUnit 3, one cannot run only a subset of them. JUnit 4 provides two orthogonal ways to group tests. The first one is test suites:

public class A {

  @Test
  public void a() {}

  @Test
  public void a2() {}
}

public class B {

  @Test
  public void b() {}
}

@SuiteClasses( { A.class, B.class })
public class ABSuite {}

From that point, you can run ABSuite to run both A and B. For more fine-grained purposes, you can also use categories.

public interface Fast {}
public interface Slow {}

public class A {

  @Test
  public void a() {}

  @Category(Slow.class)
  @Test
  public void b() {}
}

@Category({Slow.class, Fast.class})
public class B {

  @Test
  public void c() {}
}

Here's how you can run only desired categories with Maven:

mvn test -Dtest.categories=Fast

TestNG is pretty similar to categories (or the other way around):

public class A {

  @Test
  public void a() {}

  @Test(groups = "slow")
  public void b() {}
}

@Test(groups = { "slow", "fast" })
public class B {

  @Test
  public void c() {}
}
mvn test -Dgroups=fast

JUnit 5 has redesigned its approach with the @Tag annotation. Tags are labels that you annotate your class with. Then you can filter the tags you want to run during test execution:

public class A {

  @Test
  public void a() {}

  @Test
  @Tag("slow")
  public void b() {}
}

@Tag("fast")
@Tag("slow")
public class B {

  @Test
  public void c() {}
}

Both frameworks implement similarly running a subset of tests.

Test method ordering

This point is the most debatable of all because JUnit stems from unit testing. In unit testing, tests need to be independent of one another. For this reason, you can run them in parallel.

Unfortunately, you'll probably need to implement some integration testing at one point or another. My go-to example is an e-commerce shop application. we want to test the checkout scenario with the following steps:

  1. Users can browse the product catalog
  2. They can put products in their cart
  3. They can go to the checkout page
  4. Finally, they can pay

Without test method ordering, you end up with a gigantic test method. If the test fails, it's impossible to know where it failed at first glance. You need to drill down and hope the log contains the relevant information.

With test method ordering, one can implement one method per step and order them accordingly. When any method fails, you know in which step it did - provided you gave the method a relevant name.

For this reason, JUnit was not suited for integration testing. It would not stand to reason to have JUnit for unit testing and TestNG for integration testing in a single project. Given that TestNG could do everything JUnit could, that's the most important reason why I favoured the former over the latter.

TestNG implements ordering by enforcing dependencies between test methods. It computes a directed acyclic graph of the dependent methods at runtime and runs the methods accordingly. Here's a sample relevant to the e-commerce above:

public class CheckoutIT {

    @Test
    public void browseCatalog() {}

    @Test(dependsOnMethods = { "browseCatalog" })
    public void addProduct() {}

    @Test(dependsOnMethods = { "addProduct" })
    public void checkout() {}

    @Test(dependsOnMethods = { "checkout" })
    public void pay() {}
}

JUnit 5 provides a couple of ways to implement test method ordering:

Annotation Order
Random No order
MethodName Alphanumerical method name
DisplayName Alphanumerical display name set by @DisplayName on each test method
OrderAnnotation Order set by @Order on each test method

One can implement the same e-commerce testing scenario like the following:

@TestMethodOrder(OrderAnnotation.class)              // 1
public class CheckoutIT {

    @Test
    @Order(1)                                        // 2
    public void browseCatalog() {}

    @Test
    @Order(2)                                        // 2
    public void addProduct() {}

    @Test
    @Order(3)                                        // 2
    public void checkout() {}

    @Test
    @Order(4)                                        // 2
    public void pay() {}
}
  1. Define the order based on @Order
  2. Set the order for each method

TestNG implements ordering via a DAG, JUnit directly. TestNG's approach is more flexible as it allows the runtime to run some methods in parallel, but JUnit gets the job done.

Conclusion

Up until now, I've favoured TestNG because of the poor parameterization design and, more importantly, the complete lack of ordering in JUnit. Version 5 of JUnit fixes both issues. Even more so, its implementation offers multiple configuration capabilities.

There are still areas where TestNG shines:

  • It has a richer lifecycle
  • For integration testing, its dependency-between-tests capabilities are a huge asset
  • Also, I don't particularly appreciate annotating my methods with static for JUnit's parameterized tests.

Yet, given that the JUnit ecosystem is much more developed, I think I'll switch to JUnit for new projects and reassess again in a few years.

To go further:

Originally published at A Java Geek on September 19th, 2021

Topics:

Related Articles

View All
  • Java Testing with VS Code

    In our last post, we talked about starting a new Java project and running and debugging it with VS Code. In this post, we will cover testing.

    To run Java tests on VS Code, we recommend using the Java Test Runner extension or the Java Extension Pack, which includes the extension. The extension supports the JUnit4, JUnit5, and TestNG frameworks.

    Read More
    Avatar photo
    March 15, 2021
  • Towards Continuous Performance Regression Testing

    JfrUnit is an extension for JUnit 5 which integrates Flight Recorder into unit tests.

    It makes it straightforward to initiate a JFR recording for a given set of event types, execute some test routine, and then assert the JFR events which should have been produced.

    Stay tuned for next parts in this series, where we’ll explore how to trace the SQL statements executed by an application using the JMC Agent and assert these query events using JfrUnit.

    Read More
    February 25, 2021
  • Avoiding NullPointerException

    The terrible NullPointerException (NPE for short) is the most frequent Java exception occurring in production, according to a 2016 study. In this article we’ll explore the main techniques to fight it: the self-validating model and the Optional wrapper.

    You should consider upgrading your entity model to either reject a null via self-validation or present the nullable field via a getter that returns Optional. The effort of changing the getters of the core entities in your app is considerable, but along the way, you may find many dormant NPEs.

    Read More
    December 22, 2020

Author(s)

  • Nicolas Frankel

    Nicolas is a developer advocate with 15+ years experience consulting for many different customers, in a wide range of contexts (such as telecoms, banking, insurances, large retail and public sector). ... Learn more

Comments (0)

Your email address will not be published. Required fields are marked *

Highlight your code snippets using [code lang="language name"] shortcode. Just insert your code between opening and closing tag: [code lang="java"] code [/code]. Or specify another language.

Save my name, email, and website in this browser for the next time I comment.

Subscribe to foojay updates:

https://foojay.io/feed/
Copied to the clipboard