Using custom constraints to improve NUnit test quality, part 1 of 2

! Warning: this post hasn't been updated in over three years and so may contain out of date information.

Writing unit tests for your code is a laudable thing to do. Most developers who have tried using a test-orientated approach to development will agree that it is a superior approach to development, compared with not writing them. However, writing good tests can be hard. This two-part article explores the arguments for multiple asserts verses multiple single-assert tests and – having explained why neither is ideal – examines a powerful alternative that solves the problems with both approaches.

This article relates specifically to the.NET testing framework NUnit, though the techniques described should be transferable to other testing frameworks that support extending the framework. This first part however is relevant to all testing frameworks.

Testing with multiple asserts
Consider the following highly contrived code. It is written in a style that makes demonstrating the testing issues easier, whilst keeping the code simple. So please bear that in mind should you have “why has he written it that way?” thoughts whilst reading it.

CellState.cs:

Board.cs:

XMoveAllInOneTest.cs

As you can see, I have defined a simple CellState enumeration that defines the three states that the nine cells in a Tic Tac Toe (Noughts and Crosses) game can have: nothing drawn there yet, an X or an O. The Board class that holds the state of all nine cells and provides a method for setting a cell to an X. I’ve left off other code (setting the cell to an O for example,) to keep things simple.

The test is also simple (some might say simplistic) in its testing approach. It contains a single test with nine asserts (one per cell.) To understand why it’s a simplistic test strategy, let’s imagine that I got confused whilst writing the PlaceX method and it ended up like the following code:

When we run our tests, the second assert fails. There is however no indication that the third assert should fail too. That in a nutshell is the problem with multiple asserts in a test method: failing asserts are often hidden. If I remove the first line of the confused PlaceX method, then the second assert will pass and the third assert will now fail. In a real-world case, making one assert pass only to have the next fail can cause problems. Did the fix to one assert cause the next to fail, or was it already failing?

One of the key purposes of writing unit tests is to give developers confidence when refactoring code. Having asserts fail, without knowing which change caused the fail, quickly destroys that confidence. Avoiding multiple asserts in a test is therefore a really good idea if you care about keeping your code easy to refactor.

Testing with single asserts
With this in mind, consider the following reworking of the test file. Rather than having one test with nine asserts, we now have nine tests, each with a single assert:

XMoveSingleAssertionTests.cs

By constructing the test class this way, we end up with two failing tests, so we can straight away see that my barmy code change caused two problems. It is for this reason that tests with single asserts are favoured over tests with multiple asserts. However, there is an obvious disadvantage to this method too: we now have a much larger test class with nine near-identical tests. In more complex, real-world cases, this can be a serious problem. One can end up with many complex tests of near-identical code. This invites the dreaded use of copy and paste and so increases the chances of mistakes in the test class. You could end up testing the wrong this quite easily, leaving you with either tests that pass when they shouldn’t, or tests that fail even though the code is correct.

If neither testing strategy is ideal, is there an alternative? Yes there is and that is the subject of part two of this article.