Arno# - The cutting edge of developer waffle

Phil Haack recently wrote about a paper published in the Proceedings of the IEEE Transactions on Software Engineering entitled “On the Effectiveness of Test-first Approach to Programming”. The paper reported the results of an experiment into the relative merits of the test-then-code approach over the code-then-test approach and proudly concluded:

We found that test-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed.

As Phil’s post explains, the results are not the end of the discussion, but they appear to provide the much requested evidence that TDD is a cost-effective and worthwhile tool that ought to be used when developing software.

However the conclusion above and the paper’s results don’t quite tie up. Jacob Proffitt writes an excellent critique of the paper and its conclusions. He points out that the results show better quality and productivity when unit tests are written after the code.

So does this mean that Test Driven Development isn’t all it’s cracked up to be? I’d argue no, it doesn’t mean that. Both Phil and Jacob seem to miss the fundamental advantage that TDD brings to an organisation: developers actually write unit tests when doing TDD. The idea of writing the code and then the tests is both an old idea and an idea thoroughly proven to never (or very rarely) happen in my view. The paper is fundamentally flawed and compares a “code after test” verses “test after code” strawman. A real-life experiment would need to compare “code after test” versus “code and run out of time for testing”.

When someone carries out studies into whether TDD leads to more unit tests being written and whether more unit tests leads to increased quality and productivity, we’ll know the answer for sure. Until then, the notion that TDD is better than no testing is a subjective, rather than empirical, conclusion. However most managers aren’t experts on the scientific method and so the distinction is likely to be lost on them. In my view at least, it is and will remain a perfectly sensible conclusion that should stand until someone objectively shows otherwise.