Advanced Test Harness Features

In a past post I talked about the basic functionality of a test harness.  That is, it should be something that provides a reusable framework for an application, for running and reporting the results of test cases.  There is much more that a test harness can do, however.  It can provide mechanisms for lightweight test development, model based testing, and scripting.

Basic test harnesses like the Shell98 I spoke of last time or cppunit are what I would call heavyweight harnesses.  By that I mean that the testing code is statically bound to the harness at compile time.  In the most basic form the harness comes in the form of source code that is compiled along with the test code.  A slightly more advanced model involves a separate library that is statically linked with the test code to form a standalone executable.  This is fine but it means longer compile times, larger binaries, and potentially less flexibility.

There is a better way to do this.  The test harness and test cases can be separated.  The harness is compiled into an executable and the test cases are loaded by it dynamically.  I call this a lightweight harness.  As the harness no longer knows what test cases it will be tasked with executing, this requires that the tests are discoverable in some manner by the test harness.  The test cases are usually collected in dlls, jar files, or assemblies which are loaded by the harness.  The harness uses reflection (C# or Java) or a custom interface to discover which tests are in the test file.  This system is much more complex than the static binding but it offers several advantages.  It separates the test cases from the harness, allowing them to be varied independently.  It decreases compile times and reduces binary size.  It can also allow a single instance of a test harness to run many different types of tests.  With a static model, this scenario would require running many different executables.  Nunit is an example of a lightweight test harness.

Another feature that advanced test harnesses will have is support for model based testing.  Model based testing is a method of testing where test cases are generated automatically by the test framework based on a well-defined finite state machine.  A good description can be found on Nihit Kaul’s blog.  A test harness which supports model based testing will provide mechanisms for defining states and state transitions (actions) which it will use to generate and execute test cases.  The harness will also need to support a mechanism for verifying that the system is still correct after each transition.  Setting up model based testing usually requires a lot of work up front.  They payoff can be quite high though.

In a simple test harness, a test case is not provided any context in which to run.  Imagine the test cases as functions which take no parameters.  They will run exactly the same each time they are executed.  An advanced test harness will provide a mechanism to modify the test cases via parameters or scripting.  The simple method is to allow parameters to be passed to the test cases via a configuration file.  This is useful when you want several tests which vary only on a single parameter.  I wished for this feature when I was testing DVD playback.  In order to test the performance of our DVD navigator on different discs, I needed test cases to play different chapters.  I was forced to create a test case per chapter I wanted to play.  It would have been much simpler to write one test case and then allow a parameter which supplied the chapter to play.  Some harnesses might even provide a full scripting model where you could programmatically call test cases and provide parameters.  It is easy to envision embedding VB, Python, or even Lisp into a test harness and using that to control the test case execution.  The advantages to both of these methods are the ability to easily vary the test coverage without being required to program.  This makes test case creation accessible to non-programmers and saves programmers a lot of time over compilation.

Very advanced test harnesses provide mechanisms to execute the tests on a pool of machines.  We might call these test systems rather than test harnesses.  I’ll discuss these in another post.

http://blogs.msdn.com/b/steverowe/archive/2006/05/06/591764.aspx