Console of Elders - Overriding Java System.out for unit testing

This was the last Test Master Musings written before the crown was ripped from my head for the crime of hubris...

Test Master Musings - Console of Elders

Defect 20790 entered the realm today, indicating that exceptions were being written to the console during successful completion of certain processes. Since exceptions should only be written to the console during exceptional circumstances, and not during successful runs, the fix was to simply suppress the exception. As you can imagine, this was a one line code fix that was easy to spot. The unit test was more challenging.

The unit test for this should fail if anything is written to the console during the object’s construction. (The constructor takes a File object and validates certain aspects of its contents. If the file does not exist then this exception is logged, despite it being not an error). So this unit test requires a way to examine console output; easily achieved by overriding standard error to a ByteArrayOutputStream. Here is the setup, teardown, and unit test:

protected void setUp( ) throws Exception {
  originalErr = System.err;
  standardErr = new ByteArrayOutputStream( );
  System.setErr(new PrintStream(standardErr)); //override standard err
}
protected void tearDown( ) throws Exception {
  System.setErr(originalErr); //restore standard err
}
public void testConstructor( ) {
  final File zipFile = new File(TEST_TEMP_FILE);
  if (zipFile.exists( )) {
    zipFile.delete( ); //make sure file does not exist
  }
  final ClassUnderTest cut = new ClassUnderTest(zipFile);
  assertStreamedOutput("", standardErr);
}

And here is the custom assert method (currently hiding in the same test file):

private static void assertStreamedOutput(String expected, ByteArrayOutputStream actual) {
  try {
    final String output = actual.toString("ISO-8859-1").trim();
    assertEquals("Correct message not written to console", expected, output);
  } catch (UnsupportedEncodingException ex) {
    fail("UnsupportedEncodingException" + ex.getMessage());
  }
}

Here are a few notes on this assert method:

1. Despite the byte stream being an instance field, it was passed to the assert method as a parameter. This has several benefits: the method is now static and can be easily moved to a framework package, the method follows the conventions of junit assertions being static, and the method follows the junit conventions of having the parameters be named expected and actual. Arguably, all good things.
2. The assert method catches exceptions and fails instead of raising the exception. Raising the exception tends to clutter the calling code with more catch statements. The assert method is meant to fail on an error, so put a fail statement in it rather than raising the exception.
3. The ByteArrayOutputStream specifies an encoding in the toString( ) call. This was one of Joshua Bloch’s Puzzlers… without specifying a standard encoding the results of toString( ) will vary from machine to machine.
4. This unit test is possible because we are making sure that no output was written to the console. If we needed to test that output was written to the console then this test would not work so well. You would have to test that the stack trace of the exception equals a certain string… which is very, very fragile. An alternative would be to only compare x number of characters or perhaps search for a certain string. This is still less than ideal. Expect to discuss this issue more in the future!
5. So what is the bad smell of this test? Permanent fixture setup! An error on teardown might cause unrepeatable tests down the line.
6. What is a better solution? Define a Logger object (rather than a static logger such as System.out) and pass it into the system under test. This gives you the ability after exercising the SUT to verify the contents of the Logger. This is probably a better design that converting to something like log4j and adding a Unit Test Appender.
   

May the Green Bar Lead You Out of Darkness,

The Unit Test Master