Mocks and Stubs aren't Spies

At this point, we all know the difference between mocks and stubs... right? Well, perhaps not. That's OK, I'll try to explain it. And if I do a poor job you can always go read the article. (I've tried to have these samples follow Fowler's samples so that the two articles can be read together easily).

So in unit tests, when your system under test requires a collaborator you provide it with a dummy implementation that does nothing, like so.

public void testFillingOrder() {
Warehouse dummyWarehouse = new Warehouse() {
  public void remove(String product, int quantity) {
      //overridden so database query is NOT executed
    }
  }
};
Order order = new Order("TALISKER", 50);
assertTrue("order must be filled", order.fill(dummyWarehouse);
}

It isn't simple, but it isn't very complex either. Basically, the system under test requires that a warehouse be present, but beyond that it doesn't really care that the warehouse does anything. Test code gets more complex when your system under test requires reading certain values from a collaborator. You then have to provide it with a stub implementation that returns that value, like so (notice how the remove() method returns a value).

public void testFillingOrder() {
Warehouse warehouseStub = new Warehouse() {
  public boolean remove(String product, int quantity) {
      //overridden so database query is NOT executed
      return true;
    }
  }
};
Order order = new Order("TALISKER", 50);
assertTrue("order must be filled", order.fill(dummyWarehouse);
}

This example makes the difference look pretty simple, but in real unit tests stubs are a lot more complex than dummy objects because you usually need a way to modify the return value on the warehouse. It's still fairly easy to follow though, and probably doesn't represent a maintenance burden. It starts to get really complex when your system under test requires that certain methods on a collaborator get called (possibly in a certain order). Then you need to use a mock that can record how it is used and be verified later on.

public void testFillingRemovesInventory() {
  //setup - data
  Order order = new Order("TALISKER", 50);
  Warehouse warehouseMock = createMock(Warehouse.class);

  //setup - expectations (really a part of verify)
 expect(warehouseMock.remove("TALISKER", 50)).once();

  //exercise
  order.fill(warehouseMock);

  //verify
  warehouseMock.verify();
}

I tried to find a way to have the sound of a needle scratching a record play as you read that last code sample, but I couldn't figure out how to do it. Imagine one now. Why? For starters, this code is a lot more complex than a stub. As I wrote previously, recording, verifying, expecting, and playing back objects is not a language commonly used in testing. So there is that mental hurdle to jump. But do you know which line of code this test will most commonly fail on? The line that says verify(). I hope you like looking at stack traces. Or debugging. But until you have fairly extensive experience with a mock object framework, you're probably not going to know exactly what went wrong by simply looking at the error message. (Everyone I know has grappled with the problem of a mock object error message that reads something like "Failed: Expected 'result' received 'result'". For the uninitiated, this is a sign that your failure is being masked by a toString() method. Intuitive, huh?).

But my biggest issue with the mock test method is that it fails to follow the most common test structure of setup, exercise, and verify. Following this structure in your tests leads to tests as documentation and communication of intent. But mock objects do their verify just after setup, in a setup, verify, and exercise pattern. Despite the fact that the verify() method occurs after exercising the tests, to understand what the method is supposed to do you need to look several lines up in your test method. Breaking out of this pattern by using mocks has lead me to write some really hard to maintain and fragile unit tests.

Wouldn't it be easier if we didn't have to set expectations on mocks? If we could just assert that something happened at the end of the test? Like this:

public void testFillingRemovesInventory() {
  //setup - data
  Order order = new Order("TALISKER", 50);
  Warehouse warehouseSpy = createSpy(Warehouse.class);

  //exercise
  order.fill(warehouseSpy);

  //verify
assertTrue(warehouseSpy.removeWasCalledOnce("TALISKER", 50));
}

Using a Test Spy, as in this example, is a much simpler way to test how collaborators were used than creating a record/playback style mock. Moving the expectations to the end of the test method more clearly reveals the intent of what should occur. The test is also devoid of any record/playback language, which is accidental complexity and should be removed. A final complaint about mocks is that they lead to very fragile tests, in which a change in implementation leads to hard to diagnose failures in the unit tests. Extensive use of mocks has led me to write over specified and fragile unit tests.

Before moving on, I'd like to clarify and define some terms in use here, which I originally discovered in Gerard Meszaros' xUnit Patterns book.

• A Dummy Object is a placeholder object passed to the system under test but never used.
• A Test Stub provides the system under test with indirect input
• A Test Spy provides a way to verify that the system under test performed the correct indirect output
• A Mock Object provides the system under test with both indirect input and a way to verify indirect output

The still missing piece is a Test Spy framework that provides calling semantics like the spy.removeWasCalledOnce() method. This could be done in a dynamically typed language like Groovy using its metaprogramming facilities, or it could be done in Java using a more reflective style API. This would be hugely beneficial in my unit tests efforts. It would declare intent and document the system better, and unburden the unit tests from the problems associated with mocks.

In closing, when writing unit tests that require collaborators, consider that Mocks aren't a one size fits all solution. As always, figure out what you need first and then find the best tool for the job. And you can let this handy chart guide your decisions:

Watch this spot for more news on a Groovy Test Spy framework!