Mock Objects and Statistics Gathering

Unit testing tends to focus on the API. The unspoken assumption is that, if the API behaves as expected, then the internals are correct. However, that's not guaranteed; even Uncle Bob concedes that a pure test-driven approach will lead you to BubbleSort rather than QuickSort (warning: it's a bit of a rant). Clearly, the “black box” approach of testing just the API is not sufficient; some tests need a “white box” approach, where you look into the implementation. But how to do that?

One way is to add test hooks: public methods that give access to internal object state. I can see how this might work: you write all your mainline code in terms of interfaces, with factories to give you instances of those interfaces. And as mainstream software development embraces dependency injection frameworks, this might be the right answer. But it still leaves me with a queasy feeling. All it takes is one undisciplined programmer writing a cast to the implementation class so that s/he can use those methods, and you've got brittle code waiting to break.

I recently tried a different approach: using a mock object to gather statistics about the internals of the object under test. Ironically, it was a sort routine.

Mock objects have been a part of test-centric development for a while. There are many libraries that help you create mock objects, and it's easy to create your own using proxies. But most of the articles that I've read on mock-centric tests use them to track simple interactions: you want to verify that your class takes a particular action, so you inject a mock that asserts it occurred.

When testing my sort routine, I wanted to verify that the sort actually performed in NlogN time. It was a static method, so there was no place to add a test hook even if I believed in doing so. But there was one hook that was a natural part of the API:

public static class CountingIntComparator
implements Heapsort.IntComparator
{
    public int count;
    public int expectedCount;

    public CountingIntComparator(int size)
    {
        // our implementation of heapsort should perform at most 3 compares per element
        expectedCount = 3 * size * (int)Math.ceil(Math.log(size) / Math.log(2));
    }

    public int compare(int i1, int i2)
    {
        count++;
        return (i1 < i2) ? -1
             : (i1 > i2) ? 1
             : 0;
    }

    public void assertCompareCount()
    {
        assertTrue("expected < " + expectedCount + ", was " + count,
                   count < expectedCount);
    }
}

This works because “O(NlogN)” refers to the number of comparisons made by the sort. And it made for very simple test code (note that I needed a relatively large test array; this is a probabilistic tests, and I didn't want the test to fail because the random number generator returned a string of “bad” values):

public void testIntSortManyElements() throws Exception
{
    final int size = 10000;

    int[] src = createRandomArray(size);
    int[] exp = createSortedCopy(src);

    CountingIntComparator cmp = new CountingIntComparator(size);
    Heapsort.sort(src, cmp);
    assertEquals(exp, src);

    cmp.assertCompareCount();
}

The take-away from this, if there is one, is that mock objects can do more than simple interaction tests. They can gather statistics that give you insight into the long-term behavior of your objects, without the need to expose object state to the entire world.

The key seems to be whether or not the API that you're building is amenable to injecting such a mock without adding a test-specific variant. Perhaps it's a technique that only works for utility classes. But I'll be looking for similar opportunities in more complex code.