Debugging 301: Into the Code

You've tried to write a Small, Self-Contained Example program and failed. Whatever the problem is, it only manifests in your full-scale application. What now?

The Scientific Method remains your best tool, but rather than trying to fit a hypothesis to the available data, it's now time to make predictions based on your hypotheses. And, using either log messages or a debugger, watch your program execute until you find the place where your predictions become incorrect. I've written elsewhere about logging, so this post will focus on running your code in a debugger.

The power of a debugger comes from the fact that you can examine the entire state of your program and then change it at will. In some cases, you'll be able to restart methods to see how their execution changes with different starting conditions. However, this power is also a hindrance: it's easy to get overwhelmed by the amount of information available, and even more easy to make assumptions and spend hours attempting to prove them correct.

The key is to form a hypothesis, and then look only at the code and data that are affected by that hypothesis. For example: you've written code that's supposed to upload data to a server, but it doesn't. You could waste a lot of time tracing the code, but it's more efficient to come up with a few hypotheses:

1. You're not actually running the upload code.
2. The upload fails due to a communication problem between client and server; the data never gets to the server.
3. The server is unable to process the file.

Hypothesis #1 is easy to falsify: put a breakpoint at the start of the upload code and run your program. If the breakpoint is hit, then that hypothesis is done. If you don't hit the breakpoint, then you need to come up with hypotheses as to why you didn't (note, however, that falsifying this hypothesis says nothing about hypotheses #2 and #3 — you could have more bugs).

Hypothesis #2 is more difficult to falsify, because there could be many reasons for the upload to appear successful. For example, library code that ignores an exception. It might make sense to step through the code, but again all of the possibilities can waste your time. Better is to use a tool that can directly disprove the hypothesis: a TCP monitor, or perhaps using the logging that's built into your your communications library.

If you get to hypothesis #3, it will soon be time to generate a whole new set of hypotheses, to cover the different cases that would cause the server to reject your file. And it's likely that the debugger will no longer be the best tool. Because now you have enough information to create test cases to validate your code, or an SSCEP to validate the server code.

Debugging 202: The SSCEP

When following the Scientific Method, you need a way to test your hypotheses. And this is where the Small, Self-Contained Example program comes in: it's a program that attempts to isolate the specific problem, moving it out of the larger application. If you can cause a problem to appear in a SSCEP, you can start building new hypotheses about why it occurs. If you can't cause the problem to appear, then you've falsified the hypothesis which led to its creation, and can start formulating a new hypothesis.

Often, while you're creating a SSCEP the problem will become obvious — you'll move from the realm of science back into the realm of mathematics. It's remarkable how problems simply disappear when you give yourself the opportunity to step back and think about them without distractions.

So how do you start writing a SSCEP? The traditional way is to start with your program and pare it down until the bug stops appearing. As long as you keep track of what you remove at each step, sooner or later you will remove the buggy code. Then you figure out a better way to write it. However, you may need a lot of revisions to reach that point, and such programs tend to be larger than needed.

I prefer to start from the other end: with an empty main() method (or framework-specific class) and — guided by my hypothesis — add small pieces until it breaks. Many times, the SSCEP bears little resemblance to the real program: I'm isolating a hypothesis, not building an application.

One of the nice side-effects of this approach is that the result looks a lot like a unit or integration test. In fact, particularly when writing layered code, I can create the SSCEP using JUnit. And once I have such a test, there's no reason to throw it away. Instead, it goes into the test suite as a regression test.

Debugging 201: Scientific Method

I've never liked the term “computer science.” Science, and the scientific method, is all about increasing our knowledge of the unknown. Computer programs are deterministic, moving from state to state according to specific rules; there's nothing unknown about them. Well, at least until you introduce a bug into your program. And then scientific method can be extremely useful, helping you to discover the real rules that your program is using.

For those who slept through 8th grade, here's a summary of the scientific method:

1. Gather data.
2. Come up with a hypothesis to explain how that data could exist.
3. Design experiments to prove that hypothesis wrong (this is the hard part).
4. Once you've run out of experiments, call your hypothesis a theory (8th grade science teachers are now upset).
5. Repeat.

That's right: science doesn't grow by proving hypotheses correct, it grows by proving them wrong. It's impossible to prove a hypothesis about the natural world, because someone might find data that disproves it. For example, Newtonian mechanics sufficed for the 17th and 18th centuries, but by the end of the 19th century scientists had gathered evidence that showed it didn't always apply. Einstein replaced Newtonian mechanics with general relativity in the early 20th century, and that hypothesis remains with us today. But it might be overturned tomorrow, next year, or a million years from now.

So what does that have to do with debugging?

It's a way to break free from your assumptions. When following the scientific method, your goal is not to prove your assumptions correct, but to prove them wrong. And you devote all of your effort to that proof. Only when you've exhausted all attempts to prove your assumptions wrong are you allowed to form new assumptions.

The first assumption that you need to prove wrong, of course, is that your code is correct. Which is where the Small, Self-Contained Example Program comes in.