Debugging 102: Sherlock Holmes and William of Ockham

So if assumptions are a problem, and they're almost impossible to eliminate once formed, how do you prevent them from being formed? A good first step is to examine every perceived fact with Occam's razor — or more correctly, the interpretation of Occam's razor that says “simpler is more likely.”

Looking back at my “broken” network cable, with razor in hand, I should have asked myself “how did this cable break while sitting on the floor?” There's no simple explanation for that. Cables don't break unless they experience some trauma, yet this cable was stored in an out-of-the-way place. Damage, therefore, wasn't a simple explanation; time to look for something else.

Occam's razor is also known by the tagline “SELECT isn't broken.” The belief that your program is perfect and the compiler, or DBMS, or third-party open-source library is broken is common, especially among neophyte programmers. It takes time and suppression of one's own ego to trust someone else's work, to believe that “if 10,000 programmers can use this compiler without it breaking, my code is to blame.” Of course, sometimes it is the compiler: I've discovered one or two compiler bugs in my career, along with numerous bugs in open-source libraries.

And this is where Sherlock Holmes enters the picture (as quoted from The Sign of the Four):

when you have eliminated the impossible whatever remains, however improbable, must be the truth

The key part of this quote is “when you have eliminated.” Debugging is a process: start with the most likely situation (per Occam's razor), work to eliminate it as a possibility, then move on. The process fails when you either jump too quickly to an assumption or, as in the case of my network connection, don't work to eliminate your assumptions.

And one way to eliminate possibilities, even if they're based on strongly-held assumptions, is to apply the “Scientific Process” to your debugging.

Debugging 101: Assumptions

Story time: this weekend I was configuring a router for my wife's uncle. I needed a live upstream network connection to test it, and happened to have a spare cable plugged into my main router — a gray cable, that I'd used recently for a computer without wireless, which was coiled neatly next to my equipment rack.

I plugged this cable into the new router, and nothing happened. “Oh, that's right, the connector has a broken tab, it probably needs to be pushed back into its socket.” After five minutes of unplugging and replugging the cable, and testing it on my laptop, I decided that the cable simply wasn't working anymore, and grabbed another off the shelf. I configured the router and that was that … until I sat down at my desktop computer and couldn't connect to the Internet.

I had violated the first rule of debugging: don't assume.

In this case I assumed that the gray cable I was busy plugging and unplugging at my main router was the same gray cable that was plugged into the new router. In fact, it was the gray cable that was attached to my desktop computer. The end of the neatly coiled spare cable had long-ago fallen onto the floor (it was missing the tab on its connector, remember).

There were plenty of clues staring me in the face. For one thing, the “spare” cable was zip-tied to my equipment rack. When I saw this, I actually said to myself “why did I do that?” Another clue was that only three “connection” lights were lit on the main router. Oh, and then there was the fact that I had two gray cables lying next to each other. But I ignored these clues.

That's the nature of assumptions: once you believe something, you won't let go of that belief until you have incontrovertible evidence to the contrary. Perhaps this was a good thing for early human evolution: it's best to assume that rustling in the brush is something that plans to eat you until you see otherwise. Those who abandoned their assumptions too quickly were removed from the gene pool.

But to a software developer, it's a trait that gets in the way of debugging. Fortunately, there are a few tricks to break assumptions … as long as you remember to use them before the assumption takes hold.

More on those tomorrow.

Integration Tests with Maven

Maven is simultaneously one of the most useful and most frustrating tools that I have ever used. As long as you stay within its “standard build process,” you can do some amazing things with very little configuration. But if you step off that standard, even in a small way, you'll end up in a world of pain. If you're lucky, a short session of Googling will suffice; if unlucky, you'll need to read plugin source code.

In my case, all I wanted to do was create a separate directory of integration tests, and allow them to be invoked from the command line.* I thought that I could get away with just redefining a system property:

mvn -Dproject.build.testSourceDirectory=src/itest/java test

No luck: Maven used its default directory (confirmed with the -X command-line option). After some Googling, which included a bunch of Maven bug reports, I learned that some plugins load system properties before they load the properties defined in the POM — which I think misses the whole point of command-line defines.

My next step was to take a closer look at Surefire's configuration options in the hope that I could override there. And testSourceDirectory looked like it would work, but its datatype is File, and Maven doesn't try to convert non-String properties.

Some more Googling led me to build profiles. This seemed promising: I could create a named profile, and within it set the test directory to whatever I wanted. There was only one problem: profiles can't override the POM's build directories. If this post ever drops into the hands of the Maven team, I'd really like to know the reasoning behind that decision.**

OK, but perhaps I could use a profile to pass my directory directly to the Surefire plugin? This seemed promising, and when I first tried it, even appeared to work (but it turned out that I'd accidentally put source files under the wrong directory tree). Back to the documentation, and I realized that tests are compiled by the compiler plugin, not the Surefire plugin, and the compiler plugin doesn't have any directory config. This is when I downloaded the Surefire plugin code, and grepped for testSourceDirectory. It appears to be used only for TestNG; too bad I'm using JUnit.

In the end, I gave up on using a command-line override to run integration tests. I realized that Maven really, really wanted me to have a single test directory, used for all phases of the build. But to split this directory into unit and integration phases meant I would have to use a naming convention and include/exclude filters. I still rejected this: I have no desire for “big ball of mud” source trees.

Instead I created a sub-project for the integration tests. Of course, it wasn't a true sub-project: Maven is very particular about parent-child relationships. Instead, I just created an "itest" directory within my main project directory, and gave it its own POM; Maven's transitive dependency mechanism really helped here. It took about 10 minutes, versus the 4+ hours I'd spent trying to override the test directory. And it didn't affect the mainline build; for once I was happy that Maven ignores things that it doesn't expect.

---

* Yes, I know that Maven has an integration-test phase. But after reading about the configuration needed, I decided that it would be a last resort.

** Update (7-Apr-10): last night I attended a presentation by Jason van Zyl, creator of Maven and founder of Sonatype. And his answer to why Maven 2 doesn't allow profiles to override all build elements was that they didn't see a use case at the time: integration tests are outside of the normal build flow — except that now they're finding a need to do just that when building the Hudson build-management tool. For myself, I've come to accept sub-projects as the simplest solution.

Why Write Open Source Libraries

I just created my third open-source project on SourceForge, S34J. It's a set of a half-dozen objects that encapsulate the calls for Amazon's Simple Storage Service (S3) — at least, it will be once I finish the code. My other two projects are PracticalXML, a utility library hiding the (often painful) Java XML API, and SwingLib, a library of Swing GUI enhancements that currently has three classes (mostly because I haven't taken the time to upload more).

Other than PracticalXML, I don't expect anyone to ever use these libraries. And for PXML, I don't expect anyone other than the other maintainers, all former coworkers, to use it. So why write them, and why take the time to create a SourceForge project? The answer can be found in the project description for SwingLib:

Classes that I've written for Swing programming. Possibly useful for other people.

Throughout my career, I've written the same code over and over again. Such as a method that creates an XML element that inherits the namespace of its parent. Simple code, a matter of a few minutes to write, but after the third or fourth time it gets annoying. Particularly if you have a several dozen such methods. And once I type that code on an employer's computer, it becomes their property; I can't simply take a copy on to my next job (I'll note here that ideas, such as an API, are not protected by copyright; I always make a DomUtil class, and it always has an appendChildInheritNamespace() method, but the implementation has been from-scratch each time).

An auto mechanic acquires his tools over a lifetime, and takes them from job to job; they don't belong to the shop where he works. By releasing this code as open source, I can do the same. And, who knows, someone else might stumble on it and decide it's useful.

Building a Wishlist Service: The Data Access Layer

I'm a huge fan of hiding persistence behind a data access layer. In particular, a data access layer that provides the basic CRUD operations — create / retrieve / update / delete — in a form slightly less granular than the actual domain objects.

I adopted this attitude after realizing that a typical business application has different types — levels — of rules. At the top are the “business” rules that describe how data is to be transformed: for example, a withdrawal or deposit changes the value of a bank account. Below those are the “logical model” rules describing how objects interact: a deposit or withdrawal can't exist without an account. And at the bottom are the “physical model” rules of how data is structured: a transaction consists of an account number, an operation, and a dollar amount.

In my view, the data access layer should manage the logical model. For the product list service, it implements rules such as “adding an item to a list will create that list if it doesn't already exist.” I could do this in the database, using triggers, but that means writing new code for each DBMS, along with additional work for deployment or upgrades. Or I could do it in the high-level service code, manipulating the domain objects directly, but that would result in duplicated code. Using the data access layer, the high level code is responsible only for populating and formatting the domain objects, while the database is responsible only for persistence.

Along with providing a place to hang rules, a data access layer simplifies testing: it allows the use of mock or stub objects in unit tests, thereby avoiding any actual database dependencies. Prior to developing the product list service, however, I had never used them in this way. For most data-driven applications, I would rely on integration tests to exercise a real persistence layer.

The product list service was different, primarily because I wanted to focus on client-server interaction, and didn't want to think about a persistence framework until later in the development process. So I spent a couple of hours to develop a stub data access layer that used HashMap for storage. Along with a suite of tests that allowed me to work through the rules of data management. My intent was that these could start as unit tests, then migrate to implementation tests once I added a real persistence framework.

Going into the project, I had assumed that persistence would be handled via a relational DBMS, probably using Hibernate as an object-relational mapper.* As I proceeded with development, however, I realized that there was no good reason to do this: as long as I had some way to assure sequenced, atomic commits, I had no need for a relational database. And because I was coding to the data access layer rather than “active record,” I had the freedom to explore this option.

That exploration is worth its own posting.

---

* In my opinion, Hibernate's chief benefit is that it hides the actual DBMS, meaning that it's (relatively) easy to implement portable applications — a big win if you're planning to sell your app, not so big if you control its deployment. In the case of the product list service, the object model is very simple, so I wrote a data access implementation using JDBC talking to an in-memory HSQLDB database. The goal was to verify that I wasn't doing anything that relational databases couldn't support. It didn't take long to write, but I remembered just how painful explicit SQL can be.

How HashMaps Work

This was originally written as an answer to a StackOverflow question where the original poster and several responders seemed to have some misunderstandings of how hashed lookup works. Since it would end up buried under a dozen more-or-less right responses if posted there, I've decided to post here, where it might end up on the front page of Google.

The Hashcode

Hashed lookup starts with an object's hashcode. A hashcode is simply a number that is permanently associated with an object, used only when storing that object in a hashed structure; there's no magic about it. The only rule that governs hashcode values is that, given objects x and y, if x.equals(y) returns true, then both x and y must return the same hashcode. The reverse isn't true: x and y could return the same hashcode, but not be considered equal. Here's an example:

Long x = Long.valueOf(0x600000002L);
Long y = Long.valueOf(0x700000003L);

System.out.println(x.hashCode());
System.out.println(y.hashCode());
System.out.println(x.equals(y));

All objects could return 0 as their hashcode, and that would satisfy the rules. It wouldn't be very useful, however. To be useful, a hash function must evenly distribute values over the 32-bit range of an integer: you should be able to apply the same statistical tests that you would for a random number generator.

There are many ways to compute hashcodes. Integer, for example, simply returns its int value. Long does a shift and xor, to mix its high-order 32 bits with its low-order 32 bits. A common approach is to keep a running sum, multiplying that sum by a small prime number before adding each term. This is how String does it:

for (int i = 0; i < len; i++) {
    h = 31*h + val[off++];
}

Note that this causes a string's hashcode to be “driven” by its trailing characters; integer overflow during multiplication claims the leading characters. This is OK, as long as the strings end with different characters — which normal text almost always will. As I recall, the original (1.1.x) implementation of String.hashCode() gave greater weight to the leading characters in the string, which turned out to be a very bad thing when processing HTTP URLs (which all start with “http://”).

Some Myths About Hashcodes

A lot of people think that a hashcode is a unique identifier for an object. This is wrong, and the reason should be obvious: a Java hashcode is 32 bits. As soon as you have objects with 33 bits of data, you're guaranteed to have two objects with the same hashcode. Depending on the hash function, you can have collisions with less than 32 bits of object data: the strings “Az” and “C<”, for example.

Then there's the “identity” hashcode returned by Object.hashCode(). The myth is that it returns a pointer to the object, which in a 32-bit JVM would be a unique identifier. Perhaps it does return a pointer, at least on the first call; it is, after all, implemented as a native method. However, objects are permitted to move around in the heap, so if the identity hashcode is indeed a pointer, there is no guarantee that two objects won't (over time) have the same value.

HashMap

OK, so every object has a hashcode; what's the point? For just a moment, pretend that a hashcode is indeed a unique identifier for the object. If that were the case, then you could create a map around an object array that had 2^32 elements: compute the key's hashcode, and return whatever you found at that spot in the array.

But, as I've said, hashcodes aren't unique identifiers, and an array containing 2^32 objects would be too big to fit in the heap (at least on a 32-bit JVM). But the idea still holds: take an array with 1,000 elements, and divide the hashcode by 1,000, and you can return whatever is at that index.

But then what happens if two objects map to the same index? They don't even have to have the same hashcode: with our thousand element table, one object could have hashcode 1001, and the second could have hashcode 2001. This is known as a hash collision.

There are many ways to resolve a hash collision. A really dumb approach is to find the next empty cell in the array. So if your hashcode mod 1000 is 723, and cell 723 already has a value, you look in cell 724 … then 725, then 726, and so on. As the table gets full, you'll end up doing a linear search of the entire table looking for an empty cell.

A much smarter way is to have each cell (called a "bucket") in your hash table hold a list of entries. So when your hashcode resolves to index 723, you can simply add your object onto the end of that list. And this is just what the Sun HashMap does: it defines an array of buckets:

/**
 * The table, resized as necessary. Length MUST Always be a power of two.
 */
transient Entry[] table;

And an Entry object is a node in a linked list:

static class Entry<K,V> implements Map.Entry<K,V> {
    final K key;
    V value;
    final int hash;
    Entry<K,V> next;

You'll still run into a problem if many objects have the same hash value, or if they have different hash values that resolve to the same bucket index: you'll traverse a very long linked list.

As long as you have a good hash function, that produces an even distribution of 32 bit numbers from your objects, this is not an issue: if you start to get a long list on a particular bucket, you simply increase the size of the table. Since it's an even distribution, you'll be distributing the objects across more buckets, which means fewer objects to a single bucket.

The Sun implementation uses hash tables that are always a power of two in size (eg: 16 elements, 32, 64, and so on). This is a performance optimization in that you can transform a 32-bit hashcode into the appropriate index by applying a mask (eg, if you have 16 elements in your table, your index is hashcode & 0x0F). However, such tables are very susceptible to bad hash functions; the Sun implementation takes extra steps to rehash supplied values.

The Bottom Line

If you use a HashMap, you need to ensure that the objects that you use as keys have a good hashcode. If you, for example, take a population of 1,000,000 objects, and have a hash function that can produce 1,000 discrete hashcodes, you'll end up with buckets containing 1,000 entries. And this will mean your map, which has an optimal access time of O(1), will turn into a data structure that effectively has O(n) access time.

Deploying to Amazon EC2

After talking with Fred Stluka, I started using Amazon EC2 as a demo site. For me, the choice was between Amazon or a Java hosting account with GoDaddy or similar. The economics are a tossup, but I like the control that I get from Amazon: I can turn on a demo box for just as long as I need it, running exactly the software that I want (provided that I install it), and not worry (too much) about someone breaking in and installing malware while I'm not looking.

One of the interesting issues with this setup is deploying new builds. Tomcat has a nice, simple, form-based upload-and-deploy feature in its Manager application. Which falls down pretty quickly when I'm uploading a new WAR over my 40kbytes/second residential ADSL connection. My current app is under 2 Mb, but it's still enough time to get another cup of coffee.

So far, I've tried two alternatives, with their own benefits and drawbacks.

The first is to maintain synchronized exploded WAR directories on both my build machine and the EC2 machine, using rsync. The following command will upload only those files that have changed, using compression to reduce their sizes:

rsync -aczv --delete . demo.kdgregory.com:Workspace/demo.war

It works, and the files that get uploaded tend to be small because the included JARs don't change, but it adds manual steps into my deployment: I have to unpack the WAR locally, then repack it remotely. Yes, I know that I could sync the exploded WAR right into Tomcat's webapps directory, but I'm concerned about non-atomic updates.

The other alternative is to do builds on the EC2 machine. It is, after all, a fully-functional computer with unlimited shell access. I've installed a complete Java toolset, and my applications build with Maven. The only real problem with this is getting the source code out of my home repository. Including getting Subversion so that I can get the source code.

EC2 provides many different pre-built instance types. I'm currently using a bare-bones Fedora Core instance, which does not come with any development tools. I already had a start script to create my users (their home directories are persisted on S3), so simply added “yum install subversion” to that script.

Once subversion is installed, I need to be able to use it. I have an SSH port open on my home machine, and Subversion allows you to tunnel requests through SSH: it starts a new subversion server for each request. Since I'm not using the standard port, I need to add the following to ${HOME}/.subversion/config:

kdg = ssh -p 12345

Now I can check out my project with the following command:

svn co 'svn+kdg://kdghome/repo/Project/trunk'

That only leaves one step: editing /etc/hosts to map kdghome to whatever public IP Verizon has given me today (and hoping that it doesn't change in the middle of the day). It's a lot of work, but only has to happen once per session. After that, I can run svn update from the remote machine to get any changes, or make edits there and commit back to my home repository.

Building a Product List Service: HTML Templating

Along with the product list service itself, I built a demo page. The first iteration of this page was very simple: a table to display list entries, a visible form to add new entries, and some invisible forms for generating update/delete requests. Even with this primitive page, however, I ran into the problem of how to turn the JSON data from the service into HTML.

My first approach was “the simplest thing that could possibly work”: I wrote a JavaScript function that built up an HTML string from the JSON, and then inserted this string into DIV using innerHTML. It worked, and had the benefit that the various event handlers were defined in the same file — if I changed a function's name or interface, I didn't have to update multiple files. But embedding markup in a script is ugly and hard to maintain; just keeping quotes matched takes a lot of work.

My second approach was to develop a utility library containing functions that build markup based on model objects. This approach was clearly influenced by Swing; in fact, the object passed to my buildTable() function looked a lot like Swing's TableModel. Doing this got the markup out of my main script, and gave me reusable components, which I liked. However, the code defining the model object alone was larger than my original concatenation function.

If my second approach resembled Swing, my first approach resembled early servlets. Keeping with the Java analogies, what I was really looking for was an approach that resembled JSP: all markup within one file, making references to data provided elsewhere.

With some pointers from friends who are adept JavaScript programmers, I started looking at different templating solutions. John Resig's micro-templates even looked like JSP directives (which meant, unfortunately, that they couldn't be used within a JSP-generated page). I tried out a few of the existing solutions, then decided to write my own — it was only a dozen lines of code, using regular expressions.

But while the template approach did the job, and provided a fairly clean break between markup and scripts, I was still uncomfortable. In part because there was now too much of a break between the markup and the scripts that interacted with it: a script in one file would blindly access markup from another file. My JavaScript friends would say that this break is a Good Thing, but I think that it ignores the fact that the markup and scripts are tightly coupled by nature — and in fact takes us back to a 1960s view of programs manipulating data. But hat's a topic for a future post.

Refactoring: Evil, or Just Misunderstood

How did refactoring get such a bad reputation? Depending on who you talk to, it introduces bugs, makes projects more expensive, breaks unit tests, and helped the Yankees win the World Series. Now, as a Boston native and Philly resident, I hardly want to help the Yankees, but it seems that claim is as overdone as the others.

Let's start with increased project cost. In my prior job, I often had to create estimates for changes to our current codebase. After working through an estimate I would usually have a conversation with the Business Owner that went like this:

BO: You have an estimate of five days for these page changes; can you give a breakdown?

Me: Sure. These are pages that do essentially the same thing, but have about 3,000 lines of code each. Refactoring accounts for four days, and the new additions take a day.

Usually, I was interrupted in the middle of the word “refactoring,” with a loud “We can't afford that!” And while I understood the reaction, the alternate estimate was for 8 days, allowing the developers time to figure out where to make yet another set of copy-and-paste changes to those enormous, similar-but-not-the-same JSPs.

In my view, refactoring is a tool to reduce duplication and reduce the amount of code that a developer has to keep in his or her head at one time. This viewpoint is derived largely from the use of the word “factor“ in mathematics: the number 12, for example, has the factors 4 and 6, which can be reduced to prime factors 2, 2, and 3. Once you reduce to prime factors, you discover duplication.

In software, duplicate code may not be so obvious. Unless, of course, that software was built using “copy and paste coding,” with short deadlines and no refactoring during development. In that case, it becomes just as obvious as the prime factors of 12. Eliminating duplication reduces the amount of code that has to be maintained and changed, consequently reducing the number of places that bugs can hide. The result should therefore be reduced implementation costs (and in the places where I got away with page refactoring, using the technique described here, those cost reductions did indeed appear in future projects).

Which brings up the second complaint: refactoring introduces bugs. I suppose that's true: every time you edit code, you run the risk of introducing a bug. And if you write bugfree code that never has to be changed, then I fully agree: don't refactor it. For myself, although I've occasionally written (non-trivial) code that is apparently bug-free, I've always had to change it at some point down the road as the business requirements changed.

And as I see it, well-factored code should actually reduce bugs! Consider the following code, something that you might see scattered throughout a web application:

String operation = request.getParameter("operation");
if ((operation != null) && (operation.length() > 0))
{
    // do something with the operation value
}

I look at this code and see two immediate refactorings: replacing the literal value with a constant, and replacing the if condition with a method call like Jakarta Commons' StringUtils.isNotEmpty(). These refactorings would immediately reduce the chance of errors due to typos — including a typo that left out the null check (yes, I've seen this in production code). True, good code would be written like this from the start, but again, that's not code that needs refactoring.

If I were feeling ambitious — and if this were part of a large method or scriptlet I would almost certainly feel that way — I would extract the entire if operation into its own method, with a descriptive name like retrieveOperationData(). Over the long term, this will make the code easier to maintain, because developers can simply skip the implementation if the don't care about operation data. And they're less likely to put some unrelated code inside the if.

But what if my changes introduced a bug? What if the code inside that if already mucked with some unrelated variable? Assuming that the code would actually compile with such a bug, I'd expect my tests to catch it — and if they didn't, I'd write a new test once a person reported the bug.

Which brings up the last complaint (and the one that prompted this article): refactoring breaks unit tests. Before taking on this point, I want to quote Martin Fowler, from the preface to Refactoring:

Refactoring is the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure.

If you don't alter the external behavior of the system, how do you break tests? Well, one obvious way is to write tests that depend on the internal structure. It's particularly easy to do this when you use mock-object testing, or when you decompose large objects into smaller “helper” objects. If you've unintentionally made the tests dependent on internal structure, then you fix your tests, and hopefully won't repeat the mistake. But a breaking test might indicate that your mainline code has similar dependencies, in which case it would be a Good Thing to eliminate those dependencies everywhere.

A more likely cause, however, is that you're not really refactoring per the definition above, but restructuring: changing the external behavior of the code as well as its internal structure. In mathematical terms, you've decided that rather than 12, you really want 14; which happens to factor into 2 and 7 and can't be reduced further.

While you might have a perfectly valid reason for wanting this, you can't blame the tests for continuing to assert 12. Instead, ask yourself why you once thought 12 — your original design — was correct, and why it's no longer correct. That's the topic of a future post.

Objectificated

I have an admission to make: it wasn't until October 1999 that I truly “got” object-oriented programming. Perhaps that doesn't seem like a big admission, but consider this: I had been programming in C++ since 1990, and had been reading articles on object-oriented programming since the mid-1980s.

Oh, sure, at one level I understood why objects were better than procedural code. When I read Stroustrup's introduction to The C++ Programming Language, I understood and agreed with his ideas of modularization via objects. By then, structured programming and layered design were in the mainstream, and objects seemed a simple way to accomplish those goals. And, working in the financial industry, I had many examples of polymorphic behavior (all securities have a price, but that price means something very different if you're talking about stocks, bonds, or options).

But it never gelled. Although I replaced struct with class, carefully considered the public/private divide, and compiled using a C++ compiler, I continued to refer to my self as a “C programmer.” In fact, my resume doesn't mention C++ (although that's more because C++ has come a long way since I stopped using it).

In October 1999, I joined an existing Java project — my first real use of the language. And one day, looking at the output from a parser generator, it clicked: polymorphism wasn't about three classes that exposed the same set of methods, it was about 150 classes, each of which used different data to drive the same method. Since that time, I've had a clear division in my mind between classes that provide behavior, and classes that hold (and expose) data.

Most “business” applications, however, revolve around data, not behavior. If you're writing online trading software, your use of polymorphism is limited to calculating the value of a position; something else displays that value. And that something else tends to be written in a procedural style: get this piece of data, do something to it, get the next piece of data.

There have been a lot of approaches to making such processing “more object-oriented,” from the Visitor pattern to mandates that all data be accessed via getters and setters. And to be sure, there are benefits from these techniques, particularly if you come up with a clean way to encapsulate business rules.

I think that “clean” is the real problem. Business rules tend to me more exception than rule, having evolved over time, changing to suit the needs of the moment, and involving inter-locking relationships between different sources of data. In that world, “tell, don't ask” is a lot more difficult than “ask and decide.”

(which, as I'm proofing this, strikes me as an example of Conway's law in action — but that's a topic for another post)

Multitasking

Right now I have four IDE windows open: NetBeans, two instances of Eclipse running on my main development machine, and another Eclipse running on a laptop to which I'm connected via Remote Desktop. All are running some mixture of the same projects. My development box runs Ubuntu, and I'm a multiple desktop addict.

I've noted before how I like NetBeans' integrated Tomcat server, but don't particularly like its Java editor or debugger. So when working on a web app, I set up projects in both NetBeans and Eclipse that refer to the same source tree. When switching between them after edits, I do need to kick off a rebuild, but this takes a fraction of a second to complete. It works well enough, in fact, that I don't feel the need for any of the web development plugins that I can get for Eclipse.

The two Eclipse instances are a little trickier to explain. My default Eclipse workspace has pretty much all of the projects that I'm currently working on, some open, most closed. And for the most part, that's a Good Thing: when I edit, say, a utility class, all dependencies in the workspace will pick up the update.

However, I've found that there are times when I want to switch back and forth between projects. For example, I'm working on a mainline project, and think of a series of changes that I want to make to a utility library. I find it's easier to use separate Eclipse workspaces for this, since each workspace can have its own set of open windows. When I switch from one workspace to the other, I maintain the context of where I stopped working.

Unfortunately, switching workspaces within a single Eclipse instance is not a matter of a few seconds, and the new workspace opens with the folder view collapsed. Separate instances can coexist peacefully, however, as long as they're using different workspaces. The JVM has become good enough about sharing portions of virtual memory that there's no noticeable swapping when I switch between instances. The only real drawback to this technique is that changes — particularly refactorings — don't automatically update across workspaces; I have to manually refresh.

That brings me to the Eclipse instance running on my laptop. I've found that, even within a single project, I'll be working on one part of the code and think of a refactoring somewhere else. Or realize that a widely accessed method should change its signature. I tend to be anal about my checkins: the commit message should accurately describe what was changed, and I don't want to check in a lot of files that have changes incidental to the commit.

One solution is to keep a list of such changes, and postpone them until after my current checkin. But that doesn't always work: sometimes it will be hours between checkins. Sometimes an entire day. And while some refactorings don't look so good in the cold light of morning, usually the delay just makes my current coding more difficult.

The laptop solves this problem, by holding a copy of the code as-of the previous checkin. Refactorings or signature changes can happen there, get checked in, then applied to my main directory via svn update. And while this seems like a recipe for disaster, or at least eternal conflicts, I've been surprised at just how easy it is to integrate such changes — they are, after all, incidental to my “current” work.

Tests are the Story

I have a recurring discussion with my friend James, regarding test-first development. He believes is that writing tests first is like writing an outline for an academic paper or novel. And sometimes you don't know where the story is going to end up when you start writing. In the past, I've always countered that with the liberal-arts-major argument that you should know what you're going to say before you say it.

However, he makes a good point. Even when you have a goal in mind, sometimes the path to that goal takes unexpected twists and turns. If you write your tests presuming a specific order of steps, they'll become obsolete when the real story doesn't follow the outline.

So here's a different response: the tests are the story. The production code is merely a footnote.

The ultimate goal is satisfied users, and the tests are development standins for those users (even if the "user" is another piece of code). As you write your tests, you should be thinking not of the needs of your mainline code, but the needs of those users. As your understanding of those users changes, your tests will change. Some will be deleted.

You could call this wasted effort, but that's misleading. It was necessary effort, because you didn't have a perfect understanding of the users' needs. The real question is: was it better to explore the plot using tests, or to simply write some mainline code and hope that it did what it should?

Tag Reuse in JSPs

One thing that's always puzzled me about JSP taglibs is the mechanics of tag reuse — what happens when you invoke the same tag twice on a page. The Tag JavaDoc says the following:

After the doEndTag invocation, the tag handler is available for further invocations (and it is expected to have retained its properties).

This statement sent up a bunch of red flags when I first read it. And it's done the same for others: I've seen tag implementations that explicitly reset all attributes in doEndTag(). But if this was the required implementation, I'd expect to see it mentioned in the J2EE tutorial or sample code, and it isn't.

Moreover, the description makes no sense: you can't build a software system on components that will render arbitrary values depending on where they're invoked. My first hint came from examining the Java code generated by Tomcat:

private org.apache.jasper.runtime.TagHandlerPool _jspx_tagPool_prodList_form_visible_userId_operation_name_listName_htmlId_format;
private org.apache.jasper.runtime.TagHandlerPool _jspx_tagPool_prodList_form_visible_userId_timeout_operation_name_listName_htmlId_format;

Those are pretty long names. And interestingly, they include the parameters used in the tags. And they're different. That led me back to the JSP specifications, with a more focused search. And on page 2-51, we see the rules:

The JSP container may reuse classic tag handler instances for multiple occurrences of the corresponding custom action, in the same page or in different pages, but only if the same set of attributes are used for all occurrences.

I'd love to know the rationale behind this — and whether it was intentional or a “retro-specification” to cover decisions made in the reference implementation (Tomcat). To me, it seems like premature optimization: assume that it's expensive to instantiate a tag handler or set its attributes, so go to extra effort to pool those instances. If, instead, the specification required handlers to be explicitly instantiated per use it would drive developers toward lightweight handlers that collaborate with external services. Which in my opinion is a far better way to write them.

Unease

If you really don't care you aren't going to know it's wrong. The thought'll never occur to you. The act of pronouncing it wrong's a form of caring.

Zen and the Art of Motorcycle Maintenance is a book that you'll either love or hate. To some, it is a bombastic restatement of ideas that everyone already knows. To others, a lightweight gateway to heavyweight philosophy. And then there are people who believe it changed their life. I can't say that I'm one of the latter, but the book resonates with me on many levels, and I've reread it every few years since I was 15.

I recently paraphrased the above quote in a presentation on testing and coverage. The main point of my presentation was that 100% coverage does not guarantee sufficient testing; an audience member asked the obvious question “then how do you know that you've written enough tests?” My answer can be paraphrased as “you've written enough tests when you feel comfortable that you've written enough.”

Not a terribly good answer, to be honest. It did, however, set up my closing slide, which pointed out that you need committed, test-infected developers to get good tests. If you don't have that, you might as well buy a tool that throws random data at your code. But how does one arrive at the level of caring needed to write good tests? And is it inborn, or something that comes from experience?

I've been thinking about these questions this morning, because I'm refactoring some code and am left with what I consider an ugly method call: if called in one situation I want it to throw an exception, if called in another I want it to fail silently and return null. I've been puzzling over whether I really need to break the method into two parts, and also whether I should think more about another quote from Pirsig:

What's more common is that you feel unpeaceful even if it's right

Sometimes, perhaps, good enough is good enough.

Building a Wishlist Service: HTML Forms

Back to the wishlist service, and it's time to look at the client side. In particular, the mechanism that the client uses to submit requests. XML on the browser is, quite simply, a pain in the neck. While E4X is supposed to be a standard, support for it is limited. Microsoft, as usual, provides its own alternative. Since XML is a text format, you could always construct strings yourself, but there are enough quirks that this often results in unparseable XML.

Against the pain of XML, we have HTML forms. They've been around forever, work the same way in all browsers, and don't require JavaScript. They're not, however, very “Web 2.0 friendly”: when you submit a form, it reloads the entire page. Filling this gap, the popular JavaScript libraries provide methods to serialize form contents and turn them into an AJAX request. As long as you're sending simple data (ie, no file uploads), these libraries get the job done.

To simplify form creation, I created some JSP tags. This provides several benefits, not least of which is that I can specify required parameters such as the user ID and wishlist name. I also get to use introspection and intelligent enums to build the form: you specify an “Operation” value in the JSP, and the tag implementation can figure out whether it needs a GET or a POST, what parameters need to go on the URL, and what fields need to go in the body.

One of the more interesting “learning experiences” was the difference between GET and POST forms. With the former, the browser will throw away any query string provided in the form's action attribute, and build a new string from the form's fields. With the latter, the query string is passed untouched. In my initial implementation I punted, and simply emitted everything as input: the server didn't care, because getParameter(), doesn't differentiate between URL and body. This offended my sense of aesthetics however, so I refactored the code into a class that would manage both the action URL and a set of body fields. Doing so had the side benefit that I could write out-of-container unit tests for all of the form generation code.

The other problem came from separating the form's markup from the JavaScript that makes it work. This is current “best practice,” and I understand the rationale behind it, but it makes me uncomfortable. In fact, it makes me think that we're throwing away lessons learned about packaging over the last 30 years. But that's another post.

Script Kiddies

As you might have guessed from my earlier posting about del.icio.us, I pay attention to the access logs for my main website. It's interesting to see what pages are getting traffic, and where that traffic is coming from. This month, of course, the article on Java Reference Objects was far and away the winner, with almost exactly 50% of the hits. However, nearly 25% of these hits came from Google searches — which is not surprising, since it's on the first page when you Google “java reference objects”. Unfortunately for the world of software, a lot of the query strings indicate deep misunderstandings of how Java works. And then there are the strange ones: I have yet to figure out what “sensitive objects” could be.

What's really interesting about my access logs, however — and why I look at them rather than using a web beacon — are the attacks on my site. I have a homegrown page management system, which I think is fairly solid, but ever since I saw the first attack I've paid attention to them. Here's a a typical log entry, that appears to be coming from a home computer:

68.217.2.156 - - [05/Sep/2009:16:38:07 -0700] "GET /index.php?page=http://64.15.67.17/~calebsbi/logo.jpg? HTTP/1.1"

If my pages blindly took the page parameter value and passed it to include, this attack would work. And if you Google for “calebsbi” you'll see over 30,000 hits, indicating that there are a lot of people who blindly pass parameters to include. Other attacks try to read the password file, and one enterprising person wrote a script that sent himself email (to a Google address, which I promptly reported).

What's amazing to me is the amount of effort these people expend. Oh, sure, I expect that the actual hits come from a bot, but someone has gone to the trouble to figure out that my site is driven by the page parameter: I don't get hits with include= or sql=. And it's not always a botnet doing the hits: one night a person came to my site from Twitter, and spent over an hour trying different URL combinations. He (I'll assume) actually managed to uncover portions of my site's directory tree (the programming examples use a simple directory tree, which has since been “capped” with a redirect), and started trying URLs that included functions from my page fragments.

So far, my site hasn't been hacked, and I think a large part of the reason is that the attackers have been “script kiddies,” trying attacks that they don't really understand. The person who walked my programming directories, for example, kept putting needless parameters on the URLs. And if you try to retrieve “logo.jpg”, you'll find that there's no longer any webserver at that address. Yet the attacks keep coming, so I keep looking to see that my site code behaves as expected.

Intern Isn't Forever ... And Maybe Never Was

Today I managed to hit myself over the head with a cluebat, proof that there's always something to learn and that you should never accept dogma. In this case, the dogma was that interned strings never get garbage-collected. Like all good dogma, it combined a few facts with a leap of faith, and was plausible enough that I never challenged it.

The facts first (actually, only one fact): interned strings are stored in the same pool as literal strings. This is explicitly stated in the JavaDoc for String.intern(), and can be demonstrated with the following code:

String a = "Are we having fun yet?";
String b = new String(a);
System.out.println(a == b);

String c = b.intern();
System.out.println(a == c);

And now the leap of faith: the JVM doesn't clean up the constant pool. Seems plausible: after all, two literal strings are guaranteed to be the same. And intern() is a native method, so it must be doing something tricky behind the scenes. And everybody else says you'll cause bugs if you intern too many strings, so …

A skeptic might ask “how can you tell that two string literals aren't the same if you don't have references to both?” I even said as much when I wrote about canonicalizing maps (an article that got some edits today). Once all references to a string go out of scope (including any references within a class definition), then there's no need to keep that string in the pool. And the JVM doesn't — at least, the Sun JVM doesn't.

This particular cluebat entered the picture because I'm currently writing an article on out-of-memory exceptions, and wanted a program to demonstrate permgen failures. So I wrote a loop that interned big, random-content strings … and nothing happened. I killed the program after I finally realized that it wasn't going to die on its own.

But the dogma must have some basis in fact, right? It happens that I have a machine with Sun JVMs from version 1.2 on up. So I ran my test program on each revision, and while the -verbose:gc output changed, the result did not: all of these versions appear to clean up the string pool. Is it possible that a 1.1 release is the source of this dogma? Perhaps, and if someone still has one installed, here's the program:

public class InternExhaustion
{
    public static void main(String[] argv)
    throws Exception
    {
        while (true)
        {
            String str = generateRandomString(65536);
            str.intern();
        }
    }    
    
    private static String generateRandomString(int length)
    {
        char[] chars = new char[length];
        for (int ii = 0 ; ii < length ; ii++)
            chars[ii] = (char)(96 * Math.random() + ' ');
        return new String(chars);
    }
}

For myself, I have some edits to make. And a lump on the head to remind me to question dogma.

Je vais m'amuser

If you're a native French speaker, you may be giggling to yourself at the title of this post. If, like me, you had two semesters of French in college, you may see nothing wrong — until your Parisian instructor starts giggling. “I will amuse myself.” The difference is idiom: in America, amusing oneself is harmlessly waiting for another; in France … well, it's still harmless, but that's not the point.

Programming idiomatically means understanding a language at a level deeper than its syntax. In Python, it means using list comprehensions rather than explicit iteration, or a generator rather than a physical list. In Java, it means using enum rather than constants. In any object-oriented language, it means “tell, don't ask.”

A hallmark of programmers who are new to a language is their lack of idiom. They write the new language as if it were the old, often bemoaning the loss of their familiar constructs (I'm no different: as I was learning Java I wished for C-style function pointers). There's an old saying that “You can write Fortran in any language.”

A danger of idioms is that they can turn into slang, used to add color rather than value, a way to signify membership in a clique. Duff's device is an extremely clever use of C's switch statement, and arguably a brilliant optimization (although I have trouble believing that the cycles saved from unrolling loops weren't immediately lost waiting for the IO register). But when used for copying memory, it loses all cleverness: replacing a CISC hardware instruction with discrete moves, or preventing a RISC compiler from applying its own optimizations.

More dangerous are the programmers who never spend enough time with a language to learn its idioms. They are the ones who code randomly: an ArrayList here, a LinkedList there, reading all rows from a database so that they can do a sum on the client. Exposure to different languages can be a great learning experience, as long as you learn enough to know when someone wants a little time alone.

Short Attention Span Theatre, or, What Happens When Your Page Becomes Delicious

Last week my article on Java reference objects managed to find its way to the “popular Java” page of delicious.com (and, within a day or two, drop off again). Normally, this article gets hit a dozen times a day, usually by people who are Googling for “how to fix out of memory errors.” During the week that it appeared on delicious.com (which was preceeded by a lot of tweeting), it was getting hit 150+ times a day — including 49 times in short order from one London-based ISP (which I assume was a proxy).

What was interesting to me, however, was that almost none of the people visiting this page went to any other page on the site — and of the ones that did, more went to the “food” section than to the “programming” section. And they tended to do so quickly: under a minute between initial page hit and moving on. I know that it takes longer than that to read about reference objects.

To an author, this should be a blow to the ego: is my writing so bad that people don't want to read more of it? It's like they picked up my book, read the dust jacket, and put it back down. But I refuse to accept that conclusion. For one thing, I have a strong ego. But more important, enough people bookmarked the page for it to appear as a popular link.

The real answer, I think, is that people in the software industry are very quick to jump on new ideas, but don't have a lot of follow-through. And why should they, when there's a plethora of well-known pundits saying they should explore as many new things as they can, and there's always a new popular link showing up on delicious.com?

The problem with this attitude is that exposure to a lot of ideas doesn't mean that any of them will stick. In fact, I'd say the opposite is true: if you don't take the time to examine an idea and see how it fits with your existing knowledge, you're not going to ever make use of it.

Building a Wishlist Service: External Libraries

One of the great things about developing Java applications is the wealth of open-source libraries available. The Jakarta Commons libraries have been part of every project that I've done over the past five years; you can find StringUtils.isEmpty() in almost all text manipulation code that I write.

However, the wealth of open-source libraries presents a paradox of choice: which libraries do you use for a project. Each library adds to the memory footprint of your project, either directly as classes are loaded, or indirectly as the JVM memory-maps the library's JAR. External libraries also make dependency management in your build more complex, in some cases forcing you to build the library locally.

More important, every library represents a form of lock-in: once your code is written to conform to the library, it will be expensive to change. And if you discover a bug or missing feature, you'll need to develop a remediation plan. Even if you can code a patch, it will take time to integrate with the mainline code — assuming that it is accepted. In some cases you may find yourself maintaining a private fork of the library over several public releases.

All of which is to say: use open source, but pick your libraries carefully.

In the cases of the product list service, one of the places where I considered external libraries was XML management, in particular conversion between XML and Java beans. There are lots of libraries that handle this: XMLBeans and XStream are two that are commonly used, and the JDK provides its own serialization and deserialization classes as part of the java.beans package.

Of these, XStream seemed to be the best choice: XmlBeans requires a separate pre-compile step, while the JDK's serialization format would require a lot of work on the part of any non-Java client. However, I had another alternative: I am the administrator and main developer on Practical XML, an open-source library for XML manipulation. It didn't support XML-object conversion, but I also had some converter classes that I'd written before XStream became popular. I figured that it would take a minimal amount of work to flesh out those classes and integrate them into the library.

I have an incentive to evolve the Practical XML library, and to use it in all of my projects. However, adding this functionality introduced a two week diversion into my project. In this case the delay didn't matter: I have no hard deadlines on this project. And since I was already using the library in other places, I had the benefit of consistency and reduced footprint. Faced with an unmovable ship date, my decision would have been different.

Building a Wishlist Service: Template Method

When I'm giving an interview, one of my favorite questions is “Are you familiar with Design Patterns? Please describe your favorite pattern, tell me why you like it, and give an example where you've recently used it.” Historically, if I were to be asked that question, my answer would be Strategy; currently, I think it would be Template Method. Which is interesting, because those two patterns basically do the same thing from different directions.

A description first: Template Method is a pattern for building class hierarchies in which the base class of the hierarchy imposes logic on its subclasses. The base class defines public methods, then calls into abstract protected methods that must be implemented by the subclass. For example, the base class might define a validate() method:

public final void validate()
{
    validateEncryptedParam();
    validateExpiration();
    validateRequiredParams();
    subclassValidation();
}

Historically I've liked Template Method because it avoids bugs caused by subclasses that don't invoke their super's implementation of a method (I take the attitude that “subclasses are like ogres,” which is a topic for a future post). However, as I'm realizing with this service, it also highlights duplicated code, and allows that code to be moved into the superclass. In this example, there's only one method that's actually implemented by the subclass (and its name should make it obvious).

I think it also pushes you toward moving responsibilities out of the class hierarchy entirely. In this example, validateRequiredParams() was originally intended for the subclass, which knew what parameters it needed. But the actual validation code was common to all subclasses, so I changed the method to simply ask the subclass for a list of its required parameters. A little more thought, and I realized that this isn't really an attribute of the subclass, but of the operation itself. So I added the list of parameters to the enum defining the service's operations (and this spurred writing an article for my website):

public enum Operation
{
    // only the third example has a required parameter

    RetrieveList(HttpMethod.GET, null),
    AddItem(HttpMethod.POST, ProductEntry.class),
    DeleteItem(HttpMethod.POST, null, RequestParameter.itemId),
    // ,,,
}

This kind of thinking, taken to an extreme, will take you out of Template Method and into objects constructed according to a configuration file. And that's not a bad thing: if your subclasses don't have behavior, they don't deserve to exist.

Building a Wishlist Service: Testing

There are a lot of ways to test a web application: in-container testing using tools such as Cactus, full client-server testing using tools such as HttpUnit, out-of-container unit tests, and even manual tests.

I'm not a fan of any test approach that requires starting the application, because you won't want to do that on every build. So most of the tests for my service are unit tests, meant to run outside of the container. And although I knew that this would happen, I'm still amazed at how such tests push me toward decoupled code, even when I write the tests after the code.

The servlet's dispatch mechanism is a prime example. Going in, I knew that I wanted a RequestHandler class for each operation. My first pass used an interface, with a getRequestHandler() method inside the servlet class. However, the only way to test that method was to make it public or protected, and I'm not willing to violate encapsulation for testing. So RequestHandler became an abstract class, and getRequestHandler() became a static factory method. At the same time, I decided to instantiate a handler object per request, rather than reuse a table of singleton objects: the latter was easier to manage, but the former was easier to test.

Unit-testing servlet components means a lot of stubbing and mocking, and I decided to create a reflection proxy for the servlet request and response objects. I realize that there are existing mock implementations for these objects, but I figured that I wasn't using many of their methods and wanted to limit the use of third-party libraries (that's the topic of another post).

And that led to another observation: writing your own mock objects tends to reduce the number of places you touch those objects. If I have to mock both getParameter() and getParameters(), I'm going to think about why I call both methods, and probably use just one of them. This should translate to reduced chance of errors, in this case because I'll be cognizant of cases where there may be more than one parameter with the same name.

There's another effect that I'm seeing from writing for testability: I tend to use the Template Method pattern. A lot. Or perhaps it's simply a natural pattern for web services. I'll look at that more closely in the next post.

Building a Wishlist Service: The Development Environment

Working software is the primary measure of progress

There's a great feedback loop from seeing your code run in situ. Even if you're religious about writing tests, the green bar doesn't match up to pressing a “Submit” button. I think it's one of the reasons that interactive languages like Python have such a devoted following. And a big part of this feeling is the thought that “I could demo this to someone who doesn't know how to code.” Or, as you get closer to the end of the project, “I could ship this.”

This feedback loop works even if you're a one-person team, which is why I started my wishlist project with the file demo.jsp. At first this page held a hard-coded form, which allowed me to invoke the skeleton of a servlet class. From there to documentation: writing up my thoughts on the service API. And then implementing those thoughts, in data transfer objects, JSP taglibs, and more servlet code. It's a circular process: update the JSP, update the servlet, update the docs. Of course, there are tests supporting all the servlet changes, but that web page stays on the screen, ready to submit whatever form data I need.

My development tools help with this process. Eclipse is my primary IDE: I like its editor, debugger, and refactoring tools. But I don't like its support for web applications (to be fair, the last time I tried it out was four years ago; things may have gotten better). For the web-app side, I use NetBeans 5.5, which comes bundled with a Tomcat server (6.5 gives pretty much the same environment but uses an external server; there's a level of comfort from using tools released together). I can build and deploy in a few seconds, then switch right back to either editor to make changes.

For production packaging, of course, I'll need to move to a Maven build. NetBeans is a nice rapid prototyping tool, but I don't like the way that it forces you to use its own Ant scripts. But that's a ways down the road, and Subversion should help me keep my sanity as I break the pieces into three separate build projects. For now, my prototyping environment is rapid enough that I don't feel the need for an interactive language.

Design versus Implementation

My last few postings have discussed the design of my wishlist service in very broad strokes: here's what I want to accomplish, here are a few of the issues that I've thought about. They have no detail; they don't even look at choice of platform. To me, this is the essence of the divide between design and implementation.

I see it as analogous to traditional building architecture, where the lead architect sketches broad forms, the rest of the architecture studio fleshes those out as models, and the mechanical engineers make it all work. It's also similar to a corporate mission statement — at least, a mission statement for a corporation that isn't floundering in indecision.

If the design is complete, then the implementation should flow naturally. My design specifies a RESTful interface using either XML or form-encoded data in the request; this implies that my implementation will have a front-end layer to transform these inputs into a common form. The design calls for separation between base product data and any application specific data; the data model will reflect that, and the data access code won't need to perform joins or multiple queries to load objects.

There are decisions that don't derive from the design: for example, my choice of Java and Servlets. This decision is one of convenience: it minimizes the amount of stuff that I have to learn. Other choices include Java/Spring, Python/Django, and Ruby/Rails. While I have some familiarity with Python, and could learn Ruby in short order, any code that I write with those would not be idiomatic — it would probably look like Java with a different syntax. Eventually, I plan to use the service as a basis for exploring Python and Ruby in an idiomatic way, but the first pass should leverage what I know.

This dividing line still leaves a lot of implementation decisions to be made, and the next several postings will look at those decisions and the implementation process itself.

Designing a Wishlist Service: Scalability

One of the repeated themes in Founders at Work is how startups were unprepared for scalability issues, particularly in the cases where the company experienced viral growth. In some companies, the founders worked around the clock adding capacity. Viral growth is by its nature unpredictable: you have no idea how fast — or how long — your company will keep growing, and going in you have no idea who is going to like the product. The rest of us, however, should have an intended audience for our product, and can make design decisions based on that audience.

There's a lot of conventional wisdom on designing for scalability, most of which boils down to “cache everything” and “don't maintain session state.” The trouble with conventional wisdom is that it rarely applies to the real world. Caching, for example, is useless if you're constantly changing the data, or if a particular data item is accessed infrequently. And a RESTful web service doesn't store session state by its very nature.

So what usage pattern should we expect from a wishlist service? I'm designing for three types of user:

Wishlist Owner

This person will add and remove items, and perhaps change their rankings. These interactions all happen on human timescales (ie, several seconds or minutes between changes, then long periods of inactivity), and should involve more writes that reads. For this user, caching is not going to help performance: the list will probably age out of the cache before it's read again, and would need to be constantly purged as items change.

Friends and Family

Wishlist owners may choose to share their wishlist, in the hopes that other people will do the buying. On the surface, it appears that caching might be a good idea: after all, there may be a lot of people accessing the service. But again, we're looking at human timescales: the list will be shared by passing a link, probably in an email. While you may send such an email to a hundred people, they're not going to open it at the same time, so there's no point in caching.

Collaborative Shoppers

One of the original triggers for this project was a collaborative shopping tool, where several people would share a list and make updates (typically comments), using a chat service to broadcast changes. This is one situation where caching would be useful, since there will be a group of people hitting the same data at the same time. They'll also be updating the data constantly, meaning the lifetime of a cached object would be very short.

However, I think that the solution for this scenario is to use caching in front of the application rather than as part of it: either Apache's mod_cache or some sort of caching filter in the application server. To make this work, we need to use URLs that identify the version of the list along with other key data — but this is the RESTful approach anyway.

By coming up with this picture, I can implement the service as a simple database front-end; I don't need to waste time writing explicit code for scalability. And if it goes viral … well, let's just say I don't expect that, particularly with an eCommerce server gating access.

Designing a Wishlist Service: Security

Security is the red-headed stepchild of the web-service world. Sure, there's WS-Security, which may be useful if you're using SOAP in server-server communication. But most services, in particular, browser-accessed services, rely on the security mechanisms provided by HTTP and HTTPS. If you want authentication, you use a 401 response and expect the requester to provide credentials in subsequent requests. If you're paranoid, you use a client certificate.

The wishlist service doesn't quite fit into this model. There's a clear need for some form of authorization and control, if only to protect the service from script kiddies with too much time on their hands. Yet authentication has to be transparent for usability: we don't want to pop up login dialogs, particularly if the customer has already logged in to their eCommerce account.

Perhaps the biggest constraint on any security mechanism is that requests will come from a browser. This means that the service can't expect the requester to provide any secret information, because anybody can view the page or script (I'm amazed by just how many web services require a “secret key” in their URL). If a page needs to provide secret information, that information must be encrypted before it is delivered to the user.

And secret information is required, because there are different levels of authorization: some people can update a list, others can only read it. The solution that I chose is to encrypt the wishlist key and user's authorization level, and pass that to the service as a URL parameter. This does represent another departure from pure REST, in that there may be multiple URLs that refer to the same underlying resource, but it seems the most reasonable compromise.

It also has the limitation that anybody who has a URL has the access defined by that URL. In some web services, this would be a problem. In the case of the wishlist service, it's mitigated by several factors.

First among these is that the value of this data just isn't that high. Sure, if you plan to be nominated for the Supreme Court, you probably don't want to keep a wishlist of X-rated videos. But for most people, a wishlist of clothing just isn't that important (and an opt-in warning can scare away those who feel differently). A related factor is that, in normal usage, people won't be handing out these URLs — at least, not URLs that can make changes. There's simply no reason to do so.

A second mitigating factor is that each URL also has an encrypted timeout: the service will reject any requests that arrive after that timeout. While this can be used to exert control over shared lists, it is primarily intended to defeat script kiddies who might try a denial-of-service attack via constant updates.

A third mitigating factor is that we keep close control over secrets, particularly those that can be used to break the encryption — in other words, security through obscurity. Bletchely Park would not have been nearly so successful at breaking Enigma if they hadn't learned to recognize radio operators. Taking heed of that, none of the information used by the encrypted parameter is provided in plaintext, meaning that any attempts at discovering the key will require heuristic analysis rather than a simple String.contains().

Designing a Wishlist Service: Interface

One thing that has always struck me about web services is how hard they are to invoke from a web browser. SOAP requests are the worst, with the overhead from its envelope, but even simple XML services leave a lot to be desired. Particularly since Internet Explorer, the most popular browser currently in use, doesn't support the E4X extensions to JavaScript. This has led to diverging paths: either a web service is expected to be used in a browser, and is built around POST data and JSON, or it's meant to be called from another server, and is built around an XML protocol.

For the product list service, I wanted both. My main use case is browser-based requests, but I also want to support Flex application (which prefer XML) as well as server-server requests (such as managing a cart). This led me down the path of REST services.

“REST” seems to be applied to any web service that uses XML and isn't SOAP, so before continuing I should state what it means to me:

1. GET is used for retrieval, POST for update. REST purists will now be upset with me, saying “what about PUT and DELETE?!?” My answer: when HTML supports PUT and DELETE as form methods, and every browser allows them, then we can talk about using them. Until that time, practice trumps theory.
2. The URL uniquely and completely identifies the resource and action. Again, REST purists will object to including the action in the URL, but that's a necessity if we can't use PUT and DELETE.
3. The request and response bodies contain only data. Far too many “RESTful” web services are really XML-RPC in disguise, particularly for update operations.
4. HTTP status codes are used to signal success or failure. For any response other than 200 (Success), the response body will contain error information.

The result is a very simple interface. My desire to support both POST/JSON and XML requests introduces a slight wrinkle, one that's resolved front-end code that identifies the content type and converts to and from a JavaBean representation.

One thing that REST doesn't contemplate, however, is security. That's the next post.

Designing a Wishlist Service: Feature Set

I've been referring to this project as a “wishlist,” but it's really a “product” list. Product lists appear everywhere in eCommerce, with different names and purposes: wishlists and registries keep track of products that we'd like others to buy for us; favorites and re-order lists keep track of products that we like to buy for ourselves, and the cart holds products that we're buying right now. Sometimes these lists will be shared with others, sometimes they won't. Some of these applications have different data needs: quantity on the registry and cart, versus ranking on the wishlist and favorites. From a design perspective, which features are important and which aren't? Particularly if I'm planning to sell this product to multiple sites?

Before we can think about what data gets stored, however, we need to think about how the list itself will be identified. One approach is to give each list a unique identifier, and have all requests use that identifier. I rejected this approach, because it requires any potential clients to modify their database schema, adding in a lookup table to associate this unique ID to a customer or whatever data they use to identify the list.

Instead, I decided to use a two-part key of customer ID and wishlist name. Every eCommerce site has some notion of a customer ID, so I'm reusing existing data. By giving wishlists names, access to a particular wishlist can be controlled by page-level code — no need to modify the database. These two key components are managed as character strings, with no interpretation by the service: as long as the site uses reasonable-length identifiers, they will work.

The next question is how to identify each product entry. Here I went with another composite key: product ID and sub-product ID. This is a nod to my past experience at GSI, which used a second-level ID to identify the particular size/color of a base product. Looking at other clothing sites, this appears to be a common practice, and sites that don't use such an ID can simply leave the field empty. Again, the values are defined as strings, without any interpretation by the service.

Finally, we get to the information associated with each product. And here I decided that the least information stored, the better: each entry has fields for rank and quantity, nothing else. This is an application of the Agile YAGNI principle (you ain't gonna need it): I've identified cases where these two fields will be used, but don't know what others might be needed. I do, however, recognize that other data might be needed, so attached a free-form string (CLOB) to each entry: should a particular site want to store data there, they can. It won't be interpreted by the service, but will be persisted with the rest of the entry.

Entries will also have a list of comments attached to them. This feature was driven by a specific collaborative shopping application, although I think it will be useful in multiple places (for example, a shared favorites list in which the customer describes why the product is a favorite). These comments are not associated with the product per-se, they are managed via separate service calls.

This brings me to the most interesting (to me) feature of all: how the client will access the service. That deserves a posting of its own.

Why a Wishlist?

From an eCommerce perspective, wishlists are far from the leading edge. Does the world need another one, and more important, is it marketable?

There are several answers to this. While wishlists (or registries) have been a part of eCommerce for many years, it's surprising how few sites actually use them. Perhaps this is because sites tried out a wishlist, discovered that it didn't measurably add to sales, and got rid of it. Yet Amazon continues to enhance its wishlist offering. And in my former role, I talked with many clients that wanted features that our wishlist offering didn't have.

Chief among these desired features was collaboration: not only sharing a list, but allowing other people to update or comment on it. This leads to some interesting questions about managing privacy and control, and I didn't see good answers to those questions in existing products. I think I've got those answers.

Existing wishlist implementations are also tightly integrated with their host site, typically (as with Amazon) accessed via dedicated pages. However, the way that people approach eCommerce is changing: from browser-based collaborative shopping, to the opening of new channels such as social networking sites. To me, the future will be based on federated services, accessed via rich browser apps, and my wishlist service is squarely in that space.

And finally, there's a personal angle: although I've been working with the J2EE stack for 10 years, and even co-developed a servlet-enabled web-server, I've never implemented a production web-application “from scratch.” This project is a way to cement my knowledge, and also produce a template app that can be adapted for other purposes.

The Paradox of Free Time

It's been a month since my last blog posting. That's a common thing with technology bloggers big and small, often followed by an apologetic posting about how busy work has become. But I'm unemployed, so can't use that excuse … except that's exactly what happened!

OK, there were a couple of vacations interspersed. One was a motorcycle trip, and packing a laptop on a motorcycle is difficult (at least, if you want to keep it dry and unbroken). Another was spent in cabin in the woods, without Internet access. And while I suppose I could post from the iPhone, you don't want to read that (or more correctly, decipher it). But in reality, through much of July, I was at home and sitting at the computer for at least six hours out of the day (and standing on a ladder the rest of the time).

It turns out there's a paradox to having free time: when I was working for someone else, I would carve out an hour of each day during which I'd write. But now, I have the entire day to schedule as I wish, and I've been spending the time working on other projects. One of those projects is a wishlist service, and starting tomorrow, I'll be posting a series of articles about its design and development.

Ladder Safety

When I bought my house, one of my thoughts was “brick! I don't need to paint!” Over the past six years, I've learned just how much wooden trim is on a brick house. And I've learned that after 75 years, the layers of paint can be measured with a ruler. And so this week, I found myself on a ladder again, heat gun in one hand, putty knife in the other. And I tried to think of a way to turn the experience into a programming-related blog posting.

When you're standing on a ladder, there's really only one safety rule: keep your hips between the ladder rails. As long as you remember this rule, you can twist your upper body and stretch your arms as far as they can reach. You can stand on one foot. The ladder isn't going to move — provided, of course, that it was set up properly (feet on solid ground, correct angle, both tips in contact with the wall).

Which, of course, brings us right back to the idea of reinforcing practices. If you set up the ladder with one foot on concrete and the other in mud, it doesn't matter how careful you are about body position. Sooner or later one side will sink, and then you'll fall. Extending the metaphor, proper ladder setup doesn't mean that you're going to get a decent paint job.

Speaking of which: the heat gun quickly bubbled the old paint, making it (relatively) easy to remove. Yet some adjacent boards, which I'd pre-painted before installation a few years ago, were unaffected. I'm sure there's a programming metaphor to be found there…

The Resume

It's time to update my resume again.

It's starting to get too long: I've never given much thought to keeping it at a single page, but the HTML version is now at five printed pages, covering 17 years. One solution is to drop older entries, but I'd really like to keep my Fidelity experience listed: it was a major part of my career, and shows I've been something other than an individual contributor.

There's also the matter of the number of entries. When I first entered the professional workspace, my recruiter told me that the average job span was two years. I've kept pretty close to that, at least as a full-time employee, although my contract jobs tended to be a year or less. However, a bunch of short gigs raise red flags, even when listed as “contract.” That's always been a peeve of mine, since the sign of an advancing career within the same company is a new position every couple of years. I simply do it with different companies.

The skills section doesn't need too much editing. I've always taken the attitude that I should be able to stand up to an experienced person quizzing me on any skill that I've listed, and when sitting on the other side of the interview table, I tend to do just that. The database section needs some work — Teradata? what's Teradata? Actually, it's been long enough that I don't think I can call myself a “database programmer” anymore, so maybe I should just merge it into another category.

And then there's the readability issue, in particular the issue of whether I'm writing for a human or a search engine. Search engines have been good to me: I've seen many hits in my access log from people using them, and gotten more than a few follow-up emails (which is impressive, given how I don't provide an easy email link on my website). But ultimately, it's a person who's reading, who can get bored, who can throw it in the trash based on font choice or layout. HTML has some decided negatives here: no matter how much I tweak the CSS, I'm not going to match the layout that I had with Word.

One thing that I don't plan to change is the structure of the entries: early on I decided that my resume would speak to what I've done, rather than how I've done it. That causes problems with a lot of recruiters: they want to see buzzwords everywhere, on the belief that hiring managers want the same. And perhaps some hiring managers do want that, but I've decided that I don't really want to work for those managers. I'd rather work for someone who wants the same things I do.

Resources

Resources are working on the issue.

If there's one word that I'd take out of the corporate lexicon, it's ”resources.” It seems harmless enough. After all, you can't complete a project unless you have the resources to do so. Some of those resources are people, some are materiel. Yet wherever I've seen people referred to as resources, I've seen a dysfunctional company.

At first glance, the reason seems obvious: when you dehumanize someone, there's no reason to consider them in your decisions. Do you need 80 hour weeks to get the project done? No problem, we have the resources! Don't have anyone with the skillset to do a job? That's nonsense, just move a resource into the slot! These, and similar, are fodder for a host of TV shows and movies … as well as more than a few real-world companies.

But the damage goes further than that: it separates people from their work, giving them an excuse not to care. That quote at the top of this article? It came from the production support team, not some random manager. Did you notice that it doesn't say we are working on the issue?

In a world of “resources,” there's no reason for an individual to feel responsibility. After all, he or she is just a cog in the wheel. To be replaced or re-assigned at the drop of a hat. And once you adopt that attitude, it really doesn't matter what you produce. It's a short step from there to ignoring broken windows.

This is not news, and it's not an issue confined to the software industry. One of the essays in a rather obscure book on railway locomotives, first published in 1959, discusses the effects of crew assignment on the well-being of the trains themselves. Not surprisingly, the engines that “belonged” to a single crew or small number of crews were consistently in top shape. Engines from a “common pool,” which would be crewed by whomever was available, were consistently in the worst shape.

Even on an assembly line, the home of interchangeable jobs, people are not interchangeable. You can't take a person with ten years of experience threading wiring harnesses, and drop him/her into a welding station. Or vice versa. Unless you actually want to lower product quality and create unhappy workers.

The real issue is why the belief in “resources” persists, even among the people so labeled. I suspect that it comes from a feeling that humans can adapt to anything; that “specialization is for insects.” While that may be true in the long run, in the short run it's just wishful thinking. And being told “you are not a beautiful or unique snowflake” is bound to leave a few psychic scars.

Teams as Reinforcement

One of the four principles of the Agile Manifesto is “individuals over process.” It's one of the first things that I think gets missed when people adopt agile practices, but it's probably the most important.

A few years ago, I worked for a company that had a “quote page” on the development wiki. If one of your coworkers said something memorable, you'd be honor-bound to record it for posterity. One of my favorites:

I'd check this in, but James would kill me.

OK, perhaps to you this seems like a case study in hostile work environments. To me, knowing the people involved, it's a case study in high-quality software development. The person who said this realized that he had taken a shortcut, and wasn't willing to be called out for it. So he went back and did a better job.

As I've written recently, I don't think you can adopt individual practices such as test-driven development and expect your team to succeed. It's too easy to take shortcuts. Instead, you need reinforcing practices: TDD mixed with pair programming or code review, for example. And the best reinforcement is to have a team that is unwilling to disappoint each other.

I've been lucky enough to work on several high-performing teams, and to lead one of them. I've also been unlucky enough to work on several dysfunctional teams: teams that sometimes produced something that worked, but it didn't have very high quality, and the developers weren't very happy doing it. And the common thread of the high-performing teams was that the individuals on those teams were driven by maintaining each others' respect — not by money, and not even by the end result.

The desire not-to-disappoint is a powerful emotion. In its extreme form, guilt, it's a staple of one-liner comedians and mediocre managers. It's also incredibly difficult to maintain. Unless you focus on people over process.

Reinforcement

My postings on SkipList finish with observations and concerns, but no conclusions — no resolution to my concerns. Actually, when I originally posted the article, there was a concluding paragraph, but I decided that it deserved its own posting.

So, the concern: test-first-development, or even test-driven-design, is a great thing for experienced programmers. It provides a sandbox in which to explore ideas and design, with the safety — fearlessness, in the words of TDD proponents — to change code without breaking it. In the hands of an inexperienced programmer, however, I believe that it can turn into a blind alley, through a process in which small rational decisions lead to an irrational result. I can easily imagine the dead mother counter arising from TDD.

So what's the solution? Well, one simple approach is to have an experienced programmer look at the tests. Code reviews are an accepted part of many development processes, but they usually focus on the mainline code. While that's certainly important, it seems less-so when you have high test coverage (as you will in a pure-TDD process). The tests, on the other hand, not only tell you whether the requirements were met, but give insight into the mind of the programmer. If you see a series of tests that don't make sense, it's a flag to look at the mainline code: there's probably something in it that doesn't make sense either.

Another benefit of code review for tests is that it can happen early and often. Mainline code reviews generally happen after the code is done, because that's when you can see the logic flow. Test reviews, however, can happen at any time: even if your mainline method/class is half-written, you should still have a bunch of tests for it.

Taken to the extreme (pun intended), you'll have a person sitting by your side and reviewing the tests as you write them: pair programming. I imagine that most readers are cringing after that last sentence. Programmers, because who wants someone looking over your shoulder, and managers, because pair programming is a great way to take two people and get the output of one.

But is it really? If your shop has code reviews as part of its process, have you ever considered how many person-hours goes into one? As a case study from my own career, one shop scheduled 2 to 3 reviews a week, with the presenter and 3 other developers. A review took about an hour of preparation, and an hour of meeting time, giving a total of 8 person-hours. And those reviews covered only the “important” code: perhaps 10% of the codebase.

So, 2-3 developer-days per week spent on code reviews, with more time spent by the reviewed developer to clean up any issues. Of course, you can do less rigorous reviews, but where's the value in that? It's too easy to end up with a review “process” that's nothing more than a formality: a rubber stamp, as overworked developers take at best a cursory look at the code under review. And if you're in that situation, the time is well and truly wasted.

Pair programming is a practice espoused by many Agile processes, in particular Extreme Programming (XP). Now, to be fair, I've never worked in a shop that adopted XP, so you might reasonably ignore what follows. However, I have followed many of the XP practices (including limited pair programming) at other shops, so am not completely talking through my hat.

And looking at the various practices — and also at shops that claim to practice Agile but never seem to reap any benefits — I've come to the conclusion that the real benefit of XP is that the various practices reinforce each other. OK, this isn't a great insight: it's right there on the front page of extremeprogramming.org: “The rules and practices must support each other.”

But that message seems to get lost. Indeed, if you click the “How do I start” link on that page, you'll see a suggestion that you can add a few practices at a time. This is not what I remember when I was first exposed to XP: at the time, the prevailing opinion was that you should start with all dozen or so practices, and eliminate practices only once your team had enough experience to do so with reason.

Reinforcement is important. It's too easy, when you see a deadline looming, to say “there's no need to write tests for this code, it's simple.” It's surprising how often that “simple” code ends up harboring a bug that doesn't turn up until you hit production. It's a lot harder to skip that test when the person sitting next to you raises his/her eyebrow and asks why.

Which brings me to what should be the strongest reinforcement of all: not disappointing your coworkers. But that's another posting.