Building a Wishlist Service: The Development Environment

Working software is the primary measure of progress

There's a great feedback loop from seeing your code run in situ. Even if you're religious about writing tests, the green bar doesn't match up to pressing a “Submit” button. I think it's one of the reasons that interactive languages like Python have such a devoted following. And a big part of this feeling is the thought that “I could demo this to someone who doesn't know how to code.” Or, as you get closer to the end of the project, “I could ship this.”

This feedback loop works even if you're a one-person team, which is why I started my wishlist project with the file demo.jsp. At first this page held a hard-coded form, which allowed me to invoke the skeleton of a servlet class. From there to documentation: writing up my thoughts on the service API. And then implementing those thoughts, in data transfer objects, JSP taglibs, and more servlet code. It's a circular process: update the JSP, update the servlet, update the docs. Of course, there are tests supporting all the servlet changes, but that web page stays on the screen, ready to submit whatever form data I need.

My development tools help with this process. Eclipse is my primary IDE: I like its editor, debugger, and refactoring tools. But I don't like its support for web applications (to be fair, the last time I tried it out was four years ago; things may have gotten better). For the web-app side, I use NetBeans 5.5, which comes bundled with a Tomcat server (6.5 gives pretty much the same environment but uses an external server; there's a level of comfort from using tools released together). I can build and deploy in a few seconds, then switch right back to either editor to make changes.

For production packaging, of course, I'll need to move to a Maven build. NetBeans is a nice rapid prototyping tool, but I don't like the way that it forces you to use its own Ant scripts. But that's a ways down the road, and Subversion should help me keep my sanity as I break the pieces into three separate build projects. For now, my prototyping environment is rapid enough that I don't feel the need for an interactive language.

Design versus Implementation

My last few postings have discussed the design of my wishlist service in very broad strokes: here's what I want to accomplish, here are a few of the issues that I've thought about. They have no detail; they don't even look at choice of platform. To me, this is the essence of the divide between design and implementation.

I see it as analogous to traditional building architecture, where the lead architect sketches broad forms, the rest of the architecture studio fleshes those out as models, and the mechanical engineers make it all work. It's also similar to a corporate mission statement — at least, a mission statement for a corporation that isn't floundering in indecision.

If the design is complete, then the implementation should flow naturally. My design specifies a RESTful interface using either XML or form-encoded data in the request; this implies that my implementation will have a front-end layer to transform these inputs into a common form. The design calls for separation between base product data and any application specific data; the data model will reflect that, and the data access code won't need to perform joins or multiple queries to load objects.

There are decisions that don't derive from the design: for example, my choice of Java and Servlets. This decision is one of convenience: it minimizes the amount of stuff that I have to learn. Other choices include Java/Spring, Python/Django, and Ruby/Rails. While I have some familiarity with Python, and could learn Ruby in short order, any code that I write with those would not be idiomatic — it would probably look like Java with a different syntax. Eventually, I plan to use the service as a basis for exploring Python and Ruby in an idiomatic way, but the first pass should leverage what I know.

This dividing line still leaves a lot of implementation decisions to be made, and the next several postings will look at those decisions and the implementation process itself.

Designing a Wishlist Service: Scalability

One of the repeated themes in Founders at Work is how startups were unprepared for scalability issues, particularly in the cases where the company experienced viral growth. In some companies, the founders worked around the clock adding capacity. Viral growth is by its nature unpredictable: you have no idea how fast — or how long — your company will keep growing, and going in you have no idea who is going to like the product. The rest of us, however, should have an intended audience for our product, and can make design decisions based on that audience.

There's a lot of conventional wisdom on designing for scalability, most of which boils down to “cache everything” and “don't maintain session state.” The trouble with conventional wisdom is that it rarely applies to the real world. Caching, for example, is useless if you're constantly changing the data, or if a particular data item is accessed infrequently. And a RESTful web service doesn't store session state by its very nature.

So what usage pattern should we expect from a wishlist service? I'm designing for three types of user:

Wishlist Owner

This person will add and remove items, and perhaps change their rankings. These interactions all happen on human timescales (ie, several seconds or minutes between changes, then long periods of inactivity), and should involve more writes that reads. For this user, caching is not going to help performance: the list will probably age out of the cache before it's read again, and would need to be constantly purged as items change.

Friends and Family

Wishlist owners may choose to share their wishlist, in the hopes that other people will do the buying. On the surface, it appears that caching might be a good idea: after all, there may be a lot of people accessing the service. But again, we're looking at human timescales: the list will be shared by passing a link, probably in an email. While you may send such an email to a hundred people, they're not going to open it at the same time, so there's no point in caching.

Collaborative Shoppers

One of the original triggers for this project was a collaborative shopping tool, where several people would share a list and make updates (typically comments), using a chat service to broadcast changes. This is one situation where caching would be useful, since there will be a group of people hitting the same data at the same time. They'll also be updating the data constantly, meaning the lifetime of a cached object would be very short.

However, I think that the solution for this scenario is to use caching in front of the application rather than as part of it: either Apache's mod_cache or some sort of caching filter in the application server. To make this work, we need to use URLs that identify the version of the list along with other key data — but this is the RESTful approach anyway.

By coming up with this picture, I can implement the service as a simple database front-end; I don't need to waste time writing explicit code for scalability. And if it goes viral … well, let's just say I don't expect that, particularly with an eCommerce server gating access.