Packaging Java for AWS Lambda

When I want to package a Java application in a single JAR with all of its dependencies, I normally turn to Maven’s Shade plugin. This plugin builds an “UberJAR” by unpacking all of the files from the project's dependencies and repackaging them into a single JAR.

The Shade plugin is also used by the Lambda Developer's Guide. I was surprised, then to see this later in the same guide:

Reduce the time it takes Lambda to unpack deployment packages authored in Java by putting your dependency .jar files in a separate /lib directory. This is faster than putting all your function’s code in a single jar with a large number of .class files.

I was surprised because a single JAR containing class files should be the most efficient way to deploy a project. The JVM memory-maps the JAR files on its classpath, which means that it can access arbitrary parts of those files without an explicit call to the OS kernel. And UberJAR means that the JVM only has to examine a single directory structure.

After some investigation, I discovered the reason: Lamdba unpacks the contents of the deployment package into its filesystem. Which means that an UberJAR turns into a lot of little files, each of which must be read using several kernel calls.

OK, so if you’re a Maven user, how do you package your Lambdas in the prefered way? The answer is that instead of using the Shade plugin, you use the Assembly plugin. The stated goal of this plugin is to allow developers to package their project artifact with related information such as a website. But it's a very flexible plugin, using a “deployment descriptor” to specify which files should be included in the final artifact, and where they should go.

This is the descriptor that I used for my Java Lambda example. It's duplicated for each of the Lambdas, and is found at src/assembly/deployment.xml:

<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.0.0 http://maven.apache.org/xsd/assembly-2.0.0.xsd">
    <id>deployment</id>
    <includeBaseDirectory>false</includeBaseDirectory>
    <formats>
        <format>zip</format>
    </formats>
    <dependencySets>
        <dependencySet>
            <outputDirectory>/lib</outputDirectory>
            <useProjectArtifact>true</useProjectArtifact>
            <unpack>false</unpack>
            <scope>runtime</scope>
        </dependencySet>
    </dependencySets>
</assembly>

The descriptor is simple: it says that every JAR used by the Lambda — including the one built by the project itself — should be stored in the /lib directory of the final artifact. To use it, I replace the Shade configuration in my POM with this (the plugin.assembly.version property defined as 3.2.0):

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-assembly-plugin</artifactId>
    <version>${plugin.assembly.version}</version>
    <configuration>
        <descriptors>
            <descriptor>src/assembly/deployment.xml</descriptor>
        </descriptors>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
        </execution>
    </executions>
</plugin>

And that's it: when I run mvn package I get a file in my target directory with the suffix -deployment.zip.

Finalizers

I recently saw a comment on my reference objects article that said, in effect, “he mentioned finalizers but didn't say DON'T USE THEM EVER, so I stopped reading.” The capitalized words are taken exactly from the original comment; the rest is a paraphrase to tone down the rhetoric. The comment didn't bother me — if you stop reading so easily, you probably won't gain much from my articles anyway — but the attitude did. Because finalizers do indeed have a role to play.

They compensate for sloppy programming.

Good Java programmers know that non-memory resource allocation has to take place in a try / catch / finally construct. Just like good C programmers know that every malloc() must be matched by a free(). Sloppy programmers either don't know or don't care. And good programmers sometimes forget.

The question then becomes: what should the program do?

One alternative is to just explode. And this is not necessarily a bad thing: not closing your resources is a bug, and it's better to expose bugs early in the development process. In this view, forgetting to close your resource is no different than dereferencing a null pointer.

The problem with this view is that null pointer exceptions tend to show up early, the first time through the affected code. By comparison, leaked resources tend to hide until production, because you never generate sufficient load during development. Leaked file handles are a great example: a typical developer environment allows 4,096 open files. It will take a long time to run out of them, especially if you constantly restart the app. A typical server environment might allow twice or four times that much, but if you never shut down the app you'll eventually run out. Probably at your time of highest load.

And that's leads to an alternate solution: protect the programmers from themselves, by checking for open resources at the time the owning object is garbage collected. This isn't perfect: in a server with lots of memory, the garbage collection interval might exceed the time taken to exhaust whatever resource you're leaking.*

Once you accept the need for some alternate way to clean up non-memory resources, the only remaining question is how. Jave provides two mechanisms: finalizers or (since 1.2) phantom references.

So which should you use? Before answering that question, I want to point out something to the DON'T USE THEM EVER crowd: finalizers and phantom references are invoked under exactly the same conditions: when the garbage collector decides that an object is eligible for collection. The difference is what happens afterward. With phantom references the object gets collected right away, and the reference is put on a queue for later processing by the application. With finalizers, the object is passed to a JVM-managed thread that runs the finalize() method; actual collection happens once the finalizer runs.

Given the choice between a finalizer and a phantom reference, I will pick the finalizer 95% of the time. Phantom references require a lot of work to implement correctly. In the normal case, the only real benefit they provide is to avoid out-of-memory errors from large objects with long-running finalizers.**

A better answer is to avoid both. Far too many people think of finalizers as equivalent to C++ destructors, a place to put important cleanup code such as transaction commit/rollback. This is simply wrong. But it's just as wrong to invoke that code in a phantom reference.

Rather than dogmatically insist “no finalizers!”, I think the better approach is to adopt some development practices to prevent their invocation. One approach is to identify resources that are allocated but not released, using an analysis tool like FindBugs.

But if you want to recover from mistakes in the field, and can do so quickly and cleanly (so that you don't block the finalization thread), don't be afraid of finalizers.

---

* This is unlikely: if you're running through resources quickly, the associated objects will almost certainly be collected while still in the young generation. At this point, a pedantic pinhead will say “but the JLS doesn't guarantee that the garbage collector will ever run.” My best response to that: ”well, then you've got bigger problems.”

** A bigger benefit, and this is the 5% where I would use them, is to provide leaked-object tracking. For example, in a database connection pool. But I don't consider that the “normal” case.

Deprecation and Technical Debt

API designers deprecate classes and methods as a signal that they shouldn't be used. Usually because there's a better way to do the same thing, but occasionally (as in the case of Java's Date constructors) because the existing methods have unintended consequences. Deprecation is the first step in controlled breakage of backwards compatibility, a warning for users that they can't rely on the functionality always being present.

Of course, in the real world, deprecated functionality rarely gets removed. The Date constructors have been with us since Java 1.0.x and show no sign of going away. Indeed, of all the major frameworks that I've used, the only one that takes deprecation seriously is Spring — and even then, only minor things seem to get removed; you can still compile a program written for Spring 1.x with only a few changes.

However, leaving deprecated code in your codebase is a form of technical debt. You run the risk that the functionality will one day be removed, and that you'll have to scramble to replace it. But more important, if your code contains a mix of deprecated and non-deprecated features, you have to expend extra effort to maintain it.

The worst case that I've seen was a company that built websites for clients. One of their features was a content management system: a tool that allowed the client to create their own content, inserted into the page when various conditions were met. Standard stuff, but this company actually had three incompatible content management systems, all home-grown. Each was developed to replace the previous.

The reason that all three systems remained in active use was that there was a cost to convert to the newer framework, and neither the company nor their clients were willing to pay that cost. In effect, they chose to leave deprecated code in their codebase.

Since the frameworks were all built in-house, there was no reason to worry about them going away unexpectedly. But what did go away were the developers who had built the frameworks, and with them the knowledge of how the frameworks worked. Although the older frameworks didn't have a lot of bugs, when one did appear it was a crisis situation, with a mad search to find someone with enough knowledge to track it down and fix it. Even without bugs, the lack of knowledge meant that every enhancement to the client site took extra time, as each new developer learned enough to avoid breaking the CMS.

This situation continued for as long as I worked at that company; for all I know, they've added a fourth or fifth system to the mix. Although the old systems imposed an unnecessary cost on each project, that incremental cost was always lower than the cost to upgrade. But over time, the costs added up, and the client paid much more than an outright conversion (which the company didn't mind, as that increased revenue).

Worse, developers did all they could to avoid contact with the older systems, or to have their names on any checkins associated with it. Getting pulled onto a CMS problem meant that your own project schedules would slip, and you'd have to listen to upset project managers waste more of your time. Much easier to let the other team sink or swim on their own.

I'm not saying that ignoring deprecations will create disfunction and discord within your development team. But do you want to risk it?

Static Variables Considered Harmful

In my last post, I mentioned that a particular programmer had abused static initializers. I've been thinking more about this, and decided that the real problem was that this programmer actually abused static variables. The initializer abuse was simply a side effect, an attempt to make a bad design work.

To start, I'm going to give you my thoughts on static initializers: they're a code smell in and of themselves. If you're not familiar with a static initializer, here's an example: we have a static Map used for a lookup table, and want need to populate it before use. The static initializer is run at the time the class is loaded, before any code is allowed to use the class, so we can guarantee that the map will be populated.

public class MyClassWithALookupTable
{
    private static Map<String,String> states = new HashMap<String,String>();
    
    static
    {
        states.put("AK", "Alaska");
        states.put("AL", "Alabama");
        // ...
    }

This sort of code is, in my opinion, the only valid use for a static initializer: initializing an object that will never change, using code that is guaranteed never to throw. In effect, creating a constant.

The “guaranteed never to throw” part is important: static initializers run before any code is allowed to use the class; what happens if they do throw? The language spec says that static initializers are not permitted to throw checked exceptions, but they are allowed to throw unchecked exceptions (although a smart compiler will make you hide the exception in a called method). If the initializer does throw, you'll get an ExceptionInInitializerError the first time you access the class.

How do you debug such errors? You can't use a debugger, because the class has to be loaded to do set a breakpoint. You can insert a logging statement — assuming that your logging framework is able to initialize (one of the more painful things I've ever debugged was with a homegrown addition to Log4J that had a static initializer that threw). Even if you log the exception, you have to find where it's being thrown: people who abuse static initializers tend to do it a lot, so you might have to work through a long list of classes, adding exception handlers to each, before you find the one that's broken.

OK, so static initializers are a code smell. How do I make the leap from there to “static variables considered harmful?”

Going back to the legacy application in question, the original programmer used a static initializer to populate a lookup map. But the values to populate that map came from Hibernate. And since Hibernate was accessed through a DAO loaded as a Spring bean, he had to create a way to access the application context in a static manner. So he created a new class, with a static variable to hold the app context. And he wired this class into the DispatcherServlet configuration.

It's a long list of hacks, but here's the thing: every one of them logically proceeds from the one before it. And it comes down to the lookup table whose contents are allowed to vary.

There are certainly other solutions that he could have used. For example, I replaced the static initializer with a Spring post-construct from the associated DAO. But this is only one example of how static variables (versus constants) can lead you astray. More common are concurrency bugs, or “how did that value get there?” bugs. It seems a lot smarter to simply say “don't do this,” with the caveat for experts of “don't do this (yet).”

Cleverness

Delta kitchen faucets have a very clever quick-release attachment for their sprayer hose. It uses a couple of pieces of plastic and a spring clip to hold the hose onto a special fitting on the underside of the faucet, with a bushing to make a water-tight seal. To install the hose, all you have to do is slide it over the fitting until you hear it click into place. I'm sure that I appreciated that cleverness five or six years ago when I installed the faucet; it's always painful to deal with threaded fittings under the sink.

Unfortunately, I don't remember much about installing the faucet (trauma can have that effect). So when the hose started leaking a couple of weeks ago, I picked up a generic replacement hose at Home Depot, grabbed a flashlight and basin wrench, and threaded my body around the garbage disposal. And spent nearly half an hour trying to figure out how to remove the hose, with its non-standard attachment mechanism. And then spent another hour driving to Home Depot to find an adapter, only to be told that they didn't have one and I'd have to call Delta. I dreaded this call, but ended up with a very pleasant support rep, who put in an order for a free replacement.

When the new hose arrived, it only took a few minutes to install. Yet I can't help but thinking that this special fitting actually cost me several hours, versus the fifteen minutes for a standard threaded fitting. Not to mention a week's worth of annoyance, wiping up leaked water every time I used the faucet.

The lesson for software development should be obvious: clever tricks can save a lot of time in the short term. But unless you remember how they work, they'll probably cost you — or someone else on your team — more in the future.

Building a Wishlist Service: The Data Access Layer

I'm a huge fan of hiding persistence behind a data access layer. In particular, a data access layer that provides the basic CRUD operations — create / retrieve / update / delete — in a form slightly less granular than the actual domain objects.

I adopted this attitude after realizing that a typical business application has different types — levels — of rules. At the top are the “business” rules that describe how data is to be transformed: for example, a withdrawal or deposit changes the value of a bank account. Below those are the “logical model” rules describing how objects interact: a deposit or withdrawal can't exist without an account. And at the bottom are the “physical model” rules of how data is structured: a transaction consists of an account number, an operation, and a dollar amount.

In my view, the data access layer should manage the logical model. For the product list service, it implements rules such as “adding an item to a list will create that list if it doesn't already exist.” I could do this in the database, using triggers, but that means writing new code for each DBMS, along with additional work for deployment or upgrades. Or I could do it in the high-level service code, manipulating the domain objects directly, but that would result in duplicated code. Using the data access layer, the high level code is responsible only for populating and formatting the domain objects, while the database is responsible only for persistence.

Along with providing a place to hang rules, a data access layer simplifies testing: it allows the use of mock or stub objects in unit tests, thereby avoiding any actual database dependencies. Prior to developing the product list service, however, I had never used them in this way. For most data-driven applications, I would rely on integration tests to exercise a real persistence layer.

The product list service was different, primarily because I wanted to focus on client-server interaction, and didn't want to think about a persistence framework until later in the development process. So I spent a couple of hours to develop a stub data access layer that used HashMap for storage. Along with a suite of tests that allowed me to work through the rules of data management. My intent was that these could start as unit tests, then migrate to implementation tests once I added a real persistence framework.

Going into the project, I had assumed that persistence would be handled via a relational DBMS, probably using Hibernate as an object-relational mapper.* As I proceeded with development, however, I realized that there was no good reason to do this: as long as I had some way to assure sequenced, atomic commits, I had no need for a relational database. And because I was coding to the data access layer rather than “active record,” I had the freedom to explore this option.

That exploration is worth its own posting.

---

* In my opinion, Hibernate's chief benefit is that it hides the actual DBMS, meaning that it's (relatively) easy to implement portable applications — a big win if you're planning to sell your app, not so big if you control its deployment. In the case of the product list service, the object model is very simple, so I wrote a data access implementation using JDBC talking to an in-memory HSQLDB database. The goal was to verify that I wasn't doing anything that relational databases couldn't support. It didn't take long to write, but I remembered just how painful explicit SQL can be.

Unease

If you really don't care you aren't going to know it's wrong. The thought'll never occur to you. The act of pronouncing it wrong's a form of caring.

Zen and the Art of Motorcycle Maintenance is a book that you'll either love or hate. To some, it is a bombastic restatement of ideas that everyone already knows. To others, a lightweight gateway to heavyweight philosophy. And then there are people who believe it changed their life. I can't say that I'm one of the latter, but the book resonates with me on many levels, and I've reread it every few years since I was 15.

I recently paraphrased the above quote in a presentation on testing and coverage. The main point of my presentation was that 100% coverage does not guarantee sufficient testing; an audience member asked the obvious question “then how do you know that you've written enough tests?” My answer can be paraphrased as “you've written enough tests when you feel comfortable that you've written enough.”

Not a terribly good answer, to be honest. It did, however, set up my closing slide, which pointed out that you need committed, test-infected developers to get good tests. If you don't have that, you might as well buy a tool that throws random data at your code. But how does one arrive at the level of caring needed to write good tests? And is it inborn, or something that comes from experience?

I've been thinking about these questions this morning, because I'm refactoring some code and am left with what I consider an ugly method call: if called in one situation I want it to throw an exception, if called in another I want it to fail silently and return null. I've been puzzling over whether I really need to break the method into two parts, and also whether I should think more about another quote from Pirsig:

What's more common is that you feel unpeaceful even if it's right

Sometimes, perhaps, good enough is good enough.

Designing a Wishlist Service: Scalability

One of the repeated themes in Founders at Work is how startups were unprepared for scalability issues, particularly in the cases where the company experienced viral growth. In some companies, the founders worked around the clock adding capacity. Viral growth is by its nature unpredictable: you have no idea how fast — or how long — your company will keep growing, and going in you have no idea who is going to like the product. The rest of us, however, should have an intended audience for our product, and can make design decisions based on that audience.

There's a lot of conventional wisdom on designing for scalability, most of which boils down to “cache everything” and “don't maintain session state.” The trouble with conventional wisdom is that it rarely applies to the real world. Caching, for example, is useless if you're constantly changing the data, or if a particular data item is accessed infrequently. And a RESTful web service doesn't store session state by its very nature.

So what usage pattern should we expect from a wishlist service? I'm designing for three types of user:

Wishlist Owner

This person will add and remove items, and perhaps change their rankings. These interactions all happen on human timescales (ie, several seconds or minutes between changes, then long periods of inactivity), and should involve more writes that reads. For this user, caching is not going to help performance: the list will probably age out of the cache before it's read again, and would need to be constantly purged as items change.

Friends and Family

Wishlist owners may choose to share their wishlist, in the hopes that other people will do the buying. On the surface, it appears that caching might be a good idea: after all, there may be a lot of people accessing the service. But again, we're looking at human timescales: the list will be shared by passing a link, probably in an email. While you may send such an email to a hundred people, they're not going to open it at the same time, so there's no point in caching.

Collaborative Shoppers

One of the original triggers for this project was a collaborative shopping tool, where several people would share a list and make updates (typically comments), using a chat service to broadcast changes. This is one situation where caching would be useful, since there will be a group of people hitting the same data at the same time. They'll also be updating the data constantly, meaning the lifetime of a cached object would be very short.

However, I think that the solution for this scenario is to use caching in front of the application rather than as part of it: either Apache's mod_cache or some sort of caching filter in the application server. To make this work, we need to use URLs that identify the version of the list along with other key data — but this is the RESTful approach anyway.

By coming up with this picture, I can implement the service as a simple database front-end; I don't need to waste time writing explicit code for scalability. And if it goes viral … well, let's just say I don't expect that, particularly with an eCommerce server gating access.

Designing a Wishlist Service: Security

Security is the red-headed stepchild of the web-service world. Sure, there's WS-Security, which may be useful if you're using SOAP in server-server communication. But most services, in particular, browser-accessed services, rely on the security mechanisms provided by HTTP and HTTPS. If you want authentication, you use a 401 response and expect the requester to provide credentials in subsequent requests. If you're paranoid, you use a client certificate.

The wishlist service doesn't quite fit into this model. There's a clear need for some form of authorization and control, if only to protect the service from script kiddies with too much time on their hands. Yet authentication has to be transparent for usability: we don't want to pop up login dialogs, particularly if the customer has already logged in to their eCommerce account.

Perhaps the biggest constraint on any security mechanism is that requests will come from a browser. This means that the service can't expect the requester to provide any secret information, because anybody can view the page or script (I'm amazed by just how many web services require a “secret key” in their URL). If a page needs to provide secret information, that information must be encrypted before it is delivered to the user.

And secret information is required, because there are different levels of authorization: some people can update a list, others can only read it. The solution that I chose is to encrypt the wishlist key and user's authorization level, and pass that to the service as a URL parameter. This does represent another departure from pure REST, in that there may be multiple URLs that refer to the same underlying resource, but it seems the most reasonable compromise.

It also has the limitation that anybody who has a URL has the access defined by that URL. In some web services, this would be a problem. In the case of the wishlist service, it's mitigated by several factors.

First among these is that the value of this data just isn't that high. Sure, if you plan to be nominated for the Supreme Court, you probably don't want to keep a wishlist of X-rated videos. But for most people, a wishlist of clothing just isn't that important (and an opt-in warning can scare away those who feel differently). A related factor is that, in normal usage, people won't be handing out these URLs — at least, not URLs that can make changes. There's simply no reason to do so.

A second mitigating factor is that each URL also has an encrypted timeout: the service will reject any requests that arrive after that timeout. While this can be used to exert control over shared lists, it is primarily intended to defeat script kiddies who might try a denial-of-service attack via constant updates.

A third mitigating factor is that we keep close control over secrets, particularly those that can be used to break the encryption — in other words, security through obscurity. Bletchely Park would not have been nearly so successful at breaking Enigma if they hadn't learned to recognize radio operators. Taking heed of that, none of the information used by the encrypted parameter is provided in plaintext, meaning that any attempts at discovering the key will require heuristic analysis rather than a simple String.contains().