Java Object Serialization and Untrusted Code Execution

This past week we had a mini-fire-drill at work, in response to a CERT vulnerability note titled “Apache Commons Collections Java library insecurely deserializes data.” We dutifully updated our commons-collections dependencies, but something didn't quite smell right to me, so I did some digging. And after a few hours of research and experimentation, I realized that commons-collection was only a convenient tool to exploit the real weakness.

Yes, if you use commons-collection, you should update it to the latest version. Today. Now.

But more important, you need to review your codebase for any code that might deserialize untrusted data. This includes references to ObjectInputStream, but it also includes any RMI servers that you expose to the outside world. And don't stop at your code, but also look at your dependencies to make sure that they don't deserialize untrusted data. Once you've done that, read on for an explanation.

Actually, there are two things you should read first. The first is the slide deck from “Marshalling Pickles,” which describes how this exploit works (for multiple languages, not just Java). The second is my article on Java serialization, especially the section on readObject().

Now that you've read all that, I'm going to summarize the exploit:

• The Java serialization protocol is public and easy to read. Anybody can craft an arbitrary object graph and write it to a file.
• To deserialize an object, that object's classfile must be on the classpath. This means that an attacker can't simply write a malicious class, compile it, and expect your system to execute it (unless they can also get the bytecode into your system, in which case you've already lost). This, however, is a false sense of security.
• Apache Commons Collections provides a number of functor objects that perform invocation of Java methods using reflection. This means that, if Commons-Collections is on your classpath, the attacker can execute arbitrary code, by providing it as serialized data. And because it's so useful, Commons-Collections is on many classpaths, including many app-servers.
• Commons-Collection also provides a LazyMap class that invokes user-defined functors via its get(). This means that, if you can get the target JVM to accept and access such a map, you can execute arbitrary code.
• The JRE internal class sun.reflect.annotation.AnnotationInvocationHandler has a member variable as a Map (rather than, say, a HashMap). Moreover, it has a readObject() method that accesses this map as part of restoring the object's state in its readObject() method.

Putting all these pieces together, the exploit is a serialized AnnotationInvocationHandler that contains a LazyMap that invokes functors that execute arbitrary code. When this graph is deserialized, the functors are executed, and you're pwned.

There are a few points to take away from this description. The first, of course, is that uncontrolled deserialization is a Bad Idea. That doesn't mean that serialization is itself a bad idea; feel free to serialize and deserialize your internal data. Just be sure that you're not deserializing something that came from outside your application.

The second take-away is that code-as-data can be dangerous. As I said above, there's no way to put arbitrary code on your classpath, but the exploit doesn't rely on that: it relies on passing arbitrary data into your application. Clojure fans might not like the implications.

For that matter, pay attention to what goes onto your classpath! Apache Commons is an incredibly useful collection of utility classes; if you don't use them, it's likely that one of your dependencies does. Or you have a dependency that offers features that are just waiting to be exploited. My KDGCommons library, for example, has a DefaultMap; all it's missing is a reflection-based functor (and a much wider usage).

The final — and most important — take-away is to pay attention to the variables in your serialized classes. The attack vector here is not commons-collections, it's AnnotationInvocationHandler and its Map. If, instead, this member was defined as a concrete class such as HashMap, the exploit would not have worked: attempting to deserialize a LazyMap when a Hashmap is expected would cause an exception. I know, using a concrete class goes against everything that we're taught about choosing the least-specific type for a variable, but security takes precedence over purity (and you don't need to expose the concrete type via the object's API).

If you're not scared enough already, I'll leave you with this: in the src.zip for JDK 1.7.0_79, there are a little over 600 classes that implement Serializable. And there are a bunch of these that define a private Map. Any of those could be a vector for the same attack.

When Immutability Gets In The Way

Here's a recent hack that I perpetrated:

(defn- redact-headers-from-exception-info
  [ei]
  (if-not (get-in (ex-data ei) [:request :headers])
    ei)

  (let [orig-trace (.getStackTrace ei)]
    (doto (ex-info (.getMessage ei)
                   (update-in (ex-data ei) [:request :headers] dissoc "Authorization")
                   (.getCause ei))
          (.setStackTrace orig-trace))))

Clojure provides the ex-info function, which creates a RuntimeException that has an attached map of data. This is quite useful, as you can provide detailed information at the exception site without creating your own exception subclass.

However, it can provide too much information. In this case, the exception was being thrown from a web-service call, and, to assist debugging, contained the entire request. Including the Authorization header, which contains client credentials. Not something that you want to write to a logfile.

And here's how immutability got in our way. Clojure maps are immutable: you can't update them in-place. And the ExceptionInfo object doesn't provide any way to replace the map. So we were faced with a choice: either pass the exception to the logger and have sensitive data end up in the log; or extract the map from the exception, redact the sensitive data, but lose the stack trace. I don't know about you, but I consider stack traces to be one of the most important clues to debugging errors; we make web-service calls all over the place, and need to know which one is failing.

Fortunately, java.lang.Throwable is not immutable: it provides methods to get and set the stack trace. And thus the hack: I extract the stack trace and data map from the existing exception, remove the fields that I don't want to log, and create a new exception from those components. Then I explicitly replace the stack trace in the new exception. It's ugly, but it works: we log the information we need to debug the problem, but not the sensitive information that was present in the original info map.

This post is not intended as a slam on Clojure, or on immutability in general. Instead, it's a reminder that a rigid adherence to immutability can have unintended consequences.

Thoughts on Clojure, Page Two

WTF? You didn't mention Clojure at all!

Perhaps after my prior “six months” post people were expecting more vitriol. Or adulation. Sorry to disappoint.

I really can't find much to say about the language itself, positive or negative. I'm not turned off by the number of parentheses, because I simply see them as doing double duty; “C-family” languages split the job with braces. I find prefix notation somewhat hard to read, coming from an English-language subject-verb-object background, but that's easily solved by making lots of helper functions. Parameter ordering in the standard library seems haphazard, and there are some edges to the sequence abstraction that I stumble over. But overall, meh.

So rather than write about the language itself, I decided to write about attitudes. I was prompted by the book that I used to learn the language. Overall, I think it's a good book, and I'd recommend it to someone learning the language (and I still use it). But the authors are a little too focused on the word “concise.” At one point (table 9-1), they compare Java to Clojure and make the claim that Clojure is “certainly more concise” than Java, even though almost all of their examples use the same characters — just re-arranged. Examples like this perpetuate the stereotype of the “smug LISP weenie.”

I can understand why LISP was considered revolutionary back in the 1970s and 1980s. There was a generation of programmers that learned BASIC on a DEC mini, and I can imagine them going off to college and seeing a whole new world: scoped variables! first-class functions! dynamic dispatch! structuring a program as list transformations!

Actually, I don't have to imagine that, I was part of that generation. I happened to go to a college that focused on strong typing and provably correct programs, with PL/1 as the implementation language. Which is one reason why my degree, when it finally came, was in History and not Computer Science.

But that was then, this is now, and most mainstream languages have features that were once revolutionary. Indeed, one of the reasons that Clojure exists is that it leverages the JVM, which natively supports the garbage collection and dynamic dispatch that underly any LISP implementation. But those same features are also available for Java.

So, to proclaim Clojure “better” due to its language constructs seems to me as misguided as proclaiming HP calculators better because they had RPN.

That said, I think there is something revolutionary happening in the Clojure community, and that's the view that program state must be considered in relation to time. That, to me, has far more relevance than whether you use braces or parentheses, or where you put the operator.

Thoughts on Clojure After Using It For Six Months

When I was in high school, in the late 1970s, “pocket” scientific calculators were hot. They were portable computation devices that hadn't even existed a decade prior, were incredibly expensive when they came out, but were now reasonable birthday presents for the budding geek. There were several manufacturers, but the only ones that mattered were Texas Instruments (TI) and Hewlett Packard (HP). The cool kids all had HPs.

There's no question: HP calculators were better. A TI calculator was about half the price of an HP calculator with the same features, but it reached that price point by cheaper build quality. You could drop an HP calculator down a flight of stairs without ill effect, but a TI would start to have keyboard problems within a year of normal use. In fact, that's how I got my first calculator: my father's TI stopped working, he sent it to TI for repair, but bought another before it returned.

Another difference between TI and HP was that the former used algebraic notation: “1 + 2 =” while the latter used reverse Polish notation (RPN): “1 ENTER 2 +”. From the perspective of a calculator designer, RPN makes a lot of sense: algebraic notation requires more work, and the transistors that performed that work could be put to better use elsewhere. Or put another way — one intended to taunt an HP-wearing geek — RPN-based calculators were less advanced than algebraic calculators.

Here's the interesting part: the vast majority of those HP aficionados were convinced that RPN was the reason that HP calculators were better.

The Carburetor: Elegance and the Real World

The carburetor is a simple, elegant device for providing fuel to a gasoline engine. So simple and elegant that it was the primary mechanism for doing so for approximately 100 years, and remains the primary mechanism for small engines such as lawnmowers.

Here's a simplified explanation of how it works: all air entering the engine flows through the carburetor; the “jet,” a small tube connected to the supply of gasoline, is exposed to the airflow; as air flows through the carburetor body, it pulls gasoline out of the jet and vaporizes it. The more air flowing past the jet, the more fuel is drawn into it. By properly sizing the jet, you can approach a near-optimal fuel-air mixture — although more on that later.

The airflow through a carburetor is caused by the intake stroke of the engine: as the piston moves down in the cylinder, it creates a vacuum that pulls air from the intake manifold; the faster the engine turns, the more air it wants (and the more fuel it needs). Now here's the interesting part: there's a valve inside the carburetor, attached to the accelerator pedal: when your foot is off the pedal, that valve is (almost) closed; the engine can only get enough air (and fuel) to run at idle speed. When you push the pedal to the floor, the valve opens wide, and the engine gets all the air and fuel that it wants.

But that leads to the first problem with a carbureted engine: the feedback loop (and lag) between opening the throttle and actually going faster. At a normal cruise speed, the butterfly valve is only partway open; the engine doesn't get all the air that it can handle. When you open the throttle, that lets more air into the system, and indirectly, more fuel. More fuel means more power generated by the engine, which (given constant load) means that the engine will speed up. Which increases the vacuum, which means that more air is drawn into the system, which means that more fuel is provided with it, which …

But, as I said, there's lag: when you first allow more air into the system, the flow of gasoline doesn't keep up. So at some point carburetor designers added an “accelerator pump”: an actual pump that squirts extra gasoline into the airflow. Most of the time it does nothing, except when you push the pedal to the floor; then it sends that extra gasoline to the engine, to compensate for the flow through the main jet.

While the single carburetor works fairly well for small engines, it doesn't work so well for large engines that have widely differing airflow requirements between idle and full throttle. So, carburetor designers compensated with multiple barrels: separate paths for the air to follow. At idle the “primaries” provide all the air and fuel; under full acceleration, the “secondaries” open to provide vastly more air and fuel.

All of which works fairly well as long as the altitude doesn't change. A typical automotive carburetor is sized for its market, on the fairly reasonable assumption that drivers stay close to home. But take an east coast car, tuned for driving at sea level, up into the mountains of Colorado, and the mixture becomes excessively rich: the carburetor allows too much fuel to mix with the air, and (perhaps surprisingly) performance suffers. Eventually the spark plugs will get coated with a layer of soot. A worse fate meets the Colorado car driven to the shore: it doesn't provide enough fuel, which causes the engine to run hot, which eventually causes extensive (and expensive) damage to the engine's valves and pistons.

Light aircraft, which cover a similar altitude range on every flight, have a solution: the mixture control. After every significant change in altitude, the pilot has to re-adjust the mixture to match the air density at that altitude. In a typical long-distance flight, the pilot might change throttle settings three times (takeoff, cruise, landing) but adjust mixture a half-dozen times or more. Not something that the typical automobile driver would want to do (or do particularly well; all teenagers' cars would be overly rich “for more power”).

The simple, elegant carburetor is no longer so simple or elegant: to meet the needs of the real world, it's grown a bunch of features. These features expand what I'll call the “conceptual model” of the carburetor. A simple function relating airflow velocity and fuel delivery is not sufficient to implement the real-world model.

Today we use fuel injection in almost every car (and in most high-performance light aircraft). Fuel injection systems are certainly not simple: a computer decides how much fuel to inject based on inputs from a plethora of sensors, ranging from ambient temperature to position of the accelerator pedal. But fuel injectors do a much better job of providing exactly the right amount of fuel at exactly the right time.

And, although a fuel injection system is more complex than a carburetor, its conceptual model is actually simpler: there's a single sensor that measures the amount of residual oxygen in the exhaust, and the computer attempts to optimize this value.

Have I Been Hacked?

Twenty-five years later I can still remember how I felt, returning home that day. Being burgled is a surrealistic experience: you notice a progression of little things that don't belong, and it may be quite a long time before your realize that your world has changed. For me, the first anomaly was that the front door of our two-family house was unlocked. OK, maybe my neighbor was out in the back yard. The second was that I could hear the music that I left on for the bird. As I walked up the stairway (unchanged), I saw that my front door was open. I didn't notice that the wood around the lock was splintered. I walked into what seemed a perfectly normal front room, and finally realized that something was wrong when I saw the VCR hanging by its antenna cable (+1 for tightening the cable with a wrench).

Now imagine a different scenario: the front door locked, the upper door standing open but unlocked, lights on that shouldn't be, and a few dirty dishes on the counter. All things that could be explained by me being particularly forgetful that morning. But still the sense that something was not quite right.

That was my feeling this week, as several of my friends reported receiving spam emails from me, warning me that my Yahoo account might have been hacked. The first was from my neighbor, and I discounted his report, thinking that the spammer might have hit another neighbor, gotten the neighborhood mailing list, and paired up random people. After all, the “From” address wasn't even mine! But then I got reports from friends that weren't neighbors, and even found a couple of the emails in my own GMail spam folder.

OK, so my Yahoo account got hacked, big deal. Maybe next time I'll be more careful with sharing passwords.

Except … the Yahoo account has its own password, one that's not saved anywhere but my head, so I have a hard time accepting that the account was actually broken into. And Yahoo's “activity report” claims that all logins in the past 30 days came from my home IP (a nice feature, it's one of the tabs on the profile page). And, I can still log into the account. I've never heard of anyone breaking into an account and just leaving it there, untouched.

And when I looked at the message, my email address wasn't to be found anywhere. It was “kdgregory,” but some server in a .cx domain. Different people reported different domains. Nor was a Yahoo server to be found in the headers. OK, headers can be forged, but I would have expected a forgery that at least attempted to look credible. According to the IP addresses in the headers, this email originated somewhere in India, went through a Japanese server, and then to its destination.

So I'm left with wondering what happened. Clearly these emails were based on information from my account, either headers from old messages (likely) or a contact list (less likely). But how? Googling for “yahoo data breach” turns up quite a few news stories, but nothing from this year. Did whoever acquired these addresses just sit on them for a year? And if yes, what other information did they glean from my account?

It's disquieting, this sense of perhaps being compromised. I think I would have been happier if they had changed my password and locked me out of the account. At least then I'd know I was being sloppy about security. As it is, I have no idea what (if anything) I did wrong. Or whether it will happen again.

The Achilles Heel of Try-With-Resources

Here's some code that you might see since Java 7 appeared. Can you spot the bug?

public List<String> extractData(File file, String encoding)
throws IOException
{
    List<String> results = new ArrayList<String>();
    try (BufferedReader rdr = new BufferedReader(new InputStreamReader(new FileInputStream(file), encoding)))
    {
        // read each line and extract data
    }
    return results;
}

OK, here's a hint: what if encoding is invalid?

The answer is that the InputStreamReader constructor will throw UnsupportedEncodingException (which is a subclass of IOException).

But this exception happens after the FileInputStream has been created, but before the body of the try-with-resources statement. Which means that the implicit finally in that statement won't get executed, and the file will remain open until the stream's finalizer executes.

To solve this problem, you need to break apart the constructor chain, to ensure that the actual resource is assigned to its own variable in the resource clause, and will therefore be closed. There are two ways to approach this: either use multiple resource clauses, or push non-resource constructors into the body of the statement.

In this case, the InputStreamReader and BufferedReader are simple decorator objects: if you neglect to close them, it doesn't change the behavior of the program. So we can construct them inside the body of the statement:

try (InputStream in = new FileInputStream(file))
{
    BufferedReader rdr = new BufferedReader(new InputStreamReader(in, encoding));
    // ...

Bug solved: regardless of what happens when constructing rdr, the underlying stream will be closed; we don't have to worry about a “too many open files” exception.

However, if we try to do something similar with output streams, we introduce a new bug:

try (OutputStream out = new FileOutputStream(file))
{
    // DON'T DO THIS!
    BufferedWriter wtr = new BufferedWriter(new OutputStreamWriter(out, encoding));
    // ...

The problem with this code is that decorators are important when writing. Unless you close the BufferedOutputStream, the data in its buffer won't be written to the file. On the positive side, you'll see this bug the first time you try to write a file. To solve it and still use the try-with-resources construct, you need to use multiple resource clauses:

try (OutputStream out = new FileOutputStream(file) ;
        OutputStreamWriter osw = new OutputStreamWriter(out, encoding) ;
        BufferedWriter wtr = new BufferedWriter(osw))
{
    // ...

There you have it, the only safe way to use try-with-resources. If you currently have code with nested constructors, I strongly recommend that you change it. And please change any public documentation that has nested constructors; explaining the bug to someone that is “just following the docs” wastes time that could be used for better things.

The Encryption Tax

Last year I wrote a post comparing build times using an SSD and a traditional hard disk. One of the conditions that I tested was using an encrypted home directory, and was surprised that encryption added 11% to my build times — about the same as using a “spinning rust” drive.

That was Xubuntu 12.04, which used eCryptfs, a FUSE filesystem, to handle encryption. At the time, I posited that the buffer cache was holding encrypted data blocks, so you paid a “decryption tax” every time you accessed one. Xubuntu 14.04, however, provides an option to enable full-disk encryption as part of the installation. Since this hooks more tightly into the kernel, I wondered if the decryption tax was still present. This called for another day of testing.

For this series of tests, I'm using my HP ProBook 640-G1:

• Intel i5-4300M @ 2.6 GHz, 2 cores, hyperthreaded
• 800 MHz FSB
• 8 Gb RAM, 3 Mb L2 cache
• Samsung 840 SSD
• Xunbuntu 14.04 LTS 64-bit

And as a test subject, I'm building the Spring Framework 3.2.0.RELEASE. In my last test, it showed the biggest difference between SSD and platter, as well as the biggest difference between encrypted and non-encrypted access.

I tested three scenarios:

• Standard Xubuntu 14.04 install, no encryption.
• Home directory encryption (eCryptfs, running in userspace).
• Full-disk encryption (dm-crypt, which runs — I believe — in the kernel).

As before, I pre-fetched all dependencies and disconnected from the network. Each timing is the average of two runs. Between runs I cleaned the build directory, TRIMed the filesystem, and rebooted to clear the buffer cache.

Here, then, are the numbers. Execution time is wall-clock time in seconds, as reported by Gradle, because that's what we really compare about.

	Execution Time	Difference from baseline
Baseline	288	–
Encrypted home directory	300	+ 4.2 %
Encrypted filesystem	295	+ 2.5 %

I'm not sure why the penalty of an ecrypted home directory was lower this time around, but it's still there. And the full-disk encryption is a lower penalty, as expected, but again it's still there. So you still have to make a tradeoff between performance and protection. But given the numbers, protection should win.

At the end of my earlier post, I suggested that there's no substitute for RAM, and the ability to maintain a large buffer cache. I'm not sure that argument applies with a project as big as Spring. After each build, I cleaned the directory and ran a second build, figuring that 8Gb should be enough RAM to maintain everything in the buffer cache. The numbers, however, tell a different story:

	Execution Time	Difference from baseline
Baseline	284	- 1.4 %
Encrypted home directory	294	+ 2.1 %
Encrypted filesystem	289	+ 0.3 %

Yes, a slight improvement over the previous runs, but not much. I think this might be due to the fact that compilers produce disk blocks, rather than consume them — reading existing blocks is a small part of the overall execution time.

Developing in the Clouds

Deploying to the cloud — whether to a platform-as-a-service provider such as Heroku, or an infrastructure-as-a-service provider such as Amazon EC2 — is commonplace, particularly for startups that don't want to invest in infrastructure, or established companies that need resources to handle operational spikes. But not much has been written about using a cloud-based host as a development platform. I just wrapped up a six-month project where my primary development machine was an Amazon EC2 instance, and here are a few of my thoughts.

I'm going to start with the negatives:

• Latency

  I live in Philadelphia and was working on an Amazon EC2 instance in Ashburn Virgina, a distance of approximately 150 miles — at the speed of light, roughly a millisecond. However, there are multiple network hops between me and the server, all of which add up to a round-trip time (via ping) of roughly 40 milliseconds. If you confine yourself to text, that's nearly unnoticeable. If you run the X Window System, it's unbearable. If your workflow is GUI-intensive, cloud-based development might not be a good choice (although I consider VNC quite acceptable when using an IDE for development).

• Capability

  My desktop runs an Ivy Bridge Core i7 with 32 Gb of RAM. The t2.medium instance that I used for remote development has baseline performance of roughly 40% of an undisclosed Xeon and only 4 Gb of RAM. As it turns out, that's sufficient for many development tasks, especially with a rapid-turnaround platform such as Node.JS. If you have big compiles, you can always fire up a c4.8xlarge with a Haswell Xeon, 60 Gb of RAM, and disk throughput that's far better than your desktop SSD.

• Cost

  Mind you, that c4.8xlarge will cost you: as of this date, $1.68 per hour or fraction thereof. On another project, a colleague fired up a cluster of these instances and forgot to shut them down when he left the company. A month later the IT department gave me a call to ask if we really needed them, because they were costing us $5,000 a month. By comparison, the t2.medium instance costs $0.052 per hour, or $456 per year. More than a decent developer desktop on a three-year depreciation schedule, but not that bad in the larger scheme.

• Security

  This is the big one: if you're going to run machines in the cloud, you need to have at least a baseline knowledge of Internet security (or hire someone who does). Put simply, you will be attacked. To apply some numbers to that statement, I started an EC2 instance that exposed SSH and HTTP, and left it running for 24 hours. The first attempt to break into SSH happened within a half hour; there were 39 attempts over the course of the test. Yandex started exploring the site within 12 hours, followed by other web scrapers.*

  Basic security rules will get you a long way: don't use dictionary passwords for any exposed service (and for SSH, don't use passwords at all), and don't expose any unprotected services to the outside world. Use a firewall that checks origin IP. If you're running on AWS, this feature is built into security groups. If you need to share access to your instance, or access it from locations that you don't know in advance, consider a VPN.

  This is also a case where I think security by obscurity is useful — at least as a first line of defense. Most scannerbots and web crawlers look at well-known ports: 22 for SSH; 80, 8000, and 8080 for HTTP. Exposing your prototype website on port 21498 isn't going to stop a dedicated attacker (and there are bulk port scanners out there), but it will prevent your site's content from showing up in a search index before you're ready.

And now, the positives:

• Availability

  The ability to access a cloud-based host from anywhere, at any time, gives you an enormous amount of flexibility in how you do work. There's no need to lug a laptop home every night, and if the roads are snow-covered and filled with stopped traffic, you can easily work from home. With tools like screen or VNC, you can have a session that's always set up just how you want it, and which can run programs while you're not connected. Plus, it's easy to collaborate: unlike a personal computer, a cloud instance can be shared by multiple users.

• Consistency

  I don't know about you, but after a few years my computers all accumulate a significant amount of cruft: libraries or entire applications that I installed for a particular project and no longer need, along with various versions of standard tools, some of which are expected by the OS itself. Multiply that cruft by the number of members on your team, and add a fudge factor for different shell configuration files. None of which matches your production (or even test) environment. It's a mess.

  To me, this is the biggest benefit of developing in the cloud: you can be certain that all machines are configured alike — or at least start out that way. Tools such as Chef and Puppet will take you from a base image to a fully configured server in one step. With Amazon, after you've configured the server once, you can create a private AMI and stamp out as many instances as you want.

• Disposability

  The flip side of starting servers quickly is disposing of them when no longer needed. There's no reason to patch or update your machine; that just accumulates cruft. This mantra has long been used by operations teams: I was out riding with a friend last fall when his beeper went off; we pulled over, he discovered that one of his production servers was having trouble, shut it down, and started a replacement. Analysis could wait for later; there was no need to try to make an emergency patch.

Finally, lessons learned:

• Experience Helps — a Lot

  I've been using AWS since 2009, and have been responsible for administering my personal Linux machine since well before that. However, the limits of my knowledge became apparent when the company hired Eric, a “cloud architect” who lived up to the name. He quickly had us running in an isolated virtual private cloud (VPC), with separate VPCs for our test, integration, and production environments, OpenVPN to secure access, LDAP to hold credentials, and automatic deployments from our CI server. If you can find such a person, hire him or her; it will save a lot of time.

• Disposability Changes How You Work

  I got a hint of this several years ago, when using AWS to test a distributed application: we would prepare a test, start up a dozen servers, run the test, then shut them down. It allowed a form of testing that was quite simply impossible just a few years earlier; no company that I've worked for had a spare closet full of machines, and even if they did, configuration would require hours.

  As developers, we set up our machines just the way we want them; it takes a long time, and no two developers have the same configuration. But if you have the base development image preconfigured, firing up a new machine becomes a process of copying your personal configuration files and checking out your workspace. You learn to let go of “my machine.”

---

* I've written elsewhere about the questionable practices of web crawlers. If you have a site that's not ready for primetime, don't expose it to the Internet on a well-known port.