Layoffs

Where are you in your company's layoff list? Every company has one, even if it's just a vague idea in the mind of the CEO. By looking around, you should be able to determine approximately where you stand. In my department of 20 people, for example, I believe myself to be #3. Maybe #2, perhaps as low as #5. But one of the first people to go, regardless.

Let me stop here, to stress that being laid off does not mean that you're incompetent, or less competent than the people who were retained.

Layoffs are a coldly rational response to the prospect of running out of money. Sometimes they happen well in advance of a crisis, sometimes they happen when the company is on the brink of bankruptcy. But in either case, being laid off simply means that the skills you bring to the table are perceived as being less valuable than the associated cost of your salary and benefits. I realize that's not much comfort when you're the person being laid off.

I've been laid off three times in a 30+ year career. The first time was six months into my second job, and happened on the Monday that I returned from vacation. I was working for a subsidiary of a midwestern company that made laboratory instruments, an acquisition that had been an independent startup a year or so earlier. For whatever reason, the parent company decided to consolidate operations at their home office. We were all gathered into the large conference room, and told that we no longer had jobs. The third time was similar: I was working for a subsidiary of a subsidiary of a large telecom company that learned too late that flip-phones were passé.

There's not much to be learned from these two examples, nor much to be done about them; you simply have to be prepared. If you're lucky, they come with a decent severance package.

My second layoff was more interesting. It happened toward the end of the dot-com crash. We'd been through three rounds of layoffs already, along with an enforced vacation during the winter holidays. At the start of the new year I sat down with my division VP, and said “I know I'm now at the top of the layoff list, let's talk.” Six months later (after I'd been given a large retention bonus) the fourth round hit, and he said “you were right.”

So how did I know? Like many things, your likelihood of being laid off can be expressed as a quad-chart:

The left axis is salary: the more expensive you are, the bigger a target you become. You may argue that this doesn't take into account productivity, but I believe that's irrelevant. If you're at a company that values productivity, then everyone will be productive; the idea that there's a 10x difference between top and bottom is a myth. And if you're at a company that doesn't value productivity, you're just a cog in the machine with a known cost, regardless of how productive you think you are (and really, what are you doing at such a company?).

The bottom axis, strategic versus tactical, is the one that I think is more interesting. Strategic work is for the future: shaping your next product or next release. Tactical work is about now: operations, bugfixes, responding to immediate customer desires. When a company is forced to choose, they'll value tactics above strategy every time. In the case of my second layoff, I wasn't the highest-paid person but my work was entirely focused on the future; that put me at the top of the list.

So what do you do with the knowledge that you're near the top of the layoff list?

The facile answer is that you update your resume, but that's wrong on several counts. First, of course, is that your resume should always be up to date, in case the perfect job gets dropped in your lap. Second, if you change jobs you're likely to move from the top of one layoff list to the top of another, without the offsetting benefit of tenure (yeah, it isn't a simple quad-chart; it never is). And finally, because knowing where you are on the list only matters when layoffs are imminent.

If you're in a company that's sailing along, with no storms on the horizon, there's little reason to care. Yes, there's always the chance of a black swan event that puts the company in crisis, but those aren't worth sleepless nights.

If you work for a company that does have obvious financial difficulties ahead, you should consider your options. Perhaps you transition from a strategic position to one that's more tactical. Or, again, do nothing. I work for a venture-funded startup, which means that either we become profitable or go out of business. But I'm doing the job that I was hired for, I'm enjoying it, and I'm prepared for the possibility of finding myself without a job.

And if you work for a company that has had layoffs and looks like it will have more? Perhaps then it's time for an honest conversation with your manager (or better, your manager's manager). Talk about where you stand on the layoff list, and what steps you can take to either lower your position or ease your departure.

In my case, I was already thinking about moving to Philadelphia to be with my now-wife. Perhaps my conversation with our VP raised me to the very top of the list if I wasn't already there. But it meant that both of us were prepared for what happened six months later.

Java Object Serialization and Untrusted Code Execution

This past week we had a mini-fire-drill at work, in response to a CERT vulnerability note titled “Apache Commons Collections Java library insecurely deserializes data.” We dutifully updated our commons-collections dependencies, but something didn't quite smell right to me, so I did some digging. And after a few hours of research and experimentation, I realized that commons-collection was only a convenient tool to exploit the real weakness.

Yes, if you use commons-collection, you should update it to the latest version. Today. Now.

But more important, you need to review your codebase for any code that might deserialize untrusted data. This includes references to ObjectInputStream, but it also includes any RMI servers that you expose to the outside world. And don't stop at your code, but also look at your dependencies to make sure that they don't deserialize untrusted data. Once you've done that, read on for an explanation.

Actually, there are two things you should read first. The first is the slide deck from “Marshalling Pickles,” which describes how this exploit works (for multiple languages, not just Java). The second is my article on Java serialization, especially the section on readObject().

Now that you've read all that, I'm going to summarize the exploit:

• The Java serialization protocol is public and easy to read. Anybody can craft an arbitrary object graph and write it to a file.
• To deserialize an object, that object's classfile must be on the classpath. This means that an attacker can't simply write a malicious class, compile it, and expect your system to execute it (unless they can also get the bytecode into your system, in which case you've already lost). This, however, is a false sense of security.
• Apache Commons Collections provides a number of functor objects that perform invocation of Java methods using reflection. This means that, if Commons-Collections is on your classpath, the attacker can execute arbitrary code, by providing it as serialized data. And because it's so useful, Commons-Collections is on many classpaths, including many app-servers.
• Commons-Collection also provides a LazyMap class that invokes user-defined functors via its get(). This means that, if you can get the target JVM to accept and access such a map, you can execute arbitrary code.
• The JRE internal class sun.reflect.annotation.AnnotationInvocationHandler has a member variable as a Map (rather than, say, a HashMap). Moreover, it has a readObject() method that accesses this map as part of restoring the object's state in its readObject() method.

Putting all these pieces together, the exploit is a serialized AnnotationInvocationHandler that contains a LazyMap that invokes functors that execute arbitrary code. When this graph is deserialized, the functors are executed, and you're pwned.

There are a few points to take away from this description. The first, of course, is that uncontrolled deserialization is a Bad Idea. That doesn't mean that serialization is itself a bad idea; feel free to serialize and deserialize your internal data. Just be sure that you're not deserializing something that came from outside your application.

The second take-away is that code-as-data can be dangerous. As I said above, there's no way to put arbitrary code on your classpath, but the exploit doesn't rely on that: it relies on passing arbitrary data into your application. Clojure fans might not like the implications.

For that matter, pay attention to what goes onto your classpath! Apache Commons is an incredibly useful collection of utility classes; if you don't use them, it's likely that one of your dependencies does. Or you have a dependency that offers features that are just waiting to be exploited. My KDGCommons library, for example, has a DefaultMap; all it's missing is a reflection-based functor (and a much wider usage).

The final — and most important — take-away is to pay attention to the variables in your serialized classes. The attack vector here is not commons-collections, it's AnnotationInvocationHandler and its Map. If, instead, this member was defined as a concrete class such as HashMap, the exploit would not have worked: attempting to deserialize a LazyMap when a Hashmap is expected would cause an exception. I know, using a concrete class goes against everything that we're taught about choosing the least-specific type for a variable, but security takes precedence over purity (and you don't need to expose the concrete type via the object's API).

If you're not scared enough already, I'll leave you with this: in the src.zip for JDK 1.7.0_79, there are a little over 600 classes that implement Serializable. And there are a bunch of these that define a private Map. Any of those could be a vector for the same attack.

When Immutability Gets In The Way

Here's a recent hack that I perpetrated:

(defn- redact-headers-from-exception-info
  [ei]
  (if-not (get-in (ex-data ei) [:request :headers])
    ei)

  (let [orig-trace (.getStackTrace ei)]
    (doto (ex-info (.getMessage ei)
                   (update-in (ex-data ei) [:request :headers] dissoc "Authorization")
                   (.getCause ei))
          (.setStackTrace orig-trace))))

Clojure provides the ex-info function, which creates a RuntimeException that has an attached map of data. This is quite useful, as you can provide detailed information at the exception site without creating your own exception subclass.

However, it can provide too much information. In this case, the exception was being thrown from a web-service call, and, to assist debugging, contained the entire request. Including the Authorization header, which contains client credentials. Not something that you want to write to a logfile.

And here's how immutability got in our way. Clojure maps are immutable: you can't update them in-place. And the ExceptionInfo object doesn't provide any way to replace the map. So we were faced with a choice: either pass the exception to the logger and have sensitive data end up in the log; or extract the map from the exception, redact the sensitive data, but lose the stack trace. I don't know about you, but I consider stack traces to be one of the most important clues to debugging errors; we make web-service calls all over the place, and need to know which one is failing.

Fortunately, java.lang.Throwable is not immutable: it provides methods to get and set the stack trace. And thus the hack: I extract the stack trace and data map from the existing exception, remove the fields that I don't want to log, and create a new exception from those components. Then I explicitly replace the stack trace in the new exception. It's ugly, but it works: we log the information we need to debug the problem, but not the sensitive information that was present in the original info map.

This post is not intended as a slam on Clojure, or on immutability in general. Instead, it's a reminder that a rigid adherence to immutability can have unintended consequences.