Thirty Years

Thirty years ago this week I started my first full-time, paid position as a software developer. I'm not sure of the actual day; when I moved to Philadelphia I purged a lot of old papers, and the offer letter for this job was one of them. I was a nineteen-year-old who had left college (that's another story), and whose father had been very clear about the consequences. Fortunately, the winter of 1983/84 was a good year to be looking for software jobs in the Boston area.

I landed at Think Technologies, the creators of Macintosh Pascal, as an entry-level programmer. My chief contribution to that product was a demo program that ended up as the box artwork (also shown in the link above).

I started the same week that the Macintosh was released. As a result, I didn't see either it or the product while I was interviewing. What I did see was the Apple Lisa, with its mouse and graphical UI, and the sense of “this is something big” moved Think to the top of my list.

In retrospect, Think may have been one of the most valuable jobs of my career, and not just for getting in on GUIs at the beginning. I learned a lot about startups, and about the pain of real-world software development. I also learned to keep my ego in check: the company was filled with bright people, and Mel, the person whose ideas were the base of the product, combined a particularly creative career with a reserved, self-effacing exterior.

And I learned that no job lasts forever. Eventually, you reach an intellectual or perceptual plateau, and it's time to move on to something new. As a professional, you must of course balance your personal desire for growth against your responsibilities to see your work in production. But it's easy to get trapped on the plateau, and that's something that I quickly learned to avoid.

The intervening years have seen a lot of change in the industry: graphical user interfaces are now the standard, as are integrated development environments (Macintosh Pascal wasn't the first IDE that I used, but it was the first good one); computer networks are pervasive, as is the resulting knowledge-web of Internet-hosted documentation and anonymous helpful strangers; web-apps have taken us back to a central server with dumb terminals, although client-side JavaScript is an echo of the micro-computer revolution; and lastly, the machines are almost infinitely more powerful.

But programming is still about moving bits from one place to another without dropping any.

Learning Go: Imports

I'm going to finish this series of posts with an interesting but in my opinion unfinished feature: third-party imports. Unless you like to reinvent wheels, your projects will depend on libraries written by other people. To help you do this, Go's import declaration allows you to reference packages stored in open source repositories.

For example, if you want to use the Go project's example square root implementation, you could write the following:

package main

import "fmt"
import "code.google.com/p/go.example/newmath"

func main() {
    fmt.Println("The square root of 2 is ", newmath.Sqrt(2))
}

If you just try to build this code, you'll get an error message indicating that the compiler can't find the package. Before you can use it, you have to retrieve it:

go get code.google.com/p/go.example/newmath

This is, to say the least, inconvenient: if you depend on multiple libraries, you have to manually retrieve each of them before you can build your own project. Fortunately, go get resolves transitive dependencies, and if your project is stored in one of the “standard” repositories you can leverage this feature to retrieve your project and its dependencies in one step. However, at least for GitHub projects, this technique doesn't clone the repository. If you want to make changes — or get updates from your dependencies — you need to manually clone.

A bigger problem is that there's no mechanism for versioning: you always get the trunk revision. If all of your libraries are backwards compatible, that may not be a problem. My experience suggests that's a bad thing to rely upon. In practice, it seems that most developers retrieve the libraries that they want to use, then check those libraries into their own version control system: creating a “locked” revision that they know supports their code.

As I said, I think the feature is unfinished. Versioning is a necessary first step, and should not be too difficult to add. But it's not enough. If you rely on retrieving packages from an open-source repository, you also rely on the package owner; one fit of pique, and your dependency could disappear. Along with versioning, Google needs to add its own repository for versioned artifacts, much like Maven Central, CPAN, or RubyGems.org. This could work transparently to the user: remote imports of public projects would be proxied through Google's server, and it would keep a copy of anything that had a version number.

Learning Go: Goroutines and Multiple Cores

Here's a program that I've been using to explore Go concurrency. If you want to play along at home, you'll need to compile and run it locally; the Playground won't suffice.

package main

import (
    "fmt"
    "runtime"
    "syscall"
)

type Response struct {
    Received int
    Calculated int
    Handler int
    Tid int
}

func Listener(me int, in chan int, out chan Response, done chan int) {
    for val := range in {
        out <- Response{val, runCalc(me), me, syscall.Gettid()}
    }
    done <- me
}

func runCalc(num int) int {
    zz := 1
    for ii := 0 ; ii < 100000000 ; ii++ {
        zz += ii % num
    }
    return zz
}

func main() {
    fmt.Println("Main running on thread ", syscall.Gettid(), " numCPU = ", runtime.NumCPU())

    chSend := make(chan int, 100)
    chRecv := make(chan Response)
    chDone := make(chan int)

    listenerCount:= 8
    for ii := 1 ; ii <= listenerCount ; ii++ {
        go Listener(ii, chSend, chRecv, chDone)
    }

    messageCount := 100
    for ii := 0 ; ii < messageCount ; ii++ {
        chSend <- ii
    }
    close(chSend)

    for listenerCount > 0 {
        select {
            case data := <- chRecv :
                fmt.Println("Received ", data.Received, ",", data.Calculated, " from ", data.Handler, " on thread ", data.Tid)
                messageCount--
            case lnum := <- chDone :
                fmt.Println("Received DONE from ", lnum)
                listenerCount--
        }
    }

    fmt.Println("Main done, outstanding messages = ", messageCount)
}

The short description of this program is that it kicks off a bunch of goroutines, then sends them CPU-intensive work. My goal in writing it was to explore thread affinity and communication patterns as the amount of work increased. Imagine my surprise when I saw the following output from my 4 core CPU:

go run multi_listener.go 
Main running on thread  27638 , numCPU =  8
Received  0 , 1  from  1  on thread  27640
Received  1 , 1  from  1  on thread  27640
Received  2 , 50000001  from  2  on thread  27640
Received  3 , 100000000  from  3  on thread  27640
Received  4 , 150000001  from  4  on thread  27640
Received  5 , 200000001  from  5  on thread  27640
Received  6 , 249999997  from  6  on thread  27640
Received  7 , 299999996  from  7  on thread  27640
Received  8 , 350000001  from  8  on thread  27640
Received  9 , 1  from  1  on thread  27640
Received  10 , 1  from  1  on thread  27640
Received  11 , 50000001  from  2  on thread  27640
Received  12 , 100000000  from  3  on thread  27640
Received  13 , 150000001  from  4  on thread  27640
Received  14 , 200000001  from  5  on thread  27640

The thread ID is's always the same! And top confirmed that this wasn't a lie: one core was consuming 100% of the CPU, while the others were idle. It took some Googling to discover the GOMAXPROCS environment variable:

experiments, 505> export GOMAXPROCS=4
experiments, 506> go run multi_listener.go 
Main running on thread  27674 , numCPU =  8
Received  2 , 350000001  from  8  on thread  27677
Received  0 , 299999996  from  7  on thread  27678
Received  1 , 1  from  1  on thread  27674
Received  3 , 350000001  from  8  on thread  27677
Received  4 , 50000001  from  2  on thread  27679
Received  5 , 299999996  from  7  on thread  27678
Received  6 , 200000001  from  5  on thread  27674
Received  7 , 249999997  from  6  on thread  27677
Received  8 , 100000000  from  3  on thread  27679
Received  9 , 1  from  1  on thread  27678
Received  10 , 200000001  from  5  on thread  27674
Received  11 , 150000001  from  4  on thread  27677

This variable is documented in the runtime package docs, and also in the (28 page) FAQ. It's not mentioned in the Go Tour or tutorial.

I'm a bit taken aback that it's even necessary, however the comment that goes along with the associated runtime method gives a hit: “This call will go away when the scheduler improves.” As of Go 1.2, the behavior remains, one of the quirks of using a young framework.