My Clojure Workflow, Reloaded

This is a companion post to my discussion with Craig Andera on Relevance Podcast Episode 32 and my Clojure/West talk Clojure in the Large. I've talked about various bits and pieces of this workflow at other times, too, but I'll try to bring it all together here in the hopes that others will find it useful.

One of the great pleasures of working with a dynamic language is being able to build a system while simultaneously interacting with it. To make this possible, first you need the ability to redefine parts of the program while it is running: Clojure provides this capability admirably. However, some aspects of Clojure's runtime are not quite as late-binding as one might wish for interactive development. For example, the effect of a changed macro definition will not be seen until code which uses the macro has been recompiled. Changes to methods of a defrecord or deftype will not have any effect on existing instances of that type.

The facilities that Clojure provides for loading code from files are not sufficient to deal with these issues. I wrote the second version of tools.namespace to make a "smarter" require that recognizes dependencies between namespaces and reloads them appropriately.

But tools.namespace is only part of the story. To really get the benefit of interactive development, I want to ensure that the version of the application I am currently interacting with is congruent with the source files I'm editing. That means not only that the application must be running the most up-to-date version of the code, but also that any state in the application was produced by that same code. It is dangerously easy, when changing and reloading code at the REPL, to get an application into a state which could not have been reached by the code it is currently running.

Therefore, after every significant code change, I want to restart the application from scratch. But I don't want to restart the JVM and reload all my Clojure code in order to do it: that takes too long and is too disruptive to my workflow. Instead, I want to design my application in such a way that I can quickly shut it down, discard any transient state it might have built up, start it again, and return to a similar state. And when I say quickly, I mean that the whole process should take less than a second.

To achieve this goal, I make the application itself into a transient object. Instead of the application being a singleton tied to a JVM process, I write code to construct instances of my application, possibly many of them within one JVM. Each time I make a change, I discard the old instance and construct a new one. The technique is similar to dealing with virtual machines in a cloud environment: rather than try to transition a VM from an old state to a new state, we simply discard the old one and spin up a new one.

Designing applications this way requires discipline. First and foremost, all state must be local. Any global state, anywhere, breaks the whole model. Second, all resources acquired by the application instance must be carefully managed so that they can be released when the instance is destroyed.

Enough talk. Here's how it works.

Read More »