Relevance, Inc. | Agile Software Design & Development

Clojure Interview on InfoQ

Tags: clojure

Werner Schuster over at InfoQ has interviewed me about Clojure. Check it out.

Programming Clojure Beta 6 is Out

Tags: clojure and programming-clojure

Programming Clojure Beta 6 is now available. What's new:

• A new Chapter, "Functional Programming," exlains recursion, TCO, laziness, memoization, and trampolining in Clojure.
• A new Section, "Creating Java Classes in Clojure," shows how to create and compile Java classes from Clojure.
• The snake example has been moved into the concurrency chapter. The snake demonstrates how to divide your model layer into two parts: a functional model and a mutable model.
• The STM example in the introduction now uses alter instead of commute, which allowed a race condition. Given the problem domain of the example, the race condition was acceptable. However, explaining this in the introductory chapter would have been distracting.
• The lancet runonce function now uses locking instead of an an agent. Yes, you can do plain old-fashioned locking in Clojure! Agents are unsuitable for the kind of coordination lancet requires, because you cannot await an agent while inside another agent.

While I am talking about lancet: it now has its own repository. I have added integration with clojure.contrib.shell-out, so lancet can now call either Ant tasks or other applications. It is still far from being a replacement from Ant or Rake, but maybe your contributions can change that!

The book is now prose-complete, so if you have been waiting for the whole story, this is it. A number of readers have made suggestions for additional topics. If you have suggestions, please add them as comments to this post. For reasons of space and time, most new topics will appear in a series of articles on this blog, not in the book itself.

Clojure and clojure-contrib continue to evolve. To make sure you have the latest, greatest version of the sample code from the book, go and grab the github repo.

Thanks again to everyone who has been offering feedback. I have cleared almost 400 entries from the errata/suggestion page.  Keep the feedback coming!

Programming Clojure Beta 5 is Out

Tags: clojure, programming-clojure, lisp, and java

Programming Clojure Beta 5 is now available. What's new:

A new chapter, Clojure in the Wild, with information on

• unit testing with test-is
• data access with clojure.contrib.sql
• web development with Compojure
• desktop GUI development with Clojure and Swing

I have rewritten Section 1.1 "Clojure Coding Quick Start." All dependent libraries (Clojure, clojure-contrib, etc.) are now prebuilt and included with the sample code for the book. This should make it much easier for people to get started.

I have rewritten Section 2.6 "Metadata" to clearly explain the difference between user metadata and compiler metadata.

Clojure and clojure-contrib continue to evolve. To make sure you have the latest, greatest version of the sample code from the book, go and grab the github repo.

Thanks to everyone who has been offering feedback. I have cleared over 300 entries from the errata/suggestion page.  Keep the feedback coming!

On Lisp -> Clojure, Chapter 9

Tags: clojure and on-lisp

This article is part of a series describing a port of the samples from On Lisp (OL) to Clojure. You will probably want to read the intro first.

This article covers Chapter 9, Variable Capture.

Macro Argument Capture

Macros and "normal" code are written in the same language, and can share access to names, data, and code, This is the source of their power, but can also cause subtle bugs. What happens if a macro caller and a macro implementer both try to use the same name? The macro can "capture" the name, leading to unintended consequences.

OL begins with an example of argument capture, a broken definition of for. Here is a similar macro in Clojure:

  (defmacro bad-for [[idx start stop] & body]
    `(loop [~idx ~start limit ~stop]
       (if (< ~idx ~stop)
         (do
     ~@body
     (recur (inc ~idx) limit)))))

The problem is the name limit introduced inside the macro. If you call bad-for after binding the name limit, strange things will happen. What if you try:

  (let [limit 5] 
    (bad-for [i 1 10] 
      (if (> i limit) (print i))))

Presumably the intent here is to print some numbers greater than five. But in many Lisps, this would print nothing, because the bad-for macro invisibly binds limit to ten.

Clojure catches this problem early, and fails with a descriptive error:

  (let [limit 5] (bad-for [i 1 10] (if (> i limit) (println i))))
  -> java.lang.Exception: Can't let qualified name: ol.chap-09/limit

Clojure makes it difficult to accidentally capture limit, by resolving symbols into a namespace. The limit inside the macro resolves to ol.chap-09/limit, and there is no name collision.

Of course, you do not want your macros to use a shared global name either! What you really want is for macros to use their own guaranteed-unique names. Clojure provides this via auto-gensyms. Simple append # to limit in the bad-for example above, and you get good-for:

  (defmacro good-for [[idx start stop] & body]
    `(loop [~idx ~start limit# ~stop]
       (if (< ~idx limit#)
         (do
     ~@body
     (recur (inc ~idx) limit#)))))

Now the macro will use a unique generated name like limit__395, and callers can use good-for as expected:

  (let [limit 5] (good-for [i 1 10] (if (> i limit) (println i))))
  6
  7
  8
  9

Symbol Capture

Another form of unintended capture is symbol capture, where a symbol in the macro unintentionally refers to a local binding in the environment. OL demonstrates the problem with this example:

First, w is a global collection of warnings that have occurred when using a library. In Clojure:

  (def w (ref []))

This is different from the OL implementation because in Clojure data structures are immutable, and mutable things must be wrapped in a reference type that has explicit concurrency semantics. In the code above the ref wraps the immutable [].

Second, the gripe macro adds a warning to w, and returns nil. gripe is intended to be used when bailing out of a function called with bad arguments. In Clojure:

  (defmacro gripe [warning]
    `(do
       (dosync (alter w conj ~warning))
       nil))

Again, this is fairly different from OL because you must be explicit about mutable state. To update w you must use a transaction (dosync) and a specific kind of update function (such as alter).

Third, there is a library function sample-ratio that performs some kind of calculation, the details of which are irrelevant to the example. sample-ratio also uses gripe to warn and bailout for certain bad inputs. In Clojure:

  (defn sample-ratio [v w]
    (let [vn (count v) wn (count w)]
      (if (or (< vn 2) (< wn 2))
        (gripe "sample < 2")
        (/ vn wn))))

This is practically identical to the OL version, since there is no mutable state to (directly) deal with.

Since we are talking about symbol capture, you can probably guess the problem: What happens when the global w for warnings collides with the local w argument in sample-ratio?

In Common Lisp, this sort of capture would cause the error message to be added to the wrong collection: the local samples w instead of the global warnings w.

In Clojure, this just works. The global w resolves into a namespace, and does not collide with the local one.

More Complex Macros

Clojure's namespaces and auto-gensyms take care of many common problems in macros, but what if you really want capture? You can capture symbols by unquoting them with the unquote character (~, a tilde) and then requoting them with a non-resolving quote character (', a single quote). For example, here is a bad version of gripe that goes out of its way to do the wrong thing and capture w:

  (defmacro bad-gripe [warning]
    `(do
       (dosync (alter ~'w conj ~warning))
       nil))

I am not going to show more complex macros that really need this feature. My point here is to show that Clojure doesn't make macros safer by compromising their power. You can still do nasty things, you just have to be more deliberate about it.

Interestingly, Clojure protects you from bad-gripe, even after you go to the trouble of introducing inappropriate symbol capture. Here is a bad-sample-ratio that uses the buggy bad-gripe:

  (defn bad-sample-ratio [v w]
    (let [vn (count v) wn (count w)]
      (if (or (< vn 2) (< wn 2))
        (bad-gripe "sample < 2")
        (/ vn wn))))

If you try to call bad-sample-ratio with bad inputs, bad-gripe will not be able to modify the wrong collection:

  (bad-sample-ratio [] [])
  -> java.lang.ClassCastException: clojure.lang.PersistentVector cannot\
     be cast to clojure.lang.Ref

Now you see how having immutability as the default can protect you from bugs. The global w is an explicitly mutable reference. But the local w is an implicitly immutable vector. When bad-gripe tries to update the wrong collection, it is thwarted by the fact that the collection is immutable.

Wrapping up

Clojure makes simple macros easier and safer to write. The combination of namespace resolution and auto-gensyms prevents many irritating bugs.

Clojure still has the power to write more complex macros when you need it. With the right combination of unquoting and quoting, you can undo the safety net and write any kind of macro you want.

One final note: Because they are ported straight from Common Lisp, many of the examples here are not idiomatic Clojure. In Clojure most uses of imperative loops such as good-for would be replaced by a more functional style. A good example of this is Clojure's own for, which performs sequence comprehension.

Notes

• The sample code is available at http://github.com/stuarthalloway/onlisp-clojure.

Other Resources

If you find this series helpful, you might also like:

• My book Programming Clojure
• My blog series porting Practical Common Lisp to Clojure

Revision history

• 2008/12/17: initial version

On Lisp -> Clojure, Chapter 7

Tags: clojure and on-lisp

This article is part of a series describing a port of the samples from On Lisp (OL) to Clojure. You will probably want to read the intro first.

This article covers Chapter 7, Macros.

A Few Simple Macros

OL begins with a simple nil! macro that sets something to nil. nil! is implemented as a macro in Common Lisp (CL) nil needs to generate a special form. Clojure puts much more careful boundaries around mutable state, so most Clojure data structures are not set-able at all. The few things that can be set are reference types, each with an explicit API and concurrency semantics.

Because setters go through an explicit API instead of a special form, the Clojure nil! does not need to be macro at all. Here is a nil! for Clojure atoms:

  (defn nil! [at]
    (swap! at (fn [_] nil)))

The swap! function is specific to atoms. Usage for nil! looks like:

  (def a (atom 10))
  (nil! a)
  @a
  -> nil  

The next interesting macro in OL is nif, which demonstrates the use of backquoting. One way to implement Clojure nif is:

  ((use '[clojure.contrib.fcase :only (case)])
  (defmacro nif [expr pos zer neg]
    `(case (Integer/signum ~expr) 
     -1 ~neg
     0 ~zer
     1 ~pos))

There are a few interesting differences from CL here:

• Clojure unquoting uses ~ and ~@ instead of CL's , and ,@. This allows Clojure to treat commas as whitespace.
• Clojure does not have a built-in signum, but it has access to all of Java, including Integer/signum.
• Clojure's case is not part of core, and is provided by Clojure Contrib.

Defining Simple Macros

OL demonstrates the "fill in the blanks" approach to writing macros:

• Write the desired expansion.
• Write the desired macro invocation form.
• Use backquoting to create a template based on the desired expansion.
• Use unquoting to substitute forms from the macro invocation into the template.

As examples, OL uses our-when and our-while. The Clojure equivalents are:

  (defmacro our-when [test & body]
    `(if ~test
       (do
         ~@body)))

  (defmacro our-while [test & body]
    `(loop []
       (when ~test
         ~@body
         (recur))))

There is one interesting new thing here. Clojure' loop/recur is an explicit way to denote a self-tail-call so that Clojure can implement it with a non-stack-consuming iteration. (Clojure cannot optimize tail calls in a generic way due to limitations of the JVM.)

It is also worth noting that while loops are uncommon in Clojure. They rely on side effects that change the result of test, and most Clojure functions avoid side effects.

Destructuring in Macros

Both Clojure and CL support destructuring in macro definitions. The OL example of this is a when-bind macro. Here is a literal translation in Clojure:

  (defmacro when-bind [bindings & body]
    (let [[form tst] bindings]
      `(let [~form ~tst]
         (when ~form
     ~@body))))

The [form tst] is a destructuring bind. The first element of bindings binds to form, and the second element to tst. Usage looks like this:

  (when-bind [a (+ 1 2)] (println "a is" a))
  a is 3

Do not use the when-bind as defined above. Clojure provides a better version called when-let:

  ; from Clojure core
  (defmacro when-let
    [bindings & body]
    (if (vector? bindings)
      (let [[form tst] bindings]
        `(let [temp# ~tst]
           (when temp#
             (let [~form temp#]
               ~@body))))
      (throw (IllegalArgumentException.
               "when-let now requires a vector for its binding"))))

when-let adds two features not present in when-bind:

• when-let requires that the binding form be a vector. This leads to the "arguments in square brackets" style that distinguishes Clojure from many Lisps.
• when-let introduces a temporary binding temp# using Clojure's auto-gensym feature.

The temporary binding of temp# keeps the binding form from being expanded directly into the when, because some binding forms are not legal for evaluation. The following output shows the difference:

  (when-bind [[a & b] [1 2 3]] (println "b is" b))
  ->java.lang.Exception: Unable to resolve symbol: & in this context 
  (when-let [[a & b] [1 2 3]] (println "b is" b))
  -> b is (2 3)

If it is not clear to you why when-bind doesn't work, try calling macroexpand-1 on both the forms above.

Wrapping up

The concepts in OL Chapter 7 translate fairly directly from Common Lisp into Clojure. The bigger differences are choices of idiom. Many of the examples in Common Lisp presume mutable state. In the typical Clojure program these forms would be in the minority.

Notes

• The sample code is available at http://github.com/stuarthalloway/onlisp-clojure.
• Thanks to Chouser for setting me straight on why temp# appears in when-let.

Revision history

• 2008/12/12: initial version

On Lisp -> Clojure

Tags: clojure and on-lisp

I am porting the examples from the macro chapters of Paul Graham's On Lisp (OL) to Clojure.

My ground rules are simple:

• I am not going to port everything, just the code samples that interest me as I re-read On Lisp.
• Where Paul introduced macro features in a planned progression, I plan to use whatever Clojure feature come to mind. So I may jump straight into more "advanced" topics.
  

Please do not assume that this port is a good introduction to Lisp! I am cherry-picking examples that are interesting to me from a Clojure perspective. If you want to learn Lisp, read OL. In fact, you should probably read the relevant chapters in OL first, no matter what.

The Series

• Intro (this post)
  
• Chapter 7, Macros
• Chapter 9, Variable Capture
• Chapter 10, Other Macro Pitfalls

Note: Fogus is also porting On Lisp to Clojure.

Other Stuff

If you find this series helpful, you might also like:

• My book Programming Clojure
• My blog series porting Practical Common Lisp to Clojure

Talks

I am available to give conference talks on Clojure. Check the schedule for an event near you, or contact Relevance (info@thinkrelevance.com) to schedule an event.

Notes

• The sample code is available at http://github.com/stuarthalloway/onlisp-clojure.

Living Lazy, Without Variables

Tags: clojure, functional, and lazy

Programmers coming to functional languages for the first time cannot imagine life without variables. I address this head-on in the Clojure book. In Section 2.7 (free download here), I port an imperative method from the Apache Commons Lang to Clojure. First the Java version:

  // From Apache Commons Lang, http://commons.apache.org/lang/
  public static int indexOfAny(String str, char[] searchChars) {
    if (isEmpty(str) || ArrayUtils.isEmpty(searchChars)) {
      return -1;
    }
    for (int i = 0; i < str.length(); i++) {
      char ch = str.charAt(i);
      for (int j = 0; j < searchChars.length; j++) {
        if (searchChars[j] == ch) {
            return i;
        } 
      }
    }
    return -1;
  }

And now the Clojure code. I have shown the supporting function indexed as well:

  (defn indexed [s] (map vector (iterate inc 0) s))
  (defn index-of-any [s chars]
    (some (fn [[idx char]] (if (get chars char) idx)) 
          (indexed s)))

There are many things I like about the Clojure version, but I want to focus on something I didn't mention already in the book. A reader thought the Clojure version did too much work:

...the [Java] version can be seen as *more efficient* when a match is found because scanning stops right there, whereas "indexed" constructs the whole list of pairs, regardless of whether or not a match WILL be found....

The reader's assumption is reasonable, but incorrect. Clojure's sequence library functions are generally lazy. So the call to indexed is really just a promise to generate indexes if they are actually needed.

To see this, create a logging-seq that writes to stdout every time it actually yields an element:

  (defn logging-seq [s]
    (if s
      (do (println "Iterating over " (first s))
    (lazy-cons (first s) (logging-seq (rest s))))))

Now, you can add logging-seq to indexed so that each element of indexed is of the form [index, element, logged-element].

  (defn indexed [s] (map vector (iterate inc 0) s (logging-seq s)))

Test the modified indexed function at the Clojure REPL:

  user=> (indexed "foo")
  Iterating over  f
  (Iterating over  o
  [0 \f \f] Iterating over  o
  [1 \o \o] [2 \o \o])

As you can see, the indexed sequence is only produced as needed. (At the REPL it is needed to print the return value.)

Finally, you can test indexed-of-any and see that Clojure only produces enough of the sequence to get an answer. For a match on the first character, it only goes to the first character:

  (index-of-any "foo" #{\f})
  Iterating over  f
  0

If there is no match, index-of-any has to traverse the entire string:

  (index-of-any "foo" #{\z})
  Iterating over  f
  Iterating over  o
  Iterating over  o
  nil

So give up on those variables, and live lazy!

Clojure Wins Again

Tags: clojure, javascript, lisp, java, and emacs

Steve Yegge's most recent post takes a right angle turn about a third of the way through, and begins a comparison of Emacs Lisp and JavaScript.

And the winner is ... Clojure!

OK, Steve didn't say that. What he did do was call out things he liked about JavaScript and Emacs Lisp.

For JavaScript:

• momentum
• (namespace) encapsulation
• delegation (polymorphism?)
• properties (by Steve's definition)
• serialize to source

For Emacs Lisp:

• Macros
• S-Expressions

I first picked up Clojure looking for many of the same things that Steve wants. I found them. Clojure can do all the things on both lists above. (Serialize to source isn't formal yet, but check the mailing list. And of course, you will have to judge "momentum" for yourself.)

The scary thing is that Clojure wins the language war before you even learn about its signature features. When I started exploring Clojure, I quickly realized it had everything I wanted, which could be summarized as "Lisp that really embraces the Java platform."

Then Clojure changed the definition of what I wanted. Now I also want

• persistent data structures
• functional programming
• STM and agents
  

If you have half an hour, watch a compelling vision of what software development will look like in 2010.

Clojure Beta Book Available

Tags: clojure and book

The Clojure Beta book is now available. Here's the Table of Contents. (Chapters with an asterisk are included in this beta.)

• Preface*
• Getting Started*
• Exploring Clojure*
• Working with Java*
• Unifying Data with Sequences*
• Functional Programming
• Concurrency*
• Macros
• Multimethods
• Third-Party Libraries
• Case Study

Because this is a Beta book, and Clojure is continuing to evolve, there will be errata. Please let me know any problems you find, and I will address them in the next Beta.

Other Clojure resources

• Rich Hickey's excellent videos
• My blog series converting Practical Common Lisp examples to Clojure
• Bill Clementson's setup instructions for Clojure

Concurrent Programming with Clojure

Tags: clojure and training

Clojure is a dynamic language for the Java Virtual Machine with several powerful features for building concurrent applications. In this talk you will learn about:

• Functional programming. Clojure's immutable, persistent data structures encourage side-effect free programming that can easily scales across multiple processor cores.
• Software Transactional Memory (STM). STM provides a mechanism for managing references and updates across threads that is easier to use and less error-prone than lock-based concurrency.
• Agents. Agents provide a thread-safe mechanism for asynchronous, uncoordinated updates.
• Atoms. Atoms provide for synchronous, uncoordinated updates.
• Dynamic Vars. Dynamic Vars support thread-local state.
• Direct access to Java. Clojure calls Java directly, and can emit the same byte code that a handcrafted Java program would. Where it makes sense to do so, you can easily access the java.util.concurrent library.