PCL -> Clojure, Chapter 9

This article is part of a series describing a port of the samples from Practical Common Lisp (PCL) to Clojure. You will probably want to read the intro first.

This article covers Chapter 9, Practical: Building a Unit Test Framework.

Tests and reports

To build a minimal testing library, I need nothing more than tests and results. To keep reporting as simple as possible, I will start with console output. The report-result function tests a result, and prints pass or FAIL, plus a form with supporting detail:

  (defn report-result [result form]
    (println (format "%s: %s" (if result "pass" "FAIL") (pr-str form))))

Now any function can be a test. The detail message can often be the same form that caused the error, so I will pass the same form twice: once for evaluation, and again (quoted!) for use in the detail message:

  (defn test-+ []
    (report-result (= (+ 1 2) 3) '(= (+ 1 2) 3))
    (report-result (= (+ 1 2 3) 6) '(= (+ 1 2 3) 6))
    (report-result (= (+ -1 -3) -4) '(= (+ -1 -3) -4)))

The console output for test-+ looks like this:

  user=> (test-+)
  pass: (= (+ 1 2) 3)
  FAIL: (= (+ 1 2 3) 7)
  pass: (= (+ -1 -3) -4)

Inferring the detail message

The fact that I want to pass the same form twice, but with different evaluation semantics, just screams macro. Sure enough, I can clean up the code with a macro:

  (defmacro check [form]
    `(report-result ~form '~form))

The macro expands the form twice, once for evaluation and once quoted for the detail message. Now I can replace calls to report-result with simpler calls to check:

  (defn test-* []
    (check (= (* 1 2) 3))
    (check (= (* 1 2 3) 6))
    (check (= (* -1 -3) -4)))

Hmm. The calls to check are cleaner than the calls to report-result in the earlier example, but the check itself still looks repetitive. Solution: a better check macro that can handle multiple forms:

  (defmacro check [& forms]
    `(do
       ~@(map (fn [f] `(report-result ~f '~f))  forms)))

The quoting and unquoting is a little more complex--play around with macroexpand-1 to see how it works.

With the better check in place, test functions are quite simple:

  (defn test-rem []
    (check (= (rem 10 3) 1)
     (= (rem 6 2) 0)
     (= (rem 7 4) 3)))

Aggregating results

So far I have tests and console output. Next, I need some way to aggregate a set of checks into a single, top-level "checks passed" or "checks failed".

I would like to simply and together all the individual checks, but that does not quite work. As in many languages, Clojure's and short-circuits and stops evaluating when it encounters a logical false. That's no good here: Even if one test fails, I still want all the tests to run.

Since it is a question of optional evaluation, a macro is appropriate. The combine-results macro works like and, but it always evaluates all the forms:

  (defmacro combine-results [& forms]
    `(every? identity (list ~@forms)))

Now check can use combine-results instead of do.

  (defmacro check [& forms]
    `(combine-results
      ~@(map (fn [f] `(report-result ~f '~f)) forms)))

All existing functionality still works, and now I can see a useful return value from a test.

  user=> (test-*)
  pass: (= (* 2 4) 8)
  pass: (= (* 3 3) 9)
  true

Capturing test names

Tests ought to have names. In fact, tests ought to support multiple names. You can imagine a test detail report saying:

Check math->addition->associative passed: ...

Where associative is the name of a check, addition is the name of a function, and math is the name of another function that called addition.

First, I need a variable to store a sequence of names:

  (def *test-name* [])

Printing the variable as part of a result is easy:

  (defn report-result [result form]
    (println (format "%s: %s %s" 
           (if result "pass" "fail") 
           (pr-str *test-name*) 
           (pr-str form)))
    result)

Now for the hard part: populating the collection of names. For this, I will introduce a deftest macro:

  (defmacro deftest [name & forms]
    `(defn ~name []
       (binding [*test-name* (conj *test-name* (str '~name))]
         ~@forms)))

The macro expansion perfomed by deftest is nothing new: deftest turns around and defns a new function named name. The interesting part is the call to binding, which rebinds *test-name* to a new collection built from the old *test-name* plus the name of the current test.

The new binding of *test-name* is visible anywhere inside the dynamic scope of the binding form. The dynamic scope includes any function calls made inside the binding, and their function calls, and so on ad infinitum ... or until another binding performs the same trick again. This gives exactly the semantics we want:

  • The dynamic scope allows callers to influence callees without having to pass test-name an an argument all over the place. Nested functions "remember" a stack of their caller's names through *test-name*.
  • The unwinding of the dynamic scope protects readers of *test-name* outside a binding. Code after the binding will never see the values *test-name* takes during the binding.
  • Dynamic bindings are thread-local (and therefore thread-safe).

With deftest in place, I can defined a hierarchy of nested tests:

  (deftest test-*
    (check (= (* 2 4) 8)
     (= (* 3 3) 9)))

  (deftest test-math
    ; TODO: test rest of math
    (test-*))

  (deftest test-all-of-nature
    ; TODO: test rest of nature
    (test-math))

Calling test-all-of-nature will demonstrate multiple levels of nested name in a test report:

  user=> (test-all-of-nature)
  pass: ["test-all-of-nature" "test-math" "test-*"] (= (* 2 4) 8)
  pass: ["test-all-of-nature" "test-math" "test-*"] (= (* 3 3) 9)
  true                         

From here, better formatting of the console message is just mopping up.

Wrapping up

When I first read Practical Common Lisp, this was my favorite chapter. The testing library evolves quickly and naturally to a substantial feature set. (In case you didn't keep count, the entire "framework" is less than twenty lines of code.)

Try implementing the unit-testing example in your language of choice. Don't just implement the finished design. Work through each of the iterations described above:

  1. tests and results
  2. inferring the detail message
  3. aggregating results
  4. capturing test names

I would love to hear about your results, and I will link to them here.

Notes