This article covers Chapter 9, Practical: Building a Unit Test Framework.
Tests and reports
To build a minimal testing library, I need nothing more than tests and results. To keep reporting as simple as possible, I will start with console output. The
report-result function tests a
result, and prints
FAIL, plus a
form with supporting detail:
(defn report-result [result form] (println (format "%s: %s" (if result "pass" "FAIL") (pr-str form))))
Now any function can be a test. The detail message can often be the same form that caused the error, so I will pass the same form twice: once for evaluation, and again (quoted!) for use in the detail message:
(defn test-+  (report-result (= (+ 1 2) 3) '(= (+ 1 2) 3)) (report-result (= (+ 1 2 3) 6) '(= (+ 1 2 3) 6)) (report-result (= (+ -1 -3) -4) '(= (+ -1 -3) -4)))
The console output for
test-+ looks like this:
user=> (test-+) pass: (= (+ 1 2) 3) FAIL: (= (+ 1 2 3) 7) pass: (= (+ -1 -3) -4)
Inferring the detail message
The fact that I want to pass the same form twice, but with different evaluation semantics, just screams macro. Sure enough, I can clean up the code with a macro:
(defmacro check [form] `(report-result ~form '~form))
The macro expands the form twice, once for evaluation and once quoted for the detail message. Now I can replace calls to
report-result with simpler calls to
(defn test-*  (check (= (* 1 2) 3)) (check (= (* 1 2 3) 6)) (check (= (* -1 -3) -4)))
Hmm. The calls to
check are cleaner than the calls to
report-result in the earlier example, but the
check itself still looks repetitive. Solution: a better
check macro that can handle multiple forms:
(defmacro check [& forms] `(do ~@(map (fn [f] `(report-result ~f '~f)) forms)))
The quoting and unquoting is a little more complex--play around with
macroexpand-1 to see how it works.
With the better
check in place, test functions are quite simple:
(defn test-rem  (check (= (rem 10 3) 1) (= (rem 6 2) 0) (= (rem 7 4) 3)))
So far I have tests and console output. Next, I need some way to aggregate a set of checks into a single, top-level "checks passed" or "checks failed".
I would like to simply
and together all the individual checks, but that does not quite work. As in many languages, Clojure's
and short-circuits and stops evaluating when it encounters a logical
false. That's no good here: Even if one test fails, I still want all the tests to run.
Since it is a question of optional evaluation, a macro is appropriate. The
combine-results macro works like
and, but it always evaluates all the forms:
(defmacro combine-results [& forms] `(every? identity (list ~@forms)))
check can use
combine-results instead of
(defmacro check [& forms] `(combine-results ~@(map (fn [f] `(report-result ~f '~f)) forms)))
All existing functionality still works, and now I can see a useful return value from a test.
user=> (test-*) pass: (= (* 2 4) 8) pass: (= (* 3 3) 9) true
Capturing test names
Tests ought to have names. In fact, tests ought to support multiple names. You can imagine a test detail report saying:
Check math->addition->associative passed: ...
associative is the name of a check,
addition is the name of a function, and
math is the name of another function that called
First, I need a variable to store a sequence of names:
(def *test-name* )
Printing the variable as part of a result is easy:
(defn report-result [result form] (println (format "%s: %s %s" (if result "pass" "fail") (pr-str *test-name*) (pr-str form))) result)
Now for the hard part: populating the collection of names. For this, I will introduce a
(defmacro deftest [name & forms] `(defn ~name  (binding [*test-name* (conj *test-name* (str '~name))] ~@forms)))
The macro expansion perfomed by
deftest is nothing new:
deftest turns around and
defns a new function named
name. The interesting part is the call to
binding, which rebinds
*test-name* to a new collection built from the old
*test-name* plus the name of the current test.
The new binding of
*test-name* is visible anywhere inside the dynamic scope of the
binding form. The dynamic scope includes any function calls made inside the binding, and their function calls, and so on ad infinitum ... or until another
binding performs the same trick again. This gives exactly the semantics we want:
- The dynamic scope allows callers to influence callees without having to pass
test-namean an argument all over the place. Nested functions "remember" a stack of their caller's names through
- The unwinding of the dynamic scope protects readers of
binding. Code after the
bindingwill never see the values
*test-name*takes during the
- Dynamic bindings are thread-local (and therefore thread-safe).
deftest in place, I can defined a hierarchy of nested tests:
(deftest test-* (check (= (* 2 4) 8) (= (* 3 3) 9))) (deftest test-math ; TODO: test rest of math (test-*)) (deftest test-all-of-nature ; TODO: test rest of nature (test-math))
test-all-of-nature will demonstrate multiple levels of nested name in a test report:
user=> (test-all-of-nature) pass: ["test-all-of-nature" "test-math" "test-*"] (= (* 2 4) 8) pass: ["test-all-of-nature" "test-math" "test-*"] (= (* 3 3) 9) true
From here, better formatting of the console message is just mopping up.
When I first read Practical Common Lisp, this was my favorite chapter. The testing library evolves quickly and naturally to a substantial feature set. (In case you didn't keep count, the entire "framework" is less than twenty lines of code.)
Try implementing the unit-testing example in your language of choice. Don't just implement the finished design. Work through each of the iterations described above:
- tests and results
- inferring the detail message
- aggregating results
- capturing test names
I would love to hear about your results, and I will link to them here.
- The sample code is available at http://github.com/stuarthalloway/practical-cl-clojure.
- Feedback on how to improve these examples is most welcome!