Blog

May 29 2013

Comments

Stuart Sierra - Episode 032

cover art

This past February, I found myself driving across rural Virginia with Stuart Sierra enroute to Relevance World HQ. We had a few hours to kill, and I had recently had the opportunity to work extensively with Stuart on a Clojure project for a client. Stuart has a particular - and, in my experience, productive - way of arranging things, so I thought it would be fun and informative to record an episode exploring it.

Our conversation covered some of the same ground as Stuart's Clojure/West presentation "Clojure in the Large", so if you want to hear more, keep an eye out for that video, to be released in the coming months.

I hope you enjoy this episode!

Read More »

May 21 2013

Comments

Entropy and Evolution of a Codebase

Let's do a thought experiment about change over time. We will start with a "clean" codebase, organized into a bunch of modules. These can be classes, subsystems, namespaces... it doesn't matter for our purposes.

As time goes on, we have to make changes in the code. Suppose the changes are distributed randomly across the codebase. Some modules will get changed a lot more often than others. Empirically, the distribution of changes almost always follows a power law. That is, a few modules get changed very frequently while a decreasing number have an increasing interval between changes.

(In fact, concave shapes resembling a power law are very common in software. We see them in object lifespans, coupling, and the distribution of complexity across modules.)

We started with a clean codebase, but what does that mean? To me, it means that the code is simple, cohesive, and loosely coupled. In short, it is easy to change, and most changes are local.

With every change a developer makes, the module might lose cohesion, gain coupling, or become complex. These are entropy-increasing changes. On the other hand, the developer could split an incohesive module, decouple a coupled one, or factor out complexity. These are entropy-reducing changes.

Even with a very disciplined team, some proportion of changes will be entropy-increasing. We would all like to avoid them, but they happen anyway. I think of these as introducing stress into the codebase. The stress is trying to collapse the codebase into a Ball of Mud. We must eventually relieve that stress through refactoring. At every change, there is a chance that the module transitions from clean to dirty, or from dirty to clean.

If the odds of a clean-to-dirty transition and a dirty-to-clean transition were equal, then we should expect the codebase to reach a kind of thermodynamic equilibrium where dirtiness is conserved, but shuttled around. Sadly, the transition odds are not equal. The definition of clean means that it is easy to change. Dirty code is hard to change. We all intuitively understand that it is much harder to clean up a module than to dirty it in the first place. Therefore, the likelihood of clean-to-dirty transitions is higher than that of the reverse. From this alone, we could expect the fraction of dirty modules to increase over time. It is entropic decay.

There's a secondary effect at play that exacerbates this problem. Because it is easier to change a clean module than a dirty one, we will see some kinds of changes "sliding sideways" from a dirty module to a cleaner one. This is just another way of saying that kludges cluster. A developer who needs to change a dirty module in a difficult way is very likely to make a nearby change instead, especially under time or deadline pressure. That nearby change inevitably dirties the module it lands in, often by adding coupling.

If you imagine the modules in a codebase like cells in a Game of Life automaton, you could see cells fading from healthy blue to sickly red, then transmitting their disease elsewhere. Very occasionally, you'll see a cell or cluster of cells brightening to health as a developer restructures that area. Mostly, the codebase will decay until it must be discarded or rewritten.

Can this entropic decay be prevented?

Yes, but it is not easy. We know of several ways to avoid it.

  1. Surgical anti-entropy. Have people consistently tackle the gnarliest, smelliest parts of the system and bring them back to health.
  2. Ongoing stewardship. Have people nurture their part of the codebase indefinitely. They should have the time, skill, and desire to tend to their garden.
  3. Methodical attacks on suspected entropy strongholds. Use version control records to find the least-frequently-changed modules that are coupled to areas with high change rates. Odds are, the infrequently changed module is really dirty. The frequent changes in nearby modules indicate "sliding away" from the problem. The long time between changes on the module in the middle is a sign of sickness, not stability.
  4. Community spirit. Make sure the whole team knows that every single person is responsible for cleaning modules, fixing broken windows, reducing coupling, and reversing entropy. (This is the XP approach of Collective Code Ownership.)
  5. Disposable code. Discard the entire codebase periodically and recreate it. Do this often enough and you will create smaller and smaller codebases, each isolated from the others. Sam Umbach points out that many effective startups are built on this idea: if the startup fails, you throw the code away; if it succeeds, you throw it away and rewrite.

Some applications live for decades. These require strong anti-entropy measures. Others live for days or weeks. Dirty code is more expensive to maintain, but with such a short lifespan, who cares if these get dirty? Whatever your approach, think about the expected lifespan of the codebase. Be deliberate about your approach to entropy.

May 14 2013

Comments

Chas Emerick, Mostly Lazy - Podcast Episode 031

cover art

Chas Emerick is a well-known and respected member of the Clojure community. His contributions to the Clojure world include the Friend authentication library, the annual State of Clojure Survey, and, of course, his book, to name just a few. On top of that, he's a very insightful and interesting person. So I was thrilled to get the chance to sit down with him and record what I think turned out to be a fascinating conversation. We talked about his book, his business, his secret new project, "100% time", and the weight of the word "should".

As a bonus, although he and I disagree about whose idea this was (I still say it was his), we continued the conversation on Mostly Lazy, his podcast. That episode is available here.

Download the episode.

Read More »

May 07 2013

Comments

Jason Wolfe of Prismatic - Podcast Episode 030

cover art

One of the few unfortunate aspects to working at Relevance is that we sometimes know about people using Clojure in cool ways that we can't talk about. So when Ben Moss suggested that we talk to someone at Prismatic, I jumped at the chance. They have definitely put Clojure into production in a big and awesome way, and I was really glad to get the chance to talk to Jason Wolfe about their experience building Prismatic, including some of the cool stuff they've open sourced out of it. A big thanks to Jason, and to you, our listeners.

Download the episode.

Read More »

May 01 2013

Comments

Where to Find Relevancers: May Edition

Want to meet a Relevancer in person? Here's where you can find us during the month of May:

Little Rock, AR 5/4
Made by Few
Attending: Kevin Altman

Norfolk, VA 5/6 @ 6pm
757.rb Ruby Meetup Group
Speaking: Russ Olsen: Looking Inside Your Ruby Implementation

Sydney, Australia 5/9
YOW! Night
Speaking: Stuart Sierra: Clojure: Lisp for the Real World

Cambridge, MA 5/9 @ 6:30pm
Clojure Meetup
Attending: Gabriel Horner

Sydney, Australia 5/10-5/11
Intro to Clojure Training
Trainer: Stuart Sierra

Brisbane, Australia 5/14-5/15
Intro to Clojure Training
Trainer: Stuart Sierra

Portland, OR 5/15
Refresh PDX
Speaker: Michael Parenteau: The Myth of the Half-Brained Designer… And Their Magic Vacuum

Brisbane, Australia 5/16-5/17
YOW! Lambda Jam
Speaking: Stuart Sierra: Keynote, Data, Visibility & Abstraction; Datomic Jam

Melbourne, Australia 5/20-5/21
Intro to Clojure Training
Trainer: Stuart Sierra

Melbourne, Australia 5/21
YOW! Night
Speaking: Stuart Sierra: Clojure: Lisp for the Real World

Apr 23 2013

Comments

Lake Denman - Podcast Episode 029

cover art

Michael Parenteau suggested to me that we ought to have Lake Denman on the show. "He's done some cool stuff, man." I remember hearing about a crazy and awesome Arduino project that Lake had worked on, so I thought, "Yeah, sure, we should totally have Lake on." And I'm very glad we did: Lake has indeed worked on some amazingly creative projects, and he has even more ambitious and interesting ones in mind. I hope you enjoy listening to him talk about them on this episode as much as I enjoyed hearing them.

Read More »

Apr 22 2013

Comments

The Shoemakers

Over an email thread a little while back, one of my colleagues suggested that programming should be a required subject in high school. To survive in today's world, everyone should have a basic understanding of programming because technology is so essential to everyone's lives. His friend retorted that, "Since shoes are also essential, high school students should take a mandatory year of cobbling."

Read More »

Apr 09 2013

Comments

Ben Vandgrift - Podcast Episode 028

cover art

Pretty much every conversation I've ever had with Ben Vandgrift has involved him saying something profound or insightful. I think this episode of the podcast was no exception, as we talked about Ben's experience as a relative newcomer to Clojure, and he had some constructive (and slightly painful) feedback for the Clojure community. We also got to talk about (inc Relevance), the quarterly gathering of Relevance employees, which Ben spearheads. I hope you enjoy the show!

Read More »

Apr 04 2013

Comments

Abstraction or Leverage?

A colleague of mine recently accused me of speaking too precisely. He was giving me a bit of grief, but I took it as a compliment. In this industry, we torture words until they give up all meaning. Keep your eyes open for the next time you read the word "significant." Does it actually refer to something that signifies? Or is it just salt, sprinkled in for flavor?

One of the words I see overused is "abstraction." I hear developers use it when they talk about the DRY principle and reducing common code. Reducing common code to eliminate repetition improves your leverage, but not necessarily abstraction. Leverage allows you to make a change in one place that affects all uses.

As an example, think about one of my favorite code-eating patterns in OO: the Null Object pattern. It concentrates a lot of "if not null" code into one place, which means you can change the logic in that one place. That's leverage: you can make a small changes with large-scale effects.

"Extract method" and "extract superclass" transformations fall into this camp. They add leverage, which is useful.

By drawing this distinction, I'm not criticizing leverage. It's useful and important. But there's an additional thing to consider. When you extract common code to gain leverage, you have little ability to ignore details. Indeed, the specific case is often tightly coupled to the implementation of the general case.

(I will admit that when I distill code down this way, it sometimes clears away the cobwebs enough to see the abstraction that was lurking in the corner. That's always a lovely feeling. So leverage can lead to abstraction.)

The essence of abstraction is cleaving. Abstraction separates a manifold thing into independent concepts. By separating the concepts, it also allows us to see connections between things that share one of those concepts. So abstraction also illuminates connections between things.

For example, consider Clojure's seq (pronounced "seek") abstraction. It calls out the concept "Sequential Iteration," thus bringing together everything that acts like an ordered sequence that can be walked. At the same time, it separates that concept from others. For a vector, seq separates that concept of "Sequential Iteration" from the concepts of "Random Access," "Stacklike Object," and "Lazy Evaluation." For a string, seq separates "Sequential Iteration" from "Unicode Storage" and "Regex Matching." That's the part about separating a manifold thing. At the same time, "seq" shows us that there is an important characteristic shared across all those implementations.

In each case, the abstraction allows us to emphasize a particular aspect of the thing while disregarding the rest. This is the dual nature of abstraction: it obscures and reveals.

As it happens, good abstractions also provide leverage, because they separate concerns. I think this is why the words get conflated. All abstractions offer leverage, but not all leverage is an abstraction.

As you work on your designs, think about where you can split those conceptual atoms. What ideas can you separate? How can you then reconnect the shared concepts?

Apr 01 2013

Comments

Where to Find Relevancers: April Edition

Want to meet a Relevancer in person? Here's where you can find us during the month of April:

Philadelphia, PA 4/2-4/4
Emerging Technologies for the Enterprise
Speaking: Stuart Sierra: Clojure/ClojureScript: One Language to Rule the Web

St. Augustine, FL 4/4-4/5
Ancient City Ruby
Speaking: Russ Olsen: Insight, Intuition and Programming

Panama City Beach, FL 4/11-4/14
LessConf
Attending: Maggie Litton, Ben Vandgrift, Marc Phillips, Ryan Neufeld, Kevin Altman, Sam Umbach

Durham, NC 4/16
West End Ruby Hack Night @ Relevance HQ
Attending: Sam Umbach, Yoko Harada

Alexandria, VA 4/23 @ 6pm
ClojureDC Meetup Group
Speaking: Bobby Calderwood: Datomic

Durham, NC 4/25 @ 7pm
Triangle Clojure Meetup Group @ Relevance HQ
Pedestal Q&A Leader: Ryan Neufeld
Attending: Chris Redinger, Daemian Mack

Columbia, SC 4/25-4/26
ConvergeSE
Attending: Ben Vandgrift

Portland, OR 4/29-5/2
RailsConf
Speaking: Yoko Harada: Datomic, From Ruby, From Rails
Attending: Jamie Kite, Lake Denman

Popular Tags