Blog

Jul 10 2013

Comments

Rich Hickey and core.async - Podcast Episode 035

cover art

Download this podcast episode.

Listeners of the show need no introduction to Rich Hickey, even if he had not already been our guest twice. In this episode, we talk to him about his latest brain gem, the core.async library. It was an illuminating discussion, and I thank Rich for taking the time to explain the concepts to me!

Read More »

Jul 08 2013

Comments

Shopping in Infinite Space

Long hours driving the open road grant me that rare luxury: time to think about everything and nothing. I love freely associating ideas as the miles roll by. Over a brief vacation this past week, a few ideas collided in my head (somewhere near Story City, IA, if I recall correctly) and stuck together in an amusing way. I'd like to share that with you now---let's have some fun.

I was thinking about mutability, immutability, and systems design. I love immutability in the small. Pure functions over immutable arguments are so much easier to work with than squirmy, stateful goop.

When we use a wider lens, though, it's much less clear how to build full-scale systems that have the same nice properties. For instance, think about a retail system. A shopping cart is just inherently mutable, isn't it? You add stuff, remove stuff, change quantities---those are all mutations, right?

Read More »

Jul 03 2013

Comments

Turkey Tacos and the Giant Smoking Crater

Here's a short experience from a project I was on long before joining Relevance. It might be interesting or educational.

It was October in Minnesota. Fall is a time for endings, and my consulting partnership had recently decided to separate. Amid the lengthening shadows, I found myself in need of a gig. Something relaxing, where I could put in my hours, pull myself back together, and restore my drained spirit. I joined a project that had a January launch date, right after the holiday freeze lifted. The team was already in "crunch time," and had been for six months when I joined. We were working from before dawn until after dark in a converted warehouse in Minnesota. To maximize our productivity, management had lunch brought in every day from a rotating pool of local restaurants. This wasn't as nice as it sounds. There's simply no way to make lunch for 300 people palatable. Turkey tacos every Tuesday? Yikes.

Crunch time is standard practice in our industry. Everyone knows it exacts a toll on the team, but (almost) everyone does it anyway. It's that tantalizing burst of speed at the end of the project. Somehow, one extra day of work at the end seems much more valuable than it was back in the beginning. I've been through more than a few crunches in my career, and I have to confess that the big push to the finish line can be exhilarating. As perverse as it sounds, humans bond through adversity more than through success. So a hard push for the last couple of weeks of a long project can actually bring people together and create an even stronger sense of success when the project launches.

Six months is a long time to be crunching, though. And as Fall turned into Winter, it became clear that the January launch date was a complete fiction, despite all the green lights on the project managers' dashboards. Winter became Spring, and we continued in crunch time.

As you might imagine, morale suffered a bit as the release date kept receding into the distance. It seemed to slip one day for every day that we worked. The management team tried a bunch of different things to bolster our spirits. For one all-hands meeting, each subteam nominated a theme song to be the project anthem. All the songs were compiled into a CD that got pressed and distributed. You can get an idea of our collective state of mind from the tracks we chose: "A Million Miles Away," "I Wanna Be Sedated," "If I Had A Million Dollars," and so on. Someone later realized that this could be construed as a violation of a dozen copyrights, with a total liability greater than the project's budget, so nearly all the discs were recovered. Only a few survive. They are desirable collector's items.

In another amusing effort to boost morale, the management team brought in a movie-style popcorn machine. Who can be grumpy when the air is suffused with the delectable scent of coconut oil and popped corn?

The team needed a sense of progress, not novelty CDs or a popcorn machine. In fact, the whole thing kind of backfired, because we ended up cleaning the popcorn machine ourselves, on a daily rotation among the subteams. (Except, if I recall correctly, the business analysts, project managers, and architects were exempt.) Giving us dishes to wash certainly let every designer, programmer, and admin know what management thought of their value.

Progress came late that spring, when we finally passed through load testing, security testing, and functional testing. The critical bugs were fixed. The "final" release was cut, and we started the migration and cutover process. I'm sure you won't be surprised to learn that there were many emergency builds between the "final" release and the one that went live. Eventually, thanks to a team that lived in a co-opted conference room for three weeks, we reached the day that so many of us thought would never come: launch!

It was a disaster.

The system was totally unprepared for contact with the real world. It had architectural problems nobody had discovered. It would crash whenever Google tried to index it. There were critical problems in the code, configuration files, and the user interface. It needed immediate surgery, but there was nobody left to operate. The developers were gone. They were all contractors, eager to roll off this hellish project. Most of the project managers had been released back to the project management office (the dreaded PMO) to be assigned elsewhere. There was an application maintenance and support group assigned, but few of them had any knowledge of the system internals.

Almost everyone who could fix things headed for the hills the first chance they got. The patient was bleeding out on the table, and there wasn't a surgeon in the building.

The Dangers of Project Thinking

The first of the many ills on that project was that it was defined as a project at all! We were building an array of systems meant to be the future of this business. The end of the project is the true birth of the system. Like any newborn, a software-based system needs people to keep it alive and growing.

During the build phase of a product, people accumulate knowledge about how it works, how it's built, and most importantly why things are the way they are. Everyone makes thousands upon thousands of decisions. All of those decisions have to be made real, and the place they are made real is in the code. There are side effects, though: As you make decisions, you also learn how the product is built and why.

If your product team is staffed like most companies', you are part of a group of people attached to a line of business. The business defines a need, supplies a budget, and decides what is in scope for the project. The build team gets drawn from pools of people, mostly in IT. You'll need a project manager, some developers, some UI designers, and one or more database administrators. If you're lucky, some of them will have met each other before this project starts. If you're unlucky, each of them will still be supporting their last few projects. (They won't have any budget or time allocated, of course. Those projects are finished and their charge codes are closed out. It comes out of the time they bill to your project. But don't worry, you'll get that time back from the next project, after yours is already finished.) One developer joked that he was supporting ten years' worth of old projects. He said the only way to get rid of them would be to quit, so he did.

In a large company, each of these groups of people reports to someone different: project managers report to the corporate Project Management Office, developers report to a manager of application development, database administrators report to a DBA manager, and so on. At some point, the project ends, the budget is used, and the team goes back to their respective pools.

You might also be using contractors for extra capacity. Our project had hundreds. In that case, you know they will be temporary, either due to cost or policy concerns. Many companies have policies that limit how long a contractor can be engaged. (To avoid lawsuits about "perma-temps" and benefits.) One company I know had a limit of 18 months. They had a project that ran long and crossed that deadline, so they booted out the entire development team. In the last few months before launch, they brought in an entirely new crop of contractors who had no knowledge or context about the existing work! You can imagine how well that worked.

Once the original builders are gone, no support team in the world can tell which decisions were carefully thought out and which were quick, "good-enough" hacks. So the hacks will hang around for a long time. Worse yet, within the code, hacks tend to breed other hacks near them. The original team would know to beware of those hacks, but the code doesn't come with any markers to warn their successors where the quicksand is. The net result is what we affectionately call "bit rot." Each successive project will take longer and cost more, until you decide (or your development team tells you) it's time for a rewrite.

Limited Vision of a Project

Project thinking is limited-term thinking. Projects optimize for a target delivery date and budget. That induces them to make choices that favor the date, but might not help your product after the project is done. A project team may decide to skip data purging to save a developer-week of effort. That's great for the project but incurs an ongoing operations cost, because someone has to purge data manually for the life of your product. Now, the development cost may turn out to be more or less than the total of future operations costs. Either way, you should make that decision based on how it affects your product's lifetime profitability. It shouldn't be a project manager's decision based on a delivery date.

This problem happens with business processes too. I can't tell you how many times I've seen people switching between multiple internal tools to do their jobs, usually with a spreadsheet open to keep track of the IDs or codes that the tools don't share. In every case, it's because the individual tools were built one at a time without regard to the overall workflow. None of the projects could accept the extra scope of integrating the tools because it would blow the schedule. The result: saving some development cost once, but paying the price in efficiency forever.

A Better Way

Instead of project thinking, exercise "product thinking." Design the organization together with the software so you can keep it alive and growing after 1.0 goes out the door.

As much as possible, try to create a self-sufficient product team. That means the team should be able to handle routine daily activities internally: analyzing customer metrics, planning new features, implementing them, releasing them, and measuring their effect on key performance indicators. The team should also handle those ugly non-routine daily activities, too: fixing bugs, restoring service when things crash... all that stuff we lump into "operations."

This dedicated product team gets a bunch of advantages. First, they will communicate better. Second, they will be more motivated to do good work. Temporary team members assigned to a project may be loyal to the project, but team members permanently attached to the product will be loyal to the product. These two goals are often in conflict.

The third big advantage of the self-sufficient team is dramatically reduced cycle time. With a temporary team, everything becomes a project. Want a set of bugs fixed? Make a list, define the scope, get an estimate, and charter a project. Meanwhile, your customers either learn to work around the bugs or they give up on you and go somewhere else. With a dedicated team, you can deliver continuously and fix a bug every day. All you need to do is put up a page with the "hit list" of top bugs and give your team some slack, and they'll fix the bugs out of their own sense of pride.

I won't claim that product organization would have cured all the ills of my "Turkey Tacos" project. I'm quite sure, however, that people would have done better work if they knew they would live with that system for the next few years. They would have built a living ecosystem instead of a bunch of dead artifacts.

Jul 02 2013

Comments

A Decade

2003; the Space Shuttle Columbia takes its last, tragic flight, and Pioneer 10 sends its final signal. SARS is a thing we are told to fear; the Iraq war begins. Terry Wallis wakes up after 19 years in a coma. The Supreme Court decides that the state has no pertinent interest in the bedroom. China sends its first manned mission to space. The Concorde takes its final flight. Apple releases Panther and is still 10 years away from running out of cats. June Carter and Johnny Cash make it to the far side banks of Jordan.

And Relevance is incorporated. ;)

It took a full two years for us to decide what we wanted Relevance to be - but ever since then, it has been an amazing ride. We moved into our first office in 2006, and are now on our fourth. We've worked on some incredible projects with some incredible clients. We've been a part of several major upheavals in the technical landscape, and we intend to keep up that trend. We had some crazy ideas along the way: RunCodeRun and the Venture Dojo, to name two. We'd like to start by thanking our many clients over those 10 years, the companies and individuals who trusted your vision and your needs to us. You are the lifeblood of our business, and we would have had no purpose without you.

And then there are the people who worked for and with us. For anything that lasts 10 years, there will be great times and bad ones; there will be victories and defeats. And for any list of relationships, some will have been flawlessly excellent and some frayed. The following is a list of everybody we can think of that helped us make it 10 years. Some are or were employees; some contractors. Some are companies we had partnerships with, and some are mentors who helped us along the way. Each and every person or group on this list gave of their time, energy, passion, skill, intelligence and heart, and Stu and Justin want every single person listed here to know how humble and thankful we feel knowing that you did. We could not, quite literally, have made it this long without each of you. From the very bottom of our hearts, thank you.

And I'm sure we missed some. If we did, I am sorry (and if you poke me on Twitter, I'll add you ;). Here's to another 10 years, and another long list of relationships to celebrate.

Jul 01 2013

Comments

Where to Find Relevancers: July Edition

Want to meet a Relevancer in person? Here's where you can find us during the month of July:

Chicago, IL 7/8-7/10
LambdaJam
Conference Organizer: Alex Miller
Speaking: Stuart Sierra: Data, Visibility & Abstraction

Madison, WI
UXMad
Attending: Michael Parenteau

Durham, NC 7/16
West End Ruby Hack Night @ Relevance HQ
Attending: Yoko Harada

Denver, CO 7/16-7/19
ÜberConf
Speaking: Stuart Halloway:
Sessions include: Intro to Clojure, Generative Testing, Pure Fun, Simulation Testing with Simulant, edn and Fressian: Flexible Languages for Data

Jun 25 2013

Comments

Michael Nygard - Podcast Episode 034

cover art

Download this podcast episode.

One of the most popular posts on our blog in recent history is Michael Nygard's post "Three Book Ideas". The enthusiastic reaction to that post combined with the many interesting conversations I've had with Mike in the past told me it was past time to have Mike on the show. We talked about his "Three Book Ideas" post, physics, the nature of relational data, and a plethora of other interesting topics. I enjoyed it, and I thank him for spending the time with me! I hope you'll enjoy it, too.

While I've got your attention, I'd also like to mention a few things:

Read More »

Jun 12 2013

Comments

Kurt Zimmer of Room Key - Podcast Episode 033

cover art

Download this podcast episode.

I've said it before: one of the few downsides to working at Relevance is getting to hear all sorts of cool customer success stories that we can't talk about. Which is why Justin and I were thrilled to record an episode with Kurt Zimmer from Room Key, one of our clients and an enthusiastic user of both Clojure and Datomic. We talked about Relevance's relationship with Room Key, and about Kurt's experience successfully selling cutting-edge technologies into extremely large, conservative organizations.

Kurt is a sharp guy, and it was a pleasure to have him as a guest.

While I've got your attention, I'd also like to mention a few things:

Read More »

Jun 07 2013

Comments

The Rule of Three

Every solution comes with costs and consequences. If you get stuck on the first thing that comes to mind, you aren't necessarily choosing the solution with the most desired balance of costs and consequences.

I use something I call "The Rule of Three."

Read More »

Jun 04 2013

Comments

My Clojure Workflow, Reloaded

This is a companion post to my discussion with Craig Andera on Relevance Podcast Episode 32 and my Clojure/West talk Clojure in the Large. I've talked about various bits and pieces of this workflow at other times, too, but I'll try to bring it all together here in the hopes that others will find it useful.

One of the great pleasures of working with a dynamic language is being able to build a system while simultaneously interacting with it. To make this possible, first you need the ability to redefine parts of the program while it is running: Clojure provides this capability admirably. However, some aspects of Clojure's runtime are not quite as late-binding as one might wish for interactive development. For example, the effect of a changed macro definition will not be seen until code which uses the macro has been recompiled. Changes to methods of a defrecord or deftype will not have any effect on existing instances of that type.

The facilities that Clojure provides for loading code from files are not sufficient to deal with these issues. I wrote the second version of tools.namespace to make a "smarter" require that recognizes dependencies between namespaces and reloads them appropriately.

But tools.namespace is only part of the story. To really get the benefit of interactive development, I want to ensure that the version of the application I am currently interacting with is congruent with the source files I'm editing. That means not only that the application must be running the most up-to-date version of the code, but also that any state in the application was produced by that same code. It is dangerously easy, when changing and reloading code at the REPL, to get an application into a state which could not have been reached by the code it is currently running.

Therefore, after every significant code change, I want to restart the application from scratch. But I don't want to restart the JVM and reload all my Clojure code in order to do it: that takes too long and is too disruptive to my workflow. Instead, I want to design my application in such a way that I can quickly shut it down, discard any transient state it might have built up, start it again, and return to a similar state. And when I say quickly, I mean that the whole process should take less than a second.

To achieve this goal, I make the application itself into a transient object. Instead of the application being a singleton tied to a JVM process, I write code to construct instances of my application, possibly many of them within one JVM. Each time I make a change, I discard the old instance and construct a new one. The technique is similar to dealing with virtual machines in a cloud environment: rather than try to transition a VM from an old state to a new state, we simply discard the old one and spin up a new one.

Designing applications this way requires discipline. First and foremost, all state must be local. Any global state, anywhere, breaks the whole model. Second, all resources acquired by the application instance must be carefully managed so that they can be released when the instance is destroyed.

Enough talk. Here's how it works.

Read More »

Jun 03 2013

Comments

Three Book Ideas

I've been gestating three different books in my head for a while. You would expect that having three books in your head is great for productivity, but it seems that the opposite is true. It's hard to decide which one to work on, so I get vapor-locked by indecision and work on none.

With a deep sigh, it's time for me to admit that none of these is going to emerge like Athena, fully-formed, from my head. To my chagrin, it looks like I actually need to sit down and do the work of writing one of them. But which one?

In the spirit of garnering feedback early and often, I'd like your input. Leave a comment, please. Which, if any, of these three books would you most want to read?

Blueprint for a Web Company

A cross between Chasing the Rabbit and The Art of Scalability. The best companies today know how to bring together a handful of ideas across several dimensions. Each of these ideas gives an individual a bigger lever to impact the company. Instead of focusing on risk avoidance, these companies make many survivable bets.

This book would show how to combine:

  • Product-oriented organizational structure
  • Data-driven decision making
  • Decoupled software architecture
  • Cloud computing and cloud-native application architecture
  • Lean software development
  • DevOps

Doing any of these things will deliver benefits. Do them all, and it's like a turbo boost for your company.

This would be a full-length book.

The Hickey/Halloway Paradigm

Clojure and Datomic express a distinct set of ideas about software design. Rich Hickey and Stu Halloway have articulated these in various presentations. As we have worked with Clojure, Datomic, and ClojureScript, and as we have built Pedestal, many of us at Relevance have adopted this paradigm for programming.

This book would articulate the principles of this style: an explicit approach to time and context, the separation of identity from state, what makes a good abstraction, and how to structure real-world systems from functional, explicit, evident components.

This would be a medium-length ebook.

The Truth About Data

E.W. Dijkstra famously said "It is practically impossible to teach good programming to students that have had a prior exposure to BASIC: as potential programmers they are mentally mutilated beyond hope of regeneration." I would make a similar case about relational database structures. We get such early training in normalization, entities, and relations that we are tempted to think of these as the "natural" form of data. Absolutely not.

Normalized databases as we practice them are not essential to the relational model, and the relational model itself is an arbitrary set of constraints applied to a much richer fundamental view of data.

In "Data and Reality," William Kent described many ways in which our models are arbitrary. "Arbitrary" not in the sense of capricious, but in the sense of representing choices from a set of potential models with equal validity.

There is another way to understand data. We can apply some very general abstractions to apprehend a data set as a kind of Platonic ideal, from which we project our actual data sets. In this view, entire databases look like points in a many-dimensional space and data models are just projections of this space into reified memory, storage, and index structures. Queries are further projections. Transactions move the database from one point in this space to another.

This would be a medium-length ebook, and may be a bit challenging.

Need Your Voice

Please leave a comment below. Which of these, if any, sounds interesting to you?

Popular Tags