Turkey Tacos and the Giant Smoking Crater

Here's a short experience from a project I was on long before joining Relevance. It might be interesting or educational.

It was October in Minnesota. Fall is a time for endings, and my consulting partnership had recently decided to separate. Amid the lengthening shadows, I found myself in need of a gig. Something relaxing, where I could put in my hours, pull myself back together, and restore my drained spirit. I joined a project that had a January launch date, right after the holiday freeze lifted. The team was already in "crunch time," and had been for six months when I joined. We were working from before dawn until after dark in a converted warehouse in Minnesota. To maximize our productivity, management had lunch brought in every day from a rotating pool of local restaurants. This wasn't as nice as it sounds. There's simply no way to make lunch for 300 people palatable. Turkey tacos every Tuesday? Yikes.

Crunch time is standard practice in our industry. Everyone knows it exacts a toll on the team, but (almost) everyone does it anyway. It's that tantalizing burst of speed at the end of the project. Somehow, one extra day of work at the end seems much more valuable than it was back in the beginning. I've been through more than a few crunches in my career, and I have to confess that the big push to the finish line can be exhilarating. As perverse as it sounds, humans bond through adversity more than through success. So a hard push for the last couple of weeks of a long project can actually bring people together and create an even stronger sense of success when the project launches.

Six months is a long time to be crunching, though. And as Fall turned into Winter, it became clear that the January launch date was a complete fiction, despite all the green lights on the project managers' dashboards. Winter became Spring, and we continued in crunch time.

As you might imagine, morale suffered a bit as the release date kept receding into the distance. It seemed to slip one day for every day that we worked. The management team tried a bunch of different things to bolster our spirits. For one all-hands meeting, each subteam nominated a theme song to be the project anthem. All the songs were compiled into a CD that got pressed and distributed. You can get an idea of our collective state of mind from the tracks we chose: "A Million Miles Away," "I Wanna Be Sedated," "If I Had A Million Dollars," and so on. Someone later realized that this could be construed as a violation of a dozen copyrights, with a total liability greater than the project's budget, so nearly all the discs were recovered. Only a few survive. They are desirable collector's items.

In another amusing effort to boost morale, the management team brought in a movie-style popcorn machine. Who can be grumpy when the air is suffused with the delectable scent of coconut oil and popped corn?

The team needed a sense of progress, not novelty CDs or a popcorn machine. In fact, the whole thing kind of backfired, because we ended up cleaning the popcorn machine ourselves, on a daily rotation among the subteams. (Except, if I recall correctly, the business analysts, project managers, and architects were exempt.) Giving us dishes to wash certainly let every designer, programmer, and admin know what management thought of their value.

Progress came late that spring, when we finally passed through load testing, security testing, and functional testing. The critical bugs were fixed. The "final" release was cut, and we started the migration and cutover process. I'm sure you won't be surprised to learn that there were many emergency builds between the "final" release and the one that went live. Eventually, thanks to a team that lived in a co-opted conference room for three weeks, we reached the day that so many of us thought would never come: launch!

It was a disaster.

The system was totally unprepared for contact with the real world. It had architectural problems nobody had discovered. It would crash whenever Google tried to index it. There were critical problems in the code, configuration files, and the user interface. It needed immediate surgery, but there was nobody left to operate. The developers were gone. They were all contractors, eager to roll off this hellish project. Most of the project managers had been released back to the project management office (the dreaded PMO) to be assigned elsewhere. There was an application maintenance and support group assigned, but few of them had any knowledge of the system internals.

Almost everyone who could fix things headed for the hills the first chance they got. The patient was bleeding out on the table, and there wasn't a surgeon in the building.

The Dangers of Project Thinking

The first of the many ills on that project was that it was defined as a project at all! We were building an array of systems meant to be the future of this business. The end of the project is the true birth of the system. Like any newborn, a software-based system needs people to keep it alive and growing.

During the build phase of a product, people accumulate knowledge about how it works, how it's built, and most importantly why things are the way they are. Everyone makes thousands upon thousands of decisions. All of those decisions have to be made real, and the place they are made real is in the code. There are side effects, though: As you make decisions, you also learn how the product is built and why.

If your product team is staffed like most companies', you are part of a group of people attached to a line of business. The business defines a need, supplies a budget, and decides what is in scope for the project. The build team gets drawn from pools of people, mostly in IT. You'll need a project manager, some developers, some UI designers, and one or more database administrators. If you're lucky, some of them will have met each other before this project starts. If you're unlucky, each of them will still be supporting their last few projects. (They won't have any budget or time allocated, of course. Those projects are finished and their charge codes are closed out. It comes out of the time they bill to your project. But don't worry, you'll get that time back from the next project, after yours is already finished.) One developer joked that he was supporting ten years' worth of old projects. He said the only way to get rid of them would be to quit, so he did.

In a large company, each of these groups of people reports to someone different: project managers report to the corporate Project Management Office, developers report to a manager of application development, database administrators report to a DBA manager, and so on. At some point, the project ends, the budget is used, and the team goes back to their respective pools.

You might also be using contractors for extra capacity. Our project had hundreds. In that case, you know they will be temporary, either due to cost or policy concerns. Many companies have policies that limit how long a contractor can be engaged. (To avoid lawsuits about "perma-temps" and benefits.) One company I know had a limit of 18 months. They had a project that ran long and crossed that deadline, so they booted out the entire development team. In the last few months before launch, they brought in an entirely new crop of contractors who had no knowledge or context about the existing work! You can imagine how well that worked.

Once the original builders are gone, no support team in the world can tell which decisions were carefully thought out and which were quick, "good-enough" hacks. So the hacks will hang around for a long time. Worse yet, within the code, hacks tend to breed other hacks near them. The original team would know to beware of those hacks, but the code doesn't come with any markers to warn their successors where the quicksand is. The net result is what we affectionately call "bit rot." Each successive project will take longer and cost more, until you decide (or your development team tells you) it's time for a rewrite.

Limited Vision of a

Project

Project thinking is limited-term thinking. Projects optimize for a target delivery date and budget. That induces them to make choices that favor the date, but might not help your product after the project is done. A project team may decide to skip data purging to save a developer-week of effort. That's great for the project but incurs an ongoing operations cost, because someone has to purge data manually for the life of your product. Now, the development cost may turn out to be more or less than the total of future operations costs. Either way, you should make that decision based on how it affects your product's lifetime profitability. It shouldn't be a project manager's decision based on a delivery date.

This problem happens with business processes too. I can't tell you how many times I've seen people switching between multiple internal tools to do their jobs, usually with a spreadsheet open to keep track of the IDs or codes that the tools don't share. In every case, it's because the individual tools were built one at a time without regard to the overall workflow. None of the projects could accept the extra scope of integrating the tools because it would blow the schedule. The result: saving some development cost once, but paying the price in efficiency forever.

A Better

Way

Instead of project thinking, exercise "product thinking." Design the organization together with the software so you can keep it alive and growing after 1.0 goes out the door.

As much as possible, try to create a self-sufficient product team. That means the team should be able to handle routine daily activities internally: analyzing customer metrics, planning new features, implementing them, releasing them, and measuring their effect on key performance indicators. The team should also handle those ugly non-routine daily activities, too: fixing bugs, restoring service when things crash… all that stuff we lump into "operations."

This dedicated product team gets a bunch of advantages. First, they will communicate better. Second, they will be more motivated to do good work. Temporary team members assigned to a project may be loyal to the project, but team members permanently attached to the product will be loyal to the product. These two goals are often in conflict.

The third big advantage of the self-sufficient team is dramatically reduced cycle time. With a temporary team, everything becomes a project. Want a set of bugs fixed? Make a list, define the scope, get an estimate, and charter a project. Meanwhile, your customers either learn to work around the bugs or they give up on you and go somewhere else. With a dedicated team, you can deliver continuously and fix a bug every day. All you need to do is put up a page with the "hit list" of top bugs and give your team some slack, and they'll fix the bugs out of their own sense of pride.

I won't claim that product organization would have cured all the ills of my "Turkey Tacos" project. I'm quite sure, however, that people would have done better work if they knew they would live with that system for the next few years. They would have built a living ecosystem instead of a bunch of dead artifacts.

Get In Touch