How to roll out flawless software

A somewhat radical new way of thinking about software projects and how to manage customers through them.

Introduction
I have been doing software-development work for my entire career. That, to my horror, is somewhere close to 12 years now, which is starting to seem like a downright respectable length of time.

In this time, I have had the often dubious privilege of working on hundreds of software programming projects, some small, some big. And many of them have been excruciatingly hard — the actual coding being the easy part.

I want to suggest that there is a way to roll out software projects that can work, but which requires rethinking expectations and accepting some pain. I call it the Two-Hills-and-a-Valley Software Development Cycle.

Why is software development so hard?
The literature abounds with answer to this question, but I will share my own personal view anyway since my fiancée asked me this same question after my particularly gruelling week in the trenches.

Software is, according to the legendary The Mythical Man-Month by Fred Brooks, like two monsters wrestling in a tar pit. As they fight, they both invariably sink. The monsters, according to Brooks, are the programmer and the customer, fighting each other for features, time, money, whatever.

The influential extreme-programming methodology (by Kent Beck) argues that it’s hard because the customer isn’t involved enough in the development process, creating all kinds of communication gaps.

I think, however, that it’s hard for quite different reasons. In fact, I think it has little to do with the development process itself.

In any kind of contracted project, there will be gaps between what the customer wants and the contractor makes. It doesn’t matter if it’s a house, a wedding or a piece of software; what’s in one person’s head can never really be 100% communicated to someone else. In addition, the desired result often isn’t even in the customer’s head. It’s in their future head once the result has been delivered and they don’t like what they see.

But this is normal, and I believe both customer and contractor factor this into their expectations and thinking already.

There are two important things that make software rather unique, however: first, much of the complexity is hidden from view. And second, it is highly sensitive to changes in variables.

A house can still break after it has been built. It cracks and weathers, and bits of it fall off that weren’t installed properly in the first place. But, in most cases, the stuff that breaks is obvious. One day you have a wall, the next day it’s a pile of bricks. And the variables are manageable. Maybe there’s a nasty hailstorm once in a while, but most of the time the house exists in the same environment, and is used in the same way.

Software, not so much. Its myriad inner complexities are bafflingly obscure, even to other expert programmers. The visible consequences of bugs often make no sense and don’t point to the root cause. And to make matters worse, it’s almost impossible to imagine in advance all the different ways a piece of software will be used. In the house analogy, one day there’s a hailstorm, the next day it’s submerged in custard.

The upshot of all of this is that software is not so much hard to make, and where it is, we expect it. What is hard, unforeseen and crippling is keeping the stuff running.

Hills and Valleys
Making software is a steady climb. There will be setbacks, delays and budget overruns, but in the end, a system is delivered. And battle-scarred but jubilant, the team celebrates a job completed.

That’s the first hill — maybe it’s even a mountain, but it’s been scaled. And clinging to each other on the summit, the brave software soldiers feel a just sense of pride and accomplishment.

The next morning, and each morning after that, is a downward plummet. The next day is the valley.

The valley is not a fun place. It is where the once-cheerful customer is now facing endless complaints from his users about his shiny new system; where bug after bug is appearing, and with each bug fix, more crawl out; where developers don’t want to hear about it any more; and where real money is being lost with each moment that the system isn’t functioning correctly.

It is the time where blame is apportioned and the real shit hits the fan.

How does one avoid the valley? How does one make sure a system launch is carried over into a fabulous running system?

Simple answer: you don’t. In the Two-Hills-and-a-Valley Model, the valley follows the hill as surely as day follows night and exhaustion follows a 15-hour session of Medal of Honour.

What I’m arguing is that a kind of disastrous collapse is an intrinsic part of the software-development cycle — one about which the customer needs to be forewarned, and the team needs to be prepared. And if the customer refuses to accept it, the team needs to be even more prepared, because it will happen anyway.

Life in the valley
A typical valley day consists of the following kind of thing: you arrive at work in the morning, already having received three phone calls from your customer. He’s irate. His system is down. It’s throwing a different error every time someone tries to log in. What’s worse, his clients are ready to move to his competitor because its system apparently works much better than his.

You stumble into the office and rush directly to the QA team. “What the hell is going on?” you yell. “The system is totally fucked,” they say. “What do you mean it’s totally fucked?” you ask. “It was working perfectly yesterday?” Faces are blank.

Next you rush off to the development team. “Guys, the system is broken,” you say. “Works fine on my machine,” they say. “Well, it’s broken on the live server and the client is freaking out. Can someone please look into it?”

“We’re busy,” the developers say. “Tell the client we’ll look at it when we have time. We told everyone things might go wrong because of the last-minute changes that we included.”

Etc. You get the picture. These are not happy days.

And they are inevitable. Well, the first bit is. The rest of it is where the opportunity to do something different exists.

The valley of opportunity
There is a single opportunity in the valley: that is, the way to start climbing out of it. And that is to service and support the system to death. Take the flak, bear up and fix it as it breaks.

Of course, of critical importance here is not to enter into crisis patch-and-fix mode. In fact, if I were to write two golden rules for valley times, they would be:
1. Take every call from the client promptly, write everything down, confirm that you have understood the problem, give it a reference number and keep the list updated (alright, that’s not one thing).
2. Release fixes in a slow, ordered manner, never skipping a full retest for all upgrades, no matter the pressure.

Conclusion
Beyond the valley, you’ll be glad to hear, is another hill. One climbed far more slowly, and never quite as high as the first. But over time you will regain your customer’s trust, and you will have a system that everyone is pretty happy with, that works and (unlike before) runs largely bug-free.

The current software-development lifecycles — even something as novel as extreme programming — focus on the wrong stuff. It’s not about the first journey or the destination. It’s about what you do once you get there.