Not Like Civil Engineering

Many people who do not write code think that software engineering is akin to civil engineering but is this an accurate metaphor? Software is built in virtual space and the rules of virtual space are different than the rules of physical space. In civil engineering the waterfall method works well; they plan, do and then review. But in software development these distinctions aren’t so clear.

If you’re going to build a bridge across a river then the first thing you do is you bring in the surveyors to find the bedrock at the banks of the river. If they discover that the bedrock is 30 feet underground then guess where it is when they bring in the construction crew six months later? Yep, you guessed it, at 30 feet down. We can be assured that these things won’t change much. But in software development our requirements and specifications change all the time because we are constantly learning and so are our stakeholders. In physical construction the crew works from a blueprint. And the job of the hardhat is compliance.

There was a fascinating show that I sometimes caught on the Discovery Channel called Engineering Disasters that examines what happened that caused buildings or bridges to collapse. Invariably someone on the construction crew decided to take shortcuts or vary from the blueprint to save time or money.

The architects who create blueprints are aware of the tensile strength of materials and their design captures these things so that their structures will be robust. In software development this level of up front articulation is often not possible because specifications are written in English and human language is inherently ambiguous.

If I give the same blueprint for a building to 10 different construction crews and ask them to build it, what I will get are 10 identical structures. The appearance of each structure might be slightly different, for example they might’ve painted the structure a different color, but the internal structure of the building will be virtually identical.

If I give the same specification to 10 different development teams I will get vastly different programs. They may appear to behave the same way to the user but if I look at the code I’ll probably find very different implementations.

Civil engineering has a history of thousands of years behind it but software engineering has only about half a century. If I want to build an arch, for example, I know there are only two ways to do it: the right way and the wrong way. The Greeks figured it out and while there might be different variations for different arches, the fundamentals of an arch, the things that make an arch an arch, remain the same.

In software engineering we are still figuring it all out. And this is the good news! I don’t want to work in an industry where it’s all been decided and all I have to do is blindly comply.

A blueprint might be a single piece of paper but there’s a great deal of information behind it. Civil engineering has something called The Civil Engineering Handbook. This book is used to handle and resolve virtually any situation that comes up when building a physical structure.

There is no comparable handbook for software engineering and there won’t be one, at least not in the near future. What we do is to manage complexity and the rules around that are fundamentally different than the rules for assembly of physical structures.

To get a sense for this let’s look at some examples. Let’s say I manage the construction of desks. And I get a big order that requires me to expand my business quickly; say a new customer wants 100,000 desks. What do I do? I would probably create a new assembly line and run it in parallel with my current assembly line. Lo and behold the desk building capability of my company has now doubled.

Now let’s say I discovered that I need to significantly increase the amount of software that is needed for a project. What do I do? Let’s say I hire more programmers and double my head count and what happens? In all likelihood my company’s productivity is now much less than it was before.

How can this be? With manufacturing I add more resources and I get more productivity but with software development when I add more resources I get less productivity.

The reason for this has to do with parallelism. In the desk manufacturing example we were able to start a new assembly lines which worked completely independently and in parallel with the first one and thus we were able to double productivity. However, when we double the amount of developers on the team we also increase the amount of communication required among all members and this often has the effect of slowing things down, not speeding them up.

Developers know this naturally because we deal with these issues on a daily basis but many managers who are not directly involved in the development process have a very difficult time understanding these principles.

Software engineering is not like civil engineering or any other form of engineering for that matter. Software engineering is not like anything.

When I was in college I read a book by Warner Heisenberg called Physics and Philosophy. If you’re unfamiliar with Dr. Heisenberg, he was the one who came up with what is known as the Uncertainty Principle and was a contemporary Albert Einstein. Like Einstein, he was not only a brilliant physicist but also a philosopher.

In his book, Physics and Philosophy Heisenberg says, and I’m paraphrasing, when we talk about the subatomic state we are inevitably led to the question “What is it like?” He says this is the wrong question to ask because it’s not like anything in our realm of experience. It is our nature as linguistic beings to ask this question when we encounter anything new. Often it serves us well but when we encounter something completely new it can delude us.

The subatomic state is completely new to us; we have no sensory experience of it and we can only transcribe aspects of its behavior mathematically. We now know the model that was proposed by Niels Bohr of electrons traveling around protons and neutrons; like planets traveling around the sun, is completely incorrect. We do not truly know what happens at the subatomic level; we can’t. We only know that there is a certain mathematical probability that an electron will be in a particular place at a particular time. This is what Heisenberg called the Uncertainty Principle and he says this is due to the fact that the observation of an electron’s position changes it velocity and therefore the probability of where it will be in the next moment.

There is also an Uncertainty Principle in software development, although it’s for a different reason. Many managers get upset with developers when we are not able to give them accurate estimates on when we will complete a task. We are not trying to be evasive; we simply do not know. Developers do not have a book that we can look the answer up in, unfortunately. Almost all software development is building things we have not built before and we don’t know how long it will take to do this because we have no references or experience to guide us.

Having upper management decide for us how long at task will take is not the solution. Trying to figure it out up front with limited information and imposing arbitrary deadlines is also not the solution. This is one of the main reasons we are failing in software development and our success rate is so dismally low.

So if we are willing to admit the truth–that we don’t know what we don’t know–then what do we do about it?

This is where agility comes in. Rather than trying to figure it all out up front we embark on a process of discovery with our customer. At the end of each iteration we ask the customer “Is that what you wanted?” and “What would you like next?” Asking these questions lets the customer become our partner and helps them see our progress.

I’ve worked with thousands of software developers in very intimate and interactive settings. Frankly I am a bit offended when I hear managers say that most developers are geeks and all that we want to do all day is geek out. The vast majority of developers that I know and have worked with want to do the right thing. Oftentimes we can’t because of the constraints that are placed upon us. What about you? Wouldn’t you want to do the right thing if you could? Of course you would!

So how do we help our customer understand what we are facing and at the same time give them as much of what they want as we are able? The answer is to make them our partners in the process and let us both discover what we can do for them together. This is what agility is all about.

If we have done a task before then it’s easy to estimate how long it will take to do it again. I’ve written about a dozen accounting systems in my day and the last thing I want to do now is write another one. Been there done that. I want to move on. Like most developers, I enjoy facing new challenges. This is great for me but I also recognize that this is not so great for customers. I’m sure managers and customers would like to meet a developer who has done exactly what they want many times before but this is just not feasible. We developers are mavericks and we like forging new territory. We love to learn and discover and invent. We don’t like doing the same thing over and over. If we did we would be in another profession.

And that’s what development is: invention. Development is not compliance or construction or following a blueprint to the letter, at least not all of it. Development is also discovery and dynamic and inventive and this is what most of us developers love about our profession.

Most people probably don’t have the stomach for it but we developers do and it is what drives us. So when managers get frustrated with us they have to understand that if they want something we’ve never done before, that is entirely and completely new, we may hesitate to tell them how long it will take us.

I work with someone who was formerly a cancer researcher. Is it reasonable to ask her to invent a new cure for cancer with no side effects, that can be produced cheaply and effectively, and to do it in the next five minutes? It’s an absurd expectation but sometimes these are the kind of expectations that are placed upon us.

I was teaching a class a few years ago outside a San Jose at a beautiful resort. The building next to my classroom was nearly finished being constructed and the crew was just finishing up the interior walls.

I asked my class what they thought would happen if the person who commissioned the building went up to the foreman and said, “You guys are doing a great job, really outstanding. I just have one small change for you; just move the building across the street.”

The foreman would probably laugh at him. Even though it’s just one change we know that it is a big one and it invalidates just about everything else that was done. But how many times has a manager or customer ask you such a question?

Sometimes a huge change in the behavior of the system can be done with minor modifications; after all software is soft. But not all changes can be done easily and those who didn’t write the code can have a hard time understanding the difference.

The flexibility of a system has a lot to do with what the programmer expected would need to change. But we all don’t have crystal balls and our job is not to anticipate every possible change because that leads to paralysis by analysis.

We do want to do the right thing but how can we when you don’t know what the right thing is. The answer to this question is actually quite simple: write code that is easier to change. There are no panaceas but there are principles and practices that we can follow to build software that is resilient to change and these techniques are things that every developer should be familiar with.

This is what my trainings teach. We cover a handful of key principles and practices that give our software flexibility and we cover them in a way that any developer can understand. This will not solve all problems in software development but it least it will give us a good start.

If you want to learn more please check out my Software Developer Essentials class at and see for yourself.