Software estimation for great fun and profit
Have you seen a well-oiled agile team go about their work? It is a pleasure to watch. They have a rhythm for software delivery ‐ the programmers work in a sustainable pace and they meet business goals. I think this is because agile project management does two things right: story points for relative sizing and predicting future performance by using past averages (velocity and volatility). These two methods account for the fact that individual estimates are fallible and work around it.
But not all teams are agile. One reason is that customers don't understand how unpredictable software development can be.
"when will you ship?" asks the customer.
The requirements are a bunch of implied expectations scattered over many email threads. Do we fall into this trap?
"That is a trick question", says you. "Imagine that you decide to take a hike on the coast from San Francisco to Los Angeles..."
But you are interrupted.
"Hold on a sec, I'm not hiking anywhere! Just give me a date, okay?"
Thus begins a long conversation, the first, in a series of long conversations.
The other reason for projects to not use common-sense project management is that they simply can't. They may be in constant flux to allow any rhythm to take hold and their timelines too short to even start calculating a reliable average velocity. Real world, things hairy.
We have to up our game to work well in those environments. Without the luxury of sensible agile practices, we will have to get better at estimating individual tasks. The challenge then is to deliver software predictably without a working process to do so. It is an interesting challenge, and can even come with a few benefits. In this post, I'll argue that done right, estimating individual tasks well is a great productivity booster and makes software development a bit more fun.
But first, we have to understand what we are up against.
Estimates are commitments to ourselves
Is there a wall-poster which says estimates are not commitments? We should get one. We should read it in the mornings and evenings, before food and after food. Because the first casualties of an estimate are us, the developers.
You are working on a story that you had estimated to be a 1-pointer. It typically takes you a day to finish a point, but this one is taking longer. Anxiety creeps in, but you don't recognize it yet: this story is not supposed to take this long, ergo, I'm too slow.
We become wedded to the estimate, an arbitrary number ‐ a guess made from incomplete data, and it worries us when reality doesn't match up. This is "anchoring".
a cognitive bias that describes the common human tendency to rely too heavily on the first piece of information offered (the "anchor") when making decisions.
Various studies have shown that anchoring is very difficult to avoid. For example, in one study students were given anchors that were obviously wrong. They were asked whether Mahatma Gandhi died before or after age 9, or before or after age 140. Clearly neither of these anchors can be correct, but the two groups still guessed significantly differently (average age of 50 vs. average age of 67)
Dan Ariely has run significant experiments on Anchoring, about which he talks in his engaging book Predictably Irrational.
The anxiety induced by anchoring is not an articulated thought. It is a nagging thread in the back of the mind chipping into the joy of our work.
But it is a silly game that the mind plays and we can eradicate it by simply being aware of it. That is both easy and hard. Easy because once recognized, the worry is disposed off easily, and hard because mindfulness require an awful lot of practice.
We could have produced an accurate estimate in the first place and avoided this anxiety. But before we get to that, we have to look at what estimates do to our customers.
Estimates are commitments to our customers
Let me illustrate. You are asked for an estimate on when a big feature can be released, and the following happens:
You think: The stories total to around 45 points. Our average weekly velocity is 16 and volatility is negligible. We still have some nice buffer if I get four weeks.
You say: if everything goes well, our projection tells us that it looks like we should be able to ship by end of December.
Your customer hears: bla bla bla, bla bla, end of December.
They are thinking: Fred, our customer, has been asking for this feature for a while now, and their account has been shaky of late. I'll promise him the feature a few days before Christmas. The team will work harder (they always do, longer and on weekends) when I tell them how dire the situation is.
What actually happens: The customer checks in on the progress by the end of the first week. You just knocked off a 3-pointer faster than you expected. Your confidence is palpable and it shows. Customer is relieved, things are going better than plan.
But Monday morning welcomes you with bad news. Your datatable widget's filtering option isn't working ‐ a hitherto unknown bug. It seems fixed in a later version, but to use it, you'll have to upgrade your version of Angular, which simply can't be done now. There is a StackOverflow question with the exact problem you face, but it has zero answers. Your luck has run out, but you are no timid programmer. You roll your sleeves up and jump into the library code and fix it. But the delay has a cascading effect, and you are able to ship only by January mid.
That didn't go well. The feature was not a show-stopper for Fred, he was simply keeping us on our toes. However, we did promise him a timeline, and optimistic promises have a way of spreading, so he in-turn promised his VP and it became a big deal. He had to break his promise, which isn't boding well for your business.
The moral of the story is that no matter how many ifs, buts and other caveats we insert, the customer will treat our estimates with an inappropriately high level of certainty.
There are a few things we could have improved on though. At least next time, we'll:
- Re-iterate that estimates are not commitments.
- Ask what our estimates are going to be used for. We should always know the full implication of the numbers we provide.
- Be careful to not amp up expectation in others because today's good weather does not translate into tomorrow's. But we respect our business owners and understand that they are capable of dealing with reality. We keep them aware of slips in the plan as they happen. We remain stoic in the face of both progress and setback.
- Not work on weekends. We worked on weekends, and it didn't help. It never does, in the long run. Don't work on weekends, and don't give optimistic estimates thinking you would.
- Relax and reflect. When the going gets tough, we forget to think. The stress creates adrenaline, and it clouds our judgement. The datatable filtering was not a show-stopper. It could have waited and we could have shipped on time, only if we had the presence of mind to consider the possibility.
Our estimates are going to come back and bite us ‐ that is the nature of the beast ‐ we ourselves become bound to it through anchoring and our customers become bound to it because they need a number to plan things around.
Casual estimates
Casual conversations are the minefield when it comes to dangerously wrong estimates:
"So, how hard is this feature? Is it like an hour's worth of work, or a few?"
And you say, "definitely not a quick one. A day, probably."
Stop right there and take note:
- The customer did your job for you and supplied an estimate in the question itself. In fact they started from the lowest possible number, putting pressure on you to anchor on it. That is negotiation. Estimation should never be a negotiation.
- We thought this was a casual question and gave a casual (aka wrong) answer. But when you get around to actually estimating this, telling the customer that the earlier estimate was wrong is akin to backtracking on a commitment. It wouldn't matter that it was a ballpark number, that detail will be forgotten by both the customer and you. Casual promises are promises nonetheless, so don't make them.
- Have conversations based on time instead of story points. Story points are a useful abstraction, try to stick to it if you can. (Why use story points instead of hours for estimating).
Now the same conversation would go like this:
"So, how hard is this feature? Is it like an hour's worth of work, or a few?"
And you would take a moment to reflect on it, and say:
"I am not able to say off the top of my head. I'll estimate this story, probably need an hour to do that, and get back to you."
This is obvious but overlooked ‐ the accuracy of an estimate is proportional to the time & effort you spend on it. The more you invest, the better the estimate. And no matter how quick and approximate an estimate your customer needs, you cannot afford to be lazy and put out a number that you don't have maximum confidence in.
Estimation is a high-risk activity and should be treated with the gravity it deserves. Casual estimation underplays the risk involved and the time it takes to come up with a reliable estimate. It makes us rush through and produce unreliable numbers which ultimately leaves everyone worse off.
We should explicitly allocate time for estimation. This conveys the time & cognitive effort required to do it and ensures that stakeholders think through what they really need before they ask us to spend time on estimating them.
Now the question becomes what do we do with all that time we allocated for estimation? To answer which, I have to first hate on fuzziness.
Fuzziness
Fuzziness is a knot in the stomach. I feel it when I estimate a story without thinking it through. It is my body telling me that I'm painting myself into a corner. But fuzzy is easy; we see a story and put a number to it without expending the mental energy to articulate it.
We say this story is a 1-pointer or a 2-pointer based on a hazy intuitive notion. This is fuzzy, it doesn't work well when you have a short-term delivery horizon. We tend to underestimate the task at hand and overestimate our abilities, and no matter how many times we have been wrong, we fool ourselves that we can do better this time. We are also people pleasers, we hate giving bad news and tend to default on the most optimistic numbers.
Getting better at estimation involves replacing fuzzy estimates with concrete data. It is hard to defend estimates made based on gut feeling, but data can speak for itself. Data doesn't people-please ‐ it gives us the conviction needed to communicate well and set the right expectations with our customers.
We defeat fuzziness by concretizing, aka, planning a task to its actionable items.
Plan, don't estimate.
We have to know what we are estimating to be able to estimate it. Sounds obvious? It is not. We often don't know the specifics of a story when we estimate it; we simply do a high-level sizing and throw a relative number. We keep aside the figuring out of what needs to be done for the later implementation stage.
The reason we default to such fuzzy estimation has a lot to do with the economics of energy of the prefrontal cortex. This is the part of our brains that is responsible for decision making, prioritization and deliberate thought.
The prefrontal cortex is the biological seat of your conscious interactions with the world. It’s the part of your brain central to thinking things through, instead of being on “autopilot” as you go about your life.
The prefrontal cortex holds the contents of your mind at any one point. It’s where we hold thoughts that are not being generated from external sources or from the senses. We ourselves are generating them.
Conscious mental activities chew up metabolic resources, the fuel in your blood, significantly faster than automatic brain functions such as keeping your heart beating or your lungs breathing. The stage (prefrontal cortex) requires a lot of energy to function. It’s as if the lights are a long way back from the stage, so you need a lot of them, all on full, to see the actors. To make matters worse, the power to light the stage is a limited resource, decreasing as you use it, a bit like a set of batteries that constantly need recharging.
Deliberate thinking is a formidable productivity booster (duh!). But generating thought is energy intensive. So we resist it and rely on gut feeling, which is simply the brain cruising on auto-pilot. It is not an effective strategy. Here, look at what some upfront planning did to Rachel Aaron, a writer (emphasis hers):
Looking back, it was so simple I feel stupid for not thinking of it sooner. If you want to write faster, the first step is to know what you're writing before you write it. I'm not even talking about macro plot stuff, I mean working out the back and forth exchanges of an argument between characters, blocking out fights, writing up fast descriptions. Writing this stuff out in words you actually want other people to read, especially if you're making everything up as you go along, takes FOREVER. It's horribly inefficient and when you get yourself in a dead end, you end up trashing hundreds, sometimes thousands of words to get out. But jotting it down on a pad? Takes no time at all. If the scene you're sketching out starts to go the wrong way, you see it immediately, and all you have to do is cross out the parts that went sour and start again at the beginning. That's it. No words lost, no time wasted. It was god damn beautiful.
Every writing session after this realization, I dedicated five minutes (sometimes more, never less) and wrote out a quick description of what I was going to write. Sometimes it wasn't even a paragraph, just a list of this happens then this then this. This simple change, these five stupid minutes, boosted my wordcount enormously. I went from writing 2k a day to writing 5k a day within a week without increasing my 5 hour writing block. Some days I even finished early.
She managed to increase her daily output from 2000 words to 10,000 words a day, mostly thanks to planning. Programming is also in its basic form, just writing. There are many lessons to be learnt from that craft, and this one is a case in point. Instead of wallowing in the code for hours ‐ going where the code takes us, ending up in blind alleys and backtracking, and getting lost in the trees and losing sight of the forest, a little bit of thought & planning can do wonders for our days as programmers.
The earlier we plan, the better. In fact, going into as much detail of a story as we possibly can before we estimate is the simplest way to increase the accuracy of our estimates. It will also bring out any hidden assumptions before we give out invalid estimates, or worse still, get neck deep into implementing the wrong thing.
Implementation becomes a breeze once we have a solid plan to follow. It is a beautiful day of writing code when that happens.
Estimate as a by-product of a plan
So what has planning got to do with estimation? It obviously makes for a more enjoyable programming experience. But it also brings our estimates to ninja precision. That is the whole point of this post.
Estimate is a by-product of a plan. It can't help but jump out at us when we look at a well thought out, written plan. At a sufficiently fine level of granularity, we would have the very implementation steps in the plan, and to come up with an accurate estimate, all we need to answer is how hard is it to convert this into code.
Drilling into this level of granularity might not be practical in many large agile environments. There will be too many people and stories for us to go into depth for each of them. But remember we are not talking about those environments. They have their own strategies which we cannot afford to have.
However, it is easy to make mistakes even in the relative sizing of stories as is done in a pure agile practice. Some of the stories turn out to be much larger than we anticipated. That is fine, the process accounts for such errors. But the feedback that the story was not estimated correctly is a lot more useful when we have a plan backing the estimate. Without a plan, the estimate is a one dimensional number, and that is all we are left with to evaluate any feedback on. But with a plan, the feedback is more useful ‐ it is easier to find out what went wrong in our assumptions and account for that when estimating similar stories in the future.
To conclude, plan early, plan often, plan deep. And good things will follow.