Making Sense of #NoEstimates

[article]
Summary:

A couple of years ago, the Twitter hashtag #NoEstimates appeared. Its purpose was to start a discussion about alternatives to estimations, but the idea of a project without explicit estimates is odd to most people in software development. However, if you start exploring it, you may find better sources of information to rely on.

When I first heard about the Twitter hashtag #NoEstimates, it sounded weird—even heretic. How could there be a project without estimations? Isn’t it obvious to anyone that estimates are the basis for planning capacity and we can’t issue a simple quote without them?

In the two years since this became a discussion topic on social media, I’ve gone through many sessions of thinking and writing about it. This article sums up how I feel about #NoEstimates and what the movement really tries to solve.

What Is #NoEstimates?

The hashtag was created, I think, to sound controversial. But if you check out the website, you’ll see that the idea is not exactly quite as polarizing. The people who started this party, Woody Zuill, Neil Killick, and Vasco Duarte, say the discussion they raised was about exploring alternatives to estimations, rather than doing without them completely.

However, no means no, and therefore it put some people up in arms—mostly project managers. They rightfully asked, “Without estimates, how can you plan? Isn’t the customer, the one with the money, entitled to know how much the project will cost and how long it will take?”

These are very good questions.

Imagine you’re doing a kitchen makeover. You ask for an estimate from a contractor, and he tells you it will be twenty thousand dollars and will take four weeks. Now, you know he’s wrong. It will probably take twice as long and twice the cash.

But if you already knew that, why did you ask for the estimate in the first place?

If you’re thinking, “I need the estimates for the work to be done,” stop right there. If you had infinite amounts of time and money and you hired that contractor, the work would still be done, even if you didn’t ask for estimates.

We don’t need estimates. But we really want them.

Why Do We Want Estimates?

We want estimates in order to make decisions. Based on the estimate we get for a project, we can make a go/no-go decision. If it costs too much or will take too long, we can scrap the project or delay it. If we compare alternative estimates, we can choose one that fits our needs better.

Estimates also help in planning. For example, if our development project is estimated to last a year, the company will want to start marketing efforts two months before the year’s end. We’ll want to train the sales representatives a month before release. The estimates give us visibility into dependencies so we can plan better.

Obviously, estimates are important—everyone asks for them. The funny thing is nobody likes giving them.

The Problem with Estimates

Did you ever give an estimate that “magically” turned into a commitment? This organizational dysfunction happens a lot, as the inaccurate estimate becomes the deadlines against which the developer is measured.

Developers are smart, so they don’t fall into this trap twice. The developers pad their estimates, their team lead adds a little buffer, and her director makes sure the team can actually do the work based on their record by adding more spares. We can go from a few weeks’ worth of the original estimate to a few months, maybe even years.

Here, it gets messy. If we wanted the estimate to decide on a go/no-go for the project, we now have the wrong numbers to decide by!

Clever as we are, we found a solution for that, too: If we know more about the project, we’ll get a more accurate estimate (yeah, it’s a thing). If everyone agrees on the work involved, then there’s no need for the extra padding. So we convene, and discuss, and meet again, sometimes for weeks, to understand what we’re going to build. In a couple of months, we’ll have a beautiful golden number. And during that process we’ve wasted hundreds of hours of assumptions and calculations rather than actually building the product.

There are a lot of problems with estimates. Yet these are not the main issue.

Estimates Are Not Really Helpful

I’ve never seen a project canceled because of estimations. If it was valuable enough, we found a way to do it. I’ve witnessed projects delayed based on their uncertain value and risk, though. Alternatives are nice to have, but they are really worthless because at the decision point, you mistrust all of them equally. And show me the project lead who will trust my estimate enough today that he won’t ask me again in ten months where I am, so we can start doing some marketing.

We wanted estimates to help with decision-making and planning. It seems that they are not useful tools for that.

We don’t trust estimates because we suck at making them. We are biased, we’re optimistic, and we think we know what we’re talking about and that the project will be a piece of cake. Then we get surprised every time. So when we get estimates, we’re right to be suspicious.

We also have a theoretical basis for this mistrust. Steve McConnell introduced us to the Cone of Uncertainty twenty years ago. When we estimate the project, we know the least about it. The more we make progress, we clear the fog of war and estimate better. But when requirements change (and they always do), the cone opens up again, and our estimates go down the drain.

What Do We Really Want?

We want to be right. We want to make the right decision and feel confident about it. Estimates are tools, but should we rely only on them to make decisions? Maybe there is another information source available that can help us be right and confident.

Neil Killick gave this example: You need to get somewhere by train. You can ask, “When does the train leave?” and I’ll give you an estimate, which you can plan your arrival on. But if I told you I can’t estimate when the train leaves but I know for a fact trains come precisely every ten minutes, would that help you make a better decision?

Suddenly the accuracy of the estimate becomes less relevant, while the other piece of information has more impact on the decision.

What Other Types of Information Can We Rely On?

If you look around, there are other types of information besides estimates that we can base our decisions on.

  • Prioritization by value: If we can estimate how much value we get, then compare it to the estimated cost, we can make a better decision. Focusing on value rather than cost can illuminate where we should invest. Planning methods such as quantifying cost of delay put value first and make it simple to compare and prioritize projects.
  • Evaluate complexity: Consider how the project is similar to other projects and how much of that knowledge is transferable to your team. Was the project done before in your team? In your company? Or is it completely new? Answer this question, and see what your estimates are worth. Liz Keogh’s model for complexity evaluation can put estimates in context.
  • Collect the data: Past experience is by far a better predictor than gut feelings. Velocity information from the team gives better prediction about the ability to manage similar projects. Note, however, that past velocity is not applicable for different team composition, technologies, or domains, so be careful.
  • Reduce story variance: Velocity is helpful if stories don’t vary in size. If stories always take three to four days, you can predict by velocity. If story size varies, velocity as a mean doesn’t signify much and cannot be reliably used for prediction. The team needs to be skilled at cutting stories into the same size, but when they get there, prediction becomes more reliable.
  • Assume you are ignorant: Former US Secretary of Defense Donald Rumsfeld has given complexity research a valuable gift by simplifying categories of knowledge: what we know we know, what we don’t know, and what we don’t know we don’t know. The first two are what we estimate, and the latter tears the estimates apart. The most important things we can assume is that we don’t know anything and that everything we think we know is an assumption.
  • Enumerate your assumptions: Our estimates our based on assumptions, and it is better to discuss those before discussing the estimates. When we make our assumptions known, a funny thing happens: They can be criticized, approved, or disproved. This criticism is a small price to pay instead of relying on an estimate we conjured in our heads.
  • Plan for deliberate discovery: Finally, if everything is in our heads, are estimates really the way to decide? Maybe instead, we should discover what we need to build. Instead of creating a plan based on our conjured numbers, maybe our plan should consist of experiments to prove the assumptions we have. And instead of a big plan consisting of what we think the customer wants, we can plan a set of small, cheap, safe-to-fail experiments that will show us if our assumptions are correct or not and pave the way to a successful product, rather a gigantic project failure.

Estimate or #NoEstimate? That Is the Question

There are projects done without estimates. However, it takes an understanding environment to practice this approach. If your organization is not there yet, try teaching people about the alternatives described here, to be used in addition to estimating. Limit the time for estimation research and instead plan how to get quick feedback about your assumptions.

When I started researching #NoEstimates, the idea looked weird. Since then, estimates are the ones that look weird to me. While it’s so easy to ask, “How much will it cost?” I now almost automatically try to make decisions by relying less on estimates and more on actual information.

It makes better sense, doesn’t it?

User Comments

4 comments
Glen Alleman's picture

You cover lots of ground here. Several of the section describe symptoms not root causes. "We're bad at estimates so let's not do them," ignores the root cause - untrained, inexperienced, unskilled estimators should not be estimating. Learn how to estimate.

The example of the kitchen, $20K and 6 weeks, expected to go to twice is an estimate based on an "anchor" and "adjusted" for experience. It's still an estimate, but with a higher probability of completion on time and on budget. This increase started with a low probability estimate and ended with a higher probability estimate. This is part of the estimating process. Anchor, optimistic, adjust with reference class data.

The notion that estimate aren't really helpful because projects aren't canceled because of an estimate is not the case in many domains. Enterprise IT commonly makes choices based on estimates. The called "real options." It is the fiduciary responsibility of management to properly apply their limited funds for the maximum benefit of the firm. The is called Microeconomics - making choices - actually opportunity costs - in the presence of uncertainty about the future value of those choices. This is commonly baked into the IT Governance processes.

"We don't trust estimates because we suck at them." seems very naive. I can't learn to estimate, so I won't do it. http://goo.gl/Pq4XAS If you actually "suck" at estimates, your supervisor should never ask you to, you suck at estimating. Doesn't remove the business need to have an estimate to be used for decision making. Simple ROI or more advanced Real Options. "I suck at DB table design, so I'm not going to design 1st Normal Form tables, I'm just going to start throwing columns and keys together and see where it takes me." That doesn't sound like responsible behavior when spend other peoples money does it. 

When mangers misuse estimates - and low maturity mangers do - they're being low maturity managers. It not an estimating problem it's a management maturity problem. Stopping estimates will not increase their maturity.

The empirical examples of Neil are miss-stated. The knowledge of the numbers on arrival board of every 10 minutes allows me to estimate when I need to arrive at the station. I make an estimate that if I leave my home 15 minutes before the train I need one that arrives and departs in 20 minutes, then I have a nearly 100% of catching that train. That is exactly how our team commuted to the office from Eching to Munich every days. Steady departures + our travel time from our apartment to the S-Ban station, allowed us to estimate what train we'd be able to catch.

All the bullets on the last page are useful and needed. But those are development activities. The business, in order to stay in business needs to know - to some level of confidence - the cost of development for the value produce for that development. This is the simple ROI = (Value-Cost)/Cost, time phased over the life of the project.

Yes, there are projects being done without estimates. Proof by anecdote is very sporty business. Do those projects have a governance framework? Are they of "high value at risk," are they short lived.

If you're writing software for money, with someone else’s money, it's not really your decision on how to spend that money. It's not your money. Woody, Neil, and Vasco have yet to connect the dots between their personal dislike for estimating and those paying their salary's need to know how much. Until the connection is made to the business, the Microeconomics of SWDev, management in the presence of uncertainty (more MicroEcon), this discussion well never be credible, since it ignored, possibly with intent, the basic principles of all for-profit business. "The value can't be less than the cost." Since both of those are in the future, and both are probabilistic, the very notion of making decisions in the absence of estimates violates the principles of probabilistic decision-making.

The #NoEstimates advocates have started in the wrong end of the question. “If business has an interest in knowing when you will be done, how much it will cost, and what you’re producing” in the presence of uncertainty, then estimating is needed. If you “suck” at estimating, it’s not likely you’ll be asked to participate at the table when decisions are being made.

The empirical approach of Neil, assumes the future will be like the past. This is naïve best, and simply not true in fact. Uncertainty pervades all project work. Simple and small samples, with understanding the underlying statistical processes produce “uninformed by risk” projects of the future. The data provide by Vasco for his empirical examples have a ±30% variance. Not what I’d called high confidence.

There is lots of guidance on how to make good estimates in the software development business. Journals, books, articles, profession societies. The basis of for-profit business depends on knowing both value and cost to some level of confidence. Stating “we suck” or “I know a project that doesn’t estimate” is no going to remove the need. It just reinforces the notion of the Woody’s et al that “I’m disconnected from the seat at the table where decision about the business are being made,” at least any business that has serious value at risk. If the business has low value at risk, no one really cares what you do. These two domains have not been addressed by the advocates of #NoEstimates. 

February 25, 2015 - 9:34pm
Mark Hart's picture

Instead of a binary assessment regarding 'estimates' or '#NoEstimates,' I prefer an emphasis on outcomes and value. Artifacts such as story points and user stories are deprecated.

I strive to move from the acceptance of the status quo to new assessments of efforts that are more likely to provide better outcomes and value. 

I have found that the best way to get better at estimates is to leverage more from past performance and strive to improve capabilities to adapt to the inevitable emergence of challenges during projects.

 

A recent 38 minute podcast at http://giantrobots.fm/132 reviewed another evolution of insights on estimates. 

February 26, 2015 - 4:43pm
Leyton Collins's picture

Gil, kudos for sharing this! Lots of great stuff via the hashtag on Twitter too.

Those who don't understand this rudimentary concept of Agile are doing the rhetorical 'lipstick on a pig' thing.

Again, kudos for getting this out there!

February 27, 2015 - 11:12am
Glen Alleman's picture

Mark,

"I prefer to an emphasis on value and outcomes," needs to be connected with the cost and time needed to produce those outcomes that have value. Without an estimate of that cost and the time when that value will be available for use, the value cannot be used to make decisions as to it's actul "worth" until it is too late.

The purpose of estimates in the decision making paradigm is to seperate those "proposed" outcomes from each other based on the Microeconomics of sofwtare development. This is the "opportunity cost" decision process.

What is the value of a needed feature IF it cost more than the produced value and shows up later then when I need it. Estimating all three - Value, Time, and Cost are the basis of "Opportunity cost" decisions.

This principles can be found in simple rules like Return On Investment, http://goo.gl/UP6Wkz to Real Options, http://goo.gl/Q5ajhc (which Vasco claims he uses)  to Stochastic Decision modeling of System Dynamics http://goo.gl/wWNiCW 

Deciding where "value" to produce, in the presence of limited resources (time, money, talent) requires an estimate of not only the time and money, but also the probability that the "value" will be what you need to recope that investment in time and money.

March 2, 2015 - 10:33am

About the author

CMCrossroads is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.