18 May 2012

Optimism is necessary and troubling

As a follow up to this video on the optimism bias, I thought I would add a few comments.

Tali's hypothesis is that optimism is a critical element of success. Without optimism we'll just give up, or not have faith we can overcome the challenges that beset us. So optimism is a net gain for the human race.

On the other hand, optimistic estimates make us fail.  We fail because we set unrealistic expectations and thus when we deliver, we let people down.

It's a conundrum. How do we optimise delivery by taking advantage of our natural optimism, and still avoid over estimating our ability to solve the problem?

Some solutions that have been tried in the past include;

  • Separate the people who do the work from those who do the estimates so you can bypass the optimism in the team
  • Ensure your team has pessimists on board so that they can speak of the worst case, bundle this with blind delphi estimates
  • Force thinking by the team about best, likely and worst case through scenario modelling, and weight the estimates with historical performance or arbitrary numbers
None of these work particularly well.

Our preferred method today is to use historical data. And if we don't have some, to work in small bacthes until we do.

The jury is still out on how we budget for the rocket to Jupiter.


  1. Nice post, Craig. I've worked on teams that have used the first two techniques that you mentioned. They were relatively effective (compared to just having the project team make an estimate), but as you point out, these techniques are not perfect.

    I really like the sound of the third technique you mentioned. I can see that method giving team members the freedom to think openly about worst case scenarios. I've found that once an initial estimate is presented (either formally or informally) the other team members use that estimate as a starting point, rather than coming up with their own independent estimates. If the initially estimate is optimistic, it can be tough for a team to significantly change it. Especially considering that few people want to be the lone "pessimistic" voice.

    One question: when you talk about using historical data, what exactly do you mean? Do you mean finding similar work that your team has done in the past, and identifying how long it took to complete the work?


  2. Brian
    I wrote quite a long response, but it has disappeared into the ether.

    I will restate it but shorter :)

    Historical data can be

    1. As you say, similar projects from the organisation's past
    2. Just start and work out a Velocity in the first 4-6 weeks, then continue to revise your estimate to complete until it is stable
    3. Go get feature or smaller level historical data from your team and use that

    Two places to read more are the blogs of Neil Killick and Vasco Duarte.