Search This Blog

18 May 2012

Optimism is necessary and troubling

As a follow up to this video on the optimism bias, I thought I would add a few comments.

Tali's hypothesis is that optimism is a critical element of success. Without optimism we'll just give up, or not have faith we can overcome the challenges that beset us. So optimism is a net gain for the human race.

On the other hand, optimistic estimates make us fail.  We fail because we set unrealistic expectations and thus when we deliver, we let people down.

It's a conundrum. How do we optimise delivery by taking advantage of our natural optimism, and still avoid over estimating our ability to solve the problem?

Some solutions that have been tried in the past include;

  • Separate the people who do the work from those who do the estimates so you can bypass the optimism in the team
  • Ensure your team has pessimists on board so that they can speak of the worst case, bundle this with blind delphi estimates
  • Force thinking by the team about best, likely and worst case through scenario modelling, and weight the estimates with historical performance or arbitrary numbers
None of these work particularly well.

Our preferred method today is to use historical data. And if we don't have some, to work in small bacthes until we do.

The jury is still out on how we budget for the rocket to Jupiter.