Sunday, November 19, 2006

Estimating Estimations

During last few days I was doing a research on how to perform effective estimations. There are a few interesting points I have learnt in doing the research. One good theory I learnt was the “Cone of Uncertainty”.

The above picture depicts how the level of uncertainty changes during the project life cycle. As you can see the uncertainty reduces as project progresses. This may not be news to most of us, but the problem is how many of us use this instinctive knowledge in our estimation practices? For an example is we provide an estimation in the project agreement level, the deviation of actual effort can be 4 times the estimation. “Later you estimate, higher the accuracy will be”

Also if we look at the project failure (or overrun) rates, market research shows that the success probability is high for small projects.

Project Size (lines of code)

Early

On Time

Late

Failed

1000 LOC

11%

81%

6%

2%

10,000 LOC

6%

75%

12%

7%

100,000 LOC

1%

61%

18%

20%

1,000,000 LOC

<1%

28%

24%

48%

10,000,000 LOC

0%

14%

21%

65%

One reason for this fact is that it is difficult to estimate a large project than a small one (of course there are many other reasons also). If you have a large system to develop try to break it in to several small projects. “Smaller the project, higher the success probability”

Another common mistake we make is the bias towards so called ‘Expert Judgment’. How many of us believe our experience and knowledge is the best source for estimations? Probably many of us think so. But the researches have proven the converse. Expert judgment is one of the worst methodologies to derive an estimate. So what would give us a better estimate? Often the historical data provide us a better estimation than any other method. Historical data can be in 3 forms:

  • Industry average data
  • Organization average data
  • Project local data

Needless to say the data accuracy is higher in project data than industry average data. We should build an estimation process that uses historical data. It is better to use several industry accepted estimation methodologies (customized with historical data) instead of a one methodology. Simple models with less control knobs have proven to deliver effective results. “Never trust expert judgments, use count and compute methods empowered with historical data”

One other common mistake is forgetting many of the required software activities at the estimation time. It is very preferred to have omitted activity list and go through it at the estimation time. Also allow several individual estimators to work independently and then converge those to get the final estimate. The derived estimate should not be a single value but should be presented with a variance. Three point estimation techniques is a good place to start working on estimation probability.

I hope these hits will help you in making a proper estimation process for your organization. You probably need to build a customized process that match with your organization practices. Try to use estimations throughout the project life cycle after end of every phase. Also make sure to asses and improve the estimation process. Cost of over running a project for several times will be definitely higher than your estimation effort. “Invest on estimations. It always paybacks”

References:
Software Estimation: Demystifying the Black Art
Software Measurement and Estimation: A Practical Approach
CHAOS Report

2 comments:

Bruce said...

Deep in the guts of your notes about estimating is the assumption that you're giving estimates in ranges rather than single numbers. And estimating in ranges is critical to giving a good estimate.

But one of the problems I see is that project managers want a single number to plug into MS Project (or any number of similar tools). We need better tools to handle this kind of ranged estimate (shameless plug for LiquidPlanner). Until our tools support this kind of good estimation practice I think we will have an uphill fight.

Good summary of parts of McConnell's book by the way! Have you looked at his company's site (Construx)? The blogs and forums have some good discussions of these topics.

Cheers!

Hasith Yaggahavita said...

Thanks for the note Bruce.

I agree with your point that everyone looks for a single number other than we poor developers. The challenge is to convince the parties to accept a probabilistic estimation. I often present my estimation experience on a powerpoint to our customers having good amount of industrial data to backup my points. This seems to work pretty well.

One point I missed in the original post is the importance of assumptions. We never get complete set of requirements at estimations and have to put every assumption as a note in the estimation.

Yes I had a look in to 'construx' and I like the tool. It has good amount of statistical analysis built in.

Regarding planning software MS Project allows you to derive plans based on best, likely and worst case scenarios (PERT analysis). I managed to build 3 point cost estimations in to a MS Project template plan using custom tables.