https://blog.codinghorror.com/how-good-an-estimator-are-you/
Chapter 2 of Software Estimation: Demystifying the Black Art opens with a quiz designed to test your estimation abilities. It's an interesting exercise, so I thought everyone might like to give it a shot.
- For each question, fill in the upper and lower bounds so that you have a 90 percent chance of including the correct value.
- Don't make your ranges too narrow or too wide, but be sure they're wide enough to give you a 90 percent chance of hitting the correct value.
- Don't research the answers on the internet.
- You must provide an estimate range for each question.
- Spend no more than 10 minutes on this quiz.
Software developers are often asked to estimate projects in unfamiliar business areas, projects that will be implemented in new technologies, the impacts of new programming tools on productivity, the productivity of unidentified personnel, and so on. Estimating in the face of uncertainty is business as usual for software estimators. The rest of [the book Software Estimation: Demystifying the Black Art] explains how to succeed in such circumstances.
| Question | Answer |
| Surface temperature of the sun | 10,000F / 6,000C |
| Latitude of Shanghai | 31 degrees North |
| Area of the Asian continent | 17,139,000 square miles 44,390,000 square kilometers |
| The year of Alexander the Great's birth | 356 BC |
| Total value of U.S. currency in circulation in 2004 | $719.9 billion |
| Total volume of the Great Lakes | 5,500 cubic miles 23,000 cubic kilometers 2.4 x 10^22 cubic feet 6.8 x 10^20 cubic meters 1.8 x 10^23 U.S. gallons 6.8 x 10^23 liters |
| Worldwide box office receipts for the movie Titanic | $1.835 billion |
| Total length of the coastline of the Pacific Ocean | 84,300 miles 135,663 kilometers |
| Number of book titles published in the U.S. since 1776 | 22 million |
| Heaviest blue whale ever recorded | 380,000 pounds 190 English tons 170,000 kilograms 170 metric tons |
For the test takers whose results are shown in the figure, the average number of correct answers is 2.8. Only 2 percent of quiz takers score 8 or more correct answers. No one has ever gotten 10 correct. I've concluded that most people's intuitive sense of "90% confident" is really comparable to something closer to "30% confident." Other studies have confirmed this basic finding (Zultner 1999, Jrgensen 2002).
Additionally, the few people who manage to get close to the goal of ~9 correct answers typically feel they did something wrong:
When I find the rare person who gets 7 or 8 answers correct, I ask "How did you get that many correct?" The typical response? "I made my ranges too wide."My response is, "No, you didn't! You didn't make your ranges wide enough!" If you get only 7 or 8 correct, your ranges were still too narrow to include the correct answer as often as you should have.We are conditioned to believe that estimates expressed as narrow ranges are more accurate than estimates expressed as wider ranges. We believe that wide ranges make us appear ignorant or incompetent. The opposite is usually the case.
So, what have we learned from this exercise?
- When you ask someone for a range that provides 90% confidence, expect 30% confidence on average.
- People are naturally hesitant to provide wide ranges – even when the confidence level requires a wide range to be accurate – because they feel that narrow estimates are a sign of a better estimate.
Narrow estimates are self-defeating, but unfortunately they are human nature. Unless you have specific data that supports your narrow estimate, your estimate probably should be wider than you made it.
