For the test takers whose results are shown in the figure, the average number of correct answers is 2.8. Only 2 percent of quiz takers score 8 or more correct answers. No one has ever gotten 10 correct. I've concluded that most people's intuitive sense of "90% confident" is really comparable to something closer to "30% confident." Other studies have confirmed this basic finding (Zultner 1999, Jrgensen 2002).
Additionally, the few people who manage to get close to the goal of ~9 correct answers typically feel they did something wrong:
When I find the rare person who gets 7 or 8 answers correct, I ask "How did you get that many correct?" The typical response? "I made my ranges too wide."
My response is, "No, you didn't! You didn't make your ranges wide enough!" If you get only 7 or 8 correct, your ranges were still too narrow to include the correct answer as often as you should have.
We are conditioned to believe that estimates expressed as narrow ranges are more accurate than estimates expressed as wider ranges. We believe that wide ranges make us appear ignorant or incompetent. The opposite is usually the case.
So, what have we learned from this exercise?
  • When you ask someone for a range that provides 90% confidence, expect 30% confidence on average.
  • People are naturally hesitant to provide wide ranges – even when the confidence level requires a wide range to be accurate – because they feel that narrow estimates are a sign of a better estimate.
Narrow estimates are self-defeating, but unfortunately they are human nature. Unless you have specific data that supports your narrow estimate, your estimate probably should be wider than you made it.