What does the Overconfidence Test measure?
It measures calibration: whether “90% sure” really means right 90 times in 100. For each of 10 questions you give a low and a high number you are 90% sure the answer lies between. If your sense of your own knowledge is accurate, about 9 of your 10 ranges hold the answer. Fewer means your ranges were too narrow for how much you knew.
It isn't a quiz. Not knowing the answer is fine: a wide range is the honest answer when you are unsure, and it counts as a hit when the answer is inside it.
How is your Overconfidence Test result calculated?
Your result is how many answers fell inside your ranges, ends included. Even a perfectly calibrated guesser doesn't score 9 every time, because each range still misses one time in ten. This is how often such a guesser gets each score, worked out from the binomial distribution:
| Inside your ranges | How often a calibrated guesser scores this |
|---|---|
| 10 of 10 | 34.9% |
| 9 | 38.7% |
| 8 | 19.4% |
| 7 or fewer | 7% |
| 5 or fewer | 0.2% |
So one game with 8 inside could be bad luck; 5 or fewer almost never is. Your result page says how likely your own score would be for someone whose ranges are right 90% of the time.
How do you give better ranges?
- Start from the edges, not the middle. Ask what the smallest believable answer is, and the largest. Starting from a best guess and widening it a little tends to leave the range too narrow: Amos Tversky and Daniel Kahneman described this pull of a starting number, anchoring, in 1974.
- Imagine being wrong. If the answer turned out to be outside your range, would you be surprised? If not, it isn't a 90% range yet.
- Use what you do know. A year can be bounded by what it must come before or after; a size, by things you can compare it with.
Where do the questions come from?
There are 35 questions, and each game draws 10 of them at random. Each has one fixed answer and a named source, shown after you answer: an encyclopaedia, a space agency, a sport's governing body. Answers that can change carry their date, like Mount Everest's height from the survey announced in 2020. None is a trick, and every answer is a number, so a range always makes sense.
Where does the Overconfidence Test come from?
Asking for ranges and counting how many hold the answer is how researchers measure calibration. Marc Alpert and Howard Raiffa used it in their work on training people to judge probabilities, published in the 1982 collection Judgment under Uncertainty: Heuristics and Biases, alongside Sarah Lichtenstein, Baruch Fischhoff and Lawrence Phillips's review of calibration. For a different kind of self-knowledge, try How Random Are You?, or test your eye with the Estimation Game.