The 'heuristics and biases' bias in expert elicitation

In the early 1970s Tversky and Kahneman published a series of papers on 'heuristics and biases' describing human inadequacies in assessing probabilities, culminating in a highly popular article in Science. This seminal research has been heavily cited in many fields, including statistics, as the definitive research on probability assessment. Curiously, although this work was debated at the time and more recent work has largely refuted many of the claims, this apparent heuristics and biases bias in elicitation research has gone unremarked. Over a decade of research into the frequency effect, the importance of framing, and cognitive models more generally, has been almost completely ignored by the statistical literature on expert elicitation. To remedy this situation, this review offers a guide to the psychological research on assessing probabilities, both old and new, and gives concrete guidelines for eliciting expert knowledge.