Expert Political Judgment

Even the most astute observers will fail to outperform random prediction generators—the functional equivalent of dart-throwing chimps—in affixing realistic likelihoods to possible futures.

Even the most astute observers will fail to outperform random prediction generators—the functional equivalent of dart-throwing chimps—in affixing realistic likelihoods to possible futures. Overconfidence is pervasive. Experts in political and economic forecasting consistently believe they know more about the future than they do. Their subjective probability estimates for outcomes they deem most likely significantly exceed the actual frequency with which those outcomes materialize. For instance, events experts rated as 100% certain occurred only about 80% of the time. Barely beating chance. When pitted against simple performance benchmarks, experts' forecasting accuracy is humbling. They only marginally outperform random chance (dart-throwing chimps) and simple extrapolation algorithms that mechanically predict the continuation of past trends. They are significantly outperformed by sophisticated statistical models. Diminishing returns. Beyond a minimal level of general knowledge (like that of an attentive reader of a quality newspaper), increasing expertise or professional status confers little additional advantage in forecasting accuracy. Specialists are often no better than well-informed generalists when predicting outcomes outside their narrow domain, suggesting that predictive skill reaches a plateau quickly.

What experts think matters far less than how they think. Content vs. Style. The content of an expert's beliefs – whether they are liberal or conservative, realist or institutionalist, optimist or pessimist – is a poor predictor of their forecasting accuracy. Experts from across the ideological and theoretical spectrums are equally likely to be right or wrong. Cognitive style is key. However, the way experts think about the world is a significant predictor of their judgmental performance. Differences in cognitive style, particularly along a dimension related to how people handle complexity and conflicting information, strongly correlate with forecasting skill. This suggests that the mental approach to processing information is more crucial than the specific information held or the conclusions drawn. Beyond background. Traditional markers of expertise, such as educational attainment (Ph.D. vs. Master's), years of experience, professional background (academic, government, journalist), or access to classified information, show little to no correlation with forecasting accuracy. This reinforces the idea that inherent cognitive traits or learned thinking processes are more important than credentials or access.

If we want realistic odds on what will happen next, coupled to a willingness to admit mistakes, we are better off turning to experts who embody the intellectual traits of Isaiah Berlin’s prototypical fox... than we are turning to Berlin’s hedgehogs. The Hedgehog and the Fox. Drawing on Isaiah Berlin's metaphor, experts can be broadly categorized: Hedgehogs "know one big thing" and try to fit everything into a single, coherent framework. Foxes "know many little things" and draw from an…