Index Investor Forecasting Methodology Summary



Rather than statistical or machine learning, our approach to forecasting more closely resembles estimative intelligence analysis, which employs a combination of bottom-up and top-down sensemaking processes. These were described in Pirolli and Card's classic article, "The Sensemaking Process and Leverage Points for Analyst Technology”, which includes this very useful graphic:

Stacks Image 7

Our forecasting process also draws on lessons Tom Coyne learned from spending four years as a member of the Good Judgment Project team, which won the Intelligence Advanced Research Projects Activity’s forecasting tournament with results that were more than 50% more accurate than the tournament's control group of professional intelligence analysts (the team's experience is described in Professor Philip Tetlock's book, “Superforecasting").

For example, we pay careful attention to base rates, and take a Bayesian approach to using new evidence to update our prior regime probabilities.

Perhaps most important, over our years of forecasting we have always kept in mind the conclusion reached by a 1983 CIA study of failed forecasts: "Each involved historical discontinuity, and, in the early stages…unlikely outcomes. The basic problem was…situations in which trend continuity and precedent were of marginal, if not counterproductive value."

That's still just as true — and just as dangerous — today.

For that reason, we urge subscribers to keep in mind three research findings about that can be used to improve forecast accuracy:

(1) Obtain forecasts based on different methodologies, or prepared by forecasters with significantly different backgrounds (as a proxy for different mental models and information).

(2) Combine those forecasts (using a simple average if few are included, or the median if many are).

(3) The final step is to “extremize” the average (mean) or median probability by using a systematic process to move it closer to 0% or 100%. This helps to offset the impact of forecasters' tendency to underestimate the extent to which their information is either little known and/or not incorporated into other forecasters' predictions. We provide subscribers with a tool for extremizing a forecast.