Predictions by the Expert

 The studies have covered many different tasks- university admissions, recidivism or violence of criminals, clinical pathology, medical diagnosis, financial investment, sports, and weather forecasting. Thirty years after his book was published, Meehl suggested that “there is no controversy in social science that shows such a large body of qualitatively diverse studies coming out so uniformly in the same direction.”

Experts versus improper models

Despite their superiority to clinical judgment, actuarial models are difficult to use because the outcome Ye must be measured, to provide the raw data for deriving regression weights. It can be costly or time-consuming to measure outcomes (for recidivism or medical diagnosis), or definitions of outcomes can be ambiguous (What is “success” for a Ph.D.?). And past outcomes must be used to fit cross-validated regression weights to predict current outcomes, which makes models vulnerable to changes in true coeffidents over time. Therefore, “improper”5 models- which derive regression weights without using Ye – might be more useful and nearly as accurate as proper actuarial models.

In one improper method, regression weights are derived from the Ys judg- ments themselve:;;; then cues arc weighted by the derived weights and· summed. This procedure amounts to separating the overall expert judgment Ys into two components, a modeled component Ys and a residual component zs, and using only the modeled component Ys as a prediction. 6 That is,

Ys = Lb;.,J( + Zs = Ys + Zs

If the discarded residual zs is mostly random error, the modeled component Ys will correlate more highly with the outcome than will the overall judgment, Ys. (In standard terminology, the correlation between Ys and Ye, denoted r m’ will be higher than ra.)

This method is called “bootstrapping” because it can improve judgments without any outcome information: It pulls experts up by their bootstraps. Bowman  first showed that bootstrapping improved judgments in pro- duction scheduling; similar improvements were found by Goldberg  in clinical predictions based on MMPI scores7 and by Dawes in graduate admissions. A cross-study comparison showed that bootstrapping works very generally, but usually adds only a small increment to predictive accuracy (Camerer, 1981a). Table S.l shows some of those results. Accuracy can be usefully dissected with the lens-model equation, an identity relating several interesting correlations. Einhorn’sversion of the equation states

where R; is the bootstrapping model R 2 (how closely the judge resembles the linear model), and rz is the correlation between bootstrapping-model residuals zs and outcomes Ye (the “residual validity”). If the residuals zs represent only random error in weighing and combining the cues, rz will be close to zero. In this case, r m will certainly be larger than ra, and because Rs < 1, bootstrapping will improve judgments. But even if rz is greater than zero (presumably be- cause residuals contain some information that is correlated with outcomes), bootstrapping works unless s By contrast, actuarial models often are called “optimal linear models,” because by definition no linear combination of the cues can pn:uict Ye more accurately.

6 Of course, such an explanation is “paramorphic” : It describes judgments in a purely statistical way, as if experts were weighing and combining cues in their heads; the process they use might be quite different. However, Einhorn, Kleinmuntz, and Kleinmuntz argued persuasively that the paramorphic regression approach might capture process indirectly.