Statistical Significance and Replications

In the presidential debate experi ment, the average rating of Reagan in the unedited condition was 66 on a 0–100-point scale, and it was 49 in the con- dition in which the jokes remained but the audience reaction was removed. Is the difference between 66 and 49 large enough to be meaningful, or could this difference simply be due to chance (that is, just random variation, like flipping a coin 10 times and getting heads 6 times and tails 4)? The answer is that you can’t tell just by looking at these numbers alone. Results obtained in an experiment are examined with statistical analyses that allow the researcher to determine how likely it is that the results could have occurred by random chance. The standard convention is that if the statistical analyses indicate that the results could have occurred by chance fewer than 5 times in 100 possible outcomes, then the result is statistically signif- icant and should be taken seriously.

The fact that results are statistically significant does not mean, however, that they are absolutely certain. In essence, statistical significance is an attractive betting proposition. The odds are quite good that the effects didn’t occur by random chance, but there is still the small possibility. This is one reason why replication is so important—that is, to repeat the experiment and find similar results. If similar results are found, the probability that these results could have occurred by chance both times becomes exponentially more remote. Statistical significance is relevant not only for the results of experiments but also for many other kinds of data as well, such as correlations. A correlation between two variables may be statistically significant or not, depending on the strength of the correlation and the number of participants or observations in the data.