Since the 1930s, many of our top methodologists have argued that significance tests are not conducive to science. Bakan (1966) believed that “everyone knows this” and that we slavishly lean on the crutch of significance testing because, if we didn’t, much of psychology would simply fall apart. If he was right, then significance testing is tantamount to psychology’s “dirty little secret.” This paper will revisit and summarize the arguments of those who have been trying to tell us—for more than 70 years—that p values are not empirical. If these arguments are sound, then the continuing popularity of significance tests in our peer-reviewed journals is at best embarrassing and at worst intellectually dishonest.
AnastasiA. (1976). Psychological testing (4th ed.). New York, NY: Macmillan.
4.
AndreskiS. (1972). Social sciences as sorcery. London, UK: André Deutsch Limited.
5.
ArmstrongJ.S. (2007). Statistical significance tests are unnecessary even when properly done and properly interpreted: Reply to commentaries. International Journal of Forecasting, 23, 335–336.
6.
BakanD. (1966). The test of significance in psychological research. Psychological Bulletin, 66, 423–437.
7.
BakanD. (1974). On method: Toward a reconstruction of psychological investigation. San Francisco, CA: Jossey-Bass.
8.
BerksonJ. (1938). Confidence curves: An omnibus technique for estimation and testing statistical hypotheses. Journal of the American Statistical Association, 33, 526–542.
9.
BlinkhornS.JohnsonC. (1990, December27). The insignificance of personality testing. Nature, 348, 671–672. doi: 10.1038/348671a010.1038/348671a0
10.
BoringE.G. (1919). Mathematical versus scientific significance. Psychological Bulletin, 16, 335–338.
11.
BradburnN.M. (2007). A tribute to Bill Kruskal. Statistical Science, 22, 262–263.
12.
BrunswikE. (1952). The conceptual framework of psychology. In NeurathO. (Ed.), International encyclopedia of unified science (Vol. 1, pp. 655–760). Chicago, IL: University of Chicago Press.
13.
BrunswikE. (1956). Perception and the representative design of psychological experiments. Berkeley: University of California Press.
14.
CampbellJ.P. (1982). Some remarks from the outgoing editor. Journal of Applied Psychology, 67, 691–700.
15.
CarverR.P. (1976). Letter to the editor. Educational Psychologist, 12, 96–97.
16.
CarverR.P. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378–399.
17.
CarverR.P. (1993). The case against statistical significance testing, revisited. The Journal of Experimental Education, 61, 287–292.
18.
CattellR.B. (1978). The scientific use of factor analysis in behavioral and life sciences. New York, NY: Plenum.
19.
CohenJ. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 69, 145–153.
20.
CohenJ. (1990). Things I have learned so far. American Psychologist, 12, 1304–1312.
21.
CohenJ. (1992). A power primer. Psychological Bulletin, 112, 155–159.
22.
CohenJ. (1994). The earth is round (p < .05). American Psychologist, 12, 997–1003.
23.
DawesR.M. (1994). House of cards: Psychology and psychotherapy built on myth. New York, NY: Free Press.
24.
DawesR.M. (2001). Everyday irrationality: How pseudo-scientists, lunatics, and the rest of us systematically fail to think rationally. Boulder, CO: Westview.
25.
DixonP. (1998). Why scientists value p values. Psychonomic Bulletin & Review, 5, 390–396.
26.
EdgeworthF.Y. (1886). Progressive means. Journal of the Royal Statistical Society, 49, 469–475.
27.
EdwardsW. (1965). Tactical note on the relation between scientific and statistical hypotheses. Psychological Bulletin, 63, 400–402.
FalkR. (1998). In criticism of the null hypothesis statistical test. American Psychologist, 53, 798–799.
30.
FalkR. (2008). Probabilistic reasoning is not logical. Mathematics Magazine, 81, 268–275.
31.
FalkR.GreenbaumC.W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 5, 75–98.
32.
FaustD.ZiskinJ. (1988, July1). The expert witness in psychology and psychiatry. Science, 241(4861), 31–35.
33.
FidlerF.ThomasonN.CummingG.FinchS.LeemanJ. (2004). Editors can lead researchers to confidence intervals, but can’t make them think. Psychological Science, 15, 119–126.
34.
FinchS.CummingG.ThomasonN. (2001). Reporting of statistical inference in the “Journal of Applied Psychology”: Little evidence of reform. Educational and Psychological Measurement, 61, 181–210.
35.
FisherR.A. (1925). Statistical methods for research workers. London, UK: Oliver & Boyd.
36.
FisherR.A. (1929). The statistical method in psychical research. Proceedings of the Society for Psychical Research, 39, 185–189.
37.
FisherR.A. (1935). The design of experiments. Edinburgh, UK: Oliver & Boyd.
38.
FisherR.A. (1955). Statistical methods and scientific induction. Journal of the Royal Statistical Society. Series B (Methodological), 17, 69–78.
39.
FisherR.A. (1956). Statistical methods and scientific inference. Edinburgh, UK: Oliver & Boyd.
40.
GerrigR.J.ZimbardoP.G. (2002). Psychology and life (16th ed.). Boston, MA: Allyn & Bacon.
41.
GigerenzerG. (1987). Probabilistic thinking and the fight against subjectivity. In KrügerL.GigerenzerG.MorganM. (Eds.), The probabilistic revolution: Vol. II. Ideas in the sciences (pp. 11–33). Cambridge, MA: MIT Press.
42.
GigerenzerG. (1993). The superego, the ego, and the id in statistical reasoning. In KerenG.LewisC. (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 311–339). Hillsdale, NJ: Erlbaum.
43.
GigerenzerG. (2000). Adaptive thinking: Rationality in the real world. Oxford, UK: Oxford University Press.
44.
GigerenzerG. (2004). Mindless statistics. The Journal of Socio-Economics, 33, 587–606.
45.
GigerenzerG.SwijtinkZ.PorterT.DastonL.BeattyJ.KrugerL. (1989). The empire of chance. Cambridge, UK: Cambridge University Press.
46.
GillJ. (1999). The insignificance of null hypothesis significance testing. Political Research Quarterly, 52, 647–674.
GreenwaldA.G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
49.
GroveW.M.ZaldD.H.LebowB.S.SnitzB.E.NelsonC. (2000). Clinical vs. mechanical prediction: A meta-analysis. Psychological Assessment, 12, 19–30.
50.
HagenR.L. (1997). In praise of the null hypothesis test. American Psychologist, 52, 15–24.
51.
HebbD.O. (1966). A textbook of psychology. Philadelphia, PA: Saunders.
52.
HoltR.R. (1970). Yet another look at clinical and statistical prediction: Or, is clinical psychology worthwhile?American Psychologist, 25, 337–349.
53.
HubbardR. (2004). Alphabet soup: Blurring the distinctions between p’s and α’s in psychological research. Theory & Psychology, 14, 295–327.
54.
HubbardR.ArmstrongJ.S. (2006). Why we don’t really know what statistical significance means: Implications for educators. Journal of Marketing Education, 28, 114–120.
55.
HubbardR.LindsayR.M. (2008). Why p values are not a useful measure of evidence in statistical significance testing. Theory & Psychology, 18, 69–88.
56.
HubbardR.RyanP.A. (2000).The historical growth of statistical significance testing in psychology—and its future prospects. Educational and Psychological Measurement, 60, 661–681.
57.
IoannidisJ.P.A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.
58.
JonesL.V.TukeyJ.W. (2000). A sensible formulation of the significance test. Psychological Methods, 5, 411–414.
59.
KirkR.E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746–759.
60.
KirkR.E. (2003). The importance of effect magnitude. In DavisS.F. (Ed.), Handbook of research methods in experimental psychology (pp. 83–105). Oxford, UK: Blackwell.
61.
KormanA.K. (1968). The prediction of managerial performance. Personnel Psychology, 21, 295–322.
62.
LindsayR.M. (1995). Reconsidering the status of tests of significance: An alternative criterion of adequacy. Accounting, Organizations and Society, 20, 35–53.
63.
LoftusG.R. (1991). On the tyranny of hypothesis testing in the social sciences. Contemporary Psychology, 36, 102–105.
LykkenD.T. (1991). What’s wrong with psychology anyway? In CicchettiD.GroveW. M. (Eds.), Thinking clearly about psychology: Matters of public interest (Vol. 1, pp. 3–39). Minneapolis: University of Minnesota Press.
66.
MahoneyM. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1, 161–175.
67.
MeehlP.E. (1967). Theory testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103–115.
68.
MeehlP.E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 103–115.
69.
MeehlP.E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108–141.
70.
MulaikS.A.RajuN.S.HarshmanR.A. (1997). There is a time and a place for significance testing. In HarlowL.L.MulaikS.A.SteigerJ.H. (Eds.), What if there were no significance tests? (pp. 65–115). London, UK: Erlbaum.
71.
MurrayD.SchwartzJ.LichterR. (2001). It ain’t necessarily so: How the media make and unmake the scientific picture of reality. Lanham, MD: Rowman & Littlefield.
72.
NunnallyJ. (1960). The place of statistics in psychology. Educational and Psychological Measurement, 20, 641–650.
73.
PaulosJ.A. (2003). A mathematician plays the stock market. New York, NY: Basic Books.
74.
PetrinovichL. (1979). Probabilistic functionalism: A conception of research method. American Psychologist, 34, 373–390.
75.
PlattJ.R. (1964, October16). Strong inference: Certain methods of scientific thinking may produce much more rapid progress than others. Science, 146, 347–353.
76.
PopperK.R. (1959). The logic of scientific discovery. London, UK: Hutchinson. (Original work published 1935)
77.
RosenbergA. (1988). Philosophy of social science. Boulder, CO: Westview.
78.
RosenthalR. (1993). Cumulating evidence. In KerenG.LewisC. (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 519–559). Hillsdale, NJ: Erlbaum.
79.
RosnowR.L.RosenthalR. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284.
80.
RozeboomW.W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57, 416–428.
81.
SalmonW.C. (1984). Scientific explanation and the causal structure of the world. Princeton, NJ: Princeton University Press.
82.
SalsburgD. (2001). The lady tasting tea: How statistics revolutionized science in the twentieth century. New York, NY: Holt.
83.
SawyerJ. (1966). Measurement and prediction, clinical and statistical. Psychological Bulletin, 66, 178–200.
84.
SchmidtF. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173–1181.
85.
SchmidtF. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115–129.
86.
SchmidtF.HunterJ.E. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In HarlowL.L.MulaikS.A.SteigerJ.H. (Eds.), What if there were no significance tests? (pp. 37–64). London, UK: Erlbaum.
87.
SchopenhauerA. (2004). Essays and aphorisms. London, UK: Penguin. (Original work published 1851)
88.
SedlmeierP.GigerenzerG. (1989). Do studies of statistical power have an effect on the power of studies?Psychological Bulletin, 105, 309–316.
89.
ShermerM. (2000). How we believe: Science, skepticism and the search for God. New York, NY: Holt.
90.
SkinnerB.F. (1972). Cumulative record: A selection of papers (3rd ed.). New York, NY: Appleton-Century-Crofts.
91.
StiglerS. (1986). The history of statistics: The measurement of uncertainty before 1900. Cambridge, MA: Harvard University Press.
92.
TabachnickB.FidellL. (2001). Computer-assisted research design and analysis. Needham Heights, MA: Allyn & Bacon.
93.
TalebN.N. (2007). Black swan: The impact of the highly improbable. New York, NY: Random House.
ThompsonB. (1999). Statistical significance tests, effect size reporting and the vain pursuit of pseudo-objectivity. Theory & Psychology, 9, 191–196.
96.
ThompsonB. (2002). “Statistical,” “practical,” and “clinical”: How many kinds of significance do counselors need to consider?Journal of Counseling and Development, 80, 64–71.
97.
ThompsonB. (2003). Guidelines for authors reporting score reliability estimates. In ThompsonB. (Ed.), Score reliability: Contemporary thinking on reliability issues (pp. 91–102). Thousand Oaks, CA: Sage.
98.
ThompsonB. (2006). Research synthesis: Effect sizes. In GreenJ.CamilliG.ElmoreP.B. (Eds.), Handbook of complementary methods in education research (pp. 583–603). Washington, DC: American Educational Research Association.
99.
ThompsonB. (2007). Effect sizes, confidence intervals, and confidence intervals for effect sizes. Psychology in the Schools, 44, 423–432.
100.
ThorngateW. (1990). The economy of attention and the development of psychology. Canadian Psychology, 31, 262–271.
101.
TukeyJ.W. (1962). The future of data analysis. Annals of Mathematical Statistics, 33, 1–67.
102.
TukeyJ.W. (1991). The philosophy of multiple comparison. Statistical Science, 6, 100–116.
103.
UttalW.R. (2005). Neural theories of mind: Why the mind–brain problem may never be solved. Mahwah, NJ: Erlbaum.
104.
WilkinsonL., & The Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.
105.
WrightR.H.CummingsN.A. (Eds.). (2003). Destructive trends in mental health: The well-intentioned path to harm. New York, NY: Taylor & Francis.
106.
YatesF. (1951).The influence of statistical methods for research workers on the development of the science of statistics. Journal of the American Statistical Association, 46, 19–34.
