Traditional indices of effect size are designed to answer questions about average group differences, associations between variables, and relative risk. For many researchers, an additional, important question is, “How many people in my study behaved or responded in a manner consistent with theoretical expectation?” We show how the answer to this question can be computed and reported as a straightforward percentage for a wide variety of study designs. This percentage essentially treats persons as an effect size, and it can easily be understood by scientists, professionals, and laypersons alike. For instance, imagine that in addition to d or η2, a researcher reports that 80% of participants matched theoretical expectation. No statistical training is required to understand the basic meaning of this percentage. By analyzing recently published studies, we show how computing this percentage can reveal novel patterns within data that provide insights for extending and developing the theory under investigation.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
3.
CotaL. D. (2017). An examination of alternatives to null hypothesis significance testing (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 10686232)
4.
CummingG. (2014). The new statistics: Why and how. Psychological Science, 25, 7–29.
5.
DelaneyH. D.VarghaA. (2002). Comparing several robust tests of stochastic equality with ordinally scaled variables and small to moderate sized samples. Psychological Methods, 7, 485–503.
6.
DiamondD. M.RavnskovU. (2015). Additional commentary on deception in statin research. Expert Review of Clinical Pharmacology, 10, 1411–1412. doi:10.1586/17512433.2015.1102009
7.
EdwardsA.ElwynG.MulleyA. (2002). Explaining risks: Turning numerical data into meaningful pictures. BMJ, 324, 827–830.
8.
FergusonC. J. (2009a). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40, 532–538. doi:10.1037/a0015808
9.
FergusonC. J. (2009b). Is psychological research really as good as medical research? Effect size comparisons between psychology and medicine. Review of General Psychology, 13, 130–136. doi:10.1037/a0015103
10.
FisherA. J.MedagliaJ. D.JeronimusB. F. (2018). Lack of group-to-individual generalizability is a threat to human subjects research. Proceedings of the National Academy of Sciences, USA, 115, E6106–E6115. doi:10.1073/pnas.1711978115
11.
FunderD. C.OzerD. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2, 156–168. doi:10.1177/2515245919847202
12.
GenioleS. N.ProcyshynT. L.MarleyN.OrtizT. L.BirdB. M.MarcellusA. L.. . . CarréJ. M. (2019). Using a psychopharmacogenetic approach to identify the pathways through which—and the people for whom—testosterone promotes aggression. Psychological Science, 30, 481–494. doi:10.1177/0956797619826970
13.
GriceJ. W. (2020). OOM: Observation oriented modeling [Computer software]. Retrieved from http://www.idiogrid.com/OOM
14.
GriceJ. W.CraigD. P. A.AbramsonC. I. (2015). A simple and transparent alternative to repeated measures ANOVA. SAGE Open, 5(3). doi:10.1177/2158244015604192
15.
GrissomR. J. (1994). Statistical analysis of ordinal categorical status after therapies. Journal of Consulting and Clinical Psychology, 62, 281–284. doi:10.1037/0022-006X.62.2.281
16.
HarrisR. J.QuadeD. (1992). The minimally important difference significant criterion for sample size. Journal of Educational Statistics, 17, 27–49.
HsuL. M. (2004). Biases of success rate differences shown in binomial effect size displays. Psychological Methods, 9, 183–197.
19.
HubbardR. (2016). Corrupt research: The case for reconceptualizing empirical management and social science. Los Angeles, CA: Sage.
20.
HubertyC. J. (2002). A history of effect size indices. Educational and Psychological Measurement, 62, 227–240. doi:10.1177/0013164402062002002
21.
KirkR. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746–759. doi:10.1177/0013164496056005002
22.
LamiellJ. T. (2013). Statisticism in personality psychologists’ use of trait constructs: What is it? How was it contracted? Is there a cure?New Ideas in Psychology, 31, 65–71. doi:10.1016/j.newideapsych.2011.02.009
23.
MaxwellS. E.DelaneyH. D. (1993). Bivariate median splits and spurious statistical significance. Psychological Bulletin, 113, 181–190. doi:10.1037/0033-2909.113.1.181
24.
McGrawK. O.WongS. P. (1992). A common language effect size statistic. Psychological Bulletin, 111, 361–365. doi:10.1037/0033-2909.111.2.361
25.
MolenaarP. C. M. (2004). A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement: Interdisciplinary Research and Perspectives, 2, 201–218. doi:10.1207/s15366359mea0204_1
26.
MontgomeryJ. M.NyhanB.TorresM. (2018). How conditioning on posttreatment variables can ruin your experiment and what to do about it. American Journal of Political Science, 62, 760–775.
27.
RosnowR. L.RosenthalR. (2003). Effect sizes for experimenting psychologists. Canadian Journal of Experimental Psychology, 57, 221–237. doi:10.1037/h0087427
28.
RuscioJ. (2008). A probability-based measure of effect size: Robustness to base rates and other factors. Psychological Methods, 13, 19–30.
29.
RuscioJ.MullenT. (2012). Confidence intervals for the probability of superiority effect size measure and the area under a receiver operator characteristic curve. Multivariate Behavioral Research, 47, 201–223. doi:10.1080/00273171.2012.658329
30.
SauerS. (2018). Observation oriented modeling revised from a statistical point of view. Behavior Research Methods, 50, 1749–1761. doi:10.3758/s13428-017-0949-8
31.
SawaokaT.MoninB. (2018). The paradox of viral outrage. Psychological Science, 29, 1665–1678. doi:10.1177/0956797618780658
32.
SiegelE. H.WormwoodJ. B.QuigleyK. S.BarrettL. F. (2018). Seeing what you feel: Affect drives visual perception of structurally neutral faces. Psychological Science, 29, 496–503. doi:10.1177/0956797617741718
33.
SiegelS. (1956). Nonparametric statistics for the behavioral sciences. New York, NY: McGraw-Hill.
34.
SimmonsJ. P.NelsonL. D.SimonshohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
35.
ThompsonA.TempleN. J. (2004). The case for statins: Has it really been made?Journal of the Royal Society of Medicine, 97, 461–464. doi:10.1177/0141076809701002
36.
TrafimowD. (2019). A frequentist alternative to significance testing, p-values, and confidence intervals. Econometrics, 7(2), Article 26. doi:10.3390/econometrics7020026
37.
TukeyJ. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
38.
ValentineK. D.BuchananE. M.ScofieldJ. E.BeauchampM. T. (2019). Beyond p values: Utilizing multiple methods to evaluate evidence. Behaviormetrika, 46, 121–144. doi:10.1007/s41237-019-00078-4
39.
van ’t VeerA. E.Giner-SorollaR. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12.
40.
VarghaA.RudasT.DelaneyH. D.MaxwellS. E. (1996). Dichotomization, partial correlation, and conditional independence. Journal of Educational and Behavioral Statistics, 21, 264–282. doi:10.3102/10769986021003264
41.
WassersteinR. L.SchirmA. L.LazarN. A. (2019a). Moving to a world beyond “p < 0.05.”The American Statistician, 73(Suppl. 1), 1–19. doi:10.1080/00031305.2019.1583913
42.
WassersteinR. L.SchirmA. L.LazarN. A. (2019b). Statistical inference in the 21st century: A world beyond p < 0.05 [Special issue]. The American Statistician, 73(Suppl. 1).
43.
WilkinsonL., & Task Force on Statistical Inference, American Psychological Association, Science Directorate. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604. doi:10.1037/0003-066X.54.8.594
