A content analysis of 2 years of Psychological Science articles reveals inconsistencies in how researchers make inferences about indirect effects when conducting a statistical mediation analysis. In this study, we examined the frequency with which popularly used tests disagree, whether the method an investigator uses makes a difference in the conclusion he or she will reach, and whether there is a most trustworthy test that can be recommended to balance practical and performance considerations. We found that tests agree much more frequently than they disagree, but disagreements are more common when an indirect effect exists than when it does not. We recommend the bias-corrected bootstrap confidence interval as the most trustworthy test if power is of utmost concern, although it can be slightly liberal in some circumstances. Investigators concerned about Type I errors should choose the Monte Carlo confidence interval or the distribution-of-the-product approach, which rarely disagree. The percentile bootstrap confidence interval is a good compromise test.
Aptech Systems. (2012). GAUSS (Version 12) [Computer software]. Black Diamond, WA: Aptech Systems. Available from http://www.statmodel.com
2.
AroianL. A. (1947). The probability function of the product of two normally distributed variables. Annals of Mathematical Statistics, 18, 265–271.
3.
BaronR.KennyD. A. (1986). The moderator-mediator variable distinction in social psychology: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182.
4.
BiesanzJ. C.FalkC.SavaleiV. (2010). Assessing mediational models: Testing and interval estimation for indirect effects. Multivariate Behavioral Research, 45, 661–701.
5.
CapraraG. V.AlessandriG.EisenbergN. (2012). Prosociality: The contribution of traits, values, and self-efficacy beliefs. Journal of Personality and Social Psychology, 102, 1289–1303.
6.
EdwardsJ. R.LambertL. S. (2007). Methods for integrating moderation and mediation: A general analytical framework using moderated path analysis. Psychological Methods, 12, 1–22.
7.
FritzM. S.MacKinnonD. P. (2007). Required sample size to detect the mediated effect. Psychological Science, 18, 233–239.
8.
FritzM. S.TaylorA. B.MacKinnonD. P. (2012). Explanation of two anomalous results in statistical mediation analysis. Multivariate Behavioral Research, 47, 61–87.
9.
GuendelmanM. D.CheryanS.MoninB. (2011). Fitting in but getting fat: Identity threat and dietary choices among U.S. immigrant groups. Psychological Science, 22, 959–967.
10.
HayesA. F. (2013). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. New York, NY: Guilford Press.
11.
HayesA. F.CaiL. (2007). Using heteroscedasticity-consistent standard error estimators in OLS regression: An introduction and software implementation. Behavior Research Methods, 39, 709–722.
12.
HuangJ. Y.SedlovskayaA.AckermanJ. M.BarghJ. A. (2011). Immunizing against prejudice: Effects of disease protection on attitudes toward out-groups. Psychological Science, 22, 1550–1556.
13.
JohnsonI. R.FujitaK. (2012). Change we can believe in: Using perceptions of changeability to promote system-change motives over system-justification motives in information search. Psychological Science, 22, 133–140.
14.
JuddC. M.KennyD. A. (2010). Data analysis in social psychology: Recent and recurring issues. In FiskeS. T.GilbertD. T.LindzeyG. (Eds.), Handbook of social psychology (Vol. 1, pp. 115–139). New York, NY: Wiley.
15.
KelleyK. (2007). Methods for the behavioral, educational, and social sciences: An R package. Behavior Research Methods, 39, 979–984.
16.
LongJ. S.ErvinL. H. (2000). Using heteroscedasticity consistent standard errors in the linear regression model. The American Statistician, 54, 217–224.
17.
MacKinnonD. P. (2008). An introduction to statistical mediation analysis. Mahwah, NJ: Erlbaum.
18.
MacKinnonD. P.FritzM. S.WilliamsJ.LockwoodC. M. (2007). Distribution of the product confidence limits for the indirect effect: Program PRODCLIN. Behavior Research Methods, 39, 384–389.
19.
MacKinnonD. P.LockwoodC. M.HoffmanJ. M.WestS. G.SheetsV. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7, 83–104.
20.
MacKinnonD. P.LockwoodC. M.WilliamsJ. (2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39, 99–128.
21.
MathieuJ. E.TaylorS. R. (2006). Clarifying conditions and decision points for mediational type inferences in organizational behavior. Journal of Organizational Behavior, 27, 1031–1056.
22.
MinsonJ. A.MuellerJ. S. (2012). The cost of collaboration: Why joint decision making exacerbates rejection of outside information. Psychological Science, 23, 219–224.
23.
MuthénL. K.MuthénB. O. (2011). Mplus user’s guide (6th ed.). Los Angeles, CA: Author.
24.
NaJ.KitayamaS. (2011). Spontaneous trait inference is culture-specific: Behavioral and neural evidence. Psychological Science, 22, 1025–1032.
25.
NevilleL. (2012). Do economic equality and generalized trust inhibit academic dishonesty? Evidence from state-level search-engine queries. Psychological Science, 23, 339–345.
26.
OishiS.SchimmackU.DienerE. (2012). Progressive taxation and the subjective well-being of nations. Psychological Science, 23, 86–92.
27.
PreacherK. J.HayesA. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments, & Computers, 36, 717–731.
28.
PreacherK. J.HayesA. F. (2008). Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behavior Research Methods, 40, 879–891.
29.
PreacherK. J.RuckerD. D.HayesA. F. (2007). Assessing moderated mediation hypotheses: Theory, methods, and prescriptions. Multivariate Behavioral Research, 42, 185–227.
30.
PreacherK. J.SeligJ. P. (2012). Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures, 6, 77–98.
31.
R Development Core Team. (2012). R: A language and environment for statistical computing (Version 2.14). Vienna, Austria: R Foundation for Statistical Computing. Available from http://www.R-project.org
32.
RindermannH.ThompsonJ. (2011). Cognitive capitalism: The effect of cognitive ability on wealth, as mediated through scientific achievement and economic freedom. Psychological Science, 22, 754–763.
33.
ShroutP. E.BolgerN. (2002). Mediation in experimental and nonexperimental studies: New procedures and recommendations. Psychological Methods, 7, 422–445.
34.
SobelM. E. (1982). Asymptotic confidence intervals for indirect effects in structural equation models. In LeinhartS. (Ed.), Sociological methodology (pp. 290–312). San Francisco, CA: Jossey-Bass.
35.
SpencerS. J.ZannaM. P.FongG. T. (2005). Establishing a causal chain: Why experiments are often more effective than mediational analysis in examining psychological processes. Journal of Personality and Social Psychology, 89, 845–851.
36.
SweenyK.VohsK. D. (2012). On near misses and completed tasks: The nature of relief. Psychological Science, 23, 464–468.
37.
TofighiD.MacKinnonD. P. (2011). RMediation: An R package for mediation analysis confidence intervals. Behavior Research Methods, 43, 692–700.
38.
WickensC. M.WiesenthalD. L.FloraD. B.FlettG. L. (2011). Understanding driving anger and aggression: Attributional theory in the driving environment. Journal of Experimental Psychology: Applied, 17, 354–370.
39.
WilliamsJ.MacKinnonD. P. (2008). Resampling and distribution of the product methods for testing indirect effects in complex models. Structural Equation Modeling, 15, 23–51.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
