To assess the direct and indirect effect of an intervention, multilevel 2-1-1 studies with intervention randomized at the upper (class) level and mediator and outcome measured at the lower (student) level are frequently used in educational research. In such studies, the mediation process may flow through the student-level mediator (the within indirect effect) or a class-aggregated mediator (the contextual indirect effect). In this article, we cast mediation analysis within the counterfactual framework and clarify the assumptions that are needed to identify the within and contextual indirect effect. We show that unlike the contextual indirect effect, the within indirect effect can be unbiasedly estimated in linear models in the presence of unmeasured confounders of the mediator–outcome relationship at the upper level that exert additive effects on mediator and outcome. When unmeasured confounding occurs at the individual level, both indirect effects are no longer identified. We propose sensitivity analyses to assess the robustness of the within and contextual indirect effect under lower and upper-level confounding, respectively.
BaronR. M.KennyD. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173–1182. doi:10.1037/0022-3514.51.6.1173
2.
CastellanoK. E.Rabe-HeskethS.SkrondalA. (2014). Composition, context, and endogeneity in school and teacher comparisons. Journal of Educational and Behavioral Statistics, 39, 333–367. doi:10.3102/1076998614547576
3.
ColeD. A.PreacherK. J. (2014). Manifest variable path analysis: Potentially serious and misleading consequences due to uncorrected measurement error. Psychological Methods, 19, 300–315. doi:10.1037/a0033805
DavisonA. C. A. C.HinkleyD. V. (1997). Bootstrap methods and their application. Cambridge, England: Cambridge University Press.
6.
DemidenkoE. (2013). Mixed models: Theory and applications with r. Hoboken, NJ: John Wiley. doi:10.1002/9781118651537
7.
De NaeghelJ.Van KeerH.VansteenkisteM.HaerensL.AeltermanN. (in press). Promoting elementary school students autonomous reading motivation: Effects of a teacher professional development workshop. Journal of Educational Research, in press.
8.
De NaeghelJ.Van KeerH.VansteenkisteM.RosseelY. (2012). The relation between elementary students’ recreational and academic reading motivation, reading frequency, engagement, and comprehension: A self-determination theory perspective. Journal of Educational Psychology, 104, 1006–1021. doi:10.1037/a0027800
9.
GoetgelukS.VansteelandtS.GoetghebeurE. (2008). Estimation of controlled direct effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 70, 1049–1066. doi:10.1111/j.1467-9868.2008.00673.x
10.
HausmanJ. A.TaylorW. E. (1981). Panel data and unobservable individual effects. Econometrica, 49, 1377–1398. doi:10.2307/1911406
11.
HongG.RaudenbushS. W. (2006). Evaluating kindergarten retention policy. Journal of the American Statistical Association, 101, 901–910. doi:10.1198/016214506000000447
12.
HongG.RaudenbushS. W. (2008). Causal inference for time-varying instructional treatments. Journal of Educational and Behavioral Statistics, 33, 333–362. doi:10.3102/1076998607307355
13.
HoyleR. H.KennyD. A. (1999). Sample size, reliability, and tests of statistical mediation. In HoyleR. (Ed.), Statistical strategies for small sample research (pp. 195–222). Thousand Oaks, CA: Sage.
14.
ImaiK.KeeleL.TingleyD. (2010). A general approach to causal mediation analysis. Psychological Methods, 15, 309–334. doi:10.1037/a0020761
15.
KrullJ. L.MackinnonD. P. (2001). Multivariate behavioral multilevel modeling of individual and group level mediated effects. Multivariate Behavioral Research, 36, 249–277. doi:10.1207/S15327906MBR3602
16.
LedgerwoodA.ShroutP. E. (2011). The trade-off between accuracy and precision in latent variable models of mediation processes. Journal of Personality and Social Psychology, 101, 1174–1188. doi:10.1037/a0024776
17.
LiuJ.GustafsonP. (2008). On average predictive comparisons and interactions. International Statistical Review, 76, 419–432. doi:10.1111/j.1751-5823.2008.00056.x
18.
LoeysT.MoerkerkeB.RaesA.RosseelY.VansteelandtS. (2014). Estimation of controlled direct effects in the presence of exposure-induced confounding and latent variables. Structural Equation Modeling: A Multidisciplinary Journal, 21, 396–407. doi:10.1080/10705511.2014.915372
19.
LoeysT.MoerkerkeB.VansteelandtS. (2015). A cautionary note on the power of the test for the indirect effect in mediation analysis. Frontiers in Psychology, 5, 1–8. doi:10.3389/fpsyg.2014.01549
20.
LüdtkeO.MarshH. W.RobitzschA.TrautweinU.AsparouhovT.MuthénB. (2008). The multilevel latent covariate model: A new, more reliable approach to grouplevel effects in contextual studies. Psychological Methods, 13, 203–229. doi:10.1037/a0012869
21.
MacKinnonD. (2008). Introduction to statistical mediation analysis. New York, NY: Taylor & Francis Group.
22.
MoerkerkeB.LoeysT.VansteelandtS. (2015). Structural equation modeling versus marginal structural modeling for assessing mediation in the presence of post-treatment confounding. Psychological Methods, 20, 204–220. doi:10.1037/a0036368
23.
NagengastB.MarshH. W. (2012). Big fish in little ponds aspire more: Mediation and cross-cultural generalizability of school-average ability effects on self-concept and career aspirations in science. Journal of Educational Psychology, 104, 1033–1053. doi:10.1037/a0027697
24.
PearlJ. (2001). Direct and indirect effects. In BreeseJ.KollerD. (Eds.), Proceedings of the seventeenth conference on uncertainty in artificial intelligence (pp. 411–420). San Francisco, CA: Morgan Kaufmann.
25.
PearlJ. (2014). Interpretation and identification of causal mediation. Psychological Methods, 19, 459–481. doi:10.1037/a0036434
26.
PituchK. A.MurphyD. L.TateR. L. (2009). Three-level models for indirect effects in school- and class-randomized experiments in education. The Journal of Experimental Education, 78, 60–95. doi:10.1080/00220970903224685
27.
PituchK. A.StapletonL. M. (2008). The performance of methods to test upper-level mediation in the presence of nonnormal data. Multivariate Behavioral Research, 43, 237–267. doi:10.1080/00273170802034844
28.
PituchK. A.StapletonL. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods & Research, 41, 630–670. doi:10.1177/0049124112460380
29.
PreacherK. J. (2011). Multilevel SEM strategies for evaluating mediation in three-level data. Multivariate Behavioral Research, 46, 691–731. doi:10.1080/00273171.2011.589280
30.
PreacherK. J. (2015). Advances in mediation analysis: A survey and synthesis of new developments. Annual Review of Psychology, 66, 825–852. doi:10.1146/annurev-psych-010814-015258
31.
PreacherK. J.ZhangZ.ZyphurM. J. (2011). Alternative methods for assessing mediation in multilevel data: The advantages of multilevel SEM. Structural Equation Modeling: A Multidisciplinary Journal, 18, 161–182. doi:10.1080/10705511.2011.557329
32.
PreacherK. J.ZyphurM. J.ZhangZ. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15, 209–233. doi:10.1037/a0020141
R Core Team. (2015). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved fromhttp://www.R-project.org/
35.
RobinsJ. M.GreenlandS. (1992). Identifiability exchangeability effects for direct and indirect effects. Epidemiology, 3, 143–155. doi:10.1097/00001648-199203000-00013
36.
RubinD. B. (1976). Inference and missing data. Biometrika, 63, 581–592. doi:10.1093/biomet/63.3.581
37.
RubinD. B. (1980). Comment on: “Randomization analysis of experimental data in the Fisher randomization test” by D. Basu. Journal of the American Statistical Association, 75, 591–593. doi:10.1080/01621459.1980.10477517
38.
SearleS. R.CasellaG.McCullochC. E. (2006). Variance components (Vol. 391). New York, NY: John Wiley.
39.
ShroutP. E.BolgerN. (2002). Mediation in experimental and nonexperimental studies: New procedures and recommendations. Psychological Methods, 7, 422–445. doi:10.1037//1082-989X.7.4.422
40.
StefanskiL. A.BoosD. D. (2002). The calculus of m-estimation. The American Statistician, 56, 29–38. doi:10.1198/000313002753631330
41.
TchetgenE. J. T.VanderWeeleT. J. (2012). On causal inference in the presence of interference. Statistical Methods in Medical Research, 21, 55–75. doi:10.1177/0962280210386779
42.
TingleyD.YamamotoT.HiroseK.KeeleL.ImaiK. (2014). Mediation: R package for causal mediation analysis. Journal of Statistical Software, 59, 1–38. Retrieved fromhttp://www.jstatsoft.org/v59/i05/
43.
VanderWeeleT. J. (2008). Ignorability and stability assumptions in neighborhood effects research. Statistics in Medicine, 27, 1934–1943. doi:10.1002/sim
44.
VanderWeeleT. J. (2010). Direct and indirect effects for neighborhood-based clustered and longitudinal data. Sociological Methods & Research, 38, 515–544. doi:10.1177/0049124110366236
45.
VanderWeeleT. J. (2013). A three-way decomposition of a total effect into direct, indirect, and interactive effects. Epidemiology, 24, 224–232. doi:10.1097/EDE.0b013e318281a64e
46.
VanderWeeleT. J. (2015). Explanation in causal inference: Methods for mediation and interaction. New York, NY: Oxford University Press.
47.
VanderWeeleT. J.HongG.JonesS. M.BrownJ. L. (2013). Mediation and spillover effects in group-randomized trials: a case study of the 4Rs educational intervention. Journal of the American Statistical Association, 108, 469–482. doi:10.1080/01621459.2013.779832
48.
WigfieldA.EcclesJ. (2000). Expectancy-value theory of achievement motivation. Contemporary Educational Psychology, 25, 68–81. doi:10.1006/ceps.1999.1015
49.
ZhangZ.ZyphurM. J.PreacherK. J. (2009). Testing multilevel mediation using hierarchical linear models. Organizational Research Methods, 12, 695–719. doi:10.1177/1094428108327450
50.
ZyskindG. (1967). On canonical forms, non-negative covariance matrices and best and simple least squares linear estimators in linear models. The Annals of Mathematical Statistics, 38, 1092–1109. doi:10.1214/aoms/1177698779
51.
ZyskindG.MartinF. B. (1969). On best linear estimation and general Gauss-Markov theorem in linear models with arbitrary nonnegative covariance structure. SIAM Journal on Applied Mathematics, 17, 1190–1202. doi:http://dx.doi.org/10.1137/0117110
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
