Designs that facilitate inferences concerning both the total and indirect effects of a treatment potentially offer a more holistic description of interventions because they can complement “what works” questions with the comprehensive study of the causal connections implied by substantive theories. Mapping the sensitivity of designs to detect these effects is of critical importance because it directly governs the types of evidence researchers can bring to bear on theories of action under realistic sample sizes. In this study, we develop closed-form expressions to estimate the variance of and the power to detect causally defined indirect effects in two-level group-randomized studies examining individual-level mediators (i.e., 2-1-1 mediation). We formulate our approach within the purview of typical multilevel mediation models and anchor their interpretation in the potential outcomes framework. The results provide power analysis formulas that reduce calculations to simple functions of the primary path coefficients (e.g., treatment–mediator and mediator–outcome relationships) and common summary statistics (e.g., intraclass correlation coefficients). Probing these formulas suggests that group-randomized designs can be well powered to detect indirect effects when carefully planned. The power formulas are implemented in the PowerUp software (causalevaluation.org).
AllisonP. (1995). Exact variance of indirect effects in recursive linear models. Sociological Methodology, 25, 253–266.
2.
AroianL.TanejaV.CornwellL. (1978). Mathematical forms of the distribution of the product of two normal variables. Communications in Statistics: Theory and Methods, 7, 165–172.
3.
AvinC.ShipsterI.PearlJ. (2005). Identifiability of path-specific effects. Proceedings of the International Joint Conferences on Artificial Intelligence (pp. 357–363). Edinburgh, UK: Morgan-Kaufmann Publishers.
4.
BollenK. A. (1987). Total, direct, and indirect effects in structural equation models. Sociological Methodology, 17, 37–69.
5.
BollenK. A.StineR. (1990). Direct and indirect effects: Classical and bootstrap estimates of variability. Sociological Methodology, 20, 115–140. doi:10.2307/271084
6.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
7.
DongN.MaynardR. (2013). PowerUp!: A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6, 24–67.
8.
EndersC.TofighiD. (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12, 121–138.
9.
FritzM. S.MacKinnonD. P. (2007). Required sample size to detect the mediated effect. Psychological Science, 18, 233–239.
10.
HayesA. F.ScharkowM. (2013). The relative trustworthiness of inferential tests of the indirect effect in statistical mediation analysis: Does method really matter?Psychological Science, 24, 1918–1927.
11.
HedgesL.HedbergE. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87.
12.
HsiehF.BlochD.LarsenM. (1998). A simple method of sample size calculation for linear and logistic regression. Statistics in Medicine, 17, 1623–1634.
13.
HongG.RaudenbushS. W. (2006). Evaluating kindergarten retention policy: A case study of causal inference for multilevel observational data. Journal of the American Statistical Association, 101, 901–910.
14.
HongG.RaudenbushS. W. (2008). Causal inference for time-varying instructional treatments. Journal of Educational and Behavioral Statistics, 33, 333–362.
15.
Institute of Education Sciences, U.S. Department of Education & National Science Foundation. (2013). Common guidelines for education research and development (NSF 13-126). Retrieved February 15, 2014, fromhttp://ies.ed.gov/pdf/CommonGuidelines.pdf
16.
KelceyB. (2011). Assessing the effects of teachers’ reading knowledge on students’ achievement using multilevel propensity score stratification. Educational Evaluation and Policy Analysis, 33, 458–482.
17.
KelceyB.PhelpsG. (2013). Considerations for designing group randomized trials of professional development with teacher knowledge outcomes. Educational Evaluation and Policy Analysis, 35, 370–390.
18.
KennyD. A.JuddC. M. (2014). Power anomalies in testing mediation. Psychological Science, 25, 334–339.
19.
Kisbu-SakaryaY.MacKinnonD.MiocevicM. (2014). The distribution of the product explains normal theory mediation confidence interval estimation. Multivariate Behavioral Research, 49, 261–268.
20.
KreftI. G. G.de LeeuwJ.AikenL. S. (1995). The effect of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30, 1–21.
21.
MacKinnonD. P. (2008). Introduction to statistical mediation analysis. Mahwah, NJ: Lawrence Erlbaum.
22.
MacKinnonD. P.LockwoodC.BrownC.WangW.HoffmanJ. (2007). The intermediate endpoint effect in logistic and probit regression. Clinical Trials, 4, 499–513.
23.
MacKinnonD. P.LockwoodC.HoffmanJ. (1998). A new method to test for mediation. Paper presented at the Annual Meeting of the Society for Prevention Research, Park City, UT.
24.
MacKinnonD. P.LockwoodC.HoffmanJ.WestS.SheetsV. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7, 83–104.
OehlertG. W. (1992). A note on the delta method. The American Statistician, 46, 27–29.
27.
PituchK. A.StapletonL. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods & Research, 41, 630–670.
28.
PreacherK. J.SeligJ. P. (2012). Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures, 6, 77–98.
29.
RaudenbushS. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185.
30.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
31.
RaudenbushS. W.MartinezA.SpybrookJ. (2007). Strategies for improving precision in group randomized experiments. Educational Evaluation and Policy Analysis, 29, 5–29.
32.
RaudenbushS. W.SadoffS. (2008). Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness, 1, 138–154.
RowanB.CamburnE.BarnesC. (2004). Benefiting from comprehensive school reform: A review of research on CSR implementation. In CrossC. (Ed.), Putting the pieces together: Lessons from comprehensive school reform research (pp. 1–52). Washington, DC: National Clearinghouse for Comprehensive School Reform.
35.
SchochetP. Z. (2011). Do typical RCTs of education interventions have sufficient statistical power for linking impacts on teacher practice and student achievement outcomes?Journal of Educational and Behavioral Statistics, 36, 441–471.
36.
SobelM. E. (1982). Asymptotic confidence intervals for indirect effects in structural equation models. Sociological Methodology, 13, 290–312.
37.
SobelM. E. (2008). Identification of causal parameters in randomized studies with mediating variables. Journal of Educational and Behavioral Statistics, 33, 230–251.
38.
SpybrookJ.ShiR.KelceyB. (2016). Progress in the past decade: An examination of the precision of cluster randomized trials funded by the U.S. Institute of Education Sciences. International Journal of Research and Method in Education, 39, 255–267.
39.
VanderWeeleT. J. (2008). Ignorability and stability assumptions in neighborhood effects research. Statistics in Medicine, 27, 1934–1943.
40.
VanderWeeleT. J. (2010). Direct and indirect effects for neighborhood-based clustered and longitudinal data. Sociological Methods & Research, 38, 515–544.
41.
VanderWeeleT. J. (2015). Explanation in causal inference: Methods for mediation and interaction. New York, NY: Oxford University Press.
42.
VanderWeeleT. J.VansteelandtS. (2009). Conceptual issues concerning mediation, interventions and composition. Statistics and Its Interface, 2, 457–468.
43.
ZhangZ.ZyphurM.PreacherK. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12, 695–719.
