Restricted accessResearch articleFirst published online 2016-8
Meaningful Effect Sizes,Intraclass Correlations,and Proportions of Variance Explained by Covariates for Planning Two- and Three-Level Cluster Randomized Trials of Social and Behavioral Outcomes
There is a need for greater guidance regarding design parameters and empirical benchmarks for social and behavioral outcomes to inform assumptions in the design and interpretation of cluster randomized trials (CRTs).
Objectives:
We calculated the empirical reference values on critical research design parameters associated with statistical power for children’s social and behavioral outcomes, including effect sizes, intraclass correlations (ICCs), and proportions of variance explained by a covariate at different levels (R2).
Subjects:
Children from kindergarten to Grade 5 in the samples from four large CRTs evaluating the effectiveness of two classroom- and two school-level preventive interventions.
Measures:
Teacher ratings of students’ social and behavioral outcomes using the Teacher Observation of Classroom Adaptation–Checklist and the Social Competence Scale–Teacher.
Research design:
Two types of effect size benchmarks were calculated: (1) normative expectations for change and (2) policy-relevant demographic performance gaps. The ICCs and R2 were calculated using two-level hierarchical linear modeling (HLM), where students are nested within schools, and three-level HLM, where students were nested within classrooms, and classrooms were nested within schools.
Results and Conclusions:
Comprehensive tables of benchmarks and ICC values are provided to inform prevention researchers in interpreting the effect size of interventions and conduct power analyses for designing CRTs of children’s social and behavioral outcomes. The discussion also provides a demonstration for how to use the parameter reference values provided in this article to calculate the sample size for two- and three-level CRTs designs.
AguinisH.CulpepperS. A. (2015). An expanded decision-making procedure for examining cross-level interaction effects with multilevel modeling. Organizational Research Methods, 18, 155–176.
2.
BerkhofJ.KampenJ. K. (2004). Asymptotic effect of misspecification in the random part of the multilevel model. Journal of Educational and Behavioral Statistics, 29, 201–218.
BloomH. S.HillC. J.BlackA. B.LipseyM. W. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1, 289–328. doi:10.1080/19345740802400072
5.
BloomH. S.Richburg-HayesL.BlackA. R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29, 30–59.
BradshawC. P.KothC. W.ThorntonL. A.LeafP. J. (2009). Altering school climate through school-wide positive behavioral interventions and supports: Findings from a group-randomized effectiveness trial. Prevention Science, 10, 100–115.
8.
BradshawC. P.MitchellM. M.LeafP. J. (2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 133–148.
9.
BradshawC. P.PasE. T.GoldweberA.RosenbergM. S.LeafP. J. (2012). Integrating school-wide positive behavioral interventions and supports with tier 2 coaching to student support teams: The PBIS plus model. Advances in School Mental Health Promotion, 5, 177–193.
10.
BradshawC. P.WaasdorpT. E.LeafP. J. (2015). Examining variation in the impact of school-wide positive behavioral interventions and supports: Findings from a randomized controlled effectiveness trial. Journal of Educational Psychology, 107, 546–557.
11.
ChatterjiM. (2006). Reading achievement gaps, correlates, and moderators of early reading achievement: Evidence from the Early Childhood Longitudinal Study (ECLS) kindergarten to first grade sample. Journal of Educational Psychology, 98, 489–507.
12.
Conduct Problems Prevention Research Group. (2002). Evaluation of the first 3 years of the Fast Track prevention trial with children at high risk for adolescent conduct problems. Journal of Abnormal Child Psychology, 30, 19–35.
13.
CorriganA. (2003). Teacher social competence scale, grade 8/year 9 (Fast Track Project Technical Report). Retrieved from the Fast Track Project website, http://www.fasttrackproject.org
14.
De Los ReyesA.KazdinA. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131, 483.
15.
DongN.MaynardR. A. (2013). PowerUp!: A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6, 24–67. doi:10.1080/19345747.2012.673143
16.
DuckworthA. L.YeagerD. S. (2015). Measurement matters assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44, 237–251.
17.
Gifford-SmithM. (2000). Teacher social competence scale, grade 6/year 7 update (Fast Track Project Technical Report). Durham, NC: Duke University.
18.
HedgesL. V. (2011). Effect sizes in three-level cluster-randomized experiments. Journal of Educational and Behavioral Statistics, 36, 346–380.
19.
HedgesL. V.HedbergE. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87.
20.
HedgesL. V.HedbergE. C. (2013). Intraclass correlations and covariate outcome correlations for planning two- and three-level cluster-randomized experiments in education. Evaluation Review, 37, 445–489.
21.
HedgesL. V.HedbergE. C.KuyperA. M. (2012). The variance of intraclass correlations in three- and four-level models. Educational and Psychological Measurement, 72, 893–909. doi:10.1177/0013164412445193
22.
HedgesL. V.RhoadsC. (2010). Statistical power analysis in education research (NCSER 2010-3006). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. This report is available on the IES website athttp://ies.ed.gov/ncser/
23.
HillC. J.BloomH. S.BlackA. R.LipseyM. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2, 172–177.
24.
IalongoN. S.WerthamerL.KellamS. G.BrownC. H.WangS.LinY. (1999). Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and antisocial behavior. American Journal of Community Psychology, 27, 599–641.
KelceyB.PhelpsG. (2013). Strategies for improving power in school randomized studies of professional development. Evaluation Review, 37, 520–554.
27.
KellamS. G.LingX.MeriscaR.BrownC. H.IalongoN. (1998). The effect of the level of aggression in the first grade classroom on the course and malleability of aggressive behavior into middle school. Developmental Psychopathology, 10, 165–185.
28.
KellamS. G.MackenzieA.BrownC. H.PoduskaJ. M.WangW.PetrasH.WilcoxH. C. (2011). The Good Behavior Game and the future of prevention and treatment. Addiction Science & Clinical Practice, 73–84.
29.
KonstantopoulosS. (2008). The power of the test for treatment effects in three-level block randomized designs. Journal of Research on Educational Effectiveness, 1, 265–288.
30.
KothC. W.BradshawC. P.LeafP. J. (2009). Teacher Observation of Classroom Adaptation-Checklist: Development and factor structure. Measurement and Evaluation in Counseling and Development, 42, 15–30.
31.
LipseyM. W.PuzioK.YunC.HebertM. A.Steinka-FryK.ColeM. W.…BusickM. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms (NCSER 2013-3000). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education. Retrieved fromhttp://ies.ed.gov/ncser/
32.
MurrayD. W.RabinerD.CarrigM. (2014, 3). Grade level effects of the incredible years teacher training program on emotion regulation and attention. Paper presented at the 2014 meeting of the Society for Research on Educational Effectiveness, Washington, DC.
33.
National Research Council & Institute of Medicine. (2009). Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: The National Academies Press.
34.
PetrasH.ChilcoatH. D.LeafP. J.IalongoN. S.KellamS. G. (2004). Utility of TOCA-R scores during the elementary school years in identifying later violence among adolescent males. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 88–96.
35.
RaczS. J.KingK. M.WuJ.WitkiewitzK.McMahonR. J., & The Conduct Problems Prevention Research Group. (2013). The predictive utility of a brief kindergarten screening measure of child behavior problems. Journal of Consulting and Clinical Psychology, 81, 588–599. doi:10.1037/a0032366
36.
RaudenbushS. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185.
37.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
38.
RaudenbushS. W.MartinezA.SpybrookJ. (2007). Strategies for improving precision in group-randomized experiments. Educational Evaluation and Policy Analysis, 29, 5–29.
39.
RaudenbushS. W.SpybrookJ.CongdonR.LiuX.MartinezA.BloomH.HillC. (2011). Optimal design plus empirical evidence (Version 3.0). Retrieved January 20, 2015, fromhttp://www.wtgrantfoundation.org
40.
ReinkeW. M.HermanK. C.DongN. (2014, 3). A group randomized evaluation of the incredible years teacher training program. Paper presented at the Annual Meeting of Society for Research on Educational Effectiveness, Washington, DC.
41.
RosenthalR. (1994). Interpersonal expectancy effects: A 30-year perspective. Current Directions in Psychological Science, 3, 176–179.
42.
SAS Institute Inc. (2013). SAS/STAT® 13.1 User’s Guide. Cary, NC: SAS Institute Inc.
43.
SchaefferC. M.PetrasH.IalongoN.MasynK. E.HubbardS.PoduskaJ.SheppardK. A. (2006). Comparison of girls’ and boys’ aggressive-disruptive behavior trajectories across elementary school: Prediction to young adult antisocial outcomes. Journal of Consulting and Clinical Psychology, 74, 500–510.
44.
SchochetP. Z. (2008). Statistical power for randomized assignment evaluation of education programs. Journal of Educational and Behavioral Statistics, 33, 62–87.
45.
SpybrookJ.HedgesL.BorensteinM. (2014). Understanding statistical power in cluster randomized trials: Challenges posed by differences in notation and terminology. Journal of Research on Educational Effectiveness, 4, 384–406. doi:10.1080/19345747.2013.848963
46.
SpybrookJ.RaudenbushS. W. (2009). An examination of the precision and technical accuracy of the first wave of group-randomized trials funded by the Institute of Education Sciences. Educational Evaluation and Policy Analysis, 31, 298–318. doi:10.3102/0162373709339524
47.
StormshakE. A.BiermanK. L.BruschiC.DodgeK. A.CoieJ. D. (1999). Conduct problems prevention research group. The relation between behavior problems and peer preference in different classroom contexts. Child Development, 70, 169–182.
48.
VannemanA.HamiltonL.AndersonJ. B.RahmanT. (2009). Achievement gaps: How Black and White students in public schools perform in mathematics and reading on the national assessment of educational progress (Statistical Analysis Report. NCES 2009-455). Washington, DC: National Center for Education Statistics.
49.
Webster-StrattonC. (1998). Preventing conduct problems in Head Start children: Strengthening parenting competencies. Journal of Consulting and Clinical Psychology, 66, 715–730.
50.
Werthamer-LarssonL.KellamS.WheelerL. (1991). Effect of first-grade classroom environment on shy behavior, aggressive behavior, and concentration problems. American Journal of Community Psychology, 19, 585–602.
51.
WestineC. D.SpybrookJ.TaylorJ. A. (2013). An empirical investigation of variance design parameters for planning cluster-randomized trials of science achievement. Evaluation Review, 37, 490–519.
52.
ZhuP.JacobR.BloomH. S.XuZ. (2012). Designing and analyzing studies that randomize schools to estimate intervention effects on student academic outcomes without classroom-level information. Education Evaluation and Policy Analysis, 34, 45–68.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
