Multilevel modeling techniques are becoming more popular in handling data with multilevel structure in educational and behavioral research. Recently, researchers have paid more attention to cross-classified data structure that naturally arises in educational settings. However, unlike traditional single-level research, methodological studies about multilevel effect size have been rare and those that have recently appeared had an emphasis on strictly hierarchical data structure. This article extends the work on multilevel standardized mean differences from strictly hierarchical structure to both fully and partially cross-classified structures. Analytically derived formulae for calculating effect sizes and the corresponding sampling variances (or standard errors) are presented, verified by simulation results, and illustrated with real data examples. Implications for primary research studies and meta-analyses are discussed.
AhnS.MyersN. D.JinY. (2012). Use of the estimated intraclass correlation for correcting differences in effect size by level. Behavior Research Methods, 44, 490–502. doi:10.3758/s13428-011-0153-1
2.
American Educational Research Association. (2006). Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35, 33–40. doi:10.3102/0013189X035006033
3.
American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
4.
BauerD. J.SterbaS. K.HallforsD. D. (2008). Evaluating group-based interventions when control participants are ungrouped. Multivariate Behavioral Research, 43, 210–236. doi:10.1080/00273170802034810
5.
BeretvasS. N. (2008). Cross-classified random effects models. In O’ConnellA. A.McCoachD. B. (Eds.), Multilevel modeling of educational data (pp. 161–197). Charlotte, NC: Information Age.
6.
BeretvasS. N. (2010). Cross-classified and multiple membership models. In HoxJ. J.RobertsJ. K. (Eds.), Handbook of advanced multilevel analysis (pp. 313–334). New York, NY: Routledge.
7.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
8.
CohenJ. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003. doi:10.1037/0003-066X.49.12.997
9.
CoyneM. D.SimmonsD.Hagan-BurkeS.SimmonsL. E.KwokO.KimM.RawlinsonD. M. (2013). Adjusting beginning reading intervention based on student performance: An experimental evaluation. Exceptional Children, 80, 25–44.
10.
FergusonC. (2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40, 532–538. doi:10.1037/a0015808
11.
FriedelJ. M.CortinaK. S.TurnerJ. C.MidgleyC. (2010). Changes in efficacy beliefs in mathematics across the transition to middle school: Examining the effects of perceived teacher and parent goal emphases. Journal of Educational Psychology, 102, 102–114. doi:10.1037/a0017590
12.
GoldsteinH. (1986). Multilevel mixed linear model analysis using iterative generalized least squares. Biometrika, 73, 43–56. doi:10.2307/2336270
GrissomR. J.KimJ. J. (2012). Effect sizes for research: Univariate and multivariate applications (2nd ed.). New York, NY: Routledge.
15.
HedgesL. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational and Behavioral Statistics, 6, 107–128. doi:10.3102/10769986006002107
16.
HedgesL. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32, 341–370. doi:10.3102/1076998606298043
17.
HedgesL. V. (2009). Effect sizes in nested designs. In CooperH.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 337–355). New York, NY: Russell Sage Foundation.
18.
HedgesL. V. (2011). Effect sizes in three-level cluster-randomized experiments. Journal of Educational and Behavioral Statistics, 36, 346–380. doi:10.3102/1076998610376617
19.
HedgesL. V.HedbergE. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87. doi:10.3102/0162373707299706
20.
HoxJ. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.
21.
InglesS.AbrahamS. Y.RasinskiK. A.KarrR.SpencerB.FrankelM. (1990). National Educational Longitudinal Study of 1988: Base year student component data file user’s manual. Washington, DC: U. S. Department of Education.
22.
JohnsonI. Y. (2011). Contingent instructors and student outcomes: An artifact or a fact?Research in Higher Education, 52, 761–785. doi:10.1007/s11162-011-9219-2
23.
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
24.
LittellR. C.MillikenG. A.StroupW. W.WolfingerR. D. (1996). SAS system for mixed models. Cary, NC: SAS.
25.
LuoW.KwokO. (2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behavioral Research, 44, 182–212. doi:10.1080/00273170902794214
26.
LuoW.KwokO. (2010). Proportional reduction of prediction error in cross-classified random effects models. Sociological Methods & Research, 39, 188–205. doi:10.1177/0049124110384062
27.
MasonW. M.WongG. Y.EntwisleB. (1983). Contextual analysis through multilevel linear model. Sociological Methodology, 14, 72–103. doi:10.2307/270903
28.
MurrayD. M.BlitsteinJ. L. (2003). Methods to reduce the impact of intraclass correlation in group-randomized trials. Evaluation Review, 27, 79–103. doi:10.3102/0162373707299706
29.
MuthénL. K.MuthénB. O. (1998–2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.
30.
R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
31.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
32.
SearleS. R.CasellaG.McCullochC. E. (2006). Variance components. Hoboken, NJ: Wiley.
33.
ShiY.LeiteW.AlginaJ. (2010). The impact of omitting the interaction between crossed factors in cross-classified random effects modelling. British Journal of Mathematical and Statistical Psychology, 63, 1–15. doi:10.1348/000711008X398968
34.
SnijdersT. A. B.BoskerR. J. (1994). Modeled variance in two-level models. Sociological Methods & Research, 22, 342–363. doi:10.1177/0049124194022003004
35.
SnijdersT. A. B.BoskerR. J. (2011). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). London, England: Sage.
36.
ThompsonB. (2007). Effect sizes, confidence intervals, and confidence intervals for effect sizes. Psychology in the Schools, 44, 423–432. doi:10.1002/pits.20234
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
