Multisite field experiments using the (generalized) randomized block design that assign treatments to individuals within sites are common in education and the social sciences. Under this design, there are two possible estimands of interest and they differ based on whether sites or blocks have fixed or random effects. When the average treatment effect is assumed to be identical across sites, it is common to omit site by treatment interactions and “pool” them into the error term in classical experimental design. However, prior work has not addressed the consequences of pooling when site by treatment interactions are not zero. This study assesses the impact of pooling on inference in the presence of nonzero site by treatment interactions. We derive the small sample distributions of the test statistics for treatment effects under pooling and illustrate the impacts on rejection rates when interactions are not zero. We use the results to offer recommendations to researchers conducting studies based on the multisite design.
American Educational Research Association. (2006). Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35, 33–40. https://doi.org/10.3102/0013189X035006033.
2.
BancroftT. A.HanC. P. (1983). A note on pooling variances. Journal of the American Statistical Association, 78, 981–983.
3.
BloomH. S.RaudenbushS.W.WeissM. J.PorterK. (2017). Using multisite experiments to study cross-site variation in treatment effects: A hybrid approach with fixed intercepts and a random treatment coefficient. Journal of Research on Educational Effectiveness, 10(4), 917–842.
4.
BloomH. S.SpybrookJ. (2017). Assessing the precision of multisite trials for estimating the parameters of a cross-site population distribution of program effects. Journal of Research on Educational Effectiveness, 10(4), 877–902.
5.
BozivichH.BancroftT. A.HartleyH. O. (1956). Power of analysis of variance test procedures for certain incompletely specified models. Annals of Mathematical Statistics, 27, 1017–1043.
6.
CalinskiT.KageyamaS. (2000). Block designs: A randomization approach. Springer.
7.
CohenJ. (1977). Statistical power analysis for the behavioral sciences. Academic Press.
8.
DanielsH. E. (1939). The estimation of components of variance. Supplement to the Journal of the Royal Statistical Society, 6(2), 186–197.
9.
EichlerF. (1967). Limit theorems for regressions with unequal and dependent errors. In LeCamL. M.NeymanJ. (Eds.), Proceedings of the fifth Berkeley symposium on mathematical statistics and probability (vol. 1., pp. 59–82). University of California Press.
10.
EisenhartC. (1947). The assumptions underlying the analysis of variance. Biometrics, 3(1), 1–21.
11.
FisherR. A.MacKenzieW. A. (1922). Studies in crop variation II: The manurial response of different potato varieties. Journal of Agricultural Science, 13, 311–320.
12.
HanC. P.BancroftT. A. (1968). On pooling means when the variance is unknown. Journal of the American Statistical Association, 63,1333–1342.
13.
HedgesL. V. (2007a). Correcting a significance test for clustering. Journal of Educational and Behavioral Statistics, 32, 151–179.
14.
HedgesL. V. (2007b). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370.
15.
HedgesL. V.RhoadsC. (2009). Statistical power analysis in education research (NCSER 2010-3006). National Center for Special Education Research, Institute of Education Sciences, U.S. Department of Education.
16.
HendersonC. R. (1953). Estimation of variance and covariance components. Biometrics, 9(2), 226–252.
17.
HinesW. G. S. (1996). Pragmatics of pooling in ANOVA tables. The American Statistician, 50(2), 127–139.
18.
HuberP. (1967). The behavior of maximum-likelihood estimates under nonstandard conditions. In LeCamL. M.NeymanJ. (Eds.), Proceedings of the fifth Berkeley symposium on mathematical statistics and probability (pp. 221–233). University of California Press.
19.
KendallM. G.StuartA. (1968). The advanced theory of statistics (Vol. 3 Design and analysis and time series, 2nd ed.). Griffin.
KishL. (1965). Sampling organizations and groups of unequal sizes. American Sociological Review, 30, 564–572.
22.
MacKinnonJ. G.WhiteH. (1985). Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties. Journal of Econometrics, 29(3), 305–325.
23.
MeadR.BancroftT. A.HanC. P. (1973). Power of analysis of variance test procedures for incompletely specified models. Annals of Statistics, 3, 797–808.
24.
MiratrixL.WeissM.HendersonB. (2021). An applied researchers guide to estimating effects from multisite individually randomized trials: Estimands, estimators, and estimates. Journal of Research on Educational Effectiveness, 14, 270–308. https://dash.harvard.edu/handle/1/37366188.
25.
PaullA. E. (1950). On a preliminary test for pooling mean squares in the analysis of variance. Annals of Mathematical Statistics, 21, 539–556.
26.
RaudenbushS. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173–185.
27.
RaudenbushS. W.BloomH. (2015). Learning about and from a distribution of program impacts using multisite trials. American Journal of Evaluation, 36(4), 475–499.
28.
RaudenbushS. W.BrykA. S. (2002) Hierarchical linear models. Sage Publications.
29.
RaudenbushS. W.LiuX. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5(2), 199–213.
30.
SchefféH. (1959). The analysis of variance. Wiley.
31.
SchochetP. Z. (2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33(1), 62–87.
32.
SchochetP. Z. (2018). Design-based estimators for average treatment effects for multi-armed RCTs. Journal of Educational and Behavioral Statistics, 43(5), 568–593.
33.
SearleS. R. (1971). Linear models. John Wiley.
34.
SpybrookJ.RaudenbushS. W. (2009). An examination of the precision and technical accuracy of the first wave of group randomized trials funded by the Institute of Education Sciences. Educational Evaluation and Policy Analysis, 31(3), 298–316.
35.
StuartE. A.ColeS. R.BradshawC. P.LeafP. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. The Journal of the Royal Statistical Society, Series A, 174(2), 369–386.
36.
TangP. C. (1938). The power function of the analysis of variance tests with tables and illustrations of their use. Statistical Research Memoirs, 2, 126–149.
37.
TiptonE. (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38(3), 239–266.
38.
WeissM. J.BloomH. S.Verbitsky-SavitzN.GuptaH.VigilA. E.CullinanD. N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite trials. Journal of Research on Educational Effectiveness, 10, 843–876.
39.
WhiteH. (1980). A heteroscedastity-consistent covariance matrix and a direct test for heteroscedasticity. Econometrica, 48, 817–838.
40.
WindsorC.R.ClarkG.L. (1940). A statistical study of variation in the catch of plankton nets. Journal of Marine Research, 3, 1–34.
41.
Wolde-TsadiG.AfifiA. A. (1980). Comparison of sometimes pool, sometimes switch, and never pool procedures in the 2-way ANOVA random effects model. Technometrics, 22(3), 367–373.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
