In multisite trials, researchers are often interested in several inferential goals: estimating treatment effects for each site, ranking these effects, and studying their distribution. This study seeks to identify optimal methods for estimating these targets. Through a comprehensive simulation study, we assess two strategies and their combined effects: semiparametric modeling of the prior distribution and alternative posterior summary methods tailored to minimize specific loss functions. Our findings highlight that the success of different estimation strategies depends largely on the amount of within-site and between-site information available from the data. We discuss how our results can guide balancing the trade-offs associated with shrinkage in limited data environments.
AngristJ. D.HullP. D.PathakP. A.WaltersC. R. (2017). Leveraging lotteries for school value-added: Testing and estimation. The Quarterly Journal of Economics, 132(2), 871–919.
2.
AngristJ. D.HullP. D.WaltersC. R. (2022). Methods for measuring school effectiveness (Technical Report Working Paper No. 30803). National Bureau of Economic Research.
3.
AntonelliJ.TrippaL.HaneuseS. (2016). Mitigating bias in generalized linear mixed models: The case for Bayesian nonparametrics. Statistical Science: A Review Journal of the Institute of Mathematical Statistics, 31(1), 80.
4.
AntoniakC. E. (1974). Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. The Annals of Statistics, 2(6), 1152–1174.
Barrera-OsorioF.LindenL. L.SaavedraJ. E. (2019). Medium-and long-term educational consequences of alternative conditional cash transfer designs: Experimental evidence from Colombia. American Economic Journal: Applied Economics, 11(3), 54–91.
7.
BloomH. S.RaudenbushS. W.WeissM. J.PorterK. (2017). Using multisite experiments to study cross-site variation in treatment effects: A hybrid approach with fixed intercepts and a random treatment coefficient. Journal of Research on Educational Effectiveness, 10(4), 817–842.
8.
BloomH. S.WeilandC. (2015). Quantifying variation in head start effects on young children’s cognitive and socio-emotional skills using data from the National Head Start Impact Study (Technical report). MDRC.
9.
BorensteinM.HedgesL. V.HigginsJ. P.RothsteinH. R. (2009). Introduction to meta-analysis. John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470743386.
10.
BoomsmaA. (2013). Reporting Monte Carlo studies in structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 20(3), 518–540.
11.
BrownL. D. (2008). In-season prediction of batting averages: A field test of empirical Bayes and Bayes methodologies. Annals of Applied Statistics, 2(1), 113–152.
12.
ChettyR.FriedmanJ. N.RockoffJ. E. (2014). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593–2632.
13.
DorazioR. M. (2009). On selecting a prior for the precision parameter of Dirichlet process mixture models. Journal of Statistical Planning and Inference, 139(9), 3384–3390.
EscobarM. D.WestM. (1995). Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90(430), 577–588.
16.
GelmanA.CarlinJ. B.SternH. S.DunsonD. B.VehtariA.RubinD. B. (2013). Bayesian data analysis (3rd ed.). Chapman and Hall/CRC.
17.
GhideyW.LesaffreE.EilersP. (2004). Smooth random effects distribution in a linear mixed model. Biometrics, 60(4), 945–953.
18.
GhoshM. (1992). Constrained Bayes estimation with applications. Journal of the American Statistical Association, 87(418), 533–540.
19.
GilraineM.GuJ.McMillanR. (2020). A new method for estimating teacher value-added (Technical report). National Bureau of Economic Research.
20.
GoldsteinH.SpiegelhalterD. J. (1996). League tables and their limitations: Statistical issues in comparisons of institutional performance. Journal of the Royal Statistical Society: Series A (Statistics in Society), 159(3), 385–409.
21.
GuJ.KoenkerR. (2023). Invidious comparisons: Ranking and selection as compound decisions. Econometrica, 91(1), 1–41.
22.
ImbensG. W.RubinD. B. (2015). Causal inference in statistics, social, and biomedical sciences. Cambridge University Press.
23.
IshwaranH.ZarepourM. (2000). Markov chain Monte Carlo in approximate Dirichlet and beta two-parameter process hierarchical models. Biometrika, 87(2), 371–390.
24.
JiangW.ZhangC.-H. (2009). General maximum likelihood empirical Bayes estimation of normal means. The Annals of Statistics, 37(4), 1647–1684.
25.
KlineP.RoseE. K.WaltersC. R. (2022). Systemic discrimination among large US employers. The Quarterly Journal of Economics, 137(4), 1963–2036.
26.
KontopantelisE.ReevesD. (2012). Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A comparison between DerSimonian-Laird and restricted maximum likelihood. Statistical Methods in Medical Research, 21(6), 657–659.
27.
LeslieD. S.KohnR.NottD. J. (2007). A general approach to heteroscedastic linear regression. Statistics and Computing, 17, 131–146.
28.
LiuJ.DeyD. K. (2008). Skew random effects in multilevel binomial models: An alternative to nonparametric approach. Statistical Modelling, 8(3), 221–241.
29.
LockwoodJ.CastellanoK. E.ShearB. R. (2018). Flexible Bayesian models for inferences from coarsened, group-level achievement data. Journal of Educational and Behavioral Statistics, 43(6), 663–692.
30.
LockwoodJ.LouisT. A.McCaffreyD. F. (2002). Uncertainty in rank estimation: Implications for value-added modeling accountability systems. Journal of Educational and Behavioral Statistics, 27(3), 255–270.
31.
LuB.Ben-MichaelE.FellerA.MiratrixL. (2023). Is it who you are or where you are? Accounting for compositional differences in cross-site treatment effect variation. Journal of Educational and Behavioral Statistics, 48(4), 420–453.
32.
McCaffreyD. F.LockwoodJ. R.KoretzD.LouisT. A.HamiltonL. (2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29(1), 67–101.
33.
McCullochC. E.NeuhausJ. M. (2011). Misspecifying the shape of a random effects distribution: Why getting it wrong may not matter. Statistical Science, 26, 388–402.
34.
MeagerR. (2019). Understanding the average impact of microcredit expansions: A Bayesian hierarchical analysis of seven randomized experiments. American Economic Journal: Applied Economics, 11(1), 57–91.
35.
MiratrixL. W.WeissM. J.HendersonB. (2021). An applied researcher’s guide to estimating effects from multisite individually randomized trials: Estimands, estimators, and estimates. Journal of Research on Educational Effectiveness, 14(1), 270–308.
36.
MislevyR. J.BeatonA. E.KaplanB.SheehanK. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29(2), 133–161.
37.
MogstadM.RomanoJ. P.ShaikhA.WilhelmD. (2020). Inference for ranks with applications to mobility across neighborhoods and academic achievement across countries (Technical report). National Bureau of Economic Research.
38.
MountjoyJ.HickmanB. R. (2021). The returns to college(s): Relative value-added and match effects in higher education (Technical report). National Bureau of Economic Research.
39.
MurugiahS.SweetingT. (2012). Selecting the precision parameter prior in Dirichlet process mixture models. Journal of Statistical Planning and Inference, 142(7), 1947–1959.
40.
OhlssenD. I.SharplesL. D.SpiegelhalterD. J. (2007). A hierarchical modelling framework for identifying unusual performance in health care providers. Journal of the Royal Statistical Society: Series A (Statistics in Society), 170(4), 865–890.
41.
PaddockS. M.RidgewayG.LinR.LouisT. A. (2006). Flexible distributions for triple-goal estimates in two-stage hierarchical models. Computational Statistics & Data Analysis, 50(11), 3243–3262.
42.
PaganinS.PaciorekC. J.WehrhahnC.RodríguezA.Rabe-HeskethS.de ValpineP. (2023). Computational strategies and estimation performance with Bayesian semiparametric Item Response Theory models. Journal of Educational and Behavioral Statistics, 48(2), 147–188.
43.
PumaM.BellS.CookR.HeidC.ShapiroG.BroeneP.JenkinsF.MashburnA. J.DownerJ. T. (2010). Head start impact study: Final report (Technical report). Administration for Children & Families.
44.
Rabe-HeskethS.PicklesA.SkrondalA. (2003). Correcting for covariate measurement error in logistic regression using nonparametric maximum likelihood estimation. Statistical Modelling, 3(3), 215–232.
45.
Rabe-HeskethS.SkrondalA. (2022). Multilevel and longitudinal modeling using Stata, Volume I: Continuous responses (4th ed.). Stata Press.
46.
RaudenbushS. W.BloomH. S. (2015). Learning about and from a distribution of program impacts using multisite trials. American Journal of Evaluation, 36(4), 475–499.
47.
RaudenbushS. W.BrykA. S. (1985). Empirical Bayes meta-analysis. Journal of Educational Statistics, 10(2), 75–98.
48.
RubinD. B. (1981). Estimation in parallel randomized experiments. Journal of Educational Statistics, 6(4), 377–401.
49.
Rubio-AparicioM.Marin-MartinezF.Sanchez-MecaJ.Lopez-LopezJ. A. (2018). A methodological review of meta-analyses of the effectiveness of clinical psychology treatments. Behavior Research Methods, 50, 2057–2073.
50.
RudolphK. E.SchmidtN. M.GlymourM. M.CrowderR.GalinJ.AhernJ.OsypukT. L. (2018). Composition or context: Using transportability to understand drivers of site differences in a large-scale housing experiment. Epidemiology, 29(2), 199–206.
51.
SabolT. J.McCoyD.GonzalezK.MiratrixL.HedgesL.SpybrookJ. K.WeilandC. (2022). Exploring treatment impact heterogeneity across sites: Challenges and opportunities for early childhood researchers. Early Childhood Research Quarterly, 58, 14–26.
52.
SchochetO. N.PadillaC. M. (2022). Children learning and parents earning: Exploring the average and heterogeneous effects of Head Start on parental earnings. Journal of Research on Educational Effectiveness, 15(3), 413–444.
53.
ShenW.LouisT. A. (1998). Triple-goal estimates in two-stage hierarchical models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 60(2), 455–471.
54.
SkrondalA. (2000). Design and analysis of monte carlo experiments: Attacking the conventional wisdom. Multivariate Behavioral Research, 35(2), 137–167.
55.
SpybrookJ. (2014). Detecting intervention effects across context: An examination of the precision of cluster randomized trials. The Journal of Experimental Education, 82(3), 334–357.
56.
SpybrookJ.RaudenbushS. W. (2009). An examination of the precision and technical accuracy of the first wave of group-randomized trials funded by the institute of education sciences. Educational Evaluation and Policy Analysis, 31(3), 298–318.
57.
StuartE. A.ColeS. R.BradshawC. P.LeafP. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174(2), 369–386.
58.
TiptonE.SpybrookJ.FitzgeraldK. G.WangQ.DavidsonC. (2021). Toward a system of evidence for all: Current practices and future opportunities in 37 randomized trials. Educational Researcher, 50(3), 145–156.
59.
VerbekeG.LesaffreE. (1996). A linear mixed-effects model with heterogeneity in the random-effects population. Journal of the American Statistical Association, 91(433), 217–221.
60.
WeissM. J.BloomH. S.Verbitsky-SavitzN.GuptaH.VigilA. E.CullinanD. N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10(4), 843–876.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
