Sage Journals: Discover world-class research

Abstract

This article discusses estimation of average treatment effects for randomized controlled trials (RCTs) using grouped administrative data to help improve data access. The focus is on design-based estimators, derived using the building blocks of experiments, that are conducive to grouped data for a wide range of RCT designs, including clustered and blocked designs, and models with weights and covariates. Because of the linearity of the regression model underlying RCTs, the asymptotic properties of design-based estimators using group-level averages—formed randomly or by covariates for nonclustered designs and as cluster-level averages for clustered designs—match those using individual data. Furthermore, design effects from aggregation are tolerable with moderate numbers of groups and few covariates, suggesting little information is lost in these cases. Ecological inference methods for subgroup analyses, however, yield large design effects. Several empirical examples using real-world education RCT data demonstrate the theory.

Keywords

administrative data randomized controlled trials data aggregation design-based estimators clustered designs blocked designs ecological inference

Get full access to this article

View all access options for this article.

References

Boruch

Reichen

(1975). Experimental testing of public policy. Boulder, CO: Westview.

Cameron

A. C.

Miller

D. L.

(2015). A practitioner’s guide to cluster-robust inference. Journal of Human Resources, 50, 317–372.

Campbell

M. K.

Mollison

Steen

Grimshaw

Eccles

(2000). Analysis of cluster randomized trials in primary care: A practical approach. Family Practice, 17, 192–196.

Card

Chetty

Feldstein

Saez

. (2010). Expanding access to administrative data for research in the U.S. (National Science Foundation 10-069 White Paper). Washington, DC: National Science Foundation.

Decker

Glazerman

Mayer

. (2004). The effects of teach for America on students: Findings from a national evaluation. Princeton, NJ: Mathematica Policy Research.

Dhrymes

Lleras-Muney

(2006). Estimation of models with group data by means of “2SLS.” Journal of Econometrics, 133, 1–29.

Feige

Watts

H. W.

(1972). An investigation of the consequences of partial aggregation of microeconomic data. Econometrica, 40, 343–360.

Fleiss

(1986). The design and analysis of clinical experiments. Chichester, England: Wiley.

Freedman

(2008). On regression adjustments to experimental data. Advances in Applied Mathematics, 40, 180–193.

10.

Freedman

Klein

S. P.

Ostland

Roberts

(1998). On “solutions” to the ecological inference problem (Statistics Department Working Paper). Berkeley: University of California.

11.

Gerber

Green

(2012). Field experiments: Design, analysis, and interpretation. New York, NY: W. W. Norton.

12.

Goodman

(1959). Some alternatives to ecological correlation. American Journal of Sociology, 64, 610–625.

13.

Holland

P. W.

(1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–960.

14.

Imbens

Rubin

(2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge, England: Cambridge University Press.

15.

Jacob

Goddard

Kim

E. S.

(2014). Assessing the use of aggregate data in the evaluation of school-based interventions: Implications for evaluation research and state policy regarding public use data. Educational Evaluation and Policy Analysis, 36, 44–66.

16.

Johnson

N. L.

Kotz

Balakrishnan

. (1994). Continuous univariate distributions (2nd ed.). New York, NY: Wiley Series in Probability and Statistics.

17.

King

(1997). A solution to the ecological inference problem. Princeton, NJ: Princeton University Press.

18.

Kish

(1995). Survey sampling. New York, NY: John Wiley.

19.

Ding

. (2017). General forms of finite population central limit theorems with applications to causal inference. Journal of the American Statistical Association, 112, 1759–1769.

20.

Liang

Zeger

(1986). Longitudinal data analysis using generalized linear models. Biometrika, 73, 13–22.

21.

Lin

(2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. Annals of Applied Statistics, 7, 295–318.

22.

Matthews

G. J.

Harel

(2011). Data confidentiality: A review of methods for statistical disclosure limitation and methods for assessing privacy. Statistics Surveys, 5, 1–29.

23.

Mayer

Peterson

Myers

Tuttle

Howell

(2002). School choice in New York City: An evaluation of the school choice scholarships program, Washington, DC: Mathematica Policy Research

24.

Miratrix

L. W.

Sekhon

J. B.

(2013). Adjusting treatment effect estimates in randomized experiments. Journal of the Royal Statistical Society B, 75, 369–396.

25.

Murray

D. M.

(1998). The design and analysis of group-randomized trials. Oxford, England: Oxford University Press.

26.

Neyman

(1923). On the application of probability theory to agricultural experiments: Essay on principles. Section 9, Translated in Statistical Science, 5, 465–472. (Original work published 1990)

27.

Prais

S. J.

Aitchison

(1954). The grouping of observations in regression analysis. Review of the International Statistical Institute, 22, 1–22.

28.

Robinson

W. S.

(1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357.

29.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Education Psychology, 66, 688–701.

30.

Rubin

D. B.

(1977). Assignment to treatment group on the basis of a covariate. Journal of Education Statistics, 2, 1–26.

31.

Rubin

D. B.

(1986). Which ifs have causal answers? Discussion of Holland’s “Statistics and causal inference.” Journal of the American Statistical Association, 81, 961–962.

32.

Schochet

P. Z.

(2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33, 62–87.

33.

Schochet

P. Z.

(2010). Is regression adjustment supported by the Neyman model for causal inference? Journal of Statistical Planning and Inference, 140, 246–259.

34.

Schochet

P. Z.

(2013). Estimators for clustered education RCTs using the Neyman model for causal inference. Journal of Educational and Behavioral Statistics, 38, 219–238.

35.

Schochet

P. Z.

(2016). Statistical theory for the RCT-YES software: Design-based causal inference for RCTs (NCEE 2015–4011). Washington, DC: Institute of Education Sciences, U.S. Department of Education. (Original work published 2015)

36.

Schochet

P. Z.

Kautz

(2018). Design-based estimators for clustered RCTs and how they compare to robust estimators (Mathematica Policy Research Working Paper). Princeton, NJ: Mathematica.

37.

Stoker

(1993). Empirical approaches to the problem of aggregation over individuals. Journal of Economic Literature, 31, 1827–1874.

38.

U.S. Office of Management and Budget. (2018). Analytical Perspectives: Budget of the U.S. Government Fiscal Year 2018. Retrieved from https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/budget/fy2018/spec.pdf

39.

Wang

Chen

Schifano

Yan

(2016). Statistical methods and computing for big data. Statistical Interface, 9, 399–414.

40.

Yang

Tsiatis

(2001). Efficiency study of estimators for a treatment effect in a pretest-posttest trial. American Statistician, 55, 314–321.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.21 MB

0.00 MB