As any empirical method used for causal analysis, social experiments are prone to attrition which may flaw the validity of the results. This article considers the problem of partially missing outcomes in experiments. First, it systematically reveals under which forms of attrition—in terms of its relation to observable and/or unobservable factors—experiments do (not) yield causal parameters. Second, it shows how the various forms of attrition can be controlled for by different methods of inverse probability weighting (IPW) that are tailored to the specific missing data problem at hand. In particular, it discusses IPW methods that incorporate instrumental variables (IVs) when attrition is related to unobservables, which has been widely ignored in the experimental literature before.
AhnH.PowellJ. (1993). Semiparametric estimation of censored selection models with a nonparametric selection mechanism. Journal of Econometrics, 58, 3–29.
2.
AngristJ.BettingerE.KremerM. (2006). Long-term educational consequence of secondary school vouchers: Evidence from administrative records in Colombia. American Economic Review, 96, 847–862.
3.
AngristJ.LavyV. (2009). The effects of high stakes high school achievement awards: Evidence from a randomized trial. The American Economic Review, 99, 1384–1414.
4.
BarnardJ.FrangakisC. E.HillJ. L.RubinD. B. (2003). Principal stratification approach to broken randomized experiments: A case study of school choice vouchers in New York City. Journal of the American Statistical Association, 98, 299–311.
5.
BertrandM.DufloE.MullainathanS. (2004). How much should we trust differences-in-differences estimates?. The Quarterly Journal of Economics, 119, 249–275.
6.
BertrandM.MullainathanS. (2004). Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. The American Economic Review, 94, 991–1013.
7.
BloomH.OrrL.BellS.CaveG.DoolittleF.LinW.BosJ. (1997). The benefits and costs of JTPA title II-A programs: Key findings from the National Job Training Partnership Act study. Journal of Human Resources, 32, 549–576.
8.
BlundellR.PowellJ. (2003). Endogeneity in nonparametric and semiparametric regression models. In DewatripontL. H. M.TurnovskyS. (Eds.), Advances in economics and econometrics (pp. 312–357). Cambridge, UK: Cambridge University Press.
9.
BussoM.DiNardoJ.McCraryJ. (2009a). Finite sample properties of semiparametric estimators of average treatment effects. Working paper, University of California, Berkeley
10.
BussoM.DiNardoJ.McCraryJ. (2009b). New evidence on the finite sample properties of propensity score matching and reweighting estimators. IZA Discussion Paper No. 3998
11.
CardD. (1999). The causal effect of education on earnings. In AshenfelterO.CardD. (Eds.), Handbook of labor economics (pp. 1802–1863). Amsterdam, Netherlands: North-Holland.
12.
CastiglioniL.PforrK.KriegerU. (2008). The effect of incentives on response rates and panel attrition: Results of a controlled experiment. Survey Research Methods, 2, 151–158.
13.
CochranW. G.ChambersS. P. (1965). The planning of observational studies of human populations. Journal of the Royal Statistical Society Series A, 128, 234–265.
14.
CrumpR. K.HotzV. J.ImbensG. W.MitnikO. A. (2009). Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96, 187–199.
15.
DasM.NeweyW. K.VellaF. (2003). Nonparametric estimation of sample selection models. Review of Economic Studies, 70, 33–58.
16.
DiamondA.SekhonJ. S. (2006). . In Genetic matching for estimating causal effects: A general multivariate matching method for achieving balance in observational studies. Institute of Governmental Studies Working Paper. University of California, Berkeley.
17.
DiNardoJ.McCraryH.SanbonmatsuL. (2006). . In Constructive proposals for dealing with attrition: An empirical example. Ann Arbor: University of Michigan.. Working paper
18.
DufloE. (2006). Field experiments in development economics. In
BlundellR.NeweyW.PerssonT. (Eds.), Advances in economics and econometrics (Vol. 2, pp. 322–48). New York, NY: Cambridge University Press
19.
FinnH. D.AchillesC. M. (1999). Tennessee’s class size study: Findings, implications, misconceptions. Educational Evaluation and Policy Analysis, 21, 97–109.
20.
FisherR. (1925). . In Statistical methods for research workers. Edinburgh, UK: Oliver and Boyd.
21.
FisherR. (1935). . In The design of experiments. Edinburgh, UK: Oliver and Boyd.
22.
FitzgeraldJ.GottschalkP.MoffittR. (1998). An analysis of sample attrition in panel data: The Michigan panel study of income dynamics. The Journal of Human Resources, 33, 251–299.
23.
FrangakisC. E.RubinD. B. (2002). The defining role of principal stratification and effects for comparing treatments adjusted for posttreatment variables: From treatment noncompliance to surrogate endpoints. Biometrics, 58, 191–199.
24.
FreedmanD. (2006). Statistical models for causation: What inferential leverage do they provide. Evaluation Review, 30, 691–713.
25.
GertlerP. (2004). Do conditional cash transfers improve child health? Evidence form PROGRESA’s control randomized experiment. The American Economic Review, 94, 336–341.
26.
GriloC. M.MoneyR.BarlowD. H.GoddardA. W.GormanJ. M.HofmannS. G., … WoodsS. W. (1998). Pretreatment patient factors predicting attrition from a multicenter randomized controlled treatment study for panic disorder. Comprehensive Psychiatry, 39, 323–332.
27.
GroggerJ. (2009). Bounding the effects of social experiments: Accounting for attrition in administrative data. mimeo,
28.
HarrisonG. W.ListJ. A. (2004). Field experiments. Journal of Economic Literature, 42, 1009–1055.
29.
HausmanJ. A.WiseD. A. (1979). Attrition bias in experimental and panel date: The Gary income maintenance experiment. Econometrica, 47, 455–473.
30.
HeckmanJ. J. (1974). Shadow prices, market wages and labor supply. Econometric, 42, 679–694.
31.
HeckmanJ. J. (1976). The common structure of statistical models of truncation, sample selection and limited dependent variables and a simple estimator for such models. Annals of Economic and Social Measurement, 5, 475–492.
32.
HeckmanJ. J. (1979). Sample selection bias as a specification error. Econometrica, 47, 153–161.
33.
HeitjanD. F.BasuS. (1996). Distinguishing missing at random and missing completely at random. The American Statisticia, 50, 207–213.
34.
HiranoK.ImbensG. W.RidderG. (2003). Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71, 1161–1189.
35.
HorowitzJ.ManskiC. F. (1998). Censoring of outcomes and regressors due to survey nonresponse: Identification and estimation using weights and Imputations. Journal of Econometrics, 84, 37–58.
36.
HorowitzJ.ManskiC. F. (2000). Nonparametric analysis of randomized experiments with missing covariate and outcome data. Journal of the American Statistical Association, 95, 77–84.
37.
HorvitzD.ThompsonD. (1952). A generalization of sampling without replacement from a finite population. Journal of American Statistical Association, 47, 663–685.
38.
HuberB. (2009). . In Treatment evaluation in the presence of sample selection (Department of Economics Discussion Paper no. 2009–07). Switzerland: University of St. Gallen.
39.
HuberM.LechnerM.WunschC. (2010). . In How to control for many covariates? Reliable estimators based on the propensity score. IZA Discussion Paper no. 5268. Institute for the Study of Labor, Bonn, Germany
40.
ImaiK. (2009). Statistical analysis of randomized experiments with non-ignorable missing binary outcomes: An application to a voting experiment. Journal of the Royal Statistical Society Series C, 58, 83–104.
41.
ImbensG. W. (2004). Nonparametric estimation of average treatment effects under exogeneity: A review. The Review of Economics and Statistics, 86, 4–29.
42.
ImbensG. W. (2009). Better LATE than nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009). NBER Working Paper No. 14896. The National Bureau of Economic Research, Cambridge, MA
43.
ImbensG. W.NeweyW. (2009). Identification and estimation of triangular simultaneous equations models without additivity. Econometrica, 77, 1481–1512.
44.
ImbensG. W.WooldridgeJ. M. (2009). Recent developments in the econometrics of program evaluation. Journal of Economic Literature, 47, 5–86.
45.
KarlanD.ListJ. A. (2007). Does price matter in charitable giving? Evidence from a large-scale natural field experiment. The American Economic Review, 97, 1774–1793.
46.
KruegerA. B. (1999). Experimental estimates of education production functions. Quarterly Journal of Economics, 114, 497–532.
47.
KruegerA. B.ZhuP. (2004). Another look at the New York City school voucher experiment. American Behavioral Scientist, 47, 658–698.
48.
LeeD. S. (2009). Training, wages, and sample selection: Estimating sharp bounds on treatment effects. Review of Economic Studies, 76, 1071–1102.
49.
LeeS.WhangY.-J. (2010). Nonparametric tests of conditional treatment effects, . CeMMAP Working Paper CWP36/09. Centre for Microdata Methods and Practice, London, UK
50.
MealliF.PaciniB., (2008). Exploiting instrumental variables in causal inference with nonignorable outcome nonresponse using principal stratification. mimeo
51.
MulliganC. B.RubinsteinY. (2008). Selection, investment, and women’s relative wages over time. Quarterly Journal of Economics, 123, 1061–1110.
52.
NeweyW. K. (2007). Nonparametric continuous/discrete choice models. International Economic Review, 48, 1429–1439.
NeymanJ. (1923). On the application of probability theory to agricultural experiments. Essay on principles. Statistical Science, Reprint, 5, 463–480.
55.
RobinsJ.RotnitzkyA. (1995). Semiparametric efficiency in multivariate regression models with missing data. Journal of American Statistical Association, 90, 122–129.
56.
RobinsJ.RotnitzkyA.ZhaoL. (1995). Analysis of semiparametric regression models for repeated outcomes in the presence of missing data. Journal of American Statistical Association, 90, 106–121.
57.
RobinsJ.TsiatisA. (1991). Correcting for non-compliance in randomized trials using rank-preserving structural failure time models. Communications in Statistics, 20, 2069–2631.
58.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
59.
RotnitzkyA.RobinsJ. (1995). Semiparametric regression estimation in the presence of dependent censoring. Biometrika, 82, 805–820.
60.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.
61.
RubinD. B. (1976). Inference and missing data. Biometrika, 63, 581–592.
62.
RubinD. B. (1977). Formalizing subjective notions about the effect of nonrespondents in sample surveys. Journal of the American Statistical Association, 72, 538–543.
63.
RubinD. B. (1978). Multiple imputations in sample surveys—A phenomenological Bayesian approach to nonresponse. In Proceedings of the Survey Research Methods Section, American Statistical Association (pp. 20–34). Alexandria, VA: American Statistical Association.
64.
RubinD. B. (1990). Formal modes of statistical inference for causal effects. Journal of Statistical Planning and Inference, 25, 279–292.
65.
RubinD. B. (1996). Multiple imputation after 18+ Years. Journal of the American Statistical Association, 91, 473–489.
66.
RubinD. B. (2008). For objective causal inference, design trumps analysis. The Annals of Applied Statistics, 2, 808–840.
67.
ScharfsteinD. O.RotnitzkyA.RobinsJ. M. (1999). Adjusting for nonignorable drop-out using semiparametric nonresponse models. Journal of the American Statistical Association, 94, 1096–1120.
68.
SchochetP. Z.BurghardtJ.GlazermanS. (2001). . In National Job Corps study: The impacts of Job Corps on participants employment and related outcomes. Report. Washington, DC: Mathematica Policy Research.
69.
ShaikhA. M.SimonsenM.VytlacilE. J.YildizN. (2009). A specification test for the propensity score using its distribution conditional on participation. Journal of Econometrics, 151, 33–46.
70.
SianesiB. (2004). An evaluation of the Swedish system of active labour market programs in the 1990s. Review of Economics and Statistics, 86, 133–155.
71.
WooldridgeJ. (2002). Inverse probability weighted M-estimators for sample selection, attrition and stratification. Portuguese Economic Journal, 1, 141–162.
72.
WooldridgeJ. (2007). Inverse probability weighted estimation for general missing data problems. Journal of Econometrics, 141, 1281–1301.
73.
ZhangJ.RubinD. B. (2003). Estimation of causal effects via principal stratification when some outcome are truncated by death. Journal of Educational and Behavioral Statistics, 28, 353–368.
74.
ZhangJ.RubinD. B.MealliF. (2008). Evaluating the effects of job training programs on wages through principal stratification. In MillimetD.SmithJ.VytlacilE. (Eds.), Advances in econometrics: Modelling and evaluating treatment effects in econometrics (Vol. 21, pp. 117–145). New York, NY: Elsevier Science Ltd.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
