Sage Journals: Discover world-class research

Abstract

Over the past ten years, propensity score methods have made an important contribution to improving generalizations from studies that do not select samples randomly from a population of inference. However, these methods require assumptions and recent work has considered the role of bounding approaches that provide a range of treatment impact estimates that are consistent with the observable data. An important limitation to bound estimates is that they can be uninformatively wide. This has motivated research on the use of propensity score stratification to narrow bounds. This article assesses the role of distributional overlap in propensity scores on the effectiveness of stratification to tighten bounds. Using the results of two simulation studies and two case studies, I evaluate the relationship between distributional overlap and precision gain and discuss the implications when propensity score stratification is used as a method to improve precision in the bounding framework.

Keywords

propensity score overlap bounds stratification generalizability

Get full access to this article

View all access options for this article.

References

Chan

(2017). Partially identified treatment effects for generalizability. Journal of Research on Educational Effectiveness, 10(3), 646–669.

Chan

(2019). An evaluation of bounding approaches for generalization. Journal of Experimental Education. Advanced online publication. https://doi.org/10.1080/00220973.2019.1697191

Cochran

W. G.

(1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24, 295–313.

Cole

S. R.

Stuart

E. A.

(2010). Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. American Journal of Epidemiology, 172(1), 107–115.

Imai

King

Stuart

E. A.

(2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society: Series A (Statistics in Society), 171, 481–502.

Konstantopoulos

Miller

S. R.

van der Ploeg

(2013). The impact of Indiana’s system of interim assessments on mathematics and reading achievement. Educational Evaluation and Policy Analysis, 35, 481–499.

Lee

D. S.

(2009). Training, wages, and sample selection: Estimating sharp bounds on treatment effects. The Review of Economic Studies, 76(3), 1071–1102.

Lohr

S. L.

(2009). Sampling: design and analysis. CRC Press.

Long

D. M.

Hudgens

M. G.

(2013). Sharpening bounds on principal effects with covariates. Biometrics, 69(4), 812–819.

10.

Manski

C. F.

(1990). Nonparametric bounds on treatment effects. The American Economic Review, 80(2), 319–323.

11.

Miratrix

Furey

Feller

Grindal

Page

L. C.

(2018). Bounding, an accessible method for estimating principal causal effects, examined and explained. Journal of Research on Educational Effectiveness, 11(1), 133–162.

12.

Olsen

R. B.

Orr

L. L.

Bell

S. H.

Stuart

E. A.

(2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32(1), 107–121.

13.

O’Muircheartaigh

Hedges

L. V.

(2014). Generalizing from unrepresentative experiments: a stratified propensity score approach Journal of the Royal Statistical Society: Series C (Applied Statistics) 63 195–210

14.

Roschelle

Shechtman

Tatar

Hegedus

Hopkins

Empson

Knudsen

Gallagher

L. P.

(2010). Integration of technology, curriculum, and professional development for advancing middle school mathematics: Three large-scale studies. American Educational Research Journal, 47(4), 833–878.

15.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.

16.

Rubin

D. B.

(1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.

17.

Rubin

D. B.

(1977). Assignment to treatment group on the basis of a covariate. Journal of Educational Statistics, 2(1), 1–26.

18.

Rubin

D. B.

(1978). Bayesian inference for causal effects: The role of randomization. The Annals of Statistics, 6(1), 34–58.

19.

Rubin

D. B.

(1980). Randomization analysis of experimental data: The Fisher randomization test comment. Journal of the American Statistical Association, 75(371), 591–593.

20.

Rubin

D. B.

(1986). Statistics and causal inference: Comment: Which ifs have causal answers. Journal of the American Statistical Association, 81, 961–962.

21.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

22.

Stuart

E. A.

Cole

S. R.

Bradshaw

C. P.

Leaf

P. J.

(2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174, 369–386.

23.

Tipton

(2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38(3), 239–266.

24.

Tipton

(2014). How generalizable is your experiment? Comparing a sample and population through a generalizability index. Journal of Educational and Behavioral Statistics, 39, 478–501.

25.

Tipton

Hallberg

Hedges

L. V.

Chan

(2017). Implications of small samples for generalization: Adjustments and rules of thumb. Evaluation Review, 41(5), 472–505.

26.

Tipton

Hedges

Vaden-Kiernan

Borman

Sullivan

Caverly

(2014). Sample selection in randomized experiments: A new method using propensity score stratified sampling. Journal of Research on Educational Effectiveness, 7(1), 114–135.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.02 MB

0.00 MB