A common strategy for estimating treatment effects in observational studies using individual student-level data is analysis of covariance (ANCOVA) or hierarchical variants of it, in which outcomes (often standardized test scores) are regressed on pretreatment test scores, other student characteristics, and treatment group indicators. Measurement error in the prior test scores, which typically is both large and heteroscedastic, is regularly overlooked in empirical analyses and may erode the ability of regression models to adjust for student factors and may result in biased treatment effect estimates. We develop extensions of method-of-moments, Simulation-Extrapolation, and latent regression approaches to correcting for measurement error using the conditional standard errors of measure of test scores, and demonstrate their effectiveness relative to simpler alternatives using both simulation and a case study of teacher value-added effect estimation using longitudinal data from a large suburban school district.
BallouD. (2009). Test scaling and value-added measurement. Education Finance and Policy, 4, 351–383.
2.
BattauzM.BellioR.GoriE. (2011). Covariate measurement error adjustment for multilevel models with application to education data. Journal of Educational and Behavioral Statistics, 36, 283–306.
3.
BianconciniS.CagnoneS. (2012). A general multivariate latent growth model with applications to student achievement. Journal of Educational and Behavioral Statistics, 37, 339–364.
4.
BollenK. (1989). Structural equations with latent variables. New York, NY: John Wiley.
5.
BoydD.LankfordH.LoebS.WyckoffJ. (2012). Measuring test measurement error: A general approach. (National Bureau of Economic Research Working Paper 18010).
6.
BraunH. (2005). Using student progress to evaluate teachers: A primer on value-added models. Princeton, NJ: Educational Testing Service, Policy Information Center.
7.
BriggsD.DomingueB. (2011). Due diligence and the evaluation of teachers: A review of the value-added analysis underlying the effectiveness rankings of Los Angeles Unified School District teachers by the Los Angeles Times. Boulder, CO: National Education Policy Center.
8.
BriggsD.WeeksJ. (2009). The sensitivity of value-added modeling to the creation of a vertical score scale. Education Finance and Policy, 4, 384–414.
9.
CamilliG. (2006). Examination of a simple errors-in-variables model: A demonstration of marginal maximum likelihood. Journal of Educational and Behavioral Statistics, 31, 311–325.
10.
CarrollR.RuppertD.StefanskiL.CrainiceanuC. (2006). Measurement error in nonlinear models: A modern perspective (2nd ed.). London, England: Chapman and Hall.
11.
ChettyR.FriedmanJ.RockoffJ. (2013). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. (National Bureau of Economic Research Working Paper w19423). Cambridge, MA: National Bureau of Economic Research.
12.
ChoiK.SeltzerM. (2010). Modeling heterogeneity in relationships between initial status and rates of change: Treating latent variable regression coefficients as random coefficients in a three-level hierarchical model. Journal of Educational and Behavioral Statistics, 35, 54–91.
13.
CookJ.StefanskiL. (1994). Simulation-extrapolation estimation in parametric measurement error models. Journal of the American Statistical Association, 89, 1314–1328.
14.
CTB/McGraw-Hill. (2011). New York State Testing Program 2011: Mathematics, Grades 3-8. Technical Report. New York, NY: Author.
15.
CulpepperS.AguinisH. (2011). Using analysis of covariance (ANCOVA) with fallible covariates. Psychological Methods, 16, 166–178.
16.
DevanarayanV.StefanskiL. (2002). Empirical simulation extrapolation for measurement error models with replicate measurements. Statistics & Probability Letters, 59, 219–225.
17.
FoxJ.-P.GlasC. (2003). Bayesian modeling of measurement error in predictor variables using item response theory. Psychometrika, 68, 169–191.
18.
FullerW. (2006). Measurement error models (2nd ed.). New York, NY: John Wiley.
19.
GelmanA.RubinD. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–472.
HarrisD. (2011). Value-Added Measures in Education: What Every Educator Needs to Know. Cambridge, MA: Harvard Education Press.
22.
HarrisD.AndersonA. (2012, November). Bias of public sector worker performance monitoring: Theory and empirical evidence from middle school teachers. Paper presented at annual Association for Public Policy Analysis and Management conference, Baltimore, MD.
23.
HarrisD.SassT. (2006). Value-added models and the measurement of teacher quality. Unpublished manuscript, Florida State University, Tallahassee, FL.
24.
JunkerB.SchofieldL.TaylorL. (2012). The use of cognitive ability measures as explanatory variables in regression analysis. IZA Journal of Labor Economics, 1, 1–19.
25.
KaneT.McCaffreyD.MillerT.StaigerD. (2013). Have we identified effective teachers? Validating measures of effective teaching using random assignment. Bill and Melinda Gates Foundation MET Project Research Paper.
26.
KangJ.SchaferJ. (2007). Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 22, 523–539.
27.
KoedelC.BettsJ. (2007). Re-examining the role of teacher quality in the educational production function (Technical report). Columbia: University of Missouri.
28.
KoedelC.LeathermanR.ParsonsE. (2012). Test measurement error and inference from value-added models. The B.E. Journal of Economic Analysis and Policy, 12, 1–37.
29.
KolenM.ZengL.HansonB. (1996). Conditional standard errors of measurement for scale scores using IRT. Journal of Educational Measurement, 33, 129–140.
30.
LairdN. (1978). Nonparametric maximum likelihood estimation of a mixing distribution. Journal of the American Statistical Association, 73, 805–811.
31.
LehmannE. (1999). Elements of large-sample theory. New York, NY: Springer-Verlag.
32.
LockwoodJ.McCaffreyD. (2007). Controlling for individual heterogeneity in longitudinal models, with applications to student achievement. Electronic Journal of Statistics, 1, 223–252.
33.
LockwoodJ.McCaffreyD. (In press). Should nonlinear functions of test scores be used as covariates in a regression model? In LissitzR. (Ed.), Value-added modeling and growth modeling with particular application to teacher and school effectiveness. Charlotte, NC: Information Age.
34.
LordF. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
35.
LunnD.ThomasA.BestN.SpiegelhalterD. (2000). WinBUGS—A Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing, 10, 325–337.
36.
MartineauJ. (2006). Distorting value added: The use of longitudinal, vertically scaled student achievement data for value-added accountability. Journal of Educational and Behavioral Statistics, 31, 35–62.
37.
McCaffreyD.HanB.LockwoodJ. (2013). Using auxiliary teacher data to improve value-added: An application of small area estimation to middle school mathematics teachers. In LissitzR. (Ed.), Value-added modeling and growth modeling with particular application to teacher and school effectiveness. Charlotte, NC: Information Age.
38.
McCaffreyD.LockwoodJ.KoretzD.LouisT.HamiltonL. (2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29, 67–101.
39.
McCaffreyD.LockwoodJ.SetodjiC. (2013). Inverse probability weighting with error-prone covariates. Biometrika, 100, 671–680.
40.
MihalyK.McCaffreyD.LockwoodJ.SassT. (2010). Centering and reference groups for estimates of fixed effects: Modifications to felsdvreg. The Stata Journal, 10, 82–103.
41.
MillsapR. (1997). Invariance in measurement and prediction: Their relationship in the single-factor case. Psychological Methods, 2, 248–260.
42.
MillsapR. (2007). Invariance in measurement and prediction revisited. Psychometrika, 72, 461–473.
43.
MislevyR.JohnsonE.MurakiE. (1992). Scaling procedures in NAEP. Journal of Educational Statistics, 17, 131–154.
44.
PlummerM. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. Proceedings of the 3rd International Workshop on Distributed Statistical Computing (DSC 2003), Vienna, Austria.
45.
Rabe-HeskethS.PicklesA.SkrondalA. (2003). Correcting for covariate measurement error in logistic regression using nonparametric maximum likelihood estimation. Statistical Modelling, 3, 215–232.
46.
Rabe-HeskethS.SkrondalA.PicklesA. (2001). Maximum likelihood estimation of generalized linear models with covariate measurement error. The Stata Journal, 1, 1–26.
47.
Rabe-HeskethS.SkrondalA.PicklesA. (2004). GLLAMM Manual (U.C. Berkeley Division of Biostatistics Working Paper Series 1160). Berkeley, CA: Berkeley Electronic Press.
48.
RaudenbushS.BrykA. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Newbury Park, CA: Sage.
49.
ReckaseM. (2004). The real world is more complicated than we would like. Journal of Educational and Behavioral Statistics, 29, 117–120.
50.
RoederK.CarrollR.LindsayB. (1996). Nonparametric maximum likelihood estimation of a mixing distribution. Journal of the American Statistical Association, 91, 722–732.
51.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
52.
RothsteinJ. (2009). Student sorting and bias in value-added estimation: Selection on observables and unobservables. Education Finance and Policy, 4, 537–571.
53.
SchaferD.PurdyK. (1996). Likelihood analysis for errors-in-variables regression with replicate measurements. Biometrika, 83, 813–824.
54.
SchaferW. (2006). Growth scales as an alternative to vertical scales. Practical Assessment, Research & Evaluation, 11.
55.
ShangY. (2012). Measurement error adjustment using the SIMEX method: An application to student growth percentiles. Journal of Educational Measurement, 49, 446–465.
56.
SkrondalA.Rabe-HeskethS. (2004). Generalized latent variable modeling: Multilevel, longitudinal and structural equation models. Boca Raton, FL: Chapman and Hall/CRC Interdisciplinary Statistics.
Stata Corp. (2011). Stata Statistical Software: Release 12. College Station, TX: Author.
59.
Value Added Research Center. (2010). NYC teacher data initiative: Technical report on the NYC value-added model. Madison, WI: Wisconsin Center for Education Research.
60.
van der LindenW.HambletonR. (1996). Handbook of modern item response theory. New York, NY: Springer-Verlag.
61.
WansbeekT.MeijerE. (2000). Measurement error and latent variables in econometrics. New York, NY: Elsevier.
62.
WhiteH. (1982). Maximum likelihood estimation of misspecified models. Econometrica, 50, 1–26.
63.
WooldridgeJ. (2002). Econometric analysis of cross section and panel data. Cambridge, MA: MIT Press.
