While most educators assume that not all students try their best on achievement tests, no current research examines if behaviors associated with low test effort, like rapidly guessing on test items, affect teacher value-added estimates. In this article, we examined the prevalence of rapid guessing to determine if this behavior varied by grade, subject, and teacher, and evaluated if rapid guessing influenced teacher value-added estimates. We observed differences in rapid guessing across grades, subjects, and teachers; however, this behavior did not appear to have a substantive effect on teacher value-added estimates. These findings suggest that rapid guessing occurs frequently enough that educators should be mindful of its effect on the interpretations of student test results. However, based on the value-added specifications used in this research, this type of behavior did not appear to affect estimates of teacher performance.
BradlowE. T.WeissR. E.ChoM. (1998). Bayesian identification of outliers in computerized adaptive testing. Journal of the American Statistical Association, 93, 910–919.
2.
CaoJ.StokesS. L. (2008). Bayesian IRT guessing models for partial guessing behaviors. Psychometrika, 73, 209–230.
3.
DeeT. S.WyckoffJ. (2015). Incentives, selection, and teacher performance: Evidence from IMPACT. Journal of Policy Analysis and Management, 34, 267–297.
4.
DohertyK. M.JacobsS. (2013). Connect the dots: Using evaluations of teacher effectiveness to inform policy and practice. State of the States 2013. Washington, DC: National Council on Teacher Quality.
5.
EklöfH.PavešičB. J.GrønmoL. S. (2014). A cross-national comparison of reported effort and mathematics performance in TIMSS advanced. Applied Measurement in Education, 27, 31–45.
6.
FullerW. A. (2009). Measurement error models (Vol. 305). New York, NY: John Wiley.
7.
GoldhaberD.ChaplinD. D. (2015). Assessing the “Rothstein Falsification Test”: Does it really show teacher value-added models are biased?Journal of Research on Educational Effectiveness, 8, 8–34.
8.
GuarinoC. M.MaxfieldM.ReckaseM. D.ThompsonP. N.WoolridgeJ. M. (2015). An evaluation of empirical Bayes’s estimation of value-added teacher performance measures. Journal of Educational and Behavioral Statistics, 40, 190–222.
9.
HanushekE. A. (2011). The economic value of higher teacher quality. Economics of Education Review, 30, 466–479.
10.
HanushekE. A.KainJ. F.O’BrienD. M.RivkinS. G. (2005). The market for teacher quality (No. w11154). Cambridge, MA: National Bureau of Economic Research.
11.
HerrmannM.WalshE.IsenbergE.ReschA. (2013). Shrinkage of value-added estimates and characteristics of students with hard-to-predict achievement levels. Washington, DC: Mathematica Policy Research.
12.
KalogridesD.LoebS.BéteilleT. (2013). Systematic sorting teacher characteristics and class assignments. Sociology of Education, 86, 103–123.
13.
KaneT. J.StaigerD. O. (2008). Estimating teacher impacts on student achievement: An experimental evaluation (No. w14607). Cambridge, MA: National Bureau of Economic Research.
14.
KoedelC.BettsJ. R. (2011). Does student sorting invalidate value-added models of teacher effectiveness? An extended analysis of the Rothstein critique. Education, 6(1), 18–42.
15.
KongX. J.WiseS. L.BholaD. S. (2007). Setting the response time threshold parameter to differentiate solution behavior from rapid-guessing behavior. Educational and Psychological Measurement, 67, 606–619.
16.
LockwoodJ. R.McCaffreyD. F. (2014). Correcting for test score measurement error in ANCOVA models for estimating treatment effects. Journal of Educational and Behavioral Statistics, 39, 22–52.
LoebS.SolandJ.FoxL. (2014). Is a good teacher a good teacher for all? Comparing value-added of teachers with their English learners and non-English learners. Educational Evaluation and Policy Analysis, 36, 457–475.
19.
McCaffreyD. F.LockwoodJ. R.KoretzD.LouisT. A.HamiltonL. (2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29, 67–101.
20.
MeijerR. R.SijtsmaK. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107–135.
21.
PauflerN. A.Amrein-BeardsleyA. (2014). The random assignment of students into elementary classrooms: Implications for value-added analyses and interpretations. American Educational Research Journal, 51, 328–362.
22.
RaoJ. N.MolinaI. (2015). Small area estimation. New York, NY: John Wiley.
23.
RiosJ. A.GuoH.MaoL.LiuO. L. (2017). Evaluating the impact of careless responding on aggregated-scores: To filter unmotivated examinees or not?International Journal of Testing, 17, 74–104.
24.
RiosJ. A.LiuO. L.BridgemanB. (2014). Identifying low-effort examinees on student learning outcomes assessment: A comparison of two approaches. New Directions for Institutional Research, 2014(161), 69–82.
25.
RothsteinJ. (2009). Student sorting and bias in value-added estimation: Selection on observables and unobservables. Education, 4, 537–571.
26.
SetzerJ. C.WiseS. L.van den HeuvelJ. R.LingG. (2013). An investigation of examinee test-taking effort on a large-scale assessment. Applied Measurement in Education, 26, 34–49.
27.
SolandJ.JensenN.KeysT.BiS.WolkE. (2017). Are test and academic engagement related? A case study using rapid guessing. Manuscript submitted for publication.
28.
van Krimpen-StoopE. M. L. A.MeijerR. R. (2000). Detecting person-misfit in adaptive testing using statistical process control techniques. In van der LindenW. J.GlasC. A. W. (Eds.), Computerized adaptive testing: Theory and practice (pp. 201–219). Boston, MA: Kluwer.
29.
WiseS. L. (2006). An investigation of the differential effort received by items on a low-stakes Computer-Based Test. Applied Measurement in Education, 19, 95–114.
30.
WiseS. L. (2015). Effort analysis: Individual score validation of achievement test data. Applied Measurement in Education, 28, 237–252.
31.
WiseS. L.DeMarsC. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10, 1–17.
32.
WiseS. L.DeMarsC. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43, 19–38.
33.
WiseS. L.DeMarsC. E. (2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15, 27–41.
34.
WiseS. L.KingsburyG. G. (2016). Modeling student test-taking motivation in the context of an Adaptive Achievement Test. Journal of Educational Measurement, 53, 86–105.
35.
WiseS. L.KingsburyG. G.HauserC. (2009, April). How do I know that this score is valid? The case for assessing individual score validity. Paper presented at the meeting of the National Council on Measurement in Education, San Diego, CA.
36.
WiseS. L.KongX. (2005). Response time effort: A new measure of examinee motivation in Computer-Based Tests. Applied Measurement in Education, 18, 163–183.
37.
WiseS. L.MaL. (2012, April). Setting respond time thresholds for a CAT item pool: The normative threshold method. Paper presented at the meeting of the National Council on Measurement in Education, Vancouver, British Columbia, Canada.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
