Contrary to common belief, reliability estimates of number-right multiple-choice tests are not inflated by speededness. Because examinees guess on questions when they run out of time, the responses to these questions generally show less consistency with the responses of other questions, and the reliability of the test will be decreased. The surprising implication is that adding questions to a multiple-choice test may lower its reliability when the test is speeded. This article develops the mathematical derivations and shows the effects of speededness on reliability in simulations.
Anastasi, A., & Urbina, S. (1997). Reliability of speeded tests. In A. Anastasi & S. Urbina (Eds.), Psychological testing (7th ed., pp. 102-105). Upper Saddle River, NJ: Prentice Hall.
2.
Crocker, L., & Algina, J. (1986). Factors that affect reliability coefficients. In L. Crocker & J. Algina (Eds.), Introduction to classical and modern test theory (pp. 143-146). Fort Worth, TX: Harcourt Brace Jovanovich.
3.
Gulliksen, H. (1950). Theory of mental tests. New York: John Wiley.
4.
Johnson, M. E. (1987). Multivariate statistical simulation. New York: John Wiley.
5.
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
6.
Schnipke, D. L., & Scrams, D. J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34, 213-232.
7.
Schnipke, D. L., & Scrams, D. J. (2002). Exploring issues of examinee behavior: Insights gained from response-time analyses. In C. N. Mills, M. T. Potenza, J. J. Fremer, & W. C. Ward (Eds.), Computer based testing (pp. 237-266). Mahwah, NJ: Lawrence Erlbaum.
8.
Scrams, D. J., & Schnipke, D. L. (1997, March). Making use of response times in standardized tests: Are accuracy and speed measuring the same thing? Paper presented at the meeting of the National Council on Measurement in Education, Chicago.
9.
Stanley, J. C. (1971). Reliability. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 356-443). Washington, DC: American Council on Education.
10.
Traub, R. E. (1994). Factors affecting the reliability coefficient. In R. E. Traub (Ed.), Reliability for the social sciences: Theory and applications (pp. 98-114). Thousand Oaks, CA: Sage.
11.
Yamamoto, K. (1995). Estimating the effects of test length and test time on parameter estimation using the HYBRID model (ETS RR-95-02). Princeton, NJ: Educational Testing Service.
