Conditional independence (CI) between response time and response accuracy is a fundamental assumption of many joint models for time and accuracy used in educational measurement. In this study, posterior predictive checks (PPCs) are proposed for testing this assumption. These PPCs are based on three discrepancy measures reflecting different observable consequences of different types of violations of CI. Simulation studies are performed to evaluate the specificity of the procedure, its robustness, and its sensitivity to detect different types of conditional dependence and to compare it to existing methods. The new procedure outperforms the existing methods in most of the simulation conditions. The use of the PPCs is illustrated using arithmetics test data.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 395–479). Reading, MA: Addison-Wesley.
2.
BolsinovaM.de BoeckP.TijmstraJ. (2015). Modeling conditional dependence between response time and accuracy. Manuscript submitted for publication.
3.
BolsinovaM.MarisG. (2016). A test for conditional independence between response time and accuracy. British Journal of Mathematical and Statistical Psychology, 69, 62–79.
4.
BrooksS.GelmanA. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7, 434–455.
5.
CasellaG.GeorgeE. (1992). Explaining the Gibbs sampler. The American Statistician, 43, 167–174.
6.
ChenH.De BoeckP. (2014). Are slow and fast ability test responses different?Manuscript submitted for publication.
7.
ChenW.ThissenD. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265–289.
8.
CoyleT. (2003). A review of the worst performance rule: Evidence, theory, and alternative hypotheses. Intelligence, 31, 567–587.
9.
FoxJ. P.van der LindenW. J. (2007). Modeling of responses and response times with the package cirt. Journal of Statistical software, 20, 1–14.
10.
GelmanA. (2006). Prior distributions for variance parameters in hierarchical model. Bayesian Analysis, 1, 515–533.
11.
GelmanA.MengX.-L.SternH. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6, 733–807.
12.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–472.
13.
GemanS.GemanD. (1984). Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 721–741.
14.
GoldhammerF.NaumannJ.StelterA.TóthK.RölkeH.KliemeE. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106, 608–626.
15.
HoffP. D. (2009). A first course in Bayesian statistical methods. New York, NY: Springer.
IpE. H. (2001). Testing for local dependency in dichotomous and polytomous item response models. Psychometrika, 66, 109–132.
18.
Klein EntinkR.FoxJ. P.van der LindenW. J. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74, 21–48.
19.
LevyR.MislevyR. J.SinharayS. (2009). Posterior predictive model checking for multidimensionality in item response theory. Statistica Sinica, 6, 733–807.
20.
LevyR.XuY.YelN.SvetinaD. (2015). A standardised generalised dimensionality discrepancy measure and a standardised model-based covariance for dimensionality assessment for multidimensional models. Journal of Educational Measurement, 52, 144–158.
21.
MarisG.van der MaasH. (2012). Speed-accuracy response models: Scoring rules based on response time and accuracy. Psychometrika, 77, 615–633.
22.
MarsmanM.MarisG.BechgerT.GlasC. A. (2015). Markov chain Monte Carlo for large-scale Bayesian inference. Manuscript submitted for publication.
23.
MengX.-L. (1994). Posterior predictive p-values. The Annals of Statistics, 22, 1142–1160.
24.
PartchevI.De BoeckP. (2012). Can fast and slow intelligence be differentiated?Intelligence, 40, 23–32.
25.
RangerJ.OrtnerT. (2012). The case of dependency of responses and response times: A modeling approach based on standard latent trait models. Psychological Test and Assessment Modeling, 54, 128–148.
26.
RaschG. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: The Danish Institute of Educational Research. (Expanded ed., 1980. Chicago, The University of Chicago Press).
27.
RatcliffR. (1978). A theory of memory retrieval. Psychological Review, 85, 59–108.
28.
ReckaseM. D. (1997). A linear logistic multidimensional model. In van der LindenW. J.HambletonR. K. (Eds.), Handbook of modern item response theory (pp. 271–286). New York, NY: Springer Verlag.
29.
RubinD. B. (1984). Bayesianly justifiable and relevant frequency calculations for the applied statistician. Annals of Statistics, 12, 1151–1172.
30.
SinharayS.JohnsonM. S.SternH. S. (2006). Posterior predictive assessment of item response theory models. Applied Psychological Measurement, 30, 298–321.
31.
TuerlinckxF.De BoeckP. (2005). Two interpretations of the discrimination parameter. Psychometrika, 70, 629–650.
32.
van der LindenW. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31, 181–204.
33.
van der LindenW. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308. doi:10.1007/s11336-006-1478-z
34.
van der LindenW. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46, 247–272.
35.
van der LindenW. J.GlasC. A. W. (2010). Statistical tests of conditional independence between responses and/or response times on test items. Psychometrika, 75, 120–139.
36.
YenW. M. (1984). Effects of local dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125–145.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
