To improve the calibration accuracy and calibration efficiency of cognitive diagnostic computerized adaptive testing (CD-CAT) for new items and, ultimately, contribute to the widespread application of CD-CAT in practice, the current article proposed a Gini-based online calibration method that can simultaneously calibrate the Q-matrix and item parameters of new items. Three simulation studies with simulated and real item banks were conducted to investigate the performance of the proposed method and compare it with the joint estimation algorithm (JEA) and the single-item estimation (SIE) methods. The results indicated that the proposed Gini-based online calibration method yielded higher calibration efficiency than those of the SIE method and outperformed the JEA method on item calibration tasks in terms of both accuracy and efficiency under most experimental conditions.
BanJ.-C.HansonB.H.WangT.YiQ.HarrisD.J. (2001). A comparative study of online pretest item-calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38, 191–212. https://doi.org/10.1111/j.1745-3984.2001.tb01123.x
2.
BreimanL.FriedmanJ.StoneC. J.OlshenR. A. (1984). Classification and regression trees. Chapman & Hall.
3.
ChandraB.Paul VargheseP. (2009). Fuzzifying Gini Index based decision trees. Expert Systems With Applications, 36(4), 8549–8559. https://doi.org/10.1016/j.eswa.2008.10.053
4.
ChenP. (2017). A comparative study of online item calibration methods in multidimensional computerized adaptive testing. Journal of Educational and Behavioral Statistics, 42(5), 559–590. https://doi.org/10.3102/1076998617695098
5.
ChenP.XinT. (2011, April). Item replenishing in cognitive diagnostic computerized adaptive testing. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
6.
ChenP.XinT.WangC.ChangH. (2012). Online calibration methods for the DINA model with independent attributes in CD-CAT. Psychometrika, 77(2), 201–222. https://doi.org/10.1007/s11336-012-9255-7
7.
ChenY.LiuJ.YingZ. (2015). Online item calibration for Q-matrix in CD-CAT. Applied Psychological Measurement, 39(1), 5–15. https://doi.org/10.1177/0146621613513065
de la TorreJ. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343–362. https://doi.org/10.1111/j.1745-3984.2008.00069.x
10.
de la TorreJ. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115–130. https://doi.org/10.3102/1076998607309474
DemareeR. G. (1950). An investigation of homogeneity in the interpretation and scaling of test results. University of Illinois at Urbana-Champaign.
14.
DiBelloL. V.StoutW. (2007). Guest editors introduction and overview: IRT-based cognitive diagnostic models and related methods. Journal of Educational Measurement, 44, 285–291. https://doi.org/10.1111/j.1745-3984.2007.00039.x
15.
GibbonsR. D.WeissD. J.PilkonisP. A.FrankE.MooreT.KimJ. B.KupferD. J. (2012). Development of a computerized adaptive test for depression. Archives of General Psychiatry, 69(11), 1104–1112. https://doi.org/10.1001/archgenpsychiatry.2012.14
16.
ImS.CorterJ. E. (2011). Statistical consequences of attribute misspecification in the rule space method. Educational and Psychological Measurement, 71, 712–731. https://doi.org/10.1177/0013164410384855
17.
JunkerB. W.SijtsmaK. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. https://doi.org/10.1177/01466210122032064
18.
KaradS. M.PatilD. D.HalgaonkarP. S.WadhaiV. M.KharatM. U. (2012). Two phase iterative clustering for educational data. International Journal of Applied Information Systems, 1(5), 11–15. https://doi.org/10.5120/ijais12-450173
19.
LiuH.YouX.WangW.DingS.ChangH. (2013). The development of computerized adaptive testing with cognitive diagnosis for an English achievement test in China. Journal of Classification, 30(2), 152–172. https://link.springer.com/article/10.1007/s00357-013-9128-5
MaW.de la TorreJ. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253–275. https://doi.org/10.1111/bmsp.12070
22.
MehtaM.AgrawalR.RiassnenJ. (1996). SLIQ: A fast scalable classifier for data mining, in extending database technology (pp. 18–32). Springer. https://doi.org/10.1007/BFb0014141
23.
NájeraP.SorrelM. A.AbadF. J. (2019). Reconsidering cutoff points in the general method of empirical Q-matrix validation. Educational and psychological measurement, 79(4), 727–753. https://doi.org/10.1177/0013164418822700
24.
No Child Left Behind Act of 2001, Pub. L. No. 1-7-110 (2001).
25.
RuppA. A.TemplinJ. L. (2008). The effect of Q-matrix misspecification on parameter estimates and misclassification rates in the DINA model. Educational and Psychological Measurement, 68, 78–96. https://doi.org/10.1177/0013164407301545
StroblC.BoulesteixA.-L.AugustinT. (2007). Unbiased split selection for classification trees based on the Gini Index. Computational Statistics & Data Analysis, 52(1), 483–501. https://doi.org/10.1016/j.csda.2006.12.030
28.
TanP. N.SteinbachM.KumarV. (2016). Introduction to data mining. Pearson Education India.
29.
TatsuokaK. K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In FrederiksenN.GlaserR.LesgoldA.SaftoM. (Eds.), Diagnostic monitoring skills and knowledge acquisition (pp. 453–488). Erlbaum.
30.
TatsuokaK. K.TatsuokaM. M. (1997). Computerized cognitive diagnostic adaptive testing: Effect on remedial instruction as empirical validation. Journal of Educational Measurement, 34(1), 3–20. https://doi.org/10.1111/j.1745-3984.1997.tb00504.x
31.
WainerH.MislevyR. J. (1990). Item response theory, item calibration, and proficiency estimation. Computerized adaptive testing: A primer, 4, 65–102.
32.
WangW.SongL.DingS.MengY.CaoC.JieY. (2018). An EM-based method for Q-matrix validation. Applied Psychological Measurement, 42(6), 446–459. https://doi.org/10.1177/0146621617752991
33.
YuX.ChengY. (2020). Data-driven Q-matrix validation using a residual-based statistic in cognitive diagnostic assessment. British Journal of Mathematical and Statistical Psychology, 73, 145–179. https://doi.org/10.1111/bmsp.12191
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
