Sage Journals: Discover world-class research

Abstract

In the context of cognitive diagnosis models (CDMs), a Q-matrix reflects the correspondence between attributes and items. The Q-matrix construction process is typically subjective in nature, which may lead to misspecifications. All this can negatively affect the attribute classification accuracy. In response, several methods of empirical Q-matrix validation have been developed. The general discrimination index (GDI) method has some relevant advantages such as the possibility of being applied to several CDMs. However, the estimation of the GDI relies on the estimation of the latent group sizes and success probabilities, which is made with the original (possibly misspecified) Q-matrix. This can be a problem, especially in those situations in which there is a great uncertainty about the Q-matrix specification. To address this, the present study investigates the iterative application of the GDI method, where only one item is modified at each step of the iterative procedure, and the required cutoff is updated considering the new parameter estimates. A simulation study was conducted to test the performance of the new procedure. Results showed that the performance of the GDI method improved when the application was iterative at the item level and an appropriate cutoff point was used. This was most notable when the original Q-matrix misspecification rate was high, where the proposed procedure performed better 96.5% of the times. The results are illustrated using Tatsuoka’s fraction-subtraction data set.

Keywords

CDM G-DINA Q-matrix validation GDI

Get full access to this article

View all access options for this article.

References

Akaike

(1974). A new look at the statistical identification model. IEEE Transactions on Automated Control, 19, 716–723.

Chen

de la Torre

Zhang

(2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50, 123–140.

Chiu

C.-Y.

(2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598–618.

de la Torre

(2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343–362.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179–199.

de la Torre

Chiu

C.-Y.

(2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273.

de la Torre

Douglas

J. A.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353.

de la Torre

Douglas

J. A.

(2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73(4), 595–624.

Gao

Miller

M. D.

Liu

(2017). The impact of Q-matrix misspecification and model misuse on classification accuracy in the generalized DINA model. Journal of Measurement and Evaluation in Education and Psychology, 8(4), 391–403.

10.

. (in press-a). Partial identifiability of restricted latent classes models. Annals of Statistics. arXiv:1803.04353

11.

. (in press-b). Sufficient and necessary conditions for the identifiability of the Q-matrix. Statistica Sinica. arXiv:1810.03819

12.

Haertel

(1984). An application of latent class models to assessment data. Applied Psychological Measurement, 8(3), 333–346.

13.

Junker

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric IRT. Applied Psychological Measurement, 25, 258–272.

14.

Kang

Yang

Zeng

(2019). Q-matrix refinement based on item fit statistic RMSEA. Applied Psychological Measurement, 43(7), 527–542.

15.

Köhn

H.-F.

Chiu

C.-Y.

(2017). A procedure for assessing the completeness of the Q-matrices of cognitively diagnostic tests. Psychometrika, 82(1), 112–132.

16.

Köhn

H.-F.

Chiu

C.-Y.

(2018). How to build a complete Q-matrix for a cognitively diagnostic test. Journal of Classification, 35, 273–299.

17.

Kunina-Habenicht

Rupp

Wilhelm

(2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49(1), 59–81.

18.

Liu

Ying

(2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548–564.

19.

de la Torre

(2018). GDINA: The generalized DINA model framework (R Package Version 2.0.8). https://cran.r-project.org/package=GDINA

20.

de la Torre

(2020). An empirical Q-matrix validation method for the sequential generalized DINA model. British Journal of Mathematical and Statistical Psychology, 73, 142–163. https://doi.org/10.1111/bmsp.12156

21.

Guo

(2019). Cognitive diagnosis models for multiple strategies. British Journal of Mathematical and Statistical Psychology, 72, 370–392.

22.

Maris

(1999). Estimating multiple classification latent class models. Psychometrika, 64(2), 187–212.

23.

Mislevy

R. J.

(1996). Test theory reconceived. Journal of Educational Measurement, 33(4), 379–416.

24.

Nájera

Sorrel

M. A.

Abad

F. J.

(2019). Reconsidering cutoff points in the general method of empirical Q-matrix validation. Educational and Psychological Measurement, 79(4), 727–753.

25.

R Core Team. ( 2018). R (Version 3.4) [Computer Software]. R Foundation for Statistical Computing.

26.

Rupp

Templin

(2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78–96.

27.

Schwarzer

(1976). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.

28.

Sorrel

M. A.

Olea

Abad

F. J.

de la Torre

Aguado

Lievens

(2016). Validity and reliability of situational judgement test scores: A new approach based on cognitive diagnosis models. Organizational Research Methods, 19(3), 506–532.

29.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconception based on item response theory. Journal of Education Statistic, 20, 345–354.

30.

Tatsuoka

K. K.

(1990). Toward an integration of item-response theory and cognitive error diagnosis. In Frederiksen

Glaser

Lesgold

Safto

, M. (Eds.), Monitoring skills and knowledge acquisition (pp. 453–488). Lawrence Erlbaum.

31.

Templin

Henson

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305.

32.

Terzi

(2017). New Q-matrix validation procedures [Unpublished doctoral dissertation]. Rutgers, The State University of New Jersey.

33.

Terzi

de la Torre

(2018a). An iterative method for empirically-based Q-matrix validation. International Journal of Assessment Tools in Education, 5, 248–262.

34.

Terzi

de la Torre

(2018b, April). Two general iterative Q-matrix validation procedures. Paper presented at the meeting of the National Council of Measurement in Education, New York, NY, United State.

35.

von Davier

. (2005). A general diagnostic model applied to language testing data (Research Report, RR-05-16). Educational Testing Service.

36.

Wang

Song

Ding

Meng

Cao

Jie

(2018). An EM-based method for Q-matrix validation. Applied Psychological Measurement, 42, 446–459.

37.

Shang

(2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523), 1284–1295.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.23 MB