Sage Journals: Discover world-class research

Abstract

In cognitive diagnostic assessment, a Q matrix that associates each item with cognitive attributes in a test is necessary to derive the knowledge states of students. Defining the Q matrix is the most fundamental step in cognitive diagnosis. The conventional method for calibrating cognitive attributes relies mainly on the subjective judgments of experts. Herein, a method that uses a structural learning algorithm in a Bayesian network (BN) to estimate the Q matrix is proposed. Both the viability and efficacy of the suggested approach are examined by running simulations and conducting analyses on real data. The outcomes of the simulations indicate that the proposed approach mostly exceeds the performance of the available methods, and its advantages stem from not only better estimation accuracy but also its computational efficiency. The efficiency of the proposed approach is also verified via real data analysis. Compared with the stepwise method, the model data fits the estimated Q matrix via the BN method for the empirical data is more satisfactory, and the Root Mean Square Error of Approximation (RMSEA) is lower. Consequently, this Q-matrix estimation method based on a BN can improve accuracy and computational efficiency and promote the practice of cognitive diagnosis assessment.

Keywords

cognitive diagnosis Bayesian network Q matrix estimation structural learning

Get full access to this article

View all access options for this article.

References

Anderson

C. J.

H. T.

(2007). Log-multiplicative association models as item response models. Psychometrika, 72, 5–23. https://doi.org/10.1007/s11336-005-1419-2

Borsboom

Mellenbergh

G. J.

van Heerden

(2003). The theoretical status of latent variables. Psychological Review, 110, 203–219. https://doi.org/10.1037/0033-295X.110.2.203

Buntine

(1996). A guide to the literature on learning probabilistic networks from data. IEEE Transactions on Knowledge and Data Engineering, 8(2), 195–210.

Chen

(2017). A residual-based approach to validate Q-matrix specifications. Applied Psychological Measurement, 41, 014662161668602. https://doi.org/10.1177/0146621616686021

Chen

Y. H.

Culpepper

S. A.

Chen

Y. G.

Douglas

(2018). Bayesian estimation of the DINA Q-matrix. Psychometrika, 83(1), 89–108.

Chen

Culpepper

Liang

(2020). A sparse latent class model for cognitive diagnosis. Psychometrika, 85, 121–153.

Chiu

C.-Y.

Douglas

(2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns. Journal of Classiﬁcation, 30, 225–250. https://doi.org/10.1007/s00357-013-9132-9

Chiu

C.-Y.

Douglas

(2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633–665. https://doi.org/10.1007/s11336-009-9125-0

Chung

(2019). A Gibbs sampling algorithm that estimates the Q-matrix for the DINA model. Journal of Mathematical Psychology, 93, 102275. https://doi.org/10.1016/j.jmp.2019.07.002

10.

Culbertson

M. J.

(2016). Bayesian networks in educational assessment: The state of the field. Applied Psychological Measurement, 40(1), 3–21. https://doi.org/10.1177/0146621615590401

11.

Culpepper

S. A.

(2019). Estimating the cognitive diagnosis Q matrix with expert knowledge: Application to the fraction subtraction dataset. Psychometrika, 84(2), 333–357.

12.

Daly

Shen

Aitken

(2011). Learning Bayesian networks: Approaches and issues. The Knowledge Engineering Review, 26(2), 99–157.

13.

DeCarlo

L. T.

(2010). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the Q-matrix. Applied Psychological Measurement, 35(1), 8–24.

14.

DeCarlo

L. T.

(2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Applied Psychological Measurement, 36(6), 447–468. https://doi.org/10.1177/0146621612449069

15.

de la Torre

(2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 346–362.

16.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179–199.

17.

de la Torre

Chiu

C. Y.

(2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273.

18.

Ding

S. L.

Yang

S. Q.

Wang

W. Y.

(2010). The importance of reachability matrix in constructing cognitively diagnostic testing. Journal of Jiangxi Normal University (Natural Science), 34, 490–495.

19.

Embretson

(1984). A general latent trait model for response processes. Psychometrika, 49(2), 175–186. https://doi.org/10.1007/BF02294171

20.

Y. Q.

G. J.

(2020). Partial identifiability of restricted latent class models. The Annals of Statistics, 48(4), 2082–2107.

21.

Y. Q.

G. J.

(2021). Sufficient and necessary conditions for the identifiability of the Q-matrix. Statistica Sinica, 31, 449–472.

22.

Hansen

Cai

Monroe

(2016). Limited information goodness-of-fit testing of diagnostic classification item response models. The British Journal of Mathematical and Statistical Psychology, 69(3), 225–252. https://doi.org/10.1111/bmsp.12074

23.

Hartz

S. M.

(2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality [Unpublished doctoral dissertation]. University of Illinois at Urbana-Champaign, IL.

24.

Hartz

S. M.

Roussos

(2008). The fusion model for skills diagnosis: Blending theory with practicality. ETS Research Report Series, 2008(2), 1–57.

25.

Heller

Wickelmaier

(2013). Minimum discrepancy estimation in probabilistic knowledge structures. Electronic Notes in Discrete Mathematics, 42, 49–56. https://doi.org/10.1016/j.endm.2013.05.145

26.

Templin

(2019). Using diagnostic classification models to validate attribute hierarchies and evaluate model fit in Bayesian networks. Multivariate Behavioral Research. 55(2), 300–311.https://doi.org/10.1080/00273171.2019.1632165

27.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. https://doi.org/10.1177/01466210122032064

28.

Koller

Friedman

(2009). Probabilistic graphical models: Principles and techniques. MIT Press.

29.

Lee

Corter

J. E.

(2011). Diagnosis of subtraction bugs using Bayesian networks. Applied Psychological Measurement, 35, 27–47. https://doi.org/10.1177/014662161037.7079

30.

Leighton

J. P.

Gierl

M. J.

(Eds.). (2007). Cognitive diagnostic assessment for education: Theory and practices. Cambridge University Press.

31.

Lim

Y. S.

Drasgow

(2017). Nonparametric calibration of item-by-attribute matrix in cognitive diagnosis. Multivariate Behavioral Research, 52(5), 562–575. https://doi.org/10.1080/00273171.2017.1341829

32.

Liu

Ying

(2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548–564. https://doi.org/10.1177/0146621612456591

33.

Liu

Tian

Xin

(2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41(1), 3–26. https://doi.org/10.3102/1076998615621293

34.

de la Torre

(2020a). An empirical Q-matrix validation method for the sequential G-DINA model. British Journal of Mathematical and Statistical Psychology, 73, 142–163.

35.

Ouyang.

G. J.

(2023). Learning latent and hierarchical structures in cognitive diagnosis models. Psychometrika, 88, 175–207.

36.

Macready

G. B.

Dayton

C. M.

(1977). The use of probabilistic models in the assessment of mastery. Journal of Educational and Behavioral Statistics, 2(2), 99–120. https://doi.org/10.3102/10769986002002099

37.

Margaritis

(2003). Learning Bayesian network model structure from data (PhD thesis, School of Computer Science, Carnegie-Mellon University). Technical Report CMU-CS-03-153.

38.

Pearl

(1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. Morgan Kaufmann.

39.

Philipp

Strobl

de la Torre

Zeileis

(2018). On the estimation of standard errors in cognitive diagnosis models. Journal of Educational and Behavioral Statistics, 43(1), 88–115. https://doi.org/10.3102/1076998617719728

40.

Rupp

A. A.

Templin

(2008).The effects of Q-matrix misspecification on parameter estimates and classification accuracy in DINA model. Educational and Psychological Measurement, 68(1), 78–96.

41.

Scutari

(2010). Learning Bayesian networks with the bnlearn R Package. Journal of Statistical Software, 35(3), 1–22.

42.

Scutari

Denis

J. B.

(2021). Bayesian networks with examples in R. Chapman and Hall/CRC.

43.

Song

L. H.

Wang

W. Y.

Dai

H. Q.

Ding

S. L.

(2016). Bayesian networks for cognitive diagnostic modeling. Psychological Science, 39(4), 783–789.

44.

Tatsuoka

K. K.

(1985). A probabilistic model for diagnosing misconceptions in the pattern classification approach. Journal of Educational Statistics, 12, 55–73.

45.

Tatsuoka

K. K.

Tatsuoka

M. M.

(1997). Computerized cognitive diagnostic adaptive testing: Effect on remedial instruction as empirical validation. Journal of Educational Measurement, 34(1), 3–20. https://doi.org/10.1111/j.1745-3984.1997.tb00504.x

46.

Templin

J. L.

Henson

R. A.

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287.

47.

Wang

D. X.

Cai

D. B.

(2020). Q-matrix estimation methods for cognitive diagnosis models: Based on partially known Q-matrix. Multivariate Behavioral Research, 1–13. https://doi.org/10.1080/00273171.2020.1746901

48.

Wang

Xin

Liu

(2021). An improved parameter estimating method in Bayesian networks applied for cognitive diagnosis assessment. Frontiers in Psychology, 12, 665441. https://doi.org/10.3389/fpsyg.2021.665441

49.

G. J.

Shang

Z. R.

(2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523), 1284–1295.

50.

X. F.

Ding

S. L.

Qin

C. Y.

(2011).Application of Bayesian networks to identify hierarchical relation among attributes in cognitive diagnosis. Acta Psychologica Sinica, 43(3), 338–346.