Sage Journals: Discover world-class research

Abstract

Solving a constructed-response item usually requires successfully performing a sequence of tasks. Each task could involve different attributes, and those required attributes may be “condensed” in various ways to produce the responses. The sequential generalized deterministic input noisy “and” gate model is a general cognitive diagnosis model (CDM) for graded response items of this type. Although a host of dichotomous CDMs with different condensation rules can be used to parameterize the success probability of each task, specifying the most appropriate one remains challenging. If the CDM specified for each task is not in accordance with the underlying cognitive processes, the validity of the inference could be questionable. This study aims to evaluate whether several hypothesis tests, namely, the Wald test using various variance–covariance matrices, the likelihood ratio (LR) test, and the LR test using approximated parameters, can be used to select the appropriate CDMs for each task of graded response items. Simulation studies are conducted to examine the Type I error and power of the hypothesis tests under varied conditions. A data set from the Trends in International Mathematics and Science Study 2007 mathematics assessment is analyzed as an illustration.

Keywords

cognitive diagnosis G-DINA sequential CDM polytomous data model selection

Get full access to this article

View all access options for this article.

References

Akaike

(1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723.

Chalmers

R. P

. (2018). Numerical approximation of the observed information matrix with Oakes’ identity. British Journal of Mathematical and Statistical Psychology. doi:10.1111/bmsp.12127

Chalmers

R. P.

Pek

Liu

(2017). Profile-likelihood confidence intervals in item response theory models. Multivariate Behavioral Research, 52, 533–550.

Chen

de la Torre

Zhang

(2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50, 123–140.

de la Torre

(2008). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115–130.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179–199.

de la Torre

Lee

Y. S.

(2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50, 355–373.

DiBello

L. V.

Roussos

L. A.

Stout

(2007). A review of cognitively diagnostic assessment and a summary of psychometric models. In Rao

Sinharay

(Eds.), Handbook of statistics: Psychometrics (Vol. 26, pp. 979–1030). Amsterdam, the Netherlands: Elsevier.

Duan

J. C.

Fulop

(2011). A stable estimator of the information matrix under EM for dependent data. Statistics and Computing, 21, 83–91. doi:10.1007/s11222-009-9149-4

10.

Guo

de la Torre

(2017). Standard error estimation using bootstrap approaches for cognitive diagnosis models. Paper presented at the Annual Meeting of the National Council of Measurement in Education, San Antonio, TX.

11.

Haertel

E. H.

(1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301–321.

12.

Hartz

S. M.

(2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). University of Illinois at Urbana–Champaign, Urbana–Champaign.

13.

Hemker

B. T.

van der Ark

L. A.

Sijtsma

(2001). On measurement properties of continuation ratio models. Psychometrika, 66, 487–506.

14.

Henson

R. A.

Templin

J. L.

Willse

J. T.

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.

15.

Hou

de la Torre

Nandakumar

(2014). Differential item functioning assessment in cognitive diagnostic modeling: Application of the Wald test to investigate DIF in the DINA model. Journal of Educational Measurement, 51, 98–125.

16.

Huber

P. J.

(1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Le Cam

& Neyman

(Eds.) Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (pp. 221–233). Berkeley, CA: University of California Press.

17.

Jamshidian

Jennrich

R. I.

(2000). Standard errors for EM estimation. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 62, 257–270.

18.

Kunina-Habenicht

Rupp

A. A.

Wilhelm

(2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49, 59–81.

19.

Lee

Y.-S.

Park

Y. S.

Taylan

(2011). A cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the U.S. national sample using the TIMSS 2007. International Journal of Testing, 11, 144–177.

20.

Liu

Huggins-Manley

A. C.

Bulut

. (2017). Retrofitting diagnostic classification models to responses from IRT-based assessment forms. Educational and Psychological Measurement. doi:10.1177/0013164416685599

21.

Liu

Tian

Xin

(2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41, 3–26.

22.

Liu

Xin

(2017). dcminfo: Information matrix for diagnostic classification models (R package version 0.1.7) [Computer software]. Retrieved from https://CRAN.R-project.org/package=dcminfo

23.

Liu

Xin

Andersson

Tian

. (2018). Information matrix estimation procedures for cognitive diagnostic models. British Journal of Mathematical and Statistical Psychology. doi:10.1111/bmsp.12134

24.

Liu

Xin

Tian

Liu

(2016). An improved method for differential item functioning detection in cognitive diagnosis models: An application of Wald statistic based on observed information matrix. Acta Psychologica Sinica, 48, 588–598.

25.

Louis

T. A.

(1982). Finding the observed information matrix when using the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 44, 226–233.

26.

(2014). Validation of the item-attribute matrix in TIMSS: Mathematics using multiple regression and the LSDM (Unpublished doctoral dissertation). University of Denver, Denver, CO.

27.

. (2018). A diagnostic tree model for polytomous responses with multiple strategies. British Journal of Mathematical and Statistical Psychology. doi:10.1111/bmsp.12137

28.

de la Torre

(2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69, 253–275.

29.

de la Torre

. (2017). GDINA: The generalized DINA model framework [R package version 1.4.2 computer software]. Retrieved from https://CRAN.R-project.org/package=GDINA

30.

Iaconangelo

de la Torre

(2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40, 200–217.

31.

Maris

(1999). Estimating multiple classification latent class models. Psychometrika, 64, 187–212.

32.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.

33.

Paek

Cai

(2014). A comparison of item parameter standard error estimation procedures for unidimensional and multidimensional item response theory modeling. Educational and Psychological Measurement, 74, 58–76.

34.

Philipp

Strobl

de la Torre

Zeileis

(2018). On the estimation of standard errors in cognitive diagnosis models. Journal of Educational and Behavioral Statistics, 43, 88–115.

35.

R Core Team. (2017). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from https://www.R-project.org/

36.

Rojas

de la Torre

Olea

(2012). Choosing between general and specific cognitive diagnosis models when the sample size is small. Paper presented at the Annual Meeting of the National Council of Measurement in Education, Vancouver, Canada.

37.

Schwarz

(1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461–464.

38.

Sinharay

Almond

R. G.

(2007). Assessing fit of cognitive diagnostic models: A case study. Educational and Psychological Measurement, 67, 239–257.

39.

Skaggs

Wilkins

J. L.

Hein

S. F.

(2016). Grain size and parameter recovery with TIMSS and the general diagnostic MODEL. International Journal of Testing, 16, 310–330.

40.

Sorrel

M. A.

de la Torre

Abad

F. J.

Olea

(2017). Two-step likelihood ratio test for model comparison in cognitive diagnosis models. Methodology, 13, 39–47.

41.

Spiegelhalter

D. J.

Best

N. G.

Carlin

B. P.

Van Der Linde

(2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64, 583–639.

42.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354.

43.

Templin

J. L.

Henson

R. A.

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.

44.

Tutz

(2016). Sequential models for ordered responses. In van der Linden

W. J.

(Ed.), Handbook of item response theory (Vol. 1: Models) (pp. 139–150). Boca Raton, FL: CRC Press.

45.

Verhelst

N. D.

Glas

C. A. W.

de Vries

H. H.

(1997). A step model to analyze partial credit. In van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of modern item response theory (pp. 123–138). New York, NY: Springer-Verlag.

46.

Verhelst

N. D.

Verstralen

H. H.

(2008). Some considerations on the partial credit model. Psicologica, 29, 229–254.

47.

von Davier

(2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287–307.

48.

von Davier

(2014). The DINA model as a constrained general diagnostic model: Two variants of a model equivalency. British Journal of Mathematical and Statistical Psychology, 67, 49–71.

49.

Wald

(1943). Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical Society, 54, 426–482.

50.

White

(1982). Maximum likelihood estimation of misspecified models. Econometrica, 50, 1–25.

51.

Xin

Liu

Tian

(2017, 4). New item-level model selection procedures for diagnostic classification models. Paper presented at the Annual Meeting of the National Council on Measurement in Education, San Antonio, TX.

52.

Yuan

K. H.

Cheng

Patton

(2014). Information matrices and standard errors for MLEs of item parameters in IRT. Psychometrika, 79, 232–254.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.03 MB

0.13 MB

0.00 MB