Sage Journals: Discover world-class research

Abstract

One common score reported from diagnostic classification assessments is the vector of posterior means of the skill mastery indicators. As with any assessment, it is important to derive and report estimates of the reliability of the reported scores. After reviewing a reliability measure suggested by Templin and Bradshaw, this article suggests three new measures of reliability of the posterior means of skill mastery indicators and methods for estimating the measures when the number of items on the assessment and the number of skills being assessed render exact calculation computationally burdensome. The utility of the new measures is demonstrated using simulated and real data examples. Two of the suggested measures are recommended for future use.

Keywords

reliability cognitive diagnostic models diagnostic classification assessments DINA

Get full access to this article

View all access options for this article.

References

American Educational Research Association, American Psychological Association, & National Council for Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: Author.

Costner

(1965, June). Criteria for measures of association. American Sociological Review, 30, 341–353.

Cui

Gierl

Chang

H.-H.

(2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49, 19–38.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179–199.

de la Torre

Douglas

J. A.

(2004). Higher order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.

Freeman

(1986). Order-based statistics and monotonicity: A family of ordinal measures of association. Journal of Mathematical Sociology, 12, 49–69.

Genz

Bretz

(2009). Computation of multivariate normal and t probabilities. In P. Diggle, U. Gather, & S. Zeger (Series Eds.), Lecture Notes in Statistics: Vol. 195 (pp. 126). New York, NY: Springer-Verlag.

Genz

Bretz

Miwa

Leisch

Scheipl

Hothorn

. (2016). mvtnorm: Multivariate normal and t distributions (Version 1.0-5) [R package ].

Haberman

(2008). When can subscores have value? Journal of Educational and Behavioral Statistics, 33, 204–229.

10.

Haberman

Sinharay

Puhan

(2008). Reporting subscores for institutions. British Journal of Mathematical and Statistical Psychology, 62, 79–95.

11.

Hagenaars

(2015). Methodological issues in categorical data analysis: Categorization, linearity, and response effects. Methodology, 11, 126–141.

12.

Harris

D. J.

Hanson

B. A.

(1991, April). Methods of examining the usefulness of subscores. Paper presented at the annual meeting of the National Council of Measurement in Education, Chicago, IL.

13.

Henson

Templin

Willse

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.

14.

Johnson

Sinharay

(2018). Measures of agreement to assess attribute-level classification accuracy and consistency for cognitive diagnostic assessments. Journal of Educational Measurement, 55, 635–664.

15.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.

16.

Kelley

(1916). A simplified method of using scaled data for purposes of testing. School and Society, 4, 34–31, 71–75.

17.

Kelley

(1921). The reliability of test scores. Journal of Educational Research, 3, 370–399.

18.

Kent

(1983). Information gain and a general measure of correlation. Biometrika, 70, 163–173.

19.

Lee

W.-C.

Hanson

B. A.

Brennan

R. L.

(2002). Estimating consistency and accuracy indices for multiple classifications. Applied Psychological Measurement, 26, 412–432.

20.

Linfoot

(1957). An informational measure of correlation. Information and Control, 1, 85–89.

21.

Lord

Novick

(1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

22.

Muthén

Hofacker

(1988). Testing the assumptions underlying tetrachoric correlation. Psychometrika, 53, 563–578.

23.

Pearson

(1900). Mathematical contributions to the theory of evolution. VII. On the correlation of characters not quantitatively measured. Philosophical Transactions of the Proceedings of the Royal Society of London, 195, 1–48.

24.

Rupp

A. A.

Templin

Henson

R. A.

(2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford.

25.

Spearman

(1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72–101.

26.

Tatsuoka

(2002). Data-analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society, Series C, 51, 337–350.

27.

Tatsuoka

(1984). Analysis of errors in fraction addition and subtraction problems (Technical report, Final report for NIE-G-81-0002). Urbana-Champaign: University of Illinois.

28.

Templin

Bradshaw

(2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30, 251–275.

29.

von Davier

(2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287–307.

30.

Wang

Song

Chen

Meng

Ding

(2015). Attribute-level and pattern-level classification consistency and accuracy indices for diagnostic assessment. Journal of Educational Measurement, 52, 457–476.

31.

Youden

(1950). Index for rating diagnostic tests. Cancer, 3, 32–35.

32.

Yule

(1912). On the methods of measuring the association between two attributes. Journal of the Royal Statistical Society, 75, 579–652.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.21 MB