Item response theory (IRT) models have a number of advantages for developing and evaluating scales in organizational research. However, these advantages can be obtained only when the IRT model used to estimate the parameters fits the data well. Therefore, examining IRT model fit is important before drawing conclusions from the data. To test model fit, a wide range of indices are available in the IRT literature and have demonstrated utility in past research. Nevertheless, the performance of many of these indices for detecting misfit has not been directly compared in simulations. The current study evaluates a number of these indices to determine their utility for detecting various types of misfit in both dominance and ideal point IRT models. Results indicate that some indices are more effective than others but that none of the indices accurately detected misfit due to multidimensionality in the data. The implications of these results for future organizational research are discussed.
AkaikeH. (1973). Information theory and an extension of the maximum likelihood principle. In PetrovB. N.CsákiF. (Eds.), Second International Symposium on Information Theory (pp. 267–281). Budapest, Hungary: Akadémiai Kiadó.
2.
BafumiJ.GelmanA.ParkD. K.KaplanN. (2005). Practical issues in implementing and understanding Bayesian ideal point estimation. Political Analysis, 13, 171–187.
3.
BockR. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29–51.
4.
BoltD. M. (2002). A Monte Carlo comparison of parametric and nonparametric polytomous DIF detection methods. Applied Measurement in Education, 15, 113–141.
5.
BormanW. C.BuckD. E.HansonM. A.MotowidloS. J.StarkS.DrasgowF. (2001). An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. Journal of Applied Psychology, 86, 965–973.
6.
BrownA.Maydeu-OlivaresA. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71, 460–502.
7.
CarterN. T.DalalD. K. (2010). An ideal point account of the JDI Work Satisfaction scale. Personality and Individual Differences, 49, 743–748.
8.
CarterN. T.DalalD. K.BoyceA. S.O’ConnellM. S.KungM. C.DelgadoK. M. (2014). Uncovering curvilinear relationships between conscientiousness and job performance: How theoretically appropriate measurement makes an empirical difference. Journal of Applied Psychology, 99, 564–586.
9.
CarterN. T.DalalD. K.LakeC. J.LinB. C.ZickarM. J. (2011). Using mixed-model item response theory to analyze organizational survey responses: An illustration using the job descriptive index. Organizational Research Methods, 14, 116–146.
10.
ChanK. Y.DrasgowF.SawinL. L. (1999). What is the shelf life of a test? The effect of time on the psychometrics of a cognitive ability test battery. Journal of Applied Psychology, 84, 610–619.
11.
ChernyshenkoO. S.StarkS.ChanK. Y.DrasgowF.WilliamsB. A. (2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523–562.
12.
ChernyshenkoO. S.StarkS.DrasgowF.RobertsB. W. (2007). Constructing personality scales under the assumptions of an ideal point response process: Toward increasing the flexibility of personality measures. Psychological Assessment, 19, 88–106.
13.
ChernyshenkoO. S.StarkS.WilliamsA. (2009). Latent trait theory approach to measuring person-organization fit: Conceptual rationale and empirical evaluation. International Journal of Testing, 9, 358–380.
14.
DalalD. K.CarterN. T. (2015). Consequences of ignoring ideal point items for applied decisions and criterion-related validity estimates. Journal of Business and Psychology, 30, 483–498.
15.
DavisonM. L. (1977). On a metric, unidimensional unfolding model for attitudinal and developmental data. Psychometrika, 42, 523–548.
16.
De AyalaR. J.DoddB. G.KochW. R. (1992). A comparison of the partial credit and graded response models in computerized adaptive testing. Applied Measurement in Education, 5, 17–34.
17.
de La TorreJ.StarkS.ChernyshenkoO. S. (2006). Markov chain Monte Carlo estimation of item parameters for the generalized graded unfolding model. Applied Psychological Measurement, 30, 216–232.
18.
DeMarsC. E. (2004). Type I error rates for generalized graded unfolding model fit indices. Applied Psychological Measurement, 28, 48–71.
19.
DeMarsC. E. (2005, August). Scoring subscales using multidimensional item response theory models. Poster presented at the annual meeting of the American Psychology Association, Washington, DC.
20.
DoornikJ. A. (2009). An object-oriented matrix programming language Ox 6. Computer software.
21.
DrasgowF.ChernyshenkoO. S.StarkS. (2010). 75 years after Likert: Thurstone was right!Industrial and Organizational Psychology, 3, 465–476.
22.
DrasgowF.LevineM. V.McLaughlinM. E. (1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59–79.
23.
DrasgowF.LevineM. V.TsienS.WilliamsB.MeadA. D. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, 143–166.
24.
DrasgowF.LissakR. I. (1983). Modified parallel analysis: A procedure for examining the latent dimensionality of dichotomously scored item responses. Journal of Applied Psychology, 68, 363–373.
25.
DrasgowF.StarkS.ChernyshenkoO. S.NyeC. D.HulinC. L.WhiteL. A. (2012). Development of the Tailored Adaptive Personality Assessment System (TAPAS) to Support Selection and Classification Decisions(Tech. Rep. 1311). Arlington, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
26.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
27.
ForeroC. G.Maydeu-OlivaresA. (2009). Estimation of IRT graded response models: Limited versus full information methods. Psychological Methods, 14, 275–299.
28.
FosterG. C.MinH.ZickarM. J. (2017). Review of item response theory practices in organizational research: Lessons learned and paths forward. Organizational Research Methods, 20, 465–486.
29.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–472.
30.
GlasC. A. W. (1988). The derivation of some tests for the Rasch model from the multinomial distribution. Psychometrika, 53, 525–546.
31.
GoldbergL. R.KilkowskiJ. M. (1985). The prediction of semantic consistency in self-descriptions: Characteristics of persons and of terms that affect the consistency of responses to synonym and antonym pairs. Journal of Personality and Social Psychology, 48, 82–98.
32.
HabingB.FinchH.RobertsJ. (2005). A Q-sub-3 statistic for unfolding item response theory models: Assessment of unidimensionality with two factors and simple structure. Applied Psychological Measurement, 29, 457–471.
33.
HuL.-T.BentlerP. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424–453.
34.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 3, 424–453.
35.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114.
36.
HuangJ. L.LiuM.BowlingN. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828–845.
37.
JooS. H.LeeP.StarkS. (2017). Evaluating anchor-item designs for concurrent calibration with the GGUM. Applied Psychological Measurement, 41, 83–96.
38.
KangT.CohenA. S.SungH. J. (2009). Model selection indices for polytomous items. Applied Psychological Measurement, 33, 499–518.
39.
KaskowitzG. S.De AyalaR. J. (2001). The effect of error in item parameter estimates on the test response function method of linking. Applied Psychological Measurement, 25, 39–52.
40.
KieftenbeldV.NatesanP. (2012). Recovery of graded response model parameters: A comparison of marginal maximum likelihood and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 36, 399–419.
41.
KimJ. S.BoltD. M. (2007). Estimating item response theory models using Markov chain Monte Carlo methods. Educational Measurement: Issues and Practice, 26, 38–51.
42.
KlineR. B. (2005). Principles and practices of structural equation modeling. New York, NY: Guilford.
43.
LiY.BoltD. M.FuJ. (2006). A comparison of alternative models for testlets. Applied Psychological Measurement, 30, 3–21.
44.
Maydeu-OlivaresA.BrownA. (2010). Item response modeling of paired comparison and ranking data. Multivariate Behavioral Research, 45, 935–974.
45.
Maydeu-OlivaresA.JoeH. (2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49, 305–328.
46.
MeadeA. W. (2010). A taxonomy of effect size measures for the differential functioning of items and scales. Journal of Applied Psychology, 95, 728–743.
47.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.
48.
MeijerR. R.SijtsmaK. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107–135.
49.
MuthénL. K.MuthénB. O. (1998-2017). Mplus user’s guide. 6th ed. Los Angeles, CA: Muthén & Muthén.
50.
NyeC. D.DrasgowF. (2011). Effect size indices for analyses of measurement equivalence: Understanding the practical importance of differences between groups. Journal of Applied Psychology, 96, 966–980.
51.
NyeC. D.NewmanD. A.JosephD. L. (2010). Never say “always”?: Extreme item wording effects on scalar invariance and item response curves. Organizational Research Methods, 13, 806–830.
52.
OrlandoM.ThissenD. (2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50–64.
53.
PatzR. J.JunkerB. W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24, 146–178.
54.
RajuN. S.LaffitteL. J.ByrneB. M. (2002). Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 87, 517–529.
55.
ReckaseM. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational and Behavioral Statistics, 4, 207–230.
56.
ReckaseM. (2009). Multidimensional item response theory. New York, NY: Springer.
57.
RobertsJ. S.DonoghueJ. R.LaughlinJ. E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24, 3–32.
58.
RobertsJ. S.LaughlinJ. E. (1996). A unidimensional item response model for unfolding responses from a graded disagree-agree response scale. Applied Psychological Measurement, 20, 231–255.
59.
RobertsJ. S.LaughlinJ. E.WedellD. H. (1999). Validity issues in the Likert and Thurstone approaches to attitude measurement. Educational and Psychological Measurement, 59, 211–233.
60.
RoznowskiM. (1989). Examination of the measurement properties of the Job Descriptive Index with experimental items. Journal of Applied Psychology, 74, 805–814.
61.
RubinD. B. (1984). Bayesianly justifiable and relevant frequency calculations for the applied statistician. Annals of Statistics, 12, 1151–1172.
62.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores(Psychometric Monograph No. 18). Iowa City, IA: Psychometric Society.
63.
SchwarzG. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.
64.
SinharayS. (2006). Bayesian item fit analysis for unidimensional item response theory models. British Journal of Mathematical and Statistical Psychology, 59, 429–449.
65.
SinharayS.JohnsonM. S.SternH. S. (2006). Posterior predictive assessment of item response theory models. Applied Psychological Measurement, 30, 298–321.
66.
SpiegelhalterD. J.BestN. G.CarlinB. P.Van Der LindeA. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64, 583–639.
67.
StarkS.ChernyshenkoO. S.ChanK. Y.LeeW. C.DrasgowF. (2001). Effects of the testing situation on item responding: Cause for concern. Journal of Applied Psychology, 86, 943–953.
68.
StarkS.ChernyshenkoO. S.DrasgowF. (2004). Examining the effects of differential item (functioning and differential) test functioning on selection decisions: When are statistically significant effects practically important?Journal of Applied Psychology, 89, 497–508.
69.
StarkS.ChernyshenkoO. S.DrasgowF.WhiteL. A. (2012). Adaptive testing with multidimensional pairwise preference items: Improving the efficiency of personality and other noncognitive assessments. Organizational Research Methods, 15, 463–487.
70.
StarkS.ChernyshenkoO. S.DrasgowF.WilliamsB. A. (2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring?Journal of Applied Psychology, 91, 25–39.
71.
StarkS.ChernyshenkoO. S.DrasgowF.NyeC. D.WhiteL. A.HeffnerT.FarmerW. L. (2014). From ABLE to TAPAS: A new generation of personality tests to support military selection and classification decisions. Military Psychology, 26, 138–152.
72.
StarkS.ChernyshenkoO. S.NyeC. D.DrasgowF.WhiteL. A. (2017). Moderators of the Tailored Adaptive Personality Assessment System Validity(Tech. Rep. 1357). Arlington, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
73.
Suárez-FalcónJ. C.GlasC. A. (2003). Evaluation of global testing procedures for item fit to the Rasch model. British Journal of Mathematical and Statistical Psychology, 56, 127–143.
74.
TayL.AliU. S.DrasgowF.WilliamsB. (2011). Fitting IRT models to dichotomous and polytomous data: Assessing the relative model-data fit of ideal point and dominance models. Applied Psychological Measurement, 35, 280–295.
75.
TayL.DrasgowF. (2012). Adjusting the adjusted χ2/df ratio statistic for dichotomous item response theory analyses: Does the model fit?. Educational and Psychological Measurement, 72, 510–528.
76.
TayL.DrasgowF.RoundsJ.WilliamsB. A. (2009). Fitting measurement models to vocational interest data: Are dominance models ideal?Journal of Applied Psychology, 94, 1287–1304.
77.
TayL.MeadeA. W.CaoM. (2015). An overview and practical guide to IRT measurement equivalence analysis. Organizational Research Methods, 18, 3–46.
78.
ThurstoneL. L. (1927). A law of comparative judgment. Psychological Review, 34, 273–286.
79.
ThurstoneL. L. (1928). Attitudes can be measured. American Journal of Sociology, 33, 529–554.
80.
VandenbergR. J.LanceC. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4–70.
81.
WangW. (2014). A Bayesian Markov chain Monte Carlo approach to the generalized graded unfolding model estimation: The future of non-cognitive measurement (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign.
82.
YenW. M. (1981). Using simulation results to choose a latent trait model. Applied Psychological Measurement, 5, 245–262.
83.
YenW. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125–145.
84.
ZhuX.StoneC. A. (2012). Bayesian comparison of alternative graded response models for performance assessment applications. Educational and Psychological Measurement, 72, 774–799.
85.
ZickarM. J.GibbyR. E.RobieC. (2004). Uncovering faking samples in applicant, incumbent, and experimental data sets: An application of mixed-model item response theory. Organizational Research Methods, 7, 168–190.
86.
ZickarM. J.RobieC. (1999). Modeling faking good on personality items: An item-level analysis. Journal of Applied Psychology, 84, 551–563.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
