The article compares three different methods to estimate effects of task characteristics and to use these estimates for model-based proficiency scaling: prediction of item difficulties from the Rasch model, the linear logistic test model (LLTM), and an LLTM including random item effects (LLTM+e). The methods are applied to empirical data from a German large-scale study of reading comprehension in English as a foreign language (N = 10,543). A priori defined task characteristics were used as predictors for item difficulty; the estimated effects were used to define thresholds between proficiency levels. The comparison of results indicates that the LLTM is too restrictive; the Rasch model and the LLTM+e yield similar results in terms of implications for scale anchoring.
AldersonJ. C. (2000). Assessing reading. Cambridge, England: Cambridge University Press.
3.
AldersonJ. C.FiguerasN.KuijperH.NoldG.TakalaS.TardieuC. (2006). Analysing tests of reading and listening in relation to the common European framework of reference: The experience of the Dutch CEFR Construct Project. Language Assessment Quarterly, 3, 3-30. doi:10.1207/s15434311laq0301_210.1207/s15434311laq0301_2
4.
BeatonE.AllenN. (1992). Interpreting scales through scale anchoring. Journal of Educational Statistics, 17, 191-204. doi:10.3102/1076998601700219110.3102/10769986017002191
5.
BeckB.KliemeE. (Eds.). (2007). Sprachliche Kompetenzen. Konzepte und Messung [Language skills. Concepts and measurement]. Weinheim, Germany: Beltz. Retrieved from http://www.pedocs.de/volltexte/2010/3140/
6.
BorsboomD.MellenberghG. J.van HeerdenJ. (2004). The concept of validity. Psychological Review, 111,1061-1071. doi:10.1037/0033-295X.111.4.106110.1037/0033-295X.111.4.1061
7.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
8.
Council of Europe. (2001). A common European framework of reference for languages: Learning, teaching, assessment. Cambridge, England: Cambridge University Press.
9.
DanielR. C.EmbretsonS. E. (2010). Designing cognitive complexity in mathematical problem-solving items. Applied Psychological Measurement, 34, 348-364. doi:10.1177/014662160934980110.1177/0146621609349801
10.
De BoeckP.WilsonM.ActonG. S. (2005). A conceptual and psychometric framework for distinguishing categories and dimensions. Psychological Review, 112, 129-158. doi:10.1037/0033-295X.112.1.12910.1037/0033-295X.112.1.129
11.
DiBelloL. V.StoutW. F.RoussosL. A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In NicholsP. D.ChipmanS. F.BrennanR. L. (Eds.), Cognitively diagnostic assessment (pp. 361-389). Hillsdale, NJ: Lawrence Erlbaum.
12.
EmbretsonS. E. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179-197. doi:10.1037/0033-2909.93.1.17910.1037/0033-2909.93.1.179
13.
EmbretsonS. E. (1998). A cognitive design system approach for generating valid tests: Approaches to abstract reasoning. Psychological Methods, 3, 300-396. doi:10.1037/1082-989X.3.3.38010.1037/1082-989X.3.3.380
14.
EmbretsonS. E. (2006). The continued search for nonarbitrary metrics in psychology. American Psychologist, 61, 50-55. doi:10.1037/0003-066X.61.1.5010.1037/0003-066X.61.1.50
15.
FischerG. H. (1973). Linear logistic test model as an instrument in educational research. Acta Psychologica, 37, 359-374. doi:10.1016/0001-6918(73)90003-610.1016/0001-6918(73)90003-6
16.
FischerG. H. (1997). Unidimensional linear logistic Rasch models. In van der LindenW. J.HambletonR. K. (Eds.), Handbook of modern item response theory (pp. 225-243). New York, NY: Springer.
17.
FreedleR.KostinI. (1993). The prediction of TOEFL reading item difficulty: Implications for construct validity. Language Testing, 10, 133-170. doi:10.1177/02655322930100020310.1177/026553229301000203
18.
GorinJ. S.EmbretsonS. E. (2006). Item difficulty modeling of paragraph comprehension items. Applied Psychological Measurement, 30, 394-411. doi:10.1177/014662160628855410.1177/0146621606288554
19.
JanssenR.SchepersJ.PeresD. (2004). Models with item and item group predictors. In De BoeckP.WilsonM. (Eds.), Explanatory item response models: A generalized linear and nonlinear approach (pp. 189-212). New York, NY: Springer.
20.
JanssenR.TuerlinckxF.MeuldersM.DeBoeckP. (2003). A hierarchical IRT model for criterion-referenced measurement. Journal of Educational and Behavioral Statistics, 25, 285-306. doi:10.2307/116520710.2307/1165207
21.
KintschW. (1998). Comprehension. Cambridge, England: Cambridge University Press.
22.
LeeY.-W. (2004). Examining passage-related local item dependence (LID) and measurement construct using Q3 statistics in an EFL reading comprehension test. Language Testing, 21, 74-100. doi:10.1191/0265532204lt260oa10.1191/0265532204lt260oa
23.
LeightonJ. P.GierlM. J. (Eds.). (2007). Cognitive diagnostic assessment for education. Cambridge, England: Cambridge University Press.
24.
MeskauskasJ. A. (1976). Evaluation models for criterion-references standard setting. Review of Educational Research, 45, 133-158. doi:10.3102/0034654304600113310.3102/00346543046001133
25.
NoldG.RossaH. (2007). Leseverstehen [Listening comprehension]. In BeckB.KliemeE. (Eds.), Sprachliche Kompetenzen. Konzepte und Messung [Language competencies: Concepts and measurement]. Weinheim, Germany: Beltz.
26.
NoldG.RossaH.HartigJ. (2008). Proficiency scaling in DESI listening and reading EFL tests. Task characteristics, item difficulty and cut-off points. In TaylorL.WeirC. J. (Eds.), Multilingualism and assessment: Achieving transparency, assuring quality, sustaining diversity (pp. 94-116). Cambridge, England: Cambridge University Press.
27.
RudnerL. M. (2001). Computing the expected proportions of misclassified examinees. Practical Assessment, Research & Evaluation, 7. Retrieved from http://pareonline.net/getvn.asp?v=7&n=14
28.
TuerlinckxF.RijmenF.MolenberghsG.VerbekeG.BriggsD.Van den NoortgateW.MeuldersM.De BoeckP. (2004). Estimation and software. In De BoeckP.WilsonM. (Eds.), Explanatory item response models: A generalized linear and nonlinear approach (pp. 343-373). New York. NY: Springer.
29.
WangW.-C.WilsonM. (2005). Exploring local item dependence using a random-effects facet model. Applied Psychological Measurement, 29, 296-318. doi:10.1177/014662160527628110.1177/0146621605276281
WilsonM.De BoeckP. (2004). Descriptive and explanatory item response models. In De BoeckP.WilsonM. (Eds.), Explanatory item response models: A generalized linear and nonlinear approach (pp. 43-74). New York, NY: Springer.
