Differential item functioning (DIF) has been a long-standing problem in educational and psychological measurement. In practice, the source from which DIF originates can be complex in the sense that an item can show DIF on multiple background variables of different types simultaneously. Although a variety of non-item response theory-(IRT)-based and IRT-based DIF detection methods have been introduced, they do not sufficiently address the issue of DIF evaluation when its source is complex. The recently proposed least absolute shrinkage and selection operator (LASSO) regularization method has shown promising results of detecting DIF on multiple background variables. To provide more insight, in this study, we compared three DIF detection methods, including the non-IRT-based logistic regression (LR), the IRT-based likelihood ratio test (LRT), and LASSO regularization, through a comprehensive simulation and an empirical data analysis. We found that when multiple background variables were considered, the Type I error and Power rates of the three methods for identifying DIF items on one of the variables depended on not only the sample size and its DIF magnitude but also on the DIF magnitude of the other background variable and the correlation between them. We presented other findings and discussed the limitations and future research directions in this paper.
BauerD. J. (2017). A more general model for testing measurement invariance and differential item functioning. Psychological Methods, 22(3), 507. https://doi.org/10.1037/met0000077
2.
BauerD. J. (2023). Enhancing measurement validity in diverse populations: Modern approaches to evaluating differential item functioning. British Journal of Mathematical and Statistical Psychology, 76(3), 435–461. https://doi.org/10.1111/bmsp.12316
3.
BauerD. J.BelzakW. C. M.ColeV. T. (2019). Simplifying the assessment of measurement invariance over multiple background variables: Using regularized moderated nonlinear factor analysis to detect differential item functioning. Structural Equation Modeling: A Multidisciplinary Journal, 27(1), 43–55. https://doi.org/10.1080/10705511.2019.1642754
4.
BelzakW. C. M. (2023). The regDIF R package: Evaluating complex sources of measurement bias using regularized differential item functioning. Structural Equation Modeling: A Multidisciplinary Journal, 30(6), 974–984. https://doi.org/10.1080/10705511.2023.2170235
5.
BelzakW. C. M.BauerD. J. (2020). Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning. Psychological Methods, 25(6), 673. https://doi.org/10.1037/met0000253
6.
BelzakW. C. M.BauerD. J. (2024). Using regularization to identify measurement bias across multiple background characteristics: A penalized expectation–maximization algorithm. Journal of Educational and Behavioral Statistics, 49(6), 976–1012 . https://doi-org/10.3102/10769986231226439
7.
ChalmersR. P. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
8.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge.
9.
CurranP. J.ColeV. T.BauerD. J.HussongA. M.GottfredsonN. (2016). Improving factor score estimation through the use of observed background characteristics. Structural Equation Modeling, 23, 827–844. https://doi.org/10.1080/10705511.2016.1220839
10.
CurranP. J.ColeV. T.BauerD. J.RothenbergW. A.HussongA. M. (2018). Recovering predictor–criterion relations using covariate-informed factor score estimates. Structural Equation Modeling: A Multidisciplinary Journal, 25(6), 860–875. https://doi.org/10.1080/10705511.2018.1473773
11.
FinchW. H.Hernández FinchM. E. (2013). Investigation of specific learning disability and testing accommodations based differential item functioning using a multilevel multidimensional mixture item response theory model. Educational and Psychological Measurement, 73(6), 973–993. https://doi.org/10.1177/0013164413494776
HastieT.TibshiraniR.FriedmanJ. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7
14.
HollandP. W. (1985). On the study of differential item performance without IRT. Proceedings of the Military Testing Association, 1, 282–287.
HolmS. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 65–70. https://doi.org/10.2307/4615733
17.
HuangS.Svetina ValdiviaD. (2024). Wald χ2 test for differential item functioning detection with polytomous items in multilevel data. Educational and Psychological Measurement, 84(3), 530–548. https://doi.org/10.1177/00131644231181688
18.
KimE. S.YoonM. (2011). Testing measurement invariance: A comparison of multiple-group categorical CFA and IRT. Structural Equation Modeling: A Multidisciplinary Journal, 18(2), 212–228. https://doi.org/10.1080/10705511.2011.557337
19.
LeeH.KimK. Y. (2025). A comparison of anchor selection strategies for DIF analysis. Journal of Educational Measurement. https://doi.org/10.1111/jedm.12429
20.
LimH.ChoeE. M.HanK. T. (2022). A residual-based differential item functioning detection framework in item response theory. Journal of Educational Measurement, 59(1), 80–104. https://doi.org/10.1111/jedm.12313
21.
MaennerM. J.ShawK. A.BakianA. V.BilderD. A.DurkinM. S.EslerA.FurnierS. M.Hall-LandeJ.Hallas-MuchowL.HudsonA.HughesM. M.PatrickM.PierceK.PoynterJ. N.SalinasA.ShenoudaJ.VehornA.WarrenZ.CogswellM. E. (2023). Prevalence and characteristics of autism spectrum disorder among children aged 8 years—Autism and Developmental Disabilities Monitoring Network, 11 sites, United States, 2020. Morbidity and Mortality Weekly Report. Surveillance Summaries, 72(2), 1–14. https://doi.org/10.15585/mmwr.ss7202a1)
22.
MagisD.BélandS.TuerlinckxF.De BoeckP. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862. https://doi.org/10.3758/BRM.42.3.847
23.
MagisD.TuerlinckxF.De BoeckP. (2015). Detection of differential item functioning using the lasso approach. Journal of Educational and Behavioral Statistics, 40(2), 111–135. https://doi.org/10.3102/1076998614559747
NandakumarR.RoussosL. (2004). Evaluation of the CATSIB DIF procedure in a pretest setting. Journal of Educational and Behavioral Statistics, 29(2), 177–199. https://doi.org/10.3102/10769986029002177
26.
PaekI.WilsonM. (2011). Formulating the Rasch differential item functioning model under the marginal maximum likelihood estimation context and its comparison with Mantel–Haenszel procedure in short test and small sample conditions. Educational and Psychological Measurement, 71(6), 1023–1046. https://doi.org/10.1177/0013164411400734
PuhanG.MosesT. P.YuL.DoransN. J. (2009). Using log-linear smoothing to improve small-sample DIF Estimation. Journal of Educational Measurement, 46(1), 59–83. https://doi.org/10.1111/j.1745-3984.2009.01069.x
29.
R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
30.
RitvoR. A.RitvoE. R.GuthrieD.RitvoM. J.HufnagelD. H.McMahonW.EloffJ. (2011). The Ritvo Autism Asperger Diagnostic Scale-Revised (RAADS-R): A scale to assist the diagnosis of autism spectrum disorder in adults: An international validation study. Journal of Autism and Developmental Disorders, 41, 1076–1089. https://doi.org/10.1007/s10803-010-1133-5
31.
SchwartzmanB. C.WoodJ. J.KappS. K. (2016). Can the five factor model of personality account for the variability of autism symptom expression? Multivariate approaches to behavioral phenotyping in adult autism spectrum disorder. Journal of Autism and Developmental Disorders, 46(1), 253–272 . https://doi.org/10.1007/s10803-015-2571-x
32.
ShealyR.StoutW. (1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58(2), 159–194. https://doi.org/10.1007/BF02294572
33.
SturmA.HuangS.BalV.SchwartzmanB. (2024). Psychometric exploration of the RAADS-R with autistic adults: Implications for research and clinical practice. Autism, 28(9), 2334–2345. https://doi.org/10.1177/13623613241228329
34.
Svetina ValdiviaD.HuangS.BotterP. (2024, April). Detecting differential item functioning in presence of multilevel data: Do methods accounting for multilevel data structure make a DIFference? In Frontiers in Education (Vol. 9, p. 1389165). Frontiers Media SA. https://doi.org/10.3389/feduc.2024.1389165
35.
SwaminathanH.RogersH. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
36.
TibshiraniR. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
37.
TutzG.SchaubergerG. (2015). A penalty approach to differential item functioning in Rasch models. Psychometrika, 80(1), 21–43. https://doi.org/10.1007/s11336-013-9377-6
WangC.ZhuR.XuG. (2023). Using LASSO and adaptive LASSO to identify DIF in multidimensional 2PL models. Multivariate Behavioral Research, 58(2), 387–407. https://doi.org/10.1080/00273171.2021.1985950
40.
WangM.WoodsC. M. (2017). Anchor selection using the Wald test anchor-all-test-all procedure. Applied Psychological Measurement, 41(1), 17–29. https://doi.org/10.1177/0146621616668014
41.
WellsC. S.HambletonR. K.KirkpatrickR.MengY. (2014). An examination of two procedures for identifying consequential item parameter drift. Applied Measurement in Education, 27(3), 214–231. https://doi.org/10.1080/08957347.2014.905786
42.
WilsonM.De BoeckP.CarstensenC. H. (2006). Explanatory item response models: A brief introduction. In HartigJ.KliemeE.LeutnerD. (Eds.), Assessment of competencies in educational contexts (pp. 91–120). Hogrefe & Huber Publishers.
43.
WoodsC. M. (2009). Evaluation of MIMIC-model methods for DIF testing with comparison to two-group analysis. Multivariate Behavioral Research, 44(1), 1–27. https://doi.org/10.1080/00273170802620121
44.
WoodsC. M.CaiL.WangM. (2013). The Langer-improved Wald test for DIF testing with multiple groups: Evaluation and comparison to two-group IRT. Educational and Psychological Measurement, 73(3), 532–547. https://doi.org/10.1177/0013164412464875
45.
YuL.MosesT.PuhanG.DoransN. (2008). DIF detection with small samples: Applying smoothing techniques to frequency distributions in the Mantel-Haenszel procedure. ETS Research Report Series, 2008(2), i–33. https://doi.org/10.1002/j.2333-8504.2008.tb02130.x
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
