Claims about human behavior have been hampered by limited availability of comparable data across cultures. Invariance testing has been proposed to address questions about the comparability of data, yet statistical methods have been challenged on various grounds across psychology and related fields. We highlight how current debates confuse distinct issues and fail to consider the role of data within science. We aim to overcome the impasse by a) summarizing various criticisms, b) distinguishing five mapping processes that occur during a typical research project and c) exploring how thinking about invariance as mapping can move current discussions forward. Specifically, we differentiate 1) mapping ideas to theoretical constructs and concepts, 2) mapping constructs to stimuli, 3) mapping participants’ responses to stimuli onto numerical representations, 4) testing internal relations of stimuli responses (the typical focus of statistical invariance testing) and 5) mapping empirical observations back to theoretical statements. We treat invariance testing as a theory-guided process that offers important insights about instruments, construct validity and psychological theories throughout the research process that are currently missed by focusing on only the statistical details. Psychological claims that are valid for all humans depend on questions of invariance in the broad sense that we outline here.
AdamantiosD.PapadopoulosN. (2010). Assessing the cross-national invariance of formative measures: Guidelines for international business researchers. Journal of International Business Studies, 41(2), 360–370.
2.
AlexandrovaA.HaybronD. M. (2016). Is construct validation valid?Philosophy of Science, 83(5), 1098–1109. https://doi.org/10.1086/687941
3.
BauerD. J.BelzakW. C. M.ColeV. (2020). Simplifying the assessment of measurement invariance over multiple background variables: Using regularized moderated nonlinear factor analysis to detect differential item functioning. Structural Equation Modeling: A Multidisciplinary Journal, 27(1), 43–55. https://doi.org/10.1080/10705511.2019.1642754
4.
BoehnkeK. (2022). Let’s compare apples and oranges! A plea to demystify measurement equivalence. American Psychologist, 77(9), 1160–1168. https://doi.org/10.1037/amp0001080
5.
BoerD.HankeK.HeJ. (2018). On detecting systematic measurement error in cross-cultural research: A Review and critical reflection on equivalence and invariance tests. Journal of Cross-Cultural Psychology, 49(5), 713–734. https://doi.org/10.1177/0022022117749042
6.
BorsboomD.DesernoM. K.RhemtullaM.EpskampS.FriedE. I.McNallyR. J.RobinaughD. J.PeruginiM.DalegeJ.CostantiniG.IsvoranuA.-M.WysockiA. C.van BorkuloC. D.van BorkR.WaldorpL. J. (2021). Network analysis of multivariate data in psychological science. Nature Reviews Methods Primers, 1(1). Article 1. https://doi.org/10.1038/s43586-021-00055-w
7.
BorsboomD.MellenberghG. J.van HeerdenJ. (2003). The theoretical status of latent variables. Psychological Review, 110(2), 203–219. https://doi.org/10.1037/0033-295X.110.2.203
8.
BreugelmansS. M.PoortingaY. H. (2006). Emotion without a word: Shame and guilt among Rarámuri Indians and rural Javanese. Journal of Personality and Social Psychology, 91(6), 1111–1122. https://doi.org/10.1037/0022-3514.91.6.1111
9.
ChidlowA.PlakoyiannakiE.WelchC. (2014). Translation in cross-language international business research: Beyond equivalence. Journal of International Business Studies, 45(5), 562–582. https://doi.org/10.1057/jibs.2013.67
10.
CramerA. O. J.Van Der SluisS.NoordhofA.WichersM.GeschwindN.AggenS. H.KendlerK. S.BorsboomD. (2012). Dimensions of normal personality as networks in search of equilibrium: You can’t like parties if you don’t like people. European Journal of Personality, 26(4), 414–431. https://doi.org/10.1002/per.1866
11.
DavidovE.DülmerH.SchlüterE.SchmidtP.MeulemanB. (2012). Using a multilevel structural equation modeling approach to explain cross-cultural measurement noninvariance. Journal of Cross-Cultural Psychology, 43(4), 558–575. https://doi.org/10.1177/0022022112438397
12.
DeffnerD.RohrerJ. M.McElreathR. (2022). A causal framework for cross-cultural generalizability. Advances in Methods and Practices in Psychological Science, 5(3), 25152459221106366. https://doi.org/10.1177/25152459221106366
13.
EdwardsJ. R.BagozziR. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5(2), 155–174. https://doi.org/10.1037/1082-989x.5.2.155
14.
FaulknerS. L.BaldwinJ. R.LindsleyS. L.HechtM. L. (2006). Layers of meaning: An analysis of definitions of culture. In Redefining culture: Perspectives across the disciplines (pp. 27–52). Lawrence Erlbaum Associates Publishers.
15.
FischerR. (2009). Where is culture in cross cultural research? An outline of a multilevel research process for measuring culture as a shared meaning system. International Journal of Cross Cultural Management, 9(1), 25–49.
16.
FischerR. (2021a). Alternative four-factor structure of the Mini-IPIP in Thailand. International Journal of Personality Psychology, 7, 35–42. https://doi.org/10.21827/ijpp.7.37978
17.
FischerR. (2021b). Origins of values differences: A two-level analysis of economic, climatic and parasite stress explanations in the value domain. Cross-Cultural Research, 55(5), 438–473. https://doi.org/10.1177/10693971211031476
18.
FischerR.KarlJ. A. (2019). A primer to (cross-cultural) multi-group invariance testing possibilities in R. Frontiers in Psychology, 1507.
19.
FischerR.KarlJ. A. (2023). Niche diversity effects on personality measurement – evidence from large national samples during the COVID-19 pandemic. Current Research in Ecological and Social Psychology, 4, 100116. https://doi.org/10.1016/j.cresp.2023.100116
20.
FischerR.PoortingaY. H. (2018). Addressing methodological challenges in culture-comparative research. Journal of Cross-Cultural Psychology, 49(5), 691–712.
21.
FischerR.RudnevM. (2024). From MIsgivings to MIse-en-scène: The role of invariance in personality science. European Journal of Personality, Advanced Online Publication. https://doi.org/10.1177/08902070241283081
22.
FischerR.SchwartzS. (2011). Whence differences in value priorities? Individual, cultural, or artifactual sources. Journal of Cross-Cultural Psychology, 42(7), 1127–1144.
FontaineJ. R. J. (2008). Traditional and multilevel approaches in cross-cultural research: An integration of methodological frameworks. In Multilevel analysis of individuals and cultures (pp. 65–92). Taylor & Francis Group/Lawrence Erlbaum Associates.
25.
FunderD. C.GardinerG. (2024). MIsgivings about measurement invariance. European Journal of Personality, Advanced Online Publication. https://doi.org/10.1177/08902070241228338
GurvenM.von RuedenC.MassenkoffM.KaplanH.Lero VieM. (2013). How universal is the big five? Testing the five-factor model of personality variation among forager-farmers in the Bolivian amazon. Journal of Personality and Social Psychology, 104(2), 354–370. https://doi.org/10.1037/a0030841
28.
HarknessJ. A.VijverF. J. R. van deMohlerP. P. (Eds.), (2003). Cross-cultural survey methods. J. Wiley.
29.
HauckeM.HoekstraR.van RavenzwaaijD. (2021). When numbers fail: Do researchers agree on operationalization of published research?Royal Society Open Science, 8(9), 191354. https://doi.org/10.1098/rsos.191354
30.
HenselerJ.RingleC. M.SarstedtM. (2016). Testing measurement invariance of composites using partial least squares. International Marketing Review, 33(3), 405–431. https://doi.org/10.1108/IMR-09-2014-0304
31.
HofstedeG. (1980). Culture’s consequences: International differences in work-related values. Sage Publications.
32.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1). Article 1. https://doi.org/10.1080/10705519909540118
33.
JacksonJ. C.WattsJ.HenryT. R.ListJ.-M.ForkelR.MuchaP. J.GreenhillS. J.GrayR. D.LindquistK. A. (2019). Emotion semantics show both cultural variation and universal structure. Science, 366(6472), 1517–1522. https://doi.org/10.1126/science.aaw8160
34.
KangY.McNeishD. M.HancockG. R. (2016). The role of measurement quality on practical guidelines for assessing measurement and structural invariance. Educational and Psychological Measurement, 76(4), 533–561. https://doi.org/10.1177/0013164415603764
35.
KarlJ.FischerR. (2022). More than yes and No: Predicting the magnitude of non-invariance between countries from systematic features. In Klicperova-BakerM.FriedlmeierW. (Eds.), Xenophobia vs. Patriotism: Where is my Home?Proceedings from the 25th Congress of the International Association for Cross-Cultural Psychology, 300, pages 1–21. International Association for Cross-Cultural Psychology. https://doi.org/10.4087/ELED5219
36.
KleinK. J.TosiH.CannellaA. A. (1999). Introduction to special topic forum: Multilevel theory building: Benefits, barriers, and new developments. Academy of Management Review, 24(2), 243–248.
37.
KruisJ.MarisG. (2016). Three representations of the Ising model. Scientific Reports, 6(1). Article 1. https://doi.org/10.1038/srep34175
38.
LakatosI. (1970). Falsification and the methodology of scientific research programmes. In MusgraveA.LakatosI. (Eds.), Criticism and the growth of knowledge: Proceedings of the international colloquium in the philosophy of science, 1965 (Vol. 4, pp. 91–196). Cambridge University Press. https://doi.org/10.1017/CBO9781139171434.009
39.
LambertL. S.NewmanD. A. (2022). Construct development and validation in three practical steps: Recommendations for reviewers. Organizational Research Methods, 109442812211153. https://doi.org/10.1177/10944281221115374
40.
LeitgöbH.SeddigD.AsparouhovT.BehrD.DavidovE.De RooverK.JakS.MeitingerK.MenoldN.MuthénB.RudnevM.SchmidtP.van de SchootR. (2023). Measurement invariance in the social sciences: Historical development, methodological challenges, state of the art, and future perspectives. Social Science Research, 110, 102805. https://doi.org/10.1016/j.ssresearch.2022.102805
41.
LordF. M. (1953). The relation of test score to the trait underlying the test. Educational and Psychological Measurement, 13(4), 517–549. https://doi.org/10.1177/001316445301300401
42.
MaassenE.D’UrsoE. D.Van AssenM. A. L. M.NuijtenM. B.De RooverK.WichertsJ. M. (2023). The dire disregard of measurement invariance testing in psychological science. Psychological Methods. https://doi.org/10.1037/met0000624
43.
MaraunM. D. (1998). Measurement as a normative practice: Implications of wittgenstein’s philosophy for measurement in psychology. Theory & Psychology, 8(4), 435–461. https://doi.org/10.1177/0959354398084001
44.
MarshH. W.HauK.-T.WenZ. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing hu and bentler’s (1999) findings. Structural Equation Modeling: A Multidisciplinary Journal, 11(3). Article 3. https://doi.org/10.1207/s15328007sem1103_2
45.
MarsmanM.BorsboomD.KruisJ.EpskampS.BorkR. VWaldorpL. J.MaasH. L. J. V DMarisG. (2018). An introduction to network psychometrics: Relating ising network models to item response theory models. Multivariate Behavioral Research, 53(1), 15–35. https://doi.org/10.1080/00273171.2017.1379379
46.
MeehlP. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46(4), 806–834. https://doi.org/10.1037/0022-006X.46.4.806
47.
MeehlP. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1(2), 108–141. https://doi.org/10.1207/s15327965pli0102_1
48.
MeitingerK.BehrD. (2016). Comparing cognitive interviewing and online probing: Do they find similar results?Field Methods, 28(4), 363–380. https://doi.org/10.1177/1525822X15625866
MeulemanB.ŻółtakT.PokropekA.DavidovE.MuthénB.OberskiD. L.BillietJ.SchmidtP. (2022). Why Measurement Invariance is Important in Comparative Research. A Response to Welzel et al. (2021). Sociological Methods & Research, 00491241221091755. https://doi.org/10.1177/00491241221091755
MorrisM. W. (2014). Values as the essence of culture: Foundation or fallacy?Journal of Cross-Cultural Psychology, 45(1), 14–24. https://doi.org/10.1177/0022022113513400
54.
NyeC. D.DrasgowF. (2011). Effect size indices for analyses of measurement equivalence: Understanding the practical importance of differences between groups. Journal of Applied Psychology, 96(5), 966–980. https://doi.org/10.1037/a0022955
55.
PodsakoffP. M.MacKenzieS. B.PodsakoffN. P. (2016). Recommendations for creating better concept definitions in the organizational, behavioral, and social sciences. Organizational Research Methods, 19(2), 159–203. https://doi.org/10.1177/1094428115624965
56.
RobitzschA.LüdtkeO. (2023). Why full, partial, or approximate measurement invariance are not a prerequisite for meaningful and valid group comparisons. Structural Equation Modeling: A Multidisciplinary Journal, 1–12. https://doi.org/10.1080/10705511.2023.2191292
57.
RutkowskiL.SvetinaD. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74(1), 31–57. https://doi.org/10.1177/0013164413498257
58.
SaucierG.KennerJ.IurinoK.Bou MalhamP.ChenZ.ThalmayerA. G.KemmelmeierM.TovW.BouttiR.MetaferiaH.ÇankayaB.MastorK. A.HsuK.-Y.WuR.ManiruzzamanM.RugiraJ.TsaousisI.SosnyukO.Regmi AdhikaryJ.AltschulC. (2015). Cross-cultural differences in a global “survey of world views.”. Journal of Cross-Cultural Psychology, 46(1), 53–70. https://doi.org/10.1177/0022022114551791
59.
SauterD. A.EisnerF.EkmanP.ScottS. K. (2015). Emotional vocalizations are recognized across cultures regardless of the valence of distractors. Psychological Science, 26(3), 354–356. https://doi.org/10.1177/0956797614560771
60.
SchwabaT.RhemtullaM.HopwoodC. J.BleidornW. (2020). A facet atlas: Visualizing networks that describe the blends, cores, and peripheries of personality structure. PLoS One, 15(7), e0236893. https://doi.org/10.1371/journal.pone.0236893
61.
SchwartzS. H. (2014). Rethinking the concept and measurement of societal culture in light of empirical findings. Journal of Cross-Cultural Psychology, 45, 5–13. https://doi.org/10.1177/0022022113490830
SokolovB. (2018). The index of emancipative values: Measurement model misspecifications. American Political Science Review, 112(2), 395–408. https://doi.org/10.1017/S0003055417000624
64.
SteenkampJ.-B. E. M.BaumgartnerH. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25(1), 78–90. https://doi.org/10.1086/209528
65.
SternerP.PargentF.DeffnerD.GoretzkoD. (2024). A causal framework for the comparability of latent variables. Structural Equation Modeling: A Multidisciplinary Journal, 0(0), 1–12. https://doi.org/10.1080/10705511.2024.2339396
StiversT.EnfieldN. J.BrownP.EnglertC.HayashiM.HeinemannT.HoymannG.RossanoF.de RuiterJ. P.YoonK.-E.LevinsonS. C. (2009). Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106(26), 10587–10592. https://doi.org/10.1073/pnas.0903616106
TavesM. G. W.ElliottI.MaulA. A (2021). Survey item validation. In The routledge handbook of research methods in the study of religion (2nd ed.). Routledge.
70.
van BorkuloC. D.van BorkR.BoschlooL.KossakowskiJ. J.TioP.SchoeversR. A.BorsboomD.WaldorpL. J. (2022). Comparing network structures on three aspects: A permutation test. Psychological Methods. https://doi.org/10.1037/met0000476
71.
VandenbergR. J.LanceC. E. (2000). A Review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002
72.
VanderWeeleT. J. (2022). Constructed measures and causal inference: Towards a new model of measurement for psychosocial constructs. Epidemiology, 33(1), 141. https://doi.org/10.1097/EDE.0000000000001434
73.
VanderWeeleT. J.VansteelandtS. (2022). A statistical test to reject the structural interpretation of a latent factor model. Journal of the Royal Statistical Society - Series B: Statistical Methodology, 84(5), 2032–2054. https://doi.org/10.1111/rssb.12555
74.
van de VijverF. J. R.LeungK. (1997). Methods and data analysis for cross-cultural research (pp. xiii–186). Sage Publications, Inc.
75.
van de VijverF. J. R.LeungK. (2021). FetvadjievV. H.HeJ.FontaineJ. R. J. (Eds.), Methods and data analysis for cross-cultural research (2nd ed.). Cambridge University Press. https://doi.org/10.1017/9781107415188
76.
Van De VijverF. J. R.PoortingaY. H. (1982). Cross-cultural generalization and universality. Journal of Cross-Cultural Psychology, 13(4), 387–408. https://doi.org/10.1177/0022002182013004001
77.
Van De VijverF. J. R.van HemertD. A.PoortingaY. H. (Eds.), (2008). Multilevel analysis of individuals and cultures. Lawrence Erlbaum Associates.
78.
Van HerkH.GoldmanS. P. K. (2022). The advancement of measurement invariance testing in cross-cultural research in the period 1999–2020. Executing rather than scrutinizing? In BaumgartnerH.WeijtersB. (Eds.), Measurement in marketing (19, pp. 95–119). Emerald Publishing Limited. https://doi.org/10.1108/S1548-643520220000019005
79.
WanY. (2021). Why are they so quiet? Exploring reticent and passive east Asian ESL Students in the U.S. Classrooms. Open Journal of Modern Linguistics, 11(6). Article 6. https://doi.org/10.4236/ojml.2021.116073
80.
WelzelC.BrunkertL.KruseS.InglehartR. F. (2021). Non-invariance? An Overstated problem with misconceived causes. Sociological Methods & Research, 0049124121995521. https://doi.org/10.1177/0049124121995521
81.
WelzelC.InglehartR. F. (2016). Misconceptions of measurement equivalence: Time for a paradigm shift. Comparative Political Studies, 49(8), 1068–1094. https://doi.org/10.1177/0010414016628275
82.
ZumboB. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233. https://doi.org/10.1080/15434300701375832
83.
ZyphurM. J.KaplanS. A.ChristianM. S. (2008). Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: Problems and solutions. Group Dynamics: Theory, Research, and Practice, 12(2), 127–140. https://doi.org/10.1037/1089-2699.12.2.127
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
