Measurement involves numerous theoretical and empirical steps—ensuring our measures are operating the same in different groups is one step. Measurement invariance occurs when the factor loadings and item intercepts or thresholds of a scale operate similarly for people at the same level of the latent variable in different groups. This is commonly assumed to mean the scale is measuring the same thing in those groups. Here we test the assumption of extending measurement invariance to mean common measurement by randomly assigning American adults (N = 1500) to fill out scales assessing a coherent factor (search for meaning in life) or a nonsense factor measuring nothing. We find a nonsense scale with items measuring nothing shows strong measurement invariance with the original scale, is reliable, and covaries with other constructs. We show measurement invariance can occur without measurement. Thus, we cannot infer that measurement invariance means one is measuring the same thing, it may be a necessary but not a sufficient condition.
BorsboomD. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.
3.
BorsboomD.MellenberghG. J.Van HeerdenJ. (2003). The theoretical status of latent variables. Psychological Review, 110(2), 203–219. https://doi.org/10.1037/0033-295X.110.2.203
4.
BowersE. P.LiY.KielyM. K.BrittianA.LernerJ. V.LernerR. M. (2010). The five Cs model of positive youth development: A longitudinal analysis of confirmatory factor structure and measurement invariance. Journal of Youth and Adolescence, 39(7), 720–735. https://doi.org/10.1007/s10964-010-9530-9
5.
BussA. R.RoyceJ. R. (1975). Detecting cross-cultural commonalities and differences: Intergroup factor analysis. Psychological Bulletin, 82(1), 128–136. https://doi.org/10.1037/h0076156
6.
ByrneB. M.ShavelsonR. J.MuthénB. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105(3), 456–466. https://doi.org/10.1037/0033-2909.105.3.456
7.
ChenF. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504.
8.
ChenF. F. (2008). What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research. Journal of Personality and Social Psychology, 95(5), 1005.
9.
CheungG. W.RensvoldR. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5
10.
CounsellA.CribbieR. A.FloraD. B. (2020). Evaluating equivalence testing methods for measurement invariance. Multivariate Behavioral Research, 55(2), 312–328. https://doi.org/10.1080/00273171.2019.1633617
11.
CunninghamW. R. (1980). Age comparative factor analysis of ability variables in adulthood and old age. Intelligence, 4(2), 133–149. https://doi.org/10.1016/0160-2896(80)90011-2
12.
DolanC. V.ColomR.AbadF. J.WichertsJ. M.HessenD. J.VanDeSluisS. (2006). Multi-group covariance and mean structure modeling of the relationship between the WAIS-III common factors and sex and educational attainment in Spain. Intelligence, 34(2), 193–210. https://doi.org/10.1016/j.intell.2005.09.003
13.
DweckC. S.ChiuC. Y.HongY. Y. (1995). Implicit theories and their role in judgments and reactions: A word from two perspectives. Psychological Inquiry, 6(4), 267–285. https://doi.org/10.1207/s15327965pli0604_1
14.
FlakeJ. K.LuongR. (2023). Measurement invariance testing using confirmatory factor analysis and alignment optimization: A tutorial for transparent analysis planning and reporting. Psychological Methods, 28, 905–924.
15.
GignacG. E.SzodoraiE. T. (2016). Effect size guidelines for individual differences researchers. Personality and Individual Differences, 102, 74–78. https://doi.org/10.1016/j.paid.2016.06.069
16.
GriceH. P. (1975). Logic and conversation. In ColeP.MorganJ. L. (Eds.), Syntax and semantics, 3: Speech acts (pp. 41–58). Academic Press. https://doi.org/10.1163/9789004368811_003
17.
HornJ. L.McArdleJ. J. (1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18(3), 117–144. https://doi.org/10.1080/03610739208253916
18.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
19.
IrvineS. H. (1969). Contributions of ability and attainment testing in Africa to a general theory of intellect. Journal of Biosocial Science, 1(S1), 91–102. https://doi.org/10.1017/S0021932000023245
20.
JöreskogK. G. (1971). Simultaneous factor analysis in several populations. Psychometrika, 36(4), 409–426. https://doi.org/10.1007/BF02291366
21.
JungE.YoonM. (2017). Two-step approach to partial factorial invariance: Selecting a reference variable and identifying the source of noninvariance. Structural Equation Modeling: A Multidisciplinary Journal, 24(1), 65–79. https://doi.org/10.1080/10705511.2016.1251845
22.
KimS.VandenabeeleW.WrightB. E.AndersenL. B.CeraseF. P.ChristensenR. K.. . .PalidauskaiteJ. (2013). Investigating the structure and meaning of public service motivation across populations: Developing an international instrument and addressing issues of measurement invariance. Journal of Public Administration Research and Theory, 23(1), 79–102. https://doi.org/10.1093/jopart/mus027
LiC. H. (2016). The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychological Methods, 21(3), 369–387. https://doi.org/10.1037/met0000093
25.
LittleT. D. (2013). Longitudinal structural equation modeling. Guilford Press.
26.
LuceR. D.SuppesP.KrantzD. H. (2007). Foundations of measurement: Representation, axiomatization, and invariance (Vol. 3). Courier Corporation.
27.
LuongR.FlakeJ. K. (2023). Measurement invariance testing using confirmatory factor analysis and alignment optimization: A tutorial for transparent analysis planning and reporting. Psychological Methods, 28(4), 905.
28.
MarkusK. A.BorsboomD. (2013). Frontiers of test validity theory: Measurement, causation, and meaning. Routledge. https://doi.org/10.4324/9780203501207
29.
MarshH. W.LüdtkeO.MuthénB.AsparouhovT.MorinA. J. S.TrautweinU.NagengastB. (2010). A new look at the Big-Five factor structure through exploratory structural equation modeling. Psychological Assessment, 22, 471–491. https://doi.org/10.1037/a0019227
30.
MarshH. W.MuthénB.AsparouhovA.LüdtkeO.RobitzschA.MorinA. J. S.TrautweinU. (2009). Exploratory structural equation modeling, integrating CFA and EFA: Application to students’ evaluations of university teaching. Structural Equation Modeling, 16, 439–476. https://doi.org/10.1080/10705510903008220
31.
MaulA. (2017). Rethinking traditional methods of survey validation. Measurement: Interdisciplinary Research and Perspectives, 15(2), 51–69. https://doi.org/10.1080/15366367.2017.1348108
32.
McGrawB.JöreskogK. G. (1971). Factorial invariance of ability measures in groups differing in intelligence and social economic status. British Journal of Mathematical and Statistical Psychology, 24(2), 154–168. https://doi.org/10.1111/j.2044-8317.1971.tb00463.x
33.
McNeishD.WolfM. G. (2021). Dynamic fit index cutoffs for confirmatory factor analysis models. Psychological Methods, 28, 61–68. https://doi.org/10.1037/met0000425
MillsapR. E. (1997). Invariance in measurement and prediction: Their relationship in the single-factor case. Psychological Methods, 2(3), 248–260. https://doi.org/10.1037/1082-989X.2.3.248
40.
MuthénB.AsparouhovT. (2018). Recent methods for the study of measurement invariance with many groups: Alignment and random effects. Sociological Methods & Research, 47(4), 637–664.
41.
NadelhofferT.ShepardJ.NahmiasE.SripadaC.RossL. T. (2014). The free will inventory: Measuring beliefs about agency and responsibility. Consciousness and cognition, 25, 27–41.
42.
PagesR.ProtzkoJ.BaileyD. H. (2022). The breadth of impacts from the abecedarian project early intervention on cognitive skills. Journal of Research on Educational Effectiveness, 15(2), 243–262.
43.
ProtzkoJ. (2016). Does the raising IQ-raising g distinction explain the fadeout effect?. Intelligence, 56, 65–71.
44.
ProtzkoJ.ZedeliusC. M.SchoolerJ. W. (2019). Rushing to appear virtuous: Time pressure increases socially desirable responding. Psychological science, 30(11), 1584–1591.
45.
PutnickD. L.BornsteinM. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental review, 41, 71–90.
46.
RobitzschA.LüdtkeO. (2022). Why measurement invariance is not necessary for valid group comparisons. https://psyarxiv.com/cjyqp
47.
RockD. A.WertsC. E.FlaugherR. L. (1978). The use of analysis of covariance structures for comparing the psychometric properties of multiple variables across populations. Multivariate Behavioral Research, 13(4), 403–418. https://doi.org/10.1207/s15327906mbr1304_3
48.
SassD. A.SchmittT. A.MarshH. W. (2014). Evaluating model fit with ordered categorical data within a measurement invariance framework: A comparison of estimators. Structural Equation Modeling: A Multidisciplinary Journal, 21(2), 167–180. https://doi.org/10.1080/10705511.2014.882658
49.
SatorraA.BentlerP. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In von EyeA.CloggC. C. (Eds.), Latent variable analysis: Applications for developmental research (pp. 399–419). Sage.
50.
StegerM. F.FrazierP.OishiS.KalerM. (2006). The meaning in life questionnaire: Assessing the presence of and search for meaning in life. Journal of Counseling Psychology, 53(1), 80–93. https://doi.org/10.1037/0022-0167.53.1.80
51.
SteinmetzH.SchmidtP.Tina-BoohA.WieczorekS.SchwartzS. H. (2009). Testing measurement invariance using multigroup CFA: Differences between educational groups in human values measurement. Quality & Quantity, 43(4), 599–616. https://doi.org/10.1007/s11135-007-9143-x
52.
TuckerL. R.LewisC. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10. https://doi.org/10.1007/BF02291170
53.
UherJ. (2021). Psychometrics is not measurement: Unraveling a fundamental misconception in quantitative psychology and the complex network of its underlying fallacies. Journal of Theoretical and Philosophical Psychology, 41(1), 58–84. https://doi.org/10.1037/teo0000176
54.
Van VaerenberghY.ThomasT. D. (2013). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25(2), 195–217. https://doi.org/10.1093/ijpor/eds021
55.
VernonP. E. (1967). Working papers on cross-cultural applications of factor analysis [Mimeographed Report]. Institute of Education, University of London.
56.
VernonP. E. (1969). Intelligence and cultural environment. Methuen.
57.
WelzelC.BrunkertL.KruseS.InglehartR. F. (2023). Non-invariance? An overstated problem with misconceived causes. Sociological Methods & Research, 52(3), 1368–1400.
58.
WelzelC.KruseS.BrunkertL. (2023). Against the Mainstream: On the Limitations of Non-Invariance Diagnostics: Response to Fischer et al. and Meuleman et al. Sociological Methods & Research, 52(3), 1438–1455.
59.
WetzelE.BrownA.HillP. L.ChungJ. M.RobinsR. W.RobertsB. W. (2017). The narcissism epidemic is dead; long live the narcissism epidemic. Psychological Science, 28, 1833–1847. https://doi.org/10.1177/0956797617724208
60.
WetzelE.DonnellanB.TrzesniewskiH. (2018). Generational Changes in Self-Esteem and Narcissism. The SAGE Handbook of Personality and Individual Differences: Volume II: Origins of Personality and Individual Differences, 132–145.
61.
WhitehurstG. J.ArnoldD. S.EpsteinJ. N.AngellA. L.SmithM.FischelJ. E. (1994). A picture book reading intervention in day care and home for children from low-income families. Developmental Psychology, 30(5), 679–689. https://doi.org/10.1037/0012-1649.30.5.679
62.
WidamanK. F.FerrerE.CongerR. D. (2010). Factorial invariance within longitudinal structural equation models: Measuring the same construct across time. Child Development Perspectives, 4(1), 10–18. https://doi.org/10.1111/j.1750-8606.2009.00110.x
63.
WidamanK. F.LittleT. D.GearyD. C.CormierP. (1992). Individual differences in the development of skill in mental addition: Internal and external validation of chronometric models. Learning and Individual Differences, 4(3), 167–213. https://doi.org/10.1016/1041-6080(92)90002-V
64.
YentesR. D.WilhelmF. (2018). careless: Procedures for computing indices of careless responding (R package version 1.2). https://github.com/ryentes/careless/.
