The value-added method of Haberman is arguably one of the most popular methods to evaluate the quality of subscores. According to the method, a subscore has added value if the reliability of the subscore is larger than a quantity referred to as the proportional reduction in mean squared error of the total score. This article shows how well-known statistical tests can be used to determine the added value of subscores and augmented subscores. The usefulness of the suggested tests is demonstrated using two operational data sets.
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B (Methodological), 57, 289–300.
2.
BoyerJ. E.PalachekA. D.SchucanyW. R. (1983). An empirical study of related correlation coefficients. Journal of Educational Statistics, 8, 75.
3.
BrennanR. L. (2012). Utility indexes for decisions about subscores (CASMA Research Report No. 33). Iowa City, IA: Center for Advanced Studies in Measurement and Assessment.
4.
DaiS.WangX.SvetinaD. (2016). subscore: Computing subscores in classical test theory and item response theory (R package Version 2.0).
5.
DiedenhofenB.MuschJ. (2015). cocor: A comprehensive solution for the statistical comparison of correlations. PLOS One, 10, e0121945.
6.
DunnO. J.ClarkV. (1969). Correlation coefficients measured on the same individuals. Journal of the American Statistical Association, 64, 366.
7.
EfronB. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7, 1–26.
8.
EfronB.TibshiraniR. J. (1993). An introduction to the bootstrap. New York, NY: Chapman and Hall.
9.
FeinbergR.JurichD. P. (2017). Guidelines for interpreting and reporting subscores. Educational Measurement: Issues and Practice, 36, 5–13.
10.
HabermanS. J. (2008). When can subscores have value?Journal of Educational and Behavioral Statistics, 33, 204–229.
11.
HedgesL. V.OlkinI. (1983). Joint distributions of some indices based on correlation coefficient. In CarlinS.AmemiyaT.GoodmanL. A. (Eds.), Studies in econometrics, time series, and multivariate statistics (pp. 437–454). New York, NY: Academic Press.
12.
HittnerJ. B.MayK.SilverN. C. (2003). A Monte Carlo evaluation of tests for comparing dependent correlations. The Journal of General Psychology, 130, 149–168.
13.
LiuY.RobinF.YooH.MannaV. (2018). Statistical properties of the GRE Psychology test subscores(Educational Testing Service Research Report No. 18–19). Princeton, NJ: Educational Testing Service.
14.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison Wesley.
15.
LyrenP. (2009). Reporting subscores from college admission tests. Practical Assessment, Research, and Evaluation, 14, 1–10.
16.
MeijerR. R.BoevA. J.TendeiroJ. N.BoskerR. J.AlbersC. J. (2017). The use of subscores in higher education: When is this useful?Frontiers in Psychology, 8. doi:10.3389/fpsyg.2017.00305
17.
OlkinI. (1967). Correlations revisited. In StanleyJ. C. (Ed.), Improving experimental design and statistical analysis (pp. 102–128). Chicago, IL: Rand McNally.
18.
OlkinI.FinnJ. D. (1990). Testing correlated correlations. Psychological Bulletin, 108, 330–333.
19.
OlkinI.FinnJ. D. (1995). Correlations redux. Psychological Bulletin, 118, 155–164.
20.
OlkinI.SiotaniM. (1976). Asymptotic distribution of functions of a correlation matrix. In IkedaS. (Ed.), Essays in probability and statistics (pp. 235–251). Tokyo, Japan: Shinko Tsusho.
21.
PearsonK.FilonL. N. G. (1898). Mathematical contributions to the theory of evolution. IV. On the probable errors of frequency constants and on the influence of random selection on variation and correlation. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 191, 229–311.
22.
PuhanG.SinharayS.HabermanS. J.LarkinK. (2010). The utility of augmented subscores in a licensure exam: An evaluation of methods using empirical data. Applied Measurement in Education, 23, 266–285.
23.
ReiseS. P.BonifayW. E.HavilandM. G. (2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95, 129–140.
24.
SawakiY.SinharayS. (2017). Do the TOEFL iBT section scores provide value-added information to stakeholders?Language Testing. Advance online publication. doi:10.1177/0265532217716731
25.
SinharayS. (2010). How often do subscores have added value? Results from operational and simulated data. Journal of Educational Measurement, 47, 150–174.
26.
SinharayS. (2013). A note on assessing the added value of subscores. Educational Measurement: Issues and Practice, 32, 38–42.
27.
SinharayS. (2018). An interpretation of augmented subscores and their added value in terms of parallel forms. Journal of Educational Measurement, 55, 177–193.
28.
SinharayS.HabermanS. J. (2008). Reporting subscores: A survey(ETS Research Memorandum No. 08–18). Princeton, NJ: ETS.
29.
SinharayS.HabermanS. J. (2014). An empirical investigation of population invariance in the value of subscores. International Journal of Testing, 14, 22–48.
30.
SteigerJ. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245–251.
31.
SteigerJ. H.HakstianA. R. (1982). The asymptotic distribution of elements of a correlation matrix: Theory and application. British Journal of Mathematical and Statistical Psychology, 35, 208–215.
32.
ThissenD. (2013). Using the testlet response model as a shortcut to multidimensional item response theory subscore computation. In MillsapR.van der ArkL.BoltD.WoodsC. (Eds.), New developments in quantitative psychology—Presentations from the 77th Annual Psychometric Society Meeting (pp. 29–40). New York, NY: Springer.
33.
WassersteinR. L.LazarN. A. (2016). The ASA’s statement on p-values: Context, process, and purpose. The American Statistician, 70, 129–133.
34.
WedmanJ.LyrenP. (2015). Methods for examining the psychometric quality of subscores: A review and application. Practical Assessment, Research, and Evaluation, 20, 1–14.
35.
WilliamsE. J. (1959). The comparison of regression variables. Journal of the Royal Statistical Society, Series B, 21, 396–399.
36.
YaoL. (2010). Reporting valid and reliable overall scores and domain scores. Journal of Educational Measurement, 47, 339–360.
