This study was conducted to evaluate alternative analysis strategies for the meta-analysis method of reliability generalization when the reliability estimates are not statistically independent. Five approaches to dealing with the violation of independence were implemented: ignoring the violation and treating each observation as independent, calculating one mean or median from each study, selecting only one observation per study, and using a mixed-effects model. Monte Carlo methods were used to simulate samples under known and controlled population conditions. The results suggest that the type of approach does not have a noticeable impact on the accuracy of the reliability results but that researchers should be cautious when the intraclass correlation is relatively large. The simulations in this study also resulted in very poor confidence band coverage.
Barcikowski, R.S. (1981). Satistical power with group mean as the unit of analysis.Journal of Educatonal Statistics, 6, 267-285.
2.
Barnes, L.L.B., Harp, D., & Jung, W.S. (2002). Reliability generalization of scores on the Spielberger State-Trait Anxiety Inventory. Educational and Psychological Measurement, 62, 603-618.
3.
Beck, A.T., Ward, C.H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 53-63.
4.
Becker, B.J. (2000). Multivariate meta-analysis. In H. E. A. Tinsley & S. Brown (Eds.), Handbook of applied and multivariate statistics and mathematical modeling (pp. 499-525). San Diego, CA: Academic Press.
5.
Becker, B.J., & Kim, J. (2002, April). Effects of ignoring nonindependence of effect sizes on Hedge's homogeneity test. Paper presented at the annual meeting of the American Educational Research Association , New Orleans, LA.
6.
Bem, S.L. (1974). The measurement of psychological androgyny. Journal of Consulting and Clinical Psychology, 42, 155-162.
7.
Beretevas, S.N., & Pastor, D.A. (2003). Using mixed-effects models in reliability generalization studies.Educational and Psychological Measurement, 63, 75-95.
8.
Bock, R.D. (1975). Multivariate statistical methods in behavioral research. New York: McGraw-Hill .
9.
Bonett, D.G. (2002). Sample size requirements for testing and estimating coefficient alpha. Journal of Educational and Behavioral Statistics , 27, 335-340.
10.
Bradley, J.V. (1978). Robustness? British Journal of Mathematics and Statistical Psychology, 31, 144-152.
11.
Capraro, R.M., & Capraro, M.M. (2002). Myers-Briggs Type Indicator score reliability across studies: A meta-analytic reliability generalization study. Educational and Psychological Measurement, 62, 590-602.
12.
Caruso, J.C. (2000). Reliability generalization of NEO personality scales. Education and Psychological Measurement, 60, 236-254.
13.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York : Academic Press.
14.
Cooper, H. (1979). Statistically combining independent studies: A meta-analysis of sex differences in conformity research. Journal of Personality and Social Psychology, 37, 131-146.
15.
Costa, P.T., Jr., & McCrae , R.R. (1985). The NEO Personality Inventory: Professional manual. Odessa, FL: Psychological Assessment Resources.
16.
Crocker. L. & Angina, J. (1986). Introduction to classical and modern test theory . Toronto, Ontario, Canada: Holt Rinehart & Winston.
17.
Feldt, L.S. (1965). The approximate sampling distribution of Kuder-Richardson reliability coefficient twenty. Psychometrika, 30, 357-370.
18.
Fisher, R.A. (1950). Statistical methods for research workers. Edinburgh, UK: Oliver and Boyd.
19.
Fuller, J.B., & Hester, K. (1999). Comparing the sample-weighted and unweighted meta-analysis: An applied perspective. Journal of Management, 25, 803-828.
20.
Greenhouse, J.B., & Iyengar, S. (1994). Sensitivity analysis and diagnostics. In H. Cooper & L. Hedges (Eds.), The handbook of research synthesis (pp. 383-398). New York: Russell Sage.
21.
Hakstian, A.R., & Whalen, T.E. (1976). A k-sample significance test for independent alpha coefficients. Psychometrika, 41, 219-231.
22.
Hanson, B.A., & Beguin, A.A. (1999, April).Obtaining a common scale for IRT item parameters using separate versus concurrent estimation in the common item nonequivalent groups equating design. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Québec, Canada.
23.
Hedges, L.V., & Olkin, I. (1985).Statistical methods for meta-analysis. San Diego, CA: Academic Press, Inc.
24.
Helms, J.E. (1999). Another meta-analysis of the White Racial Identity Attitude Scale's Cronbach alphas: Implication for validity. Measurement and Evaluation in Counseling and Development, 32, 122-137.
25.
Henson, R.K. (2001). Understanding internal consistency reliabilities estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177-189.
26.
Hewitt-Gervais, C.M., & Kromrey, J.D. (1999, April). Analysis strategies for conducting F-tests with non-independent observations: An empirical investigation. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Québec, Canada.
27.
Hunter, J.E., & Schmidt, F.L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
28.
Iacobucci, D., & Duhachek, A. (2003). Advancing alpha: Measuring reliability with confidence . Journal of Consumer Psychology, 13, 478-487.
29.
Kenny, D.A., & Judd, C.M. (1986). Consequences of violating the independence assumption in analysis of variance. Psychological Bulletin, 99, 422-431.
30.
Koning, A.J., & Franses, P.H. (2003). Confidence intervals for Cronbach's coefficient alpha values (ERIM Report No. ERS-2003-041-MKT). Available at http://ssrn.com/abstract=423658
31.
Kromrey, J.D., & Dickinson, W.B. (1996). Detecting unit of analysis problems in nested designs: Statistical power and Type I error rates of the F test for groups-within-treatments effects.Educational and Psychological Measurement, 56, 215-231.
32.
Landman, J.T., & Dawes, R.M. (1982). Psychotherapy outcomes: Smith and Glass' conclusions stand up to scrutiny.American Psychologist, 37(5), 504-516.
33.
Maydeu-Olivares, A., Coffman, D.L., & Hartmann, W.M. (2007). Asymptotically distribution-free (ADF) interval estimation of coefficient alpha. Psychological Methods, 12, 157-176.
34.
Myers, I.B., & McCaulley, M.H. (1985). Manual: A guide to the development and use of the Myers-Briggs Type Indicator. Palo Alto, CA: Consulting Psychologists Press.
35.
Raudenbush, S.W. (1994). Random effects models. In H. Cooper & L. Hedges (Eds.), The handbook of research synthesis (pp. 301-321). New York: Russell Sage.
Raudenbush, S.W., & Bryk, A.S. (1987). Examining correlates of diversity. Journal of Educational Statistics, 12, 241-269.
38.
Robey, R., & Barcikowski, R. (1992). Type I error and the number of iterations in Monte Carlo studies of robustness. British Journal of Mathematics and Statistical Psychology45, 283-288.
39.
Romano, J.L., Kromrey, J.D., & Hibbard, S.T. (2008, March). Confidence interval methods for coefficient alpha: A Monte Carlo study of eight methods proposed in the literature. Paper presented at the annual meeting of the American Educational Research Association, New York.
40.
Rosenthal, R., & Rubin, D.B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes.Psychological Bulletin, 99, 400-406.
41.
Sawilowsky, S.S. (2000). Psychometrics versus datametrics: Comment on Vacha-Haase's ``reliability generalization'' method and some ERM editorial policies. Educational and Psychological Measurement, 60, 157-173.
42.
Scariano, S.M., & Davenport, J.M. (1987). The effects of violations of independence assumptions in an one-way ANOVA. The American Statistician, 41, 123-129.
43.
Schmidt, F.L., & Hunter, J.E. (1977). Development of a general solution to the problem of validity generalization.Journal of Applied Psychology, 62, 529-540.
44.
Smith, M.L., Glass, G.V., & Miller, T.I. (1980). The benefits of psychotherapy, Baltimore: Johns Hopkins Unversity Press.
45.
Spielberger, C.D., Gorsuch, R.L., & Lushene, R.E. (1970). Manual for the State-Trait Anxiety Inventory. Palo Alto, CA: Consulting Psychologists Press.
46.
Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement , 60, 174-195.
47.
Tracz, S.M., Elmore, P.B., & Pohlmann, J.T. (1992). Correlational meta-analysis: Independent and non independent cases. Educational and Psychological Measurement, 52, 879-888.
48.
Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.
49.
Vacha-Haase, T., Henson, R.K., & Caruso, J. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562-569.
50.
Vacha-Haase, T., Kogan, L.R., & Thompson. B. (2000). Sample composition and variabilities in published studies versus the in test manuals: Validity of score reliability inductions . Educational and Psychological Measurement, 60, 509-522.
51.
Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
52.
Yin, P., & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability Generalization across studies. Educational and Psychological Measurement, 61, 201-223.
