The authors conducted Monte Carlo simulations to compare the Hedges and Olkin, the Hunter and Schmidt, and a refinement of the Aguinis and Pierce meta-analytic approaches for estimating moderating effects of categorical variables. The simulation examined binary moderator variables (e.g., gender—male, female; ethnicity—majority, minority). The authors compared the three meta-analytic methods in terms of their point estimation accuracy and Type I and Type II error rates. Results provide guidelines to help researchers choose among the three meta-analytic techniques based on theory (i.e., exploratory vs. confirmatory research) and research design considerations (i.e., degree of range restriction and measurement error).
Aguinis, H. (2001). Estimation of sampling variance of correlations in meta-analysis. Personnel Psychology, 54, 569-590.
2.
Aguinis, H. (2004). Regression analysis for categorical moderators . New York: Guilford.
3.
Aguinis, H. (2007). Performance management. Upper Saddle River, NJ: Pearson Prentice Hall.
4.
Aguinis, H., Beaty, J.C., Boik, R.J., & Pierce, C.A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review . Journal of Applied Psychology, 90, 94-107.
5.
Aguinis, H., Boik, R.J., & Pierce, C.A. (2001). A generalized solution for approximating the power to detect effects of categorical moderator variables using multiple regression . Organizational Research Methods, 4, 291-323.
6.
Aguinis, H., Petersen, S.A., & Pierce, C.A. (1999). Appraisal of the homogeneity of error variance assumption and alternatives to multiple regression for estimating moderating effects of categorical variables. Organizational Research Methods, 2, 315-339.
7.
Aguinis, H., & Pierce, C.A. (1998). Testing moderator variable hypotheses meta-analytically . Journal of Management, 24, 577-592.
8.
Aguinis, H., & Stone-Romero, E.F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82, 192-206.
9.
Aguinis, H., & Whitehead, R. (1997). Sampling variance in the correlation coefficient under indirect range restriction: Implications for validity generalization . Journal of Applied Psychology, 82, 528-538.
10.
Bobko, P., & Stone-Romero, E.F. (1998). Meta-analysis may be another useful research tool, but it is not a panacea. In Research in personnel and human resources management (Vol. 16, pp. 359-397). Stamford, CT: JAI.
11.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
12.
Collins, D.B., & Holton, E.F. (2004). The effectiveness of managerial leadership development programs: A meta-analysis of studies from 1982 to 2001. Human Resource Development Quarterly, 15, 217-248.
13.
Cornwell, J.M., & Ladd, R.T. (1993). Power and accuracy of the Schmidt and Hunter meta-analytic procedures. Educational and Psychological Measurement, 53, 877-895.
14.
Cortina, J.M. (2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods , 6, 415-439.
15.
Dalton, D.R., Daily, C.M., Johnson, J.L., & Ellstrand, A.E. (1999). Number of directors and financial performance: A meta-analysis. Academy of Management Journal, 42, 674-686.
16.
Field, A.P. (2005). Is the meta-analysis of correlation coefficients accurate when population effect sizes vary?Psychological Methods, 10, 444-467.
17.
Fuller, J.B., & Hester, K. (1999). Comparing the sample-weighted and unweighted meta-analysis: An applied perspective. Journal of Management, 25, 803-828.
18.
Gerstner, C.R., & Day, D.V. (1997). Meta-analytic review of leader-member exchange theory: Correlates and construct issues. Journal of Applied Psychology, 82, 827-844.
19.
Glass, G.V. (1976). Primary, secondary, and meta-analysis of research . Educational Researcher, 5, 3-8.
20.
Hall, J.A., & Rosenthal, R. (1991). Testing for moderator variables in meta-analysis: Issues and methods. Communication Monographs, 58, 437-448.
21.
Hall, S.M., & Brannick, M.T. (2002). Comparison of two random-effects methods of meta-analysis. Journal of Applied Psychology, 87, 377-389.
22.
Hedges, L.V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
23.
Hedges, L.V., & Pigott, T.D. (2001). The power of statistical tests in meta-analysis . Psychological Methods, 6, 203-217.
24.
Hough, S.L., & Hall, B.W. (1994). Comparison of the Glass and Hunter and Schmidt meta-analytic techniques. Journal of Educational Research , 87, 292-296.
25.
Huffcutt, A.I., Roth, P.L., & McDaniel, M.A. (1996). A meta-analytic investigation of cognitive ability in employment interview evaluations: Moderating characteristics and implications for incremental validity. Journal of Applied Psychology, 81, 459-473.
26.
Hunter, J.E., & Schmidt, F.L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
27.
Hunter, J.E., & Schmidt, F.L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
28.
Johnson, B.T. (1993). DSTAT 1.10: Software for the meta-analytic review of research literatures. Hillsdale, NJ: Lawrence Erlbaum.
29.
Johnson, B.T., Mullen, B., & Salas, E. (1995). Comparison of three major meta-analytic approaches . Journal of Applied Psychology, 80, 94-106.
30.
Marin-Martinez, F., & Sanchez-Meca, J. (1998). Testing for dichotomous moderators in meta-analysis. The Journal of Experimental Education , 67, 69-81.
31.
Millsap, R.E. (1989). Sampling variance in the correlation coefficient under range restriction: A Monte Carlo study. Journal of Applied Psychology, 74, 456-461.
32.
Neter, J., Wasserman, W., & Whitmore, G.A. (1988). Applied statistics (3rd ed.). Newton, MA: Allyn & Bacon.
33.
Oswald, F.L., & Johnson, J.W. (1998). On the robustness, bias, and stability of statistics from meta-analysis of correlation coefficients: Some initial Monte Carlo findings . Journal of Applied Psychology, 83, 164-178.
34.
Overton, R.C. (1998). A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects. Psychological Methods, 3, 354-379.
35.
Phillips, J.M. (1998). Effects of realistic job previews on multiple organizational outcomes: A meta-analysis. Academy of Management Journal, 41, 673-690.
36.
Richard, O.C. (2000). Racial diversity, business strategy, and firm performance: A resource-based view. Academy of Management Journal, 43, 164-177.
37.
Russell, C.J., & Gilliland, S.W. (1995). Why meta-analysis doesn't always tell you what the data really mean. Journal of Management , 21, 813-831.
38.
Russell, C.J., Settoon, R.P., McGrath, R.N., Blanton, A.E., Kidwell, R.E., Lohrke, F.T., et al. (1994). Investigator characteristics as moderators of personnel selection research: A meta-analysis. Journal of Applied Psychology, 79, 163-170.
39.
Sackett, P.R., Harris, M.M., & Orr, J.M. (1986). On seeking moderator variables in a meta-analysis of correlational data: A Monte Carlo investigation of statistical power and resistance to Type I error. Journal of Applied Psychology , 71, 302-310.
40.
Sagie, A., & Koslowsky, M. (1993). Detecting moderators with meta-analysis: An evaluation and comparison of techniques. Personnel Psychology, 46, 629-640.
41.
Sharma, R., & Yetton, P. (2003). The contingent effects of management support and task interdependence on successful information systems implementation. MIS Quarterly, 27, 533-555.
42.
Spector, P.E., & Levine, E.L. (1987). Meta-analysis for integrating study outcomes: A Monte Carlo study of its susceptibility to Type I and Type II errors. Journal of Applied Psychology, 72, 3-9.
43.
Steel, P.D., & Kammeyer-Mueller, J.D. (2002). Comparing meta-analytic moderator estimation techniques under realistic conditions. Journal of Applied Psychology, 87, 96-111.
44.
Sturman, M.C. (2003). Searching for the inverted U-shaped relationship between time and performance: Meta-analyses of the experience/performance, tenure/performance, and age/performance relationships. Journal of Management, 29, 609-640.
45.
Sturman, M.C., Cheramie, R.A., & Cashen, L.H. (2005). The impact of job complexity and performance measurement on the temporal consistency, stability, and test-retest reliability of employee job performance ratings. Journal of Applied Psychology, 90, 269-283.
46.
Viswesvaran, C., & Sanchez, J.I. (1998). Moderator search in meta-analysis: A review and cautionary note on existing approaches. Educational and Psychological Measurement, 58, 77-87.
47.
Webber, S.S., & Donahue, L.M. (2001). Impact of highly and less job-related diversity on work group cohesion and performance: A meta-analysis. Journal of Management, 27, 141-162.
48.
Wolf, F.M. (1986). Meta-analysis: Quantitative methods for research synthesis. Beverly Hills, CA: Sage .
