In the discipline of psychology there are numerous examples of multiple published studies examining the same topic with contradictory results. Meta-analysis is viewed by proponents as a superior methodology that is based on a more reliable and objective process than narrative reviews in summarizing the results of multiple studies. To asses this claim, we reviewed 10 years of meta-analyses published in Psychological Bulletin, and examined the epistemic culture that informs the practice of meta-analysis. The practice of meta-analysis as examined within the sample of studies reviewed was characterized by methodological inconsistencies. This suggests that the methods set forth by the proponents of meta-analysis are not being followed by some researchers with rigour or consistency. Such preliminary findings, therefore, challenge the notion that meta-analysis is an inherently superior technique as compared to other forms of review. We discuss this in terms of the shared commitments and epistemic culture of researchers that rely on meta-analysis.
*Albarracin, D., Fishbein, M., Johnson, B.T., & Muellerleile, P. (2001). Theories of reasoned action and planned behavior as models of condom use: A meta-analysis. Psychological Bulletin, 127(1), 142-161.
2.
*Bakermans-Kranenburg, M.J., Ilzendoorn, M.H., & Juffer, F. (2003). Less is more: Meta-analyses of sensitivity and attachment interventions in early childhood. Psychological Bulletin, 129(2), 195-215.
3.
Bauer, H. (1992). Scientific literacy and the myth of the scientific method. Urbana: University of Illinois Press.
4.
Begg, C.B. (1994). Publication bias. In H.M. Cooper & L.V. Hedges (Eds.), The handbook of research synthesis (pp. 399-409). New York: Russell Sage Foundation.
5.
*Bettencourt, B.A., & Miller, N. (1996). Gender differences in aggression as a function of provocation: A meta-analysis. Psychological Bulletin, 119(3), 422-447.
6.
*Bettencourt, B.A., Chariton, K., Dorr, N., & Hume, D.L. (2001). Status differences and in-group bias: A meta-analytic examination of the effects of status stability, status legitimacy, and group permeability. Psychological Bulletin, 127(4), 520-542.
7.
*Bond, R., & Smith, P.B. (1996). Culture and conformity: A meta-analysis of studies using Asch's (1952b, 1956) line judgement task. Psychological Bulletin, 119(1), 111-137.
8.
*Bonta, J., Law, M., & Hanson, K. (1998). The prediction of criminal and violent recidivism among mentally disordered offenders: A meta-analysis. Psychological Bulletin, 123(2), 123-142.
9.
*Brow, M., & Steven, P. (1996). A meta-analysis and review of organizational research on job involvement. Psychological Bulletin, 120(2), 235-255.
10.
Brown, T.L. (2003). Making truth: Metaphor in science. Urbana: University of Illinois Press.
11.
*Burt, D., Zembar, M.J., & Niederehe, G. (1995). Depression and memory impairment: A meta-analysis of the association, its pattern, and specificity. Psychological Bulletin, 117(2), 285-305.
12.
*Byrnes, J.P., Miller, D.C., & Schafer, W.D. (1999). Gender differences in risk taking: A meta-analysis . Psychological Bulletin, 125(3), 367-383.
13.
Cohen, J. (1990). Things I have learned so far. American Psychologist, 45, 1304-1312.
14.
Cohen, J. (1994). The earth is round (p <h;.05) . American Psychologist, 49, 997-1003.
15.
*Collins, N.L., & Miller, L.C. (1994). Self-disclosure and liking: A meta-analytic review . Psychological Bulletin, 116(3), 457-475.
16.
Cooper, H.M. (1985). Literature searching strategies of integrative research reviewers. American Psychologist, 40, 1267-1269.
Cooper, H.M., & Hedges, L.V. (1994a). Research synthesis as a scientific enterprise . In H.M. Cooper, & L.V. Hedges (Eds.), The handbook of research synthesis (pp. 3-14). New York: Russell Sage Foundation.
19.
Cooper, H.M., & Hedges, L.V. (Eds.). (1994b). The handbook of research synthesis. New York: Russell Sage Foundation.
20.
*Deci, E.L., Ryna, R.M., & Koestner, R. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 125(6), 627-668.
21.
*DeNeve, K.M., & Cooper, H.M. (1998). The happy personality: A meta-analysis of 13T personality traits and subjective well-being. Psychological Bulletin, 124(2), 197-229.
22.
*Eagly, A.H., Karaw, S.J., & Makhijani, M.G. (1995). Gender and the effectiveness of leaders: A meta-analysis. Psychological Bulletin, 117(1), 125-145.
23.
*Elfenbein, H.A., & Ambady, N. (2002). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin , 128(1), 203-235.
24.
*Erel, O., & Burman, B. (1995). Interrelatedness of marital relations and parent-child relations: A meta-analytic review. Psychological Bulletin , 118(1), 108-132.
25.
*Feingold, A. (1994). Gender differences in personality: A meta-analysis . Psychological Bulletin, 116(3), 429-456.
26.
*Garb, H.N., & Schramke, C.J. (1996). Judgement research and neuropsychological assessment: A narrative review and meta-analysis. Psychological Bulletin, 120(1), 140-153.
27.
*Gershoff, E.T. (2002). Corporal punishment by parents and associated child behaviors and experiences: A meta-analytic and theoretical review. Psychological Bulletin, 128(4), 539-579.
28.
Glass, G., McGaw, B., & Smith, M. (1981). Meta-analysis in social research. Newbury Park, CA: Sage.
29.
*Hasselbold, V., & Hedges, L.V. (1995). Meta-analysis of screening and diagnostic tests . Psychological Bulletin, 117(1), 167-178.
30.
*Hebert, T.B., & Cohen, S. (1993). Depression and immunity: A meta-analytic review . Psychological Bulletin, 113(3), 472-486.
31.
Hedges, L.V. (1994). Statistical considerations. In H.M. Cooper, & L.V. Hedges (Eds.), The handbook of research synthesis (pp. 29-38). New York: Russell Sage Foundation.
32.
*Holden, G.W., & Miller, P.C. (1999). Enduring and different: A meta-analysis of the similarity in parents' child rearing. Psychological Bulletin , 125(2), 223-254.
33.
Hunter, J.E. & Schmidt, F.L. (1996). Cumulative knowledge and social policy formulation: The critical role of meta-analysis. Psychology, Public Policy, and Law, 2(2), 324-347.
34.
*Ito, T.A., Miller, N., & Pollock, V.E. (1996). Alcohol and aggression: A meta-analysis on the moderating effects of inhibitory cues, triggering events, and self-focused attention. Psychological Bulletin, 120(1), 60-82.
35.
Jackson, G.B. (1980). Methods for integrative reviews. Review of Educational Research, 50, 438-460.
*Kluger, A.N., & Denisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254-284.
39.
*Knight, G.P., Fabes, R.A., & Higgins, D.A. (1996). Concerns about drawing causal inferences from meta-analyses: An example in the study of gender differences in aggression . Psychological Bulletin, 119(3), 410-421.
40.
Knorr Cetina, K. (1999). Epistemic cultures: How the sciences make knowledge . Cambridge, MA: Harvard University Press.
41.
*Kuncel, N.R., Ones, D.S., & Hezlett, S.A. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127(1), 162-181.
42.
*Lalumiere, M.L., Blanchard, R., & Zucker, K.J. (2000). Sexual orientation and handedness in men and women: A meta-analysis. Psychological Bulletin, 126(4), 575-592.
43.
*Langlois, H.H., Kalakanis, L., Rubenstein, A.L., Larson, A., Hallam, M., & Smoot, M. (2000). Maxims or myths of beauty? A meta-analytic and theoretical review. Psychological Bulletin, 126(3), 390-423.
44.
Latour, B. (1987). Science in action. Cambridge, MA: Harvard University Press.
45.
*Lee, S.R., & Waldman, I.D. (2002). Genetic and environmental influences on antisocial behavior: A meta-analysis of twin and adoption studies. Psychological Bulletin, 128(3), 490-529.
46.
*LeFrance, M., Hecht, M.A., & Paluck, E.L. (2003). The contingent smile: A meta-analysis of sex differences in smiling. Psychological Bulletin, 129(2), 305-334.
47.
Lepper, M.R., Henderlong, J., & Gingras, I. (1999). Understanding the effects of extrinsic rewards on intrinsic motivation-uses and abuses of meta-analysis: Comment on Deci, Koestner, and Ryan (1999). Psychological Bulletin, 125(6), 669-676.
48.
Lipsey, M.W., & Wilson, D.B. (1995). Reply to comments on Lipsey and Wilson (1993) . American Psychologist, 50(2), 113-115.
49.
*Logan, T.K., Cole, J., & Leukefeld, C. (2002). Women, sex and HIV: Social an contextual factors. A meta-analysis of published interventions and implications for practice and research. Psychological Bulletin, 128(6), 851-885.
50.
*McClure, E.B. (2000). A meta-analytic review of sex differences in facial expression processing and their development in infants, children, and adolescents . Psychological Bulletin, 126(3), 424-453.
51.
*Mead, D., & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. Psychological Bulletin , 114(3), 449-458.
52.
Mertens, D.M. (1998). Research methods in education and psychology: Integrating diversity with quantitative & qualitative approaches. Thousand Oaks, CA: Sage.
53.
Miller, N., & Pollack, V.E. (1994). Meta-analytic synthesis for theory development . In H.M. Cooper & L.V. Hedges (Eds.), The handbook of research synthesis (pp. 457-484). New York: Russell Sage Foundation.
54.
*Miller, T.Q., Smith, T.W., Turner, C.W., Guijarro, M.L., & Hallet, A.J. (1996). A meta-analysis review of research on hostility and physical health. Psychological Bulletin, 119(2), 322-348.
55.
*Mor, N., & Winquist, J. (2002). Self-focused attention and negative affect: A meta-analysis. Psychological Bulletin, 128(4), 638-662.
56.
*Newcomb, A.F., Bukowski, W.M., & Pattee, L. (1993). Children's peer relations: A meta-analytic review of popular, rejected, neglected, controversial, and average sociometric status . Psychological Bulletin, 113(1), 99-128.
*Oliver, M.B., & Hyde, J.S. (1993). Gender differences in sexuality: A meta-analysis . Psychological Bulletin, 114(1), 29-51.
59.
*Oyserman, D., Coon, H.M., & Kemmelmeier, M. (2002). Rethinking individualism and collectivism: Evaluation of theoretical assumptions and meta-analysis . Psychological Bulletin, 128(1), 3-72.
60.
*Ozer, E.J., Best, S.R., Lipsey, T.L., & Weiss, D.S. (2003). Predictors of posttraumatic stress disorder and symptoms in adults: A meta-analysis. Psychological Bulletin , 129(1), 52-73.
61.
*Polich, J., Pollock, V.E., & Bloom, F.E. (1994). Meta-analysis of P300 amplitude for males at risk for alcoholism. Psychological Bulletin, 115(1), 55-73.
62.
*Postmes, T., & Spears, R. (1998). Deindividuation and antinormative behavior: A meta-analysis. Psychological Bulletin, 123(3), 238.
63.
*Rind, B., Tromovitch, P., & Bauserman, R. (1998). A meta-analytic examination of assumed properties of child sexual abuse using college samples. Psychological Bulletin, 124(1), 22-53.
64.
*Roese, J., & Jamieson, D.W. (1993). Twenty years of bogus pipeline research: A critical review and meta-analysis. Psychological Bulletin, 114(2), 363-375.
65.
Rosenthal, R. (1979). The `file drawer problem' and tolerance for null results. Psychological Bulletin, 86, 638-641.
66.
*Rothbaum, F., & Weisz, J.R. (1994). Parental caregiving and child externalizing behavior in non clinical samples: A meta-analysis. Psychological Bulletin , 116(1), 55-74.
67.
Schmidt, F.L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist , 47, 1173-1181.
68.
*See, J.E., Howe, S.R., Warm, J.S., & Dember , W.N. (1995). A meta-analysis of the sensitivity decrement in vigilance. Psychological Bulletin, 117(2), 230-249.
69.
*Shadish, W.R., Matt, G.E., Navarro, A.M., & Phillips, G. (2000). The effects of psychological therapies under clinically representative conditions: A meta-analysis. Psychological Bulletin, 126(4), 512-529.
70.
Sharpe, W. (1997). Of apples and oranges, file drawers and garbage: Why validity issues in meta-analysis will not go away. Clinical Psychology Review, 17(8), 881-901.
71.
*Sheera, P., Abraham, C., & Orbell, S. (1999). Psychosocial correlates of heterosexual condom use: A meta-analysis. Psychological Bulletin, 125(1), 90-132.
72.
Sohn, D. (1995). Meta-analysis as a means of discovery. American Psychologist, 50(2), 108-110.
73.
*Sporer, S.L., Penrod, S., Read, D., & Cutler, B. (1995). Choosing, confidence, and accuracy: A meta-analysis of the confidence-accuracy relation in eye witness identification studies . Psychological Bulletin, 118(3), 315-327.
74.
*Stajkovic, A.D., & Luthans, F. (1998). Self efficacy and work-related performance: A meta-analysis. Psychological Bulletin, 124(2), 240-261.
75.
*Stanne, M.B., Johnson, D.W., & Johnson, R.T. (1999). Does competition enhance or inhibit motor performance? A meta-analysis. Psychological Bulletin, 125(1), 133-154.
76.
*Stanton, M.D., & Shadish, W.R. (1997). Outcome, attrition, and family-couples treatment for drug abuse: A meta-analysis and review of the controlled, comparative studies. Psychological Bulletin, 122(2), 170-191.
77.
*Stice, E. (2002). Risk and maintenance factors for eating pathology: A meta-analytic review. Psychological Bulletin, 128(5), 825-848.
78.
*Symons, C.S., & Johnson, B.T. (1997). The self-reference effect in memory: A meta-analysis . Psychological Bulletin, 121(3), 371-394.
79.
*Tracey, T.J., & Rounds, J. (1993). Evaluating Holland's and Gati's vocational interest models: A structural meta-analysis. Psychological Bulletin , 113(2), 229-246.
80.
*Twenge, J.M., & Crocker, J. (2002). Race and self-esteem: Meta analyses comparing Whites, Blacks, Hispanics, Asians, and American Indians and comment on Grey-Little and Hafdahl (2000). Psychological Bulletin, 128(3), 371-408.
81.
*Verhaeghen, P., & Salthouse, T.A. (1997). Meta-analysis of age-cognition relations in adulthood: Estimates of linear and non-linear age effects and structural models. Psychological Bulletin, 122(3), 231-249.
82.
*Voyer, D., Voyer, S., & Bryden, M.P. (1995). Magnitude of sex differences in spatial abilities: A meta-analysis and consideration of critical variables. Psychological Bulletin, 117(2), 250-270.
83.
*Wampold, B.E., Mondin, G.W., Moody, M., Stich, F., Benson, K., & Ahn, H. (1997). A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, `all must have prizes'. Psychological Bulletin, 122(3), 203-215.
84.
White, H.D. (1994). Scientific communication and literature retrieval . In H.M. Cooper & L.V. Hedges (Eds.), The handbook of research synthesis. (pp. 41-55). New York: Russell Sage Foundation.
85.
*Wood, W., Lundgren, S., Ovellette, J.A., Busceme, S., & Blackstone, T. (1994). Minority influence: A meta-analytic review of social influence processes. Psychological Bulletin , 115(3), 323-345.
86.
*Wood, W., & Quinn, J.M. (2003). Forewarned and forearmed? Two meta-analytic syntheses of forewarnings of influence appeals. Psychological Bulletin , 129(1), 119-138.
87.
Wortman, P.M. (1994). Judging research quality. In H.M. Cooper & L.V. Hedges (Eds.), The handbook of research synthesis (pp. 97-109). New York : Russell Sage Foundation.
88.
*Yirmiya, N.Y., Osnat, E., Shaked, M., & Solomonica-Levi, D. (1998). Meta-analyses comparing theory of mind abilities of individuals with autism, individuals with mental retardation, and normally developing individuals. Psychological Bulletin, 124(3), 283-307.
