This paper examines the impact of quality-score weights in meta-analysis. A simulation examines the roles of study characteristics such as population effect size (ES) and its variance on the bias and mean square errors (MSEs) of the estimators for several patterns of relationship between quality and ES, and for specific patterns of systematic deviations related to quality differences. The bias and MSEs of the estimators are large when ESs from low-quality studies deviate from the population ES in specific ways, and bias does not approach zero in these cases. Because meta-analysts can never know whether biases due to quality exist, and because quality weights lead to bias in almost every condition studied, we recommend against the use of quality weights.
Abele-HornM.DauberA.BauernfeindA.RusswurmW.Seyfarth-MetzgerI.GleichP.RuckdeschelG. (1996). Decrease in nosocomial pneumonia in ventilated patients by selective oropharyngeal decontamination. Intensive Care Medicine, 23, 187–195.
2.
BalkE. M.BonisP. A. L.MoskowitzH.SchmidC. H.IoannidisJ. P. A.WangC.LauJ. (2002). Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. Journal of the American Medical Association, 287, 2973–2982.
3.
Bangert-DrownsR. L.KulikC.-L. C.KulikJ. A.MorganM. T. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61, 213–238.
4.
BérardA.AndreuN.TetraultJ.-P.NiyonsengaT.MyhalD. (2000). Reliability of Chalmers’ scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis. Annals of Epidemiology, 10, 498–503.
5.
BérardA.BravoG. (1998). Combining studies using effect sizes and quality scores: Application to bone loss in postmenopausal woman. Journal of Clinical Epidemiology, 51, 801–807.
6.
BérardA.BravoG.GauthierP. (1997). Meta-analysis of the effectiveness of physical activity for the prevention of bone loss in postmenopausal women. Osteoporosis International, 7, 331–337.
7.
BlairP.RowlandsB. J.LowryK.WebbH.ArmstrongP.SmilieJ. (1991). Selective decontamination of the digestive tract: A stratified, randomized, prospective study in a mixed intensive care unit. Surgery, 110, 303–310.
8.
BlochD. A.MosesL. E. (1988). Nonoptimally weighted least squares. The American Statistician, 42, 50–53.
9.
BolliniP.Garcia RodriguezL. A.Perez CutthannS. (1992). The impact of research quality and study design on epidemiologic estimates of the effect of nonsteroidal antiųinflammatory drugs on upper gastrointestinal tract disease. Archives of Internal Mediųcine, 152, 1289–1295.
10.
Brun-BuissonC.LegrandP.RaussA.RichardC.MontraversF.BesbesM. (1989). Intestinal decontamination for control of nosocomial multiresistant Gram-negative bacilli: Study of an outbreak in an intensive care unit. Annals of Internal Medicine, 110, 873–881.
11.
CerraF. B.MaddausM. A.DunnD. L.WellsC. L.KonstantinidesN. N.LehmannS. L.MannH. J. (1994). Selective gut decontamination reduces nosocomial infections and length of stay but not mortality or organ failure in surgical intensive care unit patients. Archives of Surgery, 127, 163–169.
12.
CockerillF. R. III.MullerS. R.AnhaltJ. P.MarshH.FarnellM.MuchaP.ThompsonR. (1992). Prevention of infection in critically ill patients by selective decontamination of the digestive tract. Annals of Internal Medicine, 117, 545–553.
13.
ColditzG. A.MillerJ. N.MostellerF. (1989). How study design affects outcomes in comparisons of therapy. Statistics in Medicine, 8, 441–454.
14.
DetskyA. S.NaylorC. D.O’RourkeK.McGeerA. J.L’Abbe´K. A. (1992). Incorporating variations in the quality of individual randomized trials into metaanalysis. Journal of Clinical Epidemiology, 45, 255–265.
EddyD. M.HasselbladV.ShachterR. (1992). Meta-analysis by the confidence profile method: The statistical synthesis of evidence. San Diego, CA: Academic Press.
17.
EmersonJ. D.BurdickE.HoaglinD. C.MostellerF.ChalmersT. C. (1990). An empirical study of the possible relation of treatment differences to quality scores in conųtrolled randomized clinical trials. Controlled Clinical Trials, 11, 329–352.
18.
FalchikovN.GoldfinchJ. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70, 287–322.
19.
FerrerM.TorresA.GonzalezJ.PuigD.el-EbiaryM.RocaM.Rodriguez-RoisinR. (1994). Utility of selective digestive decontamination in mechanically ventilated patients. Annals of Internal Medicine, 120, 389–395.
20.
FleissJ. L.GrossA. J. (1991). Meta-analysis in epidemiology, with special reference to studies of the association between exposure to environmental tobacco smoke and lung cancer: A critique. Journal of Clinical Epidemiology, 44, 127–139.
21.
GastinneH.WolffM.DelatourF.FaurissonF.ChevretS. (1992). A controlled trial in intensive care units of selective decontamination of the digestive tract with nonųabsorbable antibiotics. New EnglandJournal of Medicine, 326, 594–599.
22.
GaussorguesP.SalordF.SirodotM.TigaudS.CagninS.GerardM.RobertD. (1991).Efficacite´de la de´contamination digestive sur la survenue des bacte´ rie´ mies nosocomiales chez les patients sous ventilation me´anique et recevant des betamime´ųtiques. Re´animation, Soins Intensifs, Me´decine d’ugence, 7, 169–174.
23.
GodardJ.GuillaumeC.ReverdyM. E.BachmannP.Bui-XuanB.NageotteA.MotinJ. (1990). Intestinal decontamination in a polyvalent ICU: a doubleblind study. Intensive Care Medicine, 16, 307–311.
24.
GreenlandS.O’RourkeK. (2001). On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics, 2, 463–471.
25.
HammondJ. M. J.PotgieterP. D.SaundersG. L.ForderA. A. (1992). Double-blind study of selective decontamination of the digestive tract in intensive care. Lancet, 340, 5–9.
26.
HartenauerU.ThüligB.DiemerW.LawinP.FegelerW.KehrelR.RitzerfeldW. (1991). Effect of selective flora suppression on colonization, infection, and mortalųity in critically ill patients: A one-year, prospective consecutive study. Critical Care Medicine, 19, 463–473.
27.
HedgesL. V. (1982). Estimation of effect size from a series of independent experiments. Psychological Bulletin, 92, 490–499.
HerbisonP.Hay-SmithJ.GillespieW. J. (2006). Adjustment of meta-analyses on the basis of quality scores should be abandoned. Journal of Clinical Epidemiology, 59, 1249–1256.
30.
HunterJ. E.SchmidtF. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings(2nd ed.). Thousands Oaks, CA: Sage.
31.
JacobsS.FowerakerJ. E.RobertsS. E. (1992) Effectiveness of selective decontaųmination of the digestive tract in an ICU with a policy encouraging a low gastric pH. Clinical Intensive Care, 3, 52–58.
32.
JüniP.AltmanD. G.EggerM. (2001). Assessing the quality of controlled clinical trials. British Medical Journal, 323, 42–46.
33.
JüniP.WitschiA.BlochR.EggerM. (1999). The hazards of scoring the quality of clinical trials for meta-analysis. Journal of the American Medical Association, 282, 1054–1060.
34.
KerverA.RommesJ. H.Mevissen-VerhageE.HulstaertP. F.VosA.VerhoefJ.WittebolP. (1988). Prevention of colonization and infection in critically ill patients: A prospective randomized study. Critical Care Medicine, 16, 1087–1093.
35.
KorinekA. M.LaisneM. J.NicolasM. H.RaskineL.DeroinV.Sanson-LeporsM. J. (1993). Selective decontamination of the digestive tract in neurosurgical intensive care unit patients: A double-blind, randomized, placebo-controlled study. Critical Care Medicine, 21, 1466–1473.
36.
KunzR.OxmanA. (1998). The unpredictability paradox: Review of empirical comparisons of randomized and non-randomized clinical trials. British Medical Journal, 317, 1185–1190.
37.
LaggnerA. N.TrybaM.GeorgopoulosA.LenzK.GrimmG.GraningerW.DrumlW. (1994). Oropharyngeal decontamination with gentamicin for long-term ventilated patients on stress ulcer prophylaxis with sueralfate. Wien Klin Wochenschr, 106, 15–19.
38.
LedinghamI.EastawayA.McKayI. C.AlcockS. R.McDonaldsJ. C.RamsayG. (1988). Triple regimens of selective decontamination of the digestive tract, systemic cefotaxime, and microbiological surveillance for prevention of acquired infection in intensive care. The Lancet, 331, 785–790.
39.
LindeK.ScholzM.RamierzG.ClausiusN.MelchartD.JonasW. B. (1999). Impact of study quality on outcome in placebo-controlled trials of homeopathy. Journal of Clinical Epidemiology, 52, 631–636.
40.
LingnauW.BergerJ.JavorskyF.LejeuneP.MutzN.BenzerH. (1997). Selective intestinal decontamination in multiple trauma patients: Prospective, controlled trial. The Journal of Trauma, 42, 687–694.
41.
McClellandP.MurrayA.WilliamsP.Van SaeneH.GilbertsonA.MonstafaS.BoneJ. (1990). Reducing sepsis in severe combined acute renal and respiratory failure by selective decontamination of the digestive tract. Critical Care Medicine, 18, 935–939.
42.
MoherD.CookD. J.JadadA. R.TugwellP.MoherM.JonesA.KlassenT. P. (1999). Assessing the quality of reports of randomized trials: Implications for the conųduct of meta-analysis. Southampton, United Kingdom: Universityof Southampton, The National Coordinating Center for Health Technology Assessment.
43.
OlivoS. A.MacedoL. G.GadottiI. C.FuentesJ.StantonT.MageeD. J. (2008). Scales to assess the quality of randomized controlled trials: A systematic review. Physical Therapy, 88, 156–175.
44.
OrtizZ.SheaB.SuarezA. M. E.MoherD.WellsG. A.TugwellP. (1998). The efficacy of folic acid and folinic acid in reducing methotrexate gastrointestinal toxicity in rheumatoid arthritis: A meta-analysis of randomized controlled trials. The Journal of Rheumatology, 25, 36–43.
45.
PalomarM.Alvarez-LermaF.JordaR.BermejoB. (1997). Prevention of nosocoųmial infection in mechanically ventilated patients: Selective digestive decontamination versus sueralfate. Clinical Intensive Care, 8, 228–235.
46.
PuginJ.AuckenthalerR.LewD. P.SuterP. M. (1991). Oropharyngeal decontamiųnation decreases incidence of ventilator-associated pneumonia: A randomized, placebo-controlled, double-blind clinical trial. JAMA, 265, 2704–2710.
47.
QuinioB.AlbaneseJ.Bues-CharbitM.ViviandX.MartinC. (1996). Selective decontamination of the digestive tract in multiple trauma patients: A prospective double-blind, randomized, placebo-controlled study. Chest, 109, 765–772.
48.
RochaL. A.MartinM. J.PitaS.PazJ.SecoC.MargusinoL.DuránM. (1992). Prevention of nosocomial infection in critically ill patients by selective decontamination of the digestive tract: A randomized, double blind, placebo-controlled study. lntensive Care Medicine, 18, 398–404.
49.
Rodriguez-RoldánJ.Alma-CuestaA.LópezA.CarrilloA.GarciaJ.LeónJ.Martínez-PellusA. (1990) Prevention of nosocomial lung infection in ventilated patients: Use of an antimicrobial pharyngeal nonabsorbable paste. Critical Care Medicine, 18, 1239–1242.
50.
RubinD. B. (1992). Meta-analysis: Literature synthesis or effect-size surface estimation?Journal of Educational and Behavioral Statistics, 17, 363–374.
51.
Sanchez GarciaM.Cambronero GalacheJ. A.Lopez DiazJ.Cerda CerdaE.Rubio BlascoJ.Gomez AguinagaM.Sacristan del CastilloJ. (1998). Effectiveness and cost of selective decontamination of the digestive tract in critically ill intubated patients. A randomized, double-blind, placebo-controlled, multicenter trial. American Journal of Respiratory and Critical Care Medicine, 158, 908–916.
52.
SchulzK.F.ChalmersI.HayesR.J.AltmanD.G.(1995). Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Journal of the American Medical Association, 273, 408–412.
53.
ShadishW. R.HaddockC. K. (1994). Combining estimates of effect size. In CooperH.HedgesL. V. (Eds.), The handbook of research synthesis (pp. 261–281). New York, NY: Russell Sage Foundation.
54.
SingerG. H. S. (2006). Meta-analysis of comparative studies of depression in mothers of children with and without developmental disabilities. American Journal on Mental Retardation, 111, 155–169.
55.
SlavinR. E. (2008). What works? Issues in synthesizing educational program evaluaųtions. Educational Researcher, 37, 5–14.
56.
SwansonH. L.HoskynM. (1998). Experimental intervention research on students with learning disabilities: A meta-analysis of treatment outcomes. Review of Educaųtional Research, 68, 277–231.
57.
StoutenbeekC.van SaeneH.MirandaD.ZandstraD.BinnedijK. (1984). The effect of selective decontamination of the digestive tract on colonization and infection rate in multiple trauma patients. Intensive Care Medicine, 10, 185–192.
58.
TreadwellJ. R.TregearS. J.RestonJ. T.TurkelsonC. M. (2006). A system for rating the stability and strength of medical evidence. BMC Medical Research Methodology, 6, 1–20.
59.
TritchlerD. (1999). Modeling study quality in meta-analysis. Statistics in Medicine, 10, 2135–2145.
60.
Ulrich Harinck-de WeerdJ. E.BakkerN. C.JaczK.DoornbosL.de RidderV. A. (1989). Selective decontamination of the digestive tract with norfloxacin in the prevention of ICU-acquired infections: A prospective randomized study. Intensive Care Medicine, 15, 424–431.
61.
UnertlK.RuckdeschelG.SelbmannH. K.JensenU.ForstH.LenhartF. P.PeterK. (1987). Prevention of colonization and respiratory infections in long-term ventilated patients by local antimicrobial prophylaxis. Intensive Care Medicine, 13, 106–113.
62.
ValentineJ. C.CooperH. (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The Study Design and Implementation Assessment Device (Study DIAD). Psychological Methods, 13, 330–349.
63.
van den NoortgateW.OnghenaP. (2003). Estimating the mean effect size in meta-analysis: Bias, precision, and mean squared error of different weighted methods. Behavior Research Methods, Instruments, & Concepts, 35, 504–511.
64.
van NieuwenhovenC. A.BuskensE.van TielF. H.BontenM. J. M. (2001). Relationship between methodological trial quality and the effects of selective digestive decontamination on pneumonia and mortality in critically ill patients. Journal of American Medical Association, 286, 335–340.
65.
VerwaestC.VerhaegenJ.FerdinandeP.SchetzM.Van den BergheG.VerbistL.LauwersP. (1997). Randomized, controlled trial of selective digestive decontaminaųtion in 600 mechanically ventilated patients in a multidisciplinary intensive care unit. Critical Care Medicine, 25, 63–71.
66.
WellsK.LittellJ. H. (2009). Study quality assessment in systematic reviews of research on intervention effects. Research on Social Work Practice, 19, 52–62.
67.
WhiteK.TaylorM.MossV. (1992). Does research support claims about the benefits of involving parents in early intervention programs?Review of Educational Research, 62, 91–125.
68.
WhitesideM. F.BeckerB. J. (2000). Parental factors and the young child’s postdivorce adjustment: A meta-analysis with implications for parenting arrangements. Journal of Family Psychology, 14, 5–26.
69.
WhitingP.HarbordR.KleijnenJ. (2005). No role for quality scores in systematic reviews of diagnostic accuracy studies. BMC Medical Research Methodology, 5, 1–9.
70.
WhitingP.RutjesA. W. S.DinnesJ.ReitsmaJ. B.BossuytP. M. M.KleijnenJ. (2005). A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. Journal of Clinical Epidemiology, 58, 1–12.
71.
WienerJ.ItokazuG.NathanC.KabinsS. A.WeinsteinP. A. (1995). A randomized, double-blind, placebo-controlled trial of selective decontamination in a medical-surgical intensive care unit. Clinical Infectious Diseases, 20, 861–867.
72.
WinterR.HumphreysH.PickA.MacGowanA. P.WillattsS. M.SpellerD. C. E. (1992). A controlled trial of selective decontamination of the digestive tract in intensive care and its effect on nosocomial infection. The Journal of Antimicrobial Chemotherapy, 30, 73–87.
