Evaluators widely agree that stakeholder involvement is a central aspect of effective program evaluation. With the exception of articles on collaborative evaluation approaches, however, a systematic review of the breadth and depth of the literature on stakeholder involvement has not been published. In this study, we examine peer-reviewed empirical studies as the first step in establishing a firm foundation for a discussion of stakeholder involvement. We excluded all studies that did not provide descriptions of their methods, helping ensure that we only reviewed studies with strong methodological warrants for their findings. We address three key questions about the state of the empirical literature on stakeholder involvement, focusing on the studies' methodological and contextual characteristics, and the involvement themes and substantive nature (positive, negative/mixed, or explanatory/normative) of the findings. We provide detailed findings in an appendix at http://aje.sagepub.com/supplemental or http://goo.gl/W8oMnl.
American Evaluation Association, Task Force on Guiding Principles for Evaluators. (1995). Guiding principles for evaluators. In ShadishW. R.NewmanD. L.ScheirerM. A.WyeC. (Eds.), Guiding principles for evaluators. New Directions for Program Evaluation, 66, 19–16.
2.
BrandonP. R. (1999). Involving program stakeholders in reviewing evaluators’ recommendations for program revisions. Evaluation and Program Planning, 22, 363–372.
3.
BrandonP. R. (2012, April). Ruminations on research on evaluation. Paper presented at the meeting of the American Educational Research Association, Vancouver, Canada.
4.
BrandonP. R.HeckR. H. (1998). The use of teacher expertise in decision making during school-conducted needs assessments: A multilevel perspective. Evaluation and Program Planning, 21, 323–331.
5.
BrandonP. R.LindbergM. A.WangZ. (1993). Involving program beneficiaries in the early stages of evaluation: Issues of consequential validity and influence. Educational Evaluation and Policy Analysis, 15, 420–428.
6.
BrandonP. R.NewtonB. J.HarmanJ. W. (1993). Enhancing validity through beneficiaries’ equitable involvement in identifying and prioritizing homeless children’s educational problems. Evaluation and Program Planning, 16, 287–293.
7.
BrandonP. R.SinghJ. M. (2009). The strength of the methodological warrants for the findings of research on program evaluation. American Journal of Evaluation, 30, 123–157.
8.
BrandonP. R.VallinL. M.PhilippoffJ. (2012, October). A quantitative summary of research on evaluation published in theAmerican Journal of Evaluationfrom 1998 through 2011. Paper presented at the meeting of the American Evaluation Association, Minneapolis, MN.
9.
BrandonP. R.WangZ.HeckR. H. (1994). Teacher involvement in school-conducted needs assessments: Issues of decision-making process and validity. Evaluation Review, 18, 458–471.
10.
BrykA. (Ed.). (1983). Stakeholder-based evaluation. New Directions for Program Evaluation, 17. San Francisco, CA: Jossey-Bass.
ChouinardJ. A.CousinsJ. B. (2009). A review and synthesis of current research on cross-cultural evaluation. American Journal of Evaluation, 30, 457–494.
13.
ChristieC. A.FleischerD. N. (2010). Insight into evaluation practice: A content analysis of designs and methods used in evaluation studies published in North American evaluation-focused journals. American Journal of Evaluation, 31, 326–346.
14.
CorynC. L. S.NoakesL. A.WestineC. D.SchroeterD. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32, 199–226.
15.
CousinsJ. B. (2003). Utilization effects of participatory evaluation. In KellaghanT.StufflebeamD. L.WingateL. A. (Eds.), International Handbook of Educational Evaluation (pp. 245–266). Boston: Kluwer.
16.
CousinsJ. B.ChouinardJ. A. (2012). Participatory evaluation up close: An integration of research-based knowledge. Charlotte, NC: Information Age.
17.
DattaL. (2006). The practice of evaluation: In I. ShawF.GreeneJ. C.MarkM. M. (Eds.), The SAGE handbook of evaluation (pp. 419–428). Thousand Oaks, CA: Sage.
18.
DattaL. (2011). Politics and evaluation: More than methodology. American Journal of Evaluation, 32, 273–294.
19.
FarrarE.HouseE. R. (1983). The evaluation of Push/Excel: A case study. In BrykA. S. (Ed.), Stakeholder-based evaluation. New Directions for Program Evaluation, 17, 31–58.
20.
FettermanD. M. (2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage.
21.
GoldN. (1981). The stakeholder process in educational program evaluation. Washington, DC: National Institute of Education.
22.
GoldN. (1983). Stakeholders and program evaluation: characterizations and reflections. In BrykA. S. (Ed.), Stakeholder-based evaluation. New Directions for Program Evaluation, 17, 63–72.
23.
GreeneJ. C. (1987). Stakeholder participation in evaluation design: Is it worth the effort?Evaluation and Program Planning, 10, 375–394.
24.
HouseE. R. (1991). Evaluation and social justice: Where are we? In McLaughlinM. W.PhillipsD. C. (Eds.), Evaluation and education: At quarter century (pp. 233–247). Chicago, IL: National Society for the Study of Education.
25.
HouseE. R.HoweK. R. (1999). Values in evaluation and social research. Thousand Oaks, CA: Sage.
26.
JacobsonM. R.AzzamT.BaezJ. E. (2013). The nature and frequency of inclusion of people with disabilities in program evaluation. American Journal of Evaluation, 34, 23–44.
27.
JohnsonK.GreenseidL.ToalS. A.KingJ. A.LawrenzFVolkovB. (2009). Research on evaluation use: A review of the empirical literature from1986–2005. American Journal of Evaluation, 30, 377–410.
28.
KingJ. A.StevahnL. (2012). Interactive evaluation practice: Managing the interpersonal dynamics of program evaluation. Newbury Park, CA: Sage.
29.
LabinS. N. (2008). Research synthesis: Toward broad-based evidence. In SmithN.BrandonP. R. (Eds.), Fundamental issues in evaluation (pp. 89–110). New York, NY: Guilford.
30.
LabinS. N.DuffyJ. L.MeyersD. C.WandersmanA.LesesneC. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33, 307–338.
31.
MarkM. M. (2008). Building a better evidence for evaluation theory: Beyond general calls to a framework of types of research on evaluation. In SmithN. L.BrandonP. R. (Eds.), Fundamental issues in evaluation (pp. 111–134). New York, NY: Guilford.
32.
MarkM. M.ShotlandL. (1985). Stakeholder-based evaluation and value judgments. Evaluation Review, 9, 605–626.
33.
MertensD. M. (2009). Transformative research and evaluation. New York, NY: Guilford.
34.
MillerR. L.CampbellR. (2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27, 296–319.
35.
PattonM. Q. (1978). Utilization-focused evaluation. Beverly Hills, CA: Sage.
PeckL.KimY.LucioJ. (2012). An empirical examination of validity in evaluation. American Journal of Evaluation, 33, 350–365.
38.
RogD. J.FitzpatrickJ. L.ConnerR. (Eds.). (2012). Context and evaluation. New Directions for Evaluation, 135. San Francisco: Jossey-Bass.
39.
RossJ. A.BarkaouiK.ScottG. (2007). Evaluations that consider the cost of educational programs: The contribution of high quality studies. American Journal of Evaluation, 28, 477–492.
40.
RyanG. W.BernardH. R. (2003). Techniques to identify themes. Field Methods, 15, 85–109.
41.
SmithN. L. (1993). Improving evaluation through the empirical study of evaluation practice. Evaluation Practice, 14, 237–242.
42.
StakeR. E. (1976). A theoretical statement of responsive evaluation. Studies in Educational Evaluation, 2, 19–22.
43.
StakeR. E. (1983). Stakeholder influence in the evaluation of Cities-in-Schools. New Directions for Program Evaluation, 17, 15–30.
44.
StevahnL.KingJ. A.GhereG.MinnemaJ. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26, 43–59.
45.
SzanyiM.AzzamT.GalenM. (2013). Research on evaluation: A needs assessment. Canadian Journal of Program Evaluation, 27, 39–64.
46.
Taut, S. (2008). What have we learned about stakeholder involvement in program evaluation? Studies in Education Evaluation, 34, 224–230.
47.
TrevisanM. S. (2004). Practical training in evaluation: A literature review. American Journal of Evaluation, 25, 255–272.
48.
TylerR. W. (1987). Education: Curriculum development and evaluation, an oral history conducted 1985–1987 by Malca Chall. Berkeley, CA: University of California, Bancroft Library, Regional Oral History Office.
49.
TylerR. W. (1991). General statement on program evaluation. In McLaughlinM. W.PhillipsD. C. (Eds.), Evaluation and Education: At quarter century (pp. 3–17). Chicago, IL: National Society for the Study of Education.
50.
VoA. (2013). Visualizing context through theory deconstruction: A content analysis of three bodies of evaluation theory literature. Evaluation and Program Planning, 38, 44–52.
51.
YarbroughD. B.ShulhaL. M.HopsonR. K.CaruthersF. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.
52.
AlpertB.BecharS. (2007). Collaborative evaluation research: A case study of teachers’ and academic researchers’ teamwork in a secondary school. Studies in Educational Evaluation, 33, 229–257.
53.
ArdenC.McLachlanK.CooperT. (2009). Building capacity through sustainable engagement: Lessons for the learning community from the GraniteNet Project. Australian Journal of Adult Learning, 49, 75–101.
54.
AyersT. D. (1987). Stakeholders as partners in evaluation: A stakeholder-collaborative approach. Evaluation and Program Planning, 10, 263–271.
55.
AzzamT. (2010). Evaluator responsiveness to stakeholders. American Journal of Evaluation, 31, 45–65.
56.
BrandonP. R. (1999). Involving program stakeholders in reviews of evaluators’ recommendations for program revisions. Evaluation and Program Planning, 22, 363–372.
57.
CampbellB.MarkM. M. (2006). Toward more effective stakeholder dialogue: Applying theories of negotiation to policy and program evaluation. Journal of Applied Social Psychology, 36, 2834–2863.
58.
CampbellR.DoreyH.NaegeliM.GrubsteinK. L.BennettK. K.BonterF.,…DavidsonW.S. (2004). An empowerment evaluation model for sexual assault programs: Empirical evidence of effectiveness. American Journal of Community Psychology, 34, 251–262.
59.
CartlandJ.Ruch-RossH. S.MasonM.DonohouW. (2008). Role sharing between evaluators and stakeholders in practice. American Journal of Evaluation, 29, 460–477.
60.
ChenS.PolandB.SkinnerH. A. (2007). Youth voices: Evaluation of participatory action research. Canadian Journal of Program Evaluation, 22, 125–150.
61.
CousinsJ. B. (1996). Consequences of researcher involvement in participatory evaluation. Studies in Educational Evaluation, 22, 3–27.
62.
CousinsJ. B. (2001). Do evaluator and program practitioner perspectives converge in collaborative evaluation?Canadian Journal of Program Evaluation, 16, 113–133.
63.
CousinsJ. B.DonohueJ. J.BloomG. A. (1996). Collaborative evaluation in North America: Evaluators’ self-reported opinions, practices, and consequences. Evaluation Practice, 17, 207–226.
64.
CurranV.ChristopherJ.LemireF.CollinsA.BarretB. (2003). Application of a responsive evaluation approach in medical education. Medical Education, 37, 256–266.
65.
DavidsdottirS.LisiP. (2007). Effects of deliberative democracy on school self-evaluation. Evaluation, 13, 371–386.
66.
FleischerD. N.ChristieC. A. (2009). Evaluation use: Results from a survey of U.S. American evaluation association members. American Journal of Evaluation, 30, 158–175.
67.
ForssK.KruseS. E.TautS.TendenE. (2006). Chasing a ghost? An essay on participatory evaluation and capacity development. Evaluation, 12, 128–144.
68.
GibbsD. A.HawkinsS. R.Clinton-SherrodM.NoonanR. K. (2009). Empowering programs with evaluation technical assistance: Outcomes and lessons learned. Health Promotion Practice, 10, 38–44.
69.
GreeneJ. C. (1987). Stakeholder participation in evaluation design: Is it worth the effort?Evaluation and Program Planning, 10, 379–394.
70.
GreeneJ. C. (1988). Communication of results and utilization in participatory evaluation. Evaluation and Program Planning, 11, 341–351.
71.
GreeneJ. G. (1988). Stakeholder participation and utilization in program evaluation. Evaluation Review, 12, 91–116.
72.
HenryG. T.DickeyK. C.AresonJ. C. (1991). Stakeholder participation in educational performance monitoring systems. Educational Evaluation and Policy Analysis, 13, 177–188.
73.
IutcovichJ. M. (1993). Assessing the needs of rural elderly: An empowerment model. Evaluation and Program Planning, 16, 95–107.
74.
JivanjeeP.RobinsonA. (2007). Studying family participation in system-of-care evaluations: Using qualitative methods to examine a national mandate in local contexts. Journal of Behavioral Health Services and Research, 34, 369–381.
75.
JohnsonR. L.WillekeM. J.SteinerD. J. (1998). Stakeholder collaboration in the design and implementation of a family literacy portfolio assessment. American Journal of Evaluation, 19, 339–353.
76.
LennieJ. (2005). An evaluation capacity-building process for sustainable community IT initiatives: Empowering and disempowering impacts. Evaluation, 11, 390–414.
77.
MacLellan-WrightM. F.PattenS.Dela CruzA. M.FlahertyA. (2007). A participatory approach to the development of an evaluation framework: Process, pitfalls, and payoffs. Canadian Journal of Program Evaluation, 22, 99–124.
78.
MayberryR. M.DanielsP.YanceyE. M.AkintobiT. H.BerryJ.ClarkN.DawaghrehA. (2009). Enhancing community-based organizations' capacity for HIV/AIDS education and prevention. Evaluation and Program Planning, 32, 213–220.
79.
McDuffM. D. (2001). Building the capacity of grassroots conservation organizations to conduct participatory evaluation. Environmental Management, 27, 715–727.
80.
MillerR. W. (1987). Using evaluation to support the program advisory function: A case study of evaluator-program advisory-committee collaboration. Evaluation and Program Planning, 10, 281–288.
81.
NorthcoteJ.LeeD.ChokS.WagnerA. (2008). An email-based Delphi approach to tourism program evaluation: Involving stakeholders in research design. Current Issues in Tourism, 11, 269–279.
82.
PapineauD.KielyM. C. (1996). Participatory evaluation in a community organization: Fostering stakeholder empowerment and utilization. Evaluation and Program Planning, 19, 19–93.
83.
ParkinsonS. (2009). Power and perceptions in participatory monitoring and evaluation. Evaluation and Program Planning, 32, 229–237.
84.
PothC. A.ShulhaL. (2008). Encouraging stakeholder engagement: A case study of evaluator behavior. Studies in Educational Evaluation, 34, 218–223.
85.
PreskillH.CaracelliV. (1997). Current and developing conceptions of use: Evaluation use TIG survey results. Evaluation Practice, 18, 209–225.
86.
QuintanillaG.PackardT. (2002). A participatory evaluation of an inner-city science enrichment program. Evaluation and Program Planning, 25, 15–22.
87.
RibislK. M.StecklerA.LinnanL.PattersonC. C.PevznerE. S.MarkatosE.,…PetersonA. B. (2004). The North Carolina Youth Empowerment Study (NCYES): A participatory research study examining the impact of youth empowerment for tobacco use prevention. Health Education & Behavior, 31, 597–614.
88.
RiceM.FranceschiniM. C. (2007). Lessons learned from the application of a participatory evaluation methodology to healthy municipalities, cities and communities initiatives in selected countries of the Americas. Promotion & Education, 14, 68–73.
89.
RossJ. A.JaafarS. B. (2006). Participatory needs assessment. Canadian Journal of Program Evaluation, 21, 131–154.
90.
ThayerC. E.FineA. H. (2001). Evaluation and outcome measurement in the non-profit sector: stakeholder participation. Evaluation and Program Planning, 24, 103–108.
91.
ToalS. A.KingJ. A.JohnsonK.LawrenzF. (2009). The unique character of involvement in multi-site evaluation settings. Evaluation and Program Planning, 32, 91–98.
92.
TurnbullB. (1999). The mediating effect of participation efficacy on evaluation use. Evaluation and Program Planning, 22, 131–140.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
