Sage Journals: Discover world-class research

Abstract

In the last 20 years, approaches (i.e. concepts, theories, methods) from the systems fields have increased in popularity among evaluators and evaluation commissioners. Their popularity stems in part from a widespread view that systems approaches transform evaluation practice, despite a lack of evidence. There have been few empirical examinations of the implications of these approaches for evaluation practice. This study analyzed eight cases of seasoned evaluators using systems approaches in diverse areas of evaluation practice to identify implications for how these professionals practice evaluation. Case materials included semi-structured interviews with each evaluator and descriptions of evaluations they conducted. Findings support a variety of case-specific transformations rather than a wholesale transformation in evaluation practice. Future directions for evaluator training, evaluation commissioning and contracting, and research are identified.

Keywords

complexity evaluation practice evaluation theory interdisciplinary systems

Get full access to this article

View all access options for this article.

References

Alkin

Christie

(2005) Unraveling theorists’ evaluation reality. New Directions for Evaluation 106: 111–28.

Attenborough

(2007) Soft systems in a hardening world: Evaluating urban regeneration. In: Williams

Imam

(eds) Systems Concepts in Evaluation: An Expert Anthology. Point Reyes, CA: EdgePress of Inverness, 75–87.

Barnes

Matka

Sullivan

(2003) Evidence, understanding and complexity: Evaluation in non-linear systems. Evaluation 9(3): 265–84.

Better Evaluation Rainbow Framework (2014) Available at: http://betterevaluation.org (accessed 6 January 2014).

Britan

(1978) The place of anthropology in program evaluation. Anthropological and Education Quarterly 51(2): 119–28.

Butler

Copeland-Carson

(eds) (2005) Creating Evaluation Anthropology: Introducing an Emerging Subfield. Berkeley: University of California Press for the National Association for the Practice of Anthropology.

Byrne

(2013) Evaluating complex social interventions in a complex world. Evaluation 19(3): 217–28.

Cabrera

(2015) Systems Thinking Made Simple New Hope for Solving Wicked Problems. Ithaca, NY: Odyssean.

Cabrera

Colosi

(2008) Distinctions, systems, relationships, and perspectives (DSRP): A theory of thinking and of things. Evaluation and Program Planning 31(3): 311–17.

10.

Cabrera

Colosi

Lobdell

(2008) Systems thinking. Evaluation and Program Planning 31(3): 299–310.

11.

Callaghan

(2008) Evaluation and negotiated order: Developing the application of complexity theory. Evaluation 14(4): 399–411.

12.

Campbell

(2011) Assessing the impact of planned social change. Journal of Multidisciplinary Evaluation 7(15): 3–43.

13.

Chelimsky

(1998) The role of experience in formulating theories of evaluation practice. American Journal of Evaluation 19(1): 35–55.

14.

Cronbach

and Associates (1980) Toward Reform of Program Evaluation. San Francisco, CA: Jossey-Bass, Inc.

15.

Datta

(2008) Response to paper ‘Systems thinking’ by D. Cabrera et al.: Systems thinking: An evaluation practitioner’s perspective. Evaluation and Program Planning 31(3): 321–2.

16.

Dyehouse

Bennett

Harbor

. (2009) A comparison of linear and systems thinking approaches for program evaluation illustrated using the Indiana Interdisciplinary GK-12. Evaluation and Program Planning 32(3): 187–96.

17.

Eoyang

(2007) Human systems dynamics: Complexity-based approach to a complex evaluation. In: Williams

Imam

(eds) Systems Concepts in Evaluation: An Expert Anthology. Point Reyes, CA: EdgePress of Inverness, 123–40.

18.

Eoyang

Berkas

(1999) Evaluating performance in a complex adaptive system. In: Lissack

Gunz

(eds) Managing Complexity in Organizations. Westport, CT: Quorum Books, 1–21.

19.

Fetterman

(1988) Qualitative Approaches to Evaluation in Education: The Silent Scientific Revolution. New York: Praeger Publications.

20.

Fitzpatrick

Christie

Mark

(eds) (2009) Evaluation in Action. Thousand Oaks, CA: SAGE.

21.

Forss

Marra

Schwartz

(eds) (2011) Evaluating the Complex: Attribution, Contribution, and Beyond. London: Transaction Publishers.

22.

Foster-Fishman

Nowell

Yang

(2007) Putting the system back into systems change: A framework for understanding and changing organizational and community systems. American Journal of Community Psychology 39(3–4): 197–215.

23.

Gates

(2016) Making sense of the emerging conversation in evaluation about systems thinking and complexity science. Evaluation and Program Planning 59: 62–73.

24.

Hargreaves

(2014) Rapid evaluation approaches for complex initiatives. Mathematica Policy Research. Available at: https://aspe.hhs.gov/sites/default/files/pdf/73191/rs_EvalApproach.pdf (accessed 5 January 2016).

25.

Hargreaves

(2010) Evaluating system change: A planning guide. Available at: https://www.mathematica-mpr.com/our-publications-and-findings/publications/evaluating-system-change-a-planning-guide (accessed 5 January 2016).

26.

Henning

Chen

(2012) Systems thinking: Common ground or untapped territory? Systems Research and Behavioral Science 29: 470–83.

27.

Hieronymi

(2013) Understanding systems science: A visual and integrative approach. Systems Research and Behavioral Science 30: 580–95.

28.

Hummelbrunner

(2007) Systemic evaluation in the field of regional development. In: Williams

Imam

(eds) Systems Concepts in Evaluation: An Expert Anthology. Point Reyes, CA: EdgePress of Inverness, 161–80.

29.

Hummelbrunner

(2011a) Systems thinking and evaluation. Evaluation 17(4): 395–403.

30.

Hummelbrunner

(2011b) Applying systems thinking to evaluation of structural funds. In: Olejniczak

Kozak

Bienias

(eds) Evaluating the Effects of Regional Interventions: A Look Beyond Current Structural Funds’ Practice. Warsaw: Ministry of Regional Development, 255–75.

31.

Hummelbrunner

Reynolds

(2013) Systems thinking, learning and values in evaluation. Evaluation Connections (European Evaluation Society Newsletter), 9–10.

32.

Institute of Development Studies Bulletin (IDS) (2015) In: Befani

Ramalingam

Stern

(eds) Towards systemic approaches to evaluation and impact. Available at: http://www.ids.ac.uk/publication/towards-systemic-approaches-to-evaluation-and-impact (accessed 10 December 2015).

33.

Jones

(2012) Taking responsibility for complexity: How implementation can achieve results in the face of complex problems. Overseas Development Institute Working Paper 330. London, UK. Available at: http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/6485.pdf (accessed 2 November 2015).

34.

Mark

Henry

Julnes

(2000) Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs. San Francisco, CA: Jossey-Bass.

35.

Midgley

(2000a) Systemic Intervention: Philosophy, Methodology, and Practice. New York: Plenum Publishers.

36.

Midgley

(2007) Systems thinking for evaluation. In: Imam

Williams

(eds) Systems Concepts in Evaluation: An Expert Anthology. Point Reyes, CA: EdgePress of Inverness.

37.

Miller

Levine

McNall

. (2009) A dynamic model of client recruitment and retention in community-based HIV prevention programs. Health Promotion Practice 12(1): 135–46.

38.

Milstein

(2008) Hygeia’s constellation: Navigating health futures in a dynamic and democratic world. Available at: http://www.rethinkhealth.org/resources/hygeias-constellation-navigating-health-futures-in-a-dynamic-and-democratic-world/ (accessed 5 August 2015).

39.

Morell

(2010) Evaluation in the Face of Uncertainty: Anticipating Surprise and Responding to the Inevitable. New York: The Guilford Press.

40.

Mowles

(2014) Complex, but not quite complex enough: The turn to the complexity sciences in evaluation scholarship. Evaluation 20(2): 160–75.

41.

Murphy

(2016) Nine guiding principles to help youth overcome homelessness: A principles-focused developmental evaluation. In: Patton

McKegg

Wehipeihana

(eds) Developmental Evaluation Exemplars: Principles in Practice. New York: The Guilford Press.

42.

National Cancer Institute (2007) Greater than the sum: Systems thinking in tobacco control. Tobacco Control Monograph No. 18. Bethesda, MD: U.S. Department of Health and Human Services, National Institutes of Health, National Cancer Institute. NIH Pub. No. 06–6085.

43.

Patton

(2011) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York: The Guilford Press.

44.

Patton

(2016) What is essential in developmental evaluation? On integrity, fidelity, adultery, abstinence, impotence, long-term commitment, integrity, and sensitivity in implementing evaluation models. American Journal of Evaluation 37(2): 250–65.

45.

Patton

McKegg

Weipeihana

(eds) (2016) Developmental Evaluation Exemplars: Principles in Practice. New York: The Guilford Press.

46.

Ramalingam

(2013) Aid on the Edge of Chaos: Re-thinking International Cooperation in a Complex World. Oxford: Oxford University Press.

47.

Reynolds

Forss

Hummelbrunner

. (2012). Complexity, systems thinking and evaluation: An emerging relationship? Evaluation Connections: Newsletter of the European Evaluation Society, 7–9.

48.

Rogers

(2008) Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation 14(1): 29–48.

49.

Russ-eft

Preskill

(2009) Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change, 2nd edn. New York: Basic Books.

50.

Sanderson

(2000) Evaluation in complex policy systems. Evaluation 6(4): 433–54.

51.

Schwandt

(2016) Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice. Stanford, CA: Stanford University Press.

52.

Schwandt

(2014) On the mutually informing relationship between practice and theory in evaluation. American Journal of Evaluation 35(2): 231–6.

53.

Stake

(2005) Multiple Case Study Analysis. New York: Guilford Publications.

54.

Stern

Saunders

Stame

(2015) Standing back and looking forward: Editors’ reflections on the 20th Anniversary of Evaluation. Evaluation 21(4): 380–90.

55.

Stufflebeam

Shinkfield

(2007) Evaluation Theory, Models, and Applications. San Francisco, CA: John Wiley & Sons, Inc.

56.

Sweeney

Meadows

(2010) The Systems Thinking Playbook: Exercises to Stretch and Build Learning and Systems Thinking Capabilities. White River Junction, VT: Chelsea Green Publishing.

57.

Trickett

Beehler

Deutsch

. (2011) Advancing the science of community-level interventions. American Journal of Public Health 101(8): 1410–19.

58.

Ulrich

(1988) Churchman’s ‘process of unfolding’ and its significance for policy analysis and evaluation. Systems Practice 1(4): 415–28.

59.

Wallace

Geller

Ogawa

(eds) (2015) Assessing the Use of Agent-Based Models for Tobacco Regulation. Washington, DC: The National Academies Press.

60.

Walton

(2014) Applying complexity theory: A review to inform evaluation design. Evaluation and Program Planning 45: 119–26.

61.

Wasserman

(2010) Using a systems orientation and foundational theory to enhance theory-driven human service program evaluations. Evaluation and Program Planning 33(2): 67–80.

62.

Wasserman

Emery

(2013) Valuing Kujichagulia: A self-determination theory-based approach to evaluating Africentric rites of passage programming. Black Child Journal Special Edition: Rites of Passage Foundations and Practices Summer: 24–59.

63.

Westhorp

(2012) Using complexity-consistent theory for evaluating complex systems. Evaluation 18(4): 405–20.

64.

Williams

(2005) Systems thinking. In: Mathison

(ed.) Encyclopedia of Evaluation. Thousand Oaks, CA: SAGE.

65.

Williams

Hummelbrunner

(2011) Systems Concepts in Action: A Practitioner’s Toolkit. Stanford, CA: Stanford University Press.

66.

Williams

Imam

(eds) (2007) Systems Concepts in Evaluation: An Expert Anthology. Point Reyes, CA: EdgePress of Inverness.

67.

Wolf-Branigin

(2013) Using Complexity Theory for Research and Program Evaluation. Oxford: Oxford University Press

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.18 MB

0.00 MB