Sage Journals: Discover world-class research

Abstract

This article first makes a case for the need to establish evaluation reporting standards, support for which is rooted in the growing demand for professionalization, in the growing metaevaluation literature, and in growing efforts to develop reporting standards for inquiry efforts. Then, a case is made for a particular set of such standards introduced in this article—the CHecklist for Evaluation-Specific Standards (CHESS). In doing so, this article outlines the process used and presents the resulting checklist and associated standards, which are defined as the minimum, evaluation-specific elements that must be reported to make judgments about the quality of the evaluation. This article also describes a few of the categories to offer illustrative information to help readers better understand and consider the usefulness of the proposed checklist. A discussion of CHESS, including challenges and next steps, is also included.

Keywords

reporting standards for evaluations checklist professionalization metaevaluation

Get full access to this article

View all access options for this article.

References

Alkin

(2004). Context adapted utilization: A personal journey. In Alkin

(Ed.), Evaluation roots: Tracing theorists’ views and influences (pp. 293–303). Thousand Oaks, CA: Sage.

Alkin

M. C.

Ellett

F. S.

(1990). Development of evaluation models. In Walberg

H. J.

Haertel

G. D.

(Eds.), The international encyclopedia of educational evaluation (pp. 15–21). London, England: Pergamon Press.

Altschuld

J. W.

Engle

(2015). Accreditation, certification, and credentialing: Relevant concerns for U.S. evaluators. New Directions for Evaluation, 145, 1–113.

American Evaluation Association. (2004). Guiding principles for evaluators. Retrieved from http://www.eval.org/p/cm/ld/fid=51

American Evaluation Association Competencies Task Force. (2016, 10). Update on the development of the AEA evaluator competencies. Paper presented at the Annual American Evaluation Association Conference, Atlanta, GA.

Barela

(2005, 10). How school district evaluators make sense of their practice: A folk theory. Paper presented at the Joint Conference of the Canadian Evaluation Society and the American Evaluation Association, Toronto, Canada.

Brandon

P. R.

Fukunaga

L. L.

(2014). The state of the empirical research literature on stakeholder involvement in program evaluation. American Journal of Evaluation, 35, 26–44. doi:10.1177/1098214013503699

Brandon

P. R.

Singh

J. M.

(2009). The strength of the methodological warrants for the findings of research on program evaluation. American Journal of Evaluation, 30, 123–157. doi:10.1177/1098214009334507

Centers for Disease Control and Prevention. (2013). Good evaluation questions: Checklist to help focus your evaluation. Retrieved from http://www.cdc.gov/asthma/program_eval/AssessingEvaluationQuestionChecklist.pdf

10.

Chouinard

J. A.

Cousins

J. B.

(2009). A review and synthesis of current research on cross-cultural evaluation. American Journal of Evaluation, 30, 457–494. doi:10.1177/1098214009349865

11.

Christie

C. A.

(2003). Understanding evaluation theory and its role in guiding practice: Formal, folk, and otherwise. In Christie

C. A.

(Ed.), New directions for evaluation: No. 97. The practice–theory relationship in evaluation (pp. 91–93). San Francisco, CA: Jossey-Bass. doi:10.1002/ev.79

12.

Christie

C. A.

Fleischer

D. N.

(2010). Insight into evaluation practice: A content analysis of designs and methods used in evaluation studies published in North American evaluation-focused journals. American Journal for Evaluation, 31, 326–346. doi:10.1177/1098214010369170

13.

CONSORT Group. (n.d.). Impact of CONSORT. Retrieved July 2016, from http://www.consort-statement.org/about-consort/impact-of-consort

14.

Cook

T. D.

Gruder

C. L.

(1978). Metaevaluation research. Evaluation Quarterly, 2, 5–51.

15.

Coryn

C. L. S.

Noakes

L. A.

Westine

C. D.

Schröter

D. C.

(2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32, 199–226. doi:10.1177/1098214010389321

16.

Coryn

C. L. S.

Wilson

L. N.

Westine

C. D.

Hobson

K. A.

Ozeki

Fiekowsky

E. L.

… Schröter

D. C.

(2017). A decade of research on evaluation: A systematic review of research on evaluation published between 2005 and 2014. American Journal of Evaluation, 1–19. Advance online publication. doi:10.1177/1098214016688556

17.

Creswell

J. W.

(2011). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Columbus, OH: Pearson.

18.

Cronbach

L. J.

Ambron

S. R.

Dornbusch

S. M.

Hess

R. D.

Hornik

R. C.

Phillips

D. C.

… Weiner

S. S.

(1980). Toward reform of program evaluation: Aims, methods, and institutional arrangements. San Francisco, CA: Jossey-Bass.

19.

Davidson

J. E.

(2014, 3). It’s the very core of evaluation and makes or breaks our work: So why is it in hardly anyone’s toolkit. Evaluation Connections, 4–5. Retrieved from https://www.europeanevaluation.org/resources/connections

20.

Dewey

Montrosse

B. E.

Schröter

D. C.

Sullins

Mattox

J. R.

(2008). Evaluator competencies: What’s taught versus what’s sought. American Journal of Evaluation, 29, 268–287. doi:10.1177/1098214008321152

21.

Fitzpatrick

J. L.

Christie

C. A.

Mark

M. M.

(2009). Evaluation in action: Interviews with expert evaluators. Thousand Oaks, CA: Sage.

22.

Fournier

D. M.

(1995). Establishing evaluative conclusions: A distinction between general and working logic. New Directions for Evaluation, 68, 15–32.

23.

Fournier

D. M.

(2005). Logic of evaluation. In Mathison

(Ed.), Encyclopedia of evaluation (p. 140). Thousand Oaks, CA: Sage.

24.

Fuller

Peters

Pearson

Anderson

(2014). Impact of the transparent reporting of evaluations with nonrandomized designs reporting guidelines: Ten years on. American Journal of Public Health, 104, 110–117. doi:10.2105/AJPH.2014.302195

25.

Gawande

(2010). The checklist manifesto: How to get things right. New York, NY: Picador.

26.

Grant

S. P.

Mayo-Wilson

Melendez-Torres

G. J.

Montgomery

(2013). Reporting quality of social and psychological intervention trials: A systematic review of reporting guidelines and trial publications. PLoS ONE, 8, 1–11. doi:10.1371/journal.pone.0065442

27.

Griffith

J. C.

(2016). Finding value in epistemology: The role of value in effective inquiry (Doctoral dissertation). Retrieved from ProQuest (Publication no. 10140631). Retrieved from http://gateway.proquest.com/openurl?url_ver=Z39.88-2004&res_dat=xri:pqdiss&rft_val_fmt=info:ofi/fmt:kev:mtx:dissertation&rft_dat=xri:pqdiss:10140631

28.

Griffith

J. C.

Montrosse-Moorhead

(Eds.). (2014). Revisiting truth, beauty, and justice: Evaluating with validity in the 21st century. New Directions for Evaluation, 142, 1–110.

29.

Henry

G. T.

Mark

M. M.

(2003). Toward an agenda for research on evaluation. New Directions for Evaluation, 97, 69–80. doi:10.1002/ev.77

30.

House

E. R.

(1977). The logic of evaluative argument (CSE Monograph 7). Los Angeles: Center for the Study of Evaluation, University of California.

31.

House

E. R.

(1980). Evaluating with validity. Beverly Hills, CA: Sage.

32.

Hurteau

Houle

Mongait

(2009). How legitimate and justified are judgments in evaluation? Evaluation, 15, 307–319. doi:10.1177/1356389009105883

33.

Jacob

Affodegon

W. S.

(2015). Conducting quality evaluations: Four generations of meta-evaluation. SpazioFilosofico, 13, 165–175.

34.

Jacobson

M. R.

Azzam

Baez

J. E.

(2013). The nature and frequency of inclusion of people with disabilities in program evaluation. American Journal of Evaluation, 34, 23–44. doi:10.1177/1098214012461558

35.

Jarlais

D. C. D.

Lyles

Crepaz

, & the TREND Group. (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. American Journal of Public Health, 94, 361–366. doi:10.2105/AJPH.94.3.361

36.

Johnson

Greenseid

Toal

S. A.

King

J. A.

Lawrenz

Volkov

(2009). Research on evaluation use: A review of the empirical literature from 1986–2005. American Journal of Evaluation, 30, 377–410. doi:10.1177/1098214009341660

37.

Kahneman

(2013). Thinking, fast and slow. New York, NY: Farrar, Straus and Giroux.

38.

Labin

S. N.

Duffy

J. L.

Meyers

D. C.

Wandersman

Lesesne

C. A.

(2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33, 307–338. doi:10.1177/1098214011434608

39.

LaVelle

J. M.

Donaldson

S. I.

(2010). University-based evaluation training programs in the United States 1980—2008: An empirical examination. American Journal of Evaluation, 31, 9–23. doi:10.1177/1098214009356022

40.

Mathison

(2007). What is the difference between evaluation and research—and why do we care? In Smith

N. L.

Brandon

P. R.

(Eds.), Fundamental issues in evaluation (pp. 183–196). New York, NY: Guilford.

41.

Miller

R. L.

Campbell

(2006). Taking stock of empowerment evaluation: An empirical review. American Journal of Evaluation, 27, 296–319. doi:10.1177/109821400602700303

42.

Moher

Jones

Lepage

, for the CONSORT Group. (2001). Use of the CONSORT statement and quality of reports of randomized trials: A comparative before and after evaluation. Journal of the American Medical Association, 285, 1992–1995. doi:10.1001/jama.285.15.1992

43.

Montgomery

Grant

Hopewell

Macdonald

Moher

Michie

Mayo-Wilson

(2013). Protocol for CONSORT-SPI: An extension for social and psychological interventions. Implementation Science, 8, 1–7. doi:10.1186/1748-5908-8-99

44.

Montrosse-Moorhead

Griffith

J. C.

(2015, 11). What’s values got to do with it? A pilot study on validity in evaluation. Paper presented at the Annual Meeting of the American Evaluation Association, Chicago, IL.

45.

Nilsson

Hogben

(1983). Metaevaluation. In House

E. R.

(Ed.), New directions for program evaluation: No. 19. Philosophy of evaluation (pp. 83–97). San Francisco, CA: Jossey-Bass.

46.

Oksanen

(2016, 10). Professionalization of evaluation in Europe. Paper presented at the Annual American Evaluation Association Conference, Atlanta, GA.

47.

Patton

M. Q.

(2012). Essentials of utilization-focused evaluation. Thousand Oaks, CA: Sage.

48.

Peck

L. R.

Kim

Lucio

(2012). An empirical examination of validity in evaluation. American Journal of Evaluation, 33, 350–365. doi:10.1177/109821401243992

49.

Plint

A. C.

Moher

Morrison

Schulz

Altman

D. G.

Hill

Gaboury

(2006). Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. The Medical Journal of Australia, 185, 263–267.

50.

Rog

Fitzpatrick

J. L.

Conner

R. F.

(2012). Context: A framework for its influence on evaluation practice. New Directions for Evaluation, 135, 1–105.

51.

Roorda

M. B.

(2014, 10). Identifying criteria for public sector evaluations: What happens in practice? Paper presented at the Meeting of the American Evaluation Association, Denver, CO.

52.

Rosnow

Rosenthal

(1997). People studying people: Artifacts and ethics in behavioral research. New York, NY: Freeman.

53.

Rossi

P. H.

Lipsey

M. W.

Freeman

H. E.

(2005). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage.

54.

Schulz

K. F.

Altman

D. G.

Moher

for the CONSORT Group. (2010). CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. British Medical Journal, 340, 1–28. doi:10.1136/bmj.c869

55.

Schwandt

T. A.

(2015). Evaluation foundations revisited: Cultivating a life of the mind for practice. Stanford, CA: Stanford University Press.

56.

Scriven

(1969). An introduction to meta-evaluation. Educational Product Report, 2, 36–38.

57.

Scriven

(1975). Evaluation bias and its control. Occasional Paper Series sponsored by the National Science Center (Paper #4). Kalamazoo, MI: Evaluation Center, Western Michigan University.

58.

Scriven

(1981). Evaluation, the state of the science. In Welch

(Ed.), Educational evaluation: Recent progress, future needs: Proceedings of the Minnesota Evaluation Conference (pp. 6–18). Minneapolis: Minnesota Research and Evaluation Center.

59.

Scriven

(1991). Evaluation thesaurus. Newbury Park, CA: Sage.

60.

Scriven

(1993). The nature of evaluation. In Scriven

(Ed.), New directions for program evaluation: No. 58. Hard-won lessons in program evaluation (pp. 5–48). San Francisco, CA: Jossey-Bass.

61.

Scriven

(1994). Evaluation as a discipline. Studies in Educational Evaluation, 20, 147–166.

62.

Scriven

(2007). The logic of evaluation. In Hansen

H. V.

Godden

D. M.

Hansen

H. V.

Tindale

C. W.

Blair

J. A.

(Eds.), Dissensus and the search for common ground, CD-ROM (pp. 1–16). Windsor, ON: Ontario Society for the Study of Argumentation (OSSA).

63.

Scriven

(2009). Meta-evaluation revisited. Journal of MultiDisciplinary Evaluation, 6, iii–viii.

64.

Scriven

(2011). The three revolutions. Retrieved August 5, 2011, http://michaelscriven.info/papersandpublications.html

65.

Scriven

(2012). Formative, preformative, and proformative evaluation. Journal of MultiDisciplinary Evaluation, 8, 58–61.

66.

Scriven

(2013). Key evaluation checklist (Version: July 25, 2013). Retrieved from http://michaelscriven.info/

67.

Scriven

(2015). Evaluation revolutions. Journal of MultiDisciplinary Evaluation, 11, 14–21.

68.

Shadish

W. R.

(1999). Evaluation theory is who we are. American Journal of Evaluation, 19, 1–19. doi:10.1177/109821409801900102

69.

Smith

N. L.

Brandon

P. R.

(2007). Fundamental issues in evaluation. New York, NY: The Guilford Press.

70.

Stake

R. E.

Migotsky

Davis

Cisneros

E. J.

Depaul

Dunbar

Jr … Chaves

(1997). The evolving synthesis of program value. Evaluation Practice, 18, 89–109.

71.

Stufflebeam

D. L.

(1971). The relevance of the CIPP evaluation model for educational accountability. Journal of Research and Development in Education, 5, 19–25.

72.

Stufflebeam

D. L.

(1978). Meta evaluation: An overview. Evaluation and the Health Professions, 1, 17–43. doi:10.1177/016327877800100102

73.

Stufflebeam

D. L.

(1999). Program evaluations metaevaluation checklist (long version). Retrieved from https://wmich.edu/evaluation/checklists

74.

Stufflebeam

D. L.

(2001). The metaevaluation imperative. American Journal of Evaluation, 22, 183–209. doi:10.1016/s1098-2140(01)00127-8

75.

Stufflebeam

D. L.

(2011). Meta-evaluation. Journal of MultiDisciplinary Evaluation, 7, 99–158. (Original work published 1974).

76.

Tong

Sainsbury

Craig

(2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19, 349–357. doi:10.1093/intqhc/mzm042

77.

Trevisan

M. S.

(2004). Practical training in evaluation: A literature review. American Journal of Evaluation, 25, 255–272. doi:10.1177/109821400402500212

78.

Turner

Shamseer

Altman

D. G.

Schultz

K. F.

Moher

(2012). Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Systematic Reviews, 1, 1–7. doi:10.1186/2046-4053-1-60

79.

Turner

Shamseer

Altman

D. G.

Weeks

Peters

Kober

… Moher

(2012). CONsolidated Standards of Reporting Trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database of Systematic Reviews, 11, 1–161. doi:10.1002/14651858.MR000030.pub2

80.

Weiss

C. H.

(1972). Evaluation research: Methods for assessing program effectiveness. Englewood Cliffs, NJ: Prentice-Hall.

81.

Williams

J. E.

(1998). Mapping evaluation theory: A numerically developed approach. Dissertation Abstracts International, 49, AAT 8825488, Doctoral dissertation, University of California, Los Angeles.

82.

Wyant

D. C.

Fraser

M. W.

(2016). Journal of the society for social work & research: Author guidelines for manuscripts reporting on qualitative research. Journal of the Society for Social Work & Research, 7, 405–425. doi:10.1086/685816

83.

Yarbrough

D. B.

Shulha

L. M.

Hopson

R. K.

Caruthers

F. A.

(2011). Program evaluation standards (3rd ed.). Thousand Oaks, CA: Sage.