In response to a call for more research on evaluator education, this paper explores the use of role-play as an in-class activity for students in a graduate course in evaluation. We describe the activity in detail and use the Structure of Observed Learning Outcomes taxonomy as a tool to describe the cognitive complexity of student responses in the activity. Using a collaborative autoethnographic approach, we found that the role-play activity provided the opportunity for students to exhibit knowledge of evaluation theories and for us to observe the complexity of their responses.
ArchibaldT. (2021). The role of evaluative thinking in the teaching of evaluation. Canadian Journal of Program Evaluation, 35(3), 310–319. https://doi.org/10.3138/cjpe.69753
4.
BiggsJ. B.CollisK. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (structure of the observed learning outcome). Academic Press.
5.
BiggsJ. B.TangC.KennedyG. (2022). Teaching for quality learning at university (5th ed.). Open University Press.
6.
BlalockA. E.AkehiM. (2018). Collaborative autoethnography as a pathway for transformative learning. Journal of Transformative Education, 16(2), 89–107. https://doi.org/10.1177/1541344617715711
7.
BloomB. S.EngelhartM. D.FurstE. J.HillW. H.KrathwohlD. R. (1956). Handbook I: Cognitive domain. David McKay.
8.
BowmanN. (2021). Praxis makes perfect? Transcending textbooks to learning evaluation experientially and in cultural contexts. Canadian Journal of Program Evaluation, 35(3), 320–329. https://doi.org/10.3138/cjpe.69698
9.
BoyceA. S.McGowanB. L. (2019). An exploration of two novice evaluation educators’ experiences developing and implementing introduction to evaluation courses. American Journal of Evaluation, 40(1), 119–136. https://doi.org/10.1177/1098214018778812
10.
BrabrandC.DahlB. (2009). Using the SOLO taxonomy to analyze competence progression of university science curricula. Higher Education, 58(4), 531–549. https://doi.org/10.1007/s10734-009-9210-4
11.
ChangH.NgunjiriF.HernandezK. C. (2013). Collaborative Autoethnography. Left Coast Press.
12.
ChenM.GullicksonA. M.DattaR. P. (2020). Guide for learning objectives to analyze deep and surface learning. Supported by the US National Science Foundation under Grant No. 1853472.https://evalu-ate.org/assessment-pd-toolkit/
13.
ChouinardJ. A.BoyceA. S.HicksJ.JonesJ.LongJ.PittsR.StockdaleM. (2017). Navigating theory and practice through evaluation fieldwork: Experiences of novice evaluation practitioners. American Journal of Evaluation, 38(4), 493–506. https://doi.org/10.1177/1098214016667582
14.
ChouinardJ. A.LaVelleJ. M. (2021). Re-envisioning evaluation pedagogy with a community of scholar teachers. Canadian Journal of Program Evaluation, 35(3), 466–472. https://doi.org/10.3138/cjpe.71359
15.
ChristieC. A.AlkinM. C. (2013). An evaluation theory tree. In AlkinM. (Ed.), Evaluation roots: A wider perspective of theorists’ views and influences (2nd ed., pp. 12–65). Sage.
DaviesR.MacKayK. (2014). Evaluator training: Content and topic valuation in university evaluation courses. American Journal of Evaluation, 35(3), 419–429. https://doi.org/10.1177/1098214013520066
18.
DaviesR. S. (2021). Establishing and developing professional evaluator dispositions. Canadian Journal of Program Evaluation, 35(3), 330–349. https://doi.org/10.3138/cjpe.71156
19.
DillmanL. M. (2013). Evaluator skill acquisition: Linking educational experiences to competencies. American Journal of Evaluation, 34(2), 270–285. https://doi.org/10.1177/1098214012464512
20.
GullicksonA. M.KingJ. A.LaVelleJ. M.ClintonJ. M. (2019). The current state of evaluator education: A situation analysis and call to action. Evaluation and Program Planning, 75(2019), 20–30. https://doi.org/10.1016/j.evalprogplan.2019.02.012
21.
HenryD.McCormackS.SaeedN. (2019). An evaluation of a simulation learning task in finance education. Journal of Education for Business, 94(8), 549–560. https://doi.org/10.1080/08832323.2019.1579162
22.
HernandezK. A. C.ChangH.NgunjiriF. W. (2017). Collaborative autoethnography as multivocal, relational, and democratic research: Opportunities, challenges, and aspirations. a/b: Auto/Biography Studies, 32(2), 251–254. https://doi.org/10.1080/08989575.2017.1288892
23.
IlgüyM.IlgüyD.FişekçioğluE.OktayI. (2014). Comparison of case-based and lecture-based learning in dental education using the SOLO taxonomy. Journal of Dental Education, 78(11), 1521–1527. https://doi.org/10.1002/j.0022-0337.2014.78.11.tb05827.x
24.
KingJ.AyooS. (2020). What do we know about evaluator education? A review of peer-reviewed publications (1978–2018). Evaluation and Program Planning, 79(2020), 101785. https://doi.org/10.1016/j.evalprogplan.2020.101785
25.
KingJ. A. (2018). Book review. [Review of the books RealWorld evaluation: Working under budget, time, data, and political constraints (2nd ed.), by M. Bamberger, J. Rugh, and L. Mabry, and Practical program evaluation: Theory-driven evaluation and the integrated evaluation perspective (2nd ed.), by H. T. Chen.]. The American Journal of Evaluation, 39(2), 305–311. https://doi.org/10.1177/1098214017730124
26.
KingJ. A.StevahnL. (2013). Interactive evaluation practice: Mastering the interpersonal dynamics of program evaluation. Sage Publications.
LaVelleJ. M.ChouinardJ. A. (2021). How to teach evaluation. Canadian Journal of Program Evaluation, 35(3), 279–281. https://doi.org/10.3138/cjpe.71277
29.
LaVelleJ. M.DonaldsonS. I. (2021). Opportunities and challenges ahead for university-based evaluator education programs, faculty, and students. American Journal of Evaluation, 42(3), 314–457. https://doi.org/10.1177/1098214020937808
30.
LaVelleJ. M.NeubauerL. C.BoyceA. S.ArchibaldT. (2023). Preparing the 21st century evaluator: Critically-defined and responsive education and training. New Directions for Evaluation, 2023(177), 13–22, https://doi.org/10.1002/ev.20542
31.
LaVelleJ. M.YangZ. (2021). In their own words: Student key learning experiences in an introductory evaluation course. Canadian Journal of Program Evaluation, 35(3), 450–465. https://doi.org/10.3138/cjpe.69601
32.
MarkM. M. (2008). Building a better evidence base for evaluation theory: Beyond general calls to a framework of types of research on evaluation. In SmithN. L.BrandonP. (Eds.), Fundamental issues in evaluation (pp. 111–134). Guilford Press.
33.
MillerR. L. (2010). Developing standards for empirical examinations of evaluation theory. American Journal of Evaluation, 31(3), 390–399. https://doi.org/10.1177/1098214010371819
34.
Montrosse-MoorheadB.GambinoA. J.YahnL. M.FanM.VoA. T. (2022). Evaluator education curriculum: Which competencies ought to be prioritized in master’s and doctoral programs?American Journal of Evaluation, 43(2), 156–308. https://doi.org/10.1177/10982140211020326
35.
NeubauerL. C.LaVelleJ. M.BoyceA. S.ArchibaldT. (2023). The evaluator and evaluation: Critical considerations for the future of education and training. New Directions for Evaluation, 2023(177), 153–162. https://doi.org/10.1002/ev.20549
36.
NygaardC.CourtneyN.LeighE. (2012). Simulations, games and role play in university education. Libri.
37.
PattonM. Q. (2014). Qualitative research & evaluation methods: Integrating theory and practice. Sage Publications.
38.
RooneyD.NyströmS. (2018). Simulation: A complex pedagogical space. Australasian Journal of Educational Technology, 34(6), 53–64. https://doi.org/10.14742/ajet.4470
39.
RoyR.UekusaS. (2020). Collaborative autoethnography: “Self-reflection” as a timely alternative research approach during the global pandemic. Qualitative Research Journal, 20(4), 383–392. https://doi.org/10.1108/QRJ-06-2020-0054
SmithN. L. (1993). Improving evaluation theory through the empirical study of evaluation practice. Evaluation Practice, 14(3), 237–242. https://doi.org/10.1177/109821409301400302
44.
SowlS.Amrein-BeardsleyA.CollinsC. (2022). Teaching program evaluation: How blending theory and practice enhance student-evaluator competencies in an education policy graduate program. Evaluation and Program Planning, 94(2022), 1–13. https://doi.org/10.1016/j.evalprogplan.2022.102139
45.
StroessnerS. J.BeckermanL. S.WhittakerA. (2009). All the world’s a stage? Consequences of a role-playing pedagogy on psychological factors and writing and rhetorical skill in college undergraduates. Journal of Educational Psychology, 101(3), 605–620. https://psycnet.apa.org/doi/10.1037/a0015055https://doi.org/10.1037/a0015055
46.
StufflebeamD.ZhangG. (2016). The CIPP evaluation model: How to evaluate for improvement and accountability. Guilford.
47.
ToveyT. L. S.ArchibaldT. (2023). The relationship between reflective practice, evaluative thinking, and practical wisdom. In HurteauM.ArchibaldT. (Eds.), Practical wisdom for an ethical evaluation practice (pp. 87–101). Information Age Publishing.
48.
ToveyT. L. S.SkolitsG. J. (2022). Conceptualizing and engaging in reflective practice: Experienced evaluators’ perspectives. American Journal of Evaluation, 43(1), 5–25. https://doi.org/10.1177/1098214020983926
49.
TrevisanM. S. (2004). Practical training in evaluation: A review of the literature. American Journal of Evaluation, 25(2), 255–272. https://doi.org/10.1177/109821400402500212
