Evaluation scholars have committed decades of work to the development of evaluator competencies. The 2018 American Evaluation Association (AEA) Evaluator Competencies may be useful for evaluators to identify their strengths and weaknesses to improve their practice; however, a few empirically validated self-assessment tools based on the competencies exist. Two studies were conducted to develop a validated tool. The first study (N = 170) developed the Evaluator Competencies Assessment Tool (ECAT), a self-assessment tool based on the
AEA, 2018 Evaluator Competencies. This study provided evidence for structural validity via confirmatory factor analysis. The second study (N = 142) reconfirmed structural validity with a new sample and examined variables that are associated with evaluator competencies through correlation and t-test analyses. Having a mentor, years of evaluation experience, age, evaluation training, and education level were positively related to evaluator competencies. The ECAT can be used to foster self-reflection for practitioners to improve evaluation competence.
AllenT.EbyL.PoteetM.LentzE.LimaL. (2004). Career benefits associated with mentoring for proteges: A meta-analysis. Journal of Applied Psychology, 89(1), 127–136. https://doi.org/10.1037/0021-9010.89.1.127
2.
American Evaluation Association (2018). AEA evaluator competencies [White paper]. American Evaluation Association. https://www.eval.org/Portals/0/Docs/AEA%20Evaluator%20Competencies.pdf
3.
American Institutes for Research (2017). Teacher leadership: Teacher self-assessment tool. American Institutes for Research, Center on Great Teachers & Leaders. https://gtlcenter.org/sites/default/files/TeacherLeadership_TeacherSelf-Assessment.pdf
4.
Bertrand JonesT. (2014). Socializing emerging evaluators: The use of mentoring to develop evaluation competence. New Directions for Evaluation, 2014(143), 83–96. https://doi-org.ccl.idm.oclc.org/10.1002/ev.20095
5.
BoatengG. O.NeilandsT. B.FrongilloE. A.Melgar-QuiñonezH. R.YoungS. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149. https://doi: 10.3389/fpubh.2018.00149
6.
Canadian Evaluation Society (CES) (2018). Competencies for Canadian evaluation practice. Canadian Evaluation Society. https://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice.pdf.
7.
ChristieC. A.QuiñonesP.FierroL. (2014). Informing the discussion on evaluator training: A look at evaluators’ course taking and professional practice. American Journal of Evaluation, 35(2), 274–290. https://doi.org/10.1177/1098214013503697
8.
CookC.HeathF.ThompsonR. L. (2000). A meta-analysis of response rates in web- or internet-based surveys. Educational and Psychological Measurement, 60(6), 821–836. https://doi.org/10.1177/00131640021970934
CorynC. L. S.BechoL. W.WestineC. D.MateuP. F.Abu-ObaidR. N.HobsonK. A.RamlowM. (2019). Material incentives and other potential factors associated with response rates to internet surveys of American evaluation association members: Findings from a randomized experiment. American Journal of Evaluation, 41(2), 277–296. https://doi.org/10.1177/1098214018818371
11.
CranoW. D.BrewerM. B.LacA. (2015). Principles and methods of social research. Routledge.
12.
DeweyJ. D.MontrosseB. E.SchröterD. C.SullinsC. D.MattoxJ. R. (2008). Evaluator competencies: What’s taught versus what’s sought. American Journal of Evaluation, 29(3), 268–287. https://doi.org/10.1177/1098214008321152
13.
DillmanD. A. (2007). Mail and internet surveys: The tailored design method. Wiley.
14.
DillmanL. M. (2013). Evaluator skill acquisition: Linking educational experiences to competencies. American Journal of Evaluation, 34(2), 270–285. https://doi.org/10.1177/1098214012464512
15.
DunawayK. E.MorrowJ. A.PorterB. E. (2012). Development and validation of the cultural competence of program evaluators (ccpe) self-report scale. American Journal of Evaluation, 33(4), 496–514. https://10.1177/1098214012445280
16.
European Evaluation Society (EES) (2012). The EES evaluation capabilities framework [White paper]. European Evaluation Society. https://europeanevaluation.org/wp-content/uploads/2020/02/EES-EVALUATION-CAPABILITIES-FRAMEWORK-1.pdf
17.
Evaluator Competencies Task Force (2018). A report to the board of the American evaluation association from the evaluator competencies task force. American Evaluation Association.
18.
FieldA. P. (2013). Discovering statistics using IBM SPSS statistics: And sex and drugs and rock ‘n’ roll. Sage.
19.
GalportN.AzzamT. (2017). Evaluator training needs and competencies: A gap analysis. American Journal of Evaluation, 38(1), 80–100. https://doi.org/10.1177/1098214016643183
20.
GeorgeD.MalleryP. (2003). SPSS for windows step by step: A simple guide and reference. 11.0 update. Allyn & Bacon.
21.
GhereG.KingJ. A.StevahnL.MinnemaJ. (2006). A professional development unit for reflecting on program evaluator competencies. American Journal of Evaluation, 27(1), 108–123. https://doi.org/10.1177/1098214005284974
22.
GorsuchR. L. (1983). Factor analysis. Lawrence Erlbaum Associates.
23.
HalungaA. G.OrmeC. D.YamagataT. (2017). A heteroskedasticity robust Breusch-Pagan test for contemporaneous correlation in dynamic panel data models. Journal of Econometrics, 198(2), 209–230. https://doi.org/10.1016/j.jeconom.2016.12.005
24.
HuL.BentlerP. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
25.
IngleM.KlaussR. (1980). Competency-based program evaluation: A contingency approach. Evaluation and Program Planning, 3(4), 277–287. https://doi.org/10.1016/0149-7189(80)90043-9
26.
International Development Evaluation Association (IDEAS) (2012). Competencies for development evaluation evaluators, managers, and commissioners. International Development Evaluation Association. https://ideas-global.org/about/professionalisation/
27.
JenkinC. R.DillmanD. A. (1997). Towards a theory of self-administered questionnaire design. In LybergL.BiemerP.CollinsM.deLeeuwE.DippoC.SchwarzN.TrewinD. (Eds.), Survey measurement and process quality (pp. 165–196). Wiley-Interscience.
28.
JonesT. B. (2006) Cultural competency in evaluation: A black perspective [Dissertation]. Florida State University Libraries. http://purl.flvc.org/fsu/fd/FSU_migr_etd-1389
29.
KingJ.StevahnL. (2015). Competencies for program evaluators in light of adaptive action: What? So what? Now what?New Directions for Evaluation, 2015(145), 21–37. https://doi.org/10.1002/ev.20109
30.
KingJ. A.StevahnL.GhereG.MinnemaJ. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22(2), 229–247. https://doi.org/10.1177/109821400102200206
KlineR. B. (2016). Principles and practice of structural equation modeling (4th ed). Guilford Press. https://ebookcentral.proquest.com
33.
KumariS. S. (2008). Multicollinearity: Estimation and elimination. Journal of Contemporary Research in Management, 3, 87–95.
34.
LaVelleJ. M. (2018). 2018 Directory of evaluator education programs in the United States. University of Minnesota Libraries. https://conservancy.umn.edu/handle/11299/200790
35.
LaVelleJ. M.DonaldsonS. I. (2010). University-based evaluation training programs in the United States 1980—2008: An empirical examination. American Journal of Evaluation, 31(1), 9–23. https://doi.org/10.1177/1098214009356022
LukeD. A.BaumannA. A.CarothersB. J.LandsverkJ.ProctorE. K. (2016). Forging a link between mentoring and collaboration: A new training model for implementation science. Implementation Science, 11(1), 137. https://doi.org/10.1186/s13012-016-0499-y
38.
MaicherB.FrankC. (2015). The development and initial validation of competencies and descriptors for Canadian evaluation practice. Canadian Journal of Program Evaluation, 29(3), 54–69. https://doi.org/10.3138/cjpe.29.3.54
39.
MardiaK. V. (1985). Mardia’s test of multinormality. In KotzS.JohnsonN. L. (Eds.), Encyclopedia of statistical sciences (Vol. 5, pp. 217–221). Wiley.
40.
McGuireM.ZorziR. (2005). Evaluator competencies and performance development. Canadian Journal of Program Evaluation, 20, 73–99. https://doi.org/10.3138/cjpe.20.005
41.
MertensD. M. (1994). Training evaluators: Unique skills and knowledge. New Directions for Program Evaluation, 1994(62), 17–27. https://doi.org/10.1002/ev.1673
42.
Minnesota Evaluation Studies Institute (2018). 2018 AEA evaluator competencies self-assessment. University of Minnesota.
43.
MoshagenM.MuschJ. (2014). Sample size requirements of the robust weighted least squares estimator. Methodology, 10(2), 60–70. https://doi.org/10.1027/1614-2241/a000068
44.
National AfterSchool Association (NAA) (2016). Core knowledge and competencies self-assessment tools. National AfterSchool Association. https://naaweb.org/resources/core-competencies
PerrinB. (2005). How can information about the competencies required for evaluation be useful?Canadian Journal of Program Evaluation, 20, 169–188. https://doi.org/10.3138/cjpe.20.009
47.
Public Health Foundation (2014). Competency assessment for public health professionals. Public Health Foundation. http://www.phf.org/resourcestools/Pages/Competency_Assessments_For_Public_Health_Professionals.aspx
48.
RaginsB. R.CottonJ. (1999). Mentor functions and outcomes: A comparison of men and women in formal and informal mentoring relationships. Journal of Applied Psychology, 84, 529–550. https://doi.org/10.1037/0021-9010.84.4.529
49.
RaginsB. R.McFarlinD. B. (1990). Perceptions of mentor roles in cross-gender mentoring relationships. Journal of Vocational Behavior, 37(3), 321–339. https://doi.org/10.1016/0001-8791(90)90048-7
StevahnL.KingJ.GhereG.MinnemaJ. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26(1), 43–59. https://doi.org/10.1177/1098214004273180
54.
StufflebeamD. L.WingateL. A. (2005). A self-assessment procedure for use in evaluation training. American Journal of Evaluation, 26(4), 544–561. https://doi.org/10.1177/1098214005279730
55.
SudmanS.BradburnM.SchwarzN. (1996). Thinking about answers: The application of cognitive processes to survey methodology. Jossey-Bass Inc.
56.
TabachnickB. G.FidellL. S. (2007). Using multivariate statistics. Pearson/Allyn & Bacon.
57.
WanbergC.WelshE.HezlettS. (2003). Mentoring research: A review and dynamic process model. Research in Personnel and Human Resources Management, 22, 39–124. https://doi.org/10.1016/s0742-7301(03)22002-8
58.
WilcoxY.KingJ. A. (2014). A professional grounding and history of the development and formal use of evaluator competencies. Canadian Journal of Program Evaluation, 28(3), 1–28. https://doi.org/10.3138/cjpe.0028.005
59.
WorthenB. R. (1999). Critical challenges confronting certification of evaluators. American Journal of Evaluation, 20(3), 533–555. https://doi.org/10.1177/109821409902000312
60.
WorthingtonR. L.WhittakerT. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838. https://doi.org/10.1177/0011000006288127
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
