Sage Journals: Discover world-class research

Abstract

Evaluation capacity building is generally conceptualized as occurring either at the individual or the organizational levels. However, ongoing societal crises require organizations within or across sectors to work together to find solutions to complex problems and to evaluate joint initiatives. Interorganizational evaluation capacity is required to ensure the ongoing conduct and use of evaluations to support interorganizational decision-making and improvement. This exploratory study describes and analyzes four cases of interorganizational evaluation capacity building initiatives in the public, health and community sectors in Canada and Denmark to identify their key dimensions. Preliminary findings highlight the importance of developing individual and organizational evaluation capacity as well as the need to provide stakeholders with interorganizational evaluation training and projects in which they can work together to better learn about each other’s organizations and challenges and find solutions to common problems.

Keywords

evaluation capacity interorganizational evaluation organizational learning

In recent years, emergent international challenges, such as climate change, social inequality, increased migration, changing demographics, and a global pandemic, have required organizations to collaborate within and between sectors to achieve common goals. Such challenges, typically managed by social service and health organizations, including national and local governments, are complex and typically exceed the resources and operational capacity of individual organizations and sectors (Keast et al., 2007). Thus, collaboration is necessary to “help stakeholders negotiate shared understanding and shared meaning about the problem and its possible solutions” (Keast et al., 2007: 5).

Evaluation, as a practice focused on determining the merit and worth of social and health interventions, can play an essential role in interorganizational collaborations. Evaluators can support the development of innovative initiatives, by providing early data and background information (Patton, 2012); they can also support organizations and groups in developing theories of change, collecting and analyzing data, and formulating recommendations regarding improvement, learning, and decision-making. In this way, evaluation can not only support organizations but also broader interorganizational initiatives or sectors focused on solving global problems. However, conducting and using evaluations in interorganizational or sectoral initiatives, just like in standalone organizations, requires particular skills, systems, and structures that are generally defined as evaluation capacity (EC). EC and its corollary, evaluation capacity building (ECB), are typically described in individual or organizational terms (for instance, see Preskill and Boyle, 2008), but the oft-cited definition provided by Stockdill et al. (2002) also includes mention of multiple entities:

A context-dependent intentional action system of guided processes and practices for bringing about and sustaining a state of affairs in which quality program evaluation and its appropriate uses are ordinary and ongoing practices within and/or between one or more organizations/programs/sites. (Stockdill et al., 2002: 8)

Although early publications on ECB tended to focus on individual skill development through formal training (direct ECB) or practice (indirect ECB), current conceptions of ECB also focus on the development of tools, procedures, policies, and systems that support the ongoing conduct and use of evaluation across one or multiple organizations, as per Stockdill et al. (2002). However, a recent integrative review of the literature by Bourgeois et al. (2023) shows that most case applications and studies of EC or ECB focus on single organizations; little is known, currently, about how EC might apply in interorganizational initiatives, although systems thinking and complexity theory support the development of EC for these large-scale collaborations and projects (Braithwaite et al., 2017).

In this article, we examine how building capacity to do and use evaluation across organizations and sectors can support interorganizational initiatives and lead to potential interorganizational learning outcomes through the presentation and analysis of four case studies conducted in Denmark and Canada. More specifically, we seek to document the approaches and strategies that support ECB and learning at an interorganizational or sectoral level.

Literature review

Understanding the components of interorganizational EC requires a solid foundation in both individual and organizational EC. This section seeks to establish the theoretical and conceptual framing of the study through an examination of the various EC and ECB frameworks, models and theories found in the literature.

Individual evaluation capacity

Individual EC generally refers to the knowledge, skills and attitudes required to conduct evaluations successfully and/or to use them toward program improvement, learning, and decision-making. As per our recent integrative review (Bourgeois et al., 2023), individual EC is at the heart of most EC case applications and research studies. Commonly cited (but non-exhaustive) individual EC dimensions include the following¹:

Evaluation practice competencies, such as logic model development, evaluation theories and approach selection, data collection instrument development, and data collection and analysis approaches (Bourgeois and Cousins, 2013; Cousins et al., 2014a; Mayne, 2020; Nielsen et al., 2011; Preskill and Boyle, 2008; Volkov, 2008).

Stakeholder involvement and decision support competencies, such as facilitation skills, teaching skills, communication skills, conflict resolution skills, and reporting approaches (Bourgeois and Cousins, 2013; Compton and Baizerman, 2012; Labin et al., 2012; Nielsen et al., 2011).

Evaluative thinking competencies, that are often meant for evaluation users rather than evaluation practitioners and refer to a general understanding of the role of evaluations in organizations and how they are conducted; they also refer to how evaluations should be used to support program improvement, learning, and decision-making within the organization (Bourgeois and Cousins, 2013; Compton and Baizerman, 2012; Cousins et al., 2014b; Mayne, 2020; Volkov, 2008).

As described previously, individual ECB approaches include direct ECB, which refers to training through formal courses and/or workshops as well as written materials and online toolkits. Indirect ECB, which refers to “learning by doing” through involvement in evaluation and coaching/mentoring in practice, is also a common approach to building individual ECB (Cousins et al., 2014). Both approaches can be used to develop all of the competencies listed previously, including evaluative thinking for evaluation users or organizational decision-makers.

Organizational evaluation capacity

Organizational EC refers to the systems, structures, and procedures that exist within an organization to support ongoing evaluation practice and use. These transcend the individual and remain a permanent part of the organization, even when individuals change roles or move on to other organizations. These can include the following:

Organizational resources dedicated to evaluation, including human, technical, and financial resources (Bourgeois and Cousins, 2013; Labin et al., 2012; Taylor-Ritzler et al., 2013).

Organizational processes that facilitate evaluation, including standing committees on evaluation and decision-making processes that include evaluative evidence (Bourgeois and Cousins, 2013; Cousins et al., 2014a).

Organizational policies that support the conduct and use of evaluation, including specific evaluation policies as well as other policies that emphasize the need for evaluative evidence (Al Hudib and Cousins, 2022; Christie and Lemire, 2019; Mark et al., 2009).

Although organizational EC is an important theoretical construct in the EC and ECB literature, concrete examples of organizational EC building are found more rarely in published articles on ECB and typically describe how individual ECB practices have been applied across an organization. One notable exception is the literature on organizational evaluation policies, which has been developed substantially over the last 15 years. Research on evaluation policy consistently shows concrete linkages between such policies and organizational EC in various sectors (Al Hudib and Cousins, 2022; Christie and Lemire, 2019; Mark et al., 2009), including governmental (Bourgeois and Maltais, 2023), education (Christie and Fierro, 2012), and philanthropic foundations (Dillman and Christie, 2017).

A corollary of organizational EC is also found in the international development literature, labeled as national monitoring and evaluation capacity development (M&E ECD). This body of literature pays particular attention to the conditions or enabling environment within which EC can be built in government organizations; these are especially relevant when considering interorganizational EC, given the connections between government levels and institutions. It also includes the perspectives and roles of multiple actors within these large systems. National ECD efforts are similar to those found at the organizational level, including conferences and training to foster individual learning and collaboration as well as the development of a country-level evaluation policy, evaluation standards, evaluation plans, and quality reviews (Chirau et al., 2021; Goldman et al., 2018).

Interorganizational or sectoral evaluation capacity

Although the dimensions of individual and organizational EC have been well-documented over the years, the literature on interorganizational or sectoral EC is much sparser. An extensive search of the literature on this topic has resulted in the identification of only a few studies (e.g. Dinh et al., 2015; Goodyear, 2011; Inkleas et al., 2017). Such studies primarily examine individual or organizational EC or ECB within a broader interorganizational context, without specifically addressing how this context might influence EC or highlight different dimensions of EC from those found at the organizational level.

The definition of ECB provided by Stockdill et al. (2002) hints at interorganizational EC within a broader context. In this article, we define interorganizational EC as the systems and structures that facilitate collaborative evaluation practices between organizations and that foster learning and improvement within broader sectors such as healthcare and education. These systems and structures can include shared evaluation policies, practices and support structures as well as collective reporting mechanisms.

Conceptual framework

Interorganizational EC likely requires the development of individual and organizational EC and extends these to arrangements involving multiple organizations. We used the conceptual framework proposed by Cousins and his colleagues (2014a) to identify the key aspects of individual and organizational EC that might apply to interorganizational arrangements as well as other antecedents or mediating factors that can influence the development of EC and its logical consequences at all three levels (individual, organizational, and interorganizational). We propose some adaptations to this framework below, based on the need to expand it to the interorganizational or sectoral space.

Sources of knowledge, skills, and abilities

Cousins et al. (2014a) refer to this component of the framework as the existing evaluation competencies held by evaluators or practitioners as well as those developed as part of an ECB intervention. These typically tend to focus on evaluation methods and practice and may also include evaluation theory.

Organizational support structures

These refer to the various mechanisms, tools and supports that exist within organizations or groups of organizations to foster learning and knowledge transfer from evaluations. They can include human, technical and financial resources to conduct evaluations or to engage in direct or indirect ECB initiatives, as well as templates, tools, and policies that support evaluative practice within an organization or a group of organizations.

Capacity to do evaluation

According to Cousins et al. (2014a), capacity to do evaluation arises from individual learning opportunities as well as organizational support structures. They describe this element as the ability to conduct actual evaluation tasks. This includes both technical aspects of evaluation (e.g. data collection and analysis) and relational aspects of evaluation (e.g. facilitation).

Evaluative inquiry

This component of the conceptual framework refers to the evaluation activities that may take place during or following ECB interventions. Cousins et al. (2014a) mention that evaluative activities can take many forms, which is helpful when considering interorganizational ECB projects. Such projects often include collaborative evaluation work that may or may not fit under the guise of more traditional evaluation activities.

Mediating conditions

These refer to the factors and conditions that affect the extent to which evaluations are used in an organization, or in this case, in multiple organizations simultaneously. According to Cousins et al. (2014a), these can include timeliness, nature of feedback provided, informational needs of stakeholders, collaboration within the evaluation process, and relevance to organizational priorities.

Capacity to use evaluation

This component of the framework refers to the ability of an organization, or a group of organizations, to integrate evaluation findings into decision-making processes. This can include instrumental uses, conceptual uses, or symbolic uses, as well as process use when stakeholders are more closely involved in the evaluation process. Although Cousins et al. (2014a) do not specifically refer to evaluative thinking as a determining factor in an organization’s capacity to use, they mention “. . . an inquiry-minded approach to routine organizational business and processes” (p. 17), which implies the presence of evaluative thinking within the organization.

Organizational learning capacity

This extends further the notion of “capacity to use” and refers to organizational characteristics that support the ongoing production and use of evidence to inform decision-making. Such characteristics include leadership support, and structures that facilitate knowledge transfer, teamwork, and cooperation within the organization. Although these tend to focus on individual organizations, they can also be found in groups of organizations working collaboratively toward shared goals.

Organizational consequences

These refer to organizational learning outcomes influenced by ECB and evaluative activities. For example, Cousins et al. (2014a) refer to “shared mental representations or understandings of the organization and how it operates” (p. 18). They also include organizational learning processes including single-loop learning, where organizations routinely engage in questioning and evidence gathering to support continuous improvement, as well as double-loop learning, through which fundamental changes in the organization’s understanding and practice occur.

Research objectives

Given the scarcity of published frameworks, models, and examples that describe interorganizational EC, our study sought to compare four different cases of interorganizational evaluation activities from the public, community, and health sectors in Denmark and Canada. This comparison, based on the expanded conceptual framework of EC proposed by Cousins et al. (2014a), aimed to identify the individual, organizational, and interorganizational dimensions of EC in the four cases, as well as the approaches and processes used to build capacity at the interorganizational level.

Methodology

Research design and case description

A comparative multiple case study (Yin, 2018) was used to document interorganizational EC and evaluation practice in various sectors in Canada and Denmark. Four cases were selected based on the following criteria:

Each case describes an evaluation or ECB initiative involving multiple organizations (interorganizational initiative) or a group of organizations working on common goals or outcomes (sectoral initiative);

Each case is based on secondary data collected through other studies pertaining to evaluation capacity or evaluation projects (measured through standardized tools or described qualitatively);

Each case focuses on organizations situated either within Canada or Denmark.

The four cases include the following: (1) a sub-set of Canadian federal departments and agencies, (2) community-based organizations in Quebec, Canada, (3) quality improvement projects in the healthcare sector in Denmark, and (4) regional organizations in the healthcare sector in Northern Jutland, Denmark. A summary of each case is provided in the “Findings” section below.

Procedures—Case development

Each case was developed using secondary EC or evaluation project data collected through document reviews, key informant interviews, and EC assessments, obtained as part of previous studies of individual and organizational EC (e.g. Bourgeois and Buetti, 2019; Bourgeois and Cousins, 2013; Bourgeois and Leblanc, 2019). These studies were conducted in recent years and focused on describing existing EC and evaluation practice in each group of organizations. To build each case, data from the previous studies were extracted and organized (Yin, 2018) according to key components of the theoretical framework, including

Interorganizational or sectoral resources/antecedents on evaluation and ECB shared by individuals or organizations (e.g. communities of practice, documents/guides, conferences, or discussion tables);

Evidence of individual and organizational EC within the sector, collaboration, or partnership;

Evidence of interorganizational or sectoral EC or evaluative practice;

Strategies that have been used or implemented at an interorganizational or sectoral level to build EC (e.g. multi-organizational training activities, shared resources, studies on sector EC, frameworks that outline EC or evaluation results for a sector, meta-evaluations).

Extracted data for each case were analyzed thematically and resulted in four-case narratives that illustrate various elements of interorganizational or sectoral EC. These narratives served as primary data for the cross-case analysis. Each author was responsible for the development of two cases; once developed, each case narrative was reviewed by the other author to identify any potential gaps or needed clarifications.

Procedures—Cross-case analysis

A cross-case analysis was conducted to identify and examine similarities and differences between the four cases based on the conceptual framework by Cousins et al. (2014a). Based on the full case narratives, the cross-case analysis entailed the coding and development of data summaries for each analytical component found in the case narratives. The data summaries were then reviewed collectively for each analytical component in order to highlight commonalities as well as key differences between cases.

Findings

We begin with a brief description of each case, to provide some background and context. An overview of the cross-case analysis findings is then provided. The results from the cross-case analysis follow to provide a more complete sense of how interorganizational EC can be built and how it manifests in shared evaluative practice.

Case descriptions

Case 1: Canadian federal government departments and agencies

The Canadian federal government has a long-standing history of evaluative practice, driven by successive government-wide directives and policies over more than 40 years. Akin to evaluation policies found in other jurisdictions and described by Trochim (2009), such guiding documents describe what should be evaluated, how often evaluations should occur, what key questions should be covered by evaluations, and how evaluations should be used. Evaluation policies and directives are produced and disseminated by the Treasury Board Secretariat, which functions as a coordinator of evaluation across the federal government. Individual federal departments and agencies (organizations) are responsible, under the supervision of their lead administrator, for the establishment of an organizational evaluation unit. Such units are responsible for planning, implementing, reporting, and following-up on evaluation studies. For the most part, evaluation units focus on their own organizations’ programs and policies; however, in some cases, interdepartmental programs require a coordinated approach to evaluation and lead to the establishment of interorganizational, or horizontal, evaluation projects. In addition to these, organizations also often recognize the value of sharing best practices and discuss common evaluation issues; this happens through meetings or conferences organized by the Treasury Board Secretariat and evaluation professional associations. Professional development opportunities are also often offered to federal evaluators from various providers, including universities, professional associations, and the internal public service school.

Multiple studies have been conducted in recent years to document and assess the federal government’s EC. Such examples include Bourgeois and Maltais’ (2023) review of policy compliance as well as Bourgeois et al. (2013) validation study of an EC measurement instrument for federal government organizations. A survey of Heads of Evaluation (directors and directors general supervising departmental evaluation units), conducted in 2018, revealed that overall, departments and agencies have the required tools and competencies (3.5/4, n = 22); and that federal evaluators have access to the required tools (3.25/4; n = 22), financial resources (3/4; n = 21); and policies and procedures (2.95/4; n = 22) to conduct high-quality evaluations (Bourgeois and Leblanc, 2019).

Beyond organizational EC in departments and agencies, evaluation use is also fostered at an interorganizational level through the inclusion of evaluation findings in “horizontal” program funding requests submitted to the Treasury Board Secretariat by partnering departments and agencies. Horizontal initiatives refer to programs developed and shared by two or more departments and agencies through a focused partnership meant to address significant and broad-ranging issues for which multiple organizations have responsibility (e.g. human trafficking, gun violence, wildlife protection). Such initiatives require horizontal evaluations involving all partners to support high-level decision-making across multiple organizations. However, both of these presumed evaluation uses have yet to be investigated empirically.

Case 2: Community-based organizations in the province of Québec, Canada

The Province of Québec, Canada, is home to more than 8000 non-profit organizations providing health and social services as well as advocacy for social justice in their communities. The size, mandate, and structures of these organizations vary widely, but most are funded through private donations, philanthropic foundations, and/or government grants. Generally, these organizations are headed by a Board of Directors that can have varying degrees of influence over daily operations and decision-making. Organizations that receive funding from either philanthropic foundations or the provincial or federal governments are required to report on their outputs and outcomes on a cyclical basis. Such reports can include performance data, administrative data, and evaluations, depending on the funders’ specific requirements. It is not surprising, therefore, that most of these organizations perceive evaluative activities as an accountability mechanism that has little to no bearing on their own learning, improvement, or decision-making.

There has been increasing recognition of the potential usefulness of evaluation in providing organizations with information and data that goes beyond the strict needs of funders and focuses instead on internal informational needs. Organizations are more interested than ever in undertaking or commissioning evaluation work for their own purposes. However, they tend to lack the individual or organizational capacity to conduct and use evaluations. Since 2005, two major studies have been conducted to describe evaluation practices in the sector. The first study, by Hébert et al. (2005), showed that 95 per cent of community-based organizations were favorable to evaluation and that 80 per cent associated evaluation with learning and improvement, but lacked some of the resources and competencies to achieve this. The second study, by Tello-Rozas et al. (2022), sought to update the previous study. This study surveyed 404 community-based organizations to describe evaluation practices and activities at the organizational level. Seventy-three per cent indicated that they had conducted at least one evaluation over the past 10 years. Among these, 88 per cent indicated that evaluations are conducted internally, while 5 per cent indicated that evaluations are conducted entirely by an external expert. Forty-six per cent of the organizations that have conducted evaluations recently indicated that they received outside support and coaching during the evaluation process. The top reasons for conducting an evaluation were to “improve organizational activities” (89.4%), “taking the time to reflect on our practices” (84.1%), and “assess our outcomes” (82%). Tello-Rozas et al. (2022) also speak of an “evaluation ecosystem” in the sector, that includes both the community-based organizations as well as other key actors, such as evaluation professionals that provide coaching and support, universities that provide evaluation services as well as training, funders, and groups of organizations with common goals or mandates. In other words, several interorganizational networks and supports, including communities of practice, have been established to help non-profits develop their EC. Funders have also jumped on board by providing interorganizational training and toolkits.

One example of a strategy used to build EC across the entire sector is a project called “L’évaluation par et pour le communautaire, au service de la transformation sociale” (evaluation for and by community organizations, to support social transformation), developed and implemented by the Centre de formation populaire, an organization that supports community-based actors through training activities related to organizational development and management. In total, 324 organizations participated in this project through a series of training workshops, discussion groups, coaching and technical support, conferences, and online toolkits. After this first pilot project, the initiative was renamed “EvalPop” and was extended into a permanently funded initiative.

Organizational groups have also emerged in the past 15 or 20 years. These groups bring together smaller organizations that have shared goals or mandates, to share tools, perspectives, and resources. Such groups have enabled organizations, for instance, to borrow or use data collection instruments rather than developing their own, and therefore provide a new way to build organizational and sector-wide EC. The data collected by commonly used instruments, for instance, can then be rolled up to describe sectoral outputs and outcomes rather than just those obtained by each organization individually. The increasing complexity of the evaluation ecosystem in the community sector provides both opportunities for building interorganizational EC, through the concerted actions of various sectoral actors. The increased availability of support, training, and evaluation tools offered by third-party organizations or funders has led to an increase in EC across the community sector. However, Tello-Rozas et al. (2022) also emphasize that this has increased the number of conflicting evaluation priorities and approaches within the sector, which can lead to confusion within and between organizational actors.

Case 3: Quality improvement across the healthcare systems in Denmark

In recent years, the healthcare system in Denmark has experienced increased pressures related to managing chronic diseases found in an aging population. Elderly, frail and vulnerable patients with multiple concurrent illnesses experience poor quality of care due to a lack of coordination and cooperation between healthcare organizations. This problem can be defined as a complex problem (Rittel and Webber, 1973) that requires cross-sectoral thinking and joint action (Kettl, 2006).

The Danish healthcare system, like others internationally, has a long tradition of using Quality Improvement tools (Deming, 1986) such as the PDSA-circle to generate learning and improvement of care through iterative evaluation (Christie et al., 2017; Langley et al., 1996). In this particular case, senior managers at the University Hospital of Aalborg, cooperating municipalities, and general practitioners, have developed several cross-sectoral Quality Improvement projects, with an overall goal to improve the quality of care for frail and vulnerable patients. Five Quality Improvement Communities of Practice (QICP), each engaging 25 stakeholders (doctors, nurses, social and healthcare workers, general practitioners, and managers), were tasked to design and implement a quality improvement project to resolve a joint interorganizational problem. This involved collecting and analyzing interorganizational data as well as continuously evaluating the outcomes of improvement efforts undertaken by each QICP.

Each project took place over a period of 18 months. During this time, approximately 12 project meetings and 3 full-day seminars were held by each QICP. The seminars focused on training stakeholders to use the PDSA-circle (Langley et al., 1996) as well as conducting program evaluations using formative and collaborative evaluation approaches (Shulha et al., 2016). The evaluation tools (e.g. data collection planning tools, theory of change templates, statistical analysis tools) used in these sessions were developed through interorganizational collaboration by each QICP and were aligned with standard Quality Improvement templates.

Each QICP started by identifying a joint interorganizational problem through collaborative data collection, analysis, and reflection. Many QICP teams had difficulty accessing relevant data pertaining to interorganizational issues as well as agreeing on an issue to examine further through evaluation. Collaboration, use of templates and joint iterative evaluations of selected improvement initiatives, helped QICP members develop new knowledge and competencies related to data collection, to conducting small evaluations and to apply evaluation findings within the context of quality improvement. At the organizational level, the evaluation findings were used to inform decision-making regarding interorganizational cooperation, joint financial investments and new organizational structures, such a cross-sectoral geriatric function and a mobile laboratory for blood sampling. At the interorganizational level, the quality improvement projects resulted in new inter-sectoral evaluation procedures and policies, such as new methods and tools for assessing patient pathways through the health system and new participatory evaluation approaches that include decision-makers from multiple organizations. These procedures and policies continue to be used and have been further developed in the 3 years since their initial implementation.

Case 4: Collaborative evaluation in the Region of North Jutland, Denmark

Danish hospital emergency departments have experienced a significant increase in volume in recent years. The number of unique patient procedures in the North Jutland region’s emergency departments in Denmark increased significantly over the summer of 2021 and continued to increase thereafter. Overall, unique patient procedures increased by 14 per cent between 2019 and 2021. The number of patients seen for an acute outpatient assessment (without hospitalization) also increased by 19 percent during this period. Due to this increasing pressure, the Regional Council decided to investigate whether the “right patient is in the right place at the right time,” completing multiple patient inventories (Hermansen et al., 2020) and using collaborative evaluation approaches (Shulha et al., 2016).

In Denmark, the healthcare system is divided into the following three sectors: Two public (hospital and municipality) and one private sector (General Practitioners). In this case, stakeholders from 11 municipalities, three regional hospitals, and one university hospital, as well as general practitioners representing the entire region, were brought together to design new templates for patient inventory and for assessing the impact of cross-sectoral care and treatment. In total, 80 stakeholders were involved in this evaluative activity (including managers, nurses, and physicians), representing a total of 33 different organizations. Twelve 3-hour workshops were held over a 6-month period, supported by evaluators from various healthcare organizations. Through these workshops, the stakeholders assessed a total of 120 real-life complex patient histories, with a focus on performance data, patient management processes, quality of care and treatment before, during, and after a hospitalization, as well as the impact of the care and treatment provided. These evaluations focused on the merit and worth of health interventions across the Danish health system, and recommended new collaborative approaches to reduce the pressure on emergency departments.

The evaluation findings showed that 66 per cent of the interventions by emergency departments and 68 per cent of hospital admissions contributed positively to each patient’s condition. As a result of the evaluations, the stakeholders identified eight major organizational and interorganizational issues related to poor quality of care, lack in continuity of services, and unnecessary hospitalizations. In addition to the new collaborative relationships and increased leadership engagement in cross-sector collaboration across the healthcare sectors, a new political cross-sectoral focus was developed to better support older adults. This new focus has led to new local initiatives in the primary healthcare sector that have improved existing practices such as early detection of incipient disease in the municipalities and prevention of hospitalizations for the most vulnerable patients.

Cross-case analysis

The cross-case analysis provided us with a broader view of interorganizational EC. An important finding generated through this analysis is that all cases did not define interorganizational evaluation in the same way, leading to a very different scope in each case. For instance, cases 1 and 2, generated within a Canadian context, are much broader in nature and focus on entire sectors of organizations that conduct evaluations, either individually or collectively. Cases 3 and 4 are more specific in nature and focus on concrete assessment projects meant to bring together practitioners working within a broad sector of the Danish healthcare system. These projects sought to resolve a specific issue through an empirical process that included both traditional evaluation practices as well as other data collection and analysis approaches not typically found in program evaluation. Although the four cases are quite different from one another, both in terms of their focus (organizational type) and how EC is defined (interorganizational ECB vs interorganizational evaluation activities), several key trends can be observed between and within cases. The conceptual framework by Cousins et al. (2014a) was used as a starting point for the cross-case analysis and provides a solid foundation on which to rest our findings. Table 1 below provides an overview of the key findings for each case as well as the combined findings for all four cases.

Table 1.

Case elements by conceptual framework components.

Components of conceptual framework	Case 1: Canadian federal government	Case 2: Community-based organizations in Quebec	Case 3: Quality improvement across healthcare system in Denmark	Case 4: Collaborative evaluation in the Region of North Jutland, Denmark	Findings
Sources of knowledge, skills, and abilities	Professional evaluators who are competent in all aspects of evaluationFormal and informal training provided through professional associations, universities, and contractors	Evaluations are conducted by internal staff members with little to no trainingWorkshops and tools have been developed by funders and other organizations to support the development of evaluation-related knowledge and skills	Participants are managers and clinicians with little to no training in evaluationThree-day workshop and tools are partly developed in an interorganizational collaboration, partly aligned with standard quality improvement templates	Participants are clinicians and managers with clinical knowledge but no formal evaluation capacity/skillsWorkshop, tools, and evaluation templates developed in collaboration with the stakeholders	In all four cases, there is a training component.Practitioners learn about evaluation in cases 2, 3, and 4 through workshops and use tools and evaluation templates developed for this purpose
Organizational support structures	A government-wide policy on results mandates evaluation activities in each department and agencyFinancial and human resources are allocated to evaluation functions in the organizations	Support structures are interorganizational and provide evaluation resources to organizationsOrganizations allocate staff time and internal resources to evaluation activities when possible	Guided support from regional evaluatorsEvaluation toolkits/templates that can be used by participantsStaff time of participants	Guided support from regional evaluatorsEvaluation toolkits/templates that can be used by participantsStaff time of participants	In all four cases, financial resources are allocated to the evaluation projects or ECB initiatives through salary dollarsIn all four cases, there are interorganizational support structures that facilitate evaluation and learning activities
Evaluation capacity to DO evaluation	Professional evaluators have already typically been trained and continue to learn as they acquire more evaluation experienceHorizontal evaluations enable interorganizational ECB	Participation in network-wide evaluation training opportunities result in increased evaluation activities with external support and coaching	Participation in project group meetings to work on evaluations with external support and coaching	Participation in project group meetings to work on evaluations with external support and coaching	Interorganizational evaluation activities were observed in cases 1, 3, and 4.In case 2, organizational members do not engage in interorganizational evaluation but may participate in network learning activities
Evaluative Inquiry	Ecosystem of evaluation includes central agency policy that drives evaluation in most organizations	Funders and other organizations provide both demand and support for evaluation activities within the network	Ecosystem of evaluation includes quality improvement initiatives and interest in learning	Ecosystem of evaluation includes collaborative and utilization-focused evaluations and interest in learning	Each case shows its own ecosystem of evaluation and embedded supports.In all four cases, the organizations sit within broader networks of linked organizations. In cases 3 and 4, interorganizational evaluation activities were intentional
Evaluation capacity to USE evaluation	Use occurs at the meso and macro levels: evaluations target both senior decision-makers as well as program officials	Use occurs at the micro level: evaluation results produced for funders are not used internally but evaluation results produced by the organizations for their own needs are highly relevant and used in an instrumental fashion	Use occurs at the meso and micro levels: findings are used by municipal senior managers; and the findings are also used by individual stakeholders	Use occurs at the meso and micro levels: findings are used by municipal senior managers; and the findings are also used by individual stakeholders	Only case 1 includes use at the macro level, probably because of the type of organizations. Micro and meso levels of use occur in most cases.
Organizational learning capacity	Knowledge transfer from evaluations happens through formal reporting structures	Knowledge transfer from evaluations happens within each organization but not sectorally	Knowledge transfer at a micro-level happens through internal communication and process use in each organizationKnowledge transfer on an organizational level depends on the absorptive capacity of the organizations and their ability to use the findings	Knowledge transfer from evaluations happens through formal structures at the political levelKnowledge transfer at a micro-level happens through internal communication and process use in each organization	There is evidence of knowledge transfer from evaluations in all four cases, but the level of use (micro, meso, macro) influences learningInterorganizational learning from evaluation depends on the absorptive capacity of each participating organization
Organizational consequences	Interorganizational learning happens when heads of evaluation meet and in horizontal evaluations	Organizational representatives benefit from sharing their practices with othersFunders benefit from more knowledgeable organizations in terms of evaluations, can better meet their informational needs	Project groups benefit from working together on common quality improvement initiatives and learn about organizational interdependencies	Interorganizational learning happens when new knowledge is generated by evaluations	Signs of instrumental use in all four casesBringing people together from different organizations increases learning, whether it is learning about evaluation or learning from evaluation
Mediating conditions	Political landscape influences how and whether organizations work togetherHorizontal evaluation depends on leadership, collaboration, resources, and so on.	No current coordination of evaluation activities across sector despite availability of shared ECB opportunities	Political landscape influences how and whether organizations work togetherFeedback structures between the project groups and managersConflicts between private and public organizations in terms of absorptive capacity, resources, and management structures	Political landscape influences whether evaluation findings are usedConflicts between private and public organizations in terms of absorptive capacity, resources and management structures	Political landscape has an important influence on whether interorganizational evaluations occur as shown in cases 1, 3, and 4.Interorganizational ECB occurs in case 2 but is not subject to political influence.

Note: ECB = evaluation capacity building.

Sources of knowledge, skills, and abilities

Given the nested structure of individual and organizational ECB, interorganizational ECB also begins with the development of evaluative knowledge, skills, and abilities. In all four cases, training (direct ECB) is provided to evaluators or practitioners to support their development; for instance, federal evaluators (case 1) usually have knowledge and skills in evaluation or research methods when they are first hired, acquired through university courses or other opportunities. This training is supplemented over time through professional development courses and participation in evaluation-related conferences and webinars. In cases 2, 3, and 4, practitioners are the focus of direct ECB activities, rather than professional evaluators. In addition to evaluation workshops taught by evaluators (cases 3 and 4) and community developers (case 2), practitioners also have access to various evaluation templates and tools (indirect ECB) that guide them through an evaluation or assessment process. In cases 3 and 4, these are developed in collaboration with stakeholders to ensure greater uptake and ease of use.

Organizational support structures

In all four cases, financial resources were allocated to the evaluation projects or ECB initiatives through salary dollars. In case 1, human resources are explicitly assigned to an evaluation unit in each federal department or agency. In cases 2, 3, and 4, practitioners are provided with time to work on evaluation activities. In all of the cases, this typically represents a significant expenditure for the organizations involved and highlights the importance of developing EC at an individual and organizational level. Other significant support structures found in all cases include guidance and coaching from professional evaluators or third parties, templates and toolkits that practitioners can use in current or future interorganizational evaluations, and specific opportunities to share practices and lessons learned with practitioners from other organizations.

Evaluation capacity to DO evaluation

In cases 1, 3, and 4, the availability of interorganizational evaluation projects provide evaluators and practitioners with “real-life” evaluation learning and experience. In case 2, interorganizational evaluation does not tend to happen, given the limited mandates and resources of small community-based organizations; however, opportunities to learn about evaluation alongside practitioners from other organizations are commonplace and provide the context within which broader interorganizational sharing can take place. Therefore, individual EC may vary considerably between cases, depending on the availability of resources and structures within which either organizational or interorganizational evaluation can take place. In addition to this, we recognize that EC to DO can be developed over time at the individual and organizational levels, but that often, its sustainability is difficult to achieve due to staff turnover and fluctuating organizational priorities. Cases 3 and 4 also show that the development of individual soft skills (e.g. openly reflecting on issues faced by other stakeholders, handling potential conflicts, working with stakeholders from other organizations or groups) is critical to interorganizational EC.

Evaluative inquiry

Given our interorganizational focus, we chose to define evaluative inquiry as the broader ecosystem within which evaluation takes place and informs decision-making in each sector covered by the cases. Indeed, each case appears to be well-situated within its own ecosystem of evaluation and embedded supports. For instance, in case 1, the ecosystem of evaluation found in the Canadian federal government includes a central agency policy that drives evaluation in most of its organizations, as well as a cadre of qualified evaluation professionals that may move from one organization to another over the course of their careers, thus building their own individual capacity and sharing lessons learned with others. Here, the ecosystem defines the parameters of evaluations (through central policy) and typically requires utilization-focused evaluations that sometimes include theory-based approaches. In case 2, the ecosystem of evaluation in the community sector is made up of several different organizational actors, including the community organizations, philanthropic foundations, and government funders that provide grants and operating funds and require evaluations as well as universities and third-party organizations that provide evaluation services, training, and capacity building. The interaction between these actors creates both a demand for evaluation and support for evaluation activities throughout the network, even though evaluations tend to be conducted on an organizational level. In case 3, the ecosystem includes a paradigm shift from control and performance management to learning-oriented assessments, which creates the needed conditions for participatory and collaborative interorganizational evaluation approaches. Finally, case 4 highlights a strong focus from all actors in the ecosystem on the learning function of evaluation, which is achieved through collaborative and utilization-focused assessments that involve clinicians and managers. A key takeaway on evaluative inquiry is that in all four cases, evaluative practice is centered around utilization-focused approaches to support decision-making both within individual organizations and in their broader interorganizational networks.

Evaluation capacity to USE evaluation

Collectively, the four cases show evidence of capacity to use evaluation, but at different levels and in various ways. In all cases, evaluation findings are used by individual stakeholders such as program managers or senior decision-makers, to guide improvement at an organizational level. For example, in cases 3 and 4, assessment findings are used by participating stakeholders and managers: in case 3, the findings are used to adjust quality improvement initiatives and orient them toward desired outcomes. In case 4, the knowledge generated through the assessments is used to make practice-oriented changes at the organizational level. In both of these cases, assessment findings are also used to inform practices and decisions at an interorganizational management level; for example, in case 3, decision-makers across organizations agree to fund a geriatric team that includes employees from both the hospital and the municipality. This example illustrates how collaborative interorganizational assessment and evaluation activities can support the identification of solutions to common and complex problems (Beeby and Booth, 2000). In both cases 3 and 4, instrumental findings as well as the interorganizational ECB process have led to changes in procedures and policies related to interorganizational evaluation practice, using collaborative evaluation approaches.

Organizational learning capacity

This refers to the overarching capacity of the case organizations to learn and adapt to changing environments. Our analysis shows that organizational learning capacity can be influenced by leadership, organizational mission, vision and goals for engaging in or doing evaluation, absorptive capacity of evaluation findings, and organizational culture.

In cases 2, 3, and 4, we found that leadership support enabled organizational learning processes through the investment of salary resources and clear messaging regarding the importance and usefulness of evaluation to decision-making. The same three cases provide specific venues to encourage learning about evaluation (Beeby and Booth, 2000), either through communities of practice engaged in evaluation training activities or in project groups working on evaluative tasks together. All three cases bring together stakeholders from multiple organizations, and in cases 3 and 4, there is an expectation that the collaborative assessment and evaluation work will serve the purposes of not only individual organizations, but also interorganizational networks (Cohen and Levinthal, 1990; Lane and Lubatkin, 1998; Peronard and Brix, 2019; Van Winkelen, 2010). Thus, the absorptive capacity of the organizations involved in cases 3 and 4 also had an important influence on their ability to integrate, share, and use evaluation findings.

Organizational consequences

Bringing people together from different organizations and engaging them in collaboration, dialogue, and evaluation practice increases learning, whether it is learning about evaluation or learning from evaluation. In all four cases, there is evidence of learning and knowledge transfer within and across organizations.

In cases 2, 3, and 4, we found evidence of interorganizational sharing and learning, through which organizations receive knowledge from other organizations, or create new knowledge or practices through interaction with other organizations (Brix, 2021; Brix et al., 2021; Larsson et al., 1998; Peronard and Brix, 2019; Van Winkelen, 2010). In case 2, organizational representatives participated in joint evaluation training sessions, while in case 3, project groups learned about quality improvement principles and processes as a team, and how quality improvement in one organization or sector can influence outcomes in other organizations or sectors. In addition to this, transfer of knowledge from the stakeholder level to decision-makers, within and across organizations, appears to lead to both single-loop learning (e.g. adjustment of agreed workflows in the interorganizational collaboration) and double-loop learning (e.g. when the organizations question and change their fundamental assumptions about their collaboration and common goals).

In case 4, interorganizational learning occurred when new knowledge was generated through the patient assessments. This included new knowledge related to trends in complex and interorganizational problems unknown to the separate/individual organizations.

Mediating conditions

The political context within which organizations operate also influences organizational EC (Al Hudib and Cousins, 2022). This takes on even greater importance in an interorganizational environment, given that political dispositions can vary between sectors. In all four cases, we found that the political landscape influences whether interorganizational ECB occurs successfully. In case 1, government priorities can influence how and whether federal organizations work together on joint horizontal evaluations as well as directing government-wide evaluation policy. In cases 3 and 4, political values and priorities on a national, regional, and municipal level influence both whether the organizations invest stakeholder resources in joint evaluation activities and if decision-makers value and use evaluation findings.

Discussion

This study sought to explore interorganizational EC as an extension of existing individual and organizational ECB models and frameworks. Given the increasing complexity of our world and the open nature of most governmental, social, and healthcare organizations, interorganizational ECB provides an opportunity to develop meaningful, empirical evidence that can be used to inform decision-making by multiple organizational actors within broad sectors or systems. This study sought to examine various aspects of interorganizational ECB to illustrate how an expanded notion of more traditional EC dimensions might enrich our understanding of both EC and evaluation utilization. Our case narratives, as well as our cross-case analysis, have identified various trends that are worthy of consideration as building blocks of interorganizational EC and as potential lessons learned for future efforts. First, developing the evaluation competencies of practitioners and decision-makers through interorganizational training (direct ECB) or evaluative inquiry (indirect ECB) creates a unique context for conversations focused on interorganizational issues or problems that can be explored through empirical work. The collaborative aspect of interorganizational ECB was also found to positively influence the development of soft skills related to negotiation and conflict resolution by practitioners. These dimensions can potentially enable organizations to create better solutions to interorganizational problems than what they can achieve on their own. Second, financial resources and other support structures are a necessary element of interorganizational ECB. This is also true at the individual and organizational levels, but without a firm investment from the organizations involved, practitioners will not be able to carve out time to participate in interorganizational evaluation activities, regardless of sector. Third, the notion of evaluation ecosystem was salient in both the case narratives and the cross-case analysis. Evaluation practice and capacity building can only occur in environments that are conducive to learning, and this requires the participation of multiple organizational actors. Fourth, as with other forms of ECB, evaluation use is a critical outcome that often requires the development of reports and presentations suited to decision-making or program improvement. Finally, mediating conditions affect both how interorganizational ECB is conducted as well as its focus. Organizational and interorganizational politics are a frequent source of conflict during interorganizational ECB initiatives and should be taken into account during the design of the ECB intervention and its evaluation.

Beyond the instrumental findings of our exploratory study, other observations may be interesting to consider in future work. For instance, the use of the conceptual framework proposed by Cousins et al. (2014a) to guide the analysis was helpful in identifying key elements of EC or ECB that could be compared, regardless of the scope of each case. The framework is general enough to be adapted to various organizational or interorganizational settings, and its constituent elements provide some useful distinctions between key concepts. However, we had to adapt or expand the framework in some instances to better reflect the interorganizational nature of the four cases and to ensure the consideration of multiple organizational actors. In addition, we found that the cases reflected different aspects of interorganizational ECB; as mentioned previously, cases 1 and 2 describe sectoral evaluation systems in which organizations conduct evaluations, whereas cases 3 and 4 describe concrete ECB projects conducted by groups of stakeholders representing various organizations. These findings led us to conceptualize the interorganizational ECB cases on a continuum, where case 2 might be positioned at one end, with a heavy focus on interorganizational training and resources but not practice, and case 4 at the other end, with training, resources, and practice done in an interorganizational context, to support interorganizational decision-making. These adaptations and this conceptualization of interorganizational ECB will require the development of new, expanded conceptual frameworks that address broader dimensions found in the interorganizational sphere, such as social, relational, political, cultural, and community factors.

Given the exploratory nature of this study, we would certainly advocate for other empirical work to continue in this new interorganizational space for ECB. Key among the issues to be explored might be the following: How does interorganizational EC differ from other levels, in terms of its contextual factors and mechanisms? What does interorganizational EC look like across different sectors? How could it be measured? How might systems theory help us better understand interorganizational EC and ECB? How should interorganizational ECB interventions be evaluated? What are some of the sectoral or societal outcomes that may emerge from interorganizational evaluation practice? We believe that an empirical examination of these questions would yield important new insights, not only on interorganizational EC but also on some of the factors and conditions that also influence individual and organizational EC.

Conclusion

This study sought to document and examine interorganizational EC initiatives undertaken in various sectors in Canada and Denmark. We drew on existing secondary data to build four case descriptions, which were subsequently analyzed based on a conceptual framework by Cousins et al. (2014a). We found important variability in terms of the tools, mechanisms and supports used to build interorganizational capacity, in addition to fundamental differences between cases regarding what can be defined as EC. The first two cases focus more directly on building individual and organizational EC through policy development and training offered at a sectoral level, while the last two cases focus more explicitly on building individual EC through indirect ECB activities involving stakeholders from different organizations in evaluative work. We conclude that interorganizational ECB depends heavily on individual and organizational EC but transcends these two levels when stakeholders have an opportunity to learn from one another and work together on resolving evaluative and interorganizational challenges.

Supplemental Material

sj-docx-1-evi-10.1177_13563890231207115 – Supplemental material for Interorganizational evaluation capacity building in the public, health and community sectors

Supplemental material, sj-docx-1-evi-10.1177_13563890231207115 for Interorganizational evaluation capacity building in the public, health and community sectors by Charlotte Laubek and Isabelle Bourgeois in Evaluation

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The author(s) received no financial support for the research,authorship,and/or publication of this article.

ORCID iD

Isabelle Bourgeois

Charlotte Laubek is currently completing her PhD at the Institut for Kultur og Laering—Capacity Building and Evaluation at Aalborg University. Her thesis focuses on interorganizational evaluation capacity within the healthcare sector.

Isabelle Bourgeois is a Full Professor at the Faculty of Education,University of Ottawa. Her research focuses on organizational evaluation capacity building in the public and community sectors.

References

Al Hudib

Cousins

(2022) Understanding evaluation policy and organizational capacity for evaluation: An interview study. American Journal of Evaluation 43(2): 234–54.

Beeby

Booth

(2000) Networks and interorganizational learning: A critical review. The Learning Organization 7(2): 75–88.

Bourgeois

Buetti

(2019) Using action research to build evaluation capacity in Ontario: Lessons learned. Journal of Multidisciplinary Evaluation 33: 81–90.

Bourgeois

Cousins

(2013) Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation 34(3): 299–319.

Bourgeois

Leblanc

(2019) L’évaluation de programmes au gouvernement du Canada: Utilisation et capacités organisationnelles. Administration Publique Canadienne 62(2): 294–311.

Bourgeois

Lemire

Fierro

, et al. (2023) Laying a solid foundation for the next generation of evaluation capacity building: Findings from an integrative review. American Journal of Evaluation 44(1): 29–49. https://doi.org/10.1177/10982140221106991

Bourgeois

Maltais

(2023) Translating evaluation policy into practice in Government organizations. American Journal of Evaluation 44(3): 353–373. https://doi.org/10.1177/10982140221079837

Bourgeois

Toews

Whynot

, et al. (2013) Measuring organizational evaluation capacity in the Canadian federal government. Canadian Journal of Program Evaluation 28(2): 1–19.

Braithwaite

Churruca

Clay-Williams

, et al. (2017) Complexity science in healthcare—Aspirations, approaches, applications and accomplishments: A White paper. Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, Australia.

10.

Brix

(2021) Strategisk innovationsledelse Om balancen mellem drift og udvikling i offentlige og private organisationer, 1st edn. Copenhagen: Hans Reitzels Forlag.

11.

Brix

Tuurnas

Mortensen

(2021) Creating opportunity spaces for co-production: Professional co-producers in interorganizational collaborations. In: Thomassen

Jensen

(eds) Processual Perspectives on the Co-Production Turn in Public Sector Organizations. Hershey, PA: IGI Global, pp. 157–75.

12.

Chirau

Blaser-Mapitsa

Amisi

(2021) Policies for evidence: A comparative analysis of Africa’s national evaluation policy landscape. Evidence & Policy 17(3): 535–48.

13.

Christie

Fierro

(2012) Evaluation policy to implementation: An evaluation of scientifically based research in practice. Studies in Educational Evaluation 38(2): 65–72.

14.

Christie

Lemire

Inkelas

(2017) Understanding the similarities and distinctions between improvement science and evaluation. New Directions for Evaluation 153: 11–21.

15.

Christie

Lemire

(2019) Why evaluation theory should be used to inform evaluation policy. The American Journal of Evaluation 40(4): 490–508.

16.

Cohen

Levinthal

(1990) Absorptive capacity: A new perspective on learning and innovation. Administrative Science Quarterly 35(1): 128–52.

17.

Compton

Baizerman

(2012) The evolution of a philosophy and practice of evaluation advice. New Directions for Evaluation 136: 67–76.

18.

Cousins

Goh

Elliott

, et al. (2014a) Framing the capacity to do and use evaluation. Organizational Capacity to Do and Use Evaluation. New Directions for Evaluation, no 141, Cousins

J.B.

Bourgeois

(Ed.). San Francisco: Jossey-Bass.

19.

Cousins

Goh

Elliott

, et al. (2014b) Government and voluntary sector differences in organizational capacity to do and use evaluation. Evaluation and Program Planning 44: 1–13.

20.

Deming

(1986) Out of the Crisis. Cambridge, MA: MIT Center for Advanced Engineering Study.

21.

Dillman

Christie

(2017) Evaluation policy in a nonprofit foundation: A case study exploration of the Robert Wood Johnson Foundation. American Journal of Evaluation 38(1): 60–79.

22.

Dinh

Keys

Thomson

(2015) Finding the common thread: The charms and challenges of evaluation for a consortium-run international development program. Evaluation Journal of Australasia 15(2): 4–11.

23.

Goldman

Byamugisha

Gounou

, et al. (2018) The emergence of government evaluation systems in Africa: The case of Benin, Uganda and South Africa. African Evaluation Journal 6(1): a253.

24.

Goodyear

(2011) Building a community of evaluation practice within a multisite pro-gram. New Directions for Evaluation 129: 97–105.

25.

Hébert

Fortin

Fournier

, et al. (2005) Analyse des pratiques d’évaluation dans les organismes communautaires. Rapport de recherche, p. 135. Available at: https://www.clo-ocol.gc.ca/sites/default/files/RapportARPEOC_analyse.pdf

26.

Hermansen

Holmskov

Johnsen

, et al. (2020) Quality in practice: Applying the patient inventory method at a Danish psychiatric hospital. International Journal for Quality in Health Care 32(7): 477–79.

27.

Inkleas

Bowie

Guirguis

(2017) Improvement for a community population: The Magnolia community initiative. New Directions for Evaluation 153: 51–64.

28.

Keast

Brown

Mandell

(2007) Getting the right mix: Unpacking integration meanings and strategies. International Public Management Journal 10(1): 9–33.

29.

Kettl

(2006) Managing boundaries in American administration. Public Administration Review 66: 10–19.

30.

Labin

Duffy

Meyers

, et al. (2012) A research synthesis of the evaluation capacity building literature. American Journal of Evaluation 33(3): 307–38.

31.

Lane

Lubatkin

(1998) Relative absorptive capacity and interorganizational learning. Strategic Management Journal 19(5): 461–77.

32.

Langley

Nolan

(1996) The Improvement Guide. San Francisco, CA: Jossey-Bass Publishers.

33.

Larsson

Bengtsson

Henriksson

, et al. (1998) The interorganizational learning dilemma: Collective knowledge development in strategic alliances. Organization Science 9(3): 285–305.

34.

Mark

Cooksy

Trochim

WMK

(2009) Evaluation policy: An introduction and overview. New Directions for Evaluation 123: 3–11.

35.

Mayne

(2020) Building evaluative culture in community services: Caring for evidence. Evaluation and Program Planning 80: 101450.

36.

Nielsen

Lemire

Skov

(2011) Measuring evaluation capacity—Results and implications of a Danish study. American Journal of Evaluation 32(3): 324–44.

37.

Patton

(2012) Essentials of Utilization-Focused Evaluation. Thousand Oaks, CA: SAGE.

38.

Peronard

J-P

Brix

(2019) Organizing for interorganizational learning in service networks. The Learning Organization 26(3): 276–88.

39.

Preskill

Boyle

(2008) A multidisciplinary model of evaluation capacity building. American Journal of Evaluation 29(4): 443–59.

40.

Rittel

HWJ

Webber

(1973) Dilemmas in a general theory of planning. Policy Sciences 4(2): 155–69.

41.

Shulha

Whitmore

Cousins

, et al. (2016) Introducing evidence-based principles to guide collaborative approaches to evaluation: Results of an empirical process. American Journal of Evaluation 37(2): 193–215.

42.

Stockdill

Baizerman

Compton

(2002) Toward a definition of the ECB process: A conversation with ECB literature. New Directions for Evaluation 93: 7–26.

43.

Taylor-Ritzler

Suarez-Balcazar

Garcia-Iriarte

, et al. (2013) Understanding and measuring evaluation capacity: A model and instrument validation study. American Journal of Evaluation 34(2): 190–206.

44.

Tello-Rozas

Léonard

Lussier-Lejeune

(2022) Mise à jour du portrait des pratiques d’évaluation dans les organismes communautaires. Centre de formation populaire, Réseau québécois de l’action communautaire autonome, Relais-femmes, Service aux collectivités de l’Université du Québec à Montréal.

45.

Trochim

WMK

(2009) Evaluation policy and evaluation practice. New Directions for Evaluation 123: 13–32.

46.

Van Winkelen

(2010) Deriving value from inter-organizational learning collaborations. The Learning Organization 17(1): 8–23.

47.

Volkov

(2008) A bumpy journey to evaluation capacity: A case study of evaluation capacity building in a private foundation. Canadian Journal of Program Evaluation 23(3): 175–97.

48.

Yin

(2018) Case Study Research and Applications. Design and Methods, 6th edn. Thousand Oaks, CA: SAGE.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.01 MB

0.00 MB