Abstract
The relation between advocacy and effectiveness is increasingly assumed to be important by nongovernmental organizations (NGOs) and funding agencies (Almog-Bar & Schmid, 2013; Hudson, 2002; Netherlands Ministry of Foreign Affairs [NL MFA], 2014; Teles & Schmitt, 2011; Wong, 2012). In this context, there is a politics of results that defines effectiveness mostly in terms of assessing achieved outcomes against planned objectives (Eyben, 2015; Organization for Economic Co-Operation and Development, 2008, 2011), resulting in a narrow focus on outcomes that are measurable, visible, and tangible within the time frame of funding agreements (Eyben, 2015; Riddell, 2014; Vähämäki, Schmidt, & Molander, 2011). This involves an interaction between demands (political pressure to show results for accountability reasons) and needs (political pressure for legitimacy and credibility in demonstrating outcomes). However, measuring effectiveness is incredibly difficult with regard to advocacy (Beer & Reed, 2009; Devlin-Foltz, Fagen, Reed, Medina, & Neiger, 2012). Practitioners and evaluators working with advocacy continuously struggle to come to terms with the demand for demonstrable effectiveness and with the complexity of evaluating advocacy. This makes research on advocacy evaluation ever more important as a field of study.
The complexity of advocacy is widely recognized and discussed (Beer & Reed, 2009; Coffman, 2009; Devlin-Foltz et al., 2012; Gardner & Brindis, 2017; Wong, 2012). Advocacy is defined as a “wide range of activities conducted to influence decision makers at different levels” (Morariu & Brennan, 2009, p. 100; see also, Fagen, Reed, Kaye, & Jack, 2009). Advocates thus pursue outcomes as structural changes in social, political, and organizational systems, while challenging existing power structures (Hudson, 2001; Keck & Sikkink, 1998). 1 The literature in the field of advocacy evaluation focuses on effectiveness by stressing the nonlinear and political character of advocacy and its policy change successes (Jones, 2011; Keck & Sikkink, 1998; Teles & Schmitt, 2011; Wong, 2012). This impacts the existing evaluation methods, which do not currently unpack or deal with the problematic identification of advocacy outcomes (i.e., traceability) as such. Evaluations remain often results based, focusing heavily on outcomes as successful and visible (policy) changes, building on the assumption that outcomes self-evidently follow from actions. In this article, I question this assumption.
Drawing on ethnographic findings from a quest for advocacy outcomes as part of an evaluation in which I was involved as both a researcher and evaluator from 2012 to 2015, this article provides empirical insights in the quest for evaluating advocacy outcomes. Building on the many challenges advocacy evaluators are faced with, I demonstrate possibilities to identify outcomes as socially and politically constructed and discuss what this means for evaluating advocacy. The question that inspired this research is: How are advocacy outcomes constructed and identified and what does it mean for advocacy evaluation that advocacy outcomes are often intangible?
The recognition of the complexity of advocacy has resulted in important efforts to develop more suitable evaluation methods for complex interventions. Some scholars have pointed out the need to evaluate advocacy by focusing on the advocates’ skills and capacities to act strategically (Arensman, van Waegeningh, & van Wessel, 2017; Teles & Schmitt, 2011). Other studies stressed the need to focus on contribution rather than attribution to change, in assessing the role of advocacy programs in certain successful (policy) changes (Coffman, 2011; Gardner & Brindis, 2017; Mayne, 2008, 2012). Increasingly, scholars, practitioners, and evaluators promote working with a theory of change to increase the flexibility of planning, strategizing, and assessing advocacy programs and other complex interventions (Beer & Reed, 2009; Gardner & Brindis, 2017; Klugman, 2011). This approach often includes looking at advocacy progress in terms of interim outcomes instead of major policy outcomes that are not always achieved (Coffman, 2011).
The article begins by discussing the current discussions on how to demonstrate effectiveness in the field of advocacy evaluation. Although the literature has widely recognized the challenges faced in evaluating advocacy and sought better ways for evaluation, the identification of outcomes remains understudied. Outcomes are generally considered central to the evaluation purpose of demonstrating effectiveness, and they are usually considered to be self-evident (tangible and following from action). The complexities and challenges of nonlinearity and unpredictability have not necessarily led to the development of methods appropriate for advocacy evaluation. In the next section, the multisited ethnographic approach and qualitative methodology are explained, setting out the scope of the case study, the data collection, and the analysis. After this, the empirical findings are demonstrated providing illustrations of the quest for advocacy outcomes and the problematic nature of identifying these outcomes. By empirically demonstrating the search for identifying advocacy outcomes, this article demonstrates that there is room for interpretation, as outcomes are often intangible, socially constructed, and political. In the final section, the findings are discussed drawing out the broader implications of these findings for theory and practice regarding advocacy evaluation.
Advocacy and Its Unique Outcomes
Coming to terms with challenges associated with demonstrating effectiveness is an ongoing struggle in advocacy evaluation practice and in research on advocacy evaluation. A crucial issue is the problematic nature of advocacy outcomes and their identification. Advocacy involves processes that are unpredictable and conducive to change in the highly complex sphere of public policymaking, with many actors and factors influencing the course of events (Grantcraft, 2012; Guthrie, Louie, David, & Foster, 2005; Reisman, Gienapp & Stachowiak, 2007; Roche & Kelly, 2012a, 2012b). Many forces are at play that make or break an advocacy effort, thus influencing advocacy effectiveness. A recent study by Arensman, van Waegeningh, and van Wessel (2017) demonstrates the recursive character of advocacy and the emergent nature of advocacy outcomes. Determining the effects of strategies ahead of time in the planning process is thus likely to be impossible (Coffman, 2009; Devlin-Foltz et al., 2012; Teles & Schmitt, 2011). This means that the determination of outcomes is not “set in stone,” challenging evaluators to search for other ways of demonstrating the effectiveness of advocacy (Jones, 2011; Teles & Schmitt, 2011). At the same time, NGOs face growing pressure to demonstrate their effectiveness (Eyben, 2015; Riddell, 2014).
In this complex world of public policymaking, outcomes are mostly invisible, unpredictable, and dependent on many actors and factors (Whelan, 2008). The relations between the advocacy efforts, progress toward outcomes, and actual changes are not easily established; these links are elusive and often indirect (Teles & Schmitt, 2011). Because they come about as a result of often intangible processes that are not observable or traceable, advocacy outcomes are both unique and problematic.
The advocacy evaluation literature discusses the problematic nature of outcomes in a number of ways. Some scholars stress the impossibility of establishing attribution of advocacy programs or advocates actions to certain changes. These studies emphasize the need to evaluate advocacy outcomes by looking instead at establishing contribution between advocates’ actions and the identified changes (Devlin-Foltz et al., 2012; Fagen et al., 2009; Jones, 2011). Another issue stressed is the importance of understanding different layers of outcomes including short-term or interim outcomes. Looking at interim outcomes rather than end goals is important because the end goals are usually abstract (e.g., the prevention of violent conflict) and often not achieved within a specified time frame (Coffman, 2009, 2011; Harvard Family Research Project, 2007). Specifically, interim outcomes are the smaller steps taken toward a bigger change. Types of interim outcomes include changing attitudes or beliefs, raising awareness, creating visibility, building relations, and creating space for civil society. These steps might not result directly in explicit changes in policies, laws, practices, or institutional structures, but they are nonetheless meaningful. Focusing on interim outcomes provides the space necessary to demonstrate changes, following and assessing progress toward larger pursued objectives (Devlin-Foltz et al., 2012; Harvard Family Research Project, 2007; Starling, 2010).
Evaluating Advocacy Outcomes
Although these problems in advocacy and evaluation studies are acknowledged in the field, the advocacy evaluation approaches that are currently widely implemented are not necessarily helpful for gaining a better understanding of advocacy outcomes or how they are constructed. Approaches that are most often mentioned in relation to advocacy evaluation are outcome mapping (OM) and outcome harvesting (OH; Gardner & Brindis, 2017). Moreover, both these methods are currently widely implemented for evaluating effectiveness in the NGO community, with the intent to do justice to complex processes such as advocacy (Gardner & Brindis, 2017; Lemon & Pinet, 2018; Jones & Hearn, 2009; Roche & Kelly, 2012a, 2012b; Tsui, Hearn, & Young, 2014; Tsui & Lucas, 2013; van Ongevalle & Peels, 2012; Wilson-Grau & Britt, 2012). Although also other methods are used and developed (see Gardner & Brindis, 2017, for an overview), the widespread use of OM and OH in the NGO and policy community specifically in relation to advocacy justifies a closer look at these two methods in relation to advocacy evaluation.
OM considers outcomes to be changes in behavior, relationships, networks, or the actions and activities of individuals, groups, and communities. To map outcomes, a high degree of cooperation with those evaluated is necessary, and a shared vision on what is being evaluated and mapped should be established (Tsui & Lucas, 2013, p. 7). Three stages are specified as important: (1) the identification of intention (how, why, who, what), outlining the programs’ desired change; (2) monitoring performance to identify progress toward outcomes and goals; and (3) evaluation planning to identify priorities. Earl, Carden, and Smutylo (2001) have stressed that OM deals with the challenge of attribution by focusing on the value and intention of results achieved: It does this by focusing on the changes that are clearly within a program’s sphere of influence. While, at first glance, this appears to suggest concentrating on easier, less important, short-term achievements, in fact, it does the opposite. It focuses attention on incremental, often subtle changes, without which the large-scale, more prominent achievements in human well-being cannot be attained or sustained. (p. 14)
In OH, outcome descriptions are specific (formulated in detail), measurable (independent and objectively verifiable quantitative and qualitative information), achieved (plausible relation and logical link between the outcome and the agent’s actions), relevant (the outcome presents a significant step towards the main goal), and timely (the outcome occurred within the evaluation period; Wilson-Grau & Britt, 2012, pp. 8–9). The evaluator is meant to “harvest” the verifiable connection between the initiative and the outcomes, creating outcome descriptions that answer the harvesters’ questions and revising these descriptions to make them more specific and comprehensive. The descriptions of the harvested outcomes are then validated “by comparing [them] with information collected from other knowledgeable and authoritative, but independent, sources.” 2 Finally, these outcomes are analyzed and interpreted in relation to their significance for achieving a certain mission, goal, or strategy. OH focuses on progress and behavior change. The method concentrates on collecting evidence of achievements rather than measuring progress toward predetermined outcomes. The evaluator determines the contribution to an outcome by collecting data from reports, documents, interviews, and other resources (Lemon & Pinet, 2018).
These methods are currently widely used and implemented in advocacy evaluations. However, these methods do not inspire to unpack outcome identification empirically. OM is embedded in the tradition of logic models and theory of change frameworks and clarifies the presumed logical intended relationships between objectives, predetermined outcomes, activities, and achievements (Smutylo, 2005). OH is an evaluation approach that does not measure progress but rather works backward, collecting evidence on the outcomes achieved (Wilson-Grau & Britt, 2012). Both approaches are considered relevant for evaluating advocacy, while they both build on the assumption that outcomes can be predicted and measured, or at least that they are self-evidently following from action. The further development of these methods would benefit from a better understanding of how advocacy outcomes are constructed.
Case Studies, Data Collection, and Analysis
Advocacy for Development Evaluation
This study is part of a broader research program conducted from 2012 to 2015 focusing on an evaluation of international advocacy programs. The programs were financed by the second Dutch cofinancing system (2010–2015) known as MFS II. 3 The evaluation was commissioned by NL MFA, in cooperation with the Foundation for Joint Evaluations and administered by the Netherlands Organization for Scientific Research. I was involved as evaluator in a team of 10 evaluators and as a researcher conducting my PhD research.
The evaluation assessed eight advocacy programs funded by the NL MFA and implemented by a variety of alliances of Dutch NGOs in cooperation with partners worldwide. For this article, I draw on three advocacy programs that I was closely involved in evaluating. For confidentiality reasons, I anonymized the case studies and the interviewees. The three programs were three different types of advocacy cases: an NGO working on child rights at the Pan-African level, an international NGO working on women’s rights at local and global levels, and a transnational advocacy network on conflict prevention and peacebuilding. In all cases, the advocacy was conducted in cooperation with global, national, and regional partners.
The main purposes of the evaluation were to account for the results of the funded advocacy programs, to contribute to the improvement of future development interventions, and to develop advocacy evaluation methodology. The evaluation team was tasked with addressing evaluation questions that were predefined by the donor. These pertained to the outcomes achieved, contribution, relevance and efficiency of the programs, as well explaining the actors and factors involved in the evaluation outcomes. The methods implemented in the evaluation to answer these questions were used in triangulation: theory of change, OM and harvesting, and process tracing. The team was instructed to answer these questions looking specifically at three predefined priority result areas: agenda setting, policy influencing, and changing practice. These result areas supposedly mirrored an expected sequence of outcome areas (not necessarily in a linear fashion) of the advocacy programs.
Data Collection and Analysis
My research was a multisited ethnographic study (see Marcus, 1995) of the three above-mentioned advocacy programs and of the evaluation process itself. Marcus (1995) has highlighted that multi-sited research is designed around chains, paths, threads, conjunctions, or juxtapositions of locations in which the ethnographer establishes some form of literal, physical presence, with an explicit, posited logic of association or connection among sites that in fact defines the argument of the ethnography. (p. 105)
I studied the advocacy programs following the advocates through time (2012–2015) and space (at local, regional, national, and global levels), using multisited ethnographic and qualitative research methods. As part of the evaluation team, I collected data on the three programs discussed in this article by studying both public and internal strategic and policy documents and via semistructured interviews using stakeholder analysis (see Sloot & Gaanderse, 2010) and snowballing technique (see Flick, 2009, p. 109) to pinpoint the important interviewees. I also collected data through participant observation of internal strategic, team or board meetings, lobby meetings, public events, meetings with all stakeholders in the MFS II evaluation (in the Netherlands, Ethiopia, the Philippines, New York, and Ghana), and feedback meetings with the advocacy programs’ staff. This meant that I collected data at 29 meetings relevant to the advocacy programs, 3 meetings relevant to the evaluation process which included all stakeholders, and 15 internal evaluation team and management meetings, in which decisions were made regarding the evaluation direction, strategies, and methods.
My co-evaluators and I conducted semistructured interviews in the three studied programs in Kenya, Mozambique, Uganda, Namibia, South Africa, the Philippines, Ghana, and New York. For the NGO working on child rights at the Pan-African level, we conducted 88 interviews. For the international NGO working on women’s rights at local and global levels, we conducted 35 interviews. For the transnational advocacy network on conflict prevention and peacebuilding, we conducted 80 interviews. Interviewees were NGO representatives from the evaluated programs and other stakeholders such as their international and domestic partners, targets, and other relevant experts in the field (using both snowballing technique and stakeholder analysis). Semistructured interviews enable the researcher to address diverse topics, while allowing the interviewee to discuss issues that are relevant and important in their perspective so that the researcher can explore subjective viewpoints (see Bradford & Cullen, 2012; Flick, 2009). The programs and processes studied were traced triangulating between diverse sources (i.e., interviewees, documents) and qualitative and contribution analysis (see Mayne, 2012, 2008; Arensman et al., 2015). In interviews, I ensured to build on the information already gathered. Hence, substantiation was ensured by follow-up interviews, document study, and (in-) formal meetings with the teams.
The semistructured interview questions were geared toward unpacking the advocacy processes, the stakeholders involved, the meaning of their roles, the goals of the advocacy programs and the relation between these goals and the activities, output and outcomes, unpacking the steps in the process as well as what it resulted in. I reflected on and analyzed the processes focusing on the construction and reconstructions around the advocacy processes (using process tracing, OM, and OH) in the three diverse case studies. Analyzing which stakeholders were involved, in what roles. What steps were taken to what results and analyzing the meaning of these steps, relations, and results. The concept of outcome was an important and difficult construct that was given meaning in the processes studied and in the interviewees’ perceptions. Subsequently, the information was coded, finding overarching themes (amongst others e.g., networked advocacy, local ownership, effectiveness, the construct of outcome, results politics, practice vs. theory). These themes were subcoded into more specifically focused themes: types of outcomes, meaning of outcomes, whose outcomes, politics of outcomes, legitimacy, accountability, profiling, framing, visibility, politics of results, donor–NGO relation, claiming and nonclaiming, and underreporting and overreporting. The findings in the next section reflect this analysis.
In my position as evaluator and academic researcher, I was an insider (evaluator), an observer (evaluator and researcher), and a researcher. While I was working as an evaluator in the evaluation team in my capacity as an evaluator/researcher, I observed the processes and interactions and was able to reflect on them. Wearing different hats in this way gave me a unique opportunity as a researcher: I was able to work closely with the advocacy programs’ staff and to research the evaluation as a process. This enabled access to privileged information that allowed me to acquire useful insight, and the analysis is informed by my dual position as both evaluator and researcher.
The Quest for Advocacy Outcomes
The following section demonstrates a number of problematic dimensions in the quest for identifying advocacy outcomes by zooming in on three diverse advocacy programs in terms of type of organization, thematic and regional focus, and advocacy strategy. The findings of the advocacy evaluation revealed that outcomes have different meanings depending on the interpretation of the advocates and on their aims. Something considered an outcome for one person may not be considered an outcome for someone else. Outcomes become outcomes through the interpretation of the stakeholders. This is dependent on the interactions, interests, and contexts involved. Especially in advocacy, there is room for interpretation around outcomes because processes, outcomes (in terms of change achieved), and the relation between advocacy and the changes achieved are not obvious. To make this explicit, I discuss the cases of claiming and nonclaiming, underreporting and overreporting, whose outcomes and what outcomes, and visibility and profiling. These findings illustrate both the space for interpretation as to what an outcome is and how its meaning is used, as well as the politics involved.
Claiming and Nonclaiming
The evaluated child rights advocacy program was part of an organization with a diverse staff that was very committed to changing the lives of children throughout the African continent. Early on in the evaluation, we noted that the organization and staff had a very good reputation with their targets and partners throughout Africa; they were considered knowledgeable on the topic of African child rights. However, the outcomes identified in reports or claimed by staff members in interviews or meetings were limited. To sketch the context, the organization was situated in Ethiopia, close to the Pan-African political arena of the African Union (AU), and they worked closely together with AU, African governments, the United Nations (UN), and other civil society organizations and networks. To understand the context, in Ethiopia, advocacy is by law forbidden. Hence, this organization stressed their research and information-providing role rather than their advocacy and lobby role while working on Pan-African level. Our evaluation focused on advocacy and influencing, but this organization’s staff members did not see their work in this way. Because they were based in Ethiopia, they had to consider the country’s restrictions on NGOs’ roles (i.e., advocacy was not permitted) and funding (limitations to foreign funding).
The organization had built credibility from their years of doing research, planning events, working closely with diverse public and political stakeholders on the African continent, and advocating for the issues identified as gaps or problems in national and regional policies on child rights. The role of the organization was pointing out the issues they surfaced through their research to the stakeholders involved to raise awareness and devise solutions to existing problems. The organization could influence the awareness of their partners and targets by doing research to point out problems and by disseminating this research, translating it into diverse African languages and organizing workshops and conferences. However, the organization did not have control over what other stakeholders would do with the reports they produced, the gaps they identified, or the solutions they proposed.
The organization did not interpret processes or outreach over which they had no control as outcomes. These outcomes were therefore outside their view of direct control, and they had not thought of them as being relevant strategically. These outcomes were thus not identified or claimed. One program manager described their work as “a chain of action” in which their research was the basis for further influencing: Some is reactive and most of it is pro-active. Like […our] campaign. This started in January when we knew the department of social affairs was going to focus on [the issue of our campaign]. We are looking at this from a legal and rights perspective, because that is what the AU is looking at. We share research papers with the [inter-governmental department]. From the discussion based on the research we had done, we were asked to help draft a declaration […] and it was discussed and adopted (11 April 2014). We wrote the declaration, they asked us, but we cannot say that outside. (program manager, 2014, Addis Ababa). We want to inform the [target], so we look at [specific] policies tailored towards supporting the [.] campaign to add value to their [the target’s] process. It is not an abstract research. Research directly geared towards an outcome that will be directly influencing [the target’s] activities or CSO activities. (program manager, 2014, Addis Ababa)
For this organization, in most cases, nonclaiming was a deliberate choice to cooperate closely with their targets. Because the organization chose not to claim their work directly—and thus also engaged in the nonclaiming of outcomes—targets were more likely to ask for input, cooperate on specific issues, and extend invitations to meetings. Processes were strategically steered by cooperating closely with the main targets. One of these roles, the organization explained, was translating every report into multiple languages spoken within the AU. This was that part of their advocacy on the Pan-African level, done to ensure that their message was conveyed to all AU member states. However, the nonclaiming of outcomes also resulted from overlooking some outcomes.
To surface these kinds of outcomes, as evaluators, we had to rely on interviews to reconstruct what happened, how one thing led to another, and who was involved in what capacities. This meant that there was room for interpretation—not only because the organization interpreted their own role as supporting and facilitating rather than as advocating or influencing, but also because the information from the interviews was built on how the persons involved interpreted the events and their roles and actions. The good reputation and good working relations of the organization allowed us, as evaluators of this organization, to gain good access to key people in diverse (policy) arenas and civil society actors in Ethiopia and in other African countries (Mozambique, Uganda, Kenia, Namibia, and South Africa). In interviews with policy makers, the organization was lauded as “knowledgeable,” “real experts,” people who “know what they are talking about,” “very helpful,” and “the only African voice on these issues.” These views, of course, are also interpretations.
The issues of claiming and nonclaiming advocacy outcomes demonstrate that there is room for interpretation in the identification of outcomes. Outcomes became outcomes when invoked or claimed as outcomes by the stakeholders involved. In this process, diverse steps were taken through which meaning was attributed to outcomes based on the decisions of political and organizational considerations about the values and needs for partnerships, access, trust building, and legitimacy. Not claiming outcomes or its ownership of them, in combination with its content-driven advocacy, gained a widespread reputation for being a Pan-African expert organization. This provided them with access to be able to work closely with their targets. Nonclaiming provided the necessary space to build trust, be seen as a legitimate voice, and establish cooperative working relations through which outcomes were achieved.
Underreporting and Overreporting
In another case in the evaluation, we uncovered the underreporting of outcomes in a peacebuilding and conflict prevention network. The network committed a great deal of their global secretariat’s time to collecting and understanding outcomes achieved by their regional members. Their annual reports included their members’ reports on what was achieved. Often, the network’s reported outcomes stated that internal network agendas were set, meetings were held, or the network facilitated learning and knowledge sharing by bringing the members together regionally and globally. Although these outcomes were significant for what the network was generating and developing, it was less clear what they had achieved as a network or how the outcomes came about.
Setting a network agenda during an annual meeting, including action plans for the following year, was often described as an important outcome. In one of the regions, I spent 4 weeks observing the regional processes and 3 days observing the annual meeting. This meeting brought together six network members from countries in the region, three members of the regional secretariat, and a representative from the global secretariat. The meeting was supported and facilitated by the regional secretariat of the network. Prior to and during the annual meeting, the members were facilitated to meet with intergovernmental organizations to discuss regional issues and with other civil society stakeholders and organizations involved in a regional conflict. During these meetings, the members were surprised to be able to meet with so many important stakeholders involved in the conflict and peace processes.
Many of the discussions at the annual meeting were about the regional network agenda. This agenda included the reported outcomes of the previous year. In these reported outcomes, the annual meeting was considered an outcome in terms of “setting the regional agenda,” making “action plans for the following year,” and “knowledge sharing and learning from each other.” For many of the network members, this meeting was one of the few moments that they felt was a “network activity.” They did not interpret the work they did within their own organizations as being part of the global or regional network.
The discussions held during the annual meeting underscored that, although the reported outcomes were indeed network-strengthening outcomes, the previous year’s action plans had not been followed up on by the individual members or with action as part of the global network. Additionally, the members did not report on outcomes or actions undertaken related to being part of the global network or to the advocacy agenda. However, updates on the local and regional contexts provided during the meeting made it clear that there were outcomes of these types to be reported. The interviews I conducted with network members also highlighted local-level outcomes such as the development of local networks inspired by being part of the regional network and multistakeholder meetings inspired by shared experiences. The network members were not aware that these were outcomes, or they were uncertain regarding the relation of these outcomes to their role in the global network.
Although a regional network’s agenda formulation could signal a change, it was uncertain for us as evaluators whether this could always be considered an outcome. This was a case of overreporting internal network outcomes while underreporting outcomes that were the result of belonging to a global network (i.e., not being aware of these outcomes or not sharing the same understanding of their meaning). In the evaluation, in a few cases, the evaluation team found that outcomes were portrayed as more powerful, influential, and/or important than they were in reality (i.e., overreporting; see Arensman, van Wessel, & Hilhorst, 2017). Outcomes and the reporting of outcomes in an evaluation assessment were a main priority for the funder (as indicated in the terms of reference and the evaluation questions highlighting outcomes) and for most of the programs involved. Over the course of the evaluation, this resulted in many discussions over the meaning and assessment of outcomes (see Arensman & van Wessel, 2017). These discussions provided insight into the political dimensions of outcomes in terms of accountability and legitimacy needs that sometimes resulted in outcomes being overreported.
Whose Outcomes and What Outcomes
Advocacy outcomes are constructed through interactions among diverse actors working in varied contexts. This means that outcomes become outcomes through interactions among actors across multiple layers of social, organizational, and cultural realities. These actors create outcomes through recursive and strategic actions and decisions. These constructed outcomes are often built over multiple years; for example, this is true of outcomes regarding the nurturing and strengthening of relations of trust and reputation. Meaning is given to advocacy outcomes in the construction of social, organizational, and cultural realities resulting in multiplicities of meanings. This section shows the constructed nature of advocacy outcomes, which makes it difficult to answer questions about “which outcomes,” “whose outcomes,” and “what outcomes.”
Annually, New York hosts multiple international weeks on women’s rights. During one of these weeks, the 2014 Commission on the Status of Women (CSW) week, which is dedicated to the promotion of gender equality and empowerment, I followed and observed a number of advocates. These advocates and their partners were in New York advocating for a global fund for the implementation of United Nations Security Council Resolution (UNSCR) 1325, which pursues women’s rights and equality. The advocates I followed worked with a Dutch international NGO in cooperation with other advocates through networks and organizations in diverse policy arenas (e.g., the UN, the European Union, and the North Atlantic Treaty Organization) and contexts (priority countries including Afghanistan, Burundi, and Colombia). The advocacy was directed at pursuing women’s rights and women’s leadership in local, regional, and global contexts including the UN. During the years of my observation and research of the advocacy program (2012–2015), the advocates followed the orbit of global policy meetings, for example, making multiple trips to New York each year. The business of these trips comprised meetings with partners, networks, civil society stakeholders and policy makers within the UN, and with government representatives to the UN based in New York.
During the CSW, the advocacy program facilitated and organized country-themed events, and they used this time to meet to discuss strategy. For these meetings, they invited their local partners. The local representatives to the 2014 CSW were from South Sudan, Afghanistan, and Somalia. All of these representatives were women. Most of the women had been to New York multiple times. One of the representatives from Afghanistan said that she felt responsible for the future of her country and for Afghan women. She emphasized her reasons for coming to New York as telling “the true story, which is not pretty, but I draw it out to them [the UN] the way it is” (advocate 2014, New York). With direct and indirect support from the Dutch international NGO, much of the advocacy by the local partners was driven by personal commitments to improve the situation for women in their countries. These local voices were incredibly important for legitimacy reasons and for constructing the advocacy messages. During a breakfast meeting, a Dutch advocate proposed that key recommendations from the local partners be put in a policy brief that was being distributed during the CSW week in meetings with key targets.
During the CSW week, many meetings were held, including a meeting with a high-ranking, established UN official. The advocacy organization identified this meeting as an outcome. The advocates involved in the meeting stressed the importance of this meeting: “If she (the UN official) underscores appreciation for our idea (global fund for UNSCR 1325 implementation), this is so high up the chain that even UN staff will see the importance of the issue we advocate for and see this as an opportunity” (advocate, 2014, The Hague). While the commitment of this UN official was necessary for the process to continue, the meeting and its outcomes involved many actors, interactions, and political sensitivities. For the global fund for UNSCR 1325 to succeed, the advocacy organization needed governments on board, and the organization’s local partners at all levels were needed to contribute to getting the governments on board. The organization needed commitments in terms of political agendas and funding. The meeting with this UN official was a constructed outcome, as it represented a spiralled opportunity emerging from years of interactions initiated by many different partners in many diverse arenas. Years of relations building went into this outcome, as did lots of legwork, pushing and pulling, and creating visibility. Processes that include bureaucracies, such as those found in the UN, demand patience, much going back and forth and many follow-up meetings to keep the issue on the agenda.
Prior to the CSW week, a Dutch advocate explained to me that there was a lot of “legwork” on the part of their partners in New York—meaning many points of contact (i.e., e-mails, meetings and calls) with those involved in the decision-making processes, partners in the civil society arena, and partners in the policy arena. This advocate stressed that over 100 e-mails went back-and-forth to establish meetings, share reports, and create visibility in order to profile themselves as “thought leaders” (policy officer, 2014, The Hague). At the heart of the advocacy was “building and nurturing the relations with civil society networks and policy makers” (advocate, 2014, New York). Most of these relations became personal relations over the years.
This also included close relationships with targets who were often also involved in the networks. A partner advocate working in the UN arena and the women’s rights field for over 20 years stressed that close relations with targets were critical for their advocacy: Many donors want their partners to become stars because it draws also attention to their work and their role. But there are limited resources and donors have their priorities and it is a continuing challenge to reconcile donor priorities with our own priorities. Luckily, we have been recognised, and we don’t need to shift all the time to the sexy issues of donors. But the funding is short-term [and] often annual, so this is a problem. (advocate, 2013, New York) The traditional civil society stance to the government is adversarial, seeing government as an enemy. For us, we can’t be stuck in this traditional adversarial stance. Government bodies and elected bodies—we need them at our side. We are critical, but we cannot speak openly to criticise them. We need to build and nurture our relations, based on a good track record and being respectful while giving them space and being in constant contact. (advocate, 2014, New York)
The above demonstrates that advocacy outcomes were constructed. First, the outcomes were layered, spiralling out from years of relations building and connections on personal and professional levels forged between the actors involved across social, cultural, and political levels, including work with both civil society and advocacy targets (policy makers at national and international levels). Second, following from this process, the outcomes were constructed on multidimensional levels involving transnational processes and crossing national borders and international arenas; they involved multiple stakeholders working at diverse levels of organizations, governments, and institutions focusing on national, international, and regional processes. Third, this meant that the outcomes were constructed in the sense of involving cross-political levels, with the advocacy targets themselves being actively involved in the networks. These characteristics made the changes achieved more elusive and more difficult to articulate.
Visibility and Profiling
During the advocacy evaluation, the question of “whose outcomes are we talking about?” came up multiple times. Most advocacy programs worked with local partners in networks or partnerships. In one case, it became clear that the question of “whose outcomes” was related to making outcomes visible to claim them as your own. In 2014, as evaluators, we met with the staff of the evaluated program for a discussion of their advocacy challenges relating to connecting global-level processes with local needs and vice versa. During this meeting, advocates and program managers stressed that visibility was important for reasons of organizational accountability and legitimacy. The program needed to make certain results visible to account for funding spent (organizational) and to build credibility as a “thought leader” (advocacy).
In their advocacy, the advocates pursued ownership. Ownership by (local) women’s groups was emphasized, stressing that “what we do is really working with women’s networks” (program manager, February 17, 2014, The Hague). In one of the countries, where the advocacy program worked on strengthening diverse women’s networks and organizations, they pursued the effective implementation of UNSCR 1325. Doing this supported and facilitated local partners in strengthening their role in the peace process and in pursuing women’s rights and equality. The connection between global and local, however, was difficult, as the two realities were constructed around diverse languages. An advocate said, “I have to translate our reports and label it into UN language” (advocate, 2014, New York). The UNSCR 1325 framing of women’s rights did not connect with the needs of the local partners. The program manager stressed, “I had been working (in this specific country) for a long time already, and on 1325 as well, but it was too sensitive […] to use 1,325 language.” (program manager, 2014, The Hague). The local partners had their own framing of women’s rights and their own national and local policies and laws in place. The program manager stressed that “there are a lot of differences; UNSCR 1325 is just a big label, but the issues are so specific in the different contexts” (February 17, 2014, The Hague).
Adding to the issue of framing, the partners’ strong sense of ownership also resulted in a less articulate or explicit advocacy role for the Dutch international NGO. The local partners developed and built women’s networks, owning their progress, and becoming more like independent partners than grantees. This made it difficult for the Dutch NGO to claim outcomes for these processes, as the Dutch program manager emphasized: [This specific country] is difficult to have an impact linked to [our programme], because in [country X] the partners are very sensitive to ownership. This is about profiling, about putting a […] logo and stamp on processes, documents, etc. The link with [our programme] is there, but it is not visible per se. (program manager, 2013, The Hague) You have to put your name and logo everywhere; this is important for donors and results accountability […] It is about building a reputation and being seen as [a] thought leader. (policy officer, 2014, New York)
All of this demonstrates that identifying advocacy outcomes was a political process at multiple levels, involving ownership over the processes as well as the need to account for outcomes in a certain way. The relation of the organization’s role and objectives with the local embedding was diffuse. Making outcomes visible involved diverse political interests. For the Dutch NGO, there was a vested interest in claiming the results achieved for accountability and for leverage reasons, to be seen as a thought leader in the specific field of women’s rights. For its partners, the interest was in claiming ownership over their own processes in order to create leverage, while also translating and framing the language in their own political language.
Discussion: The Internal and External Politics at Play
The above-illustrated cases demonstrate the constructed and political nature of advocacy outcomes. First, outcomes become outcomes when they are interpreted as such by stakeholders making (strategic) decisions. Second, advocacy outcomes are constructed in the way they are given meaning in the multilayered realities that influence how they are invoked and presented. Finally, the process of identifying outcomes is political, as outcomes are given meaning in relation to diverse interests and (strategic) decisions made regarding needs (legitimacy, credibility) versus demands (accountability). In this process, diverse steps were taken through which meaning was attributed to outcomes based on the decisions of political and organizational considerations about the values and needs for partnerships, access, trust building, and legitimacy.
In the implementation of existing methods like OM and OH, the focus remains often on outcomes as visible and measurable as well as evaluation as objective. The above case studies, however, illustrate that for the identification of advocacy outcomes, approaches need to be flexible—taking interpretation into account—as well as being aware of the social and political constructed nature of outcomes and outcome identification. This section therefore underscores the complexity of advocacy evaluation and provides a basis for critical questions on whether we should let go of outcomes as the standard for effectiveness. This is directly linked to the question of measuring advocacy effectiveness (what vs. how).
The case illustrations demonstrate that a participatory approach where evaluators and evaluated work closely together is indeed necessary to evaluate advocacy (see Jones & Hearn, 2009; Wilson-Grau, 2015; Sloot & Gaanderse, 2010). This is not only because evaluators need access to information about theorizing and strategizing, which is in the heads of advocates but also to stimulate learning together with the advocates involved. My insight into strategic nonclaiming would not have come out without meaningful and open interactions with those involved. Additionally, the politically sensitive nature of making outcomes visible, claiming those outcomes, and, thus, overreporting and underreporting would have gone unnoticed. However, this kind of participatory approach presents challenges with regard to the identification of outcomes that are not self-evident. As the findings demonstrate, advocacy outcomes are given meaning in narratives, which are interpreted by those involved and constructed in multilayered realities. These narratives are also political because much of the legitimacy and credibility of advocacy is dependent on the ability of the advocates to demonstrate effectiveness. Identifying outcomes and judging effectiveness are often more related to the meaning given to outcomes in terms of the legitimacy and credibility of the organization than to some kind of objective measurement. The current approaches like OM and OH do not provide solutions or tools to deal with these kinds of challenges.
For this reason, I argue that there is a need for a different understanding of outcomes, shifting from the narrow outcome-driven focus on measurable results to an understanding of outcomes as given meaning through political interactions and constructions that provide room for interpretation. OM and OH both build on the assumption that outcomes are visible, and can be predicted and measured, or at least that they are self-evidently following from action. However, my findings underscore that outcomes are much more than just the result of achieved objectives; they can also provide insight into the politics and meaning attributed to results in certain contexts.
For evaluators, this means that the methods used must include the constructing and reconstructing of what happened, how, why, and who was involved. This provides the ability to identify meaningful outcomes constructed in meaningful interactions. If one decides to remain focused on outcomes as important to understand the effect in terms of changes and for accountability purposes, at least these should be understood in relation to the dynamics of advocacy as strategic practice, with the advocates as strategic actors at the center of this. This includes the idea that outcomes are pursued and strategized for in ways that do not always follow a predefined path. This kind of understanding means “zooming in” on the dynamics that give meaning to outcome processes and practices, including how they fail to come about. Identifying outcomes and their challenges requires the flexibility to incorporate an understanding of the roles played by interpretation, construction, and politics. This is especially the case for advocacy because there is a dependency on the advocate’s practical judgments and account of events to understand the processes and practices leading to results—or the lack thereof.
Furthermore, evaluators and practitioners need to acknowledge that the systems shaping evaluation standards are formed by how effectiveness is defined and that outcomes have been given center stage in terms of effectiveness claims. Evaluating effectiveness requires a clear understanding of what is being evaluated, why and how, as well as what evaluation questions need to be answered to assess a certain kind of effectiveness. Effectiveness is not only established in the identification of outcomes but also in the interactions between the processes of theorizing and strategizing, with practices that demand acting, reacting, and adapting to changing circumstances.
Conclusions
The findings provide important insight into the problematic nature of outcome identification, which is especially relevant for the field of advocacy evaluation. Working in an outcomes-focused manner together with the pressure to demonstrate effectiveness through measurable results (Best, 2017; Eyben, 2015; Riddell, 2014; Vähämäki et al., 2011) risks adopting evaluation approaches that trade quality for quantity. These kinds of approaches may overemphasize tangible outcomes in light of what was planned at the expense of understanding the process of how these came about. Advocacy evaluation as a field of practice is struggling to come to terms with the pressure and need to demonstrate effectiveness to account for how funding is spent and to attain legitimacy and credibility for the organization or program, while also coping with the problematic issue of measuring advocacy effectiveness and questions of how to assess effectiveness of processes that are complex, dynamic, and difficult to measure.
Although outcomes are increasingly important in the pursuit of effectiveness, they are still considered self-evident, even in approaches like OM and OH that seem to open the door to more flexible outcome identification. Approaches like OM and OH—what’s in a name—focus too much on outcomes in and of itself as self-evidently following from actions. Identifying outcomes in advance may lead to overemphasizing certain outcomes over others as well as “sticking to the plan” rather than revisiting the strategy and its outcomes. In advocacy, adaptation, strategic interactions, decision-making, and reflection are crucial. This influences not only how outcomes are interpreted (e.g., claiming, nonclaiming, overreporting, and underreporting) but also the quest for specific types of outcomes (e.g., those that are easily measured, made visible, and claimed), meaning that outcomes do not necessarily follow from certain actions (i.e., the same type of strategy may lead to a certain change in one case, while not in another). The relation between strategy or action and its results is not set in stone or clearly defined in terms of attribution.
In conclusion, evaluating advocacy is a quest. Advocacy evaluators should therefore step away from outcomes as a main point of concern. Instead, they need to (re-)construct, question, and interpret its processes in an inquisitive by placing the advocates’ strategic action at the center rather than merely focusing on what is achieved. This means unpacking the narratives that provide meaning to its course of action and its results. Evaluators thus need to train in surfacing, questioning, and critically reflecting on the meaning of effectiveness in relation to the processes they evaluate and in constant interaction with the persons who are involved, that is, Why were certain directions taken? What were the underlying considerations? What made the advocates act or interact in that way? In other words, being inquisitive by focusing on the experiences and the stories that give meaning to everyday practices. This requires an evaluator to have an open mind and to take an approach that is truly interaction- and people-centered by focusing on the advocates, their roles and understandings, the diverse realities in which they operate, and their strategic and practical judgments, considerations, and actions. This can add much value to the approaches of OM and OH and provide a meaningful place to Coffman’s (2007) interim changes in the identification of outcomes as a strategic process.
This article provided insights into the problematic identification and evaluation of advocacy outcomes. On a final note, I would like to shortly reiterate the following principles for advocacy evaluation from the above discussion: (1) evaluation of advocacy is fundamentally a learning process; (2) meaning is made and understood in interpersonal interactions; (3) advocacy is a dynamic and strategic process in which advocates act strategically at the center; (4) narratives, experiences, strategic decision-making, and interpretations are key to understanding advocacy; and (5) processes need to be constructed and reconstructed in order to understand its meaning and its outcomes. To continue building on the findings presented above, I encourage other researchers to develop what we now know about advocacy effectiveness and its evaluation into concrete steps and approaches guiding evaluators and practitioners to really do justice to the dynamic nature of advocacy in evaluation.
Footnotes
Acknowledgements
Declaration of Conflicting Interests
Funding
ORCID iD
