Participation incentives in a survey of international non-profit professionals

Abstract

Elite surveys are increasingly common in political science, but how best to motivate participation in them remains poorly understood. This study compares the effect of three treatments designed to increase participation in an online survey of international non-profit professionals: a monetary reward, an altruistic appeal emphasizing the study’s benefits, and a promise to give the respondent access to the study’s results. Only the monetary incentive increased the survey response rate. It did not decrease response quality as measured in terms of straight-lining or skipped questions, although it may have produced a pool of respondents more likely to speed through the survey. The findings suggest that monetary incentives reduce total survey error even in the context of an elite survey, perhaps especially with elite populations frequently contacted by researchers. However, such incentives may not be without trade-offs in terms of how carefully respondents engage with the survey.

Political scientists increasingly study elites using surveys and experiments. The surveyed populations include politicians, bureaucrats, activists, international organization staffers, and business leaders. This turn has prompted attention to various ethical and methodological issues raised by surveying elites (e.g., Dietrich et al., 2021).

A lacuna remains, however: an analysis of the best methods to recruit such individuals to participate in surveys in the first place. Although there is a substantial literature on response rates in public opinion surveys, how to motivate participation in elite surveys is less understood (Kertzer and Renshon, 2022: 15). This omission is surprising since elite populations are often small and difficult to reach. Moreover, the factors that influence elite response quality are understudied. An exception is a recent study by Conn et al. (2019) of survey participation by pro-social elites in India, which found that altruistic appeals were as effective as monetary appeals at encouraging participation.

We build on their study to report the results from a pre-registered experiment on participation incentives in a survey of the leaders of international non-governmental organizations (INGOs). ¹ Our study answers the call by Kertzer and Renshon (2022: 16): “researchers [should] incorporate manipulations designed to assess the relative efficacy of different approaches to recruiting political elites.” We compare the effects of a monetary reward, an appeal emphasizing the study’s benefits, and a promise to give the respondent access to the study’s results. Only the monetary incentive increased response rates in our study. It may, however, have attracted less-dedicated respondents. Our conclusion considers why.

Design

The population of interest was leaders (e.g., presidents and founders) of U.S.-based INGOs. A key motivation was to conduct a pilot for a planned survey of a smaller subpopulation of INGO elites related to Bush and Hadden (2019) and determine the best way to invite respondents.

We identified potential participants using the roster of the World Association of Non-Governmental Organizations (WANGO). According to its Web site, WANGO is “an international organization uniting NGOs worldwide in the cause of advancing peace and global well being.” ² NGOs can apply for and pay dues to WANGO and range considerably in size, issue area, and tactics. NGO members provide limited information about themselves to be listed in the WANGO member directory.

We invited 19,354 individuals listed as U.S. WANGO members via email to participate in a survey about non-profit organizations in Summer 2019. Respondents could participate by clicking on a Qualtrics link. Invitations were identical except for the randomly-assigned appeal to participate (see Table 1). We sent two reminders with the same appeals. Supporting Information (SI) §1 contains additional survey details.

Table 1.

Experimental treatments.

Condition	Invitation
Control: Egoistic appeal	“You have been specifically selected to participate because of your expertise working in this field.”
Treatment 1: Altruistic appeal	Control + “Your participation will contribute to scientific understanding of non-profit organizations and the results of the survey will be used to design educational materials for use with students.”
Treatment 2: Data incentive	Control + “As a token of our appreciation for your participation, we will provide you exclusive access to the results of the survey in advance of the public release.”
Treatment 3: Monetary incentive	Control + “As a token of our appreciation for your participation, we will send you a $10 Amazon gift card for completing this survey.”

Our choice of treatments was informed by the previous literature on survey participation within the general population, which delineates between egoistic and altruistic motivations. ³ Egoistic motivations capture self-interested reasons for participation such as in response to monetary incentives, enjoyment, personal benefit, or an interest in the results (Kropf and Blair, 2005; Singer, 2011; Singer and Ye, 2013). Monetary incentives are a common survey incentive (including in Conn et al. (2019)), and thus were an obvious choice for inclusion in our study. Our theoretical expectation was that monetary incentives would increase survey participation for egoistic reasons, although not all research on relatively elite populations supports this expectation (Kam et al., 2007).

Our other treatments promised exclusive data access and evoked the importance of the study for understanding INGOs and education, respectively. We chose these treatments after interviews with INGO staff and reviewing the INGO literature (e.g., Mitchell et al., 2020) to determine possible non-monetary motivators for participation within our target population. They were designed to appeal to non-profit staffers’ concerns with helping others (by referencing the study’s contribution to research and education) and organizational effectiveness (by offering access to unique data on INGOs). ⁴ Our theoretical expectation was that these non-monetary appeals would increase survey participation, perhaps even more so than the monetary incentive. Some have questioned whether elites might find monetary incentives less appealing, or even insulting (Renshon, 2015: 674), and we might imagine more altruistic appeals would encourage participation among INGO leaders.

About 2% of invitees (378 individuals) responded. This low response rate reflects that some emails were no longer active as well as challenges associated with recruiting elite respondents, which motivated our study. This relatively-small sample size means that some of our tests are under-powered, although we find significant effects in some cases. As discussed at more length in SI §2, our response rate was similar to some other elite surveys such as those in Nielson et al. (2019: 699–701). Because of the low response rate, we agree with Nielson et al. (2019: 698) that it makes sense to think of our sample as a convenience sample of elites and to recognize the resulting likelihood of response bias.

Results

We pre-registered several outcomes measures for this study. The first was the overall response rate, calculated according to the definition used by the American Association for Public Opinion Research. ⁵ The next outcomes were three satisficing behaviors shown to indicate poor response quality: straight-lining, non-response, and speeding (Greszki et al., 2015; Krosnick, 1991; Zhang and Conrad, 2014). Straight-lining is when a respondent provides undifferentiated (identical) answers to a series of statements. Speeding is when an individual responds so quickly that they could not possibly have read the question, let alone carefully consider their answers. Item non-response refers to the frequency of leaving questions blank.

Table 2 displays the survey response rates by experimental group. Only the monetary incentive significantly increased response rates relative to the control. Since the treatment effect is 1.2 percentage points, the response rate more than doubled. ⁶

Table 2.

Response rate by group.

Experimental Group	Response Rate	N
Control	1.1%	4834
Altruistic appeal	1.3%	4841
Data incentive	1.2%	4835
Monetary incentive	2.3% a	4845

^aindicates difference of proportions from control is significant at p < 0.05.

Motivating participation with a monetary incentive could decrease response quality. Yet the treatment and control groups did not differ significantly in straight-lining (Figure 1) or item non-response (Figure 2). SI §4-5 describes these tests.

Figure 1.

Predicted straight-lining by group. Expected counts and 95% confidence intervals calculated from negative binomial regression.

Figure 2.

Predicted skips by group. Expected counts and 95% confidence intervals calculated from negative binomial regression.

There is, however, suggestive evidence that the monetary appeal affected how dedicated the recruited respondents were. Figure 3 shows the proportion of speeding respondents. More respondents in the monetary incentive group sped than in the control group, across all survey parts. The largest difference was 12 percentage points in Part 1 of the survey (p = 0.07, one-tailed test of proportions). Similarly, respondents in this treatment were 12 percentage points more likely to speed in Part 1 than respondents in the data incentive group (p = 0.09, one-tailed test of proportions). Although these differences do not reach statistical significance, their magnitude and consistency (for more, see SI §6) suggest the monetary incentive may have produced respondents who were more likely to rush.

Figure 3.

Proportion of speeders by group. See SI §5 for details on the survey parts and how speeding is coded.

Discussion

The results provide mixed evidence regarding the links between incentives and response quality among our respondents. On the one hand, the results for straight-lining and item non-response indicate that a monetary incentive reduced total survey error by increasing response rates and maintaining response quality. On the other hand, it may have attracted less-thoughtful respondents, perhaps because the tokenistic incentive discouraged effort. The pattern of these results and their consistency with theoretical expectations suggest more research is needed to examine links between speeding behavior and incentive types.

That monetary incentives most increased participation among INGO leaders is consistent with findings about the general survey population but at odds with the findings in a previous study about pro-social elites in India (Conn et al., 2019). ⁷ It is plausible that our distinct study population may account for the difference. Our interviews indicate that some INGO staffers—similar to other oft-studied populations—receive frequent queries from academics and may be weary of academics engaging in “information extraction” without providing much benefit in return (Cronin-Furman and Lake, 2018: 609). For them, a monetary incentive could make participation in a survey more appealing by offering some compensation, even if it is of a token amount. By contrast, other selective populations such as those surveyed in India may be elite in terms of education or social class but more unused to being contacted by researchers. Further research will need to better understand how to motivate survey participation among elites from diverse contexts.