Sage Journals: Discover world-class research

Abstract

Expert surveys are a valuable, commonly used instrument to measure party positions. Some critics question the cross-national comparability of these measures, though, suggesting that experts may lack a common anchor for fundamental concepts such as economic left–right. Using anchoring vignettes in the 2010 Chapel Hill Expert Survey, we examine the extent of cross-national difference in expert ideological placements. We find limited evidence of cross-national differences; on the whole, our findings further establish expert surveys as a rigorous instrument for measuring party positions in a cross-national context.

Keywords

European parties expert surveys vignettes

Scholars of party politics need to measure the positions parties take on a variety of policy domains. They have developed a diverse set of data collection tools – including, most notably, manifestos, roll call data, voter judgment, and expert surveys – in response to this challenge. While different sources of data may be appropriate for different kinds of questions related to party positions, we argue that expert surveys offer several advantages over other alternatives.

Expert surveys allow researchers to obtain positions for many different types of parties across a range of contexts (see, for example, Benoit and Laver, 2006; Hooghe et al., 2010; Rohrschneider and Whitefield, 2009). Expert surveys’ virtues include that they are inexpensive to administer, that they draw on broad knowledge about parties by tapping into information about what parties say and do, and that they allow for a high degree of flexibility as researchers can gather information on any topic for which there are enough competent experts (Marks et al., 2007). As a result, expert placements of parties are widely used within political science today.

Yet important questions pertaining to the expert evaluations remain. For instance, Budge (2000) is concerned that expert-specific differences contribute to errors in the data, including, for example, expert political preferences leading to biased placements of right-wing parties (Curini, 2010) or knowledge differences among experts contributing to overly centrist placements (Gemenis and Van Ham, 2014). Here, we focus on a separate but related question that concerns expert survey data: whether expert placements of parties are comparable across countries. Put differently, does an expert on the French or Swedish party system think of concepts such as left and right the same way as an expert on British parties when both are asked to place the parties of their respective countries on an economic left–right scale? As Benoit and Laver (2007: 94) argue, “what experts do have in mind when they talk about left and right, in terms of substantive policy dimensions, varies in intuitively plausible ways from country to country.” The political space underlying the substantive policy dimension may differ across individual experts, countries or regions.

Benoit and Laver (2007) stress the importance of comparing expert survey-based estimates of party positions with other independent left–right scales. However, even if expert-based estimates correlate highly with other measures of left–right, these correlations do not necessarily mean that the expert surveys are free of expert, country, or other context-specific bias in their responses. Thus, we need to examine the cross-national comparability of expert placements.

Research on European Parliament (EP) party group formation and durability provides a specific example for when this cross-national comparability is necessary. As EP power grows relative to other EU institutions (Hix and Høyland, 2013), it becomes increasingly important to understand party behavior in this supranational legislature. National parties join party groups in the EP that are composed of parties with similar policy positions, and when parties switch group affiliation they do so in an attempt to minimize positional incongruence (McElroy and Benoit, 2010, 2012). Scholars of the EP therefore require measurements of party positions that they can confidently assert are cross-nationally comparable.

Survey vignettes, which are designed to identify and ultimately correct situations when survey respondents interpret identical questions differently, provide a method of directly measuring the incomparability of responses to survey questions with ordinal response categories (King and Wand, 2007; King et al., 2004). The 2010 Chapel Hill Expert Survey (CHES) implemented this method by presenting country specialists with three hypothetical vignette parties described by short vignettes. Experts from all survey countries placed these vignette parties on an 11 point economic left–right scale, which allows the analyst to identify potential incomparability in the experts’ placement of the actual parties in the survey. With these vignettes, we can assess, for example, whether a 7 in Sweden is a 7 in the United Kingdom.

Our results indicate that experts across Europe are strikingly consistent in their ordering of the vignette parties. Although limited cross-national differences in measurement do exist, taken on the whole, our findings are consistent with those showing that expert placements provide reliable and valid measures of left–right party positions (Benoit and Laver, 2007; Hooghe et al., 2010). Our analysis shows that most experts included in this sample use “left–right” in a similar way. These findings are thus an important step forward in establishing expert surveys as a rigorous instrument for measuring party positions over time and across countries.

We begin the article by summarizing the anchoring vignettes approach, and then introduce the 2010 CHES. After briefly discussing the validity and reliability of the survey, we focus our attention on expert placements of the three vignette parties. We conclude by discussing the implications of our findings for the use of expert surveys in cross-national studies of political parties.

Anchoring vignettes

Anchoring vignettes were created to identify and correct situations when different groups of survey respondents use ordinal response categories differently. When presented with a question with typical response categories such as “strongly disagree” or “strongly agree,” a respondent group with relatively higher standards for what “strongly agree” means will provide consistently lower levels of agreement than other survey respondents. As King and Wand (2007: 46–47) point out, some respondents will differ in ways (e.g. mood, propensity to use extreme categories) that affect their responses. Anchoring vignettes allow us to evaluate the extent of response-category differential item functioning (henceforth DIF).

Along similar lines, critics of expert surveys argue that experts must determine their own frame of reference when placing parties because they do not share an anchor point (McDonald et al., 2007). In short, “centrist” in one country may mean something very different in another country. If the experts use their own country’s center as a reference point, “it leaves unanchored a center position that would make measurements across nations comparable” (McDonald et al., 2007: 74). Thus, the cross-national comparability of expert placements are in question.

The fundamental concern is that the political space underlying party placements in different contexts (be it individual experts, countries or regions/cultures) is perceived differently. On the one hand, the political space may be shifted between two different contexts. A party holding center-left economic views in one country, for example, may be considered an extreme-left party in another. On the other hand, the political space may be stretched or contracted across contexts. For example, the distance between the most extreme left and right parties in one country may span the full length of the left–right continuum, while in another context, these same two parties may be considered as moderate left and right parties, placed relatively close to one another. This spatial incomparability potentially implicates research that aggregates party positions across contexts, rendering its results susceptible to bias.¹

In the 2010 CHES, the three vignettes depicting the economic left–right position of different hypothetical parties are designed to address this problem. In the next sections, we introduce these vignettes.

Chapel Hill Expert Survey

2010 expert survey

The 2010 Chapel Hill Expert Survey was conducted in spring 2011 and covers 227 national parties in 26 countries, extending earlier CHES datasets covering 1984–2006 (Hooghe et al., 2010; Marks et al., 2007; Ray, 1999). The survey covers various aspects of party positioning for 24 EU members (excluding Cyprus, Luxembourg, and Malta) and two other western European states (Norway and Switzerland). Table 1 displays the countries included and the number of parties and experts in each country.

Table 1.

Experts and political parties in the 2010 Chapel Hill Expert Survey.

Country	Number of parties	Number of experts
Austria	6	14
Belgium	14	16
Bulgaria	10	14
Czech Rep.	7	20
Denmark	9	11
Estonia	6	14
Finland	8	10
France	9	9
Germany	6	18
Greece	7	11
Hungary	7	17
Ireland	6	8
Italy	14	9
Latvia	10	7
Lithuania	11	13
Netherlands	10	14
Norway	7	11
Poland	8	15
Portugal	5	6
Romania	6	23
Slovenia	8	13
Slovakia	10	15
Spain	13	12
Sweden	10	15
Switzerland	12	12
United Kingdom	8	16
Total	227	343

Previous research has demonstrated the reliability and validity of CHES data. The data have been cross-validated with several other data sources: party manifestos, public opinion, surveys of MPs and MEPs, and other expert surveys (Bakker et al., 2014; Hooghe et al., 2010; Marks et al., 2007; Netjes and Binnema, 2007; Steenbergen and Marks, 2007; Whitefield et al., 2007).² And yet these validity tests cannot, by themselves, assess the cross-national comparability of the expert placements. Instead, we use the anchoring vignettes that were included in the 2010 survey to assess the degree to which experts in different countries are placing parties on this dimension according to a common scale.

Left–right party placement and vignettes

Evaluating the left–right placements of parties is especially important as positions and preferences of parties are most often expressed in left–right terms, and the economic dimension tends to be the primary one in most party systems. Often left–right ideology is the only dimension used to describe party positions in political space (see, for example, Downs, 1957). CHES experts were asked to place political parties on an economic left–right dimension. At the end of the survey, experts were also presented with vignettes and asked to place hypothetical parties A, B, and C on the economic left–right ideological dimension. The text for these three vignettes is below.

Party A views the provision of a social safety net as important, but believes there is a sharp trade-off between welfare spending and economic competitiveness. It favors limiting government regulation to instances of market failure, and prefers cuts in social spending over new taxes to meet rising social needs. On a 0−10 point scale with 0 being extreme left and 10 being extreme right, where would you place this party?

Party B views the equalization of life chances for all citizens as an important goal of government. It favors active government in regulating domestic and international markets, and supports steeply progressive taxes to fund redistributive social programs. On a 0−10 point scale with 0 being extreme left and 10 being extreme right, where would you place this party?

Party C believes in small government. It favors minimal regulation of domestic and international markets, supports the privatization of many government operations, and opposes high taxes to fund redistributive social programs. On a 0−10 point scale with 0 being extreme left and 10 being extreme right, where would you place this party?

In our view, these vignette parties fall on an ordered scale, ranging from left-wing (Party B) to center-right (Party A) to far-right (Party C). The vignettes each address questions about the proper size and role of government in managing the economy, questions that are relevant to all the societies we survey. The vignette for Party B depicts a party that very clearly supports active and robust government involvement in economic matters, as well as a highly redistributive welfare state. The vignette for Party A was constructed to present a party that is relatively ambivalent on government intervention in economic matters. Party A identifies the welfare state as desirable, but believes that there is a fundamental tension between the country’s economic performance and government spending on social programs, which differentiates Party A from the unambiguous position of Party B on government spending and economic management. As opposed to Party B and Party A, Party C’s vignette creates a party profile that is unambiguously right in its left–right orientation. This party explicitly states a preference for small government, limited intervention in economic matters, privatization, and an opposition to taxes for social spending.

We expect these vignettes to be cross-nationally comparable and ordered on a single dimension with Party B to the left of Party A, which is to the left of Party C, and empirical analysis supports this ordering. Out of 280 experts in 26 countries, 204 (72.9%) experts ordered the parties in the predicted order (B < A < C). Another 41 (14.6%) experts evaluated parties A and C as equal, but to the right of B (B < A = C). The only significant departure from the expected ordering is the group of experts (28, 10%) who evaluated Party C to be to the left of Party A (B < C < A). Overall, 247 of the experts did not violate the expected ordering while 277 of the 280 had at most a single violation of the expected order. Table 2 provides the frequency of vignette ordering by experts.

Table 2.

Frequency of vignette ordering.

Ordering	Frequency	Proportion	Order violations
B < A < C	204	72.9%	0
B < A = C	41	14.6%	0
B < C < A	28	10.0%	1
B = A < C	2	0.7%	0
B = A = C	1	0.4%	0
A < B = C	1	0.4%	1
A < B < C	1	0.4%	1
C < B < A	1	0.4%	2
C < A < B	1	0.4%	3

Note: B < A < C is the expected ordering.

The rank order correlations of the vignette parties across all countries provides further evidence that the vignettes fall on a cross-nationally comparable ordered scale. For the left–right vignette parties A, B and C, we computed the 325 pairwise rank order correlations between the mean placements using Kendall’s tau (Kruskal, 1958). Kendall’s tau measures the similarity in rank orderings across measured items and ranges between −1 and 1 with 1 representing identical ranking. For all pairwise comparisons, with the exception of those involving Greece, the Kendall’s tau is 1. This indicates that the mean placements of the vignette parties on the left–right scale are perceived in the same order across all countries. In Greece, the lone exception, the mean placement of vignette parties A and C is equal (B < A = C), thus reducing the Kendall’s tau to 0.67.

This stable and consistent ordering suggests that the assumption of vignette equivalence is met. King et al. (2004: 194) argue that vignette equivalence is the “assumption that the level of the variable represented in any one vignette is perceived by all respondents in the same way and on the same unidimensional scale, apart from random measurement error.” At a more basic level, this assumption implies that the variable actually exists and has a “logically coherent and consistent meaning in different cultures” (King et al., 2004: 194). The consistent ordering of the vignette parties by experts across the 26 countries in the survey suggests that the economic left–right dimension meets these criteria.

But this consistent ordering of the parties across the vast majority of experts masks interesting variation in the expert placements of the vignette parties. With the vignettes, the hypothetical parties have the same actual ideological placement, and the survey data illustrate how different experts utilize the same ideological scale to place these parties. In other words, experts from one country can place Party B at a 2 along the scale, whereas experts from another country evaluate Party B as a 2.8. If there are significant differences, these differences represent the country DIF.

In Figure 1, we present the average placements for each vignette party for every country. In each country, it is clear that vignette Party B is to the left of A and C. With the exception of Greece, Party A is consistently to the left of Party B.³ But this visual comparison of means does not tell us whether the placements of the vignette parties are statistically significantly different across countries. We therefore conduct a test to determine the degree of country-level DIF in the left–right scale by estimating an ordinary least squares (OLS) model with the expert placements for the vignette parties as the dependent variables. We use indicator variables for countries as the independent variables in these models and suppress the constant term. The coefficients on these dummy variables are then the country-specific mean placements for the three vignette parties.

Figure 1.

Average placement of vignette parties by country.

For each model (vignette party) we can now conduct all of the pairwise difference of means tests to determine whether or not the placement of a given vignette party is significantly different across countries. For our data this results in 325 independent difference of means tests across the 26 countries in the sample. Given the large number of statistical tests, we choose to graphically present the results using the R package factor plot (Armstrong, 2010, 2011). The factor plot displays the pairwise comparisons of the mean vignette party placements. Significance is reported for the p < 0.05 level.⁴

In Figure 2, we present the factor plots showing all 26 European countries in the 2010 CHES and all three vignette parties. White cells represent no significant difference in mean placement, indicating that the row and column country experts place the vignette party at a similar point along the ideological scale. Darker shaded cells mean that the experts in the row country consider the vignette party to be to the right of where the column experts evaluate the vignette party (i.e. the row party has a greater mean placement than the column party (Significantly > 0)). Lighter shading reports that the row country experts place the vignette party to the left of where the experts from the column country place that vignette party (i.e. the row country has a lower mean placement (Significantly < 0)).

Figure 2.

Factor plot comparing vignette party placements.

Starting with vignette Party A in Figure 2a, compare the placements of Latvian experts to Norwegian experts. Latvian experts’ evaluations of Party A are statistically similar to all other country experts (unshaded cell), except for Norway. Latvian experts place Party A to the left of Norwegian experts (light grey cell). Returning to Figure 1, Latvian experts are visually an outlier in their placement of Party A, but only the difference between Latvia and Norway is statistically significant.

For Party B in Figure 2b, Bulgaria’s placement is further left-wing than all other country experts, but the differences are statistically significant for the comparison with Slovenia, Latvia and Greece. Finally, Figure 2c shows the factor plots for Party C. For this party, the Greek experts evaluate Party C further to the left than any other country experts (see Figure 1); the contrast with Bulgaria is the only statistically significant difference.

These figures show broad agreement, with few exceptions. Certainly, some country experts, such as Greece and Bulgaria, see the vignette parties differently. In Bulgaria the political space is stretched: Party B tends to be placed to the left of where it is placed in other countries, while Party C is placed to the right. In Greece, the political space is contracted: Party B tends to be placed to the right of where it is placed in other countries, while Party C is placed to the left. But the takeaway message of Figure 2 is that experts across Europe have very similar understandings of the economic left–right scale, which gives confidence to researchers that expert surveys are reliable instruments for establishing cross-national party placements.

Discussion

It is for all intents and purposes impossible to describe political preferences of parties without referring to “positions” on a variety of policy domains. Analysts are in need of reliable measures of party positions that incorporate information on party behavior beyond the strategic manifesto documents. For scholars interested in coalition formation and durability, for example, the rhetorical positioning of political parties as presented in election manifestos may be of less value than other measures of party positioning that incorporate a party’s words and deeds. Expert surveys provide just this type of data, and this article increases our confidence in the cross-national comparability of an ongoing expert survey with extensive temporal, geographic, and party coverage. Significantly, the anchoring vignettes included in the 2010 wave of the CHES allow us to more confidently assert the cross-national comparability of expert evaluations of political parties in European democracies.

We have shown that there is very little DIF in the 2010 CHES data. While we find that experts in some countries conceive of left–right in slightly different ways than experts in other countries, this is largely limited to Bulgaria, Greece, Latvia, and Norway. Future research could further probe the strikingly high levels of cross-country comparability by examining possible expert-level DIF that may produce overly centrist or otherwise biased placements. Nevertheless, the present findings already bear important implications for the expert surveys. Users of expert data interested in modeling party positions should be sensitive to differences in country contexts, and future attempts to collect data on party positions using expert judgments would benefit from including anchoring vignettes which allow for correcting possible country-specific biases.

Without the vignette analysis, we did not know whether cross-national expert surveys, in general, and the CHES on party positioning, in particular, were in fact comparable across countries, because we were not certain that the experts of these various countries used a similar conception of the left/right dimension when placing the parties of their respective systems. Our findings indicate that despite concerns of comparability across different countries and cultures, a strikingly large number of the experts use left–right in a similar way.

It is important to remember that in economic left–right we are dealing with one of the most fundamental concepts and dimensions in political science. In contrast, there may be considerably less agreement in how experts understand other dimensions such as European integration. Nevertheless, we view this as an important step forward in the ongoing project of establishing expert surveys as a rigorous instrument for measuring party positions over time and across a variety of countries.

Footnotes

Declaration of conflicting interest

The authors declare that there is no conflict of interest.

Funding

This research received no specific grant from any funding agency in the public,commercial,or not-for-profit sectors.

Supplementary material

The replication files are available at:

References

Armstrong

II (2010) factorplot: Improving presentation of simple contrasts in GLMS. Working paper. Available at: http://www.quantoid.net/factorplot.pdf (accessed 16 August 2011).

Armstrong

II (2011) factorplot. Software. Available at: http://r-forge.r-project.org/R/?group_id=1111 (accessed 16 August 2011).

Bakker

de Vries

Edwards

. (2014) Measuring party positions in Europe: The Chapel Hill Expert Survey trend file, 1999–2010. Party Politics Published online before print November 29, 2012, doi: 10.1177/1354068812462931.

Benoit

Laver

(2006) Party Policy in Modern Democracies. London: Routledge.

Benoit

Laver

(2007) Estimating party policy positions: Comparing expert surveys and hand-coded content analysis. Electoral Studies 26: 90–107.

Budge

(2000) Expert judgements of party policy positions: Uses and limitations in political research. European Journal of Political Research 37(1): 103–113.

Curini

(2010) Experts’ political preferences and their impact on ideological bias: An unfolding analysis based on a Benoit-Laver expert survey. Party Politics 16(3): 299–321.

Downs

(1957) An Economic Theory of Democracy. New York: Addison Wesley.

Gemenis

Van Ham

(2014) Comparing methods for estimating parties’ positions in voting advice applications. In: Garzia

Marschall

(eds) Matching Voters with Parties and Candidates: Voting Advice Applications in a Comparative Perspective. Colchester: ECPR Press, pp.33–47.

10.

Hix

Høyland

(2013) Empowerment of the European Parliament. Annual Review of Political Science 16: 171–189.

11.

Hooghe

Bakker

Brigevich

. (2010) Reliability and validity of measuring party positions: The Chapel Hill Expert Surveys of 2002 and 2006. European Journal of Political Research 49(5): 687–703.

12.

King

Wand

(2007) Comparing incomparable survey responses: Evaluating and selecting anchoring vignettes. Political Analysis 15(1): 46–66.

13.

King

Murray

CJL

Salmon

. (2004) Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review 98(1): 191–207.

14.

Kruskal

(1958) Ordinal measures of association. Journal of the American Statistical Association 53(284): 814–861.

15.

McDonald

Mendes

Kim

(2007) Cross-temporal and cross-national comparisons of party left-right positions. Electoral Studies 26(1): 62–75.

16.

McElroy

Benoit

(2010) Party policy and group affiliation in the European Parliament. British Journal of Political Science 40(2): 377–398.

17.

McElroy

Benoit

(2012) Policy positioning in the European parliament. European Union Politics 13(1): 150–167.

18.

Marks

Hooghe

Steenbergen

. (2007) Cross-validating data on party positioning on European integration. Electoral Studies 26(1): 23–38.

19.

Netjes

Binnema

(2007) The salience of the European integration issue: Three data sources compared. Electoral Studies 26(1): 39–49.

20.

Ray

(1999) Measuring party orientation towards European integration: Results from an expert survey. European Journal of Political Research 36(6): 283–306.

21.

Rohrschneider

Whitefield

(2009) Understanding cleavages in party systems: Issue position and issue salience in 13 postcommunist democracies. Comparative Political Studies 42(2): 280–313.

22.

Steenbergen

Marks

(2007) Evaluating expert surveys. European Journal of Political Research 46(3): 347–366.

23.

Whitefield

Vachudova

Steenbergen

. (2007) Do expert surveys produce consistent estimates of party stances on European integration? Comparing expert surveys in the difficult case of central and eastern Europe. Electoral Studies 26(1): 50–61.