Science needs more external evaluation,not less

Abstract

When science is evaluated by bureaucrats and administrators, it is usually done by quantified performance metrics, for the purpose of economic productivity. Olof Hallonsten criticizes both the means (quantification) and purpose (economization) of such external evaluation. I share the concern that such neoliberal performance metrics are shallow, over-simplified and inaccurate, but differ in how best to oppose this reductionism. Hallonsten proposes to replace quantitative performance metrics with qualitative in-depth evaluation of science, which would keep evaluation internal to scientific communities. I argue that such qualitative internal evaluation will not be enough to challenge current external evaluation since it does little to counteract neoliberal politics, and fails to provide the accountability that science owes the public. To assure that the many worthy purposes of science (i.e. truth, democracy, well-being, justice) are valued and pursued, I argue science needs more and more diverse external evaluation. The diversification of science evaluation can go in many directions: towards both quantified performance metrics and qualitative internal assessments and beyond economic productivity to value science’s broader societal contributions. In addition to administrators and public servants, science evaluators must also include diverse counterpublics of scientists: civil society, journalists, interested lay public and scientists themselves. More diverse external evaluation is perhaps no more accurate than neoliberal quantified metrics, but by valuing the myriad contributions of science and the diversity of its producers and users, it is hopefully less partial and perhaps more just.

Science needs more external evaluation, not less

The value of science is increasingly measured by numbers, from simple publication counts to more complex performance metrics. Olof Hallonsten (2021) encourages us to reject the external evaluation of science which relies on the ‘means of quantitative metrics’ to measure science for the primary purpose of ‘economic growth’. Using the ‘authority of numbers’ to assert public control over professional practices is not limited to the domain of science, but is a general postwar sociological phenomenon seen in many sectors such as education, medicine, social work and accounting (Desrosières, 1998; Espeland and Stevens, 2008; Porter, 1995; Power, 1997). Since the 1990s, performance metrics are increasingly mobilized as neoliberal tools to prioritize economic values (e.g. cost savings, marketization) over other public values (Bruno and Didier, 2013; Desrosières, 2010; Hammer, 2010; Strathern, 2000). I share Hallonsten’s concerns that current neoliberal evaluation of science is shallow, over-simplified and inaccurate, but differ in how best to oppose this reductionism. Hallonsten focuses primarily on the problem of quantification, favoring a more ‘holistic, qualitative and long-sighted’ evaluation of science. He concludes evaluation must be kept ‘internal to science and its disciplinary communities’, since in-depth evaluation can only be done by those with detailed knowledge of its content (Hallonsten, 2021: 16). While content-based peer review is important, it is unlikely to be enough to challenge the economization of science encouraged by neoliberal policies. To assure science pursues a much broader range of public values, such as truth, democracy, well-being and other forms of social, economic and epistemic justice, I argue in favor of a more diverse external evaluation of science: mobilizing quantification as well as a much broader public of interested outsiders, including sociologists of science.

The simplicity of numbers: A flaw in principle, an advantage in practice

Hallonsten argues convincingly that qualitative and situated measures of the actual content of scientific activity will be less shallow and more accurate than quantified performance metrics currently used in external evaluation by administrators and bureaucrats. Unfortunately, more complex and accurate measures may do little to improve scientific evaluation in practice if they are going to be ignored by external parties. Evaluation practices such as those for research funding or university rankings compare very different kinds of science and scientists, and they will continue to favor quantified metrics because numbers provide the commensurability needed to be able to compare research that differs widely in content.

An illustrative example of how the simplicity of quantified metrics is often an advantage in practice, is provided by Teun Zuiderent-Jerak and colleagues (2015). They examined the multiple tools that are available to measure the value of health care, both clinical quality and economic value. Various non-economic ‘valuemeters’ were available to measure clinical quality, however, these were systematically excluded in favor of more simplistic quantified economic measures of care (i.e. price/cost). The authors conclude that ‘a much broader spectrum of values’ can be included in what counts in principle, but was largely considered ‘unworkable’ in practice. In their view, economic measures were privileged because the simplicity of quantification makes them easily integrated into existing tools and ideals of neoliberal governance (i.e. marketization). In short, unless neo-liberal practices of economization are also actively challenged, the production of qualitative, complex and accurate measures of value will not be successful in replacing existing quantified metrics.

The sociology of quantification has investigated the authority of numbers in many contexts, driven not only by the need for commensurability, but also the ‘ancient association of numbers with ideals of rationality and universalism’ (Espeland and Stevens, 2008: 432). Given the power of numbers, it is important sociologists highlight their limitations and partiality. However, numbers are not inherently harmful, nor are they exclusively used to further the neo-liberal goals of economization and marketization. Numbers can also be productively used to advance non-economic collective well-being, such as combating climate change, treating disease, addressing inequality and increasing accountability (Espeland and Stevens, 2008; Bruno et al., 2014). As I see it, the problem with science’s shallow and inaccurate evaluation does not lie with quantification per se, but with the reduction of science’s purpose to the single and narrow purpose of economic growth. Instead of throwing the quantified baby out with the neoliberal bathwater, we can also cultivate forms of ‘statactivism’: the re-appropriation of numbers for emancipatory and transformative purposes (Bruno et al., 2014). By using the power of quantification, we may be more successful in disrupting existing external evaluation of science. While quantification is no guarantee for success, quantified metrics make it easier for non-economic valuemeters to be built into current evaluation practices, allowing scholarship to be valued not just for ‘productivity’, but for its contributions to more complex societal benefits. Instead of rejecting quantified evaluation, sociologists of science may join ‘the project of making numbers’ to produce better forms of quantified evaluation (Berman and Hirschman, 2018: 258), including much needed evaluative tools to audit the auditors.

This leads us to the next, much bigger question: can we ever hope to measure the great diversity of public values that science may pursue, such as truth, democracy, justice or well-being, in a simple manner?

How to value public values?

How might we measure science’s value, beyond simply quantifying productivity such as publication counts or dollars of funding? How may we conceptualize, achieve and measure science’s contribution to complex – and contradictory – public values such as truth, well-being, equality or justice? We cannot expect any simple answers, as valuation is ‘a crucial problem for the social sciences and the humanities today’, with the flourishing new field of valuation studies starting to document many complex social practices of valuing things and objectifying values (Board of Editors, 2020:1; Lamont, 2012). Hallonsten suggests science can simply be ‘left to govern itself’, believing ‘spectacular results and findings that [will] benefit society in a wide variety of ways’ will follow (Hallonsten, 2021: 11). His trust in the benefits of science is based on science’s historical track-record, expressed in the rhetorical question: ‘How else do you suppose that we have achieved this level of wealth and technical standard in Europe and North America?’ This question raises several issues.

First, it reinforces the economic purpose of science, even limiting it to wealth creation for Europe and North America, thus ignoring the negative impacts that both technology and economic growth in the ‘Global North’ have had on people in the rest of the world and on the non-human world. Second, the answer is not science, but colonialism and racial capitalism supported and legitimized by science. The ‘overwhelming logic’ of this question, then, is revealing indeed. If the intent is to convince the reader that science will automatically have positive societal impacts, it has the opposite effect. It reminds us that modern science has always been at the service of wealth accumulation for Europe and North America, with or without neo-liberal quantified performance metrics (Haskaj, 2018). The role of science in colonial and capitalist wealth accumulation is a powerful argument in favor of increased external evaluation of science. If science is to be valued for its potential to promote non-economic values such as democratization and equality, it must be held accountable for the way scientific facts and technological artifacts may be implicated in the marginalization of people and destruction of non-human life forms.

From turning inwards to opening up

To assure that science’s contribution is not limited to economic growth for Europe and North America, we must welcome increased external participation in science. Since many different humans (and non-humans) stand to gain or lose from the products of science, science’s interested public is extremely diverse. Historically, modern science excluded, dehumanized or otherwise harmed women, people of color and the working class. Today, science remains a fairly exclusive endeavor as these groups remain underrepresented in science. Lack of diversity can create blind spots in which the assumptions, beliefs and interests of a majority are mistaken for universal logic and objective truths. Not just for the sake of democratization as a goal in itself, but to effectively assure science produces more than what is narrowly defined as economic growth, science must be subject to more – not less – external evaluation.

By more external evaluation, I do not mean more evaluation by administrators such as ‘politicians and bureaucrats’ who currently claim to represent the public. Such external evaluation fails to restore public trust in science, as it evaluates only a single public value of short-term efficiency and productivity. Many are critical of such neoliberal performance metrics, and together they can engage in a ‘counter-politics of data’ that aims to transform, interfere with or hijack numbers for their own purpose (Beraldo and Milan, 2019). Such ‘counterpublics’ may include activists, civil society, journalists, lay citizens as well as scientists and scholars from different disciplines that present different perspectives (Hess, 2011). In contrast to the shallowness of quantified metrics, these diverse counter-publics can create ‘hybrid forums’ that rely on both expertise and public deliberation to achieve in-depth and ongoing public evaluation of science’s impact (Callon et al., 2001). Ultimately, it is science itself that is responsible for (re)gaining the trust of a wider public, by allowing critical evaluation of science, engaging seriously with public concerns, and being willing to act upon them (Wynne, 2006).

Conclusion

The current external evaluation of science is oversimplified, as a result of both quantification and economization. To overcome this reductionism, it won’t be enough to provide more complex and qualitative evaluation by scientists themselves. Content-based evaluation by peers is important, but is unlikely to effectively counteract the economization of science evaluation. If we are to take on the enormity of the challenge – both political and practical – of evaluating the diverse and contradictory ‘societal values’ that science may produce, social scientists must go beyond criticizing existing evaluation, and start participating in the critical composition of more heterogeneous valuemeters (Zuiderent-Jerak et al., 2015). The evaluation of science can be diversified and opened up in many directions. By supporting both formalized performance metrics and qualitative, narrative-based and informal assessments. By measuring economic productivity as well as science’s contribution to much broader ethical purposes and societal values. And in addition to evaluation by administrators and public servants, we can involve social scientists, civil society, journalists, the interested ‘lay’ public and scientists themselves in the evaluation of science. Valuation of the myriad contributions of science and the diversity of its producers and users is perhaps no more accurate than current quantified performance metrics, but it is hopefully less partial and more just.