Abstract
Keywords
Theoretical Background
Societal impact is crucial in competitive research developed internationally in the last decade. The most competitive research agencies consider societal impact as a critical element in the funding of their projects. In this scenario, the importance of establishing permanent research networks between experimental and social sciences to achieve societal impact is growing. Interdisciplinary research combining all sciences is a primary point for reaching societal impact. The European Commission, for instance, funds its projects based on three fundamental pillars: excellence, impact and quality and efficiency of the implementation (Flecha et al., 2015; European Council, 2019). Suppose projects submitted to EC agencies do not demonstrate that they will have a societal impact in their development (in-itinere), especially at their completion (expost). In that case, they will not be funded.
Concerning the second element, impact, the projects presented must demonstrate how they contribute to each of the expected outcomes and impacts mentioned in the strategic plan and the work programs under the relevant topic. The impacts are clearly framed in the benefits that society can obtain from the results of the projects, beyond academia and clearly differentiated from the dissemination of research results (Reale et al., 2018).
We conceptualize the societal impact of research when not only do we go beyond academia, but the results of any research published in scientific journals, disseminated, and transferred improve people’s lives by following the social objectives prioritized by citizens (Aiello et al., 2021). Internationally, these objectives are related to the 17 Sustainable Development Goals (United Nations, 2017); Along the same lines, the International Year of Basic Science and Social Development has been working on this year, 2022, and the next one. The ONU has launched this initiative, developed based on the SDG, encouraging exchanges between scientists and all stakeholders (including associations, students, and local authorities). 1
Therefore, social impact implies the prior dissemination of research results among the international scientific community. This prior step makes it possible for policy makers and the public to be aware of them. Secondly, it implies a transfer of the knowledge that has been disseminated, which occurs when policymakers and other social actors use it for social and political actions (Girbés-Peco, et al., 2022).
Thus, showing the future societal impact that we will have with our research (ex-ante societal impact) is fundamental if we want to develop competitive research projects. This simple fact has caused research staff to seek strategies to ensure societal impact in developing their research proposals. If a research group or research staff demonstrates to have obtained societal impact with the development of their research, they have options to get national and international competitive funding. Therefore, orienting methodologies towards societal impact measurement is an increasingly relevant and challenging factor. In that sense, Bellavista et al. (2022) demonstrate how methodologies to engage citizens are crucial to advancing societal impact measurement.
With this scenario on the horizon, it is essential to have tools that allow us to evaluate/measure the societal impact of the research projects we carry out. Establishing these tools or strategies to measure the societal impact depends mainly on the procedures we follow in our research to obtain the impact. In other words, the methodology and how we approach the measurement of societal impact becomes a critical aspect if we want to ensure a positive impact on citizens.
The European Commission has financed research projects, such as IMPACT-EV, which make it possible to evaluate and measure the social impact of research (Sordé Martí, et al., 2020). At first, the interest was particular since the objective was to have a system of indicators that would allow discerning which projects submitted to calls of the Framework Program could have more social impact. But these indicators have also served to try to measure the possible societal impact of a project in a more general way. This particular interest has been extrapolated, and many researchers have taken these indicators and methods to achieve societal impact for their research proposals (Reale et al., 2018). At the same time, researchers are also taking into account, along with these indicators and methods for obtaining societal impact, the achievement of the SDGs with research projects, as some funding agencies in European countries include the obligation to reflect the impact of the research projects concerning the SDGs.
The European Commission established a group of experts led by Dr Flecha, main research of IMPACT-EV project, which produced a report containing different ways of assessing the social impact in all areas of knowledge. The group of experts, after several working meetings, published the report “
Two essential elements are noted in the report on conceptualizing societal impact and its measurement. Firstly, societal impact is understood outside academia. We cannot speak of societal impact if we fail to impact outside the academic world positively. Secondly, the importance of two concepts, replicability, and sustainability of the societal impact, can be achieved as a result of a project. Project results acquire greater relevance when they can be replicated in other scenarios and can be sustainable over time, i.e., beyond the funding received for the research project development.
Both notions are directly linked to measuring societal impact in the short, medium, and long term. The four (KIP) are presented in tables that provide basic information on methods to be applied to obtain societal impact. They are structured according to the short, medium, and long term. What interests us, both the competent bodies that finance research, as well as the research personnel and the public, is to obtain societal impact in the long term. In this sense, replicability and sustainability are fundamental since they are obtained once the project has concluded and can be replicable in different contexts (which can also be in the medium term) and sustainable years after the project ends. These (KIP) have been the base of the final tool to evaluate the societal impact through the Social Impact Ex-post Evaluation protocol (SIEP) (Gómez & Soler-Gallart, 2022).
Study Aims and Objectives
The main objective of this study is to develop a tool to evaluate the societal impact of the research qualitatively. To establish the tool, the SIEP, developed under a communicative perspective, was established to guide the result towards measuring societal impact with a transformative orientation.
The research questions that emerged and guided the work were: Can we measure societal impact qualitatively? How do we qualitatively measure societal impact? Would it be possible to develop a tool that helps in this task? Would a communicative approach in developing this tool help us in its final realization?
To answer these questions, we set out three interrelated objectives: - To collect and systematize available evidence on the societal impact of research. - To apply a protocol to obtain a tool to measure the societal impact of research. - To analyze the societal impact of research projects.
Explanation and Justification of Method
This protocol was developed using communicative methodology (Flecha & Soler, 2014). We apply this methodology because it is oriented to social transformation and several projects of the research framework program that obtained results of great societal impact used this methodology in their developments (Aiello, et al., 2021).
The communicative methodology is based on seven postulates 3 that focus on reaching the interpretation of reality through an egalitarian and intersubjective dialogue. (Gómez et al., 2006). The dialogues established in the different work sessions have respected the different positions of each participant, and each participant has been conceptualized as a transforming social agent (Puigvert et al., 2012). Hierarchical interpretations were broken by acting on the same epistemological level through communicative rationality. By proceeding in this way, knowledge was achieved in a dialogical way, and the resulting tool is the result of a genuinely egalitarian work oriented to understanding among people from very different disciplines (Roca et al., 2022).
Within the application of the communicative methodology, the communicative evaluation of social impact (CESI) has been kept in mind since it is a conceptualization totally in line with the methodology adopted for this study and oriented to evaluate the societal impact of the research. CESI implies, among other things, a participatory process involving all participants of the research through an egalitarian dialogue that makes it possible to recreate the results in different contexts (Puigvert et al., 2019). This development fits in with the approach we wanted to provide to our study since the goal lies in the application of the tool by researchers to measure the societal impact of their research works.
The communicative methodology is characterized by three elements that differentiate it from other methodologies oriented toward social transformation. Research using communicative methodology is organized communicatively, applies communicative data collection techniques, and analyzes the information communicatively. The communicative organization of the research implies promoting the participation of all the people involved in the research process from the beginning to the end through egalitarian and inter-subjective dialogue. Some of the strategies used in previous research, such as INCLUD-ED, have been establishing an advisory board, organizing groups of experts, and working groups on specific topics (Yuste et al., 2014). In the specific case of this research, as we will explain in the section on data analysis and research organization, a group of experts was organized and met from the beginning to the end of the process to define the societal impact analysis tool.
The communicative data collection techniques are the communicative discussion group (Rué et al., 2014), communicative daily life story (Ruiz-Eugenio et al., 2020), and communicative observation (Aubert et al., 2011). The common point of all of them lies in the researcher’s role. Researchers bring to the egalitarian dialogue with the participants the accumulated theory on the problem under study. In this way, anyone without prior theoretical knowledge can link these data with their experiences and reflections.
In the specific case of this research, all the work meetings were carried out under a communicative orientation. Theoretical data and practical examples of societal impact assessment were provided and discussed from the point of view of egalitarian dialogue. No traditional data collection techniques were applied, nor was it necessary to select a sample for the development of the protocol. Both, the protocol, and the tool to measure the societal and political impact of a research, has been carried out by all members of the research team.
Sampling/Recruitment
Researchers Taking Part in the Study.
The recruitment process was totally intentional and based on the research team. The tool has been tested by implementing it to research projects, already finished, from the research institutions involved from different disciplines. In fact, its evolution has been a direct consequence of the way in which the tool has been completed in previous versions.
Research Process and Data Analysis
Project Data.
The communicative analysis of the information was carried out from the first provisional chart to measure the societal impact on the final one. Following the communicative analysis of the information, the data were processed based on the exclusionary and transformative dimensions. The exclusionary dimension is conceptualized as all those barriers that make it difficult to overcome problems. On the other hand, the transformative dimension is understood as all those actions that help to overcome these barriers (Pulido et al., 2014).
Addressing Global Challenges.
aTo access the SDGs visit: https://sdgs.un.org/es/goals
bPreferably scientific articles or, if not, documentation from official agencies. This orientation is maintained for all times when evidence/sources of data are requested.
The research was organized in a communicative way (Munte et al., 2011), with an expert group formed by all the project researchers. All interactions were based on an egalitarian and inter-subjective dialogue, where decisions were made by consensus and always based on the best arguments. In this process, the seven postulates of the communicative methodology were followed, achieving objectivity through this inter-subjective dialogue (Aiello & Sorde-Marti, 2021).
Citizen Involvement.
Supporting Policy Making.
This first working document was tested by testing the information on the societal impact of a 6th Framework Program project that had achieved very positive results in its development. Some shortcomings were detected in the charts explaining the societal impact. At the second meeting, the four charts were presented with the example already filled in, and the specific problems encountered by the team that prepared the first one was discussed. The main problem was the level of abstraction of the charts, as they were directly related to the four KIPs. Without controlling the primary bibliography on societal impact and having worked on the subject concretely in research projects, it did not seem very quickly able to successfully use this future tool for people who did not have this previous work base. The tool has always worked to be simple in its application so that people not so well versed in the subject can apply it to get a concrete idea of whether their project has had a societal impact.
At the end of the second meeting, the chart was modified, and it was decided to check it using projects carried out by members of the research team. Thus, the form was tested through seven research projects in engineering, ecology, social sciences, and palaeontology. Two ad hoc meetings were held between the people who developed the first chart and those working on the engineering and environmental projects when the records were applied. Despite the improvements introduced in the second meeting, new doubts arose when assessing the projects' societal impact using the charts.
The main barriers to the completion of the chart focused on three issues: the interpretation of some headings, which were still too abstracted to be applied; the duplication of some information in two of the four tables that were part of the chart; the interpretation of citizenship and its involvement in the development of projects when these are in areas such as engineering.
After these doubts were raised in a small committee, a third meeting was held to close the societal impact measurement chart definitively. With the seven completed charts from different disciplines, the tool was discussed for the last time and finally closed and applied to a final social science project. Below, we present the tool agreed upon after the SIEP’s application.
Tool to Measure Societal Impact
This section presents the tool developed from the Social Impact ex-post Evaluation Protocol (SIEP). It is a tool to collect, as systematically as possible, the societal impact achieved by research conducted in different areas of knowledge. It consists of two sections. The first provides the essential data identifying the project. The second consists of data and evidence on the processes leading to the project’s societal impact. This second section includes three tables, which are based in the “Key Impact Pathways” defined in the report “Monitoring the Impact of EU Framework Programmes” to achieve societal impact of the research. Keeping these pathways in mind before the research is carried out can significantly help the subsequent achievement of societal impact; they can be used to establish possible indicators to measure the future societal impact that can be achieved with research.
Each of these pathways differentiates between scientific knowledge achieved and how it is transferred and has societal impact on citizens. This way of collecting the results of a research project corresponds to its measurement in the short, medium, and long term. First, the scientific knowledge achieved is collected and how it is transferred to the public, which citizens can also use for their benefit. The information collected can then be included in repositories designed to show the societal impact of scientific research (Table 2).
This first table includes the title of the project with its area of knowledge, funding agency, and project duration. This minimal information would be essential if we want a project to be considered for inclusion in an institutional repository on societal impact. Previous versions of the data sheet contained more data than the final one, but to be more direct, we reduced it to these four pieces of information.
Societal Impact
Next table provides information on the link between the SDGs and other internationally recognized objectives and the project’s objectives (Table 3). In the theoretical background, we have already stated the importance of responding to objectives set by the citizenry, which show the improvements achieved by the project. The way to demonstrate societal impact is to relate each result obtained to those objectives set by the citizens. It is not enough to obtain our research objectives if they are not oriented to the SDGs or similar ones. Our research must be in line with citizens' social demands since it aims to improve their lives. In this sense, although the SDGs are essential to the survival of society in general, it is always crucial that research reaches citizens in some way, passively and, where possible, actively.
Once the SDGs or other similar ones have been related to those of the project, a double step is taken: 1. Specify whether results have been obtained in line with the SDGs through a brief description of how they have been obtained and their relationship with each of the SDGs. 2. Delve into how the results have been able to be replicable and sustainable over time. This second step usually implies a longer-term societal impact, as it requires the transferability of its results to other contexts and that they can be sustainable over time. If the results of a project endure over time, they acquire greater relevance, and if they can be replicated in a diversity of contexts and areas of knowledge, even more so.
The most important is to provide concise narratives of the results and specially to highlight the issue of transferability by addressing the replicability and sustainability of these results. To support the reports, it is essential to provide evidence and data for each of the results reported. This evidence may include scientific articles or other productions that denote the importance of these results.
The following table shows information on how citizens participate in research (Table 4). This is a crucial process in achieving societal impact and one that many government agencies that fund international projects focus on when providing resources.
After the last meeting, citizen participation was conceptualized comprehensively since projects in areas such as engineering do not have the traditional participation of citizens in developing a product, as is the case in other areas of basic research. Research personnel in areas such as engineering collaborate closely with companies, associations, or foundations that come to represent the agents of society during the creation of a specific product.
The co-construction of knowledge occurs among a great diversity of people. Companies are part of the citizenry since they have workers who are part of it. Therefore, the citizenry tparticipates as potential workers and consumers of the products. That way, there is double participation as an active agent and consumer. As a result of the co-creation work established during the project, the product is gestated and materialized, then marketed by the company. Citizens then participate as consumers of the product. The societal impact achieved with the project will depend on the product’s success if its consumption positively impacts citizens in line with the SDGs.
Thus, this table shows citizen participation broadly understood in three phases or moments: 1. Explaining how citizens have been involved in co-creating knowledge or specific products during the research is necessary. 2. The explanations focus on how companies, entities, or associations have promoted people’s participation at the end of the research, providing data on the participating entities and the type of activities carried out. 3. Data on how people can use the research results through diverse activities are provided.
Therefore, the importance of the stories, and the explanations, falls on two moments, during the research (corresponding to an in-itinere societal impact), at the end of the research (ex-post societal impact), and highlighting how people make the knowledge or the realization of products their own through the organization of activities.
This last table focuses on how to capture the political impact that a project can achieve upon completion (Table 5). A distinction is made between international, national, and regional/local impact since projects can vary in depth and scope. The first step is to explain the concrete political results descriptively achieved by the project and provide supporting data. Secondly, it is a matter of providing data on how political decision-makers have been reached, how they have been made participants in the research results and providing data on the interactions produced. Finally, it is a matter of providing data related to political documents that concretely cite the project results. This last description is directly related to the previous step, the interactions with politicians since these interactions often lead to these political documents.
The information presented in this last table is of great importance since the availability of policy documents based on research results allows us to speak of a longer-term societal impact and to have greater possibilities of transferring the results to broader and more diverse contexts.
Rigor
Rigor is assured following the communicative approach adopted during the entire research process. Through this approach, a collective interpretation of reality is reached based on the best arguments from all the people who participated in the research activities. Rigorous control of all the work sessions and small committee meetings has been carried out, ensuring that the seven postulates of the communicative methodology have complied at all stages.
Rigorousness is also achieved due to the communicative construction of knowledge, where research personnel contribute all their scientific knowledge and make it available to the public. Objectivity is achieved by the egalitarian and inter-subjective dialogue among all participants (Gómez et al., 2006). The diversity of people involved in their areas of expertise managed based on equality has been critical in achieving the tool.
Footnotes
Declaration of Conflicting Interests
Funding
ORCID iDs
