Should We or Should We Not Include Confidence Intervals in COVID-19 Death Forecasting? Evidence from a Survey Experiment

Abstract

Forecasting during the COVID-19 pandemic entails a great deal of uncertainty. The same way that we would like electoral forecasters to systematically include their confidence intervals to account for such uncertainty, we assume that COVID-19-related forecasts should follow that norm. Based on literature on negative bias, we may expect the presence of uncertainty to affect citizens’ attitudes and behaviours, which would in turn have major implications on how we should present these sensitive forecasts. In this research we present the main findings of a survey experiment where citizens were exposed to a projection of the total number of deaths. We manipulated the exclusion (and inclusion) of graphically depicted confidence intervals in order to isolate the average causal effect of uncertainty. Our results show that accounting for uncertainty does not change (1) citizens’ perceptions of projections’ reliability, nor does it affect (2) their support for preventive public health measures. We conclude by discussing the implications of our findings.

Introduction

Citizens are exposed to forecasting on a wide variety of matters: sports competitions (which country will win the world cup?), cultural shows (which artist or band will win Eurovision?), economic conditions (what is the predicted government deficit?), elections (which party will form the government?) and so on. The forecasts that we have seen during the COVID-19 crisis are, however, quite unusual. The vast majority of countries around the world are not used to such forecasting, which aims to predict, among others, the number of positive cases, hospitalizations and deaths due to a disease in their homeland.

Despite the fact that citizens were exposed to many such projections in the early stages of the pandemic, ¹ especially those projecting the number of positive cases and deaths, little research has examined how these forecast models were presented and, ultimately how these presentations affect citizens. In terms of how the data are shown, the focus has been on the axis scale and, more specifically, on the linear versus logarithmic presentation of the numbers. On one hand, Sevi et al. (2020) found that, among Canadians surveyed in early April, the scale did not change citizens’ support for preventive public health measures, despite the fact that the logarithmic trend appears to be more dramatic (although it presents the exact same numbers). On the other hand, Romano et al. (2020) found that changing this scale altered US citizens’ preferences for some preventive measures. For example, the group exposed to the linear numbers were more favourable to the idea that nonessential businesses should remain closed. Moreover, Ryan and Evers (2020) found in April that US citizens were more supportive of COVID-19 preventive public health measures, such as social distancing or the use of a face mask, when exposed to linear instead of logarithmic scales. Thus, our knowledge on the presentation of the data, and especially forecasting, remains overall very limited – especially when compared to other topics such as the impact of different messages and framing that can affect citizens’ attitudes and behaviours (Ghio et al., 2020; Utych, 2020; van der Linden and Savoie, 2020). The research focuses on the scale of the graph axis as the main predictor of citizens’ preferences and behaviour intentions (and the results are conflicting).

However, following Holmdahl and Buckee (2020), we believe that the core aspect in the presentation of COVID-19-related predictions pertains to uncertainty. Several predictive models were developed, but they all entail a great deal of uncertainty (for a review, see Wynants et al., 2020). The upper and lower bounds of a projection are a crucial feature of a forecast because they allow citizens to have a sense of the worst- and best-case scenarios instead of only the average prediction. We argue that accounting for such uncertainty is valuable and that it is preferable to include uncertainty rather than to exclude it. In the prediction of important matters, such as electoral forecasts, we expect scholars, polling firms, public organizations and news media to include confidence intervals (CIs) – see Pétry and Bastien (2013) for an evaluation of this practice.

The reasons for graphically including CIs are simple. First, they provide a more comprehensive and accurate picture of the forecast. We prefer this to what would be at best an incomplete picture or, at worst, a potentially misleading depiction of the situation. Using an experimental design on a US population, Podkul et al. (2020) showed that ‘while individuals are unable or unwilling to accurately calculate exponential growth, a majority of them are able to understand the trend when paired with a graphic’. In other words, graphical representations can help people to better understand data. Therefore, graphically including uncertainty in the form of CIs can facilitate the individual’s understanding of the best- and worst-case scenarios that the intervals represent.

Second, the presence or absence of uncertainty could potentially affect citizens’ attitudes and behaviours. More specifically, negative bias, understood as a tendency to pay more attention to negative information (such as a worst-case scenario), could alter one’s perceptions of reality, their attitudes and behaviours. ² Including uncertainty in forecasting does precisely this: it presents the most negative information within a certain margin of error (the lower limit of the CI), which is more likely to be salient in citizens’ minds compared to the average or the most positive prediction.

Holmdahl and Buckee (2020: 303) recently argued that how uncertainty is displayed is a key question in model results. This is important to consider as we improve our understanding of forecasting. We will aim to operationalize this argument and examine the potential impact of uncertainty on citizens’ attitudes and perceptions. In our research, we use an experimental research design to manipulate the presence (or absence) of uncertainty in COVID-19 death projections and measure its impact on citizens’ perceptions of the reliability of the forecast and their support for preventive public health measures. We systematically fail to find any statistically significant or substantially important effect of the treatment on two outcomes, that is, the perceived reliability of the prediction and attitudes towards preventive public health measures. We conclude by discussing the implications of these results and highlighting the need to better understand what is going on in citizens’ understandings of COVID-19 forecasting.

Methodology

We make use of an online survey experiment, conducted in Canada by the polling firm Léger using a quota-based approach to collect a nationally representative sample (based on age, gender, region, language and education) of 1002 respondents. They took the survey between 16 and 24 June 2020. The experiment was embedded in the Canadian survey that is part of the Citizens’ Attitudes under COVID-19 Pandemic comparative project led by the CEVIPOF (Science Po Paris).

The treatment of interest manipulates uncertainty in the forecast. We manipulate this uncertainty by randomly including or omitting CIs in a graph predicting the number of deaths in Canada. Figure 1 shows the predictions based on data from the Institute for Health Metrics and Evaluation, an independent research institute part of the University of Washington. We should note that it is particularly important that the uncertainty is visually shown, and not simply mentioned in text (lower and upper limits), which makes it easier for citizens to understand since we know that they have limited understanding of how forecasting works – see Lavrakas et al. (1991), Traugott and Kang (2000) and most importantly Podkul et al. (2020) for a study on COVID-19.

Figure 1.

The (a) Absence and the (b) Presence of Uncertainty in COVID-19 Predictions in Canada

Immediately following the prediction, on the same page, respondents were asked: ‘Personally, how reliable do you think this projection is?’ Answer choices were very reliable, somewhat reliable, not very reliable and not at all reliable. We coded this on a scale from 0 (not at all reliable) to 1 (very reliable). We excluded the ‘don’t know’ and ‘prefer not to answer’ responses ³ and the mean score was 0.54.

On the very next page, respondents were asked about their support for different preventive public health measures. This question assessed how favourable respondents were to the following policies:

• The closure of day care centres, CPEs, schools, CEGEPs and universities

• The government’s recommendation to wear a mask in public places

• Prohibition of gatherings of more than 10 people

• The use of cell phones to control movements

• Closing borders to foreigners

• Testing citizens for COVID-19 in a systematic way

• Giving $1000 fines to those who do not comply with containment guidelines

• Postponing the elections

Answer choices were very unfavourable, somewhat unfavourable, neither favourable nor unfavourable, somewhat favourable, very favourable and prefer not to answer (excluded). ⁴ We know that the vast majority of Canadians were favourable to the aforementioned preventive public health measures and that they vastly complied with them (Daoust et al., 2020; Rheault and Musulan, 2020). The sum of the items was recoded on a scale ranging from 0 to 1, where greater scores mean more favourable, and the mean is 0.68. Complete distribution of these two dependent variables is shown in Figure 2.

Figure 2.

Distributions of the Dependent Variables: (a) Reliability and (b) Attitudes Towards Preventive Measures.

Findings and Implications

To examine the treatment effect, that is, accounting for uncertainty by including CIs, we show the mean for each of the dependent variables based on whether the CI were present or absent. It is clear from simply glancing at Figure 3 that we fail to find a substantial treatment effect on respondents’ perceptions of how reliable they find the COVID-19 death-related prediction. The same applies to their attitudes towards preventive public health measures.

Figure 3.

Treatment Effects on (a) Reliability and (b) Attitudes Towards Preventive Measures.

We believe that these results are very surprising. We should expect, based on what we know about people’s limited understanding of quantitative forecasting (thus being more likely to be influenced by visual information vs numbers) and most importantly its negative bias, that allowing for uncertainty (and a worst-case scenario) would increase one’s perception of reliability and boost their support and level of compliance with preventive public health measures. Moreover, our research design should allow us to see these differences emerge at the bivariate level, without covariates. However, Figure 3 is clear: there is no discernible treatment effect.

It is possible that the effect of uncertainty is conditioned by important variables such as age, sex or education (Daoust, 2020; Galasso et al., 2020; Ryan and Evers, 2020). We thus ran ordinary least squares (OLS) regressions with interaction terms between the treatment effect and these potential moderators, for both dependent variables. We also included covariates for sociodemographics (on top of the main effect of the one interacted with the treatment). The regressions can be found in Table A1 of the Online Appendix, and we illustrate the effects in Figure 4. Again, the results were very clear – and puzzling. None of the six interactions reaches the conventional level of statistical significance (p < 0.05 or even p < 0.10) or had a substantial effect on both the perception of reliability and attitudes towards preventive public health measures.

Figure 4.

Conditional Effects of Uncertainty: (a) Reliability and (b) Support Preventive Measures.

Ultimately, our experiment was designed to detect a potential causal effect of uncertainty (using CIs) in COVID-19 death predictions on citizens’ opinions about the reliability of the projection and their attitudes towards public health measures. Based on the literature on negative bias, we expected the presence of uncertainty, through the visual cue of a worst-case scenario, to affect how favourable citizens are to preventive public health measures. However, we find neither a main effect of the treatment on any of the outcome variables, nor a conditional impact.

Despite the relevance of displaying uncertainty, as argued by Holmdahl and Buckee (2020), it is not clear that such representations impact citizens’ assessments of a projection’s reliability or support for preventive public health measures. Whereas some scholars found consequences for the type of scale (logarithmic or linear) used in graphs (Romano et al., 2020; Ryan and Evers, 2020), citizens seem to be less sensitive to the appearance of CIs in COVID-19 forecasting. This might be because participants in our experiments were already familiar with similar projections and were thus ‘pre-treated’ (Druckman and Leeper, 2012). However, the fact that the research cited above found substantial effects in a context where citizens were much more exposed makes us sceptical about that possibility. Moreover, our survey was conducted in a context where life was somewhat normal – between two waves of COVID-19 – compared to the early Spring 2020 lockdown period and where the media were much less inclined to show citizens forecasting, which makes a ‘pre-treatment’ effect even less likely.

Our research is based on one snapshot in a particular context, that is, the Canadian experience in June 2020. While we cannot claim that our case is representative of the other highly developed, democratic countries, we do know that Canada’s response to COVID-19 was a quite typical one on many features. ⁵ Moreover, we do not benefit from cross-country comparisons. However, we can make use of the within-Canada variation as it is a diverse country with more than 20% of the population for whom French is their mother tongue. Further analyses show that there is no difference between anglophone and francophone – or with people who live in Quebec (where the vast majority of francophone people live) versus the rest of Canada. ⁶ As the world continues to experience the effects of COVID-19 with the ‘second wave’ that is hitting quite badly as we are writing these lines, it will be increasingly important to understand how uncertainty and forecasting can be used to encourage compliance with public health measures and ensure that citizens can rely on new data. Among others, it would be useful to have an experimental setting where the forecast would be more pessimistic (compared to our design where the death increase was quite rapidly reaching a plateau), use different measures of the COVID-19 burden such as positive cases or hospitalization (instead of deaths), and of course, would take place in different countries.

Supplemental Material

sj-pdf-1-psw-10.1177_1478929920985686 – Supplemental material for Should We or Should We Not Include Confidence Intervals in COVID-19 Death Forecasting? Evidence from a Survey Experiment

Supplemental material, sj-pdf-1-psw-10.1177_1478929920985686 for Should We or Should We Not Include Confidence Intervals in COVID-19 Death Forecasting? Evidence from a Survey Experiment by Jean-François Daoust and Frédérick Bastien in Political Studies Review