Sage Journals: Discover world-class research

Abstract

In project portfolio selection, the aim is to choose projects which are expected to offer most value and satisfy relevant risk and other constraints. In this study, we show that uncertainties about how much value the projects will offer, combined with the fact that only a subset of the proposed projects will be selected, lead to inaccurate risk estimates about the aggregate value provided by the selected project portfolio. In particular, when downside risks are measured in terms of lower percentiles of the distribution of portfolio value, these risk estimates will exhibit a systematic bias. For deriving unbiased risk estimates, we present a calibration framework in which the required calibration can be presented in closed‐form in some cases or, more generally, derived by using Monte Carlo simulation to study a large number of project selection decisions. We also show that when the decision must comply with risk constraints, the introduction of tighter (more demanding) risk constraints can counterintuitively aggravate the underestimation of risks. Finally, we present how the calibrated risk estimates can be employed to align the portfolio with the decision maker's risk preferences while eliminating systematic biases in risk estimates.

Keywords

project portfolio selection research and development optimization simulation

Introduction

Project portfolio selection is a decision problem (Salo et al. 2011) faced by many organizations which carry out activities through projects with the aim of attaining an appropriate balance of cost, reward, and risk (see Kavadias and Chao 2007). This problem is faced, for instance, by high technology companies which launch R&D projects to create new products; municipalities which carry out maintenance and repair projects to ensure the quality of built infrastructures (Mild and Salo 2009); and research councils which select research projects that generate new knowledge and contribute to economic growth and societal well‐being.

In all these problems, the decision maker (DM) seeks to maximize the value that can be gained by carrying out a subset of available project proposals subject to relevant constraints, most notably the limited availability of resources. Once completed, each project offers some value to the DM. If there are no synergies or cannibalization effects among the projects, the resulting portfolio will yield an aggregate value which is the sum of values provided by the selected projects. Typically, however, these projects’ ex post values are not known at the time the projects are selected. Rather, the DM must select projects based on ex ante value estimates which contain estimation errors and therefore differ from the values that will be actually realized. In particular, at the time of project selection, those projects whose values have been overestimated are more likely to be selected. As a result, the estimated portfolio value, obtained by summing the ex ante value estimates, tends to be higher than the realized portfolio value, causing the phenomenon of post‐decision disappointment known as the optimizer's curse (see, e.g., Brown 1974, Harrison and March 1984, Hobbs and Hepenstal 1989, Smith and Winkler 2006, Vilkkumaa et al. 2014).

To date, the optimizer's curse phenomenon has been studied by examining the impacts on the expected value (Smith and Winkler 2006). In this study, we show that the use of value estimates with random estimation errors has major impacts on the estimation and mitigation of risks in project portfolio selection. Overall, the accurate estimation of risks is important for many reasons: for instance, the DMs may have to establish a risk reserve whose size depends on the estimated risk level of the projects that are implemented (see e.g., Lenzi 2012). Here, the overestimation and underestimation of risks are both problematic: overestimation may lead to an oversized risk reserve so that less resources remain for project implementation; but underestimation will lead to an insufficient risk reserve so that the DM will be exposed to risks that are greater than what was anticipated when making the decision.

Having unbiased risk estimates is particularly important when the selected project portfolio has to comply with risk constraints. For instance, there may be a requirement that selected lower percentiles of the distribution of the portfolio value do not fall below predefined bounds (e.g., a threshold level below which the portfolio value must not be with a probability of 5% or more; for the purposes of this study, we call this the 5% VaR level) (Best 1999). The 5% VaR level can be understood in two ways, namely, it provides (i) a threshold level below which the value of the project portfolio will be with a probability of 5% and (ii) a complementary probability 100% ‐ 5% = 95% for obtaining a portfolio value which exceeds the threshold. Thus, for instance if the 5% VaR threshold level is too high and thus overestimated (i.e., there is a higher than 5% probability that the portfolio value is below this estimated threshold), the complementary 95% probability will be underestimated.

To our knowledge, Flyvbjerg (2006) is the only study on the causes of and remedies to inaccuracies in risk estimates about projects. He notes that the underestimation in risk estimates can be attributed to optimism bias and strategic misrepresentation. In our study, we show that even if the project‐specific biases associated with optimism and strategic misrepresentation are corrected so that the estimates about projects’ values are unbiased, risk estimates about the value of the project portfolio will still be systematically biased. We also show that systematic biases in VaR estimates depend on whether projects are selected in the presence of risk constraints. In particular, we show that the introduction of risk constraints does reduce risks, as one would expect. Yet, it can lower the estimated VaR level even more, which can lead to a misplaced belief that risks are under better control than what they actually are.

As a remedy, we propose alternative approaches to the calibration of risk estimates. In situations where there is an extensive record of earlier project portfolio selection processes, guidance for the required calibration can be derived from careful analyses of historical data about these processes. If not, it may be possible to characterize the key parameters of the portfolio selection problem and to use these parameters in Monte Carlo simulations to determine the appropriate calibration. In particular, we outline a portfolio selection process in which risk estimates are explicitly calibrated to derive revised estimates which do not exhibit systematic biases and thus help the DM select a project portfolio whose value is aligned with the stated risk preferences.

We contribute to the theory and practice on project portfolio selection in several ways. First, we show that risk estimates can be biased. This should be of much concern to the DM who may have to set aside reserves depending on the estimated risk level and whose decision may have to comply with risk constraints. Second, we propose approaches to the calibration of risk estimates, which help the DM select a project portfolio that is aligned with the stated risk preferences. We also illustrate this approach with a realistic example.

The study is structured as follows. Section 2 reviews the relevant literature. Section 3 describes the project portfolio selection problem and explains how conventional but biased downside risk estimates can be debiased through calibration. In section 4, we introduce risk constraints and show how risk estimates can be calibrated in the presence of constraints. Section 5 discusses the calibration of the estimates when the DM's risk aversion is captured via an exponential utility function. Section 6 illustrates the approaches of sections 3 and 4 in the context of a case study. Concluding remarks are in section 7.

Literature Review

Our work is closely related to two streams of literature. The first deals with the optimizer's curse, i.e., the expected post‐decision disappointment on the value of a selected project when the selection is made based on noisy value estimates. This phenomenon was identified by Brown (1974) and later formalized by Harrison and March (1984). Hobbs and Hepenstal (1989) analyzed this phenomenon in water resources optimization problems calling it optimistically biased optimization. They suggest that the phenomenon can be addressed by explicitly modeling the posterior distributions of the project values. Following this approach, Smith and Winkler (2006) provide a closed‐form solution for removing the optimizer's curse when the prior distribution on projects’ values is multivariate normal and the distribution of value estimates conditioned on the actual project values is also multivariate normal. In the context of these assumptions, Vilkkumaa et al. (2014) develop a framework to help the DM to identify which proposals it is optimal to re‐evaluate when resources for these additional evaluations are limited.

The second stream considers the assessment and management of risks in the selection of a portfolio of alternatives. Roy (1952) pioneered research in this area, stressing that risk and uncertainty are not the same. Instead, he notes that the risk of a portfolio occurs when the outcome is less than expected, i.e., when downside risks do occur. Subsequently, downside risk measures have become widely employed in portfolio problems (Eppen et al. 1989, Hall et al. 2015, Menezes et al. 1980, Nawrocki 1999, Sortino and van der Meer 1991). Frequently, downside risk constraints are included in portfolio selection problems due to regulations or risk budgeting (Baule 2014, Kubo et al. 2005, Stamatelatos and Dezfuli 2011). It is also possible to assess downside risk in order to anticipate and prepare for risky outcomes. For example, the Washington State Department of Transportation requires that risk estimates are employed when allocating reserve funds to the projects that are started (Lenzi 2012). The Department uses lower quantiles of the distribution of the project values to assess the risks of financial underperformance. In many contexts, quantiles such as the lower 5th percentile of the portfolio value, are employed as the downside risk measure (see e.g., Batur and Choobineh 2010, Best 1999, LaGattuta et al. 2001). Focusing on the worst 5% portfolio value is relevant, because unlike the more frequent minor losses, the less likely but more significant losses can have major negative impacts on the organization.

Our work builds on these two streams by investigating what biases the optimizer's curse causes in (i) estimating the downside risk of the selected project portfolio and (ii) introducing risk constraints in project portfolio selection. We also propose remedies to overcome the biases that are caused by the optimizer's curse.

Downside Risk Estimation in Resource Constrained Project Portfolio Selection

Project Portfolio Selection Setting

There are n project proposals i = 1, …, n from which the DM can select a project portfolio. The selection decisions are encoded by the binary variables z = [z ₁, …, z _n]^⊤ such that z _i = 1 if and only if the i‐th project is selected. The DM can select those subsets (portfolios) which satisfy the resource and other relevant constraints. These portfolios form the set of feasible portfolios Z = {z ¹, …, z ^f}, $f \in Z^{+}$ . The DM selects projects on the basis of their value estimates $v_{i}^{E} \in R$ which can be elicited, for instance, by consulting experts. For each project, this estimate is a scalar measure of the total value that the project will provide ex post if it is carried out. After the project has been completed, its value $v_{i} \in R$ will be realized. This value is often different from the value estimate due to the estimation error $ϵ_{i} \in R$ caused by uncertainties: for example, when selecting R&D projects, there are typically major uncertainties about technical and commercial success of the projects (Huchzermeier and Loch 2001). While the value estimates and values are here treated as scalars, they can nevertheless be composed by aggregating multiple criteria, see e.g., Baker and Olaleye (2013).

The real valued random variables for project values, value estimates, and estimation errors we represent by capital letters V = [V ₁, …, V _n], $V^{E} = [V_{1}^{E}, \dots, V_{n}^{E}]$ , and $E = [E_{1}, \dots, E_{n}]$ , respectively. The relationship between the realized value estimates, values, and estimation errors of these variables is $v^{E} = v + ϵ,$ where v ^E, v, and $ϵ$ are n‐dimensional column vectors. When the DM selects projects, he knows only the realized value estimates v ^E. Once the projects have been completed, then, the project values v are realized and observed (Arnott et al. 2009), see Figure 1. The realized estimation errors, $ϵ$ , represent the difference between the estimated and actual values so that $ϵ = v^{E} - v$ .

Figure 1

Project Portfolio Selection Problem

We assume that the estimates are conditionally unbiased so that the mean of each estimation error $μ_{i}^{E}$ is zero given the actual project values v _i, i.e., $μ^{E} | v = 0$ .¹ Covariances of the estimation errors $Σ^{E}$ are assumed to be known, and they can be acquired, for instance, by evaluating past portfolio selection processes or by consulting experts. Similarly, the DM is assumed to know the means μ and the covariances Σ of the distribution of the projects’ values. Such information can be derived by examining past portfolio selection problems or relying on experts (Bansal et al. 2016), for instance. Figure 1 summarizes the underlying assumptions about the distributions of estimation errors and project values in the portfolio selection problem.

Information about the correlation between project values and estimation errors is contained in their covariance matrices Σ and $Σ^{E}$ , respectively. Project values would be correlated, for example, if the portfolio consists of R&D projects targeted for the same market, because then the value of each project would depend on the size of this market. The estimation errors, in turn, would be correlated if they are based on the same underlying uncertain assumptions about the market size.

Conventional but Biased Portfolio Value and Downside Risk Estimate

The conventional (but biased) decision analytic approach to estimate the value and downside risk of the selected project portfolio is to use the observed projects’ value estimates v ^E directly without revising these estimates in view of the fact that the selected portfolio is the one with the highest sum of value estimates (Harrison and March 1984, Hobbs and Hepenstal 1989, Smith and Winkler 2006, Vilkkumaa et al. 2014). Specifically, in this approach, the DM seeks to maximize the value of the project portfolio by relying on estimates about how much value the projects will offer once completed. Thus, the selected portfolio z ^E is the one that maximizes the estimated portfolio value ${z^{E}}^{⊤} v^{E} = Σ_{i = 1}^{n} z_{i}^{E} v_{i}^{E}$ so that $z^{E} = \underset{z \in Z}{argmax} z^{⊤} v^{E} .$

The downside risk of the selected portfolio can be obtained by computing the portfolio value that corresponds to a given VaR level ρ _α, α < 0.5. This can be computed from the equation $P [V_{P}^{E} \leq ρ_{α}] = α$ , where $V_{P}^{E} = {z^{E}}^{⊤} (v^{E} - E)$ , by analogy to Equation 1, is the random variable for the value of the selected portfolio z ^E based on Equation 2. To solve ρ _α, we define the normalized portfolio value using the random variable $φ = \frac{V_{P}^{E} - E [V_{P}^{E}]}{σ_{V_{P}^{E}}}$ , which has zero mean with variance normalized to one, $\begin{matrix} P [V_{P}^{E} \leq ρ_{α}] = & α ⟺ P [φ \leq \frac{ρ_{α} - E [V_{P}^{E}]}{σ_{V_{P}^{E}}}] \\ = & α ⟺ G_{V_{P}^{E}} (\frac{ρ_{α} - E [V_{P}^{E}]}{σ_{V_{P}^{E}}}) \\ = & α ⟺ ρ_{α} = E [V_{P}^{E}] + G_{V_{P}^{E}}^{- 1} (α) σ_{V_{P}^{E}}, \end{matrix}$ where $G_{V_{P}^{E}} (.)$ is the cumulative probability distribution function of the normalized value of the selected project portfolio. Note that the derivation of Equation 3 for the downside risk estimate is correct. However, the random variable for the estimated value of the portfolio $V_{P}^{E}$ is biased (and therefore each term $E [V_{P}^{E}]$ , $G_{V_{P}^{E}}^{- 1} (α)$ , and $σ_{V_{P}^{E}}$ ) unless it is estimated by conditioning on the fact that the selected portfolio has the highest sum of value estimates.

The conventional decision analysis approach ignores this conditioning so that (i) the expected value of the selected project portfolio $E [V_{P}^{E}]$ is assessed directly from projects’ estimated values v ^E for the selected portfolio z ^E and (ii) the random variable for the value of the selected project portfolio is assessed directly from estimation errors $E$ . But then the selected portfolio is treated as any feasible portfolio without recognizing the fact that these estimates are the ones for which estimated portfolio value is maximized so that the random error terms $E$ for the selected projects tend to have an upward bias. In this case, the cumulative probability distribution function of the normalized value of the selected project portfolio $G_{V_{P}^{E}}^{- 1} (α)$ and the standard deviation of the selected project portfolio $σ_{V_{P}^{E}}$ are taken to be $G_{{z^{E}}^{⊤} E}^{- 1} (α)$ and $σ_{{z^{E}}^{⊤} E}$ , respectively. Consequently, the estimated downside percentile for the selected portfolio, using the conventional approach becomes $ρ_{α}^{E} \equiv {z^{E}}^{⊤} v^{E} + G_{{z^{E}}^{⊤} E}^{- 1} (α) σ_{{z^{E}}^{⊤} E} .$ Proposition 1 gives an estimate for the standard deviation of the selected portfolio value $σ_{{z^{E}}^{⊤} E}$ in Equation 4 when the estimation errors for the projects in z ^E are not conditioned on the fact that z ^ϵ yields the highest estimated value.

Proposition 1

The estimated standard deviation of the value of the selected portfolio z ^E is $σ_{{z^{E}}^{⊤} E} = \sqrt{\sum_{i = 1}^{n} {z_{i}^{E}}^{2} σ_{i i}^{E} + \sum_{i, j = 1, i \neq j}^{n} z_{i}^{E} z_{j}^{E} σ_{i j}^{E}} = \sqrt{{z^{E}}^{⊤} Σ^{E} z^{E}},$ where $σ_{i i}^{E}$ and $σ_{i j}^{E}$ are the variance and covariance of estimate errors, respectively.

The proof is similar to that of calculating standard deviation for a portfolio of stocks (Luenberger 1998), except that the decision variables for projects are binary instead of continuous as is the case for stocks.

The inverse of the standardized cumulative probability distribution of the selected portfolio value at the α‐percentile $G_{{z^{E}}^{⊤} E}^{- 1} (α)$ in Equation 4 can be computed numerically when the normalized portfolio value $G_{{z^{E}}^{⊤} E}^{- 1} (α)$ follows the normal cumulative distribution function $Φ^{- 1} (α) = \frac{1}{\sqrt{2 π}} \int_{- \infty}^{α} e^{- t^{2} / 2} d t$ . Conservative approximation for $G_{{z^{E}}^{⊤} E}^{- 1} (α)$ when the probability distribution for ${z^{E}}^{⊤} E$ is not known is $G_{{z^{E}}^{⊤} E}^{- 1} (α) \approx - \sqrt{\frac{1 - α}{α}}$ and when ${z^{E}}^{⊤} E$ is only known to be symmetric is $G_{{z^{E}}^{⊤} E}^{- 1} (α) \approx - \sqrt{\frac{1}{2 α}}$ (Bonami and Lejeune 2009).

A concern in the above approaches for estimating the portfolio value and its downside risk is that value estimates for the selected projects given in z ^E are biased due to the optimizer's curse (Smith and Winkler 2006). Even if the expected estimation errors are zero for all projects, the estimation error is likely to be positive for the higher valued projects which are selected resulting in the ex post disappointment on average.

Obtaining Unbiased (Calibrated) Portfolio Value and Downside Risk Estimate

Under some conditions, the unbiased downside risk estimates can be established in a closed‐form using the Bayesian approach. This is the case when the prior distributions on the project values are normally distributed, i.e., V ∼ N( μ , Σ), and the value estimates given the actual project values are normally distributed as well, i.e., $V^{E} | v \sim N (v, Σ^{E})$ (or equivalently $E | v \sim N (0, Σ^{E})$ ). Under these conditions, the unconditional distributions for project value estimates $V^{E}$ and the posterior distribution on actual values given value estimates V|v ^E are (Carlin and Louis 2000, p. 90–91, Smith and Winkler 2006) $V^{E} \sim N (μ, Σ + Σ^{E}) and V | v^{E} \sim N (v^{B}, Σ^{B}), where$ $v^{B} = β v^{E} + (I - β) μ,$ $Σ^{B} = (I - β) Σ, and$ $β = Σ {[Σ + Σ^{E}]}^{- 1} .$ Therefore, the unbiased or calibrated value estimates are the posterior means of project values given value estimates, i.e., v ^B in Equation 7. The portfolio selection problem using the unbiased values estimates is otherwise same as in Equation 2, except that v ^E is replaced by v ^B and its solution is denoted as z ^B. The updated unbiased value estimate of the selected portfolio is thus ${z^{B}}^{⊤} v^{B}$ . The unbiased downside risk estimate for the selected portfolio is obtained using Equation 3 where the estimated portfolio value distribution $V_{P}^{E}$ is replaced by the posterior distribution on actual values given value estimates V|v ^E. Therefore the unbiased downside risk is $ρ_{α}^{B} \equiv {z^{B}}^{⊤} v^{B} + Φ^{- 1} (α) σ_{{z^{B}}^{⊤} V | v^{E}} .$ The standard deviation of the selected portfolio value $σ_{{z^{B}}^{⊤} V | v^{E}}$ in Equation 10 can be computed, using Equation 5 where the covariance matrix of estimation errors $Σ^{E}$ is replaced by Σ ^B, in Equation 8, and z ^E is replaced by z ^B. Closed‐form solutions for unbiased value estimates and downside risk estimate can be given when both the value and conditional estimate error distributions follow normal, log‐normal, or beta distributions (Carlin and Louis 2000, Miller 2010), for example.

Because closed‐form solutions are not always available for deriving unbiased value and risk estimates, we propose a two‐step simulation approach to determine the appropriate calibration for deriving unbiased estimates. Calibration has been applied to subjective probability estimates in many instances (see e.g., Johnson and Bruce 2001, Lichtenstein et al. 1981). The first step in the simulation‐based calibration approach is to obtain unbiased value estimates for selecting projects which maximize the expected value of the project portfolio. The unbiased value estimates are the expected project values conditioned on the realized value estimates, i.e., $v^{B} = E [V | v^{E}]$ . We can approximate V|v ^E by simulating project values and their value estimates and retaining only those instances in which all value estimates are within δ (a small positive real number) away from the given estimates v ^E. If the project values and estimation errors are not correlated, the unbiased estimates can be obtained for each project independently via $v_{i}^{B} = E [V_{i} | v_{i}^{E}]$ , where $V_{i} | v_{i}^{E}$ can be approximated via simulations. Using the unbiased value estimates v ^B and Equation 2, the optimal project portfolio z ^B can be selected.

The second step in the simulation‐based calibration approach is to obtain unbiased estimate for the downside risk. When the projects are of same type, so that their values follow the same probability distribution and their estimation errors are identically distributed, the unbiased estimate for the downside risk is $ρ_{α}^{B} = Q_{α} [{z^{B}}^{⊤} V | v^{s}]$ , where $Q_{α} [.]$ denotes the value of a random variable at the α‐percentile and v ^s = ${z^{B}}^{⊤} v^{B}$ denotes the estimated value of the selected portfolio. The term ${z^{B}}^{⊤}$ V|v ^s can be approximated through the repeated simulation of project selection instances of which only those portfolio values are retained in which the estimated value of the selected portfolio differs from v ^s by less than δ. These retained values provide the distribution of the actual value of the selected portfolio so that the α‐percentile can be taken. When the distributions for project values and estimation errors are different for different projects, the unbiased estimate for the downside risk is $ρ_{α}^{B} = Q_{α} [{z^{B}}^{⊤} V | v^{E}]$ . Thereby, the distribution ${z^{B}}^{⊤}$ V|v ^E can be simulated by conditioning on estimated project values while otherwise applying the same approach as in the case in which the projects are of the same type. We define the required calibration at the α‐percentile as follows $C_{α} \equiv ρ_{α}^{B} - ρ_{α}^{E} .$

This calibration technique can be used to derive an unbiased risk estimate in any percentile. The required calibration can be estimated, in principle, either by examining past project portfolio selection data or by simulating the portfolio selection process. If historical data is used, the accuracy depends on the quality and quantity of available data. If the simulation procedure is used, the parameters of the selection problem (most notably value and error distributions, the number of proposals, and the proportion of projects being selected) need to be specified. When selecting projects, the last two parameters of the selection problem are known whilst the distributions of project values and estimation errors may not be fully known but can be evaluated from historical data or by relying on expert judgments.

Calibration in Expected Terms

We first investigate the expected calibration in portfolio value and downside risk (measured in the worst 5th percentile of the distribution of the portfolio value). We consider a setup where both the project value and conditional estimate error distributions are normally distributed so that the unbiased portfolio value and risk can be computed, using the closed‐form equations from section 3.3. We compute results by simulating the selection problem 500,000 times so that in each simulated trial we compute (i) the estimated portfolio value ${z^{E}}^{⊤}$ v ^E, (ii) the unbiased portfolio value ${z^{E}}^{⊤}$ v ^B, (iii) the estimated downside risk $ρ_{α}^{E}$ , and (iv) the unbiased downside risk ${z^{E}}^{⊤} v^{B} + Φ^{- 1} (α) σ_{{z^{E}}^{⊤} V | v^{E}}$ . In each trial, the required calibration for the portfolio value and its downside risk are computed as the difference between their unbiased and estimated values, i.e., for the portfolio value this is ${z^{E}}^{⊤}$ v ^B − ${z^{E}}^{⊤}$ v ^E and for the downside risk this is ${z^{E}}^{⊤} v^{B} + Φ^{- 1} (α) σ_{{z^{E}}^{⊤} V | v^{E}} - ρ_{α}^{E}$ . By averaging these calibrations across all simulated trials, we obtain the calibrations for the expected portfolio value and risk.

In Table 1, we compare the expected calibrations when one large project is selected and when a portfolio of two small projects is selected. In order to make these selection problems comparable, the means of the project values are μ _i = 10/(the number of project proposals to select) so that the expected value of a randomly selected portfolio is 10. Also, the coefficient of variation, i.e., the ratio of standard deviation and the mean, is the same across all problems. This is achieved by setting $\sqrt{σ_{i i}} / μ_{i} = 0.2$ where the variance σ _ii is solved after substituting μ _i = 10/(the number of project proposals to select). We apply the same variance for the estimation error, i.e., $σ_{i i}^{E} = σ_{i i}$ .

Table 1

The Expected Calibration When 1 Large Project or 2 Small Projects are Selected

	Expected calibration
	1 large project selected V ∼ N(μ, Σ),μ _i = 10, $σ_{i j} = 2^{2}$ , $σ_{i j} = 0$ , i ≠ j $E \| v \sim N (0, Σ^{E}), σ_{i i}^{E} = 2^{2}, σ_{i j}^{E} = 0, i \neq j$		2 small projects selected V ∼ N(μ, Σ), μ _i = 5, $σ_{i j} = 1^{2}$ , $σ_{i j} = 0$ , i ≠ j $E \| v \sim N (0, Σ^{E}), σ_{i i}^{E} = 1^{2}, σ_{i j}^{E} = 0, i \neq j$
Number of proposals	Value	5th percentile value	Value	5th percentile value
3	−1.20	−0.23	−0.60	0.08
4	−1.46	−0.49	−0.94	−0.26
5	−1.65	−0.68	−1.17	−0.49
6	−1.79	−0.83	−1.35	−0.67
7	−1.91	−0.95	−1.49	−0.81
8	−2.01	−1.05	−1.61	−0.93
9	−2.10	−1.14	−1.71	−1.03
10	−2.18	−1.21	−1.80	−1.12

We make three observations from Table 1. First, the expected calibrations of portfolio value and downside risk are less, in absolute terms, when two smaller projects are selected instead of one large project. This result is intuitive, because the expected bias due to the optimizer's curse is largest for the project with the highest estimated value. Thus, the expected bias in selecting the single (highest valued) large project is greater than the expected bias in selecting the two (highest and the next highest valued) small projects. Second, in expected terms, the portfolio value at the 5th percentile may need to be calibrated upward (see the selection of two small projects out of three proposals) or downward, in contrast to the portfolio value that needs to be calibrated only downward (Smith and Winkler 2006). Third, the expected calibration behaves so that the smaller the share of selected projects (shown in Table 1 by increasing the number of proposals), the less the amount of required calibration. For the expected calibration of the portfolio value, this implies that the magnitude of the downward calibration increases when fewer projects are selected. For the expected calibration of the portfolio value at the 5th percentile, this implies that the possible upward calibration (see the selection of two small projects out of three proposals) changes to downward calibration, and the magnitude of this calibration increases when the proportion of the selected projects decreases.

The results in Table 2 highlight that the expected calibrations are different for the expected value of the portfolio and its downside risk. For example, if the correlation among project values increases, the expected calibration of the portfolio value decreases while it can cause in a non‐monotonic change in the expected calibration of the downside risk, see e.g., when the correlation among estimate errors is 0 or 0.25. Also, if the estimate errors are perfectly correlated, then the expected calibration for the portfolio value is zero, whereas the expected calibration for the downside risk is significant.

Table 2

The Expected Calibration as a Function of Pair‐Wise Correlation in Estimation Errors and in Value When 1 Project is Selected from 4 Proposals

Expected calibration for the value
$V \sim N (μ, Σ), μ_{i} = 10, σ_{i j} = ξ_{i j} σ_{i} σ_{j}, σ_{i} = 2, i, j = 1, \dots, 4 E \| v \sim N (0, Σ^{E}), σ_{i j} = ξ_{i j}^{E} σ_{i}^{E} σ_{j}^{E}, σ_{i}^{E} = 2, i, j = 1, \dots, 4$
Equicorrelation among values (ξ _ii = 1, ξ _ij, i = j)	Equicorrelation among estimate errors $(ξ_{i i}^{E} = 1, ξ_{i j}^{E}, i \neq j)$
Equicorrelation among values (ξ _ii = 1, ξ _ij, i = j)	0	0.25	0.5	0.75	1
0	−1.46	−1.17	−0.84	−0.46	0.00
0.25	−1.56	−1.26	−0.92	−0.52	0.00
0.5	−1.68	−1.38	−1.03	−0.59	0.00
0.75	−1.84	−1.55	−1.19	−0.73	0.00
1	−2.06	−1.78	−1.45	−1.03	0.00

Expected calibration for the 5th percentile value
$V \sim N (μ, Σ), μ_{i} = 10, σ_{i j} = ξ_{i j} σ_{i} σ_{j}, σ_{i} = 2, i, j = 1, \dots 4 E \| v \sim N (0, Σ^{E}), σ_{i j} = ξ_{i j}^{E} σ_{i}^{E} σ_{j}^{E}, σ_{i}^{E} = 2, i, j = 1, \dots, 4$
Equicorrelation among values (ξ _ii = 1, ξ _ij, i = j)	Equicorrelation among estimate errors $(ξ_{i i}^{E} = 1, ξ_{i j}^{E}, i \neq j)$
Equicorrelation among values (ξ _ii = 1, ξ _ij, i = j)	0	0.25	0.5	0.75	1
0	−0.49	−0.16	0.29	0.91	1.81
0.25	−0.55	−0.30	0.09	0.63	1.48
0.5	−0.54	−0.38	−0.07	0.42	1.25
0.75	−0.47	−0.40	−0.18	0.24	1.09
1	−0.24	−0.31	−0.21	0.06	0.96

Calibration in a Single Problem Instance

Faced with observed value estimates, the DM needs to decide how much to calibrate. To illustrate the simulation‐based calibration procedure in this case, we assume that the DM has the value estimates in Table 3 for 40 project proposals from which he can choose 10 projects. Based on past experience or consultation with experts, the DM knows that the project values are normally distributed V ∼ N( μ , Σ), μ _i = 10, σ _ii = 2², σ _ij = 0, i ≠ j, i,j = 1, …, 40 and that estimation errors conditional on actual values are normally distributed $E | v \sim N (0, Σ^{E})$ , $σ_{i i}^{E} = 2^{2}$ , $σ_{i j}^{E} = 0$ , i ≠ j, i, j = 1, …, 40. When the DM relies directly on the value estimates in selecting projects to maximizes the estimated portfolio value, the projects numbered from one to ten are chosen. The estimated value of this selected portfolio is v ^s = ${z^{E}}^{⊤}$ v ^E = 125. The conventional 5th percentile downside risk estimate for the portfolio value using Equation 4 when the standard deviation of the portfolio value is computed using Proposition 1 is $ρ_{0.05}^{E} = 125 - 1.645 \times \sqrt{10 \times 4} \approx 115$ .

Table 3

Sample Projects

	Value	Unbiased value	i	Value	Unbiased value	i	Value	Unbiased value	i	Value	Unbiased value
i	Estimate (v^E)	Estimate (v^B)	i	Estimate (v^B)	Estimate (v^E)	i	Estimate (v^E)	Estimate (v^B)	i	Estimate (v^E)	Estimate (v^B)
1	13.9	12.0	11	11.6	10.8	21	10.5	10.0	31	7.8	8.9
2	13.4	11.7	12	11.5	10.8	22	10.1	10.0	32	7.6	8.8
3	12.8	11.4	13	11.5	10.7	23	9.4	9.8	33	7.6	8.8
4	12.6	11.3	14	11.3	10.7	24	9.3	9.8	34	7.5	8.8
5	12.6	11.3	15	11.3	10.6	25	9.1	9.7	35	7.3	8.6
6	12.3	11.2	16	11.3	10.6	26	8.8	9.5	36	7.0	8.5
7	12.2	11.1	17	11.2	10.6	27	8.6	9.5	37	6.9	8.5
8	11.9	11.0	18	10.8	10.4	28	8.2	9.3	38	6.7	8.3
9	11.7	10.9	19	10.6	10.3	29	8.0	9.3	39	6.1	8.1
10	11.6	10.8	20	10.5	10.3	30	8.0	9.2	40	5.5	7.7

In the simulation‐based calibration approach, the first step is to obtain unbiased value estimates for the projects $v_{i}^{B} = E [V_{i} | v_{i}^{E}]$ , shown in Table 3. We ran 1,000,000 simulations with δ = 0.005 for each estimated project value; each simulation run took about one second. These and other simulation procedures we programmed in Python and used a 64‐bit computer with Intel® Core™ i7‐4650U CPU 1.7GHz and 2.3 GHz with 8GB of memory. In maximizing the value of the project portfolio using the unbiased value estimates, the projects numbered from one to ten remain optimal. The value of the optimally selected z ^B project portfolio using the estimates v ^E is v ^s = ${z^{B}}^{⊤}$ v ^E = 125. Conditional on v ^s = 125, the unbiased risk estimate $ρ_{α}^{B} = Q_{α} [{z^{B}}^{⊤} V | v^{s}]$ can be simulated as explained in Section 3.3. We ran 1,000,000 simulations with δ = 0.005 in about one second. These simulations included about 50,000 instances in which the portfolio value distribution was conditional on v ^s, i.e., |S| ≈ 50,000. This distribution is shown in Figure 2. From this distribution, we obtain the unbiased 5th percentile value, i.e., $ρ_{α}^{B} = Q_{0.05} [{z^{B}}^{⊤} V | v^{s} = 125] \approx 105$ as shown by the vertical dashed line. The required calibration is therefore C _0.05 = 105 − 115 = −10.

Figure 2

Example of Calibration

The calibration is different for different realizations of estimated portfolio value. In any selection problem, only one set of projects’ value estimates (and therefore estimated portfolio value) is realized, but when the similar portfolio selection problem is faced repeatedly, then different estimates will be realized. Figure 3 illustrates the conventional but biased downside risk estimates and the unbiased downside risk estimates as a function of possible realizations for the estimated portfolio values on the y‐axis. In general, the higher the realized estimated portfolio value, the more the estimated 5th percentile portfolio value needs to be reduced. Figure 3 also shows that the required calibration needs to be determined separately for different correlation structures. For example, when the pairwise value correlation is 0.5, as shown in Figure 3b, the required calibration for the 5th percentile value given v ^s = 125, is C _0.05 = 112 − 101 = −18 instead of −10 for the case in which the project values are independent.

Figure 3

The Estimated 5th Percentile Portfolio Value (Dashed Line) and the 5th Percentile Portfolio Value (Solid Line), Pair‐Wise Correlation among Values is (a) 0, (b) 0.5, (c) 1

Figure 4 complements the results in Figure 3 by illustrating that the required calibration depends on how many proposals there are and how many of these are selected. The portfolio values in these results are made comparable following the approach described in section 3.3.1 so that in all problems the expected value of a randomly selected portfolio is 100 and the coefficient of variation is 0.2.

Figure 4

The Estimated 5th Percentile Portfolio Value (Dashed Line) and the 5th Percentile Portfolio Value (Solid Line) When the Ratio between the Number of Project Proposals to Select from and the Number of Project Proposals is (a) 10/40, (b) 30/40, (c) 1/4, (d) 3/4

Downside Risk Constrained Project Portfolio Selection

We next consider the calibration of risk estimates when the project selection has to fulfil constraints on downside risk. In practice, such constraints may be required due to regulations on allowed risk levels or agreed risk budgets (Baule 2014, Kubo et al. 2005). The DM may also impose a risk constraint when the realization of a risky outcome of the portfolio selection problem would cause severe harm to the organization.

The conventional downside risk and resource constrained project portfolio optimization problem (that is subject to the optimizer's curse) takes the form of a chance constrained optimization problem as follows $\begin{matrix} z^{E} = & \underset{z \in Z}{argmax} z^{⊤} v^{E} \\ z^{⊤} v^{E} + G_{z^{⊤} E}^{- 1} (α) σ_{z^{⊤} E} \geq r_{α} . \end{matrix}$

Value and Estimation Error Distributions Are Identical

When the marginal distributions for the project values and estimation errors are identical, the calibration of the risk estimate can be included in the resource and risk constrained project portfolio selection process as follows:

Obtain unbiased value estimates for projects v ^B using the closed‐form Equation 10 or simulation described in section 3.3. Solve the optimal portfolio z ^B using Equation 2 when v ^E are replaced by v ^B.

Compute the unbiased downside risk estimate $ρ_{α}^{B}$ for the selected portfolio z ^B using the closed‐form Equation 10 or simulation procedure described in section 3.3.

If $ρ_{α}^{B} \geq r_{α}$ , then z ^B is the optimal portfolio. If $ρ_{α}^{B} < r_{α}$ , then remove from the selected projects z ^B the project with the lowest estimated value by setting its binary indicator to 0. If the project portfolio is empty, i.e., z ^B = 0, then a feasible project portfolio cannot be formed. Otherwise go to step 2.

We consider the example in section 3.3.2 shown in Table 3. The calibrated risk estimate is $ρ_{0.05}^{B} = 105$ . If the DM's risk constraint r _0.05 is less than 105, the optimal portfolio is to select the first 10 projects. If the risk constraint is more than 105, a feasible project portfolio cannot be formed. This is because removing projects from the portfolio will decrease the lower 5th percentile value of the portfolio.

However, if this selection problem occurs repeatedly in the same context (i.e., in each selection problem with realized estimates and values, projects have the same value and estimation error distributions), we can investigate how the expected calibration of the downside risk behaves when the risk constraint is tightened and the portfolio is selected using the value estimates. This is illustrated in Figure 5 which shows that a tighter risk constraint increases the probability that the risk constraint is violated and that the portfolio will be empty. In expected terms, a tighter risk constraint reduces both the estimated risk given the selected project portfolio is not empty, i.e., $E_{V^{E}} [ϱ_{0.05}^{E} | Z^{E} \neq 0]$ where $ϱ_{0.05}^{E} \equiv {Z^{E}}^{⊤} V^{E} + G_{{Z^{E}}^{⊤} E}^{- 1} (0.05) σ_{{Z^{E}}^{⊤} E}$ , (dashed line) and the actual calibrated risk given the selected project portfolio is not empty, i.e., $E_{V^{E}} [ϱ_{0.05}^{B} | Z^{E} \neq 0]$ where $ϱ_{0.05}^{B} \equiv {Z^{E}}^{⊤} V^{B} + G_{{Z^{E}}^{⊤} V^{B}}^{- 1} (0.05) σ_{{Z^{E}}^{⊤} V^{B}}$ , (solid line). These impacts correspond to the increase in the expected actual and estimated 5th percentile values of the selected portfolio in Figure 5.

Figure 5

The Expected 5th Percentile Portfolio Value (Solid Line) and its Estimate (Dashed Line) Given the Portfolio is not Empty and Probability of not Investing as a Function of Risk Constraint r _0.05

We formalize the impact of tightening a risk constraint on the expected calibration for the risk estimate at the α‐percentile $E_{V^{E}} [ϱ_{α}^{B} - ϱ_{α}^{E} | Z^{E} \neq 0]$ as follows.

Proposition 2

In a resource constraint project portfolio selection problem, $Σ_{i = 1}^{n} z_{i} = b$ , b ∈ 1, …, n − 1, where the project values and estimation errors are identically distributed, tightening the risk constraint requirement r _α decreases the expected calibration for the risk estimate $E_{V^{E}} [ϱ_{α}^{B} - ϱ_{α}^{E} | Z^{E} \neq 0] .$

Proof is in Appendix.

Proposition 1 implies that a tighter risk constraint (a greater value of r _α) increases the expected portfolio value at the α‐percentile less than its conventional but biased estimate. This is shown in Figure 5, in which the dashed line increases more than the solid line as a result of tightening the risk constraint. This means that a DM who relies on the conventional risk estimate is lead to believe that risks are managed better than what they actually are.

Different Value and Estimation Error Distributions

When the distributions for the project values and estimation errors are not the same, the chance constrained optimization problem in 12 needs to be solved explicitly. To make Equation 12 computationally tractable, we reformulate it as a second‐order cone programming problem $z^{E} = \underset{z \in Z, x, τ}{argmax} z^{⊤} v^{E},$ $z^{⊤} v^{E} + G_{z^{⊤} E}^{- 1} (α) x \geq r_{α},$ $x^{2} \geq \sum_{i = 1}^{n} τ_{i}^{2},$ $τ = L^{⊤} z,$ where $x \in R^{+}$ and $τ \in {R^{+}}^{n \times 1}$ are auxiliary decision variables and $L \in {R^{+}}^{n \times n}$ is the lower triangular matrix obtained from the Cholesky factorization ${L L}^{T} = Σ^{E}$ . For a similar kind of reformulation of the chance constraints, see Weintraub and Abramovich (1995), Novoa et al. (2017). This notwithstanding the proposed formulation uses the value estimates v ^E and random estimation errors $E$ directly for the selected portfolio z ^E and is subject to the optimizer's curse and therefore does not curtail risks accurately.

To overcome the optimizer's curse, we propose that either the unbiased estimates are used (if they are available in a closed form) and then the optimization problem in Equations (13)-(16) can be directly solved or that the risk estimates are calibrated using the simulation approach and the risk constrained portfolio selection problem is solved iteratively. Recall that when the project values are normally distributed, i.e., V ∼ N( μ , Σ), and also the value estimates given their actual values are normally distributed, i.e., $V^{E} | v \sim N (μ, Σ^{E})$ , the unbiased estimates are available in a closed‐form. Then, the optimization problem in (13)-(16) can be solved using (i) v ^B, given in Equation 7, instead of v ^E, (ii) Φ⁻¹(α) instead of $G_{z^{⊤} E}^{- 1} (α)$ , and (iii) Σ ^B, given in Equation 8, instead of $Σ^{E}$ .

When closed‐form solutions are not available for unbiased estimates, we propose the following iterative approach to the calibration of risk estimates and the solution of the risk constrained portfolio selection problem:

Obtain unbiased value estimates for projects v ^B using the simulation approach in section 3.3.

Solve the resource constrained optimization problem 2 when v ^E are replaced by v ^B to find optimal portfolio z ^B.

Compute the unbiased downside risk estimate $ρ_{α}^{B}$ for the selected portfolio z ^B using the simulation procedure described in section 3.3.

If the DM's desired risk level $r_{α} \leq ρ_{α}^{B}$ , then the optimal project portfolio is z ^B. If $r_{α} > ρ_{α}^{B}$ , then remove z ^B from the set of feasible project portfolios, i.e., Z = Z∖z ^B. If the set of feasible project portfolios Z = 0, a feasible project portfolio cannot be formed. Otherwise go to step 2.

We illustrate the impacts of the optimizer's curse on the risk constrained portfolio selection problem with the example in section 3.3.2. The value estimates are in Table 3, except that for projects 1–20 $\sqrt{σ_{i i}^{E}} = 10$ (instead of 2) and $\sqrt{σ_{i i}} = 4$ (instead of 2). Thus, there are both high and low risk projects . In this setting, a tighter risk constraint results in substituting an increasing number of high risk projects with low risk projects, which reduces the risk by increasing the 5th percentile value of the portfolio, as shown in Figure 6. Analogously to Proposition 2, a tighter risk constraint increases the estimated 5th percentile portfolio value (dashed line) more than the actual 5th percentile portfolio value (solid line). Therefore, the tightening of the risk constraint decreases the required calibration (initially a positive value), decreasing the absolute value of required calibration.

Figure 6

The Expected 5th Percentile Portfolio Value (Solid Line) and its Estimate (Dashed Line) as a Function of Risk Constraint r _0.05

Risk‐Return Trade‐off

Often, the DM is interested in knowing how much value he can expect to forgo by imposing a risk constraint or how much more risk he would have to accept to achieve a higher expected value. A common way to investigate trade‐offs between risk and return is to establish the mean‐risk efficient frontier (Markowitz 1952). This frontier contains the Pareto optimal outcomes of an expected utility model where the objective function is a weighted sum of the project portfolio's mean and risk. One approach in deriving the mean‐risk efficient frontier is to solve a series of optimization problems with the chance constrained formulation (13)-(16) starting from a non‐binding risk constraint and to employ in each consecutive optimization problem a tighter risk constraint until the problem becomes infeasible.

We applied this process to obtain the estimated (dashed line) and unbiased (solid line) mean‐risk efficient frontiers for the example in section 4.2 with high and low risk projects, see Figure 7. Besides illustrating trade‐offs between risk and return, Figure 7 shows the required calibration for the mean and the 5th percentile values of the project portfolio as a function of the DM's risk aversion. If the DM focuses on maximizing the value of the project portfolio, marked with crosses, (minimizing the risk of the portfolio, marked with circles), the required calibration for the expected value and the 5th percentile value of the portfolio are −22 and 11, respectively (as compared to 4 and 7 when minimizing the risk). Therefore, due to the optimizer's curse, a risk neutral DM who maximizes the portfolio value overestimates the expected portfolio value and underestimates the 5th percentile value of the portfolio. However, in this example, a risk averse DM who minimizes the risk of the portfolio will underestimate both the expected and the 5th percentile value of the portfolio.

Figure 7

The Mean‐5th Percentile Efficient Frontiers

Notes. Actual calibrated frontier (solid line) and its estimate (dashed line), x: value maximizing strategy and o: risk minimizing strategy.

Risk‐Averse Portfolio Selection Using Exponential Utility Function

As an alternative to constraining the downside risk, the DM's risk attitude can be captured via an exponential utility function $u (V_{P}) = - e^{- a V_{P}}$ (Kirkwood 2004), where a > 0 is the degree of risk aversion and V _P is the value of the selected portfolio. When the value of the portfolio is normally distributed, i.e., V _P ∼ N(μ _P, σ _P), the expected utility is $E [u (V_{P})] = E [- e^{- a V_{P}}] = - e^{- a μ_{P} + \frac{a^{2} σ_{P}^{2}}{2}} .$ We assume that the distributions of project values and error distributions are both normally distributed, i.e., V ∼ N( μ , Σ) and $E | v \sim N (0, Σ^{E})$ . Then, the optimal project portfolio z ^E which maximizes the expected utility $E [u (V_{P})]$ based on the conventional but biased approach (in which the estimate of the expected portfolio value μ _P is z ^⊤ v ^E and the estimate of variance of the portfolio value $σ_{P}^{2}$ is $z^{⊤} Σ^{E} z$ ) is (Markowitz 1991) $z^{E} = \underset{z \in Z}{argmax} z^{⊤} v^{E} - \frac{a}{2} z^{⊤} Σ^{E} z .$ Similarly, the optimal project portfolio z ^B which maximizes the expected utility $E [u (V_{P})]$ based on the unbiased estimates given in Equations 7 and 8, i.e., z ^⊤ v ^B is used for μ _P and z ^⊤ Σ ^B z is used for $σ_{P}^{2}$ , is $z^{B} = \underset{z \in Z}{argmax} z^{⊤} v^{B} - \frac{a}{2} z^{⊤} Σ^{B} z .$

For the example in section 4.2, the results in Figure 8 show that the calibration for the expected utility first decreases (in region 1) and then increases (in region 2) as the risk aversion term a increases. When the risk aversion term a approaches zero (a → 0⁺), then the calibration approaches zero due to both expected utilities, i.e., $E [u ({z^{E}}^{⊤} v^{E})]$ and $E [{z^{B}}^{⊤} v^{B})]$ , approaching negative one. This is because of $- e^{- a μ_{P} + \frac{a^{2} σ_{P}^{2}}{2}}$ tends to −1⁺ when a → 0⁺. In region 1, the DM is nearly risk neutral and the optimal portfolio is to select the high value and high‐risk projects, 1, …, 10 in Table 3. Because the expected values for these projects are overestimated due to the optimizer's curse, the calibration for the utility function is negative and further decreases when the risk aversion term a increases. In region 2, when the risk aversion term a increases, the optimal portfolio includes an ever greater proportion of the lower value and less risky projects, 21, …, 40 from Table 3. Because in this example the estimated values of projects 21–40 are lower than or equal to their prior mean 10, the conventional value estimates for these projects can be expected to be underestimated, so that $E [u ({z^{E}}^{⊤} v^{E})]$ becomes smaller than $E [u ({z^{B}}^{⊤} v^{B})]$ ). Coupled with that the variance of the portfolio is overestimated, i.e., ${z^{E}}^{⊤} Σ^{E} z^{E} > {z^{B}}^{⊤} Σ^{B} z^{B}$ , this makes the expected utility $E [u ({z^{E}}^{⊤} v^{E})]$ smaller relative to $E [u ({z^{B}}^{⊤} v^{B})]$ ). As a result, the negative calibration at the beginning of region 2 increases and changes to positive calibration as the risk aversion term a increases. Therefore, in keeping with the results in section 4.3, this illustrates that the DM with low risk aversion overestimates the utility (coming mainly from the portfolio value) and DMs with high risk aversion underestimate the utility (coming mainly from the portfolio risk.

Figure 8

The Calibration of the Expected Utility as a Function of the Risk Aversion

A Case Study

We next illustrate with a realistic case study how the proposed calibration technique for assessing the downside risk of portfolio value can be employed in risk constrained project portfolio selection. This process is structured into three steps.

The first step in the process is to characterize the portfolio selection setup, including the assessment of the project value and estimation error distributions. In the case study, we consider the selection of a pharmaceutical project portfolio based on the data from Kloeber (2011). In this selection problem, there are 3 projects which are to be selected out of 12 proposals based on estimated expected net present values (E[NPV]). Table 4 shows the project proposals with their estimated E[NPV]s and standard deviations for estimation errors. The estimation errors are independent from each other. The project proposals come from the same pool of projects, and their E[NPV]s are normally distributed with μ = 72 and σ = 65. The values of the project proposals are independent from each other. The estimates are assumed to be conditionally unbiased and normally distributed given actual values. Obtaining the described data about the distributions for projects’ values and estimation errors can be derived from experts’ estimates, for example, Bansal et al. (2016) explain how to do this using experts’ quantile judgments.

Table 4

Sample Projects

Project	Disease	Estimated E[NPV]	Standard deviation of estimation error
1	Antibiotic	222	432
2	Osteoarthritis	190	397
3	Depression	110	221
4	Chronic pain	101	215
5	Dry eye	89	197
6	HIV/AIDS	79	130
7	High cholesterol	45	35
8	Lung cancer	40	44
9	Asthma	31	6
10	Schizophrenia	22	2
11	Anxiety	12	1
12	Antibiotic	10	6

The second step is to evaluate the trade‐offs between the expected return of the portfolio and risk. We have derived the actual calibrated mean‐5th percentile efficient frontier (using unbiased value estimates obtained from Equations (7)-(9)) and its estimate (using the value estimates directly) for the portfolio selection problem in Figure 9. This figure illustrates that if value estimates are used directly both risk and the expected value of the portfolio are significantly overestimated. In fact, a risk neutral portfolio of projects 1–3 require the 5th percentile portfolio value estimate to be calibrated upward by (−509.3 − (+44.2))/−509.3 ≈ 110%. The overall range for the required calibration is from 110% (risk neutral) to 0% (risk averse) depending on the level of risk aversion. Thus, if the biased estimates were to be employed and the portfolio is required to yield an E[NPV] of at least 50 at the 5th percentile of the portfolio value (i.e., r _0.05 = 50), only the portfolio with the least risky projects of 9, 10, and 11 can be selected. However, when the unbiased risk estimates are used, it is possible to select the projects 3, 6, and 7 with higher risk, which can be expected to yield about 199.5/65.4 ≈ 3.1 times higher portfolio value, as is shown in Table 5.

Figure 9

The Mean‐5th Percentile Efficient Frontiers

Notes. Actual calibrated frontier (solid line) and its estimate (dashed line), x: value maximizing strategy and o: risk minimizing strategy.

Table 5

The Composition and Value of the Portfolio as a Function of Risk

Unbiased values for portfolio's
5th percentile	Mean	Projects in portfolio
44.2	225.4	1, 2, 3
46.4	223.8	1, 2, 6
48.0	223.5	2, 3, 6
49.2	222.9	3, 4, 6
50.4	199.5	3, 6, 7
50.8	174.5	6, 7, 8
53.4	132.5	7, 8, 9
54.9	65.4	9, 10, 11

The final step is to select a portfolio whose risk level is acceptable. If the DM has a strict constraint for risk r _0.05 = 50, the optimal portfolio is to select projects 3, 6, and 7. However, if the DM can take slightly more risk to gain a higher return, the portfolio with projects 3, 4, and 6 should be selected as it has only (50.4 − 49.2)/50.4 ≈ 2% higher risk whilst its estimated E[NPV] is (222.9 − 199.5)/199.5 ≈ 12% higher.

The key takeaways from this case study are that using biased value estimates can result in (i) the significant overestimation of risk, (ii) selecting a too conservative portfolio of projects with only low risk and low value projects, and (iii) missing out on opportunities to achieve a greater expected return. These problems can be avoided by calibrating the portfolio value and risk estimates.

Conclusions

We have shown that estimation errors about the future value of projects, combined with the fact that only some of the projects can be selected, has major implications for estimating the risk of the selected project portfolio. By addressing this topic, we have made both theoretical and practical contributions to the literature on project portfolio selection.

First, we have shown that the direct use of uncertain value estimates about projects can lead to systematic biases in estimates about the risk of the resulting project portfolio. These biases are problematic because underestimation and overestimation of risks are both undesirable. In the case of underestimation, the DM will be exposed to greater risks than what was expected. In the case of overestimation, the DM may unnecessarily abstain from starting risky projects in the expectation that possible risk constraints would be violated, although this would not be the case.

Second, in order to reduce the estimation error in the risk, we have proposed a general framework for the calibration of estimated risks. Under some conditions, the appropriate amount of calibration can be derived in closed‐form and, more generally, by quantifying the key parameters of the project selection problem and by using these parameters to simulate a large number of problem instances. The parameters for this simulation approach can be elicited by performing statistical analyses of past project selection processes or by consulting experts.

Third, the introduction of risk constraints may lead to larger errors in risk estimates. In particular, we have shown that, in keeping with expectations, the introduction of a risk constraint does curtail expected downside risks. However, it can reduce the expected estimated downside risk even more, and consequently the introduction of a tighter risk constraint may erroneously suggest that risks are better managed than what they actually are. In practice, this is a concern of utmost importance when the selection is constrained by resources and risks alike. As a solution to the problem, we propose a procedure for calibrating risk estimates in project portfolio selection. This procedure helps the DM select a project portfolio that is better aligned with his stated risk preferences while eliminating systematic biases in the risk estimate.

This research can be applied empirically to investigating completed processes of project portfolio selection. Such empirical studies should ideally build on sufficiently extensive data sets which contain information about estimated and realized project values (even if information about realized values can be provided for selected projects only). As a complement, therefore, controlled empirical experiments could be carried out to gain further insights into the presence of risks in the project portfolio selection problem.

Finally, the problem we have identified, examined, and proposed a solution for is present in any resource constrained project portfolio selection problem in which risks matter and only a subset of a large number of alternatives are selected based on value estimates that contain random estimation errors. Such problems are encountered frequently by public and private organizations when they select R&D projects, sites for production facilities, business development initiatives, or supply chain subcontractors. Consequently, there are fertile opportunities for carrying out empirical case studies based on the results of this study.

Footnotes

Acknowledgments

We thank the referees for insightful comments and suggestions that have helped us to significantly improve this study. We are grateful for the feedback received from Professors Ahmad Jarrah,Refik Soyer,and from the attendants of the research seminar at the Drexel University and at the Institute for Integrating Statistics in Decision Sciences at the George Washington University School of Business. This research has been supported by the Project Platform Value Now,funded by the grant 293446 of the Strategic Research Council of the Academy of Finland.

This definition of conditionally unbiased estimates is analogous to Smith and Winkler ( 2006 ) because E [ V E | V = v ] = v ⟺ E [ v + E | V = v ] = v ⟺ E [ E | v ] = 0 ⟺ μ E | v = 0,where the first equivalence follows from Equation

References

Arnott

R. D.

Sherrerd

K. F.

. 2009. Clairvoyant value and the value effect. J. Portfolio Manag. 35(3): 12–26.

Baker

Olaleye

. 2013. Combining experts: Decomposition and aggregation order. Risk Anal. 33(36): 1116–1127.

Bansal

Gutierrez

G. J.

Keiser

J. R.

. 2016. Quantifying uncertainties and risks using managerial judgments in a dynamic new product development environment. Prod. Oper. Manag. 25(12): 2010–2013.

Batur

Choobineh

. 2010. A quantile‐based approach to system selection. Eur. J. Oper. Res. 202(3): 764–772.

Baule

2014. Allocation of risk capital on an internal market. Eur. J. Oper. Res. 234(1): 186–196.

Best

1999. Implementing Value at Risk. John Wiley & Sons, Chichester.

Bonami

Lejeune

M. A.

. 2009. An exact solution approach for portfolio optimization problems under stochastic and integer constraints. Oper. Res. 57(3): 650–670.

Brown

K. C.

1974. A note on the apparent bias of net revenue estimates for capital investment projects. J. Finance 29(4): 1215–1216.

Carlin

B. P.

Louis

T. A.

. 2000. Bayes and Empirical Bayes Methods for Data Analysis. Chapman and Hall/CRC, New York.

10.

Eppen

G. D.

Martin

R. K.

Schrage

. 1989. A scenario approach to capacity planning. Oper. Res. 37(4): 517–527.

11.

Flyvbjerg

2006. From Nobel Prize to project management: Getting risks right. Proj. Manag. J. 37(3): 5–15.

12.

Hall

N. G.

Long

D. Z.

Sim

. 2015. Managing underperformance risk in project portfolio selection. Oper. Res. 63(3): 660–675.

13.

Harrison

J. R.

March

J. G.

. 1984. Decision making and postdecision surprises. Adm. Sci. Q. 29(1): 26–42.

14.

Hobbs

B. F.

Hepenstal

. 1989. Is optimization optimistically biased? Water Resour. Res. 25(2): 152–160.

15.

Huchzermeier

Loch

C. H.

. 2001. Project management under risk: Using the real options approach to evaluate flexibility in R&D. Management Sci. 47(1): 85–101.

16.

Johnson

J. E. V.

Bruce

A. C.

. 2001. Calibration of subjective probability judgments in a naturalistic setting. Organ. Behav. Hum. Decis. Process. 85(2): 265–290.

17.

Kavadias

Chao

R. O.

, 2007. Resource allocation and new product development portfolio management. Loch

Kavadias

, eds. Handbook of New Product Development Management. Elsevier, Oxford, 135–163.

18.

Kirkwood

C. W.

2004. Approximating risk aversion in decision analysis applications. Decis. Anal. 1(1): 51–67.

19.

Kloeber

2011. Current and cutting edge methods of portfolio decision analysis in pharmaceutical R&D. Salo

Keissler

Morton

, eds. Portfolio Decision Analysis Improved Methods for Resource Allocation. Springer, New York, 283–332.

20.

Kubo

Ito

Kawai

Kobayashi

2005. New approach to project portfolio management for new drug development. Hitachi review.

21.

LaGattuta

Stein

J. C.

Tennican

M. L.

Usher

S. E.

Youngen

. 2001. Cashflow‐at‐risk and financial policy for electricity companies in the new world order. Electr. J. 13(10): 15–20.

22.

Lenzi

J. C.

2012. Use of risk‐based project estimates for budgeting and project management. Washington State Department of Transportation Instructional Letter IL 4071.02.

23.

Lichtenstein

Fischhoff

Phillips

1981. Calibration of probabilities: The state of the art to 1980. Technical report ptr‐1092‐81‐6., Office of Naval Research. Available at www.dtic.mil.dtic (accessed date April 19, 2017).

24.

Luenberger

D. G.

1998. Investment Science. Oxford University Press, New York.

25.

Markowitz

H. M.

1952. Portfolio selection. J. Financ. 7(1): 77–91.

26.

Markowitz

H. M.

1991. Foundations of portfolio theory. J. Financ. 46(2): 469–477.

27.

Menezes

Geiss

Tressler

. 1980. Increasing downside risk. Am. Econ. Rev. 70(5): 921–932.

28.

Mild

Salo

. 2009. Combining a multiattribute value function with an optimization model: An application to dynamic resource allocation for infrastructure maintenance. Decis. Anal. 6(3): 139–152.

29.

Miller

L. R.

2010. PMA license valuation: A Bayesian learning real options approach. Rev. Financ. Econ. 19(1): 28–37.

30.

Nawrocki

1999. A brief history of downside risk measures. J. Investing 8(3): 9–26.

31.

Novoa

L. J.

Jarrah

Morton

D. P.

. 2017. Flow balancing with uncertain demand for automated package sorting centers. Transp. Sci. https://doi.org/10.1287/trsc.2015.0662.

32.

Roy

A. D.

1952. Safety first and the holding of assets. Econometrica 20(3): 431–449.

33.

Salo

Keisler

Morton

. 2011. Portfolio Decision Analysis: Improved Methods for Resource Allocation. Springer, New York.

34.

Smith

J. E.

Winkler

R. L.

. 2006. The optimizer's curse: Skepticism and postdecision surprise in decision analysis. Management Sci. 52(3), 311–322.

35.

Sortino

F. A.

van der Meer

,1991. Downside risk. J. Portfolio Manag. 17(4): 27–31.

36.

Stamatelatos

Dezfuli

. 2011. Probabilistic risk assessment procedures guide for NASA managers and practitioners. Technical report NASA/SP‐2011‐3421.

37.

Vilkkumaa

Liesiö

Salo

, 2014. Optimal strategies for selecting project portfolios using uncertain value estimates. Eur. J. Oper. Res. 233(3): 772–783.

38.

Weintraub

Abramovich

. 1995. Analysis of uncertainty of future timber yields in forest management. For. Sci. 41(2): 217–234.