Lee Merkhofer Consulting Priority Systems
Implementing project portfolio management

"If you construct a utility function for projects, you'll have the ability to quantify the value of projects."

Assessing Utility Functions

Eliciting utility functions

As explained on the previous page, provided that you accept a few reasonable-sounding axioms (assumptions about your preferences for outcomes and lotteries over outcomes) a utility function can be constructed that will compute a utility number indicating the degree to which you should logically prefer each of the available alternatives to a decision. The higher the utility number, the more desirable you will perceive the alternative to be.

Typically, a utility function will be designed to compute your preference level for an alternative based on the estimated consequences of the alternative, but the function can be designed to compute preferences based on whatever attributes of alternatives that you feel are relevant. Importantly, the consequences and other attributes that are used to evaluate alternatives need not be certain; a utility function can compute preferences in the face of uncertainty over the levels for attributes provided that the probabilities of the various levels for the attributes can be estimated [1,2].

The reason that utility functions are important for project portfolio management is that a utility function can be obtained for computing preferences for projects. Preferences for projects can be based on the estimated consequences of conducting the projects and/or other project attributes that you, or your organization's policy makers, wish to specify. If you construct a utility function for projects, you'll have the ability to quantify the value of projects. You can then compare the value of each project with its cost and, thereby, obtain a practical means for creating value-maximizing project portfolios.

On this page I describe methods for assessing a person's utility function and how to test for independence conditions that allow you to use a simple mathematic form for the function, such as the additive form. Assessing a utility function that captures an individual's preferences is not difficult, though it can be time consuming. However, it is actually rare to need to obtain a utility function for an organization by directly assessing it from the organization's senior decision makers. Unless your goal is to develop a model for making personal decisions, such as whether to change jobs, purchase a house, or so forth, you may not ever have the need to use the assessment methods. As shown in Part 4, if the goal is to create a model for prioritizing and selecting projects, you can usually deduce the utility function based on the selected performance measures and other characteristics of decision situation and the model. The tests described on this page for determining whether or not you can use an additive or another special form for a utility or value function are, on the other hand, critical. If the value/utility function has more than three or four attributes, you will likely not be able to assess it unless it has the additive form or some simple extension of the additive form.

Creating a Utility Function for Quantifying Project Value

To recap, the first step for constructing a utility function for valuing projects is defining the objectives for your project portfolio (The following Part 4 of this paper offers advice and examples on how to specify objectives). You then define an attribute for measuring the performance achieved relative to each objective. If there are N objectives, denoted Oi, i = 1,2,...,N, for example, there will be N attributes, denoted Xi, i = 1,2,...,N. I use lower case, xi, i = 1,2,...,N to denote specific performance levels achieved relative to attributes. The utility function computes the value of obtaining any specified set (bundle) of performance levels: (x1, x2,...,xN).


Utility function components

There are two ways to define attributes so as to measure the performance advantage achieved by a project. The attribute can be designed to measure the change in the level of performance resulting from conducting the project, in other words, the performance "delta." Alternatively, the attribute can be designed to measure the absolute level of performance achieved by the organization. In this latter case, project value is the value achieved with the project minus the value achieved without the project.

I have one other reminder regarding terminology: In the case where there is no uncertainty over the performance levels of projects, by convention, the utility function is termed a value function. A value function is denoted: V(x1,x2...xN). A value function is simply a special case of a utility function applicable when there is no uncertainty about the levels, (x1,x2...xN) to be attained for the attributes.

The Interview Process

Utility functions are typically assessed from individuals through an interview process wherein the subject is asked a series of questions requiring preference judgments as answers (there are computer programs that pose the questions) [3]. With value functions, the questions ask the subject to express preference judgments over various hypothesized outcome levels for the various attributes. For utility functions, the assessment process is a bit more difficult, as it requires the subject to express preferences over lotteries that yield various attribute levels with various specified probabilities. By convention, for assessments, value and utility functions are typically scaled from zero to one (or zero to 100), although there are times when the model is simpler if such scaling is not employed [4]. Regardless, the utility function can typically be re-scaled to indicate value in dollar equivalents.

Assessing Continuous, Single-Attribute Value Functions

A continuous, single-attribute value function is one that is able to express the relative value of outcomes that can take on any value within some range of possibilities, for example, outcomes that are expressed using a continuous scale. Such utility functions may be elicited using several simple methods [5]. One such method is known as bisection:

Bisection method
  1. Ask the subject to identify the worst, xmin, and best, xmax, levels for the attribute.
  2. Set V(xmin) = 0, V(xmax) = 1
  3. Ask the subject to identify the "bisection point" m1 for which moving from xmin to m1 is just as valuable as moving from m1 to xmax. The relative value of m1 must be 0.5. You now have 3 points on the curve.
  4. To get more points, ask the subject to identify the bisection point m2 between xmin and m1. It will have value 0.25. Then ask for the bisection point m3 between m2 and xmax, which will have value 0.75.
  5. Continue obtaining bisection points until the curve is specified with your desired level of accuracy.

Assessing Continuous, Single-Attribute Utility Functions

Several methods are likewise available for eliciting a continuous single-attribute utility function [6]. One, called the certain equivalents method, proceeds as follows:

Certaom equivalent method
  1. Ask the subject to identify the best, xbest, and worst, xworst, attribute outcomes.
  2. Assign a utility of zero to the worst outcome, and a utility of one to the best outcome.
  3. Ask the subject to name the outcome CE1 (the certain equivalent) equally desirable to a 50:50 gamble between xbest and xworst. By von Neumann and Morgenstern's theorem, the utility of CE1 must be the expected value of the utilities of the best and worst attribute levels, which is 0.5.
  4. Add more attribute levels and their corresponding utilities by constructing more gambles. You can vary either the gamble's attribute levels or the probabilities p. Keep adding pairs until you reach your desired level of accuracy.
    1. One way to get more pairs is to ask the subject to name the new attribute level CE2 that is the certain equivalent of a 50:50 gamble between CE1 and xbest. The utility of the new attribute level CE2, U(CE2), equals [0.5 x U(CE1) + 0.5. x 1] = 0.75.
    2. Another way to get additional pairs is to chose a new attribute level x1 and ask the subject to estimate the probability p that makes this attribute level equally desirable to a gamble between two attribute levels for which you already have utilities, such as xworst and xbest. In this case, the utility of x1 must be U(x1) = p x U(xbest) + (1 - p) x U(xworst) = p.

Assessing a Multi-Attribute Utility Function with Discrete Attributes

If the attributes for which values or utilities are required are all discrete, so that there are only a finite number of outcome bundles, then it is a relatively easy to determine a utility for each bundle [4]:

  1. Have the subject order the bundles from least preferred to most preferred.
  2. Assign a utility of zero to the least preferred bundle and a utility of one to the most preferred bundle.
  3. By the continuity axiom (see utility function), the utility of each bundle between the least and most preferred will be the probability p for which the subject is indifferent between that bundle and a gamble that provides the most preferred bundle with probability p.

Assessing a Multi-Attribute Utility Function with Continuous Attributes

Grid

One approach for obtaining a decision maker's utility function over a multi-dimensional outcome space defined by continuous variables is to discretize each variable so as to define a grid [6]. Once you've assessed utilities for each outcome bundle on the grid, the utilities for bundles between the grid points may be found using interpolation. A problem with this approach, obviously, is the large number of bundles that need to be assessed in order to have a good approximation for the utility of continuous, multi-attribute outcomes.

Another approach is to assume some mathematical form for the multi-variate function and then to obtain from the subject enough judgments (data points) to set the parameters of the assumed functional form, The subject's answers are, of course, likely to show inconsistencies. In that case the analyst can confront the decision maker with the inconsistencies and ask him/her to reconcile them. Alternatively, the analyst can use statistical techniques to fit the function as closely as possible to the inconsistent preference data.

The Importance of Additivity

As you may infer from the above instructions, assessing multi-attribute utility functions if all of the attributes are discrete is straightforward, but time consuming. Assessing multi-attribute functions in the continuous case is much more difficult and time consuming, and becomes more so the more attributes there are. The main practical problem is the large number of questions that the subject must answer and the difficulty of those questions, especially if they involve lotteries [7]. If no special form can be assumed for the function, the subject needs to express preferences involving multiple choices for all possible combinations of the attributes—If there are 5 attributes, for example, there are 25 - 1 or 31 attribute subsets for which multiple preference judgments must be obtained. Few real-world decision makers are willing to spend hours providing the data needed to construct value/utility functions that cannot be said to have a specific mathematical form.

Suppose, however, that the value function has the additive form:

V(x1,x2...xN) = w1V1(x1) + w2V2(x2) ... + wNVN(xN)

In this case the assessment of the multi-attribute value function becomes much easier since it requires only assessing the N single attribute value functions Vi (using, for example, the bisection method above) and the N weights wi.

Similarly, in the case where there is uncertainty over the attribute levels, assessing an additive utility function:

U(x1,x2...xN) = w1U1(x1) + w2U2(x2) ... + wNUN(xN)

requires only assessing the N single attribute utility functions plus the weights.

The remainder of this page describes the tests that can be used to determine whether independence conditions apply and, if so, what independence conditions and which simplifying forms for the value/utility function do the applicable, simplifying independence conditions imply.

Independence Conditions

Numerous independence conditions have been identified that lead to specific mathematical forms for utility functions and value functions, including utility independence, standard gamble invariance, and the zero condition. For the purpose of building value models for prioritizing projects, however, four independence conditions are most useful: preferential independence, difference independence, additive independence, and the delta property.

Preferential Independence

Attribute X is said to be preferentially independent of attribute Y if preferences for levels of X do not depend on the level of Y. To use a common example, suppose that when ordering a meal at a restaurant you are concerned about two attributes: (1) the type of meat (fish or beef) and (2) the type of wine (red or white). Suppose you prefer beef to fish regardless of the color of the wine you drink. In that case, your preferences for food may be preferentially independent of wine. However, suppose you prefer white wine with fish and red wine with beef. Then your preference for wine depends on the food you order, so wine is not preferentially independent of food. As the example shows, one attribute may be preferentially independent of another without that other attribute being preferentially independent of it. That's why mutual preferential independence is needed.

Mutual preferential independence means that every subset of attributes is preferentially independent of its compliment (the attributes not in the subset). In other words, mutual preferential independence means that each attribute is preferentially independent of all other attributes, each pair of attributes is preferentially independent, each triplet of attributes is preferentially independent, and so forth. Verifying mutual preferential independence doesn't require checking all of the possible combinations, however, because it has been proven that if the pair of attributes, Xi and Xi+1, is preferentially independent of the remaining attributes for every i, then mutual preferential independence holds [8].

You might read elsewhere that mutual preferential independence, by itself, is sufficient to justify an additive value function:

V(x1,x2...xN) = w1V1(x1) + w2V2(x2) ... + wNVN(xN)

This can create confusion because, though mutual preference independence assures the existence of an additive value function, that value function may be an ordinal value function. An ordinal value function can correctly rank projects in order of preference, but it won't necessarily capture differences in preferences. You can't divide an ordinal measure of the value of each project by the project's cost and expect to obtain an ordinal ranking of projects based on the correct ranking metric, the ratio of value to cost. For that, you need a cardinal value function. In order to obtain a cardinal value function with the additive form, in addition to mutual preferential independence, you need the condition called difference independence.

Difference Independence

Difference independence means the incremental value obtained from a change in an attribute does not depend on the levels of other attributes, so long as the other attributes remain unchanged. To build on the above example, if the increment in value that you get from changing from a white to red wine when eating beef is different than the increment in value you get from changing from a white to a red wine when eating fish, then wine is not difference independent of food.

If the attributes of a value function are mutually preferentially independent and the attributes are all difference independent, that is sufficient to ensure that a value function of the additive form can be constructed that will produce a cardinal value for projects under conditions of certainty. Accordingly, the test for ensuring an additive value function is checking for mutual preferential independence with difference independence [9]. You only need to demonstrate difference independence for one attribute, however, since if the single attribute value functions are all cardinal value functions, then the overall value function will be as well [10].

Additive Independence

Additive independence for utility functions is the strongest of the various forms of independence. Additive independence means that preferences for lotteries over the levels of attributes only depend on the marginal probability distributions over the attribute outcomes, not on their joint probability distribution [10]. Additive independence is a necessary and sufficient condition for ensuring that a utility function with the additive form can be constructed.

A test for utility additive independence for the case where there are just two attributes can be conducted as follows: Let X1 and X2 denote the attributes. The outcome where X1 = x1 and X2 = x2 is denoted (x1, x2). To test whether preferences depend only on marginal probability distributions, define two lotteries as shown:


Additive independence test

Figure 7:   Two lotteries for testing for utility function additive independence.


In these lotteries, x1Best and x1 Worst denote the most and least preferred outcomes for X1, and x2Best and x2 Worst denote the best and worst outcomes for X2. Thus, L1 is a 50:50 gamble between obtaining the best outcomes for both variables and L2 is a 50:50 gamble for the best outcome for X1 along with the worst outcome for X2 versus the worst outcome for X1 along with the best outcome for X2. The marginal probability of X1 is the probability of X1 ignoring any information about X2, and for both lotteries the marginal probability distribution is 50% probability of the best outcome and a 50% probability of the worst outcome. Likewise, the marginal probabilities for X2 are the same for both lotteries: a 50% probability of the best outcome and 50% probability of the worst outcome. Since the marginal probabilities of the lotteries are identical, the utility function will be additive if and only if the decision maker is indifferent between the two lotteries (indifference must hold for all of the possible levels that may be chosen for X1 and X2). In plain language, additive independence means that preference for gambles over the levels for any variable do not depend on the gambles that exist for the other variables. With additive independence, we can arrive at an overall level of preference by adding preferences established for the each variable.

The glossary provides an example of testing for additive independence..

The Delta Property

The certain equivalent of a lottery (gamble) is the amount of money for which the decision maker is indifferent between receiving that amount of money "for certain" versus owning the lottery (i.e., receiving the lottery's possible outcomes according to their probabilities). The definition of the certain equivalent as applied to a project with uncertain outcomes matches my definition of the value of a project: The value of a project, as I have defined it, is the certain equivalent of the project's uncertain outcomes.

Suppose the following condition holds: Whenever the decision maker is confronted with a lottery, if every one of the possible outcomes of the lottery has its outcome value increased by the same dollar amount Δ, the certain equivalent of the lottery increases by exactly that amount Δ. This condition is called the delta property [11].

Because utility functions represent personal preferences and are obtained empirically through an interviewing process, it would seem that there is no particular reason to expect that the functions would have a specific mathematical form. However, many people indicate that they are comfortable with this assumption. If the delta property holds, it can be shown that the utility function must be linear or have an exponential form [11]:


U(V) = 1 - e-V/R


where V is the value of the project's outcomes, and R is a parameter called risk tolerance. As R approaches infinity, the exponential function approaches a straight line. Thus, U being linear in value is simply a special case of the exponential function where the risk tolerance increases toward infinity. If R is positive, the decision maker is said to be risk averse, and if R is negative the decision maker is said to be risk seeking [10]. The smaller R is the more risk averse the decision maker is. If R = , the certain equivalent approaches the lottery's expected value and the decision maker is said to be risk neutral. If R approaches zero, it can be shown the a lottery's certain equivalent approaches the value of the lottery's least desirable outcome [11].

Although I've used the above form for the exponential utility function, you'll see it written in various different ways depending on how it is scaled. The version above causes the function to be normalized so as to range from minus infinity to one, assuming R is positive.

As explained on the previous page, if V is a cardinal value function, the transformation accomplished by the nonlinear, exponential equation will no longer be a cardinal utility function However, it will continue to rank projects by value correctly (i.e., U will be an ordinal utility). That doesn't create practical concern, though, because what we are interested in is the certain equivalent of the lottery, not its arbitrarily scaled utility. The certain equivalent of the lottery whose utility is U will always be the inverse of the formula for U:

CE(in $) = - R × ln[1-E(U)/R]


where E(U) denotes the expected value of U, and ln[ . ] denotes the natural logarithm. All of the various ways of scaling the exponential utility function will produce the same certain equivalent. In addition to the delta property, the exponential utility function has these attractive features:

  • Empirical evidence, as indicated above, shows the delta property applies for most decision makers (with R positive). However, the delta property becomes a less accurate approximation if Δ is so large as to approach the decision maker's total wealth
  • Both U and its inverse are easy to calculate (e.g., with Excel)
  • Specifying U requires only specifying a single parameter, R
  • Although the exponential transformation will convert a cardinal value function into an ordinal utility function (assuming R ≠ ∞), U retains a couple of attractive characteristics: U(0) = 0, and, for the not uncommon case of a risk-neutral decision maker, U is a cardinal utility function that is ratio scaled (e.g., U($5) = 5 × U($1)
  • V can be multi-attribute and, provided the attributes meet the (comparatively easy to obtain) requirements of mutual preferential independence with difference independence, V can be constructed with the additive form
  • There is no need for U to meet any of the more difficult to check and obtain independence conditions involving lotteries, including additive independence

By the way, I refer to the delta property as an independence condition because it requires the assumption that the decision maker's risk tolerance is independent of his or her wealth. Intuitively, it seems reasonable that an individual would become less risk averse as his or her wealth level rises. A person who can afford to absorb larger losses should be willing to undertake greater risks. Thus, a very wealthy person should have a larger risk tolerance than someone with less wealth. Because this relationship is typically true for gambles whose value outcomes aren't likely to make a large difference in wealth, such as the sort of project gambles undertaken by most organizations, the exponential utility function that derives from the delta property is almost always a good approximation.

The exponential utility function has another feature useful for simplified, approximate methods of analysis. Suppose the uncertainty over the value of an uncertain investment (e.g., a project) may be described by the normal probability distribution with mean μ and variance σ2. Then, it can be shown that the certain equivalent of the investment is [11]:


CE(in $) = μ - σ2/2R


The above equation, it turns out, is often a good approximation even when the probability distribution describing uncertainty over project value is not normally distributed. In other words, the value of a project is approximately its expected value less a risk premium equal to the variance of the probability distribution over project value divided by two times the organization's risk tolerance.

In summary, though we've found that there is no simply answer to the question of what metrics should be used to evaluate and select projects, we know that there is a basis for computing a model to estimate project value. The next part describes in detail how to construct such a model. I call the desired model a project selection decision model. The key component of a project selection decision model is a multi-objective value model with the capability to quantify project value.

References

  1. R. A. Howard & A. E. Abbas, Foundations of Decision Analysis, Pearson, 2015.
  2. O. Morgenstern, "Some Reflections on Utility," in A. Schotter (ed.) Selected Economic Writings of Oskar Morgenstern pp. 65-70, New York University Press 1976.
  3. P. H. Farquhar, "State of the Art - Utility Assessment Methods," Management Science 30(11), 1283-1300, 1984.
  4. R. L. Keeney, "The Art of Assessing Multiattribute Utility Functions," Organizational Behavior and Human Performance, 19(2), 267-310, 1977
  5. R. L. Keeney & D. von Winterfeldt, "Practial Value Models," Chpt 13 in Advances in Decision Analysis: From Foundations to Applications, W. Edwards, R. F. Miles Jr., & D. von Winterfeldt (eds.), Cambridge University Press, 645-654, 2007
  6. P. C. Fishburn, "Methods of Estimating Additive Utilities," Management Science, 13(7), 435-445, March 1967.
  7. D. E. Bell, "Multiattribute Utility Functions: Decompositions Using Interpolation." Management Science 25(8), 744-753, 1979.
  8. R. L. Keeney, "Utility Functions for Multiattributed Consequences," Management Science 18(5), 276-287, Jan 1, 1972.
  9. J. S. Dyer, P. C. Fishburn, R. E. Steuer, J. Wallenius & S. Zionts, "Multiple Criteria Decision Making, Multiattribute Utility Theory: The Next Ten Years," Management Science, 38(5), 645-654, May, 1992.
  10. R. L. Keeney, "Analysis of Preference Dependencies among Objectives," Operations Research, 29(6), 1105-1120, December 1, 1981.
  11. R. A. Howard & J. E. Matheson, "Risk-sensitive Markov decision processes," Management Science 18(7) 356-369, 1972.
  12. I.S. Currim and R. K. Sarin, "A Comparative Evaluation of Multiattribute Consumer Preference Models," Management Science 30(5), 543-561, 1984.