Lee Merkhofer Consulting Priority Systems
Implementing project portfolio management

"A project selection model can enable the organization to consistently and efficiently make the best project choices."

Part 4:  Inability to Measure Project Value


The goal of project selection is to choose the projects for the organization's project portfolio that, collectively, will generate the greatest possible value [1]. However, to be sure that the best projects are being selected, the organization must have the ability to assess the value of its candidate projects. As Peter Drucker is credited with saying, "You can't manage what you can't measure."

The previous Part 3 of this paper demonstrated that identifying projects that generate the most value isn't simply a matter of finding and using the right metrics. Yes, you do need a specific set of metrics to accurately and comprehensively assess project performance. But, you also need to know the equation or model that allows you to translate project performance estimates into a quantitative expression of project value. In other words, in order to identify the best project choices it is necessary to create a project value model. The inputs to the value model are the proper project performance metrics, they are the project- and organization-specific estimates associated with a project that, when input to the value model, provide as model output the value of that project. You identify the appropriate project valuation metrics by building a value model based on utility theory. The project value model tells you what project measures are needed as model inputs. Without a project value model, there is no way to obtain assurance that the best projects are being selected. Inability to measure project value is the fourth reason organizations choose the wrong projects.

Project Selection Models

Too many projects

The value of a project is, logically, the worth, to the organization, of the project's consequences. In order to make the right choices, therefore, organizations need to estimate the consequences of conducting candidate projects (something organizations with good accountability practices do anyway). They, then, must determine the value of those consequences (the job of the project value model). As demonstrated in the previous part of this paper, utility theory provides the foundation for creating a project value model. A model that reliably estimates poject value enables the organization to consistently make the right project choices.

Project Selection is an Ongoing Task

For many organizations, project selection occurs mainly as a component of the capital budgeting process when resource needs for the upcoming budget period are being planned [2]. These organizations typically conduct a comprehensive review of candidate projects and choose a subset for execution in the coming budget period. However, pressing new needs and unexpected opportunities can arise at any time [3]. Also, some projects may need to be terminated due to unforeseen events or because it becomes apparent that the resources being consumed could be better used elsewhere. Project selection is, thus, not just a once-per-year activity, it is an ongoing task.

Why Choosing the Best Projects is Difficult

I've described the reasons that project selection is difficult throughout this paper. To summarize:

  • Projects differ in fundamental ways. One project may help a firm sell a product to more customers. Another might reduce the amount of waste produced in the manufacturing process. A third might increase the level of safety for workers handling hazardous chemicals. Its hard to compare projects whose motivations are so different.
  • Uncertainty—The worth of a project depends on its consequences, but project consequences are often uncertain. Projects with uncertain consequences are like gambles. How much it is worth to undertake a gamble depends in part on the willingness of the organization to accept risk. Uncertainty and risk compound the difficulty of project selection.
  • The costs and benefits of projects occur in different time periods. Costs normally begin almost immediately, but benefits generally occur sometime in the future. Also, project benefits may continue over some extended period of time. Comparing projects when the timing and duration of benefits and costs differ is hard to do intuitively.
  • Interactions among projects—the costs and/or benefits of a project can depend on whether or not other projects are conducted. Thus, it may not be possible to make project choices one at a time and without regard to what other projects get included within the project portfolio.

In sum, choosing projects involves complex considerations. Selecting the best project from many alternatives is often tough. Choosing the most desirable combination of projects; that is, creating an optimal project portfolio, is even harder.

Project Selection Models Help Organizations Choose the Best Projects

The key to being able to consistently choose the best projects is, in my opinion, creating and using a quality, project selection decision model. The challenges that make project selection difficult for people—multiple considerations, uncertainty, interdependencies, the need to compare costs and benefits in different time periods—are exactly the characteristics that make the problem amenable to solution via a computer programmed with an analytic model. As described in Part 1 of this paper, people have limited information processing skills, can be biased, and are often inconsistent when making choices [4]. An analytic model breaks a complex problem into pieces, allowing those pieces to be studied and characterized, and their relationships represented mathematically [5]. People can then focus on the simpler task of providing inputs to the model. The processing of those inputs can be handled in accordance with sound mathematical reasoning using a valid solution algorithm and a computer.

Knowledge and effort are required initially to design and construct a project selection model. Lack of the relevant knowledge; that is, not knowing how to do it, is, in my opinion, the reason that relatively few organizations have developed formal, project selection models. This part of my paper explains, step-by-step, how to build a project selection model. Once built, a project selection model may be used again and again, both during an annual budgeting exercise and as a means for properly inserting newly identified projects into the existing priority order. A project selection model makes the criteria and logic for project evaluation explicit. When the organization has the ability to quickly and systematically evaluate the attractiveness of proposed projects, experience shows that the quality of project proposals increases. In other words, a project selection model not only helps the organization to select from among its candidate projects, it shows project managers how to design projects that generate greater value. As experience with the model accumulates, the model design can be modified and improved in ways that make its application easier, its predictions more accurate, and the insights it generates more useful.

A Project Selection Model Accounts for the Three Components of a Decision

As summarized in Figure 9, a model intended to aid decision making must account for the three basic components of a decision [6]: (1) the alternatives that are available (what you can do), (2) objectives and preferences (what you want), and (3) information and beliefs about what will happen depending on the alternative to the decision that is selected (what you know, believe or suspect).


Components of a decision model

Figure 9:   A decision model must faithfully represent the 3 components to a decision.



In the case of project selection, the alternatives are the various projects and combinations of projects that may be conducted, including alternative versions of projects if available. The objectives for project selection are the objectives of the organization, clearly stated and structured in such a way that they may be used as a basis for evaluating and comparing alternatives. The information and beliefs that matter are those relevant to predicting the consequences of doing versus not doing projects. The credibility of any project selection model requires that all three of these decision components be captured and accurately represented. In addition, essential to the capability of the model is the logic by which recommended choices are derived. The logic must be defensible, understandable, and capable of being implemented on a computer.

A Project Selection Model May Be Created by Linking a Consequence Model to a Value Model

Since the value of a project is logically the value of the project's consequences, it makes sense that a project selection model is composed of two sub-models, as shown in Figure 10 [7].


Components of a decision model

Figure 10:   A project selection decision model consists of two component.


The first sub-model, labeled simulation, is a consequence model that takes as input project characteristics, including estimates of what the possible impacts of conducting a proposed project will likely be. This sub-model produces as output a forecast for how those impacts will alter the degree to which the organization will achieve its objectives. The second, sub-model, labeled valuation, is a value model that converts the estimated, project-induced changes to the organization's performance relative to its objectives into a measure of the value of that increment to performance.

The Value Model

Though it is the second of the two sub-models, the top-down approach to model construction means that the value model must be designed first. The value model identifies the specific organizational objectives that may be impacted by projects and provides performance measures (aka attributes) for quantifying the degree to which the objectives are achieved. Utility theory, as described on the two preceding pages, provides methods for constructing value models. The value model is a value function if there is no uncertainty over the impacts of projects on objectives as measured by the attributes. If there is uncertainty, the model is a utility function. In either case, the value model translates the changes that a project induces on the organization's ability to achieve its objectives into a number indicating the value of those changes.

The Consequence Model

The consequence model simulates the organization's performance relative to its objectives. The model forecasts the consequences of project decisions based on the facts, judgments, and uncertainties that determine the impact of project choices on the ability of the organization to achieve its objectives. Consequence models need not be very complex. A common function of the consequence model is to simulate the timing of project impacts: specifying when they will start, how long they will continue, and what will be left when they end. If the value model is a value function, the consequence model will necessarily be deterministic. If the value model is a utility function, the consequence model must be probabilistic [8]. For deterministic models, a good decision is judged by the estimated project outcomes alone. For probabilistic models, the decision-maker is concerned not only with estimated project outcomes, but also with the amount of risk associated with those outcomes

Two Ways to Compute Project Value

As mentioned previously, there are two ways of configuring the attributes in a project selection model. The first is to specify as inputs to the model the "deltas" (changes) in attribute levels that will result from doing the project. The model is in this case designed to produce as output the increment in value that results from the decision to conduct the project. The second way is to design the model to be applied twice, once to estimate the value achieved by the organization with the project and once to estimate the value achieved without the project. The value of the project with this design is the difference in the two computed values. Oftentimes a model will be designed so that some attributes measure deltas while others require inputs corresponding to the "with project" and "without project" cases. As described earlier, computing value twice, with and without the project, is typically easier if failing to conduct a project results in a decline in the organization's performance, such as when maintenance projects are needed to prevent deterioration of the organization's assets.

Choosing a Methodology for Constructing a Project Selection Model

Decision analysis (DA) is the professional discipline and collection of tools and techniques for creating and analyzing decision models based on utility theory. Multi-objective decision analysis (MODA), also known as multi-attribute utility analysis (MUA) is, as the name suggests, the sub-field of decision analysis focused on decisions with multiple objectives [8, 9]. MODA encompasses all of DA's methods and tools for analyzing decisions with uncertainty and adds methods for explicitly accounting for multiple decision objectives. Other multi-criteria methods have been proposed and used for constructing project selection models. The reasons that I recommend MODA are as follows:

  1. The metric computed by MODA needed for ranking projects is project value. MODA is able to express the value of projects in monetary units (though, it doesn't require this).
  2. MODA can be applied to all types of project decisions, including dynamic project choices where subsequent project decisions depend on the outcomes to initial project choices.
  3. MODA provides a formal logic for specifying the equation (e.g., additive, multiplicative, etc.) for combining estimates of project performance relative to multiple criteria. Most other decision making methods assume specific equation forms (usually additive equations). MODA shows that the correct form of the equation depends on checkable independence conditions.
  4. MODA applies to organizations that have hierarchies of objectives and sub objectives. It also works if the organization subscribes to only a single objective. In fact, if the organization's only objective is profit (with sub objectives of maximizing net income across multiple years), it can be shown that MODA produces NPV as the correct measure for computing project value [10].
  5. MODA accounts for uncertainty and risk. MODA is also able to account for organizational risk tolerance, the degree to which the organization is averse to taking risks.
  6. A MODA value model is constructed using a top-down approach starting with identifying and structuring the organization's objectives. The method includes tools and techniques for facilitating the model-development process, including methods for obtaining weights, quantifying judgmental uncertainties, designing scales for obtaining numerical and categorical inputs for the model, and incorporating judgments from subject matter experts.
  7. Components of value
  8. A MODA value model identifies the different types of project value and shows the contribution of each to total project value (see the figure to the right showing the value components estimated for five projects as produced by a MODA-based, project selection model).
  9. MODA models are well-suited for sensitivity analysis, including showing the impact on project priorities of different assumptions regarding uncertainties and organizational willingness to accept different tradeoffs among objectives.
  10. MODA is widely regarded as the most defensible of decision making approaches, and has been identified and labeled best practice by independent expert panels and government agencies in the U.S., Canada, and the United Kingdom.
  11. Perhaps most significantly, MODA derives, literally, from a widely accepted theory of how to make rational decisions.

Steps for Creating a Project Selection Model

Like any model-building process, the process of building a MODA project selection decision model involves art as much as it does engineering science. However, in the 60 plus years since the specification of utility theory, decision analysts have evolved an efficient, step-by-step process for building decision models. The 12 steps of the MODA model-building process, as I define them, are identified in the table and associated Figure 11 below.



Steps for Building a MODA-Based, Project Selection Decision Model
Model Component Step Description
Framing Redefine projects for independence If some projects are interdependent, combine them into larger, independent projects with multiple versions. The goal is to enable the optimal portfolio to be found via prioritization.
Create an objectives hierarchy Identify and structure the objectives for evaluating projects into a hierarchy. Objectives should be fundamental, decomposable, and measurable, with no overlap or double counting at lowest-level of the hierarchy.
Construct influence diagrams Identify factors and relationships that determine performance relative to each objective. No need to construct diagrams for financial objectives, since methods for measuring financial performance are well defined.
Define performance measures Define performance measures for the lowest-level objectives in the hierarchy. Performance measures may be natural measures, constructed scales, or proxy measures
Verify mutual preferential independence Goal is to obtain an additive value function. Reformulate objectives and performance measures as needed. Once validated, the objectives hierarchy should be reviewed and approved by executives.
Specify the aggregation equation Follows directly from independence assumptions achieved..
Implementation Specify single-attribute value functions. Needed to translate performance into value. Unless projects include a "bet the business" gamble, assume value is linear in dollars. At minimum, timing and duration of project benefits need to be modeled.
Decide whether to explicitly model uncertainty. Model uncertainty if there are serious discrete risks). Otherwise, probably won't significantly impact priorities. If uncertainty is modeled, transform value function using exponential (relative risk) utility function.
Create the consequence model Choose deterministic vs. probabilistic depending on whether uncertainty must be modeled. Decide how uncertainty will be propagated through model.
Assess weights & risk tolerance Use the swing weighting method with questions framed to avoid bias. Pilot test with PPM team before assessing weights from senior executives. If risk is modeled, assess risk tolerance.
Design the "scoring" process Define how project-specific, model inputs will be obtained. Typically involves creating scoring scales and scoring instructions.
Implement, test & roll out the model In software (e.g., Excel). Be sure to pilot test the software, making refinements to the model as needed before full-scale application. Prepare model documentation.


Components of value

Figure 11:   Steps for creating a project selection model.


Be aware that my description of the model-building process, like other attempts to reduce a creative exercise to a simple sequence of steps, is an oversimplification. Constructing a project selection model, or any decision-making model for that matter, cannot be accomplished by blindly following a linear sequence of tasks. Creating a successful model is often accomplished through trial and error. When difficulties are encountered at any step, in order to resolve those difficulties it is often necessary to revert back to a previous step from which an alternative course forward can be plotted. Perhaps most importantly, it is essential to keep the independence conditions in mind. Failing to define objectives and specify performance measure attributes in a way needed to achieve preferential independence can result in having to toss out previous effort and beginning again.

The remaining pages of this Part 4 of my paper describe each of the main steps of the MODA model building process with examples.

References

  1. S. Elonen & K. A. Artto, "Problems in Managing Internal Development Projects in Multi-project Environments," International Journal of Project Management, 21(6), 395-402 August 2003.
  2. L. J. Gitman & J. R. Forrester, Jr., "A Survey of Capital Budgeting Techniques Used by Major U.S. Firms," Financial Management, 6(3), 66-71, 1977.
  3. Y. Petit, "Project Portfolios in Dynamic Environments: Organizing for Uncertainty," International Journal of Project Management, 30(5), 539-553, July 2012.
  4. J. G. March, "Bounded Rationality, Ambiguity, and the Engineering of Choice," The Bell Journal of Economics, 9(2), 587-608, Autumn 1978
  5. P. R. Garvey, Analytical Methods for Risk Management: A Systems Engineering Perspective, CRC Press, Oct 20, 2008.
  6. D. W. North, "A Tutorial Introduction to Decision Theory," IEEE Transactions on Systems Science and Cybernetics 4(3): 200-210, 1968.
  7. M. W. Merkhofer, "Assessment, Refinement and Narrowing of Options," Chapter 8 in V. H. Dale and M. R. English (eds.),Tools to Aid Environmental Decision Making, Springer, 2012.
  8. R. L. Keeney, "Decision Analysis: An Overview," Operations Research, 30(5), 803-838, 1982.
  9. R. T. Clemen, Making Hard Decisions with Decision Tools, Terence Reilly Cengage Learning, 2013.
  10. S. Frederick, G. Loewenstein & T. O'Donoghue, "Time Discounting and Time Preference: A Critical Review," Journal of Economic Literature vol. XL 351-401 June 2002.