Part 1 identified available tools for project portfolio management (PPM), Part 2 described key differences, and Part 3 summarized costs and risks. Part 4 identified inability to optimize the project portfolio as the weak link for most tools. Part 5 described the components of tools, including the decision model. This part provides a framework for evaluating tools.
How should you evaluate candidate project portfolio management tools? Don't make the mistake of choosing a tool based solely on a comparison of tool features. Instead, seek a tool that will help your organization make better portfolio decisions. Six criteria are relevant for evaluating a tool's decision model: (1) accuracy, (2) logical soundness, (3) completeness, (4) practicality, (5) effectiveness, and (6) acceptability . The subsections below clarify these considerations.
The Tool Must Be Accurate
The value of a PPM tool depends, most obviously, on its ability to produce accurate outputs, including recommendations. Important questions include: Can the tool be counted on to produce reliable estimates? Is it biased toward or against certain projects, interests, or considerations? Are results highly sensitive to untested or untestable assumptions? Does the tool produce outputs with an acceptable level of confidence and precision? Does the tool indicate the confidence level or precision associated with outputs?
Flaws and omissions in decision models can produce large errors in recommendations. As evidence, see the examples in the boxes below.
Intuitive and Reasonable-Sounding Tools Can Be Inaccurate
The Los Alamos National Laboratory (LANL) was seeking a tool for prioritizing equipment purchases, seismic upgrades, and other high-cost investment projects. The lab reviewed a half dozen prioritizing models used in industry and by government agencies, noting that each involved a very different approach.
A team consisting of the author plus experts from LANL, two other national laboratories, management consultants, and a university, was formed to develop an improved prioritization model, called the Laboratory Integration and Priority System (LIPS) . Unlike the other models, LIPS was based on multi-attribute analysis (MUA) . A test was conducted to compare project rankings produced by LIPS with those produced by other models. The results surprised many. Even though the other models seemed intuitive and reasonable, projects that ranked the highest using LIPS were often ranked low by other models.
Being Well-Established Is No Guarantee that a Tool Is Accurate
As another example, in the late 1980's Congress ordered the U.S. EPA to evaluate whether its Hazard Ranking System (HRS), used to place contaminated sites on the National Priorities List for accelerated clean up, was the best-available site-ranking model. The analysis consisted of comparing site rankings produced by the HRS and other sire-ranking models (used by various states and other government agencies) with a ranking produced by an expert panel. The analysis also included developing a model based on MUA.
Once again, the results were surprising. Although the expert and MUA model rankings were nearly identical, none of the site-ranking models produced rankings that correlated with the expert panel. Also, no two models produced rankings that resembled each other. Several models, in fact, produced rankings that were negatively correlated with the expert ranking .
Unfortunately, inaccuracies in recommendations often only surface after many applications of a tool. It is difficult to collect empirical data for validating a tool's recommendations (few organizations are willing to fund recommended and not-recommended projects for the purpose of testing a tool!). Tool providers may have some evidence, but they will naturally emphasize the positive and minimize (or ignore) the negative. The data they have may not be representative of the particular tool configuration or types of projects relevant to your organization.
Conducting a pilot test before fully committing to a tool is essential. Choose a variety of projects with different characteristics and see what recommendations the tool makes. Be skeptical of any odd patterns, such as projects with certain characteristics consistently being ranked either high or low. Resist the natural temptation to rationalize the results ("garbage in, gospel out"). Drill down to fully understand why the results come out the way they do.
Since extensive tool testing is usually impossible, assessments of accuracy must be based, at least in part, on a detailed evaluation of how the tool's decision model works. In this regard, two other considerations are useful. As described below, the tool must be logically sound and it must be complete.
The Concepts Need to Be Right
Success may be in the details, but the first step is to get the concepts right. As an example, an organization was using a scoring approach wherein candidate projects were awarded points based on judged performance along several dimensions, including financial reward, strategic fit, leverage, and probability of success. The points were added and the result used to rank projects. Although no one objected to the logic, there was a suspicion that something was wrong. Large projects were nearly always high on the list and small projects near the bottom.
One major flaw of the tool is that it fails to account for the resource constraint. Projects should be ranked by the ratio of benefit-to-cost, not benefit. Another flaw is that adding points awarded in the various dimensions does not provide a measure of project benefit. The aggregation equation must correspond to the way that the various performance dimensions actually influence value (as described by an applicable theory, such as MUA). For example, points indicating probability of success should not be added. Probability of success should be a multiplier—If a project has zero probability of success the project will generate no benefit whatsoever.
The Tool Must Be Logically Sound
From a practical standpoint, the logical defensibility of a tool is primarily a function of the degree to which it can be justified by accepted and proven theory. If a model can be shown to faithfully implement accepted theory, more confidence can be attributed to the predictions and recommendations made by that model. Conversely, if a model is not grounded in theory, it will almost certainly not consistently produce reliable predictions.
What exactly is a theory? A theory is a logic predicting what outcomes will result from specified actions and why. The "why" is important. A theory must explain cause and effect. To do this, a theory contains an explanation of how things work. A theory is proven by showing that its predictions consistently match observations.
To illustrate the importance of theory, consider initial attempts at manned flight. Observing birds, early researchers thought that they could fly if they strapped feathered wings onto their arms and jumped off cliffs. Many animals fly by flapping feathered wings, but this is not the fundamental reason that they can fly. Manned flight only became possible after Daniel Bernoulli developed theory explaining how airflow around objects can produce lift.
Ranking Projects Based on Strategic Alignment is Not Logically Sound
As described in the paper Mathematical Theory, ranking projects based on the ratio of benefit-to-cost ("bang for the buck") can in some circumstances be a reasonable prioritizing technique (provided that there are no project inter-dependencies, e.g., infrastructure projects and projects that use that infrastructure). Choosing projects that implement organizational strategy is important. However, strategic alignment is not a surrogate for "bang-for-the-buck." Therefore, the approach will not necessarily lead to value-maximizing project portfolios.
Many models are like the feathered wings of early would-be aviators. Instead of being derived from theory, they are heuristics—rule-of-thumb relationships based on observations that certain characteristics tend to be associated with certain outcomes. No explanatory, cause-effect reasoning is provided.
For example, a popular heuristic for constructing project portfolios is balance. Project portfolios of successful companies often balance low-payoff "sure things" against high-payoff gambles. But, balance is not the fundamental characteristic that makes a project portfolio successful. Rather, balance is something that tends to result when the best choices are made based on needs and the nature of available options. Simply choosing a portfolio that happens to be balanced does not ensure success.
As noted previously, relevant theories for selecting and prioritizing projects include decision analysis , multi-attribute utility analysis (MUA) , modern portfolio theory , portfolio optimization theory , and real options . These theories are well established within the technical and academic communities and have been proven in many real-world applications.
Decision analysis is a theory and collection of associated methods for making decisions under uncertainty. The approach involves constructing and analyzing a model of the decision problem to identify the choice, or sequence of choices, leading to outcomes most consistent with the preferences of the decision maker.
Real options analysis is a valuation theory that views projects as creating options for an uncertain future. For example, a new factory carries options to shut down, abandon, or expand, depending on market conditions. Project value depends on the options created. The theory includes a method for computing project value based on the market prices of related assets.
Modern Portfolio Theory
Modern portfolio theory is a theory of decision making that seeks to construct a portfolio of investments offering maximum expected returns for a given level of risk. The theory quantifies the benefits of diversification as a means of reducing risk.
Theories Derive from Axioms
As an example, decision analysis is based on a theory derived from axioms intended to define "rational choice" . One such axiom, known as transitivity, states: "If there are 3 possibilities, labeled A, B, and C, then, if you prefer A to B and you prefer B to C, you will prefer A to C." The theory shows that if you accept the axioms, then there is a way to mathematically measure your preference for alternatives based on the characteristics of those alternatives and your answers to some basic questions about what you like.
The above theories are sometimes termed "axiomatic theories." Each begins with a set of axioms (assumptions, or hypotheses) about how things work, for example, about how people or organizations ought to value and choose projects. These axioms can be accepted or rejected based on observations or other evidence. The theory then derives (through mathematical "proofs") conclusions that follow from its axioms. If you can demonstrate that the axioms are acceptable (and the math is correct), the rest of the theory follows. See the side box for an example.
By the way, the fact that there are multiple theories does not mean that you will get different answers depending on the theory that you choose to apply. The well-established theories for selecting projects typically give the same answers, provided that each theory is able to address all the relevant factors, the theories are properly applied, and the assumptions for the analyses are the same. The situation is analogous to that for theories in other fields, such as physics. For example, the mathematics for Newton's laws of motion and Einstein's theory of relativity look very different, but both theories predict the same trajectories for everyday objects under everyday conditions. The predictions only differ in situations (e.g., speeds close to the speed of light or extremely small objects) that invoke considerations that aren't addressed by Newton's much simpler laws of motion. Likewise, decision analysis and real-options, for example, look very different, but they give exactly the same answers to any problems that both can fully address provided that each theory is correctly applied . Nevertheless, the choice of the theory is critical. Choosing a solution approach based on ill-suited theory may make it extremely difficult, or impossible, to satisfy the necessary assumptions for the theory (e.g., it may not be possible to provide the required inputs). Thus, the wrong theory can make it impossible to obtain useful and meaningful answers.
Rather than cite the theories on which their tools are based, most providers merely reference the analytic techniques that their tools employ, such as balanced scorecards, strategic alignment, decision trees, linear programming and Monte Carlo simulation. Such techniques, by themselves, provide no explanation for why recommended portfolios should be preferred. Instead, they merely describe or refer to mathematical calculations that are performed. For example, balanced scorecards typically use an equation that weights and adds scores assigned to projects. Projects are ranked based on total weighted score. There is no reason why this should lead to the identification of preferred projects.
Weighted scoring techniques can sometimes be used to effectively apply appropriate theories, but only if the scoring scales and weights are structured to match the requirements of the theory. For example, MUA theory for valuing projects can sometimes be implemented using scorecards and a weight-and-add equation. However, for a weight-and-add equation to work, the metrics that are scored must meet a condition known as additive independence, scaling functions must be assigned so that the computed performance measures are proportional to value, and the weights must quantify the value of specified improvements against objectives (weights are often assigned using an assessment technique known as the "swing weight method" ). Unless these conditions are met, the aggregated score will not measure value and will not serve as an indicator of decision-maker preference.
Tools Not Based on Sound Theories Are Exposed When Subjected to Technical Review
Congress required the Department of Energy (DOE) to rank potential sites for disposing of radioactive waste from nuclear power plants. To select the best site, the DOE initially used a scorecard approach. Each site was rated against each of the objectives established for a good site, the objectives were weighted, and the rates and weights were combined to rank the sites.
Hanford, a site in Washington State, ranked highest, and the DOE published the results in a draft Environmental Impact Statement. The choice was criticized, especially by officials from Washington State, and the DOE asked a board of the National Academy of Sciences (NAS) to review the ranking methodology.
The board, which included experts in decision theory, responded that DOE's method was "unsatisfactory, inadequate, undocumented, and biased" . DOE was told to redo its analysis (a site ranking model based on MUA was developed). The new analysis ranked Yucca Mountain, Nevada, highest. DOE was forced to change its decision, causing the agency considerable embarrassment.
Thus, the use of a weighted scorecard does not in any way ensure that the requirements of any accepted theory are being met. The defensibility of the recommendations made by any decision model depends on whether the techniques used to value and prioritize projects are consistent with some defensible theory and on how faithfully the model implements the requirements of that theory.
As an example of the dangers of using tools not based on sound theories, see the side box example describing the Department of Energy's initial attempt to rank potential sites for a nuclear waste repository. Logical defensibility is particularly important when using a tool to help make controversial decisions. Although most project decisions aren't as controversial as nuclear waste, some (such as an electric utility's decision to acquire right-of-way to construct a transmission line) can be.
Using a logically sound approach avoids errors associated with unsound methods and reduces the risk of successful challenges to the credibility of decisions. Although logical soundness does not guarantee accuracy (see below), it is safer and wiser to use a tool based on sound theory than one that merely "seems" reasonable.
The Tool Must Be Complete
Completeness Requires the Capability to Address "Hard" and "Soft" Considerations
A water utility desired a system for prioritizing capital improvements. To fully value such investments, it was necessary to account for hard financial benefits, such as increased revenues, and also for "soft" project benefits, such as enhanced public health and safety, reduced flood risk, increased community recreational opportunities, and expanded capability and knowledge.
Providing inputs for quantifying project value required information from databases associated with many different applications, including asset utilization systems, customer information systems, GIS, work management systems, and investment modeling applications. Furthermore, it was clear that field personnel had local knowledge of needs and the effectiveness of proposed projects that was not captured by historical data. Thus, the resulting priority system included both an extensive scoring logic (to capture judgments) plus analytic techniques for logically combining empirical and judgmental data.
Being complete means accounting for all significant and relevant considerations. A logically sound tool that is incomplete gives, at best, the right answer to the wrong question. A tool might leave out important considerations because those considerations are difficult to accommodate (e.g., it may be too hard or too costly to obtain the necessary input data) or because they are impossible to include given the nature of the selected decision model (e.g., dynamic considerations within a static decision model).
As noted earlier, project decisions tend to produce broad, enterprise-level impacts. This makes creating a complete model challenging. As illustrated by the example in the side box, if the decision problem is extremely complex, a complete model may need to be large, requiring many inputs and sophisticated mathematical algorithms.
One way to assess the completeness of a model is through sensitivity analysis. Vary the description of a complicated project in ways that should logically affect the attractiveness of that project. Do the inputs permitted by the model allow you to reflect such considerations? Do the priorities established by the model behave as they should? No model can capture everything, but a complete model should properly address those considerations most important to project selection decisions.
The Tool must be Practical
Accuracy, logical soundness, and completeness are the reasons that prioritization tools must often include sophisticated decision models. But, a tool must also be practical. Building a quality tool that is still practical is a significant challenge facing expert tool designers.
Experience Creates Capability and Demand for More Sophisticated Tools
In the early 1990's, a client operating an oil and gas pipeline sought help in designing a system for prioritizing investments to reduce risk. The new system was much more detailed and formal than existing practices. It required training, implementing new processes, constructing additional databases, and creating computationally sophisticated evaluation software. The first system documentation manual was more than 200 pages in length. Many outside the development team argued that the system was "far too complicated." However, backing of senior management was secured when it was demonstrated that the system uncovered ways to significantly reduce risks without increasing total costs. The client has continued to use the system independently for many years, and each year the system has been expanded. Experience, better documentation and internal training led to increased confidence and expertise, promoting the desire for more capability and accuracy.
To be practical, users must have sufficient expertise to understand and apply the tool. The required inputs must be available. Computational resources must be adequate, and tool applications must be completed within available time. All this must be accomplished without unacceptably degrading the accuracy, logical soundness, or completeness of the model.
How can such challenges be addressed? Obviously, skill is required and experts should guide the development of the decision model used by the tool. At the same time, the organization can take steps to make it practical for it to use more sophisticated tools. These steps can include investing in internal training, reassigning responsibilities, developing new sources of data, and adjusting budgeting schedules.
Although the above recommendations may seem daunting, it is worth noting that it is common for an organization to initially view a quality tool as "too complex," and then later, after gaining experience and comfort with its use, to want to expand the tool and make it even more sophisticated (see the above side box example). Thus, organizations should avoid simplistic tools and tools whose decision model cannot be improved as experience and understanding grows.
The Tool Must Be Effective
A Centralized Project Ranking Tool May Not Fit a Decentralized Decision-Making Environment
An electric utility desired a system for allocating its $250 million annual O&M budget. The organization had a decentralized decision-making structure. The managers of business units (e.g., line clearance, meter reading, the call center) had historically been given authority to decide how best to spend within their respective areas. A centralized, project ranking system was unacceptable. It would dictate to each manager how the projects within his or her area should be prioritized. On the other hand, allowing each business unit to have its own, separate priority system would not provide a consistent approach for the enterprise as a whole. The selected system  involved a tiered approach to PPM. It allowed each department to apply a tool to determine the value generated under different funding levels, based on the projects that would be conducted and the benefits that would be generated under those funding levels This design allowed area managers to retain authority to prioritize and select projects within their respective areas while rewarding managers who propose projects create value consistent with corporate objectives.
A PPM tool cannot be effective unless it fits the way that the organization actually makes decisions. The considerations to which the tool may need to be sensitive include:
Such considerations must be understood and the tool and its application process must be designed to ensure that there is a good fit to the organization. See the sidebox for an example wherein the approach needed to be modified to accommodate the organization's decision-making structure.
Being Effective May Require a New Approach
A recent effort was conducted for an electric utility to provide a system for prioritizing bids for long-term (greater than 10-year) energy contracts. Standard contract valuation methods were overly sensitive to discount rate assumptions. Therefore, an alternative model not dependent on discount rates was developed based on real options analysis. Real options  adopts the view that projects and assets create options (e.g., a contract creates the option to buy energy according to a predetermined price rule). The real options tool did not replace the utility's conventional contract valuation tool. Rather, it is used to make broad choices about contract durations while the utility's more conventional tool is still used to support specific decisions.
Being effective also means achieving the specific goals that motivate using a tool. For example, a tool intended for back-room use would be designed differently than one whose purpose is to demonstrate to regulators and local citizens that the organization's project decisions are in the best interests of the community. Not only would the latter have different user characteristics and features, the definitions of project benefits would be quite different.
Before building or purchasing a tool, think carefully about what the tool needs to do in order to be effective. See the second side box for another example of an application requiring a specialized approach. Although a tool may appear to offer lots of flexibility, it can only be adjusted within the limitations dictated by its underlying decision model. Few tools allow users to change the mathematical logic by which projects are valued or optimal portfolios are identified.
The Tool Must Be Acceptable to Stakeholders
Developing the Tool as a Collaborative Effort Promotes Buy-In
A client was investing heavily in information technology. A system was desired that would prioritize proposals submitted by different departments. Department managers were naturally concerned about how the new system would affect their ability to obtain funding. One department head, responsible for core systems, was particularly pessimistic about the ability of a tool to fairly evaluate his projects. "How can you quantify the value of adding capacity? Our projects don't increase revenue or reduce costs, and they are invisible to our customers. The only time we ever get more funding is when things stop working." The manager only agreed to participate after attempts to get his projects exempted from the system failed. To address the concerns, the selected design included a feature not common to priority systems--If an infrastructure project enabled or facilitated another project, the enabling project obtained a share of the benefit derived from the enabled project. The approach worked, and the core systems department obtained approval for several projects that had historically been denied funding.
A tool that is practical and effective is not always acceptable to decision makers and other stakeholders. An acceptable tool must be compatible with organizational processes and culture. It must be understandable and understood. A tool that impacts funding decisions will be perceived as a threat to some interests. All key stakeholders must have confidence that the tool will help them, as well as the organization, to succeed.
Gaining adequate acceptance is critical to the success of the tool. For a dramatic example of a tool that was widely regarded as technically defensible, complete, accurate, and practical and, yet, was rejected, read "The Rise and Fall of a Risk-Based Priority System" .
The most effective way of generating acceptance is to involve in the design effort those who will use and be impacted by the tool. A collaborative process helps ensure that the tool will have exactly those characteristics necessary to best suit the organization and its needs. Equally important, involving stakeholders in the design of the decision model creates buy-in by allowing skeptics to express their concerns and see first hand how those concerns are addressed. See the side box for an example.