My advice that organizations create a project-selection decision model in order to define project evaluation metrics is nothing more than a recommendation to follow the "scientific method." The scientific method refers to a multi-step process that forces a careful, deliberate approach to building and validating new knowledge. It is the process scientists use to develop accurate (that is, logical, reliable, and non-arbitrary) answers to tough questions.
The scientific method was formalized in the early 1600's by the lawyer and philosopher Francis Bacon. Bacon, who was heavily influenced by Copernicus, was critical of the then common scholastic approach to learning wherein accepted authorities would deduce broad "scientific truths" from limited observations. For example, casual observations of the night sky led people to conclude that the planets circle the earth, not the sun. Copernicus had a difficult time reversing this belief.
Bacon, in his book "The New Organon" (Greek word meaning tool or instrument), argued that "whatever the mind seizes and dwells upon with particular satisfaction is to be held in suspicion." To address the bias, he proposed an inductive form of reasoning committed to empirical evidence that would build understanding through a series of deliberate steps.
Figure 27 provides one representation of the scientific method.
Figure 27: Developing a decision model is an application of the Scientific Method.
The scientific method embodies several important principles. The first is that the problem to be addressed should be formulated in terms of the specific question you want solved, not in terms of some other question that you think might be easier to answer. Since the goal is to select projects that create maximum value, the relevant question for choosing projects is, "What is the value of the project?" In contrast, none of the following questions about which much has been written address what we really want to know: "How do we measure balance?" "How do we measure alignment?" "What are potentially useful project metrics?"
Second, because the scientific method is fact based, information relevant for solving the problem is collected. In this case, such information includes understanding of the goals of organizations, how projects impact success, and the relevant theories for measuring value.
Third, a model (hypothesis) is constructed to explain the relevant phenomenon. For our purposes, the necessary model is one that predicts the consequences of doing versus not doing proposed projects and assigns a value to those consequences. The model provides a causal explanation for what is observed (that some projects produce more useful and highly-valued consequences than others). Finally, central to the scientific method is the concept of testing and refining the model. This can involve "thought experiments" or the comparisons of real-world observations with model predictions. Will the projects that the model predicts will produce highly-valued consequences actually produce those consequences? Whenever inconsistencies are observed, understanding is refined and the model is revised accordingly.
Critical thinking has been defined as the application of the scientific method to everyday reasoning. In accordance with the scientific method, one should focus on the solving the problem that really matters, finding out what the relevant facts are, and being willing to revise one's beliefs on the basis of new information. One should never accept new ideas as fact without careful consideration. And, it may be useful to re-consider some things you currently accept as fact. A person that internalizes the scientific method can think critically, ask appropriate questions, gather relevant information, efficiently sort through and reason logically from the information, and come to reliable conclusions.
A decision model is not much different from the types of conceptual models developed in other areas more typically addressed using the scientific method. Because the model is quantitative, it allows statistical measures of the reliability of the results to be established. Initially the decision model may be constructed based on judgment, but it is tested and refined based on facts and observations. The key is that the model is available for scrutiny by people other than those who developed it originally. Individuals with specialized expertise, experience, or knowledge can spot questionable assumptions and suggest refinements. Thus, decision models have a built-in tendency to get better. This is why scientists claim to "stand on the shoulders of the giants who came before them." The key to progress is defining rigorously logical, explicit relationships that can be quantified with estimates that can be tested and refined over time.
Scorecards and Strategic Alignment Are Not Decision Models
The majority of currently available, project prioritization and portfolio management software tools provide capability for defining both financial and non-financial metrics. The tools are often based on a balanced scorecard approach. Scoring scales are developed to measure project contribution to various non-financial objectives. The scores are weighted and added and combined with financial metrics to provide a "balanced" measure of project performance. This aggregate measure is then used to prioritize projects. Scorecards are useful in some contexts, but the way that they are defined means that they cannot properly prioritize projects.
A company that tried prioritizing with scorecards
The following example, taken from a case study used in a Stanford University professional development course, illustrates the problems experienced by those who use balanced scorecards to prioritize projects :
"When Inspire Pharmaceuticals, a North Carolina-based biotech company, first started managing its pipeline of products for diseases of the eyes and lungs, it used an approach familiar to many: the Balanced Scorecard. Eight criteria were ranked by importance on a scale of 0 to 10. Each project in the pipeline -- discovery, development, or commercialization — was classified by how well it met these criteria. With a potential range from 0 to 100, in theory this approach could provide guidance as to which projects to fund and which to drop.
"In reality, though, the top ranking project differed from the bottom ranking project by just 9 points on the 100-point scale, and nothing scored higher than 35. Not only that, but the projects at the top of the list were in late-stage development, and future expenses to commercialize were likely to be high. The projects at the bottom of the list were found to be those in early research; did it make sense to kill those low-scoring projects when the investment at this stage was so minimal?"
According to the case study, when Inspire replaced their scorecard priority system with a value-based approach using the methods described in this paper, they were able to obtain logical and meaningful prioritizations of project alternatives.
There are four problems for using balanced scorecards to prioritize projects. First, and most fundamentally, the goal for selecting projects should be maximizing value, not creating a balanced project portfolio. Basic scorecard guidance advises that, "The measures represent a balance between external measures for shareholders and customers and internal measures of critical business processes, innovation, and learning for growth" . Assigning weights to measures of this type implies a willingness to accept lower performance in one area (e.g., lower performance for shareholders) in return for better performance in another area (e.g., better business processes). Why would an organization want to accept less value (e.g. lower shareholder value) in order to obtain a higher score (i.e. better "balance") on some internal business process? Value maximization, and not balance, is the goal.
The second problem is that, contrary to typical scorecard mathematics, it is generally not correct to weight objectives that represent means for achieving more fundamental objectives. For example, suppose we include scorecards for measuring the impacts of a project on the quality of business processes as well as scorecards for the impacts on operating costs, customer service, etc. But, improving business processes is merely a means for achieving more fundamental objectives, including reducing operating costs and improving customer satisfaction, etc. Thus, a project might get a favorable score on process improvement, but zero weight should be assigned to this score if the value of that process improvement is completely reflected in the scores assigned to metrics that represent the fundamental objectives that explain why business process improvements are important. If the weight is not zero, there will be double counting.
Failure to account for the hierarchical nature of objectives (including the fact that means objectives contribute to the achievement of fundamental objectives) is a serious error being made by many who are designing tools for project portfolio management. For example, several websites advise, "There are four goals for portfolio management, value maximization, balance, strategic direction and the right number of projects." There is only one goal, value maximization. The proper balance, strategic direction and number of projects are whatever is required to maximize value. A proper decision model computes the value of each alternative based on decision maker's objectives hierarchy, taking into account how lower-level objectives relate and contribute to the achievement of higher-level objectives (Figure 24 in the previous section provided an example). A decision model provides the capability to determine the levels of performance on lower-level objectives that enable the top-level objective (value maximization) to be achieved.
A third problem is more basic—Scorecards do not implement any defensible calculus for project valuation. Contrary to the 'weight-and-add" scorecard approach, it is generally not correct to add different types of value. This statement, which is well established by value measurement theories such as multi-attribute utility analysis, often comes as a surprise to people accustomed to adding and subtracting monetary (e.g., dollar) values. In fact, being able to weight and add sources of values is an exception. According to the theory of value functions, a weight and add approach could only produce a measure of value if all of the metrics were additive independent, each metric is linear measure of value, and each weight expresses the equivalent monetary equivalent of value added per unit of the corresponding metric. Additive independence requires the condition in which the value of achieving any level of performance on any one metric does not depend on the degree to which any other metric is achieved, a condition sometimes referred to as "preferential independence." Multi-attribute utility analysis provides tests for determining whether different types of values may be added, or whether more complicated aggregation equations are needed. (See the prioritization tutorial for an example of a case requiring a non-additive aggregation equation.)
Balanced scorecards cannot correctly prioritize projects
Scoring methods are being advocated that involve weighting and adding scores for criteria such as project risk, internal rate of return, time-to-complete, urgency, and many other criteria that fail to pass the test of preferential independence. It makes no sense, for example, to weight and add a project's score for time-to-complete to weighted scores for other criteria that indicate the value added once the project is completed. Being quick is more valuable if the project adds a lot of value than if the project adds little or no value. Weight-and-add could only make sense, in this case, if the weights are not constants; that is, if the weight assigned to time-to-complete is a function of the ultimate value of the project. (The "weight" for a metric like time-to-complete, would, of course, be determined based on time-preference, and discounting, which is not weight-and-add, would likely be the appropriate form for in the aggregation equation.)
A sound decision model addresses these issues by specifying a logically correct way of quantifying value. Prioritizing projects using a balanced scorecard approach will distort project decisions unless the weights and mathematical form of the aggregation equation are derived consistent with a defensible theory of valuation.
Strategic Alignment — The Most Popular Scorecard Approach
Many providers of project portfolio management software and even some of the recent books on project portfolio management promote an "easy way" to prioritize projects:
"Prioritize projects based on alignment with strategy." "If your budget doesn't have room for the high-end software…, you can always fall back on alignment as a sound way to rank your projects." "The approach needs to be really simple, so that small projects can come up with a strategic alignment rating in about 5-10 minutes." 
Strategic alignment (also called strategic fit) is probably the most popular of the project scoring methods being used today to prioritize projects.
Strategic alignment sounds great in principle. An organization's projects should be consistent with and implement its strategy. But, how do you measure alignment? This is where it gets shaky. Typical advice goes something like this :
This approach has all of the flaws identified above typical of most scorecard approaches, plus some additional problems:
The fundamental problem is that strategic alignment is not a decision model. To review, a decision model is an analytic model that generates as output a measure of the value of a project; that is, a number with the property that Project A is preferred to Project B if and only if the computed measure is higher for Project A than it is for Project B. We can easily show that strategic alignment is not a decision model in this sense by applying the four steps of scientific method:
Strategic alignment is the main, but not the only untenable concept for prioritization being promoted in books and by other websites. Unlike utility theory, such methods do not require learning new concepts and unfamiliar mathematics. I suspect that many, but certanly not all, of those promoting such methods know that there is no logical basis for what they recommend. They assume, without evidence, that the simplified approaches will be "close enough." Unfortunately, the application of a method intended to produce something other than what is desired (e.g., alignment rather than value) won't provide sound recommendations regardless of how much effort is expended on implementing the method.
In answer to those who advocate using concepts other than value as the basis for project selection because utility theory is "too hard," I offer the thoughts of management theorist Russell Ackoff. Ackoff, who passed away in October 2009, believed that most organizational problems arise from trying to do the wrong thing righter: "The more efficient you are at doing the wrong thing, the wronger you become. It is much better to do the right thing wronger than the wrong thing righter! If you do the right thing wrong, and correct it, you get better."