Confronting Climate Uncertainty in Water Resources Planning and Project Design The Decision Tree Framework

Water infrastructure projects are a significant portion of the World Bank’s lending portfolio and a major need for developing countries throughout the world. Many water resources projects have long periods of economic return, with significant uncertainties in the behavior of the natural system as well as that of human factors (technology, population dynamics, economic development, and the like). The uncertainties associated with climate change, however, have led to a reconsideration of whether the water development community is adequately taking into account the uncertainties that characterize the future. The goal of this book is to outline a pragmatic process for risk assessment of Bank water resources projects that can serve as a decision support framework - a decision tree to assist project planning under uncertainty. The decision tree described in this book is based on the growing consensus that robustness-based approaches are needed to address uncertainty and its potential impacts on infrastructure planning. These approaches emphasize assessment of individual projects and their ability to perform well over a wide range of future uncertainty, including climate and other uncertainties. The decision tree is designed to address the fundamental issues to provide a path forward for project planners who face decisions potentially affected by climate change uncertainty. The decision tree was also designed so that human and financial resources can be used economically. The goal of this work was to develop a tool that will be applicable to all water resources projects, but that will also allocate climate risk assessment effort in a way that is consistent with each project’s potential sensitivity to that climate risk. Report is organized as follows: chapter one gives introduction. Chapter two provides background on the risks relevant to water systems planning, describes the different approaches to scenario definition in water systems planning, and introduces the decision scaling methodology upon which the general structure of the decision tree framework is based. Chapter three describes the decision tree tool and explains each of the steps and processes that make up the tool. Chapter four focuses on a case study of a small hydropower project as an illustration of the decision tree procedure. Chapter five describes some of the tools available for decision making under uncertainty and methods available for climate risk management. Concluding thoughts on implementation are presented in chapter six.

and spatial scales that are relevant to water resources projects. As a result, climate science-led eff orts do not typically reduce the uncertainty of future climate and, in fact, are unlikely to describe the limits of the range of possible climate changes. Perhaps most important, GCMs have the least skill in generating the variables that are most important for water resources projects, such as local hydrologic extremes like fl oods and drought. Often, the results of a climate change analysis present a wide range of possible future mean climates without providing any insight into climate extremes, and give the sense that they only show the tip of the iceberg for climate uncertainty. As a result, the project planner gains little insight into the potential impact of climate change on the project. The second issue relates to risk management. If climate-related risks are quantifi ed in the process of climate change risk assessment, it remains unclear in most cases whether the eff ects of changes in climate on a certain water resources project are signifi cant relative to the impacts of changes in other nonclimate factors (such as demographic, technological, land use, and economic changes). Project planners are therefore ill-equipped to incorporate uncertain climate information into a broader (all-uncertainty) assessment of a project's probability of success, and thus to make intelligent modifi cations to the project design to reduce its vulnerabilities to failure. And if the project planner succeeds in characterizing the relative importance of various risks and system vulnerabilities, the choice remains as to how best to manage those risks to improve system robustness and fl exibility. Though a number of analytical tools have been developed, engagement with the tools can be complex and expensive.
In the typical engagement with science, the scientifi c analysis reduces uncertainty and identifi es a likely future, at which point the planner can select the best options for that future. However, given that climate science is not in a position to present a likely future of limited and reasonable range, a diff erent approach to project assessment and decision making is needed. This book puts forth a decision support tool in the form of a "decision tree" to meet this need (see fi gure ES.1).
The decision tree provides guidance on the application of proven techniques for climate change risk assessment and advanced tools for risk management. Decision scaling, upon which the decision tree's general structure is based, is a bottom-up, robustness-based approach to water system planning that uses a stress test for the identifi cation of system vulnerabilities and simple, direct techniques for the iterative reduction of system vulnerabilities through targeted design modifi cations. Decision scaling features prominently in the riskassessment aspects of the decision tree because it is effi cient and scientifi cally defensible (that is, it does not involve numerous assumptions about the future, nor does it rely on GCMs for direct climate input). Moreover, it makes the best use of available climate change projections, which, though highly uncertain, can still be useful under particular conditions. It creates those conditions by assessing the relative performance and vulnerabilities of alternatives; using that information to describe future scenarios; and then applying the available information regarding local climate trends, climate projections, and historical climate variability to answer specifi c questions that arise during the decisionmaking process.
In addition to addressing the fundamental science issues, the decision tree framework was designed with the economic use of human and fi nancial resources in mind. The goal was to develop a tool that would be applicable to all

IDENTIFYING AND MANAGING CLIMATE RISKS
A rapid project scoping exercise, using a simplified water resources system model that compares climate impacts with others such as existing variability, population growth, and other variables Climate sensitivity screening for all Bank projects: Is climate a factor to take into account?

Introduction
Water infrastructure projects are a signifi cant portion of the World Bank's lending portfolio and a major need for developing countries throughout the world. Many water resources projects have long periods of economic return, with signifi cant uncertainties in the behavior of the natural system as well as that of human factors (technology, population dynamics, economic development, and the like). Traditionally, attempts to quantify and incorporate those uncertainties in planning tools have assumed stationarity of historical trends. For example, the concept of hydrologic stationarity historically has been deemed adequate for water project design with at least partial understanding that hydrologic response (due to land use changes) and hydrologic variability (due to climate variability and change) are not fully stationary. The uncertainties associated with climate change, however, have led to a reconsideration of whether the water development community is adequately taking into account the uncertainties that characterize the future (Milly et al. 2008). This additional scrutiny of water resources projects is warranted, given the large potential regrets associated with possible stark climate changes in the future. However, the means for conducting this additional assessment is unclear. No general methodology has been accepted for assessing the signifi cance of climate risks relative to all other risks for water resources projects, nor is there an accepted process within the Bank.
The need for an accepted process for climate change risk assessment at the Bank has recently been elevated. Under the IDA17 Replenishment 1 (which added $52 billion to the resources of the International Development Association [IDA], the Bank's fund for the poorest), Special Theme 3 on climate change calls for all IDA Country Partnership Frameworks to incorporate climate-and disaster-risk considerations into the analysis of the country's development challenges and priorities, and, when agreed upon with the country, to incorporate such considerations in the content of the programs and the results framework.
The goal of this book is to outline a pragmatic process for risk assessment of Bank water resources projects that can serve as a decision support framework-a "decision tree"-to assist project planning under uncertainty.
The decision tree described in this book is based on the growing consensus that robustness-based approaches are needed to address uncertainty and its potential impacts on infrastructure planning . These approaches emphasize assessment of individual projects and their ability to perform well over a wide range of future uncertainty, including climate and other uncertainties (Brown and Wilby 2012;). Together, they are an alternative to the most prominent kind of approaches to risk assessment, so-called top-down approaches, which assess system response under a limited set of plausible future climate, demographic, and land use conditions, with climate projections downscaled from time series of general circulation models (GCMs). Considerable investment has been made in climate modeling and downscaling with the aim of benefi ting decision makers. However, a recent study by the Bank's Independent Evaluation Group (IEG) found that "climate models have been more useful for setting context than for informing investment and policy choices" (IEG 2012, 61) and "they often have relatively low value added for many of the applications described" (IEG 2012, 69).
The lack of success in the use of climate projections to inform decisions is not due to lack of eff ort in translating model outputs to a form relevant to decision makers. Instead, two fundamental and unavoidable issues limit the utility of these approaches. The fi rst could be classifi ed as a risk assessment problem. The uncertainty associated with future climate is largely irreducible in the temporal and spatial scales that are relevant to water resources projects ). Climate projections provide limited and often biased explorations of the eff ects of internal climate variability, especially precipitation variability (Rocheta et al. 2014), with amplifi ed carryover eff ects for runoff estimates (Fekete et al. 2004). As a result, climate science-led eff orts do not typically reduce the uncertainty of future climate in a way that is relevant for water systems planning and, in fact, are unlikely to describe the limits of the range of possible climate changes. Nor are they able to provide probabilistic representations of the uncertainty (Hall 2007).
Because risk is a function of both probability and impact (Dessai and Hulme 2004), the inability of climate projections to probabilistically represent uncertainty is a substantial impediment to the assessment of climate-related risks for proposed water resources projects. Perhaps most important, GCMs have the least skill in generating the variables that are most important for water resources projects, such as local hydrologic variability and extremes (for example, fl ood and drought). Those extreme events are located at the tails of distributions of climate variables and percentage-wise will change more rapidly than the mean in a changing climate (Dai, Trenberth, and Karl 1998). Often, the results of a climate change analysis present a wide range of possible future mean climates, without any insight into climate extremes, and give the sense that they are only the tip of the iceberg for climate uncertainty. As a result, the project planner gains little insight into the potential impact of climate change on the project.
The second issue relates to risk management. As described in chapter 2, some researchers have suggested that the magnitude of the eff ects of changes in climate on water resources might be small relative to the impact of changes in other variables such as population, technology, and demand, over medium-to long-range periods (for example, Frederick and Major 1997;Lins and Stakhiv 1998); others have suggested that climate change may be the most important factor for long-range development planning (for example, Arnell and Lloyd-Hughes 2014;Rockstrom et al. 2009); and still others have argued that the relative likelihoods of long-range change magnitudes are not quantifi able from our present limited perspective (for example, Allen, Raper, and Mitchell 2001; Lempert et al. 2004). Project planners are therefore ill-equipped to incorporate climate information and all its uncertainties into a broader (all-uncertainty) assessment of a project's probability of success, and thus to make intelligent modifi cations to the project design to reduce its vulnerabilities to failure. And if the project planner succeeds in characterizing the relative importance of various risks and system vulnerabilities, the choice remains as to how best to manage those risks to improve system robustness and fl exibility. Though a number of analytical tools have been developed, engagement with the tools can be complex and expensive.
In the typical engagement with science, the scientifi c analysis reduces uncertainty and identifi es a likely future, as a result of which the planner can select the best options for that future. However, given that climate science is not in a position to present a likely future of limited and reasonable range, a diff erent approach to project assessment and decision making is needed.
The decision tree is designed to address these fundamental issues to provide a path forward for project planners who face decisions potentially aff ected by climate change uncertainty. In addition to addressing the fundamental science issues described above, the decision tree was also designed so that human and fi nancial resources could be used economically. Water resources projects are quite diverse, including water sector reform, water management, development of hydrometeorological networks, and establishment of new infrastructure, including water supply, sanitation, and hydroelectric facilities. The goal of this work was to develop a tool that would be applicable to all water resources projects, but that would also allocate climate risk assessment eff ort in a way that is consistent with each project's potential sensitivity to that climate risk. To accomplish this objective, the process was designed to be hierarchical, with diff erent stages or phases of analysis potentially triggered by the fi ndings of the previous phase. The result is that diff erent categories of projects will undergo diff erent types of analysis, with eff ort that is proportionate to the need.
A project manager must account for uncertainties of many kinds when seeking approval for funding from a donor or board; most important, the project must be shown to be cost-eff ective and robust. The decision tree described here provides the project manager with a scientifi cally defensible, repeatable, direct, and clear method for demonstrating the robustness of a project to climate change. At the conclusion of the process, the project manager will be able to confi dently communicate the method by which the vulnerabilities of the project were carefully assessed, and how any necessary adjustments improved the project's feasibility and profi tability.
Before the decision tree framework is described in chapter 3, chapter 2 provides background on the risks relevant to water systems planning, describes the diff erent approaches to scenario defi nition in water systems planning, and introduces the decision scaling methodology upon which the general structure of the decision tree framework is based. As explained in chapter 2, decision scaling is a robustness-based, bottom-up approach to the integration of the best current methods of climate risk assessment, which supports simple, direct procedures for risk management. When faced with complex and interconnected system uncertainties, decision scaling can be combined with more advanced risk-management tools, such as robust decision making . Chapter 3 describes the decision tree tool, illustrated in fi gures 3.1 and 3.2, and explains each of the steps and processes that make up the tool. Chapter 4 focuses on a case study of a small hydropower project as an illustration of the decision tree procedure. Chapter 5 describes some of the tools available for decision making under uncertainty and methods available for climate risk management. Concluding thoughts on implementation are presented in chapter 6.

Risk Enumeration
As described by Arnell (1999), in the most general terms, water resources projects are subject to variable and uncertain pressures on the supply side and the demand side. Supply-side pressures include climate change (for example, reductions or increases in precipitation and changes in precipitation timing and intensity; increasing temperature, resulting in increased evaporation) as well as environmental degradation (decreasing the fraction of water available for use); infrastructure degradation (for example, the sedimentation of reservoirs, which decreases water storage capacity); and shifting agreements on the transboundary distribution of water resources. Among the demand-side pressures are population growth, migration, and concentration (for example, urbanization); shifting agricultural cropping and irrigation patterns; increased environmental demands (for example, low fl ow requirements in rivers); and economic or technological development or water tariff restructuring that increases or decreases the quantity of water used per capita. Climate change may aff ect the demand side as well as the supply side, especially as a result of changes in crop evapotranspiration.
Though climate-related pressures receive a large share of attention from governments and the research community (Kundzewicz et al. 2007), the infl uence of nonclimate factors is signifi cant and in many places even greater than that of climate. Already, many water resources systems are stressed by resource overallocation, political instability, and economic growth (UNESCO et al. 2012). Analogous to climate change projections, for which general circulation model (GCM) ensembles represent only the "lower bound on maximum range of uncertainty" (Stainforth et al. , 2163, projections of nonclimate factors are also highly uncertain. Water use scenarios are "notoriously diffi cult to make" (Arnell 1999, S33), and human population growth, one driver of water demand, is subject to myriad volatile and poorly understood factors (Cohen 2003).
Guidance on the estimation of the relative impact on water systems performance of climate change and changes in nonclimate factors is available in the form of broad global studies (for example, Alcamo, Floerke, and Maerker 2007;Arnell 1999;Arnell and Lloyd-Hughes 2014;Vorosmarty et al. 2000) and a large number of more geographically targeted case studies. The derivation of rules of thumb regarding the relative importance of climate and nonclimate uncertainties should be approached with caution because the conclusions of the studies are not easily reconcilable. Whereas Vorosmarty et al. (2000) fi nd projected changes in population to be a more signifi cant source of water stress than climate change in the short term (up to 2025), Arnell (1999) argues that the eff ects of climate change are likely to be more signifi cant in the long term (beyond 2025). Arnell and Lloyd-Hughes (2014) further conclude that uncertainty in projected future impacts of climate change on exposure to water stress and river fl ooding is dominated by uncertainty in the projected spatial and seasonal pattern of change in climate (especially precipitation), as represented by the available climate models. In studies at the local scale, Lownsbery (2014) fi nds uncertainties in projected precipitation change, not municipal or industrial development, to be the most signifi cant risk to the Apalachicola-Chattahoochee-Flint (ACF) River Basin; and Frans et al. (2013) determine that growing regional precipitation, not land use change, is the dominant driver of positive runoff trends in the Upper Mississippi River Basin. In contrast,  observe demographic change projections to have a greater impact on the future water balance of Amman, Jordan, than climate change projections. One statistical tool that might help in the partitioning and ranking of the climate and nonclimate uncertainties facing water systems is the analysis of variance, 1 similar to that presented by Hawkins and Sutton (2009) for climate models, and applied to the ACF River Basin by Lownsbery (2014).
In brief, the relative infl uence of climate and nonclimate factors is particular to each project context, and must be attended to in each unique case. When evaluating the relative importance of climate and nonclimate factors, issues that deserve particular attention are initial conditions (whether water supplies are already stressed), recent local climate and demographic trends, and project lifetime, with longer-lived projects likely to experience greater climate-related pressure.
The baseline expectation is that a proposed water project would perform satisfactorily with a continuation of current demand-and supplyside conditions. Continuation of such conditions, however, cannot be assumed. When evaluating the robustness of a water project to change, it is important to estimate, as specifi cally as possible, reasonable ranges of shift in all relevant supply-and demand-side factors. Clearly, not all factors are relevant to every water project, and furthermore, though changes in a certain factor might have an eff ect on system performance, it may be that no reliable local projections for future values of that factor can be established. Eff ort should therefore be expended according to the relative signifi cance to system performance of shifts in the various factors, as well as the limitations posed by the available data. Once approximate ranges of future values of the relevant change factors have been estimated, the process of factor-specifi c and cumulative risk assessment can begin.
Alternative Approaches to Scenario Defi nition Schwartz's The Art of the Long View (1996) popularized the businessmanagement (and business-growth) concepts of scenario planning. Internally consistent scenarios of the type endorsed by scenario planning are the basis for most approaches to climate change risk assessment and adaptation planning. Scenarios have narrative power to open the imagination to future worlds. The starting point for imagination of future climates has typically been the output of GCMs, which provide the most advanced, model-based sources of scientifi c information about future climate that are generally available. The next section describes this approach. The use of preexisting climate scenarios is often convenient because of the ready availability of output from GCMs; however, such scenarios are ill-equipped to illuminate a clear path for long-range water system planning because of the issues described in chapter 1. The "Ex Post Scenario Defi nition" section of this chapter therefore explains an alternative basis for risk analysis in water system planning that is the backbone of decision scaling and other similar approaches: to fi rst subject the proposed project to a scenario-independent climate stress test, and then (ex post) allow the vulnerabilities of the system to defi ne scenarios of interest.

Ex Ante Scenario Defi nition
Traditional decision analysis in water systems planning has used scenarios generated from internally consistent storylines of future global development, downscaled for local or regional applications (for example, Christensen and Lettenmaier 2007;Fowler and Kilsby 2007;Lettenmaier et al. 1999;Minville et al. 2009;Sun et al. 2008). In this approach to scenario development, GCMs provide estimates of future regional (a typical GCM grid is approximately 250 to 600 kilometers to a side) climate conditions (for example, temperature and precipitation) in the context of a variety of possible global economic, technological, and social trends. Because they link climate and demographics, these storylines can provide guidance for projections of both future water availability and future water consumption, although their use is typically limited to projections of future carbon emissions.
The output of a number of GCMs is available for each projected state of the world in the form of future climate time series. These future climate time series can be obtained before any examination of the project under consideration begins, and as such they defi ne an ex ante, bounded (limited, and possibly ill-selected) set of scenarios over which the performance of the project can be tested. Water systems models using ex ante scenarios, also referred to as the "scenario-led" approach, test the performance of the system using a sample of the ex ante futures described by the storylines. Studies of this type translate time series of climate parameters (for example, average daily temperature and precipitation) from the downscaled GCMs for a particular internally consistent scenario (RCP8.5, for instance) into projections of future streamfl ow using a hydrologic model (examples from fl ood risk management include Cameron, Beven, and Naden 2000;Loukas, Vasiliades, and Dalezios 2002;Muzik 2002;Prudhomme, Jakob, and Svensson 2003;Willems and Vrac 2011).
Robustness-based approaches to water systems planning make the strategically critical leap of emphasizing preparedness for a range of possible futures. The importance of this ideological shift is not to be understated. However, most planning exercises still base the projections of future climate on downscaled GCMs, tying the relative severity of climate projections to demographic, economic, and technological trends as described by IPCC storylines.
Top-down frameworks using ex ante climate scenarios can help quantify the relative contribution of diff erent components to overall uncertainty for extremes such as low fl ows (for example, Wilby and Harris 2006). Moreover, very high resolution regional climate models are now being used to investigate the sensitivity of extreme precipitation to temperature forcing (for example, Kendon et al. 2014). In other words, climate models and downscaling methods can be usefully deployed to strengthen the understanding of the physical processes or critical thresholds that drive hydrological extremes.
However, top-down climate assessments rely heavily on GCM outputs for delineation of the ex ante scenarios describing local and regional climate impacts. Most top-down approaches begin with a small selection of all available GCM output, which, because each GCM will represent the particular climate dynamics of relevance to the project with varying skill, greatly increases the risk that the wrong (unskilled) GCM subset is used. Even if the output from all available GCMs is used, the GCMs represent only a subset of all possible climate futures. This is evidenced in the increase in uncertainty in the range of climate forecasts of the Coupled Model Intercomparison Project (CMIP) from phase 3 to phase 5-the phase 3 ensemble was an underestimate of the phase 5 uncertainty, which may be an underestimate of the uncertainty in the next CMIP phase. As a result, top-down methods do not sample from the full range of climate futures, and may sample from the wrong range entirely. The process of downscaling GCMs results in a cascade of uncertainty . Furthermore, all models have similar resolution and must put parameters around the same processes (Tebaldi and Knutti 2007). Uncertainties that are related to the underlying science will be the same in diff erent models.
Top-down climate change analyses present a wide range of possible mean future climate conditions, but they do not adequately describe the range of potential future conditions more generally ). In addition, top-down analyses provide limited insight into the changes to climate drivers (such as monsoon patterns and atmospheric rivers) and climate extremes (Olsen and Gilroy 2012). They provide the least information for the variables that are most important for water resources projects, such as hydrologic variability and extremes (for example, fl oods and droughts). Therefore, deriving probability distributions of these events from an ex ante ensemble of GCMs is fraught with problems. Given the essential role of likelihood concepts in risk assessment (in which risk is a function of impact and probability of that impact), topdown methods tend not to provide the insights needed for water resources system planning.
A subtle point in this discussion deserves emphasis: scenario-led studies tend by necessity to select a small subset of all possible futures because of the large computational eff ort required for generating usable time series from GCM projections for the spatial and temporal context required, in addition to generating internally consistent combinations of uncertain parameters and repeatedly testing the performance of the system. This approach is often prohibitively ineffi cient. Without targeted information about the vulnerabilities of the system, great eff ort might be expended in the development and evaluation of ex ante scenarios that off er no further information regarding the system design because they fail to exert stress on the system, or because they do not provide information about which future is more likely nor delimit the range of climate change and variability that might be experienced.
It should be noted that downscaled GCM output is not the only source of climate information in the scenario-led approach. In fact, it is simply an updated input for traditional water resources analysis that had been based on historical time series. Scenarios have been developed through the perturbation and stochastic resampling of historical fl ow patterns using Markov Chain bootstrap techniques Sharma, Tarboton, and Lall 1997;Yates et al. 2003) or the assignment of a "change factor" to historical temperature and precipitation values based on climate shifts identifi ed by GCM output (Hay, Wilby, and Leavesley 2000). Use of these other sources of climate information could result in improved accuracy relative to downscaled GCM output (and could better represent local drought and fl ood risks, particularly), but it does not solve the problem of the ineffi ciency of scenario development and testing.
Generally, successful approaches to scenario-led planning require many thousands (if not hundreds of thousands or millions) of runs of sophisticated scenario-generating algorithms to trace out the decision space. Successful approaches also tend to require heavy involvement from high-level experts capable of guiding the process.

Ex Post Scenario Defi nition
In contrast to ex ante approaches, scenarios can be generated by parametrically  or stochastically (Brown et al. 2011; varying the climate (and other) data to identify vulnerabilities in water system performance, and elaborating scenarios ex post according to the vulnerabilities of the project or the opportunities it presents. Water system models using ex post scenarios test the performance of the system across a very wide range of potential futures (climate and nonclimate permutations), beyond the scope of the futures suggested by the IPCC narratives and associated GCM models. Scenarios are thereby defi ned as those futures in which the system struggles or fails. Scenarios defi ned ex post are meant to identify suboptimal system performance, and are less likely to underestimate vulnerabilities. In addition, the ex post defi nition of scenarios may facilitate the assignment of relative or subjective probabilities to the scenarios.
As discussed in greater depth later in the book, ex post approaches, although they may involve nearly as many model simulations as ex ante approaches, reduce the computational eff ort by saving scenario defi nitions that describe only those futures in which the project has demonstrated vulnerabilities. By taking an ex post approach to scenario development, the additional level of eff ort expended by ex ante approaches on the presimulation development of internally consistent scenarios is not required.
An additional benefi t of ex post scenario defi nition, if scenario defi nition is necessary at all, is that the considerable uncertainty associated with climate change projections does not enter the analysis until the stage of assigning probabilities. By withholding the use of the projections until the end of the modeling process, rather than inserting them at the beginning, the problem of propagation of uncertainty throughout the analysis (Carter et al. 2007;Jones 2000) is ameliorated. The identifi cation of "narrative scenarios" or clusters of scenarios that are descriptive of futures of concern (to which, for example, the system under consideration is vulnerable) also becomes possible (Groves and Lempert 2007). When only a few uncertainties are considered, such as climate sequences diff erentiated by average precipitation and temperature, such scenario clusters can be identifi ed visually. When analyses consider more dimensions of uncertainty, scenario-discovery algorithms can be useful ); these algorithms identify and display the most important combinations of uncertainties aff ecting system performance. Such scenarios can be useful in communicating information about system vulnerabilities to decision makers (Parker et al. 2013).

Exploring Vulnerabilities: Weather Generators and the Climate Stress Test
To defi ne ex post scenarios, the system's performance needs to be explored. Internally consistent coupled projections of climate and nonclimate factors are typically used to conduct this exploration. Alternatively, a stress test can be used to effi ciently explore the performance of alternatives and identify weaknesses. The stress test is a process by which an option is exposed to a variety of plausible climate and nonclimate changes, modifying means and other aspects of variability, to identify vulnerabilities. The analytical engine of the climate aspects of the stress test is an update to traditional weather generators.
Weather generators are computer algorithms that produce long series of synthetic daily weather data. The parameters of the model are conditioned on existing meteorological records to ensure that the characteristics of historical weather emerge in the daily stochastic process. Weather generators are a common tool for extending meteorological records (Richardson 1985); supplementing weather data in a region of data scarcity (Hutchinson 1995); disaggregating seasonal hydroclimatic forecasts (Wilks 2002); and downscaling coarse, long-term climate projections to fi ne-resolution, daily weather for Ex ante scenario development is the generation of scenarios from internally consistent storylines of future global conditions (including, but not limited to, demographic, economic, land use, and climate conditions, or skipping straight to radiative forcing) prior to evaluation of the climate-related vulnerabilities of (or opportunities for) the project under investigation. Water system models using ex ante scenarios, also referred to as the "scenario-led" approach, test the performance of the system across a sample of futures described by the storylines. Ex ante scenarios tend to refer to Intergovernmental Panel on Climate Change (IPCC) storylines, and water system evaluations using ex ante scenarios tend to take climate information directly from general circulation models (GCMs).
Ex post scenario development is the generation of scenarios by parametrically or stochastically varying the climate (and other) data to identify vulnerabilities in water system performance, and elaborating scenarios according to the vulnerabilities of the project or the opportunities it presents. Scenarios defi ned ex post aim to identify suboptimal system performance (in whatever future state that might occur), and are less likely to underestimate vulnerabilities. In addition, the ex post defi nition of scenarios may facilitate the assignment of relative or subjective probabilities to the scenarios. Because GCM projections are not entered into ex post scenarios until the end of the analysis, ex post scenarios are less susceptible to the welldocumented shortcomings of GCMs.

BOX 2.1 Ex Ante versus Ex Post Scenario Development
impact studies Wilks 1992). A major benefi t aff orded by most weather generators is their utility in performing climate-sensitivity analyses (Wilks and Wilby 1999). Several studies have used weather generators to systematically test the climate sensitivity of impact models, particularly in the agricultural sector (Confalonieri 2012;Dubrovsky, Zalud, and Stastna 2000;Mearns, Rosenzweig, and Goldberg 1996;Riha, Wilks, and Simoens 1996;Semenov and Porter 1995). These sensitivity studies systematically change parameters in the model to produce new sequences of weather variables (precipitation, for instance) that exhibit a wide range of change in their characteristics (such as average amount, frequency, intensity, and duration).
In the context of a climate stress test, a stochastic weather generator can be built for a region of interest and used to generate several scenarios of daily climate within which a water resources system can be tested. The fl exibility of stochastic weather generators enables many climate permutations to be generated, each of which can exhibit a diff erent type of climate alteration that the analyst may be interested in. Note that the permutations created by the weather generator are not dependent on any climate projections at this point in the analysis, thereby allowing a wide range of possible future climates to be generated while avoiding biases propagated from the projections. However, the particular permutations generated can be informed by available projections to ensure that they more than encompass the range of GCM projections. The stochastically generated climate permutations form the foundation of the climate vulnerability assessment. Many diff erent weather generators are mentioned in the scientifi c literature for use in the climate stress test; a unique model that incorporates low-frequency variability was developed for the case study presented in chapter 4.

Paleoclimatology Data
One aspect of risk estimation that has traditionally received less attention but is becoming more important to the understanding of likely future climate scenarios is the use of paleoclimatology data to inform expectations for the future. Paleoclimatology data make available much extended data sets (500 or more years), which would be important to inform the risks to which the water system could be exposed. These data are based on observations from the history of a given location, and not on projections of what recently unprecedented conditions might be realized in the future. Often, unfortunately, paleoclimatology data are only available for specifi c variables and at coarse temporal resolution (annual or decadal). Risks associated with a nonstationary climate have been presented as deviations from observations of the past 100 years or so of record; however, natural climate cycles leading to extreme fl ooding and drought that repeat on periods greater than a single century are likely to provide much better information about the risks faced in the economic lifetime of long-lived water infrastructure (e.g., dams, canals).
Paleoclimatology data have featured prominently in climate risk assessments of the Colorado River (for example, Meko and Woodhouse 2011;Prairie and Rajagopalan 2007) and in decision scaling applications for the Great Lakes (for example, Brown et al. 2011;Moody and Brown 2012).

Stakeholder Expertise: Top-Down and Bottom-Up
In climate change planning, other studies (for example, ) have defi ned "top-down" to refer to the scenario-led use of downscaled GCMs to describe future regional climate, inputting the scenarios into impact models, and then prescribing adaptations. By contrast, "bottom-up" has been described as ex post scenario development (that is, not beginning with ex ante scenarios encompassed in downscaled GCM output). Bottom-up in this sense does not solely refer to the origination of project motivations, design parameters, and performance thresholds with stakeholders. The use of the term "bottom-up" as adopted by the decision tree framework encompasses both aspects-decision-making pathways (beginning with stakeholders) and climatological (ex post climate scenario defi nition).

Background on Decision Scaling
Decision scaling (also referred to as climate informed decision analysis) is an approach to the integration of the best current methods for climate risk assessment and robust decision analysis with simple procedures for risk management. Decision scaling, as outlined in fi gure 2.1, is a robustness-based approach to water system planning that makes use of a stress test for the identifi cation of system vulnerabilities, and simple, direct techniques for the The term "bottom-up, " as adopted in this book, refers to both the emphasis on system vulnerabilities (through ex post scenario development, including late-stage incorporation of climate projection information from general circulation models), and the active involvement of stakeholders in the decision-making process.

BOX 2.2
Bottom-Up, Climate-Informed Decision Making iterative reduction of system vulnerabilities through targeted design modifications. The decision scaling methodology has been presented in a number of publications (for example, Brown 2010Brown , 2013Brown et al. 2011;. Other tools are discussed in the "Risk Assessment Tools" section of chapter 5. Decision scaling features prominently in both the risk-assessment aspects and general structure of the decision tree (see chapter 3) because it is effi cient and scientifi cally defensible (that is, it does not involve numerous ex ante assumptions about the future, nor rely on GCMs for direct climate input). C li m a t e p a r a m e t e r x 2

FIGURE 2.1 Schematic Comparison of Decision Scaling with Traditional Approach to Climate Change Risk Assessment
Note: GCM = general circulation model; RCM = regional climate model.
Moreover, it makes the best use of available climate information, which is typically highly inexact but may still be useful under certain conditions. Decision scaling creates those conditions by assessing the relative performance and vulnerabilities of alternative system designs, using that information to describe ex post scenarios, then applying the available climate information to assist in decision making.
Because various sources of climate information can be applied without rerunning the model, decision scaling makes use of all sources of climate information (for example, frequency analysis of GCM output, historical data, paleoclimatology data, stochastically generated climate simulations, and expert judgment of scientists and stakeholders) to inform the likelihoods of diff erent types of climate change. When climate information is deemed fairly reliable and projections are consistent, model-based probabilistic estimates of risk can be made and risk-weighted decision making can be used. In the event that projections based on the various sources are contradictory, not relevant, or not credible, the process enables the identifi cation of climate sensitivities and provides a framework for addressing potential hazards through robustness approaches (see chapter 5 for examples).
Decision scaling is also bottom-up in its development of decision-making pathways. The fi rst step is stakeholder consultation for identifi cation and characterization of historical system performance, desired future performance thresholds, and vulnerabilities to change. Whereas standard decision analysis requires well-characterized uncertainties, decision scaling was developed to handle poorly characterized uncertainties and make the best use of available information.
By serving as a climate stress test designed to evaluate the response of the system to a wide range of plausible climates, beyond the range of the IPCC GCMs but with guidance from those GCMs, decision scaling determines whether the time-and effort-intensive process of GCM downscaling is likely to be beneficial. The resulting climate response function provides insight into the expected performance of the system in an uncertain future. Decision scaling does not include an explicit framework for risk management, although it contributes many of the informational elements required of a decision tool. For decision scaling to be used as a decision analysis tool, it would need to trade off the benefits and costs of mitigating strategies, either by way of an optimization algorithm or by Monte Carlo simulation. Other tools, such as robust decision making, specialize in risk and vulnerability management and have the potential to greatly expand the risk-management capabilities of the decision analysis process.
A decision scaling analysis can be accomplished in a couple of months if all modeling tools are preexisting and available (at a cost below $100,000), or may take one to two years if all required models must fi rst be developed (at a cost of up to $200,000). For comparisons of level of eff ort and cost with other methods, see .

Applications of the Decision Scaling Methodology
Examples of applications of the decision scaling methodology are available for the Niger Basin (Brown 2010) and the Upper Great Lakes in North America (Brown et al. 2011;Moody andBrown 2012, 2013). The application to the Upper Great Lakes is instructive with regard to the need for additional measures to address climate risks once they have been identifi ed. An adaptive management plan has been advocated for the Great Lakes and is currently in development by the International Joint Commission (the binational governing body for the Great Lakes). Given the potential for (and indeed likelihood of ) faulty assumptions and unforeseen surprises that threaten the success of the Lake Superior regulation plan, and given the signifi cant magnitude of negative consequences to a large portion of North America in the event of a failure of the regulation plan, contingencies were established for residual risks and for unlikely but catastrophic extreme events.
Decision scaling assists in the development of adaptive management plans that establish contingencies in proportion to the negative consequences of system failure. To sustain monitoring and provide mechanisms for use of the collected data in decision making, an institutional framework for the adaptive management process is required. In other cases, alternative approaches to addressing climate risk may be warranted or more appropriate, as discussed in Phase 4 of the decision tree framework (described in chapter 3). Notes 1. Analysis of variance is a statistical method used to test diff erences between two or more means; inferences are made about the means by analyzing variance. 2. Radiative forcing, as defi ned by Moss et al. (2010 ), is "the change in the balance between incoming and outgoing radiation to the atmosphere caused by changes in atmospheric constituents, such as carbon dioxide."

Introduction
The decision tree presented in this chapter is based on the basic principles of bottom-up climate risk assessment methodologies, such as developed in decision scaling and robust decision making (RDM). In preparing this approach, other existing decision support tools were reviewed, including those produced by the United Kingdom Climate Impacts Programme, the World Bank Climate Change Group, the United States Agency for International Development, and the World Resources Institute. Insights from this review were incorporated into this tool's design. An overview of the process is presented in fi gure 3.1. This process is hierarchical, meaning that in each phase of the analysis, either the process ends because the climate risks have been adequately addressed or the process proceeds to the next phase to address remaining concerns. Phases 1 through 3 are elements of risk assessment. Phase 4 shifts to risk management. This chapter provides a detailed description of each phase in the decision scaling methodology.
The overarching objective of the decision tree framework is to provide a consistent, credible, and repeatable process for project managers to use to assess climate risks. It is designed such that eff ort expended is proportional to the climate sensitivity of the project in question. It is also designed to be implemented largely internally, although at some points outside expertise may be warranted. When outside expertise is needed, the framework provides clear guidance on the process that should be specifi ed in the contract. Finally, the framework is expected to allow a project manager to feel confi dent that climate change risks have been assessed and addressed, and for management and the larger stakeholder community to agree.

FIGURE 3.1 General Steps in the Decision Tree for Water Resources Projects
• A complete coupled hydrologic-economic water system model is built, if not already available.
• An exhaustive climate stress test is conducted to identify the climate sensitivity of the system, and is presented on a climate response map. • The addition of historical data, paleoclimatology data, and general circulation model projections to the climate response map illustrates the risk to the system within the plausible range of climate changes relevant in the economic lifetime of the project. • Projects shown to be insensitive to plausible climate changes are categorized as climate invulnerable in a Climate Risk Report. • Climate response maps revealing ambiguity in the projected climate-related impacts are subjected to a credibility assessment. • The robustness of the project to credible, plausible risks is examined.

Phase 3: Climate Stress Test
• A water resources system model (at least a low-level approximation of the system) is developed, if not already available. • An estimate of the magnitude of potential climate stressors relative to stressors of other types is developed through a rapid project scoping exercise and reported in a Climate Risk Statement.

Phase 2: Initial Analysis
• If the robustness is unsatisfactory, and cannot be improved, alternatives are pursued.
• If the robustness can be easily and directly improved, adjustments are made to the project design, and the revised project is returned to the climate stress test. • If doubt remains about the robustness of the project, and robustness improvements cannot be made with simple adjustments to design parameters, then advanced tools for decision making under uncertainty are employed using the ex post scenarios identified during Phase 3 and elaborated on in Phase 4. • The results are presented in a Climate Risk Management Plan.

Phase 4: Climate Risk Management
• Stakeholder-defined performance indicators and risk thresholds are established. • The proposed project is classified according to its climate sensitivity. • Context analysis is performed using the Four C's framework, and potential climate vulnerabilities are described relative to potential vulnerabilities of other types.

Phase 1: Project Screening
Figure 3.2 presents a schematic workfl ow of the tool designed for water resources projects. The procedure consists of four hierarchical phases-Phase 1: Project Screening; Phase 2: Initial Analysis; Phase 3: Climate Stress Test; and Phase 4: Climate Risk Management. The overarching goal of the decision tree framework is to enable the project manager to confi dently estimate the performance of the considered project, and to anticipate its impacts (positive and negative) on the natural and anthropocentric environment into which it would be introduced. More generic water resources system planning tools developed to accomplish this goal take climate (and other) information as input. The particular strength of the framework presented in this book is its handling of climate-related uncertainty. For situations in which climate information, among all sources of uncertain information, is inadequate for the task of evaluating project performance, the decision tree provides bottom-up procedures for clarifying the climate information.
Phase 1 consists of a well-defi ned, self-guided desktop screening of the project, to be conducted with the assistance of directed questions included on a Climate Screening Worksheet. The project manager would execute Phase 1 with little need for expert consultation. Projects that do not have signifi cant climate sensitivities are excused from further analysis, and projects that do have potentially signifi cant climate sensitivities may still be excused if those sensitivities are shown to be minor relative to other, larger, project-performance sensitivities. The judgment is made by the project planner and facilitated by guiding questions.
Projects classifi ed as climate sensitive in Phase 1 move to Phase 2, which consists of an initial analysis, during which it is necessary to build a fi rst-order approximate model of the proposed water system (if such a model does not already exist) and incorporate the expert input of stakeholders, regional managers, and others. Though not a thorough climate change risk assessment, the rapid scoping exercise is able to estimate a project's relative sensitivity to climate changes in the range projected by available general circulation models (GCMs) and indicate whether a more in-depth climate change risk assessment is required.
For projects with climate sensitivities that are signifi cant relative to other, nonclimate, sources of performance sensitivities, a Phase 3 stress test is recommended. This step requires the use of a coupled hydrologic-economic analytical model to assess the climate sensitivity of the project in quantitative terms. Phase 3 would likely be performed by a qualifi ed team of internal staff or expert consultants with knowledge of decision scaling applications.
Phases 1, 2, and 3 include points for exit from the decision tree, to be used when the climate information is deemed adequate (relative to other relevant sources of information) for the purpose of project evaluation. If signifi cant and

FIGURE 3.2 Decision Tree Schematic
Note: GCM = general circulation model. credible climate risk is identifi ed, Phase 4 risk management is required. In this case, climate vulnerabilities are reduced through design modifi cations (or, in the extreme, the abandonment of the original design) using tools for decision making under uncertainty. Methods for dealing with decision-making uncertainty and, in some cases, further geophysical analysis, are needed. Chapter 5 discusses decision making under uncertainty in more depth.
For each phase, a product has been specifi ed that would be appropriate for inclusion in the project folder and for presentation to reviewers to demonstrate that climate risks have been assessed according to an approved procedure. For example, Phase 1 results in a completed Climate Screening Worksheet, demonstrating the climate sensitivity (or lack thereof ) of a given project. Phase 4, when necessary, results in an in-depth report, the Climate Risk Management Plan, that outlines the climate risks and proposed means of addressing them. Each step is defi ned in greater detail in the rest of this chapter.

Phase 1: Project Screening
In Phase 1, a project is quickly screened to establish whether it is sensitive to climate factors.

Entry into Phase 1
Phase 1 is the default starting point: all projects enter Phase 1.

Description of Phase 1
The objective of project screening is to quickly assess and "clear" water resources projects that do not have climate sensitivities. To aid in this process, a Climate Screening Worksheet has been developed to guide the project evaluator through a series of screening questions. The proposed project is classifi ed into one of only two categories: signifi cantly climate sensitive or not signifi cantly climate sensitive. The classifi cation would be based on results from climate risk screening tools 1 complemented by the answers to the questions in the worksheet. Examples of complementary screening questions include the following (see the Climate Screening Worksheet in appendix B for a broader list of questions): • What are the stakeholder-defi ned performance indicators (PIs) and risk thresholds? • Is this a water infrastructure project?
• What is the proposed economic lifetime of the project? • What discount rate is preferred (for example, a social-welfare or finance-equivalent rate; see box 3.1)? • Is this a water policy adjustment, training session, or environmental or water resources study (without infrastructure)? • Is this a hydrometeorological service project? • Does the project use historical streamfl ow or climate data in the design?
The worksheet is designed to lead to a clear classifi cation. In general, projects involving infrastructure would proceed to Phase 2, while more-policyoriented projects, such as water sector reform, training activities, and The discount rate is a particularly infl uential parameter in the ranking of investment alternatives. Higher discount rates result in lower economic design lives and deemphasize benefi ts accrued long into the future (for example, by future generations). Stern (2007) uses a consumption discount rate of 1.4 percent, while others (for example, Mendelsohn 2008; Nordhaus 2007) recommend higher discount rates-nearer 5 percent. (Nordhaus [2007], for instance, endorses a consumption discount rate of 4.3 percent.) The low consumption discount rate chosen by Stern results in a recommendation for a more aggressive policy to combat climate change, whereas the higher discount rate endorsed by Nordhaus results in a considerably more modest proposed effort.
In practice, the social discount rates used to evaluate the net benefi ts of proposed projects have varied widely, with developed nations typically applying a lower rate (3-7 percent) than developing nations (8-15 percent) (Zhuang et al. 2007). Organizations such as the World Bank, the Inter-American Development Bank, and the Asian Development Bank use a discount rate of 12 percent, although in some cases (such as water supply projects), 10 percent is the norm. The essential rationale for these elevated discount rates stems from the high value of scarce capital in developing countries: projects consuming large amounts of capital are required to account for the opportunity cost of these fi nancial resources, pushing up the expected rate of return.
Goulder and Williams (2012) distinguish between a social-welfare-equivalent discount rate, appropriate for determining whether a given policy would augment social welfare, and a fi nanceequivalent discount rate, suitable for determining whether the policy would offer a potential Pareto improvement. a The distinction underscores the need for active stakeholder participation in local determination of the discount rate. The relevant question to stakeholders is, for whom are the benefi ts of development projects intended (distributed in space and in time)? Though the deliberations through which social discount rates are chosen are disconnected from evaluations of the net present value of individual proposed projects, the conclusions reached go a long way toward establishing the attractiveness of the project under consideration. studies, would be classifi ed as climate insensitive and reach the endpoint of the climate risk analysis; this latter category may, however, continue through the decision tree to analyze other sources of uncertainty.

The Four C's
A key aspect of the Climate Screening Worksheet is the context analysis, which defi nes the bounds of the project being evaluated. For the purpose of this work, the context analysis is guided by a framework described as the Four C's: Choices, Consequences, Connections, and unCertainties. The project analyst, in consultation with stakeholders, will identify available project design choices; the consequences or performance metrics that will be used to evaluate the project's success; the connections through which choices and consequences are linked; and fi nally, the uncertainties that aff ect the conclusions that can be drawn about the profi tability of the project. Through this analytical approach to assessing the project, the relative eff ects of the various uncertainties are expected to emerge.

Product:
The Climate Screening Worksheet is a standardized document with a series of questions that result in the categorization of the project as either climate sensitive (leading to Phase 2) or not climate sensitive (leading to the end of the climate assessment process, and exit from Phase 1); it is approximately two pages long (see appendix B). The Climate Screening Worksheet is completed regardless of project categorization (climate sensitive or not). It is either submitted alone (as evidence of the climate insensitivity of the project), or included as part of more thorough climate assessment reports as described in relation to later phases of the decision tree.

Exit from Decision Tree after Phase 1
If the Climate Screening Worksheet suggests that the proposed project has no measurable climate sensitivities, the project may exit the decision tree (see fi gure 3.3). In this case, a climate risk assessment is not necessary and the project manager can proceed with standard procedures for evaluating the project. 2

Phase 2: Initial Analysis
In Phase 2, a cursory analysis is conducted to determine whether the identifi ed sensitivities to climate should be considered relatively insignifi cant compared with sensitivities to nonclimate factors.

Entry into Phase 2 from Phase 1
If the project is shown to have signifi cant potential climate sensitivities in the Climate Screening Worksheet, the decision tree process proceeds from Phase 1 to Phase 2.

Des cription of Phase 2
The intent of Phase 2 is to further explore the context of the project, and to identify cases in which climate sensitivities are present but deemed unlikely to be important relative to sensitivities of other types. Expert-led climate sensitivity analysis using either an existing water system model (typically developed as part of a prefeasibility study) or a simple water balance informed by available GCMs will serve to excuse many water resources projects from further in-depth climate risk assessment in Phases 3 and 4 of the decision tree.
A number of techniques are available for diagnosing the sensitivity of the system to climate changes relative to changes in other factors, ranging from simple one-factor-at-a-time sensitivity analysis (see the section on "Extensions to Traditional Decision Analysis" in chapter 5) to rapid project scoping (see the "Rapid Project Scoping" section in this chapter) to the "patient" rule induction method (PRIM)  Projects that do not qualify for jump-out from Phase 1: These projects have potential climate sensitivities that must be explored further.

FIGURE 3.3 Phase 1 Entry and Exit Conditions
Note: Four C's are Choices, Consequences, Connections, and unCertainties. See text for discussion.
PRIM can be used to identify the combinations of input parameters under which the system performs poorly. By systematically exploring all possible combinations of input parameters simultaneously, incremental changes in hydrologic system inputs can be ranked in signifi cance relative to incremental changes in nonhydrologic inputs. If a PRIM-based analysis were to identify relatively signifi cant system sensitivities to hydrologic changes within a range that might reasonably be expected to occur during the project's design life (for example, within 1 standard deviation of historical mean annual streamfl ow), those sensitivities would be explored in greater depth in a Phase 3 Climate Stress Test. In climate change risk assessment, PRIM is best applied as a screening tool to identify general system sensitivities, with more targeted explorations of the eff ects of potential changes in specifi c climate characteristics (mean temperature and precipitation, precipitation variability, seasonal shift, monsoon intensity and duration, timing and frequency of arrival of atmospheric rivers, and so forth) undertaken in Phase 3. Because PRIM is computationally intensive and involves a substantial measure of model development or customization, this section describes a less sophisticated and more direct method for characterizing the relative signifi cance of various system sensitivities, known as rapid project scoping.
Much of the analysis will often have already been completed by external consultants as part of the prefeasibility study, and would thus not need to be repeated by the project manager at this stage. An important responsibility of the project manager is therefore to thoroughly review existing hydrologic studies completed at earlier phases of the evaluation of the proposed water resources system investment.

Rapid Project Scoping
Rapid project scoping is based on a simplifi ed evaluation of the hydrology and climate change projections (Grijsen 2014b), as described, for example, in fi gure 3.4. The scoping analysis is designed to be executable by technical staff using spreadsheet software (Excel, for instance), though in many cases this analysis would be more effi ciently performed by external expert consultants. The analysis uses regressions to relate changes in climate parameters to changes in available water, and separate relationships to relate changes in available water to changes in system performance. Though the procedure outlined in fi gure 3.4 is not a thorough climate change risk assessment, it enables estimation of a project's relative sensitivity to climate changes in the range projected by available GCMs. If the potential risks to system performance are shown to be signifi cant, further analysis may be justifi ed in Phase 3 of the decision tree. If not, the project may be allowed to exit the decision tree.
The bulk of the eff ort in this phase is spent on water resources system modeling and analysis, to determine the runoff elasticity of selected PIs. If a water resources system model has not been developed, one will be needed for this analysis, even if it is only a basic description of essential system components using spreadsheet software. At this stage, publicly available hydrologic data will also need to be collected to create a baseline runoff record, and climate change projections will need to be acquired. A hydrologic model is required at this stage only if data from the historical record are of insuffi cient length or quality.
Eff ort on the order of a month or two of analysis for most projects will suffi ce, including all model development. If all required models are preexisting

Climate elasticities of runoff (Q)
Runoff elasticities of PIs
and available, the analysis may be accomplished much more quickly (in the range of two to three weeks). Using the system model, a fi rst-cut exploration of the local historical hydrology and climate change projections translates into estimates of the likelihood that selected PIs will perform inadequately. It is useful to express all results in terms of relative (that is, percentage) changes.
Relating percentage changes in PIs to percentage changes in runoff (as opposed to using absolute values) substantially reduces systemic errors in modeling, and redirects discussions with stakeholders away from disputable absolute numbers. The rapid scoping exercise involves the following steps: 1. Establish stakeholder-defi ned PIs and acceptable risk levels (thresholds). 2. Develop (or acquire) a water resources model of the proposed system (or proposed modifi cations to existing system). Use the system model to assess the response of the anthropocentric water system to varied hydrologic inputs. This step is accomplished by developing elasticities, ε Q , of basin PIs (GWh/year, fi rm power, irrigated area, minimum fl ow, navigation, and so forth) by parametrically varying basin runoff . These elasticities are applied to risk assessment in step 7. In addition to regression relationships, fi rst-order approximate (for example, mass balance) hydrologic modeling may be required if the record of historical streamfl ow observations is inadequate. Example elasticity plots are presented in fi gure 3.5. As shown in fi gure 3.5, the rapid scoping exercise need not be conducted on a single design at a time; it can be conducted on numerous design options simultaneously (or on a single prefeasibility option and a small number of alternatives). 3. Assess the response of the hydrologic system to varied climate inputs by parametrically varying fl ow and climatic boundary conditions. This step is accomplished by fi rst generating regression relationships between climate parameters and streamfl ow using time series of historical climate and streamfl ow data. In addition to regression relationships, fi rst-order hydrologic modeling may be required. The primary purpose of this step is to generate a second set of elasticities-climate elasticities to changes in precipitation and temperature-based on (1) historical runoff (Q) data and (2) hydrometeorological or gridded data sets of precipitation (P) and temperature (T). These elasticities quantify the response of the hydrologic system (as represented by runoff and streamfl ow) to climate drivers within the bounds of the historical record. 4. Use climate information from several sources of climate projections (for example, from the Nature Conservancy's Climate Wizard or the Bank's Climate Change Knowledge Portal) to assess the likelihood of climate risks for Note: GWh = gigawatt hour; HP = hydropower plants. These data apply to alternative designs using four and seven hydropower plants on the Sanaga River in Cameroon.
the specifi ed investment horizons. GCM data should be complemented with analysis of glacial melt water and sea level rise risks, where appropriate. 5. Develop probability density functions (PDFs) 3 of projected changes in precipitation and temperature (see fi gure 3.6 for an example). 6. Develop PDFs of projected changes in streamfl ow (or available water) by way of the hydrologic analysis in step 3 (see fi gure 3.6 for illustration). 7. Using the information from step 6 in combination with the water resources system model (and associated elasticities) developed in step 2, estimate the risks and probabilities of changes in the PIs of concern. This estimation is made by developing PDFs of "projected" changes in , where ε Q may be a nonlinear function of ∆Q/Q 0 (see fi gure 3.7 for an example).
An example of conclusion to the rapid project scoping, which results in exit from Phase 2, is presented for the Sanaga Basin, Cameroon, in Grijsen (2014a). In this case, rapid project scoping indicated that the total average energy generation of four run-of-the-river hydropower plants in the basin had an elasticity to fl ow (Q-elasticity) of 0.3 to 0.5, and that it was highly unlikely that the threshold for the stakeholder-defi ned PI (in this case a change in energy production relative to the baseline of > 20 percent) would be crossed by 2050 or 2080. There was a high probability that the economic

FIGURE 3.6 Example of Changes in Precipitation, Temperature, and Runoff According to General Circulation Model Projections
Note: GCM = general circulation model; RCP = representative concentration pathway. This illustration is for the Upper Niger Basin at Bamako, Mali. The symbol represents, respectively, precipitation, temperature, and runoff changes according to GCM climate projections for RCP 4.5 of the Coupled Model Intercomparison Project phase 5 (CMIP5) generation; the symbol represents those respective changes for RCP 8.5 of the CMIP5 generation. The reduced variate of the normal distribution is defi ned as Z = (x − μ)/σ. Z has a mean of zero and a standard deviation of 1, making it easier to visualize the effect of changes in the independent variables (precipitation, temperature, runoff) on the dependent variable (runoff, water system performance). internal rate of return (EIRR) of the Lom Pangar and Nachtigal dam projects on the Sanaga basin, specifi cally, would not change signifi cantly as a result of climate change (< 5 percent of the projected 14.5 percent EIRR). In the worst case, the EIRR was estimated to fall to 13 percent from 14.5 percent, still within the range of attractive investments for the project sponsor. It was therefore concluded that the evaluated hydropower projects are economically robust and climate insensitive. Climate risk screening and adaptation tools, such as the World Bank's Climate Change Knowledge Portal (http://sdwebx.worldbank.org/climateportal/ index.cfm) and the Nature Conservancy's Climate Wizard (http://www. climatewizard.org/), are helpful tools for quickly identifying anticipated changes in temperature, precipitation, and other climate conditions relevant to the location of the planned water project. The United Nations Development Programme's (UNDP) Adaptation Learning Mechanism (http://www. adaptationlearning.net/) off ers geographically targeted resources for climate change adaptation. If the proposed project is believed to be potentially sensitive to large decreases in precipitation, for example, these resources for climate information can inform the project manager of the likely magnitude of precipitation changes in the region of interest, allowing climate risks to be considered relative to risks of other types.
Additionally, as pointed out in step 2, the rapid scoping exercise need not be conducted on a single design at a time; the exercise can be conducted on numerous design options simultaneously (or on a single prefeasibility option and a small number of alternatives).
Completion of this analysis may require consultation with internal experts, or a regional manager or colleague. The expectation is that by completing the Four C's and the rapid scoping procedure presented in fi gure 3.4, the project manager will gain the clarity necessary to judge whether further analysis of the project for climate risks is warranted. If the analyst concludes that the project does not require further analysis, a Climate Risk Statement is completed (a template is provided for this statement in appendix B).
If sensitivity to climate is determined to be a signifi cant factor in the project's expected performance, a more detailed model of the system's response to climate changes may be required. If the project cannot be excused from an in-depth climate stress test, the now intermediate Climate Risk Statement can be skipped, given that its content will be covered in the Phase 3 Climate Risk Report.

Exit from Decision Tree after Phase 2
If Phase 2 of the decision tree shows that the project has climate sensitivities, but that those sensitivities are small relative to sensitivities to uncertain factors of other types (for example, demographic or political factors; see fi gure 3.8), standard internal procedures for evaluating the project should be followed after completion of the Climate Risk Statement. For expensive or complex projects, it is recommended, given the potential sensitivities identifi ed in Phase 1, that the project manager employ some extension of traditional decision analysis (example methods are summarized in "Extensions to Traditional Decision Analysis" in chapter 5) to assist in the selection of the best project design.
As was the case with exit from Phase 1, if potential nonclimate vulnerabilities have been shown in this phase to be much more signifi cant than potential climate vulnerabilities, a nonclimate version of the stress test might be used to explore those vulnerabilities quantitatively. Such methods are not described in detail here.

Product:
The Climate Risk Statement outlines the eff ects of uncertainty on the project and the expected relative eff ect of climate uncertainty in comparison with other uncertainties; it is normally fewer than fi ve pages long. The statement should justify why further climate analysis Phase 2: Initial Analysis Phase 1 projects that have potential climate sensitivities that must be explored Projects with climate sensitivities that are small relative to sensitivities to uncertain factors of other types (e.g., demographic or political factors). Based on a rapid project scoping, the project manager determines eligibility for jump-out from Phase 2. These projects can be evaluated with traditional decision analysis (plus robustness measures such as safety margins, sensitivity analysis, and adaptive management). Hydrological models may still be required.

Jump in
to Phase 3 Projects that do not qualify for jump-out from Phase 2: These projects have relatively significant potential climate vulnerabilities that must be explored further.

FIGURE 3.8 Phase 2 Entry and Exit Conditions
is not required, for instance, because of the dominant eff ect of other uncertainties. A template is provided for the completion of this statement in appendix B. The Climate Risk Statement is completed only if Phase 2 is exited; otherwise, the climate risks will be described as part of a more in-depth report on climate vulnerabilities (the Climate Risk Report of Phase 3 or the Climate Risk Management Plan of Phase 4).

Phase 3: Climate Stress Test
In Phase 3, a project is subjected to a so-called climate stress test, as outlined in box 3.2.

Enter Phase 3 from Phase 2
The Four C's analysis of Phase 1 will help identify the pathways through which the various uncertainties (climate and nonclimate) might aff ect the First, a weather generator is developed for the region of interest to produce numerous stochastic time series that preserve the variability and seasonal and spatial correlations of the historical record. These time series can be generated either by resampling directly from the historical record or by generating new time series based on the statistical characteristics of the historical record.
Next, the parameters are systematically changed to produce new sequences of weather variables (for example, precipitation) that exhibit a wide range of change in their characteristics (average amount, frequency, intensity, duration, and so forth). For instance, linear trends could be added to the precipitation and temperature of the numerous stochastic time series to simu-late climate change on a range informed by the available downscaled general circulation models. An example of a method for conducting the stress test is provided in Steinschneider and Brown (2013).
Using the stochastic time series, the hydrologic and water resources system model is then run repeatedly for the entire period for many future climates, for each of the water system plans under consideration.
Finally, the performance of each proposed plan is evaluated for a range of future climate states and the results are presented on a climate response map. Some aspects of system performance that may be evaluated are benefi t-cost ratio, total net benefi ts, economic rate of return, and violations of performance thresholds.

BOX 3.2 General Procedure for a Climate Stress Test
project. The initial analysis in Phase 2 will indicate whether the climate sensitivities identifi ed in Phase 1 are relatively signifi cant. If potential climate vulnerabilities are not insignifi cant relative to nonclimate risks, a Phase 3 analysis is begun.
Generally, infrastructure projects with design lives longer than 10-20 years, especially projects in geographic regions with high inter-or intra-annual climate variability, will require more thorough project scoping in Phase 3. More carefully constructed hydrologic and water resources system models will need to be constructed, with particular attention given to capturing potential changes or shifts in climate other than percentage changes in annual average temperature and precipitation.

Description of Phase 3
Proposed projects entering Phase 3 have potential climate-related vulnerabilities signifi cant enough relative to other risks that they cannot be dismissed. Phase 3 is the point at which the climate concerns are so prominent that they warrant heavier computational analysis. If a formal model of the natural, engineered, or socioeconomic system is not available at this point, it must be created to relate climate conditions to the impacts on PIs identifi ed in Phase 2. The third phase of the decision tree is the fi rst that is highly technical and will require an external expert consultant or qualifi ed internal experts. Data availability, as well as cost and time constraints for the analysis, must be duly considered. The overall approach to a climate stress test is described in box 3.2.
To explore the climate sensitivity of a project, climate response functions may be developed by systematically varying climate conditions and recording changes in performance metrics. The risks to the system are exhaustively explored and identifi ed by testing a wide variety of possible climate futures, beyond the narrow range available in a typical GCM analysis . A climate response map is a useful way to portray the sensitivities of the system to climate changes, as shown in fi gure 3.9. Figure 3.9 is taken from an evaluation of a run-of-the-river hydropower facility, which will be referred to throughout this book and explained in greater depth in chapter 4. For now, let it suffi ce to illustrate the main features of a standard climate response map.
A climate response map demonstrates the performance of a system across a wide range of possible climate states. In this example, the performance of a hydropower facility with a specifi c design capacity is being tested across the range of climates shown. Note that the climate space over which the system is tested includes and substantially exceeds both the historical climate  1948-1978 1979-2008 CMIP3_Proj

CMIP5_Proj
(demonstrating the recent trend of increasing temperature and decreasing precipitation) and all available downscaled GCMs (both phases 3 and 5 of the Coupled Model Intercomparison Project [CMIP3 and CMIP5]). A performance threshold of a levelized cost regret 4 of $500/GWh has been set. The climate response map identifi es climate states that result in unacceptable performance relative to the threshold. In the red area (low precipitation and relatively high temperature), the levelized cost regret of hydropower generated by the prefeasibility design is higher than $500/GWh, 5 which would constitute a "failure." The system would perform "acceptably" in the green region. Examples of ex post scenarios that would require further attention, therefore, would be the red region and the part of the green region close to the failure threshold.
The climate stress test is conducted using a model of the system, typically driven by hydrologic time series (although other model drivers are possible). During the climate stress test, the water system model is exposed to a very large number of stochastically generated climate states. The tested climate states cover a range of possible futures much wider than those suggested by historical data and GCMs. It is recommended that the organization establish a range of climate change for each geographical region for consistency in reviews of project assessments. Data can be obtained from the World Bank's Climate Change Knowledge Portal for this purpose. The climate response map (fi gure 3.9) demonstrates the sensitivity of the system to a wide range of climate states.
If no vulnerabilities are revealed on the climate response map, a Climate Risk Report will explain the process used and the fact that the project has been designated robust to climate change (and thus exits the decision tree). If the climate response map does show project vulnerabilities within the tested range, the system is determined to be vulnerable to changes in climate.
If the system is shown to be vulnerable to changes in climate, further analysis is required to determine whether the problematic conditions are likely to arise. This step frames the available climate information with regard to its eff ect on the project. Data can also be obtained from the Bank's Climate Change Knowledge Portal for this purpose. The evaluation of the risk associated with the climate response map can be presented, for example, 6 by a simple count of GCM projections to which the project is robust (see fi gure 3.10) or a risk matrix, as shown in table 3.1. According to fi gure 3.10, only 22 of the 121 GCM runs (about one-sixth of them) project   future climate states in which the levelized cost regret of this project exceeds the threshold. However, fi gure 3.10 does not off er any information about the magnitude of the failure, and though relative magnitudes can be inferred from fi gure 3.9, further analysis is required to quantitatively describe the magnitude of the risk.
Regardless of the magnitude, if the result of the risk matrix is that the impacts are unlikely to occur (for example, not a single projection indicates the conditions are likely and the conditions have not occurred historically or in the paleoclimatology record), the Climate Risk Report is completed (see appendix B). The report should explain the climate stress test to which the system model was subjected and conclude that detrimental impacts are unlikely to occur. Completion of the Climate Risk Report marks completion of the decision tree process-the project exits.
If the risks to project performance are very high (that is, both the historical record and some climate projections indicate that the conditions are possible), methods to improve the robustness of the project to those climate impacts must be evaluated. If, however, the impact to the project is in doubt (for example, the project is vulnerable to a climate state indicated by some GCM projections, but not to others), an assessment of the credibility of each data source should be conducted. For medium or high probability of impact (regardless of the magnitude of impact), the question of whether the robustness of the project can be improved must be asked.
The example presented in this section would have directed the project manager to Phase 4, Management of Climate Risks. 7 Not all projects entering Phase 3 necessarily move to Phase 4. In fact, many projects undergoing a Phase 3 Climate Stress Test will be shown not to have climate vulnerabilities that require adaptation strategies in Phase 4 Climate Risk Management; this is precisely one of the strengths of the Phase 3 Climate Stress Test. It is the responsibility of the project manager, the stakeholders, and the project evaluators to conduct the Phase 3 assessment in such a way that climate-related system vulnerabilities are described in the context of their relative significance, so that relatively small risks do not receive analytical attention out of proportion to their anticipated eff ect on the PIs of concern.
Exit from Decision Tree after Phase 3 Projects exiting Phase 3 (and thus not entering Phase 4) will have been subjected to a climate stress test that showed the particular project design not to have substantive vulnerabilities in the climate range that might reasonably be expected to occur in the lifetime of the project (see fi gure 3.11). Therefore, advanced (and computationally expensive) tools for decision making under uncertainty are not required at this stage. However, it is recommended that any cost-benefi t analyses done for these projects (which have been shown in Phase 2 to have signifi cant climate sensitivities) give careful attention to the inclusion of safety margins and sensitivity analysis, as discussed in "Extensions to Traditional Decision Analysis" in chapter 5. Stochastic optimization procedures, such as benefi t-cost analysis under uncertainty, multiobjective robust optimization, and real options analysis (discussed in "Stochastic Optimization" in chapter 5), would also be appropriate at this point.

Product:
The Climate Risk Report details the climate stress test analysis process and its results. The sensitivity of the project is described and the plausible risks associated with problematic conditions are explained, with the likelihood of each risk described based on the range of available climate information sources. A plan for addressing any climate risks that do emerge should be included. The suggested length of the report is 10-20 pages.

P hase 4: Climate Risk Management
Phase 4 centers on the selection of tools to manage the climate risks identifi ed in Phase 3 under conditions of uncertainty, once it has been established that the robustness of the project to a plausible range of climate change cannot be further improved. A project may also be considered too risky and be abandoned altogether at this stage.

Entry into Phase 4 from Phase 3
Projects entering Phase 4 have substantial vulnerabilities in the range of climate that might reasonably be expected to occur in the lifetime of the project, as demonstrated in the Phase 3 climate stress test.

Description of Phase 4
This step guides the project manager through selection of the tools available for managing the climate risks identified in Phase 3. The input to Phase 4 is a project that has been shown to have worrisome vulnerabilities to climate change that might reasonably be expected to occur within the lifetime of the project. The relevant question in Phase 4 is whether the robustness of the project to the plausible range of climate change can be improved. If the robustness of the project can be improved, the revised project should be subjected to the Phase 3 Climate Stress Test. If doubt remains about the ability to substantially improve the robustness of the project, methodologies for decision making under uncertainty should be used and reported on in the Climate Risk Management Plan. Some considerations regarding these methodologies are described in the "Risk Management Tools" section of chapter 5. In some cases, such as projects affected by uncertainty associated with changes in snowmelt or glacier melt, further geophysical analysis may also be warranted.
Given that the occurrence of a particular climate change scenario cannot be anticipated, preferred adaptation strategies should be robust across as wide a range of potential futures as possible.  provide a summary of four methodologies that have been used to support decisions under uncertainty: cost-benefi t analysis under uncertainty (for example, Arrow et al. 1996), real options analysis (for example, ), RDM (for example, Lempert and Schlesinger 2000;, and decision scaling, as discussed throughout this book. As described by , the latter two methods are both "context-fi rst," robust decision approaches that diff er mostly in the particular analytic tools they use and their relative emphasis on climate versus the combination of climate and socioeconomic uncertainties. The fi rst two methods have routinely been incorporated into such robust analyses as a means of evaluating strategies. The "Risk Management Tools" section of chapter 5 expands on those options, adding information gap decision theory (Ben-Haim 2006), dynamic adaptive policy pathways , stochastic optimization (Loucks, Stedinger, and Haith 1981), and multiobjective robust optimization  to the suggested techniques for decision making under uncertainty. Robust optimization and many-objective optimization (Reed and Minsker 2004) are sister techniques, and ideologically the same. In recent years, multiobjective decision-making approaches have been incorporated into RDM , thereby expanding beyond an optimization framework. Safety margins and sensitivity analysis, as described in "Extensions to Traditional Decision Analysis" in chapter 5, play essential roles in decision making under uncertainty, but are more directly applicable to the levels of climate sensitivity encountered in Phase 2.
Techniques that emphasize optimality are not recommended for decision making under uncertainty. Rather, preferred techniques aim at robustness to a wide range of futures (Brown and Wilby 2012; or adaptive management techniques such as dynamic adaptive policy pathways or real options analysis, which add fl exibility to incrementally adapt to a wide range of futures (Adger, Arnell, and Tomkins 2005;HMT and Defra 2009;. Rosenhead (1989) describes robustness as a particular perspective on fl exibility. As philosophical approaches to model development, robustness and fl exibility and adaptability are founded on slightly diff erent premises. Techniques that aim at robustness skew toward the conservative because they seek solutions that perform satisfactorily even in unknown future conditions signifi cantly worse than the expected. Techniques that emphasize adaptability do not necessarily recommend water system confi gurations that perform satisfactorily in the worst case, but hold open the option to upgrade the system if, over time, it begins to look like the worst case is more likely. However, whereas robustness is a decision criterion that needs to be expressed with respect to some performance metric (as discussed in "The Concept of Robustness" in chapter 5), adaptability is better understood as an attribute of a strategy, and is not expressed with respect to any particular performance metric, but rather is a means to achieve robustness (or optimality). The best water systems plans therefore aim at robustness by way of fl exible and adaptable increments. This approach is demonstrated in the recent trend toward analytic methods that seek robust adaptive strategies (Walker and Marchau 2003;.

Jump Back to Phase 3 from Phase 4
Projects modifi ed in Phase 4 to decrease vulnerabilities to climate change, either by way of simple, direct design modifi cations or analysis using advanced tools for decision making under uncertainty, are resubmitted to the Phase 3 Climate Stress Test.

Exit from Decision Tree after Phase 4
If improvements cannot be made, the project may be deemed too risky (confi dence in the acceptable level of net benefi ts of the project cannot be claimed), and be abandoned in favor of an alternative (the "do-nothing" alternative being a viable option at this point).
Product: Climate Risk Management Plan-a project that reaches Phase 4 will have considerable climate vulnerabilities that must be addressed. Each risk management plan will be unique to the project considered but would ideally incorporate both adaptability/fl exibility and robustness components. The plan should detail the climate risks faced and the means to address those risks. In some cases, the risks may be judged by consulted experts to be acceptable, without taking additional steps. In other cases, modifi cations might be proposed to ensure that the occurrence of certain climate conditions would not cause the project to fail in its objectives. The length of the report may vary, but is likely to be more than 20 pages.

FIGURE 3.12 Phase 4 Entry and Exit Conditions
Note: IGDT = information gap decision theory; RDM = robust decision making.
The Climate Risk Report is completed only in the event of exit from Phase 3. However, the goal of the decision tree is to achieve project designs with low vulnerabilities (high robustness) to climate change. Therefore, projects modifi ed in Phase 4 (unless they are abandoned during Phase 4) are resubjected to the Phase 3 Climate Stress Test (see fi gure 3.12). If the Phase 4 design modifi cations were suffi cient to successfully pass the project out of the decision tree through Phase 3, a Climate Risk Management Plan is included as part of the Climate Risk Report. Notes 1. For example, the World Bank's Climate Policy and Finance team has developed a set of climate risk screening tools. 2. Potentially, such procedures should include nonclimate versions of Phases 2 and 3 (and in some cases Phase 4), through which climate-insensitive projects are subjected to nonclimate stress tests, with risk assessment eff ort expended in proportion to the consequences of project failure. Nonclimate versions of the stress test are sensitivity analyses that involve straightforward modifi cations to the climate stress test described in this book, but in the interest of clarity are not presented here. 3. A PDF of a variable is a function that describes the relative likelihood that this variable will take on a given value. 4. Levelized cost is a form of cost-benefi t analysis in which amortized annual cost is divided by annual average hydropower generated during the economic lifetime of the project. It represents the amount of money that must be charged per unit of energy to break even on the investment of building and operating the power plant. Levelized cost regret is the diff erence between the levelized cost for a particular design and the levelized cost for the lowest-cost design in a particular future. A robust project design is one that has low relative levelized cost regret across the considered range of climate futures.
(See chapter 4 for more detail regarding the calculation and signifi cance of this performance metric, as well as a discussion of the concept of "regret.") 5. The example is presented as an illustration of the decision tree procedure, and is not intended to be instructive with regard to the scale of benefi ts and costs at which real-world decisions are aff ected. For example, the cost regret threshold of $500/GWh ($0.0005/kWh) presented in fi gure 3.9 is unrealistically low; however, it is pedagogically useful. This chapter (and chapter 4) will carry forward the analysis at these scales, understanding that in reality the performance thresholds would be set at much higher values of cost regret. 6. GCM count is only one method for assigning probabilities to future climate states and may not be the best method available. It is shown here for its directness and simplicity, but falsely assumes the independence of each GCM result without acknowledging the common ancestry (clustering) of many GCM assumptions. For this example, only 23 separate GCMs were run, using a number of diff erent CMIP3 (A1B, A2, B1) and CMIP5 (RCP 4.5, RCP 8.5) emission scenarios. As will be described in chapter 4, a more scientifi cally defensible approach would be to assign probabilities to future climate states based on the fraction of the total projected future climate domains encapsulated by the particular climate state under consideration, using "climate change envelopes" . 7. With the reiterated caveat that, in reality, the magnitude of the risk threshold set for the example, $500/GWh, is very low, and would likely not be of concern to the hydropower project manager planning this project.

Introduction
This chapter presents an application of the decision tree to a hypothetical example, a run-of-the-river hydropower project in Sub-Saharan Africa. 1 The example is presented to illustrate the decision tree procedure, and is not intended to be instructive with regard to the scale of benefi ts and costs at which real-world decisions are aff ected. For example, the cost regret threshold of $500/GWh ($0.0005/kWh) presented in fi gure 3.9, introduced in chapter 3, is unrealistically low; however, it is pedagogically useful. This chapter will carry forward the analysis at these scales, understanding that in reality the performance thresholds would be set much higher.
The example run-of-the-river hydropower project involves the diversion of fl ow from two rivers through two small, concrete intake dams, and two identically sized tunnels (North and South) to a single hydropower turbine complex. The capacity of the hydropower plant originally proposed was 90 megawatts (31 cubic meters per second [m 3 /s] turbine capacity). It has since been suggested that the design capacity could be increased to 200 megawatts. As will be demonstrated, the performance of the proposed hydropower facility is sensitive to changes in annual precipitation, but not particularly sensitive to temperature changes. This case study explores the eff ect of changes in future climate, both precipitation and temperature, on the performance of a number of proposed hydropower systems of various capacities.

Phase 1: Project Screening
The proposed project is an infrastructure project with design life longer than 20 years in a part of the world with high seasonal and interannual precipitation variability. Based on that information alone, the Climate Screening Worksheet explains that the project is likely to continue to Phase 2 of the decision tree. However, climate sensitivities should be explored in context using the Four C's for guidance.

The Four C's
Following the Four C's framework, the Choices available are to build a hydropower facility with a capacity of 31 m 3 /s, to build a hydropower facility with an alternate (larger or smaller) capacity, or not to build a hydropower facility. To understand the Consequences, the performance metrics must fi rst be defi ned. As will be explained in greater depth in subsequent sections, the project is evaluated according to its cost effi ciency (specifi cally, the levelized cost) and its total net benefi ts. Undesirable consequences, therefore, are either a poor benefi t-cost ratio or low total net benefi ts. Consequences are understood relative to thresholds. If, for a given future climate state, the levelized cost is projected to be higher than some threshold levelized cost (for example, the levelized cost of the next-best-performing project design), or the total net benefi ts are less than zero, the project design would be considered in a state of failure. The project is being evaluated in isolation, with no Connections to other sectors and no other water projects explicitly modeled. As for unCertainties, demographic and economic factors are expected to aff ect the performance of the proposed system. However, because this is a water infrastructure project with a long design life in a part of the world that has high climate variability, it is reasonable to assume that the primary uncertainty in this evaluation would be the eff ect of a changing climate. The Four C's framework therefore points to the need for a rapid project scoping as part of a Phase 2 Initial Analysis. As part of Phase 2, performance thresholds are established in the next section.
Phase 2: Initial Analysis  In a typical application of the decision tree, Phase 2 Initial Analysis is recommended to determine whether an in-depth Phase 3 Climate Risk Assessment is necessary; however, a Phase 2 project scoping exercise was not performed for this example.

Performance Metrics
Two performance metrics were considered for this example: levelized cost and total net benefi t. Elasticities of these two performance indicators to changes in streamfl ow, ε Q , are used to estimate the risks to system performance of changes in climate. Using these performance metrics, three decision rules were evaluated in this example: minimum levelized cost regret, maximum probability of zero levelized cost, and maximum total net benefi ts.
Levelized cost is a cost-benefi t ratio in which amortized annual cost is divided by annual average hydropower generated during the economic lifetime of the project. In this example, a 5 percent discount rate was used to amortize capital costs. See box 3.1 for a discussion of the eff ect of discount rate uncertainty on water system risk assessment. For each design capacity, annual operation and maintenance costs are a specifi ed fraction of annualized capital costs, regardless of climate future (but specifi c to each design capacity).
Amortized annual cost Annual average hydropower generated kWh $ (4.1) Levelized cost regret is the diff erence between the levelized cost for a particular design and the levelized cost for the lowest-cost design in a particular future. A robust project design is one that has a relatively low levelized cost regret across the considered range of climate futures.

Phase 3: Climate Stress Test
Typically, a quantitative Phase 2 Initial Analysis (for example, a rapid project scoping exercise) would be used to indicate the necessity of a Phase 3 Climate Stress Test. However, given that a quantitative initial analysis was not performed for this example, the more qualitative results of the Four C's assessment are used as guidance on the anticipated value added by a climate stress test. In this case, the Four C's framework suggests that a Phase 3 Climate Stress Test is probably warranted.

Weather-Generator Procedure
The goals of the weather generator are to generate easily reproducible stochastic time series for the local system that preserve spatial and temporal correlations between sites (and also preserve temporal correlations between temperature and precipitation), and preserve low-frequency (interannual) variability. In this case, low-frequency variability was only explicitly preserved for the precipitation signal. The expectation is that the precipitation signal propagates to temperature, though low-frequency variability in temperature is not explicitly modeled. In context, low-frequency variability in temperature is unlikely to signifi cantly aff ect system performance, and the added complexity of multivariate correlation was deemed to not be worth the potential marginal improvement in signal preservation. A wavelet autoregressive model (WARM) was developed to identify lowfrequency, interannual variability in the aggregated (average of two climate stations) annual time series of average monthly precipitation. Then, using the low-frequency statistics, many realizations (approximately 10,000) of 50-year time series of annual precipitation were generated, each of which was faithful to the low-frequency statistics of magnitude (mean, standard deviation, and skew) and frequency patterns (in this case, with a statistically signifi cant signal at six years) in the historical record. Following a heuristic approach suggested by , a k-Nearest Neighbor (k-NN) analysis was used for spatial and monthly disaggregation of stochastically generated annual time series. The k-NN procedure identifi es a subset of years with annual precipitation totals nearest in value to the annual precipitation value of each particular stochastic realization. In this case, the subset consists of k = 7 historical years. One among the subset of seven historical years was next randomly selected, with highest likelihood of sampling the year with the annual precipitation value nearest the annual precipitation value of the stochastically resampled year. For the sampled year of the historical record, spatial disaggregation was achieved simply by substituting back into the new stochastic time series the actual climate values from the two climate stations. The two stations contain monthly values of mean precipitation, and minimum, maximum, and mean temperature, thus preserving spatial and temporal correlations.
This particular strategy for stochastic weather generation is but one among many possible for this application. It was chosen for its simplicity and because it perfectly preserves spatial and intra-annual correlations; however, the resampled precipitation variability cannot be greater than that observed in the historical record. In fact, it is almost guaranteed that the variability of the resampled subset of historical annual precipitation values will be lower than the historical, total variability. An alternative approach would be to approximately preserve spatial and intra-annual correlations by using the k-NN procedure to resample years with total precipitation similar to that of the WARM-generated annual precipitation values, but retaining the WARM-generated totals and disaggregating spatially and subannually by using the disaggregation factors relevant to each resampled year and site. Although this alternative approach would only approximately preserve spatial and intra-annual correlations, it would better represent the potentially increased range of future precipitation variability. The diff erence in methods is not expected to be particularly signifi cant in this case.

Climate Stress Test Procedure
When developing the climate response surface, it is advantageous to run the system model (in this case, the Water Evaluation and Planning [WEAP] model developed by the Stockholm Environment Institute) as many times as possible, both to trace out as wide a space of possible future climate as possible, and to fi ll each potential future climate space with as many stochastic runs as possible. However, trade-off s must be considered between comprehensiveness and computational burden. For this analysis, the WEAP model was run 10 times for each precipitation perturbation and each temperature perturbation.
Using only 10 of the 50-year stochastic time series of resampled historical values, and choosing only four-fi fths of the length of each time series (because the WEAP model is set up to receive only 40-year time series), time series of future climates were generated. First, a decision was made about the ranges over which to perform the stress test. The relevant downscaled GCMs range from approximately 650 to 1,325 mm of precipitation in 2040, and average annual temperatures of approximately 20.3 to 22.7 o C (see fi gure 4.1). Understanding that the range of future climates indicated by the GCMs should serve as a minimum indicator of uncertainty, this analysis extended somewhat beyond the range of the GCMs. Thirteen "bins" of precipitation and six "bins" of temperature were used. The binning of precipitation and temperature conditions into categories of plausible climate trends was done by placing a multiplicative trend on precipitation, to produce annual average precipitation totals for 2040 in the range of 400 to 1,600 mm, or −61 to +57 percent of historical, in increments of 100 mm; an additive trend was placed on temperature, to produce changes in annual average temperature for 2040 of −1, 0, +1, +2, +3, and +4. The ultimate temperature and precipitation increases were achieved in 2050, having begun in 2010, meaning that each year in between received one-fortieth of the total change. In total, for the prefeasibility design, the WEAP model was run for 780 climate states (13 precipitation, 6 temperature, 10 runs of each). The WEAP model calculated a time series of monthly streamfl ow at each site, as well as the monthly hydropower generated. The levelized cost and levelized cost regret were calculated during post-processing of the WEAP results.

Mean Climate Response Surface
Refer again to the climate response surface, introduced in chapter 3 and fi gure 3.9. If, for example, attention were given to the prefeasibility design (31 m 3 /s), and a performance threshold (in this case, levelized cost regret) of $500/GWh had been set, the climate response map would identify climate states of unacceptable performance relative to the threshold. In the red area (low precipitation and relatively high temperature), the levelized cost regret of hydropower generated by the prefeasibility design is higher than $500/GWh, which means the design is a failure. However, the system performs acceptably in the green region. Scenarios to be further evaluated, therefore, lie in the red region and in the part of the green region near the failure threshold.
The system is obviously sensitive to climate (Phases 1 and 2). The summary fi ndings of Phase 3, illustrated in fi gure 3.9, demonstrate that, more than being generally sensitive, this particular design has substantial climate vulnerabilities within the range of climate change explored, and, significantly, within the range suggested by both downscaled GCMs and by extrapolation of the historical trend. This is the fi rst input received from the GCMs in this analysis. The role of the GCMs has not been to project system performance, but to provide information about the relative likelihood of poor system performance, after the vulnerability assessment was completed using projected climate time series generated by the stochastic weather generator. Assuming that the decision maker, taking stock of the GCM projections and historical trend (and any available paleoclimatology data not shown here) is dissatisfi ed with the magnitude of the climate-related vulnerability of the proposed system, design modifi cations (or alternative projects altogether) would be explored in Phase 4.

Phase 4: Climate Risk Management
At this point, signifi cant vulnerabilities in the hypothetical prefeasibility design have been identifi ed. An initial step toward the management and reduction of those climate-related risks might be to test other turbine capacities. This section describes a simplifi ed version of the process used by decision scaling to manage climate risks. Because this example is simple, most of the advanced search algorithms (for example, real options analysis, robust optimization) and scenario-clustering procedures (for example, robust decision making) for decision making under uncertainty are unnecessary. Were the problem more complex, the more advanced (and more time-and computationally intensive) techniques might be warranted. These methods are briefl y discussed in chapter 5.

Return to Phase 3: Mean Climate Response Surface
There is no reason to believe that robustness in this case is not achievable. Phase 4 therefore involves strategic design modifi cations, with each revised project design confi guration being resubjected to the Phase 3 Climate Stress Test. Figure 3.9 demonstrated the climate response of the prefeasibility design, thereby identifying its mean climate-related vulnerabilities. Figure 4.2 presents the results of a climate stress test on a range of 12 design capacities (of 29, 31, 33, 35, 37, 39, 41, 43, 45, 47, 49, and 51 m 3 /s), and identifi es the band of climate space over which each design operates with the lowest cost regret. Costs are calculated using a pricing spreadsheet in which the cost of each design is discreet and nonlinear relative to the cost of each other design.
To perform this analysis, a Visual Basic for Applications (VBA) script was used to call and run WEAP (see brief description of WEAP in appendix A) for every climate time series for every design. The WEAP model was run for the 12 designs and 780 climate folders, resulting in a total of 9,360 WEAP runs. Each time WEAP was run, the VBA script provided it with a design capacity (North tunnel size, South tunnel size, and turbine size) and the appropriate number of the climate folder containing the climate time series. The WEAP model then calculated a 40-year time series of monthly streamfl ow at each site, as well as the monthly hydropower generated. Results were output to folders to be analyzed in R (an open-source software environment for statistical computing and graphics). Figure 4.2 is a combined climate response map that enables the performance of the 12 alternative designs to be compared (design 1 having the smallest capacity and design 12 the largest). The dashed domain is the climate envelope for the CMIP3 ensemble, the dotted domain is the climate envelope for the CMIP5 ensemble, and the solid box encapsulates the climate envelope for all projections plus the historical observations (see "Risk Analysis Concepts" in this chapter for a discussion of the rationale behind the development of the three climate envelopes). In brief, polygons can be drawn to  encapsulate the area on the climate response surface in which GCMs project future temperature and precipitation, and the area can be assigned a uniform distribution (giving no preference to GCM "clusters"). If the resulting encapsulated space in the climate response surface is all red (regardless of shade of red), the chance of unacceptable system performance is estimated to be 100 percent; adaptation is thus necessary. 2 If the space is only one-quarter red, the likelihood of unacceptable system performance is 25 percent, and risks (for example, net present value, discounted costs) should be calculated to determine whether adaptation measures are worthwhile. In the current example, if average annual precipitation exceeds about 1,300 mm, the preferred design would be the largest design (design 12, at 51 m 3 /s), to be able to make productive use of the available streamfl ow. However, if average annual precipitation is less than about 850 mm, the smallest hydropower design capacity (design 1, at 29 m 3 /s) would be the most effi cient. "Most effi cient," in this case, means that the design under consideration would operate at the lowest levelized cost. Temperature also aff ects the choice, but it does so to a lesser extent than precipitation; at higher temperatures, more precipitation is required to warrant the choice of a larger design. s) is the preferred design when future mean precipitation is less than about 1,000 mm. In the white area, the levelized cost regret is zero, which means that of all the considered alternatives, design 1 has the lowest levelized cost regret within that particular climate space. Design 12 (51 m 3 /s capacity, panel 4 of fi gure 4.3) produces hydropower with the lowest levelized cost for future precipitation values higher than about 1,100 mm in 2040 (with linearly increasing precipitation trends between now and then). For the entire modeled climate domain, larger designs have higher potential for higher levelized cost regrets than do smaller designs, keeping in mind that no GCMs project conditions in which the larger designs have their highest levelized cost regret (these lie outside the solid box), and only two extreme GCM projections indicate the likelihood of even "high" levelized cost regrets for large design capacities (at the lower right corner of the solid box). Looking specifi cally at the CMIP3 envelope, the CMIP5 envelope, or the region of extrapolation of the historical trends, however, may result in diff erent conclusions. Further exploration is needed, and is provided in the "Risk Analysis Concepts" section.  Turning briefl y to an evaluation of total net benefi ts, fi gure 4.4 shows that, in general, larger designs potentially yield higher total net benefi ts. This is an important fi nding in favor of larger designs, given that fi gure 4.3 indicates that the levelized cost regret metric favors smaller designs. Within the envelope of projected and historical climate (solid box), at a price of $0.05/kWh (best current estimate of the future local price of electricity), design 12 produces the largest total net benefi t (zero net benefi t regret, that is, forgone profi t, throughout). Design 1, however, could potentially result in more than $200 million of forgone profi t relative to the highest-profi t-generating design within the domain of projected and historical climate. Figure 4.4 is, of course, highly sensitive to the forecasted future price of electricity, and a sensitivity analysis should therefore be performed on this factor.

Risk Analysis Concepts
Figures 4.5 and 4.6 summarize the fi ndings of a straightforward analysis of the risks faced by various project designs. The performance of each design is presented in every panel as measured by (1) the considered futures over which it has a levelized cost regret of zero (the lowest levelized cost among all alternatives)-a measure of robustness, (2) its expected levelized cost regret ($/GWh)-a measure of robust optimality, and (3) its total net benefi ts ($/year)-an alternative measure of robust optimality. Each fi gure further subdivides the results specifi c to the performance of each design in the domain of CMIP3-projected space, the domain of CMIP5-projected space, and the total climate envelope (designated in fi gures 4.5 and 4.6 as "Full range"). The ranges of performance of each design across the 10 stochastically generated future climate realizations are presented as colored lines above and below mean performance (black dots).  As shown in equation (4.2), the basic concept of risk, applicable in this and most other instances of water resources planning and management, is a combination of impact (hazard) and the probability of that impact.
The probabilities in fi gure 4.5 are derived from GCM counts, as was demonstrated for the prefeasibility design in the "Phase 3: Climate Stress Test" section of chapter 3, and a uniform distribution is used to characterize the likelihood of each GCM result. A count of the downscaled GCMs falling in the region of preference for each design is not the only (or even the preferred) method for evaluating the relative likelihood of each climate state (and thereby the relative likelihood of the preference for each design). Were the GCMs independent, this might be a reasonable approach. However, "clustering" of GCMs in particular regions of the climate response map may  be attributable to the cross-model similarities in GCM model design and the propagation of potentially erroneous common assumptions (or data sources) throughout numerous GCMs. In this example, only 23 separate GCMs were run, using a number of diff erent CMIP3 (A1B, A2, B1) and CMIP5 (RCP 4.5,8.5) emission scenarios. Figure 4.6 therefore presents the results of an alternative approach that assigns a uniform 3 distribution to the region of the total climate envelope and weights each alternative design by the fraction of the total climate envelope in which it is preferred.
According to fi gure 4.6, larger design capacities are likely to have the highest levelized cost regret and the lowest net benefi t regret. Designs in the low-middle range have zero cost regret for the largest number of GCM projections, and also the lowest mean cost regret. Less uncertainty (shorter uncertainty bars) surrounds net benefi t regret projections than levelized cost projections (assuming the forecasted electricity price is reliable).
The key fi nding of fi gure 4.6 is that design 1 is highly cost-effi cient over a large portion of the climate envelope. However, because the region of the climate envelope in which design 1 is highly cost-effi cient contains few GCM projections, it does not show up so prominently in fi gure 4.5. Interestingly, design 1, while maximizing the probability of zero cost regret in fi gure 4.6, does not produce hydropower with the lowest expected levelized cost regret because it has very high cost regret in the small regions of the climate envelope in which it has any cost regret. In the risk matrix (table 3.1), high regret in design 1 is a high impact-low probability event, and equation (4.2) (the mathematical expression of risk) shows that the risk of that event actually occurring (the product of high impact and low probability) is small. This close examination of design 1 in fi gure 4.6 is therefore a useful exercise in understanding how the source of climate information could aff ect the design. A few anomalous climate projections in the CMIP5 ensemble point to the preferability of design 1. However, on the whole, design 1 is not very robust. The design does command attention in the CMIP3 group.

Discussion and Recommendations
If minimization of maximum net regret is the driving motivation (conservative decision making), smaller designs would be preferred. If maximization of net benefi ts is the goal (opportunistic decision making), larger designs would be preferred. Design 12 looks especially robust for decision makers driven by profi t maximization. Of course, to achieve greater future returns, a bigger investment must be made in the present to build larger hydropower facilities. Budget constraints should therefore be considered.
The results make a strong case for designs in the middle range, which have relatively low levelized cost regret and relatively low net benefi t regret. Such middle-range designs are robust to most projected future climate states. Design 1 produces hydropower with the maximum probability of zero levelized cost over a large portion of the climate envelope; however, it is not a portion of the space in which many GCM projections fall. Given the competitiveness of several of the evaluated designs, other metrics, such as capital investment required, investment resources available, and impact on existing ecological and human systems, should be used to distinguish between the choices. Along these lines, in an adaptive framework, it would be ideal if the hydropower system could be designed to be modular and expandable with marginal additional investment in the future, if higher precipitation occurs.
Furthermore, this simple case study presents design sensitivities to changes in mean annual precipitation and mean annual temperature. The hypothetical hydropower system has been shown to have sensitivities to changes in precipitation variability. A second climate stress test could therefore be developed to evaluate the relative performance of each alternative design across a range of streamfl ow means and variabilities (for example, increases in the intensity of precipitation resulting from a narrowing of the monsoon season). The ability of weather generators such as the one developed for this study to perturb many aspects of the climate signal-mean values (annual, monthly, or daily precipitation or temperature), extreme values, variability, and spatial and temporal patterns, and the like-is a particular strength of this approach. The insights gained into the broader system sensitivities through a more comprehensive analysis of climate parameters, in addition to annual average temperature and precipitation, could be instrumental in the identifi cation of preferred system designs, especially when placed in the context of best scientifi c understanding of likely changes in seasonality, variability, and other spatial and temporal patterns. Once a weather generator has been developed, the additional computational eff ort involved in such explorations is not prohibitive.
Finally, returning to the statement in the introduction to this chapter on the use in this example of unrealistically small thresholds for cost regret, the implications for the application of the decision tree to this example are clear: this example deserves a more thorough exploration in Phase 2. Were other uncertainties considered (for example, demographic factors such as population growth and per capita use of water and electricity, agricultural water use, electricity price, and discount rate), they would likely be found to be relatively more signifi cant to project performance than climate uncertainties. For the sake of illustration, this analysis jumped straight to an examination of the eff ects of climate-related uncertainties. A more thorough Phase 2 evaluation would begin with a quantifi cation of the elasticity of the prefeasibility design's levelized cost (or net benefi ts) to changes in nonclimate uncertainties. If the elasticity is signifi cantly higher for nonclimate factors, a Phase 3 Climate Stress Test may not be necessary. Instead, attempts to reduce the uncertainty in nonclimate factors would be made in order of greatest to least levelized-cost or -benefi t elasticity. Adaptations may be made to the prefeasibility design to address potential nonclimate vulnerabilities, with climate-related vulnerabilities addressed only if signifi cant potential vulnerabilities remain.

Special thanks to Mehmet Umit Taner for his contribution in the analysis and
writing of this chapter. His manuscript detailing this case study is forthcoming. The reader may recognize in this example the confi guration of the Lower Fufu Hydropower Project in Malawi (Cervigni et al. 2015). The example has been adapted and presented here only to illustrate the application of the decision tree procedure. Conclusions refer to the hypothetical example only and cannot be used to indicate the relative merits of the actual proposed project. 2. The shade of red indicates the severity of the failure, and is useful as an indication of the magnitude of vulnerability, but is not immediately relevant to the binary question of failure or not. 3. It is not necessary that the distribution be uniform. Other distributions, such as bivariate normal, are equally possible. However, the use of the uniform distribution is simplest and quickest, and requires the least evidence of applicability. If a bivariate normal distribution were assigned to the region of the total climate envelope, each alternative design would be weighted by the fraction of the total envelope in which it is preferred, and the quantile of the bivariate normal distribution in the region of preference.

Introduction
The methodology presented in chapter 3 describes a decision tree framework for bottom-up, climate-informed decision making in water resources planning and management. It guides project planners through a risk assessment (tailored to climate risk, but generalizable to most other risks a water system might face) in which analytical eff ort accounting for the uncertain eff ects of change in some conditions is expended in proportion to the project's sensitivity to those particular conditions. If the assessed risks to the project are signifi cant, tools for decision making under uncertainty may be needed to systematically modify the proposed project design to reduce its vulnerabilities. This chapter provides background on prominent examples of such tools.

Background
Reports on decision making under risk and uncertainty in water resources planning and management have proliferated recently. Most of these reports have been generated at the request of governments, international agencies, or donors (Coates et al. 2012;National Research Council 2009;Schultz et al. 2010;Vucetic and Simonovic 2011).
The reports together represent a compendium of best practices for planning water resources projects in the absence of a clear picture of future conditions. All of the reports emphasize the fundamental role of uncertainty in water resources systems decision making in light of the increasingly significant nonstationarity of the climate signal. Most of the reports make clear that water systems have always been planned under uncertainty, and that the nonclimate uncertainty facing water systems planners has often outweighed the uncertainty related to future climate. Therefore, the methods pointed to for decision making under this new, "deeper" or "more severe" climate uncertainty (as discussed in the following section) are mostly drawn from methods previously developed for incorporation of more conventional uncertainty in water systems planning. This chapter summarizes the fi ndings of those reports and organizes the information to correspond with the phases of the decision tree (see fi gure 3.1). A number of concepts and techniques are introduced that are explained in greater detail in later sections. Note that not all of the methods described herein are exclusively decision-making approaches; some include tools for risk assessment that can be incorporated into Phases 1-3.

Extensions to Traditional Decision Analysis
Operations research, developed during World War II, has provided tools for modern decision analysis. The foundation of operations research is the formulation of a mathematical model to represent a problem, and the use of a computer-based procedure for deriving solutions to the problem (Hillier and Lieberman 2005). Traditional decision analysis aims for an optimum system confi guration (for example, infrastructure development or operational policy), typically by minimizing expected cost or maximizing expected benefi ts.
Some tools for traditional decision analysis are built on search (optimization) algorithms Loucks 1970;Steinschneider and Brown 2012). Others run simulations of the system repeatedly (a so-called Monte Carlo experiment), systematically varying model parameters to identify preferred system confi gurations . If a tool based on a search algorithm is used, emphasis should be placed on the generation of trade-off s between competing objectives . Robustness with respect to one objective (for example, maintenance of low fl ows for ecological well-being) will likely be won at the expense of some other objective (reservoir storage for water supply during drought, for instance). Other concepts to be emphasized in optimization objective functions are adaptability (real options analysis ) and diversifi cation (redundancy and diff usion of risks [Brown and Carriquiry 2007]). A well-crafted optimization model for water systems planning and management will likely incorporate elements of all three objectives-adaptability, diversifi cation, and robustness.
Traditional decision analysis tools and their extensions are suitable for the types of projects that exit from Phase 2 or Phase 3 of the decision tree, but are of limited utility to problems involving "deep" or "severe" uncertainty, as would be analyzed in Phase 4. For projects exiting at Phase 3, in particular, the cost-benefi t analyses performed must pay careful attention to the inclusion of safety margins and sensitivity analysis, given that those projects were shown in Phase 2 to have signifi cant potential sensitivities to climate change, though current climate change projections do not indicate a high likelihood of resulting system failure (relative to performance threshold). Phase 3 problems might also benefi t from treatment using models based on stochastic optimization procedures, as discussed later in this chapter. The tools needed for Phase 4 analysis are summarized in the "Risk-Management Tools" section of this chapter, following an introduction to more traditional techniques and a short discussion of key concepts.

Trade-Off s between Benefi ts and Costs
In water resources applications, a system model tends to take water-related parameters as input and outputs economic metrics. A water supply system model might allocate the water according to minimum cost or maximum utility. A water distribution network (or wastewater collection network) system model would provide guidance on the relative costs and benefi ts of reconstruction or reoperation 1 of the storage, treatment, and channel system. A fl ood control system model would likely seek to minimize the cost of fl ood damage. In each case, in addition to parameters related to the current system confi guration, historical performance, and comprehensive cost and benefi t data, water system models used as planning tools need information that anticipates the behavior of water within the system during the lifetime of the project. Hydrologic models translate forecasts of temperature and precipitation into water runoff or water storage. A description of a number of the hydrologic models most commonly used in the United States is included in appendix A.
A decision-support model of the system can be developed to assist in the characterization and quantifi cation of the economic (and other) trade-off s inherent in the project installation. Use of a hydrologic model to project water (runoff , soil moisture, storage, seepage, and so on) in the system is not entirely necessary, but many such tools are available, such as lumped parameter tools (for example, the abcd model) and distributed models (for example, variable infi ltration capacity-the VIC model), summarized in appendix A. However, a model of the benefi ts and costs of each project is absolutely necessary to compare the relative merits of each project to its alternatives. Output of such a model could be monetary benefi ts, metrics of system performance (for example, reliability or invulnerability), or ecosystem costs. Because every water-related project, program, or policy entails costs, a framework is needed to permit comparison of the net benefi ts of the proposed project to its alternatives. The framework need not monetize all costs and benefi ts, and need not be overly complex or even computer based. However, without the establishment of a framework for trading off benefi ts and costs, decisions for investment in water projects lack basis. 2 Conventionally, water resources system planners, faced with uncertain future benefi ts and costs (and their more fundamental elements like demand and supply), would use deterministic tools to optimize over expected values. Being aware that the use of expected values was subject to the potential realization of futures less fortuitous than the expected, planners attach safety margins to key system elements or operating policies. Safety margins, sensitivity analysis (as described below), and adaptive management (described later in this chapter both as a concept and in practice) are tools for adding robustness to traditional decision analysis. However, traditional expectedvalue decision making, even in combination with tools for adding robustness, has signifi cant drawbacks when the sensitivities of the system are substantial (Loucks, Stedinger, and Haith 1981).

Safety Margins
A simple and eff ective strategy for decision making under uncertainty is to be conservative, traditionally accomplished by including a safety margin (also called a safety factor). For example, if it is believed that sea level will rise approximately 1 foot in the next century, it would make sense to build a 2-foot sea wall to hedge against a worst-case scenario. The more expensive the potential failure, the more reasonable it is to include a very large safety factor. If the proposed sea wall is intended to protect New York City from a 1-foot sea level rise, it might be reasonable to include a factor of safety of 1,000 percent and build a 10-foot sea wall. Of course, the magnitude of the safety margin is aff ected by many factors, including the cost of additional capacity, the consequences of system failure, the economic lifetime of the project, the fl exibility of the design, and the likelihood that better forecasts of future conditions will become available in time to add additional capacity at a later stage. The discussion of real options analysis (ROA) later in this chapter presents, as an example, a method for inclusion of extra strength in the base of a sea wall that would enable extension of the sea wall height as needed in the future.

Sensitivity Analysis
For situations in which uncertainty exists about the system model or the distribution of its inputs, practitioners of traditional decision analysis turn to sensitivity analysis. Sensitivity analysis is a simple method for assessing the eff ect of uncertainty on system performance by considering the possible costs of making other than the optimal decision. According to Loucks and van Beek (2005, 261), "A sensitivity analysis attempts to determine the change in model output values that results from modest changes in model input values. A sensitivity analysis thus measures the change in the model output in a localized region of the space of inputs." The shadow price in linear programming is a particularly useful tool in sensitivity analysis because it presents the marginal utility of relaxing a system constraint by one unit. In the run-of-the-river hydropower case study presented in chapter 4, for example, it would be possible to show the marginal value (as measured by a decrease in levelized cost) of 1 additional millimeter of rainfall per year. A sensitivity analysis is not the same as a thorough analysis of the uncertainties potentially aff ecting system performance (together with their probability of occurrence), nor does it address the question of what decision should be made when the future is unknown or unknowable (Loucks, Stedinger, and Haith 1981). Saltelli, Tarantola, and Campolongo (2000) describe four primary contributions of sensitivity analysis: • It addresses structural uncertainty and provides guidance for the identification of the weak links in a scientifi c assessment chain. • It can be used to determine which subset of input factors (if any) accounts for most of the output variance (and in what percentage). • It can be useful as a quality assurance tool, to make sure that the assumed dependence of the output on the input factors in the model makes physical sense and can be reconciled with the analyst's understanding of the system. • It can be used before and during model identifi cation and parameter estimation to calibrate the degree of detail of the model to the task at hand, saving the analyst time.
However, as argued by , the attachment of sensitivity analysis to traditional decision-analysis techniques is an adequate measure for risk exploration only if the optimum strategy is relatively insensitive to key assumptions. If it is not, sensitivity analysis techniques "can encourage analysts and decision makers to downplay uncertainty to make predictions more tractable" (Lempert et al. 2006, 515). Furthermore, they can lead to strategies vulnerable to surprises that might have been countered had available information been used diff erently (Lempert, Popper, and Bankes 2002). Thus, conclusions drawn from sensitivity analysis should be handled judiciously.

Key Concepts in Decision Making under Uncertainty
Before proceeding to a discussion of the types of advanced tools for decision making under uncertainty that are useful in Phase 4, beyond the simple extensions to traditional decision analysis just presented, two key concepts that surface repeatedly in the literature on decision making under uncertainty should be defi ned: the concepts of "deep" or "severe" uncertainty, and of "robustness." These terms are ambiguous, so various perspectives on them are off ered here. The community of academics and practitioners developing these ideas continues to assess the degree to which a formal defi nition of these concepts is necessary, and under what conditions.

Deep and Severe Uncertainty
According to robust decision making (RDM), "deep uncertainty is the condition in which analysts do not know or the parties to a decision cannot agree upon (1) the appropriate models to describe interactions among a system's variables, (2) the probability distributions to represent uncertainty about key parameters in the models, or (3) how to value the desirability of alternative outcomes" (Lempert et al. 2006, 514). Information gap decision theory (IGDT) defi nes severe uncertainty as "conditions where the evidence upon which to base a decision is scarce and only of limited relevance to predicting what may happen in the future" (Hall et al. 2012, 4-5). Such uncertainty leads to an information gap-a disparity between what is known and what needs to be known to make a dependable decision.
For the purpose of decision making under uncertainty, the common thread between the two concepts is that they cannot be characterized by a single probability distribution. (See box 5.1.) In this way, both are examples of Knightian (1921) uncertainty. A common modeling response to deep or severe uncertainty is to characterize all events in the possibility space as equally likely, and to assign them a uniform distribution. This assignment does not solve the problem of needing to select bounds for the uniform distribution, and the net eff ect is that all future scenarios considered "possible" are assigned equal probability of 1/N (N = number of scenarios), whereas all future scenarios outside of the set explicitly considered are assigned a probability of zero (thus labeled impossible). Better solutions have been adopted by both the IGDT and RDM schools. The former does not use probabilities but rather considers nested sets of future states of the world representing the range of uncertainty around multiple parameters of interest. In this way, bounds on uncertainty can be usefully quantifi ed. RDM can consider imprecise probabilities, represented as sets of alternative joint probability distributions over multiple parameters of interest, and then seeks strategies that are robust over a wide range of such distributions (Lempert and Collins 2007).

The Concept of Robustness
Rosenhead describes robustness as a particular perspective on fl exibility that is concerned with "situations where an individual, group or organization needs to make commitments now under conditions of uncertainty, and where these decisions will be followed at intervals by other commitments" (Rosenhead 1989, 188). A robustness perspective focuses alternately between the present, in which decisions must be made with the best available information about future conditions, and a regular updating of best estimates of Deep uncertainty refers to the condition in which probability distributions cannot be assigned to the key uncertainties, the appropriate models to describe interactions among a system's variables are lacking, the relative desirability of various alternative outcomes cannot be quantifi ed, or any combination of the three.
Severe uncertainty refers to conditions in which an unbreachable disparity exists between what is known and what needs to be known to make a dependable decision.
For the purpose of decision making under uncertainty, the common thread between the two concepts is that they elude characterization by a single probability distribution.

BOX 5.1
Deep and Severe Uncertainty those future conditions. The robustness of any initial decision is defi ned as "the number of acceptable options at the planning horizon with which it is compatible, expressed as the ratio of the total number of acceptable options at the planning horizon" (Rosenhead 1989, 190).
RDM has typically defi ned robustness as performing reasonably well compared with the alternatives over a wide range of plausible futures . A system fi tting this defi nition might be described as being "reliable" over a wide range of plausible futures, or possibly, depending on context, as having relatively low vulnerability. Applications of RDM have also defi ned robustness as "trading some optimal performance for less sensitivity to broken assumptions." Collins 2007, 1017) Variants of both these robustness criteria have been used in a recent World Bank report on water supply infrastructure (World Bank 2014). In particular, this report examines and compares climate adaptation strategies using three robustness criteria within an RDM framework: minimize maximum regret, satisfi ce over a wide range of future conditions, and satisfi ce over a wide range of likelihoods for future conditions. In most situations these criteria yield similar adaptation strategies, but in some cases the strategies diverge. Depending on the application, RDM applies robustness criteria either directly to measures of system performance (such as reliability, cost, and environmental impact) or to "regret" calculated from one or more of these measures. Regret is the diff erence between the performance of some strategy in a particular future and the performance of the best strategy in that future.
IGDT defi nes robustness as "the maximum uncertainty, measured by the parameter α: α ≥ 0, over which a strategy achieves a certain level of performance… Robust-satisfi cing seeks to identify acts that perform acceptably well under a wide range of conditions… Robustness decreases as the requirement for reward becomes increasingly demanding" (Hall et al. 2012, 6-7).  explain that robustness, as with optimality or any other decision criterion, must be expressed with respect to some performance metric. A system cannot be said to be robust, generally, because a system cannot be said to be optimal generally. A water supply system that reduces water shortages by increasing spot market water transfers should not be described as "robust"; instead, it should be described as robust to water shortages (but certainly not robust to water transfers, which might have numerous undesirable qualities of their own). Specifi c to water shortages, for example, a robustness of 0.25 would mean that, based on the chosen water system capacity and the previous climate-and demand-related probabilities, there is a 25 percent chance that there will be a shortage in the modeled year. The system is thus robust to 75 percent of futures. However, this does not mean that it is robust to 75 percent of future scenarios, because the future scenarios are not necessarily all equally likely. But, based on the probability space, one would expect a 75 percent chance that the system will not fail (that is, a shortage will not occur).
When robustness concepts are applied to water systems decision models, precise language should be used. In the context of climate change risk screening, for example, water systems can be categorized according to (among others) each of four descriptors: (1) climate sensitive, that is, whether its performance is aff ected by climate at all; (2) reliable over a wide range of climate risks, that is, though it might be sensitive to climate change, its performance thresholds might not be violated; (3) vulnerable to very costly failures, that is, though it might resist failure, when it does fail, it might fail catastrophically; and (4) resilient, that is, able to recover quickly from failure to previous levels of performance. (See box 5.2.) Robustness can be presented with respect not only to system reliability in the face of fl ood or drought (as is typically done), but also with respect to a variety of occurrences of concern to stakeholders (for example, water transfers in response to drought or controlled fl ooding of farmland in response to a fl ood).
Robust project designs perform reasonably well compared with alternative designs over a wide range of plausible futures. Robustness is a criterion that can be used to compare alternative decisions. One way to express the robustness of any decision is as a function of the number of possible futures (or size of the projected future domain) with which it is compatible, divided by the total number of projected futures (or total size of the projected future domain). A system fi tting this defi nition might be described as being "reliable" over a wide range of plausible futures, or possibly, depending on context, as having relatively low vulnerability.
Adaptability or fl exibility is an attribute of a decision that can be used to make a strategy more robust (or optimal). Adaptability or fl exibility generally has costs, and so is preferable if (1) uncertainty is dynamic-knowledge is expected to improve over time; and (2) the project involves irreversible creation or destruction of capabilities. Certain adaptation strategies are more fl exible than others to the possibility of upgrade in the future in the event that the impacts of climate change are high. Real options analysis is an established process by which adaptability can be explicitly incorporated into project designs through the use of Bayesian probability trees.

Robustness and Adaptability or Flexibility Alternative Decision Rules
In traditional decision analysis, costs are minimized or benefi ts maximized. However, there are simple ways to represent a decision maker's degree of aversion to risk within the objective function. More-risk-averse decision makers might maximize the minimum profi t or minimize the maximum regret. An objective function formulated to minimize the maximum regret would select the system confi guration that results in the least bad worst-case performance (relative to other system confi gurations). The minimax regret formulation is the one used most often by those seeking a project robust to climate change, although methods like RDM and IGDT often use other robustness criteria, as described above.
Less-risk-averse decision makers, more interested in maximizing potential gains (for example, windfalls), might structure the objective function as a maximization of maximum profi t. The optimization function in this case would seek the system confi guration that would result in the occurrence of the maximum benefi t achieved in any single scenario, regardless of the adequacy of that system confi guration in other less fortuitous scenarios.
The Hurwicz criterion is a compromise between the maximax and maximin criteria. It uses a weighting method for balancing pessimism with optimism (Revelle, Whitlatch, and Wright 2004).

Risk Assessment Tools
The following subsections summarize procedures for climate risk assessment-the scenario-neutral approach and IGDT-that are similar in concept to the stress-test elements of decision scaling, as presented in Phase 3 of the decision tree.  perform a procedure very similar to the climate stress test described in relation to decision scaling. The authors describe the procedure as the "scenario-neutral" approach to climate change impact studies, and present it in contrast to "conventional, 'top-down' (scenarioled) approaches to climate change adaptation" (Prudhomme et al. 2010, 199). Both precipitation and temperature sensitivity tests sample from a range of scenarios signifi cantly larger than those indicated by the projections. The study uses a "change factor" to apply an absolute percentage change to temperature and precipitation according to that suggested by the general circulation models (GCMs), and then uses a harmonic function to model the seasonal pattern of precipitation and temperature. The authors explain that "this is to allow for any signifi cant diff erence in future projections from the next generation of climate models or new emission pathways to be part of the sensitivity domain" (Prudhomme et al. 2010, 206). By performing repeated simulations using a hydrologic model to observe fl ood peaks across scenarios, the authors gain valuable information (risk analysis) on the critical climate conditions at which the fl ood control system fails. The scenario-neutral approach does not present a method by which the decision maker can judge the relative merits of fl ood prevention strategies (as measured, for example, by cost). Robustness is characterized strictly as reliability. Hipel and Ben Haim (1999) argue that three fundamentally distinct approaches are available for formally describing uncertainty: probability, fuzzy set theory, and information gap modeling. Though there are ample examples of probability (the accepted standard and basis for most texts, see Revelle, Whitlatch, and Wright 2004;Loucks, Stedinger, and Haith 1981;Loucks and van Beek 2005;and Sen and Higle 1999) and fuzzy sets (Karmakar and Mujumdar 2006;Li, Huang, and Nie 2007;Li et al. 2009;Maqsood, Huang, and Yeomans 2005;Qin et al. 2007;Wang et al. 2009) used to describe uncertainty in water resources problems, IGDT modeling had not until recently been used in water resources. Since 2009, however, a number of studies have applied IGDT to a variety of water resources planning problems under great uncertainty, especially in fl ood mitigation (Hine and Hall 2010;Korteling, Dessai, and Kapelan 2013;Manning et al. 2009;Matrosov, Woods, and Harou 2013;Woods, Matrosov, and Harou 2011).

Information Gap Decision Theory
IGDT  consists of three elements: (1) a nonprobabilistic, quantifi ed model of uncertainty; (2) a system model that projects the outcome of decisions contingent on the model of uncertainty; and (3) a set of performance requirements that specify the value of the outcomes the decision makers require or aspire to achieve. The technique asks decision makers to set minimum and aspirational performance levels, and to favor the strategies that meet these levels over, respectively, the widest and narrowest range of uncertainty.
IGDT characterizes the uncertainty of system performance as a group of nested sets. The method requires the user to identify a best estimate of each unknown parameter from which to start the uncertainty analysis. Next, each of the input parameters is bounded in an interval, the range of which is meant to encompass "most" of the uncertainty particular to that parameter.
Whereas in the stress tests developed within decision scaling and scenarioneutral modeling a single increment of uncertainty is explored (the total range of average annual temperature and precipitation over which the performance of a water project is evaluated, for example), IGDT explores the range of performance within subsets of the total uncertainty space, which it refers to as "horizons." IGDT scales the diff erence between the user-defi ned best estimate of a future parameter, u, and some particular realization, ũ. The deviation between u and ũ is scaled by h, which becomes the increment or uncertainty horizon. The IGDT uncertainty model is constructed as a nested set based on ũ and h such that U(h, ũ). A water system plan will have a range of performance for each uncertainty horizon specifi ed by U(h, ũ). The minimum performance level is referred to as the robustness, and the maximum performance level is termed opportuneness. By quantifying opportuneness as well as robustness, the approach aims to be value-neutral-neither optimistic nor pessimistic, acknowledging that uncertainty means that outcomes could be both worse and better than expected.
IGDT requires the development of a system model and a nonprobabilistic quantifi ed model of uncertainty. As with most other risk analysis tools presented here, the technique requires computing power to run thousands of simulations.
IGDT is in the early stages of establishing itself as a viable alternative to probabilistic conceptions of uncertainty. Given that probabilistic notions of deep and severe uncertainty are often ineff ective, an eff ective alternative would be welcome. IGDT functions well in extreme uncertainty, when agreement cannot be reached on the probability distribution. It could be simply interval bounded, with all uncertain parameters nested and scaled by a common term α.
Slowing its broader adoption throughout water resources planning and management are the diffi culties presented to practitioners trained in probability theory who must learn IGDT concepts, which is a paradigm shift as an approach to uncertainty characterization. Eff ort must be expended to clearly communicate the model structure and fi ndings, given that the technique requires a series of judgments from analysts and decision makers in constructing the uncertainty model and specifying the required minimum aspirational performance levels.
Another limitation of IGDT is that, in cases of great uncertainty, the user-defi ned best estimate of the parameter whose uncertainty is being explored may, in fact, be a poor estimate of the true value. In such cases, the largest nested uncertainty space drawn around the IGDT value might not be large enough to include the true value. In this sense, IGDT is not dissimilar from decision scaling and scenario-neutral modeling, both of which explore a limited space of potential future realizations of unknown parameters. Yet IGDT is particularly vulnerable to this shortcoming. Therefore, the "best estimate" of the uncertain parameter must be made with great care, and the uncertainty horizon explored should be drawn large enough to encompass all reasonable parameter realizations. Decision scaling improves on IGDT by starting from a logical point (climate normal) and then using projections to inform the probabilities of the space that can be derived.
This section has discussed IGDT as a risk-assessment paradigm. The aspects of IGDT that relate to decision making under uncertainty are discussed in the next section.

Risk Management Tools
Traditional decision analysis and its extensions are ill-equipped to deal with the types of great uncertainty encountered in Phase 4 of the decision tree. This section therefore summarizes some of the more prominent tools available for decision making under deep (or severe) uncertainty. Decision scaling makes iterative modifi cations to the design to trace out alternative performance and fi nd robust designs. IGDT and RDM take a similar approach, if by diff erent procedures. Stochastic optimization, multi-(or many-) objective robust optimization, and ROA use search algorithms to quickly home in on the most effi cient design modifi cations to improve system robustness.

Information Gap Decision Theory
The risk assessment and uncertainty characterization aspects of IGDT have been discussed in relation to other tools for risk assessment. This section focuses on IGDT as a tool for decision making under uncertainty.
During risk assessment, the parameters of a candidate water plan are perturbed to explore the uncertainty space around a best estimate of future parameter values. The process is similar to that described for Phase 3 in chapter 3, but uses nested uncertainty increments instead of a single uncertainty space as illustrated in a climate response map. Both the robustness and the opportuneness of the candidate water plan are evaluated within each uncertainty increment.
Risk management, Phase 4 of the decision tree, begins with the identifi cation of a number of other candidate strategies developed to reduce the vulnerabilities in the initial water plan revealed in Phase 3. If IGDT were used, the performance of each new system would be simulated, and the results of the simulations presented with robustness and opportuneness curves. In the instance of the run-of-the-river hydropower dam case study in chapter 4, the alternative water plans would be designs of diff erent generating capacity. Previous studies have explored the relative robustness and opportuneness of diff erent infrastructure portfolios for water supply (desalination, water import, and so on, as in Matrosov, Woods, and Harou [2013]), fl ood risk management options (for example, raising levee height or channel widening, as in Hine and Hall [2010]), or emission reduction paths (for example, Hall et al. [2012]), among others.
Robustness curves provide a count of the number of uncertainty increments in which the project being evaluated performs acceptably well. For the run-of-the-river hydropower development project, this might be a count of uncertainty increments in which levelized cost for each proposed design capacity is below some designated threshold. In an example from water resources planning, Matrosov, Woods, and Harou (2013) present robustness curves of storage susceptibility, total cost over the economic lifetime of the infrastructure portfolio, total energy consumption, supply reliability, and environmental factors. The larger the number of uncertainty horizons for which performance exceeds a given threshold, the more robust the infrastructure portfolio is to that threshold. Steep robustness gradients indicate minimal performance loss as uncertainty increases, and the points at which robustness curves cross mark the locations at which one infrastructure portfolio becomes relatively more robust than another.
Opportuneness curves show the potential windfall payoff for the optimistic planner. Riskier plans might have both higher potential robustness and higher potential opportuneness. The decision maker will want to weigh the potential for windfall in a riskier design against the consistent robustness of a more conservative design.
The decision maker using IGDT would evaluate the trade-off s between total system cost (or net benefi ts) and the robustness and opportuneness of the candidate water strategy and its various alternative designs with respect to all of the evaluated performance criteria. The ultimate decision, as is true for each tool for decision making under uncertainty, will be left to human judgment as informed by values, ethics, and risk tolerance.

Robust Decision Making
RDM combines attributes of scenario planning, Monte Carlo simulation, and sophisticated analytical tools to identify vulnerabilities of proposed strategies, to help decision makers identify potential responses to those vulnerabilities, and then to identify which responses are most robust. 3 RDM is built on the premise that if adequate information were available about the future, one could conduct a traditional probabilistic decision analysis, which might seek to maximize net benefi t or minimize cost. However, in the absence of high-confi dence information about the relative likelihood of various possible futures, the most reasonable choice is to seek a strategy that satisfi es one of the robustness criteria described in "The Concept of Robustness" above.
RDM inverts traditional sensitivity analysis, seeking management strategies (or project designs) whose good performance is insensitive to the most signifi cant uncertainties. The process typically begins by running a simulation model multiple times with sets of input parameters that represent many diff erent future states of the world. This formulation enables RDM to use many diff erent types of climate information, including outputs from downscaled GCMs (top-down) (Groves et al. 2015), from stochastic weather generators that provide permutations based on historical climate records (bottom-up) (Groves et al. 2008), or from combinations of the two (Bureau of Reclamation 2012; . Regardless of how the climate scenarios are developed, the next step in the RDM process is to identify a candidate robust strategy by initially ranking or screening all proposed strategies according to a stakeholder-defi ned measure of performance, and to subsequently identify and characterize one or more clusters of future states in which each of the strategies performs poorly. These clusters, unweighted by probabilities, are designed to be considered unlikely or inconvenient even by decision makers. Faced with potential futures (or clusters of futures) in which a proposed strategy (or system) performs poorly, the RDM procedure makes modifi cations to the strategy (or system) that hedge against vulnerabilities. For the run-of-the-river hydropower project, once the risk assessment for the candidate strategy has been completed, RDM would simulate the performance of a number of alternative designs (for example, hydropower-generating capacities). Finally, trade-off s involved in the choice among the hedging options would be developed.
Because RDM samples from all combinations of uncertain system parameters, it explores futures both more benign and direr than the present. In a full RDM analysis, various aspects of candidate strategies (more than just the design capacity, as in the case of the run-of-the-river hydropower project) would be successively altered and resubmitted to the RDM process until a suitably robust strategy could be identifi ed. A particular strength of RDM is its ability to consider a wide range of climate, socioeconomic, and other uncertainties, and to provide information on how system performance may be aff ected by combinations of such factors. The RDM framework makes it possible to analyze very large numbers of futures in which any or all of the system and design parameters are altered in any number of confi gurations.
Because of its dependence on modeling and computational complexity, RDM is computationally intensive. In Matrosov, Woods, and Harou's (2013) water resources system planning study, RDM implementation required approximately 310,000 simulations, whereas IGDT required only 1,600. Some RDM applications, however, have used as few as 200 futures. RDM incorporates scenario discovery algorithms, such as  "patient" rule induction method, to identify and characterize clusters of future states in which the strategy fails to meet its goals, and requires many thousands of simulation iterations to be run to trace out system performance. The method has to be guided by an expert who understands the process of homing in from all possible strategies on those that are "robust" to their vulnerabilities and who can weigh the trade-off s between clusters of future world possibilities.
RDM is well developed and has been widely tested. The process presents the decision maker with trade-off s between robust strategies and identifi es the vulnerabilities in every one of the proposed iterations' designs. It can consider a wide range of diff erent types of uncertainties. For instance, it can identify water system vulnerabilities that arise from the interaction of climate and socioeconomic uncertainties. Such vulnerabilities may not appear when considering climate uncertainties in isolation.
RDM's primary weakness is that it can be computationally intensive. In some cases it can be useful to use a qualitative RDM analysis (McDaniels et al. 2012) or conduct an initial screening analysis to determine whether a full RDM analysis could prove valuable (World Bank 2014). When RDM is applied to climate-related uncertainty in particular, care should be taken not to rely too heavily on direct output from downscaled GCMs, and to sample a breadth of uncertainty beyond that suggested by the current generation of GCMs. In narrowing down the best-performing design alternatives, particular thought should be given to the conceptualization of all future states, extreme and otherwise, and especially whether each should be weighted equally in decision making.

Dyn amic Adaptive Policy Pathways (DAPP)
The DAPP approach 4  combines the concepts of adaptive policy making (Walker, Rahman, and Cave 2001;Kwakkel, Walker, and Marchau 2010) with those of adaptation tipping points (Kwadijk et al. 2010) and adaptation pathways Haasnoot et al. 2012). The key idea is to keep plans "yoked to an evolving knowledge base" (McCray, Oye, and Peterson 2010, 952). DAPP emerged in response to the desire among Dutch policy makers to make it easier to update plans in light of new climate scenarios (Kwadijk et al. 2010), and was adopted in the Dutch Delta Program. At the same time, in the United Kingdom, pathways were explored for the Thames Estuary 2100 study (Reeder and Ranger 2011).
Adaptation tipping points are a key concept of DAPP. An adaptation tipping point specifi es the conditions under which a given plan will fail. This concept can be linked to other techniques such as RDM and decision scaling, which also focus on identifying the vulnerabilities of existing or proposed policies. An adaptation tipping point is reached when the magnitude of external change is such that the current management strategy no longer meets its objectives, and new actions are thus called for. The timing of an adaptation tipping point is scenario dependent.
The DAPP approach begins with the identifi cation of objectives, constraints, and uncertainties relevant to a particular decision-making process. The uncertainties are next used to generate a variety of plausible futures. System performance in each of these futures is compared with the objectives to determine if problems are likely to arise or if opportunities are likely to occur. This analysis determines whether and when policy actions are needed, either through simulation-based techniques or optimization-based techniques. If simulation-based techniques are used, then a large number of model runs are performed, and the statistics of the results are analyzed to determine the general eff ectiveness of various design (or policy) alternatives. If optimization-based techniques are used, the best-performing suite of design alternatives and policies are identifi ed by the optimization algorithm (and subject to the algorithm's particular biases and limitations).
To assemble a rich set of possible actions, the approach distinguishes between four types of actions: (1) shaping actions, (2) mitigating actions, (3) hedging actions, and (4) seizing actions (Kwakkel, Walker, and Marchau 2010). In subsequent steps, these actions are used as the basic building blocks for the assembly of adaptation pathways. The performance of each of the actions is assessed in light of the defi ned objectives to determine the adaptation tipping point of the action. Once a set of actions seems adequate, potential pathways (that is, a sequence of actions) may be constructed, and one or more preferred pathways can subsequently be selected as input to a dynamic, robust plan. The aim of this plan is to stay on the preferred pathway as long as possible. Contingency actions for staying on the pathway are defi ned, and a trigger for each contingency action is specifi ed and subsequently monitored (Kwakkel, Walker, and Marchau 2010). Figure 5.1 shows a stylized example of an adaptation pathway map, including a scorecard for each of the pathways. In the map, starting from the current situation, targets are fi rst missed after four years. Following the grey line of the current plan, four alternatives become available. Based on an assessment of current projections of future conditions, Actions A and D are expected to be able to satisfy the performance threshold for the next 100 years in all scenarios. If Action B is chosen, a tipping point is reached in about another fi ve years: the tipping point indicates that some performance metric will dip below the associated threshold in at least one future scenario. Achieving the set targets will require a shift to one of the other three actions defi ned (A, C, or D). If Action C is chosen, a shift to Action A, B, or D will be needed after approximately 85 years, the set targets would be achieved for the next 100 years, except in the case of the worst-case scenario (dashed green line). In the worst-case scenario, Action C would need to be abandoned after 85 years. The colors in the scorecard refer to the actions: A (red), B (yellow), C (green), and D (blue).
The adaptation pathway approach has several strengths. First, it is relatively easy to understand and explain. The metro map visualizations, as used in fi gure 5.1, are evocative, and help in visualizing the diff erent routes that are available for achieving the desired objectives. Second, the approach

FIGURE 5.1 Example of an Adaptation Pathways Map and Scorecard
Sour ce: Reproduced from .

Note:
The map on the left shows the possible adaptation pathways, and the scorecard on the right presents the costs and benefi ts of the nine possible pathways depicted in the map. + = positive costs or effects (positive costs being "bad" and positive target or side-effects being "good"); -= negative costs or effects. More + or -presented sequentially indicates the more positive or negative the costs or effects. encourages decision makers to think about "what if" situations and their outcomes, and to make decisions over time to adapt, while maintaining the fl exibility to make future changes (Jeuken and Reeder 2011). This approach also helps designers anticipate undesirable lock-ins or other path dependencies, which can consequently be avoided. Third, the adaptation pathway approach explicitly frames adaptation as a dynamic process that takes place over time. It forces those involved to consider transient scenarios, rather than one or a few points in time (Haasnoot et al. 2012).
Although initially applied to managing climate change-related uncertainties and perspectives (Off ermans, Haasnoot, and Valkering 2011), recent work has expanded the approach to cover other uncertainties as well, such as land use change (Kwakkel, Walker, and Marchau et al. 2014). The approach has also been put into actual practice. For instance, it is a cornerstone of the Dutch climate adaptation strategy in the water domain ( van Rhee 2012). It was also a key element of the Thames Estuary 2100 study, and is being used in the development of climate adaptation strategies in New York (Rosenzweig et al. 2011;, Bangladesh, Vietnam, and New Zealand (Lawrence and Manning 2012).
The adaptation pathway approach is continuing to be developed further. Related challenges include coping with multiple diverging objectives and a lack of clarity at present on exactly how to apply existing economic evaluation approaches to adaptation pathways. It is particularly diffi cult to account for the dynamic nature of adaptation pathways-the metro maps must be continually updated to refl ect new information and new understandings of the anticipated best path forward. Similarly, most current examples of DAPP refl ect a conservative, risk-averse, decision-making approach in which the switch is made from one pathway to another (with potentially signifi cant transaction costs) when system performance in even a single future scenario is unsatisfactory in the fi rst pathway (dashed lines in the "metro maps" notwithstanding), however dire and unlikely that future scenario may be. Improvements to the method could include probabilistic decision-making frameworks that are not dominated by the worst case. A particular diffi culty in using adaptation pathways is that the metro maps, although visually highly instructive, can become overly complicated and cluttered for more complex problems. This complexity can be ameliorated somewhat by the relabeling and grouping of candidate actions.

Stoc hastic Optimization
Stochastic optimization is an optimization technique in which uncertain future scenarios are weighted probabilistically and the resulting "best" design performs reasonably well across the range of considered futures. Stochastic optimization off ers a fi rst level of hedging against infeasibility, and thus is a step toward robustness. For summaries of stochastic optimization techniques that make use of probabilistic uncertainty paradigms for water systems decision making, see Revelle, Whitlatch, and Wright (2004), Loucks, Stedinger, and Haith (1981), Loucks and van Beek (2005), and Sen and Higle (1999). Multiobjective robust optimization (RO), briefl y discussed earlier, extends stochastic optimization to make it more explicitly robust to challenging scenarios.

Cost-Benefi t Analysis under Uncertainty
Traditional decision analysis emphasizes optimality under an expected future. Cost-benefi t analysis (CBA) is an example of this technique . CBA under uncertainty (Arrow et al. 1996) is an improvement on CBA in the face of an unclear future, and is applicable to problems in which uncertainty is quantifi able. However, when representing the uncertainty associated with climate change indices (for example, temperature and precipitation) with Gaussian or other asymptotically diminishing probability distribution functions, the CBA under uncertainty method is extremely sensitive to tails of the distribution functions Weitzman 2009). In situations of deep uncertainty, therefore, CBA under uncertainty is best used as a screening tool, and not as a replacement for the type of indepth, bottom-up analysis adopted here .

Multiobjective Robust Optimization
According to Sahinidis (2004), there are three general methods for optimization under uncertainty: stochastic programming, fuzzy programming, and stochastic dynamic programming. Stochastic programming includes (1) the standard approaches using recourse models (termed two-stage or multistage stochastic linear and nonlinear programs; see Sen and Higle [1999] for an introductory tutorial on stochastic programming); (2) RO, as described below; and (3) probabilistic models (chance constraints, attributed to Charnes and Cooper [1959]); see Loucks, Stedinger, and Haith (1981) and Tung (1986) for an introduction to and applications of chance constraints to water resources problems. Potentially, a fourth class of methods for optimization under uncertainty-evolutionary optimization algorithms-can be connected to Monte Carlo simulation models of water resources systems (for example, Kasprzyk et al. 2009).
In contrast to chance constraint techniques, for example, which can only limit the probability of a violation of a model constraint, RO off ers a means of simultaneously controlling the sensitivity of the solution to any uncertain parameters or inputs and penalizing exponentially larger violations of one or multiple model constraints. There are two forms of RO models: those that guarantee satisfaction of hard constraints, and those that apply penalties to violations of soft constraints. The fi eld of manufacturing and engineering science has tended to emphasize formulations that guarantee the satisfaction of hard constraints, thus leading to single optimal robust solutions (Ben-Tal andNemirovski 1998, 1999;Taguchi 1989). By contrast,  recommend a mathematical programming approach based on a trade-off between solution robustness (nearness to optimality across all scenarios) and feasibility robustness (nearness to feasibility across all scenarios). Their RO formulation extends stochastic programming to a multiobjective optimization framework that includes higher moments of the objective value (variance, most commonly) and a penalty function or functions on violations of one or more chosen constraints. The scenarios used in RO are discrete points in an empirical probability distribution (or joint probability distribution), generated to represent best current understanding of the relative likelihood of potential future system states.
Applications of this type of RO in water resources range from water distribution system design (Cunha and Sousa 2010) and wastewater treatment design (Afonso and Cunha 2007) to the design of large-scale water systems (Escudero 2000), as well as the design of groundwater pump and treatment systems (Ricciardi, Pinder, and Karatzas 2009). Nearly all previous water resources applications involve only feasibility robustness, and do not consider solution robustness. Most such examples deal with groundwater remediation applications (for example, Alcolea et al. 2009;Bau and Mayer 2006;Bayer, Buerger, and Finkel 2008;Ko and Lee 2009;Ricciardi, Pinder, and Karatzas 2007). Only a few applications of RO to water resources systems also include solution robustness through the minimization of variance or standard deviation of direct cost in the objective function (for example, Kasprzyk et al. 2009;Kawachi and Maeda 2004;Suh and Lee 2002;.

Real Options Analysis
As was pointed out in "Phase 4: Climate Risk Management" in chapter 3, a strong water system management plan will combine elements of adaptability and fl exibility, diversifi cation, and robustness. ROA is applicable when (1) uncertainty is more "dynamic" than "deep"-knowledge improves over time, and (2) the project involves irreversible creation or destruction of capabilities. Certain adaptation strategies are more fl exible than others to the possibility of future upgrading if climate change impacts are high. The expected value of each option (more fl exible and less) can be calculated and compared. The objective in this formulation is still to maximize net present value, but the adaptability of design options is explicitly considered. The government of the United Kingdom, for example, requires that climate change adaptation analyses account for "the value of fl exibility in the structure of the activity" (HMT and Defra 2009, 14).
ROA is an established probabilistic decision process (and a subset of stochastic optimization) by which adaptability can be explicitly incorporated into project designs, and large potential regrets associated with either overinvestment or underinvestment in adaptation measures can be avoided. ROA encourages staged decision making, through which more expensive and more highly irreversible decisions are delayed until more information is available on which to base those decisions. The philosophical underpinning of ROA has roots in the work of Dewey (1927), who promoted policies that incorporate continual learning and adaptation in response to experience over time, and Rosenhead (1989), who presented "fl exibility" as an indicator by which to evaluate the robustness of strategies under uncertainty. The mechanism for real options is founded in the analysis of fi nancial decision making Myers 1984;Copeland and Antikarov 2001).
ROA focuses on making changes to the system confi guration in reaction to reductions to uncertainty through future learning (de Neufville 2003). Wang and de Neufville (2005) present real options analyses as subsets of real options "on" systems (focusing on the external factors of a system, with greatest benefi t gained through fi nancial valuation methods), and real options "in" systems (incorporating fl exibility into the structural design). Most water resources engineering design problems are of the latter type.
The adaptive policy-making paradigm, of which ROA is one element, has received increased attention in water resources planning and management in recent years. As of 2013, dynamic adaptive plans were being developed or had already been developed for water management in New York, New Zealand, the Rhine delta, and the Thames estuary. In the Netherlands, ROA has been used to assess optimal costs and benefi ts of pathways for fresh water supply of the South-West Netherlands Delta, and for studying how fl exibility can be built into fl ood risk infrastructure .
Examples of real options for water supply include investments in pumps to draw upon dead storage, pipelines to connect to storage at another impoundment, or infrastructure to tap groundwater resources. Demand-oriented real options for water supply are also possible, such as investments in household metering and strong public outreach campaigns that could be activated at some cost to help enforce future conservation eff orts (Steinschneider and Brown 2012). Real option water transfers provide a mechanism by which water supply can be augmented without the need for large-scale infrastructure expansion.
Assume that there is an equal probability of high and low climate change impacts in the future. The standard wall costs 75, and has benefi ts of 100 from avoided fl ooding. The upgradeable wall costs 50, its upgrade costs 50, and the upgraded wall would give benefi ts of 200 from avoided fl ooding. The information can be presented in a decision tree, as shown in fi gure 5.3. Simplifying assumptions: Residual damages under "do not invest" strategies have been ignored; the discount factor is 0.8. The expected value of investing in the standard wall is derived through a simple NPV calculation of the expected costs and benefi ts of the investment. The NPV is (0.5 × 25) + (0.5 × −75) = −25. This suggests the investment should not proceed.

FIGURE 5.2 Illustration of an Upgradeable Wall as an Example of Flood Risk Management Real Options
Flexibility with regard to the investment decision allows for upgrading in the future if the impacts of climate change are high. The expected value of this option can be calculated. If the impacts of climate change are high enough to warrant upgrading, the value of the investment is 120. If the impacts are low, upgrading is not justifi ed, since the payoff is negative (−40). Because the investment costs of the upgrade are not realized in practice in the low outcome, they are not incorporated into the NPV. The expected value of investing now with the option to upgrade in the future is (0.5 × 120) − 50 = +10.
Comparing the two approaches shows an NPV of −25 for the standard approach, and +10 for the ROA. Flexibility to upgrade in the future is refl ected in the higher NPV, and switches the investment decision.

Summary of Decision Making under Uncertainty
This chapter presents options for use once the water resources program designer (or other user of the decision tree) has deemed climate (and other) information suffi ciently well-characterized to proceed to the step of decision making under uncertainty. The options are presented in relation to the phase of the decision tree for which they are most appropriate. This chapter further expands the description of the options for decision making under uncertainty presented in Phase 4 of the decision tree. Though all these tools generally aim at robustness, they diff er signifi cantly in their conceptualizations of uncertainty, modeling philosophies, and solution techniques. The tool chosen for decision making under uncertainty should be the tool that best fi ts the particular water resources context, the project budget and time frame for project evaluation, and the program designer's personal level of comfort with the available methodologies. Notes 1. Reoperation is the redevelopment of operation rules for a water system. For example, the operator of a multipurpose reservoir facing a decrease in precipitation might choose to raise the normal maximum water level to better safeguard against drought (at the expense of the capacity of the dam to capture fl ood fl ows).
2.  give an example of a well-constructed study that presents future fl ood risks without providing any method by which planners can decide whether investment in increased robustness is warranted. It can only be assumed that the fl ood risks presented to the decision maker in this case were somehow weighed against the costs of alternative mitigating strategies, but that the discussion was outside the scope of the publication. 3. Special thanks to David Groves and Robert Lempert for their contributions in the writing of this section about RDM. 4. Special thanks to Jan Kwakkel for his contribution in the writing of this section about DAPP.
into fl ood damage. The water system model will usually include elements of control where the economic impact of infrastructure or policy decisions can be explored or optimized. For a fl ooding river, the decisions might involve the construction of levees of various heights, reclamation of land within the fl oodplain, or development of reservoir storage capacity.
Water system decision tools have been discussed throughout this book. This appendix provides a very brief introduction to hydrologic models and off ers recommendations for the best tools available for various purposes, with an eye to compatibility with a climate stress test.
In choosing a hydrologic model, the questions to ask are the following: (1) What type of output is needed? (2) What is the basin size? (3) What type of modeling speed is needed?
For a small basin, a lumped model should suffi ce, and could be built and run quickly. Lumped models have no geographic heterogeneity, and represent all features of the watershed at a point. For larger basins, distributed models are preferred. Distributed models are typically gridded, with each grid cell of the watershed carrying unique model parameters (for example, elevation, ground cover, impervious area). In general, distributed models can be conceptualized as a gridded array of lumped submodels, each connected by routing models (fl ow vectors) to collect streamfl ow at the point of interest (typically the lowest elevation point, which is the basin outlet). Likewise, two or more distinct lumped models, each developed at its own time for its own purpose can usually be combined into nongridded (somewhat more organic) versions of distributed models with the inclusion of a routing model. When preexisting, nongridded lumped models are combined using a routing model, the result is referred to as a semi-distributed model (not fully distributed, given that they lack the continuity and uniformity of fully distributed models). The question of how to divide a larger catchment into smaller catchments and when to add a routing model is an area of active research and requires expert judgment.
Typically, by "hydrologic" model, what is meant is "rainfall-runoff " model. The most commonly requested output from a hydrologic model is streamfl ow, though other outputs are possible, such as soil moisture and groundwater seepage. Explorations of groundwater movement are more appropriate in the realm of porous-media hydraulics than hydrology. The most popular software package for modeling groundwater movement is called MODFLOW. A modular hydrologic system might be a piecemeal assemblage of elements of the total hydrologic system, for example, an element for evapotranspiration (for example , Hamon 1961;Hargreaves 1975), one for groundwater movement (for example, MODFLOW), one for soil moisture (for example, variable infi ltration capacity, or VIC), and one for streamfl ow routing (for example, Muskingum [Cunge, 1969], Soil Conservation Service unit hydrograph, Saint-Venant [1871] equation).
In the United States, the most widely used hydrologic models are the Precipitation Runoff Modeling System (PRMS, used by the United States Geological Survey), the Sacramento Model (used by the National Weather Service for its Advanced Hydrological Prediction Service), the Community Land Model (CLM, of the National Center for Atmospheric Research, and used as an element of general circulation models [GCMs]), the University of Washington's VIC Model, the Soil and Water Assessment Tool (SWAT, developed by the United States Department of Agriculture and Texas A&M), the Water Evaluation and Planning system (WEAP, of the Stockholm Environment Institute, the hydrological modeling component of which is called WATBAL, short for Water Balance), TOPMODEL (of Lancaster University), and the simple, pedagogically excellent abcd model. Some of these models are physically based and inherently distributed (for example, VIC, CLM, PRMS), meaning that they approximately represent actual natural processes including water, energy, and soil interactions. Some of these models are conceptual (for example, the Sacramento model, the abcd model, WEAP), and keep track of the water balance without reproducing the complicated relationships with energy and soil. Some models straddle the line between conceptual and physically based. If the Sacramento model is coupled with Snow-17, 2 it approximately fi ts into the partially physically based category. SWAT and TOPMODEL could similarly be described as partially physically based, while TOPMODEL is semi-distributed.
The analogy could be made with methods of GCM downscaling. Conceptual models are like statistical downscaling techniques, which are fast, highly effi cient, and typically quite accurate. Physically based models fi t the dynamic downscaling metaphor: they are slower, more complex, and more diffi cult to fi t to historical observations. However, the output of physically based models is more useful, easier to interpret, and generally more meaningful to the modeler.
In the opinion of this book's authors, once computing power improves to a suffi cient level (and effi ciency improvements are made in the hydrologic model algorithms), distributed, physically based watershed modeling systems will be preferred for most applications. Until then, conceptual models will be useful in many cases because of their speed, accuracy, and simplicity. Once certain modifi cations have been made, and given continuing eff orts at better representation of true physical processes in combination with the availability of parallel computing power, physically based models hold great utility.

Variable Infi ltration Capacity (VIC) Macroscale Hydrologic Model
Source: University of Washington Site: http://www.hydro.washington.edu/Lettenmaier/Models/VIC/ Lumped/Distributed: Distributed Conceptual/Physically based: Physically based Appropriateness for climate stress test: Slow, probably needs parallel computing power to run many thousands of times.
Other comments: The VIC model is a macro-scale hydrologic model that solves full water and energy balances, originally developed at the University of Washington. It is physically based and distributed, allowing variation in land cover (vegetation type) in the horizontal plane, as well as variation in soil layer characteristics in the vertical plane. It solves the water balance and energy balance at the same time. The output is physically meaningful and comprehensive (soil moisture layer interactions, runoff from sub-areas, and so forth). Many good sources of input data are available from remote sensing and interpolated ground-based readings. VIC (as well as CLM) is part of global climate models, which integrate GCMs with sea ice and landsurface components.

Sacramento (originally named the Stanford Watershed Model)
Source: National Weather Service River Forecast System Site: http://www.nws.noaa.gov/iao/iao_hydroSoftDoc.php Lumped/Distributed: Lumped Conceptual/Physically based: Conceptual Appropriateness for climate stress test: Highly appropriate, fast.
Other comments: What is now known as the Sacramento model was fi rst developed in the 1960s as the Stanford Watershed model-the fi rst known computer-based watershed model. The Stanford Watershed model has many branches. Its immediate successor, the Hydrological Simulation Program-Fortran, "HSPF," is the FORTRAN programming language version developed in the 1970s. When it was translated into the C programming language it became the Sacramento model. The Sacramento model is a conceptual lumped-parameter model focused on soil moisture, which it divides into several components (tension water storage for upper layer and free water storage for lower layer). The Sacramento model also supports diff erent vegetation types and diff erent soil layers. It solves the water balance, and can take into account the energy balance if it is coupled with the SNOW-17 module.
Other comments: TOPMODEL encompasses a set of programs for rainfallrunoff modeling in single or multiple subcatchments in a semi-distributed way and using gridded elevation data for the catchment area. It is considered a physically based model because its parameters can be, theoretically, measured in situ (Beven and Kirkby 1979;Beven, Schoffi eld, and Tagg 1984). TOP-MODEL 95.02, written in Fortran 77, is suited to catchments with shallow soils and moderate topography, and that do not suff er from excessively long dry periods. TOPMODEL is a variable contributing area conceptual model, in which the major factors aff ecting runoff generation are the catchment topography and the soil transmissivity, which diminishes with depth. The present model version includes two mechanisms to estimate surface runoff production: infi ltration excess and saturation excess (http://s1004.okstate.edu/ S1004/Regional-Bulletins/Modeling-Bulletin/TOPMODEL.html).
Lancaster uses the generalized likelihood uncertainty estimation (GLUE) methodology to carry out calibration, sensitivity analysis, and uncertainty estimation based on many thousands of runs. The current version of TOP-MODEL provides an option for output of Monte Carlo simulation results for later use with the compatible GLUE package. Three options are available in the program: (1) the Hydrograph Prediction Option, (2) the Sensitivity Analysis Option, and (3) the Monte Carlo Analysis Option. The results fi le produced will be compatible with the GLUE analysis software package.
Water Evaluation and Planning/Water Balance (WEAP/WATBAL) Source: Stockholm Environment Institute Site: http://weap21.org/index.asp Lumped/Distributed: Lumped models, with the critical added benefi t of simplicity (parameter parsimony), and the supplemental added benefi t that each of its parameters has a physical interpretation. Note that "monthly" water balance models may be useful at shorter time scales (that is, daily) if the basin is small enough that all important dynamic processes not included in the model occur within the timestep of the model.
The abcd model is a nonlinear watershed model that accepts precipitation and potential evapotranspiration as input, and produces streamfl ow as output. Internally, the model also represents soil moisture storage, groundwater storage, direct runoff , groundwater outfl ow to the stream channel, and actual evapotranspiration. The abcd model was originally introduced by Thomas (1981) and Thomas et al. (1983), at Harvard University. Because of its simplicity and the ability of its few parameters to be described in physical terms, the model makes an excellent pedagogic tool. It also can be developed quickly, and runs quickly, making it an attractive basis for a climate stress test. The abcd model is unrelated to, and has a completely diff erent structure from, the linear "abc" model (Fiering 1967).

Precipitation Runoff Modeling System (PRMS)
Source: United States Geological Survey Site: http://wwwbrr.cr.usgs.gov/projects/SW_MoWS/PRMS.html Lumped/Distributed: Distributed Conceptual/Physically based: Physically based Appropriateness for climate stress test: Slow, probably needs parallel computing power to run many thousands of times.
Other comments: PRMS was developed to evaluate the response of various combinations of climate and land use on streamfl ow and general watershed hydrology. The primary objectives are (1) simulation of hydrologic processes including evaporation, transpiration, runoff , infi ltration, and interfl ow, as determined by the energy and water budgets of the plant canopy, snowpack, and soil zone, on the basis of distributed climate information (temperature, precipitation, and solar radiation); (2) simulation of hydrologic water budgets at the watershed scale for temporal scales ranging from days to centuries; (3) integration of PRMS with other models used for natural resource management or with models from other scientifi c disciplines; and (4) to provide a modular design that allows for selection of alternative hydrologicprocess algorithms from the standard PRMS module library (ftp://brrftp. cr.usgs.gov/pub/mows/software/prms/prmsSummary.txt).
Other comments: SWAT (Arnold and Allen 1992;Arnold, Williams, and Maidment 1995) operates continuously, on a daily time step, and is designed to predict the impacts of management practices on hydrology, sediment, and water quality on an ungauged watershed. Major model components include weather generation, hydrology, sediment, crop growth, nutrient, and pesticides. Integration with geographic information systems (GIS) was accomplished by Srinivasan and Arnold (1994). SWAT is a small watershed to river basin-scale model for simulating the quality and quantity of surface and ground water, and predicting the environmental impact of land use, land management practices, and climate change. SWAT is widely used in assessing soil erosion prevention and control, non-point-source pollution control, and regional management in watersheds. SWAT has a high-quality user interface and is particularly user friendly.

Notes
1. This appendix presents a collection of hydrologic models that, in the authors' experience, are the most trusted and most commonly used hydrologic models in the United States. Inclusion of a particular hydrologic model in this section does not constitute an endorsement of the model by the authors, but rather an expression of its relatively common use and citation. There are, of course, many other hydrologic models not included in this appendix, and their omission should be interpreted as nothing other than the authors' lack of familiarity with them. 2. SNOW-17 is a conceptual model. Most of the important physical processes that take place within a snow cover are explicitly included in the model, but only in a simplifi ed form.

Worksheets and Report Templates
Phase 1: The Climate Screening Worksheet Step 1 Describe the proposed project and its context. Describe the known and poorly understood parameters that aff ect project performance.
Step 2 Is this a water infrastructure project?
• What are the stakeholder-defi ned performance indicators and risk thresholds? • What is the project's anticipated economic lifetime? as climate insensitive, indicate that here. If the proposed project, despite being of this type, may potentially be sensitive to climate change, conduct a Four C's analysis.

The Four C's: Choices, Consequences, Connections, unCertainties
Within the contextual guidance provided by the Four C's, strategic questions are suggested, and the project designer is encouraged to add or delete questions based on relevance to the particular project.
Choices: What are the design options? Are diff erent system sizes and confi gurations possible? Are there substitutable parts? Is the project fl exible in time or space? Can the project be modular? What is the timing of the project? Can the project be delayed, or must it be initiated immediately? What are the fi nancing options? Is there wide agreement on the choices available?
Consequences: What are the project's benefi ts and costs, or are other performance indicators primary? How are benefi ts and costs distributed spatially, temporally, and socioeconomically? How are the project's benefi ts and costs measured? Can all benefi ts and costs be monetized? What are the performance thresholds, that is, the criteria for designation as failure or success? What are the windfall possibilities (best case), and what are the worst-case

Introduction
This process is especially relevant to proposed projects shown in Phase 1 to have climate sensitivities, but with questions remaining about the significance of those sensitivities relative to vulnerabilities of other kinds. The Initial Analysis should be used to evaluate the relative signifi cance of climate change vulnerabilities. Phase 1 projects that are likely to exit the decision tree after a Phase 2 evaluation are wastewater projects; other infrastructure projects with design lives shorter than 20 years; and continuous or long-lasting policies, training activities, or hydrometeorological services. However, hydrometeorological services may have signifi cant sensitivities to climate change, depending on the specifi c context. Generally, infrastructure projects with design lives longer than 10-20 years, especially projects in geographic regions with high inter-or intra-annual climate variability, will require more thorough project scoping in Phase 3 (more carefully constructed hydrologic and water resources system models), with particular attention given to capturing potential changes or shifts in climate other than percentage changes in annual average temperature and precipitation. Depending on the results of the Initial Analysis, a number of these projects may require more in-depth climate risk assessment in Phase 3 of the decision tree.

Rapid Project Scoping
The rapid project scoping procedure described in chapter 3 is one option for execution of a Phase 2 Initial Analysis. Other options are the patient rule induction method (PRIM), or more conventional single-factor sensitivity analysis, for example. Rapid project scoping is based on a simplifi ed analysis of the hydrology and climate change projections (for example, Grijsen 2014) and the evaluation of the elasticity of system performance with respect to climate and other, nonclimate uncertainties. Details of the procedure may be found in "Description of Phase 2" in chapter 3.

Climate-Related Data Resources
Tip: For quick identifi cation of anticipated climate changes in the location of the planned project, consult the following: • The World Bank's Climate Change Knowledge Portal (http://sdwebx.worldbank.org/climateportal/index.cfm). • The Nature Conservancy's Climate Wizard (http://www.climatewizard. org/).

Synthesis
If the project scoping phase confi rms that, though the project has climate sensitivities, those sensitivities are small relative to sensitivities to uncertain factors of other types (for example, demographic or political factors), standard means for evaluating the project (that is, traditional decision analysis) are expected to be suffi cient. For expensive or complex projects, it is recommended, given the potential sensitivities identifi ed, that the project designer employ some measure of robustness to the traditional decision analysis, such as safety margins, sensitivity analysis, or adaptive management (see "Extensions to Traditional Decision Analysis" in chapter 5). If climate change-related uncertainties are shown to be relatively signifi cant, more in-depth exploration of those potential vulnerabilities in Phase 3 is warranted.
Phase 3: Guidance for the Climate Risk Report Design, execution, and interpretation of the climate stress test require specialized staff or outside expertise.

Stress Test Procedure
1. At this point, if a formal model of the natural, engineered, or socioeconomic system is not available, it must be created so that climate conditions can be related to the impacts to performance indicators identifi ed in Phase 2. 2. Performance measures and thresholds are established. 3. With the help of expert consultants or highly qualifi ed internal specialists, a standard climate response map is generated for the base design to exhaustively explore the vulnerabilities of the project to change (in climate and other relevant factors). This step is completed using a weather generator and a wide range of projections of uncertainties of other kinds (for example, demographic, economic, land use). 4. Hazards to the project (and their magnitudes) are noted. 5. Probabilities are assigned to subsets of the projected future domain in which particular vulnerabilities are apparent (ex post scenarios). General circulation models fi rst enter the analysis at this point, and are used to inform probabilities in combination with historical trends, paleoclimatology data, and all other useful sources of climate information. 6. Risks are quantifi ed as a function of impacts (hazards) and probabilities.

The Climate Risk Report
1. Summarizes the procedure used to generate the climate response map, and all assumptions and mathematical relationships. 2. Fully describes all statistical, hydrologic, and water system models (or models of other kinds) constructed for or applied to the climate stress test. The weather generator should receive special attention here.