From Waste to Resource Shifting paradigms for smarter wastewater interventions in Latin America and the Caribbean Background Paper III: The Role of Modeling in Decision Making in the Basin Approach © 2019 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington, DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Please cite the work as follows: “World Bank, 2019. From Waste to Resource: Shifting paradigms for smarter wastewater interventions in Latin America and the Caribbean. Background Paper III: The Role of Modeling in Decision Making in the Basin Approach” World Bank, Washington, DC. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org Cover and report design: Alejandro Scaff Herrera. From Waste to Resource Background Paper III: The Role of Modeling in Decision Making in the Basin Approach The World Bank is working with partners around This background paper is part of the supporting the world to ensure that wastewater’s inherent material for the report “From Waste to Resource: value is recognized. Energy, clean water, fertilizers, Shifting Paradigms for Smarter Wastewater and nutrients can be extracted from wastewater Interventions in Latin America and the Caribbean”, and contribute to the achievement of the a product of the “Wastewater: from waste to Sustainable Development Goals. Wastewater resource”, an Initiative of the World Bank Water can be treated up to different qualities to Global Practice. It contains an overview of the main satisfy demand from different sectors, including types of models used in basinwide water quality industry and agriculture. It can be processed assessments, and their data requirements. in ways that support the environment, and can even be reused as drinking water. Wastewater Models are an integral part of the basinwide treatment for reuse is one solution to the world’s water quality management process, in that they water scarcity problem, freeing scarce freshwater provide a quantitative link between pollutant resources for other uses, or for preservation. In sources and receiving water quality. addition, by-products of wastewater treatment can become valuable for agriculture and energy Models are mathematical representations of natural generation, making wastewater treatment processes and thus are approximations of reality. In plants more environmentally and financially the water sector there are myriad models to analyze sustainable. Therefore, improved wastewater a wide array of issues, for example, flooding, water management offers a double value proposition availability, water quality, aquatic ecosystems, and if, in addition to the environmental and health climate change, to name a few. Some processes benefits of wastewater treatment, financial are easier to model, for example, determining the returns can cover operation and maintenance extent of inundation caused by a flood event. Other costs partially or fully. Resource recovery from processes, especially those that have biochemical wastewater facilities in the form of energy, reusable components, are exceedingly complex and the water, biosolids, and other resources, such as mathematical approximations do not always nutrients, represent an economic and financial capture this complexity well. Such is the case of benefit that contributes to the sustainability of aquatic ecosystems that involve food webs and water supply and sanitation systems and the water energy fluxes. Nevertheless, with good data and in utilities operating them. One of the key advantages the hands of an experienced modeler, models can of adopting circular economy principles in the provide valuable information to make decisions, processing of wastewater is that resource recovery specifically when evaluating various options. and reuse could transform sanitation from a costly service to one that is self-sustaining and adds value As shown in figure 1, models serve as a tool to to the economy. estimate the effects of various water quality controls on actual water quality. Figure 1 Evaluating the effects of various management options on the water quality of receiving water bodies Control actions Input Output Wastewater treatment Model Water quality Nonpoint source controls Source: LimnoTech, 2018 3 From Waste to Resource Evaluating a range of management actions considered. Options that provide acceptable water helps determine which set of actions will provide quality can be further evaluated to determine acceptable water quality (figure 2). If the predicted which is most cost-effective. This section provides water quality is unacceptable for one option, guidance on how models can be used to support that option can be rejected and alternatives basinwide water quality management assessments. Figure 2 Application of models in basinwide planning Control actions Input Output Wastewater treatment Model Water quality Nonpoint source controls NO Evaluate additional Is water quality controls acceptable? YES Evaluate cost feasibility Source: LimnoTech, 2018 Data needs Models are tools that integrate available data to have specific data requirements. Figure 3 provides provide increased understanding of a system. The an overview of the two main types of models used, models used in basinwide water quality assessments and the model inputs that require site-specific data. 4 From Waste to Resource Figure 3 Types of models used in basinwide planning Depending upon the level of detail at which the model is applied, additional data may be required Example inputs defining soil type, subbasin boundaries, and Basin loading model installed drainage networks. (data needs) Precipitation Urban Rural Topography land land The second type of model provides information on Land use the water quality, that is, pollutant concentrations that result from pollutant loads, and is commonly referred to as a Receiving Water Quality Model. This model accepts runoff and pollutant load data from the Basin Loading Model. It also requires data to describe bathymetry (e.g., lake or river Example outputs depth and width), pollutant loading rates from Tributary flow point source discharges (e.g., existing or proposed Pollutant loads wastewater treatment plants [WWTPs]), and current water quality. This type of model simulates the physical processes that take place as the pollutants enter the receiving water body, be it a river, lake, or reservoir: hydrodynamic dispersion, Example inputs Receiving water biological transformation, and chemical reactions. (data needs) quality model Bathymetry Table 1 presents examples of commonly used Hydraulics Point sources models of both types. In some of the examples, the Observed water quality Water quality Basin Loading Model is coupled with the Receiving Water Quality Model. Table 1 should be used only for information purposes and not to select models to address a given problem. The model selection process requires specialized knowledge and is described below. Example outputs Concentrations of pollutants in rivers, lakes, Model selection or reservoirs A wide range of models exist that can Source: LimnoTech, 2018 support basinwide planning, so an initial step in the planning process is the selection of The first type of model used in basinwide an appropriate model. Several good model assessment provides information on pollutant selection compendiums are available (USEPA loads generated via runoff from the land surface 1997; LimnoTech 1999; Shoemaker et al. 2005). (nonpoint sources), and is commonly referred to as There is no one best model for all applications; a Basin Loading Model. These models require data model selection should be driven by an explicit describing: consideration of (i) management objectives, (ii) site-specific characteristics, and (iii) resource • Precipitation constraints (DePinto et al. 2004). • Land use and land cover • Topography (i.e., land slope) 5 From Waste to Resource Table 1 Examples of commonly used models Supporting organization Model Model name Source and country type U.S. Department of Agriculture– Soil and Water Assessment Agricultural Research Service BL (rural) http://swat.tamu.edu Tool (SWAT) (USDA-ARS)/Texas A&M University, United States Agricultural Nonpoint http://www.ars.usda.gov/Research/docs. Source Pollution Model USDA-ARS, United States BL (rural) htm?docid=5222 (AGNPS) Generalized Watershed Pennsylvania State University, BL http://www.mapshed.psu.edu/ Loading Function (GWLF) United States https://www.epa.gov/water-research/ Storm Water Management U.S. Environmental Protection storm-water-management-model- BL (urban) Model (SWMM) Agency (USEPA), United States swmm Hydrologic Simulation AquaTerra Consultants, California, BL and Program—FORTRAN (HSPF) http://water.usgs.gov/software/HSPF/ United States RW MIKE SHE model Danish Hydraulic Institute, BL and http://www.mikepoweredbydhi.com/ Denmark RW Watershed Analysis Risk Systech Water Resources, Inc., BL and http://systechwater.com/warmf_ Management Framework California, United States RW software/ (WARMF) River Basin Simulation Model BL and https://www.deltares.nl/en/software/ Deltares, Netherlands (RIBASIM) RW ribasim/ https://cfpub.epa.gov/si/si_public_ QUAL2 USEPA, United States RW record_Report.cfm?dirEntryID=75862 Water Quality Analysis https://www.epa.gov/exposure- Simulation Program (WASP) USEPA, United States RW assessment-models/water-quality- analysis-simulation-program-wasp Environmental Fluid https://www.epa.gov/exposure- Dynamics Code (EFDC) USEPA, United States RW assessment-models/efdc Source: LimnoTech 2018, Adapted from Stone Environmental and LimnoTech (2014). Note: WL = Basin Loading Model; RW = Receiving Water Quality Model. As discussed above, the selected model must models are only as good as the data upon which be capable of addressing the management they are based. Time and schedule are other objectives defined during problem specification. important considerations, as sufficient time must The final consideration in model selection is be made available to collect the necessary data, resource constraints, which can be grouped into develop the model, and conduct predictive data, time, and human resources. Site-specific scenarios. Human resources are a third important data are essential to any model application, as component, both in terms of quality and quantity. 6 From Waste to Resource Sufficient staff time must be made available for is part of the adaptive management component model development and application. The proper of the basin approach, in which adjustments are application of models also requires significant made as more data become available. This course training and judgment, making the modeler’s of action is needed frequently in developing skill an important consideration. One additional countries, where lack of data is often an issue. resource is budget, as this can dictate the amount Basic data at a large scale (e.g., satellite imagery) of data to be collected as well as the size and may be available to support simplified modeling experience of staff resources. efforts. As the basin planning process advances, measurable project characteristics help guide future When selecting a model, it is important to data collection efforts. Throughout this iterative understanding the trade-offs between model process, two considerations are critical: (i) selection complexity and data requirements. It is easy to list of a model that fits both the problem at hand and the many factors to consider in the model, with a the data available, and (ii) judicious interpretation tendency toward desiring increased complexity. of the modeling results, particularly in view of It must be noted, however, that any addition the uncertainty surrounding sparse data and the of complexity requires additional data for its capabilities of a model, to avoid making decisions successful application. The simplest model that that the model cannot support. adequately addresses management objectives is the most desirable, as it is more likely that the Data can be scarce in developed countries as well, data requirements of simpler models can be although scarcity is a relative term. Examples of readily satisfied. data-scarce and data-rich modeling efforts are presented in boxes 1 and 2. Model development Once a specific model framework has been selected, the next step is to develop the model application that best describes site-specific conditions. Model development can be divided into two separate steps: (i) initial specification of model inputs, and (ii) model calibration. Specification of model inputs Initial specification of model inputs consists of compiling all available data that can serve as model inputs. This compilation process provides an opportunity to verify that sufficient data exist to apply the selected model. While there is no universal criterion for the minimum quantity of data needed for a successful model application, it is best to focus on model input values that involve direct measurement rather than assumption. If a significant fraction of inputs must be assumed, it is advisable to either delay the modeling until data gaps can be filled or select a simpler model with fewer data requirements. This evolution 7 From Waste to Resource Box 1 A data-scarce model used for San Luis • Discharge monitoring reports for the San Luis Obispo Creek, California, United States Obispo WWTP Site background • Maps defining the soils, topography, and land The San Luis Obispo Creek basin is located in west- use throughout the basin central California, covering approximately 215 square kilometers. The main stem of the creek is 29 miles Modeling approach long and it flows through the City of San Luis Obispo. Given the lack of data to support a rigorous The basin is a rapidly growing area and the creek modeling effort, simple screening-level models provides a spawning habitat for trout. were applied to define the relative contribution of different source categories to the stream impairment. The model selected was a linkage of the Universal Soil Loss Equation (USLE) (for rural areas) and general loading functions (for urban areas). USLE is an empirical equation predicting soil loss by sheet and rill erosion (Wischmeier and Smith 1978) from rainfall, soil erodibility, topography, and land use. The United States Environmental Protection Agency (USEPA 1985) provides a general urban loading function that defines pollutant loads from urban areas as a function of population density and annual precipitation. Point source loads were calculated from measurements of flow and Source: Coastal San Luis Resource Conservation District effluent concentrations available from the WWTP. Water quality problems The results indicated that 96 percent of the The creek was impaired by sediments and phosphorus load was coming from the WWTP. nutrients. Sediment was destroying trout Conversely, 99 percent of the sediment was spawning areas within the creek, as well as coming primarily from agricultural lands. impairing other beneficial uses. Nutrients were a problem because they caused frequent algal Management approach blooms. Both point (a municipal wastewater An adaptive management approach was selected treatment plant [WWTP]) and nonpoint for this basin. While the results of the modeling (agricultural and urban runoff) sources were were very uncertain due to the lack of data, they suspected to contribute to the problems, were sufficiently robust to identify that nutrient although specific contributions from these controls were required at the WWTP to prevent sources were unknown. algal blooms, and that nonpoint source controls on agricultural lands were required to reduce Available data sediment loads. This allowed the management A limited data set was available describing water agency to collect more targeted data to define quality and pollutant loading sources: the level of nutrient removal required at the WWTP, and to educate local farmers about • Twenty-one instream measurements of total practices to prevent soil erosion. phosphorus concentration • Six instream measurements of suspended solids Source: LimnoTech, 2018 8 From Waste to Resource Box 2 A data-rich model used for Sanitation Water quality, biological habitat, and physical District No. 1 (SD1), Kentucky, United States monitoring programs were developed and implemented over several years to fill data gaps. Site background Next, appropriate basin loading and receiving SD1 is responsible for the collection and treatment water model frameworks were selected to of wastewater from 100,000 people living in 33 simulate the relationship between pollution communities in a service area of nearly 480 square controls and resulting water quality. A geographic kilometers (km2) that drains into the Ohio River. information system (GIS)-based basin assessment Three wastewater treatment plants are in SD1’s model (Watershed Assessment Tool, WAT) was service area. SD1 also manages stormwater from an developed to assess the potential pollutant area of 560 km2, which enters combined sewers in contribution from various source types (e.g., older parts of the region and separate storm sewers stormwater runoff, combined sewer overflows, in newer areas. etc.). The Hydrological Simulation Program— FORTRAN (HSPF) was selected as the primary Modeled bacterial loads basin loading model and receiving water quality model in small tributaries. The Storm Water Management Model (SWMM) was applied to the portion of the basin serviced by sewer networks. The linked hydrodynamic/water quality models, Environmental Fluid Dynamics Code (EFDC)/RCA Model, were used to simulate receiving water quality in the Ohio River and its key tributaries. Management approach The models were applied as part of an integrated strategy to evaluate a range of options under Source: LimnoTech, 2018. consideration. The basin planning process identified a range of potential pollution controls, Water quality problems covering both the basin (e.g., green infrastructure Regulatory agencies required SD1 to develop for stormwater management) and conventional long-term, capital improvement programs to infrastructure (e.g., sewer system improvements). control combined sewer overflows and discharges The models were used to define the costs and from separate storm sewer systems, and eliminate resulting water quality associated with various sanitary sewer overflows. These requirements combinations of controls. The results of this focus solely on the control of point source assessment were used to prioritize which controls discharges and cost hundreds of millions or even to implement. billions of dollars to implement. Available data Model results demonstrated that an integrated SD1 has a considerable amount of data including solution combining basin controls with long-term records of flow, water quality, biology infrastructure improvements provided more water and aquatic habitat, land use, soils, cadastral quality benefits at a lower cost than a traditional information, and maps of existing infrastructure, solution based solely on infrastructure controls. and detailed geographic data available in This approach was used to guide expenditures so geographic information systems. that money would be spent first on controls that resulted in cost-effective improvements. It also allowed SD1 to gain stakeholder and regulatory Modeling approach agency support for basin planning. The available data characterizing the basin and receiving waters were collected and evaluated. Source: LimnoTech, 2018 9 From Waste to Resource Model calibration Uncertainty analysis Models of environmental systems, being Before a model is used to assess basinwide simplified descriptions of the real world, management alternatives, it should first be inherently contain some degree of uncertainty confirmed that model predictions are consistent in their predictions. In many applications, the with real-world observations. This is achieved uncertainty is quite large. It is important to through what is called model calibration, which explicitly consider this uncertainty to understand involves adjusting model coefficients until the potential risks associated with making the model is able to reproduce site-specific decisions based upon incorrect model results. observations. The process of adjusting the model coefficients to calibrate to data is an art with Rigorous uncertainty analyses can be difficult scientific foundations. A few considerations that to perform on environmental models, but should be kept in mind are as follows: simpler approaches can be readily applied. The simplest approach is termed sensitivity analysis, • Define as many coefficients independently as where the modeler uses professional judgment possible, and try to leave these fixed while other to specify the uncertainty in key model input coefficients are adjusted. parameters, then conducts separate simulations using the upper and lower bound of the • Always a priori establish an expected range parameter values. The range in results between for each coefficient that is to be adjusted in these bounds provides an estimate of model calibration. The range should be based on prediction uncertainty in response to parameter scientific experience from laboratory and other uncertainty. field studies. Avoid deviating from the range. • To the extent possible, justify coefficient A slightly more rigorous method (from Beck 2001) adjustments based on logical scientific consists of: observations or theory, qualitative if not quantitative. 1. Dividing the calibration data set into two • Avoid arbitrary adjustment of coefficients independent periods; spatially and/or temporally solely to improve the 2. Calibrating the model to one of the subsets of model performance without logical justification. the overall data set; This would be merely an attempt to fit output to desired values, not calibration. 3. Testing the calibrated model against the second subset of data; and Matching the model to real-life data requires a multilevel approach. Global quantities such 4. Using the resulting error as a measure of as seasonal water volumes and total pollutant uncertainty. loads should be the first values to be attempted 5. As discussed below, the presence of this to be matched. Once model parameters are uncertainty does not necessarily render model adjusted in this level, matching volumes and results useless. loads for individual events (e.g., a rainstorm) can be attempted. The next level to be matched is peak flows and concentrations for specific Model application occurrences. The calibration effort hinges on an in-depth knowledge of the validity range for model The final activity in the modeling process is parameters. The level of success in calibrating to apply the calibrated model to evaluate the model provides a measure of accuracy and the environmental response to management uncertainty. alternatives. This task consists of four steps: 10 From Waste to Resource 1. Formulation of potential solution alternatives Modeling of alternatives 2. Modeling of alternatives Once specific pollution control alternatives have 3. Decision process to select preferred alternative been identified, the next step is to apply the 4. Detailed simulation of preferred alternative models to simulate the water quality outcomes of the alternatives under consideration. This step first In the past, this was a linear process developed by requires specifying the background environmental a relatively small group of stakeholders supported conditions (e.g., precipitation, river flow) to be by modelers. As discussed below, advances in considered by the model, as the types and severity computing technology now allow for an interactive of water quality impacts vary substantially over process in which stakeholders contribute to the different environmental conditions. For example, identification of solutions, and modelers can quickly water quality problems caused by runoff occur examine and display the impact of those ideas. during wet weather; impacts from continuous sources such as poorly performing WWTPs are Formulation of potential solution alternatives worst during dry weather and low stream flows. This step consists of identifying the specific When conducting scenario evaluations, it is pollution control alternatives to be modeled. It important that the model inputs be selected begins by identifying the entire range of pollution to represent a realistic range of environmental control alternatives under consideration. This list conditions, i.e., one that includes the full range should consider wastewater treatment and other of conditions expected to cause water quality measures that control sources of pollution in the problems. These conditions should be selected basin. Wastewater treatment alternatives should to represent reasonable worst-case conditions consider the number of facilities as well as the level (e.g., large rains, droughts), but not conditions so of treatment at each facility. Selection of potential severe that there is an extremely low probability control alternatives for nonpoint sources should be of them occurring. The selection of overly critical guided by model results, with alternatives being conditions can result in unnecessarily strict focused on those land uses shown by the model to requirements—for example, developing controls be primary contributors of pollutant loads. to prevent against environmental conditions that never occur. Conversely, selecting too limited a set Once a range of alternatives has been identified, of environmental conditions can lead to controls it is necessary to preliminarily dimension the that will not meet water quality objectives during required facilities, estimate their cost, evaluate all times of the year. their pollutant removal efficiency, and assess the social and political feasibility of each alternative. As mentioned above, current models and their Data on the cost and effluent quality associated visualization capabilities enable a participatory with various wastewater treatment alternatives process in which a wide spectrum of stakeholders are widely available, while cost and pollutant can collaborate in the development of alternatives. removal efficiency for various source controls have Today’s computational capabilities facilitate been published in the scientific literature. The communication of benefits and impacts using original list of alternatives under consideration visualization tools ranging from simple plots can be narrowed down for modeling purposes to three-dimensional graphics to fly-through by removing those that are determined to be animations. Depending on the complexity and unacceptably expensive, inefficient, or unfeasible extent of the model, results could be available for other reasons. interactively so that stakeholders can visualize what the different alternatives do and begin to identify the most advantageous ones, which facilitates consensus building. 11 From Waste to Resource Box 3 Modeling in real time District No. 1 (SD1), University’s Decision Theater Network (https:// Kentucky, United States dt.asu.edu/) in the United States. These facilities visualize modeling results in real time. Visual To most stakeholders, models are a “black box” representations enable stakeholders from diverse that they are told to trust. There is also a time lag backgrounds and with differing levels of technical between proposing alternatives and receiving expertise to collaborate and feel confident about the information from the models to gauge the impact results of the modeling process. of interventions. Today’s computing technology alleviates these shortcomings of the modeling process by bringing together models, data, visualization tools, and human expertise and making them available to stakeholders and decision makers. An example is the Deltares iD-Lab (Interactive Data Research Laboratory; https://www.deltares.nl/ en/facilities/idlab-integrated-service-facility/) in the Netherlands. Another example is Arizona State Photo: Arizona State University Decision process to select preferred alternative configuration of the solution, for example, if some Chapter 2 of the main document From Waste of the components are found to be unnecessary or to Resource. Shifting Paradigms for Smarter unfeasible. Wastewater Interventions in Latin America and the Caribbean describes the process of how to Once the individual projects have been refined, arrive at a preferred management alternative. the calibrated model is modified accordingly to represent the updated configuration and Detailed modeling of the preferred alternative performance of the individual projects that Once the preferred alternative is chosen, it is comprise the selected alternative. This step verifies possible to define additional details beyond that the proposed solution still meets the goals for the conceptual sizing that was used in the the basin. It is possible that this refined analysis will selection process. The preferred solution often result in a more compact, less expensive facility. Of consists of a number of projects, for instance, a course, the opposite may also occur; therefore, it is WWTP and source controls in various locations. worthwhile revisiting the selection criteria to verify Sewer expansion projects may be used to bring that the solution is still acceptable. wastewater to the facility. Without going into a final design, this step involves the provision of This step yields the “official” model of the basin, additional detail to refine the overall dimensions of at least for the scope initially envisioned. This each of the components so as to better evaluate model would be the tool of choice to evaluate all their performance and cost. It is possible that other future projects as it represents a benchmark this additional level of analysis will change the derived with the best available data. 12 From Waste to Resource Box 4 A model of a model users understand the impacts through a suite of visualization utilities. Some models are massive due to their extent, scale, and the level of detail they represent. Such is the case of the Chesapeake Bay water quality model that simulates loads and transport of nitrogen, phosphorus, and sediment over an area of 166,000 square kilometers containing 150 major rivers and more than 100,000 tributaries. This massive model incorporates information about land use, fertilizer applications, wastewater plant discharges, septic systems, air deposition, farm animal populations, and meteorology, among many other basin variables. Even with ample computing resources, the time to obtain results of a new run can be in the order of days to weeks. To evaluate solutions without running the entire basin model, the Chesapeake Assessment Scenario Tool (CAST) was developed. CAST is a “model of the model.” Output from the Chesapeake Bay model was preprocessed and staged in CAST so that users can rapidly evaluate corrective actions at various scales. CAST is a planning tool designed to compare various scenarios and to help Source: Limnotech, 2018 The role of modeling in decision making information on pollutant concentrations that result in the basin approach: A summary from the pollutant loads, and is commonly referred to as the Receiving Water Quality Model. Models are mathematical representations of natural processes and thus are approximations of reality. They Models need to be selected to fit the problem at are an integral part of the basin management process, hand so that they will properly address the stated in that they provide a quantitative link between objectives. The selection also depends on site- pollutant sources and receiving water quality. specific characteristics and resource constraints. Models are only as good as the data available. The Models serve as a tool to get information on simplest model that adequately addresses the various water quality control actions of interest objectives is the most desirable, as it is more likely and provide output on the resulting water quality. that simpler models’ data requirements can be As such, models allow the basin plan to define the readily satisfied. Models need to be calibrated with optimal location, timing, and phasing of wastewater real data. treatment infrastructure, as well as controls for other sources. Also, models can assist in defining There are many levels of modeling expertise, all strategies that will protect public health and provide of which have a role in developing and running environmental benefits with the resources available. basin models. It is essential that the corresponding knowledge and abilities be clearly understood, There are two main types of models useful for especially those related to the model’s limitations. wastewater planning in a basin context. The first The adaptive management feature of the basin type is called the Basin Loading Model and provides approach allows for incremental improvements in information on pollutant loads generated via runoff the accuracy of models as better site-specific data from the land surface. The second type provides become available. 13 From Waste to Resource References Beck, M.B., 1987. Water quality modelling: a review of the analysis of uncertainty. Water Resources Research, 23 (8), 1393-1442. ———. 2001. Model evaluation and performance. In: El-Shaarawi, A.H. and Piegorsch, W.W. eds. Encyclopedia of environmetrics. Vol. 3. Wiley, New York, 1275-1279. DePinto, J.V., P.L. Freedman, D.M. Dilks, and W.M.Larson. 2004. Models quantify the total Maximum daily load process. J. Environmental Engineering 130:703–713. LimnoTech, 2018. From Waste to Resource: A Watershed- Based Framework for Planning and Investing in Wastewater Treatment. Unpublished technical background paper prepared for the World Bank. Shoemaker, L., Dai, T. &. Koenig, J. 2005. TMDL Model Evaluation and Research Needs. EPA/600/R-05/149. U.S. EPA, ORD, National Risk Management Research Laboratory, Cincinnati, OH. Stone Environmental & LimnoTech. 2014. Comprehensive Eutrophication Model Development and Management Scenario Evaluation/Prespa Lake, Macedonia Phase I: Model Selection and Data Evaluation. Prepared for: United Nations Development Programme (UNDP). United States Environmental Protection Agency. (1985). Water Quality Assessment: A Screening Procedure for Toxic and Conventional Pollutants in Surface and Ground Water—Part I (Revised–1985). EPA/600/6-85/002a. Environmental Research Laboratory, Athens GA. ———. 1997. Compendium of Tools for Watershed Assessment and TMDL Development. EPA841-B-97-006. U.S. Environmental Protection Agency, Office of Water, Washington, DC. ———. 2008. Handbook for Developing Watershed Plans to Restore and Protect Our Waters. EPA- 841-B-08-002. Washington, DC. Wischmeier, W. H., and Smith, D.D. 1978. Predicting rainfall erosion losses— A guide to conservation planning. Agriculture Handbook No. 537. U.S. Department of Agriculture. 14