Approach Paper Implementing the World Bank’s Environmental and Social Framework: A Developmental Evaluation November 6, 2024 1. Background and Context 1.1 The World Bank adopted a new Environmental and Social Framework (ESF) in 2016 that aims to protect people and the environment from potential adverse project impacts while promoting sustainable development. The Board of Executive Directors approved the ESF on August 4, 2016. It consists of a Vision for Sustainable Development; 10 Environmental and Social Standards (ESSs), which set out the requirements that apply to borrowers; an Environmental and Social Policy for Investment Project Financing (IPF), which sets out the requirements that apply to the World Bank; an Environmental and Social Directive/Procedure for IPF; and a Directive on Addressing Risks and Impacts on Disadvantaged or Vulnerable Individuals or Groups (World Bank 2017, 2021, 2023). It applies to all IPF projects initiated on or after October 1, 2018. The World Bank has also developed a Good Practice Note to assist task teams in identifying risks of sexual exploitation and abuse and sexual harassment (SEA/SH) and to advise borrowers on how to best manage such risks. Although the ESF itself does not explicitly mention SEA/SH, various ESSs align with the recommendations of this Good Practice Note for addressing SEA/SH. The ESF is supplemented by many other guidance notes, templates, and tools. 1 At the time of ESF adoption, the Board and management explicitly recognized that the ESF has the potential to create better long-term development outcomes in IPF and that evidence of improved development outcomes will be collected over time by the World Bank (World Bank 2016). 1 Figure 1.1. Differentiated Responsibilities of the Environmental and Social Framework Source: Independent Evaluation Group, based on World Bank Environmental and Social Framework Training. Box 1.1. The Environmental and Social Framework Environmental and Social Standards (ESS) ESS1: Assessment and Management of Environmental and Social Risks and Impacts ESS2: Labor and Working Conditions ESS3: Resource Efficiency and Pollution Prevention and Management ESS4: Community Health and Safety ESS5: Land Acquisition, Restrictions on Land Use and Involuntary Resettlement ESS6: Biodiversity Conservation and Sustainable Management of Living Natural Resources ESS7: Indigenous Peoples/Sub-Saharan African Historically Underserved Traditional Local Communities ESS8: Cultural Heritage ESS9: Financial Intermediaries ESS10: Stakeholder Engagement and Information Disclosure Environmental and Social Policy, Directives, and Procedures Environmental and Social Policy for Investment Project Financing; Environmental and Social Directive/Procedure for Investment Project Financing; Directive on Addressing Risks and Impacts on Disadvantaged or Vulnerable Individuals or Groups 2 Good Practice Notes for Bank Staff Addressing Sexual Exploitation and Abuse and Sexual Harassment (SEA/SH) in Human Development Operations; Addressing SEA/SH in IPF involving Major Civil Works; Animal Health and Related Risks; Assessing and Managing the Risks and Impacts of the Use of Security Personnel; Assessing and Managing the Risks of Adverse Impacts on Communities from Project- Related Labor Influx; Dam Safety; Gender; Non-Discrimination and Disability; Non-Discrimination: Sexual Orientation and Gender Identity; Road Safety; Third-Party Monitoring; Water Use Source: World Bank Group. Note: IPF = investment project financing; SEA/SH = sexual exploitation and abuse and sexual harassment. 1.2 ESF achievement is guided by five main objectives laid out in a 2016 Board policy paper (World Bank 2016). The first objective focuses on the need to address a wider range of environmental and social (E&S) risks. The scope of social issues explicitly addressed has been expanded, for example, to include labor and working conditions; worker and community health and safety; free, prior, and informed consent; and nondiscrimination provisions. The ESF highlights additional environmental issues, such as addressing climate change and promoting climate resilience, the sustainable management of living natural resources, and water management. The second objective is to strengthen national ESF systems in borrowing countries. The ESF enables a borrower to propose using relevant parts of its ESF for the assessment, development, and implementation of a project when this is likely to address the risks and impacts of the project and enable it to achieve objectives that are materially consistent with the ESSs. The third objective is to increase the effectiveness, efficiency, and timeliness of E&S risk management. The ESF was developed to use a risk-based and proportionate approach designed to apply increased oversight and resources to complex projects and to allow for greater flexibility or responsiveness to changes in project circumstances through adaptive risk management and stakeholder engagement throughout the project cycle; before the ESF, much of the attention was focused upstream of project approval. The fourth objective focuses on policy harmonization, coherence, and alignment with development partners. The fifth objective is to enhance transparency through stakeholder engagement and information disclosure. 1.3 The ESF reform is broadly aligned with recommendations outlined in Safeguards and Sustainability Policies in a Changing World: An Independent Evaluation of World Bank Group Experience (World Bank 2010), particularly the need to address social issues and strengthen national E&S systems. The Independent Evaluation Group (IEG) recommended, and management acted to ensure, adequate coverage of social effects to integrate community and gender impacts and assess labor and working conditions, as well as health, safety, and security issues not covered in the former safeguard policies. IEG also recommended, and management acted to put in place, a central grievance redress and conflict resolution mechanism (in Operations Policy and Country Services [OPCS]). In addition, the ESF placed IEG’s recommendation of enhancing client 3 capacity, responsibility, and ownership at the core of the reform. IEG recommended that the World Bank invest in upstream analytic work, technical assistance, and lending to strengthen country and sector institutions and client capacities while requiring regular reporting by the borrower on implementation and outcomes of ESF activities in World Bank–supported projects. To ensure effective oversight, IEG recommended that the World Bank revise its guidelines, instruments, and incentives to strengthen supervision arrangements and its ESF monitoring, evaluation, and completion reporting. The World Bank shifted budget responsibility, launched an accreditation program, and has made considerable progress on staff capacity since 2016, while noting that this requires continued efforts. Although a system was put in place to track borrower E&S performance, less progress has been made on including performance indicators on E&S outcomes in project results frameworks, as recommended by IEG (according to the Management Action Record). Areas that are lagging, per IEG’s recommendations, include the need to continue strengthening monitoring and reporting, improving the quality of grievance redress, and harmonizing ESF approaches more meaningfully with the International Finance Corporation (IFC) and the Multilateral Investment Guarantee Agency (MIGA), including on criteria for project risk classification and by reorienting more resources toward risk identification management and oversight during the project cycle. 1.4 The adoption of the ESF included an agreement to conduct an evaluation of the ESF after the fifth year of implementation. Modernizing the safeguard policies was one of the most complex multiyear policy reforms undertaken by the World Bank. The decision to adopt the ESF was based on the realization that while much consensus had been achieved, remaining policy issues could be examined through evaluation, beginning with an evaluation after the fifth year, as requested by the Board. A timeline of major ESF events is included in figure 2.1. The timeline highlights the adoption of the ESF in 2016, a two-year preparatory phase that took place between 2016 and 2018, and implementation beginning in October 2018. 2 In its 2020 update to the Board, the World Bank reported making “progress on all five objectives set out in the 2016 Board paper [, but] substantial works remain[ed] in order that the ESF delivers consistently intended improvements in E&S risk management at the project level” (World Bank 2020b, 1). Subsequent progress, especially on national capacity building, was interrupted by the COVID-19 pandemic, during which the World Bank also allowed COVID-19 ESF waivers. During this time, the Group Internal Audit (GIA) conducted implementation assessments and, as of June 2023, the World Bank also began to address efficiency issues by putting in place a “streamlining” that is ongoing. 4 Figure 1.2. Key Environmental and Social Framework Milestones Source: Independent Evaluation Group. Note: ESF = Environmental and Social Framework; GIA = Group Internal Audit; IEG = Independent Evaluation Group; IPF = investment project financing. 5 2. Purpose, Objectives, and Audience 2.1 The purpose of this evaluation is to assess how relevantly and effectively the World Bank is using the ESF to protect people and the environment from potential adverse project impacts while promoting sustainable development. The evaluation will do this by assessing progress against the ESF objectives laid out in its 2016 policy paper—that is, adaptive management, client capacity building, results, and donor harmonization. This evaluation is being undertaken at the Board’s request. 2.2 This evaluation is a developmental evaluation because it seeks to engage management on emerging findings during the evaluation process to inform ongoing ESF management decisions. The ESF is very much a system in flux because it includes, among other ongoing efforts to improve the system, efforts to streamline decision- making and the application of tools, supported by internal audit assessments; staff guidance updates; updates to the data management system and its use;. As such, and as discussed in chapter 6, we intend to bring emerging findings periodically to management to enable reflection and use and to deepen the evaluation analyses throughout the process (in ways that will also lead to a richer, more contextualized final report). 2.3 The evaluation’s primary audience is the Bank Group Board of Executive Directors, Bank Group management, and all staff working on investment project finance. The primary institutional counterpart is the Operations Standards, Environment, and Social Department within OPCS. This department supports operations, ESF learning and training, knowledge management, and reporting; interfaces with the Inspection Panel; and manages the Grievance Redress Service. Other key institutional counterparts include Regional Sustainable Development directors; Regional Directors of Strategy and Operations; Committee on Development Effectiveness managers; Regional Safeguard advisors; Environment, Natural Resources, and Blue Economy Global Practice managers; the Social Sustainability and Inclusion Practice Board; Social Sustainability and Inclusion Global Practice Regional Safeguard coordinators; Environment, Natural Resources, and Blue Economy Global Practice Regional Safeguard coordinators; and the Environmental and International Law Practice Group within the Legal Vice Presidency, which provides support to internal and external World Bank clients to promote environmentally and socially sustainable development in accordance with the World Bank policies and relevant international laws. The evaluation findings will also be relevant to country clients, including government officials and agencies responsible for ESF implementation; other multilateral and bilateral agencies that have adopted or updated their ESF standards; nongovernmental organizations; civil society; and citizens. 6 3. Evaluation Scope, Theory, and Questions Evaluation Scope 3.1 The evaluation’s scoping decisions pertain to institutional coverage, timing, instruments, and portfolio. The evaluation focuses on the World Bank only. It covers IFC and MIGA in its benchmarking exercise, which includes questions on policy harmonization, but it does not assess the effectiveness of IFC and MIGA performance standards and does not fully address questions about policy coherence across the Bank Group. Separate evaluative undertakings would be needed to cover these important Bank Group ESF policy and performance areas. The evaluation covers the period from 2016 (when the policy was put forth) to December 2024 but focuses on the period after ESF implementation, beginning in October 2018. Although the evaluation covers all IPF, it also includes relevant technical assistance, training, analytic and advisory work, and policy lending that supports ESF capacity building. The portfolio scope includes all IPF projects with a project concept note (PCN) approved on or after October 1, 2018, that use the ESF, with a portfolio cutoff date of approval by June 2025. As of July 8, the portfolio comprises 1,546 IPF projects under the ESF, of which 593 projects have more than three years of implementation experience, and 283 IPF projects are anticipated to close by December 2024. The full portfolio, including relevant analytic work and technical assistance that supports ESF implementation, will be determined early in the evaluation process through several methods, including by engaging relevant World Bank staff. 3.2 The evaluation scope will consider the effects of the Streamlining Action Steps while noting that these action steps are one of many means of achieving policy objectives; the specific validation of the Streamlining Action Steps is being undertaken by the GIA. For example, the evaluation covers the issue of adaptive management, which relates to the effectiveness and efficiency of ESF implementation and includes the application of the principles of proportionality and flexibility. As planned, in FY 2025, the GIA will review (i) the ESF implementation during project supervision, (ii) ESF simplification initiatives and the changes they are bringing about, and (iii) behavioral aspects of the ESF implementation (shift from rule-based to principles- and risk-based approach). GIA will also conduct an E&S Incident Response Toolkit Insight Note. In addition, it will conduct an implementation audit of the Environmental and Social Grievance Redress Service expected to start in FY25 (second or third quarter). Theory of Change 3.3 The ESF is guided by a theory of change that is anchored in its objectives and that, if effectively implemented, will achieve compliance with ESF standards and enhanced development outcomes (figure 3.1). The ESF is designed to identify and 7 address negative adverse project impacts while promoting sustainable development. The first key tenet of the ESF is adaptive risk management. This tenet includes attention to the principle of proportionality, which ensures that E&S classifications and associated management measures are accurately assigned in line with risk levels and impacts so that higher-risk projects are allotted higher amounts of oversight and resources compared with lower-risk projects and provides for flexibility or responsiveness to changes in project circumstances (which may require reclassification of risks, and adjustments of resources and management actions). Stakeholder engagement helps identify and address risks as they emerge. The World Bank is responsible for ensuring project compliance with the ESF, while the borrower oversees its effective monitoring and implementation in line with agreed-on environment and social commitment and management plans. Adaptive management is an important way to achieve compliance in the face of changing circumstances. The second key tenet of the ESF is the World Bank’s commitment to building borrower ESF capacity and using borrower systems when they can mitigate project risks at a level that is equal to or greater than that of the World Bank’s ESF. The World Bank helps build national ESF system capacity by conducting capacity assessments and project risk analyses and by providing training and technical assistance. The third key tenet of the ESF is the commitment to support ESF harmonization among donor partners in ways that are mutually reinforcing. At the portfolio level, the World Bank has also emphasized the need to harmonize ESF approaches in cofinanced projects. Although the primary purpose of the ESF is to protect people and the environment from the potential adverse impacts of IPFs, the ESF stipulates that the World Bank’s vision goes beyond “do no harm” to maximizing development gains. According to the ESF, “where the Borrower’s environmental and social assessment has identified potential development opportunities associated with the project, the [World] Bank will discuss with the Borrower the feasibility of including these opportunities in the project. Where appropriate, such opportunities may be utilized to promote further development” (World Bank 2017, 2). 8 Figure 3.1. Environmental and Social Framework Theory of Change Source: Independent Evaluation Group. Note: E&S = environmental and social; ESF = Environmental and Social Framework. Evaluation Questions 3.4 The evaluation asks the overarching question: To what extent has the ESF progressed to achieve its objectives of protecting people and the environment from potential adverse project impacts while promoting sustainable development? To answer this question, we included three specific subquestions: 3.5 Evaluation question 1. To what extent has the World Bank implemented the key principles of the ESF since 2018? What challenges remain? a. How well has the World Bank applied the principles of adaptive risk management in its rollout of the ESF, especially regarding the principles of proportionality and flexibility? b. How well are borrowers performing in line with ESF requirements, and what are the determinants of these performance ratings? c. How well is the World Bank contributing to donor ESF harmonization and common approaches? 3.6 Evaluation question 2. How well has the World Bank been helping borrowers to build their national systems for managing E&S risks? 9 a. What capacity-building efforts have been deployed by the World Bank to date? b. How well has the World Bank helped borrowers build ESF capacity and strengthen borrower frameworks, especially for new ESF policy areas and standards? Based on this evidence, how scalable is the ESF approach regarding the use of borrower frameworks? 3.7 Evaluation question 3. To what extent is there evidence that ESF application has contributed to enhanced development outcomes? 4. Evaluation Design and Methods 4.1 The evaluation answers the overarching evaluation question by using a triangulated set of methods aligned with each of the three subquestions. The methods are also designed to be mutually reinforcing across evaluation questions. The building blocks of the evaluation are laid out in figure 4.1, and the methods are further explained in this chapter. 10 Figure 4.1. Evaluation Design Source: Independent Evaluation Group. Note: E&S = environmental and social; ESF = Environmental and Social Framework; IFI = international financial institution; SEA/SH = sexual exploitation and abuse and sexual harassment. Methods 4.2 To answer all evaluation questions, we will first need to build a usable ESF database. ESF data are only partly accessible. An Environmental and Social Management System (ESMS) was developed by the World Bank to facilitate quality and compliance reviews and to manage reporting of E&S issues and risks in IPF. It records the World Bank’s due diligence for project-related E&S risks and impacts. The ESMS was also developed to capture projects’ E&S management performance within a single system, which assists with portfolio monitoring. However, for the purposes of this evaluation, the data are not in a ready-to-use format. To overcome this data challenge, the evaluation team will engage OPCS and Information and Technology Solutions to source and transform the underlying data so that they can be used to answer the evaluation questions. IEG will validate the data and database with OPCS and Information and Technology Solutions. We view this activity both as integral to our evaluation process and as a public good that the World Bank can use as a model for future data collection, management, and decision-making. OPCS is currently developing ESMS 2.0. Through 11 this IEG data exercise, we aim to enhance the use of ESF data within the World Bank for decision-making. 4.3 The evaluation will also develop and administer a staff survey instrument covering all evaluation questions. The survey will capture lessons learned from experience across different stakeholder groups involved in ESF implementation, including ESF specialists, task team leaders, and management, among other constituencies. The nature of the survey (coverage, technology choice, number, and style of questions) will be decided as part of the survey design in consultation with survey design experts. It will be administered early in the evaluation period so that its data can be used as probing questions in subsequent key stakeholder interviews, ESF specialist focus groups, and country workshop designs. Evaluation Question 1: ESF Implementation 4.4 The evaluation will answer the question “How well has the World Bank applied the principles of adaptive risk management in its rollout of the ESF, especially regarding the principles of proportionality and flexibility?” through data collection, pattern identification and hypotheses testing, and semistructured interviews. Data will be collected from the ESMS, project documentation (standard reports and the operational portal), staff surveys, and key interviews. We will focus our pattern identification on several tenets of adaptive management in the ESF system—namely, (i) risk identification, classification, and reclassification; 1 (ii) allocation of staff and resources; and (iii) monitoring, grievance redress, reporting, and other forms of stakeholder engagement. Patterns will be analyzed across time, country income group and typology (including fragility, conflict, and violence; and Small Island Developing States), and level of project risk, among other factors. The main hypothesis that will be tested is that adaptive management practices should be made evident at key decision points across the project cycle and that ESF-related considerations should be reflected in changes to ESF management (risk classification, resources, monitoring, and reporting). Findings emerging from the pattern analyses and hypotheses testing will be discussed with key ESF specialists to broaden the evaluation’s understanding of the factors (behavioral, institutional) that lie beneath these phenomena. 4.5 The evaluation will address borrower E&S performance issues at the system and portfolio level. Specifically, we ask, “How well are Borrowers performing in line with ESF requirements and what are the determinants of these performance ratings?” We answer this question by taking stock of borrower E&S performance as rated by E&S specialists across the portfolio and by assessing how these ratings are determined within the ESF system with a view toward the quality, transparency, and integrity of the underlying data. At the portfolio level, we will also analyze the presence and evenness 12 of evidence for similar projects. First, the evaluation will develop a database of these performance ratings for projects using ESF, as reported throughout the project cycle and at project close. Second, the evaluation will determine and make evident the methods and criteria being used by the World Bank to rate borrower E&S performance. In making evident the criteria, the evaluation will consider how borrower E&S performance data are collected and analyzed, and how performance ratings decisions are determined. Third, the evaluation will collect and make evident ESF-related qualitative data in projects that provide explanatory information about borrower E&S performance. Fourth, the evaluation will examine patterns among projects with similar ESF risks to determine the evenness of information provided to support ratings decisions—that is, the quality and consistency of the evidence used to determine borrower E&S performance ratings. 4.6 The evaluation will answer the question “How well is the World Bank contributing to donor ESF harmonization and common approaches?” through a focused benchmarking exercise, interviews, and review of cofinanced project experiences. First, the evaluation will conduct a focused benchmarking exercise covering specific harmonization questions for the Asian Development Bank, the African Development Bank, the Inter-American Development Bank, the European Bank for Reconstruction and Development, IFC, and MIGA. The benchmarking exercise will focus on those policies, standards, and implementation procedures that have either been similarly adopted or that differ from those of the World Bank’s to determine the extent to which ESF systems are harmonizing and to learn about experiences that are different. Except for IFC and MIGA, all other agencies modeled their adoption of the updated ESF system on that of the World Bank; IFC and MIGA are planning an update in fiscal year 2025. Lessons from the World Bank’s early experience will also help inform IFC and MIGA updates. Benchmarking guidance and tools will be developed as part of the evaluation process. 4.7 Through interviews, we will capture lessons about the World Bank’s convening role in relation to ESF harmonization. Since the ESF’s inception, the World Bank has reached out to other international financial institutions, particularly multilateral development banks and bilateral agencies, to support the convergence of ESSs. Outreach has been conducted through the Multilateral Financial Institutions Working Group on ESSs, at the World Bank Annual and Spring Meetings, and through a number of specially convened meetings. The evaluation will conduct a series of semistructured interviews to gather perceptions and quantifiable evidence of World Bank influence. Interviewee triangulation (World Bank management, staff, partner, and client) and fact checking will support the validation of evidence collected. We will also include questions about harmonization and the role of the World Bank in promoting alignment in the client workshops and staff survey. 13 4.8 We will use a portfolio review to identify ESF projects with donor cofinance and interviews to determine the extent to which the World Bank is supporting a common approach. The ESF refers to the concept of a “common approach”—that is, when a project is jointly financed by the World Bank and other multilateral or bilateral agencies, a common approach can be proposed for the assessment and management of E&S risks and impacts if it enables the project to achieve objectives materially consistent with the ESSs. In deciding on a common approach, the World Bank will consider the policies, standards, and implementing procedures of the other agencies. According to the World Bank’s update to the ESF Board, during the ESF’s initial implementation period, there was modest uptake in developing a common approach to risk management at the project level (World Bank 2020b). Over the evaluation period, several agencies updated their ESF. The evaluation will therefore assess the extent to which cofinanced projects using ESF are making efforts to harmonize their ESF approach. This will be done through a review of project documentation and interviews with project teams. Evaluation Question 2: Borrower Frameworks and Capacity Building 4.9 To answer the evaluation subquestions “What capacity building efforts have been deployed by the World Bank to date?” and “How well has the World Bank helped borrowers build ESF capacity and strengthen borrower frameworks, especially for new ESF policy areas and standards?”, the evaluation will first take stock of relevant activities, analyze trends, and derive insights from surveys and semistructured interviews. We will identify ESF-related capacity-building activities, including analytics, training, and technical assistance in both lending and nonlending projects (using World Bank databases and ESF-related communities of practice). Next, we will use portfolio review and analysis to examine the nature of this support and analyze trends (depth, breadth, frequency, and distribution, including across country income groups and typologies such as fragility, conflict, and violence; and Small Island Developing States). To assess World Bank support for borrower frameworks, we will examine whether the ESF has facilitated steps to (i) improve client countries’ legal frameworks and implementation of existing policies and operational procedures linked to E&S management; (ii) build the coordination capacity of core agencies tasked with E&S risk management; and (iii) improve the capacity of E&S professionals in client countries. We will then conduct targeted, semistructured interviews with ESF trainers and a sample of trainees stratified across regions and country typologies. These interviews will triangulate perspectives on training quality, application, and uptake. We will collate, analyze, and quantify responses and validate uptake claims by requesting evidence. The interviews will include probing questions related to country systems development, draw on project data, and country data where relevant. The interviews will also examine capacity development for new ESF policy areas and standards. 14 4.10 To answer the subquestion “Based on this evidence, how scalable is the ESF approach regarding the use of borrower frameworks?” we will conduct project-level case analysis of those projects that use borrower frameworks under the ESF. There are 89 IPF projects in 41 countries that use borrower frameworks either partially (78) or fully (11). For these projects, we will use a structured template designed as part of the evaluation process to examine (i) when borrower systems have been used to date; (ii) what led to the decision to use borrower frameworks; (iii) what changes were put in place to enable the borrower framework to be used, if any; (iv) whether the use of borrower frameworks at the project level is leading to leveraging or spillover effects— that is, whether the borrower is using systems or skills acquired from the ESF process or other donor or country projects; (v) the evidence of capacity on new standards and policy areas; and (vi) to the extent that information is available, how well the ESF is being implemented using the borrower frameworks. Data collection for each case will include project data and documentation, as well as interviews to validate and expand on emerging findings. A consideration of borrower frameworks will also include analysis of how this support varies between country income groups and typologies (for example, fragility, conflict, and violence; and Small Island Developing States). 4.11 We will also conduct country workshops to obtain feedback from borrowers and a triangulated set of local stakeholders within and outside of the World Bank to gain deeper contextual insights. The selected countries will be regional hubs where Regional ESF advisors and teams for multiple ESF projects across countries are located. To ensure the comprehensiveness of responses and promote responder safety, the workshops will include separate sessions for different levels of government (for example, senior officials versus technical staff), other donors and multilateral development partners, and nongovernment stakeholders (civil society, private sector, and so on). In certain cases, we will gather insights from project beneficiaries. Data collected from these workshops will be systematically analyzed using qualitative methods, such as thematic analysis to identify common themes, challenges, and good practices. We will focus closely on obtaining information from borrowers and World Bank staff about capacity-building efforts for new standards and policy areas. Evaluation Question 3: Sustainable Development Outcomes 4.12 According to the ESF, “where the Borrower’s environmental and social assessment has identified potential development opportunities associated with the project, the [World] Bank will discuss with the Borrower the feasibility of including these opportunities in the project. Where appropriate, such opportunities may be utilized to promote further development” (World Bank 2017, 2). 15 4.13 The World Bank has articulated the link between some ESF mechanisms and enhanced development outcomes in its policy papers and ESF framework. The most salient examples are provided here to give more context for answering this evaluation question. For example, ESF requirements on nondiscrimination, if well implemented, can widen the benefit stream for traditionally disadvantaged citizens. In one example noted by the World Bank’s 2020 ESF Implementation Board Update, an education project amended its design to include people with disabilities. Using social assessments and the stakeholder engagement plan, the project included parameters to build disability-responsive school infrastructure and differentiated curriculum and to implement differentiated monitoring aligned with disability standards. The enhanced development outcome relates to both the number of students reached as well as the longer-term impacts associated with educational contributions to poverty reduction for people with disabilities and associated household welfare benefits. In the area of biodiversity, ESF requires that biodiversity offsets are designed and implemented to achieve measurable, additional, and long-term conservation outcomes that can reasonably be expected to result in no net loss and, preferably, a net biodiversity gain (World Bank 2021). 4.14 To answer the question “To what extent is there evidence that ESF application has contributed to enhanced development outcomes?” the evaluation will use a focused literature review and a theory-based approach that incorporates contribution analysis. First, we will conduct a focused literature review on the ways in which ESF mechanisms make plausible contributions to development outcomes. We will then develop nested theories of change for each of these identified mechanisms. This nested approach will allow for the identification of multiple subtheories of change (aligned to specific ESF mechanisms), which will help explain the potential contribution pathways of the ESF mechanisms to broader development outcomes. Using a comparative project analysis, the evaluation will then trace plausible outcome pathways and assess the contribution of the ESF mechanism in line with the nested theory. Semistructured interviews will also be used to complement, triangulate, and validate the pathways and contributions identified in the literature review and comparative project analysis. 4.15 As part of its design, the evaluation will cover both the “doing good” aspects of the ESF and the “do-no-harm” aspects. The do-no-harm aspects will be covered through several sources of information, including the borrower E&S performance analysis, staff interviews, and nongovernmental organization and civil society organization interviews. After gathering this information, we will determine system-level trends related to implementation impediments. 16 Sexual Exploitation and Abuse and Sexual Harassment 4.16 To evaluate the treatment of the cross-cutting risk of SEA/SH, as requested by the Committee on Development Effectiveness, the evaluation will compile an evaluation compendium based on recent IEG gender evaluations and evidence emerging from all evaluation questions. SEA and SH are types of gender-based violence that may be exacerbated by World Bank–financed projects, particularly in countries experiencing fragility, conflict, and violence. SEA/SH risks are covered by several World Bank Good Practice Notes and appear in four ESSs (box 4.1). The compendium will be designed based on the findings from IEG’s 2023 evaluation of World Bank support to address gender inequalities in these countries as well as the Gender Strategy Mid-Term Review and forthcoming gender strategy evaluation. In addition, the approach for each evaluation question will incorporate an analysis of SEA/SH risks based on the compendium. For evaluation question 1, the team will analyze data from the ESMS on ESF risk classifications to determine how SEA/SH risks influence a project’s ESF risk profile as well as shifts in risk classification and mitigation activities across the project cycle. It will also include SEA/SH in the review of borrower E&S performance and the harmonization exercise. For evaluation question 2, the team will assess how the World Bank is building borrower capacity to address SEA/SH risks. Finally, for evaluation question 3, the evaluation will include SEA/SH in the literature review, outcome harvesting, and comparative case design methods. Box 4.1. Mitigating Sexual Exploitation and Abuse and Sexual Harassment Risks through Environmental and Social Framework Instruments Sexual exploitation and abuse (SEA) involves the abuse of power or trust for sexual purposes, including exploitation for monetary, social, or political gain, and includes acts such as rape and sexual assault. In contrast, sexual harassment (SH) encompasses any form of unwanted verbal, nonverbal, or physical conduct of a sexual nature with the purpose or effect of violating the dignity of a person or creating an intimidating, hostile, degrading, humiliating, or offensive environment. Following an Inspection Panel case in Uganda with SEA/SH allegations, the World Bank established a Gender-Based Violence (GBV) Task Force and issued directives, guidance notes, and updated requirements for SEA/SH. These include (i) a 2017 briefing note to the Board detailing recommendations of the GBV Task Force; (ii) an Environmental and Social Framework Directive that screens World Bank–financed projects for the potential for SEA/SH and GBV; (iii) a Good Practice Note on addressing GBV in investment project financing involving major civil works published in 2018; (iv) an updated Good Practice Note on addressing SEA/SH in investment project financing involving major civil works in 2020; (v) a Good Practice Note on addressing SEA/SH in human development operations in 2022; (vi) the 2019–20 revised World Bank procurement requirements including measures to address SEA/SH risks in World Bank– financed projects; and (vii) procurement measures for contractors and subcontractors applied to projects considered high risk for SEA/SH to improve prevention and response efforts. Although the Environmental and Social Framework itself does not explicitly mention SEA/SH, 17 various Environmental and Social Standards (ESSs) align with the recommendations of the Good Practice Notes for addressing SEA/SH, including the following: • ESS1: Assessment and Management of Environmental and Social Risks and Impacts; • ESS2: Labor and Working Conditions; • ESS4: Community Health and Safety; and • ESS10: Stakeholder Engagement and Information Disclosure. SEA/SH risk and mitigation measures are also integrated into the Environmental and Social Framework instruments: • Environmental and Social Assessment: Reviews existing data on violence against women, assesses risks associated to the project, and maps existing GBV services. • Environmental and Social Review Summary: Discusses SEA/SH risks according to relevant standards (ESS1, ESS2, ESS4, and ESS10). Outlines mitigation measures. • Environmental and Social Compliance Plan: Describes the mitigation actions including timelines. • Environmental and Social Management Framework/Plan (ESMF/P) and Contractor’s Environmental and Social Management Plan (C-ESMP): Includes project-specific risks and mitigation measures; for moderate risk projects, an SEA/SH action plan is part of the ESMP and C-ESMP. • Stakeholder Engagement Plan: Includes women and girls in consultations, identifies entry points for grievance redress mechanism procedures. • Labor Management Procedures: Outlines requirements related to codes of conduct and trainings. Although environmental and social policy and directives and ESSs are mandatory for the World Bank and borrowers, Good Practice Notes provide advice only and are not legally binding or mandatory. Good Practice Notes give task teams additional information, tools, and good practice to help meet these requirements and responsibilities. However, the mitigation measures introduced based on the Good Practice Notes are documented in the Environmental and Social Compliance Plan, which forms part of the project’s legal agreement approved by the Board. Sources: World Bank n.d., 2018, 2020a, 2020b, 2022. Evaluability and Data Limitations 4.17 ESF data are dispersed across various data platforms and are not readily usable for the purpose of this evaluation. To overcome this data limitation, the evaluation team will engage OPCS, Information and Technology Solutions, Global Practices, and the Regions to source and transform the underlying data so that they can be used to answer the evaluation questions. IEG will develop a database that will support this evaluation, be a resource for the institution, and inform the development of future ESF data systems. This is one characteristic of the developmental evaluation process. Accurate 18 interpretation of ESF-related data also requires meticulous understanding and mapping of the data generation process in relation to the ESF-related steps within the IPF project cycle (see appendix C). Moreover, ESF-related data captured by data systems during the IPF project cycle may not fully reflect the intricacies of when and why certain ESF- related decisions are made and data points are generated. For example, system data may omit the deliberations surrounding ESF-related decisions and only record the outcomes of these decisions. Consequently, our analytic approach will carefully address these limitations and we will triangulate the findings from data analysis with insights gathered through our other methods to ensure robustness and reliability and to explore explanatory factors. 4.18 The evaluability and generalizability of ESF outcomes is challenged by the relatively low number of closed ESF projects. Currently, 188 out of 1,546 IPF projects under the ESF have closed (with 283 IPF projects anticipated to be closed by December 2024), and of these, just 33 have an Implementation Completion and Results Report available. Consequently, the ESF is not yet ready for comprehensive ground truthing, which precludes us from conducting extensive field evaluations. Our methods are designed accordingly and will focus on identifying implementation results, issues, and lessons. For closed and mature projects (593 projects with three or more years of implementation experience) and overall ESF capacity-building efforts, we will search as far down the results chain as possible in our analyses, including case analysis and process tracing. Our survey, in-country workshops, and virtual workshops with civil society organizations associated with projects will facilitate the inclusion of client and beneficiary voice into the evaluation. 4.19 The effects of COVID-19, including its impact on portfolio composition and project supervision, will also be factored into analyses. These considerations will support a comprehensive and contextually aware assessment of ESF implementation. 5. Quality Assurance Process 5.1 The Approach Paper and evaluation will undergo standard IEG quality assurance processes, including internal IEG and World Bank Group management review and external peer review. This evaluation will be peer reviewed by three topical and evaluation experts: • Andrew Brubaker, director, thematic and country division, independent Evaluation Department of the Asian Development Bank and former lead, Asian Development Bank’s Independent Evaluation Department’s 2020 review of the effectiveness of the Safeguard Policy Statement. 19 • Elana Berger, executive director, Bank Information Center and former director of social inclusion and accountability. She received her JD from the Columbia University School of Law and holds a BA in political science and history from the University of Maryland. • Osric Tening Forton, chief environmental safeguards and compliance officer and regional E&S coordinator at the African Development Bank. He has a PhD in Waste Management from the University of Brighton and an MSc in Environmental Engineering from the University of Nottingham. 6. Expected Outputs, Outreach, and Tracking 6.1 Expected outputs. The main output will be a final evaluation report that will be delivered to the World Bank Board’s Committee on Development Effectiveness after integrating feedback from Bank Group management. As a developmental evaluation, the team will produce intermediate outputs for discussion with and use by management; these engagements will also enable us to have a richer set of evaluation findings that are grounded in context, using explanatory factors derived (and validated) from these exchanges. 6.2 Engagement. The evaluation will be conducted in collaboration with internal stakeholders, clients, and members of civil society. It includes country workshops to solicit client voice, a survey to comprehensively collect management and staff perceptions, and structured interviews of technical experts. While developing the Approach Paper, the evaluation team consulted with Bank Group management and technical staff. The evaluation team will intermittently engage with key internal stakeholders to keep abreast of any future changes in ESF implementation and to provide findings as they emerge during the evaluation. 6.3 Noncompliance. The evaluation will adhere to IEG and Bank Group protocols in addressing any instances where, during the evaluation, the team identifies a case of ESF noncompliance that has not yet been reported to the operational team or to relevant World Bank oversight bodies. 6.4 Outreach and tracking. A communications and influence strategy—including both internal and external forums—will be developed with IEG’s Knowledge and Communications team. This strategy will include launching and disseminating the evaluation once it is disclosed. 20 7. Resources 7.1 This evaluation will be task managed by Lauren Kelly, lead evaluation officer, and co-task managed by Joy Butscher, evaluation officer, under the guidance of Penelope Jackson, manager, Finance, Private Sector, Infrastructure and Sustainable Development, and Carmen Nonay, director, Finance, Private Sector, and Sustainable Development. The team will include as core team members Azada Hussaini, evaluation officer; Sanittawan Nikki Tan, evaluation analyst; Tobias Fast, strategy officer; Tao Tao, extended-term consultant; Cecil John, short-term consultant; Christian Freymeyer, short- term consultant; Elizabeth Dodds, short-term consultant; Gabriel Stephan, short-term consultant; Iren Bagdasarian, short-term consultant; Josh Fuchs, short-term consultant; and Stephanie Keene, short-term consultant. Harsh Anuj, data scientist, will provide core advisory support for data collection, validation, management, and analysis; and Munib Qasim Zia, short-term consultant, will provide data science inputs. Ryan Watkins, professor at George Washington University, will advise the team on assessing capacity building. Bill Rahill, short-term consultant, will support the benchmarking exercise. Elena Bardasi, senior economist, will support the compendium volume on SEA/SH. Michael Woolcock, lead social scientist, and Sakuntala Akmeemana, short-term consultant, will support the sustainable development outcomes module. The evaluation team will also work closely with IEG’s Methods Advisory Function team. Policy and technical expertise on E&S risk management will be brought on to the evaluation team during implementation. Jean Jacques Alain Ildevert Ahouansou, program assistant, will provide administrative support. 7.2 This evaluation will be delivered to the Committee on Development Effectiveness in the second quarter of FY26. The longer time frame is associated with the evaluation complexity, including the multiple methods that will be used to answer the evaluation questions. Foremost among these methods is the development of a usable database on which the evaluation is predicated. The longer time frame is also associated with the developmental nature of the evaluation, including a commitment to engage and provide feedback to management and to conduct country workshops. The proposed budget is US$1,050,000, including outreach and dissemination. These costs cover core staffing, senior ESF expertise, advanced methods and related cross-support from the Methods Advisory Function and Knowledge and Communications, and six country workshops requiring facilitation and logistical support. 1These include guidance notes for borrowers on Environmental and Social Standards 1–10; Good Practice Notes for Staff on Gender; Non-Discrimination and Disability; Addressing Sexual Exploitation and Abuse/Sexual Harassment in Investment Project Financing involving Major 21 Civil Works; Third Party Monitoring; Assessing and Managing the Risks and Impacts of the Use of Security Personnel; Road Safety; Non-Discrimination: Sexual Orientation and Gender Identity; Dam Safety; Guidance on Conducting an Overview Assessment; Advisory Note on Technical Assistance and the Environmental and Social Framework (ESF) Greenhouse Gas Emissions Accounting under the ESF; Stakeholder Engagement Plan Templates; Grievance Redress Mechanism Checklist; Tip Sheet for Project Screening and Preparing Concept Stage Environmental and Social Review Summary (ESRS) and Project Information Document; Environmental and Social Commitment Plan Template and associated Tip Sheet; Environmental and Social Review Summary Concept Stage Template; Tip Sheet for Project Screening and Preparing Concept Stage Environmental and Social Review Summary and Project Information Document; Environmental and Social Review Summary Appraisal Stage Template; and Labor Management Procedures Template. 2The ESF applies to all new investment project financing projects with Concept Note meetings on or after October 1, 2018. However, between October 18, 2018, and December 31, 2019, all additional financing (AF) designed to scale up activities of existing projects governed by the safeguard policies would apply the safeguard policies rather than the ESF. This included 319 AF associated with investment lending projects (parent projects) that were approved before the implementation of the ESF. To qualify for this extension, activities to be financed by the AF should not raise the safeguards category for the project or trigger any new safeguard policies. As part of the World Bank’s rapid response to COVID-19, as of June 2020, teams were permitted to seek management waivers to enable the exceptional use of AF with the application of safeguard policies to scale up existing pre-ESF investment project financing operations. To qualify, the waiver could only be applied to scale-up where it is determined that the permitted activities are in support of the COVID-19 response. 1Risk categorization refers to the World Bank’s classification of environmental and social risks of projects into one of the four classifications (high, substantial, moderate, and low). This categorization is determined by the World Bank based on relevant issues and is to be reviewed by the World Bank on a regular basis, including during implementation, and is to be changed, where necessary, to ensure that it continues to be appropriate. The monitoring and response to changes in a project’s ESF risk profile is a core tenet of adaptive management. 22 Bibliography Beach, Derek. 2020. “Process Tracing Methods.” Oxford Bibliographies April 22. https://www.oxfordbibliographies.com/view/document/obo-9780199743292/obo- 9780199743292-0227.xml. IBM. n.d. “What Is Exploratory Data Analysis (EDA)?” Exploratory Data Analysis, IBM, New York (accessed July 7, 2024), https://www.ibm.com/topics/exploratory-data-analysis. Raimondo, Estelle, Harsh Anuj, and Virginia Ziulu. 2023. “Setting Up Experiments to Test GPT for Evaluation.” Experimenting with GPT and Generative AI for Evaluation (blog), August 16, 2023. https://ieg.worldbankgroup.org/blog/setting-experiments-test-gpt-evaluation. Vaessen, Jos, Sebastian Lemire, and Barbara Befani. 2020. Evaluation of International Development Interventions: An Overview of Approaches and Methods. World Bank. https://doi.org/10.1596/IEG155317. World Bank. n.d. “Guidance Notes for Borrowers.” Environmental and Social Framework Resources, World Bank. https://www.worldbank.org/en/projects- operations/environmental-and-social-framework/brief/environmental-and-social- framework-resources#guidancenotes. World Bank. 2010. Safeguards and Sustainability Policies in a Changing World: An Independent Evaluation of World Bank Group Experience. Independent Evaluation Group. World Bank. World Bank. 2016. “Review and Update of the World Bank’s Safeguard Policies: Environmental and Social Framework (Proposed Third Draft).” August 4, 2016, World Bank. World Bank. 2017. The World Bank Environmental and Social Framework. World Bank. World Bank. 2018. “Addressing Gender Based Violence in Investment Project Financing Involving Major Civil Works.” Good Practice Note. World Bank. https://documents1.worldbank.org/curated/en/399881538336159607/Environment-and- Social-Framework-ESF-Good-Practice-Note-on-Gender-based-Violence-English.pdf. World Bank. 2020a. “Addressing Sexual Exploitation and Abuse and Sexual Harassment (SEA/SH) in Investment Project Financing involving Major Civil Works” Environmental & Social Framework for IPF Operations. Good Practice Note. World Bank. https://thedocs.worldbank.org/en/doc/741681582580194727- 0290022020/original/ESFGoodPracticeNoteonGBVinMajorCivilWorksv2.pdf. World Bank. 2020b. The World Bank Environmental and Social Framework (ESF): Implementation Update. World Bank. World Bank. 2021. Bank Directive: Addressing Risks and Impacts on Disadvantaged or Vulnerable Individuals or Groups. World Bank. 23 World Bank. 2022. Addressing Sexual Exploitation and Abuse and Sexual Harassment (SEA/SH) in Human Development Operations.” Environmental & Social Framework for IPF Operations. Good Practice Note. World Bank. https://thedocs.worldbank.org/en/doc/e2ff01be0f07c82d73bc0c5e7ddf394f- 0290032022/original/ESF-Good-Practice-Note-on-Addressing-SEA-SH-in-HD-Operations- First-Edition-September-16-2022.pdf. World Bank. 2023. Bank Directive/Procedure: Environmental and Social Directive/Procedure for Investment Project Financing. World Bank. World Bank. 2024. “Environmental and Social Framework (ESF) Training: ESF Fundamentals.” World Bank. World Bank Group. 2021. Unlocking Nature-Smart Development: An Approach Paper on Biodiversity and Ecosystem Services. World Bank Group. http://hdl.handle.net/10986/36047. 24 Appendix A. Evaluation Design Matrix This appendix includes the key questions, information sources, data collection and analysis methods, and the strengths and limitations associated with these (table A.1). Table A.1. Evaluation Design Matrix Limitations and Mitigation Key Questions Method Information Sources Data Collection and Analysis Measures Overarching EQ: To Build a usable ESF database ESF-related data (including Consolidate all available ESF-related data into Relevant data are stored on different what extent has the ESF that consolidates relevant underlying ESMS data) from one comprehensive database. The data platforms; therefore, we will progressed to achieve ESF and project data across various World Bank data data/database will be validated with consult with various World Bank its objectives of sources. sources including standard Operations Policy and Country Services and staff to ensure that our consolidated protecting people and reports, EDC, and so on. ITS. database is complete and accurate. the environment from Design a World Bank staff Staff across grade levels and The nature of the survey (coverage of The survey design will ensure potential adverse survey to garner functions that are implicated in management and staff), technology choice, anonymity and confidentiality to project impacts while anonymized insights ESF decision-making and use. number of questions, and question style will encourage honest participation and promoting sustainable directly from implicated be decided as part of the survey design in minimize response biases. It will also development? staff. consultation with survey design experts. consider length and complexity to (cross-cutting methods) The anonymized survey will be administered ensure the survey is concise and early in the evaluation period so that the user-friendly, preventing staff from survey data can be used as probing questions feeling overburdened and increasing in subsequent interviews, focus groups, and their likelihood of responding. country workshops. The survey will help identify areas of interest for further probing through other evaluation methods, and follow-up interviews and sequenced methods will clarify areas of convergence and divergence, further deepening the insights gained. Evaluate the treatment of Recent IEG gender and FCV For EQ1, we will analyze ESF risk classification the cross-cutting risks of evaluations and evidence data to determine how SEA/SH risks influence SEA/SH. emerging from all evaluation a project’s ESF risk profile, as well as shifts in questions. risk classification and mitigation activities across the project cycle. For EQ2, we will 25 Limitations and Mitigation Key Questions Method Information Sources Data Collection and Analysis Measures assess how the World Bank is building borrower capacity to address SEA/SH risks. For EQ3, we will conduct specific focus groups on compliance with SEA/SH standards with ESF staff and other experts in this space. For EQ4, questions on SEA/SH will be incorporated into the benchmarking exercise. Key stakeholder interviews Internal: World Bank staff Across the evaluation methods, we will Individual interviews will be and group interviews and including the chief conduct semistructured key informant triangulated to ensure reliability and focus group discussions environmental and social interviews, group interviews, and/or FGDs) to to identify any potential outliers. across all evaluation standards officer, ESF validate and triangulate emerging findings Group dynamics in FGDs may methods to validate implementation support unit and to provide depth on explanatory factors present challenges, such as emerging findings and leadership, regional and contextual nuance. dominant participants identify explanatory factors. Environmental and Social Thematic content analysis: We will code and overshadowing quieter members or Standards advisers, ESF categorize key informant interview, group groupthink biasing the discussion accredited practice managers interview, and FGD transcripts to identify outcomes. To manage these and E&S specialists, project common themes and patterns (using dynamics, facilitators will employ task team leaders and teams, generative AI, where appropriate). strategies such as structured turn- and so on. taking, using smaller breakout External: Former World Bank groups, and actively encouraging staff, other multilateral or input from all participants. bilateral staff, borrower country officials, civil society, private sector, and so on. EQ1: To what extent has Data analysis of ESF ESF database (see above), key Adaptive management: We will focus our It is important to note that the the World Bank implementation, especially stakeholder interviews and pattern identification on several tenets of evaluation will not weigh in on implemented the key adaptive risk management group interviews/FGDs, World adaptive management in the ESF system— whether compliance was achieved principles of the ESF and borrower E&S Bank staff survey, and project namely, (i) risk identification, classification, but rather highlight the quality and since 2018? What performance. data and documentation. and reclassification;iii (ii) allocation of staff consistency of the evidence used to challenges remain? and resources; and (iii) monitoring, grievance determine borrower E&S redress, reporting, and other forms of performance ratings. stakeholder engagement. Patterns will be ESF-related data captured by data analyzed across time, country income group systems in the IPF project cycle may not fully reflect the reality of when 26 Limitations and Mitigation Key Questions Method Information Sources Data Collection and Analysis Measures and typology (including FCV, SIDS), and level and why certain ESF-related data of project risk, among other factors. points are generated; for example, borrower E&S performance: First, the the system data may not capture evaluation will develop a database of these deliberations about ESF-related performance ratings for projects using ESF, as decisions, only the results of the reported throughout the project cycle and at final decisions. Therefore, our project close. Second, the evaluation will analytic approach will carefully determine and make evident the methods consider these issues and the and criteria being used by the World Bank to findings from data analysis will be rate borrower E&S performance. In making triangulated with those from other evident the criteria, the evaluation will data collection and analysis consider how borrower E&S performance methods. data are collected and analyzed, and how performance ratings decisions are determined. Third, the evaluation will collect and make evident ESF-related qualitative data in projects that provide explanatory information about borrower E&S performance. Fourth, the evaluation will examine patterns among projects with similar ESF risks to determine the evenness of information provided to support ratings decisions (that is, the quality of evidence). Focused benchmarking E&S frameworks (policies, Document review of E&S frameworks and— A limitation of benchmarking other exercise on ESF standards, and key using a structured template—perform agency E&S frameworks is that harmonization across select implementation mechanisms) content and gap analysis to note the areas of implementation challenges may not international financial of other multilateral convergence and divergence across agencies. be visible from the frameworks institutions. development banks and Interviews will help the team understand themselves. To mitigate this, evaluations thereof (where reasons for gaps, per each institution’s ESF interviews with agency staff and available)—for example, the development. stakeholders will be conducted to Asian Development Bank, the uncover and understand any African Development Bank, the implementation issues. Additionally, Inter-American Development differences in context, scale, and Bank, the European Bank for scope between agencies may limit the comparability of frameworks. 27 Limitations and Mitigation Key Questions Method Information Sources Data Collection and Analysis Measures Reconstruction and This will be mitigated by selecting Development, IFC, and MIGA. agencies with similar operational environments and carefully contextualizing findings. Targeted look at “common Project documentation of We will assess the extent to which cofinanced Project documentation may not approaches” in cofinanced cofinanced projects; interviews projects using the ESF are making efforts to provide extensive detail on the use projects. (for example, project teams, harmonize their ESF approach. This will be of a common approach. We will E&S staff from cofinancing done through a review of project therefore conduct thorough agencies, and so on); documentation and interviews with project interviews with project teams to independent evaluations. teams, considering updated ESF policies, gain deeper insights and fill in gaps standards, and procedures. left by the documentation. EQ2: How well has the Stocktake of ESF-related World Bank lending and Identification of dedicated projects, project Relevant capacity-building activities World Bank been capacity-building activities nonlending portfolio from components, or trust-funded activities will need to be carefully identified helping borrowers to and portfolio review and World Bank operational (lending and nonlending) that support through manual screening, as in- build their national analysis to analyze trends. databases; information from borrower E&S risk management capacity. scope activities may not explicitly systems for managing ESF-related communities of Portfolio review and analysis to examine the indicate their link to the ESF. Clear E&S risks? practice, such as the Repository nature of this support and analyze trends inclusion and exclusion criteria will of E&S country system (depth, breadth, frequency, and distribution, be developed as part of the strengthening initiatives from including across country income groups and evaluation process. the social risk management typologies such as FCV and SIDS). team; relevant publications; official communications from ESF leadership (for example, Board updates, presentations, newsletters, and so on). Semistructured interviews ESF trainers and a sample of Targeted, semistructured interviews with ESF One limitation is challenges with with ESF trainers and a trainees stratified across trainers and a sample of trainees (stratified recall, as respondents may not stratified sample of regions and country across regions and country typologies). accurately remember details of past trainees. typologies. Interviews will triangulate perspectives on training sessions. To mitigate this, training quality, application, and uptake. We we will conduct interviews with will collate, analyze, and quantify responses, those more recently trained and, and validate uptake claims by requesting where possible, interviewees will be evidence. The interviews will include probing provided with prompts or materials questions related to country systems to aid their memory. To avoid 28 Limitations and Mitigation Key Questions Method Information Sources Data Collection and Analysis Measures development, drawing on data provided to response bias, where participants indicate whether capacity was built at the may give desirable answers rather project level, country level, or both. Interviews than truthful ones, we will ensure will also probe capacity development for new anonymity and confidentiality and ESF policy areas and standards. use neutral, nonleading questions. Quantitative data from training evaluations Additionally, to ensure a and feedback forms will be analyzed, if representative sample, we will available to IEG. carefully select participants from various regions and country typologies. Project-level case analysis Project documentation for ESF For those projects using borrower Causal attribution/contribution of of projects using borrower projects using borrower frameworks, we will use a structured template capacity-building activities may be frameworks frameworks (partially or fully); designed as part of the evaluation process to challenging to ascertain from interviews with project teams. examine (i) when borrower systems have documentation alone; therefore, been used to date; (ii) what led to the triangulated interviews with project decision to use borrower frameworks; (iii) teams and in-country workshops what changes were put in place to enable the with numerous stakeholders will borrower framework to be used, if any; (iv) provide contextual understanding whether the use of borrower frameworks at and additional perspectives that will the project level is leading to leveraging or enrich the analysis. spillover effects—that is, whether the borrower is using systems or skills acquired from the ESF process or other donor or country projects; (v) the evidence of capacity on new standards and policy areas; and (vi) to the extent that information is available, how well the ESF is being implemented using the borrower frameworks. Data collection for each case will include project data and documentation, as well as interviews to validate and expand on emerging findings. A consideration of borrower frameworks will also include analysis of support to different country income groups and typologies (for example, FCV, SIDS). 29 Limitations and Mitigation Key Questions Method Information Sources Data Collection and Analysis Measures Regional/country Various stakeholders in six In-country workshops: Gather qualitative To ensure the fidelity of responses workshops to ground truth countries (that is, ESF regional insights through semistructured group and promote responder safety, the the process tracing exercise hubs). Participants will include discussions and participatory activities with workshops will include separate and to obtain client voice. the following: (i) government— diverse stakeholders to capture sessions for different levels of senior borrower country comprehensive and context-specific government (for example, senior officials and staff of project information. officials versus technical staff) and implementation units; and (ii) Thematic content analysis: Code and nongovernment stakeholders (for nongovernment—academics, categorize workshop transcripts to identify example, civil society, private sector, civil society organizations, common themes and patterns (using and so on). private sector representatives, generative AI, where appropriate). in-country bilaterals and multilaterals, other donors, and so on. EQ3: To what extent is Focused literature review of We will conduct a focused literature review to there evidence that ESF existing evidence on the understand and document existing evidence application has plausible contribution of on the plausible contribution of ESF contributed to ESF mechanisms to mechanisms to development outcomes. enhanced development development outcomes. outcomes? Theories of change and Project documents, interviews, We will then develop nested theories of contribution analysis annual reports, blogs, Board change for each of these identified papers, or those stories shared mechanisms. This nested approach will allow by communities of practice for the identification of multiple subtheories through meetings or listserv, of change (aligned to specific ESF among other sources. mechanisms) which help explain the potential contribution pathways of the ESF mechanisms to broader development outcomes. Comparative project Project documentation for Using a comparative project analysis, the Desk-based analysis may miss some analysis and semistructured projects that achieve enhanced evaluation will then trace plausible outcome contextual nuances or recent interviews. development outcomes; pathways and assess the contribution of the changes in project implementation, project documentation for ESF mechanism in line with the nested theory. and project documentation may not comparator projects; interviews Semistructured interviews will also be used to provide extensive detail. To address with project teams. complement, triangulate and validate the this, interviews will be conducted for pathways and contributions identified in the 30 Limitations and Mitigation Key Questions Method Information Sources Data Collection and Analysis Measures literature review and comparative project key cases to deepen contextual analysis. understanding. Source: Independent Evaluation Group. Note: AI = artificial intelligence; E&S = environmental and social; EDC = enterprise data catalog; EQ = evaluation question; ESF = Environmental and Social Framework; ESMS = Environmental and Social Management System; FCV = fragility, conflict, and violence; FGD = focus group discussion; IEG = Independent Evaluation Group; IFC = International Finance Corporation; IPF = investment project financing; ITS = Information and Technology Solutions; MIGA = Multilateral Investment Guarantee Agency; SEA/SH = sexual exploitation and abuse and sexual harassment; SIDS = Small Island Developing States. 31 Appendix B. Preliminary Portfolio This appendix outlines the methodology used to identify the set of World Bank investment project financing (IPF) operations that make up the evaluation portfolio and provides a basic description of it along project attributes of interest. The appendix also provides a basic distribution of the portfolio along some key Environmental and Social Framework (ESF)-related attributes of interest. Portfolio Identification and Description To identify the relevant potential World Bank ESF portfolio, the evaluation team searched for active and closed IPF projects approved since October 1, 2018 (the date the ESF came into effect, with some exceptions) and covered by ESF. This search resulted in a portfolio of 1,546 disclosed projects. As shown in table B.1, only 12 percent of projects in the portfolio were closed as of July 8, 2024; the rest were still active. This is to be expected given that IPFs on average have an implementation life span of about four years and because the ESF has been in place for less than six years at the time of writing (figure B.1). Table B.1. Distribution of Project Counts and Commitments by Status Total Lending Proportion of Proportion of Projects Commitments Projects Commitments Project Status (no.) ($US, millions) (%) (%) Active 1,358 165,200 88 95 Closed 188 8,391 12 5 Total 1,546 173,591 100 100 Source: Independent Evaluation Group. 32 Figure B.1. Trends in Project Counts and Commitment by Approval Year, Additional Financing versus Nonadditional Financing a. Trend in count of projects 350 307 300 256 235 237 250 217 Projects (no.) 200 150 75 83 100 57 53 50 18 5 1 2 0 2019 2020 2021 2022 2023 2024 2025 Approval FY Additional Financing New Projects b. Trend in total commitments 40,000 34,640 33,070 Total commitments (US$, 28,980 30,000 27,070 millions) 19,220 20,000 8,541 9,994 10,000 4,850 4,338 2,451 110 200 133 0 2019 2020 2021 2022 2023 2024 2025 Approval FY Additional Financing New Projects Source: Independent Evaluation Group. Note: FY = fiscal year. Figure B.2 shows the distribution of project approval counts and total commitment amounts by lead Global Practice. As can be seen from the figures, the Health, Nutrition, and Global Population Global Practice and the Social Protection and Jobs Global Practices had the highest and second-highest project approval counts and commitment volumes, respectively, with the former quite a bit larger than the latter. This is to be expected given the World Bank’s large health and economic response to the COVID-19 pandemic starting in fiscal year 2020. 33 Figure B.2. Distribution of Project Counts and Commitments by Lead Global Practice a. Distribution of project counts Health, Nutrition & Population Social Protection & Jobs Education Urban, Resilience and Land Energy & Extractives Agriculture and Food Transport Environment, Natural Resources & the Blue Economy Lead GP Finance, Competitiveness and Innovation Governance Social Sustainability and Inclusion Water Digital Development Poverty and Equity Other Macroeconomics, Trade and Investment 0 100 200 300 400 Projects (no.) b. Distribution of total commitments Health, Nutrition & Population Social Protection & Jobs Transport Urban, Resilience and Land Agriculture and Food Energy & Extractives Governance Lead GP Finance, Competitiveness and Innovation Education Water Environment, Natural Resources & the Blue Economy Social Sustainability and Inclusion Digital Development Poverty and Equity Other Macroeconomics, Trade and Investment 0 5,000 10,000 15,000 20,000 25,000 Total commitments (US$, millions) Source: Independent Evaluation Group. Note: GP = Global Practice. Figure B.3 presents the geographic distribution of the evaluation portfolio by World Bank Regions and table B.2 presents the same geographic distribution based on client countries’ fragility, conflict, and violence status. Both figures indicate that there was a concentration of project counts and commitment volumes in the Africa East Region. Finally, table B.2 shows that almost a third of the project counts and commitment volumes were in fragility, conflict, and violence countries. 34 Figure B.3. Distribution of Project Counts and Commitments by Region a. Distribution of project counts 400 350 300 Projects (no.) 250 200 150 100 50 0 AFE AFW ECA LCR EAP MNA SAR OTH Region b. Distribution of total commitments 50,000 Total commitments (US$, millions) 45,000 40,000 35,000 30,000 25,000 20,000 15,000 10,000 5,000 0 AFE ECA AFW SAR LCR EAP MNA OTH Region Source: Independent Evaluation Group. Note: AFE = Eastern and Southern Africa; AFW = Western and Central Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia; OTH = other. Table B.2. Distribution of Project Counts and Total Commitments by Country Fragility, Conflict, and Violence Status Proportion of Proportion of Country FCV Projects Total Commitments Projects Commitments Status (no.) ($US, millions) (%) (%) Non-FCV 927 119,400 69 69 FCV 416 54,210 31 31 Total 1,343 173,610 100 100 Source: Independent Evaluation Group. Note: Country FCV status is as of project approval, as reported in World Bank operations data systems. FCV = fragility, conflict, and violence. 35 Environmental and Social Framework Characteristics of Preliminary Portfolio Risk categories for overall, environmental, social, and sexual exploitation and abuse and sexual harassment (SEA/SH) risk. Out of the 1,546 IPF projects in the evaluation portfolio, the latest environmental and social risk categorization (and its environmental and social subcategories) was available for 1,269 projects (82 percent), and sexual exploitation and abuse and sexual harassment risk categorization for 1,214 projects (79 percent). Tables B.3 to B.6 outline the distribution of project counts and total commitment volumes by the overall, environmental, and social risk categories, respectively. As seen in these tables, most projects had moderate or substantial levels of overall, environmental, and social risks (79 percent, 75 percent, and 80 percent, respectively). This was in contrast to the sexual exploitation and abuse and sexual harassment risks in projects, where almost half (49 percent) of projects were categorized as having low risk and another third (30 percent) as moderate risk. Table B.3. Distribution of Project Counts and Commitments by Environmental and Social Risk Category Total World Bank Proportion of Risk Projects Commitments Proportion of Projects Commitments category (no.) ($US, millions) (%) (%) High 125 26,248 10 18 Substantial 525 74,037 41 51 Moderate 481 40,922 38 28 Low 138 3,716 11 3 Total 1,269 144,923 100 100 Source: Independent Evaluation Group. Note: ESF data are from Standard Report ESF5.0 and should be considered preliminary at this stage. ESF = Environmental and Social Framework. Table B.4. Distribution of Project Counts and Commitments by Environmental Risk Category Total World Bank Proportion of Risk Projects Commitments Proportion of Projects Commitments Category (no.) ($US, millions) (%) (%) High 73 15,419 6 11 Substantial 463 68,638 36 47 Moderate 493 47,942 39 33 Low 240 12,924 19 9 Total 1,269 144,923 100 100 Source: Independent Evaluation Group. Note: ESF data are from Standard Report ESF5.0 and should be considered preliminary at this stage. ESF = Environmental and Social Framework. 36 Table B.5. Distribution of Project Counts and Commitments by Social Risk Category Total World Bank Proportion of Risk Projects Commitments Proportion of Projects Commitments Category (no.) ($US, millions) (%) (%) High 107 23,245 8 16 Substantial 490 70,513 39 49 Moderate 515 46,686 41 32 Low 157 4,478 12 3 Total 1,269 144,923 100 100 Source: Independent Evaluation Group. Note: ESF data are from Standard Report ESF5.0 and should be considered preliminary at this stage. ESF = Environmental and Social Framework. Table B.6. Distribution of Project Counts and Commitments by Sexual Exploitation and Abuse and Sexual Harassment Risk Category Total World Bank Proportion of Risk Projects Commitments Proportion of Projects Commitments Category (no.) ($US, millions) (%) (%) High 47 8,357 4 6 Substantial 209 37,485 17 26 Moderate 368 49,961 30 35 Low 590 47,358 49 33 Total 1,214 143,160 100 100 Source: Independent Evaluation Group. Note: ESF data are from Standard Report ESF5.0 and should be considered preliminary at this stage. ESF = Environmental and Social Framework. Use of borrower ESF. Out of the 1,546 IPF projects in the evaluation portfolio, information on whether the borrower’s ESF was used was available for 980 projects (63 percent of total). As shown in table B.7, only 11 of 980 projects used the borrower framework in full, and another 78 in part, while 891 (91 percent) did not use the borrower framework at all. Table B.7. Distribution of Project Counts and Commitments by Use of Borrower Environmental and Social Framework Total World Bank Proportion of Proportion of Use of Borrower Projects Commitments Projects Commitments Framework (no.) ($US, millions) (%) (%) In part 78 11,285 8 9 In whole 11 1,118 1 1 No 891 108,589 91 90 Total 980 120,992 100 100 Source: Independent Evaluation Group. Note: ESF data are from Standard Report ESF5.0 and should be considered preliminary at this stage. ESF = Environmental and Social Framework. 37 Relevance of the 10 Environmental and Social Standards (ESSs). Out of the 1,546 IPF projects in the evaluation portfolio, information on the relevance of each of the 10 ESSs was available for 1,271 projects (75 percent). As outlined in table B.8, ESS1, ESS2, and ESS10 were relevant for all projects (for which this information was available); ESS5, ESS6, and ESS8 were relevant for about half; ESS7 was relevant for just over one-third; and ESS9 was relevant for about one-tenth. Table B.8. Distribution of Project Counts and Commitments by Relevance of ESS Total World Bank Proportion of Proportion of Projects Commitments Projects Commitments ESS (no.) ($US, millions) (%) (%) ESS1: Assessment and Management of 1,271 145,018 100.0 100.0 Environmental and Social Risks and Impacts ESS2: Labor and Working Conditions 1,265 144,492 99.5 99.6 ESS3: Resource Efficiency and Pollution 1,114 135,143 87.6 93.2 Prevention and Management ESS4: Community Health and Safety 1,152 139,431 90.6 96.1 ESS5: Land Acquisition, Restrictions on 653 98,046 51.4 67.6 Land Use and Involuntary Resettlement ESS6: Biodiversity Conservation and 661 96,123 52.0 66.3 Sustainable Management of Living Natural Resources ESS7: Indigenous Peoples/Sub-Saharan 452 66,411 35.6 45.8 African Historically Underserved Traditional Local Communities ESS8: Cultural Heritage 654 96,259 51.5 66.4 ESS9: Financial Intermediaries 131 23,833 10.3 16.4 ESS10: Stakeholder Engagement and 1,271 145,018 100.0 100.0 Information Disclosure Source: Independent Evaluation Group. Note: ESF data are from Standard Report ESF5.0 and should be considered preliminary at this stage. ESF = Environmental and Social Framework; ESS = Environmental and Social Standards. Overall ESS performance ratings. Of the 1,546 IPF projects in the evaluation portfolio, the six-point scale performance rating for overall ESS performance was available for 1,155 projects (75 percent). As outlined in table B.9, almost all projects (1,112 out of 1,155, or 96 percent) were rated as being satisfactory (69 percent) or moderately satisfactory (28 percent). 38 Table B.9. Distribution of Project Counts and Commitments by Overall Environmental and Social Standard Performance Rating Total World Bank Proportion of Proportion of Overall ESS Performance Projects Commitments Projects Commitments Rating (no.) ($US, millions) (%) (%) Highly satisfactory 1 100 0 0 Satisfactory 793 82,846 69 62 Moderately satisfactory 319 45,731 28 34 Moderately unsatisfactory 37 4,816 3 4 Unsatisfactory 5 766 0 1 Highly unsatisfactory 0 0 0 0 Total 1,155 134,259 100 100 Source: Independent Evaluation Group. Note: ESF data are from Standard Report ESF5.0 and should be considered preliminary at this stage. ESF = Environmental and Social Framework; ESS = Environmental and Social Standards. 39 Appendix C. The Environmental and Social Framework Across the Project Cycle Source: World Bank Environmental and Social Framework Training. Note: E&S = environmental and social; ERP = emergency response plan; ESF = Environmental and Social Framework; ESCP = Environmental and Social Commitment Plan; ESHG = Environmental, Health, and Safety Guidelines; ESMS = Environmental and Social Management System; ESRS = Environmental and Social Review Summary; ESS = Environmental and Social Standards; GHG = greenhouse gas; GRM = grievance redress mechanism; ICR = Implementation Completion and Results Report; ISR = Implementation Status and Results Report; MTR = Mid-Term Review; RHA = risk hazard assessment. 40 Appendix D. Glossary Borrower: In the Environmental and Social Framework Policy, “unless the context requires otherwise, the term “Borrower” means a borrower or recipient of World Bank financing for an investment project, and any other entity responsible for the implementation of the project” (World Bank 2017, 3). Borrower environmental and social (E&S) framework: “The Borrower’s [E&S] Framework will include those aspects of the country’s policy, legal and institutional framework, including its national, subnational, or sectoral implementing institutions and applicable laws, regulations, rules and procedures, and implementation capacity, which are relevant to the environmental and social risks and impacts of the project. The aspects that are relevant will vary from project to project, depending on such factors as the type, scale, location, and potential environmental and social risks and impacts of the project and the role and authority of different institutions.... Where there are inconsistencies or lack of clarity within the Borrower’s E&S Framework as to relevant authorities or jurisdiction, these will be identified.... The use of all, or part, of the Borrower’s ES Framework will be agreed between the [World] Bank and the Borrower, following completion of the assessment referred to below.... The decision to use all, or parts, of the Borrower’s [E&S] Framework will not relieve the Bank of any of its due diligence obligations set out in Section C of this Policy [environmental and social due diligence]” (World Bank 2017, 7). Environment and Social Management Plan: The borrower, in consultation with the World Bank, identifies and uses appropriate methods and tools including the use of the Environmental and Social Management Plan. The plan is an instrument that details (i) the measures to be taken during the implementation and operation of a project to eliminate or offset adverse environmental and social impacts, or to reduce them to acceptable levels; and (ii) the actions needed to implement these measures. Depending on the project, an Environmental and Social Management Plan may be prepared as a stand-alone document (particularly relevant when the borrower is engaging contractors) or the content may be incorporated directly into the Environmental and Social Commitment Plan (ESCP). Environmental and Social Compliance Plan: The World Bank agrees on an ESCP with the borrower. The ESCP sets out the material measures and actions required for the project to meet the Environmental and Social Standards over a specified time frame. The ESCP forms part of the legal agreement. It sets out the obligations of the borrower to implement the measures and actions, in accordance with specified time frames specified, and to review the status of implementation of the ESCP as part of its monitoring and reporting. The draft ESCP is disclosed as early as possible, and before project appraisal. 41 Overview assessment: “To support increasing use and strengthening of the Borrower‘s [E&S] Framework, the [World] Bank may, at the request of the Borrower and where the Bank deems this to be feasible, conduct an overview assessment of the Borrower’s existing policy, legal and institutional framework for addressing environmental and social risks and impacts, and related implementation capacity. The overview assessment will identify aspects of the existing framework that can be strengthened, and the capacity-building needed to support this. The overview assessment will not be a prerequisite for the use of all, or part of, the Borrower’s [E&S] Framework for a specific project. However, where the overview assessment has been completed, it will inform the [World] Bank’s assessment of the Borrower’s [E&S] Framework.” (World Bank 2017, 7). Proportionality: It refers to the depth and breadth to which environmental and social assessments, mitigation measures, and implementation and management plans are commensurate with the level of risk and impact associated with a project. This allows for a tailored approach rather than a “one-size-fits-all” model, such that higher-risk projects are allotted higher amounts of necessary resources compared with lower-risk projects. The Environmental and Social Framework explicitly states that “the environmental and social assessment will be proportionate to the potential risks and impacts of the project” (World Bank 2017, 15). It “will assess, in an integrated way, all relevant direct, indirect and cumulative environmental and social risks and impacts throughout the project life cycle” (World Bank 2017, 18). This proportional approach extends to project management and project monitoring: “The extent and mode of monitoring will be agreed upon with the [World] Bank, and will be proportionate to the nature of the project, the project’s environmental and social risks and impacts, and compliance requirements” (World Bank 2017, 21). Reference World Bank. 2017. The World Bank Environmental and Social Framework. Washington, DC: World Bank. 42