A PPE NDI X E S Results and Performance of the World Bank Group 2018 AN INDEPENDENT EVALUATION Contents Appendix A. Additional Information on World Bank Performance .....................................1 Appendix B. Results and Performance of International Finance Corporation Investment Projects ....................................................................................................................... 75 Trends in Performance .................................................................................................................. 75 Indicators of Development Outcome....................................................................................... 76 Performance of Infrastructure Industry Group ...................................................................... 78 Performance of the Manufacturing, Agribusiness, and Services Industry Group ...... 79 Performance of Financial Institutions Industry Group ........................................................ 79 Performance of Telecom, Media, Technology, Venture Capital and Funds Industry Group .................................................................................................................................................. 84 Performance of IFC Projects in IDA versus Non-IDA Countries ...................................... 85 Performance of Equity Investments .......................................................................................... 89 Performance by Investment Size ............................................................................................... 90 Quality of IFC’s Supervision and Administration ................................................................... 91 Appendix C. Results and Performance of International Finance Corporation Advisory Services Projects ............................................................................................................................. 94 Trends in Performance .................................................................................................................. 94 Performance of IFC Projects in IDA and Non-IDA Countries ........................................... 97 Quality of IFC’s Work in Project Preparation and Design for Advisory Services Projects ............................................................................................................................................. 102 Appendix D. Results and Performance of Multilateral Investment Guarantee Agency Projects ............................................................................................................................................104 Trends in Performance ................................................................................................................ 104 Appendix E. Analysis of Quality at Entry in World Bank Projects ....................................110 Contents............................................................................................................................................ 110 Overview ............................................................................................................................................ 111 Context .............................................................................................................................................. 113 Analysis of Quality Assurance Group Ratings and Independent Evaluation Group Ratings .............................................................................................................................................. 123 iii Contents Quality at Entry in Project Appraisal Documents ............................................................... 126 Quality at Entry in Implementation Completion and Results Reports Reviews ........ 130 Outcome from Workshops and Interviews on Quality at Entry with World Bank Staff 133 Concluding Questions for Consideration.............................................................................. 147 References ....................................................................................................................................... 150 Appendix F. Codebook for PAD and ICRR Qualitative Analysis .......................................151 Purpose ............................................................................................................................................. 151 General Instructions ...................................................................................................................... 151 Appendix G. Background on World Bank Project Performance Ratings ......................161 What Is an Implementation Completion and Results Report? ........................................ 161 What Is an ICR Review?............................................................................................................... 162 Changes to ICR Reviews in 2017 .............................................................................................. 162 Which Projects Are Covered by IEG’s ICRR Ratings?......................................................... 165 Coverage of Project Ratings Data for FY16 and FY17 Arising from Waiver for the Republic of Yemen Portfolio ..................................................................................................... 165 Coverage of Project Ratings Data for This Report ............................................................. 166 References ........................................................................................................................................167 Appendix H. Analysis of Safeguard Reporting in Recent Implementation Completion and Results Reports ....................................................................................................................168 Analysis of the Quality of Reported Environment and Social Framework Safeguard System Outcomes in Category A Projects ............................................................................ 169 Reporting on Resettlement in Category A Projects ............................................................ 171 Appendix I. Gender Incorporation in Project Development Objectives, Components, and Indicators................................................................................................................................173 Appendix J. Management Action Record: Additional Data and Methodology..........187 Definition and Data Collection Process ..................................................................................187 Appendix K. Major and Meso Evaluations, Independent Evaluation Group ...............191 A Thirst for Change ...................................................................................................................... 193 iv Contents Data for Development..................................................................................................................197 Engaging Clients for Increased Development Impact ...................................................... 199 Mobile Metropolises: Urban Transport Matters .................................................................204 Toward a Clean World for All.................................................................................................... 206 Growing the Rural Nonfarm Economy to Alleviate Poverty ........................................... 210 Growth for the Bottom 40 Percent ......................................................................................... 214 Bank Group Engagement in Upper-Middle-Income Countries ..................................... 216 Bank Group Support to Health Services................................................................................ 221 Engaging Citizens for Better Development Results ........................................................... 223 IFC’s Experience with Inclusive Business ................................................................................ 228 Maximizing the Impact of Development Policy Financing in IDA Countries............. 231 References .......................................................................................................................................245 v Appendix A. Additional Information on World Bank Performance Figure A.1. World Bank Projects: Outcome Ratings over Time, Three-Year Rolling a. Outcome ratings of World Bank lending projects, with the total percentage of projects rated MS+ indicated above each bar 100 80 77 74 72 76 70 70 70 70 69 71 60 40 Percent 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=753) (N=713) (N=749) (N=831) (N=940) (N=995) (N=961) (N=869) (N=768) (N=704) b. Outcome ratings of World Bank lending projects, with the total percentage of volume rated MS+ indicated above each bar 100 84 83 86 83 80 81 81 79 79 80 80 60 40 Percent 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=57,501) (V=52,576) (V=61,530) (V=76,295) (V=87,883) (V=93,863) (V=87,329) (V=86,857) (V=77,070) (V=71,623) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); MS+ = moderately satisfactory or above; V = total volume of projects ($, millions). All dollar amounts are U.S. dollars unless otherwise indicated. 1 Appendix A Additional Information on World Bank Performance Table A.1. World Bank Projects: Outcome Ratings over Time, Three-Year Rolling a. Percentage distribution of outcome ratings for World Bank lending projects, by number of projects Rating FY06–08 FY07–09 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 FY15–17 Highly satisfactory (%) 3 3 3 2 2 2 2 2 2 2 Satisfactory (%) 39 33 27 24 22 21 19 21 23 29 Moderately satisfactory (%) 35 38 41 45 47 47 48 48 47 45 Moderately unsatisfactory (%) 14 18 19 19 20 21 22 20 17 13 Unsatisfactory (%) 8 8 10 10 9 8 8 8 10 10 Highly unsatisfactory (%) 1 0 0 1 1 1 1 1 1 1 Moderately satisfactory or above (%) 77 74 70 70 70 70 69 71 72 76 Projects with ratings (no.) 753 713 749 831 940 995 961 869 768 704 b. Percentage distribution of outcome ratings for World Bank lending projects, by volume of projects Rating FY06–08 FY07–09 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 FY15–17 Highly satisfactory (%) 5 5 4 3 3 2 2 2 2 3 Satisfactory (%) 46 44 37 33 29 27 28 31 35 38 Moderately satisfactory (%) 32 32 39 45 48 50 49 51 46 45 Moderately unsatisfactory (%) 14 16 15 12 14 14 16 13 12 10 Unsatisfactory (%) 3 3 4 7 7 7 5 3 4 5 Highly unsatisfactory (%) 0 0 0 0 0 0 0 0 0 0 Moderately satisfactory or above (%) 83 80 81 81 79 79 80 84 83 86 Total volume of projects with ratings 57,501 52,576 61,530 76,295 87,883 93,863 87,329 86,857 77,070 71,623 ($, millions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). All dollar amounts are U.S. dollars unless otherwise indicated. 2 Appendix A Additional Information on World Bank Performance Figure A.2. World Bank Projects: Quality at Entry Ratings over Time, Three-Year Rolling a. Quality at entry ratings of World Bank lending projects, with the total percentage of projects rated MS+ indicated above each bar 100 80 76 73 69 64 64 60 60 61 59 58 60 40 Percent 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=754) (N=710) (N=738) (N=822) (N=932) (N=996) (N=958) (N=870) (N=767) (N=703) b. Quality at entry ratings of World Bank lending projects with the total percentage volume rated MS+ indicated above each bar 100 83 83 78 76 75 75 80 73 72 73 70 60 40 Percent 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=56,686) (V=51,533) (V=60,303) (V=76,344) (V=88,843) (V=94,780) (V=87,435) (V=86,685) (V=77,345) (V=71,594) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); MS+ = moderately satisfactory or above; V = total volume of projects ($, millions). All dollar amounts are U.S. dollars unless otherwise indicated. 3 Appendix A Additional Information on World Bank Performance Table A.2. World Bank Projects: Quality at Entry Ratings over Time, Three-Year Rolling a. Percentage distribution of quality at entry ratings of World Bank lending projects, by percentage of projects Rating FY06–08 FY07–09 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 FY15–17 Highly satisfactory (%) 4 4 2 2 2 2 2 1 1 1 Satisfactory (%) 45 35 29 26 22 20 18 19 19 21 Moderately satisfactory (%) 27 34 37 36 36 37 39 40 41 43 Moderately unsatisfactory (%) 12 18 22 27 30 32 32 31 30 29 Unsatisfactory (%) 11 8 9 9 9 8 9 9 8 6 Highly unsatisfactory (%) 0 0 0 1 1 1 0 0 1 0 Moderately satisfactory or above (%) 76 73 69 64 60 59 58 60 61 64 Projects with ratings (no.) 754 710 738 822 932 996 958 870 767 703 b. Percentage distribution of quality at entry ratings of World Bank lending projects, by percentage of volume Rating FY06–08 FY07–09 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 FY15–17 Highly satisfactory (%) 6 7 4 4 4 4 3 1 1 0 Satisfactory (%) 53 46 40 37 31 28 23 27 29 30 Moderately satisfactory (%) 23 29 34 33 35 40 47 48 45 44 Moderately unsatisfactory (%) 11 13 17 21 23 22 22 20 20 21 Unsatisfactory (%) 6 4 5 6 7 6 5 4 5 4 Highly unsatisfactory (%) 0 0 0 0 0 0 0 0 0 0 Moderately satisfactory or above (%) 83 83 78 73 70 72 73 76 75 75 Total volume of projects with ratings 56,686 51,533 60,303 76,344 88,843 94,780 87,435 86,685 77,345 71,594 ($, millions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). All dollar amounts are U.S. dollars unless otherwise indicated. 4 Appendix A Additional Information on World Bank Performance Figure A.3. World Bank Projects: Quality of Supervision Ratings over Time, Three- Year Rolling a. Quality of supervision ratings of World Bank lending projects, with the total percentage of projects rated MS+ indicated above each bar 100 87 83 80 78 78 78 77 78 78 80 80 60 40 Percent 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=742) (N=702) (N=731) (N=813) (N=922) (N=985) (N=953) (N=867) (N=767) (N=702) b. Quality of supervision ratings of World Bank lending projects, with the total percentage of volume rated MS+ indicated above each bar 100 91 89 89 89 88 85 85 87 83 83 80 60 40 Percent 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=55,208) (V=50,490) (V=59,511) (V=73,740) (V=86,238) (V=92,074) (V=87,182) (V=86,433) (V=77,345) (V=71,539) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = project closing fiscal year; MS+ = moderately satisfactory or above; V = total volume of projects ($, millions). All dollar amounts are U.S. dollars unless otherwise indicated. 5 Appendix A Additional Information on World Bank Performance Table A.3. World Bank Projects: Quality of Supervision Ratings over Time, Three-Year Rolling a. Percentage distribution of quality of supervision ratings of World Bank lending projects, by percentage of projects Rating FY06–08 FY07–09 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 FY15–17 Highly satisfactory (%) 5 4 4 3 3 3 2 1 2 3 Satisfactory (%) 61 54 46 40 35 32 29 30 31 33 Moderately satisfactory (%) 21 25 31 35 40 43 46 47 45 44 Moderately unsatisfactory (%) 7 12 15 16 16 17 18 16 16 15 Unsatisfactory (%) 6 4 5 6 5 5 5 6 6 5 Highly unsatisfactory (%) 1 0 0 0 0 0 0 0 0 0 Moderately satisfactory or above (%) 87 83 80 78 78 78 77 78 78 80 Projects with ratings (no.) 742 702 731 813 922 985 953 867 767 702 b. Percentage distribution of quality of supervision ratings of World Bank lending projects, by percentage of volume Rating FY06–08 FY07–09 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 FY15–17 Highly satisfactory (%) 8 7 6 5 5 4 3 2 2 3 Satisfactory (%) 63 59 52 44 38 40 43 46 45 43 Moderately satisfactory (%) 20 23 28 35 40 41 41 41 42 42 Moderately unsatisfactory (%) 6 9 12 12 11 10 9 8 8 10 Unsatisfactory (%) 3 2 2 5 6 5 4 3 3 3 Highly unsatisfactory (%) 0 0 0 0 0 0 0 0 0 0 Moderately satisfactory or above (%) 91 89 85 83 83 85 87 89 89 88 Total volume of projects with ratings 55,208 50,490 59,511 73,740 86,238 92,074 87,182 86,433 77,345 71,539 ($, millions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). All dollar amounts are U.S. dollars unless otherwise indicated. 6 Appendix A Additional Information on World Bank Performance Figure A.4. World Bank Projects: Bank Performance Ratings over Time, Three-Year Rolling a. Bank performance ratings of World Bank lending projects, with the total percentage of projects rated MS+ indicated above each bar 100 81 78 76 80 73 71 70 70 70 71 72 60 Percent 40 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=766) (N=727) (N=758) (N=843) (N=953) (N=1,013) (N=971) (N=875) (N=769) (N=704) b. Bank performance ratings of World Bank lending projects, with the total percentage of volume rated MS+ indicated above each bar 100 87 84 84 84 86 80 80 82 80 77 77 60 40 Percent 20 0 20 40 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=57,840) (V=52,786) (V=61,576) (V=77,894) (V=90,672) (V=96,931) (V=89,085) (V=87,412) (V=77,371) (V=71,623) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = project closing fiscal year; MS+ = moderately satisfactory or above; V = total volume of projects ($, millions). All dollar amounts are U.S. dollars unless otherwise indicated. 7 Appendix A Additional Information on World Bank Performance Table A.4. World Bank Projects: Bank Performance Ratings over Time, Three-Year Rolling a. Percentage distribution of Bank performance ratings of World Bank lending projects, by percentage of projects Rating FY06–08 FY07–09 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 FY15–17 Highly satisfactory (%) 3 2 1 1 1 1 1 0 0 1 Satisfactory (%) 51 38 29 24 20 17 15 16 16 18 Moderately satisfactory (%) 27 37 43 45 49 51 54 55 56 58 Moderately unsatisfactory (%) 12 17 20 21 23 23 23 21 20 17 Unsatisfactory (%) 7 5 6 8 7 7 7 8 7 6 Highly unsatisfactory (%) 1 0 0 0 1 0 0 0 1 1 Moderately satisfactory or above (%) 81 78 73 71 70 70 70 71 72 76 Projects with ratings (no.) 766 727 758 843 953 1013 971 875 769 704 b. Percentage distribution of Bank performance ratings of World Bank lending projects, by percentage of volume Rating FY06–08 FY07–09 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 FY15–17 Highly satisfactory (%) 5 5 3 3 2 3 2 1 0 0 Satisfactory (%) 56 47 39 33 27 24 21 22 25 26 Moderately satisfactory (%) 26 32 38 42 48 53 60 61 59 59 Moderately unsatisfactory (%) 10 14 18 17 17 14 14 12 12 10 Unsatisfactory (%) 3 2 2 5 6 6 4 3 4 4 Highly unsatisfactory (%) 0 0 0 0 0 0 0 0 0 0 Moderately satisfactory or above (%) 87 84 80 77 77 80 82 84 84 86 Total volume of projects with ratings 57,840 52,786 61,576 77,894 90,672 96,931 89,085 87,412 77,371 71,623 ($, millions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = project closing fiscal year. All dollar amounts are U.S. dollars unless otherwise indicated. 8 Appendix A Additional Information on World Bank Performance Figure A.5. World Bank Projects: Rating of Quality of Monitoring and Evaluation over Time, Three-Year Rolling a. M&E quality ratings of World Bank lending projects, with the total percentage of projects rated S+ indicated above each bar 100 80 60 36 35 40 40 36 31 29 28 30 30 33 Percent 20 0 20 40 60 80 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=571) (N=659) (N=704) (N=797) (N=914) (N=979) (N=950) (N=868) (N=768) (N=704) b. M&E quality ratings of World Bank lending projects, with the total percentage of volume rated S+ indicated above each bar 100 80 60 48 45 48 45 46 43 43 44 39 42 40 20 Percent 0 20 40 60 80 FY06-08 FY07-09 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=44,170) (V=48,228) (V=57,943) (V=73,752) (V=84,972) (V=91,542) (V=85,829) (V=86,701) (V=77,365) (V=71,623) Source: Independent Evaluation Group data. Note: FY = fiscal year (of project closing); M&E = monitoring and evaluation; S+ = substantial or above; V = total volume of projects ($, millions). All dollar amounts are U.S. dollars unless otherwise indicated. 9 Appendix A Additional Information on World Bank Performance Table A.5. World Bank Projects: Quality of Monitoring and Evaluation Ratings over Time, Three-Year Rolling a. Percentage distribution of M&E quality ratings of World Bank lending projects, by percentage of projects Rating FY06–08 FY07–09 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 FY15–17 High (%) 4 3 3 2 2 1 1 1 2 2 Substantial (%) 33 32 28 27 26 29 29 32 33 38 Modest (%) 46 47 54 56 56 55 56 56 56 52 Negligible (%) 17 17 16 16 15 15 14 11 10 8 Substantial or above (%) 36 36 31 29 28 30 30 33 35 40 Total (no.) 571 659 704 797 914 979 950 868 768 704 b. Percentage distribution of M&E quality ratings of World Bank lending projects, by percentage of volume ($, millions) Rating FY06–08 FY07–09 FY08–10 FY09–11 FY10–12 FY11–13 FY12–14 FY13–15 FY14–16 FY15–17 High (%) 3 5 7 7 5 3 4 4 5 2 Substantial (%) 43 43 38 39 38 40 35 38 39 46 Modest (%) 39 40 44 42 44 45 52 54 52 48 Negligible (%) 16 12 11 12 13 12 9 5 4 4 Substantial or above (%) 45 48 45 46 43 43 39 42 44 48 Total (no.) 44,170 48,228 57,943 73,752 84,972 91,542 85,829 86,701 77,365 71,623 Source: Independent Evaluation Group data. Note: FY = fiscal year (of project closing); M&E = monitoring and evaluation. All dollar amounts are U.S. dollars unless otherwise indicated. 10 Appendix A Additional Information on World Bank Performance Figure A.6. World Bank Projects: Outcome Ratings over Time, Year by Year a. Outcome ratings of World Bank lending projects, with the total percentage projects rated MS+ indicated above each bar 100 81 82 78 72 76 80 73 71 70 70 71 68 67 60 40 Percent 20 0 20 40 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 (N=281) (N=238) (N=234) (N=241) (N=274) (N=316) (N=350) (N=329) (N=282) (N=258) (N=228) (N=218) b. Outcome ratings of World Bank lending projects, with the total percentage of volume rated MS+ indicated above each bar 100 87 88 88 81 81 83 80 82 81 79 80 78 75 60 40 Percent 20 0 20 40 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 (V=22,880)(V=16,304)(V=18,317)(V=17,955)(V=25,258)(V=33,082)(V=29,542)(V=31,239)(V=26,548)(V=29,070)(V=21,452)(V=21,101) Source: Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); MS+ = moderately satisfactory or above; V = total volume of projects ($, millions). All dollar amounts are U.S. dollars unless otherwise indicated. 11 Appendix A Additional Information on World Bank Performance Table A.6. World Bank Projects: Outcome Ratings over Time, Year by Year a. Percentage distribution of outcome ratings of World Bank lending projects, by percentage of projects Rating FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 Highly satisfactory (%) 2 3 4 2 2 1 2 2 1 2 3 3 Satisfactory (%) 43 41 31 27 23 22 20 19 18 26 27 35 Moderately satisfactory (%) 36 33 38 42 43 49 47 46 50 49 41 44 Moderately unsatisfactory (%) 10 15 19 19 20 18 22 23 21 15 14 9 Unsatisfactory (%) 8 7 9 10 12 8 7 8 8 8 14 6 Highly unsatisfactory (%) 1 0 0 0 0 2 1 1 2 1 1 2 Moderately satisfactory or above (%) 81 78 73 71 68 72 70 67 70 76 71 82 Projects with ratings (no.) 281 238 234 241 274 316 350 329 282 258 228 218 b. Percentage distribution of outcome ratings of World Bank lending projects, by percentage of volume Rating FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 Highly satisfactory (%) 3 4 7 3 3 2 2 3 2 0 6 3 Satisfactory (%) 49 50 39 43 32 29 25 26 32 36 38 40 Moderately satisfactory (%) 35 27 33 35 47 49 47 53 48 52 36 46 Moderately unsatisfactory (%) 8 15 20 14 12 11 18 13 16 9 11 9 Unsatisfactory (%) 4 4 2 5 5 9 7 4 2 3 10 2 Highly unsatisfactory (%) 0 0 0 0 0 0 0 1 1 0 0 0 Moderately satisfactory or above (%) 87 81 78 81 83 80 75 82 81 88 79 88 Total volume of projects with ratings 22,880 16,304 18,317 17,955 25,258 33,082 29,542 31,239 26,548 29,070 21,452 21,101 ($, millions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). All dollar amounts are U.S. dollars unless otherwise indicated. 12 Appendix A Additional Information on World Bank Performance Figure A.7. World Bank Projects: Quality at Entry Ratings over Time, Year by Year a. Quality at entry ratings of World Bank lending projects, with the total percentage of projects rated MS+ indicated above each bar 100 78 74 80 75 70 65 66 63 59 58 61 59 58 60 40 Percent 20 0 20 40 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 (N=280) (N=239) (N=235) (N=236) (N=267) (N=319) (N=346) (N=331) (N=281) (N=258) (N=228) (N=217) b. Quality at entry ratings of World Bank lending projects, with the total percentage of volume rated MS+ indicated above each bar 100 85 81 81 86 79 77 74 80 71 72 70 71 68 60 40 Percent 20 0 20 40 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 (V=22,474) (V=16,084) (V=18,128) (V=17,321) (V=24,854) (V=34,168) (V=29,820) (V=30,791) (V=26,823) (V=29,070) (V=21,452) (V=21,072) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = FY (of project closing); MS+ = moderately satisfactory or above; V = total volume of projects ($, millions). All dollar amounts are U.S. dollars unless otherwise indicated. 13 Appendix A Additional Information on World Bank Performance Table A.7. World Bank Projects: Quality at Entry Ratings over Time, Year by Year a. Percentage distribution of quality at entry ratings of World Bank lending projects, by percentage of projects Rating FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 Highly satisfactory (%) 3 6 4 2 1 2 3 1 1 0 1 1 Satisfactory (%) 58 42 34 29 25 24 19 19 16 24 16 22 Moderately satisfactory (%) 18 27 37 39 36 34 37 39 40 41 44 44 Moderately unsatisfactory (%) 7 15 17 22 26 31 33 33 32 27 31 30 Unsatisfactory (%) 15 10 9 6 11 9 8 9 10 7 7 3 Highly unsatisfactory (%) 0 0 0 1 0 1 1 0 0 1 0 0 Moderately satisfactory or above (%) 78 74 75 70 63 59 59 58 58 65 61 66 Projects with ratings (no.) 280 239 235 236 267 319 346 331 281 258 228 217 b. Percentage distribution of quality at entry ratings of World Bank lending projects, by percentage of volume Rating FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 Highly satisfactory (%) 3 12 5 4 4 3 6 2 2 0 0 1 Satisfactory (%) 63 50 45 45 33 36 24 23 21 37 28 23 Moderately satisfactory (%) 20 19 31 37 34 29 41 52 49 42 46 46 Moderately unsatisfactory (%) 8 12 15 11 22 24 22 19 24 17 18 28 Unsatisfactory (%) 7 7 3 3 7 8 6 4 4 3 8 2 Highly unsatisfactory (%) 0 0 0 0 0 0 0 0 0 0 0 0 Moderately satisfactory or above (%) 85 81 81 86 71 68 71 77 72 79 74 70 Total volume of projects with ratings 22,474 16,084 18,128 17,321 24,854 34,168 29,820 30,791 26,823 29,070 21,452 21,072 ($, millions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). All dollar amounts are U.S. dollars unless otherwise indicated. 14 Appendix A Additional Information on World Bank Performance Figure A.8. World Bank Projects: Quality of Supervision Ratings over Time, Year by Year a. Quality of supervision ratings of World Bank lending projects, with the total percentage of projects rated MS+ indicated above each bar 100 88 87 84 78 80 84 79 77 79 78 76 80 76 60 40 Percent 20 0 20 40 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 (N=275) (N=236) (N=231) (N=235) (N=265) (N=313) (N=344) (N=328) (N=281) (N=258) (N=228) (N=216) b. Quality of supervision ratings of World Bank lending projects, with the total percentage of volume rated MS+ indicated above each bar 100 92 92 89 91 89 85 88 86 83 83 83 84 80 60 Percent 40 20 0 20 40 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 (V=22,039) (V=15,683) (V=17,486) (V=17,321) (V=24,704) (V=31,715) (V=29,820) (V=30,539) (V=26,823) (V=29,070) (V=21,452) (V=21,017) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = project closing fiscal year; MS+ = moderately satisfactory or above; V = total volume of projects ($, millions). All dollar amounts are U.S. dollars unless otherwise indicated. 15 Appendix A Additional Information on World Bank Performance Table A.8. World Bank Projects: Quality of Supervision Ratings over Time, Year by Year a. Percentage distribution of quality of supervision ratings of World Bank lending projects, by percentage of projects Rating FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 Highly satisfactory (%) 4 6 4 3 4 3 3 2 0 1 4 3 Satisfactory (%) 69 60 53 48 37 38 31 27 30 34 31 35 Moderately satisfactory (%) 15 21 27 28 37 39 44 46 48 45 41 47 Moderately unsatisfactory (%) 3 8 11 17 16 14 18 19 15 15 18 12 Unsatisfactory (%) 8 5 4 4 6 6 3 6 6 5 6 3 Highly unsatisfactory (%) 0 0 1 0 0 0 0 0 0 0 0 0 Moderately satisfactory or above (%) 88 87 84 79 77 79 78 76 78 80 76 84 Projects with ratings (no.) 275 236 231 235 265 313 344 328 281 258 228 216 b. Percentage distribution of quality of supervision ratings of World Bank lending projects, by percentage of volume Rating FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 Highly satisfactory (%) 5 13 7 3 7 4 5 4 0 1 4 3 Satisfactory (%) 73 56 56 64 41 35 38 45 47 47 41 41 Moderately satisfactory (%) 14 24 26 18 35 44 41 39 42 43 38 43 Moderately unsatisfactory (%) 4 5 9 13 15 8 11 9 8 7 11 12 Unsatisfactory (%) 4 3 2 2 3 8 5 3 3 2 5 2 Highly unsatisfactory (%) 0 0 0 0 0 0 0 0 0 0 0 0 Moderately satisfactory or above (%) 92 92 89 85 83 83 83 88 89 91 84 86 Total volume of projects with ratings 22,039 15,683 17,486 17,321 24,704 31,715 29,820 30,539 26,823 29,070 21,452 21,017 ($, millions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). All dollar amounts are U.S. dollars unless otherwise indicated. 16 Appendix A Additional Information on World Bank Performance Figure A.9. World Bank Projects: Bank Performance Ratings over Time, Year by Year a. Bank performance ratings of World Bank lending projects, with the total percentage of projects rated MS+ indicated above each bar 100 84 83 80 78 75 76 80 68 70 71 69 70 70 60 40 Percent 20 0 20 40 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 (N=283) (N=243) (N=240) (N=244) (N=274) (N=325) (N=354) (N=334) (N=283) (N=258) (N=228) (N=218) b. Bank performance ratings of World Bank lending projects, with the total percentage of volume rated MS+ indicated above each bar 100 90 88 89 84 85 84 80 83 82 79 80 73 78 60 40 Percent 20 0 20 40 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 (V=23,019) (V=16,443) (V=18,377) (V=17,965) (V=25,234) (V=34,695) (V=30,743) (V=31,493) (V=26,849) (V=29,070) (V=21,452) (V=21,101) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); MS+ = moderately satisfactory or above; V = total volume of projects ($, millions). All dollar amounts are U.S. dollars unless otherwise indicated. 17 Appendix A Additional Information on World Bank Performance Table A.9. World Bank Projects: Bank Performance Ratings over Time, Year by Year a. Percentage distribution of Bank performance ratings of World Bank lending projects, by percentage of projects Rating FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 Highly satisfactory (%) 2 4 3 1 1 2 1 1 0 0 Satisfactory (%) 65 47 37 29 23 22 15 16 13 18 Moderately satisfactory (%) 16 29 38 45 45 46 55 52 57 58 Moderately unsatisfactory (%) 6 14 17 20 22 21 24 23 22 17 Unsatisfactory (%) 10 5 5 5 9 8 5 8 8 7 Highly unsatisfactory (%) 0 0 1 0 0 1 0 0 0 1 Moderately satisfactory or above (%) 84 80 78 75 68 70 71 69 70 76 Projects with ratings (no.) 283 243 240 244 274 325 354 334 283 258 b. Percentage distribution of Bank performance ratings of World Bank lending projects, by percentage of volume Rating FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 Highly satisfactory (%) 2 11 4 2 2 3 2 3 0 0 Satisfactory (%) 70 49 45 46 29 29 22 20 19 28 Moderately satisfactory (%) 19 24 36 35 42 45 56 59 64 60 Moderately unsatisfactory (%) 5 14 13 15 24 14 15 14 14 9 Unsatisfactory (%) 5 2 2 2 3 8 5 3 3 3 Highly unsatisfactory (%) 0 0 0 0 0 1 0 0 0 0 Moderately satisfactory or above (%) 90 84 85 84 73 78 80 83 82 88 Total volume of projects with ratings 23,019 16,443 18,377 17,965 25,234 34,695 30,743 31,493 26,849 29,070 ($, millions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). All dollar amounts are U.S. dollars unless otherwise indicated. 18 Appendix A Additional Information on World Bank Performance Figure A.10. World Bank Projects: Ratings of Quality of Monitoring and Evaluation over Time, Year by Year a. M&E quality ratings of World Bank lending projects, with the total percentage rated S+ indicated above each bar 60 46 40 36 38 37 40 31 30 30 32 29 26 29 20 0 Percent 20 40 60 80 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 (N=135) (N=218) (N=218) (N=223) (N=263) (N=311) (N=340) (N=328) (N=282) (N=258) (N=228) (N=218) b. M&E quality ratings of World Bank lending projects, with the total percentage rated S+ indicated above each bar 60 55 49 51 45 48 48 46 37 39 37 39 41 40 20 0 Percent 20 40 60 80 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 (V=12,921) (V=14,502) (V=16,747) (V=16,979) (V=24,217) (V=32,555) (V=28,199) (V=30,788) (V=26,843) (V=29,070) (V=21,452) (V=21,101) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); M&E = monitoring and evaluation; S+ = substantial or above; V = total volume of projects ($, millions). All dollar amounts are U.S. dollars unless otherwise indicated. 19 Appendix A Additional Information on World Bank Performance Table A.10. World Bank Projects: Ratings of Quality of Monitoring and Evaluation over Time, Year by Year a. Percentage distribution of M&E quality ratings of World Bank lending projects, by percentage of projects Rating FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 High (%) 2 6 3 2 3 1 2 0 1 2 3 2 Substantial (%) 29 34 33 28 23 28 27 32 28 37 34 44 Modest (%) 50 42 48 51 60 56 54 54 60 54 53 49 Negligible (%) 19 18 16 18 14 15 17 14 11 8 10 5 Substantial or above (%) 31 40 36 30 26 30 29 32 29 38 37 46 Projects with ratings (no.) 135 218 218 223 263 311 340 328 282 258 228 218 b. Percentage distribution of M&E quality ratings of World Bank lending projects, by percentage of volume Rating FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 High (%) 2 4 2 9 9 5 2 0 12 1 3 1 Substantial (%) 36 43 47 39 31 46 35 39 30 44 43 54 Modest (%) 38 40 38 41 51 36 47 55 54 52 48 43 Negligible (%) 24 13 13 12 9 13 16 7 5 3 6 2 Substantial or above (%) 37 48 49 48 39 51 37 39 41 45 46 55 Total volume of projects with ratings 12,921 14,502 16,747 16,979 24,217 32,555 28,199 30,788 26,843 29,070 21,452 21,101 ($, millions) Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); M&E = monitoring and evaluation. All dollar amounts are U.S. dollars unless otherwise indicated. 20 Appendix A Additional Information on World Bank Performance Figure A.11. Country Strategy Development Outcome Ratings, by Region a. Percentage distribution of development outcome ratings in CLR reviews, with the total percentage rated moderately satisfactory or above indicated on the right, FY07–18 World Bank–wide (N=219) 62 Europe and Central Asia (N=54) 76 South Asia (N=12) 67 Latin America and Caribbean (N=58) 66 East Asia and Pacific (N=21) 62 Middle East and North Africa (N=13) 54 Africa (N=61) 48 b. Percentage distribution of development outcome ratings in CLR reviews, with the total percentage rated moderately satisfactory or above indicated on the right, FY14–18 World Bank–wide (N=92) 68 Europe and Central Asia (N=23) 91 South Asia (N=4) 75 Latin and Caribbean (N=25) 72 Middle East and North Africa (N=6) 67 East Asia and Pacific (N=10) 50 Africa (N=24) 50 Source: Independent Evaluation Group Completion and Learning Review ratings data. Note: CLR = Completion and Learning Review; FY = fiscal year; N = number of CLRs. 21 Appendix A Additional Information on World Bank Performance Table A.11. Country Strategy Development Outcome Ratings, by Region (percent) a. Percentage distribution of development outcome ratings in CLR reviews, FY07–18 Highly Moderately Moderately Region Unsatisfactory Unsatisfactory Unsatisfactory Satisfactory Satisfactory MS+ World Bank–wide (N = 219) 0 6 31 56 6 62 Europe and Central Asia (N = 54) 0 2 22 65 11 76 South Asia (N = 12) 0 0 33 67 0 67 Latin America and Caribbean (N = 58) 0 3 31 62 3 66 East Asia and Pacific (N = 21) 5 10 24 43 19 62 Middle East and North Africa (N = 13) 0 8 38 54 0 54 Africa (N = 61) 0 13 39 44 3 48 b. Percentage distribution of development outcome ratings in CLR reviews, FY14–18 Highly Moderately Moderately Region Unsatisfactory Unsatisfactory Unsatisfactory Satisfactory Satisfactory MS+ World Bank–wide (N = 92) 1 12 18 62 7 68 Europe and Central Asia (N = 23) 0 4 4 78 13 91 South Asia (N = 4) 0 0 25 75 0 75 Latin America and Caribbean (N = 25) 0 8 20 68 4 72 Middle East and North Africa (N = 6) 0 17 17 67 0 67 East Asia and Pacific (N = 10) 10 10 30 40 10 50 Africa (N = 24) 0 25 25 46 4 50 Source: Independent Evaluation Group Completion and Learning Review ratings data. Note: CLR = Completion and Learning Review; FY = fiscal year; MS+ = moderately satisfactory or above; N = number of CLRs. 22 Appendix A Additional Information on World Bank Performance Figure A.12. Country Strategy World Bank Group Performance Ratings, by Region a. Percentage distribution of World Bank Group performance ratings in CLR reviews, with the total percentage rated good or superior indicated on the right, FY07–18 World Bank–wide (N=219) 69 Europe and Central Asia (N=54) 81 East Asia and Pacific (N=21) 76 South Asia (N=12) 75 Middle East and North Africa (N=13) 69 Latin and Caribbean (N=58) 64 Africa (N=61) 59 b. Percentage distribution of World Bank Group performance ratings in CLR reviews, with the total percentage rated good or superior indicated on the right, FY14–18 World Bank–wide (N=92) 62 Europe and Central Asia (N=23) 83 East Asia and Pacific (N=10) 70 Middle East and North Africa (N=6) 67 Latin and Caribbean (N=25) 60 South Asia (N=4) 50 Africa (N=24) 42 Source: Independent Evaluation Group Completion and Learning Review ratings data. Note: CLR = Completion and Learning Review; FY = fiscal year; N = number of CLRs. 23 Appendix A Additional Information on World Bank Performance Table A.12. Country Strategy World Bank Group Performance Ratings, by Region (percent) a. Percentage distribution of World Bank Group performance ratings in CLR reviews, FY07–18 Region Poor Fair Good Superior GS World Bank–wide (N = 219) 2 29 55 14 69 Europe and Central Asia (N = 54) 0 19 59 22 81 East Asia and Pacific (N = 21) 0 24 62 14 76 South Asia (N = 12) 0 25 67 8 75 Middle East and North Africa (N = 13) 0 31 46 23 69 Latin and Caribbean (N = 58) 0 36 55 9 64 Africa (N = 61) 7 34 48 11 59 b. Percentage distribution of World Bank Group performance ratings in CLR reviews, FY14–18 Region Poor Fair Good Superior GS World Bank–wide (N = 92) 2 36 59 3 62 Europe and Central Asia (N = 23) 0 17 74 9 83 East Asia and Pacific (N = 10) 0 30 70 0 70 Middle East and North Africa (N = 6) 0 33 67 0 67 Latin and Caribbean (N = 25) 0 40 60 0 60 South Asia (N = 4) 0 50 50 0 50 Africa (N = 24) 8 50 38 4 42 Source: Independent Evaluation Group Completion and Learning Review ratings data. Note: CLR = Completion and Learning Review; FY = fiscal year; GS = good or superior; N = number of CLRs. 24 Appendix A Additional Information on World Bank Performance Figure A.13. Country Strategy Development Outcome Ratings, by Lending Category a. Percentage distribution of World Bank Group development outcome ratings in CLR reviews with the total percentage rated moderately satisfactory or above indicated on the right, FY07–18a World Bank–wide (N=219) 62 IDA (N=118) 52 IBRD (N=98) 74 FCS (N=25) 44 Non-FCS (N=194) 64 b. Percentage distribution of World Bank Group development outcome ratings in CLR reviews, with the total percentage rated moderately satisfactory or above indicated on the right, FY14–18a World Bank–wide (N=92) 68 IDA (N=48) 56 IBRD (N=43) 81 FCS (N=13) 46 Non-FCS (N=79) 72 Source: Independent Evaluation Group Completion and Learning Review ratings data. Note: CLR = Completion and Learning Review; FCS = fragile and conflict-affected situation; FY = fiscal year; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; N = number of CLRs. a The total number of IBRD and IDA country results is one less than the total number of CLR reviews because Antigua and Barbuda are categorized as Other in the World Bank lending category. 25 Appendix A Additional Information on World Bank Performance Table A.13. Country Strategy Development Outcome Ratings, by Lending Category (percent) a. Percentage distribution of World Bank Group development outcome ratings in CLR reviews, FY07–18 Highly Moderately Moderately Lending Category Unsatisfactory Unsatisfactory Unsatisfactory Satisfactory Satisfactory MS+ World Bank–wide (N = 219) 0 6 31 56 6 62 IDA (N = 118) 1 8 40 47 5 52 IBRD (N = 98) 0 5 20 67 7 74 FCS (N = 25) 4 20 32 44 0 44 Non-FCS (N = 194) 0 5 31 57 7 64 b. Percentage distribution of development outcome ratings in CLR reviews, FY14–18 Highly Moderately Moderately Lending Category Unsatisfactory Unsatisfactory Unsatisfactory Satisfactory Satisfactory MS+ World Bank–wide (N = 92) 1 12 18 62 7 68 IDA (N = 48) 2 13 29 52 4 56 IBRD (N = 43) 0 12 7 72 9 81 FCS (N = 13) 8 31 15 46 0 46 Non-FCS (N = 79) 0 9 19 65 8 72 Source: Independent Evaluation Group Completion and Learning Review ratings data. Note: CLR = Completion and Learning Review; FCS = fragile and conflict-affected situation; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; MS+ = moderately satisfactory or above; N = number of CLRs. 26 Appendix A Additional Information on World Bank Performance Figure A.14. Country Strategy World Bank Group Performance Ratings, by Lending Category a. Percentage distribution of World Bank Group performance ratings in CLR reviews, with the total percentage good or superior indicated on the right, FY07-18 World Bank–wide (N=219) 69 IDA (N=118) 68 IBRD (N=98) 70 FCS (N=25) 52 Non-FCS (N=194) 71 b. Percentage distribution of World Bank group performance ratings in CLR reviews, with the total percentage rated good or superior indicated on the right, FY14–18 World Bank–wide (N=92) 62 IDA (N=48) 63 IBRD (N=43) 60 FCS (N=13) 54 Non-FCS (N=79) 63 Source: Independent Evaluation Group Completion and Learning Review ratings data. Note: CLR = Completion and Learning Review; FCS = fragile and conflict-affected situation; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; N = number of CLRs. 27 Appendix A Additional Information on World Bank Performance Table A.14. Country Strategy World Bank Group Performance Ratings, by Lending Category a. Percentage distribution of World Bank Group performance ratings in CLR reviews, FY07–18 Lending Category Poor Fair Good Superior GS World Bank–wide (N = 219) 2 29 55 14 69 IDA (N = 118) 3 30 53 14 68 IBRD (N = 98) 1 29 57 13 70 FCS (N = 25) 8 40 40 12 52 Non-FCS (N = 194) 8 28 57 14 71 b. Percentage distribution of World Bank group performance ratings in CLR reviews, FY14–18 Lending Category Poor Fair Good Superior GS World Bank–wide (N = 92) 2 36 59 3 62 IDA (N = 48) 4 33 60 2 63 IBRD (N = 43) 0 40 56 5 60 FCS (N = 13) 8 38 54 0 54 Non-FCS (N = 79) 1 34 59 4 63 Source: Independent Evaluation Group Completion and Learning Review ratings data. Note: CLR = Completion and Learning Review; FCS = fragile and conflict-affected situation; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; GS = good or superior; N = number of CLR reviews. 28 Appendix A Additional Information on World Bank Performance Figure A.15. World Bank Projects: Outcome Ratings, by Regiona a. Projects rated moderately satisfactory or above, by number FY12-14 FY15-17 World Bank–wide East Asia and Pacific Europe and Central Asia South Asia Africa Latin America and Caribbean Middle East and North Africa 0 20 40 60 80 100 Percent b. Projects rated moderately satisfactory or above, by volume FY12-14 FY15-17 World Bank–wide East Asia and Pacific Europe and Central Asia South Asia Middle East and North Africa Africa Latin America and Caribbean 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). a World Bank–wide includes projects not tagged to a region. 29 Appendix A Additional Information on World Bank Performance Table A.15. World Bank Projects: Outcome Ratings, by Region a. Projects rated moderately satisfactory or above, by number Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Region (%) (%) (no.) (no.) World Bank–widea 69 76 961 704 East Asia and Pacific 67 88 163 121 Europe and Central Asia 76 85 157 104 South Asia 77 82 97 83 Africa 61 71 306 199 Latin America and the Caribbean 76 69 157 134 Middle East and North Africa 63 67 80 61 b. Projects rated moderately satisfactory or above, by volume Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Region (%) (%) ($, millions) ($, millions) World Bank–widea 80 86 87,329 71,623 East Asia and Pacific 75 96 17,662 12,672 Europe and Central Asia 88 95 13,247 14,574 South Asia 86 91 13,995 11,927 Middle East and North Africa 61 80 5,425 4,185 Africa 63 77 15,433 14,190 Latin America and the Caribbean 90 73 21,562 14,060 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); MS+ = moderately satisfactory or above. a World Bank–wide includes projects not tagged to a region. 30 Appendix A Additional Information on World Bank Performance Figure A.16. World Bank Projects: Quality at Entry Ratings, by Regiona a. Projects rated moderately satisfactory or above, by number FY12-14 FY15-17 World Bank–wide East Asia and Pacific Europe and Central Asia South Asia Middle East and North Africa Latin America and Caribbean Africa 0 20 40 60 80 100 Percent b. Projects rated moderately satisfactory or above, by volume FY12-14 FY15-17 World Bank–wide Europe and Central Asia South Asia East Asia and Pacific Latin America and Caribbean Middle East and North Africa Africa 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). a World Bank–wide includes projects not tagged to a region. 31 Appendix A Additional Information on World Bank Performance Table A.16. World Bank Projects: Quality at Entry Ratings, by Region a. Projects rated moderately satisfactory or above, by number Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Region (%) (%) (no.) (no.) World Bank–widea 58 64 958 703 East Asia and Pacific 60 79 160 121 Europe and Central Asia 62 73 154 103 South Asia 66 71 95 83 Africa 60 61 81 61 Latin America and the Caribbean 60 60 160 134 Middle East and North Africa 52 52 307 199 b. Projects rated moderately satisfactory or above, by volume Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Region (%) (%) ($, millions) ($, millions) World Bank–widea 73 75 87,435 71,594 East Asia and Pacific 72 85 13,030 14,545 Europe and Central Asia 75 85 14,685 11,927 South Asia 70 85 17,062 12,672 Africa 84 70 21,857 14,060 Latin America and the Caribbean 75 68 5,425 4,185 Middle East and North Africa 60 54 15,370 14,190 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = Fiscal year (of project closing); MS+ = moderately satisfactory or above. a World Bank–wide includes projects not tagged to a region. 32 Appendix A Additional Information on World Bank Performance Figure A.17. World Bank Projects: Quality of Supervision Ratings, by Regiona a. Projects rated moderately satisfactory or above, by number FY12-14 FY15-17 World Bank–wide Europe and Central Asia East Asia and Pacific South Asia Latin America and Caribbean Africa Middle East and North Africa 0 20 40 60 80 100 Percent b. Projects rated moderately satisfactory or above, by volume FY12-14 FY15-17 World Bank–wide Europe and Central Asia South Asia East Asia and Pacific Latin America and Caribbean Middle East and North Africa Africa 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). a World Bank–wide includes projects not tagged to a region. 33 Appendix A Additional Information on World Bank Performance Table A.17. World Bank Projects: Quality of Supervision Ratings, by Region a. Projects rated moderately satisfactory or above, by number Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Region (%) (%) (no.) (no.) World Bank–widea 77 80 953 702 East Asia and Pacific 78 89 154 103 Europe and Central Asia 76 89 160 121 South Asia 78 87 94 83 Africa 78 77 158 134 Latin America and the Caribbean 77 72 305 198 Middle East and North Africa 77 70 81 61 b. Projects rated moderately satisfactory or above, by volume Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Region (%) (%) ($, millions) ($, millions) World Bank–widea 87 88 87,182 71,539 East Asia and Pacific 92 97 13,030 14,545 Europe and Central Asia 87 94 14,435 11,927 South Asia 80 92 17,062 12,672 Africa 94 87 21,856 14,060 Latin America and the Caribbean 84 81 5,425 4,185 Middle East and North Africa 81 71 15,368 14,135 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). a World Bank–wide includes projects not tagged to a region. 34 Appendix A Additional Information on World Bank Performance Figure A.18. World Bank Projects: Bank Performance Ratings, by Regiona a. Projects rated moderately satisfactory or above, by number FY12-14 FY15-17 World Bank–wide East Asia and Pacific Europe and Central Asia South Asia Latin America and Caribbean Middle East and North Africa Africa 0 20 40 60 80 100 Percent b. Projects rated moderately satisfactory or above, by volume FY12-14 FY15-17 World Bank–wide Europe and Central Asia East Asia and Pacific South Asia Middle East and North Africa Africa Latin America and Caribbean 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). a World Bank-wide includes projects not tagged to a region. 35 Appendix A Additional Information on World Bank Performance Table A.18. World Bank Projects: Bank Performance Ratings, by Region a. Projects rated moderately satisfactory or above, by number Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Region (%) (%) (no.) (no.) World Bank–widea 70 76 971 704 East Asia and Pacific 68 89 164 121 Europe and Central Asia 75 85 157 104 South Asia 80 84 99 83 Africa 73 69 160 134 Latin America and the Caribbean 70 69 81 61 Middle East and North Africa 63 67 309 199 b. Projects rated moderately satisfactory or above, by volume Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Region (%) (%) ($, millions) ($, millions) World Bank–widea 82 86 89,085 71,623 East Asia and Pacific 87 97 13,247 14,574 Europe and Central Asia 76 96 17,662 12,672 South Asia 89 94 15,445 11,927 Africa 72 81 5,425 4,185 Latin America and the Caribbean 68 73 15,437 14,190 Middle East and North Africa 90 72 21,864 14,060 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year of project closing. a World Bank-wide includes projects not tagged to a region. 36 Appendix A Additional Information on World Bank Performance Figure A.19. World Bank Projects: M&E Quality Ratings, by Regiona a Projects rated substantial or above, by number FY12-14 FY15-17 World Bank–wide Europe and Central Asia East Asia and Pacific South Asia Middle East and North Africa Latin America and Caribbean Africa 0 20 40 60 80 100 Percent b. Projects rated substantial or above, by volume FY12-14 FY15-17 World Bank–wide Europe and Central Asia East Asia and Pacific South Asia Middle East and North Africa Latin America and Caribbean Africa 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). a World Bank–wide includes projects not tagged to a region. 37 Appendix A Additional Information on World Bank Performance Table A.19. World Bank Projects: M&E Quality Ratings, by Region a. Projects rated substantial or above, by number Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Region (%) (%) (no.) (no.) World Bank–widea 30 40 950 704 East Asia and Pacific 34 50 154 104 Europe and Central Asia 29 48 161 121 South Asia 37 42 93 83 Africa 30 41 80 61 Latin America and the Caribbean 28 37 158 134 Middle East and North Africa 27 32 303 199 b. Projects rated substantial or above, by volume Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Region (%) (%) ($, millions) ($, millions) World Bank–widea 39 48 85,829 71,623 East Asia and Pacific 37 61 13,030 14,574 Europe and Central Asia 30 59 17,082 12,672 South Asia 46 48 13,170 11,927 Africa 40 47 5,425 4,185 Latin America and the Caribbean 47 39 21,856 14,060 Middle East and North Africa 32 36 15,260 14,190 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year of project closing. a World Bank–wide includes projects not tagged to a region. 38 Appendix A Additional Information on World Bank Performance Figure A.20. World Bank Projects: Outcome Ratings, by Global Practicea a. Projects rated moderately satisfactory or above, by number FY12-14 FY15-17 World Bank–wide Poverty and Equity Social Protection and Labor Education Transport and Digital Development Social, Urban, Rural, and Resilience Agriculture Finance, Competitiveness, and Innovation Health, Nutrition, and Population Environment and Natural Resources Macroeconomics, Trade, and Investment Energy and Extractives Water Governance Trade and Competitiveness 0 20 40 60 80 100 Percent b. Projects rated moderately satisfactory or above, by volume FY12-14 FY15-17 World Bank–wide Poverty and Equity Social Protection and Labor Education Finance, Competitiveness, and Innovation Governance Social, Urban, Rural, and Resilience Macroeconomics, Trade, and Investment Trade and Competitiveness Energy and Extractives Environment and Natural Resources Transport and Digital Development Agriculture Health, Nutrition, and Population Water 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year of project closing. a World Bank–wide includes projects not tagged to a Global Practice. 39 Appendix A Additional Information on World Bank Performance Table A.20. World Bank Projects: Outcome Ratings, by Global Practice a. Projects rated moderately satisfactory or above, by number Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Global Practice (%) (%) (no.) (no.) World Bank–widea 69 76 961 704 Poverty and Equity 50 100 4 5 Social Protection and Labor 89 93 44 29 Education 65 89 89 53 Transport and Digital Development 73 84 84 83 Social, Urban, Rural, and Resilience 77 80 121 90 Agriculture 70 75 74 73 Finance, Competitiveness, and Innovation 71 75 56 24 Health, Nutrition, and Population 78 75 74 52 Environment and Natural Resources 59 74 63 61 Macroeconomics, Trade, and Investment 68 73 106 41 Energy and Extractives 65 70 92 61 Water 61 67 69 67 Governance 54 64 68 44 Trade and Competitiveness 69 62 13 21 40 Appendix A Additional Information on World Bank Performance b. Projects rated moderately satisfactory or above, by volume Projects Rated MS+ FY12–14 FY15–17 FY12–14 FY15–17 Global Practice (%) (%) ($, millions) ($, millions) World Bank–widea 80 86 87,329 71,623 Poverty and Equity 97 100 557 738 Social Protection and Labor 95 100 6,816 4,054 Education 81 94 7,188 4,047 Finance, Competitiveness, and Innovation 93 94 6,207 4,550 Governance 63 91 4,508 2,876 Social, Urban, Rural, and Resilience 82 90 9,930 7,977 Macroeconomics, Trade, and Investment 81 88 14,195 10,720 Trade and Competitiveness 53 86 986 475 Energy and Extractives 81 83 8,861 6,264 Environment and Natural Resources 85 82 2,609 1,831 Transport and Digital Development 69 82 10,605 13,287 Agriculture 82 81 3,328 5,143 Health, Nutrition, and Population 82 80 5,875 3,915 Water 63 67 5,552 5,745 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); MS+ = moderately satisfactory or above. Although the FY18 restructuring of the Equitable Growth, Finance, and Institutions Practice Group phased out the Trade and Competitiveness (T&C) Global Practice, some projects mapped to T&C remained in World Bank business systems as of October 3, 2018, the data cutoff date for this report. Therefore, T&C is still reported as a Global Practice in this report. Some T&C- mapped projects are expected to be remapped to Finance, Competitiveness, and Innovation; and others, to Macroeconomics, Trade, and Investment. a World Bank–wide includes projects not tagged to a Global Practice. 41 Appendix A Additional Information on World Bank Performance Figure A.21. World Bank Projects: Quality at Entry Ratings, by Global Practicea a. Projects rated moderately satisfactory or above, by number FY12-14 FY15-17 World Bank–wide Poverty and Equity Macroeconomics, Trade, and Investment Social Protection and Labor Finance, Competitiveness, and Innovation Transport and Digital Development Education Energy and Extractives Health, Nutrition, and Population Social, Urban, Rural, and Resilience Agriculture Water Environment and Natural Resources Governance Trade and Competitiveness 0 20 40 60 80 100 Percent b. Projects rated moderately satisfactory or above, by volume FY12-14 FY15-17 World Bank–wide Poverty and Equity Social Protection and Labor Macroeconomics, Trade, and Investment Finance, Competitiveness, and Innovation Governance Education Energy and Extractives Social, Urban, Rural, and Resilience Agriculture Environment and Natural Resources Health, Nutrition, and Population Water Transport and Digital Development Trade and Competitiveness 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal Year of project closing. a World Bank–wide includes projects not tagged to a Global Practice. 42 Appendix A Additional Information on World Bank Performance Table A.21. World Bank Projects: Quality at Entry Ratings, by Global Practice a. Projects rated moderately satisfactory or above, by number Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Global Practice (%) (%) (no.) (no.) World Bank–widea 58 64 958 703 Poverty and Equity 50 100 4 5 Macroeconomics, Trade, and Investment 71 80 107 41 Social Protection and Labor 84 79 44 29 Finance, Competitiveness, and Innovation 52 71 56 24 Transport and Digital Development 56 70 85 83 Education 52 70 89 53 Energy and Extractives 57 64 87 61 Health, Nutrition, and Population 64 63 73 52 Social, Urban, Rural, and Resilience 65 63 119 90 Agriculture 55 63 74 73 Water 45 58 69 67 Environment and Natural Resources 49 57 63 60 Governance 48 50 69 44 Trade and Competitiveness 60 43 15 21 43 Appendix A Additional Information on World Bank Performance b. Projects rated moderately satisfactory or above, by volume Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Global Practice (%) (%) ($, millions) ($, millions) World Bank–widea 73 75 87,435 71,594 Poverty and Equity 97 100 557 738 Social Protection and Labor 94 97 7,041 4,054 Macroeconomics, Trade, and Investment 89 89 14,496 10,720 Finance, Competitiveness, and Innovation 80 87 6,207 4,550 Governance 65 83 4,508 2,876 Education 73 76 7,126 4,047 Energy and Extractives 65 73 8,126 6,264 Social, Urban, Rural, and Resilience 62 72 9,600 7,977 Agriculture 68 72 3,328 5,143 Environment and Natural Resources 83 70 2,604 1,802 Health, Nutrition, and Population 81 67 5,560 3,915 Water 48 63 5,552 5,745 Transport and Digital Development 68 62 11,630 13,287 Trade and Competitiveness 34 40 988 475 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year of project closing; MS+ = moderately satisfactory or above. Although the FY18 restructuring of the Equitable Growth, Finance, and Institutions Practice Group phased out the Trade and Competitiveness (T&C) Global Practice, some projects mapped to T&C remained in World Bank business systems as of October 3, 2018, the data cutoff date for this report. Therefore, T&C is still reported as a Global Practice in this report. Some T&C- mapped projects are expected to be remapped to Finance, Competitiveness, and Innovation; and others, to Macroeconomics, Trade, and Investment. a World Bank–wide includes projects not tagged to a Global Practice. 44 Appendix A Additional Information on World Bank Performance Figure A.22. World Bank Projects: Quality of Supervision, by Global Practicea a. Projects rated moderately satisfactory or above, by number FY12-14 FY15-17 World Bank–wide Poverty and Equity Social Protection and Labor Finance, Competitiveness, and Innovation Macroeconomics, Trade, and Investment Education Social, Urban, Rural, and Resilience Trade and Competitiveness Agriculture Energy and Extractives Environment and Natural Resources Transport and Digital Development Health, Nutrition, and Population Water Governance 0 20 40 60 80 100 Percent b. Projects rated moderately satisfactory or above, by volume FY12-14 FY15-17 World Bank–wide Poverty and Equity Social Protection and Labor Finance, Competitiveness, and Innovation Macroeconomics, Trade, and Investment Trade and Competitiveness Governance Energy and Extractives Environment and Natural Resources Social, Urban, Rural, and Resilience Education Agriculture Health, Nutrition, and Population Transport and Digital Development Water 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). a World Bank–wide includes projects not tagged to a Global Practice. 45 Appendix A Additional Information on World Bank Performance Table A.22. World Bank Projects: Quality of Supervision Ratings, by Global Practice a. Projects rated moderately satisfactory or above, by number Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Global Practice (%) (%) (no.) (no.) World Bank–widea 77 80 953 702 Poverty and Equity 75 100 4 5 Social Protection and Labor 91 97 43 29 Finance, Competitiveness, and Innovation 70 92 56 24 Macroeconomics, Trade, and Investment 88 88 107 40 Education 81 87 89 53 Social, Urban, Rural, and Resilience 82 81 119 90 Trade and Competitiveness 92 81 13 21 Agriculture 76 81 74 73 Energy and Extractives 77 80 87 61 Environment and Natural Resources 56 78 62 60 Transport and Digital Development 84 78 85 83 Health, Nutrition, and Population 81 77 72 52 Water 67 69 69 67 Governance 62 66 69 44 46 Appendix A Additional Information on World Bank Performance b. Project rated moderately satisfactory or above, by volume Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Global Practice (%) (%) ($, millions) ($, millions) World Bank–widea 87 88 87,182 71,539 Poverty and Equity 99 100 557 738 Social Protection and Labor 98 99 6,791 4,054 Finance, Competitiveness, and Innovation 92 99 6,207 4,550 Macroeconomics, Trade, and Investment 93 98 14,496 10,665 Trade and Competitiveness 97 97 986 475 Governance 62 91 4,508 2,876 Energy and Extractives 86 91 8,126 6,264 Environment and Natural Resources 88 90 2,603 1,802 Social, Urban, Rural, and Resilience 85 90 9,600 7,977 Education 90 88 7,126 4,047 Agriculture 85 83 3,328 5,143 Health, Nutrition, and Population 84 80 5,560 3,915 Transport and Digital Development 85 78 11,630 13,287 Water 75 70 5,552 5,745 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year of project closing; MS+ = moderately satisfactory or above. Although the FY18 restructuring of the Equitable Growth, Finance, and Institutions Practice Group phased out the Trade and Competitiveness (T&C) Global Practice, some projects mapped to T&C remained in World Bank business systems as of October 3, 2018, the data cutoff date for this report. Therefore, T&C is still reported as a Global Practice in this report. Some T&C- mapped projects are expected to be remapped to Finance, Competitiveness, and Innovation; and others, to Macroeconomics, Trade, and Investment. a World Bank–wide includes projects not tagged to a Global Practice. 47 Appendix A Additional Information on World Bank Performance Figure A.23. World Bank Projects: Bank Performance Ratings, by Global Practicea a. Projects rated moderately satisfactory or above, by number FY12-14 FY15-17 World Bank–wide Poverty and Equity Social Protection and Labor Transport and Digital Development Education Macroeconomics, Trade, and Investment Social, Urban, Rural, and Resilience Agriculture Finance, Competitiveness, and Innovation Health, Nutrition, and Population Environment and Natural Resources Energy and Extractives Water Trade and Competitiveness Governance 0 20 40 60 80 100 120 Percent b. Projects rated moderately satisfactory or above, by volume FY12-14 FY15-17 World Bank–wide Poverty and Equity Social Protection and Labor Finance, Competitiveness, and Innovation Governance Social, Urban, Rural, and Resilience Macroeconomics, Trade, and Investment Education Trade and Competitiveness Energy and Extractives Transport and Digital Development Environment and Natural Resources Agriculture Health, Nutrition, and Population Water 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year of project closing. a World Bank–wide includes projects not tagged to a Global Practice. 48 Appendix A Additional Information on World Bank Performance Table A.23. World Bank Projects: Bank Performance Ratings, by Global Practice a. Projects rated moderately satisfactory or above, by number Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Global Practice (%) (%) (no.) (no.) World Bank–widea 70 76 971 704 Poverty and Equity 50 100 4 5 Social Protection and Labor 91 93 45 29 Transport and Digital Development 69 86 86 83 Education 69 83 90 53 Macroeconomics, Trade, and Investment 76 80 107 41 Social, Urban, Rural, and Resilience 79 80 121 90 Agriculture 70 75 74 73 Finance, Competitiveness, and Innovation 66 75 56 24 Health, Nutrition, and Population 80 75 74 52 Environment and Natural Resources 55 74 64 61 Energy and Extractives 67 72 93 61 Water 62 66 69 67 Trade and Competitiveness 73 62 15 21 Governance 54 59 69 44 49 Appendix A Additional Information on World Bank Performance b. Projects rated moderately satisfactory or above, by volume Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Global Practice (%) (%) ($, millions) ($, millions) World Bank–widea 82 86 89,085 71,623 Poverty and Equity 97 100 557 738 Social Protection and Labor 95 100 7,066 4,054 Finance, Competitiveness, and Innovation 90 94 6,207 4,550 Governance 68 91 4,508 2,876 Social, Urban, Rural, and Resilience 85 90 9,930 7,977 Macroeconomics, Trade, and Investment 89 89 14,496 10,720 Education 83 87 7,188 4,047 Trade and Competitiveness 53 87 988 475 Energy and Extractives 82 86 8,861 6,264 Transport and Digital Development 70 83 11,807 13,287 Environment and Natural Resources 85 82 2,610 1,831 Agriculture 82 81 3,328 5,143 Health, Nutrition, and Population 84 81 5,875 3,915 Water 66 67 5,552 5,745 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); MS+ = moderately satisfactory or above. Although the FY18 restructuring of the Equitable Growth, Finance, and Institutions Practice Group phased out the Trade and Competitiveness (T&C) Global Practice, some projects mapped to T&C remained in World Bank business systems as of October 3, 2018, the data cutoff date for this report. Therefore, T&C is still reported as a Global Practice in this report. Some T&C- mapped projects are expected to be remapped to Finance, Competitiveness, and Innovation; and others, to Macroeconomics, Trade, and Investment. a World Bank–wide includes projects not tagged to a Global Practice. 50 Appendix A Additional Information on World Bank Performance Figure A.24. World Bank Projects: Monitoring and Evaluation Quality, by Global Practicea a. Percentage of projects rated substantial or above FY12-14 FY15-17 World Bank–wide Social Protection and Labor Poverty and Equity Finance, Competitiveness, and Innovation Education Health, Nutrition, and Population Energy and Extractives Social, Urban, Rural, and Resilience Macroeconomics, Trade, and Investment Governance Environment and Natural Resources Trade and Competitiveness Transport and Digital Development Water Agriculture 0 20 40 60 80 100 Percent b. Percentage of volume rated substantial or above FY12-14 FY15-17 World Bank–wide Social Protection and Labor Energy and Extractives Finance, Competitiveness, and Innovation Health, Nutrition, and Population Environment and Natural Resources Macroeconomics, Trade, and Investment Education Social, Urban, Rural, and Resilience Water Trade and Competitiveness Governance Agriculture Transport and Digital Development Poverty and Equity 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year of project closing. a World Bank–wide includes projects not tagged to a Global Practice. 51 Appendix A Additional Information on World Bank Performance Table A.24. World Bank Projects: Monitoring and Evaluation Quality Ratings, by Global Practice a. Percentage of projects rated substantial or above Projects Rated S+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Global Practice (%) (%) (no.) (no.) World Bank–widea 30 40 950 704 Social Protection and Labor 57 62 44 29 Poverty and Equity 50 60 4 5 Finance, Competitiveness and Innovation 27 58 56 24 Education 24 55 88 53 Health, Nutrition and Population 31 52 72 52 Energy and Extractives 44 46 88 61 Social, Urban, Rural and Resilience 23 42 118 90 Macroeconomics, Trade and Investment 41 41 106 41 Governance 28 36 69 44 Environment and Natural Resources 24 34 62 61 Trade and Competitiveness 36 33 14 21 Transport and Digital Development 20 30 83 83 Water 19 30 68 67 Agriculture 30 29 74 73 52 Appendix A Additional Information on World Bank Performance b. Percentage of volume rated substantial or above Projects Rated S+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Global Practice (%) (%) ($, millions) ($, millions) World Bank–widea 39 48 85,829 71,623 Social Protection and Labor 87 90 7,041 4,054 Energy and Extractives 56 67 8,145 6,264 Finance, Competitiveness, and Innovation 43 67 6,207 4,550 Health, Nutrition, and Population 23 63 5,560 3,915 Environment and Natural Resources 35 62 2,603 1,831 Macroeconomics, Trade, and Investment 41 56 14,396 10,720 Education 39 51 7,126 4,047 Social, Urban, Rural, and Resilience 23 48 9,284 7,977 Water 10 35 5,546 5,745 Trade and Competitiveness 29 30 986 475 Governance 36 30 4,508 2,876 Agriculture 51 30 3,328 5,143 Transport and Digital Development 25 27 10,427 13,287 Poverty and Equity 97 18 557 738 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); S+ = substantial or above. Although the FY18 restructuring of the Equitable Growth, Finance, and Institutions Practice Group phased out the Trade and Competitiveness (T&C) Global Practice, some projects mapped to T&C remained in World Bank business systems as of October 3, 2018, the data cutoff date for this report. Therefore, T&C is still reported as a Global Practice in this report. Some T&C-mapped projects are expected to be remapped to Finance, Competitiveness, and Innovation; and others, to Macroeconomics, Trade, and Investment. a World Bank–wide includes projects not tagged to a Global Practice. 53 Appendix A Additional Information on World Bank Performance Figure A.25. World Bank Projects: Outcome Ratings, by IBRD/IDA, IPF/DPF, and FCS Status a. Percentage of projects rated moderately satisfactory or above FY12-14 FY15-17 Project Type IDA IBRD Other IPF DPF Country Type All FCS All Non-FCS IDA FCS IDA non-FCS IBRD FCS IBRD non-FCS 0 20 40 60 80 100 Percent b. Percentage of volume rated moderately satisfactory or above FY12-14 FY15-17 Project Type IDA IBRD Other IPF DPF Country Type All FCS All Non-FCS IDA FCS IDA non-FCS IBRD FCS IBRD non-FCS 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCS = countries affected by fragility, conflict, or violence, based on the World Bank's list of fragile and conflict-affected situation countries in effect in the fiscal year when the project closed http://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-situations; FY = fiscal year (of project closing); IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing. 54 Appendix A Additional Information on World Bank Performance Table A.25. World Bank Projects: Outcome Ratings, by IBRD/IDA, IPF/DPF, and FCS Status a. Percentage of projects rated moderately satisfactory or above Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Type (%) (%) (no.) (no.) Project type IDA 68 76 492 318 IBRD 71 78 288 255 Other 66 75 181 131 IPF 68 76 786 643 DPF 72 78 172 60 Country type All FCS 68 65 164 101 All non-FCS 69 79 747 558 IDA FCS 69 65 150 91 IDA non-FCS 68 79 404 233 IBRD FCS 57 70 14 10 IBRD non-FCS 70 78 343 325 55 Appendix A Additional Information on World Bank Performance b. Percentage of volume rated moderately satisfactory or above Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Type (%) (%) ($, millions) ($, millions) Project type IDA 75 83 31,613 24,606 IBRD 82 87 52,314 44,930 Other 74 81 3,402 2,087 IPF 79 85 59,998 55,599 DPF 80 87 27,282 16,024 Country type All FCS 73 74 6,588 3,833 All non-FCS 80 86 79,591 64,981 IDA FCS 76 75 5,973 3,292 IDA non-FCS 75 84 29,208 16,795 IBRD FCS 45 72 615 541 IBRD non-FCS 83 87 50,383 48,186 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCS = countries affected by fragility, conflict, or violence based on the World Bank's list of fragile and conflict-affected situation countries in effect in the fiscal year when the project closed http://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-situations; FY = fiscal year (of project closing); IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing; MS+ = moderately satisfactory or above. 56 Appendix A Additional Information on World Bank Performance Figure A.26. World Bank Projects: Quality at Entry Ratings, by IBRD/IDA, IPF/DPF, and FCS Status a. Percentage of projects rated moderately satisfactory or above FY12-14 FY15-17 Project Type IDA IBRD Other IPF DPF Country Type All FCS All Non-FCS IDA FCS IDA non-FCS IBRD FCS IBRD non-FCS 0 20 40 60 80 100 Percent b. Percentage of volume rated moderately satisfactory or above FY12-14 FY15-17 Project Type IDA IBRD Other IPF DPF Country Type All FCS All Non-FCS IDA FCS IDA non-FCS IBRD FCS IBRD non-FCS 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCS = countries affected by fragility, conflict, or violence, based on the World Bank's list of fragile and conflict-affected situation countries in effect in the fiscal year when the project closed http://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-situations; FY = fiscal year (of project closing); IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing. 57 Appendix A Additional Information on World Bank Performance Table A.26. World Bank Projects: Quality at Entry Ratings, by IBRD/IDA, IPF/DPF, and FCS Status a. Percentage of projects rated moderately satisfactory or above Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Type (%) (%) (no.) (no.) Project type IDA 59 61 490 318 IBRD 63 69 289 254 Other 50 63 179 131 IPF 55 63 783 240 DPF 73 83 172 10 Country type All FCS 57 56 165 101 All non-FCS 60 67 742 557 IDA FCS 56 57 151 91 IDA non-FCS 60 62 400 233 IBRD FCS 71 50 14 10 IBRD non-FCS 60 71 342 324 58 Appendix A Additional Information on World Bank Performance b. Percentage of volume rated moderately satisfactory or above Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Type (%) (%) ($, millions) ($, millions) Project type IDA 70 65 31,982 24,606 IBRD 76 81 52,084 44,901 Other 70 73 3,369 2,087 IPF 71 71 59,828 15,963 DPF 80 87 27,557 2,008 Country type All FCS 66 50 6,588 3,833 All non-FCS 74 78 79,695 64,952 IDA FCS 66 49 5,973 3,292 IDA non-FCS 73 69 29,581 16,795 IBRD FCS 72 60 615 541 IBRD non-FCS 75 81 50,114 48,157 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCS = countries affected by fragility, conflict, or violence, based on the World Bank's list of fragile and conflict-affected situation countries in effect in the fiscal year when the project closed http://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-situations; FY = fiscal year (of project closing); IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing; MS+ = moderately satisfactory or above. 59 Appendix A Additional Information on World Bank Performance Figure A.27. World Bank Projects: Quality of Supervision Ratings, by IBRD/IDA, IPF/DPF, and FCS Status a. Percentage of projects rated moderately satisfactory or above FY12-14 FY15-17 Project Type IDA IBRD Other IPF DPF Country Type All FCS All Non-FCS IDA FCS IDA non-FCS IBRD FCS IBRD non-FCS 0 20 40 60 80 100 Percent b. Percentage of volume rated moderately satisfactory or above FY12-14 FY15-17 Project Type IDA IBRD Other IPF DPF Country Type All FCS All Non-FCS IDA FCS IDA non-FCS IBRD FCS IBRD non-FCS 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCS = countries affected by fragility, conflict, or violence, based on the World Bank's list of fragile and conflict-affected situation countries in effect in the fiscal year when the project closed http://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-situations; FY = fiscal year (of project closing); IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing. 60 Appendix A Additional Information on World Bank Performance Table A.27. World Bank Projects: Quality of Supervision Ratings, by IBRD/IDA, IPF/DPF, and FCS Status a. Percentage of projects rated moderately satisfactory or above Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Type (%) (%) (no.) (no.) Project type IDA 79 80 489 317 IBRD 77 83 286 254 Other 71 73 178 131 IPF 75 79 778 136 DPF 87 93 172 4 Country type All FCS 81 71 165 100 All non-FCS 77 82 737 557 IDA FCS 82 69 151 90 IDA non-FCS 78 82 399 233 IBRD FCS 64 90 14 10 IBRD non-FCS 76 83 338 324 61 Appendix A Additional Information on World Bank Performance b. Percentage of volume rated moderately satisfactory or above Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Type (%) (%) ($, millions) ($, millions) Project type IDA 85 80 31,732 24,551 IBRD 88 92 52,082 44,901 Other 77 79 3,368 2,087 IPF 85 84 59,575 8,674 DPF 90 99 27,557 185 Country type All FCS 85 73 6,588 3,778 All non-FCS 87 89 79,442 64,952 IDA FCS 88 72 5,973 3,237 IDA non-FCS 82 80 29,331 16,795 IBRD FCS 54 77 615 541 IBRD non-FCS 90 92 50,111 48,157 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCS = countries affected by fragility, conflict, or violence, based on the World Bank's list of fragile and conflict-affected situation countries in effect in the fiscal year when the project closed http://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-situations; FY = fiscal year (of project closing); IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing; MS+ = moderately satisfactory or above. 62 Appendix A Additional Information on World Bank Performance Figure A.28. World Bank Projects: Bank Performance Ratings, by IBRD/IDA, IPF/DPF, and FCS Status a. Percentage of projects rated moderately satisfactory or above FY12-14 FY15-17 Project Type IDA IBRD Other IPF DPF Country Type All FCS All Non-FCS IDA FCS IDA non-FCS IBRD FCS IBRD non-FCS 0 20 40 60 80 100 Percent b. Percentage of volume rated moderately satisfactory or above FY12-14 FY15-17 Project Type IDA IBRD Other IPF DPF Country Type All FCS All Non-FCS IDA FCS IDA non-FCS IBRD FCS IBRD non-FCS 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCS = countries affected by fragility, conflict, or violence, based on the World Bank's list of fragile and conflict-affected situation countries in effect in the fiscal year when the project closed http://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-situations; FY = fiscal year (of project closing); IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing. 63 Appendix A Additional Information on World Bank Performance Table A.28. World Bank Projects: Bank Performance Ratings, by IBRD/IDA, IPF/DPF, and FCS Status a. Percentage of projects rated moderately satisfactory or above Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Type (%) (%) (no.) (no.) Project type IDA 70 76 497 318 IBRD 72 78 292 255 Other 66 72 182 131 IPF 68 76 795 157 DPF 78 83 173 10 Country type All FCS 71 65 165 101 All non-FCS 70 79 755 558 IDA FCS 72 65 151 91 IDA non-FCS 70 78 407 233 IBRD FCS 64 70 14 10 IBRD non-FCS 70 80 348 325 64 Appendix A Additional Information on World Bank Performance b. Percentage of volume rated moderately satisfactory or above Projects Rated MS+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Type (%) (%) ($, millions) ($, millions) Project type IDA 79 82 33,065 24,606 IBRD 84 88 52,617 44,930 Other 78 80 3,403 2,087 IPF 80 85 61,452 8,258 DPF 85 88 27,582 1,988 Country type All FCS 76 72 6,588 3,833 All non-FCS 82 87 81,344 64,981 IDA FCS 78 72 5,973 3,292 IDA non-FCS 80 84 30,658 16,795 IBRD FCS 54 72 615 541 IBRD non-FCS 84 88 50,687 48,186 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCS = countries affected by fragility, conflict, or violence, based on the World Bank's list of fragile and conflict-affected situation countries in effect in the fiscal year when the project closed http://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-situations; FY = fiscal year (of project closing); IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing; MS+ = moderately satisfactory or above. 65 Appendix A Additional Information on World Bank Performance Figure A.29. World Bank Projects: Monitoring and Evaluation Quality Ratings, by IBRD/IDA, IPF/DPF, and FCS Status a. Percentage of projects rated substantial or above FY12-14 FY15-17 Project Type IDA IBRD Other IPF DPF Country Type All FCS All Non-FCS IDA FCS IDA non-FCS IBRD FCS IBRD non-FCS 0 20 40 60 80 100 Percent b. Percentage of volume rated substantial or above FY12-14 FY15-17 Project Type IDA IBRD Other IPF DPF Country Type All FCS All Non-FCS IDA FCS IDA non-FCS IBRD FCS IBRD non-FCS 0 20 40 60 80 100 Percent Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCS = countries affected by fragility, conflict, or violence, based on the World Bank's list of fragile and conflict-affected situation countries in effect in the fiscal year when the project closed http://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-situations; FY = fiscal year (of project closing); IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing. 66 Appendix A Additional Information on World Bank Performance Table A.29. World Bank Projects: Monitoring and Evaluation Quality Ratings, by IBRD/IDA, IPF/DPF, and FCS Status a. Percentage of projects rated substantial or above Projects Rated S+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Type (%) (%) (no.) (no.) Project type IDA 30 36 485 318 IBRD 33 45 287 255 Other 25 40 178 131 IPF 27 39 776 391 DPF 44 53 171 28 Country type All FCS 29 25 164 101 All non-FCS 31 43 736 558 IDA FCS 30 23 150 91 IDA non-FCS 29 43 396 233 IBRD FCS 21 40 14 10 IBRD non-FCS 32 43 340 325 67 Appendix A Additional Information on World Bank Performance b. Percentage of volume rated substantial or above Projects Rated S+ Total Projects FY12–14 FY15–17 FY12–14 FY15–17 Type (%) (%) ($, millions) ($, millions) Project type IDA 40 42 30,365 24,606 IBRD 39 51 52,083 44,930 Other 36 55 3,382 2,087 IPF 37 46 58,322 30,193 DPF 43 57 27,457 6,881 Country type All FCS 37 27 6,588 3,833 All non-FCS 39 49 78,091 64,981 IDA FCS 40 26 5,973 3,292 IDA non-FCS 42 47 27,960 16,795 IBRD FCS 8 38 615 541 IBRD non-FCS 38 50 50,131 48,186 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: DPF = development policy financing; FCS = countries affected by fragility, conflict, or violence, based on the World Bank's list of fragile and conflict-affected situation countries in effect in the fiscal year when the project closed http://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-situations; FY = fiscal year (of project closing); IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing; S+ = substantial or above. 68 Appendix A Additional Information on World Bank Performance Table A.30. Portfolio of Projects with Completion Reporting and Ratings in Each Reporting Period, by Project Size Category Total Volume of Percent Percent Project Sizea Projects Closed Projects Closed Percent of Percent of Change in Change in Category (no.) ($, millions) Projects Volume Number of Project ($, millions) FY12–14 FY15–17 FY12–14 FY15–17 FY12–14 FY15–17 FY12–14 FY15–17 Projects Volume (1) ≤20 (very small) 349 242 3,408 2,453 36 34 4 3 −31 −28 (2) >20 and ≤50 239 164 8,376 5,675 25 23 10 8 −31 −32 (3) >50 and ≤100 145 114 11,148 8,709 15 16 13 12 −21 −22 (4) >100 and ≤200 126 86 18,140 12,891 13 12 21 18 −32 −29 (5) >200 (very large) 97 96 46,257 41,895 10 14 53 58 −1 −9 N/A 5 2 — — 1 0 0 0 −60 — Total 961 704 87,329 71,623 100 100 100 100 −27 −18 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing). a Projects are designated N/A under project size when the net commitment value for this project is blank in Business Intelligence. All dollar amounts are U.S. dollars unless otherwise indicated. 69 Appendix A Additional Information on World Bank Performance Table A.31. Portfolio of Projects with Completion Reporting and Ratings in Each Reporting Period, by Source of Funds Total Volume of Percent Percent Projects Closed Projects Closed Change in Change in (no.) ($, millions) Percent of Projects Percent of Volume Number of Project Source of Funds FY12–14 FY15–17 FY12–14 FY15–17 FY12–14 FY15–17 FY12–14 FY15–17 Projects Volume IBRD 288 255 52,314 44,930 30 36 60 63 −11 −14 IDA 492 318 31,613 24,606 51 45 36 34 −35 −22 Recipient Executed 91 60 1,872 1,446 9 9 2 2 −34 −23 Trust Fund Special Fund 23 11 371 173 2 2 0 0 −52 −53 Global Environment 60 56 539 433 6 8 1 1 −7 −20 Fund Carbon Initiative 1 0 — n.a. 0 0 0 0 .. n.a. Montreal Protocol 6 3 620 34 1 0 1 0 −50 −95 Other 0 1 n.a. — 0 0 0 0 .. n.a. Total 961 704 87,329 71,623 100 100 100 100 −27 −18 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; FY = fiscal year (of project closing); other = projects are designated “other” when the net commitment value for this project is blank in Business Intelligence. All dollar amounts are U.S. dollars unless otherwise indicated. 70 Appendix A Additional Information on World Bank Performance Table A.32. Portfolio of Projects with Completion Reporting and Ratings in Each Reporting Period, by Country Designation Total Volume of Percent Percent Country Projects Closed Projects Closed Percent of Percent of Change in Change in Designation (no.) ($, millions) Projects Volume Number of Project Category FY12–14 FY15–17 FY12–14 FY15–17 FY12–14 FY15–17 FY12–14 FY15–17 Projects Volume IBRD FCS 14 10 615 541 1 1 1 1 −29 −12 IBRD non-FCS 343 325 50,383 48,186 36 46 58 67 −5 −4 IDA FCS 150 91 5,973 3,292 16 13 7 5 −39 −45 IDA non-FCS 404 233 29,208 16,795 42 33 33 23 −42 −42 Other 50 45 1,150 2,809 5 6 1 4 −10 144 Total 961 704 87,329 71,623 100 100 100 100 −27 −18 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FCS = fragile and conflict-affected situation; FY = fiscal year (of project closing); IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; other = projects are designated “other” when the net commitment value for this project is blank in Business Intelligence. All dollar amounts are U.S. dollars unless otherwise indicated. 71 Appendix A Additional Information on World Bank Performance Table A.33. Portfolio of Projects with Completion Reporting and Ratings in Each Reporting Period, by World Bank Region Total Volume of Percent Percent Projects Closed Projects Closed Change in Change (no.) ($, millions) Percent of Projects Percent of Volume Number of in Project World Bank Region FY12–14 FY15–17 FY12–14 FY15–17 FY12–14 FY15–17 FY12–14 FY15–17 Projects Volume Africa 306 199 15,433 14,190 32 28 18 20 −35 −8 East Asia and Pacific 163 121 17,662 12,672 17 17 20 18 −26 −28 Europe and Central Asia 157 104 13,247 14,574 16 15 15 20 −34 10 Latin America and the 157 134 21,562 14,060 16 19 25 20 −15 −35 Caribbean Middle East and North Africa 80 61 5,425 4,185 8 9 6 6 −24 −23 South Asia 97 83 13,995 11,927 10 12 16 17 −14 −15 Other 1 2 7 14 0 0 0 0 100 100 Total 961 704 87,329 71,623 100 100 100 100 −27 −18 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); other = projects are designated “other” when the country/region for this project is blank in Business Intelligence. All dollar amounts are U.S. dollars unless otherwise indicated. 72 Appendix A Additional Information on World Bank Performance Table A.34. Portfolio of Projects with Completion Reporting and Ratings in Each Reporting Period, by Practice Group Total Volume of Percent Percent Projects Closed Projects Closed Change in Change in (no.) ($, millions) Percent of Projects Percent of Volume Number of Project Practice Group FY12–14a FY15–17 FY12–14a FY15–17 FY12–14a FY15–17 FY12–14a FY15–17 Projects Volume Equitable Growth, Finance, 247 135 26,453 19,359 26 19 30 27 −45 −27 and Institutions Human Development 207 134 19,878 12,016 22 19 23 17 −35 −40 Infrastructure 176 144 19,466 19,552 18 20 22 27 −18 0 Sustainable Development 327 291 21,419 20,696 34 41 25 29 −11 −3 Other 2 0 113 — 0 0 0 0 −100 −100 Total 959 704 87,329 71,623 100 100 100 100 −27 −18 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); other = projects are designated “other” when practice group for this project is blank in Business Intelligence. a Two projects closed in FY12–14 are excluded because of missing data for Global Practice and net commitment. All dollar amounts are U.S. dollars unless otherwise indicated. 73 Appendix A Additional Information on World Bank Performance Table A.35. Portfolio of Projects with Completion Reporting and Ratings in Each Reporting Period, by Global Practice Total Volume of Percent Percent Projects Closed Projects Closed Percent of Percent of Change in Change in (no.) ($, millions) Projects Volume Number of Project Global Practice FY12–14a FY15–17 FY12–14a FY15–17 FY12–14a FY15–17 FY12–14a FY15–17 Projects Volume Finance, Competitiveness, and Innovation 56 24 6,207 4,550 6 3 7 6 −57 −27 Governance 68 44 4,508 2,876 7 6 5 4 −35 −36 Macroeconomics, Trade, and Investment 106 41 14,195 10,720 11 6 16 15 −61 −24 Poverty and Equity 4 5 557 738 0 1 1 1 25 32 Trade and Competitiveness 13 21 986 475 1 3 1 1 62 −52 Education 89 53 7,188 4,047 9 8 8 6 −40 −44 Health, Nutrition, and Population 74 52 5,875 3,915 8 7 7 5 −30 −33 Social Protection and Labor 44 29 6,816 4,054 5 4 8 6 −34 −41 Energy and Extractives 92 61 8,861 6,264 10 9 10 9 −34 −29 Transport and Digital Development 84 83 10,605 13,287 9 12 12 19 −1 25 Agriculture 74 73 3,328 5,143 8 10 4 7 −1 55 Environment and Natural Resources 63 61 2,609 1,831 7 9 3 3 −3 −30 Social, Urban, Rural, and Resilience 121 90 9,930 7,977 13 13 11 11 −26 −20 Water 69 67 5,552 5,745 7 10 6 8 −3 3 Other 2 0 113 n.a. 0 0 0 0 −100 n.a. Total 959 704 87,329 71,623 100 100 100 100 −27 −18 Source: World Bank Business Intelligence and Independent Evaluation Group World Bank project ratings data. Note: FY = fiscal year (of project closing); other = projects are designated “other” when global practice for this project is blank in Business Intelligence. a Two projects closed in FY12–14 are excluded because of missing data for Global Practice and net commitment. All dollar amounts are U.S. dollars unless otherwise indicated. 74 Appendix B. Results and Performance of International Finance Corporation Investment Projects Results and Performance of the World Bank Group 2018 (RAP18) updated the analysis of the development results of investment projects by the International Finance Corporation (IFC) and IFC’s performance by including findings from 87 project evaluations completed in the calendar year (CY)17 Expanded Project Supervision Report (XPSR) program. A total of 253 IFC investment projects with completed evaluations as of October 5, 2018 were included in the three-year review period of CY15–17, with approval years spanning fiscal year (FY)09–13 and evaluated in the CY15, CY16, and CY17 XPSR programs. This appendix presents project results data based on three-year rolling averages. Development outcome rating reflects the overall contribution of the project to the development of its host country and thus implicitly addresses how well the project has contributed to fulfilling IFC’s purpose. A project’s development outcome encompasses all effects on a country’s economic and social development. Development outcome has four elements or indicators: project business success, economic sustainability, environmental and social effects, and contribution to private sector development. Trends in Performance Performance of IFC investment projects on development outcome has continued to decline, with an increasing share of highly unsuccessful projects. The proportion of IFC investment projects rated mostly successful or better has declined significantly from 57 percent in CY12–14 to 45 percent in CY15–17 by number of projects (figure B.1, panel a),1 and from 69 percent in CY12–14 to 53 percent in CY15–17 by commitment amount (figure B.1, panel b), on a three-year rolling average. During the same period, the proportion of projects rated highly unsuccessful increased from 5 to 12 percent. Among the Regions, the share of highly unsuccessful projects in CY15–17 was concentrated in Europe and Central Asia (about one-quarter). Among the industry groups, Infrastructure had the highest concentration of highly unsuccessful projects (about one-third), but Infrastructure also had higher than average positive ratings (about one-half) and the highest proportion of positive ratings (61 percent) when weighted by commitment volume. 75 Appendix B Results and Performance of International Finance Corporation Investment Projects Figure B.1. Performance of IFC Investment Projects by Development Outcome Rating, Three-Year Rolling Basis, CY06–17 a. Share of number of IFC investments, by development outcome rating (three-year rolling basis), CY06-17 80 10 10 8 7 5 4 60 2 2 2 35 2 37 36 30 23 Percent 40 36 26 17 14 14 20 33 35 34 25 27 30 26 29 30 30 0 10 13 17 20 20 22 23 21 20 21 20 12 11 9 10 6 3 1 12 14 17 19 20 21 40 1 2 3 5 7 10 60 12 CY06-08 CY07-09 CY08-10 CY09-11 CY10-12 CY11-13 CY12-14 CY13-15 CY14-16 CY15-17 (N=176) (N=213) (N=220) (N=242) (N=243) (N=235) (N=232) (N=231) (N=242) (N=253) b. Share of amount of IFC investments, by development outcome rating (three-year rolling basis, IFC net commitment; $, billions), CY06–17 14 12 10 80 10 9 7 3 2 2 60 1 38 42 39 31 20 16 39 39 34 16 40 Percent 20 31 32 35 43 42 36 29 27 27 29 0 10 11 13 17 17 19 17 18 21 22 5 6 7 20 1 6 7 10 12 14 2 0 1 1 14 16 40 1 2 4 6 9 60 CY06-08 CY07-09 CY08-10 CY09-11 CY10-12 CY11-13 CY12-14 CY13-15 CY14-16 CY15-17 (V=3.0) (V=4.2) (V=4.6) (V=5.9) (V=6.8) (V=7.1) (V=6.7) (V=6.4) (V=6.7) (V=6.5) Source: Independent Evaluation Group evaluative notes. Note: CY = calendar year; V = total volume of projects ($, billions). All dollar amounts are U.S. dollars unless otherwise indicated. Percentages may not add up to 100 percent because of rounding. Indicators of Development Outcome Three of the four indicators of development outcome showed significant decline in CY15–17. Environmental and social effects was the only indicator of development 76 Appendix B Results and Performance of International Finance Corporation Investment Projects outcome that increased from 64 percent in CY12–14 to 72 percent in CY15–17.2 The remaining three indicators of development outcome (project business success, economic sustainability, and private sector development) declined in the range of 8 to 17 percent during the same period (figure B.2, panels a and b). The increase in environmental and social effects performance was mainly because IFC’s work quality related to environmental and social effects, both at appraisal and supervision, has been improving steadily over the years due to (i) well-articulated environmental and social effects sustainability policy; (ii) well-structured and improved environmental and social effects review procedures and guidelines; (iii) improved environmental and social effects capacity in terms of more staff resources, trainings, regional expertise, and so on; and (iv) more project monitoring and client interactions on environmental and social effects in all sectors. Among the four indicators of development outcome, economic sustainability was strongly associated with development outcome, followed by project business success and private sector development; whereas, the association between environmental and social effects and development outcome was weak in the CY15– 17 evaluation cohort.3 To some extent, project business success and economic sustainability were associated with each other, indicating that projects with poor business results experienced sustainability challenges.4 Among the indicators of development outcome, both project business success (at 39 percent satisfactory or better) and economic sustainability (at 41 percent) had the lowest ratings among the indicators in CY15–17. The associations between the various dimensions of development outcome suggest a decoupling between environmental and social effects performance and project business success. An important implication of this result is that good environmental and social effects performance is not necessarily predicated on strong profitability. An important question is whether strong environmental and social effects performance can become a driver for improvements in a project’s business success by, for instance, enhancing resilience and, by extension, in overall development outcome. Additionally, for repeat clients, good environmental and social effects performance might be a useful proxy indicator (perhaps necessary but not sufficient) for client commitment when IFC is considering a new investment with that client. 77 Appendix B Results and Performance of International Finance Corporation Investment Projects Figure B.2. Indicators of Development Outcome, CY12–14 and CY15–17 a. By number of projects b. By net commitment 80 80 74 72 70 Rated satisfactory or better (%) Rated satisfactory or better (%) 64 67 70 65 64 64 58 58 60 56 60 47 47 50 42 39 41 40 40 30 20 20 10 0 0 PBS ECSUS ENV PSD PBS ECSUS ENV PSD Development outcome Development outcome 2012-2014 2015-2017 2012-2014 2015-2017 Source: Independent Evaluation Group evaluative notes. Note: CY = calendar year; ECSUS = economic sustainability; ENV = environmental and social effects; PBS = project business success; PSD = private sector development. Performance of Infrastructure Industry Group Infrastructure had the largest decline in development outcome ratings. Among the industry groups, the development outcome performance for infrastructure projects declined significantly from 69 percent in CY12–14 to 48 percent in CY15–17.5 Deteriorating macro, market, and regulatory environments and declining commodity prices resulted in weakening performance. Weaknesses in IFC’s front- end work, especially for projects evaluated in CY17, contributed to the decline in development performance of infrastructure projects. The factors contributing to IFC’s weak front-end work relate to the assessment of (i) weak sponsor or company capabilities in terms of lack of relevant expertise, track record, or commitment in the targeted business areas; (ii) shortcomings in the new business models or strategies; (iii) increased competition in the market and lower than expected demand for the products or services provided; and (iv) regulatory and political issues. 78 Appendix B Results and Performance of International Finance Corporation Investment Projects Performance of the Manufacturing, Agribusiness, and Services Industry Group Decline in the performance of the manufacturing, agribusiness, and services (MAS) industry group in CY15–17 was due to weak performance of projects in tourism, retail, construction, and real estate and manufacturing sectors. IFC needs to pay greater attention to its tourism, retail, construction, and real estate portfolio because only 2 of 12 projects evaluated in CY15–17 were rated mostly successful or better. The weak performance in this sector was due to two main factors: (i) the sponsors’ lack of experience in entering opportunistically into difficult and underdeveloped market segments in this sector and (ii) insufficient IFC industry specialist coverage in this sector, resulting in gaps in assessment of risks and market. The latter occurred mostly in small countries, where many of the evaluated projects are located, and/or in projects experimenting, and in projects experimenting with innovative business models. As a result, transactions were often carried out by teams without sector expertise. In the manufacturing sector, only two of every five projects were rated mostly successful or better on development outcome in CY15– 17. This was due to three main factors: (i) shortage of IFC staff with industry knowledge in this very diverse sector, contributing to weaknesses in risk assessment at the appraisal stage of projects and inadequate follow-up during the supervision stage of projects; (ii) optimistic ex ante market assessment and business prospects; and (iii) changes in government regulation and policies, which had negative implications on projects. Performance of Financial Institutions Industry Group There was a decline in the performance of the financial institutions industry group in CY15–17 caused by weak performance of projects in three areas. These areas were (i) commercial banking (from 63 percent in CY12–14 to 56 percent in CY15– 17), (ii) nonbanking financial institutions (from 100 to 53 percent), and (iii) insurance and pensions (from 93 to 0 percent, because all four projects were rated mostly unsuccessful or worse in CY15–17). Microfinance (at 69 percent) had the highest rating on development outcome within the financial institutions industry group in CY15–17 (figure B.3, panels a and b). 79 Appendix B Results and Performance of International Finance Corporation Investment Projects Figure B.3. Performance of Commercial Banking, Nonbanking Financial Institutions, and Insurance and Pensions Sectors, CY12–14 and CY15–17 a. By number of projectsa b. By net commitmenta 100 100 100 93 90 Rated mostly successful or better (%) Rated mostly successful or better (%) 90 90 80 80 69 70 70 63 64 56 57 54 56 60 53 50 60 53 50 52 50 50 46 50 40 40 40 30 30 20 20 10 0 10 0 0 0 CB NBFI I&P MF Others CB NBFI I&P MF Others Institution type Institution type 2012-2014 2015-2017 2012-2014 2015-2017 Source: Independent Evaluation Group evaluative notes. Note: The category “others” includes Distressed Asset Recovery Program, digital finance, funds, housing finance, other financial institutions group sector, and trade and commodity. CB = commercial banking; CY = calendar year; I&P = insurance and pension; MF = microfinance; NBFI = nonbanking financial institutions. a. The decline for I&P is not statistically significant because there are only six I&P projects. Financial Institutions Industry Group: What Works and Does Not Work The analysis was based on an examination of the Evaluative Notes associated with a sample of 22 IFC financial institutions industry group projects that have been evaluated by IEG. The sample was partitioned into a group of 10 projects for which the development outcome was rated mostly successful or better, and a second group of 12 projects for which the development outcome was rated mostly unsuccessful or worse. For the first group, the analysis attempted to identify the recurring factors contributing to their successful development outcome rating. For the second group, the analysis attempted to identify the recurring factors contributing to their unsuccessful development outcome rating. In both groups of projects, factors were mostly identified from clear statements made in the relevant Evaluative Note for the project. In a few cases, factors were inferred from the evaluative description even though the factor as presented in this analysis was not explicitly mentioned.6 80 Appendix B Results and Performance of International Finance Corporation Investment Projects Financial Institutions Industry Group: Factors that Seem to Work A positive business (or enabling) environment and benign or favorable market conditions appear to be factors present in every project rated mostly successful or better (table B.1). However, it is advisable not to stretch this interpretation to infer that they should be deemed a necessary condition for project success. Another factor present in all 10 projects in the sample is satisfactory environmental and social effects, which is possibly meaningful as a proxy for client commitment.7 The final factor present in all 10 projects that is likely more robust as an explanatory factor is selection of IFC investment instruments in local currency. This is significant given that all the projects involved bank or financial institution onlending to relatively less-sophisticated end borrowers who otherwise might have been adversely affected by bearing the brunt of foreign exchange risk. Table B.1. IFC Financial Institutions Industry Group Projects Rated Mostly Successful or Better: Factors that Seem to Work Serial No. Factor 1 2 3 4 5 6 7 8 9 10 Frequency Positive business environment x x x x x x x x x x 10 Benign or favorable market x x x x x x x x x x 10 conditions IFC relationship bank/FI x x x x x x 6 Good transaction structure x x x 3 Good ID of risk and mitigants x x x x x x 6 Achieved development objectives x x x x x 5 IFC investment in local currency x x x x x x x x x x 10 Satisfactory environmental and x x x x x x x x x x 10 social effects Satisfactory AS or knowledge x x x x x x 6 transfer Satisfactory use of specialists x x x 3 Satisfactory due diligence x x x x x x x x 8 Source: Independent Evaluation Group evaluative notes. Note: AS = Advisory Services; ID = identification; IFC = International Finance Corporation; FI = financial institution; no. = number. The next important factor in terms of frequency of occurrence in the sample (8 of 10 projects) was adequacy of overall due diligence. Three important factors (each present in 6 of 10 projects) were good identification of risks and mitigants (actually 81 Appendix B Results and Performance of International Finance Corporation Investment Projects a subcomponent of due diligence but considered worth treating separately because of its significance), satisfactory delivery and coordination of either Advisory Services (as promised to IFC’s Board) or supervisory (or possibly other stakeholder) transfer of technical knowledge, and the selection of an IFC relationship bank as the partner in the project, the significance of which is that IFC is more likely to be successful in assessing both client commitment to and capacity for undertaking the project. Regarding Advisory Services and knowledge transfer, not all projects with development outcome rated as successful actually required inputs from Advisory Services interventions. In the cases in which Advisory Services projects were explicitly mentioned in IFC Board Reports, all were delivered and, moreover, showed evidence of having been coordinated to some extent with the IFC investment in question. This is significant to the extent that at investment approval, key ingredients to the project’s success were identified as depending on reasonable execution of an Advisory Services project, without which the investment project’s success would be at risk. Two factors (each present in 3 of 10 projects) that may not be as universally compelling but were explicitly observed in the Evaluative Notes were good transaction structuring and good use of IFC specialists. The former becomes particularly important when the IFC investment involves more than the usual degree of complexity, and the second is important where sectoral challenges are best met through specialized domain knowledge. Financial Institutions Industry Group: What Does Not Work The factor “weak enabling environment” in table B.2 covers macroeconomic conditions and government policy framework. Moreover, it includes problematic environments showing incipient signs at approval and other cases in which conditions deteriorated only post-disbursement. This is a difficult factor to categorize as being under IFC control because, as a development institution, IFC is expected to assume a certain amount of risk under difficult conditions and may not be able to avoid this risk to the extent a commercial institution would. However, the two most important controllable factors for IFC that seem to account for unsuccessful project performance are weak due diligence and other front-end work (other than risks and mitigants) (present in 11 of 12 projects in this group), and 82 Appendix B Results and Performance of International Finance Corporation Investment Projects weak identification of risks and mitigants (present in 6 of 12 projects and which is a subset of IFC’s front-end work but presented separately because of its significance). Next in terms of frequency of occurrence was setting objectives and targets that cannot be verified (7 of 12 projects). Weak delivery or coordination of Advisory Services (or other knowledge transfer) in connection with the relevant investment was present in 6 of 12 projects and largely meant that the project was missing certain essential inputs as presented to IFC’s Board for approval. In some cases, the Advisory Services project promised in the IFC Board Report never materialized. However, in other cases, Advisory Services projects were implemented but considered ineffective for one of two reasons: (i) an Advisory Services project had been initiated and was ongoing before approval of the relevant investment project with attendant limitations for changing previously and contractually established advisory objectives and priorities, or (ii) for Advisory Services projects undertaken post-IFC investment, early involvement of the IFC Advisory Services team at investment concept and Investment Review Memorandum stages (or at least minimal communication between the two teams) appears to have been lacking. Also present in 6 of 12 projects was weak client capacity. Moreover, weak client commitment and incentives was discernible in 5 of 12 projects; weak partner selection was present in 4 projects. In 3 projects, crystalized foreign exchange risk adversely impacted either the bank or financial institution or else their sub- borrowers. 83 Appendix B Results and Performance of International Finance Corporation Investment Projects Table B.2. IFC Financial Market Projects Rated Mostly Unsuccessful or Worse: Factors that Seem Not to Work Serial No. Factor 1 2 3 4 5 6 7 8 9 10 11 12 Frequency Weak enabling environment x x x x x x x 7 Weak client commitment and incentive x x x x x 5 Weak client capacity (for project) x x x x x x 6 Negative impact foreign exchange risk x x x 3 Weak or negative demo effect x x x x x x x x 8 Weak due diligence and other IFC’s x x x x x x x x x x x 11 front-end work Weak identification of risks and mitigants x x x x x x 6 Weak partner selection x x x x 4 Weak AS or knowledge transfer x x x x x x 6 Weak supervision x x x x x 5 Objectives or targets not verifiable x x x x x x x 7 Source: Independent Evaluation Group evaluative notes. Note: AS = Advisory Services; FX = foreign exchange; IFC = International Finance Corporation; no. = number. Performance of Telecom, Media, Technology, Venture Capital and Funds Industry Group Among the four industry groups, Telecom, Media, Technology (TMT), Venture Capital and Funds had the lowest rating on development outcome (about one- quarter of the projects rated mostly successful or better) in CY15–17. This was due largely to weak performance of the IFC-supported investment funds of the cohort in this industry group, which is not representative of the performance of IFC’s investment fund portfolio overall. A review of IFC’s investment fund projects in the TMT, Venture Capital and Funds industry group identified the following factors of underperformance: • Weak measurability of the fund’s development objectives related to (i) collecting information on the achievement of developmental objectives, particularly information on reaching targeted beneficiaries such as small and medium enterprises; and (ii) targets that are too broad and not necessarily reflective of the achievement of desired objectives (for example, 84 Appendix B Results and Performance of International Finance Corporation Investment Projects the number of additional investee company board directors may not have necessarily implied that IFC helped improve corporate governance). • Departure from the fund’s mandate. This often assumed the form of investing in companies that fall outside of the definition of target companies stated both in the fund’s investment strategy and development objectives. • Weaknesses in the fund’s business model related to (i) overambitious regional coverage without having achieved sufficient investment scale to avoid the excessive burden of high operational costs that this produced; (ii) tension between objectives (for example, targeting small entities while setting demanding emissions-reduction targets); and (iii) lack of a cap on fund manager expenses. • Weaknesses in IFC’s front-end work related to (i) adequate structuring of the IFC investment to address any inexperience or skill gaps in the fund manager; (ii) appropriate fund manager screening ratios (that is, the number of proposals reviewed per investments made); and (iii) the existence of a suitable fund manager control framework (for example, caps on fund manager expenses, budget ceilings for Advisory Services to investee companies, procedures for managing conflicts of interest, resolving or mitigating partner disputes, and changing or removing the fund manager). • Undercapitalization. In most cases of underperforming projects, the funds proceeded despite being undercapitalized. For some, this increased costs or even impaired the fund’s viability from the outset. • Difficult macroeconomic environment. This was often identified as a factor for underperformance combined with overambitious objectives. Performance of IFC Projects in IDA versus Non-IDA Countries Improving the performance of IFC investments in IDA countries is important because IFC has promised to deliver 40 percent of its commitments in IDA countries by 2030 as part of commitments under the capital increase to its stakeholders. After years of convergence in development outcome performance in IDA and non-IDA countries, the performance of IFC investments in IDA (at 41 percent mostly successful or better) slipped and was below that of non-IDA (at 85 Appendix B Results and Performance of International Finance Corporation Investment Projects 47 percent) countries in CY15–17 (figure B.4, panels a and b). About 60 percent of the IFC projects in IDA countries in CY15–17 were concentrated in Sub-Saharan Africa. Therefore, the weak performance of IFC projects in Sub-Saharan Africa (at 30 percent mostly successful or better) in CY15–17—especially in the commercial banking sector (where only one of seven projects was rated mostly successful or better) and in the oil, gas, and mining sector (at 17 percent)—contributed to the decline in IDA’s performance. For non-IDA countries, the weak performance of projects (reflected in percentages of those projects in each group rated mostly successful or better) in CIV (0 percent), tourism, retail, construction, and real estate (20 percent), and manufacturing (40 percent) sectors in CY15–17 contributed to the decline in performance (figure B.4). Figure B.4. Performance of IDA and Non-IDA Countries, CY12–14 and CY15–17 a. By number of projects b. By net commitment 80 80 73 Rated mostly successful or better (%) Rated mostly successful or better (%) 67 70 70 56 58 57 60 60 53 50 47 50 41 40 40 30 30 20 20 10 10 0 0 IDA Non-IDA IDA Non-IDA Country type Country type 2012-2014 2015-2017 2012-2014 2015-2017 Source: Independent Evaluation Group evaluative notes. Note: IDA = International Development Association. Fragile and Conflict-Affected Situations: What Works and Does Not Work The Evaluative Notes for a sample of eight fragile and conflict-affected situations (FCS) projects were reviewed to ascertain those factors that contributed to apparent 86 Appendix B Results and Performance of International Finance Corporation Investment Projects project success (in terms of a development outcome rating of mostly successful or better) and those that seem to have been responsible for unsuccessful performance (in terms of a development outcome rating of mostly unsatisfactory or worse). Because of the very small sample size, caution is recommended in inferring causality to all or even most other projects in the same sector. Nonetheless, applied with caution, the findings are believed to have heuristic value. The project sample is eclectic, drawing from diverse subsectors from among the financial sector (a bank and two microfinance institutions) and the real sector (telecommunications, cement, a gold mine, port infrastructure, and liquid natural gas energy). This resulted in the factors explaining what worked being unique in a few cases, whereas the factors contributing to unsuccessful performance were most common to all three projects in this group. A factor common to all five successful projects was selecting an experienced partner that knew how to operate in challenging FCS working conditions (table B.3). Next in terms of frequency of occurrence was satisfactory IFC front-end work (four of five projects) and satisfactory IFC supervision and administration (four of five projects). Deficiencies in IFC’s front-end work (one project) and supervision and administration (one project) may not be crucial factors. Positive factors found in three of five projects were reasonably conservative projections (ideally based on probability-based expected value rather than projections that were too optimistic or too conservative) combined with good technical analysis. Factors present in two of five projects were special loan protection features because of the additional risk in an FCS environment and, for the two microfinance institution projects, effective microfinance institution strategy formulation, sound Advisory Services and Investment coordination, and low interest-rate policy that helped maintain asset quality in an FCS environment. Regarding Advisory Services and Investment coordination, despite some variations in expectations set for project targets and coordination achieved for the Advisory Services and Investment projects respectively, there was sufficient alignment of Advisory Services and Investment broader objectives and priorities, and effective and timely transfer of knowledge (both through training and on-the-job) to underpin project success. Moreover, coordination was most effective in those areas where there was flexible adaptation to changing project needs in a fluid FCS environment. Other factors identified as contributors to project success in this group tended to be project- specific. 87 Appendix B Results and Performance of International Finance Corporation Investment Projects Table B.3. Fragile and Conflict-Affected Situations: Factors that Seem to Work Serial No. Factor 1 2 3 4 5 Frequency Conservative projections x x x 3 Good technical analysis x x x 3 Good analysis of policy and regulatory framework x 1 Identification of corruption and COIs as risks x 1 Governance issues addressed x 1 Effective MFI strategy formulation x x 2 Effective use of government subsidy x x 2 Experienced partner for challenging conditions x x x x x 5 Loan protection features x x 2 Sound AS and IS coordination x x 2 FCS low-interest-rate policy helps sustainability x x 2 Port solid FCS priority x 1 Effective consultant use at appraisal x 1 Satisfactory IFC front-end work x x x x 4 Satisfactory supervision and administration x x x x 4 Source: Independent Evaluation Group evaluative notes. Notes: AS =Advisory Services; COI = conflict of interest; FCS = fragile and conflict-affected situation; IS = investment services; MFI = microfinance institution. A common contributing factor to unsuccessful FCS project performance was weak IFC front-end work (table B.4). This was manifest in weak understanding of business strategy, flawed structuring,8 weak risk assessment, weak due diligence, and weak partner selection. In one case, an ill-suited funding source contributed to an unsuccessful project performance. This factor, which involved relying on public securities markets to raise additional capital to address cost overruns instead of on a few patient, experienced investors with deep pockets, is likely worth considering where a challenging environment may create the need for additional funding. This is because it can become a costly distraction to the management team, as it was in this case. In only one of three projects, weak IFC supervision and administration was a contributing factor to unsuccessful project performance. 88 Appendix B Results and Performance of International Finance Corporation Investment Projects Table B.4. Fragility, Conflict, and Violence: Factors that Seem Not to Work Serial No. Factor 1 2 3 Frequency Weak IFC front-end work caused by: x x x 3 Weak understanding of business strategy x x x 3 Flawed structuring x x x 3 Weak risk assessment x x x 3 Weak due diligence x x x 3 Selection of weak partner x 1 Ill-suited funding source x 1 Weak IFC supervision and administration x 1 Source: Independent Evaluation Group evaluative notes. Notes: IFC = International Finance Corporation. Performance of Equity Investments Improving the performance of IFC equity investments is important because IFC 3.0 calls for IFC to scale up equity investments, providing scarce equity capital in challenging markets. Equity (at 33 percent mostly successful or better) had the lowest rating on development outcome compared with loan and mix instrument types in CY15–17 (figure B.5, panels a and b). This was due to weak performance of equity projects in (i) TMT, Venture Capital and Funds industry group (with only 9 percent of the projects rated mostly successful or better)—especially in IFC- supported investment funds, where all the eight equity projects were rated mostly unsuccessful or worse in CY15–17; and (ii) infrastructure industry group (with only 20 percent of the projects rated mostly successful or better)—especially early exploration investments with junior mining companies in Africa. These mining companies had limited resources to overcome major challenges when faced with financial, operational, or project structure issues, which was the case during the commodities price downturn. 89 Appendix B Results and Performance of International Finance Corporation Investment Projects Figure B.5. Performance of IFC Instrument Types, CY12–14 and CY15–17 a. By number of projects b. By net commitment 90 90 83 Rated mostly successful or better (%) Rated mostly successful or better (%) 80 80 68 70 62 65 70 58 58 60 50 50 60 52 50 43 50 39 40 33 40 30 30 20 20 10 10 0 0 Equity Loan Both Equity Loan Both Instrument type Instrument type 2012-2014 2015-2017 2012-2014 2015-2017 Source: Independent Evaluation Group evaluative notes. Performance by Investment Size Large-size investments performed better than medium- and small-size investments. Large-size investment projects (at 60 percent mostly successful or better) performed better than medium-size investment projects (at 41 percent) and small-size (at 40 percent) in CY15–17 (figure B.6). Based on a review by IEG and IFC on IFC’s development outcomes and work quality, the difference in performance reflects the presence in large projects of more senior staff, the participation of industry specialists, and a greater managerial involvement of both IFC and its clients. There has been a significant decline in the performance of medium-size investments from 54 percent in CY12–14 to 41 percent in CY15–17 (figure B.6).9 This was due to weak performance of projects in the TMT, Venture Capital and Funds industry group (15 percent mostly successful or better), tourism, retail, construction, and real estate (14 percent), and oil, gas, and mining (18 percent) sectors in CY15–17. 90 Appendix B Results and Performance of International Finance Corporation Investment Projects Figure B.6. Performance of IFC Investment-Size Projects, CY12–14 and CY15–17 80 71 Rated mostly successful or better (%) 70 60 60 54 50 42 40 41 40 30 20 10 0 Small (<=$4.1 m) Medium (>$4.1 m & <=$38.1 m) Large (>$38.1 m) Investment size 2012-2014 2015-2017 Source: Independent Evaluation Group evaluative notes. Quality of IFC’s Supervision and Administration Quality of IFC’s Supervision and Administration has declined significantly from 82 percent in CY12–14 to 75 percent in CY15–17.10 The decline is significant, especially in the Middle East and North Africa Region, IDA projects, and in the infrastructure industry group. Evaluative notes finalized by IEG in CY17 for nine infrastructure projects for which IFC supervision and administration was rated as partly unsatisfactory or worse were reviewed to identify the main types of supervision deficiency. The deficiencies were (i) insufficiency of monitoring follow- up, where monitoring had been undertaken (observed in five projects);11 (ii) insufficiency of IFC’s environmental and social effects supervision and site visits applicable in situations in which IFC’s environmental and social effects monitoring, follow-up, or client engagement was found to be less than satisfactory (five projects); (iii) insufficiency regarding enforcement of covenants, rights, and remedies (four projects);12 and (iv) lack of adequate (non–environmental and social effects ) engagement (four projects).13 Given that supervision and administration is one of the important elements of IFC’s work quality, the recent deterioration in the quality of supervision and administration is a concern, especially when IFC’s 3.0 strategy is focusing on riskier markets. 91 Appendix B Results and Performance of International Finance Corporation Investment Projects 1 Statistically significant at 99 percent confidence interval. 2 Not statistically significant. 3Correlation coefficient (r) between: economic sustainability and development outcome is 0.8; project business success and development outcome is 0.73; private sector development and development outcome is 0.68; and environmental and social effects and development outcome is 0.28. 4Correlation coefficient (r) between project business success and economic sustainability is 0.58. 5 Statistically significant at 95 percent confidence interval. 6In some cases, the absence of any indication that a particular factor was present may be because it had not been discussed (or no supporting analysis was present) in the Evaluative Note; in others, it was because it was not present. 7Several, if not most, projects categorized as unsuccessful also had satisfactory environmental and social effects, thus caution in drawing conclusions is warranted. 8The most notable case of flawed structuring involved lending funds to a bank holding company with the funds destined to a subsidiary bank (within a regional bank group) without adequate controls and without IFC having an equity position or other leverage over the subsidiary to encourage timely reporting or compliance with loan covenants. 9 Statistically significant at 99 percent confidence interval. 10IFC’s work quality consists of the following two dimensions: (i) front-end work, and (ii) supervision and administration. Decline in the quality of IFC’s supervision and administration from 82 percent in CY12–14 to 75 percent in CY15–17 is statistically significant at 99 percent confidence interval. 11Such as, perhaps more proactive initial supervision on fine-tuning business plan and corporate governance, needed more active supervision for project site visits, or shortfall in monitoring systems and resource allocations. 12This is applicable to situations in which the IFC supervisory team failed to adequately enforce covenants, or exercise rights or take remedies allowed by contract or statutory law (for example, lapse in preparation of waiver memo for covenant breaches, needed more active supervision to enforce covenants or equity rights, and inadequate monitoring of capital expenditure [recorded but not flagged by analysis and ST financing]). 13This is applicable to instances in which the IFC supervision team did not engage adequately with the client or with other key shareholders (for example, the World Bank or 92 Appendix B Results and Performance of International Finance Corporation Investment Projects one or more relevant governmental agencies) regarding actions required to honor IFC Board Report representation or that were otherwise critical to the project’s performance. For example, failure to appoint IFC's board director, insufficient follow-up with state government, or insufficient engagement with government and the World Bank on sufficiency of tariff increase. 93 Appendix C. Results and Performance of International Finance Corporation Advisory Services Projects Results and Performance of the World Bank Group 2018 (RAP18) updated the analysis of the development results of Advisory Services projects by the International Finance Corporation (IFC) and IFC’s performance by including findings from 40 Advisory Services project evaluations completed in the fiscal year (FY)17 Project Completion Report (PCR) Program. A total of 140 IFC Advisory Services projects with completed evaluations as of October 5, 2018 were included in the three-year review period of FY15–17. This appendix presents project results data based on three-year rolling averages. Development effectiveness reflects the extent to which an intervention achieved its intended development results and private sector development. The development effectiveness rating implicitly addresses how well the project has contributed to fulfilling IFC's purpose and mission. Development effectiveness is a synthesis of the following five dimensions: strategic relevance, output achievement, outcome achievement, impact achievement, and efficiency. Trends in Performance Performance of IFC Advisory Services on development effectiveness has continued to decline. The proportion of IFC Advisory Services projects rated mostly successful or better on development effectiveness has declined significantly from 65 percent in FY12–14 to 37 percent in FY15–17 by number of projects1 (and from 72 to 34 percent by commitment amount), on a three-year rolling average (figure C.1, panels a and b). During the same period, the proportion of projects rated successful or better has also declined from 28 to 15 percent, whereas the proportion of projects rated unsuccessful or worse has increased from 13 to 26 percent. The net disconnect or gap between the development effectiveness ratings as reported by IFC (in its Self- Evaluation Reports) and Independent Evaluation Group (IEG; in its Evaluative Notes) has more than tripled from 11 percent in FY12–14 to 34 percent in FY15–17. The net disconnect or gap was higher than the IFC-wide average of 34 percent in FY15–17 for International Development Association (IDA; net disconnect or gap of 40 percent in FY15–17), Fragile and Conflict-affected situations (FCS; 42 percent), Sub-Saharan Africa (47 percent), and large-size projects (45 percent). 94 Appendix C Results and Performance of International Finance Corporation Advisory Services Projects All five dimensions of development effectiveness (strategic relevance, output achievement, outcome achievement, impact achievement, and efficiency) showed significant decline in their ratings. The ratings across all five dimensions of development effectiveness declined significantly in the range of 9 to 27 percent in FY15–17 (figure C.2, panels a and b).2 Among the indicators, outcome was strongly associated with development effectiveness—both declined at almost the same rate (26–27 percent) between FY12–14 and FY15–17. The efficiency indicator declined by 20 percent during the same period. IFC’s strategic relevance dimension was rated mostly successful or better in two-thirds of the projects in FY15–17 compared with approximately 80 percent of the projects in FY12–14. Among the dimensions of development effectiveness, impact achievement had the lowest rating of 16 percent in FY15–17 (figure C.2, panels a and b). Both IEG and IFC need to revisit the PCR Review Guidelines on rating the impact achievement indicator of development effectiveness because a high proportion of Advisory Services projects (42 projects or 30 percent) evaluated by IEG in FY15–17 did not have a numerical rating on impact achievement. That is, 18 projects were rated as “cannot be verified,” 16 projects were rated as “not applicable,” and 8 projects were rated as “too early to judge” on impact achievement in IEG Evaluative Notes. 95 Appendix C Results and Performance of International Finance Corporation Advisory Services Projects Figure C.1. Performance of IFC Advisory Services on Development Effectiveness, Three-Year Rolling Basis, FY08–17. a. Share of number of IFC advisory projects by development effectiveness rating, three-year rolling basis, FY08–17. 70 1 1 3 2 1 2 50 1 30 27 23 33 29 2 36 18 30 13 33 37 36 29 10 24 27 22 19 Percent 10 26 27 23 26 30 32 34 36 30 10 13 12 15 11 10 0 0 1 18 50 0 1 1 26 1 70 1 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (N=214) (N=249) (N=222) (N=210) (N=189) (N=185) (N=154) (N= 140) b. Share of expenditure of IFC advisory projects by development effectiveness rating, three-year rolling basis, $, millions), FY08–17 80 4 3 4 4 1 0 60 1 30 32 31 25 40 34 34 17 2 8 20 32 31 37 38 32 Percent 23 25 24 0 22 30 29 26 20 24 20 35 43 15 7 8 8 40 9 9 0 0 1 14 0 0 0 22 60 1 80 1 FY08-10 FY09-11 FY10-12 FY11-13 FY12-14 FY13-15 FY14-16 FY15-17 (V=64) (V=114) (V=146) (V=198) (V=210) (V=222) (V=186) (V= 176) Source: IEG evaluative notes. Note: FY = fiscal year; IFC = International Finance Corporation; V = total volume of projects ($, millions). All dollar amounts are U.S. dollars unless otherwise indicated. 96 Appendix C Results and Performance of International Finance Corporation Advisory Services Projects Figure C.2. Ratings for the Five Indicators of Development Effectiveness, FY12–14 and FY15–17 (percent mostly successful or better) a. By number of projects b. By project expenditure 100 89 100 91 85 Rated satisfactory or better 90 90 Rated satisfactory or better 78 80 80 76 80 71 71 66 64 65 70 70 62 60 60 50 44 50 45 39 37 37 40 40 35 30 30 16 16 20 20 10 10 0 0 2012-2014 2015-2017 2012-2014 2015-2017 Source: IEG evaluative notes. Notes: DE = development effectiveness; FY = fiscal year; SR = strategic relevance. Performance of IFC Projects in IDA and Non-IDA Countries Performance of Advisory Services projects in IDA countries (at 31 percent mostly successful or better) was below non-IDA countries (at 43 percent) in FY15–17. This was due to weak performance of Advisory Services projects (i) across three Regions (Sub-Saharan Africa, East Asia and Pacific, and Latin America and the Caribbean) with a combined performance of only 27 percent mostly successful or better; (ii) in large-size Advisory Services projects, especially in South Asia and Sub-Saharan Africa, with a combined performance of only 15 percent mostly successful or better; and (iii) in medium-size Advisory Services projects, especially in East Asia and Pacific and Latin America and the Caribbean, with a combined performance of only 8 percent mostly successful or better. Advisory Services constitutes one of IFC’s most important vehicles for market creation and supporting the private sector in 97 Appendix C Results and Performance of International Finance Corporation Advisory Services Projects challenging environments. Furthermore, IFC has an advisory portfolio of more than 700 projects worth more than $1.5 billion, which are strongly focused on priority areas (for example, 60 percent of IFC’s advisory work is in IDA countries).3 Review of Large Advisory Services Projects in IDA Countries A review was undertaken of 15 large IFC Advisory Services projects in IDA countries that were evaluated by IEG (information taken from IEG Evaluative Notes for each project) and for which the projects’ development effectiveness was rated as either mostly unsuccessful or worse. The objective of the review was to identify factors responsible for the projects’ weak performance. The assessments made in this review were not constrained to conform to the IEG ratings. For example, this review in some cases attributes weaknesses in project preparation and design to IFC front-end work, despite the IEG evaluation having rated that work as satisfactory. All 15 projects reviewed involved governmental agencies, if not directly as the client, then as a stakeholder with either a participatory or an oversight role. Most projects involved other institutional stakeholders, including private sector (commercial lenders or advisory institutions) and official international institutions (such as the World Bank, the International Monetary Fund, or regional development banks). What Seems Not to Work The factors presented appear to be the main ones responsible for Advisory Services project underperformance. In nearly all cases, performance shortfall was attributable to more than a single factor; in a few cases, most (though not all) of these factors were present in a single project. The factors are presented in three groups: (i) IFC project preparation and design, (ii) IFC project supervision, and (iii) external factors. Weak IFC Project Preparation and Design • Project design too complex. Excessive complexity (present in 10 of 15 projects) was manifest in too many components being addressed in the same project to the extent that project management was not able to exercise effective control. In some cases, it involved setting expectations too high 98 Appendix C Results and Performance of International Finance Corporation Advisory Services Projects regarding the ability of project clients and counterparts in a low-capacity country to deal with the complexity. It also entailed shortcomings at the project design stage in ascertaining the effective ceiling for the number of client institutions involved, beyond which their effective coordination became too cumbersome. The reason for the excessive complexity is not clear in many cases. In some, it might have been project preparation staff wanting to demonstrate their mastery of developmental issues and their complexity. In other cases, because many Advisory Services components are funded by donor concessional financing, donor pressure to undertake more with allocated resources within shorter time frames (both in terms of complexity and in breadth and depth of effort) might have been a major influence. • Lack of understanding of the theory of change. This factor, which was present in 13 of 15 projects, showed up in different ways. However, in several cases, an important reason seemed to be the implicit assumption that the project addressed a “binding constraint,”4 which proved not to be true after the fact (other essential factors and contributions may be missing, thus resulting in no evidence of positive impact).5 Another reason is that the presumed behavioral response to project achievement was wrong (private sector clients find Advisory Services products too sophisticated, too risky,6 or having some other drawback despite those outputs being technically sound). • Deficiency in establishing appropriate targets (subobjectives). This factor (present in 10 of 15 projects) revealed deficiencies at the project design stage of being overambitious or, in some cases, irrelevant (not measuring the right concepts). • Failure to establish at the design stage the appropriate data or information required for monitoring and evaluation (M&E) purposes. In 10 of 15 projects, inadequate attention was given to subsequent M&E, thus thwarting verification that outcomes and impacts were achieved. 99 Appendix C Results and Performance of International Finance Corporation Advisory Services Projects Weak IFC Project Supervision • Failure to revise project objectives and budget as circumstances change. This factor was present in 8 of 15 projects. In some cases, major changes in projects occurred without the objectives being modified accordingly. In other cases, additional work components were added but not adequately funded, or project scope was reduced without a commensurate reduction in budget. In one case, the project team revised the project objectives downward to what had already been achieved. • Lack of enough effort for the uptake of project products. This factor was present in 8 of 15 projects. Generally, this entailed the project team focusing more on technical achievement rather than final impact. It might be attributable in some cases to not having the right skills on the team, including not having selected the right team leader for the project. In other cases, political factors, changes in key client personnel, and lack of client ownership of project results were responsible. • Not enough effort to record information. This factor was present in 10 of 15 projects. Given the significant (though not exact) overlap with inadequate attention to M&E at the project design stage, it was not possible in this review to conclude whether this was mainly caused by shortcomings on the part of staff responsible for implementing and supervising the project or was the result of lack of guidance from project design as to what information to collect and report. In any event, this resulted in difficulties in verifying the achievement of targets, outcomes, and impacts. A concomitant weakness was failure to leave an audit trail. External Factors • Client or government commitment. Weak commitment on the part of the client institution or higher levels of government was present in 13 of 15 projects. In a few cases, this should have been anticipated during project design. In others, the reasons for lack of commitment seemed to be more obscure. This is a factor that will not be easy to ascertain except through more careful upfront risk assessment to truly align interests and through greater attention to organizational arrangements to meet ostensibly shared 100 Appendix C Results and Performance of International Finance Corporation Advisory Services Projects objectives. To the extent feasible and to enhance client commitment, IFC should in future strive to (i) reduce the number and sharpen the focus of project objectives, (ii) select project components that are essential for achieving a common purpose (that is, that are ideally “indecomposable” in terms of achieving outcomes and impacts), and (iii) ensure that organizational and working arrangements pay due regard to the optimum roles and functions of key stakeholders to achieve project success (including whether the project is designed as a consultancy or an advisory engagement). • Actions or inactions of others. This factor was present in 13 of 15 projects. Generally, it reflected a pronounced dependency of project impacts (if not always project outputs and outcomes) on contributions of other institutions (government, commercial banks, other donors, or international agencies and the World Bank). In some cases, this factor was manifest in others not fulfilling their contributions to objectives that were shared with the project. It is notable that at least a few of these likely became material problems because of the implicit assumption that the project was addressing a binding constraint, thereby taking other essential contributions for granted. In one case, another entity (including donor funding) was addressing similar, if not the same, objectives. The project design had materially underestimated the possibility of competition and the need for coordination. In at least two cases, IFC and World Bank priorities were misaligned, causing government confusion when it came to uptake. • Force majeure. This factor was present in 2 of 15 projects—political instability in one case, and a combination of extreme weather conditions and a breakdown in the rule of law in the other. • Changes in government (client) organization. This factor was present in 4 of 15 projects. It entailed institutional changes and the retirement of a senior official who had been the project’s champion in one case. 101 Appendix C Results and Performance of International Finance Corporation Advisory Services Projects Quality of IFC’s Work in Project Preparation and Design for Advisory Services Projects There was a strong association between development effectiveness and IFC’s work in project preparation and design. Weakness in IFC’s work in project preparation and design is a common contributing factor in Advisory Services projects rated mostly unsuccessful or worse on development effectiveness. Development effectiveness was associated with IFC’s work in project preparation and design in 75 percent of the Advisory Services projects in FY15–17 (figure C.3, panels a and b).7 Additionally, IFC Advisory Services projects with poor project preparation and design (those rated partly unsatisfactory or worse) rarely achieved mostly successful or better ratings on development effectiveness.8 This indicates a strong association between development effectiveness and IFC’s work in project preparation and design. Assessment of a sample of recently approved Advisory Services projects found recurring issues (such as broad scale and scope of project activities, unrealistic and ambitious time frames, and insufficient M&E planning) in IFC’s work in project preparation and design. Figure C.3. Association between Development Effectiveness and IFC’s Project Preparation Quality a. By number of evaluated projects b. By project expenditure Project Preparation Quality Project Preparation Quality LOW HIGH LOW HIGH Development Effectiveness Development Effectiveness 18 percent 26 percent High High 9 percent 8 percent $ 22 M n=9 n=27 $ 10 M 55 percent 16 percent 19 percent Low Low $ 69 M 49 percent n=16 $ 24 M n=50 Source: IEG. Note: M = millions (dollars). 102 Appendix C Results and Performance of International Finance Corporation Advisory Services Projects 1 Statistically significant at 99 percent confidence interval. 2 Statistically significant either at 99 percent or 95 percent confidence interval. 3 IFC Strategy Business Outlook Update FY19–21: Implementing IFC 3.0 (April 12, 2018). 4 A binding constraint is a constraint used in linear programming equations whose value satisfies the optimal solution; any changes in its value changes the optimal solution. When trying to find an optimal solution, the binding constraint is the factor on which the solution is more dependent. 5For example, in one case, the Advisory Services project did not adequately take the position of the banks into account while designing and implementing the Advisory Service. That is, the project did not consider whether the banks in the country considered Advisory Services project outcomes to have sufficiently improved small and medium enterprise creditworthiness to the extent that the banks would feel comfortable lending to them. This implies that in certain types of projects, contributions from other players are essential to positively affect the project’s outcomes. 6For example, in a country lacking legal safeguards and reliable electrical power supply, advisory recommendations encouraging small and medium enterprises or individuals to switch from traditional cash-based commercial transactions to adopting Advisory Services– provided online digital payment systems may be perceived as carrying excessive risk —this could be from possible cyberfraud or identity theft and risk that frequent power outages might either curtail livelihoods or force additional cost by necessitating backup power supply (for example, buying generators that are not needed with traditional methods). 7Twenty-six percent of IFC Advisory Services projects were rated positive on both development effectiveness and IFC’s work in project preparation and design. Similarly, 49 percent of IFC Advisory Services projects were rated negative on both development effectiveness and IFC’s work in project preparation and design. 8Only 9 percent by number and 8 percent by commitments have development effectiveness rated mostly successful or better when IFC’s work in project preparation and design is rated partly unsatisfactory or worse. 103 Appendix D. Results and Performance of Multilateral Investment Guarantee Agency Projects Results and Performance of the World Bank Group 2018 (RAP18) updates the analysis of the development results of projects by the Multilateral Investment Guarantee Agency (MIGA) and MIGA’s performance by including findings from nine project evaluations completed in the fiscal year (FY)17 Project Evaluation Report (PER) Program. Sixty-five MIGA projects with evaluations completed as of October 5, 2018, were included in the six-year review period of FY12–17. This appendix presents project results data based on six-year rolling averages. All success rate percentages in this appendix are based on number of projects unless stated otherwise. Development outcome aims to capture the project’s overall impact on a country’s economic and social development and, thus, is important as an implicit proxy for how well the project has contributed to fulfilling MIGA’s purpose and mission. Development outcome is a synthesis of the four indicators: project business performance, economic sustainability, environmental and social effects, and private sector development impact. Development outcome for MIGA projects are rated on a four-point scale: excellent, satisfactory, partly unsatisfactory, and unsatisfactory. Trends in Performance Performance of MIGA projects on development outcome has increased because of strong performance in both energy and extractive industries and infrastructure sectors.1 The proportion of MIGA projects rated satisfactory or better has increased from 62 percent in FY06–11 to 65 percent in FY12–17 by number of projects (and from 66 percent in FY06–11 to 70 percent in FY12–17 by gross issuance amount) on a six-year rolling average (figure D.1, panels a and b). This was because of the strong performance of MIGA projects in both energy and extractive industries and infrastructure sectors—a combined increase in performance from 52 percent in FY06–11 to 77 percent in FY12–17. The factors contributing to project success in these sectors include (i) projects with strong strategic relevance to countries together with a stable regulatory environment, (ii) sponsors or companies with strong experience and track records in relevant sectors, and (iii) stable market or 104 Appendix D Results and Performance of Multilateral Investment Guarantee Agency Projects demand and competitive products (for example, lower production costs of power generation projects supported by MIGA). Figure D.1. Development Outcome Ratings for MIGA, Six-Year Rolling Basis, FY06– 17 a. Share of MIGA guarantees by development outcome rating, by number, six-year rolling basis, FY06– 17 70 8 7 6 5 8 8 8 50 30 56 59 57 60 54 54 54 Percent 10 10 24 24 25 28 27 28 29 30 12 10 11 12 6 10 10 50 FY06-11 FY07-12 FY08-13 FY09-14 FY-10-15 FY11-16 FY12-17 (N=50) (N=50) (N=58) (N=63) (N=78) (N=78) (N=65) b. Share of MIGA guarantees by development outcome rating, by gross issuance $billion, six-year rolling basis, FY06–17 70 4 9 8 15 11 8 8 50 30 59 66 51 58 56 51 50 10 Percent 10 26 22 23 22 18 28 27 11 12 30 6 11 14 13 14 50 FY06-11 FY07-12 FY08-13 FY09-14 FY-10-15 FY11-16 FY12-17 (V=2.9) (V=4.2) (V=5.1) (V=5.4) (V=7.5) (V=7.7) (V=7.6) Source: Independent Evaluation Group evaluation notes and MIGA systems. Note: The gross issuance amounts include guarantees issued and additions to existing guarantees during the evaluation period. FY = fiscal year; MIGA = Multilateral Investment Guarantee Agency. 105 Appendix D Results and Performance of Multilateral Investment Guarantee Agency Projects Three of the four indicators of development outcome showed decline in FY12–17. The environmental and social indicator increased from 52 percent satisfactory or better in FY06–11 to 77 percent in FY12–17 (figure D.2, panels a and b). This was due to strong environmental and social performance across the energy and extractive industries (at 100 percent satisfactory or better), infrastructure (at 82 percent), and International Development Association (IDA) projects (at 83 percent). Among the indicators, the decline was notable in private sector development, from 84 percent satisfactory or better in FY06–11 to 65 percent in FY12–17. This was due to decline in ratings across finance and capital markets (from 88 percent in FY06–11 to 50 percent in FY12–17), IDA projects (from 86 percent to 63 percent), and Sub-Saharan Africa (from 83 percent to 52 percent). Figure D.2. Indicators of Development Outcome in FY06–11 and FY12–17 a. By number of projects b. By gross issuance 90 84 100 96 77 90 Rated satisfactory or better Rated satisfactory or better 80 81 70 70 65 66 65 80 70 62 60 66 60 55 70 66 66 62 62 52 57 60 50 50 50 40 40 30 30 20 20 10 10 0 0 DO PBS ECSUS ENV PSD DO PBS ECSUS ENV PSD 2006-2011 2012-2017 2006-2011 2012-2017 Source: Independent Evaluation Group evaluation notes and MIGA systems. Note: DO = development outcome; ECSUS = economic sustainability; ENV = environmental and social effects; FY = fiscal year; PBS = project business success; PSD = private sector development. The performance of financial markets was the weakest among the sectors. Both by number of projects (at 44 percent satisfactory or better) and by gross issuance amount (at 60 percent), the performance of MIGA projects in finance and capital markets was below the MIGA-wide average of 65 percent by number of projects and 70 percent by gross issuance amount in FY12–17 (figure D.3, panels a and b). This was due to weak performance of projects in Europe and Central Asia, where the concentration of finance and capital markets projects was high. Projects created in response to the global financial crisis (that is, MIGA guarantee support was 106 Appendix D Results and Performance of Multilateral Investment Guarantee Agency Projects provided as part of the global international financial institution response to the Europe and Central Asia financial crisis under the Vienna initiative) contributed to the decline in performance in Europe and Central Asia. Of 18 MIGA projects evaluated in FY12–17 in the finance and capital markets sector, 14 projects (or approximately 80 percent) were concentrated in Europe and Central Asia. Finance and capital markets also had the lowest rating across all four indicators—project business success (44 percent), economic sustainability (33 percent), environmental and social (64 percent), and private sector development (50 percent)—compared with the other three sectors (energy and extractive industries, agribusiness and general services, and infrastructure). Figure D.3. Development Outcome Ratings, by Sector a. By number of projects b. By gross issuance 90 90 89 81 77 Rated satisfactory or better 80 80 Rated satisfactory or better 6967 69 70 70 6265 70 65 66 60 52 59 5860 60 50 44 50 40 40 30 30 20 20 10 10 0 0 AMS EEI/INFRA FINCAP Grand AMS EEI/INFRA FINCAP Grand total total 2006-2011 2012-2017 2006-2011 2012-2017 Source: Independent Evaluation Group evaluation notes and MIGA systems. Note: AMS = agribusiness, manufacturing, and services; EEI/INFRA = energy and extractive industries/infrastructure; FINCAP = finance and capital markets; FY = fiscal year. IDA projects have performed better than non-IDA projects did in FY12–17. One of three elements of MIGA’s FY18–20 strategy is a reaffirmed commitment to the poorest through support for projects in IDA countries.2 Also, under the 18th Replenishment of IDA, the private sector window seeks to expand private investment in IDA-only countries, with a focus on IDA-eligible fragile and conflict- affected situations, according to the MIGA Strategy and Business Outlook FY18–20. That said, the development outcome performance for IDA (at 70 percent satisfactory or better) was above non-IDA performance (at 61 percent) in FY12–17, and IDA performance increased compared with 59 percent in FY06–11 (figure D.4, 107 Appendix D Results and Performance of Multilateral Investment Guarantee Agency Projects panels a and b). This was due to the strong performance of IDA projects in the energy and extractive industries sector (at 78 percent mostly successful or better). Figure D.4. Development Outcomes for International Development Association and Non-IDA a. By number of projects b. By gross issuance 90 90 85 80 80 80 Rated satisfactory or better 70 Rated satisfactory or better 70 64 70 66 59 61 59 60 60 50 50 40 40 30 30 20 20 10 10 0 0 2006-2011 2012-2017 2006-2011 2012-2017 IDA Non_IDA IDA Non_IDA Source: Independent Evaluation Group evaluation notes and the Multilateral Investment Guarantee Agency. Note: IDA = International Development Association. MIGA’s assessment, underwriting, and monitoring quality was weak in the financial markets sector. At 39 percent satisfactory or better, both by number of projects and by gross issuance amount, MIGA’s assessment, underwriting, and monitoring quality for projects in the finance and capital markets sector was below the MIGA-wide average (61 percent by number of projects and 56 percent by gross issuance amount) (figure D.5, panels a and b). Lack of indicators to measure expected impacts, failure to explain outcomes and impacts, and lack of monitoring of development impacts and environmental and social aspects are linked to the decline in ratings. 108 Appendix D Results and Performance of Multilateral Investment Guarantee Agency Projects Figure D.5. MIGA’s Assessment, Underwriting, and Monitoring in the Finance and Capital Markets Sector a. By number of projects b. By gross issuance 90 90 88 81 81 75 77 Rated satisfactory or better 80 80 73 Rated satisfactory or betetr 70 61 61 70 63 60 54 58 56 50 60 50 50 39 39 40 40 29 30 30 20 20 10 10 0 0 2006-2011 2012-2017 2006-2011 2012-2017 Source: Independent Evaluation Group evaluation notes and MIGA systems. Notes: AMS = agribusiness and general services; EEI/INFRA = energy and extractive industries/infrastructure; FINCAP = financial and capital markets. 1The following are the four MIGA management sectors: (i) agribusiness and general services, (ii) energy and extractive industries, (iii) infrastructure, and (iv) financial and capital markets. 2The three elements of MIGA’s FY18–20 strategy are (i) a reaffirmed focus on the poorest through support for projects in International Development Association countries; (ii) a continuing emphasis on fragile and conflict-affected situations, where MIGA can have impact where private political risk insurance insurers are unwilling to go; and (iii) an expanded commitment to climate change mitigation and adaptation, targeting 28 percent of new issuance related to climate change mitigation or adaptation in 2020. 109 Appendix E. Analysis of Quality at Entry in World Bank Projects Contents Overview ............................................................................................................................................ 111 Context .............................................................................................................................................. 113 Literature Review and Background....................................................................................................... 113 Definition and Rating of Quality at Entry ........................................................................................... 115 Analysis Method .......................................................................................................................................... 116 Analysis of Quality Assurance Group Ratings and Independent Evaluation Group Ratings .............................................................................................................................................. 123 Quality at Entry in Project Appraisal Documents ............................................................... 126 Quality at Entry in Implementation Completion and Results Reports Reviews ........ 130 Outcome from Workshops and Interviews on Quality at Entry with World Bank Staff ............................................................................................................................................................ 133 Enablers of Quality at Entry .....................................................................................................................134 Challenges to Quality at Entry ................................................................................................................ 136 Levers that Facilitate Quality at Entry if Sufficient and Hinder Quality at Entry if Not ........ 138 Suggestions for Overcoming Challenges to Quality at Entry ......................................................143 Concluding Questions for Consideration.............................................................................. 147 References ....................................................................................................................................... 150 110 Appendix E Analysis of Quality at Entry in World Bank Projects Overview Previous work by the Independent Evaluation Group (IEG) and others notes an association between quality at entry and project performance. Findings from the literature and past Results and Performance of the World Bank Group (RAP) reports note that project performance, usually assessed in terms of outcome ratings, is correlated with quality at entry ratings. Other factors associated with project performance include project design, attention to risk, government commitment, and applying past lessons. Ex ante quality at entry assessment ratings by the Quality Assessment Group (QAG) were weakly correlated with IEG quality at entry ratings, but no discernible effect was found in comparing projects with and without QAG assessments. Overall, when analyzed through propensity score matching, controlling for project and country characteristics, no statistically significant difference was found between IEG ratings (on outcome, quality at entry, and other indicators) of projects with and without QAG ratings. Although this analysis did not find an effect of QAG assessment—in the form that assessment took in QAG’s Seventh and Eighth Quality at Entry Assessment (QAE7 and QAE8)—on outcome or other ratings at the end of the project, the analysis also does not disprove a connection. Attention to quality at entry is still warranted. Analysis identifies several recurring aspects of poor quality at entry, consistent with previous findings by IEG and others. Issues identified included the following: • Projects that were too complex given the country context. • Capacity assessments not done or done poorly. • Inappropriate results frameworks; examples included projects with logical flaws in design of monitoring and evaluation (M&E) or where key indicators were not appropriate or not measurable. • Lack of readiness for implementation, including procurement arrangements not in place or feasibility studies not completed. • Unrealistic timelines given the project objectives; participants noted pressure to deliver a greater number and volume of projects, limiting the time available to learn from the past and engage with clients. 111 Appendix E Analysis of Quality at Entry in World Bank Projects This analysis triangulated findings from: review and coding of recent project appraisal documents (PADs), recent Implementation Completion and Results Report Reviews (ICRRs), and interviews and workshops with World Bank staff about their recently approved projects. Analysis also found several examples of high quality at entry. Structured review and coding of 30 PADs of projects approved in FY17–18 and 100 recent ICRRs noted positive examples in: • Application of past lessons • Appropriate partnership arrangements with other donors • Fiduciary arrangements, and arrangements for conducting environmental and social safeguard assessments and for monitoring environmental and social safeguards during implementation. Many of the descriptions coded as examples of high quality at entry were found in the stratified random sample of the 30 PADs (not in the ICRRs or interviews and workshops). The fact that participants noted pressure to appraise and present their project with more optimism than realism suggests that some of the examples of high quality at entry found in the 30 PADs may be aspirational. Structured interviews and workshops with 104 World Bank operational staff and managers brought into focus several positive examples of quality at entry in recent projects. Key enablers of quality at entry mentioned included the following: • Local presence of World Bank staff that facilitates strong client relationships, which is especially important in fragile and conflict-affected situation (FCS) countries. • Effective internal processes that enable rather than hinder quality at entry. • Support to project teams from the country director and the Country Management Unit (CMU). Comparison of findings from the ICRRs with output from interviews and workshops suggest that some aspects of quality at entry may not be well covered by IEG’s ICRRs. Aspects noted in interviews and workshops but mostly absent from ICRRs were: 112 Appendix E Analysis of Quality at Entry in World Bank Projects • Political economy considerations • Borrower ownership or government commitment • Relationship with the client (or trust between the client and the World Bank) • Country knowledge on the part of the World Bank team • World Bank presence in the country • Task team leader (TTL) experience and team experience Suggestions for improving quality at entry arose from interview and workshop participants’ experience in their recently approved projects with decreased contestability in the review process, pressure to deliver, and inadequate attention to political economy. Participants advocated for more independence in the review process (for example, changing the practice of reviewers often reporting to the same management as the project team); greater discussion of technical aspects of projects; and more time, budget, and specialist support to address corporate priorities. Participants suggested greater efforts be made to recognize and acknowledge governance and political economy issues. Others proposed stronger incentives for working in FCS countries. Context The RAP 2018 team conducted this analysis of quality at entry in World Bank projects to complement the presentation of project and country performance data and to respond to requests by directors and management during discussion of RAP 2017 for information on the relatively low levels of quality at entry ratings over time. The analysis began with construction of a design matrix (table E.5). Literature Review and Background Previous work by IEG and others notes associations between quality at entry and project performance, usually assessed in terms of outcome rating. A structured literature review on project performance for RAP 2017 concluded that “measures of preparation effort, including the IEG rating of quality at entry and the quality of project design, indicated that higher preparation quality was associated with stronger project outcomes” (World Bank 2017, 181). Raimondo (2016) found that when controlling for quality at entry, good M&E quality is positively and 113 Appendix E Analysis of Quality at Entry in World Bank Projects significantly associated with project outcome. Vawda et al. (2003) found that the “probability of unsatisfactory outcomes for a project with ‘poor’ economic analysis was three to four times higher than that for a project with ‘good’ economic analysis at entry” (19). The quality of (project) preparation and supervision was found to have a positive and significant effect on a project’s probability of success in postconflict environments (Chauvet, Collier, and Duponchel 2010) and for investment project financing (IPF). 1 An in-depth portfolio review for RAP 2014 0F found that poor quality at entry was associated with poor outcomes, with no significant differences between small and large projects. Empirical work as far back as the Annual Review of Development Effectiveness 1997 suggested that the quality of World Bank performance at entry and during implementation was responsible for a 15–20 percentage point increase in the likelihood of a satisfactory project outcome. 2, 3 Further analysis showed that 1F 2 F, improved quality at entry (proxied by individual ratings for World Bank performance during identification 4 and appraisal) correlate with project outcome 3F ratings, while improving the quality of appraisal in line with the annual target for quality at entry (85 percent acceptable or good) 5 would see project outcome 4F improve from 71 percent (sample average) to 73 percent based on sensitivity tests of portfolio performance during FY95–97. Elements identified across various documents as worthy of focus for quality at entry were project design; articulation of project objectives and results frameworks; attention to risk (including fiduciary, governance, institutional capacity, and political); attention to learning (through systematic monitoring and evaluation); government commitment; implementation capacity; and applying past lessons. The remaining elements—unrealistic cost estimation, inadequate safeguards, and poor links to other projects—posed challenges but if addressed would work toward better quality at entry (see table E.1). 114 Appendix E Analysis of Quality at Entry in World Bank Projects Table E.1. Elements of Quality at Entry Noted as Opportunities for Improvement in Past IEG Reviews of Results and Performance Elements ARDE 1997 RAP 2014 RAP 2015 RAP 2016 RAP 2017 Project design ✓ ✓ ✓ ✓ Articulation of project objectives ✓ Articulation of results frameworksa ✓ ✓ ✓ Attention to riskb ✓ ✓ ✓ Attention to learningc ✓ Government commitment ✓ ✓ ✓ Implementation capacity ✓ ✓ Applying past lessons ✓ ✓ ✓ ✓ Unrealistic cost estimation ✓ Inadequate safeguards ✓ Poor links to other projects ✓ Source: World Bank 1997, 2014, 2015, 2016a, 2017. Note: ARDE = Annual Review of Development Effectiveness; IEG = Independent Evaluation Group; RAP = Results and Performance of the World Bank Group. a. May include setting up of a project monitoring information system and conducting baseline surveys. b. Risks include fiduciary, governance, institutional capacity, and political. c. Through systematic monitoring and evaluation. Definition and Rating of Quality at Entry IEG’s internal ICRR manual defines quality at entry as the extent to which the World Bank identified, facilitated preparation of, and appraised the operation such that it was most likely to achieve planned development outcomes and was consistent with the World Bank’s fiduciary role. World Bank performance in ensuring quality at entry is rated against 10 criteria that are weighted according to the operational, sector, and country context of each (see table E.2). Table E.2. Criteria for Rating Quality at Entry Criteria Strategic relevance and approach Technical, financial, and economic aspects Poverty, gender, and social development aspects Environmental aspects Fiduciary aspects Policy and institutional aspects Implementation arrangements Monitoring and evaluation arrangements Risk assessment World Bank inputs and processes Source: Independent Evaluation Group 2017. Quality at entry of World Bank projects is not systematically rated at appraisal or at the first Implementation Status and Results report. 6 Moreover, in contrast to the 5F 115 Appendix E Analysis of Quality at Entry in World Bank Projects practice at many multilateral development banks, the World Bank does not have a centralized quality control mechanism at entry (World Bank 2016b, 94). From 1996 to 2010, QAG monitored portfolio quality at entry. 7 Thereafter, units in the 6F Operations Policy and Country Services Vice Presidency took on quality monitoring. During IEG’s ICRR process, evaluators provide a rating for quality at entry based on their review of the description of the project’s components in the Implementation Completion and Results Report (ICR), and explanation of key factors affecting implementation and outcome of the project. Evaluators assess reasonableness of project components in relation to (i) achieving the project or program objectives, (ii) the capacity of the implementing agency (including administrative and financial management capacity), (iii) the likely availability of local funding, and (iv) lessons learned from relevant prior projects. Ratings of quality at entry are assigned using the scale shown in table E.3. Table E.3. Definition of Ratings Used by IEG for Quality at Entry Quality at Entry Rating Definition Highly satisfactory There were no shortcomings in identification, preparation, or appraisal. Satisfactory There were minor shortcomings in identification, preparation, or appraisal. Moderately There were moderate shortcomings in identification, preparation, or appraisal. satisfactory Moderately There were significant shortcomings in identification, preparation, or appraisal. unsatisfactory Unsatisfactory There were major shortcomings in identification, preparation, or appraisal. Highly There were severe shortcomings in identification, preparation, or appraisal. unsatisfactory Source: Independent Evaluation Group 2017. Analysis Method This analysis of quality at entry followed two paths. First, an econometric study using data from QAG and IEG investigated possible associations between the distribution of quality at entry ratings from the two data sources and then between projects with IEG outcome ratings and those with and without QAG assessments. The analysis first examined the correlation between the quality at entry rating from QAG-assessed projects from QEA7 and QEA8 (the last two rounds of QAG assessment) and IEG’s quality at entry rating based on a sample of 222 projects. 8 7F 116 Appendix E Analysis of Quality at Entry in World Bank Projects The analysis then used propensity score matching and applied probit and logit models to compare two samples of projects approved during FY01–07. The first group comprised 412 projects with both QAG assessments and IEG outcome ratings, and the second data group comprised 1,503 projects without QAG assessments and with IEG outcome ratings. Second, a multisource analysis, incorporating PADs, ICRRs, and workshops and interviews with World Bank staff, assessed the factors that helped or hindered quality at entry (see table E.5). A framework disaggregating quality at entry into 40 component elements based on the questionnaire used by QAG (FY06–07) 9 and the ICRR evaluator manual 10 8F 9F conceptualizes quality at entry in terms of five broad categories: strategic relevance and approach; technical, financial, and economic aspects; policy and institutional aspects; monitoring and evaluation; and risk. The framework was used to review and code the PADs of 30 projects drawn from 411 World Bank IPF projects approved in FY17 and FY18 and 100 ICRRs drawn from 396 World Bank IPF 10F projects with completion reporting and IEG ICR reviews in FY17–18.11 A codebook was developed to define how the narrative elements within the documents reviewed (PADs and relevant sections of ICRRs) should be interpreted with respect to each element of the framework (see appendix F). Using categorical and magnitude coding in the qualitative software DeDoose, coders assigned deductive codes to the 40 component elements (where applicable), assigning a magnitude to each text segment. If the text segment represented an example or aspect of high quality at entry, it was assigned a magnitude of +1 or “positive.” If the text segment represented an example or aspect of poor quality at entry, it was assigned a magnitude of –1 or “negative.” If the text segment represented an ambiguous aspect of quality at entry, it was assigned a magnitude of 0 or “neutral.” A search of the World Bank’s Business Intelligence database for IPF projects approved in FY17 and FY18 identified colleagues with potential insight on quality at entry. This search yielded a list of 378 TTLs assigned to those projects as of October 2018. Those TTLs were invited to attend one of a series of workshops or to participate in a telephone conversation and share their experience with quality at entry when conceptualizing and preparing their most recent project. Telephone conversations took place from October 8, 2018 to October 28, 2018. Workshops took 117 Appendix E Analysis of Quality at Entry in World Bank Projects place on October 23, 24, and 25, 2018. These workshops were complemented by structured interviews with senior staff in project identification and preparation roles. A total of 104 colleagues participated in either the workshops or telephone conversations (see table E.4). Table E.4. Characteristics of Participants in Quality at Entry Interviews and Workshops Number of Number of Characteristics Participants Characteristics Participants Location Field 44 Practice Group and Region EFI 14 HQ 60 (VPU) HD 19 Duty AFR 12 INFR 17 station EAP 13 SD 29 ECA 4 AFRVP 7 LAC 5 EAPVP 5 MNA 0 ECAVP 2 SAR 10 LACVP 5 Grade GC 2 MNAVP 1 GE 5 SARVP 5 GF 9 GG 55 GH 31 Gender Female 52 GI 2 Male 52 Source: Independent Evaluation Group. Note: AFR = Africa; AFRVP = Office of the Vice Presidency Africa Region; EAP = East Asia and Pacific; EAPVP = Office of the Vice President East Asia and Pacific Region; ECA = Europe and Central Asia; ECAVP = Office of the Vice President Europe and Central Asia Region; EFI = Equitable Growth, Finance, and Institutions (Practice Group); grade = job level, that is, the generic profile and minimum requirements for staff members at each hiring level; HD = Human Development (Practice Group); HQ = headquarters; INFR = Infrastructure (Practice Group); LAC = Latin America and the Caribbean; LACVP = Office of the Vice President Latin America and the Caribbean Region; MNA = Middle East and North Africa; MNAVP = Office of the Vice President Middle East and North Africa Region; SAR = South Asia Region; SARVP = Office of the Vice President South Asia Region; SD = Sustainable Development (Practice Group); VPU = vice presidential unit. All 104 participants in the interviews and workshops on quality at entry were asked to consider their most recently prepared project in the context of the following questions: • What aspects of identification, concept development, and preparation went well? What were examples of high quality at entry? 118 Appendix E Analysis of Quality at Entry in World Bank Projects • What aspects (of identification, concept development, and preparation) did not go so well? What (if any) were the obstacles to high quality at entry? • What would have to be different to remove the obstacles (if any) to high quality at entry? What changes would create the conditions needed to make overall improvements in quality at entry more possible or more likely? 119 Appendix E Analysis of Quality at Entry in World Bank Projects Table E.5. Design Matrix for Quality at Entry Analysis Main Questions 2. What are challenges to or 1. What is quality at entry? What are its weaknesses in quality at 3. What are possible Source of Information determinants? entry? solutions? Ratings over time Trends in quality at entry over time; 1 Existing data on projects, Quantitative analysis of determinants of countries, task team leads, and so quality at entry ratings; to be on complemented with more elaborate in- depth analysis for a random sample of recently closed projects (see below). Observations in previous IEG and Summary of challenges identified 2 other reports (for example, QAG) in the past (mostly past RAPs) Content analysis and coding a Challenges identified in recent 3 random sample of PADs of PADs (recent projects) projects approved during the past New versus old challenges: two years PADs of recent projects, compared with six Focus groups, with tracking of What defines quality at entry: Focus group What focus group participants What focus group 4 participant: participants can validate or add to the initial identified as challenges to or participants identified as Role framework. weaknesses in quality at entry possible solutions Location Global Practice Structured interviews, with several What defines quality at entry: Key What key informants identified as What key informants 5 types of key informants: informants can validate or add to the initial challenges to or weaknesses in identified as possible framework. quality at entry solutions 120 Appendix E Analysis of Quality at Entry in World Bank Projects Main Questions 2. What are challenges to or 1. What is quality at entry? What are its weaknesses in quality at 3. What are possible Source of Information determinants? entry? solutions? Role(s) Geography, including country office or headquarters Global Practice From ICRR data available since Trends in quality at entry over time Challenges identified in recent 6 2015 (from the portal-based ICRR Quantitative analysis of determinants of ICRRs (older projects but recently system) quality at entry ratings closed) Text from quality at entry sections Text from lessons sections Quality at entry ratings Efficacy ratings {would need to decide how to aggregate across efficacy ratings for multiple objectives} Efficiency ratings QAG data plus ICRR rating data: Is having an ex ante 7 Take the group of projects with process of quality QAG assessments and with ICRR assessment (a QAG ratings; identify a comparison review) associated with group (matching) of projects from higher ratings? 121 Appendix E Analysis of Quality at Entry in World Bank Projects Main Questions 2. What are challenges to or 1. What is quality at entry? What are its weaknesses in quality at 3. What are possible Source of Information determinants? entry? solutions? the same cohort, with ICRR ratings, that did not have QAG assessments QAG data plus ICRR rating data: To what extent do ICRR ratings of quality at 8 Take the (smaller) group of entry reflect QAG ratings for the same projects with QAG and ICRR projects? ratings of quality at entry; compare them Note: ICRR = Implementation Completion and Results Report Review; IEG = Independent Evaluation Group; PAD = project appraisal document; QAG = Quality Assessment Group; RAP = Results and Performance of the World Bank Group. 122 Appendix E Analysis of Quality at Entry in World Bank Projects Analysis of Quality Assurance Group Ratings and Independent Evaluation Group Ratings IEG conducted an analysis comparing ratings of quality at entry by the former QAG with ratings of quality at entry and project outcome by IEG for the same projects. This analysis was based on data from the last two rounds of QAG assessment, specifically, 245 projects in QEA7 and QEA8, of which 222 (90 percent) had received IEG ratings as of end-September 2018. This analysis found the overall quality at entry ratings for these projects (approved in FY04–05 and FY06–07) showed a statistically significant correlation between IEG’s quality at entry and outcome ratings at the 0.05 percent confidence level but explained only a small percent of variance in IEG’s quality at entry and outcome ratings (7 percent and 2 percent, respectively). In other words, the correlation was weak. Table E.5 shows the distribution of quality at entry ratings across both QAG and IEG rating scales by number of projects. Table E.5. Distribution of Quality at Entry Ratings: Quality Assessment Group versus Independent Evaluation Group IEG Quality at Entry Rating QAG Overall Quality at Entry Rating HS S MS MU U HU HS 2 S 6 31 10 2 MS 4 45 25 7 MU 5 31 25 3 1 U 10 6 6 1 HU Source: Independent Evaluation Group. Note: HS = highly satisfactory; HU = highly unsatisfactory; MS = moderately satisfactory; MU = moderately unsatisfactory; QAG = Quality Assessment Group; S = satisfactory; U = unsatisfactory. This finding is, perhaps, not surprising given that QAG and IEG ratings of quality at entry had several types of underlying differences, detailed as follows: • Criteria for measurement—QAG’s overall rating was based on eight dimensions,1 similar to IEG’s criteria, but the latter also considers M&E arrangements and World Bank inputs and processes. 123 Appendix E Analysis of Quality at Entry in World Bank Projects • Definition—IEG uses a different six-point scale from that used by QAG (see table E.6). • Approaches to data collection—QAG’s assessment relied on structured interviews based on a guidance questionnaire and included all relevant project documents in its evaluation; IEG reviews only the ICR and a few key documents such as the PAD and the relevant country strategy, and interviews only the most recent TTL of the project (who may not always be available). • Timing of the evaluation—QAG’s assessment was conducted immediately after Board of Executive Directors (Board) approval; IEG’s assessment, based on the self-evaluation done for the project, comes after project closure. This ex post approach means that some aspects of quality at entry may no longer be observable; however, the timing can allow for reflection on quality at entry and its contribution to development outcome. Table E.6. Quality at Entry Rating Scale: Quality Assessment Group versus Independent Evaluation Group Scale IEG QAG Highly satisfactory There were no shortcomings in An exemplary operation, demonstrating good identification, preparation, or practice in several areas of preparation and appraisal. design, part of a subset, at the upper end, of satisfactory operations Satisfactory There were minor Satisfactory or above on all key areas of shortcomings in identification, preparation and design, likely to meet stated preparation, or appraisal. development objectives, and complying with World Bank policies and procedures. The operation also seeks to achieve rewards commensurate with risks. Moderately satisfactory There were moderate Satisfactory on all key areas of preparation shortcomings in identification, and design, but showing some deficiencies preparation, or appraisal. and missed opportunities, part of a subset at the lower end of satisfactory operations Moderately There were significant Significant deficiencies in one or two areas of unsatisfactory shortcomings in identification, preparation and design, affecting prospects of preparation, or appraisal. reaching stated development objectives, or/and compliance with World Bank policies and procedures; the risks are not commensurate with the rewards. At the upper end, in quality, of the set of less than 124 Appendix E Analysis of Quality at Entry in World Bank Projects Scale IEG QAG moderately satisfactory (below-the-line) operations. Unsatisfactory There were major shortcomings Significant deficiencies in several key areas of in identification, preparation, or preparation/design appraisal. Highly unsatisfactory There were severe A broad pattern of deficiencies shortcomings in identification, preparation, or appraisal. Source: Independent Evaluation Group. Note: IEG = Independent Evaluation Group; QAG = quality assessment group. A total of 760 projects were assessed by QAG between 1997 and 2008. IEG’s analysis focused on a sample of these, excluding those not evaluated by IEG and those that were approved outside of the FY01–07 period, because not all QAG- assessed projects approved before FY01 could be individually identified. An initial comparison did not identify a statistically significant difference in IEG outcome ratings (or other ratings) between the 412 projects with QAG assessments and the 1,503 projects without (see figure E.1). Figure E.1. IEG Outcome Ratings for Projects Approved During FY01–07, with versus without QAG Assessment Percentage of Projects With QAG (N=412) Without QAG (N=1,503) 40 20 0 20 40 60 80 100 Source: Independent Evaluation Group. Note: HS = highly satisfactory; HU = highly unsatisfactory; MS = moderately satisfactory; MU = moderately unsatisfactory; QAG = Quality Assessment Group; S = satisfactory; U = unsatisfactory. 125 Appendix E Analysis of Quality at Entry in World Bank Projects Given a lack of detailed information on how projects were sampled in all rounds of QAG assessments and the need to avoid selection bias, IEG’s analysis refined this comparison with the use of propensity score matching. Project characteristics that existed before the QAG assessment were used to calculate the propensity score, including agreement type, product line, Practice Group, Region, large volume versus small volume, country status, and additional financing. The analysis applied both probit and logit models and calculated the average treatment effect and average treatment effect for the treated. This analysis did not find a statistically significant effect of having a QAG assessment (as it was done in QEA7 and QEA8) on IEG outcome ratings. This finding, however, does not necessarily suggest that early assessment of quality of entry would bring no benefits in terms of project outcomes. Quality at Entry in Project Appraisal Documents Output from the review and coding of the 30 PADs across the five framework categories incorporating 40 elements and subelements is shown in figure E.2 and figure E.3. In general, the descriptive elements found in PADs were coded as mostly positive, that is, contributing to quality at entry. This was especially evident for the strategic relevance and approach, monitoring and evaluation, and risk categories. Just one element of the strategic relevance and approach category (country finance—the assessment of government’s own funding for the overall program to be supported) was coded mostly negative. Links between analytical work and design of the operation subelements (part of the project design element in the strategic relevance and approach category) was another aspect of quality at entry that was coded mostly negative (see figure E.2). Fiduciary elements and macroeconomic policy framework elements emerged as challenges to quality at entry in the technical, financial, and economic aspects category, as did readiness of the first year’s program element in the policy and institutional aspects category (see figure E.2). Some framework elements were mentioned in some PADs as examples of good quality at entry and in other PADs as examples of poor quality at entry; these included implementation capacity (under the policy and institutional aspects category) and the availability of baseline data for relevant indicators (under monitoring and evaluation; see figure E.2). 126 Appendix E Analysis of Quality at Entry in World Bank Projects Figure E.3 shows the percentage of PADs with at least one text section coded as positive or negative for the particular quality at entry element. In PADs (the interview and workshop participants noted that PADs often take a quite optimistic view of project preparedness), far more elements were coded as positive examples of quality at entry (29 elements and subelements) than were coded as negative examples of quality at entry (19 elements and subelements). Additionally, of the positive elements and subelements, 16 of these were coded as positive in more than 70 percent of PADs (see figure E.3). 127 Appendix E Analysis of Quality at Entry in World Bank Projects Figure E.2. Assessment Framework for Quality at Entry, with Elements Frequently Assessed in Project Appraisal Documents as Positive and/or Negative Source: Independent Evaluation Group analysis. Note: M&E = monitoring and evaluation; negative = presenting challenges to quality at entry; positive = contributing to quality at entry. 128 Appendix E Analysis of Quality at Entry in World Bank Projects Figure E.3. Quality at Entry Framework Elements Assessed as Positive or Negative, Percent of Project Appraisal Documents Reviewed Percent of PADs with at least Percent of PADs with at least one one section coded as negative section coded as positive for this for this quality at entry… quality at entry element… 1. Strategic relevance and approach 93 1.2 Appropriate partnership arrangements with… 23 7 93 1.4 Borrower ownership/government commitment 80 30 10 1.6 Country strategy 100 1.7.1 Alignment between project design and… 93 57 1.7.3 Linkage between analytical work and design of… 3 3 100 1.7.5 Results framework 1.7.5.2 Appropriateness and adequacy of M&E design 2.1 Country knowledge 3 97 2.3 Fiduciary 27 40 27 2.3.2 Procurement arrangement and capacity 30 20 7 2.4.1 Economic rationale and analysis 13 73 2.5 Macroeconomic policy framework 67 20 3.1 Arrangement to ensure accountability to citizens 93 3.2.1 Arrangements for institutional capacity building 60 37 3.2.1.1 Commitment 3 3 3.2.3 Readiness for the first year's program 17 7 17 3.3 Quality of stakeholder analysis and consultation 7 90 4.1 Appropriateness of arrangements for evaluating… 3 80 10 70 4.3 Arrangements for monitoring safeguards 7 3 93 4.3.2 Arrangements for monitoring poverty and… 73 27 57 5. Risk 90 5.2 Risk mitigation measures 3 87 Negative Positive Source: Independent Evaluation Group evaluation based on review and coding of 30 project appraisal documents (PADs) of investment project financing projects approved in FY17–18. Note: Positive = contributing to quality at entry or examples of high quality at entry; negative = elements representing challenges to quality at entry or examples of low quality at entry; PAD = project appraisal document. Because PADs may include text segments that represent both positive and negative examples of quality at entry, 129 Appendix E Analysis of Quality at Entry in World Bank Projects the coding is not mutually exclusive, and therefore the percentages shown may add to less or more than 100 percent. Quality at Entry in Implementation Completion and Results Reports Reviews Output from review and coding of the relevant sections of 100 ICRRs of recently closed projects (FY17–18) across the quality at entry analysis framework categories is shown in figure E.4 and figure E.5. In contrast to the PADs, the sample of ICRRs shows a more even distribution of elements coded as positive versus negative examples of quality at entry. Elements coded relatively frequently as examples of high quality at entry were application of past lessons, appropriate partnership arrangements with other donors, fiduciary arrangements, and arrangements for conducting environmental and social safeguard assessments, along with arrangements for monitoring those safeguards during implementation (see figure E.4). Elements and subelements noted as areas of concern for quality at entry were complexity of project design, results framework, appropriateness and quality of M&E design, indicators (measurability and appropriateness), procurement arrangements and capacity, implementation readiness, issues with capacity assessment (that is, capacity assessment not done or done poorly), realism of implementation schedule, and lack of baseline data for relevant M&E indicators (see figure E.4). Elements for which both positive and negative examples of quality at entry emerged included project design (notably, the alignment between project design and objectives, and the link between analytical work and design of the operation), and risk (that is, quality and comprehensiveness of risk assessment, and risk mitigation measures; see figure E.4). Figure E.5. shows the percentage of ICRRs with at least one text section coded as positive or negative for quality at entry. Thirty-five elements or subelements of quality at entry were coded negatively in the 100 ICRRs, and 34 were coded positively. Among the positive elements, application of past lessons was coded positively in 64 percent of ICRRs, while project design was coded negatively for quality at entry in 36 percent of projects (see figure E.5). 130 Appendix C Special Analysis of Quality at Entry in World Bank Projects Figure E.4. Assessment Framework for Quality at Entry, with Elements Frequently Assessed in Implementation Completion and Results Report Reviews as Positive and/or Negative Source: Independent Evaluation Group analysis. Note: Positive = contributing to quality at entry or examples of high quality at entry; M&E = monitoring and evaluation; negative = elements representing challenges to quality at entry or examples of low quality at entry. 131 Appendix E Analysis of Quality at Entry in World Bank Projects Figure E.5. Quality at Entry Framework Elements Assessed as Positive or Negative, Percent of Implementation Completion and Results Report Reviews Reviewed Percent of ICRRs with at least Percent of ICRRs with at least one one section coded as negative section coded as positive for this… for this quality at entry… 1. Strategic relevance and approach 3 64 1.2 Appropriate partnership arrangements with other… 5 17 5 3 1.4 Borrower ownership/government commitment 1 2 1 1.6 Country strategy 3 36 21 1.7.1 Alignment between project design and objectives 9 5 19 3 1.7.3 Linkage between analytical work and design of the… 13 26 1.7.5 Results framework 15 3 30 3 1.7.5.2 Appropriateness and adequacy of M&E design 34 11 2.1 Country knowledge 1 1 2.3 Fiduciary 5 18 7 7 2.3.2 Procurement arrangement and capacity Negative 12 1 Positive 2.4.1 Economic rationale and analysis 2 5 2 1 2.5 Macroeconomic policy framework 1 3.1 Arrangement to ensure accountability to citizens 1 13 3 3.2.1 Arrangements for institutional capacity building 9 30 1 3.2.1.1 Commitment 16 6 3.2.3 Readiness for the first year's program 8 1 9 1 3.3 Quality of stakeholder analysis and consultation 2 5 1 1 4.1 Appropriateness of arrangements for evaluating… 1 5 3 4.3 Arrangements for monitoring safeguards 3 19 3 7 4.3.2 Arrangements for monitoring poverty and social… 1 6 9 1 5. Risk 1 26 36 5.2 Risk mitigation measures 24 25 Source: Independent Evaluation Group evaluation based on review and coding of 100 Implementation Completion and Results Report Reviews. Note: ICRR = Implementation Completion and Results Report Review; M&E = monitoring and evaluation; negative = elements representing challenges to quality at entry or examples of low quality at entry; positive = contributing to quality at entry or examples of high quality at entry. Because ICRRs may include text segments that represent both positive and negative examples of quality at entry, the coding is not mutually exclusive and therefore the percentages shown may add to less or more than 100 percent. 132 Appendix E Analysis of Quality at Entry in World Bank Projects Outcome from Workshops and Interviews on Quality at Entry with World Bank Staff To further inform the analysis of quality at entry, 104 World Bank staff who were recently involved in project preparation participated in interviews and workshops in which they reflected on three questions: What worked (for quality at entry)? What did not work? What would have to be different to improve quality at entry? Participants’ responses and reflections for the first two questions were classified subsequently into 15 themes: (i) relationship with the client, World Bank history in the country; (ii) implementation readiness and client capacity; (iii) political economy, country context, and client commitment; (iv) quality at entry in FCS environments; (v) analytical work to feed project preparation; (vi) project design; (vii) corporate priorities; (viii) the recent “drive to deliver;” (ix) increase in project complexity; (x) TTL turnover, teams, experience, skills, and composition; (xi) importance of local presence, country director, and CMU in decision making; (xii) the World Bank’s convening power as knowledge leader; (xiii) the review process and decreased contestability; (xiv) incentives; and (xv) institutional support, policies, and guidance systems. Further examination of the responses identified elements that acted as enablers of quality at entry, elements that posed a challenge for quality at entry, and elements that acted as levers—at times sufficient for quality at entry, at other times, hindering quality at entry. Figure E.6 provides a summary. The third and final question on which participants reflected was on what would have to be different to improve quality at entry. Participants responded with suggestions for overcoming challenges to quality at entry as these related to the project, the team, and Bank Group processes and organization. 133 Appendix E Analysis of Quality at Entry in World Bank Projects Figure E.6. Quality at Entry: Enablers, Levers, and Challenges Enablers of quality at entry Levers that facilitate if Challenges to quality at entry • Good relationship with sufficient and hinder if not • Political economy issues client—for example from • Support by the country • Challenges of operating in previous projects director and the Country fragile environments • Analytical work related to Management Unit • Drive to deliver, rush to country and sector done to • Experience and expertise of premature approval feed into project team • Push for including corporate • World Bank global knowledge • Cross–Global Practice priority topics without • Local presence, especially in collaboration adequate support FCS countries • Time and budget for • Insufficient depth and • Clear and appropriate results preparation contestation of quality frameworks with relevant • Internal review process reviews indicators • Implementation readiness • Team leader turnover and client capacity • Supportive internal policies, guidance, and systems Source: Independent Evaluation Group analysis. Note: FCS = fragile and conflict-affected situations. Enablers of Quality at Entry Relationship with Client—World Bank History in the Country Participants referred to this enabler in different ways: dialogue with the client, relationship with the client, getting the client on board, having the client in the driver’s seat, and ensuring the client owned the project. Many participants referred to local World Bank capacity in helping to engage the client. Participants noted the importance of having a local office—for ease of mobilizing a local co-TTL who understands the country context and can overcome communication barriers—and having on-site staff available to speak with client counterparts to help ensure that the project responded to the needs identified in the project development objective. Many participants noted that when the project was a continuation of a previous project and the client was familiar with the World Bank, then dialogue, commitment, and the relationship were much better. In other cases, participants 134 Appendix E Analysis of Quality at Entry in World Bank Projects pointed to the helpfulness of analytical work in beginning the dialogue with the client and to pilot studies and workshops for getting the client on board. Analytical Work Related to Country and Sector Done to Feed into Project Many participants spoke of the importance of analytical work—including good economic analysis—if the project was new (that is, in a new country or new sector), whereas other participants noted that technical studies are important when the project is complex. Many participants expressed appreciation for trust funds that allowed them to carry out analytical work. Participants noted that risk assessments as part of a sound economic analysis were critical, particularly in FCS countries or situations and also for category A projects. World Bank Global Knowledge A number of participants mentioned the importance of the World Bank’s convening power and its global knowledge base in helping to improve quality at entry. Some participants indicated that the World Bank’s ability to marshal resources from other agencies helped to bring knowledge and learning and create solutions. Participants said that being able to speak with other TTLs with similar project experience was extremely beneficial in project preparation. Local Presence, Especially in FCS Countries Several participants mentioned that local presence of the World Bank was critical in FCS countries. Participants noted that a small project can have a big impact in FCS environments and that a TTL in the field can have a better grasp of the challenges than a TTL from headquarters could. Some participants noted that the identification, conceptualization, preparation, and related processes function more smoothly with a strong country director who is willing to make difficult decisions, for example, to push back or call a halt if a proposed project is technically weak. Other participants noted the importance of clear guidance from the CMU when preparing projects. Clear and Appropriate Results Frameworks with Relevant Indicators Some participants said that there has been an improvement in the results frameworks of project documents. Several participants indicated that time spent on developing a good results framework was key to a project’s later success and that 135 Appendix E Analysis of Quality at Entry in World Bank Projects defining, clarifying, and developing the right indicators for a project made for good outcomes and ensured a more successful handover from one TTL to another, mitigating the negative effects of TTL turnover. Some participants indicated that core sector indicators ensure better monitoring and evaluation while other participants said that Program-for-Results has also helped to ingrain a culture of results and verification. In addition, several participants mentioned the importance of having a solid theory of change linking activities, results, and outcomes. Several participants noted that it is important to keep implementation in mind when designing a project and ensure that the concept note is aligned with the Country Partnership Framework, incorporates corporate priorities, and defines a realistic path for implementation. Challenges to Quality at Entry Political Economy Issues Many participants expressed the view that political economy is a major factor affecting quality at entry. A few participants cited examples of projects that were dropped even though they were prepared well, had met safeguard requirements, and were demanded by the client. Reasons cited by participants for these situations include a change of government or sometimes the emergence of governance issues. A few TTLs noted that governance issues are often known to the team or management but are withheld from the PAD because their inclusion would be considered too sensitive; in such cases, the TTLs noted, the issues emerge at implementation and cause a “great headache.” Challenge of Operating in Fragile Environments Several participants noted that achieving good quality at entry is difficult in environments where security issues and restrictions on travel make project preparation difficult. Some participants also noted that mobilizing teams for such environments can be problematic, despite recent attempts to create incentives for more staff to work in FCS countries. Drive to Deliver, Rush to Premature Approval A very large share of participants in structured interviews and workshops commented with concern on the recent push to prepare projects faster and with less support. Several TTLs noted that projects prepared in six months can be expected 136 Appendix E Analysis of Quality at Entry in World Bank Projects to have problem project status two years into implementation. Many participants said that when there is a time pressure to deliver a project to the Board, the speed sometimes affects the client’s readiness adversely, or it can mean that the project goes to the Board even when the client is not fully committed. Several participants said that this time pressure to deliver also means that teams are not in a position to devote sufficient attention to how the project will be implemented (versus only identifying what components the project will contain). Push for Including Corporate Priority Topics without Adequate Support Many participants stated that teams are under a lot of pressure to address corporate priorities with little guidance and often without an assigned specialist. Several participants mentioned specific challenges in keeping up with constantly-changing World Bank processes; inadequate support in procurement, financial management, and safeguards; and insufficient time or resources to conduct relevant economic analysis. Many participants expressed the view that the increasing demands from management to incorporate more corporate requirements pose significant obstacles to quality at entry. TTLs acknowledged the importance of corporate priorities (for example, climate and gender) but also stated that these priorities took up significant time and resources while TTLs and teams are trying to do more with less. Many TTLs said they felt overburdened. Some participants recounted instances when the incorporation of corporate priorities was required, even when the specific priority seemed irrelevant to the particular project. Many participants cited examples in which clients expressed dissatisfaction with the requirements arising from corporate priorities. Several participants noted that corporate requirements added complexity to the project, thus making the project objectives more difficult to achieve. Insufficient Depth and Contestation of Quality Reviews Many participants also questioned the review system, in particular the decline in contestability within the peer review process. Some participants identified the moral hazard of peer reviewers working for the same Global Practice in which the project sits; this led participants to question the independence of the review. Several TTLs noted that the reorganization has amplified this issue—in the past, reviewers were external, coming from different Regions and reporting to different 137 Appendix E Analysis of Quality at Entry in World Bank Projects managers. Other participants noted a lack of focus on technical aspects of the concept notes and PADs, with little questioning of the design, which participants also attributed to decreased contestability of the peer review process. Other participants noted that the emphasis on corporate requirements has reduced the time available for discussion of the technical aspects of the project in the review process and that review comments often focus on how the corporate requirements are being dealt with rather than the technical aspects of the project. Several participants expressed the view that the Quality Enhancement Review (QER) is seen as a rehearsal for the decision meeting and that the reduced time available for the QER means that the technical and design aspects of the project suffer. Participants also mentioned the lack of technical inputs by colleagues within the unit during the QER sessions as a shortcoming in project preparation. Furthermore, some participants said that placing the QER in the portal made it like another formal review rather than an informal discussion to alleviate issues that might arise at concept review. Team Leader Turnover A number of participants cited inexperienced TTLs as obstacles to quality at entry. Several participants reported that many experienced personnel left the organization during the corporate restructuring and downsizing. Other participants noted that the resulting increased project load means that more inexperienced staff are now leading projects earlier in their careers. Levers that Facilitate Quality at Entry if Sufficient and Hinder Quality at Entry if Not Support by the Country Director and the Country Management Unit Some participants viewed the role of the CMU as gatekeeper, with more responsibility for ensuring quality at entry as a result. Participants commented that teams do not always fully consider the positive role that the CMU can play and that the CMU’s local knowledge gives them insight into what works and what does not work in terms of the project. Several participants said that in their recent projects, the CMU helped enormously to ensure high quality at entry by interacting positively with the client and by making funds available for analysis. 138 Appendix E Analysis of Quality at Entry in World Bank Projects At the same time, many participants said that teams often feel pressure to incorporate mandates of the CMU, and the CMU may be under pressure to please (new) governments by, for example, promising fast project preparation. In such cases, the TTLs said it becomes difficult to deliver. In a few cases, the TTL said that the CMU had pressured them to finalize the project before it was ready. A few TTLs noted that they found collaboration with the CMU was better in the past when the CMU would share comments from the review team in advance, whereas now the project team is sometimes blindsided by comments at the review meeting. A few participants commented on the conflicts that can arise between the Global Practices and country directors/CMUs, while some noted that this tension can help to improve the review process. Other participants suggested that the Global Practices view the CMUs as policing their work. One participant noted that what helped quality at entry in their project was full support from the CMU with no interference from the Global Practices. Some CMU participants lamented the lack of metrics to assess quality at entry, constraining their ability to share their findings with Global Practices. Some participants said that although good gatekeeping for quality at entry requires substantive knowledge and diplomacy skills on the part of the country director, not all country directors have these skills and knowledge. Many participants commented on the desirability of a country director and CMU that communicates effectively, makes judicious and well-timed decisions, and has the emotional intelligence to work with the client, World Bank management, and the project team. Experience and Expertise of Team Many participants stressed the importance of an experienced, expert project team. One commented that when the team is experienced and knows what it is doing, everything works. Others noted that clients appreciated the varying experiences and expertise that team members brought to the process, including lessons learned from other countries. However, most participants cited constraints with respect to experience and expertise, mentioning the loss of experienced, senior TTLs during the reorganization process, not enough experienced TTLs to lead projects, and/or TTLs not having the requisite experience for the particular project. Several participants noted that TTLs need experience to understand the issues and must know when to push back on clients, but this is difficult for junior TTLs who may 139 Appendix E Analysis of Quality at Entry in World Bank Projects not ask the right questions of the client (or who may not ask enough questions). Some participants noted that the rotation requirement means that TTLs often do not have sufficiently broad experience. Collaboration across Global Practices Many participants noted that project design has become more complex with the need to include many Global Practices and a greater need to collaborate, which absorbs the TTL’s time. However, some participants spoke of the cross-learning that arises when many Global Practices are on the team and project preparation can be streamlined. Other participants spoke of little incentive to collaborate and noted a more siloed approach in which they try to do everything within their Global Practice. Many participants noted that requirements by management for larger projects, incorporating a range of skills and covering a number of sectors, made it difficult to avoid “Christmas tree” projects. Some participants said that with four to five lead specialists on a team, it is a challenge to narrow the scope of the project and/or keep the project simple. Other participants noted that often, projects were too ambitious for the capacity of the client, with complex designs making implementation difficult. Many TTLs reported that they faced pressure not to present too simple a project even when their view was that simpler is better. Time and Budget for Preparation Reflecting on their recent projects, many participants noted that good project preparation needs time. However, many participants noted that project preparation time has declined markedly in recent years and now lasts for one year or less, which works against satisfactory engagement with the client and the carrying out of relevant technical studies. Participants cited the demands on time arising from corporate requirements, leaving the TTL with even less time to engage with the client. Several participants also noted that there is insufficient time to learn from past lessons. Some participants also noted the negative impact of time on quality at entry, in cases where there was a “rush to the Board,” in particular for projects that were facing the end of an International Development Association (IDA) cycle. Many participants opined that building trust with a client is not something to be done quickly or sporadically. 140 Appendix E Analysis of Quality at Entry in World Bank Projects Several participants commented that the “how” of doing the project is not receiving sufficient attention or time, and that insufficient attention to foundational work can lead to an underestimation of risks and uncertainties that the project may encounter and that often risk mitigation is not well thought through. Other participants noted that risk analysis for projects involving the private sector was often only superficial and that technical work for reform projects is particularly important but sometimes not possible, given time and budget constraints. Two newer TTLs had positive comments on the operations portal. Most participants, however, expressed frustration with the time needed to input information, word count limitations, and instances when the portal stops working at critical junctures in the project preparation. In addition to time, many participants noted that good project preparation needs an adequate budget. In some cases, participants said that they had been lucky that the CMU had funds—or had access to trust funds—that they could use to finance preparation work. A number of participants noted that economic analysis in particular is often absent because of limited budget, especially in IDA or new International Bank for Reconstruction and Development member countries, and teams must either rely on donors or eke out space in the already-limited preparation budget. Some participants noted that the ceilings on preparation and variable costs constrained their allocations for analytical work and/or hiring decisions. A few participants said that hiring external consultants with the required expertise often meant overrunning the budget. Some participants noted that the budget exists for fiduciary and safeguards but not for technical studies. Other participants noted that the time for preparation has increased even with budget cuts such that projects now absorb the budget for supervision, which leaves no money for preparation of the next project. Several participants stated that they are also increasingly expected to identify and prepare bigger, more complex projects but with fewer resources for identification and preparation. Internal Review Process Some participants expressed the view that the internal review process, including peer reviewers, was a positive, noting the helpful comments and advice from colleagues, the benefits of a clearly defined system—the concept note, QER, and 141 Appendix E Analysis of Quality at Entry in World Bank Projects decision meeting—and the step-by-step process that facilitated better project preparation and quality at entry. Other participants expressed a different view, opining that the length of the review process causes stakeholders to lose sight of the big picture while the focus on reviews takes time away from technical analysis. For some participants, the informal nature of the QER was becoming more formalized and therefore less beneficial. Some participants noted that the reduced time for the QER meant that discussion often did not have time to delve into the technical aspects of the project. Implementation Readiness and Client Capacity A number of participants mentioned the challenge of finding the right balance between the global knowledge of the World Bank, the capacity of the client, and the political economy. Some participants expressed a view that World Bank investment in capacity is geared toward short-term gaps and that assessing capacity is not always straightforward—counterparts may change, individuals involved at the beginning of a project may not continue, and even with adequate capacity, buy- in/commitment may not be present. Some participants noted that the client may not always understand the project design, the procurement process, the way that the World Bank works, or the corporate requirements. Other participants said that ascertaining implementation readiness, particularly when decisions need to be made about client capacity, is a challenge. Supportive Internal Policies, Guidance, Systems Some participants reported that in their experience, poor quality at entry does not attract much management attention because of the understanding that things can change during implementation, and that negative quality at entry aspects in a project can be fixed during supervision. Several participants said they found the distinction between quality at entry and quality of supervision unhelpful, in part because it may give the impression that the world of preparation and the world of implementation are distinct. Some participants noted that flexibility in project design is key for success in implementation, while others expressed the view that demand for quality preparation is not high. A few participants expressed the view that the theory of change does not nurture creativity in projects and noted that project design tends to be a linear process that pays little attention to feedback loops or alternative approaches. Some participants 142 Appendix E Analysis of Quality at Entry in World Bank Projects viewed the Agile initiative as positive—for example, for project restructuring and also for simplifying and streamlining processes in project preparation—while others viewed it negatively, expressing the view that Agile is undermining the quality of projects. One participant cited an example of a team that used the Agile initiative as a reason for omitting discussion of the country context from the PAD. Many participants mentioned the option of restructuring projects as helpful, noting that although it takes time, effort, and client willingness to initiate restructuring, doing so can salvage an otherwise unworkable project. Suggestions for Overcoming Challenges to Quality at Entry Participants were also canvassed for their views on overcoming challenges to quality at entry. The suggestions are discussed below in terms of how they relate to the project, the team, and Bank Group processes and organization. This approach approximates the congruence model of organizational performance (see figure E.7), which suggests that to understand performance, one must first understand all the elements. If performance is taken to be quality at entry, then the project, the team, the Bank Group processes and organization (the formal organization), and the informal organization need to be understood. Figure E.7. Overcoming Challenges to Quality at Entry: Suggestions from Workshop and Interview Participants Informal organization INPUT World Bank Quality Group process Project at entry and Resources organization History Team Source: Adaptation of framework from Mercer Delta Consulting 2003. 143 Appendix E Analysis of Quality at Entry in World Bank Projects The Project Many participants suggested adding more time and budget to project preparation given that projects had become overcomplex, arising from increasing corporate requirements and demands from management for multisectoral projects. Several participants stated that aligning a project’s risks and complexity with its budget were critical to removing obstacles to quality at entry. Many participants said that they like the Agile framework but cautioned that it takes time to prepare a project, and management does not always recognize this. Several participants asked whether the move toward Agile calls into question the way the World Bank measures quality at entry and also the timing of assessment of quality at entry. Some participants suggested that it may be better to assess quality at entry after the first year of implementation. Some participants commented that currently there is no real-time systematic assessment of quality at entry. Some participants suggested that Operations Policy and Country Services could usefully disseminate information on “what works” or on “how Regions compare” in terms of quality at entry. Several participants suggested that the results matrix could be improved further and could also be used to gauge quality at entry. Many participants suggested that economic analysis should be used to justify selected projects and to better allocate funds to project preparation, training, and setting up structures for project implementation. A few participants noted that trust funds are used for these efforts but project preparation advances are underused and often returned because of poor planning by teams. Many participants suggested that the relationship with the client was key to keeping the project design realistic. A number of participants suggested that a project that is doable and measurable should be the guiding principle. Some participants suggested pilot testing the project and then rolling it out gradually. Several participants also noted, however, that sufficient time—for capacity building and pilot testing—is not always available. Several participants also expressed the view that political economy issues should be documented and addressed. The Team A few participants suggested that teams be provided with specialists and budgets to fulfill corporate requirements or be allowed the discretion to waive the 144 Appendix E Analysis of Quality at Entry in World Bank Projects requirements that are not directly relevant to the project. In addition, participants stressed the importance of having sufficient support from financial management, procurement, and safeguard specialists. Some participants suggested additional elements that may help overcome challenges to quality at entry, such as the team insisting on an implementation support plan, having realistic targets, and having greater incentives for TTLs to stay with a project beyond preparation through implementation. Ensuring a good level of quality at entry is critical when working in an FCS environment, and a number of participants had some ideas for improving quality at entry in these situations. Some participants expressed the view that it was important to examine the true cost of operating in FCS environments and the incentives for staff to work there. Other participants noted that the headline news that the 18th Replenishment of IDA has committed additional resources and staff placements for FCS countries failed to differentiate between the different working environments. Several participants suggested that staff would be more amenable to going to countries that have good duty of care. Other participants suggested hiring a local security officer as part of the preparation team. A few participants highlighted the potential for projects in FCS countries arising from new technologies such as geomapping, which can help ensure that benefits are reaching those most in need. Some participants noted that it is important to have committed champions of the project in government, especially the Ministry of Finance and/or Ministry of Economic Development, and that the prospective project should have strong links to the government program. Other participants suggested that the country director or CMU can help in ensuring that the project responds to the needs of the client. Several participants suggest that the CMUs could become active in organizing and funding technical studies that would then allow the TTL additional time to finish preparing the project. Several participants noted that clients understand their own country context best and have increasing access to knowledge. A number of participants noted that in many cases, however, clients often suggest a simple, effective solution, and the World Bank needs to appreciate this and work with it. Many participants expressed the view that they are overburdened, with little time to engage the client. Several participants noted that they are spread thinly with little time to codify lessons learned, and a few participants suggested introducing a 145 Appendix E Analysis of Quality at Entry in World Bank Projects system of easily accessible lessons. A few participants commented that there is no incentive to learn from the past. Participants said that they are spending too much of their time managing internal processes with little time to discuss project components. A few participants suggest hiring more E- and F-level operational staff to address these challenges. At the same time, several participants noted that the loss of GH staff during the reorganization has left a gap in terms of the experience and expertise that these staff brought to project preparation. Many participants commented that senior staff are too stretched, with reduced opportunity for mentoring. World Bank Group Processes and Organization A number of participants suggested that country directors should take the lead and provide the country context to management with the aim of making some of the corporate requirements redundant (for a particular project). Some also suggested allocating more time to project preparation, or stalling a project that is proving to be a problem project. Many participants commented that there was a reluctance to call a halt to problem projects, even with the option of restructuring. Several participants expressed the view that the World Bank could alleviate the pressure of the corporate priorities by reorienting toward its technical expertise and its knowledge base. Many participants noted that success is judged by projects going to the Board, and this poses a challenge for quality at entry. A number of participants expressed the view that there should be greater involvement of experienced, expert staff earlier on in the project (more involvement at the concept note stage), and that anyone associated with the project should know what is in the PAD. The Informal Organization A number of participants expressed the view that open conversations between colleagues are vital to designing good projects, although reaching out informally can be difficult—how does one know who are the right people? Other participants suggested that personal connections may not be enough, although this is often all they have, that is, “coffee with colleagues.” A few participants noted that the QER process has become too formal and commented that it should be an informal discussion with key people, lasting longer than the current one-hour discussion. 146 Appendix E Analysis of Quality at Entry in World Bank Projects Concluding Questions for Consideration The congruence model cautions that a change to any one of its components has repercussions for the other parts and the efficiency of the whole. Participants pointed to changes that were taking place in projects, the team, World Bank processes and organization, and the informal organization. Although some participants—mostly those relatively new to the World Bank—saw these changes as good for quality at entry, others (the majority) were less convinced. Changes in projects (with larger, more complex projects involving multiple Global Practices), staffing (with more junior staff preparing projects), organization (corporate requirements and review process), instruments (Agile framework), and clients (particularly in FCS environments) raise questions about how the World Bank views quality at entry. For example, the World Bank may wish to consider the following: • Do new strategies create a need to redefine quality at entry? • What would a systematic, commonly agreed definition of high quality at entry in projects look like? • What is the optimal balance between quality and delivery? • What is the right balance between optimism and realism in project design and planning? • What are the appropriate levels of optimism, realism, and candor in PADs (and other relevant project documents)? • To what extent has the way the World Bank undertakes analytical work and economic analysis changed? • How is the World Bank dealing with contestability in the review process? • What do new strategic priorities mean for staffing, skills, and incentives overall and related to the World Bank presence in the field in countries? • How can the World Bank ensure strong technical and leadership skills of country directors in all countries? 147 Appendix E Analysis of Quality at Entry in World Bank Projects • How should time and resources be prioritized in the new institutional environment? What should be let go? 1Project performance for investment project financing correlated with quality at entry, quality of supervision, monitoring and evaluation quality, and project size (World Bank 2015). 2This was second to borrower performance (a rating that was dropped as part of Implementation Completion and Results Report reform in 2017), which raised the probability of a satisfactory outcome rating by about 35 –40 percentage points (World Bank 1997). 3 This underscores the message from previous annual reviews: “Commitment and ownership of project goals by the borrower, together with high-quality project appraisal and supervision by the World Bank, can offset some of the effects of a weak policy and institutional environment. (Although these country-level factors can also affect the likelihood of project success)” (World Bank 1997, 19). 4This was already highly rated, and thus most improvement would come from targeting appraisal (World Bank 1997). 5The World Bank’s Corporate Scorecard monitors the share of projects with reported baseline data for all development objectives in the first Implementation Status and Results Report. Current targets (2017) for quality at entry and quality of supervision are 80 percent of project volume rated moderately satisfactory or above (World Bank Group Corporate Scorecards, October 2017). 6If quality at entry was systematically rated early in World Bank projects, “it would be extremely useful for predicting project performance” (World Bank 2015). 7The Quality Assessment Group was disbanded in 2010. The World Bank Group –wide Results Steering Group that was established in 2006 was also discontinued in 2010. 8The Quality Assessment Group conducted eight rounds of assessment on quality at entry of 760 lending projects from 1997 to 2008 with the last two rounds (QEA7 and QEA8) conducted in FY06 and FY08 covering projects approved in FY04–05 and FY06–07, respectively. 9The Quality Assessment Group questionnaire featured 141 individual subcriteria underlying the criteria (noted in italics). There were 13 subelements to assess strategic relevance and approach; 25 subelements for technical, financial, and economic aspects; 14 for poverty, gender, and social development aspects; 14 for environmental aspects; 2 for fiduciary 148 Appendix E Analysis of Quality at Entry in World Bank Projects aspects; 18 for political and institutional aspects; 11 for implementation arrangements; 13 for risk assessment; and 31 subelements for bank inputs and processes. 10Strategic relevance and approach; technical, financial, and economic aspects (for investment lending projects); poverty, gender, and social development aspects; environmental aspects; fiduciary aspects; policy and institutional aspects; implementation aspects; monitoring and evaluation arrangements; risk assessment; and World Bank inputs and processes. 11Of those 30 projects, 24 were initially sampled, and six more were sampled among the remaining population. Both samplings were stratified by Global Practice. The stratified sampling method first defined the population of projects that met the following conditions: approved in FY17–18; International Bank for Reconstruction and Development/International Development Association (IBRD/IDA) only; excluding small recipient-executed trust funds; excluding projects less than $5 million. This resulted in a population size of 411 projects. In the first round of the second step, 24 projects were stratified from the population by a combination of Global Practice and agreement type. A second round extracted 5 projects from the remaining population (387 projects). A third round extracted one project stratified by a combination of Practice Group and agreement type. The list of the combinations of Practice Group and agreement type were: EFI_IBRD; EFI_IDA; HD_IBRD; HD_IDA; SD_IBRD; SD_IDA; INFR_IBRD; INFR_IDA. 1The eight dimensions are: (i) strategic relevance and approach; (ii) technical, financial, and economic aspects; (iii) poverty, gender, and social development; (iv) environmental aspects; (v) fiduciary aspects; (vi) policy and institutional aspects; (vii) implementation arrangements; and (viii) risk assessment. 149 Appendix E Analysis of Quality at Entry in World Bank Projects References Chauvet, Lisa, Paul Collier, and Marguerite Duponchel. 2010. “What Explains Aid Project Success in Post-Conflict Situations?” Policy Research Working Paper 5418, World Bank, Washington, DC. Vawda, Ayesha Yaqub, Peter Moock, J. Price Gittinger, and Harry Anthony Patrinos. 2003. “Economic Analysis of World Bank Education Projects and Project Outcomes.” International Journal of Educational Development 23 (6): 645–60. Independent Evaluation Group. 2017. Guidelines on Review of ICRs. Evaluator Manual. Washington, DC: Independent Evaluation Group, World Bank. Raimondo, Estelle. 2016. “What Difference Does Good Monitoring and Evaluation Make to World Bank Project Performance?” Policy Research Working Paper 7726, World Bank, Washington, DC. Mercer Delta Consulting. 2003. “The Congruence Model. A Roadmap for Understanding Organizational Performance.” Mercer Delta Consulting LLC. World Bank 1997. Annual Review of Development Effectiveness. Independent Evaluation Group. Washington, DC: World Bank. ———. 2014. Results and Performance of the World Bank Group 2014. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/rap2014. ———. 2015. Results and Performance of the World Bank Group 2015. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/results-and-performance-2015. ———. 2016a. Results and Performance of the World Bank Group 2016. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/rap2016. ———. 2016b. Behind the Mirror. A Report on the Self-Evaluation Systems of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/roses. ———. 2017. Results and Performance of the World Bank Group 2017. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/rap2017. 150 Appendix F. Codebook for PAD and ICRR Qualitative Analysis Purpose A codebook was developed so that recent Implementation Completion and Results Report Reviews (ICRRs) and project appraisal documents (PADs) could be analyzed for elements of quality at entry. This appendix describes the codebook according to the general instructions for coding the ICRRs and PADs and presents a matrix of the elements of quality at entry classified according to the codes used by level, name, description, and risk. DeDoose software was used for coding. The outcome of the coding analysis is outlined in appendix E. General Instructions • Each record contains four fields derived from the Independent Evaluation Group’s ICRR: o Project development objectives (PDOs; first field). PDOs are included in each file for background information. The PDO itself is not coded. Comments in the ICR review indicating judgment on the objectives (specific, vague, and so on) are coded. o The other three fields used for coding are: ▪ Quality at entry text ▪ Quality of supervision text: Codes are not attached to this field unless the text comments on project design. ▪ Lessons learned text: Codes are not attached to this field unless the text comments on project design. • The codebook has four columns: o Level (hierarchy of codes in the decimal system): ▪ First level: no digits after the decimal ▪ Second level: two digits after the decimal 151 Appendix F Codebook for Qualitative Analysis of Quality at Entry ▪ Third level: four digits after the decimal ▪ Fourth level: six digits after the decimal o Code: name of the code as it will appear on DeDoose o Coding description: definition and instructions for coding o Risk assessment: whether the code pertains to risk assessment (yes or no) o Additional instructions: notes for coding • Read each piece of text carefully and use the guide to highlight text and attach relevant codes in DeDoose. • The codes are divided into four hierarchical categories: o Strategic relevance and approach of the project o Technical, financial, and economic aspects o Policy and institutional aspects o Monitoring and evaluation • Each code is supplemented with a positive, negative, or neutral weighting. A positive weight indicates a generally positive review of the ICRR author’s comment, whereas a negative weight highlights the author’s negative statement in the text. If the text makes a neutral statement, attach a zero-weight code. (See examples in table F.1.) • Avoid making judgments for ICRR coding; instead, rely completely on the ICRR’s comments. For example, if a PDO statement is confusing or unclear, do not attach any code to it unless the ICRR text contains a statement noting that the PDO is unclear, vague, narrow, and so on. • Use your judgment for PAD coding when reviewing and coding text. Experienced ICRR evaluators and ICRR authors will code PAD. 152 Appendix F Codebook for Qualitative Analysis of Quality at Entry A single text segment might require two or more codes. For example, consider the following excerpt from an ICRR: “The hurried design diluted project capacity…” This text segment necessitates two identification codes: one that identifies a flaw in the project design and another that identifies capacity constraints. 153 Appendx F Codebook for Qualitative Analysis of Quality at Entry Table F.1. Codebook for Implementation Completion and Results Report Reviews and Project Appraisal Documents Level One Two Three Four Code Name Coding Description Risk 1.00 Relevance Strategic relevance and approach of the project 1.05 Articulate Articulation of project objectives: Use this code for text that comments on articulation of No the project development objective (PDO). Look for text that indicates whether the Implementation Completion and Results Report Review (ICRR) found the project objective: • Clearly articulated • Specific • Measurable • Vague • Convoluted • Not coherent • Too broad • Too narrow, and so on 1.15 Design Project Design 1.1505 Align Alignment of project design with the project objectives: Use this code to identify text Yes that comments on complementarity between project design and project objectives. Look for comments related to quality of program logic, results chain, theory of chain, and so on. An ICRR author may also comment on the quality of inputs and outputs in relation to expected outcomes. 1.1510 Complex Complexity: Use this code for text that comments on the complexity of the project No design. In other words, if the ICRR mentions that the project design was: • Too simple • Too complex 154 Appendix F Codebook for Qualitative Analysis of Quality at Entry • Of appropriate complexity 1.1515 Analytical Analytical framework: Use this code for text that remarks on the link between analytical No work and design of operation. Look for comments that identify the extent to which the analytical work informed design of the operation. 1.1520 Link Links between project and sectoral monitoring and evaluation (M&E): Use this code to Yes identify the ICRR author’s comment on links between project and sectoral monitoring and evaluation. Look for comments that focus on the compatibility or incompatibility of project and sectoral M&E. 1.1525 Results Results framework: Use this code to identify comments on coherence and consistency of No the results framework. 1.152005 Measure Measurability: Use this code for comments on the validity or appropriateness of No indicators defined in the results framework or M&E design at entry used to measure underlying elements. Look for comments that state if underlying indicators were: • Specific • Measurable • Achievable • Relevant • Time bound 1.152010 M&E design M&E design: Use this code to categorize text segments that comment on the M&E Yes design. Identify segments that comment on the appropriateness of evaluation design. Look for statements that indicate appropriateness and adequacy of M&E design. 1.20 Country strategy Consistency with country strategy: Use this code for remarks about alignment between No PDOs and country strategy. Look for statements that mention relevance of PDOs with: • Country Partnership Framework • Country Assistance Strategy 155 Appendix F Codebook for Qualitative Analysis of Quality at Entry • Country strategy 1.30 Lessons Applying past lessons: Use this code to identify the ICRR author’s observations about use No of (or learning from) experience to inform project design. Look for comments that identify the project’s (or project appraisal document’s) inclusion of past lessons or whether discussion of past lessons was absent. These lessons can include: • Previous projects • Industry experience • Sector experiences 1.35 Borrower Borrower ownership: Use this code to categorize statements about government No commitment. Look for text that comments on the role and engagement of government with the project. 1.40 Country finance Country finance: Use this code to identify comments on the degree to which a country’s No overall program operation is fully funded, which includes finance from the operation being considered. Look for statements that indicate whether or the extent to which project preparation included: • Assessment of the government’s own funding for the overall program to be supported by this project • Whether that funding was likely to be stable 1.45 Partnership Appropriate partnership arrangements with other donors: Use this code to categorize No statements about the extent to which project design took account of partnership arrangements with donors other than the World Bank. 2.00 TFE Technical, financial, and economic aspects 2.05 Sector knowledge Adequacy of sector knowledge: Use this code for statements about whether or to what No extent sectoral knowledge underpinned the project design. 2.10 Country Adequacy of country knowledge: Use this code to identify comments about whether or No knowledge to what extent knowledge about the country itself underpinned project design. 156 Appendix F Codebook for Qualitative Analysis of Quality at Entry 2.15 Economics Economic framework 2.1505 Macroeconomic Macroeconomic policy framework: Use this code to identify comments on the World Yes Bank’s assessment of the adequacy of macropolicy framework. Look for comments that identify the presence or absence of: • Consistent and realistic macroeconomic variables • Quality of macroeconomic analysis 2.20 Financial 2.2005 Economic Economic rationale: Use this code to categorize statements that comment on quality and Yes rationale coherence of economic rationale and analysis underpinning the project. 2.2010 Financial analysis Financial analysis: Use this code to identify comments on financial analysis underpinning Yes PDOs. If the project is a revenue-generating project, look for comments on financial soundness indicators such as: • Financial rate of return or net present value • Appropriateness of tariffs in relation to marginal cost • Operation and management cost • Reliability of financial reporting 2.2510 Procurement Adequacy and quality of procurement arrangements: Use this code to identify the ICRR Yes author’s comments on appropriateness of procurement arrangements. Look for statements that assess: • Significant implementation delays caused by procurement-related issues and their causes • Evidence of timely World Bank intervention in resolving procurement difficulties, providing procurement advice, or in giving nonobjection. 3.00 Policy Policy and institutional aspects 157 Appendix F Codebook for Qualitative Analysis of Quality at Entry 3.10 Stakeholder Quality of stakeholder analysis and consultation: Use this code to identify text that Yes comments on stakeholder analysis and consultation underpinning the design and preparation plans of the project. Look for statements that evaluate: • Exercises undertaken to identify stakeholders • Exercises undertaken to consult identified stakeholders 3.15 Implementation readiness 3.1505 Capacity Implementation capacity: Use this code to identify comments on existing Yes assessment implementation capacity. Look for statements that comment on: • Presence or absence of any existing implementation capacity assessment as part of project preparation • Quality of existing implementation capacity assessment 3.151005 Mandate Mandate: Use this code to identify the ICRR author’s comments that indicate the No presence or absence of clear mandate for project execution. 3.151510 Commitment Commitment: Use this code to identify comments on: No • Borrower’s or government’s commitment and incentive to implement the project • Borrower’s or government’s promotion of the stated objectives 3.1510 Institutional Capacity building: Use this code to identify text that comments on institutional capacity Yes capacity building building measures. Look for statements that comment on: • Implementation assumptions and their design and plans of capacity building • Any proposed measures to overcome gaps in implementation capacity 3.1525 Time Time: Use this code to identify the ICRR author’s comments on: Yes • Prospects for completing the project within the prescribed time frame • Readiness of the first year’s program for implementation 158 Appendix F Codebook for Qualitative Analysis of Quality at Entry 3.152505 Project Look for statements that comment on implementation delays caused by incomplete or No preparedness hasty preparation of the project: • Unrealistic implementation schedule, including failure to schedule sufficient time for start-up activities and mobilization • Changes in project or program scope 3.20 Accountability Accountability: Use this code to identify the ICRR author’s comments on stakeholder Yes involvement in implementation and oversight to ensure accountability to citizens. 4.00 M&E 4.05 Baseline Availability of baseline data: Use this code to identify the ICRR author’s comments about No availability availability and quality of baseline indicators. Look for comments that: • Identify the availability of baselines and targets for all underlying indicators • Assess the quality of the available baseline indicators 4.10 Evaluation Appropriateness of arrangements for evaluating impact and measuring outcomes: Use No arrangement this code to categorize text that comments on arrangements for evaluation and monitoring using relevant national, sectoral, or project-level data. Look for comments that identify the level of preparedness of evaluation planning. 4.15 Safeguards 4.1505 Social Arrangements for monitoring poverty and social aspects: Use this code to identify Yes arrangement comments that assess arrangements for monitoring poverty and social aspects. Look for statements that note whether the social assessment plans were prepared and prearranged. 4.1510 Environment Arrangement for monitoring environmental aspects: Use this code to identify comments Yes arrangement that assess arrangements for monitoring environmental aspects. Look for statements that note whether the environmental assessment plans were prepared and prearranged. 4.20 Gender Gender: Use this code to identify comments that assess arrangements for monitoring Yes gender aspects. 159 Appendix F Codebook for Qualitative Analysis of Quality at Entry 5.00 Good quotes Good quotations: Use this code to identify good quotes from the text for report writing. No 6.00 Risk 6.10 Risk assessment Use this code to identify comments on risk assessment. Look for words or statements that note presence or absence of a risk assessment. 6.15 Risk mitigation Use this code to identify comments on risk mitigation. Look for words or statements that note presence or absence of a mitigation plan. 7.00 Questions Use this code to: • Highlight about categorizing text • Questions for the group, task team leader, analyst • Uncategorized code 160 Appendix G. Background on World Bank Project Performance Ratings Trends, comparisons, and other analyses in this report that examine World Bank project performance often use quantitative or categorical data based on outcome ratings and World Bank performance ratings of World Bank projects. The ratings used to indicate World Bank project performance in this report are based on validation reviews of Implementation Completion and Results Reports (ICRs) completed by the Independent Evaluation Group (IEG). This appendix provides background on the methodology behind these ratings and some contextual analysis. What Is an Implementation Completion and Results Report? An ICR is one of the World Bank’s main instruments of self-evaluation. It is prepared by the World Bank at the close of every operation funded by the International Development Association or the International Bank for Reconstruction and Development or, for a series of programmatic policy operations, at the end of that series. According to the guidelines supplied to World Bank staff for preparing ICRs for investment project financing, the ICR is intended to (i) provide accountability and transparency for the performance and results of each operation and (ii) capture and disseminate experience from operational design and implementation to improve the design and implementation of future interventions. An ICR is expected to constitute a complete and systematic account of the project’s performance and results. In addition to telling the project’s results story, the ICR contains ratings of the project’s performance. The ratings often used in monitoring are the outcome rating (which is based on the subratings of the project’s relevance, efficacy, and efficiency), and the World Bank performance rating, which takes into account both the World Bank’s performance in ensuring quality at entry and the it’s performance in supervision of the project. Ratings scales and criteria were developed through a collaboration between the World Bank’s Operations Policy and Country Services (OPCS) and IEG. 161 Appendix G Background on World Bank Project Performance Ratings What Is an ICR Review? The ICR Review (ICRR) conducted by IEG is an independent, desk-based, critical review of the evidence, results, and ratings of the ICR in relation to the project’s design documents. Based on the evidence provided in the ICR and an interview with the final task team leader, IEG arrives at its own ratings for the project based on the same evaluation criteria the World Bank uses. In reviewing the findings and ratings in the ICR, IEG provides an independent view of the results and ratings conditioned on both the evidence presented in the ICR and the evidence provided by the project’s final task team leader. However, IEG is not privy to evidence that is not included in the ICR. The ICRR is thus an independent validation of the World Bank’s self-evaluation and ratings; it is not an independent evaluation of the project based on evidence collected outside the World Bank’s self-evaluation process. Ratings assigned by IEG in its ICRRs are aggregated into the performance percentages reported for World Bank projects in IEG’s Results and Performance of the World Bank Group (RAP) reports and in the World Bank’s Corporate Scorecard. Definitions of the criteria and scales for each rating element are provided in IEG’s Guidelines for Reviewing World Bank Implementation Completion and Results Reports: A Manual for Evaluators (World Bank 2014). Changes to ICR Reviews in 2017 Based in part on the response to the recommendation in IEG’s report Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group to avoid excessive focus on ratings and make the ICR system more flexible and geared toward learning, OPCS and IEG undertook a collaborative reform process to simplify the structure of the ICR for investment project financing and to improve related guidance (World Bank 2016). OPCS rolled out a new ICR structure on July 1, 2017 to be implemented in phases: Any ICR initiated on July 1, 2017 or later uses the new ICR structure and guidance, whereas any ICR initiated before this date uses the previous ICR structure and guidance, regardless of the project’s closing date. Ratings data used for this RAP, therefore, include ratings from ICRs and ICRRs that use the new structure and guidance as well as ratings based on the 162 Appendix G Background on World Bank Project Performance Ratings previous structure and guidance. Figure G.1 provides a summary of IEG’s ICRR ratings under the old and new structure. 163 Appendix G Background on World Bank Project Performance Ratings Figure G.1. Ratings Structure for IEG’s IPF ICR Reviews before and after ICR Reform Before ICR Reform After ICR Reform Relevance of Objectives Relevance of Objectives Relevance Relevance of Design Efficacy Achievement of Objective 1 Efficacy Outcome Achievement of Objective 1 Achievement of Objective 2 Achievement of Objective 2 Outcome Achievement of Objective 3 Achievement of Objective 3 Risk to Development Efficiency Outcome Efficiency Risk to Development Outcome Quality at Entry Bank Performance Quality at Entry Quality of Supervision Bank Performance Quality of Supervision M&E Quality Government Performance Borrower Performance IEG Only: ICR Quality Implementing Agency IEG Only: M&E Quality Legend: HU – U – MU – MS – S – HS (NR) Performance IEG Only: ICR Quality N – M – S – H (NR/NA) Source: Independent Evaluation Group. Note: ICR = Implementation Completion and Results Report; IEG = Independent Evaluation Group; IPF = investment project financing; M&E = monitoring and evaluation. Elements outlined in blue are rated on a six-point scale (from lowest to highest: highly unsatisfactory [HU], unsatisfactory [U], moderately satisfactory [MS], satisfactory [S], and highly satisfactory [HS]). Elements outlined in green are rated on a four-point scale (from lowest to highest: negligible [N], modest [M], substantial [S], high [H]). Elements in gray boxes are not rated but are discussed. 164 Appendix G Background on World Bank Project Performance Ratings Which Projects Are Covered by IEG’s ICRR Ratings? World Bank policy requires an ICR for each completed financing project to be submitted to the approving authority for that project. For financing projects funded through the International Bank for Reconstruction and Development or the International Development Association and approved by the World Bank’s Board of Executive Directors (Board), ICRs are submitted to the Board and, therefore, also submitted to IEG for validation. Financing projects for which an ICR is sent to an approving authority other than the World Bank’s Board—and that, as such, are not required to be submitted to IEG for validation—include financing projects in the following funding categories: • Recipient-Executed Trust Funds (although ICRs of recipient-executed trust funds of $5 million or more are not required to be submitted to IEG, many are submitted to IEG, and IEG completes ICRRs for any ICRs submitted.) • Global Environment Fund (medium size) • Carbon Initiative • Rainforest Initiative • Institutional Development Fund • Special Fund • Debt Reduction Fund Coverage of Project Ratings Data for FY16 and FY17 Arising from Waiver for the Republic of Yemen Portfolio On November 23, 2016, World Bank management waived the completion reporting requirement of Operation Policy and Bank Procedure 10.0 for 23 projects under implementation in the World Bank’s Republic of Yemen portfolio as of March 2015, when disbursements were suspended because of an ongoing civil conflict. Of these 23 projects, an ICR would have been required for 11 that closed in FY16 and FY17 (two projects totaling $53 million and nine projects totaling $183 million in disbursements, respectively). Without an ICR, IEG cannot provide ratings data. 165 Appendix G Background on World Bank Project Performance Ratings Coverage of Project Ratings Data for This Report As of October 3, 2018, IEG reached a coverage rate of 97 percent, completing 704 ICR reviews among 725 projects closed in FY15–17 and with ICRs submitted to IEG. This is similar to the 97 percent coverage rate in the Results and Performance of the World Bank Group 2017. Figure G.2 outlines the number of ICRs received by IEG for projects closed in each fiscal year and the percentage of ICRRs completed by IEG. Figure G.2. IEG’s ICR Review Coverage (as of October 3, 2018) Share of Completed ICR Reviews Among ICRs Received by IEG ICRs received (N) ICR reviews completed (%) 400 100% 100% 100% 350 300 100% 100% 100% 98% 100% 100% 100% 94% 250 99% 200 355 335 150 325 41% 284 275 283 243 240 244 263 100 230 232 50 121 0 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 FY18 Fiscal year of project closing Source: World Bank Business Intelligence; IEG ICR review process data. Note: FY = fiscal year; ICR = Implementation Completion and Results Report; ICRR = Implementation Completion and Results Report Review; N = number. This report is based on 97 percent coverage (704 ICRRs completed among 725 projects closed in FY15–17 with ICRs completed and received by IEG). Ratings data are as of October 3, 2018. By comparison, Results and Performance of the World Bank Group 2017 was based on 97 percent coverage (709 ICRRs completed among 734 projects closed in FY14–16 with ICRs completed and received by IEG, with ratings reported as of October 2, 2017). An important caveat is that the fiscal year of project closing is assigned based on information from the World Bank Business Intelligence system, which relies on accurate completion of the actual final closing date in the operations portal record for the project. The algorithm used to calculate the final closing fiscal year of a project is as follows: • Calculate the Derived Closing Date based on data availability of three fields in the following sequence: REV_CLOSE_DATE, CLOSING_DATE, and Deactivation Date; • Create a variable for Derived Closing FY based on the Derived Closing Date • Determine the Final Closing FY based on the following: o If the project status (PROJECT_STAT) is not Closed (that is, status is Active, Dropped, Legacy, or Pipeline), then make the Final Closing FY equal to the Evaluation FY (Eval FY); 166 Appendix G Background on World Bank Project Performance Ratings o If not (that is, if the project status (PROJECT_STAT) is Closed, then: ▪ If Derived Closing FY is blank (has no value), then make Final Closing FY equal to the Evaluation FY. ▪ If Derived Closing FY has a value, then compare that with the Eval FY as follows: • If Derived Closing FY is after Eval FY, then make Final Closing FY equal to Eval FY. • If Derived Closing FY is the same as or earlier than Eval FY, then make Final Closing FY equal to Derived Closing FY. References World Bank. 2014. Guidelines for Reviewing World Bank Implementation Completion and Results Reports: A Manual for Evaluators. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/ICRR_EvaluatorManualAug ust2014.pdf. ———. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/behindthe mirror_0716.pdf. 167 Appendix H. Analysis of Safeguard Reporting in Recent Implementation Completion and Results Reports Underlying this analysis is an assessment of the quality of safeguards reporting in the Implementation Completion and Results Reports (ICRs) of investment lending projects, which is conducted alongside the ICR Reviews (ICRRs) performed by the Independent Evaluation Group (IEG). In the safeguards reporting portion of the ICRR assessment, IEG evaluators examine the following: • Environmental Assessment category. Evaluators record the risk level category assigned to lending projects: category A (full, comprehensive environmental impact assessment following all the requirements specified in Operational Policy (OP) 4.01 for this category), category B (some type of environmental assessment, review, or plan is required but limited in scope and flexible in structure), category C (no environmental review is required), or category FI (project provides funds to a bank, credit institution, and so on, for onlending at financial intermediary’s own risk). • Safeguard policies. Evaluators note which safeguards were triggered in project appraisal documents and ICRs. These safeguards triggered may include environment assessments, natural habitats, pest management, indigenous people, physical cultural resources, involuntary resettlements, forests, safety of dams, projects on international waterways, and projects in disputed areas. • Evaluators also note which information is provided in the ICR on implementation of safeguard plans and on safeguard ratings. IEG conducted an analysis of the quality of safeguard reporting for 610 investment projects as part of the ICRR process for reviews posted between FY16 and FY18 (as of November 2018). Of these, 468 projects (77 percent) were assigned a risk level of either category A or B. Of the reports in these two categories, 71 projects (12 percent) were assigned to category A, and 397 projects (65 percent) were assigned to category B (see figure H.1). 168 Appendix H Analysis of Safeguard Reporting in Recent Implementation Completion and Results Reports Figure H.1. Distribution of Safeguard Risk Level Categories for ICRRs Posted between FY16 and FY18 28, 4% 71, 12% Safeguard risk level category 114, 19% Catergory A Category B Category C 397, 65% Category FI Source: Independent Evaluation Group analysis. Note: ICRR = Implementation and Completion Results Report Review. Labels indicate the number and percentage in each category. Analysis of the Quality of Reported Environment and Social Framework Safeguard System Outcomes in Category A Projects Of the 70 category A projects with final Implementation Status and Results Report (ISR) safeguard ratings, 33 (47 percent) received a satisfactory (S) rating and 2 received a highly satisfactory (HS) rating. An analysis of these two HS projects (the Southern West Bank Solid Waste Management Project [Social, Urban, Rural, and Resilience Global Practice] and the Huai River Basin Flood Management Project in China [Water Global Practice]—whose outcomes were also rated HS and S by IEG—would yield useful lessons on exemplary safeguard implementation in different contexts (which could inform the new environmental and social framework system upstream). The analysis also found a correlation between the safeguard ratings and IEG’s project outcome ratings. Twenty-eight (85 percent) of 33 category A projects that received an S or above final ISR safeguard rating received a moderately satisfactory or above (MS+) project outcome rating from IEG. Six category A projects received a moderately unsatisfactory or below (MU–) or lower final ISR safeguard, and these six also received an IEG rating of MU– on outcome. 169 Appendix H Analysis of Safeguard Reporting in Recent Implementation Completion and Results Reports Thirty-one (44 percent) of all 70 category A projects received a moderately satsifactory (MS) final ISR rating. In these cases, the ICRs often lacked an adequate description of how identified risks were managed. • In almost half of these projects, there was inconsistency between the final ISR rating of MS and the statement of successful or satisfactory implementation of safeguards in the ICR. Most of these ICRs also lacked explanation as to how the deficiencies identified at the final ISR stage were addressed at project closure. This reporting inconsistency often occurred in projects in China across Global Practices (such as Social, Urban, Rural, and Resilience; Transport; and Water) and was also noted in India, Kosovo, and Morocco and across Global Practices. Only two ICRs clearly reported on both the deficiencies identified at the final ISR stage and the reason why the rating was considered fully satisfactory at the ICR stage. (A project in China stipulated that the satisfactory implementation of the relevant operational policies was a condition of project closing, and a water supply project in Tanzania clarified that dam-related concerns were dropped from project plans because the activities were never implemented.) • In other cases, implementation deficiencies were recorded, but these varied widely, and it is not always clear why an MS rating was chosen instead of a lower rating. The descriptions of these deficiencies ranged from reporting on “health and safety concerns, leading to the unnecessary and preventable loss of lives and properties in the project contracts” in an Ethiopian road project, to a “lack of proper cleaning of a river bed from construction waste” in Georgia, to a “lack of reporting on the status of the Resettlement Action Plan” in an African regional project. • In four of the Category A projects with an MS final ISR safeguard rating, there was no environmental and social indicator outcome reporting in the ICR. In these cases, there is a description of the required procedures, but there is no reporting on whether the identified mitigation activities were satisfactorily implemented. 170 Appendix H Analysis of Safeguard Reporting in Recent Implementation Completion and Results Reports Reporting on Resettlement in Category A Projects Of the 70 category A projects with IEG ICRRs between FY16 and FY18 and a final ISR rating, 58 projects at appraisal and an additional 3 projects during restructuring triggered OP 4.12 on involuntary resettlement. Almost 50 percent (n = 30) of these projects have an S or above final ISR safeguard rating. Almost all of the projects included a clear description in the ICR of how the project complied with OP 4.12 (or why it was no longer applicable during implementation). They also included relevant information on how the World Bank helped clients overcome delays and difficulties, with many good practice resettlement techniques cited that could be mined for further learning (see figure H.2). Category A projects that received an MS rating at the final ISR stage often experienced challenges with the resettlement process, and these challenges were often cited as the reason for the MS rating. Of these, 25 percent did not include sufficient information in the ICR to validate compliance with the resettlement policy or safeguard, or they included language that did not explain the disconnect between the MS rating and the description of a fully satisfactory safeguard implementation experience, despite not completing the resettlement activities. The reporting in projects that triggered the resettlement policy but that had an MU or lower final ISR rating related to resettlement is also mixed. These important cases, in Brazil and India, provided a detailed description of the status of the resettlement process at project closure, including the reasons for the implementation shortfalls. In other cases, in Africa (a regional project and a project in Botswana), safeguard information was lacking in the ICR despite the below-the-line ISR rating related to the triggered resettlement policy. In all but one case, category A projects that trigger resettlement and that receive an MU or lower final safeguard rating were rated MU or lower by IEG with regard to the overall outcome rating. 171 Appendix H Analysis of Safeguard Reporting in Recent Implementation Completion and Results Reports Figure H.2. Distribution of Final Safeguard ISR Ratings and IEG Outcome Ratings for 70 Category A Projects with IEG ICRRs Completed between FY16 and FY18 IEG outcome Rating Final ISR Safeguards Rating 35 31 31 30 25 23 21 20 14 15 10 7 4 4 5 2 2 2 0 0 Source: Independent Evaluation Group analysis. Note: ICRR = Implementation and Completion Results Report Review; IEG = Independent Evaluation Group; ISR = Implementation Status and Results Report. 172 Appendix I. Gender Incorporation in Project Development Objectives, Components, and Indicators In fiscal year (FY)16, the Independent Evaluation Group piloted a systematic documentation of gender dimensions in individual World Bank projects as part of the Implementation Completion and Results Report Review (ICRR) process. Implementation Completion and Results Report (ICR) reviewers examined the following issues: • Whether gender is an explicit part of the project development objective (PDO) or one of the project components; and • Whether the ICR reports sex-disaggregated or female- or male-specific indicators. Building on the pilot analysis in the Results and Performance Report of the World Bank Group 2015 (RAP 2015), RAP 2016, and RAP 2017, RAP 2018 found that gender was an explicit part of the PDO or at least one of its components in only 114 (26 percent) of 440 projects reviewed during FY14–18 (see figure I.1, table I.1). Of the 114 projects that incorporated gender, almost one-third (37 ICRs) mentioned female beneficiaries. Of the 114 projects, 74 (65 percent) reported some sex-disaggregated or male- or female-specific indicators in the ICR while 13 (11 percent) reported female beneficiaries only in the ICR (see figure I.3 and table I.3). Among the 29 projects in which gender was explicitly part of the PDO, 22 (76 percent) reported gender markers at the PDO level in the ICR. Among the 326 projects that did not incorporate gender as an explicit part of the PDO or at least one of its components, 69 projects (21 percent) reported some sex-disaggregated or male- or female-specific indicators in the ICR while 47 projects (14 percent) reported indicators for the share of female beneficiaries (see figure I.4). The FY14–18 analysis also noted variations across Regions and Global Practices. The Human Development Practice Group had the highest proportion of projects that integrated gender into a PDO or component (45 percent), followed by Equitable Growth, Finance, and Institutions (29 percent) and Sustainable Development (22 percent). The Infrastructure Practice Group had the lowest 173 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators proportion of projects that integrated gender into a PDO or component (12 percent) (see figure I.1, table I.1). Among projects that integrated gender, the Human Development Practice Group had the highest proportion of projects reporting some level of gender markers in data (90 percent), followed by Sustainable Development (85 percent); Equitable Growth, Finance, and Institutions (51 percent); and Infrastructure (50 percent) (see figure I.3, table I.3). Among the Regions, South Asia had the highest proportion of projects that integrated gender into a PDO or component (42 percent), followed by the Middle East and North Africa (29 percent), Africa and Latin America and the Caribbean (28 percent), and East Asia and Pacific (22 percent). Europe and Central Asia had the lowest proportion of projects that incorporated gender into a PDO or components (10 percent) (see figure I.1 and table I.1). The presence of gender-related indicators in projects that incorporated gender in the PDO or at least one component ranged from all projects in the Middle East and North Africa Region to only 43 percent of projects reporting gender-disaggregated markers in Europe and Central Asia. Of the 36 projects in the Africa Region, 92 percent featured gender-related indicators, followed by 81 percent of projects in South Asia, 70 percent in Latin America and the Caribbean, and 47 percent in East Asia and Pacific (see figure I.3, table I.3). The time series analysis for FY15, FY16, and FY17 noted that the percentage of projects incorporating gender in their PDOs or at least one of the PDO’s components was highest in 2015, at 35 percent. This percentage decreased to 21 percent in FY16 and increased to 24 percent in FY17. Among the projects analyzed, 47 percent reported sex-disaggregated data at some level in 2015. This decreased to 41 percent in 2016 and increased to 51 percent in 2017 (see figure I.5, table I.5). 174 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators Figure I.1. Projects That Incorporated Gender in PDOs or at Least One Component, FY14–18, by Practice Group and Region Projects incorporating gender (percent) Bank-wide (n=440) 26 Equitable Growth, Finance, and Institutions (n=83) 29 Human Development (n=92) 45 Practice group or region Infrastructure (n=86) 12 Sustainable Development (n=179) 22 Africa (n=130) 28 East Asia and Pacific (n=76) 22 Europe and Central Asia (n=67) 10 Latin America and the Caribbean (n=83) 28 Middle East and North Africa (n=34) 29 South Asia (n= 50) 42 Gender was an explicit part of the PDO. Gender was an explicit part of at least one component (but not the PDO). Source: Independent Evaluation Group analysis. Note: n = number of projects; PDO = project development objective. 175 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators Table I.1. Projects Incorporating Gender in PDOs or at Least One Component, FY14–18 (by Practice Group and Region) Gender an Explicit Part of at Gender an Explicit Part of Least One Component but Practice Group and Number the PDO Not the PDO of Projects (percent) (percent) World Bank–wide (n = 440) 7 19 Equitable Growth, Finance, and 7 22 Institutions (n = 83) Human Development (N = 92) 15 29 Infrastructure (n = 86) 0 12 Sustainable Development 5 17 (n = 179) Africa (n = 130) 5 23 East Asia and Pacific (n = 76) 7 16 Europe and Central Asia 1 9 (n = 67) Latin America and the 8 19 Caribbean (n = 83) Middle East and North Africa 12 18 (n = 34) South Asia (n = 50) 12 30 Source: Independent Evaluation Group analysis. Note: n = number of projects; PDO = project development objective. 176 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators Figure I.2. Projects Incorporating Gender in PDOs or at Least One Component, Human Development and Sustainable Development Global Practices, FY14–18 Projects incorporating gender (percent) Bank-wide (n=440) 26 Human Development (n=92) 45 Practice Group and Global Practice Education (n=38) 32 Health, Nutrition, and Population (n=34) 59 Social Protection and Labor (n=20) 45 Sustainable Development (n=179) 22 Agriculture (n=49) 27 Environment and Natural Resources (n=32) 13 Social, Urban, Rural, and Resilience Global Practice 23 (n=56) Water (n=42) 21 Gender was an explicit part of the PDO. Gender was an explicit part of at least one component (but not the PDO). Source: Independent Evaluation Group analysis. Note: n = number of projects; PDO = project development objective. 177 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators Table I.2. Projects Incorporating Gender in PDOs or at Least One Component, Human Development and Sustainable Development Global Practices, FY14–18 Gender an Explicit Part of at Gender an Explicit Part of Least One Component but Not Practice Group and Number the PDO the PDO of Projects (percent) (percent) World Bank–wide (n = 440) 7 19 Human Development (n = 92) 15 29 Education (n = 38) 11 21 Health, Nutrition, and 26 32 Population (n = 34) Social Protection and Labor 5 40 (n = 20) Sustainable Development 5 17 (n = 179) Agriculture (n = 49) 8 18 Environment and Natural 3 9 Resources (n = 32) Social, Urban, Rural, and 7 16 Resilience (n = 56) Water (n = 42) 0 21 Source: Independent Evaluation Group analysis. Note: n = number of projects; PDO = project development objective. 178 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators Figure I.3. Presence of Gender-Relevant Indicators among Projects That Incorporated Gender as Part of the PDO or at Least One Component, FY14–18 Bank-wide (n=114) 24 76 Sub-Saharan Africa (n=36) 8 92 East Asia and Pacific (n=17) 53 47 Europe and Central Asia (n=7) 57 43 Latin America and the Caribbean (n=23) 30 70 Middle East and North Africa (n=10) 100 South Asia (n=21) 19 81 Equitable Growth, Finance, and Institutions (n=24) 49 51 Infrastructure (n=10) 50 50 Human Development (n=41) 11 90 Sustainable Development (n=39) 15 85 Education (n=12) 8 92 Health, Nutrition, and Population (n=20) 10 90 Social Protection and Labor (n=9) 11 89 Agriculture (n=13) 15 85 Environment and Natural Resources (n=4) 25 75 Social, Urban, Rural, and Resilience (n=13) 15 85 Water (n=9) 11 89 Governance (n=7) 71 29 Macroeconomics, Trade, and Investment (n=6) 50 50 Poverty and Equity (n=2) Trade and Competitiveness (n=8) 25 75 Energy and Extractives (n=6) 33 67 Transport and Digital Development (n=4) 75 25 ICR contains sex-disaggregated or male- or female-specific indicators. ICR reports only the share or percentage of female beneficiaries. No gender-specific reporting. Source: Independent Evaluation Group analysis. Note: ICR = Implementation Completion and Results Report; n = number of projects; PDO = project development objective. 179 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators Table I.3. Presence of Gender-Relevant Indicators among Projects That Incorporated Gender as Part of the PDO or at Least One Component, FY14–18 ICR Contains Sex- Disaggregated or ICR Reports Only the Region, Practice Male- or Female- Share or Percentage No Gender-Specific Group, or Global Specific Indicators of Female Reporting Practice and Number (percent) Beneficiaries (percent) (percent) World Bank–wide 65 11 24 (n = 114) Africa (n = 36) 69 22 8 East Asia and Pacific 47 0 53 (n = 17) Europe and Central 43 0 57 Asia (n = 7) Latin America and the 57 13 30 Caribbean (n = 23) Middle East and North 80 20 0 Africa (n = 10) South Asia (n = 21) 81 0 19 Equitable Growth, 38 13 49 Finance, and Institutions (n = 24) Infrastructure (n = 10) 30 20 50 Human Development 78 12 11 (n = 41) Sustainable 77 8 15 Development (n = 39) Education (n = 12) 67 25 8 Health, Nutrition, and 90 0 10 Population (n = 20) Social Protection and 67 22 11 Labor (n = 9) Agriculture (n = 13) 85 0 15 180 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators ICR Contains Sex- Disaggregated or ICR Reports Only the Region, Practice Male- or Female- Share or Percentage No Gender-Specific Group, or Global Specific Indicators of Female Reporting Practice and Number (percent) Beneficiaries (percent) (percent) Environment and 75 0 25 Natural Resources (n = 4) Social, Urban, Rural, 69 15 15 and Resilience (n = 13) Water (n = 9) 78 11 11 Governance (n = 7) 29 0 71 Macroeconomics, 17 33 50 Trade, and Investment (n = 6) Poverty and Equity 0 0 100 (n = 2) Trade and 63 13 25 Competitiveness (n = 8) Energy and Extractives 33 33 33 (n = 6) Transport and Digital 25 0 75 Development (n = 4) Source: Independent Evaluation Group analysis. Note: ICR = Implementation Completion and Results Report; n = number of projects; PDO = project development objective. 181 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators Figure I.4. Presence of Gender-Relevant Indicators among Projects That Did Not Incorporate Gender as Part of the PDO or at Least One Component, FY14–18 Bank-wide (n=326) 65 35 Sub-Saharan Africa (n=94) 45 55 East Asia and Pacific (n=59) 69 31 Europe and Central Asia (n=60) 80 20 Latin America and the Caribbean (n=60) 73 27 Middle East and North Africa (n=24) 71 29 South Asia (n=29) 62 38 Equitable Growth, Finance, and Institutions (n=59) 77 23 Infrastructure (n=76) 81 19 Human Development (n=51) 57 43 Sustainable Development (n=216) 54 46 Education (n=26) 50 50 Health, Nutrition, and Population (n=14) 57 43 Social Protection, and Labor (n=11) 73 27 Agriculture (n=36) 44 56 Environment and Natural Resources (n=28) 68 32 Social, Urban, Rural, and Resilience (n=43) 47 53 Water (n=33) 61 39 Governance (n=17) 76 24 Macroeconomics, Trade, and Investment (n=19) 79 21 Poverty and Equity (n=1) 100 Trade and Competitiveness (n=6) 50 50 Energy and Extractives (n=35) 74 26 Finance, Competitiveness, and Innovation (n=16) 81 19 ICR contains sex-disaggregated or male- or female-specific indicators. ICR reports only the share or percentage of female beneficiaries. No gender-specific reporting. Source: Independent Evaluation Group analysis. 182 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators Note: ICR = Implementation Completion and Results Report; n = number of projects; PDO = project development objective. Table I.4. Presence of Gender-Relevant Indicators among Projects That Did Not Incorporate Gender as Part of the PDO or at Least One Component, FY14–18 ICR Contains Sex- ICR Reports Only Disaggregated or the Share or No Gender- Male- or Female- Percentage of Specific Region, Practice Group, or Specific Indicators Female Beneficiaries Reporting Global Practice and Number (percent) (percent) (percent) World Bank–wide (n = 326) 21 14 65 Africa (n = 94) 32 23 45 East Asia and Pacific (n = 59) 24 7 69 Europe and Central Asia (n = 60) 13 7 80 Latin America and the Caribbean 13 13 73 (n = 60) Middle East and North Africa 13 17 71 (n = 24) South Asia (n = 29) 21 17 62 Equitable Growth, Finance, and 15 8 77 Institutions (n = 59) Infrastructure (n = 76) 5 14 81 Human Development (n = 51) 29 14 57 Sustainable Development 29 17 54 (n = 216) Education (n = 26) 27 23 50 Health, Nutrition, and Population 43 0 57 (n = 14) Social Protection and Labor 18 9 73 (n = 11) Agriculture (n = 36) 44 11 44 Environment and Natural 21 11 68 Resources (n = 28) 183 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators ICR Contains Sex- ICR Reports Only Disaggregated or the Share or No Gender- Male- or Female- Percentage of Specific Region, Practice Group, or Specific Indicators Female Beneficiaries Reporting Global Practice and Number (percent) (percent) (percent) Social, Urban, Rural, and 33 21 47 Resilience(n = 43) Water (n = 33) 15 24 61 Governance (n = 17) 18 6 76 Macroeconomics, Trade, and 5 16 79 Investment (n = 19) Poverty and Equity (n = 1) 0 0 100 Trade and Competitiveness 33 17 50 (n = 6) Energy and Extractives (n = 35) 3 23 74 Finance, Competitiveness, and 19 0 81 Innovation (n = 16) Transport and Information and 7 7 85 Communication Technology (n = 41) Source: Independent Evaluation Group analysis. Notes: ICR = Implementation Completion and Results Report; n = number of projects; PDO = project development objective. 184 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators Figure I.5. Percentage of Projects That Incorporated Gender in PDOs or Components Projects incorporating gender (percent of projects closed in each fiscal year) 35 24 21 28 16 16 7 5 8 FY 15 (n=114) FY 16 (n=150) FY 17 (n=148) Gender was an explicit part of at least one component (but not the PDO). Gender was an explicit part of the PDO. Source: Independent Evaluation Group analysis. Note: n = number of projects; PDO = project development objective. Table I.5. Projects That Incorporated Gender in PDOs or Components Gender an Explicit Part of at Gender an Explicit Part of the Least One Component but not Project Closing FY and PDO the PDO Number (percent) (percent) FY15 (n = 114) 7 28 FY16 (n = 150) 5 16 FY17 (n = 148) 8 16 Source: Independent Evaluation Group analysis. Note: FY = fiscal year; n = number of projects; PDO = project development objective. 185 Appendix I Gender Incorporation in Project Development Objectives, Components, and Indicators Figure I.6. Percentage of Projects Reporting Gender Markers Projects reporting gender markers (percent of projects closed in each fiscal year) 51 47 41 10 14 14 37 37 27 FY 15 (n=114) FY 16 (n=150) FY 17 (n=148) ICR reports only the share or percentage of female beneficiaries. ICR contains sex-disaggregated or male- or female-specific indicators. Source: Independent Evaluation Group analysis. Note: ICR = Implementation Completion and Results Report; n = number of projects. Table I.6. Projects Reporting Gender Markers ICRs Containing Sex- Disaggregated or Male- or ICRs Reporting Only the Project Closing FY and Female-Specific Indicators Share or Percentage of Number (percent) Female Beneficiaries (percent) FY15 (n = 114) 37 10 FY16 (n = 150) 27 14 FY17 (n = 148) 37 14 Source: Independent Evaluation Group analysis. Note: FY = fiscal year; ICR = Implementation Completion and Results Report; n = number of projects. 186 Appendix J. Management Action Record: Additional Data and Methodology Definition and Data Collection Process The Management Action Record (MAR) is an annual update process that follows up on the implementation of management’s action plans in response to sector, thematic, and corporate evaluations performed by the Independent Evaluation Group (IEG). MAR aims to create transparency about the progress made by World Bank Group management in addressing IEG recommendations and management’s action plans. Thus, MAR data—including IEG’s recommendations, management’s action plans, and subsequent updates and reviews—are available on IEG’s public website, making them accessible to external audiences. MAR tracks only those recommendations that are agreed or partially agreed on by Bank Group institutions’ management. A reform of the MAR process in 2011 resulted in the creation of a joint agreement between IEG and management to introduce management’s action plans that had specific targets and indicators and to rate progress against those action plans (and to no longer rate progress against adoption of IEG’s recommendations). For each agreed recommendation, management provides an action plan, progress against which is then assessed by Bank Group management and IEG independently. IEG provides comments on management’s action plans, but the responsibility for finalizing the action plans lies with management. During the MAR updates, each recommendation may either be tracked for four years to complete the implementation cycle or may be retired earlier than the fourth year. If a recommendation is retired because either management or IEG considers implementation to be complete, the recommendation remains in the set of active recommendations (with an inactive status) and exits the cycle at the end of the fourth year. The fourth-year ratings calculations are based only on the recommendations that are still being followed up on in their fourth year of implementation. The analysis of this year’s active MAR data provided in chapter 4 was based on the implementation of 126 recommendations drawn from 19 evaluations produced in fiscal year (FY)14–17 and their action plans and covering all three institutions. Approximately 72 percent of all recommendations targeted the World Bank, 20 187 Appendix J Management Action Record percent targeted the International Finance Corporation, and 5 percent targeted the Multilateral Investment Guarantee Agency. The analysis included both management and IEG’s ratings and the review of relevant updates. Focus Group Discussions: Methodology In this year’s MAR update cycle, the following IEG evaluations were in their final year of implementation tracking and will no longer be tracked through the MAR update: • World Bank Group Support for Innovation and Entrepreneurship: An Independent Evaluation • World Bank Group Support to Public Private Partnerships: Lessons from Experience in Client Countries, FY02–12 • World Bank Support to Health Financing: An Independent Evaluation • The Big Business of Small Enterprises: Evaluation of the World Bank Group Experience with Targeted Support to Small and Medium-Size Enterprises, 2006–12 • World Bank Group Assistance to Low-Income Fragile and Conflict-Affected States: An Independent Evaluation • The World Bank Group and Public Procurement: An Independent Evaluation To understand how IEG’s recommendations may have influenced the policy and operational work of the Bank Group over the past four to six years, IEG conducted focus group discussions with management counterparts and IEG staff on the evaluations exiting the MAR update cycle. The focus groups were conducted with IEG and management counterparts who were the original task team leaders of IEG’s evaluations and management responders to these evaluations, MAR update responders and managers, and some sector specialists. In total, IEG conducted eight focus group discussions and an additional four interviews for the above-noted evaluations. To ensure heterogeneity of focus groups and stimulate candid discussions, IEG held focus groups for IEG staff separately from management representatives, and each focus group targeted one evaluation only. All groups used the same standardized questionnaire to ensure uniformity and impartiality of data collection. The discussions probed perceptions 188 Appendix J Management Action Record of how IEG’s recommendations may or may not have contributed to the change in a given field. The discussions also tried to identify bottlenecks with translating IEG’s recommendations into specific management’s action plans and implementation of the action plans during the four-year cycle. Status of Management’s Action Plans As noted in chapter 4, Bank Group management delayed all action plans in response to IEG’s evaluations this year. Table J.1 outlines the status of all action plans as of March 31, 2019, as well as the number of days lapsed between the Committee on Development Effectiveness meeting date and the receipt of the final action plan, taking into consideration the 90-business-day period that management has to put together a draft action plan as agreed in the IEG–Bank Group working arrangement. The working arrangement also provides a 15-business-day period for IEG to comment on draft action plans and 20 working days for management to finalize the action plans after receiving IEG’s comments. Table J.1. Status of Management’s FY18 Action Plans (as of March 31, 2019) Committee on Development Effectiveness Green Sheet Final Action Number of Evaluation Meeting Issuance Plan Delayed Days Growing the Rural Non- AOB on September n.a. Overdue 285 business Farm Economy to 28, 2017 days Alleviate Poverty (FY18) Data for Development: August 30, 2017 December 1, Overdue 305 business An Evaluation of World 2017 days Bank Support for Data and Statistical Capacity (FY18) Mobile Metropolises: An August 30, 2017 December 15, Overdue 285 business IEG Evaluation of the 2017 days World Bank Group's Support for Urban Transport (FY18) 189 Appendix J Management Action Record Committee on Development Effectiveness Green Sheet Final Action Number of Evaluation Meeting Issuance Plan Delayed Days Growth for the Bottom October 25, 2017 November 9, Overdue 256 business 40 Percent: The World 2018 days Bank Group’s Support for Shared Prosperity (FY18) Toward a Clean World October 30, 2017 February 2, 2018 Overdue 253 business for All: An IEG days Evaluation of the World Bank Group’s Support to Pollution Management (FY18) The International January 10, 2018 December 3, Received 75 working days Finance Corporation’s 2018 September 7, (prior to Approach to Engaging 2018 receiving) Clients for Increased Development Impact (FY19) An Evaluation of the January 10, 2018 December 3, Received 150 business World Bank Group’s 2018 January 10, days Support for Water 2019 Supply and Sanitation with Focus on the Poor (FY19) Note: AOB = absence of objection; FY = fiscal year; IEG = Independent Evaluation Group. 190 Appendix K. Major and Meso Evaluations, Independent Evaluation Group Enhancing development effectiveness through excellence in independent evaluation is the driving force behind the 12 evaluation reports prepared by the Independent Evaluation Group (IEG) during fiscal year (FY)18. Summaries of the nine major evaluations and three meso evaluations are arranged alphabetically in this appendix, beginning with the major evaluations. The summaries fall loosely into three categories that cut across IEG’s dual goals: (i) what works and (ii) real- time learning. The first category includes summaries of four major evaluations that have a thematic or sectoral focus and involve in-depth studies of World Bank Group performance in pollution, health, urban transport, and water supply and sanitation. The second category comprises the summaries of three reports (two major and one meso) that examine the ways in which the business model of the Bank Group can be improved. Lessons from Bank Group partnerships with upper- middle-income countries, its support for data and statistical capacity, and the drivers of success for the use of development policy financing contribute to the goals of real-time learning and what works. The theme of the third category, which includes summaries from five reports (three major and two meso), is engagement and inclusiveness. Common themes across the sectoral evaluations highlight relevant sustainable development goals, the Bank Group’s convening and knowledge management powers, and maximizing finance for development. In this group of evaluations, sector studies inform on what works and why in health, urban transport, water supply and sanitation, and pollution. A number of the Sustainable Development Goal (SDGs) help inform recommendations for the health (World Bank Group Support to Health Services: Achievements and Challenges), water supply and sanitation (A Thirst for Change: An Evaluation of the World Bank Group’s Support for Water Supply and Sanitation with Focus on the Poor), and urban transport (Mobile Metropolises: Urban Transport Matters: An IEG Evaluation of the World Bank Group’s Support for Urban Transport) evaluations. For example, SDG 3 seeks to ensure healthy lives and promote well-being for all at all ages, SDG 6 seeks to ensure the availability and sustainable management of water and sanitation for all, and SDG 11 seeks to make cities and human settlements inclusive, safe, resilient, and sustainable. All sector studies remark on the importance of the Bank Group’s knowledge and convening powers and call on the Bank Group to use these to promote good practices and 191 Appendix K IEG Major and Meso Evaluations learning. Bank Group engagement in pollution management (Toward a Clean World for All: An IEG Evaluation of the World Bank Group’s Support to Pollution Management) is multisectoral and cross-cutting. The sector evaluations also examine the extent to which the private sector is being leveraged to support development and the obstacles to this involvement, including an overreliance on subsidies and transfers in certain sectors, underdeveloped private markets, inadequate regulation and regulatory enforcement, and low capacity that curtails synergies between the private and public sectors. Addressing knowledge and data gaps forms the core recommendations of the evaluations loosely grouped in the second category. The Bank Group’s approach to supporting development data—its production, sharing, and use—relies on its technical expertise and its ability to link global needs to national needs. Extending these areas further will greatly improve the way in which the Bank Group does business for itself and for its clients, even with the challenges noted in the evaluation (Data for Development: an Evaluation of World Bank Support for Data and Statistical Capacity). Lessons learned from Bank Group engagement in upper- middle-income countries highlights the extremely diverse development needs of these countries and the differing roles that the Bank Group plays in meeting the second-generation challenges facing them (World Bank Group Engagement in Upper Middle-Income Countries: Evidence from IEG Evaluations). As knowledge and data gaps are addressed, lessons are learned that can lead to improvements in the Bank Group business model. The meso evaluation provides evaluative insights into drivers of success and risks for a sample of development policy financing (DPF) operations. Lessons learned are timely in the context of the record 18th Replenishment of International Development Association (IDA18) and a declining share of DPF in IDA commitments (Maximizing the Impact of Development Policy Financing in IDA Countries: A Stocktaking of Success Factors and Risks). Engaging clients creates greater prosperity, inclusiveness, and development outcomes. Reports detailing the approach of the International Finance Corporation (IFC) to engaging clients for increased development impact (IFC’s Approach to Engaging Clients for Increased Development Impact), the Bank Group’s commitment to citizen engagement (Engaging Citizens for Better Results), the Bank Group’s record on implementation of the shared prosperity goal for the bottom 40 percent (Growth for the Bottom 40 Percent: the World Bank Group’s Support for Shared Prosperity), how the Bank Group can engage the rural nonfarm economy to alleviate poverty (Growing 192 Appendix K IEG Major and Meso Evaluations the Rural Nonfarm Economy to Alleviate Poverty), and IFC’s experience with inclusive business (IFC’s Experience with Inclusive Business: An Assessment of IFC’s Role, Outcomes, and Potential Scenarios) focus on the client in detailing the progress made on the shared prosperity and inclusiveness theme. The Bank Group’s commitment to citizen engagement—a term adopted by the Bank Group to denote the two-way interaction between citizens and governments or the private sector with the scope of Bank Group interventions—gives citizens a stake in decision-making with the goal of improving development outcomes. Client engagement is essential for IFC to support the private sector, maximize finance for development, and contribute to achieving the Bank Group’s twin goals. The Bank Group’s record on implementation of the twin goals since 2013 uses the official definition of the goal of fostering income growth of the bottom 40 percent, and the evaluation examines the extent to which the Bank Group was already incorporating distributional issues in its various activities from 2005, before the adoption of the shared prosperity goal in 2013. The distributional impacts of rural nonfarm opportunities can be significantly pro-poor. Engaging rural nonfarm activity is critical to alleviating poverty and promoting shared prosperity. Shared prosperity is related to IFC’s definition of inclusive business, referring to companies’ business models and their value chains. Such businesses provide goods, services, and livelihoods on a commercially viable basis, either at scale or scalable, to people living at the base of the economic pyramid, thereby making them part of the value chain. A Thirst for Change The major evaluation A Thirst for Change: An Evaluation of the World Bank Group’s Support for Water Supply and Sanitation with Focus on the Poor assesses the Bank Group’s effectiveness in supporting improved access to adequate, reliable, and sustained water and sanitation services (WSS) in client countries.1 It also examines how well the Bank Group is equipped to support countries in moving toward sustained WSSs for all, especially the poor (see figure K.1). 193 Appendix K IEG Major and Meso Evaluations Figure K.1. A Thirst for Change: Highlights from IEG’s Water Supply and Sanitation Evaluation (2007–16) Source: World Bank 2017a: http://ieg.worldbankgroup.org/sites/default/files/Data/Water%20Supply%20and%20Sanitation%20Infographic_onli ne.pdf. Note: AfDB = African Development Bank; IDB = International Development Bank; MDB = multilateral development bank; SDG = Sustainable Development Goal; WBG = World Bank Group. 194 Appendix K IEG Major and Meso Evaluations Global WSS suffers from a lack of financial viability and accountability of service providers. The Bank Group provided $30.3 billion for WSS to client countries during FY07–16.2 Approximately 71 percent of WSS projects completed during this time had moderately satisfactory or above outcomes, and 42 percent of those had significant or high risk to project development outcome ratings.3 Monitoring and evaluation quality is substantial or high for just 22 percent of projects. IFC and Multilateral Investment Guarantee Agency (MIGA) investments are made predominantly in middle-income countries. Half of IFC investments and advisory services were rated only moderately satisfactory or better because of weak government execution capacity, lack of political will, and a politicized tariff adjustment system. Bank Group programs and projects skew toward augmenting bulk water supply and away from addressing issues of reliability, quality, and affordability of services. Low tariffs and insufficient government transfers for WSS in most lower-middle- income and low-income countries hinder financial viability of the sector. Subsidies and transfers in the sector are insufficient to compensate service providers, in particular those providing superior performance. Furthermore, service providers have little incentive to manage their assets efficiently because of a culture of financial handouts. This culture acts as a deterrent to the needed private sector involvement to provide financing or operate WSS infrastructure when tariffs are not remunerative, unless the government assumes the demand risk. Of the $30 billion support for WSS, IFC and MIGA account for just 5 percent and 2 percent, respectively, which is a challenge for the successful application of the Bank Group’s Maximizing Finance for Development approach to the sector. Bank Group policy dialogue and investments emphasize institution building in project approval and strategy documents. Transparency in rules, accountability of organizations responsible for service delivery to customers, active stakeholder participation, and leadership at the country and sector levels are critical for determining sustainable WSS outcomes. The level and degree of institutional accountability varies across country income groups. The evaluation notes that upper-middle-income countries are successful in leveraging World Bank support for policy frameworks and institution building that hold WSS providers accountable for service delivery performance, resulting in more professional WSS organizations responsive to consumer demand. Although similar policy frameworks exist for low- and lower-middle-income countries, institution building 195 Appendix K IEG Major and Meso Evaluations successes are less visible in these instances, and the Performance Standards (PSs) for which WSS providers are accountable are much lower, with varying levels of responsiveness to consumer demand. The Bank Group has not yet developed a systematic response to cross-sectoral challenges that are approaching crisis proportions in many areas. Competing demand for water resources from agriculture and industry, climate change, and the pollution of water bodies with untreated wastewater and poorly managed solid waste and industrial effluents pose interrelated development challenges. The evaluation notes that, apart from a few exceptions, the relevant Global Practices in the Bank Group are not coordinating their efforts to address these issues or providing the necessary support to client governments to deal with these challenges. Financial viability, institutional accountability, and cross-sectoral challenges inhibit progress toward SDG 6. This is particularly evident in fast-developing low-income and lower-middle-income countries for which there are few sustainable development models to address these issues. Best practice World Bank projects in Brazil and Morocco have helped improve WSS utility performance and accountability, and in Colombia have facilitated private sector participation. Community-based rural WSS has been mainstreamed in several countries with positive results, and decentralized cell phone–based monitoring and evaluation systems in Indonesia have promoted service provider accountability. In addition, extreme water-stress situations in Brazil and Mexico are being helped by World Bank projects, and World Bank projects are targeting the impact of wastewater on water bodies in the Arab Republic of Egypt, Argentina, and India. Replicating these experiences elsewhere is a challenging but necessary task. The report makes five recommendations: First, the Bank Group should increase the diagnostic efforts for enhanced engagement to reduce disparities in WSS within and across Regions, particularly in rural areas and among low- and lower-middle- income countries. Second, the Bank Group should align the results frameworks and key performance indicators of projects with SDG 6 and support client countries’ need in building their evidence base for WSS access and delivery.4 Third, the Bank Group should engage with client governments on reforms to the WSS sector to strengthen the financial viability of service providers and help with opportunities for increased access to commercial finance (cascade approach).5 Fourth, the Bank Group should increase cross-sectoral collaboration (with Global Practices and with country authorities) to address complex WSS-related challenges in lending, 196 Appendix K IEG Major and Meso Evaluations technical assistance, and knowledge support. Fifth, the World Bank should maintain and enhance its role in generating and sharing knowledge through analytical work—the Water and Sanitation Program and Water Partnership Program—and technical assistance and capacity building through investment projects with a clear link to project outcomes in their results framework. Data for Development Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity assesses how effectively the World Bank has supported development data production, sharing, and use, and suggests ways to improve its approach.6,7,8 The evaluation reviews World Bank support in FY04–16 (lending, technical assistance, and trust fund grants) for developing countries’ capacity and data systems, data for the national and global public good, engagements in international partnerships, and technological innovations, particularly relating to big data. The World Bank is an effective leader and partner in development data for global audiences and the global good, although support for data production is greater than for data use, and competing demands for funding place existing partnerships at risk.9 World Bank support for statistical capacity building through World Bank– led partnership programs (for example, the Marrakech Action Plan for Statistics and the Busan Action Plan for Statistics) align well with the global development agenda and country priorities.10 However, the evaluation suggests that engagement with data users is feeble and that no strategies are in place for stimulating demand for data from government, the private sector, civil society, academia, or the media. New partnerships for data innovation are not well-articulated, and the overall picture is one of individually relevant initiatives that are sometimes disjointed. The risk is that the existing, well-functioning partnership architecture will stop receiving adequate funding as donors’ attention shifts to a new generation of less- clearly articulated data partnerships with lofty ambitions, overlapping goals, and insufficient funding.11 Clearer goals for World Bank engagements in global data partnerships to protect the gains made will help maintain a coherent, focused approach. Effective national statistical systems that track progress across a broad spectrum of development objectives are a work in progress in many countries. Country clients need better coordinated, long-term support to strengthen their administrative data systems and support statistical capacity building beyond national statistical 197 Appendix K IEG Major and Meso Evaluations offices.12 Using its comparative advantage, the World Bank can assist and encourage country clients to continue their trajectory of data production, sharing, and use with the following steps:13 (i) strengthen domestic and international long- term funding for data and statistical capacity building;14 (ii) make data more central in policy dialogue, promote evidence-based decision-making, and demonstrate the value of using data; (iii) move toward a data capacity building model that reaches beyond the national statistical office’s boundaries to other parts of the national statistical system; and (iv) scale up support for administrative data systems in collaboration with global practices and other development partners, aligned with country priorities. Support for national statistical systems enhances data production more than it promotes in-country data sharing and use.15 The World Bank promotes an open data agenda using legal reforms and technical updates to make official data and microdata more accessible and has been most effective at supporting data sharing in the countries in which it engages national statistical offices. Nevertheless, it has not used its leverage fully with governments that are reluctant to share data. No framework or approach exists to ensure that interventions in different sectors align to foster a user-centered data culture.16 Staff have mixed views on the merit of pursuing a data use objective.17 Some of the factors underlying data use or nonuse by government decision makers, particularly those related to government capacity, can be reconciled through communication, policy dialogue, and training. The rise in demand for data to monitor the SDGs, increased interest in performance-based budgeting among some governments, and the surge of citizen surveys of service quality also provide opportunities to promote a more inclusive, user-centered data culture. Efforts to use big data are curtailed by a lack of understanding of when and how these data can complement traditional data in answering key development questions and a lack of corporate agreements to ensure the World Bank has access to such data. The World Bank has not systematically analyzed the potential contributions and drawbacks from big data for its mission or how, when, and where it can best support the development of clients’ big data capacity. The advantages of big data—faster, less costly, more reliable, more frequent, and more disaggregated than traditional data—could represent the future for many types of use (for example, geographic targeting). The World Bank should examine its own and other relevant organizations’ experience with the usefulness of big data 198 Appendix K IEG Major and Meso Evaluations complementing traditional data and implement coordinated actions to ensure that sufficient, advanced big data analytics underpin its own decisions and that it provides effective support to country clients for big data use. In conclusion, the report notes that World Bank success with development data is attributable to its technical expertise and its ability to link global needs to national needs, sustain long-term initiatives, and align partnership engagements. The report notes that the World Bank is mostly effective at the country level in supporting data production, promoting open data, encouraging some country clients to share data, and building the capacity of national statistical organizations in countries where it adopts a systemwide approach. It is less effective in adapting to the changed global partnership landscape where the complementarity of new partnerships is less clear. The World Bank is also less effective in fully using its leverage to encourage data sharing by client countries reluctant to do so and in promoting data use in government decision-making, building subnational data capacity, strengthening country clients’ administrative data systems, and staying at the forefront in analyzing the potential and pitfalls of big data for development. The report makes five recommendations: First, the World Bank needs to implement goals and priorities to support global data and global partnerships, country data capacity, and a user-centered data culture. Second, the World Bank needs to mobilize and deliver additional support to countries’ statistical systems using a comprehensive model of statistical capacity building that also factors in needs and opportunities to strengthen administrative data systems. Third, the World Bank needs to step up its engagement with global partners and client governments on long-term funding for development data. Fourth, the World Bank needs to increase the promotion of data sharing and data use. Finally, the World Bank needs to implement coordinated actions so that its operations benefit from big data’s insights and clients receive appropriate support for big data use. Engaging Clients for Increased Development Impact The International Finance Corporation’s Approach to Engaging Clients for Increased Development Impact assesses how strategic and effective IFC has been in its approaches to client engagement. IFC introduced client engagement as a strategic priority in the early 2000s. This was a relevant and timely response to competitive pressures from new players in the private sector development space, making more financing and advisory options available to private investors in developing 199 Appendix K IEG Major and Meso Evaluations countries and making developing country enterprises more confident. The evaluation examines three modalities of IFC’s engagement with clients: (i) client- focused partnerships,18 (ii) programmatic interventions to support upstream market expansion and creation activities, and (iii) country-focused interventions. Overall, the evaluation finds that since the adoption of the strategic approach to client engagement, IFC has become a more client-focused institution. IFC’s engagement with strategic clients has been associated with positive effects on its behavioral outcomes and performance. IFC has gained a deeper understanding of client objectives and needs and has improved its access to key decision makers. Clients appear to appreciate the appointment of client service leaders since FY15 as a step in the right direction because it provides a consistent point of contact and interlocutor in IFC. Clients also noted, however, the limited scope of accountabilities of the client leaders and the importance of the right fit in terms of seniority and relationship skills. IFC’s engagement with strategic clients is associated with better development results compared with those of one-off clients. Strategic clients’ projects tend to perform better in higher-risk country contexts compared with the projects of one- off clients. This performance pattern holds true for business performance, economic sustainability, and private sector development impacts. However, the environmental and social effects performance of strategic clients is the same as that of one-off clients. This performance pattern mainly reflects a selection effect rather than a learning- through-repeat-projects effect. Empirically, a self-reinforcing selection effect is observed, through which client quality and strategic fit enable success that leads to a deepening of the relationship and its transformation over time into a de facto strategic engagement. IFC’s ability to influence client capacities and strategic orientation for greater development focus has remained somewhat limited. IEG found little or no effect on the client’s environmental and social capacity beyond individual projects. Although IFC had a positive effect on strategic clients’ access to new markets and sectors, in most cases, the client was already preparing to enter the new area and typically approached IFC as the lender of last resort and provider of comfort to key stakeholders. 200 Appendix K IEG Major and Meso Evaluations Little evidence was found of IFC’s ability to influence the clients’ poverty focus and base-of-the-pyramid orientation. Rather, this becomes effective mainly through IFC’s support to social enterprises that already have such a focus; its support for clients’ expansion into previously unattended or underserved, often rural, areas where they can be inferred to have reached poorer populations—especially in IDA countries; and its support for clients’ corporate social responsibility programs. Key objectives of IFC’s strategic client engagement remain unachieved. Although the investment volume in strategic priority areas—IDA, fragile and conflict-affected situations, infrastructure, and others—grew over the past decade, the growth was not driven by strategic clients. With strategic clients, efficiency showed modest gains as measured by average processing days for investments, mostly for financial markets projects. Client satisfaction has not seen significant gains. Finally, IFC’s ability to influence client capacities and development orientation appears somewhat limited. The evaluation finds that client characteristics, supported by IFC’s financial and nonfinancial additionality, have been the key factors associated with the superior performance experienced by IFC’s engagements with strategic clients. Thus, IEG observes that strategic clients are on average more sophisticated companies in terms of the quality of their corporate governance and financial strength. Strategic engagements tend to evolve organically from project interactions, with IFC pursuing repeat engagements with relatively stronger clients with a track record of implementing projects competently and successfully. Stronger client capacity may also mitigate the effects of country risks. IFC’s ex post financial and nonfinancial additionality emerge as the key motivators for the client’s engagement with IFC in support of its strategic priorities (higher- risk markets and countries). Such relationships tend to exhibit a life-cycle pattern where at some point clients outgrow the need for IFC’s services. The need for IFC to maintain additionality throughout this life cycle in client relationships has implications for IFC’s future selection and segmentation of clients. In its client engagements, IFC still predominantly seeks investment opportunities at an advanced stage of preparation that can be executed within a short period. In a portfolio of client engagement approaches that varies by types of clients, products, and project preparation stages, the project financial structuring stage is likely to remain the dominant entry point for future client engagements. However, such a 201 Appendix K IEG Major and Meso Evaluations predominantly short-term, market-opportunistic orientation limits IFC’s opportunities to be truly strategic. IFC’s ability to translate pressing country development needs into bankable investment projects needs to be strengthened. Numerous IFC initiatives have attempted to strengthen its upstream and programmatic approaches at the client and country level. IEG reviewed a sample of such initiatives and found mixed outcomes. Successful approaches have not been mainstreamed within IFC. These findings point to the challenges of moving toward the upstream and market-creating activities envisioned under IFC’s new strategy, IFC 3.0. The reviewed country and programmatic initiatives were highly relevant to IFC’s own strategic priorities and addressed important development needs in the host country’s development strategy. The reviewed country-focused interventions provided IFC with a platform for strategic, upstream client engagement focusing on addressing binding constraints to private sector development in the targeted countries and sectors. It is, however, too early to assess their contribution to IFC’s strategic priorities and impact. IEG identified the following factors of success in programmatic and country interventions: (i) high relevance in terms of the potential for private sector solutions to development needs and priorities; (ii) careful design using bankability criteria to target binding constraints to private sector development (for example, policy, regulatory, and access to finance); (iii) joint teams combining transaction and policy expertise in seamless transactions with public and private sector clients for package solutions; and (iv) leveraging IFC’s convening power and credibility with governments. Although upstream and programmatic activities are complex, resource intensive, and high risk, the review also identified internal factors that deserve greater attention. IFC’s incentives and culture do not appear to be conducive to the innovation and risk-taking inherent in finding solutions to move further upstream. IFC’s upstream and market-creating initiatives have often been ad hoc, isolated, and motivated by personal initiative and relationships. Success is not mainstreamed or scaled up within operations. Institutional encouragement to focus on upstream interventions and market creation activities and facilitate access to resources for innovation and promotional activities and cross-cutting teams also 202 Appendix K IEG Major and Meso Evaluations appeared inadequate. As IFC implements the new IFC 3.0 strategy, these issues merit greater attention. The findings indicate that although the client strategic approach has not been fully and consistently implemented, indications are that its key elements have had positive effects on IFC’s behavior and suggest that a fuller implementation would yield enhanced positive results. The limited experience to date with programmatic and country-based interventions—while highlighting the challenges of upstream work with clients—points to their relevance and potential contribution to IFC’s future strategic priorities. Given this, the strengthening and more systematic implementation of the strategic approach to client engagement, fully integrated with market- and country-focused upstream interventions, appears to offer good potential to support IFC’s strategic priorities and to increase its development impact, as envisioned in IFC 3.0. The evaluation makes three recommendations: First, IFC should systematically implement the strategic approach to client engagement with a focus on areas where implementation has lagged. To do so, IFC needs to (i) adopt clear criteria for the selection and segmentation of clients according to IFC strategic priorities, including for upstream project development, cascade, and creating markets; (ii) streamline procedures and processes and strengthen criteria for additionality for strategic clients, including for the justification of incremental additionality in repeat projects addressing the same market failure; and (iii) identify areas in which client approaches can be integrated with sector and country strategies and programmatic approaches. Second, IFC should strengthen the client relationship management function to enhance its capacity to build a pipeline of new clients. To do so, IFC needs to (i) establish clear accountabilities and incentives for strategic client relationship management; and (ii) strengthen the business development function and support systems by, among other things, increasing the transparency and accountability for client interactions and their outcomes and integrating client management databases into IFC’s information architecture. Third, IFC should enhance its capacity for upstream engagement on projects to increase the number and quality of IFC's pipeline of strategic clients. To do so, IFC needs to (i) strengthen the capacity to operationalize findings of country and sector diagnostic work to develop new clients and investment opportunities; and (ii) bolster IFC’s internal incentives and resources to systematize support for upstream approaches across IFC departments. 203 Appendix K IEG Major and Meso Evaluations Mobile Metropolises: Urban Transport Matters Mobile Metropolises: Urban Transport Matters: An IEG Evaluation of the World Bank Group’s Support for Urban Transport assesses the Bank Group’s effectiveness in supporting countries’ efforts to achieve mobility for all, urban transport service delivery that is financially and environmentally sustainable, and institutional development to strengthen the urban transport sector.19 The objective of SDG 11.2 is to provide access to safe, affordable, and sustainable transport systems for all by 2030. The Bank Group has played a strong role in supporting urban transport development through enhancing the capacity and quality of service delivery, promoting private sector provision of services, intervening to help clients mitigate environmental harm from urban transport systems, and helping strengthen institutional capacity to administer urban transport service provision. In addition, the Bank Group uses its knowledge and convening power to promote good practices and South-South learning, despite its relatively low investment in a global context and challenges from rapid urbanization with associated pressures to improve policies,20 systems services, sectoral management, and oversight (see figure K.2). Figure K.2. Factors Affecting Urban Transportation More than half Rapid the people on urbanization is earth live in exerting pressure cities—a share on all urban that will grow to systems and 60 percent by services. 2030. Why Urban Transport Matters Urban transport can Mobility and provide arteries for accessibility are the movement of deteriorating in most people and goods cities because for efficient, increasing sustainable cities. motorization overwhelms the system. 204 Appendix K IEG Major and Meso Evaluations Source: Based on World Bank 2017d: http://ieg.worldbankgroup.org/evaluations/urban-transport. IEG finds that projects that incorporate both supply and demand measures can commit more resources to mobility, achieve greater mobility improvements, and obtain more sustainable mobility outcomes compared with projects focusing solely on demand or supply measures. During FY07–16, 245 urban transport projects (about 82 percent of the portfolio) were approved with 104 projects delivered, increasing the supply of urban transport infrastructure and services and improving service quality. By contrast, relatively few projects supported travel demand measures—123 projects included one or more type of travel demand measure activity.21 The narrow support and implementation of demand management measures accounts for the shortfall in improving citywide mobility—no project managed to improve this, although most projects did improve local mobility. Targeted measures that connect disadvantaged groups to urban transport services and systems have good outcomes but are not yet broadly applied. Only 36 percent of projects target the poor, 7 percent target women, and 10 percent target disabled and elderly persons. Projects that support fare integration encourage public transit use, particularly among low-income users; however, these projects, in addition to those targeting affordability (through studies and subsidy policy support), account for less than one-fifth of evaluated projects. Financial sustainability is hindered by over-optimism at the design stage. Of 16 closed mass transit projects, none present a financial analysis at closing. None of the projects with private sector involvement analyze the financial sustainability of systems, and none discuss public sector elements of urban transport systems.22 Cost overruns are more likely in road sector projects (occurring in 70 percent of projects) and in Sub-Saharan Africa (35 percent of projects). Social benefits are not quantified, especially those benefits arising from improved accessibility to health and education services and employment opportunities. There is some evidence to suggest that direct private sector involvement, which is less likely in the urban road sector, is more successful in achieving financial sustainability of operations than projects that rely only on the public sector. The operational sustainability success rate of mass transit projects with private sector participation is 44 percent compared with just 14 percent without private sector participation. However, direct private sector involvement is less likely in the urban road sector.23 Moreover, collaboration between the World Bank and IFC or between the World Bank and MIGA on engaging the private sector in urban transport is 205 Appendix K IEG Major and Meso Evaluations infrequent and, in some cases, disconnected from relevant upstream (policy and institutions) work of the World Bank, precluding possible investment opportunities involving the private sector and limiting the cascade approach. The Bank Group is successful in achieving its downstream (operations) environmental mitigation efforts. Success is often confined to the specific project area, whereas citywide environmental sustainability occurs from projects with both upstream and downstream support measures. Seventy percent of cities receiving comprehensive Bank Group support realize environmental benefits—for example, reductions in air pollution and greenhouse gas emissions. Structural change outcomes in the urban transport sector are usually weak in contexts of one-time support and when both government commitment and implementation capacity are absent. More than 90 percent of capacity building activities are delivered successfully, although more than half (20 closed and evaluated projects) failed to show evidence of their effectiveness, which may stem from weak monitoring and evaluation frameworks.24 Only five projects address the fiscal capacity of urban transport–related institutions in their design, and two of these are still active. Projects that address the financing capacity of local government to provide local services are successful in enhancing the fiscal capacity of local governments. Four of the eight closed and evaluated operations supporting structural change in the urban transport sector were successful—an outcome that appears correlated with cities receiving continuous support from the World Bank. Important gaps remain. There is a broad imbalance in support with regard to low- income countries, particularly in the Africa Region. IEG found insufficient and inconsistent attention to mobility of the disadvantaged. Furthermore, opportunities for more comprehensive engagements that may yield stronger benefits in mobility and environmental sustainability were missed. Finally, IEG found opportunities to strengthen the work quality in ex ante analysis, monitoring and evaluation, and communications and collaboration between different parts of the Bank Group. Toward a Clean World for All Toward a Clean World for All: An IEG Evaluation of the World Bank Group’s Support to Pollution Management represents IEG’s first comprehensive assessment of pollution management efforts to examine the extent to which the Bank Group has been relevant, effective, and efficient in addressing pollution concerns in client 206 Appendix K IEG Major and Meso Evaluations countries.25 Bank Group engagement in pollution management is multisectoral and cross-cutting, comprising 534 projects (317 World Bank investment and policy operations, 77 IFC investments, 123 IFC Advisory Services projects, and 17 MIGA projects) with a total commitment of $43 billion targeting pollution management during FY04–17. The report highlights the need to increase and recalibrate the approach to fighting pollution, noting that over the past two decades, pollution and its effects on the poor have worsened in many parts of the world (see figure K.3). Strengthened engagement in climate change has positive externalities for pollution, most readily seen in addressing ambient air pollution. In 804 climate change interventions undertaken during the evaluation period, 45 percent of components were likely to have cobenefits for air pollution through reducing both greenhouse gases and pollution.26 IEG notes that a careful analysis of the portfolio of pollution-targeted Bank Group interventions indicates that the Bank Group provides sustained financing for wastewater infrastructure but may miss opportunities for fighting outdoor and indoor air pollution.27 Integrating pollution into other sectoral interventions has an important role to play—the climate change mitigation portfolio might also provide an opportunity to address outdoor air pollution directly. 207 Appendix K IEG Major and Meso Evaluations Figure K.3. Take-Away Points from Toward a Clean World for All: An IEG Evaluation of the World Bank Group’s Support to Pollution Management Nine million people die annually from pollution-related causes. Within developing countries, the poor are hit particularly hard by pollution based on several factors, including living in areas with the worst environmental conditions. Pollution can cost between 4 and 5 percent of a country's gross domestic product. Development institutions should focus on the circumstances of the poor. Strengthening pollution monitoring is a critical first step. Significant cobenefits arise from combining pollution and climate change interventions. Source: Based on World Bank 2017e: http://ieg.worldbankgroup.org/evaluations/pollution. Because of its field presence and access to policy makers, the World Bank is in a strong position to integrate pollution management into the development agenda of its client countries through its knowledge work. Bank Group Country Partnership Strategies and Country Partnership Frameworks (CPFs) rarely reflect the most serious pollution concerns—only 28 percent of Country Partnership Strategies for countries with pressing pollution concerns reference these consistently, and 56 percent of strategy documents do not mention them at all. The Systematic Country Diagnostic tool captures pollution concerns more accurately, identifying them in 57 percent of diagnostics. The lack of quality data on pollution, its cost, and the cost-effectiveness of pollution abatement interventions increases the difficulty of building a case for governments to undertake pollution abatement interventions or to borrow from the World Bank for that purpose. This may explain why the Bank Group’s overall portfolio has not grown at a pace commensurate with the increase in pollution. Improving the enabling environment recognizes the complex and multidimensional nature of pollution management, with responsibilities often spread across multiple agencies and jurisdictions. Capacity building is the most prevalent support mechanism in investment lending, with interventions being successful in 53 percent of cases.28 Many pollution interventions attempt to prevent 208 Appendix K IEG Major and Meso Evaluations or treat pollution, whereas relatively few seek to build pollution monitoring systems.29 Interventions targeting pollution monitoring, though critical, are relatively rare and also tend to fail more often—70 percent of such interventions failed to achieve their targets,30 compared with 50 percent of interventions meant to prevent pollution and fewer than 30 percent meant to treat pollution. Enforcing pollution regulations is common across country cases but is constrained by conflict of interest from the parties involved, lack of political will, lack of incentives for local authorities to enforce regulations, and politicization of decision-making. Significant progress in managing pollution in client countries is being made by the Bank Group, and drivers of pollution management intervention are sector specific. For example, urban transport interventions are relatively successful in achieving emission reduction objectives and setting up air quality monitoring systems but are often project specific. Achieving sustained environmental benefits requires projects that include both upstream and downstream measures.31 Interventions addressing pollution through solid waste management and water treatment are less successful.32 IFC Advisory Services help client companies reduce or recycle waste, increase energy or resource efficiency, or introduce renewable energy to good effect. Advisory Services also support the structuring of public-private partnerships (PPPs) in wastewater or waste management, but to lesser effect. Efforts are inhibited by a lack of political support for private sector participation, and they frequently fail to reach commercial closure. A similar scenario characterizes waste and wastewater utilities. The Bank Group uses performance standards and safeguard policies to control the potential negative pollution effects from its projects.33 Most IFC investments in pollution-intense industries are successful in meeting pollution prevention requirements,34 though there is room for improvement because one-third do not fully meet the relevant requirements for air emissions and wastewater management guidelines. Success in meeting the requirements depends on the client’s or project enterprise’s commitment and technical and financial capacity in managing the pollution associated with its operations. Additionally, engaging with the client or project enterprise throughout the entire project improves the commitment and capacity for reducing pollution. Sufficient information on pollution prevention implementation is not available for a significant number of MIGA guarantee projects.35 For those that have this information available, 62–75 percent meet pollution-prevention requirements. It is not possible to assess the extent to which 209 Appendix K IEG Major and Meso Evaluations the World Bank contributes to better pollution management through safeguards because data on safeguard outcomes are not reported in World Bank documentation.36 The report makes five recommendations: First, the World Bank should strengthen its efforts—through technical assistance and capacity and institutional building—to develop client country pollution measurement and monitoring systems, especially in countries where such capacity is low. Second, the World Bank should strengthen its country analytical work on pollution to prioritize pollution concerns based on countrywide assessments. Third, the World Bank needs to intensify the scale-up and recalibration of its efforts to address the most important pollution priorities, in particular the circumstances of the poor. Fourth, the Bank Group should better leverage its climate change portfolio to combat local and regional air pollution and other applicable forms of pollution. Fifth, IFC should strengthen support on pollution by offering advisory services for clients that lack the required knowledge. Growing the Rural Nonfarm Economy to Alleviate Poverty Growing the Rural Nonfarm Economy to Alleviate Poverty: An Evaluation of the Contribution of the World Bank Group assessed the Bank Group’s contribution to the creation of sustainable income generation for the poor within the rural nonfarm economy (RNFE) and the extent to which this has led to reduced poverty.37,38 Rural nonfarm activity refers to value chain activities such as agro-processing, transport, distribution, marketing, and retail, in addition to tourism, manufacturing, construction and mining, and self-employment activities (see figure K.4). 210 Appendix K IEG Major and Meso Evaluations Figure K.4. Growing the Rural Nonfarm Economy Source: World Bank Group 2017f: http://ieg.worldbankgroup.org/sites/default/files/Data/RNFEbrochureweb.pdf. The Bank Group invested $46.5 billion between FY04 and FY14 (1,141 implemented projects) in support of RNFE activities. Although such activities with a poverty aim have reached the rural poor, reducing vulnerability and increasing service access, they have not generated sufficient, sustained income to lift a large number of the poor out of poverty. Other activities with a growth aim, specifically value chain activities, have increased revenues but neither target nor provide evidence that the poor benefit. IFC activities in food processing have strong links to rural areas with positive employment effects, whereas IFC’s agro-processing portfolio, with some exceptions, lacks pro-poor and gender-sensitive chain analysis and poverty targeting and also requires attention to market power risks where relevant. 211 Appendix K IEG Major and Meso Evaluations Although rural poverty has declined over the past two decades, 80 percent of the worlds’ poor live in rural areas, with wide regional distributions in Sub-Saharan Africa and South Asia, in particular. Income gains for the very poor and landless are generally associated with earnings from rural nonfarm employment opportunities. Households that rely on farm labor tend to be among the poorest. The rural nonfarm sector contributes to economic growth, household income diversification, rural employment, poverty reduction, and a more spatially balanced population distribution. RNFE activities are growing and account for 35– 50 percent of rural income in developing countries. To benefit from opportunities in the RNFE, the poor will need to overcome a host of institutional, connectivity, human, social, and capital constraints. IEG found marked differences in the way that projects supported rural nonfarm income generation in countries, based on their stage of agricultural transformation (agrarian, transition, or urbanized) identified in the World Development Report 2008: Agriculture for Development. In agrarian areas, the focus was on connectivity, resilience, welfare smoothing through the provision of public and private assets, and services. In transitioning and urbanized economies, factor markets received greater focus. The Bank Group’s current approach to the agriculture sector and RNFE stems from the 2008 World Development Report and emphasizes the particular growth pathways of client countries with attention to their dependence on agriculture and the nature of rural poverty. The approach highlights the need for improved enabling conditions for RNFE, including the investment climate, skills and basic education in rural areas, and a need to think spatially. The Bank Group has a bifurcated portfolio in the RNFE space with a significant gap between its poverty-oriented and growth-oriented interventions. Projects focused on reaching the rural poor implemented in agrarian and early transition settings have generally been effective in reducing vulnerability and increasing access to services but less so in supporting sustained income or employment and reducing poverty.39 Exceptions to this arise when sustained income is achieved through a rigorous commitment to targeted enterprise training and when binding constraints (often credit) are identified and addressed and strong social accountability tools are provided. This approach is more effective in less transient areas where citizens share a similar set of values or norms or can hold one another to account through a set of project tools. Projects with a growth aim, including agricultural value chain projects and productive partnerships across agrarian, transition, and urbanized 212 Appendix K IEG Major and Meso Evaluations economies, frequently achieve increased sales, revenues, and incomes, although reach to the poor is not targeted or tracked and is often not evident. Bank Group efforts to bridge the gap have not generally been successful because instruments designed to reduce vulnerabilities (most notably social protection projects with a productive aim) have been asked to achieve employment and income goals that are not their comparative advantage. Rather, opportunities to blend or pilot new hybrid approaches that would more effectively target and link the poor to more productive opportunities in the RNFE should be pursued. For this, skills will need to be blended across different approaches and between different parts of the Bank Group. Although value chain approaches do achieve increases in production, revenues, and sales, the lack of pro-poor and gender- sensitive value chain analysis at the design stage, in addition to associated metrics to monitor these projects during implementation and evaluation, obfuscates their reach to the poor. Addressing binding constraints is key to linking the rural poor to the RNFE. For this, a holistic approach that integrates an understanding of rural demographics, skills, mobility, and remittance flows is vital, yet few countries pursue this approach.40 Diagnostics to assist an integrated country approach are also critical, and although the Bank Group is a leader in researching and documenting the RNFE, a significant gap exists on the diagnostic and analytical side.41 Enhanced rural connectivity is being achieved through synergies between transport and agriculture, though mostly for transitioning economies.42 Progress in other important areas, such as skills and financial services, has moved more slowly, particularly for the poorest rural segments. World Bank support for financial services raises questions about sustainability, crowding out, and politicization, and IFC investments often reach countries that have high exclusion rates; however, only a fraction of projects cater to the lower end of the retail segment. The evaluation makes six recommendations: First, at the corporate level, the World Bank and IFC should clarify their approaches to the RNFE to inform engagement across institutions, Global Practices, and partnerships in promoting nonfarm economic development to reduce rural poverty. Second, projects should be selected and designed in a way that is tailored to the specific stage and nature of a country or area’s structural transformation and its rural population by pooling knowledge and sharing ideas across Global Practices and Regions. Third, the Bank Group should work to close knowledge gaps about what works to reduce rural poverty 213 Appendix K IEG Major and Meso Evaluations within the nonfarm economy. Fourth, IFC should include poverty and gender impacts in its value chain analysis and monitoring and evaluation and articulate the risks of market power and associated mitigation strategies in its project documents. Fifth, where the rural economy is a key part of the solution to end poverty, the Bank Group should, in partnership with donors and client countries, collect information on formal and informal rural enterprises and their constraints and performance to better inform the Systematic Country Diagnostic. Finally, the Bank Group should strengthen and deepen gender analysis during project preparation of rural nonfarm projects. Growth for the Bottom 40 Percent The evaluation Growth for the Bottom 40 Percent: The World Bank Group’s Support for Shared Prosperity, covering FY05–16, assesses the Bank Group’s record on implementation of the shared prosperity goal since 2013 (using the official definition of the goal of fostering income growth of the bottom 40 percent [B40]) and prior efforts to include distributional issues (FY05–13). The overarching finding is that the Bank Group has made a solid effort to incorporate the shared prosperity goal into its strategies, projects, and knowledge work, but more needs to be done to translate these efforts into better development results (see figure K.5). Figure K.5. Take-Away Points from Growth for the Bottom 40 Percent: The World Bank Group’s Support for Shared Prosperity • Addressing high levels of inequality at the country level is essential for making progress toward the World Bank Group’s vision of a world free of poverty. • Results frameworks must adequately monitor impacts on the bottom 40 percent. • Core diagnostics must contain rigorous analysis of the distributional effects of interventions. Source: Based on World Bank 2017g: http://ieg.worldbankgroup.org/evaluations/shared-prosperity. The Bank Group’s knowledge work, strategy, and lending projects with associated theories of change provide a means to communicate and operationalize the shared prosperity goal, although limitations exist. The asset-based framework underlying most knowledge work suggests five pathways for shared prosperity.43 Most of the 42 Systematic Country Diagnostics performed during FY13–16 cover the goal, 214 Appendix K IEG Major and Meso Evaluations although data limitations in 60 percent of these diagnostics prevent certain groups (internally displaced persons and ethnic groups) from being fully represented. However, only 20 percent of the 64 non–Systematic Country Diagnostics (public expenditure reviews, poverty and gender assessments, and Country Economic Memorandums) contain an analysis of the B40, with the Country Economic Memorandums containing more of an analysis. The importance of core diagnostics for shaping the agenda on distributional issues is confirmed in the report’s 12 case studies.44 Knowledge work is less prevalent in IFC projects,45 although the FY18 Anticipated Impact Measurement and Monitoring tool aims to assess a project’s contribution to the twin goals. CPF objectives generally do not fully articulate the theory of change pertaining to the B40.46 Only 32 percent of World Bank projects (but no IFC projects)47 have an explicit theory of change (better articulated in investment projects than in development policy financing) that links project interventions to benefits for the B40.48 Institutional requirements in terms of staff awareness, skills and adaptability,49 availability of data and analytical tools to monitor progress and efforts, and continued emphasis on convening power are critical to promoting the shared prosperity agenda globally and in member countries. Only 63 percent of staff are fully familiar with the official World Bank definition of shared prosperity, ranging from 71 percent of International Bank for Reconstruction and Development staff to 42 percent of IFC staff. Staff advocate for the goal to focus on reducing inequality of opportunities rather than to focus solely on reducing inequality in outcomes (essentially, incomes).50 Improvement in assets (human, physical, financial, social, and natural) is the most common channel through which staff contribute to the shared prosperity goal. Other important channels aim at improving resilience to shocks, improving infrastructure, and increasing the demand for labor. The use of indicators to measure shared prosperity varies across agencies and country programs with little standard coverage and disaggregation, though with increasing use of inequality and access measures. The Bank Group demonstrates considerable convening power at the global and country levels in promoting, financing, and advocating for shared prosperity–focused interventions. Bank Group support for equity before 2013 is evident in Global Practice knowledge work, the distributional content of country strategies, development policy financing, and the geospatial analysis of World Bank projects.51 One of six Advisory Services and Analytics products produced in FY05–13 includes content related to 215 Appendix K IEG Major and Meso Evaluations distribution. Such content is most evident in Poverty and Equity Global Practice projects (47 percent of its Advisory Services and Analytics), followed by Education; Health, Nutrition, and Population; and Social Protection and Labor (25 percent, on average); Macroeconomics and Fiscal Management (21 percent); and Agriculture; Social, Urban, Rural, and Resilience; and Water (14 percent on average). Analysis of the 12 country case studies shows that more than half of the Country Assistance Strategy/CPF pillars closed and evaluated in FY05–16 have significant distributional content. Distributional content among Bank Group projects during FY05–13 trends upward and is highest among upper-middle-income country clients (33 percent) and International Bank for Reconstruction and Development clients (18 percent versus 12 percent in IDA countries) and in Latin America and the Caribbean (33 percent), followed by the Middle East and North Africa (20 percent). Social sector–focused Global Practices account for almost three-quarters of projects with high distributional content along with Agriculture and Macroeconomics and Fiscal Management.52 Early Program-for-Results projects have the highest distributional focus (38 percent, although there were few Projects-for-Results projects in the sample), followed by development policy financing projects (18 percent) and investment project financing projects (14 percent). The evaluation makes five recommendations: First, the Bank Group should provide funding to address knowledge and data gaps on the characteristics of the B40 in the CPF and encourage clients to ensure greater availability, quality, and comparability of distributional data. Second, the evaluation advocates support by management to ensure the financing of strategies and projects targeting the shared prosperity goal with clear descriptions of the results chains linking the interventions to outcomes for the B40. Third, strategies and projects should have clear results frameworks with measurable indicators. Fourth, the Bank Group should monitor the extent to which the geographical distribution of beneficiaries of its projects and that of populations in the B40 are congruent. Finally, Bank Group management should ensure that operational staff have a clear understanding of and the necessary skills to incorporate the shared prosperity goal in Bank Group–financed strategies and projects. Bank Group Engagement in Upper-Middle-Income Countries World Bank Group Engagement in Upper-Middle-Income Countries: Evidence from IEG Evaluations synthesizes existing evaluative evidence on the outcomes and lessons 216 Appendix K IEG Major and Meso Evaluations learned from the World Bank Group’s partnership with upper-middle-income countries,53 drawing mainly from IEG thematic, corporate, country, and select project evaluations produced in 2007–16.54 The report does not highlight recommendations per se but expands on the findings, which can be summarized as follows: (i) Bank Group engagement with and support to upper-middle-income countries is highly relevant in helping these countries address their specific development challenges and has a potential demonstration effect for Bank Group clients in the lower-income category; (ii) despite this engagement effect, important challenges remain, especially for structural issues that underlie upper-middle-income countries’ vulnerability to various shocks and hinder progress in several important development areas; (iii) Bank Group financing proves important for sending positive signals to markets during crisis, providing credible incentives to private investors in relatively underdeveloped markets, helping to launch innovative and catalytic public and private sector initiatives, and fine-tuning existing national programs to enhance their effectiveness; (iv) new mechanisms are required to facilitate knowledge transfer, particularly between upper-middle-income countries and lower-income countries (South-South knowledge transfer); and (v) portfolio performance during the review period (FY07–16) in upper-middle-income countries exceeds that of other income groups, although effectiveness across Global Practices varies, even within the same countries. (See box K.1.) Box K.1. World Bank Engagement in Upper-Middle-Income Countries Second-generation challenges are much more evident and vividly reflected in upper- middle-income countries, the upper-tier subset of middle-income countries as measured by income. There are currently 56 countries with gross national income per capita ranging from $4,036 to $12,475. These countries are extremely diverse and have differing development needs. The World Bank Group can have a valuable role in responding to upper-middle-income countries’ demand to address second-generation challenges through support for inclusive and sustainable economic growth, effective investment in people, and fostering resilience to global shocks and threats. Source: World Bank 2017h, viii: https://openknowledge.worldbank.org/bitstream/handle/10986/28294/119055-WP- PUBLIC.pdf?sequence=1&isAllowed=y Addressing challenges for inclusive and sustainable economic growth focuses on economic management and institutions and competitiveness and productivity. The Bank Group helped upper-middle-income countries build resilience and cope with the global economic crisis through fostering countercyclical policies, primarily 217 Appendix K IEG Major and Meso Evaluations through development policy loans and, to a lesser extent, through strengthening public sector institutions.55 IEG evaluations show that some development policy loans paid insufficient attention to the available space for fiscal stimulus, the reversibility of stimulus measures, and forward-looking measures to attain fiscal sustainability. IEG finds that the response of IFC was relevant and creative, as was MIGA’s support (through its guarantees) of several key financial institutions in Eastern Europe.56 At the same time, IFC support during the crisis was often procyclical, reducing its investments in larger upper-middle-income countries.57 Postcrisis, credit growth took longer to recover in countries where the crisis was transmitted through the capital account. The 2016 evaluation The World Bank Group's Support to Capital Market Development notes that the Bank Group had a pioneering role in facilitating capital market upgrades in several upper-middle- income countries by reforming market regulations, promoting the modernization of payment systems, and providing innovative support to the development of local currency government bond markets. Most upper-middle-income countries with active public sector reform programs saw performance improvements. This was particularly evident in the public financial management area. Public financial management and tax administration proved to be key points of entry to strengthen public sector institutions in upper- middle-income countries. An example is helping countries meet the required standards for European Union membership (for example, Bulgaria) or coping with the aftermath of a financial crisis (for example, Argentina, Colombia, and the Russian Federation). Strengthened public financial management and revenue administration have the potential to improve fiscal outcomes for any given fiscal measure in place. The report finds gaps in average competitiveness indicators between high-income and middle-income countries, especially for technological readiness and innovation. IEG findings suggest that support to improve competitiveness and innovation is largely relevant, but design and implementation of interventions often remain at the sector level and fail to address systemic weaknesses of innovation ecosystems at the country level. Lending to support innovation is heavily concentrated in education and agriculture with less attention paid to highly relevant areas such as manufacturing and management capacity. Measuring the effectiveness of the comprehensive support for investment climate reforms is challenging. Bank Group support to alleviate binding constraints, particularly IFC 218 Appendix K IEG Major and Meso Evaluations support, is compromised by political economy constraints. Hence, the emphasis is on streamlining administrative procedures and not addressing core underlying policy issues.58 IEG notes that IFC’s financial support for PPPs is concentrated in countries with an established enabling environment. Thus, the Bank Group–supported PPPs in the upper-middle-income countries are largely successful in achieving their development outcomes. Data are scarce on the effects of PPPs on the poor. The report echoes the findings in the previous 2014 evaluation of PPP activity—that IFC’s financial support for actual PPP projects too often reaches already-developed PPP countries and endorses the recommendation that IFC should identify avenues to invest increasingly in PPPs in countries and markets that do not yet have a well- developed enabling environment. Bank Group assistance for traditional infrastructures suggests significant sectorwide impact, even with modest financing. Evaluations of the Bank Group portfolio in upper-middle-income countries contain many examples of triggering critical policy reforms and achieving significant sectorwide impact.59 IFC and MIGA tend to support infrastructure projects in upper-middle-income countries, whereas World Bank support was spread across all country income groups. The evaluative evidence suggests that Bank Group engagement in investing in people and quality of life (education, health, social protection, urban sectors, and employment creation programs) added value in upper-middle-income countries.60 Successful and transformative programs include conditional cash transfers, building social safety net systems and institutions, early childhood education, and health financing.61 IFC involvement—project finance, promoting private sector financing, and setting performance standards—is crucial for PPPs in the health sector. The World Bank supports a wide range of interventions to improve the quality of life in urban areas, including support for core services, housing for lower- income groups, urban amenities, and crime prevention.62 The evaluative evidence for results in employment and job creation programs is limited; only a few Bank Group interventions specifically addressed job creation, and evaluations of these suggest limited effectiveness. Many upper-middle-income countries experience rapid growth, which creates pressures on the physical environment. The Bank Group helps alleviate these pressures by mobilizing necessary funds and helping to build institutional capacity. 219 Appendix K IEG Major and Meso Evaluations The evaluative evidence suggests that the Bank Group helps countries address their commitments under international agreements. The evaluative evidence also suggests that achievement of outcomes for building institutional capacity is lagging, highlighting systemic issues in the weak design and monitoring of results frameworks in the environment sector. Conflict and violence are no longer primarily a low-income-country problem. Fragility and violence affect development outcomes for almost 2 billion people, of which 37 percent live in upper-middle-income countries. Many of the conflicts are subnational in nature and driven by grievances based on ethnic or religious identity. The Bank Group has a significant role to play in these situations through its diplomatic skills, creativity, and appreciation of the local political economy in finding entry points that do not undermine political sensitivities. In some situations, the probability of a short-term crisis becoming a serious long-term development challenge is high,63 thus the emphasis of Bank Group support remains in helping countries address essential, longer-term developmental challenges (chronic unemployment, rapid urbanization). Furthermore, the Bank Group needs to develop financial mechanisms or fast-response facilities to use in situations of crisis.64 In addition, experience with fragility, conflict, and violence in upper- middle-income countries points to the importance of maintaining a credible and robust macroeconomic framework to withstand impacts from unforeseen crises. One of the most robust conclusions emerging from several evaluations in the knowledge agenda is the importance of Bank Group Advisory Services and Analytics in shaping the quality of overall assistance, particularly in quality at entry of development policy financing. Several IEG evaluations, however, find weak monitoring of knowledge service results in upper-middle-income countries for both individual activities and country programs. The evaluations emphasize the importance of reimbursable advisory services to ensure the sustainability of the World Bank’s business model in upper-middle-income countries and generate new knowledge that can be transferred to low-income countries, but also note that the tension between reimbursable advisory services and the World Bank’s mandate as a global development agency often manifests itself in poor coverage of sensitive but important areas—poverty diagnostics in upper-middle-income countries or various aspects of governance. Several evaluations also refer to the need to strengthen knowledge transfer—the full potential of South-South exchanges is hampered by excessive geographic, thematic, and organizational fragmentation. IEG also notes 220 Appendix K IEG Major and Meso Evaluations that the Bank Group could do more to ease the confidentiality of many knowledge activities conducted through reimbursable advisory services with a view to improving dissemination of World Bank knowledge services. Bank Group Support to Health Services World Bank Group Support to Health Services: Achievements and Challenges assesses the roles and contributions of the Bank Group in supporting health services in client countries and provides lessons and recommendations for achieving greater development effectiveness in future support to health services.65 It is the first comprehensive health sector evaluation carried out by IEG since 2009 and complements the 2014 health financing evaluation that examined Bank Group support to revenue collection for health, pooling of health funds and risks, and health financing reforms in 68 countries during FY03–12.66 The evaluation covers FY05–16. Sustainable Development Goal 3 Ensure healthy lives and promote well-being for all at all ages The Bank Group contributes to universal health coverage through its support to health services (project financing and Advisory Services and Analytics).67 Support since 2007 has focused on results-based approaches while sustaining emphasis on the needs of the poor, health systems performance, and sustainable health financing.68 The evaluation finds that the performance of World Bank–financed health projects improved markedly over the FY05–16 evaluation period and especially since 2010, when it performed better than the entire World Bank portfolio (74 percent of projects achieved a rating of moderately satisfactory or above).69,70 About 75 percent of IFC Investment Services projects are rated moderately satisfactory or better, and 64 percent of IFC Advisory Services health projects are rated satisfactory or better.71 The evaluation finds that Bank Group support to health services shows good alignment with the health needs and priorities of client countries. The focus of support in low-income countries stresses prevention, primary care, and maternal and child health.72 IFC investment support mostly covers lower- and upper-middle- 221 Appendix K IEG Major and Meso Evaluations income countries and targets hospitals, clinics, and pharmaceutical companies.73 IFC advisory support concentrates on lower-middle-income countries, with advice on PPPs heavily represented in IFC advisory support (69 percent of projects). Improvements in access, quality, and health systems are the most frequent drivers identified in Bank Group–financed projects, while objectives related to health outcomes and equity are pursued less frequently.74 Improvement in access during FY05–16 (an objective in 54 percent of World Bank–financed projects) is achieved in 70 percent of the 259 evaluated World Bank projects.75 Projects with conditional cash transfers and performance-based financing interventions perform better— access objectives are rated positively in 81 percent and 87 percent of projects, respectively. Quality improvement is an objective in 27 percent of World Bank– financed projects and 27 percent of IFC projects.76 World Bank–financed projects show a limited capacity to monitor and evaluate all relevant aspects of quality— structures, processes, and outcomes. Strengthening health systems is included as an objective in about 37 percent of World Bank–financed projects. Objectives for health outcomes are present in 29 percent of World Bank projects (but with a reduced frequency over time) and in 1 percent of IFC projects. About half of the World Bank–financed projects reach the desired health improvement objectives, while projects supporting conditional cash transfers and performance-based financing perform better.77 Few World Bank–supported projects have objectives that explicitly aim at improving equity, although there seems to be an implicit equity focus in many projects. The distributional impact of World Bank and IFC projects is not known. The Bank Group has had limited success in integrating private provision with public financing.78 IFC investee companies face challenges in improving access for the underserved (for example, limited availability of public resources and capabilities, underdeveloped private markets for health services, and inadequate regulation and regulatory enforcement) and in blending private provision with public financing. Moreover, complementarities and synergies between the public and private sectors are difficult in low-capacity and low-resource settings. A key lesson from World Bank support in pandemic situations is that capable health systems are a necessary ingredient in mounting a successful response to deadly virus outbreaks.79 To respond effectively to such outbreaks requires adequately staffed health services, a supply of essential personal protective equipment, capacities for laboratory diagnosis, clinical management, and 222 Appendix K IEG Major and Meso Evaluations surveillance for quick diagnosis and rapid contact tracing. Commitments under IDA18 aim to support 25 countries in developing pandemic preparedness plans. The Bank Group plays multiple and distinct roles in global partnership programs. The evaluation noted that World Bank financing and coordination roles are more important in countries with lower institutional capacity and more complex development partner networks.80 In countries with high government capacity, the World Bank has a narrower focus (monitoring and evaluation and targeting and reorganization of service delivery networks, which is instrumental in the development of PPPs). It is recognized as a leader in providing technical assistance and knowledge in select areas relevant to health services. The evaluation notes that there is room to achieve greater strategic alignment between the Bank Group’s involvement in global partnerships and its institutional focus and comparative advantage. The evaluation makes four recommendations: First, the Bank Group should improve the measurement of the quality of health services and the distributional effects of health services projects. To this end, monitoring and evaluation should include appropriate indicators of the relevant dimensions of health service quality and provide measures of the improvements to beneficiaries relative to nonbeneficiaries. Second, the World Bank and IFC need to strengthen their synergy to support public-private interactions that would contribute to SDG 3 and universal health coverage in client countries.81 Third, the Bank Group needs to help client countries develop sustainable capacity to address pandemics through the systematic integration of health service–related Bank Group–financed projects and advisory services with awareness and preparedness plans and governance frameworks for pandemic control within the client country’s own health system. Finally, the Bank Group needs to enhance its strategic alignment and selectivity in ongoing and future global partnership programs by applying clear selectivity criteria that reflect Bank Group comparative advantage and the broader global development agenda. Engaging Citizens for Better Development Results Engaging Citizens for Better Development Results: An Independent Evaluation assesses how effectively the Bank Group mainstreamed citizen engagement at the project, country, and corporate levels, and demonstrates how this process contributed to the achievement of development outcomes. Achieving development goals requires 223 Appendix K IEG Major and Meso Evaluations not only the active involvement of citizens and local governments but full engagement with the public, which means listening to those affected by policies and programs. The Bank Group’s commitment to citizen engagement—a term adopted by the organization to denote the two-way interaction between citizens and governments or the private sector with the scope of Bank Group interventions—gives citizens a stake in decision-making with the goal of improving development outcomes. Four questions guided the evaluation:82 (i) What is the magnitude and nature of citizen engagement mainstreaming within Bank Group operations, and how have these changed over time? (ii) What is the quality of design of citizen engagement activities? How much attention is paid to social inclusion and closing the feedback loop with citizens? (iii) What are the extent and quality of monitoring and evaluation of citizen engagement activities, and what is the evidence of results? (iv) To what extent is the Bank Group corporate environment enabling citizen engagement mainstreaming? (See box K.2.) Box K.2. Definition of Citizen Engagement Citizen engagement denotes multiple interactions between citizens and governments or the private sector within the scope of the World Bank Group’s interventions. It is a two - way relationship that implies the existence of a tangible response to citizens’ feedback. As described in official Bank Group statements, the objective of mainstreaming citizen engagement in operations is to give citizens a stake in decision-making to improve development outcomes. Source: World Bank 2018a, ix: https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/Engaging_Citizens_for_Better_Development _Results_FullReport.pdf The evolution of citizen engagement activities in the Bank Group culminated in a corporate commitment in 2013 and the adoption in 2014 of a strategy outline in the Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations. In 2013, the Bank Group committed to integrating, by FY18, beneficiary feedback into 100 percent of investment projects where beneficiaries can be clearly identified. The 2014 strategic framework built on lessons from Bank Group– financed operations across Regions and sectors and underscored the importance of country context, government ownership, and clear objectives for citizen engagement. More recently, the new country engagement model introduced in 2014, the World Development Report 2017: Governance and the Law, and the recently reformed Environmental and Social Framework (which followed the 2012 224 Appendix K IEG Major and Meso Evaluations Performance Standards on Environmental and Social Sustainability update that expanded the magnitude of stakeholder engagement in IFC operations) all confirmed the Bank Group’s commitment to a more holistic approach to citizen engagement. (See figure K.6.) Figure K.6. Four Quality Principles for Citizen Engagement MONITORING “Thick” approach Feedback loop Quality Capacity Inclusion Source: World Bank Group 2018a: https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/Engaging_Citizens_for_Better_Development _Results_FullReport.pdf The corporate commitment and buy-in of management and staff helped increase the percentage of projects with citizen engagement mechanisms,83 especially those that were not safeguard related. The percentage of projects with non-safeguards- related citizen engagement mechanisms increased from 67 percent to 76 percent between FY11–13 and FY14–16 on a portfolio of 299 randomly selected World Bank investment projects. Despite the fact that projects were found to increasingly use multiple mechanisms and pay closer attention to inclusion, citizen engagement in most cases remained “thin,”84 and capacity building for engagement was limited. 225 Appendix K IEG Major and Meso Evaluations Crucially, closing the feedback loop with citizens was not properly mainstreamed or monitored. IFC expanded the magnitude and improved the depth of its stakeholder engagement since the 2012 update in its PS 185 that required stakeholder engagement to occur in all investment projects, with special attention paid to riskier ones. An analysis of IFC investment projects86 from FY07–11 (137 projects) and FY15–17 (30 projects) suggested more frequent plans by IFC clients to engage strategically with stakeholders through a range of mechanisms.87 Clients progressively moved away from corporate social responsibility as the only form of engagement and adopted mechanisms for engagement that are business related. Nevertheless, despite some progress, IFC clients do not fully comply with PS 1.88 The increase in the share of projects, including a beneficiary feedback indicator in the results framework over the past two years,89 hid three important challenges. First, the level of reporting during implementation was low—only 57 percent of approved projects in FY14–17 reported on at least one engagement indicator included in the results framework. Second, there was a discrepancy between the mechanisms included in project design and what the organization reported. Third, very little was known about the contribution of citizen engagement to project outcomes based on Bank Group monitoring because most indicators are process or output oriented, and reporting in self-evaluation is scarce. IFC reporting is driven by compliance with PSs and focused more on processes and outputs rather than outcomes achieved. High reporting standards were not systematically enforced— one-quarter of IFC projects did not elaborate on how the client engaged with the community except in high-risk projects, where reports were slightly more complete although generally insufficient. The evaluation assessed the extent to which Bank Group citizen engagement activities incorporated four quality principles: aiming for results, closing the feedback loop, ensuring inclusion, and building citizens’ and governments’ capacity to engage. Case studies conducted for the evaluation confirmed the literature findings that “thick” approaches were more promising than “thin” approaches when aiming for results.90 The share of World Bank projects included in this evaluation with thick engagement increased from 27 percent in FY11–13 to 38 percent in FY14–16. Giving a tangible response to citizen feedback is fundamental to producing results, sustaining participation, and improving trust. However, there was some confusion on how “closing the feedback loop” should be measured. Bank 226 Appendix K IEG Major and Meso Evaluations Group guidance notes suggested “acting on the feedback of the citizens” while the strategic framework suggested “informing those engaged how the information they provided has been used.” The compliance rate for customer engagement mechanisms included in a sample of projects with the former definition was far higher than the compliance rate for the latter definition.91 Progress was made in ensuring inclusion, particularly at project design, where the share of projects describing participation by women and vulnerable groups for at least one engagement mechanism increased from 40 percent in the FY11–13 sample to 58 percent in the FY14–16 sample. However, the percentage of projects reporting on at least one beneficiary feedback indicator disaggregated by gender or other social group is still low. The new country engagement model foreshadows a more inclusive process that reaches a broader set of stakeholders than in the past.92 Less than one-quarter (23 percent) of projects reviewed had at least one engagement mechanism that included capacity enhancement activities in their design. To this end, the potential of the Global Partnership for Social Accountability,93 one of the few channels that the Bank Group has for building capacity, remains underexploited and could be better used by the Bank Group. Although the link between citizen engagement and positive development results is hard to prove, intermediate changes (in behaviors and relationships, and operations and institutions) that are essential to achieving this ultimate goal were observed in some cases. IEG noted that where citizen engagement mechanisms provided an interface between citizens and service providers for the identification of problems and solutions, conflicts were reduced, and there was increased ownership of the project. The Bank Group was not observed to use citizens’ feedback to adapt its strategies, portfolio composition, project design, and implementation in response to their demands. At the institutional level, the Bank Group used the three entry points94—convening multiple voices in policy dialogue in the formulation of CPFs, triggering reforms that create an enabling environment for citizen-state collaboration through DPF prior actions, and using investment projects to spark institutional changes that can reinforce country systems through scale-up or replication—to varying extents.95 To summarize, the evaluation notes that the Bank Group has been successful in making citizen engagement a top priority and has created awareness and buy-in. However, aspects related to quality are not given sufficient attention at the design and monitoring stage. Five key recommendations are made to address this gap. 227 Appendix K IEG Major and Meso Evaluations First, the World Bank should reflect in future corporate priorities for citizen engagement the need to achieve greater depth and quality of the citizen engagement activities it supports. Second, the World Bank should encourage and support the efforts of its regional, country, and Global Practice teams to establish, where appropriate, “thick” citizen engagement that is regular and continuous, uses multiple tools, and is embedded in country systems. Third, the World Bank should strengthen the monitoring of its citizen engagement activities by systematically adopting results framework indicators that are results oriented. Fourth, the World Bank should harness the opportunity of the implementation of the Environmental and Social Framework to leverage citizen engagement mechanisms—beyond consultations and grievance redress mechanisms—to reach the objectives of managing social risks, strengthening country systems, and promoting social inclusion. Fifth, IFC should ensure that its clients’ stakeholder engagement activities required by PS 1 in projects with affected communities are carried out during appraisal and supervision of the projects and are systematically documented. IFC’s Experience with Inclusive Business The meso evaluation IFC’s Experience with Inclusive Business recognizes the importance of inclusive businesses, but its support remains largely passive rather than proactive or strategic. IFC mainly tags projects that meet the criteria of “inclusive businesses.” The inclusive business dimensions of IFC projects do not influence their selection, structuring, or implementation. Inclusive businesses are not treated any differently than IFC’s regular investment projects. In addition to supporting client companies that focus on the base of the pyramid through funding and capacity building, IFC-issued social bonds and its role alongside the G-20 Global Platform on Inclusive Business elevates IFC’s profile on inclusive business (see figure K.7).96 228 Appendix K IEG Major and Meso Evaluations Figure K.7. International Finance Corporation Approach to Inclusive Business Models Leadership: Operationalize the concept of inclusive business Clients: Social Bonds: Enhance focus on base Support the emerging of the pyramid through market through financing and advisory treasury operations support IFC's approach to inclusive business models Source: Based on World Bank 2018b: http://ieg.worldbankgroup.org/sites/default/files/Data/reports/meso_ifcinclusivebusiness.pdf. Note: IFC = International Finance Corporation. Notwithstanding the passive nature of support, IFC committed about $16 billion to inclusive business companies between FY05 and FY17—about 14 percent of its long-term commitments, consisting of 19 percent of total projects and weighted toward IDA and fragile and conflict-affected countries. About 51 percent of projects are in financial markets (reflecting support to microfinance institutions), with 27 percent in manufacturing, agribusiness, and services; 15 percent in telecom, media, and technology; and only 5 percent in infrastructure.97 Projects with inclusive business models perform at par with other IFC projects, despite higher risks in this project segment. IEG did not find a trade-off between the profitability of projects with inclusive business models and their inclusion objectives, even though inclusive business projects tend to involve factors associated with lower development outcomes (higher-risk countries, sponsors, and markets). Other mitigating factors, such as use of repeat clients, expansion of existing operations, and choice of instruments, have helped to offset such risks. 229 Appendix K IEG Major and Meso Evaluations The clients’ preexisting orientation to integrate the base of the pyramid into its value chain is a key driver for IFC reaching this population in its operations. IFC’s main contribution for inclusive agribusiness projects has focused on the expansion and stabilization of clients that had already made the base of the pyramid part of their business model. IFC’s role was mainly through financial support, but in a few cases, IFC’s Advisory Services directly assisted clients with environmental and social management systems and corporate governance, thus deepening the clients’ engagement with people at the base of the pyramid.98 IFC needs to document the application of its definition and criteria for the identification of its inclusive business projects more rigorously to substantiate their eligibility. IEG’s review of a sample of evaluated inclusive agribusiness projects, using IFC’s definition and criteria, found that most projects are classified correctly, but information to validate whether the people reached by the projects are at the base of the pyramid is inadequate, despite these data being generally available.99 IFC does not adequately monitor the impact of its inclusive business projects on the base of the pyramid. In most cases, IFC monitors and reports on aggregate “reach” indicators—for instance, number of farmers, without distinction between small, medium, and large-size farms—making it impossible to assess the effects on the base of the pyramid. The absence of additional information, for example, on the share of total inputs delivered by small-scale versus medium- or large-scale farms, and the prices paid, or payments made to these farmers for their produce—and the consequent inability to validate the intended welfare benefits on the base of the pyramid—risks undermining the credibility of IFC’s current approach to inclusive business. Three options for future support from IFC arise from the meso evaluation. Based on a review of the entire portfolio and a deep dive into the agribusiness sector, the findings suggest the following options for IFC’s work with inclusive businesses (those businesses that integrate people—suppliers, customers, distributors—at the base of the pyramid in a commercially viable way): First, keep the current approach, noting the report’s cautions about the reputational risks emanating from the absence of robust methods to measure the reach and impact on the population at the base of the pyramid. Second, upgrade IFC’s approach to an operational framework (an incremental approach to proactively identify and select clients with inclusive business models and identify client gaps for deploying IFC’s advisory services program). Third, design a corporate strategy for inclusive business (adopt 230 Appendix K IEG Major and Meso Evaluations a proactive approach with changes to the current approach, risk appetite, and instrument mix supported by a shift in corporate culture and incentives with implications for staff skills and diagnostic work). In addition, IFC could collaborate with the World Bank to (i) work with governments on issues that may affect the viability of inclusive business models, and (ii) correct market failures to provide a conducive framework for inclusive business growth. Maximizing the Impact of Development Policy Financing in IDA Countries The meso evaluation Maximizing the Impact of Development Policy Financing in IDA Countries provides evaluative insights into drivers of success and risks for a sample of DPF operations closed over FY09–17.100, 101, 102 Three groups of findings are associated with DPF success: The first group notes that maximizing the congruence of DPF policies and objectives offers high potential for improving success. Furthermore, ensuring that DPF has strong analytical underpinnings and provides timely technical assistance during implementation offers potential for improving success. The second group notes that government ownership risks are an issue in many DPF programs and in those cases tend to significantly reduce success rates. The better the macropolicies at the outset of DPF, the greater the chances of achieving expected development outcomes. Countries with low capacity operations that secure high government ownership and use simpler designs tend to be more successful. In high-capacity contexts, the borrower’s readiness to pursue structural reforms is still important for success. The third group of findings notes that when government priorities shift toward addressing the impact of large external shocks, operations tend to be more successful when they pursue policy reforms that are congruent with new government priorities or when they address key roadblocks for gaining access to expanded aid. Democracy was insignificant in the econometric analysis, but desk-based case studies show that operations that pursued reforms aligned with the objective of a government seeking reelection were successful. Corruption was insignificant in the econometric model, but case studies confirm its potential to reduce DPF success. 231 Appendix K IEG Major and Meso Evaluations IDA countries’ use and performance of DPF varies across World Bank Regions and A record replenishment for International Development Association (IDA) 18 in the face of a declining share of development policy financing in IDA commitments during the last three IDA cycles raises the question: What are the drivers of success and risks for the use of development policy financing in IDA countries? Practice Groups. Africa accounts for the largest share of DPF (64 percent of 275 operations) and Equitable Growth, Finance, and Institutions (EFI) has the highest share among the Practice Groups (86 percent). DPF performance is highest in East Asia and Pacific and Europe and Central Asia (more than 90 percent of operations achieve outcome ratings of moderately satisfactory or above [MS+] in each Region)103 and lower in South Asia (70 percent MS+), Africa (68 percent MS+), and Latin America and the Caribbean (57 percent). All six Human Development operations show above-the-line outcomes (that is, MS+), and 71 percent of the 98 EFI operations and 78 percent of the nine Sustainable Development operations have outcome ratings of MS+. A literature review identifies 10 factors driving DPF effectiveness.104 Institutional capacity, government ownership, macroeconomic conditions, program design, and analytical work are associated with greater reform effectiveness of DPF operations, whereas external shocks, also with a strong effect, show reduced reform success. The evidence is inconclusive for 4 of the 10 variables studied, with corruption tending toward a negative association, and democracy, technical assistance, and donor coordination tending toward a positive association with reform success. The portfolio review examines 308 objectives from a sample of 111 operations for relevance and efficacy. Ratings for relevance of objectives are substantial or high in 99 percent of IDA DPF operations.105, 106 In 47 percent of cases, operations achieve or nearly achieve their objectives (substantial rating), and in 4 percent of cases, expected outcomes are exceeded. The Human Development and Sustainable Development Practice Groups lead 13 percent of operations (32 percent of objectives).107 Macroeconomic and Fiscal Management (now Macroeconomics, Trade, and Investment) in EFI leads 78 percent of operations (only 10 percent of objectives relate to macropolicy reforms).108 Efficacy is rated substantial or high in 49 percent of EFI-related objectives and is highest within the Human Development 232 Appendix K IEG Major and Meso Evaluations Practice Group (60 percent) compared with 44 percent for Sustainable Development and 49 percent for EFI. Sustainable Development objectives show the highest share of negligible efficacy ratings (16 percent compared with 5 percent for Human Development and EFI). Output from the econometric analysis using the results from the literature and portfolio reviews suggests that the drivers of DPF outcomes are the following: (i) relevance of operation design (large estimated marginal effect); (ii) sound macropolicies at the outset; (iii) high government ownership; (iv) analytical work (as proxied by Public Expenditure Review before DPF operations); (v) technical assistance during implementation; (vi) East Asia and Pacific performance better than other Regions; (vii) multisector operations (50 percent EFI and 50 percent Sustainable Development or Human Development) have higher DPF outcome ratings; and (viii) Joint Policy Assessment Frameworks appear negatively correlated with DPF success (all else being equal). The desk-based case studies categorized projects to illustrate the mechanisms through which the factors of success from the literature review affect the outcomes of IDA DPF operations in practice. Four categories were defined. The first— agreement between the literature and econometric analysis—featured projects characterized by strong program design, underlying analytical work, favorable initial macroconditions, and strong government ownership. The second—strong evidence from the literature not supported by the econometric analysis—includes projects reviewed that feature institutional capacity and external shocks. The third—inconclusive findings from the literature but statistically significant coefficients from the econometric analysis—focuses on projects with technical assistance during implementation. The fourth category—inconclusive findings from both the literature and econometric studies—emphasizes those projects for which the political environment (whether democracy exists) can affect DPF results, in addition to projects implemented in environments where corruption exists and projects that foster donor coordination in the country of implementation. Balancing the predictability of financing flows with DPF success benefits from a programmatic approach to DPF. Risks to this trade-off increase when DPF accounts for larger shares of government budgets because of the moral hazard associated with the (mis)perception of budget support as an entitlement. Furthermore, the opportunity for aid to have a high social return may be lessened in countries with relatively poor public sector management. Compared with stand-alone operations, 233 Appendix K IEG Major and Meso Evaluations longer DPF programs have, on average, a higher success rate (MS+ outcome), although the positive effect decreases with the size of commitment amount.109 1 The evaluation methodology includes a review and assessment of lending, investments, and guarantee portfolios; a literature review; 13 field- and desk-based country case studies; and interviews with government officials, implementing agencies, and World Bank staff. 2 This amount was for 516 projects (300 active and 216 closed). 3Risks were related to the lack of financial sustainability of service providers and inadequate institutional capacity, especially in rural areas. 4Sustainable Development Goal 6 is the universal and equitable provision of water supply and sanitation services by 2030. 5This refers to maximizing financing for development by leveraging the private sector and optimizing the use of scarce public resources, while World Bank Group support will continue to promote good governance and ensure environmental and social sustainability (Maximizing Finance for Development [2017] World Bank: Development Committee). 6This evaluation defines development data as data produced by country systems, the World Bank, or third parties on countries’ social, economic, and environmental issues. 7The International Finance Corporation (IFC) and Multilateral Investment Guarantee Agency (MIGA) are not included in the evaluation. 8The evaluation is informed by a literature review, development of a theory of change, World Bank staff interviews with key informants, and a portfolio review of nine major partnership programs for development data. 9 For example, the Development Grant Facility was phased out, ending World Bank financial support for PARIS21, a partnership to promote better use and production of statistics throughout the developing world. 10The Marrakech Action Plan for Statistics is a global plan for improving development statistics. The Busan Action Plan for Statistics provides legitimacy and funding for statistical capacity building. 11No mechanism exists for medium- to long-term financing of data. Trust funds for statistical capacity were central to past successes, but relying on these creates uncertainty and dependency on a few donors and precludes any long-term planning. 12Capacity building has four aspects: institutions (including laws and enabling environment), human capital (knowledge, skills, and staff incentives), organizations 234 Appendix K IEG Major and Meso Evaluations (budget, infrastructure, leadership, collaboration, and coordination between statistical stakeholders), and data systems and technologies. 13Based on past evaluations by the Independent Evaluation Group (IEG) and project documents, surveys, interviews with clients, and 11 in-depth cases studies of statistical capacity building-initiatives. 14Several governments are reluctant to borrow from the International Bank for Reconstruction and Development or International Development Association (IDA) for data. The World Bank’s Data Council—part of the internal framework for governance and coordination for data production—has proposed principles (though not binding) for funding data production and vital registration systems in IDA–eligible countries based on blending donor funding with an increasing share of domestic funding over time. 15Only 68 of the 201 projects reviewed for the evaluation include support for increasing public access to development data. 16IEG’s review of 201 projects finds that just 27 of these projects support activities to build capacity for data use. 17IEG’s structured survey of staff (655 staff members) reports that 36 percent of staff rate data production, 23 percent rate data sharing, and 27 percent rate data use as either effective or highly effective. 18 Because IFC has not implemented clear criteria for the selection of strategic clients, this evaluation used two criteria to identify de facto strategic client-focused partnerships: repeat client groups and client groups identified by IFC as potential long-term partners. 19 This evaluation includes a portfolio review, a review of internal and external literature, five mission-based and six desk-based country case studies, nine Project Performance Assessment Reports, two partnership reviews, and interviews with Bank Group staff, clients, counterparts, beneficiaries, other stakeholders, and sector experts. The evaluation focused on behavioral change and service delivery. 20The Bank Group had a portfolio of $24.5 billion in urban transport during the review period FY07–16. 21Bank Group activity focuses on the following: (i) improving traffic flows through traffic management interventions, (ii) mitigating travel demand through integrated transport and land use planning, and (iii) shifting travel demand from private vehicles to public transit systems through policies to discourage private vehicle usage. 22Ex ante analysis often does not include an explicit consideration of subsidy, maintenance costs, and alternative options. 235 Appendix K IEG Major and Meso Evaluations 23 Nearly 90 percent of road activities did not have private sector participation. 24This includes missing and inconsistent mobility indicators, the absence of a measurement baseline and measurement over the project life, and the absence of sustained monitoring and evaluation of urban transport services beyond the project life. 25 The evaluation applied six methodological instruments: (i) a strategic mapping of global and regional environmental concerns and the Bank Group response; (ii) a portfolio review of Bank Group projects and activities; (iii) desk reviews of 52 country strategies and a sample of 30 Country Environmental Analyses; (iv) five country case studies involving field missions (the Arab Republic of Egypt, Colombia, Croatia, Ghana, and Indonesia); (v) two literature reviews addressing the links between climate change and pollution agendas and the trade-offs between managing pollution versus sustainable economic development, respectively; and (vi) field-based project evaluations in China, Croatia, and Egypt, and a regional program in Africa. 26 Quantifying these is not possible because their realization is highly contextual. 27Interventions that explicitly address outdoor and indoor air pollution account for only 33 percent and 9 percent of all pollution-targeted interventions, respectively, although these account for 42 and 49 percent of deaths caused by pollution in developing countries, respectively. 28Success relies on strong multistakeholder participation that includes all relevant agencies and strong government commitment. 29The World Bank’s Pollution Management and Environmental Health Program, established in 2015, could help strengthen pollution monitoring, but many of its interventions have not yet closed, hence the effects are not assessed in this evaluation. 30Commonly cited reasons for failure are overambitious goals, lack of government commitment, and failure to address legal and procedural prerequisites. 31Upstream interventions might include measures to support the enabling environment, such as capacity or institution building. Downstream measures might include financing bus rapid transit or required infrastructure. 32 Success depends on the ability to recover costs and the institutional capacity of the municipalities in charge. 33The evaluation assesses the pollution footprint of 956 IFC investments and 168 MIGA guarantee projects in pollution-intense industries (chemicals, oil and gas, and pulp and paper) and 114 World Bank projects classified as posing elevated environmental risks (category A projects). 236 Appendix K IEG Major and Meso Evaluations 34About two-thirds of IFC investment projects meet Performance Standard 3: Resource Efficiency and Pollution Prevention; 72 percent meet pollution prevention requirements for air emissions, 71 percent for wastewater, and 92 percent for waste management. 35No-opinion-possible ratings ranged from 17 to 48 percent of MIGA projects. This lack of information resulting in no-opinion-possible ratings has been addressed by MIGA with the introduction of the 2013 Environmental and Social Sustainability Policy and the 2014 Environmental and Social Review Procedures, which require annual monitoring reports from project enterprises and monitoring site visits by MIGA environmental and social specialists. 36 The World Bank safeguard system does require clients to take actions to avoid or minimize the effect of pollution issues generated from its projects. 37This is the first multisectoral assessment to learn about what works in the rural nonfarm economy (RNFE). 38The methodology consists of a portfolio review of the products in the RNFE space across multiple Global Practices (40 percent of the activity is situated under the Agriculture Global Practice), a review of RNFE products in other spheres—IFC, community-based approaches (with a productive aim), and solution groups (for example, the Global Agriculture and Food Security Program, country case studies, review of country documents [such as Systematic Country Diagnostics, a review of Implementation Completion and Results Report Reviews and Project Performance Assessment Reports], and interviews with staff). This review underpins the framework developed for the evaluation that links a diagnostic understanding of opportunities and constraints in the RNFE to a set of interventions designed to address them. 39Example of such projects are those targeting improvements in rural livelihoods, social safety nets with a productive inclusion theme, and community approaches with a productive aim. 40 The country case studies of Bangladesh and Indonesia are exceptions. 41This gap has at times been addressed with the help of external partners, but not with any consistency. 42World Bank–financed rural transport projects have not measured their contribution to local economic gains. 43These are building income-earning assets, facilitating public and private transfers, promoting fair and transparent institutions, supporting equitable and sustainable macroeconomic policies, and promoting well-functioning markets. 237 Appendix K IEG Major and Meso Evaluations 44 The case studies are Arab Republic of Egypt, Azerbaijan, Bangladesh, Cambodia, Colombia, and Ethiopia. Ghana, Haiti, Indonesia, Serbia, South Africa, and Vietnam. 45Recent studies provide insights into how IFC can contribute to the Bank Group’s shared prosperity goal; see IFC Jobs Study (2013) and Lines of Sight (2014). [[Are IFC Jobs Study and Lines of Sight titles of publications? If not, they should be in sentence case and regular font, not italics. If they are publications, the full titles should be used or add them to the references and cite in author-date format.]] 46The Costa Rica good practice Country Partnership Framework (CPF) features four objectives with a high-quality theory of change on shared prosperity that articulates several causal steps, context variables, and outcomes related to the B40. 47 In a review of the design of approved projects during FY14 and FY16 48Theories of change were strongest in projects aiming to improve transfers to the B40 (50 percent were of high quality), followed by projects aimed at building (their) productive assets (33 percent were of high quality). Theories of change were weakest in projects addressing shared prosperity through improvements in fiscal and monetary policies, better functioning markets, and more fair and transparent institutions. 49According to a comprehensive survey carried out for this evaluation that reports on staff awareness and the extent to which staff are incorporating the shared prosperity goal into their day-to-day activities 50Equality of opportunity in access to basic services, factor and financial markets, and political voice and representation. On average, 68 percent of staff hold this view, rising to 82 percent in the Middle East and North Africa Region, with 71 percent of staff on average expressing support for a larger focus on improving equality in opportunities. 51 Examining the extent to which investment projects target the bottom 40 percent 52Almost 70 percent of Social Protection and Labor projects have a high distributional content; 30 percent of Health, Nutrition, and Population projects; 20 percent of Agriculture and Education projects; and 15 percent of Macroeconomic and Fiscal Management projects. 53A lack of analytic material on Bank Group activities in upper-middle-income countries was a constraint for this report. Many Bank Group reports do not distinguish between categories of client countries by income levels. Scope exists to redress this by analyzing Bank Group performance effectiveness across various groups and subgroups of clients and identifying narrower technical areas in upper-middle-income countries. 54The report is organized along the Bank Group’s three-ways approach to achieving its twin goals: (i) working to accelerate inclusive and sustainable economic growth, (ii) helping 238 Appendix K IEG Major and Meso Evaluations countries invest more effectively in people, and (iii) fostering resilience to global shocks and threats. It covers areas for which sufficient and relevant evaluative information is available. 55A significant gap remains between Organisation for Economic Co-operation and Development countries and upper-middle-income countries on quality of institutions and governance, which underscores the need for Bank Group long-term engagement. 56With MIGA through the Joint International Financial Institution Action Plan for the Europe and Central Asia Region (Vienna Initiative) and with IFC asking for funds from external partners. 57 This was to lessen portfolio losses and focus on short-term trade financing. 58IEG’s evaluations of investment climate reforms note that the World Bank focuses more on higher-level reforms (revising and harmonizing laws and codes, reforming institutions, and developing strategies), whereas IFC focuses mostly on streamlining and simplifying procedures and processes. This division of labor rarely generates synergy or enhances effectiveness. 59For example, Bank Group support in the transport sector for performance-based road maintenance contracts helped the governments strengthen the policy and institutional environment for managing their entire road network. 60Adopting new or innovative programs, and adapting and fine-tuning existing programs to enhance effectiveness and enable knowledge transfer. 61Including risk pooling to increase the poor’s participation with concurrent steps to subsidize their enrollment. 62IFC contributes through investment in public-private partnerships or privately owned urban service providers. 63 The refugee crisis in Jordan and Lebanon. 64The report cites the case of Jordan and Lebanon, where the civil war in neighboring Syria resulted in an influx of refugees and pressure on state systems and resources. 65The evaluation uses a mixed-method approach that triangulates evidence from the portfolio analysis, six country case studies, and intervention case studies of delivery mechanisms (conditional cash transfers and performance-based financing, and public- private interactions) and of the World Bank’s response to pandemic outbreaks. 66Health financing reforms to improve equity in health financing and service use, financial protection, and efficiency. 239 Appendix K IEG Major and Meso Evaluations 67In 2013, more than 400 million people worldwide were not receiving at least one of the seven essential health services identified as priority areas in the Millennium Development Goals. 68The results-based approach benefits from research funded by the Health Results Innovation Trust Fund and the Global Financing Facility, which builds country institutional capacity for results-based financing. 69This was due largely to the turnaround in health project performance in the Africa and Sub-Saharan Africa Region. 70 Performance was 71 percent for the entire World Bank. 71 The overall IFC investment portfolio achieves a rating of 57 percent, and the overall IFC Advisory Services portfolio achieves a rating of 58 percent. 72Support for maternal and child care increased from 39 percent (in closed projects) to 60 percent (in open projects) over the evaluation period. 73IFC projects show a high frequency of repeat support to client networks, although the realization of incremental additionality diminishes over time. 74Monitoring and evaluating distributional impacts of World Bank and IFC health service projects for specific disadvantaged population groups should be pursued. 75About 88 percent of IFC investment projects have an objective related to improvements in access. However, limitations of the monitoring frameworks of these projects preclude a determination of whether they contribute to expanding coverage. 76Only 46 percent of objectives in World Bank projects are rated moderately satisfactory and above. Projects have limited capacity to monitor and evaluate all aspects of quality. An exception is performance-based financing—projects show better results in quality improvement and stronger monitoring and evaluation frameworks. 77 The low rating may reflect the limitations of the indicator chosen because mortality rates may not be sufficiently sensitive over the project life. 78This is despite the Joint World Bank Group Approach to Harnessing the Private Sector in Health (2015)[[AQ: Is “Joint World Bank Group Approach…” the title of a publication? If not, it should be in sentence case and regular font.]] . The Health, Nutrition, and Population Global Practice has recently appointed a Global Solutions lead for private sector engagement to facilitate this. 79 The World Bank supported 63 countries under the Global Program on Avian Influenza Control and Human Preparedness during 2006 –13. It was a vital member of the coalition 240 Appendix K IEG Major and Meso Evaluations against the Ebola outbreak in West Africa in 2014–15 and mobilized needed financial resources. 80 In reference to the six country case studies from the evaluation. 81The Bank Group’s newly launched Maximizing Finance for Development approach can be applied to achieve greater synergies between the public and private sector. 82The evaluation adopted a multilevel, mixed-methods, and case-based design. Six portfolio reviews were conducted along with 8 case studies of country-level engagement and 40 case studies of engagement mechanisms embedded in 19 projects across 11 countries that were all corroborated by evidence from literature reviews. The corporate environment was assessed through a survey and interviews with staff and management, and the evaluation team also held online consultations with civil society organizations. 83Beneficiaries can be tracked using three process indicators: the percentage of investment projects with a citizen-oriented design (having at least one citizen engagement mechanism among the project activities); the percentage of investment projects with at least one beneficiary feedback indicator in their results frameworks; and the percentage of projects that report on a beneficiary feedback indicator (the latter is still in the process of being rolled out). 84The types of engagement mechanisms have not changed from what they were before the introduction of the corporate mandate. Consultations and grievance redress mechanisms remain the most common. Mechanisms implying a light degree of engagement (for example, informing or consulting) are much more frequent than more intense forms of engagement (for example, collaborating or empowering). 85IFC and MIGA require clients to adhere to a set of eight Performance Standards, which constitute an integrated sustainability and risk management framework that was revised in 2012 by IFC and in 2013 by MIGA. Performance Standard 1 (Assessment and Management of Environmental and Social Risks and Impacts) requires stakeholder engagement to occur in all investment projects, with special attention paid to riskier ones. 86 MIGA was excluded from the evaluation because of its highly intermediated relationship with its final beneficiaries. 87 The mix of activities that IFC clients selected to engage with stakeholders included more proactive rather than reactive and/or risk-mitigation activities with respect to the past and, since 2012, systematically included forms of stakeholder engagement directly related to clients’ business and the project activities. 241 Appendix K IEG Major and Meso Evaluations 88Several IFC projects approved between FY15 and FY17 were without grievance redress mechanisms or a system of disclosing project information. Many did not feature stakeholder engagement plans. 89Before the corporate commitment, 42 percent of projects had at least one beneficiary feedback indicator, increasing to 63 percent in FY14–16 (after the introduction of the corporate commitment) and to 95 percent in FY16. 90“Thick” approaches refer to approaches that combine multiple tools to enable collective action and public sector responsiveness. “Thin” approaches refer to approaches that are not matched with vertical integration of independent monitoring and oversight or do not include support to increase a government’s capacity to respond. 91When closing the feedback loop is interpreted as “acting on the feedback of citizens” (definition included in Bank Group guidance notes), IEG found that 70 percent of the mechanisms included in a sample of FY16 projects met this criterion, while just 4 percent of projects met the criterion when it was interpreted as “informing those engaged how th e information they provided has been used” (definition included in the strategic framework). 92CPF teams now consult much more frequently with local governments and local nongovernmental organizations—in some cases partnering with local civil society organizations helped improve the outreach and quality of consultations. 93Despite positive exceptions, better synergies between Global Partnership for Social Accountability grants and World Bank projects taking place in the same country would have helped the World Bank extend its engagement with civil society organizations in broader governance reforms while improving the prospects of institutionalizing Global Partnership for Social Accountability–supported social accountability mechanisms and sustaining their results. 94These were suggested by the structured reviews and case studies conducted for the evaluation. 95There were several examples of how the Bank Group promoted citizen engagement through its country strategies, although it made less use of its convening power to bring citizens to the policy discussion table outside of the CPF consultations. Only 10 percent of all prior actions in development policy financing (DPF) since the early 1990s have broadly supported citizens’ capacity to act by improving access to law and justice, introducing conflict prevention mechanisms, or enhancing state capacity to respond to citizens’ demand through decentralization or judicial or civil service reforms. There were quite a few examples of projects that strengthened country systems for engagement and sectorwide operations that embedded citizen engagement practices. 242 Appendix K IEG Major and Meso Evaluations 96As part of the call to action at the G-20 Leaders’ Summit in 2015, the G-20 endorsed the G- 20 Inclusive Business Framework, which presents policy options for G-20 and non-G-20 governments, companies, and international financial institutions to promote and support inclusive business. 97 Compared with a 23 percent share of infrastructure projects in the rest of the portfolio. 98Of the 20 sample projects, 4 showed that advisory services by IFC directly assisted with the deepening of clients’ engagement with base-of-the-pyramid farmers. 99For example, IFC project documents often lack data on farmer incomes, the share of smallholder farmers, or their expected revenues. 100This is pertinent in the context of the record 18th Replenishment of IDA and in the context of a declining share of DPF in IDA commitments. The share of DPF in IDA declined from 24 percent in FY08 to 15 percent in FY15. 101Factors are informed by the methodology, comprising a literature review, a portfolio review of the sample of DPF operations, econometric analysis, and desk-based case studies. The evaluation focused on two scenarios that suggested a number of questions to be considered: (i) Where the World Bank has provided DPF in IDA countries, to what extent and under what conditions has DPF been successful in supporting improvements in policy and institutions? What are the key factors for success?; and (ii) Where the World Bank has provided multiyear DPF in IDA countries, has there been a trade-off between predictability of financing and strength of policy reforms? Under what conditions has multiyear DPF eroded the strength of policy reforms? What approaches have been effective in minimizing such trade-offs? 102The composition of the 175 operations used in the evaluation is not statistically different from the total universe of 275 operations closed during FY09 –17. 103Based on IEG outcome ratings and the six-point classification system (highly satisfactory, satisfactory, moderately satisfactory, highly unsatisfactory, moderately unsatisfactory, and unsatisfactory). Performance ratings need to be interpreted with care given the small sample sizes outside of the Africa and Sub-Saharan Africa Regions and Equitable Growth, Finance, and Institutions. 104 A broad literature review highlighted 10 relevant variables that were then ranked based on a coding of 36 influential papers in terms of the frequency with which the variables were covered and their role in driving DPF effectiveness. 105 Based on the ratings from the Implementation Completion and Results Reports —high, substantial, modest, and negligible. 243 Appendix K IEG Major and Meso Evaluations 106This suggests a significant consistency of objectives with countries’ development priorities and Bank Group country strategies and corporate goals. 107Equitable Growth, Finance, and Institutions led 87 percent of IDA DPF operations and accounts for 69 percent of operations’ objectives. 108By contrast, 38 percent of objectives focus on governance issues and 24 percent on finance, trade, and competitiveness. 109The econometric analysis shows that when commitment amounts are 1.1 percent of government expenditures (25th percentile of DPF in the sample database), programmatic series with at least three operations are 18 percentage points more likely to succeed than shorter series or stand-alone operations. At the 90th percentile of commitments (6.25 percent of government expenditures), the success probability of such a program is 24 percentage points lower. 244 Appendix K IEG Major and Meso Evaluations References World Bank. 2017a. A Thirst for Change: An Evaluation of the World Bank Group’s Support for Water Supply and Sanitation with Focus on the Poor. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/Water%20Supply%20and%2 0Sanitation%20Infographic_online.pdf. ———. 2017b. Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/evaluations/data-for-development. ———. 2017c. The International Finance Corporation’s Approach to Engaging Clients for Increased Development Impact. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/IFCClientE ngagement.pdf. ———. 2017d. Mobile Metropolises: Urban Transport Matters: An IEG Evaluation of the World Bank Group’s Support for Urban Transport. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/urban- transport. ———. 2017e. Toward a Clean World for All: An IEG Evaluation of the World Bank Group’s Support to Pollution Management. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/pollution. ———. 2017f. Growing the Rural Nonfarm Economy to Alleviate Poverty: An Evaluation of the Contribution of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/evaluations/rural-non-farm- economy. ———. 2017g. Growth for the Bottom 40 Percent: The World Bank Group’s Support for Shared Prosperity. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/shared-prosperity. ———. 2017h. World Bank Group Engagement in Upper-Middle-Income Countries: Evidence from IEG Evaluations. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/upper-middle-income-countries. ———. 2017i. World Bank Group Support to Health Services: Achievements and Challenges. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/world-bank-group-health-services. 245 Appendix K IEG Major and Meso Evaluations ———. 2018a. Engaging Citizens for Better Development Results: An Independent Evaluation. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/engaging-citizens-better-development- results. ———. 2018b. IFC’s Experience with Inclusive Business. Independent Evaluation Group. Washington, DC: World Bank. http://ieg.worldbankgroup.org/sites/default/files/Data/reports/meso_ifcinclusivebus iness.pdf. ———. 2018c. Maximizing the Impact of Development Policy Financing in IDA Countries. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/reports/meso- devpolfinancing.pdf. 246