52606 FAST TRACKBRIEF September 23, 2008 The IEG evaluation "The World Bank's Economic and Sector Work and Technical Assistance, FY00-06" was discussed by CODE on July 9, 2008 The World Bank's Economic and Sector Work and Technical Assistance, FY00-06 Economic and sector work (ESW) and nonlending technical assistance (TA) are two of the analytical and advisory services (AAA) through which the Bank provides knowledge support to its client countries. The objectives of ESW are to inform lending, inform government policy, build client capacity, stimulate public debate, and influence the development community. The objectives of TA are to assist in policy implementation, strengthen institutions, and facilitate knowledge exchange. ESW and TA are an essential part of the Bank's engagement with its clients--it spent $910 million (or 26 percent of its spending on country services) on these products during FY00-06. The majority of ESW and TA met their objectives at least to an average extent during FY00-06, although there were substantial differences across countries and tasks. ESW and TA of higher technical quality were clearly more effective in meeting their objectives. Close collaboration with clients from task initiation through the formulation of recommendations was important for ESW and TA to be effective, whether clients were involved in the production of the task or not. Sustained follow-up after the completion of the tasks was important for effectiveness. Whether clients requested the tasks did not matter for effectiveness, although all tasks needed to be tailored to client needs and interests to be effective. ESW and TA were less effective in countries where government capacity was lower. Clients in middle-income countries prefer nonlending to lending services, and clients in all countries prefer TA over ESW. IEG has five recommendations for Bank action to obtain better results from ESW and TA. First, reinvigorate the mandate (which underpinned the ESW reforms in FY99) to maintain a strong knowledge base on countries and sectors where the Bank is providing or planning to provide funds. Second, ensure ESW in IDA countries are adequately-resourced (even if it means fewer ESW), since cost matters for quality, and quality matters for effectiveness. Third, enhance institutional arrangements for ESW and TA by ensuring substantive task team presence in country offices--particularly in countries with low institutional capacity--to facilitate closer client collaboration. Fourth, recognize, receive, and build on client feedback to counter-balance current Bank incentives for lending over nonlending and ESW over TA. Fifth, take the results tracking framework for ESW and TA more seriously, including systematizing client feedback. countries gave ratings of average and above on the extent to The Extent to Which ESW and TA Met Their which ESW and TA met their stated objectives, compared to Stated Objectives between 74 and 87 percent of such users in the Bank (task team leaders for loans and strategies). ESW had the largest Most ESW and TA met their stated objectives at least to effects on informing Bank strategy, which is not a stated an average extent, though they were more effective in ESW objective. shaping Bank lending and strategy than in directly providing support to client countries. The indirect effects The presence of relevant ESW was associated with of ESW and TA on client countries--through Bank better loan design. The evaluation period was not long lending--were greater than the direct effects. Between enough for ascertaining the effect of ESW on loan 65 and 80 percent of users of Bank ESW and TA in client outcomes. ESW in a range of sectors and of different report types, The effectiveness of ESW and TA ranged from including tasks initiated by the Bank, had positive substantially above average to substantially below effects both within the Bank and in client countries. average across countries and across tasks. The wide Country Economic Memoranda (CEMs) and Public range of effectiveness is in part due to country-specific Expenditure Reviews (PERs) were especially useful for factors, such as government capacity and government informing Bank strategies and development policy loans receptivity, and in part due to factors that are more amenable (DPLs). Within client countries, PERs, Financial Sector to being addressed by the Bank. These factors are discussed Assessment Programs (FSAPs), and Investment Climate next. Assessments (ICAs) were among the more prominently cited ESW products that had, variously, informed Factors in ESW and TA Effectiveness government policy, built capacity, stimulated public debate, When government capacity was lower, ESW and TA and influenced the development community. The box on were less effective. ESW and TA were less effective in post- this page provides examples of particularly influential ESW conflict and otherwise low-capacity countries. Policy makers and TA. Examples of less effective ESW and TA are in such countries had limited capacity to absorb all the ESW provided in the box on the next page. produced. High turnover of government officials also ESW and TA had effects beyond the stated objectives in negatively affected capacity and, in turn, the effectiveness of some cases. The more prominent ones included effects on ESW and TA in some countries. countries other than those for which they were intended, Government receptivity also influenced the private investment decisions by domestic and international effectiveness of ESW. There were instances where investors, support for local research, and the creation of political constraints influenced government receptivity to informal networks between Bank staff and policy makers. Influential ESW: Examples from Vietnam and Mauritius Stakeholders in most but not all The PER is one of the core diagnostic ESW products that was influential both within the of the twelve countries reviewed Bank (in informing DPLs and Country Assistance Strategies) and directly in client in-depth indicated that ESW and countries (in informing government policy and building capacity). The Vietnam PERs TA had made a difference to the (2000, 2005) are among those that had important direct effects. The two reports helped reforms in their countries. This shape legislation to increase the transparency in the state budget law (2002) and led to view was expressed in countries of the adoption of a medium-term expenditure framework and unification of the public a wide range of income, accounts system. In addition to informing government policy in these ways, the PERs government capacity, and policy also built government capacity through the collaborative way they had been done. This orientation. In some countries, is reflected in the increasing involvement of the government in the production of the stakeholders indicated that the PER, from providing data (1996) to writing some sections of the report and providing reforms in their countries would data (2002) to writing the report on an equal footing with the Bank (2005). not have happened at all without Bank ESW and TA. Others The Mauritius Aid for Trade TA (2006) exemplifies influential TA. The government indicated that without Bank ESW requested the TA to help define its reform program and to mobilize funds as part of the and TA, the reforms would have, multidonor Aid for Trade initiative. Within two months of the request, a Bank mission variously, been of lower quality, provided analysis of possible reforms, including estimates of adjustment costs of opening taken longer, and been more up, tax reforms, and strategies to lower the cost of key services. At the end of the two- piecemeal and less focused. Some week mission, the Bank team presented the government with a summary of the main indicated that Bank ESW and TA elements of the reform options. Among the important factors underlying the had helped them avoid or reduce effectiveness of the TA was the timely nature of the support and the leveraging of the cost of policy mistakes. Bank existing ESW. After extensive internal debate, many of the mission's recommendations ESW and TA also provided a level were incorporated into the government's reform program, which was supported by a of comfort or confidence for some follow-up Bank loan. countries to undertake reforms. A few pointed to the credibility that Bank ESW and TA certain ESW products, either rendering such ESW less provided. Clients valued ESW and TA for their high effective or delaying its effectiveness until the political technical quality, objectivity and neutrality, and provision of situation changed. international perspectives. In 2 of the 12 countries reviewed, ESW products of lower technical quality were less however, a majority of the in-country stakeholders indicated effective. In some instances the Bank did not have staff and Bank ESW and TA made no difference to the reforms in consultants with the requisite skills for particular ESW and TA their countries. tasks, which undermined the Bank's credibility. 2 ESW tasks of higher technical quality cost more. needed. Having Bank staff or consultants that speak the local Furthermore, the average cost of ESW tasks was lower in language clearly enhanced the effectiveness of these tasks. IDA than in IBRD countries. The Bank budget but not ESW and TA were not effective when some of these the trust fund component of cost was associated with the factors were not in place (see the following box). quality of ESW. Close collaboration with clients, from the initiation of What Clients Want from the Bank the task through the formulation of recommendations, was important for ESW and TA to be influential. This Stakeholders in IBRD countries indicated a strong may or may not have entailed client involvement in the preference for the Bank's nonlending services over its production of the task, such as writing sections of a report. lending services. The preference is less strong in IDA countries. TA is preferred to ESW in both groups of Where partnership with clients made a difference, a countries. Middle-income countries (MICs) value the variety of approaches was equally effective depending on Bank's advice, and some of them borrow from the Bank in country conditions. In countries with more limited order to obtain that advice. Stakeholders in some MICs government capacity, partnering in the production of narrowly expressed a strong view that Bank ESW should be de- focused tasks was more effective than partnering in broadly linked from its lending operations. IDA countries need the scoped tasks. Bank's funds but also value the Bank's advice. TA is Whether the client specifically requested the ESW and preferred to ESW in countries with high government TA did not matter for effectiveness, although client capacity as well as in those with lower capacity. buy-in was still important. In-country stakeholders in Stakeholders found the Bank's reports and TA more both low- and middle-income countries indicated that the useful than those provided by other institutions. This Bank has a useful role in proposing topics for ESW and TA, drawing on Bank knowledge and international experience. When ESW and TA Were Not Effective: Examples from the Country Reviews Nevertheless, the Bank needs to ensure there is genuine client interest Limited government capacity and insufficient collaboration/partnership to garner or to engender such interest. government buy-in were the main reasons behind the lack of success for several ESW Tailoring tasks such as core in the Democratic Republic of Congo (DRC). In contrast, a few ESW in DRC had diagnostics to clients' needs and very positive effects because of substantial partnering and collaboration. making efforts to collaborate from the Even in countries with high levels of capacity, purely supply-driven tasks without conception stage were both effective government buy-in were not effective. An example is economic monitoring ESW in in engendering client interest. middle-income countries that had access to such analysis from the private sector or Sustained follow-up beyond one- international investors. The Bank's economic monitoring ESW had very little value- off dissemination was important added in such circumstances. for ESW and TA effectiveness. The Lack of government receptivity due to political constraints could also affect or delay the ESW and TA with the greatest effects usefulness of ESW. In Peru, the Sierra Rural Development Strategy became influential were those for which there was only four years after its completion, after a change in government. sustained follow-up upon completion. The follow-up could be in the form of Poor quality and slow translation were other reasons behind the poor effects of some lending or nonlending activities, with ESW in some francophone West African and Latin American countries. In one of the the latter encompassing formal or countries, the translation took so long that when the documents finally became available informal TA. in the local language, they had lost much of their importance for policy making. Whether dissemination was broad finding was stronger for TA than for ESW. or narrowly targeted did not matter for effectiveness. In some cases, targeted dissemination to the relevant The Bank's clients have a strong desire for greater counterparts resulted in important changes, while in other Bank presence in country. Having ESW and TA cases wide dissemination stimulated the public debate that undertaken out of country offices facilitates the close spurred changes. How wide dissemination needed to be for collaboration on these tasks that is important for them to ESW and TA to be effective also varied within the same be effective. Follow-up is also best sustained out of country for different topics. country offices given the long-term close interactions that are typically entailed. Related to this is the need to further Translation is important to client countries. devolve decision-making powers to the country offices, as Stakeholders in a majority of the countries reviewed for this they are front-line recipients of government requests, evaluation indicated that better and more rapid translation is especially for rapid-response TA. 3 The Bank's Results Tracking Framework for offices particularly in countries with low institutional capacity. This will facilitate closer collaboration with clients ESW and TA from task initiation through follow-up. In addition, The Bank needs to take the results tracking formulate a dissemination and implementation strategy for framework for ESW and TA much more seriously. ESW and TA at the concept paper stage. Such a strategy Although such a framework exists, its implementation is should identify the target audience, the mode of not enforced. This can be seen in the poor quality of dissemination, and the follow-up arrangements after information on ESW and TA in the Bank's administrative dissemination, all of which should be explicitly budgeted database, as well as in the incomplete information in the for as integral parts of the task. results tracking framework themselves. This in turn Fourth, recognize and build on client preferences, reflects the lack of serious monitoring or reporting of these whether for nonlending over lending services or for TA tasks. Finally, there is no systematic follow-up with clients over ESW. Institute a mechanism to obtain client to evaluate the effects of these products. feedback on a periodic basis on delivered ESW products. Recommendations Such feedback should include clients' views on collaboration, follow-up, and usefulness of the tasks During FY00-06, the Bank spent nearly $1 billion on ESW (including specifics of how the tasks were used). The client and TA, slightly more than one-quarter of its spending on feedback should be requested at a set period (around one country services. While it is not possible to quantify the year) after the delivery of the task to the client to allow time impact of this spending, this evaluation found that in the for follow-up, and it should be the last milestone for ESW aggregate, Bank ESW and TA met their objectives to at and TA. Obtaining client feedback would encourage a least an average extent. Furthermore, Bank clients find stronger results focus for ESW and TA and help to these products useful, with clients in MICs finding them counterbalance current Bank incentives for lending over more useful than lending. In light of these findings, IEG nonlending and for ESW over TA. has several recommendations to help the Bank obtain better results for ESW and TA. These recommendations Fifth, take the results tracking framework seriously, will also help support the Bank's six strategic directions, as including by incorporating systematic client feedback as knowledge is important for all of them. noted in the above recommendation. First, reinvigorate the mandate--which under-pinned the FY99 ESW reforms--for country teams to maintain a strong knowledge base on countries and The views expressed here are those of IEG and should not be attributed sectors where the Bank is providing or planning to to the World Bank Group. The findings do not support any general provide funds. Bank country strategies and lending inferences beyond the scope of the evaluation, including any references activities need to continue to be supported by requisite about the World Bank Group's past, current or prospective overall analysis, although a return to strictly defined "core performance. diagnostic" ESW is unnecessary. The Fast Track Brief, which summarizes major IEG evaluations, will Second, ensure ESW in IDA countries are adequately be distributed to World Bank Group staff. If you would like to be resourced, even if it means fewer ESW in some added to the subscription list, please email us at ieg@worldbank.org, countries. This will help to address the lower level of with "FTB subscription" in the subject line and your mail-stop resources for individual ESW tasks in IDA countries number. If you would like to stop receiving FTBs, please email us at compared to those in IBRD countries, and is supported by ieg@worldbank.org, with "FTB unsubscribe" in the subject line. the findings that cost matters for quality and quality matters for effectiveness. Greater selectivity will also help to reduce the burden on limited government capacity in some IDA Contact IEG: Director-General, Evaluation: Vinod Thomas countries, and will free up resources for more collaboration Director: Cheryl Gray (IEG-WB) and follow-up needed to enhance the effectiveness of ESW. Senior Manager: Ali M. Khadr (IEGCR) Selectivity could be enhanced by giving priority to ESW that Task Manager: Helena Tang (IEGCR) informs Bank lending and strategy or is clearly desired and needed by the client. Copies of the report are available at: Third, enhance the institutional arrangements for http://www.worldbank.org/ieg/esw undertaking ESW and TA. To the extent allowed by IEG Help Desk: (202) 458-4497 budget, ensure substantive task team presence in country E-mail: ieg@worldbank.org 4