THEWORLD BANK 40162 Using Training to Build Capacity for Development An Evaluation of the World Bank's Project-Based and WBI Training THE WORLD BANK GROUP WORKING FOR A WORLD FREE OF POVERTY The World Bank Group consists of five institutions--the International Bank for Reconstruction and Development (IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), the Multilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of Investment Disputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their envi- ronment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public and private sectors. THE INDEPENDENT EVALUATION GROUP ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses on assessment of IFC's work toward private sector development, and IEG-MIGA evaluates the contributions of MIGA guarantee projects and services. IEG reports directly to the Bank's Board of Directors through the Director-General, Evaluation. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group's work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. W O R L D B A N K I N D E P E N D E N T E V A L U A T I O N G R O U P Using Training to Build Capacity for Development An Evaluation of the World Bank's Project-Based and WBI Training 2008 The World Bank http://www.worldbank.org/ieg Washington, D.C. ©2008 The International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved 1 2 3 4 5 11 10 09 08 This volume is a product of the staff of the International Bank for Reconstruction and Development / The World Bank. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgement on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development / The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; Internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org. Cover photo: Trainee midwives at Calcutta College of Nursing. Cover photo by: Liba Taylor/Corbis. ISBN: 978-0-8213-7378-1 e-ISBN: 978-0-8213-7379-8 DOI: 10.1596/978-0-8213-7378-1 World Bank InfoShop Independent Evaluation Group E-mail: pic@worldbank.org Knowledge Programs and Evaluation Capacity Telephone: 202-458-5454 Development (IEGKE) Facsimile: 202-522-1500 E-mail: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 Printed on Recycled Paper Contents vii Abbreviations ix Acknowledgments xi Foreword xiii Executive Summary xvii Management Response xxi Chairperson's Summary: Committee on Development Effectiveness (CODE) 1 1 The Role of Training in Capacity Building 3 The Nature of World Bank Training Support 5 Scope of the Evaluation 5 Evaluating the Training Results Chain 11 2 Effectiveness of Training 13 Learning Outputs 14 Workplace Performance Outcomes 15 Impact on Development Capacity 17 Attributing Training Results 19 3 What Training Works: Training Design 23 Pedagogy 24 Support for Transferring Learning to the Workplace 25 Targeting of Training 26 Diagnosis 28 Training-Needs Assessment 29 Participant Selection 31 From Process to Context: Making Learning Work for the Organization 33 4 When Training Works: The Organizational Context for Training 36 Importance of Organizational Context for Training Success 37 Organizational Incentives and Resources for Applying Learning 38 Importance of Client Commitment 41 5 Bank Processes and Systems 43 Identifying and Measuring the Results of Training 46 Bank Support for Training 49 Bank Resources for Effective Training 51 6 Findings and Recommendations 53 Findings 55 Recommendations i i i USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT 57 Appendixes 59 A: Methodology 65 B: Assessing Training for Successful Capacity Development: Literature Review Summary 69 C: Summary of Client Surveys in Six Countries 81 D: Monitoring and Evaluation of Training Outcomes in Closed Bank Projects 83 E: Summary of Interviews with Bank Task Team Leaders 91 F: Summary of WBI's Partner Training Institute Survey 97 G: Assessment of WBI's Activity Initiation Summary Forms 99 H: List of WBI Interviewees 101 I: List of Projects in Country Review Studies 103 J: Detailed Management Comments 109 Endnotes 115 Bibliography Boxes 8 1.1 Evaluation Tools 16 2.1 How Work Changed Following Training 22 3.1 Drivers of Training Success 23 3.2 Diverse Pedagogical Methods Support Learning 27 3.3 When Not to Train 35 4.1 Why Learning Cannot Be Implemented 36 4.2 Importance of Workplace Environment to Training Success 40 4.3 Setting Up Training-Feedback Loops in the Bangladesh Public Procurement Reform Project 45 5.1 Limited Focus of WBI Level-1 Evaluation 46 5.2 Participant Satisfaction Questionnaires: What Can They Tell Us? 49 5.3 Collaboration between WBI and Operations: Contrasting Cases 50 5.4 Insufficient Building of Partner Training Institute Capacity by WBI Figures 4 1.1 Projects in the Health, Nutrition, and Population Sector Involve the Most Training 4 1.2 Adaptable Program Loans Have the Highest Shares of Training 5 1.3 WBI Trained Mainly Government Staff and Academics 7 1.4 A Results Chain for Training 14 2.1 Most Training Examined in Field Studies Resulted in Learning Gains 15 2.2 About Half of Trainees Reported Substantial Positive Changes to Work Performance 26 3.1 Project-Based Training Provided Adequate Follow-up Support More Frequently Than WBI 28 3.2 Adequate Training-Needs Assessments Were Done in a Majority of Training Programs Reviewed in Field Missions 31 3.3 Participant Selection Strategies Reveal a Wide Range of Weaknesses 37 4.1 Projects Rate Higher Than WBI on Attention to Capacity Context 39 4.2 Client Commitment Is Stronger in Projects 47 5.1 Most TTLs Would Prefer Having an Internal Unit of Experts 47 5.2 Most TTLs Seek Advice from Bank Operational Colleagues and External Consultants i v C O N T E N T S Tables 18 2.1 Same Project, Different Training Outcomes 21 3.1 Design Factors for Successful Training 24 3.2 Courses Are Interesting but Too Short and Lack Hands-On Work 27 3.3 Targeting of Training 30 3.4 Four Strategies for Selecting Participants 30 3.5 Different Participant Expertise Levels Impair Training Results 36 4.1 Lack of Relevance Is the Main Reason Why Training Lacks Impact 38 4.2 Organizational Incentives for Implementing Learning Are Satisfactory but Material Resources Are Insufficiently Available 45 5.1 Four Levels of Training Evaluation v Abbreviations AIS Activity Initiation Summary ICR Implementation Completion Report IEG Independent Evaluation Group IMF International Monetary Fund InWEnt Internationale Weiterbildung und Entwicklung (Germany) ITCILO International Training Centre of the International Labour Organization JICA Japan International Cooperation Agency M&E Monitoring and evaluation MASHAV Israeli Center for International Cooperation NGO Nongovernmental organization PAD Project appraisal document TTL Task Team Leader WBI World Bank Institute WBIEG World Bank Institute Evaluation Group v i i Acknowledgments This report was prepared by a team led by Aliza University), and Nils Fostvedt (IEG) served as Belman Inbal, with Sue Ellen Berryman (project- peer reviewers of the final evaluation report. financed training and benchmarking), Deepa Chakrapani, Shalini Ishwar Ahuja, Ingrid IEG is particularly appreciative of the willingness Knutson, Mirafe Marcos, Nara Meli, and Rupa of the staff at the following benchmarked institu- Ranganathan. Yacouba Yaro and Kattie Lussier tions to share their experience and expertise: supported the field missions. William Hurlbut Gudrun Kochendorfer-Lucius, Hinrich Mercker, edited an earlier version of the manuscript. The and the staff from InWEnt; Francois Eyraud, report was edited by Helen Chin for publication. George Jadoun, Frans Lenglet, and the staff from The team received overall assistance from Yezena ICTILO; Masaei Matsunaga from JICA; Haim Yimer, Julia Ooro, and Erika Lorenzana. Divon and Mazal Renford from MASHAV; Donogh Colm McDonald from the IMF Institute; and Valuable early contributions were made by Edward Bales from Motorola. William Battaile, Yang-Yang Chen, Mariola Chrostowska, and Anita Yang. The report also The WBI provided extensive support for data benefited from the advice of Kenneth Chomitz collection. In particular, IEG would like to thank and Anju Vajja. Donald Foster and Charles Kabir Ahmed, Caroline Adams-Brezina, Kenneth Abelmann provided useful advice on evaluation King, Richard Tobin, Kawaljit Bhutani, Manar El- design. Kattie Lussier wrote the literature review Iriqsousi, Lisa de los Angeles, and Ellen de and Globescan coordinated the six-country Vreede. client survey and WBI-partner training institute surveys. The team also benefited from discus- The Independent Evaluation Group (IEG) sions and advice on survey design from Kathryn acknowledges the support provided by the Newcomer and Ann Doucette (George Washing- Norwegian Agency for Development Coopera- ton University), and from Gordon Willis and tion (NORAD) for this evaluation. Kristin Miller (National Institutes of Health). The evaluation was conducted under the The team is grateful for the field mission support guidance of Victoria Elliott, Manager, IEG provided by Theodore Ahlers (Tunisia), Ellen Corporate Evaluation and Methods. Goldstein (Burkina Faso), Isabel M. Guerrero (Mexico), and Christine Wallich (Bangladesh). Vinod Thomas, Director General of the IEG, was recused from this evaluation owing to his past Marissa Espineli (Aga Khan Foundation), Peter tenure as Vice President of the World Bank Taylor (Institute of Development Studies, Sussex Institute from 2000 to 2001. Director, Independent Evaluation Group, World Bank: Ajay Chhibber Manager, Corporate Evaluation and Methods: Victoria M. Elliott Task Manager: Aliza Belman Inbal i x Foreword To achieve the goals of sustainable growth and respective roles of a central training unit and poverty reduction, developing countries need Regional programs." strong national institutions. To this end, the World Bank has devoted significant resources to This evaluation addresses these questions and building the capacity of institutions and organi- examines how training is used alongside other zations in client countries. The Bank helps Bank support to achieve clients' capacity- countries build capacity through a variety of building objectives. The World Bank invests an diverse but complementary means, including estimated $720 million annually in support of technical assistance, studies, equipment, and client training. Over 90 percent is financed training. through investment projects, with the remainder provided through the World Bank Institute IEG's 2005 evaluation of capacity building in (WBI). This evaluation represents the first time Africa found that the World Bank's capacity that the World Bank's training investments have support was too fragmented, that capacity- been examined to determine the impact of building tools could be more effectively utilized, training on clients' capacity-building objectives and that quality assurance needed improvement. and to obtain insights into the factors driving Among the evaluation's recommendations for success or failure. It presents a rich set of ideas improving the Bank's support for capacity for improving the effectiveness of both WBI and building was reassessment of "what role training project-financed training, with the aim of should play in its capacity-building support, how enhancing the impact of the Bank's capacity- it should be provided, and what should be the building investments. Ajay Chhibber Director Independent Evaluation Group, World Bank x i Executive Summary T o achieve sustainable growth and poverty reduction, developing coun- tries need strong institutional capacity. The World Bank devotes sig- nificant resources to building stronger institutions and organizations in client countries. It helps build capacity through a variety of means, includ- ing technical assistance, studies, equipment, and training. This evaluation fo- cuses on the efficacy of one of the primary instruments for capacity building--training individuals so they are better able to contribute to their coun- try's development goals. It was found that most Bank-financed training Training is one of the primary means by which resulted in individual participant learning, but the Bank helps build the capacity of countries to improved the capacity of client institutions and reduce poverty. Moreover, it is often fundamen- organizations to achieve development objectives tal to the success of other investments. Without only about half the time.1 Where training did not trained road maintenance crews, highways succeed, it was because its design was flawed or crumble. Without trained teachers, school insufficient attention was paid to the organiza- buildings remain empty. Overall, in an estimated tional and institutional context in which training 60 percent of projects, training is either integral took place. The Bank could significantly improve to the achievement of the goals of one or more the impact of its training investments through (i) components or supportive of all project the development of training design guidance to components. enhance quality assurance and (ii) by making available resource staff with expertise in training The Bank supports training in two ways. First, design to Bank project managers. many investment projects include dedicated training components or training activities Training provided by the World Bank Institute embedded within project components. Second, (WBI) was found to be insufficiently targeted to the Bank has a separate unit devoted to capacity client needs, and inadequately embedded in building--the WBI, which aims to "build skills broader capacity-building strategies, to substan- among groups of individuals involved in tially impact development capacity. If the WBI is performing tasks, and also to strengthen the expected (as stated in its mandate) to play a organizations in which they work, and the socio- capacity-building role, its training processes political environment within which they need to be substantially reengineered. operate." Over the past decade, the Bank has financed approx- This evaluation examined the extent to which imately $720 million in training annually, over 90 Bank-financed training contributed to capacity percent through projects and the remainder through building. Most Bank-financed training was found the WBI. The importance of training to the to result in individual participant learning, but only achievement of development objectives goes about half resulted in substantial changes to work- well beyond these dollar terms, however. place behavior or enhanced development capacity. x i i i USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Project-based training was more successful than up support was provided to project trainees in WBI training in this regard. Where learning did half of the 29 training programs reviewed in field not result in changed workplace performance-- studies, and to WBI trainees in only two of the and thus did not have an impact on development eight cases reviewed. Finally, of the nearly half of capacity--this could be attributed to one of three survey respondents who stated that training had reasons: insufficient participant understanding of less than a substantial impact on key functions of how to apply learning in the workplace, their work, over a third said it was because inadequate incentives or resources for implemen- training lacked relevance to key work functions. tation of learning, or inadequate targeting of This last issue is indicative of inadequate target- learning to organizational needs. ing of training content. For example, inadequate incentives for Targeting of training content was found to be the implementation of learning were found in the most important design factor driving training Administrative Capacity Building project success. For training to be well targeted, organi- reviewed in Burkina Faso. Government human- zational and institutional capacity gaps need to resource managers were trained in performance- be correctly diagnosed, specific training needs evaluation techniques long before the decision must be assessed, and participants should be was made to implement performance-based selected in a strategic manner. Project-based evaluation in government. Making the leap from training reviewed in field missions performed individual learning to workplace performance better than WBI training in all of these targeting outcomes and, subsequently, to development processes. Projects were better targeted mainly capacity impact requires both good training design because they have more resources to conduct and an appropriate organizational and institutional capacity assessments and because they involve context in which to apply the learning from training. clients more fully in the design of interventions. The WBI does not generally mandate or finance Training success is predicated on adequate design. in-depth diagnosis of capacity gaps or assess- Good training design was found to involve three ment of training needs and does not consistently characteristics: consult with clients on training objectives and design. · Use of appropriate and professional pedagogic design, including opportunities to practice The organizational context for implementation of learned skills; knowledge and skills learned was a second · Provision of follow-up support to trainees to important determinant of successful capacity help them implement knowledge and skills building through training. Training builds develop- acquired; and ment capacity only when trainees have adequate · Targeting of training content, anchored in di- resources and incentives to implement learning agnosis of institutional and/or organizational ca- in the workplace. One-third of training partici- pacity gaps, formal assessment of participant pants surveyed stated that they lacked sufficient training needs, and strategic participant material resources to implement learning in the selection. workplace. Some trainees also lacked incentives to implement learning. Insufficient incentives Much of the Bank-financed training reviewed was were found to be particularly problematic in two found to have design flaws that affected results. contexts. First, in decentralized training While over 90 percent of survey respondents programs like in-service teacher training, the found their training to be interesting and lectur- central government's commitment to training ers to be of high quality, half stated that course goals did not necessarily translate into strong length was too short for the topics covered, and commitment among regional government that the course did not devote significant time to officials, training participants, or their managers. practical exercises and projects. Adequate follow- Second, in public sectors of countries with x i v E X E C U T I V E S U M M A RY weaker government capacity, low salary levels financed projects provide an opportunity for and lack of merit-based promotion systems effective use of training as part of an integrated reduced the incentives of staff to pursue their capacity-building strategy. The project model can own professional development. ensure that training is integrated into a compre- hensive, multiyear relationship with the target Even where resources or incentives were initially organization, financing a range of complemen- lacking, training succeeded as long as there was tary capacity-building interventions. However, strong client commitment to training goals and the lack of expert support for training design and adequate support was given to addressing related defined design standards makes it difficult for workplace capacity gaps. For example, the WBI's project teams to adequately supervise the design partnership with the Bangladesh central bank and implementation of training. This also resulted in the creation of a highly effective policy prevents quality assurance mechanisms from analysis unit. Successful outcomes could be being applied to training activities. attributed to the strong support of the governor of the central bank for the program, as well as the The Bank does not adequately monitor or evaluate integration of training with technical assistance training results. Most project-based and WBI and the financing of equipment and salaries training reviewed in field studies did not include through a related World Bank­financed project. sufficient monitoring and evaluation of training. Project Implementation Completion Reports Field studies revealed examples of successful seldom report on more than the numbers of World Bank­financed training activities provided persons trained and include little or no informa- by local training institutions, client governments, tion on training results in terms of the workplace international consultants and training providers, behavior of participants and impact on develop- as well as the WBI. In all cases, training succeeded ment capacity, even where training is fundamen- when its design was good and the organizational tal to the achievement of project goals. The WBI and institutional capacity context was adequately systematically monitors at the program level only addressed in conjunction with training. the number of participant training days and participant satisfaction, neither of which The WBI's training procedures and practices do not provides information on the impact of training sufficiently anchor training within comprehensive on capacity-building objectives. Hence, clients, capacity-building strategies, and are thus not project task teams, and WBI task managers generally conducive to building sustainable generally lack sufficient information to detect capacity. The WBI lacks systemic mechanisms for training weaknesses and improve training in-depth diagnosis of organizational capacity performance where necessary. gaps or formal needs assessment of training participants. It also lacks standardized Recommendations procedures for meaningful direct consultation The Bank can enhance the vital contribution of with clients on training needs and priorities. In training to client capacity building by ensuring most cases, the WBI does not directly provide that the training it supports follow-up support to facilitate workplace implementation of learning. It also does not · Is linked to the Bank's support for development systematically link its training programs to objectives in client countries, complementary capacity-building support · Is embedded within broader capacity-building provided by operations or other partners. strategies that provide complementary sup- port for the implementation of learning, and The quality of project-financed training is uneven due · Conforms with best practice in training design. to the lack of explicit design standards for all World Bank training activities, and lack of expert support The following three recommendations are for training activities embedded in projects. Bank- intended to lead to this outcome: x v USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT 1. The Bank needs to develop guidance and goals. If the WBI is to play a capacity-building quality criteria for the design and implemen- role in client countries, its training processes tation of training, to enable quality assurance should be substantially reengineered to en- and monitoring and evaluation of all its train- sure that training is likely to contribute to sus- ing support. This guidance should be applied tainable change. New WBI training processes to all training financed by the Bank, including should ensure that all training meets the fol- training that is directly provided by units such lowing criteria: as the WBI. Design guidance should include · Is based on a comprehensive capacity as- · Diagnosis and training-needs assessment sessment of the target organization(s)/in- requirements for training initiation; stitution(s)--done in cooperation with · Participant selection criteria; clients--identifying (i) clear and specific · Standards for the use of practical exercises capacity-building objectives; (ii) the human, and other active-learning techniques within institutional, and organizational capacity training; support that is necessary in order to achieve · Use of follow-up support; and these objectives; and (iii) measurable indi- · Provisions for monitoring and evaluation, in- cators of success; cluding specification of performance-change · Is undertaken after work is done with op- objectives and key monitorable indicators. erations and partners to identify and con- firm, in advance, what resources for all 2. The Bank could improve the quality and impact capacity-building support are required to of training by making available to its Regional achieve the objectives, including, where staff and borrowers, resource persons with needed, (i) multiyear training programs, technical expertise in the design, implemen- (ii) follow-up technical assistance, and (iii) tation, and monitoring and evaluation of organizational and institutional support training. measures, such as policy support and fi- nancing of implementation of learning; and 3. Management must clarify the WBI's mandate · Is subject to external quality review and on provision of training with capacity-building evaluation of results. x v i Management Response M anagement finds the Independent Evaluation Group (IEG) review1 to be an informative presentation on design and implementation is- sues related to training for capacity development. Management con- curs with a number of the conclusions and, for the most part, with the recommendations. However, Management does have issues concerning the analytic framework and the reach of some of the conclusions, relative to the evidence presented. Management Views on IEG's Analysis always achieve monitorable and measurable and Conclusions changes in workplace behaviors. Such an as- Management believes that the review contains sumption is unrealistic and inappropriate for much of value. It provides an informative the many activities that provide partners with summary of design and implementation issues information about policy options or emerg- related to training for capacity development. ing global issues and that are likely to con- Management concurs that improvements are tribute to development objectives in the longer needed in its support for partner countries' run.2 training activities with respect to pedagogical · A methodological foundation that judges out- design; linking training to follow-up and other comes and impact mostly on the basis of client support for change; and designing content and surveys with a low response rate and a few determining participation, when feasible, on the field visits3 (see box below). basis of diagnoses of organizational and other · Analysis and conclusions that seem to go be- capacity constraints. These points are relevant yond what the data can show (see box below). for most learning activities. Management also agrees with IEG's assessment that monitoring In Management's view, given these study limita- and evaluation (M&E) around partner training tions, some of the key findings and conclusions needs to improve; indeed, M&E needs to be in the IEG review must be interpreted cautiously. strengthened across the board in most develop- (A more detailed discussion is provided in ing countries. This implies a need for stronger appendix J.) support from the Bank and other donors to strengthen country M&E capacity--a major Recommendations element of the Bank's results agenda. Management is in agreement with the overall thrust of the first two recommendations (see the Points of Disagreement. Despite these many points attached Management Action Record), with one of agreement, Management has serious reserva- caveat: the recommendation regarding guidelines tions concerning the evaluative framework and is well-taken, but the general recommendation data limitations that affect a number of findings and the response to it require nuancing. The and conclusions in the review: guidance to staff must be scaled to the objective of the training, its size and cost, and the utility of the · The choice of evaluative framework. IEG's eval- recommended steps for the activity in question. uation framework assumes that training should Management takes note of IEG's recommenda- x v i i USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Data Concerns The evaluation bases its findings on two main sources of data: and 3 WBI activities were assessed for drawing conclusions a survey of participants in six countries and field studies in only on the impact of Bank-supported training.a four countries. Data from these sources provide a very limited basis for the review's sweeping conclusions (see appendix J): In several instances the report presents information and inter- · A survey of participants with a response rate as low as 11 per- pretations that go beyond what the data can support or that need cent (in Mexico). In three of the other five countries, re- further contextualization. Examples: sponse rates ranged from 21 to 30 percent. In Tunisia, a · The study's authors interviewed task team leaders (TTLs; 28 higher response rate was achieved but with a small number in 15 focus countries) about their collaboration with WBI with- of respondents. The 548 respondents combined in all six out first establishing that their responsibilities matched WBI's countries represent a raw response rate of 23 percent, with areas of support in the countries in which the TTLs worked.b 37 of those 548 then dropped for reasons related to their job · The evaluation's executive summary concludes unequivocally responsibilities. The danger of substantial bias with this re- that WBI training was found to be insufficiently targeted to sponse rate calls for caution in making broad generalizations client needs, but the evaluation (chapter 3) notes the absence to the population of interest (participants in training sup- of differences in several important respects between proj- ported by the Bank), especially given the wide variation in the ect-based and WBI training regarding targeting of client purposes and types of training reviewed. needs, and WBI's lead in some respects. · A sample size of only 29 projects and 8 WBI activities in four · The review reported country-level statistics without attention countries, among which, the review notes, only 18 projects tothelargemarginsoferrorattendantonverysmallsamplesizes. a. IEG notes that field studies were primarily used to examine the extent to which standard Bank training management practices and procedures conform to interna- tional best practice. Management notes that despite being few in number, field studies feature prominently in the discussion regarding training outcomes and are a pri- mary source for information on level of impact in chapter 2. b. IEG notes that the TTL interviews were used to determine the prevalence of WBI-TTL collaboration, including the extent to which TTLs rely on WBI as a resource for expertise on project training activities. Therefore, a random sample of TTLs was drawn for interviewing. Management notes that WBI-TTL potential for collaboration ex- ists for countries and areas in which WBI works. Thus, the TTL random sample should have been drawn from countries and areas in which WBI works to assess the prevalence, nature, or quality of WBI-TTL collaboration. Simply assessing prevalence, without taking into account whether collaboration is a distinct possibility, pro- vides little information. tion concerning WBI and notes that WBI has work nature of WBI's mandate is beyond the scope of under way on several of the topics mentioned, this study. As the Bank reviews, adopts, and including better aligning its country-focused and implements its long-term strategy, notably with long-term programmatic work with Bank regard to its knowledge role, it will also review the operations. The discussion about the precise mandate of WBI. x v i i i M A N A G E M E N T R E S P O N S E Management Action Record Major IEG recommendation Management response Develop guidance Agreed with clarifications The Bank should develop guidance and quality criteria for the de- Management will ensure the development and dissemination of sign and implementation of training, to enable the quality as- guidance for staff on training support. The guidance will cover surance, monitoring, and evaluation of all its training support. the components noted by IEG, as appropriate to the scope, ob- This guidance should be applied to all training financed by the jectives, and costs of the training. The guidance will indicate, with Bank, including that directly provided by units such as WBI. De- examples, how these components should be tailored to the pur- sign guidance should include: pose and size of the training activity being supported, to be · Diagnosis and training-needs assessment requirements for flexible and cost-effective and to avoid undue burden on part- training initiation; ner countries. Management will also seek the views of partner · Participant selection criteria; agencies to ensure that it is not unnecessarily imposing con- · Standards for use of practical exercises and other action-learn- straints on joint support. The guidance will be drafted by the end ing techniques within training; of FY08 and put into use at the start of FY09. This action will be · Use of follow-up support; and considered complete with the dissemination of the guidelines. · Provisions for monitoring and evaluation, including specifi- cation of performance change objectives and key moni- torable indicators. Provide expertise Agreed The Bank should make available to its Regional staff and bor- Management will ensure the compilation and maintenance of rowers resource persons with technical expertise in the design, a roster of resource persons with these skills and a system for implementation, and monitoring and evaluation of training. assigning these resource persons to assist task teams and part- ner countries. These will be in place for use by staff and part- ner countries by the end of FY08. This action will be considered complete with the posting of the roster and the announcement of the assignment mechanism. Clarify WBI's role and re-engineer training processes Not agreed in the context of this review Management should clarify the WBI's mandate on provision of The Bank has begun the process of developing, reviewing, and training with capacity-building goals. If the WBI is to play a ca- implementing an updated long-term strategy. WBI's business lines pacity-building role in client countries, its training processes include training, knowledge exchange, and technical assistance, should be substantially re-engineered to ensure that training will among others, but IEG considered just a specific type of train- be likely to contribute to sustainable change. New WBI training ing in its evaluation. Management will consider WBI's overall processes should ensure that all its training meets the follow- mandate, including its business lines, in the context of the ing criteria: knowledge component of the long-term strategy. · Be based on a comprehensive capacity assessment of the tar- get organization(s)/institution(s), done in cooperation with clients, and identifying (a) clear and specific capacity-build- ing objectives, (b) the human, institutional, and organizational capacity support that is necessary in order to achieve these objectives, and (c) measurable indicators of success. (Continues on the following page.) x i x USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Management Action Record (continued) Major IEG recommendation Management response · Identify and secure in advance the resources for all capac- ity-building support needed to achieve objectives, including, where needed, (a) multiyear training programs, (b) follow-up technical assistance, and (c) organizational and institutional support measures, such as policy support and financing of implementation of learning. · Be subject to external quality review and evaluation of re- sults. x x Chairperson's Summary: Committee on Development Effectiveness (CODE) O n September 12, 2007, the Committee on Development Effectiveness considered the report Using Training to Build Capacity for Devel- opment: An Evaluation of the World Bank's Project-Based and WBI Training, prepared by the Independent Evaluation Group (IEG), together with the Draft Management Response. Background. On October 10, 2005, CODE objectives; is embedded within broader capacity- discussed the report World Bank Institute: building strategies; and conforms with best Review of Progress FY03­05, prepared by WBI. practice in training design. The following three The Committee reviewed in March 2005 the IEG recommendations are intended to lead to this evaluation on World Bank Support for Capacity outcome: Building in Africa, which included WBI's role in capacity building. CODE also considered the · The Bank needs to develop guidance and qual- WBI report World Bank Institute: Developing ity criteria for the design and implementation Capacities in Countries--FY06 in Review on of training, to enable the quality assurance of October 4, 2006. all its training. · The Bank could improve the quality and impact Main findings and recommendations. The Bank oftrainingbymakingavailabletoitsRegionalstaff provides, on average, $720 million per year for and borrowers, resources and personnel with training to build client capacity. More than 90 technical expertise in design, implementation, percent is financed through projects and the and monitoring and evaluation of training. remainder is provided through WBI. The IEG · Management must clarify WBI's mandate on report finds that while most participants learned provision of training with capacity-building from training, only about half the time did goals. If the WBI is to play a capacity-building learning lead to substantial changes in workplace role, its training processes should be substan- behavior or enhanced development capacity. tially re-engineered so that it may better con- The most important factors driving training tribute to sustainable client capacity. success are good design and targeting of training content, a supportive organizational context for Draft Management response. Management concurs implementation of knowledge and skills learned, with a number of IEG conclusions and for the and strong client commitment to training goals. most part, with the recommendations. However, Management expresses serious reservations IEG recommends that the Bank enhance the concerning the evaluative framework, method- contribution of training to client capacity ological foundation, and data limitations that building, by ensuring that the training is linked affect a number of findings and conclusions of to the Bank's support for development the study. In addition, Management takes note of x x i USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT the recommendation concerning WBI and wider look at WBI and all its business lines, and responds that work is under way to address come back with a timeline of the review process. several issues mentioned in the IEG report, including better aligning its country-focus and The following main issues were raised during the long-term programmatic work with Bank meeting: operations. Management states that it will consider WBI's overall mandate, including its Scope of the report. Some speakers noted the business lines in a different and wider context, relevance of the report to the ongoing work the knowledge component of the Bank's long- related to the LTSE. They found that the IEG term strategy. report provided valuable inputs in strengthening training activities. Members expressed different Overall conclusions. The Committee welcomed views on the comparison between Bank-financed the opportunity to discuss the report which was training through investment projects and WBI very timely given the prominence of nonlending training, their contribution to capacity building, services and capacity building in the Long-Term and Management's concern on the methodology Strategy Exercise (LTSE) currently undertaken. of IEG evaluation. A member felt that project- Members commended IEG for an informative related training and WBI training were substan- and valuable evaluation that provided previously tially different, and noted that most training (90 unavailable statistics. While members acknowl- percent) was financed through projects. Another edged the methodological issues flagged by member agreed on the viability of comparing the Management, they viewed the IEG findings to two types of training within a uniform Bank's be consistent with their own knowledge and framework. Regarding the methodology, some experience. They also felt the findings should members would have preferred further discus- not be viewed as overly critical or negative. sions and agreement between IEG and Manage- Some thought the findings quite positive and in ment before conducting the evaluation. In this this regard queried whether there were vein, one member said that methodological benchmarks against which these results could differences may have been linked to differences be assessed. Going forward, members noted in training design. Relatedly, the Committee that the key issue was how to improve matters Chair invited IEG to think further about the and in this regard, they appreciated Manage- general challenge of measuring noneconomic ment's concurrence with several of IEG's effectiveness of development interventions substantive recommendations. With respect to including training. Some members found the the third recommendation to clarify WBI's role apparent differences between the IEG evaluation and re-engineer training processes, the Commit- and the Management Response to be less tee accepted Management's approach of substantive than they may first appear. One undertaking a comprehensive strategic review member thanked IEG for the new data on the of WBI in the wider context of the LTSE. The Bank's financed training activities. Committee Chair also suggested IEG focus on research of methodologies for assessing results Training and capacity building. A few members in nonlending service, which is also important concurred with IEG and Management that the for accountability. impact of training can be assessed within the broader framework of capacity building, Next steps. The Committee looked forward to highlighting the WBG's leading role in this area. discussing a comprehensive strategic review of Some speakers also noted that not all training WBI in due course. It suggested the review and learning activities are designed to lead to should look at the broader context of training institutional capacity building. In addition, they activities in the World Bank Group (WBG), and noted that capacity building involves a long-term the division of labor within different parts of the effort. Other speakers added that some of IEG's organization. Management committed to take a findings apply to training and capacity building x x i i C H A I R P E R S O N ' S S U M M A RY: C O M M I T T E E O N D E V E L O P M E N T E F F E C T I V E N E S S ( C O D E ) activities undertaken by the donor community in managed from Washington headquarters. IEG general. explained that the evaluation methodology did not specifically look at proximity to clients, Effectiveness of training. Although IEG found that although it found that clients are not always training only resulted in substantial workplace sufficiently involved in the diagnosis and design performance outcomes about half of the time, of training and TTLs lack adequate specialized some members found this finding very positive skills for designing training components of given the difficulty of trying to measure activities Bank projects. like training and the lack of benchmarks for judging success. They also noted that other Organizational context for training. Some members factors may have influenced the outcomes such felt that Management should consider consistency as weak public sector capacity, rigid adoption of of training with the Bank's development organizational transformation, and lack of objectives, increase accountability, and promote personal incentives. In addition, questions were adaptation. One of them mentioned that a raised about possible limitations in evaluating comprehensive project-based training approach training due to the lack of clearly defined undertaken by a donor agency has induced policy benchmarks, and about the relevance of training changes in recipient governments. Another and transformational effects. One member member regretted that the Bank's training service encouraged further development of monitoring might be associated with dissemination of overly and evaluation. Another member questioned the processed knowledge. In this regard, a few sustainability of project-financed training, partic- members stressed the need to focus on demand- ularly after the full disbursement of Bank's loans. driven aspects, and client-oriented focus of IEG noted the subjectivity of indicating whether training activities. One of them sought further the glass was half-empty or half-full. In this analysis on costs, including cost-sharing opportu- regard, IEG suggested that the more relevant nities. The fragmented efforts in providing question was whether the Bank could do more training and capacity building at the WBG level to assure that those attending training could were noted as an issue. Some members felt that use the information to help achieve their other development partners, including universi- organizational goals. IEG also noted that there ties and local institutions in developing countries, are no international benchmarks which could could provide training services as well. WBI noted be used. Management indicated that measur- its more than 200 partnerships with local institu- ing impact outcomes should be considered tions and that efforts were being taken to beyond training per se. It added that there was integrate its work with IFC and MIGA, such as the a need to move away from the individual recent case of investment climate work in India. product to a more programmatic approach, in a country or program context, to achieve some Bank processes and systems. Some members form of transformation. agreed with IEG on the need to develop guidance and quality criteria for the design and Training design. Some Executive Directors agreed implementation of training provided by the with IEG that design of training activities aimed Bank, including those provided by units such as at institutional capacity building requires an WBI. In this regard, one member asked Manage- institutional setting, assessment of capacity gaps ment to advance the timeline for drafting the and training needs, and consideration of partici- guidelines to mid-2008. She also asked Manage- pants' roles and capacities which can be more ment to prepare a proposal for streamlining the costly for the Bank and the WBI. One member Bank's training activities as a whole. Manage- asked whether the quality of training design and ment, while agreeing on the urgency for understanding of the institutional context was appropriate guidelines, also noted practical better when Task Team Leaders are based closer difficulties and the need for appropriate time to to clients (e.g. field presence) rather than prepare them. x x i i i USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT The role of WBI. While recognizing the impor- few speakers asked whether WBI's limitations tance of WBI's work, several speakers noted that were linked to resource constraints, although WBI needs to be aligned with the core objective one member commented on the use of trust of building capacity in client countries. In this funds by WBI. This member also observed the vein, they asked Management to review WBI's difficulty of measuring results of WBI training, core mandate and vision; business model, and noted the need for long-term evaluation. including programmatic approach; WBI role vis- IEG indicated that the evaluation did not focus à-vis other knowledge units (e.g. DEC, PREM, on division of labor among units delivering Regions); and integration with Bank Group's training. It only pointed out the current activities, including country assistance and fragmentation of work, and the need for better partnership strategies. There was also a sugges- integration of human, organizational, and tion for WBI to develop some flagship products. institutional capacity-building measurers. It Some members remarked that WBI is frequently also added that training results were linked to viewed as a capacity-building institution, but it resource availability, client involvement, and also provides other services such as dissemina- training expertise. Management indicated that tion of knowledge through the Shanghai Confer- WBI work with Regions is not covered in IEG's ence, which was regarded as very successful. A report. Jiayi Zou, Chairperson x x i v Chapter 1 Evaluation Highlights · Training is an integral element of the Bank's support for capacity build- ing in client countries. · The impact of training on develop- ment capacity depends not only on learning but also on participants' ability to implement learning in the workplace and on the relevance of that learning to development goals. · The evaluation assesses the effec- tiveness of WBI and project-based training in building capacity and highlights the factors that drive suc- cessful training. The Role of Training in Capacity Building The Nature of World Bank Training Support chieving the goals of sustainable growth and poverty reduction re- A quires developing countries to have strong national institutions. The World Bank has devoted significant resources to building the capacity of institutions and organizations in client countries. The Bank helps build ca- pacity through a variety of diverse but complementary means, including tech- nical assistance, studies, equipment, and training.1 This evaluation focuses on the efficacy of one of the Bank's primary means of capacity building--training-- and examines how training is used, alongside other Bank support, to achieve clients' capacity-building objectives. The Bank supports training in two ways. First, the achievement of the goals of one The World Bank provides many investment projects include dedicated or more components or supportive of an estimated $720 training components or training activities all project components.3 million per year, on embedded within one or more of a project's average, in support of components. Second, the Bank has a separate Training programs financed through client training. unit that organizes learning events--the World Bank loans and grants have a diverse Bank Institute (WBI), which aims to "build skills range of characteristics. They may be among groups of individuals involved in implemented by government ministries, local performing tasks, and also to strengthen the training or academic institutions, international organizations in which they work, and the training institutes and consultants, or other sociopolitical environment in which they donors or international organizations. They may operate."2 be used to build the capacities of senior policy makers and public sector officials at all levels, Over 90 percent of training is financed through teachers, health service workers, local govern- projects and the remainder is provided through ment officials, community leaders, business the WBI. Project-based training accounted for people, and farmers. Projects in all sectors finance approximately 6 percent of the total cost of training to some degree. investment projects during fiscal years 2002­06. Training may be even more important to the As shown in figure 1.1, training accounts for the achievement of project goals than these budget- highest share of project costs in the health, ary figures suggest because successful training is nutrition, and population sector (16 percent).4 sometimes essential for other project invest- Training expenditures in projects are used to ments to operate correctly and efficiently. For finance a multitude of different example, building schools will improve training objectives, with goals ranging Projects in the health, educational outcomes only if there are trained from developing specific skills to nutrition, and population teachers to work in them. In an estimated 60 creating behavioral or institutional sector involve the most percent of projects, training is either integral to change. For example, training in the training. 3 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Figure 1.1: Projects in the Health, Nutrition, and Population Sector Involve the Most Training cost 18 ojectrp 15 total 12 of 9 tagen cer 6 pe a as 3 gnin airT 0 Infrastructure Education Social Financial Environment Urban Economic Rural sector Health, development and private development policy and nutrition, and sector public sector population Source: Implementation Completion Report review used for estimating project-based client training (see appendix A). health, nutrition, and population sector builds The projects financed by both the International nursing skills, raises public health awareness, and Development Association and the International strengthens institutional capacity of health Bank for Reconstruction and Development insurance funds by providing training in general spend, on average, 6 percent of total project management as well as in technical functions of costs on training. Almost all investment project health insurance. In the rural sector, where types finance some degree of training.5 training accounts for, on average, 10 percent of Adaptable Program Loans have the highest share project costs, training is used to build skills in of training, relative to other types of investment firefighting, fish farming practices, and the use of loans (figure 1.2).6 new agricultural applications, among other things. The WBI, unlike projects, directly organizes and Figure 1.2: Adaptable Program Loans Have the Highest Shares of Training 12 10 tagen cost cer 8 pe a ojectrp 6 as gnin total 4 airT of 2 0 Technical Sector Investment Emergency Specific Learning and Adaptable Assistance and Maintenance Recovery Investment Innovation Program Loans Loans Loans Loans Loans Loans Source: Implementation Completion Report review used for estimating project-based client training (see appendix A). 4 THE ROLE OF TRAINING IN CAPACITY BUILDING implements the training programs that it · Be a scheduled activity that occurs WBI implements finances. Of these, 51 percent are done in in a setting other than where the approximately 700 partnership with other training institutes. The knowledge will be applied; and courses annually, rest are conducted directly by the WBI. The · Be conducted by lecturers, train- involving about 80,000 institute implements about 700 courses annually, ers, or facilitators. trainees. involving approximately 80,000 trainees a year, of whom 65 percent are men and 35 percent are The definition excludes training for the primary women. Approximately 40 percent of WBI purpose of personal growth, general education of trainees are government officials and 30 percent individuals to prepare them for the labor market, are academics (figure 1.3).7 on-the-job technical assistance, learning among peers, and events whose primary The WBI works in four main thematic areas: (i) purpose is networking, rather than This evaluation assessed governance, regulation, finance, and private learning.9 Also excluded from this the extent to which Bank- sector development; (ii) poverty reduction and evaluation is training of borrower financed training economic management; (iii) human develop- officials to enable them to implement contributes to country ment; and (iv) environment and sustainable Bank operations, such as procure- capacity to achieve development.8 Thirty-three percent of the WBI's ment training for staff in project development goals. training days are in courses related to management units. governance, regulation, finance, and private sector development activities; 31 percent to The evaluation focuses on client training poverty reduction and economic management conducted at the country level, and excludes Bank- activities; 19 percent to human development sponsored global training programs such as the activities; and 17 percent to environment and International Program for Development Evalua- sustainable development activities. tion Training and the WBI's global and regional programs, except when they are integrated into Scope of the Evaluation country-level capacity-building programs. This evaluation defines "client training" to include World Bank support for the training of persons in Evaluating the Training Results Chain developing countries so as to affect their workplace This evaluation assesses the extent to which behavior, for the purposes of supporting the Bank-financed training contributes to the achievement of broader development objectives. This definition encompasses a wide variety of Figure 1.3: WBI Trained Mainly Government Staff and training, such as training government officials to Academics better select and elaborate policies in specific fields, training health care workers to provide better Ministers 0.5% services in their communities, training private Private Parliamentarians sector 1.8% sector firms to help them build export capacity, as 15.8% well as training community group leaders, farmers' collectives, and nongovernmental organizations (NGOs). The ultimate goal of all training examined NGOs 9.7% in this evaluation was to build capacity in organiza- Government staff tions or institutions, in order to achieve develop- 42.0% Journalists ment objectives, rather than individual learning for 1.7% its own sake. Academia A training activity is defined in this evaluation as 28.9% having three characteristics. It must: Source: Data based on all WBI training programs in focus countries conducted between fiscal 2002 and fiscal 2006 in 13 countries. · Have specific learning objectives; 5 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT achievement of country-level development these capacity dimensions are interlinked, goals. Specifically, it seeks to answer the follow- evaluation of the impact of training on develop- ing questions: ment capacity must go beyond simply testing learning or asking training participants about · To what extent does Bank-financed training their level of satisfaction with the training. In have an impact on the capacity of target insti- order to evaluate how training contributes to tutions and organizations? client capacity to meet development objectives, · What are the factors that contribute to suc- it is necessary to understand how the resources cessful training? and incentives present in the organizational and · To what extent are these factors present in institutional context affect the implementation Bank-financed training? of learning and how human capacity building · To what extent do Bank systems and processes affects the organizational and institutional facilitate the financing and supervision of ef- context.12 fective training?10 This evaluation assesses the impact of WBI and This evaluation builds on IEG's 2005 evaluation project-based training on country development of capacity building in Africa, which looked at a capacity. To do so, it traces evaluated training broader range of capacity-building interventions. offerings along the length of the results chain As shown in that evaluation, client capacity to shown in figure 1.4, measuring the extent to achieve development objectives has interrelated which training activity inputs lead to learning dimensions, which often must be addressed in outputs, to changes in workplace behaviors tandem in order to achieve overall capacity- (outcomes), and, subsequently, to enhanced building goals:11 institutional capacity or improved organizational performance (impact on development capacity). · Human capacity: the skills and knowledge of individuals working in the organization; As is often the case in evaluation, the further one · Organizational resources: the staff, equipment, proceeds along the results chain, the more and finances an organization has at its dis- difficult it is to attribute change to specific inputs. posal; and For example, in a case where in-service teacher · Institutional incentives: the formal and informal training is initiated as part of a project to improve incentive structures, rules, and norms internal the quality of basic education, learning outputs to the organization and in its external can be measured through trainee tests and environment. assignments. Performance outcomes may, in turn, be measured (though with somewhat less These dimensions are highly interdependent. certainty) through classroom observation. But Trained individuals can be effective only to the the ultimate result--improved performance of extent that organizational resources and institu- high school students--is difficult to attribute to tional incentives enable them to apply what they teacher training, especially when other interven- learn. Training people to use a computer tions, such as extension of school hours or software package, for instance, will build client changes in the curriculum, may have also taken capacity only if trainees have access to and are place as part of the project. expected to use a computer in their Evaluation of training workplace. Training government The challenge of attribution was coupled in this must go beyond testing human-resource managers in the use evaluation with the challenge of a lack of monitor- learning or asking of performance-based employee ing and evaluation data for Bank-financed training. training participants evaluation methodologies will As discussed further in following chapters, the about their level of improve public sector capacity only if Bank rarely gathers data on the training outcomes satisfaction with the the civil service regulations mandate of project-based training programs. The WBI, for training. merit-based promotions. Because its part, does administer participant surveys to 6 THE ROLE OF TRAINING IN CAPACITY BUILDING Figure 1.4: A Results Chain for Training Input Output Outcome Impact Training Learning Workplace Enhanced Results behavior institutional chain change capacity or organizational performance · Learning · Learning is · Training program based relevant for curriculum on correct trainees' work based on assessment of · Trainees organizational the present understand how needs capacities of to apply learned · Human participants skills and capacity · Competent knowledge building Bridging trainers used at work correctly assumptions · Curriculum and · Trainees have identified as didactic methods adequate necessary for appropriate for resources and achievement of learning goals incentives to development apply learning objectives Capacity Training-needs diagnosis Capacity assessment Training-needs Clearandspecific diagnosis assessment learning goals Strategic Training-needs assessments Training Professional participant processesa curriculum design selection Attention to and pedagogy. Attention to organizational/ oragnizational/ institutional institutional capacity context capacity context Practical exercises and action learning techniques Follow-up suppport a. Further explanation about these processes can be found in appendix A. measure workplace outcomes of training, but on learning outputs, performance The Bank rarely gathers these are done at the country level rather than at outcomes, and impact on develop- data on the outcomes the program level, and thus provide no program- ment capacity in desk reviews of and impact of training. specific information. program and WBI documentation. This evidence was supplemented by a six- In order to overcome the shortcomings in country survey of training participants, and field existing data, this evaluation adopted a complex, reviews of 37 training programs in four countries. multitiered approach, which involved the nine All of these evidentiary sources were triangulated main evaluation tools listed in box 1.1, to and, taken together, provided a detailed and construct an accurate picture of workplace consistent picture of training results. performance outcomes and the impact on development capacity. Information was sought The six-country survey was designed primarily to 7 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Box 1.1: Evaluation Tools The findings of this evaluation are based on the following evi- ticipants in project-funded and WBI training were surveyed dentiary sources: in Azerbaijan, Bangladesh, Burkina Faso, Mexico, Nigeria, and Tunisia (appendix C). · Review of project training expenditures: estimation of the vol- · Review of project-based and WBI training in 13 desk-review ume of project-financed client training is based on a desk re- countries: analysis of documentation on project-based and view of 88 closed investment projects (appendix A). WBI training programs to determine the volume of project- · Field studies of 37 training programs: field missionsassessed based client training, the objectives of training and training 29 training components in 16 investment projects in results (appendixes D and G). Bangladesh, Burkina Faso, Mexico, and Tunisia, as well as · Task Team Leader (TTL) Survey: 43 randomly selected TTLs 8 WBI training programs in the three WBI focus countries: of ongoing Bank investment projects were surveyed (ap- Bangladesh, Burkina Faso, and Mexico.a In all, the missions pendix E). interviewed 105 training participants, 114 government officials, · WBI-Partner Training Institute Survey: 30 WBI training part- including policy makers and training administrators, 56 Bank ner institutes in 21 countries were surveyed on various as- staff, and 16 international agency representatives and other pects of their relationship with the WBI (appendix F). stakeholders.b · Benchmarking Survey: an analysis of selected features of six · Literature review: a review of relevant academic and prac- benchmarked training organizations.b titioner literature was commissioned for this evaluation (ap- · Interviews with 47 WBI Staff: extensive interviews were pendix B). carried out to map WBI operational procedures and practices · Six-country training participant survey: a total of 548 par- (appendix H). a. The WBI directs 70 percent of its training budget to 36 designated "focus countries." These countries are intended to be the beneficiaries of multiyear, country-specific programs linking activities to operations. b. Available on this evaluation's Web site at http://www.worldbank.org/ieg. provide evidence on training outcomes. Partici- though the small number of programs reviewed pants were asked to report on the extent to could not provide statistically significant quanti- which training had an impact on their work and tative information on overall success rates of to provide information on the characteristics of Bank-financed training. For each program, the the training programs in which they participated. evaluation mission reviewed extensive formal Respondents were not asked whether their and informal documentation and interviewed training had achieved the learning goals, because Bank staff, policy makers, training providers, and it could not be assumed that participants were trainees and their direct supervisors in order to aware of the specific learning goals of their obtain an accurate picture of all stages of the training. For similar reasons, they were not asked results chain. As presented in figure 1.4, these whether their training had contributed to reviews gathered three layers of evidence: direct development capacity. evidence of results, evidence on the validity of bridging assumptions, and evidence on the The field reviews were used to provide a richer prevalence of good training-management picture of the factors that determine training practices. results. In total, 37 training programs, including 29 project-based and 8 WBI programs, were The bridging assumptions depicted in figure 1.4 reviewed in four countries. The cases capture the characteristics that training needs at Evaluation field missions reviewed provided substantial qualita- each point of the results chain if it is to meet its interviewed policy tive information on Bank procedures objectives. Where the bridging assumptions did makers, trainees, and and practices on training, and how not hold, it was possible in some of the cases to their supervisors. they affect training outcomes, even infer that training was unsuccessful. For example, 8 THE ROLE OF TRAINING IN CAPACITY BUILDING the determination could be made that trainees mandate to assess the performance of Direct evidence of had not used skills acquired in computer training other training institutions, this survey training results was if it was found that they did not have computers was used to provide examples of good supplemented by in their workplace. In this case, the bridging training-management practice--to examining the bridging assumption that trainees have adequate illustrate how principles identified in assumptions needed to resources to implement learning would not hold. the literature review could be move from one stage of Similarly, in-service teacher training aimed at operationalized, rather than to deter- the results chain to improving literacy rates in a region could be mine the quality of Bank-financed another. assumed to have little effect if the primary cause training relative to that of other of low literacy rates was not teacher practice but training providers. critically low school-attendance rates. In this example, the bridging assumption that human Finally, surveys of Bank Task Team Leaders (TTLs) capacity building was necessary for the achieve- and WBI-partner training institutes, interviews ment of development objectives would be false. with WBI senior managers and task managers, and reviews of project and WBI documentation Training-management processes depicted in the were used to provide additional information on bottom tier of figure 1.4 were then evaluated to the nature and extent of Bank-financed training determine drivers of successful training. and on Bank training-management systems and Elements of good training management were processes. identified, based on a review of academic and practitioner literature (appendix B). The field This report presents the evaluation Training-management reviews and six-country survey were used to findings as follows: chapter 2 presents processes were assessed to determine the prevalence and quality of these evidence on the results of training; identify drivers of processes in Bank-financed training. This chapters 3 and 4 explore the training- successful training. information was matched with evidence of management processes that under- training results in order to identify the process pinned these results, and chapter 5 evaluates the factors that most contributed to training success extent to which the Bank's systems, resources, in building client capacity. and incentive structures are configured to support the design, implementation, and evalua- The evaluation of Bank training management was tion of successful training. The report then reinforced by a benchmarking survey of six concludes with a synopsis of the findings and international training institutions.13 As IEG has no offers recommendations in chapter 6. 9 Chapter 2 Evaluation Highlights · While client training programs gen- erally result in learning, only about half result in substantially improved workplace performance or devel- opment capacity. · Individual learning results are poor predictors of enhanced workplace performance. · Well-designed training can succeed in both high- and low-capacity contexts. Effectiveness of Training I n this evaluation, training was considered effective when it contributed to the achievement of the development objectives of a project or, in the case of WBI training, to the country assistance strategy. The effectiveness of train- ing was examined along three dimensions: · Learning outputs: Did training result in acquisi- · Medium: training partially achieved its objec- tion of new knowledge and skills relevant to the tives, but with significant shortcomings; and achievement of development objectives? · Poor: training did not achieve objectives, or · Workplace behavior outcomes: Are trainees ap- there were major shortcomings in the achieve- plying acquired skills in the workplace in a ment of objectives. manner likely to contribute to the achieve- ment of broader development goals? Ratings for project-based training were awarded · Impact on development capacity: Is there evi- separately for each training program, even where dence of improved institutions or enhanced or- multiple training programs were reviewed within ganizational performance as a result of training? a project. WBI training programs chosen for review were multicourse programs in thematic The six-country survey of training participants areas that the WBI had designated as high and the field studies of 29 project-based and 8 priority for that country. Ratings represent WBI training programs were used to construct a assessments of the program as a whole or of rich picture of results in all three of the above principal training offerings within the program. dimensions.1 Evidence on WBI training results was further supplemented by data from the Learning Outputs World Bank Institute Evaluation Group (WBIEG), No formal review of learning in project-based where relevant. Similar self-evaluation data on training has ever been completed by the Bank training results could not be presented for and it was impossible to determine from supervi- project-financed training because project sion documentation the extent to which the Implementation Completion Reports rarely learning goals of most training programs were report on training outcomes or impact. achieved. In the past, the WBI, for its part, did compile information on learning in some of its Each training program reviewed in the field training courses. The most recent WBIEG evalua- missions was rated on the basis of the following tion of learning results was completed in August criteria: 2005. Based on pre- and post-test data from 45 client skill-building courses in different subject · Good: training largely achieved its objectives, areas completed in fiscal 2002­03, WBIEG with minor or no shortcomings; determined that average learning gains were 10.4 13 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT percent, with average pretest scores of 42.6 In eight project-based and three WBI training percent correct answers and post-test scores of programs reviewed in the field mission, it was 53.0 percent. Twenty-two percent of WBI courses not possible to rate learning. In some training reviewed in this study did not result in any statis- offerings, the lack of specificity of learning tically significant learning gains.2 objectives made even qualitative measurement of achievement of objectives impossible. In The field studies of training programs suggested others, there was no available evidence of a more positive picture of overall learning gains learning achievements. The absence of even a than did the WBIEG study, although it basic level of results measurement in Bank- Training generally results must be noted that in most cases no financed training represents a major shortcom- in learning gains. testing had been done of participants ing in the Bank results framework. and, therefore, evaluators had to use less robust evidence of learning. Where test Workplace Performance Outcomes scores or other existing evaluative evidence of As noted in chapter 1, learning is a necessary but learning were not available, interviews with insufficient condition for training that participants and supervisors, or evidence of contributes to country development capacity. changed workplace performance in ways related Trainees must also be able to successfully apply to training received, was used to infer learning. acquired knowledge and skills in the workplace. On this more demanding criterion, results were Most training examined in the field studies substantially lower than for learning outputs. The resulted in demonstrable learning. The majority six-country survey of training participants of training programs for which it was possible to indicated that 50 percent of WBI respondents, evaluate learning had good results on this compared with 60 percent of project respon- dimension (see figure 2.1). The six-country dents, reported that training resulted in substan- survey of training participants also suggested tial positive changes to key functions of their that participant learning was high. Of work.3 A further 42 percent of WBI respondents The absence of even basic the 45 percent of survey respondents and 30 percent of project respondents reported results measurement in who said that training did not have a small changes to key work functions or changes much Bank-financed significant impact on key work to secondary work functions. When probed, training is a major functions, only 8 percent attributed however, less than one-sixth of these respon- shortcoming. this to failure to gain new knowledge. dents were able to give any examples of how Figure 2.1: Most Training Examined in Field Studies Resulted in Learning Gains Project WBI 3 3 8 5 3 15 Good Medium Poor Unrated Source: Based on field study findings. Note: The number shown within each pie-chart section indicates the number of training programs reviewed with that section's rating. 14 EFFECTIVENESS OF TRAINING their work had changed.4 Country statistics the field reviews that IEG rated were rated ranged from a low of 37 percent of WBI respon- "good" on workplace outcomes. Another four dents in Mexico who stated that training had a project-based programs were rated "medium," substantial positive impact on key work indicating that while the training did accomplish functions to a high of 90 percent of project some positive changes, there was significant respondents in Nigeria. Approximately 75 room for improvement. In the remaining five percent of project respondents in all countries, WBI programs reviewed and seven project- when probed with regard to how their work had based training programs, training outcomes changed, were able to give at least general could not be measured, either due to lack of examples. Figure 2.2 and box 2.1 present survey data or because it was too early in the program data and comments from interviewed partici- to reach definitive conclusions about training pants about the impact of training on their work. results. These findings on workplace outcomes of Impact on Development Capacity training correspond with data gathered by As discussed in chapter 1, building WBIEG in surveys of training participants sustainable client capacity to achieve Bank-financed training administered 8 to 24 months after course development goals generally requires results in substantial completion. WBIEG surveys of course partici- a multipronged approach that workplace performance pants completed in fiscal 2001­05 found that, on addresses resource, incentive, and outcomes about half of average, 55 percent of participants, when asked human capacity constraints in a strate- the time. to rate on a scale of 1­7 (1 being the minimum) gic manner. As a result, it is difficult to the extent to which they use WBI-acquired skills disentangle the impact of training from the and knowledge, rated WBI courses at 5 or impact of other interventions that enable the higher.5 achievement of capacity-building goals. The field studies made determinations of impact through Fourteen out of 22 project-based training in-depth interviews with participants, their programs and one out of three WBI programs in supervisors, government policy makers, and Figure 2.2: About Half of Trainees Reported Substantial Positive Changes to Work Performance WBI Project 8% 9% 7% 15% 50% 60% 23% 27% Substantial positive changes to key work functions Small positive changes to key work functions Positive changes to secondary functions Little or no change to work Source: IEG six-country survey of training participants. 15 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Box 2.1: How Work Changed Following Training The following particpant statements are from surveys of course "We improved the assignment delivery time by up to 15 percent. participants: We are better organized in the business and offer better service "My approach to work changed after the course. I better un- to the client. We are able to detect where we are failing." (National derstand the various procedures to be followed to set up a part- Competitiveness in Mexico Industry and Government Dialogue) nership."(PublicPrivatePartnershipinInfrastructure,BurkinaFaso) "I noted an improvement in my diagnosis methods following this "I am able to better organize and mobilize human resources of training course and techniques which are based on the scientific my service around specific objectives; I can better manage the med- method. [I acquired] new knowledge [allowing] easier identifica- ical team." (Health Workers Policies and Practices, Burkina Faso) tion of certain species." (Comprehensive Fight Against Crop Dis- "I developed a better general view of corruption and how the eases, Tunisia) federal government fights to eradicate it. It gave me the tools nec- "The knowledge from the training course helped me to write essary to solve issues of corruption and the culture of filing com- new business plan for our municipal government. We introduced plaints on behalf of the citizens. I perform audits for the federal a new system for the preparation and registration of documents public services with the training." (WBI Open and Participatory Gov- and we use computers more than before"(Community-to- ernment, Mexico) Community Learning Visits, Azerbaijan) Source: IEG six-country survey of training participants. Bank staff in order to better understand how the training of these institutions' staffs in training affected capacity to achieve develop- banking functions. In Tunisia, training of ment objectives. On this basis, it was possible to exporters enhanced their knowledge of export make evaluative assessments in 18 project-based markets and thereby contributed to an increase and 3 WBI training programs. In the remaining in Tunisian exports. training programs, the difficulty of attributing impact to training or the lack of evidence on For five WBI programs there was not enough impact made such assessments impossible. information available to rate their impacts. This is related to the lack of information gathered by In 10 project-based training programs, training the WBI on the capacity of participants' organi- could be linked to significant impact leading to zations or institutions, either prior to or follow- a "good" rating. In another three cases, training ing training programs (discussed in chapters 3 offerings led to some positive impacts, resulting and 4). In only one of the WBI training programs in a "medium" rating, and in five cases, impact reviewed was it possible to establish enhanced was rated as "poor." Examples of project-based development capacity. In Bangladesh, the WBI's training with good impact ratings include the partnership with the central bank led to the Procurement Reform Project in Bangladesh, creation of a Policy Analysis Unit, which is now which adequately prepared a large number of producing high-quality analytical material. This public sector officials to use new procurement program is linked to the Enterprise Growth and procedures, and helped build support for the Bank Modernization project in Bangladesh and transition to a more transparent procurement benefits from both project and other donor regime. In Burkina Faso, training for financing. community groups empowered them to plan, implement, and evaluate their own develop- Two WBI offerings and five project-based training ment plans as part of the Rural Development programs received poor ratings for impact. In program. In Mexico, 125 savings and credit one case, the Burkina Faso Basic Education institutions have been, or are about to become, project, a midterm review of the project legally accredited, a result attributable in part to determined that teaching quality had actually 16 EFFECTIVENESS OF TRAINING dropped during the project, a fact that was attrib- provided by governments, the WBI, The nature of the training uted to the decision to replace the standard, two- local training institutions, or interna- provider was not found year, preservice teacher-training course with a tional training providers. No evidence to be strongly correlated one-year preservice course and supplementary was found in the field studies to with training success. in-service teacher training. In-service training suggest that one type of training was not a good substitute for preservice training provider consistently achieved better training because courses were poorly targeted to teacher results. Of the 15 training programs reviewed in needs, and teacher attendance was low due to field studies with satisfactory outcomes, 7 were resource and incentive constraints. implemented through the government (in one case with the support of a local training institute), Attributing Training Results 4 by NGOs, 3 by international consultants (in The above review of training efficacy cooperation with local partners or the WBI), and demonstrates that, while most project-based and 1 through an international organization. WBI training reviewed resulted in learning, training only resulted in substantial workplace Evidence on the impact of country capacity on performance outcomes about half of the time. In training was inconclusive. While the six-country other words, individual learning is a poor predic- survey suggested that training was somewhat tor of impact. While most training participants more successful in countries with lower Country may gain new skills or knowledge in Bank- Policy and Institutional Assessment ratings than in financed training courses, many cannot or do not countries with higher ratings, the field studies use this learning after returning to the found a somewhat higher proportion of successful workplace. training programs in the higher-capacity countries of Mexico and Tunisia than in Good training outcomes Transfer of learning outcomes to the workplace is the lower-capacity countries of were found in both affected by both the design of the training itself Bangladesh and Burkina Faso. higher-capacity and and by the context in which it takes place. The However, examples of good training lower-capacity countries. following chapters analyze the reasons for insuffi- results were found in field studies in cient transfer of learning to the workplace and Bangladesh and Burkina Faso when that training comment on the key drivers of successful had been well designed, adequately addressing training. Chapter 3 examines how training design capacity constraints likely to affect workplace affects the likelihood of learning transfer. Chapter implementation of learning. In the case of 4 explores how the training context affects the Bangladesh's Export Diversification Project (table contribution of training to the achievement of 2.1), examples of good and poor training designs organizational and development goals. leading to corresponding good and poor outcomes were found in the same project. In this Excluded from this analysis of factors contribut- case, varying training success, while partially attrib- ing to training success are two variables that were utable to the greater complexity of the training not found to be strongly correlated with training goals of one of the programs, was also largely success: the nature of the training provider and owing to differences in training design and client the capacity level of the client. As noted in commitment. Both of these factors will be chapter 1, Bank-financed training may be discussed in subsequent chapters. 17 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Table 2.1: Same Project, Different Training Outcomes Two training programs in the Export Diversification Project were reviewed in the Bangladesh Field Study. The first program, involving training for the Bangladesh Tariff Commission, received a poor rating on performance outcomes, and the other, involving computer training for the National Board of Revenue, received a good rating. The differences in the ratings can be partially attributed to the relatively simple nature of the training in the latter project. However, there were also very significant differences in the quality of training design, which were found to have a signifi- cant impact on training results. These differences were related to targeting, client commitment, and follow-up support. Design element Bangladesh Tariff Commission National Board of Revenue Training objectives To strengthen capacity in the Bangladesh Tariff To train National Board of Revenue personnel in the Commission for trade data and policy analysis, and use of customs software, as part of the larger proj- for international trade cooperation; to provide a ect objective to computerize customs transactions. technical basis to foster trade liberalization. Simple but adequate performance objectives and No performance objectives or indicators of key indicators were specified. training success specified. Training provider International consulting firm. United Nations Conference on Trade and Development. Diagnosis Poor--Significant institutional weaknesses were Good--Training was based on adequate diagnosis not diagnosed and inadequate assessment of capacity of the need for computerization and associated needs created a course program that insufficiently capacity gaps, such as the need for ongoing techni- targeted job needs. cal support for software use. Training-needs assessment Poor--The training provider did not sufficiently Good--The starting experience levels of partici- consult the Bangladesh Tariff Commission. pants were known in advance. Participant selection Medium--Training was open to persons outside the Good--Very wide cross-section of National Board Bangladesh Tariff Commission, and thus the training of Revenue personnel were trained to build support was less focused on the commission's needs. for new systems. Client commitment Poor--Bangladesh Tariff Commission leadership Good--National Board of Revenue staff coordi- changed frequently and was not committed to reform. nated the training. Pedagogical design Poor--Training was very theory based. Participants Good--Strong use of practical exercises and partic- had difficulty relating theory to their work. ipatory techniques. Attention to capacity context Poor--Uncertainty of length of tenure of the Good--United Nations consultants built up in- staff. Necessary institutional reforms were not house capacity to continue training and support for executed before implementation. the customs software after project's end. Follow-up support for Poor--Insufficient technical assistance to aid Good--United Nations consultants provided in- trainees implementation of learning. house technical support for two years, and are con- tracted to give off-site support for software for as long as it is used. Source: Interviews and Bank documentation on the Bangladesh Export Diversification Project. 18 Chapter 3 Evaluation Highlights · Good pedagogy, support for imple- mentation of learning in the work- place, and adequate targeting all drive successful training. · Bank training is often too short to meet capacity-building goals, un- derutilizes practical exercises, and lacks follow-up. · Targeting of training is the most im- portant design factor for successful training. · WBI and project-based training perform equally well on pedagog- ical design of training, but WBI training performs much more poorly on targeting of training and sup- port for workplace implementation of learning. What Training Works: Training Design E ffective training design for capacity building takes into account not only how best to achieve learning goals, but also how best to ensure that trainees can apply learning in the workplace and that training con- tent responds to organizational or institutional needs. As such, good training design is important for success along the entire length of the training results chain. Table 3.1 presents the three design factors and organizational and/or institutional needs is seven associated training procedures needed for necessary if performance change is to have an successful training. Each design factor is associated impact on development capacity. with a link in the results chain. Good pedagogy is needed for successful learning outputs. Adequate The field reviews and six-country survey of partic- support for transfer of learning to the workplace ipants assessed these three factors, and seven is needed for individual workplace performance associated training-management processes, outcomes. Adequate targeting of training to according to the following parameters: Table 3.1: Design Factors for Successful Training Factors for training efficacy Associated training processes Good pedagogy · Professional curriculum design matched to training needs · Didactic methods are varied and appropriate for participant-level and training goals Adequate support for transfer of · In-class preparation to facilitate implementation of learning in the workplace through action learning to workplace learning and practical exercises · On-the-job follow-up support Adequate targeting of training to · Organizational capacity diagnosis (What capacity gaps exist? Is training an appropriate means of organizational needs addressing these gaps?) · Training-needs assessment (What is the present capacity of those to be trained? What training is needed to address existing capacity gaps?) · Strategic participant selection (Who should be trained to meet organizational goals?) Source: IEG evaluation literature review. 21 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Quality: How well did Bank-financed training Poor: training process was not implemented or perform on the three factors for training efficacy was implemented with major shortcomings. and their associated training processes? The relative importance of various processes to Importance: How important were these training success was assessed in the six-country factors/processes in the achievement of training survey through two methods.2 First, a "drivers- results? of-training-success" analysis was completed using the six-country survey data (see box 3.1).3 To assess process quality, respondents to the six- The drivers analysis found that the most country survey were asked the extent to which important factor for training success was the these training management processes were organizational context in which training was present in the courses they attended.1 In done (discussed in chapter 4). Design factors addition, the field studies rated each WBI and associated with targeting of training were found project-based training program on the extent to to be of substantial secondary importance. which it successfully implemented these pro- cesses. Training management processes were Second, participants who had stated that training rated using the following scale: had less than a substantial impact on key aspects of their work (45 percent of respondents) were Good: trainingprocesswasadequatelyimplemented, asked the reasons for this lesser impact. with no more than minor shortcomings; Inadequate targeting of training was cited as the most important cause. Fifty-seven percent of Medium: training process was partially imple- WBI respondents to this question, and 44 mented, with significant shortcomings; percent of project-based training respondents, Box 3.1: Drivers of Training Success To determine the importance of training processes and con- then ranked in importance on the basis of a discriminate textual factors to training success, respondents were asked function analysis that linked the respondent's rating of train- to agree or disagree with specific statements. A principal-com- ing impact on his or her work to the respondent's ratings on ponents analysisa was then used to group processes and each of the course attributes. The stronger the predictive contextual factors based on participants' responses and to power of course attributes, the higher the importance of the identify key drivers of successful training. These drivers were driver overall. Driver Correlated survey statements Course targeting The level of the course was appropriate for a person with my knowledge and experience. The course content specifically addressed my country's circumstances. The course content addressed issues that are important to my work. Course quality The lecturers were of good quality. The course content was interesting. The course was in a language I am fluent in. Participant input I was given the opportunity to provide feedback on my satisfaction with the course. The course organizers asked me to share with them my needs or objectives in the course, either before the course or at its start. Participant mix/interaction Course participants had about equal levels of knowledge/experience coming in to the course. I learned from the experience of other participants in the course. Source: IEG six-country survey of training participants. a. In principal-components analysis, individual survey questions are grouped together based on respondent answer patterns. Through correlation analysis, it groups the variables that respondents tend to rate similarly. 22 WHAT TRAINING WORKS: TRAINING DESIGN said that training was not relevant to key aspects A comprehensive, in-depth assess- The most important of their work.4 Inadequate resources or ment of pedagogical methods was drivers of training success incentives for implementation of training beyond the scope of this evaluation, were targeting of training (organizational context) was the second most because it would have required de- and trainees' workplace important reason cited (table 4.1). In sum, these tailed investigation of course attrib- context. two methods both identified targeting of training utes by subject-area experts. Instead, content and the organizational context for interviews with participants and training implementation of learning as the two primary providers in the field studies and the six-country determinants of training success. survey responses about course interest and lecturer quality were used to indicate pedagogi- Pedagogy cal design quality. Box 3.2 illustrates a range of Good pedagogy involves the matching of curricu- good pedagogical techniques identified in the lum and learning methods to training goals and field studies. participant characteristics. There are many questions that need to be addressed by instruc- As noted in chapter 2, Bank-financed Participants interviewed tional designers: Should training content be training has been successful, overall, in in both the six-country delivered all at one time or spread out over weeks meeting learning objectives, which survey and the field or months? To what extent are participatory suggests that pedagogy was adequate. reviews expressed methods or practical exercises necessary in order Indeed, participants interviewed both satisfaction with teaching to achieve sustainable learning? Are small class in the six-country survey and the field standards. sizes necessary to facilitate learning? Can course reviews expressed satisfaction with content be covered using electronic or distance pedagogical design and teaching standards. learning, or is face-to-face, classroom-based However, two major shortcomings were found training necessary? How should training content repeatedly in the field studies and also cited by be sequenced to maximize understanding? How participants in the six-country survey--inadequate much time should be devoted to specific topics? course length and insufficient use of practical Box 3.2: Diverse Pedagogical Methods Support Learning The WBI's partner in Mexico is the Instituto Tecnológico y de Es- anced theory, application, and practice. A three-day workshop on tudios Superiores de Monterrey, which works primarily through producing for export was followed by a one-and-a-half-day study a Web-based, interactive distance-learning platform, with par- tour to a farm to assess producers' processes and conditions. The ticipants signing onto the World Wide Web to pursue the course. instruction ended with another three-day workshop on other ex- In designing a course, the Mexican institute assembles a teach- port issues such as marketing and contracting. ing team that consists of professors/lecturers and tutors, and a The Savings and Credit Sector Strengthening project in Mex- design team that includes an instructional designer, graphic de- ico used highly practice-oriented pedagogical techniques to signer, media producer, and Web editor, with systems develop- train over 5,000 representatives of savings and credit institu- ers, technical support staff, and multimedia staff available as tions. Presentations by mentors were combined with group ac- needed. All content and exercises are practical and action ori- tivities and practical exercises about real-life cases. Training ented. The content is structured around problems that have to was structured in an initial series of courses that were 10 days be solved in the workplace and feasible solutions based on long, followed by a series of shorter courses for updates and more good practice examples from around the world. specialized topics. This multifaceted approach, with emphasis on The Agricultural Services Support project in Tunisia used a blend real-life examples, helped participants bridge the gap between of pedagogical techniques to help agricultural producers increase the course and the jobs where they had to apply what they the quality of their products and to increase exports. Pedagogy bal- learned. Source: Data based on field study findings. 23 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT exercises in courses (table 3.2). Half of all survey suggest that insufficient consideration is given to respondents agreed that their course covered too matching course length to content and to many topics for the amount of time capacity-building goals. About half of respondents allotted. Similarly, about half of respon- indicated that training dents reported that their courses had Support for Transferring Learning to the did not devote significant not devoted significant time to practical Workplace time to practical exercises and projects, an issue that is Good training design seeks to achieve learning exercises. further discussed in the following and to support the trainee in applying new section on support for transfer of knowledge and skills in the workplace. The two learning to the workplace. training-design practices most commonly associ- ated in the literature with supporting transfer of Almost all the training courses examined in the learning to the workplace are field reviews were less than five days long, regard- less of the course objectives. A review of all WBI · The use of practical learning techniques such fiscal 2006 courses confirms that the median as exercises, action plans, and projects where length of all WBI courses was three days, with 80 the trainee has the opportunity to explore percent of courses five days or less. The average how learned knowledge and skills relate to or length of project-based training examined in the could be implemented in his or her workplace field reviews and of WBI training is significantly environment;5 and shorter than in many of the benchmarked institu- · Follow-up support through on-the-job technical tions. For example, the average course length is assistance or access to off-site expert coaching two months at the Japan International Coopera- or advice.6 tion Agency (JICA), 25 days at the Israeli Center for International Cooperation (MASHAV), and two The use of practical learning techniques is weeks at the International Training Centre of the recognized in the literature as fundamental to the International Labour Organization (ITCILO). The sustainable acquisition of skills through training.7 two- to five-day training model, while Research on adult learning indicates that tasks Two to five days of appropriate for certain limited training learned through practice are more likely to be training appears goals, is inadequate for achieving the remembered, particularly where more complex inadequate to achieve more ambitious capacity-building goals skills are involved.8 Despite this well-known many of the Bank's that Bank-financed training often finding, only 43 percent of project respondents capacity-building goals. targets. The results of this evaluation and 53 percent of WBI respondents in the six- country survey indicated that significant time during the training was devoted to practical Table 3.2: Courses Are Interesting but Too Short and exercises or projects (table 3.2). Similarly, in Lack Hands-On Work several of the training courses reviewed in field studies, participants stated that there were Respondents agreeing insufficient opportunities to practice skills. Project training WBI training Diagnosis (%) (%) The use of action learning methods by the WBI The course was interesting. 97 93 may even be decreasing. A 2007 WBIEG evalua- The lecturers were of good quality. 91 91 tion found that use of a key form of practical The course covered the right amount of learning--the preparation of participant action topics for the amount of time allotted. 48 53 plans for implementation upon return to the The course devoted significant time to workplace--had dropped from 47 percent in practical exercises or projects. 43 53 fiscal 2004 to 39 percent in fiscal 2005,9 even Source: IEG six-country survey of training participants. though previous WBIEG evaluations had Note: The differences between WBI and project ratings are significant at the 95 percent concluded that such action plans improved confidence interval for "the course was interesting" and "the course devoted significant time to practical exercises." learning rates. 24 WHAT TRAINING WORKS: TRAINING DESIGN The Bank's inconsistent use of action plans and request online expert sessions in the subject area. other practical learning techniques contrasts In contrast, follow-up is only sometimes provided with the practices of some benchmarked institu- in Bank-financed training programs. tions. For example, MASHAV and Germany's Internationale Weiterbildung und Entwicklung The six-country survey of training participants (InWEnt) use the design of action plans for found that most participants had not had any form implementation upon return to the workplace as of follow-up contact with instructors or on-the-job the cornerstone of almost all their courses. technical assistance. Figure 3.1 presents data on InWEnt often conducts follow-up training follow-up support from the field sessions with training participants who reviews. Fourteen of the 26 project- Some of the benchmarked developed action plans in-course, to provide based training programs and two of the institutions use action expert support and guidance on the implemen- six WBI training offerings in the field plans to support tation of the action plans. JICA has recently reviews that IEG rated included workplace adopted policies to increase the use of action adequate follow-up support. Where implementation of plans and practical learning in its courses. JICA follow-up support for WBI training was training in almost all requires target organizations with which it has found in field reviews, it was not their courses. multiyear partnership agreements to commit to financed by the WBI, but was made implementing action plans developed in-course possible through partnerships with other donors, by training participants. partner training institutes, or related Bank operations. Follow-up support for trainees is recognized in the literature as a second important method for Targeting of Training assisting trainees in applying learning. Research Well-targeted training addresses capacity needs has indicated that learning, particularly of skills, that affect the achievement of development is far less likely to be retained and implemented objectives. As noted in table 3.1, three processes if it is not reinforced by follow-up support once are used to target training: trainees return to the workplace.10 Where follow- up support is not given, short-term learning · Diagnosis of capacity gaps involves assessment gains often do not translate into sustainable of existing organizational, institutional, and behavioral change, due to participant un- human capacity gaps and of the ap- pro- certainty about how to apply the learning or lack priateness of training as a means to address of positive reinforcement in the workplace for these gaps. learning application.11 As one training expert · Training-needs assessment roots training design (Taylor 2001) notes, "Training should not be seen in an understanding of the present capacities as an end, but as a means to achieving the organi- of the individuals to be trained and the specific zational objective. The changing of behavioral knowledge and skills that participants must patterns is a long-term undertaking which, to be acquire in order to meet development objec- successful, needs to be continually monitored tives. and reinforced . . . It is necessary to look beyond · Strategic participant selection is necessary to en- training by establishing a coaching and mentor- sure that the participants trained are those ing period to follow the training."12 whose capacities must be built up in order to meet development objectives. Among the benchmarked institutions, Motorola has the most comprehensive follow-up program. As noted earlier, good targeting of Diagnosis of capacity It provides coaches to help employees adapt and training was determined to be one of gaps, training-needs implement the training to the job, and sets up a the most important drivers of training assessment and strategic Web page to facilitate dialogue among course success. Project-based training participant selection are alumni. InWEnt provides participants with access reviewed in field missions performed all needed for accurate to an Internet group, through which they can better than WBI offerings on all targeting of training. 25 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Figure 3.1: Project-Based Training Provided Adequate Follow-up Support More Frequently Than WBI Project In the Tunisia Export Development Project I, the government created a market for follow-up technical 3 assistance. The project financed training of export 8 consultants, which private sector firms could hire to complement their own training. The government gave these consultants a one-time tax exemption and, upon 4 14 the submission of an acceptable business plan, the public-private sector steering committee gave export firms a one-time, 50-percent subsidy for using these consultants. In the Mexico Savings and Credit Sector WBI Strengthening Project, training that was focused on general topics was combined with on-the-job techni- cal assistance. Consultants worked directly with the 2 2 staff of savings and credit institutions to customize and apply the knowledge obtained in training courses to institutions' particular circumstances. 2 2 Good Medium Poor Unrated Source: IEG field study findings. Note: The number shown within each pie-chart section indicates the number of training programs reviewed with that section's rating. Where training did not targeting criteria, while the six- reported that the course did not address issues have a significant country survey did not find any signif- important to their work (see table 3.3). impact, poor targeting of icant difference between WBI and training was at fault project-based training, with regard to Diagnosis approximately half of the targeting. Good diagnosis of capacity needs is the first step time. in targeting training.13 Diagnosis involves identi- The six-country survey of training fication of human, institutional, and organiza- participants indicates that where training did not tional capacity gaps that must be addressed in have a substantial impact on workplace perform- order to achieve development objectives. ance, poor targeting was at fault in approximately Diagnostic exercises should also consider what half of the instances. Fifty-seven percent of WBI means are most appropriate for addressing these respondents and 44 percent of project respon- gaps: some human capacity gaps are better dents who reported that training had less than a addressed through on-the-job technical significant impact on key functions of their work assistance or the provision of independent attributed this to lack of relevance of the training learning materials, as shown in box 3.3. content to key work functions. In a separate set of Moreover, diagnosis must also determine questions, almost a quarter of all survey respon- whether there are critical contextual conditions, dents reported that training did not address their such as resource or incentive constraints, that country's circumstances, and an equal number are likely to block successful implementation of 26 WHAT TRAINING WORKS: TRAINING DESIGN learning. For example, at Motorola University, Table 3.3: Targeting of Training new course topics are initiated by the company's production managers, on the basis of produc- tion problems in their units. Instructional design- Respondents ers are then commissioned to determine agreeing whether weak performance is attributable to Project WBI knowledge or skills gaps--which are best training training Diagnosis (%) (%) addressed through training--or to other Course content did not specifically address my constraints, such as incentives, resources, or country's circumstances. 22 21 production processes. Course content did not address issues Diagnosis was found to be good in all project- important to my work. 25 24 based training reviewed in Mexico and Tunisia, Source: Data based on IEG six-country survey of training participants. but in only half of the project-based training programs reviewed in Bangladesh and Burkina relevant country team members, but dialogue Faso. This wide variation in performance was with Bank staff alone proved insufficient to found to be associated with varying client capaci- identify capacity needs and ensure client buy-in ties and commitment levels in the field-review to training goals. countries. Mexico and Tunisia had higher overall client ownership of project goals, and higher When adequate diagnosis was done in reviewed levels of client involvement in diagnosis and WBI training, it involved extensive client consul- design of training.14 Examples of good diagnosis tations and was financed by related Bank projects were also found in Bangladesh and Burkina Faso, rather than by the WBI. While no WBI data were where there was strong client commitment to available on the amount of funding allocated to training goals and involvement in training diagnostic exercises and for specific WBI training management, and where clients received programs, interviews with WBI task managers support from external training experts. and senior management indicated that the WBI does not generally provide funding for diagnosis Only two out of eight WBI programs reviewed in of organizational/institutional capacity-needs or the field studies had adequate diagnosis. In WBI training-needs assessment, even in the case of programs reviewed that did not adequately multiyear training programs. As a result, in-depth diagnose capacity problems, this was associated preparatory work to better target with limited or no dialogue with target organiza- training programs generally necessi- Good diagnosis is tions about training goals. Some WBI training tates outside funding. Indeed, in all associated with strong programs were based on consultations with the cases found in field studies where client involvement. Box 3.3: When Not to Train Training is not always the best way to address human capacity · Tasks are easy to learn, gaps. Other types of capacity-building interventions may be · Learners have a strong background in the topic and new more appropriate when: knowledge and skills can be self-acquired, and · Solving a problem requires applying a technique rather than · Learners are not used to or comfortable in classroom envi- building knowledge or skills, ronments. · The number of persons requiring knowledge and skills is limited, Source: IEG evaluation literature review. 27 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT WBI training was based on adequate diagnosis, needs assessments of these NGOs. As a result, training-needs assessment, and strategic partici- an incorrect assumption was made that more pant selection, WBI training was linked to Bank experienced NGOs had sufficient capacity to projects. train less experienced NGOs. Only later, after the bigger NGOs proved incapable of playing Training-Needs Assessment this role, was a training-needs assessment Training-needs assessment is used to determine done, and a more suitable training program what specific knowledge or skills must be designed. Similarly, the Second Basic Education learned to achieve capacity goals.15 A training- Development project in Mexico correctly needs assessment can be done as part of a diagnosed a need for parental training to diagnostic exercise or separately from it. For support preschooler learning. However, lack of example, for InWEnt's multiyear training a training-needs assessment resulted in projects in the field of education, one training being at an inappropriate level for the Capacity diagnosis education expert is contracted to 20 percent of parents who were functionally without training-needs diagnose what capacity gaps the illiterate and had difficulty understanding the assessment can lead to program should address (for texts. incorrect assumptions example, improving math instruc- about training needs. tion) and a second expert in the Adequate training-needs assessment was done in specific target field, is subsequently 22 out of 29 project-based training programs (85 contracted to do a training-needs assessment percent of Mexico and Tunisia project-based and to design a learning program. training, and 55 percent of project-based training in Bangladesh and Burkina Faso), and in five of Whether diagnosis and training-needs assess- the eight WBI training courses reviewed (figure ment are done as part of the same exercise or 3.2). All five highly rated WBI offerings used separately, it is important that both be done for external financing for their training-needs assess- training to be properly targeted. For example, ments: three from WBI's partner training the HIV/AIDS Prevention project in Bangladesh institute in Mexico, and two from projects. For correctly diagnosed the need to build the example, project financing enabled WBI's Rural capacity of NGOs working with at-risk popula- Development program in Burkina Faso to do a tions through training, but did not do training- series of participatory needs-assessment work- Figure 3.2: Adequate Training-Needs Assessments Were Done in a Majority of Training Programs Reviewed in Field Missions Project WBI 1 4 2 2 5 1 22 Good Medium Poor Unrated Source: Data based on field mission study findings. Note: The number shown within each pie-chart section indicates the number of training programs reviewed with that section's rating. 28 WHAT TRAINING WORKS: TRAINING DESIGN shops with agricultural producers, so they could selection, these shortcomings in participant help shape a training program that would best selection suggest the need for greater Bank suit their needs. supervision in the design of participant selection processes, resulting in better targeted The three WBI programs without external financ- training. ing did not have adequate training-needs assess- ments. As in the case of in-depth diagnosis, the The six-country survey could provide only WBI does not generally finance formal training- limited information on participant selection needs assessments or ensure that they be done. because participants were not necessarily aware Indeed, a 2004 WBIEG evaluation that surveyed of how they were selected. The survey WBI task managers reported that formal training- found that approximately half of the WBI does not generally needs assessments were done in 31 percent of time, course participants did not have finance formal training- the courses evaluated. While economic and equal levels of knowledge or experi- needs assessments or sector work can be used to assess training needs, ence coming into the course (table ensure that they be done. the WBI does not apply planning and review 3.5). There are circumstances where processes to ensure that training be undertaken assembling training participants with a broad only where it or others have first done a training- range of experience is desirable. However, the needs assessment.16 driver analysis in the six-country survey found that having equal levels of experience and Participant Selection knowledge was an important contributor to For training to contribute to development perceptions of successful training. This finding objectives, it has to involve the right people, and suggests that where participants of similar the right combination of people, in any given backgrounds were preferable, this condition was classroom.17 Optimal participant selection strate- insufficiently ensured. gies vary depending on the development objectives. For certain types of training, it is Among strategies that should be more essential for all trainees to have similar levels of frequently employed in Bank-financed training experience and expertise in order to facilitate is competitive selection. Although not learning, whereas for others, it may be useful to appropriate for all training goals, competitive train an entire unit together, placing high-level selection of participants is seen as an essential managers and low-level assistants in the same contributor to course quality in benchmarked classroom. Table 3.4 provides some examples of institutions such as MASHAV and the Interna- participant selection strategies. tional Monetary Fund (IMF) Institute. MASHAV selects, on average, one out of every Both the field studies and the six-country survey four course applicants for its interna- Poor participant selection highlighted significant shortcomings in partici- tional courses. Lack of a sufficient strategies pant selection (see figure 3.3). The field studies pool of qualified applicants is seen as were associated with a determined that participant selection strategies evidence of lack of demand for a lack of Bank supervision were better for projects than for the WBI. Poor course. Acceptance into IMF of participant selection. participant selection strategies were associated Institute courses is based on a with a lack of Bank supervision of participant rigorous competitive application process. selection. Detailed information on participant Qualified applicants are reviewed and ranked selection strategies is rarely included in project by the IMF's resident representatives, the or WBI planning documents, and the WBI relevant area departments, and the Institute's typically sends letters of invitation to govern- admissions committee. Final selection is done ment ministries with only loosely specified by a committee, chaired by the Institute, with participant profiles. While it is generally neither representatives from the IMF departments that feasible nor desirable for Bank TTLs or WBI task are relevant to the region and to the topic of managers to micromanage participant the training. 29 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Table 3.4: Four Strategies for Selecting Participants Type of strategy Strategy description Appropriate use of strategy Example Competitive Selection based on a Recruitment of new personnel to be In the Bangladesh Procurement Reform Project, competitive application trained for specific roles in the target participants in the training of trainers program process. organization. Selection may be phased, were selected through a multistage competi- with performance in the first stage of the tive application process. Initial training was course determining continued participation. done of trainers who had passed the screening Strategy may be used in multicountry or process. The best performers in the first train- multiorganization courses to ensure uniform ing course were selected to proceed to subse- high quality of training participants. quent courses. Targeted Invitation of Training needed only for persons Trainees for plant protection and seed/plant participants based on fulfilling specific key functions within certification as part of the Tunisia Agricultural highly specified job an organization. These may be key Support Services project were selected on the profiles. decision makers or persons with basis of job profiles. Only technicians or high- specialized technical skills. level members of laboratories were accepted. Widespread Training large numbers Training objectives are organizationwide, The Initial Education Project in Mexico trained of people in an necessitate building support for 1.3 million parents of children aged 0­4 in organization or across change, or call for the coordinated action rural communities of 500­2,500 inhabitants, in a sector, often with of persons serving a range of order to help them play a positive role in their different job functions organizational functions. children's education. and levels of expertise. Demand-driven Training is open to Training for private sector organizations or The Competitiveness and Enterprise Develop- (qualified) participants in the context of community-driven ment project in Burkina Faso offered 50 per- on a willingness-to-pay development programs, where the cent cofinancing credits to businesses request- basis. Includes fee-based participants are in the best position ing training support. Businesses were respon- training and training with to evaluate their own needs. sible for procuring their own training once the cofinancing credits. credits had been awarded. Source: IEG. Table 3.5: Different Participant Expertise Levels Impair Training Results Respondents agreeing Project training WBI training Diagnosis (%) (%) I learned from the experience of other participants in the course. 75 85 Course participants had about equal levels of knowledge/ experience coming into the course. 51 52 Source: Data based on IEG six-country survey of training participants. Note: The differences in the results between WBI and project respondents were significant at a 95 percent confidence interval for the statement: "I learned from the experience of other participants in the course." 30 WHAT TRAINING WORKS: TRAINING DESIGN Figure 3.3: Participant Selection Strategies Reveal a Wide Range of Weaknesses The field studies revealed a number of weaknesses in participant Project selection processes. While none of these phenomena affected 1 1 more than a few training offerings, in aggregate they indicate the many potential pitfalls in participant selection: · Participants were sometimes selected based on distortionary 6 incentives. Where per diems were offered to cover expenses or study tours were provided, the selection of course partic- 21 ipants sometimes reflected the desire to "reward" certain em- ployees rather than substantive criteria. This was particularly a factor in low-income countries. · Participants were sometimes chosen on the basis of their WBI availability or "expendability" from work, when there was low client commitment to training. In one such WBI program in Bangladesh, one trainee reported that 30 percent of partici- 2 pants in the course were "serious" ones whose work re- 3 quired the training, 30 percent were sent solely to represent their ministries or organizations, and 40 percent were there to "fill seats." 3 · Course participation was sometimes expanded to include participants with limited subject-matter experience and ex- pertise who did not need the training to fulfill their work func- tions. This negatively affected participants whose work Good Medium Poor Unrated related more directly to training topics, by lowering the course level and raising the class size so that the use of effective par- ticipatory learning techniques and practical exercises was impossible. This was primarily found in WBI courses, re- flecting, perhaps, WBI management's use of number of par- ticipant training days as an indicator of success. Source: Data based on field study findings. Note: The number shown within each pie-chart section indicates the number of training programs reviewed with that section's rating. From Process to Context: Making depend on the quality of the training design. Learning Work for the Organization Rather, the organizational and institutional Targeting of training, use of practical exercises in context in which training occurs is also a signifi- learning, and follow-up support were all cant determinant of whether learning outputs determined to be important factors in facilitating result in workplace outcomes and in impact on transfer of learning to the workplace. However, development capacity. The next chapter explores success in implementing learning does not only these contextual factors. 31 Chapter 4 Evaluation Highlights · Availability of resources and incen- tives for the implementation of learning are essential for training success. · Over one-third of training partici- pants who tried to apply what they learned in their jobs lacked mate- rial resources to do so. · Lack of incentives posed a particu- lar problem in civil service training in low-capacity countries and in de- centralized training programs. · Projects are better positioned than the WBI to address organizational and institutional capacity constraints affecting the implementation of learning. · Client commitment influences both the organizational context and the design factors that drive successful training. When Training Works: The Organizational Context for Training O ne of the strongest determinants of training success is the organiza- tional context in which training is done. For training to be success- ful, participants must have the resources and incentives to implement acquired skills and knowledge (box 4.1). Where these resources and incen- tives are not in place prior to training, training must be accompanied by prop- erly sequenced interventions in order to address organizational and institutional constraints. This chapter evaluates the extent to which Bank-financed train- ing is done in contexts where the necessary resources and incentives exist for implementation of the knowledge and skills acquired. Box 4.1: Why Learning Cannot Be Implemented The following statements are from surveyed course participants: have in order to bring about bigger changes." (WBI Corporate So- cial Responsibility, Mexico) "After the course, we were asked to write a proposal for possible "The positive aspect is that people do speak about what they assistance (small grant) but to date nothing has come...If they feel about the Millennium Development Goals. The negative aspect didn't promise, I wouldn't be bothered. I had big plans but no funds is there are no resources to carry out the job." (WBI Support for to back it up." (WBI Debate to Action: Building Capacity in Niger- NEEDS Implementation Module: NYSC Dev Knowledge Seminar, ian Youth Organizations, Nigeria) Nigeria) "I could not apply new knowledge in my current work mostly "We lack equipment in the laboratories to apply what we because material and technical capacities of our organization learned." (Organization and Management of a Detection Service, are rather limited." (Social Provision for Families Below the Poverty Tunisia) Line, Azerbaijan) "Unfortunately I could not apply my new knowledge in my "The decisions are not up to me. As much as I would like to apply practical work. It is not easy to persuade other people to behave social responsibility, it's the higher ranks who make the decisions. and work differently."(WBI Lessons Learned for Community Driven I can place my little grain of rice but I need more support than I Development, Azerbaijan) Source: IEG six-country survey of training participants. 35 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Importance of Organizational Context for or support in the workplace in implementing Training Success results of training as the second most important The context in which training is done was found reason for low training impact, after targeting of to be one of the two most important determi- training (table 4.1). nants of training success, along with targeting of training to organizational needs. The "drivers These findings correspond with those of the field analysis" in the six-country survey data, cited in studies and the benchmarking survey (box 4.2), as chapter 3, identified the support of managers and well as with the literature on workplace implemen- colleagues in the implementation of learning as tation of training, which cites the importance of the single most important determinant of training addressing institutional incentives and organiza- success, with availability of material resources and tional resources in order to ensure the sustainabil- targeting of training emerging as other important ity of human-capacity-building efforts. As the variables.1 The 45 percent of survey respondents United Nations Development Programme argued who reported that training had less than a signifi- in a 2006 capacity-development practice note: cant impact on their work cited lack of resources "Attempts to address capacity issues at any one Table 4.1: Lack of Relevance Is the Main Reason Why Training Lacks Impact Respondents agreeing Project training WBI training Diagnosis (%) (%) I did gain significant new knowledge and skills, but they were not very relevant to important aspects of my work. 44 57 I knew how to apply what I learned but did not have the resources or support to do so. 13 17 The course content was relevant to my job but I did not know how to apply what I learned. 11 5 I didn't gain significant new knowledge or skills in the course. 7 9 Other. 18 4 Don't know/ Not applicable. 6 8 Source: IEG six-country survey of training participants. Note: The differences between ratings of WBI respondents and project respondents are not significant at the 95 percent confidence level, using a t-test for sig- nificance, except for the statement: "I did gain significant new knowledge and skills, but they were not very relevant to important aspects of my work." Box 4.2: Importance of Workplace Environment to Training Success In order to gather information on the likely impact of workplace · Job aids are available to support what I learned in this environment on training outcomes, Motorola University asks all training. training participants to rate the following statements. The re- · The procedures taught in this training are ones I can use on sponses to these questions are stronger predictors of course suc- the job. cess than evaluation questions on course content, quality, and · I feel my coworkers will help me if I have problems using this self-assessment of learning. training on the job. · My managers know what I was taught in training. · In general, applying training on the job helps employees in · The training was built to match the way I need to do my job. their careers with Motorola. Source: IEG benchmarking analysis. 36 WHEN TRAINING WORKS: THE ORGANIZATIONAL CONTEXT FOR TRAINING level, without taking into account the others, are enrollment, they neither targeted WBI does not generally likely to result in developments that are skewed, specific organizations nor inquired assess or address resource inefficient, and, in the end, unsustainable."2 into the capacity of the participants' and incentive constraints workplaces. In the single case where in the trainee's workplace Organizational Incentives and Resources WBI training adequately addressed environment. for Applying Learning the capacity context--the Bangladesh Findings from interviews with WBI managers, the Bank Partnership program--training was six-country survey, and the field reviews indicate embedded within an International Development that insufficient attention is being paid to the Association project and received project and workplace context of both project-based and donor financing for related interventions such as WBI training. The WBI does not generally assess long-term, on-the-job, technical assistance, and or address resource and incentive constraints in salary supplements for participants. The field the trainee's workplace environment, unless it studies indicated that projects performed receives project or external donor financing to considerably better than the WBI in this regard, do so. This was illustrated in the case studies. As with over half of projects adequately addressing seen in figure 4.1, only one of eight WBI training the capacity context. offerings adequately addressed the workplace capacity context. In many of the cases that The six-country survey indicated that trainees received medium or poor ratings, inadequate often lacked material resources to apply resources and incentives impaired implementa- learning. Over one-third of respondents who tion of learning. In others, there was no assess- tried to apply what they learned in their jobs said ment of the organization's capacity to facilitate that they had insufficient material resources to implementation of the skills and knowledge implement knowledge and skills gained and no monitoring of whether trainees acquired in training (table 4.2).3 Lack of incentives was a were able to implement learning upon returning more frequent problem to their workplaces. Lack of incentives was found in the field for training civil servants studies to be a more frequent problem than for training For example, in the case of the three WBI for training civil servants than training community groups, programs in Mexico, poor ratings were awarded community groups, farming collectives, farming collectives, or because, as distance-learning courses with open or private sector firms. These latter private sector firms. Figure 4.1: Projects Rate Higher Than WBI on Attention to Capacity Context Project WBI 1 7 1 16 4 6 2 Good Medium Poor Unrated Source: Data based on field study findings. Note: The number shown within each pie-chart section indicates the number of training programs reviewed with that section's rating. 37 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Table 4.2: Organizational Incentives for Implementing Learning Are Satisfactory but Material Resources Are Insufficiently Available Respondents agreeing Project training WBI training Diagnosis (%) (%) My managers encourage my efforts to use what I learned in the course 92 81 My colleagues provide me with the support I need to apply what I learned in the course 80 80 Policies at my organization allow me to apply what I learned 91 85 I have the resources to apply what I learned 65 61 Source: Data based on six-country survey of training participants. Note: The differences between ratings of WBI respondents and project respondents are significant at the 95 percent confidence level for the statement: "Poli- cies at my organization allow me to apply what I learned" and "My managers encourage my efforts to use what I learned in the course." groups were more likely to have "built-in" accompanied by strategies to deal with problems incentives for applying learning, arising from the of staff turnover and weak incentives, such as possibility of direct personal benefits, such as supplementing training with salary bonuses, or higher profits or more financing of community building local capacity to continue training new projects. Incentives to apply learning fall into two personnel, in environments with high staff categories: general and specific. General institu- turnover.4 tional incentives, such as merit-based promotion systems or competitive salary levels, affect the Adequate specific incentives for implementing participant's motivation to improve workplace training, unlike general institutional incentives, performance. Specific incentives for were found to be as likely to occur in higher- Incentives to improve application of new knowledge and skills capacity countries as in lower-capacity countries. workplace performance include the support and encourage- Specific incentives for implementation were were lacking in countries ment of managers and colleagues and found to be particularly problematic in cases of with weak public sector the existence of legal/regulatory diffuse or decentralized training programs, such capacity. frameworks for implementation of as in-service teacher training. In such cases, even learning. where commitment to training goals was high in the central government, it did not necessarily General institutional incentives were a problem translate into strong commitment levels among primarily in Bangladesh and Burkina Faso, the the trainees, their managers, or their colleagues two field mission countries with low public sector in the field. capacity. Where general institutional incentives are weak, staff have less incentive to pursue their Importance of Client Commitment own professional development, and bureaucra- Timely availability of resources and incentives cies have difficulty retaining trained personnel. In was also found to be highly correlated with client the most extreme cases, the building capacity of commitment to the objectives of learning. Where individuals working in the public sector can high-level decision makers see the training weaken organizational capacity through a "brain objectives as a priority, they are more likely to drain," if trained personnel are in ensure that adequate organizational and institu- Incentives to apply the demand elsewhere. This does not tional conditions exist to enable implementation specific knowledge and mean, however, that human capacity of learning. Where this is not the case, significant skills acquired were building should not be undertaken in delays in the disbursement of training resources particularly lacking in contexts without adequate general may ensue, even when clients have committed diffuse or decentralized institutional incentives. Such training to such disbursements up front. For example, training programs. can help build capacity as long as it is bureaucratic problems delayed the launch of 38 WHEN TRAINING WORKS: THE ORGANIZATIONAL CONTEXT FOR TRAINING training in the National Nutrition Project in training goals and priorities (figure Where high-level decision Bangladesh by approximately two years. 4.2). This finding corresponds with a makers see training Moreover, once the program began, training recent WBIEG evaluation of the WBI`s objectives as a priority, organizers reported difficulties in obtaining the primary, country-level, program they are more likely to promised financing, and former "nutrition strategy document--the country ensure that the conditions promoter" trainees did not consistently receive program brief--which found that, for success exist. government payments for their activities. "Although WBI has suggested that its learning programs are country led and fully Client commitment to training, where the owned at the country level, and that all focus clients are government policy makers, was countries `meet the essential criterion of better in Mexico and Tunisia where projects are ownership of the capacity development process,' loan-financed than in Bangladesh and Burkina few CPBs [(country program briefs)] provide Faso where projects are grant-financed. Sixteen evidence to support these statements."5 out of 20 project-based training programs that could be rated in Mexico and Tunisia scored As mentioned in chapter 3, client commitment highly on client commitment, whereas only to training also influences training design and three out of nine project-based training implementation processes such as the targeting offerings in Bangladesh and Burkina Faso of training and the monitoring and evaluation of received high scores in this category. Judgments training. For example, the Bangladesh on client commitment were based on Public Procurement Reform project Client commitment interviews with government policy makers was one in which strong client influences the about the importance of training goals, as well commitment to training goals, and availability of resources as assessments by Bank staff and other inform- strong involvement in implementing and the support for ants involved in training organization. learning, resulted in the establish- transferring learning to ment of feedback loops between the workplace. Overall, client commitment was better in projects training and the workplace, thus than in the WBI, reflecting lack of meaningful enabling workplace realities to better inform direct dialogue between the WBI and clients on training content (see box 4.3). Figure 4.2: Client Commitment Is Stronger in Projects Project WBI 1 1 9 1 3 19 3 Good Medium Poor Unrated Source: Data based on field study findings. Note: The number shown within each pie-chart section indicates the number of training programs reviewed with that section's rating. 39 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Box 4.3: Setting Up Training-Feedback Loops in the Bangladesh Public Procurement Reform Project In medium- or long-term training programs, monitoring and eval- side training provider was contracted to train trainers and manage uation systems can help improve outcomes by establishing feed- the training, the Central Procurement Technical Unit remained ac- back loops between the workplace and training programs. tively involved in the design, implementation, and monitoring of Information on the implementation of knowledge and skills by for- training. The unit used data from its ongoing monitoring of imple- mer trainees can be used to shape training content. mentation of procurement regulations to refine and enhance the Monitoring and evaluation was successfully used in the training program and to provide strategic follow-up support to Bangladesh Public Procurement Reform Project to improve train- trainees in instances where there were evident implementation ing. The project included widespread training of both public sec- problems. Presently, with the training program stabilized after a tor officials and private sector agents, to facilitate the transition to lengthy pilot period, the unit is working to phase out its involvement newprocurementregulations.ACentralProcurementTechnicalUnit in training by building the capacity of a local training institute. The was established in the Ministry of Planning to implement and mon- unit's involvement in the initial phases was crucial to the success itor the transition to new procurement regulations. Although an out- of the training, in support of the transition to the new regulations. Source: Data based on field study findings. 40 Chapter 5 Evaluation Highlights · The Bank's client-training programs are neither systematically monitored nor adequately evaluated. · There are no specific or clearly de- fined Bank standards for training, against which the quality of design implementation can be measured. · Task Team Leaders lack adequate expert advice and the WBI is not their preferred source of expertise. · The WBI's focus-country approach has improved coordination with the country team but does not appear to have significantly enhanced the WBI's contribution to the Bank's de- velopment objectives. Bank Processes and Systems T he preceding chapters established that successful training results are strongly correlated with good training-management processes. Train- ing succeeds in building development capacity when there is adequate support for the transfer of learning to the workplace, sufficient targeting of training to organizational needs, and an organizational context that facilitates implementation of learning. Training-management processes that were found WBI and project-based training have separate to be associated with these conditions include: structures for the management of training, with (i) using practical learning techniques and Bank staff playing very different roles in each. follow-up support to facilitate transfer of learning The WBI is generally directly responsible for the to the workplace; (ii) using diagnostic exercises, design, implementation, and evaluation of training-needs assessment, and strategic partici- training, either independently or in partnership pant selection to ensure proper targeting of with local training institutes. Project task teams, training; and (iii) embedding training in a on the other hand, support clients in their fulfill- broader capacity-building strategy that addresses ment of the above functions. This difference was resource constraints and incentive gaps likely to taken into account in the following analysis. affect the implementation of training. Identifying and Measuring the Results How well do Bank systems, procedures, and of Training allocation of resources facilitate the implementa- The monitoring of Bank-financed Bank documentation tion of these training processes? This chapter training is poor, owing to two short- inadequately specifies examines the extent to which the Bank ensures comings: (i) projects and WBI pro- training objectives. the quality of training management by mandat- grams do not consistently establish ing design standards that can form the basis for clear and monitorable training objectives at the quality control and supervision of training. It also design stage, and (ii) there are no clear Bank looks at whether Bank evaluation procedures standards against which to benchmark progress provide an accurate picture of performance on in training design and implementation. training. Finally, it examines whether Bank resources and expertise are configured to facili- Lack of clear and monitorable training objectives tate effective training. For adequate monitoring and supervision of 43 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT training, it is necessary to define, at the outset, training are rarely adequately evaluated. This training objectives that specify the knowledge, evaluation used Kirkpatrick's four-level model (see skills, and attitudes to be learned through table 5.1) as the basis for analyzing training evalua- training, performance changes that the learning tion adequacy. The prevalence of each of the should generate, and key indicators of perform- evaluation levels was checked through documen- ance. A review of appraisal documents for 38 tary reviews of Bank Implementation Completion randomly selected closed projects with high Reports (ICRs), WBIEG evaluations, WBI monthly training content in 13 countries, during fiscal management reports, and field reviews. 1997­2006, found that while 27 Monitoring of Bank- projects detailed numerical training The field reviews revealed that few training financed training is targets, only 5 specified desired programs assessed the results of training on consistently poor. learning outcomes, and 10 detailed workplace performance (level 3). Only 7 of the workplace performance outcomes 29 training programs and one of the eight WBI (see appendix D). Similarly, a review of the courses examined in the field reviews evaluated Activity Initiation Summaries (AISs) for 60 workplace behavior outcomes.2 Similarly, a randomly selected WBI programs in fiscal 2006 review of 38 randomly selected ICRs of projects found that while three-quarters (47) of the with high training content found that while over programs specified desired learning outputs, 90 percent (35) reported on numbers of people only 17 percent (9) specified desired workplace trained in at least some of the training financed performance outcomes, and half a percent (3) by the project, only 16 percent (6) reported on specified performance indicators for outcomes.1 learning outputs and 26 percent (10) on performance changes (see appendix D). Given the insufficient clarity and specificity of training objectives in required Bank documents, Evaluation of client training by WBI is done by such as project appraisal documents the WBIEG. WBI program managers are not Results of training (PADs) and AISs, the country cases expected or provided budgets to conduct self- activities are attempted to obtain supplementary evaluation of workplace performance outcomes inadequately evaluated. documentation or descriptions of of training. The WBIEG is responsible for compil- learning and performance change ing and analyzing results from level-1 evalua- objectives from stakeholders interviewed on tions, as well as for preparing more in-depth field missions. The field studies found that: evaluation reports. The WBIEG is not independent. · Learning output objectives were adequately spec- ified in program documentation in 23 of the 29 The only form of evaluation to which WBI training programs reviewed, and in 6 of the 8 training courses are uniformly subject is level-1 WBI offerings. participant satisfaction questionnaires, which are · Workplace performance outcome objectives were distributed at course end (box 5.1). Although the adequately specified in 17 out of 29 training WBI provides its task managers with optional programs and in 3 out of 8 WBI offerings. questions that can be used to customize and · Performance indicators were adequately speci- expand the knowledge obtained from level-1 fied in 15 of 29 training programs. One of the evaluations, no examples were found in the field eight WBI offerings had a good rating on spec- reviews of their use.3 ification of performance indicators, one had a medium rating, and the rest were rated as The information generated by standard, WBI poor. level-1 questionnaires is insufficient for two reasons. First, most research has indicated that Evaluation of training results there is little or no correlation between Field studies and reviews of Bank documentation responses on level-1 evaluations and learning established that the results of Bank-financed outcomes (discussed in box 5.2). Second, the 44 BANK PROCESSES AND SYSTEMS Table 5.1: Four Levels of Training Evaluation Most training evaluation models today are based on the Kirkpatrick four-level model, described in the table below. Level 1 gauges participant satisfaction and the other three evaluation levels correspond with stages in the results chain. Level Measures Common Means of Verification Level 1 Participant satisfaction End-of-course participant questionnaires. Level 2 Learning outputs Posttests, sometimes as compared with pretests. Level 3 Performance change outcomes Multiple, including observation, interviews and surveys of participants, colleagues, and supervisors. Level 4 Organizational impact/results Multiple, including comparisons with baseline organizational performance measures, surveys, and interviews with key informants. Source: Kirkpatrick 1998. standard WBI questionnaire does not ask for pretests and posttests in WBI courses Participant satisfaction information on specific topics, course sessions, to compile data on learning. The last questionnaires are of or lecturers, and, as such, does not elicit informa- review of level-2 results completed by limited use in improving tion that can be used to identify aspects of the the WBIEG was done in August 2005. training quality. course that should be altered to improve quality. The use of testing in WBI courses has Of the WBI task managers interviewed for this since been largely discontinued. evaluation, none reported that level-1 question- naires provided useful information for designing The WBIEG has also mostly discontinued level-3 future training. A 2007 review of new approaches evaluations of specific sector and thematic to the evaluation of learning programs commis- programs. Three level-3 evaluations of thematic sioned by the WBIEG noted that "it may be programs were published in fiscal 2005, only one difficult to draw inferences about the quality of in fiscal 2006, and none are planned for fiscal 2007. training" from level-1 evaluations and thus The WBIEG continues to send level-3 evaluation suggested that "primary reliance on participants' questionnaires to training participants eight reactions to evaluate training should be months to two years after completion of training. reconsidered."4 The responses provide information on a country basis but not on a program basis, and thus include In addition to collating and analyzing level-1 little information that program managers can use results, in the past the WBIEG used level-2 to improve the quality of their courses. Box 5.1: Limited Focus of WBI Level-1 Evaluation The WBI's standard, level-1 questionnaire asks six questions, · Usefulness for you of the information that you have obtained. each rated on a 5-point scale. Participants can also check "no · Focus of this activity on what you specifically needed to opinion" for any question. learn. · Extent to which the content of this activity matched the an- · Relevance of this activity to your current work or functions. nounced objectives. · Extent to which you have acquired information that is new · Overall usefulness of this activity. to you. Source: WBI evaluation toolkit, client level-1 templates. 45 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Box 5.2: Participant Satisfaction Questionnaires: What Can They Tell Us? Level-1 evaluations provide information on participants' level of cautions against using level-1 evaluations as an indicator of the satisfaction with the training. Level-1 evaluations are "designed value of training to the achievement of organizational goals (Lee to measure participants' attitudes and feelings toward the interest and Pershing 2002; Bramley and Newby 1984). and usefulness of the content, effectiveness of the instructor, and While participant reaction questionnaires may have limited quality of the materials" (Keller 1996). value for assessing results, they can help improve future pro- Level-1 evaluations do not provide information about learning grams if used correctly. Level-1 evaluations can provide useful in- outputs, behavior outcomes, or organizational impact. Rodin & formation on didactic methods and instructor quality, as long as Rodin (1972) found a negative correlation between participants' re- questions are formulated to provide specific information on the lec- actions as recorded in level-1 evaluations and learning. Dixon turers, course modules, and topics needing improvement. General (1990), as well as Warr and Bunce (1995), found no correlation be- questions on course satisfaction do not enable course organizers tween learning outcomes and participant assessments of course to identify what aspects of the course can be altered to improve enjoyment, usefulness, and difficulty. Furthermore, the literature quality. Source: IEG literature review. For example, in fiscal 2006, level-3 reviews were adequate expert support for the supervision of completed in Senegal, Tanzania, and the Philip- training design and implementation, and (iii) the pines. In each country, surveys were distributed WBI is configured to do effective training. to participants in a wide range of training programs, providing general statistics on Bank standards for training design learning implementation, and its correlation with While Bank-financed projects have ample quality general training features, such as control mechanisms to ensure acceptable WBI has largely course length, but no information on standards of project design, including on aspects discontinued outcome specific program themes or content. such as financial management, engineering, evaluations of specific This issue was raised in many environment, and procurement, they lack clearly sector and thematic interviews with WBI task and program defined standards in the particular area of programs. managers conducted within the training design. This makes it difficult for existing context of this evaluation. Several quality review mechanisms in the Regional Vice- stated that neither these general level-3 evalua- Presidential Units and in the Quality Assurance tions, nor the standard level-1 evaluations, Group to assess the quality of training design and provide useful information to help them implementation, even when such components improve program content. In two cases, WBI are critical to the achievement of the project's program managers suggested that having development objectives. program managers do self-evaluations of workplace performance outcomes would The WBI, for its part, has no formal quality substantially improve their ability to gauge the assurance mechanisms. The Quality Assurance success of their programs and to improve Group does not assess the quality of WBI training training content. programs. The WBIEG did introduce, in 2004, the option for WBI task managers to undertake Bank Support for Training quality enhancement reviews by expert panels The following section explores the extent to that could advise the task managers on preiden- which Bank structures and procedures facilitate tified issues and questions. Owing to lack of effective training. Specifically, it explores the interest from WBI task managers, quality extent to which (i) the Bank defines standards enhancement reviews were discontinued for training design, (ii) project TTLs have beginning in fiscal 2006. 46 BANK PROCESSES AND SYSTEMS Support to project task team leaders preferred options were the establish- Project task teams do not Project-based client training programs do not ment of in-house training expertise, get sufficient expert benefit from sufficient expert advice on design and and having a training expert on the advice on training design implementation issues. In a survey of 43 randomly project team (figure 5.1). and implementation. selected TTLs, all of whom had managed training in the context of their projects, most reported the TTLs reported that they did not generally use the need for more technical help in designing and WBI or view it as an important in-house resource supervising training components of projects. Half for training support. When TTLs did need advice of TTLs surveyed who had previously received on training, they were far more likely to go to advice on training design expressed a desire for other sources of expertise than to WBI staff even more expert support for training. The (figure 5.2). Figure 5.1: Most TTLs Would Prefer Having an Internal Unit of Experts Preferred options for additional training advice 20 total 34 18 oftuo 16 14 tsn 12 den 10 8 espor 6 ofr 4 2 mbeu 0 N Unit of experts Training for Web-based or Training expert Updated No answer within the TTLs written on project database of Bank guidelines team experts Source: IEG survey of 43 TTLs. Figure 5.2: Most TTLs Seek Advice from Bank Operational Colleagues and External Consultants Sources and quality of advice total 34 35 oftuo 30 25 tsn 20 den 15 espor 10 ofr 5 mbeu 0 N None External WBI Bank operational colleagues Highly satisfactory Satisfactory Partially satisfactory No help Source: IEG survey of 43 TTLs. 47 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT WBI contribution to country assistance · Two-thirds of survey respondents in WBI focus strategy goals countries stated that they had never worked In fiscal 2002, the WBI introduced a focus-country with the WBI on the design or delivery of WBI- approach. In cooperation with the Regions, it financed training, and had never been con- designated countries that were to receive the sulted by the WBI in shaping WBI training majority of WBI training resources. At present, 70 priorities. percent of WBI programming budgets are · Ten of the 20 TTLs surveyed from focus coun- allocated for focus countries. In these countries, tries who expressed an opinion on changes in WBI Regional Coordinators are appointed to set their relationship with the WBI in the past two training programs, based on dialogue with the years reported that the relationship had stayed country teams, so as to better align WBI training the same, with no improvements. Only one TTL with country assistance strategy priorities. reported significant improvement in their co- operation. Eight others reported that cooper- A recent WBIEG survey of country directors, ation had improved somewhat. managers, and other team members "most likely to be knowledgeable about WBI's activities" in WBI Evidence gathered in the field reviews presents focus countries found that WBI consul- an uneven picture of collaboration and coopera- Bank country teams are tation with the country team on the tion. Burkina Faso had very little collaboration satisfactorily consulted former's work program was satisfactory. between the WBI and the country team on the on WBI training Ninety-eight percent of respondents provision of training, Mexico had some collabora- programs. said that the WBI satisfactorily tion, and Bangladesh had extensive and fruitful consulted with the country team to cooperation and collaboration. Differences in the select its activities, and 87 percent found that the extent of collaboration and cooperation could be WBI was aligned either to a large extent or traced to different management practices of WBI completely with country capacity needs.5 Field Regional Coordinators (box 5.3). study interviews with key country team informants confirmed that country directors or managers are The greater coordination of WBI activities with generally consulted on WBI training programs in country assistance strategy priorities has also not their countries. led to WBI country programs with sufficient coherence and concentration to make a substan- The focus-country Increased consultations with the tial contribution to the achievement of country structure has not led to country team on the WBI's overall work assistance strategy goals. The WBI's work better collaboration program has not led to systematically program in its focus countries continues to be between WBI task better collaboration on the level of WBI diffuse. Over 80 percent of fiscal 2006 country managers and Bank TTLs. task managers and operational task program briefs designated six or more priority teams. There are no official WBI areas for WBI activities in the following year.6 On guidelines or incentives to encourage average, in fiscal 2006, the WBI provided 2.7 collaboration between WBI task managers and training activities in each priority area, with each project task teams. A survey of 43 TTLs completed activity training a median of 31 participants over for this evaluation, which included respondents three days.7 Such a small amount of training is from 15 WBI focus countries and 14 nonfocus likely to have a significant impact on country countries, indicated that capacity only where it is strategically focused and highly leveraged, such as the IMF Institute's · Two-thirds of TTLs surveyed from focus coun- training of senior officials in highly specialized tries had never used the WBI to provide train- economic management skills. The WBI's weak ing in the context of their projects. targeting of training content and participant · Seventeen of the 28 TTLs surveyed in focus selection, as discussed in chapters 3 and 4, is not countries replied that they were aware of only consistent with a highly strategic use of training some WBI activities or none in their sector. resources. 48 BANK PROCESSES AND SYSTEMS One of the ways in which the WBI tries to that they strongly value their relation- WBI country programs leverage its impact and to contribute to country ships with the WBI, but the scope of lack sufficient coherence capacity is through building local training cooperation is limited. Forty-three and concentration to capacity. According to the WBI percent of institutes surveyed make a substantial reported that they had been contribution to country Partnerships help to fulfill WBI's overall partnered with the WBI for no more assistance strategy goals. mandate by building the capacity of than three years. Two-thirds (67 partners, especially in WBI's focus percent) of the training institutes had organized countries; helping share the cost of deliver- between one and five training courses with the ing WBI activities and improving local WBI over the past year. In addition, many relevance, and/or strengthening WBI's respondents stated a desire for greater capacity- ability to reach its target audiences and to building support. Only 27 percent strongly achieve its program objectives.8 agreed that the WBI supports building the capacity of its lecturers, suggesting that there is To these ends, half of the WBI's training courses room for improvement. When asked if there are conducted in cooperation with partner were aspects of their relationship with the WBI training institutes. However, while such partner- that they would like to highlight, the most ships are undoubtedly useful as a cost-sharing frequent issue raised by respondents was the arrangement, this evaluation found no evidence need to build the capacity of their institutes. that the WBI's partnerships with local training Box 5.4 provides some of the comments made institutes have had a significant impact on by local training partners surveyed. country training capacity. A survey of 30 randomly selected WBI-partner training institutes com- Bank Resources for Effective WBI-partner training pleted as part of this evaluation (appendix F) Training institutes strongly value suggested that the WBI's contribution to building This evaluation indicates that project- their relationships with the capacity of training institutions is slight. based training and WBI training that is WBI but would like more linked to Bank projects has been more capacity-building Most WBI-partner training institutes reported successful in translating learning into support. Box 5.3: Collaboration between WBI and Operations: Contrasting Cases The appointment of a WBI Regional Coordinator for South Asia notes are used in other Regions, country team input appears to be has helped to foster collaboration and cooperation between much less substantive than in Bangladesh in many cases. the WBI and the country team in Bangladesh. Burkina Faso presents a different picture. Most of the TTLs in- WBI task managers work closely with country team members terviewed in Burkina Faso reported that they were only rarely to design course offerings. Operational staff that are active in made aware of WBI activities in their sectors, and had never Bangladesh are aware of WBI activities in their sector and report being consulted by WBI personnel on training content or partici- viewing the WBI as a resource for collaborating on achieving pant selection. Lack of coordination with the country team was high- capacity-building goals, including, in some cases, using WBI ex- lighted in a 2004 WBIEG evaluation of WBI activities in Burkina Faso, pertise in the context of active or planned projects. which found insufficient collaboration between operations and the The use of WBI activity inception notes, drafted jointly with WBI, with operations staff being consulted only occasionally and country team members, was found to be instrumental in facilitat- inconsistently. The WBI implemented a WBIEG recommendation ing collaboration. These notes provide detailed information on the to appoint an in-country WBI representative, but this person is re- reasons for and importance of the training activities given partic- sponsible for course logistics rather than the substantive aspects ular country assistance strategy goals. While similar inception of the WBI program. Source: Field study findings. 49 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Box 5.4: Insufficient Building of Partner Training Institute Capacity by WBI Several of the WBI-partner training institutes surveyed spoke of studies . . . WBI most of the time takes primary responsibility in all their desire for greater WBI focus on building partner capacity activities." and for greater devolution of responsibilities from the WBI to the "I wish WBI [would] localize and let us participate more." partner institutes. The following sample comments are from a sur- "Participants benefit from the courses, but not our institution. vey of WBI partners: We would like to have a partnership that offers us help in devel- opment as an institution and in building our own capacity." "It seems that WBI wants to execute the project, achieve the "It was a very good initiative; the World Bank was instrumen- goal, and that's it, when what is needed from WBI is to establish tal but it is about time to move it to the region... and have the part- a way to strengthen nationally and regionally the programs." ners hosting [the conference]." "WBI doesn't share its knowledge, training modules, or case Source: Partner institute survey. enhanced workplace behaviors than "free- knowledge can be implemented effectively in the standing" WBI training. Project-based training is workplace, without addressing any other better positioned to adequately manage the capacity constraints or providing substantial contextual and process requirements for effective follow-up support. Therefore, it is not surprising training. Projects are more conducive to diagnos- that the one WBI training offering (WBI- ing country contexts and assessing training needs Bangladesh Bank cooperation)--in the field relative to capacity-building goals, to addressing study reviews that received a good rating on the capacity constraints that might affect training training outcomes and impact--was embedded outcomes, and to follow up support after in an investment project that provided other training. However, while projects may be better forms of capacity support, as well as received positioned to use training for capacity building, financing from other donors for continuing they do not always succeed because of technical assistance. WBI training works best a lack of adequate design, quality where capacity control based on elaborated design The lack of training expertise in projects and the constraints have been or standards, and expert support. Proj- lack of a full range of capacity-building support are being addressed in ects can do training right, but for this available under the WBI business model suggest associated capacity- to happen, project managers have to that greater synergy is needed between Bank building projects. know how to do it right. resources for capacity building and Bank training expertise. Evidence presented above indicates The WBI, for its part, can do effective capacity that the present relationship between the WBI building, independent of external financing, only and Bank staff and systems has not addressed in limited instances where learned skills and this need. 50 Chapter 6 Findings and Recommendations Findings O ver the past decade, the Bank has financed approximately $720 million annually in training, more than 90 percent of it through projects and the remainder through the WBI. Bank-financed training supports a wide variety of capacity-building goals, with target populations ranging from small farmers and community groups to high-ranking government officials. While training occurs in all sectors of Bank from training, only about half the time did activity, it plays a particularly important role in the learning lead to substantial changes to health, nutrition, and population sector, where it workplace performance or enhanced accounts for an estimated 16 percent of invest- development capacity of target institu- ment project costs. However, the importance of tions. Project-based training was more success- training to the achievement of development ful than WBI training in this regard. Sixty percent objectives goes well beyond its dollar value. of project respondents and 50 percent of WBI respondents to the six-country survey reported Training is one of the primary means by which the substantial changes to key work functions as a Bank helps to build the capacity of countries to result of training. Evidence of workplace per- reduce poverty. Moreover, it is often fundamental formance improvements, as a result of training, to the success of other investments. Overall, in was found in 15 of the 29 project-based training an estimated 60 percent of projects, training is programs reviewed in field studies, but the same either integral to the achievement of the goals of was found in only one of the eight WBI one or more project components or supportive programs. Where learning did not result in of all project components. changed workplace performance and, therefore, did not have an impact on development capacity, Most Bank-financed training results in individual this could be attributed to one of three reasons: participant learning. Of all the training partici- (i) insufficient participant understanding of how pants surveyed for this evaluation, only a small to apply learning in the workplace, (ii) inade- percentage claimed that they had not gained quate incentives or resources for implementa- significant new knowledge through training. tion of learning, or (iii) inadequate targeting of However, while most participants learned learning to organizational needs. Making the 53 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT leap from individual learning to workplace be assessed, and participants should be selected performance outcomes and, subsequently, in a strategic manner. Project-based training to development capacity impact requires reviewed in field missions performed better than both good training design and an appropri- WBI training in all of these targeting processes. ate organizational and institutional context Projects were better targeted mainly because they in which to apply the learning from have more resources to conduct capacity assess- training. ments and they involve clients more fully in the design of interventions. The WBI does not Training success is predicated on adequate design. generally mandate or finance in-depth diagnoses Good training design was found to involve three of capacity gaps or assessments of training needs, characteristics: and it does not consistently consult with clients on training objectives and design. · Use of appropriate and professional pedagogic design, including opportunities to practice The organizational context for implementing learned skills; knowledge and skills learned was a second · Provision of follow-up support to trainees to important determinant of successful capacity facilitate implementation of knowledge and building through training. Training builds develop- skills acquired; and ment capacity only when trainees have adequate · Targeting of training content, anchored in di- resources and incentives to implement learning agnosis of institutional and organizational ca- in the workplace. One-third of training partici- pacity gaps, formal assessment of participant pants surveyed stated that they lacked such training needs, and strategic participant sufficient material resources. Some trainees also selection. lacked incentives to implement learning. Insuffi- cient incentives were particularly problematic in Much of the Bank-financed training reviewed was two contexts. First, in decentralized training found to have design flaws that affected results. programs, like in-service teacher training, the While over 90 percent of survey respondents central government's commitment to training found their training to be interesting and the goals did not necessarily translate into strong lecturers to be of high quality, half stated that commitment among regional government course length was too short for the topics officials, training participants, or their managers. covered and that their course did not devote Second, in the public sectors of countries with significant time to practical exercises and weaker government capacity, low salary levels and projects. Adequate follow-up support was lack of merit-based promotion systems reduced provided to project trainees in half of the 29 the incentive of staff to pursue their own profes- programs reviewed in field studies and to WBI sional development. But even where trainees in only two of the eight cases reviewed. resources or incentives for implementing Finally, of the nearly one-half of all survey respon- learning were initially lacking, training dents who stated that training did not have a reviewed in the field studies succeeded as significant impact on key functions of their work, long as there was strong client commitment over a third attributed this to lack of relevance of to training goals and adequate support was training content to key work functions. This last given to addressing related workplace issue is indicative of inadequate targeting of capacity gaps. training content. Field studies revealed examples of successful Targeting of training content was found to be the most Bank-financed training activities provided by local important training design factor driving training training institutions, client governments, interna- success. For training to be well targeted, organi- tional consultants, and training providers, as well zational and institutional capacity gaps need to be as the WBI. In all cases, training succeeded correctly diagnosed, specific training needs must when its design was good and the organiza- 54 FINDINGS AND RECOMMENDATIONS tional and institutional capacity context systematically monitors at the program level only was adequately addressed in conjunction the number of participant training days and with training. participant satisfaction, neither of which provides information about the impact of The WBI's training procedures and practices do not training on capacity-building objectives. Hence, sufficiently anchor training within comprehensive clients, project task teams, and WBI task capacity-building strategies and are, therefore, not managers, alike, generally do not have sufficient generally conducive to building sustainable capacity. information to detect training weaknesses and The WBI lacks systemic mechanisms for in-depth improve training performance, where necessary. diagnoses of organizational capacity gaps or formal training-needs assessments of participants. Recommendations It also lacks standardized procedures for meaning- The Bank can enhance the vital contribution of ful, direct consultation with clients on training training to client capacity building, by ensuring needs and priorities. In most cases, the WBI does that the training it supports not directly provide follow-up support to facilitate workplace implementation of learning. It also · Is linked to the Bank's support for development does not systematically link its training programs objectives in client countries, to complementary capacity-building support · Is embedded within broader capacity-building provided by operations or other partners. strategies that provide complementary sup- port for the implementation of learning, and The quality of project-financed training is uneven · Conforms with best practice in training design. due to a lack of explicit design standards for all Bank training activities, and lack of expert support for The following three recommendations are training activities embedded in projects. Bank- intended to lead to this outcome: financed projects provide an opportunity for effective use of training as part of an integrated Recommendation 1: The Bank needs to develop capacity-building strategy. The project model can guidance and quality criteria for the design and ensure that training is integrated into a compre- implementation of training, so as to enable quality hensive, multiyear relationship with the target assurance, and monitoring and evaluation of all its organization, financing a range of complemen- training support. This guidance should be applied tary capacity-building interventions. However, to all training financed by the Bank, including the lack of defined design standards and expert training that is directly provided by units such as support make it difficult for project teams to the WBI. Design guidance should include adequately supervise the design and implemen- tation of training. This also prevents quality · Diagnosis and training-needs assessment re- assurance mechanisms from being applied to quirements for training initiation; training activities. · Participant selection criteria; · Standards for the use of practical exercises The Bank does not adequately monitor or evaluate and other action-learning techniques within training results. Most project-based and WBI training; training reviewed in field studies did not include · Use of follow-up support; and sufficient monitoring and evaluation of training. · Provisions for monitoring and evaluation, in- Project Implementation Completion Reports cluding specification of performance-change seldom report on more than the number of objectives and key monitorable indicators. persons trained and include little or no informa- tion on training results in terms of workplace Recommendation 2: The Bank could improve the behavior of participants and impact on develop- quality and impact of training by making available ment capacity, even where training is fundamen- to its Regional staff and borrowers, resource tal to the achievement of project goals. The WBI persons with technical expertise in the design, 55 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT implementation, and monitoring and evaluation identifying (i) clear and specific capacity-building of training. objectives, (ii) the human, institutional, and or- ganizational capacity support that is necessary Recommendation 3: Management must clarify the in order to achieve these objectives, and (iii) WBI's mandate on provision of training with measurable indicators of success; capacity-building goals. If the WBI is to play a · Is undertaken after work is done with opera- capacity-building role in client countries, its tions and partners to identify and confirm the training processes should be substantially reengi- resources required to achieve the capacity- neered to ensure that training is likely to building objectives, including, where needed, contribute to sustainable change. New WBI (i) multiyear training programs, (ii) follow-up training processes should ensure that all training technical assistance, and (iii) organizational meets the following criteria: and institutional support measures, such as policy support and financing of implementa- · Is based on a comprehensive capacity assess- tion of learning; and ment of the target organization(s)/institu- · Is subject to external quality review and eval- tion(s)--done in cooperation with clients-- uation of results. 56 Appendixes APPENDIX A: METHODOLOGY This section describes the methodology used for (Bangladesh and Burkina Faso). In each country, the major evaluation components. evaluators interviewed key informants using common guidelines. In all, field mission person- Definition and Scope nel interviewed 136 training participants, 103 The evaluation covers project-financed and WBI government officials, including policy makers and client-training programs undertaken during fiscal training administrators, 60 Bank staff, and 11 1995­2006. It defines "client training" as includ- international agency representatives. Details of ing all World Bank support for training national the field missions are provided in table A.1. stakeholders of developing countries when the activity has the following characteristics: WBI programs were evaluated only in the three case-study countries that were also WBI-focus · Has specific learning objectives, countries: Bangladesh, Burkina Faso, and · Is a scheduled activity that occurs in a setting Mexico.1 Projects and WBI training offerings for other than where the knowledge will be ap- review were chosen to plied, and · Is conducted by trainers or facilitators. · Represent a broad range of sectors and the- matic areas; and This evaluation focuses on client training · Include high training content. For projects, this undertaken at the country level. It excludes meant that training was integral to the achieve- training of World Bank staff and of borrower ment of the objectives of at least one project officials to enable implementation of Bank component, or training was supportive of all operations, for example, procurement training project components. For the WBI, preference for staff in project management units. WBI or was given, where available, to multiyear pro- other Bank global programs were not considered grams involving multiple training offerings. unless they were included in specific country- level training programs. All four field reviews drew on core program documentation as well as program progress Evaluation Tools reports, existing self-evaluations and independ- The evaluation used a wide range of methods to ent evaluations, related Bank country assistance assess the efficacy of Bank-financed client strategies and sector strategies, and interviews training. with clients and Bank staff. Data on training results was sought in program documentation. A. Field reviews Where such evidence did not exist, assessments The review conducted in-depth field reviews of on training results were made through the Bank's support for training, examining 29 interviews with a wide range of stakeholders, project-based and 8 WBI training programs in including Bank staff, clients, training participants four countries. The review chose two countries and their supervisors. In addition, interviews with relatively high public sector capacity (Mexico with these stakeholders were used to gather and Tunisia), and two with relatively less capacity evidence on the effectiveness of the training 59 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Table A.1: Field Mission Details Stakeholders interviewed Training International Training providers/ Policy agency Bank Country Mission dates Mission team participants administrators makers representatives staff Bangladesh July 26-August 13, 2006 Aliza Inbal Kattie Lussier 29 4 28 4 15 Burkina Faso April 30­May 13, 2006 Aliza Inbal Yacouba Yaro 83 8 14 4 22 Mexico September 16­28, 2006 Kattie Lussier Sue Berryman 16 28 -- 13 Tunisia April 29­May 14, 2006 Mirafe Marcos Sue Berryman 8 21 3 10 Source: World Bank. process and features of the workplace capacity · Training-needs assessment--the extent to which context that helped or hindered the success of course design included identification of pres- the training. Details on the dates of the country ent capacities of potential trainees and the missions and individuals interviewed in govern- specific course content that would be neces- ment, donor agencies, program staff, and civil sary to bring trainees from their present ca- society are given in table A.1.2 pacity levels to those defined in course objectives, and Programs were rated according to the following · Strategic participant selection--the extent to criteria: which participant selection was linked to the development capacity objectives to be achieved Training results through training.3 The criteria used to assess success of training programs were Training follow-up The extent to which training participants received · Learning outputs--achievement of stated learn- any technical assistance or other forms of follow- ing objectives, up support necessary to facilitate transfer of · Behavior outcomes--effect of training on work- learning to the workplace environment. place performance, and · Impact on development capacity--impact of such Training context behavior change on organizational or institu- tional capacity. · Client commitment to training objectives--the ex- tent to which clients believed training objec- Training targeting tives to be important for the achievement of their capacity-building goals, and · Diagnosis--the extent to which the decision to · Attention to capacity context-- the extent to pursue training and the selection of training ob- which institutional and organizational con- jectives were based on an adequate assess- straints that are likely to impact application ment of capacity gaps and the appropriateness and sustainability of learning were addressed. of training as a means to address those gaps, Specifically, the following aspects of capacity · Client involvement in diagnosis--the extent to context were examined: which clients were meaningfully involved in di- (i) The sequencing of training vis-à-vis related agnostic exercises, organizationalandinstitutionalinterventions; 60 APPENDIX A: METHODOLOGY (ii) Incentives for individuals to learn and to of a project component. To estimate the volume apply their learning in the workplace; and of project-based client training, the review (iii) Attention to the sustainability of training generated a random sample of 179 projects from through the building up of local training the total of 1,129 completed investment projects capacity, where appropriate. that were exited between 2002 and 2006.6 The Implementation Completion Report (ICR) for Clarity of objectives each randomly selected project was reviewed to Training workshops or courses were assessed for assess the actual amount of Bank support for the extent to which their objectives specified client training. In cases where it was not possible to isolate training costs because they were a small · The knowledge, skills, and attitudes trainees aspect of a much larger project component, the were expected to learn through the training; projects were tagged as having no clear informa- · The performance change the learning was or tion and, instead, another project with clearer is expected to generate; and information on training was used. Thus, the 179 · The existence of related key performance projects included in this analysis represent only indicators. those projects that provide sufficient informa- tion to estimate training costs. Monitoring and evaluation The extent to which training design included use The review estimated the total amount spent on of well-designed participant satisfaction (level 1)4 training activities for the 179 projects between evaluations to inform future training, evaluations 2002 and 2006 to be $773 million ($155 million a of learning (level 2), and evaluations of the effect year). The total cost of the 179 projects was $12.9 of training on workplace behavior (level 3).5 billion ($2.58 billion a year). There was large variation in training, as a share of project cost The field study reports on this evaluation's Web ranging from 0 percent to as high as 80 percent. site give ratings for all training components by The review used the median of 5.7 percent and criterion. the share of 6 percent as the best average estimate to obtain the range of lending amounts B. Portfolio review to training. Extrapolating the sample to annual The portfolio of Bank investment operations was Bank lending for investment projects between analyzed to identify patterns and trends in the 2002 and 2006, the review estimated that Bank Bank's support of training, including (i) estimat- projects invest between $642 million (the ing the amount of project-based client training, median) and $676 million (the share) a year, on and (ii) evaluating the extent to which training average, on client training. design, objectives, and results are detailed in core program documentation. Each of these is A sectoral analysis of the 179 projects reviewed discussed below. In addition, the evaluation suggested that health, nutrition, and population reviewed the Quality Assurance Group's quality- had the highest share of training costs relative to at-entry and supervision-assessment guidelines the total cost of the project.7 Adaptable Program for assessing capacity building and its applicabil- Loans devote the highest identifiable percentage ity to training, and did a quantitative analysis of of project costs to training investments. It was WBI client training programs on variables, not possible to determine the significance of including the number of training participants, these results because they may simply reflect the course duration, and training in priority areas. clearer delineation of training expenditures and training components in Bank documents in Estimation of the amount of project-based some sectors andor instruments. The number of client training projects within each sector and instrument type Most projects do not explicitly identify the costs was also too small to draw any general of training, especially where training is one part conclusions. 61 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Estimation of the amount of WBI client training supportive of all components; moderate The 2006 WBI annual report notes that the WBI training content when a project included training spent about 85 percent of its budget of $71.2 that was supportive of some of the components; million in fiscal 2005 on client training and and negligible training content when training related capacity-building activities. The evalua- was not financed by the project or was support- tion reviewed WBI data on client training courses ive of no more than one training component. conducted during fiscal 2002­06.8 As noted in Project assessment documentation was reviewed table A.2, the WBI conducted nearly 3,500 for all projects with high training content in courses and trained about 287,000 stakeholders order to determine the range and specificity of from various sections of society during fiscal training goals. 2002­06. Government officials and academics, together, accounted for over 70 percent of A random sample of 38 closed projects was then training participants. generated from all the projects with high training content in countries selected for desk Desk review of projects with high training review. Project appraisal documents (PADs) and content in WBI courses ICRs of these 38 projects were reviewed to The methodology for selecting projects with assess the extent to which the Bank captured high training content for review was as follows: a expected training output, outcome, and impact sample of 13 countries with significant project- objectives at design, and reported subsequently financed training was compiled to represent a on training results. Only 5 projects out of 38 range of borrower sizes, institutional conditions, reported on training outcomes. The review also and regions. These 13 countries are Azerbaijan, assessed the extent to which PADs reported on Bangladesh, Bosnia and Herzegovina, Burkina the planned use of a training-needs assessment, Faso, Guatemala, Lao People's Democratic or on participant selection strategies or follow- Republic, Mexico, Nigeria, the Philippines, Sri up technical assistance accompanying the Lanka, Tunisia, Turkey, and Uganda. Projects training. The findings of this review are approved in these countries between fiscal 1995 presented in appendix D. A similar review was and fiscal 2005 were then reviewed for training completed of a random sample of 61 WBI content. Activity Initiation Summaries (AISs) for selected WBI client training programs conducted in fiscal Training content was rated on a scale: high, 2006, in order to determine if there was moderate, and negligible. Projects were rated as adequate information in the AIS on the design having high training content when the training of training and training objectives. The findings was integral to the achievement of the goals of of this review are presented in appendix G. more than one component, or when training was C. Literature review The evaluation commissioned a review of the Table A.2: Professionals Trained through WBI extensive literature on the attributes of success- Courses, FY02­06 ful training, to guide the evaluation's measures for success and metrics, for the Bank's support Professionals Number of trainees for client training. A summary of the findings of Government ministers 1,368 the literature review is provided in appendix B. Parliamentarians 5,124 Government staff 121,020 D. Benchmarking review Academics 81,703 The training policies and practices of six organi- Journalists 4,788 zations were reviewed to provide a basis for NGO staff 27,735 benchmarking Bank-funded training and to Private sector employees 45,094 enable the Bank to learn from best-practice Source: IEG calculations based on data provided by WBI. models in use outside of the World Bank Group. 62 APPENDIX A: METHODOLOGY The six organizations studied were the Interna- Mexico, Nigeria, and Tunisia). The survey aimed tional Monetary Fund (IMF) Institute, the to assess the impact of selected World International Training Centre of the International Bank­funded training activities on individual job Labour Organization (ITCILO), Motorola Univer- performance and, by extension, on clients' sity of the Motorola Corporation,9 Germany's development capacity. The survey targeted Internationale Weiterbildung und Entwicklung people who had participated in either WBI (InWEnt), Japan International Cooperation training courses or World Bank project-funded Agency (JICA), and the Mount Carmel Training training courses between July 2004 and July 2006. Center of the Israeli Center for International The results of the survey are summarized in Cooperation (MASHAV). These six organizations appendix C. In Tunisia alone, only 40 percent of were selected to represent a broad range of the interviews were completed, due to adminis- structures and missions. Two represent the trative difficulties. training arms of multinational donor organiza- tions (the IMF Institute and ITCILO); Motorola F. Survey of Bank task team leaders University is the training arm of a private sector The evaluation interviewed 43 Bank Task Team multinational corporation; and the last three are Leaders (TTLs) about their views on the training arms of the foreign aid agencies of resources and support they receive to help them individual countries (Germany, Japan, and design, supervise, and evaluate training activities Israel). in the projects they manage. In addition, TTLs whose projects are located in WBI focus The mission of Motorola University is to improve countries were asked about their experience the productivity of the corporation by training its with the WBI. The results of the survey are staff. The primary mission of the other five presented in appendix E. training organizations is to build capacity within developing countries. G. Survey of WBI-partner training institutes The evaluation contracted an international The data for four of the six were collected research firm to survey 30 training institutes through face-to-face interviews. Telephone worldwide in order to assess the collaboration interviews were used to obtain information from between the World Bank and its training partners, JICA and MASHAV. and to assess the Bank's impact on the capacities of these partners. The firm randomly conducted E. Six-country survey of training participants telephone interviews with 30 institutes in 21 The evaluation commissioned an independent countries, from a list of 200 contacts (provided by international survey research company to survey the Bank) at training institutes with which the former participants in World Bank­funded WBI either currently partners or has recently training and WBI training programs in six partnered to organize or cofinance courses. The countries (Azerbaijan, Bangladesh, Burkina Faso, results of the survey are detailed in appendix F. 63 APPENDIX B: ASSESSING TRAINING FOR SUCCESSFUL CAPACITY DEVELOPMENT: LITERATURE REVIEW SUMMARY This summary presents the main findings of a starting point is a stakeholder analysis, which literature review exploring the different stages helps identify the best sources of information of the training process. Works included in the regarding training needs.3 Informants can report literature review are indicated by an asterisk in on whom they would like to see trained, what the bibliography. The full literature review is type of training is needed, or even who could available on this evaluation's Web site at potentially be a trainer. Involving many http://www.worldbank.org/ieg. stakeholders allows the information to be triangulated and avoids bias caused by too much Design focus on any one type of respondent. Designing training begins with a thorough diagnosis and training-needs assessment. Based Training has two sets of objectives. First, the on information obtained from the assessment, general objectives or aims, encompassing the appropriate training objectives can be set, changes expected to take place, or competencies criteria for participant selection established, and that will improve. The second are learning training content decided. objectives, describing what participants should be able to do at the end of the course. Good Correct diagnosis of training needs should both learning objectives derive from the results of the identify organizational and/or institutional training-needs assessment, and cover all the capacity gaps to be targeted and the best means different areas of learning the training activities of addressing these gaps. The human, resource, have to offer.4 Learning objectives should be and incentive dimensions of capacity should be timely and context-specific, measurable, achiev- considered in order to determine whether able, and realistic.5 training is needed, and what other forms of capacity support are needed, in order to facili- The training-needs assessment usually provides tate the implementation of training. The target the necessary information for selecting partici- organization should play an active role in pants or identifying the target group. Participant diagnostic exercises, both to better identify selection, including the number of people to be capacity problems and to ensure client commit- trained, as well as how diverse they are, must be ment to training goals.1 linked with course objectives and the context in which change is taking place.6 The levels and Once diagnosis has determined the capacity backgrounds of the participants help determine needs to be addressed by training, a training- the appropriate depth and level of detail of the needs assessment provides specific information training. Furthermore, task and job analyses help on what participants need to learn, the issues prioritize learning needs, in order to assign they face, and what is expected from the training. proper time allocation during training delivery. When done well, such an assessment ensures This aggregated information should, in turn, that training addresses the most relevant issues influence the method in which knowledge and efficiently.2 Steps and procedures to conduct skills will be imparted: face-to-face communica- training-needs assessments vary. A preferred tion, interactive exercises, lectures, etc. 65 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT It is also important that content areas be properly Capacity development occurs once participants sequenced, so as to ensure linkages among transfer what they have learned to their everyday different topics, and to enhance learning. jobs, and improve their performance. The impact and sustainability of training is the successful Implementation of Training Activities application of new skills and knowledge. There is no single method for teaching or learning--the trainer must choose the strategy Major obstacles to the transfer of new skills and that best fits the needs and backgrounds of the knowledge tend to be the types of power participants, the facilities and equipment structures within an organization, entrenched available, and the learning objectives. The follow- attitudes, and lack of resources. ing six principles provide useful guidelines for successful adult learning.7 To increase the likelihood of transfer, factors such as the timing of the training (that is, · Participation: People learn better and remem- sequencing it or conducting it when it is most ber more when they are actively engaged in the needed), feedback to the participants, practical learning process. Practical exercises, discus- application of the training (with numerous real- sions, simulations, and games also prompt par- life and relevant examples), and support from ticipants to share their knowledge and supervisors are essential. experiences, thus fostering a collaborative environment. Follow-up · Understanding: Checking participants' under- Follow-up activities assist in the transfer of standing by asking questions, listening to their training to the workplace, by helping to address discussions, or using a vocabulary appropriate any barriers to the working environment, and by to their level (as determined by the training- maintaining participants' motivation to apply needs assessment) can all improve the learn- new learning. Follow-up, through technical ing process. assistance or supervision, can help trainees adapt · Feedback: Providing opportunities to practice the learning to real-life situations.8 and giving trainees specific feedback informa- tion on the quality of their work gives partici- Furthermore, follow-up can contribute to self- pants a sense of their progress toward the efficacy (confidence in one's ability to perform learning objectives. successfully) and motivation, two factors shown · Interest: The relevance of the training objectives to impact skill maintenance.9 Trainees experienc- to the needs of the participants is crucial in de- ing difficulties in applying new learning to their veloping the participants' interest and owner- work should have the opportunity to seek advice ship of the training. Providing various case immediately after the training. studies and activities will help stimulate and maintain interest. Evaluation · Emphasis: Focusing the attention of partici- Accountability and decision making are the most pants on the most important points of the cited reasons for evaluating training. Feedback training, by spending more time on important evaluations (or formative evaluations) help topics and allowing sufficient time to practice monitor the quality of design and delivery of new skills, will help them remember the key training. They provide information on the lessons from the training. effectiveness of the methods used, the achieve- · Results: Keeping participants informed of their ment of the training objectives, or the appropriate- progress, and recognizing them for work well ness of management and training design. Decision done will increase their confidence in newly ac- making evaluations focus instead on the value and quired skills and knowledge, and encourage contribution of the training for an organization or them to use what they have learned. a project. These evaluations assist managers, team 66 APPENDIX B: ASSESSING TRAINING FOR SUCCESSFUL CAPACITY DEVELOPMENT leaders, and donors in making informed decisions Conclusion regarding training activities. Other types of evalua- The process of training preparation for capacity tions are input evaluations and process evalua- development begins with a diagnosis, followed tions. An input evaluation shows how resources by a training-needs assessment. Based on the were employed and whether they were adequate information collected on participants' needs and for achieving the goals of the program. A process background, learning objectives and content can evaluation examines why certain objectives were be set and developed during the training design achieved and others were not, and where improve- phase. The training process can be enhanced ments are needed.10 through participation, checking the understand- ing of the participants, giving feedback to partici- There are several evaluation "levels," ranging pants, stimulating their interest, emphasizing the from simply providing information on partici- important lessons, and focusing on results. Also, pants' reactions to the training--the effectiveness following up on the training through supervision of the instructor, the relevance of the content, and on-the-job support helps adapt the training the quality of the materials--to measuring the lessons to real-life situations, thereby helping to outcomes or results of the training program.11 ensure long-term impact. Finally, evaluations are a necessary step that can provide valuable Several tools can be used to evaluate training. information on how to improve future training. These include evaluation sheets, questionnaires, and interviews, as well as direct observation of training results in the workplace. 67 APPENDIX C: SUMMARY OF CLIENT SURVEYS IN SIX COUNTRIES The evaluation commissioned an independent In each country, the firm surveyed approxi- research company to survey former participants in mately 100 respondents through face-to-face or World Bank­funded training and WBI training telephone interviews, conducted with roughly programs in six countries (Azerbaijan, Bangladesh, equal proportions of WBI and project partici- Burkina Faso, Mexico, Nigeria, and Tunisia). The pants, as shown in table C.1. In all, the survey survey aimed to assess the impact of selected firm successfully interviewed 548 respondents. World Bank­funded training activities on individ- The response rates were as follows: Azerbaijan, ual job performance and, by extension, on clients' 30 percent; Bangladesh, 21 percent; Burkina development capacity. Faso, 64 percent; Mexico, 11 percent; Nigeria, 27 percent; and Tunisia, 47 percent. The average of The survey targeted people who had participated these response rates is 33 percent. (The overall in either WBI training courses or Bank project- response rate for Mexico was unusually low funded training courses between July 2004 and because the survey firm found that a large share July 2006. Participants in WBI programs were of the WBI participants did not remember the selected randomly from participant lists provided course, could not be reached when called, or by the WBI. For project-based training, the Bank declined to be interviewed.) The survey instru- provided the firm with course details and relevant ment is included at the end of this appendix. contact people (for example, training organizers, Because the number of respondents surveyed in Bank task managers, government representatives each individual course varied significantly, involved in planning the training), and the firm results were weighted by number of respon- was responsible for obtaining lists of past partici- dents per course to give equal weight to each pants and targeting some of them at random. course. Table C.1: Respondents by Type of Training and Interview Technique WBI participants Project participants Country (# of interviews) (# of interviews) Language Face-to-face/telephone Azerbaijan 54 50 Azeri/Russian Face-to-face/telephone Bangladesh 58 42 English Telephone Burkina Faso 60 40 French Face-to-face Mexico 62 41 Spanish/English Telephone Nigeria 60 40 English Face-to-face/telephone Tunisia 13 26 French Face-to-face/telephone Total 307 241 Source: IEG six-country survey of training participants. 69 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Results quality of lectures to the appropriateness of time Results were analyzed according to the type of allocated to cover course topics. Overall, respon- training received--either WBI or project dents were positive on most aspects of their training. The results do not include 37 respon- training, finding courses to be interesting and the dents (7 percent) in the sample who left their lectures to be of high quality. Participants noted jobs within six months after the training or who room for improvement, especially the time reported that the training was not relevant to allocated to cover course topics, the number of their current job. practical exercises and projects during the training, the mix of expertise levels among the Training success: For the purposes of this survey, participants, and the availability of resources to successful training was defined as that which led implement the training. to substantial positive changes in the way partic- ipants perform key functions of their work. A principal-components analysis was then Training success was defined as positive changes conducted on this data to better understand the in workplace behavior on the basis of the specific factors that most contribute to partici- assumption that participants would not be able pants' perceptions of successful training.1 Ten to speak of the impact of their training on their drivers of program success were derived statisti- organizations as a whole with any degree of cally using data from all six countries. The drivers reliability. In addition, participant self-reports and the corresponding attributes that comprise are unlikely to be entirely accurate on perform- them are detailed in table C.3. ance change, and there is a danger of positive bias in participant self-reporting on their own Because a driver is comprised of several different behavior. Nonetheless, a participant survey was attributes, the survey computed a performance found to be a useful as: (i) a means of analysis of score for each driver by aggregating the respon- both the prevalence of certain practices in Bank- dent ratings on each of the driver's individual financed training, such as the use of follow-up attributes.2 Four drivers were consistently among instruction or practical exercises in training, and the lowest performing, across countries and type (ii) a source of information on the impact of of training--course time allocation, practical these process factors on behavior change. learning, participant mix, and material resources at workplace. Fifty-five percent of all respondents surveyed noted that the training resulted in substantial The importance of each of the drivers was positive changes to primary work functions. In derived through discriminant function analysis.3 addition, a quarter of the respondents noted The discriminant function analysis indicated that small positive changes in their primary work for most training participants, the more they feel functions. Those who participated in project they are supported and encouraged by their training were somewhat more likely than those managers and colleagues to apply their training, who participated in WBI training (60 percent and and have the resources available to apply their 50 percent, respectively) to agree that the learning, the more likely they are to state that training resulted in substantial positive change. training led to substantial change in their work. When probed on how their work had changed, 75 percent were able to cite, in at least general The importance and performance of the drivers terms, how training had contributed to their enable comparisons across drivers (see table work. Table C.2 details participants' responses to C.4). The quadrant analysis suggests that the the question of how their work had changed. Bank needs to pay greater attention to strategic participant selection and to ensuring the Drivers of success: Respondents were asked to resources for implementation of the skills and rate a number of statements on the delivery and knowledge from training (high importance, low administration of training, ranging from the performance). 70 APPENDIX C: SUMMARY OF CLIENT SURVEYS IN SIX COUNTRIES Table C.2: How Participants' Work Changed after Training Six-country average WBI Project Knowledge acquisition Acquired/enhanced knowledge 25 28 24 Learned from other country's experience/other participants 3 5 2 Skills acquisition Acquired new skill/technique 17 14 21 Consquences of Implemented new project/program/policy 17 14 20 knowledge/skill: action Shared info/trained others at work 13 16 11 Improved productivity/efficiency/quality 13 9 18 Improved job efficacy 11 12 11 Applied new tools/skills to work functions 9 6 13 Applied new knowledge to work functions 8 6 10 Improved communication with colleagues/clients 7 9 6 Adapted internal processes 7 5 10 Improved time management/organization 5 3 6 Improved problem solving 4 5 3 Promotion/new responsibilities 3 3 3 Consequences of Broadened understanding of country/issue/self 12 19 3 knowledge/skill: raised Improved understanding of work context 7 11 3 awareness Improved understanding of client needs 5 6 4 Broadened professional network 4 7 1 Source: IEG six-country survey of training participants. Table C.3: Drivers of Program Success Driver name Driver attributes Course targeting The level of course was appropriate for a person with my experience and knowledge. The course content specifically addressed my country's circumstances. The course content addresses issues that are important to my work. Course quality The lectures were good quality. The course content was interesting. The course was in a language I am fluent in. Participant input/feedback I was given the opportunity to provide feedback on my satisfaction with the course. The course organizers asked me to share with them my needs or objectives in the course, either before the course or at its start. Practical learning I was given course materials (schedule and/or learning materials) before the course start date. The course devotes significant time to practical exercises or projects. Participant mix/interaction Course participants had about equal levels of knowledge/experience coming into the course. I learned from the experience of other participants in the course. Course time allocation The course covered the right amount of topics for the amount of time allotted. Organizational support My colleagues provided me with the support I need to apply what I learned in the course. My managers encourage my efforts to use what I learned in the course. Material resources at workplace I have the resources (for example, equipment, software) to apply what I learned. Instructor follow-up Communication with course instructor (either online or `other'). Participant follow-up Communication with course participants (either online or `other'). Source: IEG six-country survey of training participants. 71 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Table C.4: Importance and Performance of Drivers Low High WBI Project capacity capacity Course quality Course targeting Organizational support Instructor follow Participant follow Participant needs assessment/feedback Course time allocation Practical learning Participant mix/interaction Material resources at workplace Key: High performance/high importance High performance/low importance Low performance/low importance Low performance/high importance Source: IEG six-country survey of training participants. 72 APPENDIX C: SUMMARY OF CLIENT SURVEYS IN SIX COUNTRIES World Bank Training Evaluation Questionnaire FIELD REGISTRATION INFORMATION ­ TO BE COMPLETED OR CODED BY OR UNDER SUPERVISION OF FIELD MANAGER OR SUPERVISOR RECORD FOR ALL RF1 Unique respondent ID RF2 Unique interviewer ID RF3 Unique supervisor ID RF9 Was this interview controlled 1. Yes 2. No FIELD REGISTRATION INFORMATION ­ TO BE COMPLETED BY INTERVIEWER WITHOUT ASKING THE RESPONDENT RECORD FOR ALL: RI1 Interview date (day/month/year) RI2 Interview start (enter hour and minutes using 24-hr clock; e.g. 21:09) RI3 Interview end (enter hour and minutes using 24-hr clock; e.g. 21:09) RI4 Respondent name RI5 Name of course attended (included in contact information) RI6 Length of course attended (number of days, included in contact information) RI7 End date of course (day/month/year, included in contact information) RI8 Number of course participants in course attended (included in contact information) RI9 WBI or non-WBI respondent 1. WBI 2. non-WBI RI10 WBI product line [insert options here] Introduction Suggested introduction only, may be adapted by field manager as appropriate. Good morning/afternoon/evening. My name is ______________ and I am here/calling from , an independent re- search firm. We have been engaged by the Independent Evaluation Group at the World Bank to ask you about your experience as a participant in training sponsored by the World Bank. We're interested in hearing your thoughts about the training, whether you learned from it and, if so, whether that learning helped you to do your job better. Our conversation will take about 15 to 20 minutes. Please be assured that anything you say will be kept totally confidential and anonymous. Your name was selected randomly from course participant lists. The Independent Evaluation Group of the World Bank has not been given a copy of our sample list and your name will not be released to them. Our report to them will be in aggregate form only. We want to be sure that you feel comfortable speaking freely and candidly about your opinions and experiences. Your candid feedback--that is, both your positive comments and your thoughtful criticism--will help the World Bank understand how to make its training more useful to participants. Would you be willing to participate in this interview? IF NO, THANK AND TERMINATE. (Continues on the following page.) 73 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Before we begin, I would like to ask you a few preliminary questions. Screening 1. I understand that you attended a course called - IS THIS CORRECT? REMIND RESPONDENT OF COURSE DATE IF NECESSARY (RI7). 01 Yes 02 No - THANK AND TERMINATE 2. Was the course part of a series of related courses that you took, or did it stand alone as a single course? 01 Part of series ­ READ THE FOLLOWING TO THE RESPONDENT: "For this interview, please answer our questions for the series of courses that you took and rate them together if you can." 02 Single course 3. How was the course conducted? INTERVIEWER TO READ OUT OPTIONS AND CHECK ALL THAT APPLY. 01 In person, in a classroom 02 Internet/online 03 Video conferencing 04 Study tour VOLUNTEERED (DO NOT READ) 97 Other (specify): ________________________ Thanks. Let's talk now about your experiences in the course and how it relates to your work: Course details 4. Can you tell me a little about what was covered in the course? VERBATIM NOT NEEDED; INTERVIEWER SHOULD NOTE AND UNDERSTAND WHAT THE COURSE COVERED. 5a. After you were trained, did you remain in the same job for at least 6 months? 01 Yes ­ READ THE FOLLOWING TO THE RESPONDENT: "Please answer all remaining questions thinking of the job you had 6 months after completing your training." [GO TO 6A] 02 No [GO TO Q5B] [ASK ONLY THOSE WHO ANSWER 02 in Q5A]: 5b. Is the training you received relevant for your present job? 01 Yes ­ READ THE FOLLOWING TO THE RESPONDENT: "Please answer all remaining questions thinking of your present job." [GO TO Q6A] 02 No [GO TO Q8 AND THEN TO Q15 - PROFILING] 74 APPENDIX C: SUMMARY OF CLIENT SURVEYS IN SIX COUNTRIES Overall impression of the course ASK ALL 6a. The following is a list of possible statements on the impact of the course on your work. Which one most accurately describes your experience? 01 The course resulted in substantial positive changes to the way I perform key or primary functions of my work. [GO TO 6B] 02 The course resulted in small positive changes to the way I perform key or primary functions of my work. [GO TO 6B] 03 The course resulted in positive changes to the way I perform non-key or secondary functions of my work. [GO TO 6B] 04 The course resulted in little or no change to my work. [GO TO 6C] 05 The course resulted in negative changes to the way I do my work. [GO TO 6B] VOLUNTEERED (DO NOT READ) 99 Don't know / no answer [GO TO Q6C] [ASK ONLY THOSE WHO ANSWER 01, 02, 03, or 05 in Q6A] 6b. Can you give me some examples of how your work has changed? RECORD VERBATIM RESPONSE. Probe for several examples of what respondent thinks is different as a result of taking the course. You may find it useful to use probes which ask for further details on the respondent's job responsibilities, on the content of the course, or on his workplace environment. [ASK ONLY THOSE WHO ANSWER 04 OR 99 in Q6A]: 6c. Can you please explain why you say [either "little / no change to my work" or "don't know"]? INTERVIEWER TO BRIEFLY SUMMARIZE RESPONDENT'S ANSWER ON Q6B OR Q6C, THEN ASKS: ASK ALL 6d. On the basis of what you told me, I'd like to confirm the impact of the course on our work. So, using the same statements from the previous question, which one most accurately describes your experience? Please feel free to give the same answer. 01 The course resulted in substantial positive changes to the way I perform key or primary functions of my work. [GO TO Q8] 02 The course resulted in small positive changes to the way I perform key or primary functions of my work. [GO TO Q7] 03 The course resulted in positive changes to the way I perform non-key or secondary functions of my work. [GO TO Q7] 04 The course resulted in little or no change to my work. [GO TO Q7] 05 The course resulted in negative changes to the way I do my work. [GO TO Q7] VOLUNTEERED (DO NOT READ) 99 Don't know / no answer [GO TO Q8] (Continues on the following page.) 75 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT ASK ONLY THOSE WHO ANSWERED 02, 03, 04, OR 05 IN Q6D 7. Which one of the following reasons best describes why you feel the course did not have greater impact on your day-to-day work? READ STATEMENTS. CODE ONE ONLY. 01 I didn't gain significant new knowledge or skills in the course. 02 I did gain significant new knowledge and skills, but they were not very relevant to important aspects of my work 03 The course content was relevant to my work, but I did not know how to apply what I had learned to my job. 04 I knew how to apply what I had learned, but I did not have the necessary resources or support to do so. VOLUNTEERED (DO NOT READ) 97 Other (specify) _________________________________ Rating specific aspects of course and organizational environment ASK ALL ENSURE THAT THOSE WHO SAY 02 IN Q5B ARE ONLY ASKED THIS QUESTION AND THEN SKIP TO Q15. 8. For the next set of questions, I'd like you to think about the course that you took and rate your level of agreement. For each statement, please tell me whether you strongly agree, agree, disagree, or strongly disagree. I would like to note before I begin reading these statements that some will be positive and other statements will be negative. READ AND ROTATE STATEMENTS ENSURE FIRST STATEMENT READ IS POSITIVE a. I learned from the experience of other participants in the course. 01 Strongly disagree 02 Disagree 03 Agree 04 Strongly agree VOLUNTEERED (DO NOT READ) 05 Not applicable 99 Don't know / no answer b. Course participants had about equal levels of knowledge/experience coming into the course. c. The course did not devote significant time to practical exercises or projects. d. The course content specifically addressed my country's circumstances. e. I was not given course materials (schedule and/or learning materials) before the course start date. f. The course organizers asked me to share with them my needs or objectives in the course, either before the course or at its start. g. The course was in a language I am not fluent in. h. The level of the course was appropriate for a person with my experience and knowledge. i. The course content was not interesting. j. The course content did not address issues that are important to my work. k. The course covered too many topics for the amount of time allotted. l. I was given the opportunity to provide feedback on my satisfaction with the course. m. The lectures were not of good quality. 76 APPENDIX C: SUMMARY OF CLIENT SURVEYS IN SIX COUNTRIES The next question set refers to your experiences in trying to apply what you learned on the job. 9. First of all, have you tried to apply what you learned on the job? 01 Yes [GO TO Q10] 02 No [GO TO Q11] ASK ONLY THOSE WHO SAY 01 IN Q9 10. For the next set of statements that I read, I'd like you to think about your organization and your work since completing the course. For each statement, please tell me whether you strongly agree, agree, disagree, or strongly disagree. Again, please note that some statements I read will be positive and other statements will be negative. READ AND ROTATE STATEMENTS. FIRST STATEMENT SHOULD BE POSITIVE. a. My managers encourage my efforts to use what I learned in the course. 01 Strongly disagree 02 Disagree 03 Agree 04 Strongly agree VOLUNTEERED (DO NOT READ) 97 Not applicable 99 Don't know / no answer b. My colleagues do not provide me with the support I need to apply what I learned in the course. c. I have the resources, (e.g., equipment, software) to apply what I learned. d. Policies at my organization allow me to apply what I learned. e. I have trouble understanding how to apply at work the theories that I have learned in the course. ASK ALL 11. Since completing the course, have you had any follow-up instruction or contact with the people who ran or attended the course? 01 Yes [GO TO Q12] 02 No [SKIP TO Q14] ASK ONLY THOSE WHO SAY 01 IN Q11 12. What type of follow up have you had? Interviewer to read out options and check all that apply. 01 Communication with the course instructor through an Internet forum or email listserve 02 Communication with course participant(s) through an Internet forum or email listserve 03 Other communication with the course instructor 04 Other communication with course participant(s) 05 Technical assistance on-the-job 06 Additional course(s) that built on the learning from this one VOLUNTEERED (DO NOT READ) 97 Other (Specify):_____________________________ (Continues on the following page.) 77 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT ASK ONLY THOSE WHO SAY 01 IN Q11 13. Please rate the extent to which the follow up has helped you apply what you learned in the course in your day-to-day work. Would you say it was READ STATEMENTS, CODE ONE ONLY. 01 Not helpful 02 Somewhat helpful 03 Very helpful 04 Essential VOLUNTEERED (DO NOT READ) 99 Don't know / no answer ASK ALL 14. Is there anything else you would like to share with me, about the course you took, how it was organized, followed up, or the work environment in which you applied the course content? RECORD VERBATIM IF RESPONDENT UNABLE TO MENTION ANYTHING, PROBE: "Perhaps you'd like to talk about the positive or negative aspects of the course?" INTERVIEWER MIGHT ALSO REMIND RE- SPONDENT ABOUT ITEMS DISCUSSED IN QUESTION 10 AND ASK THE RESPONDENT "Perhaps you can explain some of your answers in Question 10 a little further?" ASK ALL Profiling 15. Which of the following best describes your level of responsibility within your organization? 01 I am self-employed 02 Head of the organization (e.g. Minister, Director, CEO, etc.) 03 Management (e.g. departmental manager, vice-president, project manager, etc.) 04 Professional / technical / research VOLUNTEERED (DO NOT READ) 97 Other, specify:_____________________________ 16. Education. ASK EDUCATION LEVEL AS YOU WOULD NORMALLY DO IN YOUR COUNTRY AND THEN RE-CODE AS FOLLOWS FOR GLOBESCAN. Re-code list: 01 - No formal education / cannot read or write 02 - Some elementary school 03 - Completed elementary school 04 - Some high school / secondary school 05 - Completed high school / secondary school 06 - Some college / university 07 - Completed university or equivalent / university degree/diploma 08 - Post-graduate degree 99 - Don't know / no answer 78 APPENDIX C: SUMMARY OF CLIENT SURVEYS IN SIX COUNTRIES Thank you very much for your time and input! INTERVIEWER TO COMPLETE AFTER THE INTERVIEW: 17. Gender NOTE, DO NOT ASK RESPONDENT 01 Male 02 Female 18. Country CHOOSE ONE ONLY 01 Azerbaijan 02 Bangladesh 03 Burkina Faso 04 Mexico 05 Nigeria 06 Tunisia 19a. Which of the following best describes respondent's location: 01 City 02 Town 03 Village 19b. Record name of respondent's location (e.g., name of the city, town, or village): ____________________________________________ 79 APPENDIX D: MONITORING AND EVALUATION OF TRAINING OUTCOMES IN CLOSED BANK PROJECTS This evaluation reviewed project appraisal completed at least a minimal training-needs documents (PADs) and Implementation assessment in the design phase. Only seven Completion Reports (ICRs) of 38 randomly projects noted plans for follow-up technical selected projects with high training content, assistance to ensure transfer of learning to the from 13 desk-review countries, in order to assess workplace. the extent to which Bank documents specified training output, outcome and income goals Results of Training during project design, and subsequently Nearly all (35 out of 38) ICRs reported on reported on training achievements and impacts. numerical outputs for at least some of the project The review also assessed the extent to which training. Seventeen reported on all numerical PADs reported on the use of a training-needs outputs listed in the PAD, and 18 reported on assessment, or follow-up technical assistance some of them. Only 3 ICRs did not mention accompanying the training. numerical outputs at all. Conversely, few ICRs reported on the achievement of learning Preparing for Training objectives, or on the actual outcomes of the Twenty-seven out of the 38 projects contained training in terms of changes in workplace details on numerical training output targets. Ten behaviors. Less than half (16 out of 38) reported set numerical targets for all planned taining, and on either training outputs (learning) or 17 for some training components. All training outcomes (workplace behavior). Of these 16 mentioned in PADs was within the context of projects, 6 reported on the achievement of project components and had well-elaborated learning objectives. However, there are no institutional or organizational capacity-building means of verification for this information as none goals, but only five of the PADs specified how of the ICRs report on whether trainees had been training was supposed to contribute to these goals tested for the knowledge and skills they were by establishing desired training outputs or said to have gained. Only 10 out of the 38 posttraining performance indicators. Most projects identified behavior changes in the projects (22 out of 38) had planned or already workplace (outcomes). 81 APPENDIX E: SUMMARY OF INTERVIEWS WITH BANK TASK TEAM LEADERS The evaluation interviewed 43 Bank Task Team Leaders (TTLs) about resources and support that Table E.1: Characteristics of TTLs they currently have or need to help them design, supervise, and evaluate training activities in their Manage projects in WBI focus countries 28 projects. The 28 TTLs whose projects were Manage projects in nonfocus countries 15 located in one of the World Bank Institute (WBI) Located in Washington, DC 30 focus countries (2006) were also asked specifi- cally about their experience with the WBI. In Located overseas 13 total, TTLs from 15 focus countries were Average time at the Bank 12.5 years surveyed. TTLs were selected randomly, based Managed projects with significant client traininga 43 on a list of active fiscal 2006 projects with training a. In all 30 projects, respondents confirmed that client training was part of at least one project com- ponent and necessary to the achievement of the component's objective. components. Thirty of the TTLs surveyed were located in Washington, DC, and 13 were located (almost two-thirds) said they would find it very in the field. The original survey questionnaire is useful, 8 said they would find it somewhat useful, included at the end of this appendix. and 5 reported that they would not find it partic- ularly useful. Among TTLs, the preferred source Sources of and Need for Training for such advice--chosen from a prepared list of Expertise options--was a dedicated unit of experts. Eleven out of 43 respondents noted that they had not received any technical advice for training. The second-choice answer among TTLs for more Figure E.1 shows that 18 TTLs obtained advice expert training advice was an "other" source from Bank colleagues. Of these, half found the received advice highly satisfactory, while eight found it satisfactory. The second most popular Figure E.1: Sources of Training Advice Used by TTLs choice for training advice was external consult- ants: 16 of the 32 TTLs who requested technical advice obtained it externally. Half of these TTLs total 34 35 found it satisfactory, six found it highly satisfac- 30 tory, and two found it only partially satisfactory. oftuo 25 Only 10 of the 43 TTLs with client training tsn 20 component(s) as part of their projects sought den 15 technical help from the WBI. Five of those TTLs espor 10 found the advice highly satisfactory, three found ofr 5 it satisfactory, and two found it partially mbeu 0 satisfactory. N None External WBI Bank operational colleagues When the 32 TTLs who had received advice for Highly satisfactory Satisfactory Partially satisfactory No help training design were asked whether they would Source: Derived from question 10 in IEG survey of the TTLs. find it useful to have more advice for training, 19 83 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT (indicated by the lighter columns in figure E.2): As figure E.3 shows, most respondents had never 12 TTLs expressed the desire to have a training used the WBI to provide training. expert, preferably a Bank staff member who is knowledgable about Bank processes and organi- Furthermore, as shown in figure E.4, most TTLs zation, and is part of the team throughout the were aware only of some WBI activities in their project, especially during the preparation phase. sector. Several TTLs (most of whom had Two others wanted an up-to-date database that is answered "some") said they were somewhat easily accessible and lists training experts with aware of what the WBI was doing through emails their areas of specialization. that it periodically sends out, but noted that they were often too busy to read them in detail. Some Relationship between WBI and TTLs in stated that the emails were too general, and WBI Focus Countries rarely covered the specific area(s) in which the The 28 TTLs from the focus countries were asked TTLs work. whether they used the WBI to provide training within the context of the project they managed. Figure E.5 shows how TTLs in focus countries rate their level of cooperation with the WBI. Figure E.2: Preferred Options for Additional Training Nine respondents reported they had no Advice cooperation at all. Nine respondents found it satisfactory, and seven found it partially satisfactory. Two respondents found it highly 20 total satisfactory, adding that this was the result of 23 18 excellent personal relationships they had built oftuo 16 14 with individual WBI staff members. When focus tsn 12 countries' TTLs were asked whether they den 10 thought the relationship between the WBI and 8 espor operations in their sector had improved, stayed 6 ofr 4 the same, or worsened over the past two years, 2 most felt it had stayed the same (see figure mbeu 0 N E.6).1 Eight thought it had improved Unit of TrainingWeb-based Training Updated No answersomewhat, and eight stated that they did not experts within for TTLS or written expert on database the Bank guidelines project team of experts know enough to answer. Four noted that they Source: Derived from question 13 in IEG survey of the TTLs. were not aware of any relationship between the WBI and operations in their sector. Figure E.3: Use of the WBI to Deliver Project-based WBI Use of Operational Resources Training in Focus Countries TTLs in focus countries were asked two questions to evaluate how much the WBI draws upon their expertise. total 20 28 18 oftuo 16 First, they were asked whether the WBI had 14 sought their help, as country or sector experts, tsn 12 to design or deliver training that the WBI den 10 8 financed. As shown in figure E.7, most respon- espor 6 dents (19) had never contributed to the design ofr 4 or delivery of the WBI's training programs. The 2 mbeu 0 nine TTLs who had worked with WBI said that N Never Once Occasionally Frequently the nature of their collaboration either involved designing a program, or lecturing for Source: Derived from question 2 in IEG survey of the TTLs. a program. 84 APPENDIX E: SUMMARY OF INTERVIEWS WITH BANK TASK TEAM LEADERS Second, TTLs were asked whether the WBI Figure E.4: TTL Awareness of WBI Activities in Their consulted with them in shaping its training Sectors in Focus Countries priorities in their sectors, in the countries where they work. As seen in figure E.8, most had never 12 been consulted. oftuo 10 tsn 8 Findings den total 6 This survey revealed three broad patterns in the espor 28 4 relationship between the WBI and operations. ofr 2 mbeu First, operational staff needs training expertise, N 0 but TTLs reported that although they see the None Some Many All WBI as a possible source of knowledge on Source: Derived from question 5 in IEG survey of the TTLs. training, they do not know enough about what it does, or how to tap into its resources. Although all respondent TTLs knew that the Figure E.5: TTL Cooperation with the WBI in Focus WBI gives training courses in a range of topics Countries and countries, 24 said they lacked knowledge about what kind of specific activities the WBI 10 does. When TTLs seek training advice, half of tuo them informally consult with Bank colleagues tsn 8 about their training needs, finding colleagues to den 6 be more easily accessible than the WBI. Ideally, total espor 28 4 TTLs would like to have a training expert be part ofr of of their project team, to go on missions with 2 them to help identify training needs and design mbeu N 0 the training program(s). Sixteen of the TTLs No coop- Unsatisfactory Partially Satisfactory Highly that sought training advice made use of external eration satisfactory satisfactory expertise by hiring training consultants. Source: Derived from question 7 in IEG survey of TTLs. However, seven TTLs noted that having external consultants is often rather costly, and budget constraints in the preparation phase, when it would be most useful, often rule out this Figure E.6: TTL Assessment of Relationship with the option. WBI in the Past Two Years Second, the survey found no evidence that 10 operational staff in the WBI's focus countries tuo 8 have a strong relationship with the WBI. Most tsn den TTLs are somewhat aware of WBI activities in 6 total their sector and their countries but do not ask espor 28 4 the WBI to provide training in their projects, and ofr of feel there has been no change in their relation- 2 mbeu ship with the WBI over the past two years. N 0 Neither does the WBI tap into TTL's operational Improved Improved Stayed the Not been Don't know significantly somewhat same working as knowledge and subject area expertise. Most TTLs TTL for 2 in focus countries have never been consulted years about training priorities, and most never assisted Source: Derived from question 8 in IEG survey of TTLs. the WBI in the design or delivery of training Note: The option "the relationship worsened" was not chosen by any TTL, and therefore does not appear in this graph. courses. 85 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Figure E.7: Extent of TTL Input on the Design or Figure E.8: Extent of WBI Consultation Delivery of WBI Training with TTLs on Training Priorities 20 20 18 18 tuo 16 16 tsn 14 tuo 14 den 12 tsn 12 total den espor 10 10 28 total ofr of 8 espor 28 8 of mbeu 6 ofr 6 N 4 mbeu 4 N 2 2 0 0 Never Once Occasionally Frequently No consultation Briefly Intensively Source: Derived from question 3 in IEG survey of the TTLs. Source: Derived from question 6 in IEG survey of the TTLs. Finally, there are opportunities to reach out more and makes us look like we don't know what the to operations: 17 TTLs said that the WBI needs to World Bank does." For the WBI to be relevant to become more involved at the project level, operational work, as TTLs believe it should, it instead of remaining an external, independent needs to be aware of what goes on in operations, knowledge silo. "They make us feel stupid in the and make an effort to tailor its courses to meet field," commented one TTL, "we show up with the training needs of Bank projects. One TTL our training courses for our project, and WBI is suggested that the WBI read operational also there, conducting a similar training but for documents, such as project concept notes, and their own purposes. This confuses our clients present ways in which it could help. 86 APPENDIX E: SUMMARY OF INTERVIEWS WITH BANK TASK TEAM LEADERS Survey of Bank Task Team Leaders for High-Training-Component Projects This survey is in the context of an IEG evaluation of the relevance and efficacy of WBI and project-financed client training. We are interested to hear your views on whether you have the resources and the support you need in order to help design, supervise, and evaluate effective training. In addition, we would like to hear about your experience with WBI. This survey will take about 15 minutes. Please be assured that anything you say will be kept totally confidential and anonymous. Screening questions 1. Have you managed projects where training is part of at least one project component and is necessary to the achievement of that component's objectives? No Ask ONLY Q3­10 and 16­18 (Skip Q11­15) Yes Ask all questions WBI-related questions: First, we'd like to ask you about your interactions, if any, with WBI. 2. Do you ever use WBI to provide training within the context of projects you've managed? a. Never b. Once c. Occasionally d. Frequently 3. In the role of sector expert and/or country expert, have you ever worked with WBI to design or deliver training that it finances? a. Never [skip to question 6] b. Once c. Occasionally d. Frequently 4. Could you describe the nature of your cooperation? You can choose more than one. a. I helped them design the program b. I recommended participants or lecturers for the program c. I lectured in the program d. Other ____________ 5. How aware are you of WBI activities in the sector in which you work? a. All b. Many c. Some d. None 6. Over the past year, has WBI consulted with you in shaping its training priorities in your sector, in the countries in which you work? a. Intensively b. Briefly c. No consultation (Continues on the following page.) 87 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT 7. How would you describe your cooperation with WBI? a. Highly satisfactory b. Satisfactory c. Partially satisfactory d. Unsatisfactory e. Highly unsatisfactory f. I don't have any cooperation with WBI 8. Over the past two years, do you think that the coordination and collaboration between WBI and Operations in your sector have a. Improved significantly b. Improved somewhat c. Stayed the same d. Became worse e. I have not worked as a TTL for two years 9. Do you think that the coordination and collaboration between WBI and Operations could be improved? [IF YES] How could they be improved? ___________________________________________________________________________________ Project-related questions We'd like to now move to the use of training in the context of projects you've managed: 10. Have you received technical advice in designing training components in your project? If so, where have you obtained this advice? a. WBI b. Network for my sector c. Bank colleagues d. My Region's quality team e. Other ______________ f. Have not received technical advice on designing training components [Skip to question 15] 11. How useful was the advice that you obtained? a. Highly satisfactory b. Satisfactory c. Partially satisfactory d. Unsatisfactory e. Highly unsatisfactory 12. How useful to you would it be to have more technical advice on how to design effective training? a. Very b. Somewhat c. Not particularly 88 APPENDIX E: SUMMARY OF INTERVIEWS WITH BANK TASK TEAM LEADERS IF A OR B: 13. What forms of technical advice would you like to have available to you? (Check as many as necessary.) a. Dedicated unit of experts within the Bank on the design and evaluation of training for clients b. Training for TTLs on the design and evaluation of training c. Web-based or written guidelines for effective design and evaluation of training d. Other_________________ 14. Please share any other thoughts that you have about how the Bank can support you in designing and managing effective training in your projects. ________________________________________________________________________________ Demographic information 15. Sector________ 16. Field/HQ based 17. Years in Bank ________ 89 APPENDIX F: SUMMARY OF WBI'S PARTNER TRAINING INSTITUTE SURVEY The evaluation contracted with an international their relationship with the WBI, and described it research firm to survey 30 training institutes as "very rewarding," "highly appreciated," worldwide, in order to assess collaboration "wonderful," "well coordinated," and "very between the World Bank and its training beneficial." Respondents urged the WBI to offer partners, and to assess the Bank's impact on the more courses, to a broader audience, and to capacities of these partners. The research firm make training available in more countries. conducted telephone interviews with directors or senior officials in 30 institutes, in 21 countries, Partner institutes are satisfied with their relation- drawn from a list of 200 contacts (provided by ship with the WBI. More than three-quarters of the the WBI) of training institutes with which the institutes surveyed (76 percent) were generally WBI either currently partners, or had recently satisfied (strongly agree and somewhat agree) partnered, to organize or cofinance courses. The with their institute's interaction with the WBI. original survey instrument is included at the end Seventy-seven percent were satisfied with their of this appendix. opportunity to advise the WBI on course design and course feedback. Seventy percent reported Institutes were screened in advance of the adequate opportunity to give feedback on their interview to ensure that respondents came from partnership with the WBI. However, results institutes that were in World Bank client suggest that this satisfaction is "soft." Less than countries and that had jointly organized or half of respondents do not "strongly agree" that cofinanced at least one course with the WBI in they are satisfied with each of these facets of their fiscal 2006. In all, the research firm conducted 30 relationship with the WBI, indicating room for telephone interviews with individuals from 30 improvement exists (see figure F.1). different training institutes, in 21 countries. Forty-three percent of respondents had partnered with the WBI for one to three years, 33 Figure F.1: "I Am Satisfied with the Opportunities percent for four to seven years, and the remain- My Institute Has" ing 24 percent had a relationship with the WBI lasting for over seven years. Most participating To give input on course 47 30 17 7 institutes (67 percent) reported having organized design/content between 1 and 5 courses with the WBI in fiscal 2006. Few institutes (6 percent) had organized To give course feedback 37 40 13 10 to WBI, post-course more than 10 courses with the WBI in the past fiscal year. To give feedback on our 47 23 17 13 partnership with WBI Results Strongly Somewhat Somewhat Strongly Partner institutes value their relationship with the agree agree disagree disagree WBI. Most respondent institutes highly valued 91 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Partner institutes are generally satisfied with the WBI doesn't share its knowledge, training support they receive from the WBI. Partner modules, or case studies . . . WBI most of institutes reported they were generally satisfied the time takes primary responsibility in with the support they get from the WBI in their all activities. day-to-day work. Institutes were particularly positive about receiving sufficient advance notice I wish WBI [would] delocalize and let us about new courses (80 percent strongly agreed participate more. or somewhat agreed), receiving guidelines for participant selection (77 percent), and receiving It was a very good initiative; the World training materials in a timely matter (64 percent). Bank was instrumental but it is about time to move it to the region . . . and have Respondents are less positive (64 percent) about the the partners hosting [the conference]. WBI's capacity-building support. Only 27 percent strongly agreed that the WBI supports building Participants benefit from the courses, but the capacity of its lecturers, suggesting that there not our institution. We would like to have is room for improvement (see figure F.2). Several a partnership that offers us help in respondents, when asked if there were aspects development as an institution and in of their relationship with the WBI that they building our own capacity. would like to highlight, spoke of the need to build the capacity of local training institutes. Division of responsibilities between the WBI and its There were different views on how this might partner institutes can be improved. A slight majority play out. For some, this should involve a devolu- of respondents noted that they were satisfied tion of responsibilities, away from the WBI, with the current division of responsibilities (see toward institutes and local partners. Others felt figure F.3). However, a small majority also said that the WBI should improve how it shares its that additional input was needed either from the knowledge. Sample quotations from the survey WBI or from their own institute. are given below: · The partner institute is usually entirely re- It seems that WBI wants to execute the sponsible for tasks such as making logistical project, achieve the goal, and that's it, arrangements (63 percent) and booking and when what is needed from WBI is to training facilities (50 percent). (Figure F.4.) establish a way to strengthen nationally · The responsibilities for designing written ma- and regionally the programs. terials and the program content, and choosing course topics were most likely to be shared be- tween the institute and the WBI. · Few respondents said their institute is entirely responsible for either providing in-house lec- Figure F.2: Partner Institutes Satisfied with WBI Input turers (13 percent) or designing course pro- grams or content (13 percent). WBI provides sufficient advanced 40 40 7 7 · In addition to these tasks, respondents also said notice for new courses they were involved in raising awareness about WBI provides sufficient guidelines 40 37 13 7 the courses (that is, marketing), identifying for participant selection new partnership opportunities, and arranging WBI sends training materials 47 17 13 7 conferences and training programs. in a timely manner WBI assists in building capacity 27 37 27 7 Respondents noted interest in increasing their of lectures/coordinators influence. This is particularly so in choosing Strongly Somewhat Somewhat Strongly agree agree disagree disagree course topics, identifying and selecting partici- pants, and in designing course program and 92 APPENDIX F: SUMMARY OF WBI'S PARTNER TRAINING INSTITUTE SURVEY content. Shifting additional responsibility to Figure F.3: Satisfaction Levels with Division of Labor institutes in these areas may take advantage of their local insight and knowledge of the country's context and training needs. As one Designing 3 23 43 10 20 program/content respondent said, "To increase the effectiveness Providing in-house of courses is possible through adaptation to local 7 17 47 7 20 lecturers education needs." Indentifying/selecting 13 10 47 3 27 participants Indentifying external 3 20 53 20 3 lecturers Choosing course 10 53 13 20 topics Much more from WBI Somewhat more from WBI Satisfied Somewhat more from my institute Much more from my institute Figure F.4: WBI Rarely Has Primary Responsibility for Training Providing logistical arrangements 63 13 10 7 3 Identifying/selecting participants 33 30 30 3 3 Providing/booking training 50 7 13 10 10 facilities Designing written materials 20 30 30 17 3 Identifying external lecturers/ 23 27 33 17 trainers Designing programs/content 13 30 33 20 3 Providing in-house lecturers 13 30 23 20 13 Choosing course topics 20 20 37 10 13 Entirely my institute Mainly my institute Both equally Mainly WBI Entirely WBI 93 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Survey of WBI-Partner Training Institutes 1. About how long has been partnering with the World Bank Institute? _____ years 2. We would like to ask about the extent of your collaboration with WBI. How many courses did you jointly organize and/or cofinance with WBI over 2006? 01 None (TERMINATE INTERVIEW) 02 1­5 courses 03 6­10 courses 04 11­20 courses 05 More than 20 courses Please answer the remainder of these survey questions with reference to courses that you have organized with WBI over 2006. 3. I would like to discuss the nature of your partnership with WBI. I'm going to read you several statements related to different aspects of this partnership. For each statement I read, please tell me who has taken primary responsibility. The response options are: 01 Entirely your institute 02 Mainly your institute 03 Equally your institute and the World Bank Institute 04 Mainly WBI 05 Entirely WBI READ AND ROTATE ORDER OF STATEMENTS. a. Providing or booking training facilities (classrooms, etc.) b. Providing logistical arrangements (coffee breaks, communications with participants prior to course, etc.) c. Identifying and selecting course participants d. Choosing course topics e. Designing course program and content f. Designing written course materials (manuals, etc.) g. Identifying and engaging lecturers/trainers not affiliated with WBI or your institution h. Providing in-house lecturers 4. Other than the activities we just discussed, does your institute do anything else to organize or assist with WBI training? [OPEN-ENDED, RECORD VERBATIM RESPONSE.] 5. On the basis of the division of responsibilities that we just discussed, I would like to hear your perspective on whether your partnership with WBI could result in more effective training if there was a different division of responsibilities. I'm going to read you several statements. For each, please tell me which of the following best reflects your opinion, using the following response options: 01 Much more input from WBI is needed. 02 Somewhat more input from WBI is needed. 03 I am satisfied with the contribution of both WBI and my institute. 04 Somewhat more input from my institute is needed. 05 Much more input from my institute is needed. 94 APPENDIX F: SUMMARY OF WBI'S PARTNER TRAINING INSTITUTE SURVEY READ AND ROTATE STATEMENTS. a. Choosing course topics b. Identifying and selecting course participants c. Designing course program or content d. Identifying and engaging lecturers not affiliated with WBI or your training institute e. Providing in-house lecturers 6. For each of the following statements I read, please tell me if you strongly agree, somewhat agree, somewhat disagree, or strongly disagree. [READ AND ROTATE ORDER OF STATEMENTS.] a. Overall, WBI provides my institute with sufficient advance notice for planned new courses. 01 Strongly agree 02 Somewhat agree 03 Somewhat disagree 04 Strongly disagree b. WBI sends course materials to my institute in a timely manner. c. WBI provides sufficient guidelines for course participant selection. d. WBI assists my institute in building the capacity of our lecturers and/or course coordinators. e. I am satisfied with the opportunities my institute is given to provide input on course content and design. f. I am satisfied with the opportunities my institute is given to give course feedback to WBI at the end of courses. g. I am satisfied with the opportunities my institute is given to provide feedback on our partnership with WBI. 7. Which of the following best reflects the extent to which the courses that you have organized with WBI have charged participant fees? READ STATEMENTS. CODE ONE ONLY. 01 None of the courses 02 Some of them (less than 50 percent of the courses) 03 Many of the courses (more than 50 percent of the courses) 04 All of the courses In closing, are there any aspects of your partnership with WBI that you would like to highlight? Specifically, are there areas in which you believe your partnership could be improved, and/or any aspects of your present collaboration that have been particularly ben- eficial to your institute? [OPEN-ENDED, RECORD MULTIPLE VERBATIM RESPONSE.] 95 APPENDIX G: ASSESSMENT OF WBI'S ACTIVITY INITIATION SUMMARY FORMS This analysis evaluated the content of the World AISs were rated as having information in any the Bank Institute's (WBI) Activity Initiation above categories even when there was only a Summary (AIS) forms.1 A random sample of AIS vague reference. The scores were then collec- forms was reviewed to examine the level of detail tively tabulated. Table G.1 provides a quantitative and specificity of information on training design, breakdown of the various aspects of the results content, and objectives in mandatory WBI chain of a training program. course-planning documentation. Main Messages Data and methodology: From the master list of all There is poor quality control of AISs and lack of a 168 fiscal 2006 WBI training activities--in the clear results chain. Of the 61 AISs that were evaluation's 10 desk-review countries that are reviewed, only two (3 percent of the sample) also WBI focus countries2--a random sample of showed a clear results chain leading from inputs 61 in-country training activities was generated.3 to expected outcomes, outputs, and impacts. The following information was taken from the Most course details in AISs lacked adequate AIS forms: information about how objectives would be achieved and how to measure the impacts. Only · Diagnosis--organizational and/or institutional three courses provided monitoring and evalua- capacity gaps to be addressed through training, tion indicators to measure results. and the reasons why training was determined to be the appropriate response for filling the gaps; The objectives of courses are not realistically · Training-needs assessment--training needs of defined. The course objectives are very broadly the target population; defined and present goals that are beyond the · Participant selection--desired participant scope of what the course can realistically achieve. profiles; · Client participation--information on client con- sultation in design of training; Table G.1: AIS Counts and Percentages, by category · Target numbers--number of individuals to be trained; · Learning outputs--knowledge and skills to be AIS number (N=61) Percentage of AIS taught; Diagnosis 30 49 · Workplace performance outcomes--behavioral Training-needs assessment 10 16 or performance changes that can be expected Participant selection 2 3 as a result of the training; Client participation 56 61 · Development impact--the larger development Target numbers 8 13 objective that the training will contribute to Learning 47 77 (where applicable); and Performance 9 15 · Monitoring and evaluation--performance indi- cators for measuring the achievement results Impact 22 36 of the course (where applicable). Monitoring and evaluation 3 5 97 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT The diagnostic exercises before the start of courses could contribute to the expected outcome. Less are insufficient. Although 49 percent of the courses than 20 percent indicated that a training-needs made some reference to capacity gaps, none of assessment was planned or implemented to them explained why training was the appropriate ensure that the training delivered was the most means to address the capacity gaps or how training relevant to the given situation or context. 98 APPENDIX H: LIST OF WBI INTERVIEWEES Caroline Adams-Brezina, Senior Resource Manage- Mohini Malhotra, Regional Coordinator, WBIRC ment Officer Darius Mans, former Director of Regional Kabir Ahmed, Chief Administrative Officer, Coordination WBICA Maurya West Meiers, Evaluation Officer, WBIEG Chirine Alameddine, Consultant, WBIICD Raj Nallari, Lead Economist, WBIPR Kofi Anani, WBIRC Rakesh Nangia, Director, Operations Juan Blazquez Ancin, Extended Consultant, Mark Nelson, Sr. Operations Officer, WBICD WBIRC Michele de Nevers, Director, Capacity Develop- Victor Ferrer Andreu, Consultant ment and Sector & Thematic Programs Marian de los Angeles, Sr. Environmental Pietronella van den Oever, Sr. Environmental Economist, WBIEN Specialist, WBIEN Maria Gonzalez de Asis, Senior Public Sector Samuel Otoo, Manager, WBIGP Specialist, WBIPR Azerdine Ouerghi, Lead Social Protection Adrian Bianchi, Senior Institutional Develop- Specialist, WBIHD ment Specialist, WBIEN Djordjija Petkoski, Lead Enterprise Restructur- Jean Luc Bosio, Rural Development Specialist, ing Specialist, WBIFP WBISD Moira Hart Poliquin, Sr. Operations Officer, Evangeline Kim Cuenco, Sr. Operations Officer, WBIRC WBISD Francesca Recanatini, Sr. Economist, Anticorrup- Guy Darlan, Regional Coordinator, WBIRC tion diagnostic, PRMPS Gilles Dussault, former Health Specialist, Violaine Le Rouzic, Sr. Evaluation Officer, WBIEG WBIHD Salomon Samen, WBIPR Qimiao Fan, Lead Economist, WBIFP Anwar Shah, WBIPR Alexander Fleming, Sector Manager, Finance Tsutomu Shibata, Sr. Adviser, WBISP and Private Sector Development Rosa Alonso Terme, Sr. Public Sector Specialist Vincent Greaney, Lead Education Specialist, (formerly WBI) WBIHD Richard Tobin, Manager, WBIEG Michael Jarvis, Private Sector Development Emiko Todoroki, Financial Sector Specialist, Specialist, WBIFP WBIHD Philip Karp, Lead Specialist, WBIRC Victor Vergara, Lead Urban Sector Specialist, Shahidur Khandker, WBIPR WBIFP Nidhi Khattri, Sr. Evaluation Officer, WBIEG Patrick Verissimo, Sr. Sector Economist (for- Kenneth King, Manager, WBIKP merly WBI) Ruben Lamdany, former Director of Sector and Myriam Waiser, Consultant, Education Specialist, Thematic Programs WBIHD Tatyana Leonova, Regional Coordinator, WBIRC Ronald MacLean-Abaroa, Sr. Operations Officer, WBIFP 99 APPENDIX I: LIST OF PROJECTS IN COUNTRY REVIEW STUDIES Bank Projects Implementation Completion Report Bank Institutional Project approval Closing development number Country Project name date date Outcomea impactb P075016 Bangladesh Public Procurement Reform 05/02/2002 06/30/2007 -- -- P049790 Bangladesh Export Diversification 06/01/1999 06/30/2004 Satisfactory Modest P050751 Bangladesh National Nutrition Program 05/25/2000 08/31/2006 Moderately Moderate unsatisfactory P069933 Bangladesh HIV/AIDS Prevention 12/12/2000 12/31/2007 -- -- P035673 Burkina Faso Community-based Rural 11/30/2000 06/30/2007 -- -- Development P000309 Burkina Faso Basic Education Secondary 01/22/2002 06/30/2008 -- -- project P071443 Burkina Faso Competitiveness and Enterprise 03/04/2003 06/30/2008 -- -- Development P078596 Burkina Faso Administration Capacity 03/22/2005 02/28/2011 -- -- Building Project P077602 Mexico Tax Administration Institutional 06/18/2002 06/30/2007 -- -- Development P070108 Mexico Savings and Credit Sector 07/02/2002 12/01/2007 -- -- Strengthening and Rural Microfinance Capacity Building Technical Assistance P057531 Mexico Second Basic Education 03/21/2002 06/30/2004 Satisfactory High Development Project P050945 Tunisia Education Quality Improvement 06/27/2000 08/30/2006 Satisfactory Not rated Program 1 P005750 Tunisia Agricultural Support Services 06/26/2001 12/30/2008 -- -- Project P055814 Tunisia Export Development Project 1 05/20/1999 09/30/2004 Satisfactory High P005745 Tunisia Second Training and 06/13/1996 06/30/2003 Satisfactory Substantial Employment P005741 Tunisia Higher Education Reform 03/17/1998 12/31/2004 Satisfactory Modest Support Project 1 a. The standard scale for ICR Principal Performance Ratings includes: highly satisfactory, satisfactory, unsatisfactory, and highly unsatisfactory. b. Possible ratings include: high, substantial, modest, and negligible. 101 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT WBI Programs Bangladesh Bangladesh Central Bank Poverty Monitoring Trade Negotiations Burkina Faso Rural Development Social Protection Mexico Public Sector Governance Program: Open and Participatory Government course Urban and Local Government Programa: Electronic Governance and Municipal Headstart courses Private Sector Development Program: Corporate Social Responsibility course 102 APPENDIX J: DETAILED MANAGEMENT COMMENTS This appendix provides a more detailed response design and implementation issues related to to the IEG review. The review seeks to assess training for capacity development. Management training for capacity development provided concurs that improvements are needed with through Bank-financed projects and by the World respect to pedagogical design; linking training to Bank Institute (WBI). Management appreciates follow-up and other support for change; and IEG's attention to client training programs that designing content and determining participa- the Bank sponsors and believes that the evalua- tion, when feasible, on the basis of diagnoses of tion report contains much of value regarding organizational and other capacity constraints. training design and implementation. However, Many of these points are relevant for most this is an ambitious effort, given what are typically learning activities. Management also agrees with substantial differences in objectives, scale, and IEG's assessment that monitoring and evaluation form between project-financed training and WBI (M&E) of client training should be improved, but training. In Management's view, the evaluation it would note that this M&E can be done only by could have better acknowledged these differ- the partner countries where the training takes ences, and it finds issues regarding the compar- place. The role of the Bank (and of other donors) isons of outcomes and impact between the two is to help countries strengthen their capacity for fundamentally different cases. Additionally, the M&E. Management notes that strengthening Bank has many other important training M&E capacity is a general issue that goes beyond programs--those provided through partnerships training and is part of the Bank's overall results such as the Cities Alliance; the program for agenda. building statistical capacity in the Development Economics (DEC) group; the Treasury's Reserve Reservations with Regard to the Findings and Asset Management Program; and the growing Conclusions. Despite the evaluation's contribu- volume of nonproject external training provided tions, Management has serious reservations by Regions and Networks--most of which are about a number of the findings and conclusions closer to the type and intent of training provided in the report. through WBI than is project-financed training. Regrettably, this review missed the opportunity · Training versus capacity development. As the re- to examine these programs. This appendix view notes, the Bank and other donors support presents Management's general observations on capacity development in many ways in addition WBI and training provided through operations to training, including support for policy and in- supported by the Regions. A related document stitutional change, technical assistance, long- with specific chapter-by-chapter comments on term sectoral engagement often linked to the review is available on this evaluation's Web operations, joint economic and sector work site at http://www.worldbank.org/ieg.1 (which is becoming the norm), and physical in- vestment in buildings and equipment. Im- Management's Views on the Overall proved capacity is measured by, among other Analysis and Conclusions things, higher quality of public services, re- The report provides an informative summary of duced transaction costs, more efficient ser- 103 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT vice delivery, and achieving international per- agrees that there is room for improvement, but formance standards. Despite the review's ac- the evaluation does not provide benchmarks that knowledgement of the wider scope of capacity place the Bank's efforts and outcomes in development, it tends to define its measure of perspective. It does not report comparative data changes in workplace behavior as measuring from other organizations' training results with capacity development.2 respect to organizational impact, the Bank's and · The evaluation framework. IEG's evaluation WBI's own historical trends (how the results framework assumes that training should al- compare with what was achieved before), or ways achieve monitorable and measurable information about the difficulty of the work itself. changes in workplace behaviors. Such an as- Although the evaluation provides a review of sumption is inappropriate for the many activ- benchmark organizations, it does not compare ities that provide clients with information about the results to any objective standards of accept- policy options or emerging global issues and able level of results for the type of work that are likely to contribute to development ob- performed. As the Standards for Evaluation in jectives in the long run.3 the UN System notes, an "explanation of context · Methodological foundation. Outcomes and im- contributes to the utility and accuracy" of evalua- pact are judged mostly on the basis of client sur- tions.5 Describing and acknowledging the veys with a low response rate and a limited challenges of this context would put the Bank's number of field visits.4 accomplishments--as well as the opportunities · Evidence basis. Interpreting and using the data for improvement--into perspective. go beyond what the data can show. Management's Views on Analysis and Evaluation Model. Management considers IEG's Conclusions Relating to WBI evaluation model to be inappropriate for evaluat- WBI supports the development of country ing the different types of training provided with capacity by addressing constraints in three areas: Bank support. The evaluation sets out to (a) the enabling environment or societal level; examine training that had "as its ultimate goal (b) institutions and policies; and (c) organiza- building the capacity of organizations or institu- tional resources. WBI's instruments include tions to achieve development objectives, rather training, learning that blends online training with than individual learning for its own sake." The face-to-face exchanges, nonlending technical evaluation sought to answer the question, "To assistance, peer-to-peer learning, knowledge- what extent did Bank-financed training have an exchange sessions, publications, web portals, impact on the capacity of target institutions and workshops, and networking events. These activi- organizations?" Management agrees that project- ties and products are usually targeted at individ- supported and WBI-provided training is not for uals and groups who can serve as catalysts for individual learning for its own sake. Nonetheless, change, and they aim to expose participants to, individual learning is a key foundation for and raise their awareness of, innovative building capacity, and the way in which training approaches to development; disseminate best translates into behavioral change and workplace practices; and consider policy alternatives. The performance varies substantially across different literature on adult learning, including much of training methods, immediate training objectives, the literature mentioned in the evaluation, and the timeline expected for institutional and shows that raising awareness is a valid objective organizational impact. for training and is often the first step in the process of organizational change. The box below Context and Tone. Finally, Management is describes some of WBI's nontraditional training concerned about the omission of valuable work. contextual information that puts the Bank's efforts into perspective and about the way some Choice of Activities for Review. The evaluation of the evidence is presented. Management does not distinguish among the different 104 APPENDIX J: DETAILED MANAGEMENT COMMENTS Box J.1: Examples of WBI's Nontraditional Learning Programs WBI facilitated the efforts of the government of Madagascar example of a WBI program aimed at organizational capacity de- to prioritize and initiate action on its poverty reduction strat- velopment is its support to the Community Development and In- egy through a series of cabinet retreats, workshops, and peer vestment Agency of the Kyrgyz Republic, which aims to build exchanges that fundamentally changed the government's ap- capacity at the regional/local government levels to implement proach to implementation of its strategy. Similarly, WBI has sup- intergovernmental fiscal reforms outlined in the proposed ported the establishment of parliamentary networks such as the poverty reduction strategy grant. The IEG study fails to cover African Parliamentarians Network against Corruption, which these kinds of programs, which are typical of WBI's portfolio facilitates information exchange among parliamentarians. An of work. objectives and modes of WBI's activities. IEG activities may lie in the fact that during most of limited its focus to training that (a) has specific the evaluation period all of WBI's activities were learning objectives; (b) is held away from settings coded within the Bank as External Training (TE), other than where knowledge is to be applied; regardless of their primary purpose. This and (c) is conducted by lecturers, trainers, or situation has been addressed, and WBI now uses facilitators; and it specifically excluded "on-the- a broader and more accurate range of work job technical assistance, learning among peers, codes than were used during the evaluation, and events whose primary purpose is network- including nonlending technical assistance ing, rather than learning." Application of these (NLTA) and "knowledge products" (KP). criteria led IEG to exclude many of the objectives and modes of WBI's learning programs. Comparison of Project-Supported and WBI Training. However, in the end, the review examined many The IEG evaluation compares project-supported activities that do not appear to match IEG's and WBI training and finds a 10 percentage point criteria. For example, IEG examined 60 randomly difference in the training's contribution to selected Activity Information Summaries (AISs) "substantial" positive changes to work perform- that summarize the purposes of each WBI ance. Yet the evaluation does not thoroughly activity. However, almost 50 percent of IEG's examine the critical contextual differences sample included AISs that seemingly did not between project-funded and WBI training. The meet IEG's initial criterion because the events report rightly highlights the fact that "successful were "knowledge-exchange" activities whose capacity building often requires a multipronged primary objective was the exchange of informa- approach that addresses resource, incentive, and tion, knowledge, and experience among peers human capacity," and that "it is difficult to rather than having a specific learning objective.6 disentangle the impact of training from the Judging activities on whether they contribute to impact of other interventions enabling the an "ultimate" goal of building organizational and achievement of capacity building goals." Project- institutional capacity regardless of their immedi- supported training is embedded in other ate objectives is akin to judging all projects on interventions that typically provide additional their "ultimate" impact on poverty reduction and resources (for example, buildings and growth, regardless of their immediate objectives. equipment) and powerful incentives for change These sampling issues may affect the survey (project agreements and long-term engage- sample as well, which was developed on the basis ment), whereas WBI's interventions may not be of lists of participants, without first assessing the directly linked with such interventions. Although objectives of the activities they attended. the data indicate that resource support is clearly an important condition, there is little analysis Coding of Activities. The origins of the differences about how other resources support capacity in understanding regarding the purpose of WBI development. Furthermore, the evaluation 105 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT ignores the considerable differences in cost and in chapter five, for example, IEG does not distin- size among training events and also between WBI guish among the different types of activities training and Bank-supported projects that, in under review, and therefore among the appropri- most cases, have wider objectives supported by ate levels of M&E. Specification of performance several components besides training, and indicators and skills to be taught may be neither therefore it does not consider the cost-effective- feasible nor appropriate for a three-day ness of either. knowledge-exchange activity on policy alterna- tives for high-level public officials. In contrast, WBI's M&E Functions. Management further notes performance indicators would be essential for an that although the report emphasizes the extended in-service teacher training program. importance of M&E, IEG is critical of the Rather than applying a standard approach to advanced M&E functions that do exist within every activity, regardless of objectives, what is WBI.7 Like all of the "benchmark" organizations vitally important for every operation is to work that IEG identifies as models of best practice, with partner countries (including through WBI's evaluation group (WBIEG) is internal to training) on the quality of their design and WBI, and its methods appear to be equal to or implementation of the components; to assist better than those of the comparator organiza- them in designing cost-effective measures of key tions cited by IEG. In fact, IEG's own approach to outputs and outcomes that are commensurate measuring outcomes is similar to the one that with the training objectives; and to assist them in WBI pioneered in 2001. More importantly, IEG impact analysis where appropriate and cost states that WBI systematically monitors at the effective. program level only the number of participant training days. This is incorrect. WBI completed WBI's Mandate. Management acknowledges IEG's five program-based evaluations of outcomes contribution to the discussion on WBI's between FY03 and FY07, and another is mandate. However, the discussion and debate scheduled for completion in early FY08. System- about the nature of WBI's mandate is a long- atic evaluation does not imply universal standing one and will not be resolved by this coverage. IEG criticizes the lack of specific review. WBI's mandate will, of course, be covered program-level learning functions of the evalua- in the process of formulation, review, and roll out tions WBI has conducted and fails to mention the of the Bank's long-term strategy, specifically its internal accountability and management such "knowledge business line." evaluations also support. IEG does not acknowl- edge the variety of purposes that evaluations are Detailed Recommendations for WBI. Management intended to address. finds the recommendations regarding WBI appropriate for some of WBI's lines of business Management's Views on IEG's but not for all training. In particular, the Recommendations recommendation that all of WBI's training must Management agrees with the broad thrust of the be based on comprehensive assessments of report's recommendations in principle. target organization(s)/institutions(s) done in However, Management notes that they require cooperation with clients, and only after securing further analysis on the way to implementation. financing for implementation of learning, is inapplicable under many circumstances. For Guidance to Staff. IEG recommends that the Bank example, WBI sponsors knowledge-exchange develop guidance for training, including the forums and policy debates whose objective is to assessment and evaluation of training. Manage- raise awareness and provide policy options for ment sees the need to nuance the recommenda- the client to consider on a quick-turnaround, on- tion on the basis of cost, objectives, and utility demand, and fee basis; in such cases, compre- under different circumstances and types of hensive assessments are neither feasible nor training functions. In its analysis of training M&E appropriate. 106 APPENDIX J: DETAILED MANAGEMENT COMMENTS WBI Work on Improvements. Management agrees and is piloting a results framework for assessing that improvements are needed in the Bank's the design, monitoring, and evaluation of all its support to client training, and WBI is already activities, including those directly aimed at addressing pedagogy, learning-needs analysis, organizational and institutional capacity design, and follow-up. WBI's country-focused building. The framework captures six outcomes and long-term programmatic work is also of external training and knowledge sharing: evolving. Both were designed to improve awareness raising; skill enhancement; facilitating alignment and synergy with Bank operations and consensus and teamwork; helping clients to capitalize on information already available formulate strategies, policies, and plans; helping (notably priorities in Country Assistance Strate- clients implement strategies, policies, and plans; gies and sector and organizational needs) to and fostering networks. Elements of the design relevant and ongoing capacity-building framework are being institutionalized Bankwide services. for external training and knowledge-sharing activities in FY08. The framework will also be Results Framework for WBI Activities. Finally, in used to extend M&E for assessing WBI's consultation with Operations Policy and Country nonlending technical assistance work beyond Services (OPCS), WBI has recently developed the systematic evaluations conducted by WBIEG. 107 ENDNOTES Executive Summary 3. See appendix A for the details on the costs of 1. Management notes that, while it finds much of the training and the rating of training content in projects. evaluation useful, it has reservations regarding (i) the 4. Because Bank documents often do not track train- use of a model that assumes that all training is directly ing expenditures separately, these sector results are aimed at changing workplace behavior, which is inap- based on a random sample of 179 projects for which propriate for training and knowledge-sharing activities training costs could be estimated. The sample size aimed at providing clients with information and raising within each sector is not large enough to test for sta- their awareness regarding, for example, policy options tistical significance. or emerging global or country issues; and (ii) method- 5. The breakdown by investment project type is ological and data issues that call for more caution in pre- based on a random sample of 179 projects. Only one senting the findings and reaching conclusions. See the financial intermediary loan was in the sample of proj- Management Response. ects reviewed and is, therefore, not reflected in the graph. Management Response 6. The sample of projects within each investment loan 1. Using Training to Build Capacity for Develop- type was not large enough to determine statistical sig- ment: An Analysis of Project-Based and WBI Training, nificance. Independent Evaluation Group. 7. The WBI also carries out limited nonlearning 2. IEG notes that this evaluation considered learn- client-capacity-building activities. For example, in fiscal ing about policy options and emerging global issues to 2006, WBI did one piece of economic sector work and have positive outcomes insofar as the learning was used 12 nonlending technical assistance activities, in addition to inform workplace decisions. The participant survey to offering 812 client-learning events (World Bank In- was designed to capture such outcomes through a stitute 2006b). WBI nonlearning activities were not re- combination of close-ended and open-ended ques- viewed as part of this evaluation. tions, with the latter specifically aimed at eliciting all train- 8. WBI 2006c. ing outcomes relevant to the workplace. 9. Peer-to-peer learning or vocational training may 3. IEG notes that all conclusions in this evaluation also contribute to the overall client development capacity are based on a triangulation of evidence from various but was excluded from this evaluation, which focuses sources, including results from WBIEG's own participant on the impact of the training delivered, as part of WBI surveys; interviews with Bank task team leaders, coun- or operations country programs with organizational try team members, and WBI staff; comparison of Bank and institutional capacity-building goals. practices with international best practice, as determined 10. This evaluation did not look at the cost-effec- through a literature review; and extensive interviews with tiveness of Bank-financed training or compare costs of stakeholders in the field. WBI training with that of other training providers. Di- rect unit cost comparisons between training courses are Chapter 1 not meaningful. Depending on training goals and con- 1. IEG 2005, p. 9. ditions, $100 per participant training day may be rea- 2. WBI, "About the World Bank Institute," World sonable in some cases, while $10 per participant training Bank, http://www.worldbank.org/wbi. day is unduly expensive in others. 109 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT 11. IEG 2005, p. 7. by the program was lost (Rackham 1979; Phillips 1983), 12. UNDP 2006. leading the researchers to conclude that "However 13. Available on this evaluation's Web site at http:// good your skills training in the classroom, unless it's fol- www.worldbank.org/ieg. lowed up on the job, most of its effectiveness is lost" (Rackham 1979, p. 13). Rackham mentions, however, Chapter 2 that in the Xerox case, training--the purpose of which 1. As noted in chapter 1, the six-country survey was to enhance knowledge--generally showed a much was used to provide evidence primarily about work- smaller loss than skills-building when follow-up was place performance outcomes. Respondents were not missing. Rackham asserts that these two types of learn- asked whether their training had achieved its learning ing are different in this regard. goals because it could not be assumed that participants 12. Taylor 2001. were aware of the specific learning goals of their train- 13. For further information on diagnostic methods ing. For similar reasons, participants were not asked for capacity assessment, please refer to IEG 2005; Mis- whether their training had contributed to organizational sika 2006; Bramley 1996; Reid and Barrington 1994; capacity. and Taylor 2003. 2. WBIEG 2004. 14. Higher levels of ownership may be associated with 3. The difference between the WBI and project re- the fact that these countries borrow on International spondents on their ratings of positive changes to work Bank for Reconstruction and Development terms rather functions is significant at the 95 percent confidence in- than receive grants, but this hypothesis was not tested terval using a t-test for proportions. as part of this evaluation. 4. Question 6b of the survey questionnaire (see ap- 15. Research has shown that training-needs assess- pendix C). ment is one of the most important steps in training de- 5. WBI 2007a. velopment because it provides the basis for decisions about who should be trained and in what subject area. Chapter 3 Training-needs assessment is also critical to the formu- 1. Question 8 in the six-country survey instrument lation of adequate training and learning objectives. See (see appendix C). For further details on training man- Salas and Cannon-Bowers 2001. agement processes rated in the field studies, see ap- 16. WBIEG 2004, p. viii. pendix A. 17. Missika 2006; PRIA 1995; FIT 1983. 2. For the complete survey questionnaire, please see appendix C. Chapter 4 3. Results of the analysis of the importance and per- 1. Determinations about the importance of support formance of training-process factors, and the method- from managers and colleagues were made based on ology used, are detailed in appendix C. analysis of respondent ratings of the following survey 4. The difference between WBI and project respon- questions: "My colleagues provide me with the sup- dents on this rating is significant at the 95 percent con- port I need to apply what I learned in the course" and fidence interval. "My managers encourage my efforts to use what I 5. Chambers 2002. learned in the course." For more details on the drivers 6. Reid and Barrington 1994. analysis methodology, see appendix C. 7. Baldwin and Ford 1988. 2. UNDP 2006, p. 7. 8. Reid and Barrington (1994) found that the use of 3. Participants were asked to comment on the avail- structured exercises and case studies helps the trans- abilityofmaterialresourcesintwowaysinthesurvey.First, fer of knowledge gained in the training to real-life situ- as reflected in table 4.1, participants who stated that ations. training did not have a significant impact on their work 9. WBIEG 2007c. were asked to cite the primary reason why. One of the 10. Stammers and Patrick 1975; Missika 2006. possible options given was lack of availability of material 11. Studies carried out of Xerox Corporation's train- resources. In a separate question, all survey respondents ing programs showed that, in the absence of follow-up were asked to comment on the availability of material re- coaching, 87 percent of the skills-change brought about sources. Results from this question are cited in table 4.2. 110 ENDNOTES 4. Ford and others 1992; Baumgartel and Jeanpierre 2. The evaluation team undertook a pilot mission to 1972; Baumgartel, Reynolds, and Pathan 1984; Wexley the Lao People's Democratic Republic to test possible and Thornton 1972; Baldwin and Ford 1988; Huczyn- approaches. After the mission, the methodology and cri- ski and Lewis 1980; Axtell and others 1996; Hand and teria for the evaluation were significantly altered. others 1973. 3. The evaluation team recognized that different 5. WBIEG 2007, p. 21. forms of participant selection are appropriate in different contexts. For example, a competitive selection process Chapter 5 was judged to be important for high-quality training, 1. At the time of this evaluation, an AIS was the only whereas in other contexts, widespread, noncompetitive document required for all WBI training programs. participant selection was considered to be optimal. In 2. Good ratings for monitoring and evaluation, over- all cases, however, participant selection strategies were all, were given only to training offerings with adequate evaluated according to whether the strategy was linked level-3 evaluations, providing evidence on the extent to to the organizational capacity-building goals. which learning had been implemented in the workplace. 4. As detailed in chapter 4, there is considerable ev- 3. Four of the eight WBI training programs reviewed idence that participant reaction, as determined through in field studies did use more detailed level-1 question- level-1, end-of-course questionnaires, do not correlate naires, which asked about specific course modules. with learning or with workplace behavior change. As However, these were all distributed by WBI-partner such, the presence of level-1 questionnaires was not eval- training institutes, which chose to use the WBI's own, uated as adequate monitoring and evaluation. more detailed questionnaires rather than the WBI's op- 5. Adequate outcome evaluation (that is, evaluation tional questions. of training-participant behavioral change) was consid- 4. WBIEG 2007b. ered to be the most important type of evaluation to con- 5. WBIEG 2006. duct as a way to understand the impact of training on 6. Country Program Briefs are strategy documents organizational capacity. As such, good (green) evalua- prepared annually for each WBI focus country. This tion ratings were awarded only to programs that con- data was compiled for a draft WBIEG evaluation of WBI ducted such outcome evaluations or are planning to do Country Program Briefs. The evaluation was pending so at the appropriate time. Level-1 (reaction) and Level- clearance from WBI management at the time of release 2 (learning) evaluations were considered valuable only of this evaluation. to the extent that evidence was found that evaluation 7. These statistics are based on a review of WBI results affected future training design or follow-up course offerings in all 10 WBI focus countries-- strategies. Bangladesh, Bosnia, Burkina Faso, Guatemala, Lao Peo- 6. The sample was statistically significant at the 95 per- ple's Democratic Republic, Mexico, Nigeria, Philippines, cent confidence interval. Sri Lanka, and Turkey--out of the 13 countries selected 7. Because of the small number of projects in each by this evaluation for desk review. sector, some sectors were grouped together in the 8. WBI 2007b. analysis. The sectors that were grouped were the Eco- nomic Policy and Public Sector; Financial and Private Sec- Appendix A tor; Social Development, Social Protection, and Gender; 1. The WBI selects a group of "focus countries" in Transport; Global Information Communication Tech- which it develops multiyear, country-specific training nology; Energy and Mining; and Water and Sanitation programs, which are linked to Bank operations, and (called Infrastructure). evaluates impacts. Focus countries are selected in con- 8. The WBI's definition of client training includes con- sultation with the Regions, and are based on a number ferences for sharing knowledge and building aware- of criteria, most notably, the maintenance of a mix of mid- ness, regional and global dialogues, seminars, dle-income and lower-income countries, a high level of workshops, study tours, and training courses. demand for WBI programs from the client and country 9. Motorola University conducts internal training team, and countries where knowledge and capacity de- and by contract, training for employees of other com- velopment are key elements of the Bank's country strat- panies. The practices reported here are those for internal egy. WBI had 36 focus countries in fiscal 2006. training. 111 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT Appendix B are emerging lessons from an event that may benefit col- 1. Missika 2006. leagues working on training design and content. 2. Salas and Cannon-Bowers 2001. 2. The courses from the following 10 countries were 3. Taylor 2003. included in the AIS review: Bangladesh, Bosnia and 4. Taylor 2003; Loos and others 1999. Herzegovina, Burkina Faso, Guatemala, Lao People's 5. PRIA 1995; FIT 1983; and Taylor 2003. Democratic Republic, Mexico, Nigeria, the Philippines, 6. Missika 2006; PRIA 1995; and FIT 1983. Sri Lanka, and Turkey. Azerbaijan, Tunisia, and Uganda 7. Adapted from the Canadian School of Cadets In- were excluded from the sample because they were not structors by the Canadian Centre for International Stud- WBI focus countries. At the suggestion of the WBI, the ies and Cooperation and the Rural Development Services summaries included training programs from all WBI Centre in Vietnam. product-line codes (including knowledge exchange, 8. Taylor 2001; Stammers and Patrick 1975. skills building, and policy service) because the codes 9. Gist and others 1991; Marx 1982. were not found to differentiate meaningfully among 10. Warr, Bird, and Rackam 1970; Stufflebeam and types of training. The assumption was made that all in- others 1980. country WBI training courses have objectives beyond 11. Kirkpatrick 1959. that of individual capacity building and thereby fall within the scope of this evaluation. Appendix C 3. The sample of 61 training courses was represen- 1. Principal-components analysis groups individual tative of all WBI fiscal 2006 courses in the 10 desk- survey questions together, based on how respondents review countries. answer them, to identify patterns in the data for cor- relation analysis. Appendix J 2. On a 5-point scale, ratings between 1 and 2 con- 1. "Chapter-by-Chapter Management Observations stitute a "poor" rating, between 3 and 3.5 is considered on Methodological and Interpretation Issues." "average," and 3.5 and above is considered "good to 2. IEG notes that the evaluation states that the World excellent." Bank helps build capacity through a variety of comple- 3. Discriminant function analysis is a statistical tech- mentary means, of which training is one. The objective nique that accurately predicts program success or failure of this evaluation is to examine how training contributes through the analysis of participants' ratings of different to building client capacity. The evaluation focuses on course attributes. The predictive strength of the driver workplace outcomes as a necessary but insufficient is treated as an indicator of importance. For example, a condition for training to contribute to capacity devel- respondent may claim the course had a substantial pos- opment objectives. itive impact on his or her work functions. Discriminant 3. IEG notes that this evaluation considered learn- function analysis then enables establishing a link be- ing about policy options and emerging global issues to tween the respondent's stated outcome of course impact have positive outcomes insofar as the learning was used and how he or she rated the course on each of the tested to inform workplace decisions. The participant survey attributes. The stronger the predictive power of course was designed to capture such outcomes through a attributes, the higher the importance of the driver combination of close-ended and open-ended ques- overall. tions, with the latter specifically aimed at eliciting all training outcomes relevant to the workplace. Appendix E 4. IEG notes that all conclusions in this evaluation 1. The other half of respondents who had an opin- are based on a triangulation of evidence from various ion noted that it had improved. sources, including results from WBIEG's own participant surveys; interviews with Bank task team leaders, coun- Appendix G try team members, and WBI staff; comparison of Bank 1. AISs are filed by WBI task managers to initiate a practices with international best practice, as determined training activity. It is the only document that WBI task through a literature review; and extensive interviews with managers are required to submit. End-of-event reports stakeholders in the field. are not mandatory, but are recommended when there 5. Standards for Evaluation in the UN System, 112 ENDNOTES http://www.uneval.org/indexAction.cfm?module=Li about which WBI activities fit the definition of the eval- brary&action=GetFile&DocumentAttachmentID=1496. uation. As explained in paragraph 8 of this appendix, this 6. IEG notes that the evaluation looked only at WBI may stem in part from the use of the same administra- activities that conformed to the evaluation's definition tive code by WBI for all training and knowledge ex- of training. It was decided to include courses under the change activities until recently. "knowledge exchange" product line after an IEG con- 7. See, for example, Cristina M. Ling, Heidi S. Zia, tent review of a sample of WBI courses in all product Basab Dasgupta, and Izlem Yenice 2007. The Effective- lines, and in light of consultations with WBI on this ness and Impact of WBI FY01-05 Activities: Results from issue. Management notes that there appears to have 25 Focus Countries. Report No. EG07-126. Washing- been some misunderstanding during the consultations ton, DC: World Bank Institute. 113 BIBLIOGRAPHY Abadzi, H. 2006a. "Efficient Learning for the Poor." ing and Development:Multicultural Stories. Mel- Washington, DC, World Bank. (*) bourne, Florida: Krieger Publishing. ------. 2006b. "Making Training Unforgettable: From Bramley, P. 1996. Evaluating Training Effectiveness. Neurocognitive Research to the Evaluation of Train- London: McGraw-Hill. (*) ing Courses." Washington, DC, World Bank. Bramley, P., and A. Newby. 1984. "The Evaluation of Alliger, G., and E. Janak. 1989. "Kirkpatrick's Levels of Training, Part 1: Clarifying the Concept." Journal of Training Criteria: Thirty Years Later." Personnel Psy- European Industrial Training 8 (6): 10­16. (*) chology 42 (2): 331­42. Brennan, M.. 2003. Blended Learning and Business ANTA (Australian National Training Authority). 1999. Change: Key Findings. IDC. Training Package for Assessment and Workplace Briggs, G., and J. Naylor. 1962. "The Relative Efficiency Training. Melbourne: Australian Training Products, of Several Training Methods as a Function of Trans- Ltd. fer Task Complexity." Journal of Experimental Psy- Arthur, W., W. Bennett, and others. 1998. "Factors That chology 64: 505­12. Influence Skill Decay and Retention: A Quantitative Brinkerhoff, R. 1987. Achieving Results from Training. Review and Analysis." Human Performance 11: San Francisco: Jossey-Bass. 57­101. Bushnell, D. 1990. "Input, Process, Output: A Model for AusAID. 2006. A Staged Approach to Assess, Plan, and Evaluating Training." Training and Development Monitor Capacity Building. AusAID. Journal 44 (3): 41­43. ------. 2004. Capacity Development Principles and Chambers, R. 2005. Ideas for Development. London: Practices. AusAID. Earthscan. Axtell, C., S. Maitlis, and others. 1996. "Predicting Im- ------. 2002. Participatory Workshops: A Sourcebook mediate and Longer-Term Transfer of Training." Per- of 21 Sets of Ideas and Activities. London: Earthscan. sonnel Review 26 (3): 201­13. (*) Cormier, S. 1984. Transfer of Training: An Interpreta- Baldwin, T., and J. Ford. 1988. "Transfer of Training: A tive Review. Technical Report No. 608, Army Institute Review and Directions for Future Research." Per- for the Behavioral and Social Sciences, Alexandria, sonnel Psychology 41 (1): 63­105. (*) VA. Banez-Ockelford, J. 1995. Partners in Creative Train- Dawson, R. 1995. "Fill This In before You Go: Under- ings. Phnom Penh, PACT. Utilized Evaluation Sheets." Journal of European In- Baumgartel, H., and F. Jeanpierre. 1972. "Applying New dustrial Training 19 (2): 3­7. Knowledge in the Back-Home Setting: A Study of In- DFID (Department for International Development). dian Managers' Adoptive Efforts." Journal of Apllied 2002. Capacity Development: Where Do We Stand Behavioral Science 8 (6): 674­94. (*) Now? DFID Governance Department, London. Baumgartel, H., M. Reynolds, and R. Pathan. 1984. "How Dixon, N. 1990. "The Relationship between Trainee Re- Personality and Organizational-Climate Variables sponses on Participation Reaction Forms and Posttest Moderate the Effectiveness of Management Devel- Scores." Human Resource Development Quarterly opment Programmes: A Review and Some Recent Re- 1: 129­37. (*) search Findings." Management and Labour Studies Esterby-Smith, M. 1985. "Turning Course Evaluation 9: 1­16. from an End to a Means." Personnel Management Baumgartner, L., and S. Merrian, eds. 2000. Adult Learn- 17 (4): 25­27. 115 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT FIT (Foundation for International Training). 1983. Train- Holton, E. 1996. "The Flawed Four-Level Evaluation ing of trainers. Markham, Ontario, Canada: FIT. (*) Model." Human Resource Development Quarterly Fitz-Enz, J. 1994. "Yes, You Can Weigh Trainers' Value." 7 (1): 5­21. (*) Training 31 (7): 54­58. Horton, D., A. Alexaki, and others. 2003. Evaluating Ca- Fleishman, E. 1953. "Leadership Climate, Human Rela- pacity Development: Experiences from Research tions Training, and Supervisory Behavior." Personnel and Development Organizations around the World, Psychology 6: 205­22. ISNAR, CTA, IDRC. Flexter, A., and E. Tulving. 1978. "Retrieval Independence Huczynski, A., and J. Lewis. 1980. "An Empirical Study in Recognition and Recall." Psychological Review into the Learning Transfer Process in Management 85: 153­71. Training." Journal of Management Studies 17 (2): Ford, J., M. Quinones, D. Sego, and J. Sorra. 1992. "Fac- 227­40. tors Affecting the Opportunity to Perform Trained IEG (Independent Evaluation Group). 2005. Capacity Tasks on the Job." Personnel Psychology 45 (3): Building in Africa: An (IEG) Evaluation of World 511­27. (*) Bank Support. Washington, DC: World Bank. Freire, P. 1994. Pedagogy of Hope. New York: Continuum. Irvine, R., R. Chambers, and R. Eyben. 2004. Learning ------. 1973. Education for Critical Consciousness. from Poor People's Experience: Immersions. Lessons New York: Continuum. for Change in Policy and Organisations Series. In- ------. 1970. Pedagogy of the Oppressed. New York: stitute of Development Studies, University of Sussex, Continuum. Brighton, UK. Gaventa, J. 2004. "Towards Participatory Governance: As- Kabeer, N. 2000. Resources, Agency, Achievements: Re- sessing the Transformative Possibilities." In From flections on the Measurement of Women's Empow- Tyranny to Transformation?: Exploring New Ap- erment. Gendered Poverty and Well-Being. S. Razavi. proaches to Participation, ed. G. Hickey and S. 30. Mohan. London, Zed Books. Kaplan, R., and G. Pascoe. 1977. "Humorous Lectures Gaventa, J., and A. Cornwall. 2001. "Power and Knowl- and Humorous Examples: Some Effects upon Com- edge." In Handbook of Action Research: Partici- prehension and Retention." Journal of Education pative Inquiry and Practice, ed. P. Reason and H. Psychology 69 (1): 61­65. Bradbury. London: Sage Publications. Kaufman, R., J. Keller, and others. 1995. "What Works Gist, M., C. Stevens, and A. Bavetta. 1991. "Effects of Self- and What Doesn't: Evaluation Beyond Kirkpatrick." Efficacy and Post-Training Interventions on the Ac- Performance and Instruction 35 (2): 8­12. quisition and Maintenance of Complex Interpersonal Keller, J. 1996. Evaluating Diversity Training: 17 Ready- Skills." Personnel Psychology 17: 183­211. to-Use Tools. San Diego: Pfeiffer. (*) Goldstein, I. 1993. Training in Organizations: Needs Kirkpatrick, D. 1998. Evaluating Training Programs: The Assessment, Development, and Evaluation. Pacific Four Levels. San Francisco: Berrett-Koehler. (*) Grove, California: Brooks/Cole. ------. 1959. "Techniques for Evaluating Training Pro- Hamblin, A. 1974. Evaluation and Control of Training. grammes." Journal of the American Society of Train- London: McGraw Hill. ing Directors 13 (11): 3­9. Hand, H., M. Richards, and J. Slocum. 1973. "Organiza- Knowles, M. 1980. The Modern Practice of Adult Edu- tion Climate and the Effectiveness of a Human Re- cation: From Pedagogy to Andragogy. Englewood lations Training Program." Academy of Management Cliffs: Cambridge Adult Educator. Journal 16 (2): 185­95. (*) Kolb, D. 1984. Experiential Learning. New Jersey: Pren- Haskell, R. 2001. Transfer of Learning. London: Aca- tice-Hall. demic Press. Lee, S., and J. Pershing. 2002. "Dimensions and Design Hirsh, W., and P. Tamkin. 2005. Planning Training for Criteria for Developing Training Reaction Evalua- Your Business. IES Report No. 422, Institute for Em- tions." Human Resource Development International ployment Studies, University of Sussex, Brighton, UK. 5 (2): 175­97. (*) Holden, L., and Y. Livian. 1992. "Does Strategic Training Loos, G., T. Fowler, and others. 1999 A Model for Re- Policy Exist?: Some Evidence from Ten European search on Training Effectiveness. Cincinnati: NIOSH. Countries." Personnel Review 21 (1): 12­22. Mager, R. 1997. Analyzing Performance Problems or You 116 BIBLIOGRAPHY Really Oughta Wanna. Atlanta: Center for Effective havior: A Component Analysis." Academy of Man- Performance. agement Journal 27 (3): 544­60. Mager, R., and P. Pipe. 1970. Analyzing Performance Reid, M., and H. Barrington. 1994. Training Interven- Problems. Fearon, California: Fearon Pitman tions. London: Institute of Personnel and Develop- Publishers. ment. (*) Manpower Services Commission. 1981. Glossary of Remmers, H., D. Martin, and others. 1949. "Student Training Terms. Achievement and Instructor Evaluations in Chem- Marx, R. 1982. "Relapse Prevention for Managerial Train- istry: Studies in Higher Education." West Lafayette, ing: A Model for Maintenance of Behavior Change." Indiana: Purdue University. Academy of Management Review 7 (3) 433­41. Rodin, M., and B. Rodin. 1972. "Students' Evaluations McGehee, W., and P. Thayer. 1961. Training in Business of Teachers." Science 29: 1164­66. (*) and Industry. New York: John Wiley & Sons. Salas, E., and J. Cannon-Bowers. 2001. "The Science of Mezirow, J. 2006. "An Overview on Transformative Learn- Training: A Decade of Progress." Annual Review of ing." In Lifelong Learning: Concepts and Contexts, Psychology 52: 471­99. (*) eds.,P.SutherlandandJ.Crowther.London:Routledge. Shore, E., and L. Sechrest. 1961. "Concept Attainment ------. 1991. Transformative Dimensions of Adult as a Function of Number of Positive Instances Pre- Learning. San Francisco: Jossey-Bass. sented." Journal of Educational Psychology 52 (6): ------. 1990. Fostering Critical Reflection in Adulthood: 303­07. A Guide to Transformative and Emancipatory Sleezer, C., D. Cipicchio, and D. Pitonvak. 1992. "Cus- Learning. San Francisco: Jossey-Bass. tomizing and Implementing Training Evaluation." Missika, B. 2006. The Challenge of Capacity Develop- Performance Improvement Quarterly 5 (4): 55­75. ment: Working towards Good Practice. DAC Network Stammers, R., and J. Patrick. 1975. The Psychology of on Governance Capacity Paper, Organisation for Eco- Training. London: Methuen & Co. nomic Co-operation and Development, Paris. (*) Stufflebeam, D., W. Foley, and others. 1980. L'évaluation Newby, A. 1992. Training Evaluation Handbook. Alder- en éducation et la prise de décision. Ottawa: Edi- shot, England: Gower. tion NHP. Newstrom, J. 1978. "Catch-22: The Problems of In- Stymulo, T. 2001. "Crouching Impact, Hidden Attribu- complete Evaluation of Training." Training and De- tion: Overcoming Threats to Learning in Develop- velopment Journal 32 (11): 22­24. ment Programs." Block Island Workshop on Across Noe, R. 1986. "Trainees' Attributes and Attitudes: Neg- Portfolio Learning, May 22­24, International Devel- lected Influences on Training Effectiveness. Academy opment Research Centre. of Management Review 11(4): 736­49. Tamkin, P., J. Yarnall, and others. 2002. "Kirkpatrick and Patrick, J. 1992. Training: Research and Practice. Lon- Beyond: A Review of Models of Training Evaluation." don: Academic Press Limited. Institute of Employment Studies, Brighton, UK. Phillips, J. 1983. Handbook of Training Evaluation Taschereau, S. 1998. Evaluating the Impact of Training and Measurement Methods. Houston: Gulf and Institutional Development: A Collaborative Publishing. Approach. Washington, DC: World Bank. Preskill, H., and R. Torres. 1999. Evaluative Inquiry Taylor, P. 2003. How to Design a Training Course. Lon- for Learning in Organizations. Thousand Oaks, don: Continuum. (*) California: Sage. ------. 2001. Ten Key Stages towards Effective Partic- PRIA. 1995. A Manual for Participatory Training ipatory Curriculum Development. Zurich: Helvetas. Methodology in Development. New Delhi: Society for (*) Participatory Research in Asia. (*) Teskey, G. 2005. Capacity Development and State Build- Pulley, M. 1994. "Navigating the Evaluation Rapids." ing Issues: Evidence and Implications for DFID. Training & Development 48 (9): 19­24. Department for International Development, London. Rackham, N. 1979. "The Coaching Controversy." Train- Thorndike, E., and R. Woodworth. 1901. "The Influ- ing and Development Journal 33 (11): 12­16. ence of Improvement in One Mental Function upon Reber R., and J. Wallin. 1984. "The Effects of Training, the Efficiency of Other Functions." Psychological Goal Setting, and Knowledge or Results on Safe Be- Review 8 (3): 247­61. 117 USING TRAINING TO BUILD CAPACITY FOR DEVELOPMENT UNDP (United Nations Development Programme). Wilson, J. 1995. Mapping a Winning Training Approach. 2006. "Capacity Assessment Practice Note." UNDP London: Kogan Page Ltd. Practice Note, UNDP, New York. WBI (World Bank Institute). 2007a. Monthly Manage- Vella, J. 1989. Learning to Teach: Training of Trainers ment Report (March). Washington, DC: World Bank. for Community Development. Washington, DC: Save ------. 2007b. "Principles for Considering Partnership the Children; OEF International. Arrangements." http://siteresources.worldbank.org/ VeneKlasen, L., and V. Miller. 2002. A New Weave of WBI/Resources/Partnerships/Partnership_Principles_ Power, People, and Politics. Oklahoma City: World ext.pdf. Neighbors. ------. 2006a. Does Training Work? Re-examining Voluntary Service Overseas. 2004. Participatory Ap- Donor-Sponsored Training Programs in Develop- proaches: A Facilitator's Guide. London: VSO. ing Countries. Capacity Development Brief, World Warr, P., C. Allan, and others 1999. "Predicting Three Lev- Bank, Washington, DC. els of Training Outcome." Journal of Occupational ------. 2006b. Monthly Management Report (June). and Organizational Psychology 72 (3): 351­75. Washington, DC: World Bank. Warr, P., M. Bird, and N. Rackam. 1970. Evaluation of ------. 2006c. World Bank Institute Annual Report: De- Management Training. London: Gower Press. veloping Capacities in Countries. Washington, DC: Warr, P., and D. Bunce. 1995. "Trainee Characteristics and World Bank. the Outcomes of Open Learning." Personnel Psy- WBIEG (World Bank Institute Evaluation Group). 2007a. chology 48 (2): 347­75. (*) An Assessment of the World Bank Insitute's Coun- Wenger E. 1998. Communitites of Practice: Learning, try Program Briefs. Washington, DC: World Bank. Meaning, and Identity. Cambridge: Cambridge Uni- ------. 2007b. Are There New Approaches to the Eval- versity Press. uation of Learning Programs? Washington, DC: Wexley, K., and T. Baldwin. 1986. "Post-training Strate- World Bank. gies for Facilitating Positive Transfer: An Empirical Ex- ------. 2007c. Effectiveness and Impact of WBI FY01­05 ploration." Academy of Management Journal 29 Activities: Results from 25 Focus Countries. Wash- (3): 503­20. ington, DC: World Bank. Wexley, K., and G. Latham. 1981. Developing and Train- ------. 2006. Perspectives on WBI's FY06 Performance ing Human Resources in Organizations. Glenview, in Focus Countries. Washington, DC: World Bank. Illinois: Scott Foresman and Co. ------. 2005. An Evaluation of the Effectiveness and Wexley, K., and W. Memeroff. 1975. "Effectiveness of Outcomes of WBI FY01­03 Activities: Results from 12 Positive Reinforcement and Goal Setting as Methods Focus Countries. Washington, DC: World Bank. of Management Development." Journal of Applied ------. 2004. Learning Outcomes, Course Features, Psychology 60 (4): 446­50. and Learning Factors of WBI Client Learning Wexley, K., and C. Thornton. 1972. "Effect of Verbal Courses. Washington, DC: World Bank. Feedback of Test Result upon Learning." Journal of Educational Research 66 (3): 119­21. (*) (*) Included in the evaluation literature review. 118 IEG PUBLICATIONS Study Series 2004 Annual Review of Development Effectiveness: The Bank's Contributions to Poverty Reduction Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank's Approach to Global Programs Agricultural Extension: The Kenya Experience Assisting Russia's Transition: An Unprecedented Challenge Bangladesh: Progress Through Partnership Brazil: Forging a Strategic Partnership for Results--An OED Evaluation of World Bank Assistance Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy Capacity Building in Africa: An OED Evaluation of World Bank Support The CIGAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research Country Assistance Evaluation Retrospective: OED Self-Evaluation Debt Relief for the Poorest: An OED Review of the HIPC Initiative Developing Towns and Cities: Lessons from Brazil and the Philippines The Drive to Partnership: Aid Coordination and the World Bank Economies in Transition: An OED Evaluation of World Bank Assistance The Effectiveness of World Bank Support for Community-Based and ­Driven Development: An OED Evaluation Evaluating a Decade of World Bank Gender Policy: 1990­99 Evaluation of World Bank Assistance to Pacific Member Countries, 1992­2002 Financial Sector Reform: A Review of World Bank Assistance Financing the Global Benefits of Forests: The Bank's GEF Portfolio and the 1991 Forest Strategy and Its Implementation Fiscal Management in Adjustment Lending IDA's Partnership for Poverty Reduction Improving the Lives of the Poor Through Investment in Cities India: The Dairy Revolution Information Infrastructure: The World Bank Group's Experience Investing in Health: Development Effectiveness in the Health, Nutrition, and Population Sector Jordan: Supporting Stable Development in a Challenging Region Lesotho: Development in a Challenging Environment Mainstreaming Gender in World Bank Lending: An Update Maintaining Momentum to 2015? An Impact Evaluation of Interventions to Improve Maternal and Child Health and Nutrition Outcomes in Bangladesh The Next Ascent: An Evaluation of the Aga Khan Rural Support Program, Pakistan Nongovernmental Organizations in World Bank­Supported Projects: A Review Poland Country Assistance Review: Partnership in a Transition Economy Poverty Reduction in the 1990s: An Evaluation of Strategy and Performance The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank's Support Through 2003 Power for Development: A Review of the World Bank Group's Experience with Private Participation in the Electricity Sector Promoting Environmental Sustainability in Development Putting Social Development to Work for the Poor: An OED Review of World Bank Activities Reforming Agriculture: The World Bank Goes to Market Sharing Knowledge: Innovations and Remaining Challenges Social Funds: Assessing Effectiveness Tunisia: Understanding Successful Socioeconomic Development Uganda: Policy, Participation, People The World Bank's Experience with Post-Conflict Reconstruction The World Bank's Forest Strategy: Striking the Right Balance Zambia Country Assistance Review: Turning an Economy Around Evaluation Country Case Series Bosnia and Herzegovina: Post-Conflict Reconstruction Brazil: Forests in the Balance: Challenges of Conservation with Development Cameroon: Forest Sector Development in a Difficult Political Economy China: From Afforestation to Poverty Alleviation and Natural Forest Management Costa Rica: Forest Strategy and the Evolution of Land Use El Salvador: Post-Conflict Reconstruction India: Alleviating Poverty through Forest Development Indonesia: The Challenges of World Bank Involvement in Forests The Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF Support Uganda: Post-Conflict Reconstruction Proceedings Global Public Policies and Programs: Implications for Financing and Evaluation Lessons of Fiscal Adjustment Lesson from Urban Transport Evaluating the Gender Impact of World Bank Assistance Evaluation and Development: The Institutional Dimension (Transaction Publishers) Evaluation and Poverty Reduction Monitoring & Evaluation Capacity Development in Africa Public Sector Performance--The Critical Role of Evaluation All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual selection, please visit http://www.worldbank.org/ieg ISBN 978-0-8213-7378-1 THEWORLD BANK SKU 17378