60082 S P D I S C U S S I O N PA P E R NO. 1105 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Social Safety Nets Task Teams Gloria Rubio February 2011 Results Readiness in Social Protection and Labor Operations Technical Guidance Notes for Social Safety Nets Task Teams1 Social Safety Nets (SSN) are defined as noncontributory transfer programs targeted to the poor or those vulnerable to poverty and shocks2. About half of World Bank social protection projects in the reviewed cohort are Social Safety Nets. They are mostly nonemergency investment operations with a higher presence in Latin America and the Caribbean and Africa regions. Projects aimed at strengthening country's safety nets system, including their targeting, administration and service quality, are the most common type of SSN interventions (25%). These are closely followed by conditional cash transfers (20%), and health, nutrition and education projects (15%). The remaining projects are a mixture of public works; food crisis mitigation measures and other types of safety nets (social inclusion, housing, and technical assistance). a. Project Development Objectives i. Description of General Practice Project development objectives (PDO) are in line with SSN project types described above. Close to 80 percent of SSN projects have national or sectoral government institutional capacity building as one element of their PDO. This reflects the large proportion of safety net system reform projects in the SP portfolio, as well as widespread efforts to strengthen country's social protection sector institutional capacity through different types of safety net projects. As expected, reaching the poor or specific vulnerable groups is very often part of SSN projects goals (65 percent). Around a third of projects also seek to improve human development outcomes, quality of service delivery or access to public services. Explicit mention of providing temporary income support in PDO is remarkably low ­only 23 percent, considering these are safety net projects. Table 1: SSN Project Development Objectives by Focus Area Access to Access to Temporary Human Reaching Community Local National/ Quality of Fiscal public services economic income developmen specific empowerm governmen sectoral services/ objectives/ and opportun support t outcomes vulnerable ent and t governmen service efficiency infrastructure ities (%) (%) groups/ capacity institutiona t delivery (%) (%) (%) targeted building l capacity institutiona (%) groups (%) building l capacity (%) (%) building (%) Total 30.0 25.0 22.5 32.5 65.0 5.0 12.5 77.5 37.5 15.0 Safety net 20.0 10.0 60.0 10.0 80.0 80.0 30.0 system reform Cash transfers 18.2 18.2 54.5 63.6 72.7 9.1 72.7 9.1 9.1 Health, nutrition and 66.7 33.3 66.7 50.0 16.7 50.0 33.3 education 1 This Note has been prepared by Gloria Rubio. 2 Definition from World Bank Safety Net website 1 Public works 25.0 75.0 25.0 75.0 25.0 75.0 75.0 50.0 Food crisis 25.0 75.0 25.0 25.0 Other 80.0 80.0 60.0 20.0 40.0 20.0 PDO vary substantially across SSN project types. Most safety net system reform projects aim at building institutional capacity at the national or sectoral level; improving targeting mechanisms and quality of social assistance services. Providing temporary income support is a goal found only in public works and cash transfer projects. The latter also seek to enhance human development outcomes as health, nutrition and education projects do. Few SSN projects focus on capacity building at the community and local level except for public works. ii. Assessment of Performance Adequate PDO definition is a key element in results readiness. Well defined, specific, realistic and outcome driven PDO are necessary not only for a well designed intervention, but also for a solid results framework. Overall quality of the general expression of project development objectives in SSN projects is close to satisfactory. Ratings were based on whether PDO were concise and outcome driven, whether they reflect what the project can directly contribute, and whether target groups were identified. Moreover, PDO should be sufficiently well constructed that the main emphasis of a program is easily discernible. Table 2: PDO ratings Frequency Rating All SSN projects SIL DPL ERL Highly Satisfactory 3 2 0 1 Satisfactory 13 11 1 1 Moderately Satisfactory 15 9 3 3 Moderately Unsatisfactory 9 2 5 2 Unsatisfactory 0 0 0 0 Highly unsatisfactory 0 0 0 0 Average score 2.75 2.7 3.4 2.75 There are differences in PDO quality across SSN project types and lending instruments. Investment loans, particularly cash transfer projects tend to have more concise, realistic and outcome driven PDO with well defined target groups. By contrast development policy and emergency recovery loans PDO often lack clarity and specificity on the outcomes they seek to achieve. In general, SSN PDO focus on outcomes for which projects can be reasonably held accountable for. However, there are some exceptions due to projects PDO inadequate specification of their scale or target population. This will be further discussed in the following section. iii. Key Issues 2 Strengthen development outcome focus in PDO, particularly when dealing with institutional reform projects. For example, the first Colombia Safety Net Project does not fully capture its expected development change in its PDO of: "to strengthen the country's social safety net by consolidating and expanding the successful Familias en Accion Conditional Cash Transfer program and improving the monitoring and evaluation of the country's safety net portfolio." This was revised in the second Colombia Safety Net Project when stating "the principal project development objectives are: complement the income of poor families with children; promote human capital formation of poor children by increasing regular checkups, including growth monitoring, and by increasing enrollment and school attendance (basic and/or secondary education); and strengthen program quality." Including also in the PDO the next level in the results chain clarifies the project's outcome orientation and facilitates the identification of key performance indicators (KPI). Adequate specification of project scale and target populations. A number of projects involve pilot testing of new interventions without immediate scaling up. However, this is not acknowledged in the PDO leading to higher expectations than the project could actually accomplish. For example, the Jordan Social Protection Enhancement project PDO is "to improve the management and operations of the cash social assistance programs and to improve the access to and quality of social care services". There is no mention that the second component will be implemented in a pilot basis, thus benefiting only a subset of social care facilities. Adequate specification of project scope is important to have realistic expectations and also keep consistency between PDO and performance measurement framework. Similarly, clarity on PDO target populations helps define better KPI and measurement requirements. Contrasting examples of target population specification can be seen in Table 3. PDO clarity and specificity is also important in development policy and emergency recovery loans. Despite projects different implementation conditions in a crisis situation or in a programmatic context, explicit and well defined objectives are necessary for performance assessment and appropriate course of action. For instance Brazil programmatic loan for Housing Sector Reform is an example of a development policy loan with a clear and specific PDO: "to support the Government's efforts to improve access of the poor to improved housing and serviced land and to promote greater efficiency of the housing finance systems and government subsidies to low income households." In contrast to Ethiopia PRSC 2 PDO "to support improvements in: (i) the investment climate; (ii) governance and service delivery; and (iii) vulnerability, by continuing to support the Government's mediumterm reform program." Appropriate balance between comprehensiveness and capturing the main project emphasis in the PDO. For example, the Social Protection Project PDO list all project components as equally important: "will support Jamaica in strengthening its social protection system. To this end, it will: (i) further improve the effectiveness of the Program for Advancement Through Health and Education (PATH) to foster investment by poor families in human capital accumulation; (ii) develop a structured system for assisting workingage persons of PATH households seek and retain employment; (iii) enable the formulation of a reform program for the pension schemes for public sector workers; and (iv) develop a social protection strategy." However, when 3 reviewing project costs more than 90 percent of resources are allocated to improving PATH effectiveness by applying a new benefits scheme and strengthening institutional capacity. iv. Identification of Good Practice Table 3 illustrates stronger and weaker examples across desirable PDO features such as clarity of development change and target population, as well as project types. Other useful examples of PDO formulation are available in the Technical Guidance Notes for Social Funds, Service Delivery and Labor Markets. Table 3: Examples of Safety Nets projects PDO Type of project/issue Weaker PDO Stronger PDO Outcome focus The Project seeks significant improvement of To improve Honduras social safety net for the effectiveness of Ukraine's social assistance children and youth. This would be achieved by system by better targeting of the cash benefits (i) improving nutritional and basic health status and reduction of burden on beneficiaries of young children through expanding the successful AINC program, and (ii) increasing employability of disadvantaged youth through piloting a First Employment program. Identification of To reduce the risk of HIV infections by scaling The development objective of the SIP is to target population up prevention interventions and to increase improve the living conditions and the social access to and utilization of HIV counseling, inclusion of the most disadvantaged vulnerable testing, care and support services. people in the Romanian society by: (i) improving the living conditions and social inclusion o f Roma living in poor settlements; (ii) increasing the inclusiveness of ECE services in targeted areas; and (iii) improving the quality o f services for PWD, youth at risk and victims of domestic violence. Concise PDO This second phase of the Social Sector To improve the quality of life of people with Programmatic Development Policy Credit disabilities and their families in areas affected (SSPC) series continues, through a DPC by the October 8, 2005 earthquake, by operation, to support Bolivia's efforts to ensuring better mobility, improved physical achieve selected MDGs through the and mental health, increased participation in implementation of a number of key social social and economic life, and strengthened sector policies that were designed during the empowerment. first phase of this program. The SSPC II is expected to support social sector policies with three broad goals in mind: (i) protecting the positive results reached in the last decade in increasing access to and quality of health and nutrition, education, water and sanitation, and social protection; (ii) implementing the sectoral policy frameworks designed under the first phase of this programmatic series to continue progress in social outcomes and move closer to achieving a number of specific Millennium Development Goals (MDGs) by 2015; and (iii) strengthening the capacity of both the Government and civil society to monitor and evaluate policies, increasing accountability and effectiveness. Implementing these policies will advance achievement of specific MDGs by 2015. Development Policy To support improvements in: (i) the To support the Government's efforts to 4 Loan investment climate; (ii) governance and service improve access of the poor to improved delivery; and (iii) vulnerability, by continuing to housing and serviced land and to promote support the Government's mediumterm greater efficiency of the housing finance reform program. systems and government subsidies to low income households. Emergency Recovery To accelerate and improve the quality of the Improve conflictaffected communities' and Loan implementation of the package of social sector individuals' opportunities for economic reforms presented by GOL at the Paris III reintegration and access to social services, Donor Conference in the areas of social which will accelerate Cote d'Ivoire's crisis insurance, safety nets and health expenditures. recovery and strengthen the prospects for sustainable peace in its territory Food crisis The Food Crisis Response Development Policy To mitigate the negative nutritional impact of Grant of US$3 million is a development policy the food price increase on preprimary and grant intended to support the Government's primary school children and to promote poor poverty reduction strategy by providing the rural household's food security by increasing authorities with needed fiscal space to partially agricultural production compensate for the lost revenues resulting from the recently reduced tariffs on food and fuel imports. Cash transfers The development objective of the Project is to The principal project development objectives improve the education, health and nutrition of are: complement the income of poor families Salvadorian children living in the rural areas of with children; promote human capital 100 poorest municipalities. formation of poor children by increasing regular checkups, including growth monitoring, and by increasing enrollment and school attendance (basic and/or secondary education); and strengthen program quality Public works (i) reduce poverty by supporting the workfare component of the Heads of Household Program, including continued improvements in Program management and governance; and (ii) promote activities aimed at getting more beneficiaries of this Program into fulltime employment, thereby enabling them to graduate from the Program. Activities to promote labor market participation by beneficiaries would include: (i) workfare sub projects designed especially to promote employability, for example, by combining them with training, skill certification, and targets for job placement; (ii) support for productive microprojects; and (iii) the establishment and support to municipal offices of employment services. b. Key Performance Indicators i. Description of General Practice Key performance indicators (KPI) are designed to measure a project's performance in meeting its intended outputs and outcomes. General observations on the use of KPI in SSN projects include: KPI distribution closely mirrors SSN PDO classification analyzed in the previous section. The most frequent KPI were aimed at measuring national or sectoral institutional capacity 5 building, followed by KPI assessing the targeting of the poor population or other vulnerable groups (see Table 3). In contrast, community empowerment and capacity building appears least as part of PDO and in terms of frequency of KPI. SSN have on average a smaller number of KPI per project compared with other social protection interventions. On average SSN projects include 10 indicators to measure its performance, compared to an average of 15 indicators per social protection project. SSN projects rely on average on 4 outcome indicators and the rest are intermediate or output indicators. Although there is no rule about the ideal number and mix of indicators, an average of 10 KPI per project seems within the range of a manageable number of indicators for frequent monitoring but still comprehensive enough for a meaningful performance measurement. A fairly small share of KPI are not directly related to PDO areas. Only around 3 percent of outcome and intermediate KPI were outside of the main SSN project development areas. It seems that SSN performance measurement efforts are very much focused on reporting PDO results. SSN projects rely considerably on institutional development indicators. Two thirds of intermediate indicators are aimed at measuring institutional development aspects. In particular, the most common indicators are those related to service delivery (28%) and institutional capacity building (22%). Safety net system reform and cash transfers projects account for more than half of institutional development indicators used in safety nets. Table 4: Distribution and Quality of SSN Indicators Frequency Clear link to Average Share of Share of PDO rating SMART indicators indicators rating with target reported in values ISRs established Outcome indicators 166 2.22 2.77 0.75 0.73 Access to public services and infrastructure (schools, health 11 2.0 2.1 0.91 0.78 centers, roads etc.) ­ local public goods Access to economic opportunities (microcredit, active labor 9 2.11 2.67 0.67 0.67 market programs, skills development, etc.) Temporary income support (Unconditional transfers, public works and temporary employment, CCTs, wage subsidies, 28 2.33 2.57 0.82 0.69 etc.) Human development outcomes (improved education, health 50 2.08 2.71 0.60 0.88 and nutrition, HIV status etc.) Reaching specific vulnerable groups/targeted groups 47 2.11 2.41 0.79 0.67 Community empowerment and capacity building (non 1 3.0 4.0 1.0 1.0 government) Local government institutional capacity building (anything 4 2.75 4.25 0.75 0.5 subnational) National/sectoral government institutional capacity building (benefits administration, management and operation of 22 2.45 3.55 0.77 0.50 programs, targeting system,etc ) Quality of services/service delivery 24 2.5 2.5 1 0.75 Fiscal objectives/efficiency 14 2.08 3.0 0.57 0.36 6 Other 5 2.6 3.8 0 0.60 Intermediate outcome indicators 236 2.42 3.14 0.80 0.74 Access to public services and infrastructure (schools, health 25 2.22 2.42 0.56 0.65 centers, roads etc.) ­ local public goods Access to economic opportunities (microcredit, active labor 3 2.0 1.0 1.0 market programs, skills development, etc.) Temporary income support (Unconditional transfers, public works and temporary employment, CCTs, wage subsidies, 10 2.7 2.9 0.8 1.0 etc.) Human development outcomes (improved education, health 18 2.28 2.63 0.89 1.0 and nutrition, HIV status etc.) Reaching specific vulnerable groups/targeted groups 30 2.17 2.53 0.77 0.63 Community empowerment and capacity building (non 7 2.14 2.57 1.0 0.67 government) Local government institutional capacity building (anything 10 2.60 3.60 0.80 0.88 subnational) National/sectoral government institutional capacity building (benefits administration, management and operation of 86 2.40 3.39 0.76 0.62 programs, targeting system,etc ) Quality of services/service delivery 52 2.33 3.08 0.85 0.70 Fiscal objectives/efficiency 18 2.50 3.39 1.0 0.43 Other 13 3.0 4.0 .75 0.42 ii. Assessment of performance Project results readiness depends heavily on appropriate selection and definition of KPI and their satisfactory operationalization. Thus, KPI quality is assessed at project design as well as during project implementation. Project preparation should include the selection of a manageable number of specific, measurable, attributable, realistic and targeted (SMART) key performance indicators outcome and intermediate that are relevant to PDO. In addition, it is important that KPI baseline values are known at project design and taken into consideration in setting targets. Early identification of KPI data sources as well as clear responsibility for their measurement improves the likelihood of actually using them. Overall quality of KPI at project design is close to satisfactory. These are the main findings: Clarity on KPI linkage to PDO is fairly common. Two thirds of outcome indicators have a satisfactory or highly satisfactory clarity in their linkage with PDO. This percentage is lower in the case of intermediate indicators (57%). SMART rating of outcome indicators is close to satisfactory (2.77) and is higher compared to intermediate indicators (3.14). Outcome indicators related to access to public services, targeting and service delivery tend to be SMARTer compared to capacity building and fiscal objectives or efficiency indicators. A similar pattern is observed among intermediate indicators, except for service delivery indicators that worsen their SMART ratings when measuring project outputs. Institutional development indicators are responsible for the lower ratings of intermediate indicators (3.27 institutional development indicators vs. 2.89 remaining intermediate indicators) 7 One third of outcome indicators and close to half of intermediate indicators have specified a baseline value for performance measurement in the Project Appraisal Document (PAD); a larger percentage of indicators do specify a target value. Despite not having a baseline 10 and 25 percentage points more outcome and intermediate indicators respectively have set a target. This raises the question of how targets are set and how realistic they are. About half of indicators have adequately specified the data source for their measurement. Lack of data source specification is particularly an issue for development policy loans, possibly due to deficiencies in the results framework guidelines for this type of lending. Another common problem is vagueness in data source specification. For example, one project states "document available by the end of 2007" or "information provided by MSP" as data sources. This lack of clarity may result in a hard to measure indicator in a timely fashion. Good KPI are those that being SMART and relevant to PDO are actually used for timely performance measurement and taking corrective action. Unfortunately, SSN KPI operationalization tends to be moderately satisfactory. These are the main findings. Around 70 percent of indicators selected in the PAD were entered into Implementation Supervision Reports (ISRs) for eventual tracking. Outcome and intermediate indicators are equally likely to be reported during project implementation. However, there is some variability in KPI reporting in ISRs across topic areas and project types. Safety nets system reform projects have a lower percentage of indicators followed through compared to cash transfers, public works and health, nutrition and education projects. This is consistent with a lower percentage of indicators related to national or sectoral institutional capacity building or to fiscal objectives and efficiency being reported during project implementation. However, few projects update their KPI regularly during implementation and use this information to measure progress and inform decisions. Less than one third of SSN projects update regularly their KPI and report them in the ISR. Only 16% of projects have a satisfactory or highly satisfactory use of KPI when reporting their progress and taking corrective action. iii. Key Issues Deficient measurability is the most common problem in SSN projects KPI. There are a number of examples of KPI that are unclear about how they will be measured like "number of targeted municipalities with adequate provision of education and health and nutrition services" or "percentage of BISP positions that are filled with qualified staff". How "adequate provision of services" or "qualified staff" are defined have very different implications on the data sources needed to measure these indicators, as well as the feasible frequency for their measurement and their cost. In addition, the lack of specificity 8 may lead to different measurements of the same indicator depending on the interpretation of whoever quantifies it. A better definition of the second indicator would be the percentage of BISP technical staff in the central and local offices that have been certified according to a specific "X" evaluation process. Frequency of indicators measurement during project life span should be considered when selecting the mix of intermediate and outcome KPI. Many indicators, particularly outcome indicators that rely on specialized surveys, are at best measured annually and sometimes can only be measured once during the project life. One common example in SSN projects are poverty targeting indicators such as the percentage of program cash transfers received by households in the lowest quintile of the population. This type of indicators usually requires a nationally representative household survey measuring income or consumption and including a social programs module. Since most likely this indicator can only be measured once every two years, it is important to complement it with an intermediate indicator that can be measured more often and may be a proxy for program targeting such as the percentage of program beneficiaries that meet the eligibility criteria according to a performance audit conducted twice a year. Some basic KPI such as program coverage are missing or not well defined. A number of projects lack well defined coverage indicators that are critical for frequent project performance monitoring. Measuring the covered population is a useful short term progress indicator and a necessary condition (but not sufficient) for achieving the PDO. It is preferable to measure it as a proportion of the project target population, but if using the absolute number of beneficiaries there should be a target value. Beneficiary satisfaction indicators need to be strengthened. Indicators such as "percentage of beneficiaries satisfied with program implementation" or "level of satisfaction with payments" are usually not very informative. One alternative is to address quality issues by identifying key processes in beneficiaries' service quality perception and measuring their adequacy or efficiency. Failure to adequately specify target population. For example, health, nutrition and education indicators are often set too broadly, like number of children having received full set of immunization or malnutrition rate, without identifying if this is among a specific subset of the population (e.g. targeted regions, households in bottom quintile, etc.).Also, there is a widespread use of generic terms like `among the poor', or `vulnerable groups' that fail to sufficiently identify the target population, as in for instance, percentage decrease of parents from vulnerable groups perceiving discriminatory teaching practices. There are a number of challenges associated particularly with institutional development indicators. Using goal statements instead of measurable variables, for example, "improved design and implementation of the wage subsidy and unemployment subsidy programs" or "automated management information system effectively implemented". A specific way to measure these is required for a SMART KPI. Using project activities as indicators such as "standard office model is developed, piloted, evaluated and formally approved" or "NAF adopts an Operations Manual 9 governing its business processes". In some cases, these may serve as implementation milestones to track project execution progress, but they are not adequate measures of medium term PDO achievement. A better alternative would be, for example, the proportion of NAF business processes having an ISO 9000 certification. iv. Identification of Good Practice Annex 2 presents a list of indicators found in SP projects by focus area. This section will focus primarily on institutional development indicators given their broad use and need for improvement. As mentioned earlier, a large number of SSN project seek to strengthen institutional capacity to provide well targeted, efficient, good quality, and transparent social assistance services. In addition, there are widespread efforts to improve program or policy accountability through evaluation, and some attempts to strengthen policy analysis and design capabilities. All these activities require specific KPI that can be grouped as follow. Table 5 presents weaker and stronger examples of institutional development KPI by type. MIS or beneficiary registry development indicators that help assess progress in building reliable and timely administrative records of who is getting what program benefit. Service delivery quality indicators including compliance with service standards or models indicators and processes efficiency. They can be a complement to beneficiary satisfaction measurements to gauge the quality dimension of service provision. Transparency and accountability indicators that seek to assess performance in promoting monitoring and evaluation as well as access to program basic information. Social mobilization, participation and public awareness indicators aim at measuring institutional efforts to foster social awareness and participation. Institutional capacity and structure indicators measure improvements in human resources skills and capabilities as well and organizational issues. Improved policy indicators can be found in projects involving technical assistance to assess progress in implementing better planned, coordinated, or more effective policies. Table 5: Examples of Safety Nets projects institutional development KPI Type of project/issue Weaker KPI Stronger KPI Targeting indicators Percentage of social assistance cash Percentage of disbursed BISP cash benefits received by poor families transfers received by beneficiaries in Number of low income youth enrolled in quintiles 1 and 2 the program during expansion % of all revised SISBEN 1 families in newly Number of households to which the entered municipalities that are registered poverty scorecard has been applied in the program. A pilot SSN mechanism will have been % recertification of PATH families with implemented using a robust targeting children due to be recertified after 4 years formula. The pilot project will have been enrolment evaluated and lessons learned/best 10 practices extracted for national rollout MIS development A management information system that Percentage of households in bottom provides sufficient and timely information quintile included in program beneficiary has been established and is operational database Social Development Database improved Number of quarterly progress reports MIS system captures disability specific produced by the MIS, according to agreed information in adequate detail reporting standards Registry of beneficiaries designed and Duplication rate in beneficiary registry tested, and beneficiaries of the ZMP, FEP, Average time taken to update in the MIS PROPAS, JPP programs registered changes in household composition (newborns or deaths) or address changes Percentage of cash transfers made automatically through MIS Service delivery Compliance with standards/attention models Compliance with standards quality Number of primary County CSWs that % of social welfare facilities that have have a fully functioning one stop shop acceptable hygienic standards in place to model meet national criteria Percentage of key standards of care Process efficiency/adequacy adopted Percentage of NAF beneficiaries that use Three pilot integrated social work and smart cards to collect assistance care centers fully functional and Percentage of grievance redressal claims adequately funded by the Government settled within three months of application Minimum standards of care for working Average enrollment application with street children are developed and processing time applied across implementing NGOs % of cash transfer payments delivered to Process efficiency/adequacy local offices by the 13th of the payment Number of complaints by types month Transparency and Level of transparency on budget % of social programs that update accountability allocations and execution quarterly their budget allocation and Summary payment information available execution information on internet for public review % of target population aware of eligibility Two (2) program impact evaluation criteria for social assistance benefits studies completed in 2006 and 2008 Results based indicators of four main social assistance programs publicly available M&E unit in MOSD issues its annual report System of process evaluation, internal audit and quality controls with spot checks in place Periodic evaluations of progress made in achieving the MDGs followed by participatory discussions and feedback to improve social sectors policies Social mobilization, Public awareness increases participation and Public information and education strategy public awareness developed and approved in 2006 Beneficiary information booklets designed, published and distributed to new municipalities with training Number of annual consultations of Roma in poor settlements with local authorities for addressing community needs Institutional capacity Number of oblast social welfare offices At least 70 percent of the MoSD staff and structure capable of carrying out analyses and completes training envisaged in the monitoring of the benefit payments Human Resources Development Strategy 11 Functional review of MOLSP's DSDS % of staff with competences in integrated completed and business plan prepared in services according to official examination 2006 process MoSD adopts a Strategic Development Plan and Human Resources Development Strategy Standard packages for statistical data entry, cleaning, imputation, management and analysis introduced and staff trained in using them. NAF HQ and local branches adequately staffed and staff skills improved. % increase in number of qualified staff in PWD services Improved policy Cabinet adopts consolidation of selected federal Social Safety Net Programs under BISP Integration of the existing network of service providers to vulnerable population within the Social Protection System. Draft social protection policy for children and youth w/3 year implementation plan which defines roles, responsibilities and products of participating institutions Mediumterm strategy for reintegration of other conflictaffected populations defined c. Design and implementation of M&E i. Description of General Practice M&E systems include different components aimed at collecting and systematizing data required for project management, monitoring and evaluation, as well as different assessment tools depending on project performance information needs. All SSN projects have at least one additional M&E component, most frequently Management Information Systems or beneficiary registries (75%). Some form of operational assessment through process evaluation (38%), spot checks (30%) or technical/operational audits (30%) is often included. About half rely on impact evaluations, though the robustness is not always evident in the PAD. Qualitative approaches are also common through beneficiary assessments and participatory M&E. Substantial heterogeneity in SSN projects prevents the identification of a core package of M&E tools. Hence, it is more useful to analyze variations in M&E components by project type. 12 Table 6. Use of M&E Components by SSN Project Type (%) MIS Process Beneficiary Spot Impact Technical/ Participatory Expenditure Other evaluation Assessment checks Evaluation operational M&E Tracking tools audits Stdies Total 75 38 43 30 55 30 33 8 40 Safety net system 80 30 30 10 40 40 50 reform Cash transfers 91 55 55 55 82 55 45 27 Health, nutrition 50 33 33 17 33 33 17 50 and education Public works 75 50 75 25 75 75 50 25 50 Food crisis 50 25 25 75 25 75 Other 80 40 40 40 80 Investment operations include on average 4 different M&E components, twice as much as in development policy loans. Cash transfers and public works projects count on a larger variety of M&E tools and have a clearer pattern for M&E arrangements. A typical cash transfer program has an M&E system composed of a beneficiary registry and an impact evaluation, a few exceptions are found in emergency context interventions3. This reflects the operational need of having an adequate registry of transfer recipients and CCT's well established impact evaluation tradition. Most of them also include some tool for assessing their operation such as a process evaluation, spot checks, or operational audits. These are different means to verify the correct and timely delivery of transfers, as well as the implementation of conditionalities, in the case of CCTs. In addition, cash transfer projects benefit from direct feedback from participants through beneficiary assessment or participatory M&E. Public works projects also have well defined M&E systems aligned with its operational demands. They usually include an MIS system and technical audits for management and quality control of public works subprojects, as well as beneficiary assessments. Safety net system reform projects have a clear tendency to include an MIS system, except in the case of development policy loans. However, their use of other evaluation tools is less systematic. The scatter use of M&E components in HNE projects reflects the heterogeneity of interventions within this group. Despite the urgency circumstances of the food crisis projects, all of them have some kind of basic M&E arrangements relying on existing systems, use of audits and other M&E tools. There could be substantial learning across SP projects in using different M&E tools. For example, social funds long acquired experience in developing MIS in different country settings can be useful for safety net system reform or public works projects. Likewise, valuable lessons can be drawn from service delivery M&E systems, particularly for conditional cash transfers, social assistance and HNE safety nets (see respective Technical Guidance Notes). 3 The Maldives Post Tsunami Emergency project only includes audits. 13 Cost information on M&E component is scarce. About half of the projects have specified M&E costs, but only in 28% of projects it is not mixed with other costs. Costs range from half a million to 10 million dollars with an average of 1.2 million dollars4. ii. Assessment of performance Successful operationalization of an M&E system depends not only on well chosen KPI and assessment tools, but also on adequate institutional arrangements and capacity to manage the necessary design, implementation and feedback processes. Thus, it is important to assess institutional resources (organization, skills, etc.) available to implement the proposed M&E system and consider well in advance the necessary actions ­for instance, tailored technical assistance to strengthen it. Furthermore, it is critical to have early on clear roles and responsibilities on data collection, analysis and reporting, as well as overall management and supervision, and a defined process for M&E results discussion and feedback. M&E arrangements in SSN projects need to be strengthened to ensure a successful operationalization. Less than half of projects had a definition of institutional arrangements rated at least satisfactory5. Plans to strengthen client capacity are missing or need improvement in 47% of projects. Performance ratings vary considerably across lending instrument. While the overall rating for M&E arrangements among investment operations is 2.13, it is 3.89 for DPLs and 3.71 for ERL. This probably reflects the short term relief focus of ERLs as opposed to medium and long term perspective of investment operations more amenable for institutional capacity building. It is surprising, however, to observe such low ratings for DPLs considering their broader institutional and policy focus. Table 7: Quality of M&E Arrangements by lending instrument (Ratings) Institutional PAD Plan to Project M&E Overall M&E arrangements for analysis strengthen integration with arrangements performing M&E of M&E client national/sectoral activities client capacity M&E system capacity SIL 2.3 2.4 2.1 2.5 2.1 DPL 4.1 3.9 3.7 4.1 3.9 ERL 4.0 3.7 3.4 3.7 3.7 TOTAL 3.0 3.0 2.7 3.1 2.8 Ideally and in the medium term, project M&E systems may be integrated with sectoral or national systems to exploit synergies in data collection, promote institutional coordination and broaden policy discussion. Close to 40 percent of SSN projects are satisfactorily integrated with higher level M&E systems. Again, this is more common among investment operations compared to emergency 4 Average based on 11 observations. 5 This assessment is based on information included in Project Appraisal Document. 14 interventions for the same reasons discussed earlier. Here too, DPLs low performance is a missed opportunity to strengthen countries M&E activities. iii. Experience with impact evaluation Although 55% of SSN projects mentioned that they plan to conduct an impact evaluation, only one third provided detailed methodology information in the PAD to consider them potentially robust impact evaluations. Robust impact evaluations are concentrated among cash transfers projects, but other types of projects have at least one case, except food crisis interventions. Most common impact evaluation methodologies are randomization and regression discontinuity. Once a robust impact evaluation is planned, most likely there will also be early plans to collect baseline and follow up data. Few projects have explicitly costed impact evaluations. The average cost found is $850,0006. Table 8: SSN Projects with robust impact evaluation Projects with Impact evaluation methods used Projects baseline and Average IE Projects with (Frequency) with a follow up cost Summary IE explicitly rigorous IE data (thousands quality rating Regression costed Randomization Matching Other planned collection USD) discontinuity planned 33% 85% 4 850 2.3 7 3 5 3 iv. Key Issues The main issues in M&E design and implementation include: Mix of evaluation components respond to project information needs at a particular point in time: M&E systems evolve as operation consolidates and new operational or impact issues arise. Emphasis on developing an MIS system or selecting a particular evaluation tool depends on the operational and performance information needs of projects. These conditions are not static and will lead to M&E system evolution in time, see for example, Ethiopia Productive Safety Net in Box 2. Early and careful planning is key in setting up an M&E system. Project design should involve a thorough consideration of technical, institutional and political aspects of M&E. Existing M&E activities, arrangements and capacity should be assessed to ensure appropriate resources and actions to support the development of the proposed M&E system. Most common delays in implementing M&E activities are due to administrative matters or problems with data collection. Many times, delays occur due to difficulties in developing the required Terms of Reference or lack of experience with the procurement process. Set a realistic timeframe for M&E activities contracting process and ensure needed technical 6 Average based on 4 observations. 15 support. Many things can go wrong during filed work. Keep instruments as simple as possible, simplify data collection process and allow extra time for unforeseeable situations (see Box 1). Project implementation pace has an impact on M&E system operationalization; project supervision should consider this to propose needed adjustments. For example, Colombia Safety Net rapid implementation compromised the integrity of the impact evaluation control group. Thus, it is important to agree well in advance on an implementation schedule that is consistent with planned M&E activities and monitor carefully its observance. In other cases, delays in project implementation result in delays in starting M&E activities. Project supervision should consider how implementation delays will affect planned M&E activities, flag this issue with clients and take appropriate actions. DPL broader focus on policy issues could provide an opportunity to integrate M&E activities with sectoral and national systems. Among the different lending instruments, DPL are probably the most amenable to promoting the integration of M&E systems at the sectoral or even national level. It is important to review DPL guidelines on M&E issues to seize this opportunity. Box 1: Key steps in operationalizing M&E activities Once it has been defined project performance information needs and corresponding evaluation tools, it is important to set a realistic timeframe to implement them and ensure timely results for decision making. Some of the key steps, time frame and technical assistance required in each of them are: 1. Institutional arrangements and capacity assessment: Identify the unit or individual responsible for M&E implementation, reviewing institutional risk factors in carrying out M&E activities such as lack of technical staff, high staff turnover, increased workloads, or low decision making power. Based on this, anticipate personnel requirements, technical assistance or training needs at each of the key implementation steps, as well as other institutional risk mitigation mechanisms. 2. Developing TOR: Developing TORs that are comprehensive, clear and realistic involve a fair amount of knowledge and experience with the technical requirements of the evaluation tool used, as well as the individuals' qualifications and level of effort required to apply it, and the timeframe involved in the activities. Many times this experience is limited, thus different degrees of technical assistance, ranging from providing TOR models and examples to hiring a consultant, may be needed to help client countries. Depending on the complexity of the assessment and the stakeholders involved in the evaluation, it may take one to two months to develop and agree on the TOR. 3. Procurement process: Depending on the cost of the M&E activity, different procurement arrangements involving very diverse timeframes will be required. Task involving an individual consultant for a few months may be contracted within a shorter period of time if they can be procured by using a selection based on consultant qualifications. Higher cost activities such as impact evaluations that involve data collection and a larger team to implement them may require a more complex procurement process that may take a few months. Make sure that individuals managing the procurement process are familiar with the Bank´s procurement guidelines and know who to contact in case they need any support. 4. Methodological report and implementation plan: Once the consultant or firm is on board allow one to three months, depending on M&E activity complexity, to prepare a methodological report and implementation plan. Early agreement with the evaluator on a clear outline of the methodology to be used, sample design, information needs, key activities and timeframe that will be required is critical for evaluation success. If the consultancy involves data collection make sure that there is a careful implementation plan including data collection instruments design, pilot test, 16 interviewers training, field work, verification and data entry and cleaning. 5. Data collection: this is a key activity that requires a well designed and pilot tested data collection instrument; adequately trained interviewers; carefully managed and supervised field work and quality control mechanisms for data collection and entry. Data cleaning and data set preparation are also very important. Depending on evaluation complexity and scope allow two to six months for each round of data collection. When planning data collection consider events such as rainy season, holidays, elections that may affect the implementation schedule. 6. Data analysis and reporting: This should take one to three months depending on the complexity of the analysis and the evaluation team composition. Consider having a Power point presentation with the main results to initiate the evaluation discussion, highlighting projects strengths, weakness and recommendations for design or operation improvement. In addition, there should be a main report including an executive summary and methodological annex. Data sets, including codebook and instructions to use them should be delivered as part of the final product. 7. Use of results: Make sure evaluation results are widely discussed with stakeholders and there is a well defined feedback process including a concrete plan to implement the main evaluation recommendations. 8. Sustainability and institutionalization: Discuss the convenience of implementing regularly selected M&E activities, agreeing on the scope, frequency, individuals or units responsible, budget allocation and feedback process. v. Identification of Good Practice The Jamaica Social Protection Project provides a comprehensive and well described system for monitoring and evaluation that is detailed separately as part of Annex 3. This includes the rationale and description of components and methodologies for all of the M&E instruments highlighting the complementarities among them. The M&E system encompasses: (a) an enhanced MIS, more efficient and easier to operate; (b) process evaluations to examine quality of service delivery, develop a proposal of service standards, and inform the design of an internal system of "spot checks"; (c) an impact evaluation to assess the effect of program redesign including introduction of increased benefits, differentiated benefits for secondary school and higher benefits for boys7; (e) assessment of program targeting accuracy; and (f) strengthening of the internal audit capacity. In addition, annual independent audits conducted by the Auditor General will also provide useful information on project implementation progress. In addition to this overall strong example of M&E at the design stage, this review highlights some interesting components or approaches, including8: The Argentina Heads of Household Transition Project PAD has an effective monitoring strategy that combines administrative data and a permanent household survey. Administrative data enables the Ministry and the Bank to monitor closely trends in participation, overall eligibility compliance, and program dropouts. In addition, program beneficiaries are surveyed quarterly by the National Statistical Institute (INDEC) as part of 7 The PAD also considered an impact evaluation of the stepstowork pilot initiative, but was postponed by the Government until the new program component has been rolled out. 8 For specific guidance on impact evaluation see World Bank PREM Impact Evaluation website. 17 the labor force survey. Survey data helps crosschecking administrative records by providing independent and statistically representative (for urban areas) information on participants' income levels; whether they meet certain eligibility requirements (have a minor or handicapped person in the household); have a job with social benefits; and how many hours they are working in an eligible activity or attending school. Survey data is also a tool for the public at large to track the overall program performance. Also of note is the approach of the Nigeria HIV/AIDS Program Development Project to integrate a national M&E system. The first phase of the project established most of the elements of an effective national M&E system, the Nigerian National Response Information Management System (NNRIM). The second phase will support state, local and civil society level M&E, scaling up existing capacity through technical assistance. Major emphasis will be on data management, dissemination and use. Surveillance will be strengthened through existing data including National Demographic and Health Survey (NDHS) and social behavioral research. d. Use of results i. Description of General Practice The value of a resultsfocused approach depends on the extent to which results are used as a management tool. Monitoring and evaluation is not a stand alone, technical activity, but rather closely linked to decisionmaking processes at all levels. Information on results provides feedback on program performance to project managers, policy makers, civil society and researchers. The use of results by Bank task teams and management has two main objectives: (i) to `prove' (accountability), or measure if objectives are being achieved and gauge project performance, and (ii) to `improve' (management), or identify issues that need to be addressed to improve performance. Once effective, WB projects implementation are supervised at least twice a year. Implementation Status and Results Reports are the main documents that track implementation advance and progress towards PDO achievement. At the end of the project results and implementation are assessed through an Implementation Completion Report. These two are the main venues for reporting KPI monitoring and evaluation results. In some cases, projects with interesting features or good results may be featured as good practices in seminars or toolkits and their results get more widely disseminated. There are also a few examples such as CCTs or social funds in which substantial performance information has been accumulated through M&E and it has been systematized in a metaanalysis. 18 ii. Assessment of performance Of the 40 SSN projects reviewed close to half of them have been effective long enough to report KPI progress in the ISR and only 5 have an ICR. Based on these documents, general performance on results monitoring and its use by the Bank is moderately satisfactory9. Lack of reporting of KPI and failure to fulfill data collection plans were critical aspects behind this rating. Only 11 percent of SSN projects where ISRs were available carried out the data collection plans outlined in the PAD satisfactorily. Typical problems were delays in establishing MIS systems capable of reporting consistently on key output indicators, as well as delays in contracting the various external studies, like impact evaluations or baselines surveys. As discussed earlier, regular updating of KPIs is not common practice among SSN projects. Only 29 percent did it in a satisfactory way, and even a lower percentage (16%) actually used them to measure progress and take corrective action when needed. Lack of reporting is usually associated with delays in data collection or disregard of the frequency of data availability, and in some cases is due to inadequate selection of indicators. Table 9: Results readiness at project implementation Was data Are KPI Are KPIs Are M&E Is Bank Summary collection updated used to issues management rating of plan in PAD regularly? measure discussed in commenting results actually % progress "key issues on results and monitoring implemented? and for M&E issues? and use by % inform management % the Bank decisions? % % % Highly Satisfactory 0.0 4.2 4 6.1 3.2 3.3 Satisfactory 10.5 25.0 12 30.3 38.7 13.3 Moderately Satisfactory 42.1 16.7 48 27.3 16.1 40.0 Moderately Unsatisfactory 15.8 12.5 8 15.2 25.8 30.0 Unsatisfactory 21.1 20.8 16 6.1 3.2 10.0 Highly unsatisfactory 10.5 20.8 12 15.2 12.9 3.3 Average Rating 3.8 3.8 3.6 3.3 3.3 3.4 SIL 3.5 3.5 3.1 2.7 2.7 3.1 DPL 3.5 4.7 4.6 5.0 4.6 4.0 ERL 5.3 3.3 3.5 3.5 3.6 3.6 Attention to M&E issues by task teams and Bank management also needs to be strengthened. There is some disconnect between M&E rating reported in ISR and what is happening with KPI updating and progress in implementing data collection plans. While 60 percent of projects reported a satisfactory M&E implementation in the ISR, only 27 percent of projects had at least a 9 Since the assessment of this section is based solely on ISR and ICR reports, the analysis of how results are used as a management tool look only at the use of results by Bank task teams and management, not by Governments. 19 moderately satisfactory progress on data collection or KPI monitoring. Adjustments to ISR M&E ratings to reflect project progress in M&E activities were not a generalized practice. iii. Key Issues The main issues in terms of use of results by the Bank include: KPI, particularly intermediate indicators, should be a key project management tool routinely used during project implementation. Naturally, project supervision is largely focused on disbursement and implementation milestones. However, results readiness also involves a great deal of attention by Task Teams and Bank Management to other middle links in the results chain. Intermediate indicators such as coverage or service/benefits delivery measure critical aspects in achieving the PDO. Hence, their use in the ISR is important for a resultsbased project management. Use of outcome indicators during project implementation may be limited by frequency of data availability, but should be reported in ICR. In many cases, data sources for outcome indicators are household surveys or impact evaluations that are only available near project closing or afterwards. This frequency of measurement should be acknowledged at project design and in the ISR, clarifying expectations about the reporting and use of the indicator. As discussed earlier, it is important to ensure a good mix of indicators including complementary intermediate indicators that can be used to track progress in achieving the PDO during project implementation. M&E rating in ISR is an opportunity to flag issues related to successful implementation of data collection plans and M&E activities as well as KPI regular update. Early planning of M&E activities together with close implementation follow up are crucial to ensure timely availability of performance information. Delays in data collection or M&E implementation should be detected as soon as possible and corrective action taken. PDO ratings in ISR should be wellsubstantiated by KPI reporting. In cases where KPI were not consistently reported, it is difficult to see how PDO ratings are justified. Use of performance and results information requires adequate incentives. Bank management attention to KPI reporting, M&E implementation, as well as PDO rating substantiation on performance information is critical for results readiness. ISR and ICR are instruments already available to foster results focus during project implementation, as well as in future Bank interventions. 20 iv. Identification of Good Practice Good practice examples of task team and management use of M&E results include: The Romania Social Inclusion Project ISRs shows a very proactive approach by the Bank Team. For example, the M&E rating has been downgraded to MU due to delays in launching baseline M&E studies. A few months later, it is reported that none of the project components had initiated any M&E activity, although a consultant was contracted to help with the baseline survey. Unfortunately, the consultant failed to fulfill his contractual obligations and his contract was terminated. It was thus decided that each project component will take care by its own M&E activities. Finally, two of the three project components manage to have a baseline. Ethiopia Productive Safety Net project is an example of effective use of M&E results throughout project cycle and different phases (see Box 2). Early and effective feedback was possible before fully computerized MIS can be set up thanks to a combination of real time data coming in through the Information Center and a Rapid Response Team in charge of spotting and addressing implementation bottlenecks. Task team supervision is clearly resultsoriented. Although KPI are mostly discussed in the ICR, there is continuous and wellsubstantiated report on progress towards PDO using M&E results from project public works review and panel surveys. Potential threats to PDO achievement and proposed corrective actions are followed through ISRs, with a constant reflection on how to improve the next phase. Recently, Bank management has engaged in fruitful discussion with task team on project results. Box 2: Effective use of M&E results in Ethiopia Productive Safety Net Project Ethiopia faces the challenge of improving food security for a large number of chronically food insecure households and stabilizing the long term trend of increasing numbers of food insecure people. PSNP seeks to address this challenge in three phases. The first phase (20052006) was aimed at supporting the transition from a relieforiented to a productive and developmentoriented safety net by providing predictable, multiannual resources, and replacing food with grants as the primary medium of support. The second phase (20072009) was conceived as a consolidation stage to improve the efficiency, effectiveness and fairness of the program. A third phase (20102014) was added later on to fully focus on households' graduation out of chronic food insecurity. Performance information needs and feasible M&E tools in each phase are different. In phase I, M&E information focused on assessing progress on getting the new scheme up and running by shifting to public works and cash, with improvements in predictability, targeting, participatory planning of public works and better productivity of community investments. Thus, M&E system original components included: MIS to report process and output indicators, including financial and physical progress Beneficiary database and automated payroll Periodic technical reviews of the quality and sustainability of public works Annual needs assessment to dimension appropriate safety net response A series of evaluation studies including baseline beneficiary survey; studies on poverty targeting and program institutional linkages; process evaluation; local grain market analysis; a random audit of transfers and a beneficiary assessment. 21 During this phase implementation, MIS remained a cumbersome manual compilation of reports from lower levels rather than a fully automated system. To compensate it, two innovative M&E elements to monitor humanitarian risk were added: Information Center to collect realtime information from a sample of districts on transfers of cash and food and grain market prices Rapid Response Teams at federal and regional levels comprised of Government and donor staff to detect and respond to bottlenecks. The combination of the Rapid Response Teams, the real time data and the evaluation studies carried out in the first 18 months of operations was very effective at delivering feedback to program managers early on and inform phase II adjustments. For instance, due to seasonal increases in food prices in some localities, some beneficiaries switched back to food for part of the year. This flexibility was deemed desirable and was included in program design. In addition, the targeting study found that almost 90% of recipients were classified as food insecure, indicating that communitybased targeting was working reasonably well. The public works review revealed that only 56% of subprojects were assessed as technically sound, which lead to the establishment of regional teams in charge of technical planning, oversight and backstopping districts in the implementation of high quality PW subprojects. Phase II plans to strengthen and complement previous M&E activities, now with a stronger focus on measuring program improvements in efficiency (timeliness and predictability of transfers), effectiveness (quality of public works and food security interventions efficacy) and fairness (coverage, targeting, appeals and transparency). Specifically, it will: Reinforce the Information Center collection of weekly data from a sample of districts and the Rapid Response Mechanism to address critical implementation problems as they occur Scale up the annual Public Works Review, incorporating GPS/GIS technologies Conduct an impact evaluation survey that will follow the baseline survey Implement another round of process evaluation and beneficiary assessments, as well as quantitative surveys to identify errors of inclusion and exclusion. In addition, it will support analytical and piloting efforts to develop a better understanding of how to define, measure and promote graduation in preparation for phase III. Moreover, a common and integrated M&E system for the whole Food Security Program of which PSNP is a part is being developed. e. Conclusions and recommendations Results readiness in SSN varies considerably across lending instruments. While investment operations show overall good results focus, DPLs are lagging behind and need a major overhaul. Progress on results readiness in investment operations is uneven throughout project cycle. Although there are specific issues that need to be improved, in general considerable efforts are made at project design to ensure adequate PDO and KPI, as well as sound M&E arrangements. However, results readiness during project implementation is still a challenge. Specifically, data collection plans implementation; use of KPI to measure progress and inform decisions; and overall 22 result focus by Task Teams and Bank management in ISR preparation and review need to be strengthened. Key issues and recommendations to consolidate results readiness in SSN projects are the following. Specific recommendations for development policy and emergency recovery loans are in the main report. PDO SSN PDOs are relatively wellspecified, particularly among investment operations. They have a strong focus on national or sectoral government institutional capacity building. Reaching poor areas and households, as well as other vulnerable groups such as ethnic minorities, people with disabilities and at risk groups is also a common objective among SSN. About a third of SSN also seek to improve human development outcomes, or access to and quality of services and public infrastructure. SSN rarely take on local and community level objectives dealing with capacity building or empowerment issues. To ensure quality of PDOs, SSN should make explicit the desired development outcome while not losing sight of the institutional strengthening means to achieve it. In addition, SSN PDO should make sure to specify as narrowly as possible the target beneficiaries as well as project scale (pilot vs. rural/urban vs. national). PDO should be clear on whether the goals apply to poor households or targeted geographic areas and when vulnerable groups are addressed, these should be specified. KPI Four main issues arise in the review of KPI for SSN projects. First, deficient measurability is the most common difficulty in getting SMART KPI. More clarity on how indicators will be measured is needed. Use of a technical card for each indicator including an exact definition of numerator and denominators as background material in PAD preparation may be useful to ensure measurability. Second, technical guidance on how to develop SMART institutional development KPI is highly needed to ensure that project goals on capacity building are adequately assessed. Third, KPIs should be identified along the stages of the results chain. In particular, there is a need for clarity and measurability around a "missing middle" in service delivery. Finally, frequency of data availability for KPI regular update should be considered when selecting the mix of indicators along the results chain. Performance information is needed throughout the project cycle. Therefore, it is important to be clear on which KPI could be updated regularly and be used for monitoring project implementation, and which ones could be measured less frequently, but are useful to assess overall performance at project completion. M&E design and implementation Overall, design and implementation of M&E systems is satisfactory among investment operations. Core M&E components vary by project type, a large proportion of SSN projects include an MIS or 23 beneficiary registry. Some form of operational assessment through process evaluation, spot checks or technical/operational audits is often included. One out of two SSN projects plan to conduct an impact evaluation, but only a third provided detailed methodology information in the PAD to consider them potentially rigorous. Several improvements over current practice are possible by: Providing greater specificity and a deeper level of detail in the preparation of M&E frameworks, bearing in mind that early planning leads to better results. Assessing client capacity for M&E and anticipating technical assistance needs Having a more realistic timeframe for evaluations, taking into consideration the need to develop terms of reference, carry out the contracting process, as well as unforeseeable situations during data collection. Using existing systems for early monitoring start and build up from them. Ensuring a wide application of MIS and operational assessments tools and a selective and strategic use of impact evaluation, acknowledging individual project performance information needs. Use of results by the Bank Results readiness will not be complete until incentives are in place to report, discuss and make decisions informed by M&E results. General performance on results monitoring and its use by the Bank in SSN projects is moderately satisfactory. Lack of reporting of KPI and failure to fulfill data collection plans were critical aspects behind this rating. Attention to M&E issues by Task Teams and Bank management also needs to be strengthened. Performance could be improved in several ways, including (i) more consistent reporting by task team as to whether key components of the M&E framework are on track; (ii) identification of M&E issues under Priority Actions with time bound actions required, and (iii) more consistent attention by Bank management to M&E beyond disbursements. 24 Annex 1: Cohort of SSN Projects approved FY0509 Lending Project Type Country Project FY instrument Safety Net Croatia Social Welfare Devt 2005 SIL System Reform Ukraine Social Asstance System Mod 2006 SIL Jordan Social Protection Enhancement 2008 SIL Dominican Republic Social Sectors Investment Program 2008 SIL Bangladesh Disability and Children at Risk 2009 SIL Pakistan Social Safety Net Technical Assistance 2009 TAL Georgia PRSO 3 2007 DPL West Bank and Gaza PRDP Support 2008 DPL Lebanon Emergency SP Implementation Support 2008 ERL Lebanon ESPISP 2 2009 ERL Cash Transfer Ecuador Bono de Desarrollo Humano 2006 APL Colombia Social Safety Net Project 2006 SIL El Salvador Social Protection and Local Dev (FISDL) 2006 SIL Jamaica Social Protection 2008 SIL Panama Social Protection 2008 SIL Bolivia Investing in Children And Youth 2008 SIL Colombia Second Social Safety Net 2009 SIL Mexico Support to Oportunidades 2009 SIL Kenya Cash Transfer for OVC 2009 SIL Maldives Post Tsunami Emergency 2005 ERL Nepal Peace Support 2008 ERL Health, Honduras Nutrition and Social Protection 2006 SIL Nutrition and Bulgaria Social Inclusion 2009 SIL Education Nigeria HIV/AIDS Prog. Dev. II 2009 SIL Bolivia Social Sector Programmatic Credit II 2005 DPL Ethiopia PRSC 2 DPL 2005 DPL Pakistan PRSC II 2007 DPL Public Works Ethiopia Productive Safety Nets I 2005 APL Argentina Heads of Household Transition Project 2006 SIL Ethiopia Productive Safety Nets II 2007 APL Rwanda 1st Comm Living Standards 2009 DPL Food Crisis Sierra Leone Food Crisis Response 2009 DPL Philippines GFRP DPO 2009 DPL Nicaragua Emergency Food Price Response 2009 ERL West Bank and G Food Price Crisis Response 2009 ERL Other Colombia (CRL2) TAL to Support the 2nd PSAL 2005 SIL Romania Social Inclusion 2006 SIL Pakistan Earthquake Disability 2007 SIL Brazil Housing Sector Reform 2005 DPL Cote d'Ivoire PostConflict Assistance 2008 ERL 25 Appendix 2: Sample KPIs by Project Objective Area Focus area Sample KPIs Data Collection SP thematic area Lending Instrument Instruments Access to public services and infrastructure (local public goods) Net enrollment grade (by grade Household surveys Social Funds SIL and/or gender) More final outcome % of people with access to safe Household surveys Social Funds SIL oriented drinking water Increase in population with access to Household surveys or Social Funds SIL improved health facilities within 5km health information system if functioning Increase with population with access Household surveys or Social Funds SIL to improved economic infrastructure Project MIS (roads, irrigation, markets) % of subprojects operational and Facility surveys Social Funds ERL maintained one year after completion % households reporting utilization of Household surveys Service Delivery SIL health centers Proportion of births attended by Household surveys Service Delivery SIL skilled health personnel Availability of essential drugs at Facility survey or health Service Delivery SIL health facilities (%) (Percentage of information system if More project output months with tracer drugs functioning oriented availability) % of births in rural targeted Household survey including Social Safety Nets DPL communities in accredited facilities demographic information 26 % of children in poor households Household survey (linked Social Safety Nets SIL (quintiles 1 and 2) in targeted with national income or communities participating in Early consumption household Childhood Education Services data) % of public works assessed to be Technical audits Social Safety Nets ERL satisfactory (using PW Review performance criteria eligibility, appropriateness, effectiveness, quality and sustainability) Number of communities benefiting Social Fund MIS Social Funds ERL from at least one subproject % of targeted Roma settlements Administrative data Social Safety Nets SIL water points constructed or rehabilitated # of classrooms/health centers Social Fund MIS Social Funds ERL constructed % of classrooms replaced in 40 Administrative data Social Safety Nets ERL schools damaged by the Tsunami % of kilometers of rural roads Administrative data Social Safety Nets ERL rehabilitated # of bednets, contraceptives etc. Administrative data from Service Delivery SIL distributed sector # of service workers deployed Administrative data from Service Delivery SIL (teachers, health workers, etc.) sector Access to economic opportunities Number of participants exiting the Household surveys Labor markets SIL program into formal employment (Annual). More final outcome An increase in the employment rate National surveys (eurostat) Labor markets DPL oriented of younger workers to at least 23 percent (age 1524) 27 % increase of the rate of Household survey or Social Safety Nets SIL employment of youth benefiting national labor force survey from multifunctional centers services with beneficiary oversampling An increase in the activity rate of National surveys (eurostat) Labor markets DPL older workers to at least 45 percent (age 5564) Percentage of graduates employed Household surveys, tracer Labor markets SIL or selfemployed six months after studies of exparticipants program completion Employment rate for labor market Tracer studies of ex Service Delivery program graduates participants Wages of graduates of long term Household surveys, tracer Labor markets SIL training program of the project studies institutions by student origin. # of households with annual increase Household surveys Social Funds APL in average incomes % of microfinance savers/borrowers Tracer studies, beneficiary Social Funds SIL confirming an improvement in their surveys household living standards Cumulative # of farming HH Project or sectoral Service Delivery SIL receiving and using extension information system More project output packages (millions) (distribution of packages), oriented household or farm surveys (using packages) # of income generating local Social fund MIS Social Funds SIL community groups formed 28 % of supported MFIs reaching Survey of MFIs Social Funds SIL operational selfsufficiency Reduction in job search waiting time Household survey or Service Delivery SIL program MIS if reported Number of beneficiaries of program Project MIS Labor markets SIL X. Number of Employment Offices Administrative information Labor markets SIL providing Program X Services systems Number of firms registered in Project MIS Labor markets SIL program X # of clients of supported MFIs Social Fund MIS Social Funds SIL # of beneficiaries trained in income Social Fund MIS Social Funds SIL generating activities Cumulative # of people trained Project or sectoral Service Delivery SIL (farmers, microentrepreneurs etc.) information systems Temporary income support (unconditional transfers, public works CCTs, etc.) Increased income from wages Household surveys Social Funds SIL (income transfers) from PWP works % of beneficiaries of grants for Household survey Social Safety Nets productive projects engaged in activities generating income at least equal to AR$150/mo after 18 months of operation % of beneficiary households that Beneficiary survey Social Safety Nets SIL More final outcome receive PSNP resources reporting no oriented distress sales of assets to meet food needs 29 Net improvement in household total, Household surveys Social Safety Nets SIL food (including fruits and vegetables) including consumption and protein consumption in large module for treatment and municipalities. comparison groups Average number of months that Beneficiary survey Social Safety Nets SIL PSNP households report being food insecure % of labor intensity of public works Social fund MIS Social Funds subprojects Proportion of bimonthly payments Administrative data Social Safety Nets SIL based on a complete cycle of co responsibilities verification and application of sanctions Persondays provided in labor Social fund MIS Social Funds SIL intensive public works program (number) Percentage of farmers who have Administrative data Social Safety Nets ERL More project output been affected by the Tsunami oriented receiving compensation grants. # of communities benefitting from Social fund MIS Social Funds public works subprojects % of transfer payments delivered to Administrative data Social Safety Nets SIL beneficiaries by the 13th of the payment month Human development outcomes Increased primary school enrollment Household surveys Social Funds SIL and completion rates in SF supported communities 30 Net change in chronic malnutrition Household survey with Social Safety Nets SIL amongst children under 5 in large anthropometric module municipalities. applied to treatment and More final outcome comparison groups; oriented possibly, Health MIS if available and feasible to distinguish treatment and comparison groups Net change in secondary school Household survey applied Social Safety Nets SIL completion rate of PATH students to treatment and boys and girls comparison groups; possibly, Education MIS if available and feasible to distinguish treatment and comparison groups Increase DPT, Triple Viral, and polio Sectoral information Labor markets SAL vaccination of the population age 0 systems, Health surveillance 5 resulting in 3.6 million new system children vaccinated. Increase in primary completion rate Household surveys or Service Delivery PRSC sectoral information systems Increase basic and secondary school Household surveys , Labor markets SAL enrollment rates by creating 1.4 Sectoral information million new spaces, in addition to systems More project output the existing 7.8 million spaces. oriented Annual dropout rate (%) of students Facility or household Social Funds SIL in primary and secondary schools surveys Reduction in child underweight rate Household survey (with Service Delivery PRSC anthropometric module) or health information system % of underfives malnutrition using Household surveys Social Funds SIL weight for age method 31 Incidence of diarrhea in communities Household surveys Social Funds SIL implementing water and sanitation subprojects % of children vaccinated Household surveys and Social Funds SIL health information system % of women receiving prenatal care Household and facility Social Funds SIL in project communities surveys Average daily medical consultations Facility surveys of health Social Funds SIL in health centers information system HIV/AIDS prevalence rate Health surveillance system Service Delivery PRSC Infant/child mortality rate Household surveys Service Delivery SIL DTP 3 vaccination rate of children Sectoral information Service Delivery PRSC below 1 year systems Contraceptive Prevalence Women Household survey Service Delivery SIL ages 1549 % of children under 3 with anemia in Household survey including Social Safety Nets 165 high risk municipalities blood samples % of married women aged 1549 DHS type survey Social Safety Nets DPL using modern contraceptives Percentage of children 05 years old Household survey; Health Social Safety Nets SIL with complete immunizations for MIS if available their age group Education test scores Sectoral information system Service Delivery DPL Share of PATH secondary school Household survey, Program Social Safety Nets SIL boys attending school 85% of the MIS time 32 Proportion of children under 2 years Household survey; Health Social Safety Nets SIL old who participate in the growth MIS if available monitoring according to agreed protocols in the areas of intervention Percentage of families meeting their Program MIS Social Safety Nets SIL education coresponsibilities Reaching specific vulnerable groups/targeted groups Number of workers that receive Specific assessment and Labor markets SAL severance pay (job loss surveys compensation) Increase the number of poor Project MIS/Beneficiary Labor markets SIL children age 05 affiliated with the registry nutrition program Percentage of program beneficiaries Project MIS Labor markets SIL women Number of workers receiving labor Project MIS Labor markets SAL redeployment services. Percentage of beneficiaries that Administrative information Labor markets SIL come from priority areas as system identified in the poverty map. % of project resources (or Household surveys (linked Social Funds SIL More final outcome beneficiaries) from the lowest 20% with national household oriented of poverty deciles surveys for national poverty rankings) Increased number of poor people Household surveys Social Funds ERL with access to social services. 33 Number of kindergarten children Facility survey Social Funds SIL with disability benefiting from the kindergarten centers Community Management Social Fund MIS Social Funds SIL Committees have at least 50% of elected women % of project resources disbursed to Social Fund MIS Social Funds SIL the poorest districts/municipalities Number of communities mapped Social Fund MIS Social Funds SIL and profiled (with participatory poverty assessments) % of citizens with access to services Household survey Service Delivery by poverty quintile Increased access to services (e.g. Sectoral data crossed with Service Delivery DPL institutional births, school poverty mapping enrollment etc.) in poorest information geographical areas % of vulnerable groups (e.g orphans, Household survey Service Delivery elderly) using health and education services % of program expenditures (or % of Household survey Service Delivery TA More project output beneficiaries) on poorest household oriented quintile At least X percent of grants provided Project MIS Service Delivery SIL for inclusion activities to marginalized groups in project municipalities are disbursed % of programs utilizing proxy means Administrative data Service Delivery TA test and other targeting criteria 34 Percentage of disbursed cash Household surveys (linked Social Safety Nets SIL transfers received by households in with national income or quintiles 1 and 2 consumption household data) including social programs module Share of program participants in Household surveys (linked Social Safety Nets SIL lower half of income distribution with national income or consumption household data) including social programs module % of children in Q1 registered for Household surveys (linked Social Safety Nets SIL PATH with national income or consumption household data) including social programs module % reduction in the gap between Depending on exact Social Safety Nets SIL targeted poor Roma settlements and definition of living neighboring communities as conditions index may need measured by the living conditions census data, community index surveys or household survey representative at the community level % of revised SISBEN 1 families in Administrative data Social Safety Nets SIL newly entered municipalities registered in the program % of ECE interventions in Census data to identify Social Safety Nets SIL communities with more than 50% target communities and Roma population administrative data % of districts with at least 85% of Administrative data Social Safety Nets SIL households with completed needs assessment % recertification of PATH families Administrative data Social Safety Nets SIL with children due to be recertified after 4 years enrollment 35 Community empowerment and capacity building (nongovernment) % of participating communities have Social Fund MIS, Social Funds SIL improved their capacity to Community surveys More final outcome implement the CDD approach, and oriented comply with at least 10 CDD capacity indicators # of local executing agencies with Community surveys Social Funds ERL improved capacity for planning and implementing community development projects % of communities applying acquired Community surveys Social Funds DPL skills in activities beyond social fund project (use of community management committee, social audits etc.) % of facilities adequately maintained Facility surveys Social Funds by the community More project output oriented At least 90% of subprojects Household surveys, Social Funds ERL undertaken reflect the priorities of participatory M & E targeted communities and instruments beneficiaries % of beneficiaries reporting Household surveys, Social Funds SIL participating in identification and participatory M & E execution of subprojects instruments # of capacitybuilding events carried Social fund MIS Social Funds ERL out for beneficiary CBOs each year 36 % of communities benefiting from Social fund MIS Social Funds ERL social mobilization Increase in citizen satisfaction with Participatory M & E Service Delivery SIL social services as measured by instruments Citizen Report Cards and Community Score Cards % of citizens who report knowledge Household survey or Service Delivery SIL about local budgets participatory M & E instruments Improved women's understanding of Household survey or Service Delivery DPL their right to an institutional birth participatory M & E and the standards for a goodquality instruments birth Increased connectivity to and usage Administrative information, Service Delivery SIL of national information management service provider surveys networks, by local service providers # of local management committees Project (or sectoral) Service Delivery TA established for social services information system Share of coverage in community Sectoral information Service Delivery services by NGOs and community systems groups % of Roma population from targeted Household surveys, Social Safety Nets SIL poor settlements agreeing that participatory M&E subprojects reflect community instruments priorities 37 Proportion of OVC households aware Beneficiary surveys, Social Safety Nets SIL of program information, such as participatory M&E objectives, eligibility criteria and use instruments of funds % of kebeles that have developed Administrative data, Social Safety Nets APL and approved safety net plans, performance audits taking intoaccount community preferences % of public sector organizations that Administrative data Social Safety Nets SIL hold partner forums on a regular basis to report on planning decision and progress. Local government institutional capacity building Household perceptions of local Household surveys, Social Funds SIL government transparency, capacity participatory M & E and responsiveness instruments % of local governments able to set Local government surveys Social Funds APL objectives and achieve at least X% of their annual targets % of local government annual Local government surveys Social Funds SIL investments consistent with Community Development Plans More final outcome oriented Beneficiary satisfaction with training Community and local Social Funds SIL received at the local government government surveys, level participatory M & E instruments 38 Legal and administrative texts have Administrative information Social Funds SIL been adopted allowing local governments to pass through funds to community groups % of social fund projects executed by Social fund MIS Social Funds SIL local governments # of local government officials Social fund MIS Social Funds SIL trained Citizens reports of quality of basic Participatory M & E Service Delivery services delivered by local instruments or household government surveys Subnational governments Sectoral budget information Service Delivery SIL expenditures on basic services Allocation of financing from State National and sectoral Service Delivery SIL government to local self budget information More project output governments for health and oriented education services Share of local government budgets Local government budget Service Delivery DPL spent of social services information (systematized or via sample surveys where systems lacking) Number of local service providers Sectoral information Service Delivery SIL (PHCs, schools, CSWs and NGOs) systems accredited by the relevant institution in their sector 39 # of local government units trained Administrative information Service Delivery TA in sectoral service programs systems National/sectoral government institutional capacity building Number of job seekers, as reported Administrative information Labor markets SIL More final outcome by the LFS, served by the National system oriented Employment Service Formulation of a viable Pension Administrative data Labor markets SIL System Development Strategy and progress in its implementation Labor market information (LMI) Sectoral information Labor markets SIL system in place and two reports on systems labor market situation and trends published Number of new small business Project MIS Labor markets SIL incubators established. Number of new module training Project MIS Labor markets SIL programs developed and in use and number of new module training centers fully functional Operations guidelines and manuals Administrative information Labor markets SIL for a new menu of ALMPs prepared system and in use More project output Central and regional Career Administrative information Labor markets SIL oriented Counseling Centers operational system Citizen knowledgeable about basic Participatory M & E Service Delivery services (availability, eligibility, instruments transparency) 40 % of social program budget Sectoral budget information Service Delivery TA distributed based on new program system based model of allocation Number of external evaluations of Administrative data Service Delivery TA social program performance conducted Monitoring and evaluation system Administrative data Service Delivery TA for prioritized national strategic social programs operating as evidenced by timely monthly MIS reports Number of Results Agreements Administrative data Service Delivery TA signed between national planning office and sectors Decrees published relating to social Administrative data Service Delivery program operations Percentage of civil registry offices Administrative data, Social Safety Nets ERL which have their records updated, technical/operational audits modernized and operational, as evidenced by the establishment of a new computerized system % of households in the targeting Administrative data, Social Safety Nets DPL database whose poverty status is technical/operational audits verified through home visits at least once every three years % of grievance redressal claims Administrative data Social Safety Nets TA settled within three months of application 41 % of BISP positions that are filled Administrative data Social Safety Nets TA with qualified staff according to official examination system % of social welfare facilities have Administrative data, Social Safety Nets SIL acceptable hygienic standards in operational audits, facilities place to meet national criteria survey, spot checks % of MoSD staff with completed Administrative data Social Safety Nets SIL training envisaged in the Human Resources Development Strategy % of municipalities with indicators of Administrative data Social Safety Nets SIL payment process in yellow and red alert % of departmental offices for the Administrative data, Social Safety Nets SIL First Employment Program (DOFEP) operational audits, spot operating and connected to the MIS checks (as described in the operational manual) Quality of services/service delivery Increase in average percentage Impact evaluation surveys Labor markets SIL earnings of PJE graduates compared to the control group. Improved user satisfaction with Participatory Labor markets SIL program X and its affiliated M&E/Beneficiary agencies# services Assessments Amounts claimed to agency X for Project MIS/ Participatory Labor markets SIL reimbursement due to negative M&E balances in the payment of allowances through compensatory system. 42 Number of beneficiaries receiving Project MIS/Administrative Labor markets SIL their transfers directly in their bank data accounts. Number of peopletimes of use of Project MIS Labor markets SIL services (job counseling, referral, and guidance) provided by project invested employment institutions, monthly average, by origin of the user Number of migrantstimes receiving Specific surveys Labor markets SIL legal assistance in projectinvested pilot localities annually More final outcome Increase in the number of new Project MIS Labor markets SIL oriented courses offered by program X Teachers trained and certified in Sectoral information system Labor markets SIL adult education methodology. Material improved and provided to Participatory M & E Labor markets SIL each student instruments Life skills modules incorporated in all Project MIS Labor markets SIL programs Community satisfaction with service Participatory M & E Social Funds SIL quality instruments, household surveys Student/teacher (or Facility surveys, education Social Funds SIL student/classroom) ratio system information % of health centers staffed to Facility surveys or health Social Funds sectoral norms information system if functioning 43 Availability of basic medicines in Facility surveys Social Funds supported health centers # of desks, books and other inputs Facility surveys Social Funds per student Quality rating of local infrastructure Facility surveys, technical Social Funds (schools, health centers, water audit systems, roads etc.) More project output # of facilities with local community Facility surveys, social fund Social Funds oriented management committees MIS established Average pupilteacher ratio (e.g. Service Delivery SIL grades 14) Share of qualified health staff in rural Sectoral information system Service Delivery DPL areas Public perception of service quality Participatory M & E Service Delivery SIL instruments, household surveys % of health posts without shortages Sectoral information Service Delivery SIL of injectable contraceptives in last 3 systems months Increase in the use of Standard Sectoral information Service Delivery SIL Treatment Protocols in project systems supported hospitals Number of textbooks per primary Sectoral information Service Delivery PRSC school student systems % of teachers receiving inservice Sectoral information Service Delivery SIL training systems Fiscal objectives/efficiency 44 Average unit costs of infrastructure Social fund MIS Social Funds (compared to other programs) More final outcome % of operating costs as a share of Social fund MIS Social Funds SIL oriented total costs Local contributions as a share of Social fund MIS Social Funds total social fund investments % of subproject executed within Social fund MIS Social Funds SIL timeframe and budget Total Federal block grants to basic National budget Service Delivery SIL services as a share of total Federal information system discretionary expenditures % of treatment municipalities Administrative information Service Delivery SIL making financial allocations based on system agreed formula in health sector % of non wage executed budget Sectoral budget information Service Delivery DPL More project output system oriented Annual budget of national strategic National budget Service Delivery TA social programs information system % of priority social program Program or sectoral Service Delivery expenditures on administrative information systems overhead 45 Social Protection Discussion Paper Series Titles No. Title 1106 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Social Service Delivery Projects by Julie Van Domelen, February 2011 1105 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Social Safety Nets Task Teams by Gloria Rubio, February 2011 1104 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Social Funds Task Teams by Julie Van Domelen, February 2011 1103 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Labor Markets Task Teams by Maddalena Honorati, February 2011 1102 Natural Disasters: What is the Role for Social Safety Nets? by Larissa Pelham, Edward Clay and Tim Braunholz, February 2011 1101 North-South Knowledge Sharing on Incentive-based Conditional Cash Transfer Programs by Lawrence Aber and Laura B. Rawlings, January 2011 1008 Social Policy, Perceptions and the Press: An Analysis of the Media's Treatment of Conditional Cash Transfers in Brazil by Kathy Lindert and Vanina Vincensini, December 2010 (online only) 1007 Bringing Financial Literacy and Education to Low and Middle Income Countries: The Need to Review, Adjust, and Extend Current Wisdom by Robert Holzmann, July 2010 (online only) 1006 Key Characteristics of Employment Regulation in the Middle East and North Africa by Diego F. Angel-Urdinola and Arvo Kuddo with support from Kimie Tanabe and May Wazzan, July 2010 (online only) 1005 Non-Public Provision of Active Labor Market Programs in Arab-Mediterranean Countries: An Inventory of Youth Programs by Diego F. Angel-Urdinola, Amina Semlali and Stefanie Brodmann, July 2010 (online only) 1004 The Investment in Job Training: Why Are SMEs Lagging So Much Behind? by Rita K. Almeida and Reyes Aterido, May 2010 (online only) 1003 Disability and International Cooperation and Development: A Review of Policies and Practices by Janet Lord, Aleksandra Posarac, Marco Nicoli, Karen Peffley, Charlotte McClain-Nhlapo and Mary Keogh, May 2010 1002 Toolkit on Tackling Error, Fraud and Corruption in Social Protection Programs by Christian van Stolk and Emil D. Tesliuc, March 2010 (online only) 1001 Labor Market Policy Research for Developing Countries: Recent Examples from the Literature - What do We Know and What should We Know? by Maria Laura Sanchez Puerta, January 2010 (online only) 0931 The Korean Case Study: Past Experience and New Trends in Training Policies by Young-Sun Ra and Kyung Woo Shim, December 2009 (online only) 0930 Migration Pressures and Immigration Policies: New Evidence on the Selection of Migrants by Johanna Avato, December 2009 (online only) 0929 Ex-Ante Methods to Assess the Impact of Social Insurance Policies on Labor Supply with an Application to Brazil by David A. Robalino, Eduardo Zylberstajn, Helio Zylberstajn and Luis Eduardo Afonso, December 2009 (online only) 0928 Rethinking Survivor Benefits by Estelle James, December 2009 (online only) 0927 How Much Do Latin American Pension Programs Promise to Pay Back? by Alvaro Forteza and Guzmán Ourens, December 2009 (online only) 0926 Work Histories and Pension Entitlements in Argentina, Chile and Uruguay by Alvaro Forteza, Ignacio Apella, Eduardo Fajnzylber, Carlos Grushka, Ianina Rossi and Graciela Sanroman, December 2009 (online only) 0925 Indexing Pensions by John Piggott and Renuka Sane, December 2009 (online only) 0924 Towards Comprehensive Training by Jean Fares and Olga Susana Puerto, November 2009 0923 Pre-Employment Skills Development Strategies in the OECD by Yoo Jeung Joy Nam, November 2009 0922 A Review of National Training Funds by Richard Johanson, November 2009 0921 Pre-Employment Vocational Education and Training in Korea by ChangKyun Chae and Jaeho Chung, November 2009 0920 Labor Laws in Eastern European and Central Asian Countries: Minimum Norms and Practices by Arvo Kuddo, November 2009 (online only) 0919 Openness and Technological Innovation in East Asia: Have They Increased the Demand for Skills? by Rita K. Almeida, October 2009 (online only) 0918 Employment Services and Active Labor Market Programs in Eastern European and Central Asian Countries by Arvo Kuddo, October 2009 (online only) 0917 Productivity Increases in SMEs: With Special Emphasis on In-Service Training of Workers in Korea by Kye Woo Lee, October 2009 (online only) 0916 Firing Cost and Firm Size: A Study of Sri Lanka's Severance Pay System by Babatunde Abidoye, Peter F. Orazem and Milan Vodopivec, September 2009 (online only) 0915 Personal Opinions about the Social Security System and Informal Employment: Evidence from Bulgaria by Valeria Perotti and Maria Laura Sánchez Puerta, September 2009 0914 Building a Targeting System for Bangladesh based on Proxy Means Testing by Iffath A. Sharif, August 2009 (online only) 0913 Savings for Unemployment in Good or Bad Times: Options for Developing Countries by David Robalino, Milan Vodopivec and András Bodor, August 2009 (online only) 0912 Social Protection for Migrants from the Pacific Islands in Australia and New Zealand by Geoff Woolford, May 2009 (online only) 0911 Human Trafficking, Modern Day Slavery, and Economic Exploitation by Johannes Koettl, May 2009 0910 Unemployment Insurance Savings Accounts in Latin America: Overview and Assessment by Ana M. Ferrer and W. Craig Riddell, June 2009 (online only) 0909 Definitions, Good Practices, and Global Estimates on the Status of Social Protection for International Migrants by Johanna Avato, Johannes Koettl, and Rachel Sabates-Wheeler, May 2009 (online only) 0908 Regional Overview of Social Protection for Non-Citizens in the Southern African Development Community (SADC) by Marius Olivier, May 2009 (online only) 0907 Introducing Unemployment Insurance to Developing Countries by Milan Vodopivec, May 2009 (online only) 0906 Social Protection for Refugees and Asylum Seekers in the Southern Africa Development Community (SADC) by Mpho Makhema, April 2009 (online only) 0905 How to Make Public Works Work: A Review of the Experiences by Carlo del Ninno, Kalanidhi Subbarao and Annamaria Milazzo, May 2009 (online only) 0904 Slavery and Human Trafficking: International Law and the Role of the World Bank by María Fernanda Perez Solla, April 2009 (online only) 0903 Pension Systems for the Informal Sector in Asia edited by Landis MacKellar, March 2009 (online only) 0902 Structural Educational Reform: Evidence from a Teacher's Displacement Program in Armenia by Arvo Kuddo, January 2009 (online only) 0901 Non-performance of the Severance Pay Program in Slovenia by Milan Vodopivec, Lilijana Madzar, Primoz Dolenc, January 2009 (online only) To view Social Protection Discussion papers published prior to 2009, please visit www.worldbank.org/sp. Abstract The Results Readiness Review assessed progress to date on results- based management in the Social Protection & Labor (SP&L) portfolio and generated operationally relevant knowledge on how to strengthen Monitoring & Ev aluation (M&E). Specifi cally, the Review took stoc k of the status and quality of M&E in the SP&L portfolio, including both investment and policy-based lending. The Review identified trends, strengths and weaknesses, and good pr actice M&E approac hes and indicators to incorpor ate a better results focus in project design and implementation. This related Note provides guidance for World Bank Task Teams working in the area of Social Safety Nets. HUMAN DEVELOPMENT NETWORK About this series... Social Protection Discussion Papers are published to communicate the results of The World Bank's work to the development community with the least possible delay. The typescript manuscript of this paper therefore has not been prepared in accordance with the procedures appropriate to formally edited texts. The findings, interpretations, and conclusions expressed herein are those of the author(s), and do not necessarily reflect the views of the International Bank for Reconstruction and Development / The World Bank and its affiliated organizations, or those of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. For free copies of this paper, please contact the Social Protection Advisory Service, The World Bank, 1818 H Street, N.W., Room G7-703, Washington, D.C. 20433 USA. Telephone: (202) 458-5267, Fax: (202) 614-0471, E-mail: socialprotection@worldbank.org or visit the Social Protection website at www.worldbank.org/sp.