60080 S P D I S C U S S I O N PA P E R NO. 1103 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Labor Markets Task Teams Maddalena Honorati February 2011 Results Readiness in Social Protection and Labor Operations Technical Guidance Notes for Labor Markets Task Teams1 1. Introduction Labor allocation to its most efficient use, promoting employment and human capital investment as well as functioning labor markets can contribute to longterm economic growth, poverty reduction and to help workers manage their risks. A labor market policy framework includes both regulations and programs. However, the optimal framework is not standard and universal but varies country by country depending on the level of economic and financial development, culture and other structural characteristics. The nine projects comprising this cohort2 are a rather disparate lot. They range from facilitating migration of rural labor into urban areas (China) to safety nets for the unemployed (Argentina) to assisting youth at risk (Dominican Republic) to facilitating labor market convergence with EU (Bulgaria), to strengthening protections for the labor force (Azerbaijan). Due to the heterogeneity of this group of projects, the following analysis of quality of M&E tools has been conducted also by type of labor market project. Specifically, the nine projects have been classified in five groups according to their prevailing component: training programs, social assistance (unemployment benefits), employment subsidies, labor market regulation and administration and a miscellaneous group including two development policy lending (DPL) operations with equally important pillars on improving labor market policies, health, education and social protection3. Looking at the portfolio composition by type of lending instrument, five out of nine projects are investment lending operations and four are programmatic DPLs. Among Social Protection (SP) investment lending operations, labor markets investment lending operations account for the highest average disbursement per project ($203.4 million) whereas labor market total DPLs disbursement ($750 million) represent 30 percent of total development policy lending in SP. Labor market projects are equally concentrated in Latin America and the Caribbean and Eastern Europe and Central Asia regions and one is China. Interestingly, the number of projects having "improving labor market" as the primary component has increased over time (three projects have been approved in FY09 versus one in FY06 and FY07). Given the small cohort size and oneofakind nature of many of the projects in the cohort, caution is needed in generalizing any findings or doing comparative analysis for establishing best practice 1 This Note has been prepared by Maddalena Honorati. 2 They are projects in which the "Improving labor markets" component is more relevant than other components. Excluded from this cohort are projects having a labor market component but a higher safety net or pension component. 3 A complete list of projects revises and their classification is shown in annex 1 ­ table7. examples. 2. Project Development Objectives (PDOs) i. Description of General Practice As expected almost all project development objectives in the cohort of projects reviewed focus on promoting higher employment and increasing economic opportunities as the main objective especially via training programs (30 percent of projects). About half of the projects also seek to reach specific vulnerable groups by improving targeting mechanisms and to improve the quality of social assistance services by reducing the cost of job search through access to enhanced employment services and by improving employability. DPLs' common objectives are to improve the quality of service delivery and to build institutional capacity at the national or sectoral level. This latter would be achieved through improved regulatory environment (strengthened worker protections) and better institutional practices. Table 1: Project Development Objectives Access to Temporary Human Reaching National/ Quality of Fiscal Number economic income development specific sectoral services/ objectives/ of opportune support outcomes vulnerable government service efficiency projects ties groups/ institutional delivery targeted capacity groups building Labor markets (total) 8 2 3 5 5 5 3 9 Training or retraining for workers, unemployed 3 0 0 2 1 2 1 3 persons, and labor market entrants Labor administration and institutional capacity 1 0 0 0 1 1 0 1 building Employment subsidies 1 0 1 0 0 0 1 1 Social assistance and 1 2 0 1 1 0 1 2 income support Other specify in comments 2 0 2 2 2 2 0 2 Consistently with their PDOs, the main activities supported by labor market projects involved training and retraining of workers, unemployed persons, and youth (three projects), social assistance and income support (two projects), labor administration and institutional capacity building (three project), employment subsidies (one project) and a miscellaneous group including equally important goals related to health, education, labor markets and general social protection system improvements (two projects). ii. Assessment of Performance The general expression of PDOs in labor market projects is satisfactory. Ratings were based on whether PDO were concise and outcome driven, whether they reflect what the project can directly contribute, and whether target groups were identified. With all the nine projects rated above the line (six projects rated "satisfactory" and three projects rated "moderately satisfactory") this is, relatively speaking, one of the stronger aspects of this cohort. However, none of the operations was rated as "1", so genuine best practice examples are difficult to find. Differently from other results measure, PDO statements do not show to have improved over time relative to the extent they are outcome driven and to project accountability. Table 2: PDO Ratings Rating Frequency Highly Satisfactory 0 Satisfactory 6 Moderately Satisfactory 3 Moderately Unsatisfactory 0 Unsatisfactory 0 Highly Unsatisfactory 0 Average Score 2.33 iii. Key issues Ratings were low when there was either a mixing up of means and ends or a too broad and poorly specified objective. The main areas for improvement in the PDOs include: PDOs are often stated too broadly promising higher level outcomes (e.g. "strengthen prospects for sustainable peace") beyond the reach of the project; should try to keep the PDO statement clear, precise, and stated as realistic results not overly broad or too ambitious; Mixing means and ends: PDOs are often unnecessarily prescriptive about how the project goals are to be achieved. More focus on the outcome that the project can directly influence, given its duration, resources, and approach. Focus on what is going to be transformed (not what is going to be done) ; Focus on the expected outcome for the primary beneficiaries: at the close of the program, what problem has been solved for the key recipient of the program outputs; Inconsistent treatment of the PDOs between the PAD body text and Annex 3. In the context of PDOs in DPLs operations, there is often a problem of attribution/accountability. Since by definition in DPLs operations funds are not assigned to any particular spending item but are made available to finance government's budget together with other financial partners and tax revenues, it becomes therefore difficult to attribute achieved objectives to the development lending operation. Moreover, very often DPLs have very broad and ambitious objectives as MDGs. Despite the different nature of DPLs with respect to investment operation, there is still some scope for additional clarity and specificity on the objective of the supported reform program and the targeted population. iv. Identification of good practice Table 3: Examples of Labor Market PDOs Issue Stronger example Weaker example Focus on The proposed Labor Reform and Social To support the transition of rural workers to outcomes rather Development Policy Loan (LaRSDPL 111) would: (a) urban areas to access better employment than outputs strengthen Colombia's social protection system and opportunities that improves their incomes and improve the delivery of social services, (b) raise working conditions. This objective will be human capital formation, (c) improve employability, achieved by: (i) improving the access of rural and (d) enhance monitoring and evaluation systems workers to skills development opportunities, (ii) for better transparency, social oversight, and results reducing the cost of their job search through management in the social sectors access to enhanced employment services, and (iii) removing the worst excesses of their transition through strengthened worker protections. Targeted to To support GoDR in improving the employability of Support the Government of Bulgaria (GoB) to primary poor atrisk youth by building their work experience meet some of the challenges associated with EU beneficiaries and life skills and expanding second chance accession --in the case of DPL I-- and post education programs to complete their formal accession --DPL II and III. The DPL series would education. support the adoption and implementation of policies to (i) increase employment and lay the foundations for longterm productivity growth by providing incentives for job creation and improving quality of education and; (ii) promote fiscal sustainability through efficiency gains in social sectors. Concise and To support the Government's privatization program To improve delivery of labor market and social specific through mitigating the negative social and protection interventions through strengthened economic impact of the privatization of state institutions enhanced institutional and human owned enterprises (SOEs). resources capacity, and improved targeting of social safety net programs. Accountability/att The PDO is to increase the effectiveness of ribution Argentina's income transfer programs for the unemployed and families with children, by improving selected designed features and the transparency and accountability of the "Family allowances" and "Seguro" program, and by transferring beneficiaries from other, less effective schemes and programs, to the Family allowances and seguro programs. 3. KPIs i. Description of general practice Compared to other thematic groups, the average quality (based on the extent to which KPIs are SMART, have a baseline and target established as well as sources and responsibilities for data collection well defined at the design stage) of either outcome or intermediate outcome KPIs is the worst among SP thematic groups as well as their linkage to respective PDOs. As expected the most frequent outcome indicators in labor market projects are oriented at measuring access to economic opportunities consistently with the fact that 89 percent of labor projects have this as their expressed objective. KPIs aiming at monitoring training programs account for about 40 percent of all outcome indicators. Outcome indicators measuring fiscal and efficiency objectives as well as improving the quality of service delivery and reaching specific vulnerable group are also dominant consistently with project development objectives. Table 4: Distribution and quality of LM indicators Frequency Clear link Average Share of Share of to PDO SMART indicators indicators rating with target reported in values ISRs established Outcome indicators Access to economic opportunities (microcredit, active labor market programs, skills development, etc.) 15 2.87 2.87 0.93 0.73 Temporary income support (unconditional transfers, public works and temporary employment, CCTs, wage subsidies, etc.) 1 3.00 3.00 1.00 1.00 Human development outcomes (improved education, health and nutrition, HIV status etc.) 7 2.29 2.86 0.71 0.86 Reaching specific vulnerable groups/targeted groups 9 2.22 3.33 0.78 1.00 Local government institutional capacity building (anything subnational) 3 3.00 3.00 0.67 0.00 National/sectoral government institutional capacity building (benefits administration, management and operation of programs, targeting system, etc) 8 2.25 3.63 0.63 0.63 Quality of services/service delivery 11 2.27 3.00 0.55 0.73 Fiscal objectives/efficiency 12 2.75 2.92 0.50 0.25 Intermediate outcome indicators Access to economic opportunities (microcredit, active labor market programs, skills development, etc.) 8 2.13 2.50 0.88 0.75 Reaching specific vulnerable groups/targeted groups 10 2.22 3.20 0.50 0.90 Local government institutional capacity building (anything subnational) 5 2.00 3.00 1.00 1.00 National/sectoral government institutional capacity building (benefits administration, management and 22 2.50 3.32 0.91 0.82 operation of programs, targeting system, etc ) Quality of services/service delivery 19 2.40 2.74 0.37 0.53 Fiscal objectives/efficiency 4 3.00 2.50 0.75 0.25 Interestingly, a big share (80 percent) of intermediate outcome indicators are aimed at measuring institutional development and process efficiency aspects4. Most of them aim at measuring institutional capacity building, delivery of service and systems `transparency and accountability. For instance, 40 percent and 35 percent of intermediate outcome KPIs measure activities aiming at strengthening national and sectoral institutional capacity building5 and at improving quality of service delivery respectively. ii. Assessment of Performance Across outcome and intermediate outcome indicators there is a substantial difference in terms of performance6. Intermediate outcome indicators perform relatively better having baseline for 75 percent of indicators, source of data collection better specified and about 92 percent of them being reported in ISRs. By contrast, 20 percent of outcome indicators present in Annex 3 are not reported in ISR, data sources and frequencies are poorly specified and 35 percent of them lack baseline at the design stage. The average SMART rating is moderately satisfactory for both outcome and intermediate outcome KPIs with intermediate indicators being slightly better. The weakest indicators were found to be institutional indicators related to local, national and sectoral capacity building and indicators aiming at measuring targeting objectives. In aggregate, over half of the outcome and intermediate indicators poorly related to their PDOs, reflecting multiple foci of this cohort and perhaps overly complex project designs. The specification of intended data sources as well as data collection responsibilities was found to be on average moderately unsatisfactory. This was the main reason for the poor performance in tracking results during project implementation. The average number of both outcome and intermediate outcome indicators per project is seven, but projects varied greatly in the use of KPIs with some projects (e.g., Bulgaria DPL II) having as many as 15 outcome indicators while other sensibly limiting themselves to three or four indicators (e.g., China: Migrant Skills Dev., Dominican Republic: Youth Dev Project). Similarly, the number of intermediate indicators varied from zero in case of Colombia: 2nd Programmatic Labor and Social Sector Reform to 10 for Azerbaijan: Social Protection Development Project. 4 All 10 of the intermediate indicators for Azerbaijan concerned institutional issues and four4 of those were rated "2" on the SMART scale. 5 Benefits administration, management and operation of programs, targeting systems... 6 Of the 30 labor markets related indicators just one6 was rated as "1" while almost a third of the indicators were, on the other hand, rated below the line. The situation was somewhat better for the intermediate indicators with only 3 of the 29 indicators rated below the line. iii. Key Issues The main areas for improvement as regards KPIs include improving their quality (average SMART rating of only 3.07 for outcome indicators and 2.93 for intermediate indicators with four of the ten projects falling below the line in this respect), their linkage to PDOs and the specification of data sources and frequencies. Greater selectivity in choosing indicators should be another priority, recognizing that collecting accurate data in a timely manner is not a costless activity. A more thorough assessment of the client capacity to undertake the proposed M&E activities and integrating them into their regular MIS systems will also be useful in keeping the M&E designs and expectations at more realistic levels and for assuring their operational usefulness for the implementing agencies. No clear linkage to PDO. Especially among outcome indicators, the review found cases of indicators not relevant to PDOs or project subcomponents and activities not tracked by any indicator (Turkey PSSP 2). For example: the outcome is "improved employability" and the outcome indicator is "Percentage of program participants employed or selfemployed six months after program completion". It could be argued that the latter measures an outcome like "improved access to employment or selfemployment" rather than "improved employability". Employability is not a standard concept so it will be important to be clear upfront on what it is meant by it (probability of getting a job based on life skills or else; probability of keeping a job based on what characteristics..) before finding the proper indicator to measure. Measurability and implementation issues. Indicators are often not well defined in what they are meant to measure by using vague adjectives like "systems working effectively" or "improved quality of..." or "adequate resources for...". Alternatively, there are indicators that are not specific on what they want to measure like "increased productivity in a particular type of firms" without specifying what type of productivity (labor productivity, total factor productivity, capital intensity, unit labor costs) is supposed to be measured. Hence, the need to be clear and unambiguous about what is measured. It is good practice to choose indicators that are simple, can be measured and for which data is available and easy to get. The challenge is then not to come up with good (in a SMART sense) indicators, rather pick the ones that are "implementable" and easily measurable. Data source, frequency/schedule, and responsibility for its collection poorly specified. It would be good practice to measure and track indicators, especially outcome indicators, annually (to have at least one observation point in time, every year for the duration of the project life). If this is not possible because indicators rely on data sources for which the frequency of data collection is lower, they should be complemented with intermediate outcome indicators that can be measured with higher frequencies. Moreover, it is often not specified if target values are either cumulative or annual values. Failure to specify target population. Indicators could be more specific about the targeted population/area. It would be good practice to better define who are the "vulnerable" or "disadvantaged" or "at risk" groups targeted by programs. iv. Identification of Good Practice Once the PDO has been agreed and identified, identify indicators to measure progress towards the goals and retain logic of causality along the chain (inputsoutputsoutcomesimpacts) The choice of indicators should depend on the type of data available in the country as well as on what can be monitored given country capacity. Before choosing indicators, it is good practice to perform an analysis of what data are available and what data can be collected given resources and feasibility constraints. Choose indicators that can be tracked annually. Usually in the case of intermediate indicators they either rely on administrative data, when available and accurate, or on data collected by ministries ­ at the local and central level through MIS and other instruments. Outcome indicators on the other hand usually rely more on population data, or on ad hoc impact evaluation surveys, quick monitoring data, citizen scorecards, tracer studies, beneficiary assessments or other participatory studies. Set baselines and target values at the design stage, possibly by gender, income deciles, age, and geographical area. Maintain unit of measure (i.e. number or percentage) Performance indicators should be identified at all levels of the results chain: it is good practice to include output indicators measuring activities and implementation milestone among intermediate outcome indicators. Table 5: Example of Good Practice KPIs in Labor Market Projects Issue Weaker KPIs Stronger KPIs Specific Increase in contribution compliance (as a Increase DPT, Triple Viral, and polio proxy for the incentive to job creation and vaccination of the population age 05 from 80 reducing informality associated with the percent to 95 percent, resulting in 3.6 million reduction in social security contribution rate new children vaccinated. by 6% points) evident in a lower actual Increased efficiency as shown by halting the decrease in contribution revenue. decreasing trend in pupilteacher ratio in Productivity of former state owned public schools (for grades 113) observed enterprises will increase when privatized. since 2002. That is, the pupilteacher ratio Intermediate: remains roughly unchanged at its 2006 level Books distributed for EBA and PREPARA. (13.1). Intermediate: Percentage of EBA students that drop out of the program. Measurable Create an effectively functioning Ministry of Increased efficiency as shown by reductions in Social Protection. average dropouts as a percent of total Create 350,000 new jobs, above those enrolments in small mountainous created through normal growth. municipalities (proxies for regions that Number of participants in active labor struggle with largest declines in population) market programs inserted into jobs, Intermediate: including in the occupational field for which Percentage of students graduating from they are trained. Independent workers project training institutions and obtaining a have access to the social security system. diploma or certificate by student origin. Intermediate: Number of job seekers, as reported by the Number of workers receiving labor LFS, served by the National Employment redeployment services. Service. Attributable Reduce the unemployment rate from 15.7 to Number of participants exiting the program 12.1 percent. into formal employment (Annual). Reduction in the percentage of senior Percentage of PJE graduates employed or self managers in private enterprises stating that employed six months after program the interpretations of laws and regulations completion. by public officials is unpredictable and Increase in average percentage earnings of inconsistent. PJE graduates compared to the control group. Realistic Matura examination results are used by An increase in the employment rate of older Bulgarian universities as credential for workers to at least 40 percent (age 5564, admission. Eurostat data) from the 2005 baseline of 34.7 Intermediate: percent (intermediate status (2006) of 39.6 INFOTEP cofinances 400 learning courses for percent). youth at risk. Targeted Coverage of the very poor population by the Wages of graduates of long term training targeted social assistance program. program of the project institutions by student Decrease in risky behaviors. origin. Intermediate: Intermediate: Number of vulnerable children and youth Relative share of individuals benefiting from enrolled in primary and secondary education training out of the total number of through innovative targeted programs. beneficiaries of active labor market programs increases, from the 2005 baseline of 19.4 percent to 30 percent, with intermediate value of 26.5 percent (December 2007). Percentage of PJE, EBA and PREPARA beneficiaries that come from priority areas as identified in the DR poverty map. As mentioned earlier, 80 percent of intermediate outcome indicators in labor market projects are institutional development or process indicators. For the purpose of the analysis institutional indicators were classified according to the following groups: MIS or beneficiary registry development indicators that help assess progress in building reliable and timely administrative records of who is getting what program benefit. Service delivery quality indicators including compliance with service standards or models indicators and processes efficiency. They can be a complement to beneficiary satisfaction measurements to gauge the quality dimension of service provision. Transparency and accountability indicators that seek to assess performance in promoting monitoring and evaluation as well as access to program basic information. Social mobilization, participation and public awareness indicators aim at measuring institutional efforts to foster social awareness and participation Institutional capacity and structure indicators measure improvements in human resources skills and capabilities as well and organizational issues. Improved policy indicators can be found in projects involving technical assistance to assess progress in implementing better planned, coordinated, or more effective policies. Table 6: Example of Good Practice Institutional Development KPIs in Labor Market Projects Issue Weaker KPIs Stronger KPIs Service delivery quality Materials improved and provided to each Number of peopletimes of use of student. services (job counseling, referral, and Central and regional Career Counseling guidance) provided by projectinvested Centers operational. employment institutions, monthly average, by origin of the user. Number of child allowance benefits paid by ANSES through SUAF. Number of SUAF beneficiaries receiving their transfers directly in their bank accounts. Compliance with standards Central and three regional occupational 200 occupational standards, based on safety laboratories modernized and fully ISCO88/ISCO08, and 100 training functional and two reports on work and standards developed and in use. occupational safety published. Disability certification is based on Placement of students in upper secondary international standards and is expertly institutions is done on the basis of performed; DCS publishes its reports objective ability testing (using the new 7th annually. grade national test) Table 6: Example of Good Practice Institutional Development KPIs in Labor Market Projects (Continued) Issue Weaker KPIs Stronger KPIs Transparency/accountability Evaluation of the piloted "School to work" Percent of municipal education budget programs finished and at least three are delegated to schools increases (from mainstreamed. baseline of 0 to target of 100 percent; Integrated provision of social protection value 20062007: 80 percent). interventions evaluated. Automated, up to date information on Targeting mechanism and implementation nationwide vacancies available in all procedures adjusted after each round of local NES offices monitoring survey. Amounts claimed to ANSES for Social Protection survey is designed and reimbursement due to negative implemented. balances in the payment of allowances Enhance the quality of social programs through compensatory system through regular impact evaluations and a (Argentina Familias). system for the public to access and respond to the evaluation. MIS/beneficiary registry MIS system generates intended reports on Computerized record keeping in all programs, participants and flow of funds MLSPP and its affiliated agencies offices. and feeds into M&E activities Number of beneficiaries of Seguro MIS enables integrated work flow and (disaggregated by gender) sharing of business information within and Number of Employment Offices among MLSPP and its affiliated agencies providing Seguro Services departments and branches. Number of firms registered in SUAF Public awareness, Increase public access to information to Reports on Quality assessment prepared participation, social facilitate oversight by CKOKO and made available to the mobilization public increase (from a baseline of 0 to a target of 5). Labor market information (LMI) system in place and two reports on labor market situation and trends published. Institutional capacity Increase in diversity of private firms 30 new module training programs (staffing and structure) involved in PJE. developed and in use and three new Operations guidelines and manuals for a module training centers fully functional. new menu of ALMPs prepared and in use. EBA and PREPAPRA increase their ICT Department fully capable to provide coverage. administrative and technical support and Teachers trained and certified in adult maintain corporate network at all education methodology. territorial and functional and administrative levels. Improved policy Labor Code amended. White Paper on Pension System Development with a corresponding Action Plan prepared; implementation commences. Issuance of regulations to facilitate broader access to social security benefits While Table 5 and Table 6 draw examples of stronger and weaker KPIs from the pool of labor market projects in the SP portfolio considered in this report, Table 7 aims at recommending good practice indicators by type of intermediate and final outcomes and by type of labor market program7. Table 7: Good Practice Indicator by Type of Labor Market Failure and by Type of Program Type of labor Type of Intermediate Intermediate outcome Outcome Outcome indicator market program to outcome indicator constraint address it Improved Persons enrolled in vocational training programs and business (number, percentage of Targeted development target group) population self skills Program participants employed/ receiving a certificate at Improved employed 6 Vocational/life the end of training access to months/a year skills training (number, percentage) employment after program programs / self completion Lack of labor Strengthen Training providers employment supply (not capacity to meeting specified Employed earning adequate skills) deliver performance criteria (%) increased more than ( %) training income programs earnings Income earnings of Improved Number of internships targeted work provided households (% Apprenticeships experience cost/benefit ratio per increase) / Internships increased beneficiary efficiency of ALMP Number of beneficiaries Personsdays provided in Improved Income earnings of labor intensive public access to targeted works temporary households (% Increased Wage bill to total employment increase) Public works effectiveness expenses ratio and coverage Number of projects Improved Consumption/ completes access to expenses in food Number of assets infrastructur of targeted constructed/ e households (% rehabilitated increase) Wage subsidies Lack of labor Increase target population Targeted demand efficiency of reached by business population starting business support services a microenterprise development (number, %) (%) Improved support and (disaggregated by Survival rate of Business access to microenterpri income, gender) microenterprises development income se services targeted entrepreneurs started under and generating with a sound business program after X microfinance activities plan (according to months support criteria X) income generating Active microfinance activities with borrowers financial rate of Number of loans return above X% number of ban 7 Some of the indicators are drawn from the results chain matrix for Social Protection developed by the Africa Results Team. Table 7: Good Practice Indicator by Type of Labor Market Failure and by Type of Program (Continued) Type of labor Type of Intermediate Intermediate outcome Outcome Outcome indicator market program to outcome indicator constraint address it Target population covered (number, %) Job counseling Increased Number of Number of (local) and placement awareness beneficiaries Employment Offices assistance and job exiting the providing job Job search services information Improved program into placement/counseling constraints/asy access to formal services mmetric formal employment Number of firms information employment Number of registered in Job Improved beneficiaries who intermediation centers intermediation functioning of found a job in line Number of matching centers labor markets with their skills provided Rigidity of working hours index (Doing Revise labor Business) regulations, Difficulty in hiring/firing codes and tax Targeted workers index system Improved Improved population Labor market Firms' compliance with functioning of access to employment/une regulation labor regulation (rate) labor markets employment mployment rate Regular reports on Strengthen labor market situation Labor Market and trends published by Information labor market Systems information system 4. Design and Implementation of M&E v. Description of general practice Monitoring and evaluation plans are pretty much homogeneous in their components across labor market types of projects. A core packet can be identified consisting of MIS, beneficiary assessments, spot checks and impact evaluations (either only mentioned or designed in details). The use of a particular M&E tool reflects the type of project and its operational needs in terms of monitoring/evaluating performance. For instance, training and income support programs have always a MIS in place reflecting the need to produce results easy to compare and summarize as well as beneficiary assessments (especially in pilot projects) as a way to get direct feedback from beneficiaries. Impact evaluations are the most used tool for project evaluation being at least planned at appraisal in two third of projects. Contrary, process evaluations and technical audits are never found to be part of M&E plans in labor market reflecting either a lack of operational problems signaled by monitoring systems or, most likely, that they are substituted by other evaluation tools like impact evaluations and participatory methods or by other monitoring tools like spot checks. It is worth noting a difference in the use of M&E tools by lending instrument. In general, investment operation design and develop a more comprehensive and sophisticated M&E setup than DPLs do: all investment operations rely on at least three to four M&E components with a clear tendency of using Management Information Systems and spot checks. By contrast, M&E plans in DPLs operations are not articulated around many components; they rely on results based monitoring systems as well as qualitative approaches like beneficiary assessments and social audits. Interestingly, all four DPLs operations have planned to conduct impact evaluations of one or more pillars even though not many details are provided in PADs. Among all social protection projects labor market projects are the ones for which the M&E component is more often costed at appraisal: the M&E component is explicitly costed in one third of projects budgeting on average 2 million dollars. Table 8: Use of M&E Components in Labor Market Projects Number of projects MIS/ Process Beneficiary Spot Impact Technica Partici Expendi "Other" Beneficiary Evaluation Assessment Checks evaluati l/Op. patory ture registry on Audits M&E tracking planned Labor markets (total) 5 0 4 3 7 0 1 1 3 Training or retraining for workers, unemployed persons, and labor market entrants 2 0 1 1 2 0 8 0 1 Labor administration and institutional capacity building 1 0 1 0 1 0 0 0 0 Employment subsidies 0 0 0 0 1 0 0 0 0 Social assistance and income support 2 0 1 2 1 0 1 1 1 Other specify in comments 0 0 1 0 2 0 0 0 1 vi. Assessment of Performance Overall ratings on quality of M&E arrangements at appraisal were moderately satisfactory with DPLs operations performing slightly better than investment operations. Especially on the degree of integration of project M&E with existing national and sectoral M&E systems, almost all projects tried to integrate where feasible and appropriate and performed above average with Argentina Basic Protection as the best example (see box 1). Investment operations were found to be more integrated compared to DPLs. Moreover, analysis of client capacity to carry out M&E activities as well as plans developed to strengthen client capacity if weaknesses in the systems were identified appear to be well spelled out in PADs in almost all projects. However, explicit collaboration with other donors on M&E work was found in only 3 projects; none of those 3 was a DPL operation. Table 9: M&E Arrangements Readiness by Lending Instrument Average quality Project M&E PAD analysis of Plan to Summary rating of M&E integration with M&E client strengthen of M&E institutional the capacity client capacity arrangements in arrangements national/sectoral PAD beyond M&E system indicators Highly Satisfactory 1 1 0 0 0 Satisfactory 2 5 1 5 4 Moderately Satisfactory 4 1 6 4 4 Moderately Unsatisfactory 2 1 2 0 1 Unsatisfactory 0 1 0 0 0 Highly Unsatisfactory 0 0 0 0 0 NA 0 0 0 0 0 Average rating 2.78 2.56 3.11 2.44 2.6 Investment 2.8 2.2 3 2.4 2.8 DPLs 2.75 3 3.25 2.5 2.5 The performance of M&E arrangements during implementation is clearly the weakest aspect of the results agenda for this cohort. With most of the projects in the cohort suffering from long implementation delays, the M&E arrangements spelled out in the PADs have remained mostly on paper with none of the projects meriting even a satisfactory rating in terms of actual data collection efforts. For 4 projects the average rating of results readiness of M&E arrangements at the design stage was higher than the average rate of M&E arrangements performance during implementation and quality of results tracking. Typical problems found are related to delays in procurement, in establishing consistent MIS reporting activities and delays from external consultancies implementing beneficiary assessments and impact evaluations. As a consequence of the delays and operational constraints found in implementing data collection plans, KPIs are very often not updated regularly (one exception again is Argentina Basic Protection8). Hence indicators are rarely used to inform program managers on the operational problems. These findings suggest that greater attention should be paid at appraisal on implementation arrangements and operationalization of M&E tools. Table 10: M&E arrangements performance at implementation Was the data collection Are KPIs updated Are the KPIs used to plan described in the PAD regularly? measure progress and to actually implemented? inform decisions to correct any problem identified? Highly Satisfactory 0 1 0 Satisfactory 0 0 3 Moderately Satisfactory 0 3 1 Moderately Unsatisfactory 1 1 1 Unsatisfactory 1 1 2 Highly Unsatisfactory 2 0 0 NA 5 3 2 Average rating 4.25 3.16 3.28 Impact evaluations: For almost half of the projects it is mentioned that impact evaluations are under way but PADs do not provide any detailed information on the type of methodology applied, data collections plans, type of research questions to be addressed. China and Dominican Republic are exceptions having impact evaluations designs well described at appraisal and both baseline and followup data collections planned. Regarding costing, only the impact evaluation for the China migration project was explicitly costed ($220.000). It is worth noting that three out of four DPLs plan to carry out an impact evaluation of one or more project components. In all cases PADs take into account past impact evaluations and analytical work but never present a proper discussion on strategies to identify a counterfactual and methodologies to be applied. It is then difficult to assess whether planned impact evaluations are rigorous ­ experimental or quasiexperimental ­ but at the same time capacity building work is supported for impact evaluation and household surveys. For example, the reform program in the third Bulgaria programmatic DPL supports the gathering of new data through a targeted survey of beneficiaries who were affected by benefit eligibility tightening. 8 All indicators have baseline and targets and have been updated regularly every six months except indicators related to component 3. vii. Key Issues KPIs are found rarely updated in ISRs. The main causes were found to be related to data collection delays and implementation problems as discussed above. In addition, unclear definition of the frequencies of regular reporting as well as unclear specific responsibilities for indicator reporting activities (beyond the general MIS) generate problems in tracking indicators on time. Moreover, when KPIs rely on data that are not produced by the program but come from external sources, the monitoring becomes more problematic because data are less owned by project stakeholders ultimately resulting in KPIs not updated. Inconsistencies in KPIs reporting. The review found lots of unexplained disconnections between indicators presented in PAD body text, Annex 3 and indicators reported in ISRs. For example, the migrants' skill development project in China does not include two of the three outcome indicators and six out of the ten intermediate indicators. These inconsistencies reflect the fact that often KPIs selected and reported in ISRs are those more easily tracked ­ those produced by MIS or other internal tools not necessarily those needed tom monitor progress. Quality of KPIs is rarely measured. Spot checks information is never integrated to the rest of M&E tools (at least base on ISRs) when they are part of the M&E plan at the design stage. Technical audits to check on quality and process indicators are absent in the M&E package of these cohort of projects. A first step in design and developing an M&E plan should always include a stocktaking effort in analyzing and determining client's M&E capabilities and skills across all possible implementing actors: central offices, line ministries, local governments, provinces, districts officials, NGOs, research centers and civil society organizations. It may be better to put in place a simpler M&E plan with fewer mechanisms that can be implemented immediately rather than design a sophisticated. In the case of Azerbajan for example, the M&E framework appears too overburdening and taxing on the implementing agencies. The analysis of implementing agency capacity found in the PAD was too superficial. Delays in implementation are the main issues found. In Dominican Republic task team and managers pursued diligently M&E aspects but delays in project implementation have been a major handicap. Although more than half of the projects mention to support evaluations or intend to evaluate the impact of specific programs (when described in appraisal documents), few proposed evaluations designs are actually conducted (from ISRs reviews). Usually activities without an evaluation design at appraisal are never evaluated. When client capacity didn't appear to be a constraint in implementing M&E arrangements, establishing partnerships with local implementing actors resulted in a key issue. In Turkey for example, the main problem was not so much country capacity rather the intent of the counterparts who turned out to be uncooperative. viii. Identification of good practice Box 1. Argentina Basic Protection Project ­ Using and consolidating existing M&E systems The Argentina Basic Protection project supports the GoA to slowly transfer all beneficiaries of Jefes (emergency public works program) to Asignaciones Familiares (transfer program to formal salaried workers supported by the IADB) and Seguro (training, unemployment benefits and job placement program) considered better managed and more effective. Since both programs have the potential to be scaled up, lot of resources ($5 million) are allocated to a technical assistance component aiming at strengthening a social protection M&E systems by designing and implementation of a National Social Protection Survey to assess the effectiveness of current SP policies and promote reforms when necessary, evaluation studies and their dissemination. The project M&E plan builds on the experience of the GoA and WB during the implementation of previous programs. Important lessons are drawn from the experience of Jefes, notably the importance of transparency and accountability in the administration of benefits. Specifically, social accountability was enhanced through: (i) regular monthly crosschecks control of the registry database with other databases by an independent agency; (ii) greater use of public information and participatory mechanisms through workshops in order to inform and promote participation of local organizations); (iii) more supervision missions and media monitoring (the Bank contracted a service to provide reports on Jefes and Seguro covering all national newspapers). Both Seguro and Asignaciones Familiares have efficient accountable and transparent systems for transferring resources. Both have well design participatory M&E like complaints systems where participants can communicate on process mistakes or fraud claims. Complaints systems include local offices, internet portals and free telephone numbers to receive such complaints, internal process to deal with problems and mistakes and to correct them. The monitoring of Seguro is based on the existing administrative systems (social security contribution database), MIS and beneficiary registry used in Jefes, improved and refined over time. The MIS is central to the execution and management of beneficiaries transferring from Jefes to Seguro. In addition to the beneficiary registry of Jefes, a newly designed system called Emplear was put in place in order to collect information about individuals seeking a job placement and employed workers seeking career advancements. Monthly crosschecks of the registry are carried out centrally with key databases (formal private and public employees, pensioners, other social programs, tax lists, and health insurance beneficiaries) before payment is authorized. Transfer payments are monitored by Banco Nacional and overseen by a national audit agency. The Minister of Labor consolidates data on a monthly basis and transmits them to the Ministry of the Economy and the Bank. This information will enable the Bank and the Ministry to monitor trends in transfer payments, participation, compliance with eligibility criteria and exit from the program by reasons and to effectively use results in order to improve Seguro. 5. Use of Results i. Description of general practice A crucial element of an M&E system is the use of results. Monitoring and evaluation shouldn't be stand alone technical activities, but rather closely linked to decision making processes at all levels and provide feedback on program performance to project managers, policy makers, civil society and researchers. Using results and triggering an efficient feedback mechanism is extremely linked to the quality of indicators and M&E tools selected, the quality of institutional arrangements to carry out M&E activities, the extent to which they and to the quality of M&E implementation. Any failure or bottleneck in either the design or implementation phase reflects in the feedback process. Lack of results, delays in reporting make impossible to activate the feedback mechanism and to ultimately allow M&E systems to evolve over time adapting themselves to the country context, capacity and needs (when not adequately captured at the design stage). Since the assessment of this section is uniquely based on ISRs and ICRs reports9, the analysis of how monitoring and evaluation results are used as a management tool to decide on future course of action is restricted only to use of results by Bank task teams and management and not much by borrowers10. It should be also noted that many projects have been launched recently (in the past two fiscal years), thus only few ISRs were available at the time of the desk review making it difficult to assess implementation, existence and use of results. ii. Assessment of Performance General performance on use of results in labor market projects is moderately satisfactory above the average rating of all social protection projects. The average rating is based on whether and how deeply M&E issues are discussed by Bank task teams under the "key issues and actions for management" in ISRs and on whether Bank management is commenting on M&E issues raised in the respective final section. A particular encouraging feature in results tracking is the attention managers are beginning to M&E coverage. Turkey PSSP 2 and the Azerbaijan Social Protection Dev. are commendable examples of that. Clearly however, we have a long way to go before we realize the full potential of the results agenda in improving the impact of the Banksupported projects and programs. In the Turkey project, the manager and country director provided thoughtful comments on M&E but mostly ignored by the first TTL. Apparently current TTL tried to fix the M&E performance but without much success due to non cooperation from the borrower. However commendable attention by the managers; followup corrective actions by the current TTL came too late and were ineffective due to borrower resistance. Similarly, Dominical Republic DO Youth Development Project is a best practice example of the level of attention paid by task teams on M&E and impact evaluation implementation problems. As the results frameworks had focused only on outcomes and impacts, they have remained disconnected from the issues tracked by TTLs which in the early stages must necessarily focus on inputs and outputs needed for launching project implementation. Perhaps reflecting this "feeding of the beast perception" on part of the TTLs, most ISRs seem to do a fairly sloppy job of reporting on the agreed indicators (Argentina Basic Protection Project is a notable exception in this respect). In many cases the indicators tracked in the ISRs differ from those in the PADs and often time the ISRs report data that is really stale, making it practically useless for tracking of results. 9 None of the projects in this cohort has an ICR. 10 Only to the extent that the information on use of results by borrowers in contained in ISRs/ICR. In terms of restructuration, KPIs have been substantially restructured in only one project (Turkey). Lastly, project ratings in ISRs often bear little relationship to the data reported in the Results Matrices. For example, in the case of Bulgaria SIR second programmatic DPL, indicators are reported with no comments on their progress but the ISR rate to M&E s satisfactory. Neither upgrading nor downgrading in M&E rating was found in this group of projects. Table 11: Use of Results by Bank Team and Management M&E issues discussed Is Bank management Summary rating of under Key Issues and commenting on results monitoring Most recent M&E Actions for results and M&E and use by the rating in ISR Management issues? Bank Highly Satisfactory 0 1 2 0 Satisfactory 7 4 1 4 Moderately Satisfactory 0 0 2 2 Moderately Unsatisfactory 0 3 1 3 Unsatisfactory 1 0 0 0 Highly Unsatisfactory 0 1 1 0 NA 1 0 2 0 Average score 3 2.9 2.9 iii. Key Issues Limited use of KPIs in informing stakeholders on project progress and performance. Poor reporting activity and often absent KPI updating in ISRs are not reflected in M&E ratings which are satisfactory. Most importantly, poor monitoring of KPIs translates into poor tracking of PDOs. Hence, it becomes difficult to assess if PDOs can be achieved and to justify PDO ratings. iv. Identification of good practice Box 2: Colombia Labor and Social Sector (3rd programmatic): M&E Support Through DPL Operation One of the objectives of the third Labor Reform and Social Development Policy Loan is to build monitoring and evaluation tools and processes to enable public oversight and improve policymaking in the social and labor sectors. The prior actions supported by the DPL included the increase in public access to information to facilitate oversight and the creation of mechanisms for public feedback on government performance. On the monitoring side, the GoC used a webbased system for monitoring program results and progress. Second, to give citizens better information about how public resources are spent and to move towards a resultsbased budgeting system, the National Planning Department (National Directorate of Evaluation division) introduced programmatic classification of investment budgets into the monitoring system: programlevel budgets are accessible to citizens. Third, subnational governments are supported in creating their own webbased results tracking systems of their municipal development plans. On the evaluation side, the DPL continue to support the impact evaluation agenda (which has been growing counting one completed impact evaluation and 11 in process) and the establishments of partnerships between civil society organizations as well as the private sector, universities and NGOs to exercise social control of national and subnational programs. The civil societygovernment engagement has been formalized also at the subnational level to improve oversight. Finally, as good practice example of mechanism for public feedback on government performance it is worth mentioning the online user evaluation of the national monitoring system (SIGOB). The survey was launched at the end of 2005 and results analyzed in order to assess the usefulness and accessibility by citizens of the monitoring system. 5. Main Messages and Conclusions Overall monitoring remain weak and evaluation (except impact evaluations) almost nonexistent during project implementation. The main reasons found in the review are related to the low linkage of KPIs to respective PDOs, poorly specified data collection sources, frequency and responsibilities at the appraisal which translated into delays and lack of proper monitoring during project implementation. For instance, the moderately satisfactory quality of project M&E performance during implementation is poor reporting and updating of KPIs and designed data collection plans not implemented. None of the intermediate KPIs included any measures to track project implementation milestones and related outputs thus leaving a gap in the results framework for project monitoring, especially in the early phases of implementation when the focus has to be on inputs and outputs. It is therefore recommended to identify performance indicators at all levels of the results chain, also including process/institutional development indicators and output indicators measuring activities and implementation milestone among intermediate outcome indicators. Need to create new incentives for M&E for both the Bank and the borrower. This would include specific requirements for baseline data, explicit evaluation designs for pilot activities in project at the appraisal and periodic monitoring and evaluation tools of main project activities during implementation. For example, ISRs are the most important instruments the Bank has in the supervision phase of the project cycle. The monitoring function of ISR should be pushed ahead of the "flagging issues" function which ISRs seem to have now. The use of results as a management tool won't be effective as long as there are no incentives for reporting in ISRs. Specific requirements for ISRs would include reporting all KPIs present in PADs and the frequencies at which they are collected in order to update indicators accordingly. Appendix 1: Sample KPIs by Project Objective Area Focus area Sample KPIs Data Collection SP thematic area Lending Instrument Instruments Access to public services and infrastructure (local public goods) Net enrollment grade (by grade Household surveys Social Funds SIL and/or gender) More final outcome % of people with access to safe Household surveys Social Funds SIL oriented drinking water Increase in population with access to Household surveys or Social Funds SIL improved health facilities within 5km health information system if functioning Increase with population with access Household surveys or Social Funds SIL to improved economic infrastructure Project MIS (roads, irrigation, markets) % of subprojects operational and Facility surveys Social Funds ERL maintained one year after completion % households reporting utilization of Household surveys Service Delivery SIL health centers Proportion of births attended by Household surveys Service Delivery SIL skilled health personnel Availability of essential drugs at Facility survey or health Service Delivery SIL health facilities (%) (Percentage of information system if More project output months with tracer drugs functioning oriented availability) % of births in rural targeted Household survey including Social Safety Nets DPL communities in accredited facilities demographic information % of children in poor households Household survey (linked Social Safety Nets SIL (quintiles 1 and 2) in targeted with national income or communities participating in Early consumption household Childhood Education Services data) 23 % of public works assessed to be Technical audits Social Safety Nets ERL satisfactory (using PW Review performance criteria eligibility, appropriateness, effectiveness, quality and sustainability) Number of communities benefiting Social Fund MIS Social Funds ERL from at least one subproject % of targeted Roma settlements Administrative data Social Safety Nets SIL water points constructed or rehabilitated # of classrooms/health centers Social Fund MIS Social Funds ERL constructed % of classrooms replaced in 40 Administrative data Social Safety Nets ERL schools damaged by the Tsunami % of kilometers of rural roads Administrative data Social Safety Nets ERL rehabilitated # of bednets, contraceptives etc. Administrative data from Service Delivery SIL distributed sector # of service workers deployed Administrative data from Service Delivery SIL (teachers, health workers, etc.) sector Access to economic opportunities Number of participants exiting the Household surveys Labor markets SIL program into formal employment (Annual). More final outcome An increase in the employment rate National surveys (eurostat) Labor markets DPL oriented of younger workers to at least 23 percent (age 1524) % increase of the rate of Household survey or Social Safety Nets SIL employment of youth benefiting national labor force survey from multifunctional centers services with beneficiary oversampling An increase in the activity rate of National surveys (eurostat) Labor markets DPL older workers to at least 45 percent (age 5564) 24 Percentage of graduates employed Household surveys, tracer Labor markets SIL or selfemployed six months after studies of exparticipants program completion Employment rate for labor market Tracer studies of ex Service Delivery program graduates participants Wages of graduates of long term Household surveys, tracer Labor markets SIL training program of the project studies institutions by student origin. # of households with annual increase Household surveys Social Funds APL in average incomes % of microfinance savers/borrowers Tracer studies, beneficiary Social Funds SIL confirming an improvement in their surveys household living standards Cumulative # of farming HH Project or sectoral Service Delivery SIL receiving and using extension information system More project output packages (millions) (distribution of packages), oriented household or farm surveys (using packages) # of income generating local Social fund MIS Social Funds SIL community groups formed % of supported MFIs reaching Survey of MFIs Social Funds SIL operational selfsufficiency Reduction in job search waiting time Household survey or Service Delivery SIL program MIS if reported Number of beneficiaries of program Project MIS Labor markets SIL X. Number of Employment Offices Administrative information Labor markets SIL providing Program X Services systems 25 Number of firms registered in Project MIS Labor markets SIL program X # of clients of supported MFIs Social Fund MIS Social Funds SIL # of beneficiaries trained in income Social Fund MIS Social Funds SIL generating activities Cumulative # of people trained Project or sectoral Service Delivery SIL (farmers, microentrepreneurs etc.) information systems Temporary income support (unconditional transfers, public works CCTs, etc.) Increased income from wages Household surveys Social Funds SIL (income transfers) from PWP works More final outcome oriented % of beneficiaries of grants for Household survey Social Safety Nets productive projects engaged in activities generating income at least equal to AR$150/mo after 18 months of operation % of beneficiary households that Beneficiary survey Social Safety Nets SIL receive PSNP resources reporting no distress sales of assets to meet food needs Net improvement in household total, Household surveys Social Safety Nets SIL food (including fruits and vegetables) including consumption and protein consumption in large module for treatment and municipalities. comparison groups Average number of months that Beneficiary survey Social Safety Nets SIL PSNP households report being food insecure % of labor intensity of public works Social fund MIS Social Funds subprojects Proportion of bimonthly payments Administrative data Social Safety Nets SIL More project output based on a complete cycle of co oriented responsibilities verification and application of sanctions 26 Persondays provided in labor Social fund MIS Social Funds SIL intensive public works program (number) Percentage of farmers who have Administrative data Social Safety Nets ERL been affected by the Tsunami receiving compensation grants. # of communities benefitting from Social fund MIS Social Funds public works subprojects % of transfer payments delivered to Administrative data Social Safety Nets SIL beneficiaries by the 13th of the payment month Human development outcomes Increased primary school enrollment Household surveys Social Funds SIL and completion rates in SF More final outcome supported communities oriented Net change in chronic malnutrition Household survey with Social Safety Nets SIL amongst children under 5 in large anthropometric module municipalities. applied to treatment and comparison groups; possibly, Health MIS if available and feasible to distinguish treatment and comparison groups Net change in secondary school Household survey applied Social Safety Nets SIL completion rate of PATH students to treatment and boys and girls comparison groups; possibly, Education MIS if available and feasible to More project output distinguish treatment and oriented comparison groups 27 Increase DPT, Triple Viral, and polio Sectoral information Labor markets SAL vaccination of the population age 0 systems, Health surveillance 5 resulting in 3.6 million new system children vaccinated. Increase in primary completion rate Household surveys or Service Delivery PRSC sectoral information systems Increase basic and secondary school Household surveys , Labor markets SAL enrollment rates by creating 1.4 Sectoral information million new spaces, in addition to systems the existing 7.8 million spaces. Annual dropout rate (%) of students Facility or household Social Funds SIL in primary and secondary schools surveys Reduction in child underweight rate Household survey (with Service Delivery PRSC anthropometric module) or health information system % of underfives malnutrition using Household surveys Social Funds SIL weight for age method Incidence of diarrhea in communities Household surveys Social Funds SIL implementing water and sanitation subprojects % of children vaccinated Household surveys and Social Funds SIL health information system % of women receiving prenatal care Household and facility Social Funds SIL in project communities surveys Average daily medical consultations Facility surveys of health Social Funds SIL in health centers information system HIV/AIDS prevalence rate Health surveillance system Service Delivery PRSC Infant/child mortality rate Household surveys Service Delivery SIL DTP 3 vaccination rate of children Sectoral information Service Delivery PRSC below 1 year systems 28 Contraceptive Prevalence Women Household survey Service Delivery SIL ages 1549 % of children under 3 with anemia in Household survey including Social Safety Nets 165 high risk municipalities blood samples % of married women aged 1549 DHS type survey Social Safety Nets DPL using modern contraceptives Percentage of children 05 years old Household survey; Health Social Safety Nets SIL with complete immunizations for MIS if available their age group Education test scores Sectoral information system Service Delivery DPL Share of PATH secondary school Household survey, Program Social Safety Nets SIL boys attending school 85% of the MIS time Proportion of children under 2 years Household survey; Health Social Safety Nets SIL old who participate in the growth MIS if available monitoring according to agreed protocols in the areas of intervention Percentage of families meeting their Program MIS Social Safety Nets SIL education coresponsibilities Reaching specific vulnerable groups/targeted groups Number of workers that receive Specific assessment and Labor markets SAL severance pay (job loss surveys compensation) Increase the number of poor Project MIS/Beneficiary Labor markets SIL children age 05 affiliated with the registry nutrition program 29 Percentage of program beneficiaries Project MIS Labor markets SIL More final outcome women oriented Number of workers receiving labor Project MIS Labor markets SAL redeployment services. Percentage of beneficiaries that Administrative information Labor markets SIL come from priority areas as system identified in the poverty map. % of project resources (or Household surveys (linked Social Funds SIL beneficiaries) from the lowest 20% with national household of poverty deciles surveys for national poverty rankings) Increased number of poor people Household surveys Social Funds ERL with access to social services. Number of kindergarten children Facility survey Social Funds SIL with disability benefiting from the kindergarten centers Community Management Social Fund MIS Social Funds SIL Committees have at least 50% of elected women % of project resources disbursed to Social Fund MIS Social Funds SIL the poorest districts/municipalities Number of communities mapped Social Fund MIS Social Funds SIL and profiled (with participatory poverty assessments) % of citizens with access to services Household survey Service Delivery by poverty quintile More project output Increased access to services (e.g. Sectoral data crossed with Service Delivery DPL oriented institutional births, school poverty mapping enrollment etc.) in poorest information geographical areas % of vulnerable groups (e.g orphans, Household survey Service Delivery elderly) using health and education services 30 % of program expenditures (or % of Household survey Service Delivery TA beneficiaries) on poorest household quintile At least X percent of grants provided Project MIS Service Delivery SIL for inclusion activities to marginalized groups in project municipalities are disbursed % of programs utilizing proxy means Administrative data Service Delivery TA test and other targeting criteria Percentage of disbursed cash Household surveys (linked Social Safety Nets SIL transfers received by households in with national income or quintiles 1 and 2 consumption household data) including social programs module Share of program participants in Household surveys (linked Social Safety Nets SIL lower half of income distribution with national income or consumption household data) including social programs module % of children in Q1 registered for Household surveys (linked Social Safety Nets SIL PATH with national income or consumption household data) including social programs module % reduction in the gap between Depending on exact Social Safety Nets SIL targeted poor Roma settlements and definition of living neighboring communities as conditions index may need measured by the living conditions census data, community index surveys or household survey representative at the community level % of revised SISBEN 1 families in Administrative data Social Safety Nets SIL newly entered municipalities registered in the program 31 % of ECE interventions in Census data to identify Social Safety Nets SIL communities with more than 50% target communities and Roma population administrative data % of districts with at least 85% of Administrative data Social Safety Nets SIL households with completed needs assessment % recertification of PATH families Administrative data Social Safety Nets SIL with children due to be recertified after 4 years enrollment Community empowerment and capacity building (nongovernment) % of participating communities have Social Fund MIS, Social Funds SIL improved their capacity to Community surveys More final outcome implement the CDD approach, and oriented comply with at least 10 CDD capacity indicators # of local executing agencies with Community surveys Social Funds ERL improved capacity for planning and implementing community development projects % of communities applying acquired Community surveys Social Funds DPL skills in activities beyond social fund project (use of community management committee, social audits etc.) % of facilities adequately maintained Facility surveys Social Funds by the community More project output oriented At least 90% of subprojects Household surveys, Social Funds ERL undertaken reflect the priorities of participatory M & E targeted communities and instruments beneficiaries 32 % of beneficiaries reporting Household surveys, Social Funds SIL participating in identification and participatory M & E execution of subprojects instruments # of capacitybuilding events carried Social fund MIS Social Funds ERL out for beneficiary CBOs each year % of communities benefiting from Social fund MIS Social Funds ERL social mobilization Increase in citizen satisfaction with Participatory M & E Service Delivery SIL social services as measured by instruments Citizen Report Cards and Community Score Cards % of citizens who report knowledge Household survey or Service Delivery SIL about local budgets participatory M & E instruments Improved women's understanding of Household survey or Service Delivery DPL their right to an institutional birth participatory M & E and the standards for a goodquality instruments birth Increased connectivity to and usage Administrative information, Service Delivery SIL of national information management service provider surveys networks, by local service providers # of local management committees Project (or sectoral) Service Delivery TA established for social services information system Share of coverage in community Sectoral information Service Delivery services by NGOs and community systems groups 33 % of Roma population from targeted Household surveys, Social Safety Nets SIL poor settlements agreeing that participatory M&E subprojects reflect community instruments priorities Proportion of OVC households aware Beneficiary surveys, Social Safety Nets SIL of program information, such as participatory M&E objectives, eligibility criteria and use instruments of funds % of kebeles that have developed Administrative data, Social Safety Nets APL and approved safety net plans, performance audits taking intoaccount community preferences % of public sector organizations that Administrative data Social Safety Nets SIL hold partner forums on a regular basis to report on planning decision and progress. Local government institutional capacity building Household perceptions of local Household surveys, Social Funds SIL government transparency, capacity participatory M & E and responsiveness instruments % of local governments able to set Local government surveys Social Funds APL objectives and achieve at least X% of their annual targets % of local government annual Local government surveys Social Funds SIL investments consistent with Community Development Plans 34 Beneficiary satisfaction with training Community and local Social Funds SIL received at the local government government surveys, More final outcome level participatory M & E oriented instruments Legal and administrative texts have Administrative information Social Funds SIL been adopted allowing local governments to pass through funds to community groups % of social fund projects executed by Social fund MIS Social Funds SIL local governments # of local government officials Social fund MIS Social Funds SIL trained Citizens reports of quality of basic Participatory M & E Service Delivery services delivered by local instruments or household government surveys Subnational governments Sectoral budget information Service Delivery SIL expenditures on basic services Allocation of financing from State National and sectoral Service Delivery SIL government to local self budget information governments for health and education services More project output oriented Share of local government budgets Local government budget Service Delivery DPL spent of social services information (systematized or via sample surveys where systems lacking) Number of local service providers Sectoral information Service Delivery SIL (PHCs, schools, CSWs and NGOs) systems accredited by the relevant institution in their sector 35 # of local government units trained Administrative information Service Delivery TA in sectoral service programs systems National/sectoral government institutional capacity building Number of job seekers, as reported Administrative information Labor markets SIL More final outcome by the LFS, served by the National system oriented Employment Service Formulation of a viable Pension Administrative data Labor markets SIL System Development Strategy and progress in its implementation Labor market information (LMI) Sectoral information Labor markets SIL system in place and two reports on systems labor market situation and trends published Number of new small business Project MIS Labor markets SIL incubators established. Number of new module training Project MIS Labor markets SIL programs developed and in use and number of new module training centers fully functional Operations guidelines and manuals Administrative information Labor markets SIL for a new menu of ALMPs prepared system and in use More project output Central and regional Career Administrative information Labor markets SIL oriented Counseling Centers operational system Citizen knowledgeable about basic Participatory M & E Service Delivery services (availability, eligibility, instruments transparency) % of social program budget Sectoral budget information Service Delivery TA distributed based on new program system based model of allocation 36 Number of external evaluations of Administrative data Service Delivery TA social program performance conducted Monitoring and evaluation system Administrative data Service Delivery TA for prioritized national strategic social programs operating as evidenced by timely monthly MIS reports Number of Results Agreements Administrative data Service Delivery TA signed between national planning office and sectors Decrees published relating to social Administrative data Service Delivery program operations Percentage of civil registry offices Administrative data, Social Safety Nets ERL which have their records updated, technical/operational audits modernized and operational, as evidenced by the establishment of a new computerized system % of households in the targeting Administrative data, Social Safety Nets DPL database whose poverty status is technical/operational audits verified through home visits at least once every three years % of grievance redressal claims Administrative data Social Safety Nets TA settled within three months of application % of BISP positions that are filled Administrative data Social Safety Nets TA with qualified staff according to official examination system % of social welfare facilities have Administrative data, Social Safety Nets SIL acceptable hygienic standards in operational audits, facilities place to meet national criteria survey, spot checks 37 % of MoSD staff with completed Administrative data Social Safety Nets SIL training envisaged in the Human Resources Development Strategy % of municipalities with indicators of Administrative data Social Safety Nets SIL payment process in yellow and red alert % of departmental offices for the Administrative data, Social Safety Nets SIL First Employment Program (DOFEP) operational audits, spot operating and connected to the MIS checks (as described in the operational manual) Quality of services/service delivery Increase in average percentage Impact evaluation surveys Labor markets SIL earnings of PJE graduates compared to the control group. Improved user satisfaction with Participatory Labor markets SIL program X and its affiliated M&E/Beneficiary agencies# services Assessments Amounts claimed to agency X for Project MIS/ Participatory Labor markets SIL reimbursement due to negative M&E balances in the payment of allowances through compensatory system. Number of beneficiaries receiving Project MIS/Administrative Labor markets SIL their transfers directly in their bank data accounts. 38 Number of peopletimes of use of Project MIS Labor markets SIL services (job counseling, referral, and guidance) provided by project invested employment institutions, monthly average, by origin of the user Number of migrantstimes receiving Specific surveys Labor markets SIL legal assistance in projectinvested pilot localities annually More final outcome Increase in the number of new Project MIS Labor markets SIL oriented courses offered by program X Teachers trained and certified in Sectoral information system Labor markets SIL adult education methodology. Material improved and provided to Participatory M & E Labor markets SIL each student instruments Life skills modules incorporated in all Project MIS Labor markets SIL programs Community satisfaction with service Participatory M & E Social Funds SIL quality instruments, household surveys Student/teacher (or Facility surveys, education Social Funds SIL student/classroom) ratio system information % of health centers staffed to Facility surveys or health Social Funds sectoral norms information system if functioning Availability of basic medicines in Facility surveys Social Funds supported health centers # of desks, books and other inputs Facility surveys Social Funds per student 39 Quality rating of local infrastructure Facility surveys, technical Social Funds (schools, health centers, water audit systems, roads etc.) More project output # of facilities with local community Facility surveys, social fund Social Funds oriented management committees MIS established Average pupilteacher ratio (e.g. Service Delivery SIL grades 14) Share of qualified health staff in rural Sectoral information system Service Delivery DPL areas Public perception of service quality Participatory M & E Service Delivery SIL instruments, household surveys % of health posts without shortages Sectoral information Service Delivery SIL of injectable contraceptives in last 3 systems months Increase in the use of Standard Sectoral information Service Delivery SIL Treatment Protocols in project systems supported hospitals Number of textbooks per primary Sectoral information Service Delivery PRSC school student systems % of teachers receiving inservice Sectoral information Service Delivery SIL training systems Fiscal objectives/efficiency Average unit costs of infrastructure Social fund MIS Social Funds (compared to other programs) More final outcome % of operating costs as a share of Social fund MIS Social Funds SIL oriented total costs Local contributions as a share of Social fund MIS Social Funds total social fund investments 40 % of subproject executed within Social fund MIS Social Funds SIL timeframe and budget Total Federal block grants to basic National budget Service Delivery SIL services as a share of total Federal information system discretionary expenditures % of treatment municipalities Administrative information Service Delivery SIL making financial allocations based on system agreed formula in health sector % of non wage executed budget Sectoral budget information Service Delivery DPL system More project output Annual budget of national strategic National budget Service Delivery TA oriented social programs information system % of priority social program Program or sectoral Service Delivery expenditures on administrative information systems overhead 41 42 Social Protection Discussion Paper Series Titles No. Title 1106 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Social Service Delivery Projects by Julie Van Domelen, February 2011 1105 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Social Safety Nets Task Teams by Gloria Rubio, February 2011 1104 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Social Funds Task Teams by Julie Van Domelen, February 2011 1103 Results Readiness in Social Protection & Labor Operations: Technical Guidance Notes for Labor Markets Task Teams by Maddalena Honorati, February 2011 1102 Natural Disasters: What is the Role for Social Safety Nets? by Larissa Pelham, Edward Clay and Tim Braunholz, February 2011 1101 North-South Knowledge Sharing on Incentive-based Conditional Cash Transfer Programs by Lawrence Aber and Laura B. Rawlings, January 2011 1008 Social Policy, Perceptions and the Press: An Analysis of the Media's Treatment of Conditional Cash Transfers in Brazil by Kathy Lindert and Vanina Vincensini, December 2010 (online only) 1007 Bringing Financial Literacy and Education to Low and Middle Income Countries: The Need to Review, Adjust, and Extend Current Wisdom by Robert Holzmann, July 2010 (online only) 1006 Key Characteristics of Employment Regulation in the Middle East and North Africa by Diego F. Angel-Urdinola and Arvo Kuddo with support from Kimie Tanabe and May Wazzan, July 2010 (online only) 1005 Non-Public Provision of Active Labor Market Programs in Arab-Mediterranean Countries: An Inventory of Youth Programs by Diego F. Angel-Urdinola, Amina Semlali and Stefanie Brodmann, July 2010 (online only) 1004 The Investment in Job Training: Why Are SMEs Lagging So Much Behind? by Rita K. Almeida and Reyes Aterido, May 2010 (online only) 1003 Disability and International Cooperation and Development: A Review of Policies and Practices by Janet Lord, Aleksandra Posarac, Marco Nicoli, Karen Peffley, Charlotte McClain-Nhlapo and Mary Keogh, May 2010 1002 Toolkit on Tackling Error, Fraud and Corruption in Social Protection Programs by Christian van Stolk and Emil D. Tesliuc, March 2010 (online only) 1001 Labor Market Policy Research for Developing Countries: Recent Examples from the Literature - What do We Know and What should We Know? by Maria Laura Sanchez Puerta, January 2010 (online only) 0931 The Korean Case Study: Past Experience and New Trends in Training Policies by Young-Sun Ra and Kyung Woo Shim, December 2009 (online only) 0930 Migration Pressures and Immigration Policies: New Evidence on the Selection of Migrants by Johanna Avato, December 2009 (online only) 0929 Ex-Ante Methods to Assess the Impact of Social Insurance Policies on Labor Supply with an Application to Brazil by David A. Robalino, Eduardo Zylberstajn, Helio Zylberstajn and Luis Eduardo Afonso, December 2009 (online only) 0928 Rethinking Survivor Benefits by Estelle James, December 2009 (online only) 0927 How Much Do Latin American Pension Programs Promise to Pay Back? by Alvaro Forteza and Guzmán Ourens, December 2009 (online only) 0926 Work Histories and Pension Entitlements in Argentina, Chile and Uruguay by Alvaro Forteza, Ignacio Apella, Eduardo Fajnzylber, Carlos Grushka, Ianina Rossi and Graciela Sanroman, December 2009 (online only) 0925 Indexing Pensions by John Piggott and Renuka Sane, December 2009 (online only) 0924 Towards Comprehensive Training by Jean Fares and Olga Susana Puerto, November 2009 0923 Pre-Employment Skills Development Strategies in the OECD by Yoo Jeung Joy Nam, November 2009 0922 A Review of National Training Funds by Richard Johanson, November 2009 0921 Pre-Employment Vocational Education and Training in Korea by ChangKyun Chae and Jaeho Chung, November 2009 0920 Labor Laws in Eastern European and Central Asian Countries: Minimum Norms and Practices by Arvo Kuddo, November 2009 (online only) 0919 Openness and Technological Innovation in East Asia: Have They Increased the Demand for Skills? by Rita K. Almeida, October 2009 (online only) 0918 Employment Services and Active Labor Market Programs in Eastern European and Central Asian Countries by Arvo Kuddo, October 2009 (online only) 0917 Productivity Increases in SMEs: With Special Emphasis on In-Service Training of Workers in Korea by Kye Woo Lee, October 2009 (online only) 0916 Firing Cost and Firm Size: A Study of Sri Lanka's Severance Pay System by Babatunde Abidoye, Peter F. Orazem and Milan Vodopivec, September 2009 (online only) 0915 Personal Opinions about the Social Security System and Informal Employment: Evidence from Bulgaria by Valeria Perotti and Maria Laura Sánchez Puerta, September 2009 0914 Building a Targeting System for Bangladesh based on Proxy Means Testing by Iffath A. Sharif, August 2009 (online only) 0913 Savings for Unemployment in Good or Bad Times: Options for Developing Countries by David Robalino, Milan Vodopivec and András Bodor, August 2009 (online only) 0912 Social Protection for Migrants from the Pacific Islands in Australia and New Zealand by Geoff Woolford, May 2009 (online only) 0911 Human Trafficking, Modern Day Slavery, and Economic Exploitation by Johannes Koettl, May 2009 0910 Unemployment Insurance Savings Accounts in Latin America: Overview and Assessment by Ana M. Ferrer and W. Craig Riddell, June 2009 (online only) 0909 Definitions, Good Practices, and Global Estimates on the Status of Social Protection for International Migrants by Johanna Avato, Johannes Koettl, and Rachel Sabates-Wheeler, May 2009 (online only) 0908 Regional Overview of Social Protection for Non-Citizens in the Southern African Development Community (SADC) by Marius Olivier, May 2009 (online only) 0907 Introducing Unemployment Insurance to Developing Countries by Milan Vodopivec, May 2009 (online only) 0906 Social Protection for Refugees and Asylum Seekers in the Southern Africa Development Community (SADC) by Mpho Makhema, April 2009 (online only) 0905 How to Make Public Works Work: A Review of the Experiences by Carlo del Ninno, Kalanidhi Subbarao and Annamaria Milazzo, May 2009 (online only) 0904 Slavery and Human Trafficking: International Law and the Role of the World Bank by María Fernanda Perez Solla, April 2009 (online only) 0903 Pension Systems for the Informal Sector in Asia edited by Landis MacKellar, March 2009 (online only) 0902 Structural Educational Reform: Evidence from a Teacher's Displacement Program in Armenia by Arvo Kuddo, January 2009 (online only) 0901 Non-performance of the Severance Pay Program in Slovenia by Milan Vodopivec, Lilijana Madzar, Primoz Dolenc, January 2009 (online only) To view Social Protection Discussion papers published prior to 2009, please visit www.worldbank.org/sp. Abstract The Results Readiness Review assessed progress to date on results- based management in the Social Protection & Labor (SP&L) portfolio and generated operationally relevant knowledge on how to strengthen Monitoring & Ev aluation (M&E). Specifi cally, the Review took stoc k of the status and quality of M&E in the SP&L portfolio, including both investment and polic y-based lending. The Review identifi ed trends, strengths and weaknesses, and good pr actice M&E approac hes and indicators to incorpor ate a better results focus in project design and implementation. This related Note pro vides guidance for World Bank Task Teams working in the area of Labor Markets. HUMAN DEVELOPMENT NETWORK About this series... Social Protection Discussion Papers are published to communicate the results of The World Bank's work to the development community with the least possible delay. The typescript manuscript of this paper therefore has not been prepared in accordance with the procedures appropriate to formally edited texts. The findings, interpretations, and conclusions expressed herein are those of the author(s), and do not necessarily reflect the views of the International Bank for Reconstruction and Development / The World Bank and its affiliated organizations, or those of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. For free copies of this paper, please contact the Social Protection Advisory Service, The World Bank, 1818 H Street, N.W., Room G7-703, Washington, D.C. 20433 USA. Telephone: (202) 458-5267, Fax: (202) 614-0471, E-mail: socialprotection@worldbank.org or visit the Social Protection website at www.worldbank.org/sp.