WPS7375 Policy Research Working Paper 7375 Do Performance Agreements Help Improve Service Delivery? The Experience of Brazilian States Lorena Viñuela Laura Zoratto Governance Global Practice Group July 2015 Policy Research Working Paper 7375 Abstract A growing number of states and municipalities in Brazil rely 15 fewer homicides per 100,000 inhabitants than those on results-based management, and many other local and that do not, all else equal. Similarly, states that have intro- state governments are considering adopting the practice. duced performance agreements and a bonus for teachers This paper examines the experiences of the Brazilian states and school staff have improved their Basic Education Devel- that have implemented results agreements linked to variable opment Index score for public secondary schools by 0.3 pay. The analysis compares current with pre-intervention additional points compared with the scores of states with outcomes in the education, health, and security sectors. similar characteristics. The conclusions are in line with the The changes are examined in relation to regional trends to findings of in-depth impact evaluations and case study work determine whether the improvements depart in meaning- in the education and security sectors (Bruns, Evans and ful ways from the overall trend. In addition, a truncated Luque 2011, Milagres de Assis 2012). The paper does not time-series cross-section model is used to control for sev- analyze unit or team level data, which would be necessary eral additional factors influencing service delivery outcomes. to draw more rigorous conclusions about how results- The results suggest that, at least in the short and medium based interventions affect the behavior of civil servants and term, the implementation of results agreements is associ- outcomes over time. Therefore, the results should be inter- ated with significant and positive changes in outcomes preted with caution, as some of the assumptions behind in the security and education sectors. On average, states the models cannot be examined with the available data. using team-level targets and performance-related pay have This paper is a product of the Governance Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at lvinuela@ worldbank.org and lzoratto@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Do Performance Agreements Help Improve Service Delivery? The Experience of Brazilian States1 Lorena Viñuela and Laura Zoratto JEL Classification: JEL: D02 Institutions: Design, Formation, Operations, and Impact, H11 Structure, Scope, and Performance of Government, and O54 Latin America • Caribbean. Keywords: Brazil, Performance-based management, variable pay, performance agreements, service delivery, education, health, and security. 1 The analysis builds on the World Bank’s extensive and innovative engagement on public sector reform at the state level in Brazil since 2007, notably including the work led by Deborah L. Wetzel, Roland Clarke, and Tarsila Velloso. This paper is directly linked to the various past and on-going state-level operations in Brazil. The authors especially acknowledge the contributions of Ana Mie H. Reis (LCSPS) provided valuable research assistance. William Dillinger, Philip Keefer, Barbara Nunberg, and Pedro Olinto offered important advice and insights for the research agenda as peer reviewers. I. INTRODUCTION This paper reviews performance management models implemented by the states of Minas Gerais, Pernambuco, Rio de Janeiro, and Ceará, and examines whether one of its variants— results agreements linked to variable pay—has led to meaningful improvements in education, health, and security outcomes. The analysis identifies questions for future in-depth work on the types of interventions and conditions under which results-based management (RBM) can contribute to greater performance gains. Measuring performance has always been a core and important part of the public administration literature and practice. However, in the last two decades, growing attention has been given to the design and implementation of RBM systems and considerable resources have been dedicated to collecting performance data. Some of the most commonly adopted systems include performance-related pay, total quality management, managerial flexibility, strategic planning, performance measurement, and contracting out. The movement emerged first in developed countries as a response to tighter fiscal conditions, declining public sector efficiency, rising expectations on service delivery, and new complex policy issues that require the action of multiple actors to solve them (Kettl 2000). In that context, results-oriented management was seen as a ‘new way of conducting public business’ that could address the problems created by the traditional bureaucratic model (Osborne and Gaebler 1992). Performance management models rapidly disseminated across developed and developing countries. This propagation sparked the interest of students of public management and generated a copious body of literature on the subject, particularly in the context of OECD countries. In spite of this, systematic analyses of the impact of performance management on service delivery remain limited, and the reported results mixed. The available evidence shows, nonetheless, that outcomes are often contingent on the overall environment in which performance models are implemented. For these to translate into sustained gains, they require adjustments and perfecting over time. Similarly, in Brazil, the RBM model is being used by an increasing number of states and municipalities, but the literature on the subject is thus far largely normative or descriptive. The few existing empirical pieces identify performance gains resulting from linking rewards to improvement targets in the education and security sectors (Bruns, Evans, and Luque 2011, Milagres de Assis 2012). However, others find that performance interventions may result in unintended outcomes (Odelius and Santos 2007, Rodrigues 2009, and Milagres de Assis 2012). Examining what types of reforms have been adopted at the subnational level in Brazil and how these influence service delivery outcomes is of great policy relevance. At the same time, given the within-country variation, Brazil offers an exceptional opportunity to assess comparatively the impact of these reforms and extract lessons for other countries. To address this knowledge gap, the paper focuses on the experiences of states that have adopted results-based models involving performance agreements and systematic monitoring of progress against targets. Performance 2 agreements are seen as a key mechanism through which managers and civil servants align their actions with goals; they also provide a clear platform for accountability and organizational learning over time. The present paper draws on background qualitative sectoral analysis and provides an assessment of the four selected state case studies, describing the features of the different models and interviews conducted with state civil servants. The analysis compares current with pre- intervention outcomes in the education, health, and security sectors. The changes are examined in light of regional developments to determine if the improvements depart in meaningful ways from the overall trend. Furthermore, a fixed effect model is used to control for additional factors influencing service delivery and outcomes. The results of the empirical analysis support the hypothesis that the implementation of team- level results agreements is associated with significant and positive changes in outcomes in the security and education sectors, at least in the short and medium term. Using these tools, Minas Gerais has achieved positive results in the education sector that go beyond the gains made by other states in the southeastern region. This state also has a lower homicide rate than other states with the same GDP per capita. Similarly, Pernambuco has reversed a rising homicide rate and recorded considerable gains in the education sector. In the case of Rio de Janeiro, the introduction of performance agreements and a teacher bonus coincided with the substantial improvements recorded in educational achievements. Ceará is using a results monitoring system that has translated into sizable gains in the education and health sectors. While the analysis points to a positive effect of results agreements in combination with performance-related pay, it is important to note that the adoption of these models did not take place in isolation. They required clear planning and the disaggregation of goals to the team level. Fiscal reforms and efforts to strengthen planning and budgeting functions were important preconditions for the adoption of results-based management. In the education sector, the introduction of a common metric and the development of state systems for monitoring learning outcomes was crucial for making possible the implementation of the reforms as well as motivating management change. The studied states also took other complementary steps, such as creating special careers with a differentiated pay scale within the civil service for planning and management professionals. Some states also established permanent education programs and government schools to train staff and facilitate behavioral change. Finally, it is important to underscore that this paper does not analyze unit- or team-level data, which would be necessary to draw more rigorous conclusions about how results-based interventions affect civil servants’ behavior and outcomes over time. Therefore, the results should be interpreted with caution, as some of the assumptions behind performance interventions cannot be tested with the available data. The paper offers a brief overview of the literature on the subject in section II. Section III describes the results-based models adopted in the four case studies. Section IV presents trends in education, health, and security outcomes for the four states and section V correlates the 3 implementation of results-based management with changes in those state-level outcomes. Section VI discusses additional evidence supporting the conclusion, and section VII concludes. II. IN SEARCH OF BETTER OUTCOMES This section summarizes some of the most relevant insights from the literature on results-based management. The review highlights potential benefits and unintended consequences associated with performance management to guide the empirical analysis of their effects. The Case for Performance Management Results-based management is a process that involves defining and monitoring indicators to measure and improve the performance of organizations and programs on a regular basis (Poister 2008). The targets could be set at the agency, unit, team, and individual level, and/or have various objectives. Some of the most common concentrations include operating efficiency, cost- effectiveness, productivity, quality, and citizen satisfaction. Performance management systems are typically designed to support a particular process—strategic planning, budgeting and financial management, program management and evaluation, quality/process improvements, and/or communication with stakeholders. The approach is based on the premise “that management matters to performance and effectiveness, and that performance is the ultimate goal of public management systems and actions” (Moynihan 2005: 2). It relies on the adoption of systems that provide managers and organizations with greater discretion, techniques to measure and improve performance, and market-like mechanisms to replace input-focused and hierarchical steering with results-oriented management. By focusing on outcome measures, results-based models give managers more space to adapt programs to different or changing conditions. Measuring performance is expected to impact positively the behavior of employees and managers and motivate people and organizations to improve productivity by shifting the focus from processes and activities to results (Behn 2003). Critically, these systems can contribute to better aligning organizational activities and processes with the organization’s goals and connect activities and outputs to outcomes. The information allows principals to compare achievements in programs and services against targets, support processes of internal change and adjust programs as needed. However, performance systems and interventions are embedded within an institutional and political context and should not be considered in isolation (Radin 2006). While the interventions may contribute to improving performance by affecting distinct subsets of the incentives in place, the institutional context influences the extent to which these interventions may, in fact, be able to drive performance. The combination of institutional context and interventions has come to be referred to in the performance literature as a “performance regime” (Moynihan 2008b). As explained by Talbot (2010: 81), considering performance management systems together with 4 their context is particularly useful because “what evidence does exist (...) suggests that it is the totality of a performance regime which potentially shapes or steers performance for specific services rather than the narrow purchase-provider or principal-agent assumptions often made about performance drivers.” Translating results-based management into sustained gains requires adjustments and perfecting over time. In the United States, local and state governments have had different degrees of success in implementing performance management models. Yet, those that have continued to develop and refine them have observed positive impacts on outputs and the efficiency of public spending (Moynihan 2006). In the United Kingdom, where the broadest array of results-based models has been introduced, the government recorded sizable improvements in output and outcome indicators during the first years of their implementation. However, subsequent gains have been smaller. There is also evidence that performance interventions can in some cases lead civil servants to focus excessively on the aspects of their work that are being measured (Hood and Wilson 2009). Performance Agreements as a Central Mechanism for Improving Results Performance agreements are one of the tools or modalities of results-based management. They rely on a consultative process to develop a document or work plan setting goals and parameters of quality and productivity for a given department, team, or individual. They normally detail the external factors that may influence results and assessments. The targets are derived from the overall organizational goals and the intermediary hierarchy of results. Performance agreements are a critical mechanism through which interventions drive results improvements. Agreements help ensure that managers and employees are fully aware of the targets and the intermediary steps needed to achieve them. In doing so, this process substantially reduces informational asymmetries between principals and agents. As they produce specific commitments and deploy resources, agreements also provide a way of integrating goals into a common policy vision and thus facilitate cooperation. To be successful, performance agreements need to set feasible and reasonable targets and have clear and credible consequences. The process of preparing the agreements and setting targets contributes to building intrinsic motivation and increasing staff engagement. Empirical and theoretical work in developed countries suggests that performance measurement systems are more effective in positively influencing managers’ and civil servants’ behavior if these buy into the system (Rainy 2003). In turn, the likelihood of having commitment and ownership regarding performance targets is higher if managers and employees are involved in the process of developing them and defining the related measures (Poister and Streib 1999, Moynihan and Ingraham 2003). In addition, performance agreements contribute to solving frequent problems of over-aggregation of performance data encountered in results measurement by breaking down goals and indicators. Crucially, they provide data at a level that is useful for managers and allow individuals and teams to take concrete steps towards achieving results. Performance agreements also set the 5 channel and the frequency of the reporting (Behn 2003). The data extracted from these agreements not only inform managers but are also useful for external accountability purposes. Potential Concerns around Performance Management Several concerns have been expressed about the implementation of performance management that should also be considered. Performance measurement systems can generate large amounts of descriptive information, which require time and effort to be processed and distributed. If the information does not reach managers and employees in a timely and accessible fashion, it runs the risk of being ignored (Poister 2008). Additionally, large amounts of information can lead to ‘analysis paralysis’ while too little can lead to tunnel vision. By focusing on a specific set of goals, results-based systems could neglect other objectives that are more difficult to quantify. In the same vein, rewarding specific behaviors may leave out or even discourage intrinsic motivation and have an overall negative impact on the organization’s performance. It is also possible that, over time, performance indicators could lose their capacity to capture performance and discriminate between individuals who have a high performance from those who have a low one. And, that over time performance result may tend to concentrate on targets, rather than follow a normal distribution (Dooren, Bouckaert, and Halligan 2010). A leadership and organizational culture that does not reinforce individual or team achievements can limit the use of performance information (Moynihan 2008b). Also, if internal communication and employee participation in the goal-setting stage are not sufficiently strong, the impact of information is likely to be more limited (Poister and Streib 1999). Another note of caution is that results-oriented management may be adopted because of mimicry or normative goals rather than on the basis of evidence. The diffusion of this approach, as with many other management practices, could have been the result of the assimilation of practices by organizations that carry out similar functions (Di Maggio and Powell 1983). In many cases, institutions adopt the management structures or practices of organizations that they perceive as efficient and whose performance they want to emulate, regardless of whether those techniques have an effect on performance. Likewise, targets could be used as minimum pledges of performance rather than tools for stretching and improving efficiency (James 2004). If targets are misrepresented or are frequently changed, it can undermine the credibility of the entire system. Nevertheless, adopting performance-based models can be a very effective strategy for principals to signal priorities and an entry point for broader reforms that streamline processes and structures. 6 III. SELECTED STATE MODELS: LINKING RESULTS TO PAY This section presents the experiences of the four states under study: Minas Gerais, Pernambuco, and Rio de Janeiro, where results agreements and performance-related pay were implemented, and Ceará, where some results-based tools in the state management were introduced. Throughout Brazil, several states, and more than 15 municipalities are using performance agreements and bonuses. The latest state to announce this type of reform is Bahia, which following Pernambuco’s Pacto pela Vida model, adopted variable remuneration linked to results in the security sector in 2013. Many more governments are considering their adoption (see Map 1), yet, only in a handful of them have performance agreements been linked with detailed objectives at the unit level and systematic target monitoring been implemented. In even fewer states do results-based models establish a threshold of performance for teams to be eligible to receive annual monetary rewards.2 Map 1: Rapidly Disseminating Use of Performance Management and Results Agreements in Brazil Source: Authors. 2 On the other hand, some states, such as São Paulo, give civil servants rewards that are proportional to the progress made towards a given goal. Rio Grande do Sul uses team targets and performance-related pay in the revenue generating agencies, the Secretariat of Finance, and the Attorney General’s Office. 7 Minas Gerais adopted performance agreements linked to variable pay in 2004 and universalized them to all sectors in 2008. Such agreements were implemented in Pernambuco in the security, education, and health sectors in 2007, and in the state of Rio de Janeiro in the security and education sectors in 2009 and 2010, respectively. In these cases, goal setting, performance review, and rewards are group-based (at the secretariat, school, hospital, or unit level depending on the case). A portion of the pay state employees receive is linked to the achievement of a goal set for the relevant group. The variable pay is typically substantial, being equal or higher than a month’s salary. Ceará adopted other performance-based management tools, including strategic planning and high-level monitoring of priority projects. This state is also planning to adopt a results agreement and a productivity bonus in the education sector. The adoption of performance management tools by the federal government and a number of the state and municipal governments has been part of a broader set of reforms introduced in the Brazilian public sector. Within the Federal Government, alongside with fiscal and economic reforms, the restructuring and downsizing of the public sector in the mid-1990s was complemented with several measures aimed to strengthen careers and to attract and retain a qualified labor force. An important consideration driving the introduction of performance assessment and bonuses (Gratificação de Desempenho) was to improve the remuneration of the qualified staff without having to grant linear readjustments to the entire workforce (Pires 2005). While subnational reforms have been influenced by the initiatives at the federal level, in many cases reforms have been more comprehensive and innovative at the state or local level (Abrúcio and Gaetani 2006). These results-based models have not been implemented in isolation, but rather they have been preceded and accompanied by a wide-ranging set of reforms. Fiscal reforms and efforts to strengthen planning and budgeting functions have been important preconditions for the adoption of results-based management.3 In the education sector, the introduction of a common metric— the Basic Education Development Index (Índice de Desenvolvimiento da Educação Básica or IDEB)—and state monitoring systems for learning outcomes that allow gauging relative performance of every school, municipality, and state has facilitated the measurement of results and the application of performance-related rewards. Similarly, in the health sector the introduction of a unified health system (Sistema Único de Saúde or SUS) and a set of common indicators have had a similar effect. These common metrics have not only been effective in providing a basis for comparison across states and over time, but they have also been important in motivating reform. 3 In the case of Minas Gerais, the initial round of the performance agreement had fiscal rationalization as a key objective. 8 Table 1: Timeline of Performance-based Interventions in Selected Cases   2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013                          Results  Management  Agreements  Shock  in the  Pacto pela  Security  Bahia                  Vida     Sector                                                  Priority  Management  Projects  Ceará  for Results        Monitoring                                                              Results  Agreements  Second  Management  in Revenue  Generation  for Citizens  Mobilization  Results  (Emphasis in  Management  and Some  Agreements  citizen  Minas Gerais  Shock  Sectors      in All Sectors        participation)                                                       Management  Shock / Pacto  Pela Vida   Results  Results  Agreements  Agreements  in the  in the Security  Education  Pernambuco          Sector  Sector                                                            Results  New Strategic  Results  Agreements  Plans for  Agreement  in the  Education and  Security  Education  Rio de Janeiro          Security  Sector    Sector                                                        Management  Shock  Results  Agreements  Rio Grande  in Revenue  do Sul          Mobilization                                                               Management  Shock   Results  Agreement in  Education and  São Paulo          Health                                      9 Additionally, the studied states have taken other complementary steps, such as creating special careers for planning and management professionals within the civil service with a differentiated pay scale. In some cases, these specialists have been deployed in front-line service units or at the level where performance is being measured to provide technical assistance and support to managers. A few states have also established permanent education programs and government schools to train staff and facilitate behavioral change. The states that chose to implement results-based models shared similarly challenging conditions at the outset, although their institutional capacity and resource endowments varied. Public sector performance in key areas of service delivery, such as health or education, exhibited stagnant, smaller-than-expected, and often mediocre results. Security had deteriorated in the late 1990s, and early 2000s across the country, and these states had to tackle rapidly rising levels of crime. At the same time, they faced difficult fiscal conditions and new restrictions on borrowing. In turn, the limited fiscal space for new programs or investments meant that they had to focus on spending more strategically and efficiently. Moreover, the northeastern states had to grapple with a historical legacy of high levels of poverty and limited socio-economic development. Poor public sector performance was driven, among other factors, by difficulties in attracting qualified professionals, low public salaries, and high levels of absenteeism of front-line staff. Within the civil service, there were few tools for motivating performance available to managers, as pay was based on educational attainment, training, and seniority. The politicization of appointments and deeply rooted clientelistic practices in the recruitment of civil servants, together with the presence of numerous spoilers, complicated matters. The governors implementing such programs met opposition from civil servants, unions, and legislators. While implementing results agreements and performance-related pay may be costly,4 part of the motivation behind the reform was to provide an increase in civil servants’ remuneration that distinguishes high from low performers. A complementary goal was to de-politicize and improve the transparency of promotions and appointments. On the other hand, the states have benefitted from having continuity in the political leadership and policy priorities. While reforms have been politically difficult in the short run, in most cases they have given governors and their parties greater public approval ratings and electoral gains, sometimes even helped projecting these leaders into the national sphere. Importantly, in some cases there is evidence of institutional learning and that governments are using lessons from previous cycles to improve program design and implementation. In the case of Minas Gerais, the longest running performance-based system, the iterative application of the system lead to considerable adjustments. The lessons from the implementation of the first three management cycles were reflected in the introduction of additional performance agreements that 4 The state of Minas Gerais spent an estimated BRL 720 million between 2004 and 2010, while the state of Rio de Janeiro invested BRL 140 million in 2010 and BRL 538.5 million in 2011 in teachers’ bonuses. In 2013, Rio de Janeiro budgeted BRL 590.2 million for the variable pay in education. 10 disaggregated goals at the team level. Also, to design their performance management programs, states are using and building on the lessons from other states. The experience in Minas Gerais informed reforms in Pernambuco. The state of Rio de Janeiro introduced performance agreements and a teacher bonus modeled after Pernambuco’s but has added additional requirements to mitigate free-riding and reduce absenteeism. The Rio de Janeiro model is now informing reforms in Acre, while the Pernambuco model for the security sector is also being replicated in Bahia, and will soon be adopted by Amazonas. Some of the most relevant features of the programs examined are succinctly described below. Minas Gerais The state of Minas Gerais first introduced performance agreements in 2004, as part of the Management Shock (Choque de Gestão) policy initiated a year earlier. This policy was aimed primarily at promoting fiscal balance, reducing costs, and modernizing the state government through the use of new management models. One of the new tools introduced was the use of performance agreements. A first round of performance agreements were signed by the governor and two-thirds of the state secretariats. Initially, 340 indicators derived from the strategic plan were defined to monitor and systematically assess performance across the sectors. A productivity bonus was established for secretariats that met specific goals, along with spending rationalization and revenue generation targets. In 2008, a second tier of performance agreements was added between line secretariats and the teams responsible for the implementation of programs and priority projects (projetos estruturadores). First-level agreements (between the governor and the secretaries) follow the strategic priorities set in the multiannual plan, while in the second-level agreements (between the secretaries and the implementing teams) greater discretion has been given to secretaries and managers to define targets for their teams. The productivity bonus was universalized in 2008 as well. The second-stage agreements detail and give different weights to four items: achievement of outcome targets, completion of priority projects, sectoral objectives, and rationalization of spending. A committee reviews both levels of agreements and grants a productivity prize to the teams that have met their targets. This bonus constitutes a sizable incentive, as it can represent up to a month of basic salary. The bonus acquired renewed importance after the introduction of the second-stage agreements because these allow to link better strategic goals to annual programs. The implementation of this policy was accompanied by intense training programs and efforts to attract qualified staff. The state recruited a new cadre of young private sector managers to help change the culture within the state administration. A number of information systems were deployed to support the process. In addition to adopting performance agreements and setting expenditure reduction targets, the program cut positions and reduced by a third the number of departments. The functions of planning and management were integrated into a same unit, the State Department of Planning and Management. 11 Pernambuco In 2007, a results-based management model was adopted to integrate planning, budgeting, and implementation. The ‘Pernambuco para Todos’ program put a strong emphasis on changing processes and facilitating problem-solving and coordination across the entire state government. Initially, the model was based on three main components: (i) a participatory process for defining priorities, involving consultations with civil society organizations in all regions; (ii) the adoption of high-level monitoring meetings, in which senior officials are held accountable for the achievement of targets; and (iii) the use of data in solving problems, with the continuous monitoring of project performance and outcomes. In subsequent years, additional actions were undertaken, including expanding training opportunities, deploying monitoring and evaluation specialists in front-line teams, and the introduction of group-based bonuses. Special policies or pacts were adopted in the security sector in 2007 and in the education and health sectors in 2011. These pacts provided a more focused approach to monitor goals. The first one, the Pact for Life (Pacto pela Vida), has as primary goal of reducing violent deaths in the state by mobilizing and fostering collaboration among different government institutions and civil society. With specific targets for six areas,5 data collection and analysis is used to monitor program performance. Sectoral committees (Câmaras Técnicas) have been established to coordinate the activities of 12 state secretariats, the Prosecutor’s Office, and municipalities. A bonus is paid to the staff of the secretaries that meet yearly targets. The education and health pacts were created in 2011 with the objectives of universalizing access to education and increasing the supply and quality of health services, respectively. Prior to the adoption of the education pact, the state had introduced variable pay in the sector in 2009. The system rewards schools for meeting the improvement targets in IDEB scores. Schools that achieve 50 percent or more of their target receive a proportional bonus. The bonus is at least one month’s salary, but depending on the proportion of schools that meet their targets it can be much higher. Targets are set following different rules, depending on which quartile of the performance distribution the school falls into, creating discontinuities in how ambitious targets can be (Bruns, Evans, and Luque 2011). The state had also adopted several reforms in the education sector. In 2001, a program called Democratic School (Escola Democrática) established a new modality to appoint school principals for three-year terms through an election by teachers, parents, and students. The candidates are prescreened, and only the three top-ranked ones are allowed to run for the position. Several additional actions have been taken to improve the quality of the state school system. These include a new curriculum with detailed guidelines on materials to be taught in each grade, improved provision of books and materials, and the posting of teaching assignments and class schedules to incentivize parents to monitor teacher attendance (Bruns, Evans, and Luque 2011). A new cadre of education management advisors (técnicos de gestão educacional) 5 The six areas are: social prevention of crime and violence, qualified repression of violence, institutional improvement, information and knowledge management, training and education, and democratic management. 12 was created in 2009 and assigned to every school to monitor key indicators and support the implementation of individual school improvement plans. In the health sector, performance agreements and related rewards were used for the first time in 2011–2012. In parallel, some complementary changes were introduced. A civil service career was created for public management specialists in 2008 with the goal of supporting agencies in the development of results-based practices. In 2011, an Executive Secretariat of Management Development was created within the Secretariat of Planning and Management, and a program of continued training was established. Rio de Janeiro Following a decade of acute security problems and dismal results in the education sector,6 the state of Rio de Janeiro designated public security as a priority area in 2007 and education in 2010. Interventions in the prioritized sectors were guided by new strategic plans and involved the establishment of new units to fast-track programs and monitor results, the training and recruitment of new staff, and the introduction of productivity bonuses to incentivize staff and front-line personnel.7 In the security sector, Pacifying Police Units (Unidades de Polícia Pacificadora or UPPs) and a performance management model and bonus for security forces were introduced in 2008 and 2009, respectively. The program is implemented through a partnership including local, state, and federal governments as well as different actors of civil society. Its objective is restoring law and order in communities dominated by drug trafficking gangs and increasing the footprint of the state and its proximity to citizens. By 2013, 33 UPPs had been established, covering approximately 231 communities. In the education sector, the state adopted a new plan at the beginning of 2011 with the aim of improving learning performance and establishing a meritocratic selection process for key functions—such as the regional directors—that have traditionally been captured by the state’s Assembly. The program provides bonuses to school units and Regional Boards that meet or exceed annual improvement goals. The bonus is considerable, as it can be up to three times the basic monthly salary. The goals are not assigned to each individual, but rather to the school. However, to be eligible to receive a bonus payment teachers have to fulfill certain requirements. Teachers have to cover 100 percent of the minimum curriculum, participate in all internal and external evaluations, and have at least 70 percent attendance during the school year.8 6 In 2009, the state of Rio de Janeiro had the second lowest score in the Basic Education Development Index (IDEB) for secondary students in the country. 7 The Environment Secretariat is attempting to introduce a similar model. 8 The Municipality of Rio de Janeiro is using a similar system, but with stronger sanctions against teacher absenteeism. Only teachers and school employees that have five or fewer absences in the school year are eligible to receive the bonus (Bruns, Evans, and Luque 2011). 13 In order to assess goals, Rio de Janeiro, like Minas Gerais and Pernambuco had done before, adopted an annual version of the IDEB evaluation and three other evaluations. As well, a career plan for regional directors and principals was established, and a competitive recruitment process was made mandatory to fill those positions. By the end of 2012, all 28 regional directors and 172 school principals had been hired through a competitive selection process, despite the strong opposition from state representatives. Permeating the reform agenda was a strong focus on data, management and implementation. To allow school directors to focus on education quality, young, information technology-savvy managers were assigned to each school, which improved the timeliness of school data reporting and budget management rapidly. The Secretary established a routine meeting that brought the entire state management team ―over 400 people―together every six months to review progress, celebrate successes and collectively address problems. The new model put priority on assessing the performance of, and a developing a career plan for, every superintendent and principal. Ceará Since the late 1980s, successive governments have reformed Ceará’s public administration and invested heavily in social programs. These efforts have been reflected in economic growth, balanced public finances, sustainable levels of indebtedness, and significant improvements in social welfare. Access to education and health services has improved markedly over the past two decades, and there has also been significant progress in the quality of service provision. Building on earlier reforms, Ceará has introduced numerous results-based elements in the state management in the last decade, including the adoption of a new strategic government plan in 2003. The plan sought to improve fiscal space and modernize the state’s management practices, and focused government strategic objectives under four programs: Ceará Empreendedor; Ceará Vida Melhor; Ceará Integração; and Ceará a Serviço do Cidadão. A system of detailed indicators was developed to facilitate monitoring and program evaluation and a Committee for Results-based Management and Fiscal Administration (Comitê de Gestão por Resultados e Gestão Fiscal) was established to oversee the implementation of the state’s results management model. The Committee is formed by the secretaries of Planning, Finance, Internal Audit, and Casa Civil and meets in a ‘situation room’. Since then, progress in the implementation of priority programs is regularly reported to the Committee. The introduction in 2007 of an integrated Priority Actions and Projects Monitoring system (Monitoramento de Ações e Projetos Prioritários), which allows high-level decision makers to track projects from their design to their implementation, further strengthened the capacity of the state to monitor investment programs. To date, the model has focused on the implementation of the priority investments and functions mainly as an information tool for the Governor and the Cabinet. However, there are various elements in place that can help facilitate the introduction of a more complete performance-based management model. Ceará has already adopted an effective system for setting framework goals, through a medium-term planning framework, and for transmitting these goals to the heads of the sector agencies. Holding bimonthly meetings has proven to be a useful tool for uncovering 14 problems and finding multi-sectoral responses when warranted. Lastly, Ceará is planning to introduce performance agreements and a bonus for teachers. IV. MEASURING RESULTS: HOW STATES USING PERFORMANCE MANAGEMENT FARE This section examines how the performance-based systems fare in the health, security, and education sectors in the states using results agreements and variable pay. The before-and-after outcome measures are compared to learn if there are any discernible discontinuities that coincide with the implementation of results agreements. The states’ trajectories are related to the national and regional trends, which serve as a possible counterfactual of what the likely path would have been without the intervention. In the following section, these discontinuities are further examined using a truncated time-series model to establish if those discontinuities are statistically significant. It is important to stress that the validity of the analysis is limited because the reforms studied have not been designed as randomized controlled treatments, with clear treatment and control groups. The adoption of a results agreement system is not likely to be random. Moreover, in some cases this model has been adopted in a system- or government-wide fashion. To mitigate problems of causal inference created by the lack of a control group, a regression discontinuity, and difference-in-difference designs are used to approximate, albeit imperfectly, random treatment assignment. In this section, only nationally comparable outcome indicators are used—they include child mortality, homicide rates, and IDEB scores. Some caveats should be highlighted when considering the results. Outcome indicators can be difficult to interpret. There are often time lags between changes in outputs and outcomes. Outcomes can be affected by external factors that are beyond the service providers’ control—education and health outcomes, for instance, may be influenced by the socio-economic composition of the population served and changes in economic circumstances. There are ways of controlling for these factors, but they add complexity and require additional information. Besides, responsibilities for the provision of health, education, and security services are shared across levels of government and attributing results to state interventions can in some cases be difficult. One exception is secondary education, which is the sole responsibility of state governments. State and Regional Trends Comparing a state and regional averages helps address the potential threat that events or factors external to the state (federal or regional programs, economic or political shocks, among others) are responsible for the observed changes in outcomes. In the case of Brazil, many of the social indicators have improved across the country as a result of federal programs, such as the cash transfer program Bolsa Família or the universalization of healthcare coverage, and good economic performance. In addition, it is possible to make such comparisons because the 15 examined states are similar to the other states in their respective regions in terms of population, population density, GDP per capita, and poverty levels, which are some of the key control variables used in this analysis (Table 2). However, it should be noted that Minas Gerais falls below the regional average in terms of GDP, and Rio de Janeiro has a higher-than-average population density. Table 2: Selected States and Regional Comparators, 2002–2003   State  Other States in  Difference  the Region          Pernambuco (Northeast)  Population (millions), 2002–2003  7.92  4.98  2.94  Population Density (per sq. km), 2002–2003  80.37  49.83  30.54  GDP per Capita (constant 2000), 2002–2003  3,538.35  3,062.30  476.05  Poverty, 2002–2003  60.80  59.21  1.59  Minas Gerais (Southeast)  Population (millions), 2002–2003  17.90  18.20  ‐0.28  Population Density (per sq. km), 2002–2003  30.50  181.69  ‐151.19  GDP per Capita (constant 2000), 2002–2003  5,762.00  9,041.67  ‐3279.67**  Poverty, 2002‐2003  26.08  23.20  2.88  Rio de Janeiro (Southeast)  Population (millions), 2002–2003  14.40  19.30  ‐4.95  Population Density (per sq. km), 2002–2003  328.59  82.33  246.26***  GDP per Capita (constant 2000), 2002–2003  9,357.82  7,843.06  1,514.76  Poverty, 2002‐2003  22.89  24.27  ‐1.38  Source: Authors’ calculations, based on data from IBGE. Note: *** Significant at the 0.01 level (2-tailed); ** Significant at the 0.05 level (2-tailed). Health sector. Child mortality has declined rapidly across the country, with the national average rate dropping 34 percent from 24.9 deaths per 1,000 born alive in 2002 to 16.4 in 2011. The largest reductions have been achieved in the northern states and Alagoas in the northeast, which initially had rates above 40. On the other extreme, Amapá was the state in which the reduction in child mortality was the smallest. Regarding the states of interest, both Pernambuco and Ceará are among the states that have observed the largest reductions between 2007 and 2011 in child mortality relative to their initial levels (Figure 1). These states also have lower child mortality rates than the predicted value for their GDP per capita (Figure 2). On the other hand, Minas Gerais falls on the average rate predicted for its GDP per capita while Rio is above it. Consistent with the trend for the Northeast region, Pernambuco and Ceará have rapidly and steadily reduced child mortality rates since 2002. Child mortality in Pernambuco was higher than the regional average and the national rate in 2002. By 2011, the state had surpassed the regional reduction average and was below the national average. Figure 3 shows that from 2006 to 2007 the slope abruptly changes but then reverts to a similar rate of decline as before. This change coincides with the introduction of the Pernambuco para Todos policy. Ceará displays a 16 similar change, suggesting that there may have been a common program or change affecting some states in the region. However, the change is not as noticeable for Bahia. Figure 1: Child Mortality Change 2007–2011 and Figure 2: Child Mortality and GDP per Capita, 2011 Child Mortality in 2007 10 25 AP AP DF PI PA MT 20 MA BA AM 0 PR RJ MW GO RR SP RS SO SC ESSE MG RO AM TO NR PA AC TO MT BR MS PI PB SE BA AL RN RO GO CE AC MA PE RR MG -10 CE 15 NE SE MS RJ RN PE PB PRES DF SP RSSC -20 10 AL -30 5 10 20 30 40 8 8.5 9 9.5 10 Child Mortality (per 1,000 born alive) 2007 LN GDP per Capita (constant 2000) 95% CI Fitted values 95% CI Fitted values Difference Child Mortality (per 1,000 born alive) 2007-2011 Chil Mortality (per 1,000 born alive) Source: Ministry of Health, SVS 2013. Source: IBGE, Ministry of Health, SVS 2013. Figure 3: Child Mortality in Bahia, Ceará, and Figure 4: Child Mortality in Minas Gerais and Rio de Pernambuco (per 1,000 born alive), 2002–2011 Janeiro (per 1,000 born alive), 2002–2011 40 40 PE: MG: MG: Pernambuco Chil Mortality (per 1,000 born alive) Chil Mortality (per 1,000 born alive) Performance Performance para Todos Agreements Agreements – 1 stage – 2 stage 30 30 20 20 10 10 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Year Year Bahia Ceará Minas Gerais Rio de Janeiro Northeast Pernambuco Southeast Source: Ministry of Health, SVS 2013. Source: Ministry of Health, SVS 2013. In the case of Minas Gerais, there is no clear discontinuity in its declining trend that could be linked to the 2004 introduction of performance agreements in the health sector or their refinement in 2008 (Figure 4). While the rate decreased by an average of 3.7 percent between 2007 and 2011, child mortality levels remain above the regional average, and the distance between the two has been fairly constant. Minas Gerais’ trend is similar to that of the southeast region and that of Rio de Janeiro, a state that does not use results agreements in the health sector. Security Sector. After three consecutive years of decline, the national homicide rate per 100,000 inhabitants started increasing again in 2007. Since that time, only eight states have recorded reductions (Figure 5). The largest drop was observed in Pernambuco and Rio de Janeiro, with a 17 28 and 47.5 percent reduction in their homicide rate respectively. Roraima, Sergipe follow them, and Mato Grosso do Sul. Yet, it is important to bear in mind that these states had some of the highest homicide rates in the country, and their rates remain higher than the value predicted by GDP (Figure 6). Figure 5: Homicide Rate Change, 2007–2011 Figure 6: Homicide Rate and GDP per Capita, 2011 20 80 PB AM AL RN BA GO AL 10 60 TO CE SE PA MA NE BR DF NR AP MW ES SC AC PR MT PI SO MG RO PB 40 PA PE 0 RS BA SP AM GO MS SE DF CE RN MT PR SE AP RJ ES RO MS RR TO MA AC MG -10 20 RR RS RJ PE PI SC SP -20 0 10 20 30 40 50 60 8 8.5 9 9.5 10 Homicide Rate (per 100,000 inhabitants) 2007 LN GDP per Capita (constant 2000) 95% CI Fitted values 95% CI Fitted values Difference Homicide Rate (per 100,000 inhabitants) 2007-2011 Homicide Rate (per 100,000 inhabitants) Source: Ministry of Health, SVS 2013. Source: IBGE, Ministry of Health, SVS 2013. Figure 7: Homicide Rate in Bahia, Ceará and Figure 8: Homicide Rate in Minas Gerais and Rio de Pernambuco (per 100,000 inhabitants), 2002–2011 Janeiro (per 100,000 inhabitants), 2002–2011 60 60 MG: Performance Homicide Rate (per 100,000 inhabitants) Homicide Rate (per 100,000 inhabitants) PE: PPV - Bonus MG: 2nd stage Agreements– 1st stage 50 50 RJ: PBM Agreements - Bonus 40 40 30 30 20 20 10 10 0 0 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Year Year Bahia Ceará Minas Gerais Rio de Janeiro Northeast Pernambuco Southeast Source: Ministry of Health, SVS 2013. Source: Ministry of Health, SVS 2013. When examining regional patterns, the declining trend in Pernambuco contrasts with the increasing incidence of homicide in the rest of the region, including in reformer states such as Ceará. Pernambuco had a homicide rate of 53 in 2007, but by 2011 it had dropped to 39.2. The change in trend corresponds well with the introduction of the Pact for Life policy and a performance-related reward for state employees in the security sector (Figure 8). Although the state’s homicide rate is rapidly converging to the (growing) regional average, Pernambuco’s homicide rate is still well above the national average. In the southeast, Minas Gerais’ homicide rate has remained below the regional average and below the level predicted for its GPD per capita. Currently, the state has the second-lowest rate 18 in the Southeastern region after São Paulo and the sixth lowest in the country. The state achieved considerable reductions between 2004 and 2010, although it was surpassed by São Paulo, which currently has the lowest rate in the region. It is, nonetheless, important to highlight that the introduction of performance-related pay in 2004 coincided with the change in homicide trends, but recently the trend seems to have decelerated, even recording a small increase in 2011. Between 1999 and 2004, the homicide rate had more than doubled in the state of Minas Gerais, reaching the highest point in the state’s contemporary history. During that period, the state trend was going in the opposite direction than that of the other three states, which had started to see declining homicide levels since 2002. But comparing with the regional average is difficult because Rio de Janeiro particularly high rates, driving the average up. In the case of Rio de Janeiro, homicide rates have been in decline since 2002, and the gap with the regional average is closing. The rate was cut in half in a short period. But despite the sizable gains, homicide levels remain high. Education sector. The national average IDEB score for public secondary schools has improved at an average of 4.3 percent every two years since the index was adopted in 2005. However, the improvement observed in the 2005–2013 period was more modest than that observed in primary education. Between 2007 and 2013, Goiás, Pernambuco, and Rio de Janeiro had much larger improvements in their scores than other states that had similar scores at the beginning of the period (Figure 9). Many observed no change or declining scores. Still, among others, Ceará and Minas Gerais have scores above the predicted value for their GDP, while Rio de Janeiro and most southern and northeastern states still falls far below the score predicted for its income level. Figure 9: IDEB Public Secondary School Score Figure 10: IDEB Public Secondary School Score and Change in 2007–2013 and IDEB Score in 2007 GDP per Capita, 2013 4.5 GO 1 PE RJ 4 .5 PI GO PA SE RS SP RO SP RS PE MG RJ SC 3.5 NE AM SE AP NR CE ES MW ES RO MS PR RN PB RR DF TO MG CE AC DF MA AL BA BR AC MS TO RR 0 SO PI PB AM 3 SC AP MT PR MA SE BA PA RN MT AL 2.5 -.5 2 2.5 3 3.5 4 8 8.5 9 9.5 10 10.5 IDEB Secondary School Score 2007 LN GDP per Capita (constant 2000) 95% CI Fitted values 95% CI Fitted values Difference IDEB Secondary School Score 2007-2013 IDEB Secondary School Source: INEP 2013. Source: INEP 2013. 19 Figure 11: IDEB Public Secondary School Score in Figure 12: IDEB for Public Secondary School Score Bahia, Ceará and Pernambuco (0-10), 2005-2013 4 in Minas Gerais and Rio de Janeiro (0-10), 2005-2013 4 PE: PE: Performance Bonus Schools MG: Performance MG:– RJ: Performance Bonus Agreements 2 stage PPT 1 stage IDEB Secondary School IDEB Secondary School 3.5 3.5 3 3 2.5 2.5 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 Year Year Bahia Ceará Minas Gerais Rio de Janeiro Northeast Pernambuco Southeast Source: INEP 2013. Source: INEP 2013. In the northeast, Pernambuco shows improvements in the IDEB scores since 2009. These gains differ from the flat trend for the Northeast region. Several states in the region recorded no change or even reductions in their scores, as in the case of Bahia and Ceará. In the southeast, Minas Gerais showed steady improvement in the IDEB grade for secondary education between 2005 and 2011 but retroceded in 2013. The introduction of performance agreements is associated with the implementation of school-level performance agreements. While the state remains above the regional average, the gap is rapidly closing because Rio de Janeiro and Espiritu Santo recorded large improvements. Similarly, in the case of Rio de Janeiro, there is a clear discontinuity that is concomitant to the introduction of the results agreements and bonus system for school employees in 2010. After showing no change in three rounds of IDEB, the state recorded a sizable increase of 14 percent in 2011 and 12.5 percent in 2013. As a result, Rio de Janeiro rose from the 23rd position in the national ranking of IDEB 2009 to 5th place in 2013. The state achieved one of Brazil’s most remarkable turn-arounds in the education sector. Before-and-After Comparison of Outcomes The comparison of the average measures corresponding to the periods of time before and after the introduction of performance agreements and variable pay shows that in several cases there are significant differences in outcomes after the implementation of the intervention (Table 3). The before-and-after design is a quasi-experimental model that compares two groups of observations, before and after time points. The state-years preceding the program are averaged to estimate the counterfactual and the later ones to measure results. 20 Table 3: Comparison of Means Before and After the Adoption of Performance-Related Pay    Before   After  Difference  Pernambuco  Child Mortality (per 1,000 born alive), 2007–2011  33.98  19.84   ‐14.14***  Homicide Rate (per 100,000 inhabitants), 2007–2011  52.78  45.52   ‐7.26***  IDEB Public Secondary School Score, 2007–2011  2.70  3.10  0.40        IDEB Public Secondary School Score, 2008–2013  2.70  3.23  0.53**        Minas Gerais        Child Mortality (per 1,000 born alive), 2004–2011  20.4  17.38   ‐3.03***  Homicide Rate (per 100,000 inhabitants), 2004–2011  18.6  20.68  2.08*        Child Mortality (per 1,000 born alive), 2008–2011  18.95  16.53   ‐2.43***  Homicide Rate (per 100,000 inhabitants), 2008–2011  20.68  19.45  ‐1.06  IDEB Public Secondary School Score, 2008–2011  3.45  3.63  0.18***        Rio de Janeiro        Homicide Rate (per 100,000 inhabitants), 2008–2011  49.87  34.45   ‐16.29***  IDEB Public Secondary School Score, 2010‐2013  2.80  3.20  0.60***  Source: Authors’ calculations, based on data from IBGE and MEC. Note: *** Significant at the 0.01 level (2-tailed); ** Significant at the 0.05 level (2-tailed). In the case of Pernambuco, there are substantive and statistically significant reductions in child mortality and homicide rates, and an increase in the IDEB score for public secondary schools after 2007. For Minas Gerais, there are two cut-off points, after 2004 when the first stage performance agreements were introduced and after 2007 when an additional agreement was added between Secretaries and teams. In both cases, there are statistically significant reductions in child mortality vis-à-vis the average from the previous period. For the second comparison, there is a significant difference in the secondary school outcome measure. For the state of Rio de Janeiro, there is a substantial and statistically significant reduction in the average homicide rate when comparing the period before and after the adoption of a results- based model and the introduction of a performance-related bonus for the state security forces. The negative trend in homicide rates had started prior to the adoption of the results-based approach and the UPPs. In secondary education outcomes, the IDEB results for the state show a statistically significant increase equivalent to a fifth of the prior score. This method is, however, vulnerable to threats to internal validity, in particular, the threat of external events affecting outcome scores. Given that there are, in fact, reasons to think that the after series might be different from the before series even without the treatment, the results could be spurious. To address this risk, the results are also compared with those of the other states in the respective regions. For each of the groups, the differences between changes over time 21 (statistically adjusted) are compared. The cut-off years differ from sector to sector and for different states. Table 4: Comparison of Means After Adoption of Variable Pay with Other States in the Region   Before    After      Difference  State  Other States  Difference   in the  Region  Pernambuco (Northeast)          Child Mortality (per 1,000 born alive), 2007–2011  ‐0.25  19.84  21.81  ‐1.97  Homicide Rate (per 100,000 inhabitants), 2007–2011  32.56***  45.52  31.20  14.32**  IDEB Public Secondary School Score, 2007–2013  0.05  3.10  2.88  0.22*            IDEB Public Secondary School Score, 2008–2013  0.01  3.23  2.90  0.30**            Minas Gerais (Southeast)          Child Mortality (per 1,000 born alive), 2004–2011  5.75***  17.38  13.94  3.44***  Homicide Rate (per 100,000 inhabitants), 2004–2011  ‐18.99***  20.68  36.64   ‐15.96***  IDEB Public Secondary School Score, 2004‐2013    3.56  3.30  0.26*            Child Mortality (per 1,000 born alive), 2008–2011  3.32  16.53  13.07  3.46***  Homicide Rate (per 100,000 inhabitants), 2008–2011  ‐22.67***  19.63  33.79   ‐14.17**  IDEB Public Secondary School Score, 2008–2011  0.36  3.62  3.43  0.20            Rio de Janeiro (Southeast)          Homicide Rate (per 100,000 inhabitants), 2008‐2011  17.64***  33.58  29.14  4.43  IDEB Public Secondary School Score, 2010‐2011  ‐0.59***  3.40  3.60   ‐0.20  Source: Authors’ calculations, based on data from IBGE and MEC. Note: *** Significant at the 0.01 level (2-tailed); ** Significant at the 0.05 level (2-tailed). In the case of Pernambuco, while child mortality has decreased by a sizable amount after 2007, the reduction is not significantly different from that of other states in the northeast region. In the education sector, the state has surpassed the average for the region, and it is improving at a faster rate its IDEB score. In the security sector, the change is not only statistically significant but also goes in a different direction than the regional average. During that period, the gap between the state and the rest of the region has decreased by 18.3 points is the combined result of a sizable reduction of the homicide rate in Pernambuco and its increase in the rest of the states. When comparing the changes in Minas Gerais to other states in the southeast region, it is possible to conclude that only improvements in education outcomes go beyond the general trend. In child mortality, the change is very similar, and the difference between the state and the region remains fairly constant. The homicide rate for the period after the introduction of the Management Shock policy is lower than the rest of the region, but on average other states have had larger reductions. 22 The comparison for the state of Rio de Janeiro reveals that the state’s improvement in the IDEB score is larger than the average change of other states in the region. The state is converging fast with the regional averages. Because there are only two observations after the introduction of the bonus system, it is not yet possible to determine whether there is statistical significance. On the other hand, the homicide rate for the years 2008 and 2011 has significantly decreased in the state of Rio de Janeiro, but on average the rest of the states in the region did not register a significant change. The gap between the state and the region is quickly closing, and the difference is no longer statistically significant. V. EMPIRICAL ANALYSIS Using an interrupted time-series model of the state health, security, and secondary education indicators data from 2002 to 2011, this section further analyzes if any systematic change in outcome measures can be attributed to the introduction of results agreements linked to variable pay. Fixed effects are added to control for unobserved time-invariant factors associated with particular states. Using “state-year” as the unit of analysis, the following regression model is used: Yit β0 ∑ βk xkit β1itT + β2itM + eit Where i = 1,2,….; N; refers to a cross-sectional unit; t = 1,2,….; T; refers to a time period; and k = 1,2,….; K; refers to each specific explanatory variable. Yit and Xit refer respectively to the dependent and independent variables for unit i and time t; and eit is a random error and β0 and βk refer, respectively, to the intercept and the slope parameters. The variable T is the treatment dummy in this case scored 0 for all periods in the before series and 1 for all periods after the intervention, β1 is the treatment effect, in this case the difference in intercept t, on the vertical space between the lines repressing the before and after series. The variable M is a dummy for state-years in which government-wide management reforms, known in Brazil as ‘management shocks,’ have been implemented. The disturbance term is assumed to be randomly distributed to the periods in the series. Using this technique to estimate the pay-related effects on key outcomes, child mortality, the homicide rate, and the IDEB score are regressed on a dummy variable for the state-years in which the model was in place. The panel comprises ten yearly observations for 26 states and the Federal District. Each observation represents the aggregate state measure. Partial models are used to testing the explanatory power of the treatment and control for some relevant factors (Table 5). The control variables include sector expenditure per capita (to proxy for the resources allocated to the sector), the poverty rate and GDP per capita (to proxy for the level of economic development), and GDP per square km and the population density (reflecting density effects). 23 Table 5: Multivariate Regressions Dependent variable:    IDEB Secondary School Score (0‐10)  Child Mortality  Homicide Rate  Results Agreements and Variable Pay  0.34  0.32  0.24  0.26  ‐6.51  ‐1.76  ‐2.67  ‐0.20  ‐7.47  ‐14.78  ‐10.84  ‐14.78     (0.07)***  (0.09)***  (0.07)***  (0.07)***  (1.98)***  (1.88)  (1.47)*  (1.43)  (2.38)***  (2.89)***  (2.12)***  (2.53)***  Management Shock     0.07     ‐0.08     ‐10.73     ‐6.41     8.90     5.31        (0.07)     (0.07)     (1.37)***     (1.08)***     (2.12)***     (1.93)***  Sector Expenditure per Capita        0.0004  0.0004        0.001  ‐0.0004        ‐0.002  ‐0.003           (0.00)*  (0.00)        (0.01)  (0.004)        (0.01)  (0.01)  Poverty        ‐0.02  ‐0.02        0.55  0.47        ‐0.56  ‐0.53           (0.00)***  (0.004)**       (0.06)***  (0.06)***        (0.07)***  (0.07)***  *  GDP per Capita (constant 2000)        0.0001  0.00        0.001  0.001        ‐0.003  ‐0.003           (0.00)**  (0.00)**        (0.001)**  (0.001)        (0.001)** (0.001)** *  *  GPD per Sq. Km        0.00  0.00        0.00  0.00        0.00  ‐0.002           (0.00)***  (0.00)        (0.00)  (0.00)        (0.00)  (0.01)***  LN Population        0.22  0.21        ‐1.47  ‐1.77        6.08  6.31           (0.75)***  (0.08)***        (1.43)  (1.33)        (2.08)***  (2.05)***  Population Density (per sq. km)        ‐0.004  ‐0.004        ‐0.12  ‐0.09        0.05  0.03           (0.003)  (0.00)        (0.06)*  (0.06)        (0.09)  (0.09)  Constant  3.07  3.05  1.88  1.97  21.84  23.64  11.82  18.33  28.75  27.71  23.70  ‐14.78     (0.01)***  (0.01)***  (0.69)***  (0.60)***  (0.32)***  (0.35)***  (10.47)  (9.85)*  (0.39)***  (0.45)***  (15.22)  (12.54)** *  N  243  243  243  243  270  270  270  270  270  270  270  270  N Groups  27  27  27  27  27  27  27  27  27  27  27  27  R Square  0.08  0.07  0.50  0.511  0.03  0.03  0.42  0.67  0.03  0.11  0.31  0.33  F‐Ratio  25.64  38.56  23.28  27.20  10.73  237.48  37.98  42.40  9.83  14.05  15.27  14.67  Prob > F  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  *** Significant at the 0.01 level (2-tailed); ** Significant at the 0.05 level (2-tailed). 24 The results reveal that there are statistically significant differences in education and security outcomes for the states implementing performance-based management, but not in the health sector. States that have performance-related rewards have reduced the child mortality rate by an average of an additional 6.51 deaths per 1,000 children born alive than those that do not. However, the size of the difference shrinks and the statistical significance disappears when controls are added. Instead, GPD per capita and general management reforms explain the difference. On average, states implementing performance-related pay have 14.78 fewer homicides per 100,000 inhabitants than those that are not, all else equal. Spending BRL 100 more in personnel per capita decreases the homicide rate by one decimal point. A BRL 100 increase in GDP per capita is reflected in a three decimal point reduction in the homicide rate. Similarly, all else equal, states that have introduced results agreements and a bonus for teacher and school staff have improved the IDEB score for public secondary schools by 0.26 additional points. The results should be interpreted with caution, as some of the predictions in the literature hinge on assumptions whose relevance the available data cannot examine. It is also difficult to generalize across sectors. Properly analyzing the effects may require longer time series and, therefore, it may be too early to draw definitive conclusions. VI. OTHER EMPIRICAL EVIDENCE Despite the limitations of our analysis, the conclusions are in line with the findings emerging from in-depth impact evaluations and case study work that explore the mechanisms through which performance-related pay influences the behavior of public servants and ultimately affects outputs and outcomes in a given sector. They also highlight that results are contingent on the environment in which policy interventions are implemented. Emerging Results from Impact Evaluations In the education sector, there are several impact recent evaluations that allow a better understanding of the design features and conditions under which teacher bonuses are most effective. The preliminary information available for Pernambuco supports the expectation that results agreements in combination with variable pay positively impacts learning outcomes. In the first year the variable pay system was implemented, 52 percent of schools met targets and received a bonus, while in the second year 79 percent of schools did (Bruns, Evans, and Luque 2011). Those schools that had more ambitious targets achieved the more progress, all else equal. Similarly, schools that narrowly missed receiving the bonus for the first round appear to have improved more than schools that barely achieved the bonus (Ferraz and Bruns forthcoming). In the comparable program implemented by the municipality of Rio de Janeiro, in 2010 290 of the 1,044 municipal schools qualified for the bonus on the basis of their IDEB improvements (Bruns, 25 Evans, and Luque 2011). But more than 11,000 employees did not meet the minimum criteria to be eligible for the bonus. In the security sector, there is also evidence supporting the results. Milagres de Assis (2012) analyzes the effect of Minas Gerais’ result-based management in the security sector; more precisely, on results achieved by the Military Police (Polícia Militar), the Investigative Police (Polícia Civil), and the Fire Department. Survey data and in-depth interviews with policemen, firemen, and senior managers reported by the author show that the establishment of goals and financial incentives created profound changes in these agencies. Increases in productivity were especially sizable in the Investigative Police. In the Fire Department, the new incentive structure nudged the institution to conceive new strategies, and generated important productivity gains, and improvements in service delivery outcomes. However, Milagres de Assis (2012) also warns that the strong focus on the number of police investigations in the Military Police led to a reduction in their quality. The absence of obstacles made it possible for some police officers to reclassify crimes to improve their departments’ statistics in the key indicators that were being monitored. In sum, results agreements and financial incentives are very powerful and may generate positive or negative results, depending on the way they are implemented. Perception Data In addition to having increased political support from the population, performance management appears to be associated with greater satisfaction rates in the security and education sectors. In this section, we draw on data from the AmericasBarometer and the System of Social Perception Indicators (SIPS) of the Brazilian Institute of Applied Economics Research (Instituto de Pesquisa Econômica Aplicada or IPEA). These two surveys ask questions related to citizens’ perceptions of different public policies, including the three sectors of focus in this paper, to a representative sample of households. Because the surveys are relatively new, it is not possible to compare changes over time. However, we can weigh how states using performance management get rated against other states. The level of citizens’ trust in the Military Policy, for example, is considerably higher in the states of Pernambuco and Minas Gerais, when compared with their respective regional averages (Figure 12 and Figure 13). Ceará and Bahia also have above-average ratings in this question. The difference of over ten percentage points is statistically significant in both cases. 26 Figure 13: Trust in the Military Police in Figure 14: Trust in the Military Police in Northeastern States (positive responses), 2012 Southeastern States (positive responses), 2012 60 60 50 50 40 40 percentage percentage 30 30 20 20 10 10 0 - Espírito Minas Rio de São Paulo Santo Gerais Janeiro Source: AmericasBarometer, LAPOP, 2012. Source: AmericasBarometer, LAPOP, 2012. Note: Blue line represents regional average. Likewise, over half of the interviewees reported that the quality of education has improved in the states of Ceará, Pernambuco, and Minas Gerais (Figure 15 and Figure 16). These states have averages that are above those of their respective region. However, other states that were not implementing performance management at the time that the survey was conducted, such as Bahia, Piauí, Maranhão, and Espírito Santo, also scored high marks. Figure 15: Perceptions of Quality of Education in Figure 16: Perceptions of Quality of Education in Northeastern States (percentage), 2012 Southeastern States (percentage), 2010 Sergipe Rio Grande do Norte São Paulo Piaui Pernambuco Rio de Janeiro Paraíba Maranhão Ceará Minas Gerais Bahía Alagoas Espírito Santo 0 20 40 60 80 100 percentage 0 20 40 60 80 100 percentage Improved No change Worsened Improved No change Worsened Source: System of Social Perception Indicators (SIPS), Source: System of Social Perception Indicators (SIPS), IPEA, 2012. IPEA, 2012. The state of Pernambuco has the highest percentage of respondents reporting good quality in health services. Bahia, which has used private-public partnerships in the health sector, also scores high, along with Alagoas. Nevertheless, health sector results are less differentiated in the Southeast region. 27 Figure 17: Perceptions of Quality of SUS Services in Figure 18: Perceptions of Quality of SUS Services in Northeastern States (percentage), 2012 Southeastern States (percentage), 2010 Sergipe São Paulo Rio Grande do Norte Piaui Pernambuco Rio de Janeiro Paraíba Maranhão Minas Gerais Ceará Bahía Espírito Santo Alagoas 0 20 40 60 80 100 0 20 40 60 80 100 percentage percentage Good Intermediate Poor Good Intermediate Poor Source: System of Social Perception Indicators (SIPS), Source: System of Social Perception Indicators (SIPS), IPEA, 2012. IPEA, 2012. VII. CONCLUSION AND OUTLINE OF A PROPOSED RESEARCH AGENDA The paper explores the relationship between performance management and changes in sectoral outcomes using a variety of methods. The results of the empirical analysis support the expectation that the implementation of team-level results agreements and performance-related pay is associated with significant and positive changes in outcomes in the security and education sectors, at least in the short and medium term. Minas Gerais registered positive results in the education sector, surpassing the average of the southeastern region. This state also has a lower homicide rate than other states with the same GDP per capita. Similarly, Pernambuco has reversed a rising homicide rate and recorded considerable gains in the education sector through the implementation of performance agreements. In the case of Rio de Janeiro, the introduction of performance agreements and a teacher bonus corresponded with the first positive change recorded in educational achievements since 2005. Although Ceará is not using performance-related pay yet, its monitoring system and results-based management model has translated into sizable and consistent gains in the education and health sectors. It is important to underscore that the paper does not analyze unit or team level data, which would be necessary to draw more rigorous conclusions about how results-based interventions affect the behavior of civil servants and outcomes over time. Therefore, the results should be interpreted with appropriate caution. However, from an empirical perspective, the relationship between performance management and outcomes in service delivery remain largely understudied, in particular in the context of developing countries. Having reviewed the existing evidence, this paper highlights the pressing need to expand data collection and analysis efforts at the subnational level. Many of the 28 predictions found in the literature and the assumptions that these hinge on can only be tested using more disaggregated data. In order to properly address the question of what the impact of performance agreements is, it would be important to drill down and compare units within the state and over time, ideally through a panel. Furthermore, an agenda for analytical work to more directly address the question at hand requires several efforts. First, completing and regularly updating an inventory of state and municipal results-based management reforms and practices will facilitate their monitoring. Second, changes in behaviors and outcomes should be observed over a longer period to confirm if the gains recorded in the early years of implementation are sustained in the long term. Third, there is a need to examine more carefully how interventions affect existing employees and to what extent the gains observed are the result of a personnel turnover. Anecdotally, in many of the cases, there have been important efforts to recruit new employees and more qualified applicants have been attracted by the new environment and pay incentives. Finally, future work should also seek to understand better which factors make these reforms succeed and what explains how reform champions make it to strategic decision-making positions in the first place. Likewise, more in-depth empirical work is needed to identify the sources of breakdowns in their implementation. 29 ANNEX Table A.1: Variable Definition Variable and expected relationship  Variable definition and source  Dependent variable     Child Mortality Rate (1997–2011)  Number of child deaths (younger than 1 year old) over 1,000 born alive.   Rates calculated by the Ministry of Health using indirect demographic  methods. Annual.  Source: Ministry of Health, MS/SVS ‐ Sistema de Informações sobre  Nascidos Vivos – SINASC  Homicide Rate (1990–2011)  Specific Mortality Rate: Deaths by homicide over 100,000 inhabitants.  Source: Ministry of Health /SVS ‐ Sistema de Informações sobre  Mortalidade – SIM  Secondary School Student Outcome  IDEB’s combined measure of student learning results and student flows  (2005–2013)  (grade progression, repetition, and graduation rates). Indicator ranges  between 0 and 10.  The IDEB for each grade‐subject is calculated as the product of the  standardized Prova Brasil score and the average pass rate for the cycle  evaluated (π): IDEBasj = ProvaBrasilasj *πasj where a is the subject  evaluated (Portuguese or mathematics), s is the cycle evaluated, and j is  the school.  The average pass rate in the cycle varies between 0 and 1 (1 if the pass  rate equals 100 percent). The standardized IDEB measure varies  between 0 and 10. Biannual.  Source: Ministry of Education, INEP.  http://ideb.inep.gov.br/resultado/resultado/resultado.seam?cid=21029  Independent and Control Variables    Capital Budget Execution Rate  Source: Ministério da Fazenda ‐ Secretaria do Tesouro Nacional –  DESPEENE  Budget Execution Rate  Source: Ministério da Fazenda ‐ Secretaria do Tesouro Nacional –  DESPEENE  Expenditure per Capita  Source: Ministério da Fazenda ‐ Secretaria do Tesouro Nacional –  DESPEENE  Personnel Expenditure per Capita  State Expenditure in Investment R$.  Source: Ministério da Fazenda ‐ Secretaria do Tesouro Nacional –  DESPEENE  Health Expenditure per Capita  State Expenditure in Health and Sanitation in R$.  Source: Ministério da Fazenda ‐ Secretaria do Tesouro Nacional –  DESPEENE  Education Expenditure per Capita  State Expenditure in Education and Culture in R$.  Source: Ministério da Fazenda ‐ Secretaria do Tesouro Nacional –  DESPEENE  GDP  Gross Domestic Product of Federal Units.  Source: IBGE  Surface Area (sq. km)  Source: IBGE  Population Density  Source: IBGE  Population  Source: IBGE ‐ Census    30 Table A.2: Correlation Coefficients, All States IDEB  Child  Homicide  Populatio Populatio GDP per     Poverty  Score  Mortality  Rate  n  n Density  Capita  Child Mortality  −0.66***  1            Homicide Rate  −0.11  −0.08  1          Population  0.32***  −0.29***  −0.06  1        Population Density  0.01  −0.27***  0.29***  0.28***  1      GDP per Capita (constant 2000)  0.39***  −0.64***  0.10  0.25***  0.73***  1    Poverty  −0.65***  0.86***  −0.12*  −0.30***  −0.29***  −0.69***  1  Personnel Exp. per Capita  0.30***  −0.36***  0.11*  −0.19***  0.22***  0.47***  −0.33***  *** Significant at the 0.01 level (2-tailed); ** Significant at the 0.05 level (2-tailed).     Table A.3: Difference in Means in Selected States and Regional Comparators State   Other States in the Region  Difference  Pernambuco (Northeast)  Population (millions), 2002–2003  7.92  4.98  2.94  Population Density (per sq. km), 2002–2003  80.37  49.83  30.54  GDP per Capita (constant 2000), 2002–2003  3,538.35  3,062.30  476.05  Poverty, 2002–2003  60.80  59.21  1.59  Personnel Expenditure per Capita, 2002–2003  452.20  393.83  58.38  Minas Gerais (Southeast)  Population (millions), 2002–2003  17.90  18.20  −0.28  Population Density (per sq. km), 2002–2003  30.50  181.69  −151.19  GDP per Capita (constant 2000), 2002–2003  5,762.00  9,041.67   −3279.674**  Poverty, 2002–2003  26.08  23.20  2.88  Personnel Expenditure per Capita, 2002–2003  514.65  666.85   −152.2**  Rio de Janeiro (Southeast)  Population (millions), 2002–2003  14.40  19.30  −4.95  Population Density (per sq. km), 2002–2003  328.59  82.33  246.26***  GDP per Capita (constant 2000), 2002–2003  9,357.82  7,843.06  1,514.76  Poverty, 2002–2003  22.89  24.27  −1.38  Personnel Expenditure per Capita, 2002–2003  708.15  602.35  105.80  *** Significant at the 0.01 level (2-tailed); ** Significant at the 0.05 level (2-tailed). 31 f Means in States and Regions in Years Before and After the Adoption of Performance-related Pay Before Period  After Period  Other States  Other States  State   in the  Difference  State   in the  Difference  Region  Region            orn alive), 2007–2011  33.98  34.23  ‐0.25  19.84  21.81  ‐1.97  inhabitants), 2007–2011  52.78  20.22  32.56***  45.52  31.20  14.32**  ol Score, 2007–2013  2.70  2.65  0.05  3.10  2.88  0.22*            ol Score, 2008–2013  2.70  2.69  0.01  3.23  2.90  0.30**                      orn alive), 2004–2011  20.4  14.65  5.75***  17.38  13.94  3.44***  inhabitants), 2004–2011  18.6  37.59   ‐18.99***  20.68  36.64  ‐15.96***  ol Score, 2004‐2013        3.56  3.30  0.26*              orn alive), 2008–2011  18.95  15.63  3.32  16.53  13.07  3.46***  inhabitants), 2008–2011  20.68  43.35   ‐22.67***  19.63  33.79  ‐14.17**  ol Score, 2008–2013  3.45  3.07  0.38  3.62  3.43  0.20                          inhabitants), 2008‐2011  49.87  32.23  17.64***  33.58  29.14  4.43  ol Score, 2010‐2013  2.80  3.39  ‐0.59***  3.40  3.60  ‐0.20  level (2-tailed); ** Significant at the 0.05 level (2-tailed). REFERENCES Brazil Specific Abrúcio, Fernando Luiz, and Francisco Gaetani. 2006. “Avanços e Perspectivas da Gestão Pública nos Estados: Agenda, Aprendizado e Coalizão.” Paper prepared for the Council of State Secretaries of Administration (Conselho de Secretários Estaduais de Administração - Consad). Assis, Luis Otávio Milagres. 2012. “Efeitos organizacionais da implantação de metas e remuneração variável por desempenho—o caso da segurança pública em Minas Gerais.” Dissertação de Mestrado, FGV-São Paulo. Bernardi, Mônica Moreira Esteves, and Tadeu Barreto Guimarães. 2008. “Empreendedores públicos: uma experiência de gestão estratégica de pessoas na Administração Pública do Governo do Estado de Minas Gerais.” XIII Congreso Internacional del CLAD sobre la Reforma del Estado y de la Administración Pública, Buenos Aires, Argentina, Nov. 4–7. Bresser-Pereira, Luiz Carlos. 2003. “The 1995 Public Management Reforms in Brazil: Reflections of a Reformer.” In Reinventing Leviathan: The Politics of Administrative Reform in Developing Countries, eds. Ben Ross Schneider and Blanca Heredia, pp. 89–109. Miami, FL: North-South Center Press. Bruns, Barbara, Deon Filmer, and Harry Anthony Patrinos. 2011. Making Schools Work: New Evidence on Accountability Reforms. Washington, DC: World Bank. Bruns, Barbara, David Evans, and Javier Luque. 2011. Achieving World Class Education in Brazil: The Next Agenda. Washington, DC: World Bank. Ferraz, Claudio and Barbara Bruns. Forthcoming. Incentives to Teach: The Effects of Pay-for-Performance in Education. World Bank, Washington, DC. Busjeet, Gita. 2012. “The State Results-Based Management System of Minas Gerais, Brazil.” PREM Notes No. 18, World Bank. Duarte Araujo, Maria Arlete. 2010. “Responsabilização pelo controle de resultados no Sistema Único de Saúde no Brasil.” Revista Panaméricana de Salud P;ublica 27 (3): 230–6. Gaetani, Francisco. 2007. “The Reform of the Brazilian State Apparatus: a Rhetoric Analysis.” Available at: http://www.bresserpereira.org.br/Terceiros/Autores/Gaetani,Francisco/gaetani1.pdf. Guimarães, Tadeu Barreto, and Eder Sá Alves Campos. 2008. “Gestão da estratégia no governo do estado de Minas Gerais: em busca da harmonia entre a ação e os resultados.” XIII Congreso Internacional del CLAD sobre la Reforma del Estado y de la Administración Pública, Buenos Aires, Argentina, Nov. 4–7. ———. 2009. “Programa Estado para Resultados: Balanço da implementação de um escritório de apoio à gestão estratégica e gestão por resultados em Minas Gerais.” II CONSAD Public Management Conference, Brasília. La Forgia, Gerard. 2008. “Brazil: Innovative Approaches to Extending Family Health Services.” En Breve Brief, No. 123, World Bank. La Forgia, Gerard M., and Bernard F. Couttolenc. 2008. Hospital Performance in Brazil: The Search For Excellence. Washington, DC: World Bank. Melo, F. César Silva, and Cynthia Lanna de Miranda. 2008. “Mecanismos de incentivos atrelados ao modelo de gestão por resultados em Minas Gerais.” XIII Congreso Internacional del CLAD sobre la Reforma del Estado y de la Administración Pública, Buenos Aires, Argentina, Nov. 4–7. Odelius, Catarina Cecília, and Paulo Ricardo Godoy dos Santos. 2007. “Avaliação de desempenho individual na administração pública federal: aspectos intervenientes no processo e nos resultados.” Revista Economia e Gestão 7(15): 10–30. OECD. 2010. Avaliação da Gestão de Recursos Humanos no Governo: Relatório da OECD: Brasil. Paris: OECD. Pires, Charles. 2005. “Considerações a propósito do adicional trintenário previsto no Estatuto dos Servidores Públicos Militares do Estado de Minas Gerais e suas alterações legislativas.” Cadernos da Ejef: 31. Silveira, Jaqueline Insaurriaga, and Ivan Antonio Pinheiro. 2010. “A seleção dos critérios de avaliação de desempenho de seis Estados versus o Decreto Federal: uma análise comparativa da Legislação.” Mimeo. 33 Universidade Federal do Rio Grande do Sul. Escola de Administração. Available at: http://hdl.handle.net/10183/34512. Toscano, Thiago Coelho, P. Barros de Souza, and A. Victor dos Santos Barrence. 2008. “A experiência do Project Management Offices Central do Estado de Minas Gerais: a busca da maturidade em gerenciamento de projetos governamentais.” XIII Congreso Internacional del CLAD sobre la Reforma del Estado y de la Administración Pública, Buenos Aires, Argentina, Nov. 4–7. Trosa, Sylvie. 2001. Gestão Pública por resultados: quando o Estado se compromete. Rio de Janeiro: Revan/ Brasília, DF: ENAP. Rodrigues, Welsei Alves. 2009. A Influência das Metas e Recompensas na Motivação: Um Estudo no Setor Público Mineiro. Master Thesis, Facultade de Ciências Empresariais, Universidad FUMEC, Belo Horizonte. Woortmann, Mario F. 2012. “Performance Management in the Central Governments of New Zealand and Brazil: A Comparative Study.” Dissertation, Auckland University of Technology. World Bank. 2006. “Brazil – Enhancing Performance in Brazil’s Health Sector: Lessons from Innovations in the State of São Paulo and the City of Curitiba.” Report No. 35691. General Barzelay, Michael. 1992. Breaking through Bureaucracy: A New Vision for Managing in Government. Berkeley: University of California Press. Behn, Robert D. 2003. “Why Measure Performance? Different Purposes Require Different Measures.” Public Administration Review 63 (5): 586–606. Bevan, Gwyn. 2006. “What’s Measured is What Matters: Targets and Gaming in the English Public Health Care System.” Public Administration 84 (3): 517–38. Bevan, Gwyn, and Christopher Hood. 2006. “Have Targets Improved Performance in the English NHS?” BMJ (332): 419–42. Boland, Tony, and Alan Fowler. “A Systems Perspective of Performance Management in Public Sector Organizations.” International Journal of Public Sector Management 13 (5): 417–46. Bouckaert, Geert, and John Halligan. 2007. Managing Performance: International Comparisons. New York: Routledge. Chan, Yee-Ching Lillian. 2004. “Performance Measurement and Adoption of Balanced Scorecards.” International Journal of Public Sector Management 17 (3): 204–11. De Lancer Julnes, Patria, and Marc Holzer. 2001. “Promoting the Utilization of Performance Measures in Public Organizations: An Empirical Study of Factors Affecting Adoption and Implementation.” Public Administration Review 61 (6): 693–708. DiMaggio, Paul, and Walter W. Powell. 1983. “The iron cage revisited: Collective rationality and institutional isomorphism in organizational fields.” American Sociological Review 48 (2): 147-60. Dooren, Wouter Van, Geert Bouckaert, and John Halligan. 2010. Performance management in the public sector. New York: Routledge Hasnain, Zahid, Nick Manning, and Jan Henryk Pierskalla. 2012. “Performance-related Pay in the Public Sector: A Review of Theory and Evidence.” Policy Research Working Paper No. 6043, World Bank, Washington, DC. Hood, Christopher, Ruth Dixon, and Deborah Wilson. 2009. “‘Managing by Numbers’: the Way to Make Public Services Better?” ESRC Public Services Programme, University of Oxford. Ingraham, Patricia W., Philip G. Joyce, and Amy Kneedler Donahue. 2003. Government Performance: Why Management Matters. Baltimore: Johns Hopkins University Press. James, Oliver. 2004. “The UK core executive's use of public service agreements as a tool of governance.” Public administration 82 (2): 397-419. Kamensky, John M. 2006. “Key Ingredients for Successful Performance Management.” Successful Performance Management Forum, IBM Center for the Business of Government, Washington, DC. Kettl, Donald F. 2000. The Global Public Management Revolution: A Report on the Transformation of Governance. Washington, DC: Brookings Institution Press. 34 Manning, Nick. 2010. “Improving Performance: Foundations of Systemic Performance.” Global Expert Team (GET) Brief, World Bank, Washington, DC. Melkers, Julia E. 2005. “Models of Performance-Measurement Use in Local Governments: Understanding Budgeting, Communication, and Lasting Effects.” Public Administration Review 65 (2): 180–91. Melkers, Julia E., and Katherine G. Willoughby. 2000. “Implementing PBB: Conflicting Views of Success.” Public Budgeting and Finance 20 (1): 105–20. Moynihan, Donald P. 2005. “What Do We Talk About When We Talk About Performance? Dialogue Theory and Performance Budgeting.” Journal of Public Administration Research and Theory 16 (2): 151–168. ———. 2006. “Managing for Results in State Government: Evaluating a Decade of Reform.” Public Administration Review 66 (1): 77–89. ———. 2008a. “Public Management in North America 1998–2008.” Public Management Review 10 (4): 481– 492. ———. 2008b. The Dynamics of Performance Management: Constructing Information and Reform. Washington, DC: Georgetown University Press. Moynihan, Donald P., and Patricia W. Ingraham. 2003. “Look for the Silver Lining: When Performance-Based Accountability Systems Work.” Journal of Public Administration Research and Theory 13 (4): 469–90. Moynihan, Donald P., and Sanjay K. Panday. 2010. “The Big Question for Performance Management: Why Do Managers Use Performance Information?” Journal of Public Administration Research and Theory 20 (4): 849–66. Osborne, David, and Ted Gaebler. 1992. Reinventing Government: How the Entrepreneurial Spirit Is Transforming the Public Sector. Reading, MA: Addison-Wesley. Poister, Theodore H. 2008. Measuring Performance in Public and Nonprofit Organizations. San Francisco: John Wiley & Sons. Poister, Theodore H., and Gregory Streib. 1999. “Performance Measurement in Municipal Government: Assessing the State of the Practice.” Public Administration Review 59 (4): 325–35. Radin, Beryl A. 2000. “The Government Performance and Results Act and the Tradition of Federal Management Reform: Square Pegs in Round Holes?” Journal of Public Administration Research and Theory 10 (1): 111–35. ———. 2006. Challenging the Performance Movement: Accountability Complexity and Democratic Values. Washington, DC: Georgetown University Press. Rainey, Hal G. 2003. Understanding and Managing Public Organization. San Francisco: Jossey-Bass. Schick, Allen. 2001. “Getting Performance Measures to Measure Up.” In Quicker, Better, Cheaper? Managing Performance in American Government, ed. Dall W. Forsythe, pp. 39 – 60. Ithaca, NY: Rockefeller Institute Press. Talbot, Colin. 2010. Theories of Performance: Organizational and Service Improvement in the Public Domain. New York: Oxford University Press. Talbot, Colin, Jay Wiggan, and Carole Johnson. 2005. “Exploring Performance Regimes—Comparing Wales and Westminster.” A report for the Wales Audit Office, Centre for Public Policy and Management, Manchester Business School. Van Thiel, Sandra, and Frans L. Leeuw. 2002. “The Performance Paradox in the Public Sector.” Public Performance & Management Review 25 (3): 267–81. Wholey, Joseph S., and Harry P. Hatry. 1992. “The Case for Performance Monitoring.” Public Administration Review 52 (6): 604–10. 35