WPS5918 Policy Research Working Paper 5918 Identifying and Evaluating Large Scale Policy Interventions What Questions Can We Answer? Sandesh Dhungana The World Bank Africa Region Poverty Reduction and Economic Management Unit December 2011 Policy Research Working Paper 5918 Abstract Using a data-driven empirical case study method, the policy interventions in the last decade that were seen paper evaluates the impact of one identified reform as being sustained and successful for IDA countries. program on development outcomes. The paper uses the Robustness checks are performed to show the efficacy World Bank Country Policy and Institutional Assessment of the method in particular cases. It was found that the Ratings to identify large scale structural and macro-level method attains robustness in the case of Nigeria. This paper is a product of the Poverty Reduction and Economic Management Network, Africa Region It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at sandeshd@gmail.com. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Identifying and Evaluating Large Scale Policy Interventions: What questions can we answer? Sandesh Dhungana1 JEL Classification: O50, O11, O22; Keywords: Reform evaluation, Case-studies, Synthetic Controls 1 Author is affiliated to John F. Kennedy School of Government at Harvard University Contents 1 Introduction 3 2 Identifying Successful Interventions 5 2.1 How to identify large interventions? . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 CPIA History and Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 Strengths of CPIA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3 Methodology 9 3.1 Regression Methods with Dynamic Panel Data Models . . . . . . . . . . . . . . . 9 3.2 Comparative Case Studies with Synthetic Control Methods . . . . . . . . . . . . 10 3.2.1 Motivating Model: Technical Details . . . . . . . . . . . . . . . . . . . . . 11 3.3 Feasibility of Synthentic Control Case Studies . . . . . . . . . . . . . . . . . . . . 12 3.4 Intervention Selection Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.5 Factor Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4 Nigeria’s March towards Stronger Economic Management 16 4.1 Nigeria before the Reforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.2 Nigeria’s Reforms since 2003 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.3 Nigeria’s Growth Performance post-reforms . . . . . . . . . . . . . . . . . . . . . 17 4.4 Running Synthetic Controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.5 Placebo Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 4.6 Robustness Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.7 Beyond Average Income Per Capita . . . . . . . . . . . . . . . . . . . . . . . . . . 21 5 Conclusions 22 List of Figures 25 List of Tables 26 2 Chapter 1 Introduction The current literature on empirical development economics contains a vigorous debate as regards the variety of approaches to understanding the development paradigm. Economic and regulatory policies, human and physical resources, and governance are all seen as drivers of economic development and yet there is no agreement on the exact roles each play in driving development. The fundamental puzzle retains two questions: Do the quality of one of these factors matter more for sustained economic growth and development? Is it the case that their relative importance is homogenous across countries and stages of development? One strand of development economics contends that a combination of a stable macro-economy with market and investment climate-friendly set of economic policies is the necessary and suffi- cient condition for economic growth, which then drives other concepts of development, such as living standards and health outcomes. This view is associated with the Washington Consensus, which was the dominant paradigm of the 1980s and 1990s. It is under this consensus that mul- tilateral lending agencies advised successive developing country governments on the need to �x budgets, cut inflation, and watch the economy take off with the dynamism of its private busi- nesses. The experience of various countries, especially in Latin America that followed structural reforms to heart and yet ended up with sub-par growth performance in the 1990s however was a crude blow to this approach. This prompted even the World Bank, one of the proponents of the Washington Consensus to focus beyond stable macro-economic conditions and business-friendly reforms, as it did on its report on the growth experience of the 1990s. A rival strand, which goes back to Gerschenkron’s theories on backwardness, believes in the context-speci�c and more subtle nature of development. Recent proponents contend that in- stitutions matter, and well-governed countries are likely to grow faster, be more effective in alleviating poverty, and are also likely to be more successful in creating equitable growth and redistribution. A radical strand of this approach is one taken by Rodrik, Hausmann and Velasco, as the authors deviate signi�cantly from a model of growth that speci�es a nature of homogeneity in reforms to catalyze growth. Their growth diagnostics approach attempts to identity binding constraints, which are notions of the most important areas where reforms can bring the largest and quickest bene�ts. In this framework, improving trade barriers can either strongly acceler- ate the growth process; or have no impact on growth trends at all, depending on the need of the economy at the hour. The binding constraint of an economy hence dictates how successful particular reforms are. Empirical work on development macroeconomics has not been able to converge on one strand or another. Initial theoretical and empirical work into growth accounting delivered results that both trends in total factor productivity and capital accumulation (physical and human) have sig- ni�cant roles in driving growth, while productivity trends appear to be more important. These results however move us no further in the above questions, because various economic policies, governance efficiency and institutional quality all affect trends in productivity and capital accu- mulation. Without a strong macro-economic environment, it is hard to create signi�cant private 3 investment. Rampant corruption mostly goes along with weak incentives for �rms to compete and hence to lower productivity growth. We can think of numerous other examples, but the point remains; the question of relative impact of reforms in one dimension against another is still very much left unanswered. A large part of the problem is the inability of empirical macro-research to converge on methods and techniques that can attribute causal effects of interventions on outcomes. Traditionally, OLS regression methods have been used on country cross-section data to study the links be- tween macro policy/governance interventions and outcome variables. These methods have been criticized for the lack of accuracy emanating from problems of omitted variable bias and en- dogeneity, model uncertainty and heterogeneity. Researchers have attempted to solve some of these problems by using instrumental variables approach, for example Acemoglu et al. (2001). These studies are important steps towards estimating causal inference in cross-country studies and yet problems remain. Strong external instruments are hard to �nd, and even if they exist, the effect obtained from such regressions are limited to the variation that is affected by the instrument. These problems make the instrumental variables approach limited in its ability to evaluate macro interventions. This has led some researchers to use dynamic panel data methods, which account for endogeneity issues and yet other problems remain, such as the instability of results from cross-country regressions when variables, time periods and countries in the data change. Our goal in this paper is not to attempt to resolve the two aforementioned questions in their en- tirety. Such an endeavor requires a much larger and longer research agenda. We merely attempt to identify speci�c cases of reforms in the disparate dimensions, ranging from macroeconomic and debt dynamics to governance reforms in the last decade, and attempt to assess their impact on development indicators such as economic growth. We do this by using recently developed meth- ods of inference, which help us attribute with larger certainty the impact on outcome variables to the interventions themselves. By selecting case studies where changes in one dimension were not accompanied by changes in other dimensions, the paper provides a window of understanding into whether reforms in certain dimensions affected development outcomes by themselves or not. We also perform robustness checks to see whether the results of these methods are sensitive to minor modeling changes. We believe that by building a larger catalogue of such results, we can attempt to start understanding the questions highlighted above. In order to complete this task, we need to �rst identify signi�cant large scale interventions in policy and governance dimensions. Chapter II of the paper explains our methodology on how to identify and measure large recent interventions on policy and governance dimensions. Chapter III explains the methodology we use to understand how successful these interventions have been in contributing towards development outcomes. We describe the results of the particular case studies in Chapters IV and V. 4 Chapter 2 Identifying Successful Interventions Before embarking on the identi�cation and measurement of interventions on the dimensions of economic policy, institutions and governance, we attempt to de�ne each of the terms clearly. The de�nition of institutions has always been fraught with large problems. The early de�nition by Douglas North was very broad, encompassing all "rules of the game" in a society. (North, 1991) Acemoglu characterizes the structure of property rights and the presence and perfection of markets as the crucial economic institutions in a society. (Acemoglu, Chapter 6) Unlike geography, economic and social institutions are endogenously determined by societies in the long run. Acemoglu also posits the idea that the inability to clearly compare across institutional frameworks has meant that the development literature focuses more on proximate determinants of economic growth, which include policies. Public policies, on the other hand, are rules set up, and discretionary actions taken within the institutional structure to manage society, including its economy. These include rules on �scal budgets, exchange rate management, taxation, and in numerous other arenas. They also include discretionary actions, for example ones taken during recessions to expand money supply, or actions taken during wars to ration food and so forth. It is easy to see that the structure of economic, political and social institutions affect the choice of policies. Policies are also much more volatile than institutions, and provide much larger variation in the data. Hence, empirical literature tends to focus on the impact of policy differences on economic performance. Governance is another concept that is harder to grasp at. The World Bank, in 2007, de�ned governance as "the manner in which public officials and institutions acquire and exercise the authority to shape public policy and provide goods and services." This suggests the distinction between policies and governance lies in the fact that policies are the rules and discretionary actions, while governance is the concept of how well such rules and actions are shaped and implemented. Examples of governance quality include the quality of budgetary management, accountability, corruption control among others. When a policy intervention takes place, the success of such an intervention depends not only on the choice of the policy itself, but also on whether the governance quality in the particular case is strong enough to allow the policy to work to its effect. The underlying effect of institutions always remains, as both the choice of the policy and the quality of governance is shaped by the institutional frameworks that characterize a society. We believe that there has been much work on the importance of institutions in causing long run differences in economic growth. (Acemoglu, Johnson, Robinson, 2001) Institutions can also only be studied with an emphasis on the very long run. We thus focus our efforts on the evaluation of interventions on the policy and governance dimensions, while cognizant of the fact that institutional underpinnings are always affecting any results we come up with. 5 In the �rst two sections of the paper, we use the term ’intervention’ to generalize across inter- ventions in all of the dimensions we are interested in. It is only when we get down to the speci�c case studies that we distinguish between interventions in various dimensions. 2.1 How to identify large interventions? Understanding the impact of interventions on outcomes such as levels of income or corruption is a difficult and yet important task. The �rst step in ful�lling such a task accurately is the speci�cation of variables that identify interventions on a large scale. This is no trivial task, as most measures used in the literature fail to make the distinction between intervention inputs and intervention outputs. When a government launches an anti-corruption drive, or changes its exchange rate manage- ment strategy, these are intervention inputs. These intervention inputs are then mapped onto intervention outputs, which are, for example, the levels of corruption after the launch of the anti-corruption drive. This mapping depends not only on the choice of the right input, but also on random shocks that can map bad intervention inputs into great outputs, or vice versa. This is the central reason why the measurement of intervention inputs is so critical to understand the effect of interventions on outcomes such as income levels and growth. On a separate note, the identi�cation and measurement of intervention, however, needs to also avoid looking at de jure codi�cation of laws on policy and institutional matters. A law that codi�es bribery as illegal is not enough to suggest that the government has implemented an anti- corruption program. Ratings that meet these criteria should not be measuring legal statutes that do not hold. We believe that the Country Policy and Institutional Assessment (CPIA) ratings provided by the World Bank for the IDA borrower countries meets both the criteria described earlier. Hence, we use it to identify national-level reforms and thus understand the links between large scale interventions and development outcomes. We note that this restricts our sample to International Development Association (IDA) countries only. 2.2 CPIA History and Methodology Each year, the World Bank undertakes an assessment of a country’s policies and institutions, also referred to as the Country Policy and Institutional Assessment (CPIA). The CPIA assesses the quality of a country’s present policy and institutional framework. "Quality" refers to how conducive that framework is to fostering poverty reduction, sustainable growth, and the effective use of development assistance. The Bank initiated country assessments in the late 1970s to help guide the allocation of IDA lending resources. The CPIA consists of a set of criteria representing the different policy and institutional dimensions of an effective poverty reduction and growth strategy. The criteria have evolved over time, reflecting lessons learned and mirroring the evolution of the development paradigm. In 1998, the criteria were substantially revised: coverage was expanded to include governance and social policies, the number of criteria was set at 20 (where it remained until 2004), and the ratings scale was changed from a 5 to a 6 point scale. In the context of the Board deliberations on CPIA disclosure, in early 2004, the Bank assembled an external Panel to review the ratings and methodology. This Panel concluded that the CPIA criteria focus on the right set of issues and produce robust results, but it also found unnecessary overlap in some of the criteria and outlined steps to address some methodological and process issues. This Panel broadly supported the CPIA practice of rating implemented rather than intended policy actions, and it strongly favored disclosure of the ratings for IDA-eligible countries. 6 Revising the CPIA methodology to reflect this Panel’s recommendations involved the Regions, Networks, and Central Departments. Some of the criteria were deleted and others were combined and streamlined. As a result of these revisions, the 2004 CPIA and since then each annual assessment consists of 16 criteria (see Box 1). In addition, statistical analysis at the time corroborated earlier �ndings that informed the choice of the CPIA weights: it showed that using statistically (principal components) derived weights for the CPIA would yield essentially the same results as the equal weighting. An equal weighting procedure has, moreover, the added advantage of simplicity and transparency. Accordingly, beginning with CPIA 2004, equal weights are attached to each of the four clusters, a procedure that is broadly in line with the earlier approach. The 16 CPIA criteria are grouped into four clusters: Economic Management, Structural Policies, Policies for Social Inclusion and Equity, and Public Sector Management and Institutions (Figure 2.1). Figure 2.1: CPIA Ratings 2.3 Strengths of CPIA The �rst important strength of the CPIA ratings system is the fact that it is the only rating system available that actually covers the whole gamut of important policy and governance di- mensions in similar scales that are set by one institution. This provides natural advantages for research, as we can compare the size of ratings changes in one scale to another without losing much in the way of accuracy. There are two other important strengths of the CPIA ratings system. Firstly, they are not intended to measure de jure versions of policies and institutional frameworks only. There is no inherent problem in attempting to assess the impact of de jure interventions. However, de jure interventions are often disconnected from the implementation on the ground, and hence studying their impact rarely provides a deeper understanding of how interventions can affect development outcomes. We are con�dent that the CPIA ratings do not suffer from this problem because 7 analysts at the Bank are explicitly asked to look at evidence stronger than legal codi�cation when assessing the CPIA ratings. Given that the process of rating involves constant back and forth between various groups inside the organization, we are comfortable with the assessment that the ratings measure more than de jure policy statutes. On the flip side, a ratings system that rates intervention inputs cannot, by de�nition, have the exact same mapping as intervention outputs. For example, even with a strong past system of public �nances, a random shock such as a tsunami or an earthquake can ruin public �nances. If the ratings system is based mechanistically on the annual outcome of the state of public �nances, the ratings will decrease precipitously, although the policy environment has not worsened, and �nances themselves could recover quickly. The CPIA ratings system describes itself as set up in a way to avoid this problem as well. It is much harder to assess this claim however, because analysts would �nd information that is separated from outcomes very hard to �nd. There is a safeguard even with such a problem. A 2007 report by the World Bank on the CPIA distinguishes between naïve and subtle endogeneity. If analysts look at intervention outcomes while informing themselves of the strength of policy/institutions, such endogeneity is subtle and harder to avoid and test. However, the second form of endogeneity, the case where analysts change ratings blindly in response to large and exogenous shocks, can be tested. The study �nds that weather and terms of trade shocks did not signi�cantly impact the ratings. Hence, we can at least be con�dent that naïve endogeneity is not a signi�cant problem with the CPIA ratings system. 8 Chapter 3 Methodology We now turn to the discussion of methods which can help estimate the actual impact of inter- ventions on development outcomes. The CPIA ratings can be used to empirically assess the effect of particular changes in economic policy or governance regimes on development outcomes of interest. The methodological problems are however signi�cant, as policy outcomes get determined in an environment embedded by social, political institutions and governance quality. Research can thus pursue two separate methodological avenues, which we describe and assess. 3.1 Regression Methods with Dynamic Panel Data Models The �rst method uses classical regressions with dynamic panel data models which can estimate the average effect of particular interventions on development outcomes. Given that OLS and �xed effects estimators are biased due to unobserved time-invariant country-speci�c effects, and the availability of external instruments that meet exclusion restrictions is weak, these estimators have become central to the empirical growth literature. Bond et al. (2001) use the system GMM estimators to account for the endogeneity of variables such as investment, and to remove time invariant effects. There are some critical problems associated with the use of these methods in our context. Firstly, there is no convergence in the literature on the functional form for capturing differential effects of particular policy interventions. Neither the augmented Solow/Ramsey nor the endogenous growth models that are workhorses of modern macroeconomics really have institutional struc- tures as endogenous parameters in the models. Till date, the role of institutions and even most policy interventions is mostly seen in improving allocative efficiency, and hence increasing total factor productivity, which is generally calculated as the residual from the regressions that are based on the workhorse models. This creates a problem, as empirical papers generally use re- duced form equations and lump various variables of policy and institutional quality on the right hand side of the regression equations. Bhagwati and Srinivasan (1999) argue that cross-country regressions are hence not the correct way to understand the impact of reforms (such as trade liberalization) on growth outcomes, because they are not based on any strong theoretical models, and have no description of the functional forms of the relationships between the dependent and independent variables. Using more advanced methods such as system GMM estimators cannot solve these problems. Also signi�cantly, these models use cross-country variation in growth outcomes as predicted by policy changes to estimate the average effect of a particular policy/governance change on growth outcomes. This, however, assumes that the relative importance of particular policy/governance inputs is similar across the diversity of countries in the sample group. Here, outliers become 9 problematic, as really weak (or really strong) interventions are the ones that affect estimates the most, thus biasing the average effects to outliers. Authors like Bhagwati and Srinivasan (1999) and Rodrik (2005) also argue that regression results are dependent on period, country sample and variables chosen. This suggests a lack of stability in the parameters estimated through the regressions. Levine and Renelt (1992) use extreme- bounds analysis to show that the results from cross country regressions are sensitive to modeling design and hence fragile. This problem points to a larger issue, which is related to a debate on how the growth paradigm itself is imagined. If growth occurs through a certain process, which is catalyzed by certain policy steps and institutional changes, and this process retains some structural similarity across a myriad of cross-country and across-time experiences, then a regression approach retains its viability. However, recent literature focuses on a highly context- speci�c idea of development needs. For example, Hausmann et. al. argue that the binding constraint of an economy might dictate how successful particular policy interventions are. In such a situation, improving trade barriers can either strongly accelerate the growth process; or have no impact on growth trends at all, depending on the need of the economy at the hour. A cross-country regression basically flattens out this useful variation and provides us with an average effect, which in this case, could describe something that is irrelevant to any one country’s development needs. We believe this considerable list of problems is enough for us to eschew the use of cross-country regressions in assessing the impact of large scale interventions. We thus choose not to follow the system GMM approach, while noting of its ability in avoiding endogeneity problems that OLS regressions face. 3.2 Comparative Case Studies with Synthetic Control Meth- ods Current literature on evaluation of interventions, until very recently, focused signi�cantly on micro interventions, which are amenable to the use of Randomly Controlled Trials (RCTs). This approach has yielded many signi�cant results, as well-designed and well-run experiments allow the researcher to interpret the impact of interventions with a high certainty about causality. This literature, which perhaps began with Kremer and Miguel’s pioneering work on de-worming, is now used extensively in understanding questions such as the impact of classroom cameras on teacher absenteeism, and the evaluation of free bed-net distribution programs. These develop- ments are welcome, and yet the lack of opportunity in experimentation at the macro level makes true RCTs close to impossible in the macro realm. This literature has however helped economists focus on some important methodological questions and problems that are important to causal inference, whether the subject be a micro or macro intervention. The search for methods that allow rigorous analysis on aggregate outcomes such as growth has led to considerable recent work in the comparative case-study literature. Card and Krueger (1994) use the differences-in-differences method to estimate the effect of policy changes, such as US State minimum wage laws on employment outcomes. The method of matching units on propensity scores is widely used in the micro-development literature. Bhagwati and Srinivasan (1999) point out the fact that comparative case studies can be likely avenues to further the path of empirical macro research, given the problems identi�ed with cross-country regressions. In this tradition, the synthetic control method, started by Abadie and Gardeazabal (2003) uses an empirically rigorous design to estimate the effect of aggregate interventions. Abadie and Gardeazabal (2003) and Abadie, Diamond and Heinmeuller (2007) are two empirical papers that use the synthetic control approach to understand the impact of large scale policy interventions on outcome variables. In their 2003 paper, Abadie and Gardeazabal use the ap- proach to estimate the impact of the Basque separatist movement on growth trends of the state. The Abadie et. al. paper (2007) focuses on the estimation of tobacco sales trends in California after the passage of Proposition 99, which made it harder to sell tobacco. At the heart of the synthetic control method used in these papers is the idea that the effect of a particular interven- 10 tion can be empirically assessed only by comparison with the correct counterfactual, a concept of how the unit would have behaved had the intervention not happened. In difference-in-difference studies, the researcher is allowed the discretion of choosing the comparison group that mimics the counterfactual. This leads to substantive ambiguity on whether the comparison group really is an effective counterfactual. (Duflo et al. 2003) The synthetic control method resolves this problem. The method �rst divides data into a pre and post intervention period, intervention in this case being a signi�cant policy/governance intervention that is reflected in a strong change in the CPIA ratings. The method then creates a "synthetic" unit that resembles the treated unit (in this case the country that goes through the policy/governance intervention) in the pre-intervention period. This is achieved by using a weighted average of all other countries in the sample (donor pool as in the matching literature). The weights are set, unlike in the differences-in-differences approach, not by the researcher. Instead, the weights are chosen such that the synthetic country mimics the pre-intervention income per capita path and other observable characteristics of the country that undergoes the intervention. This method also safeguards against extrapolation that plagues regression models, as long as it is not used for countries with extreme values for observable characteristics. After the synthetic unit is chosen, the method looks at the growth performance of the treated country against the performance of the synthetic country after the intervention. The gap between the two can be used to estimate the effect of particular changes in policy/governance on the evolution of growth outcomes of the particular country in question. Unlike in a panel regression framework, this is not asking the question of whether particular policy changes have, on average, positive/negative effects on growth across all countries. Hence, it avoids some of the problems of the �rst method. It also helps research lead to context-speci�c understanding of the effects of policy/governance changes on growth outcomes. The largest bene�t of the method however, is the fact that it can account for time variant country characteristics as long as there are enough pre-treatment periods and the pre-intervention match is strong. After evaluating both methods, we decide to use the synthetic control method, because of its strength in helping create a counterfactual that can more accurately mimic the treated unit, hence making causal inference more probable. Below, we motivate the method using a simple model as shown by Abadie et. al. 3.2.1 Motivating Model: Technical Details Abadie et. al. motivate the method by using a model, which we describe now. Let’s say we have n J+1 countries in our dataset, and country 1 is the one that undergoes reforms. Let Yit be the level of income per capita for a country i at time t without reforms, and Yit be the level of income per capita for country i at time t with reforms. Let’s say we have time periods t = 1, ..., T0 as the pre-reform periods, and t = T0 + 1, ..., T as the post-reform periods. For the country which n undergoes reforms, the level of income per capita can also be given by Yit = Yit + αit Dit , n where αit = Yit − Yit ; and Dit = 1when t > T0 , and 0 otherwise. For countries without reforms, n Yit = Yit Now, we want to estimate how much faster/slower the country with reforms grew after the reforms, which is given by the set α1t = α1T 0+1,...1T for cases where t > T0 . We know from n earlier that α1t = Yit − Yit We can obviously observe the post-reform trend of the country with the reforms (Yit ), hence we n need to estimate Yit , which is the income per capita trend of the country with the reforms had the reforms not happened. We use a linear factor model for this estimation. Now, let’s say that output per capita is determined by the following factor model: n Yit = αt + θt Zi + λt µi + it n Here Yit is the income per capita of a certain country i, at time t. Now, αt are the time-variant 11 but unit-invariant �xed effects that can be differenced out by taking time-dummies. Zi are the observed variables that affect income per capita, while µi are the unobserved variables that affect income per capita. it are the error terms with zero mean. The synthetic controls method creates a control unit for each country by assigning a certain weight to each country in the donor pool. There can be many such combinations of weights, which is denoted by the vector V = (w2 , w3 , ...., wJ+1 ) which includes many such synthetic controls. Abadie et. al. show that if there are weights where the weighted income per capita before the intervention matches the real income per capita, and the weighted value of the observed covariates matches the real values of the covariates before the intervention, then the size of the bias in the difference between the post-intervention incomes for the real and the synthetic unit go to zero as pre-intervention period increases. They also show that this result holds when we have a close but imperfect match. ∗ ∗ ∗ Mathematically, such weights are given by the set (w2 , w3 , ...., wJ+1 ) where J+1 ∗ J+1 ∗ J ∗ j=2 wj Yj1 = Y11 , j=2 wj YjT 0 = Y1T 0 , and j=2 wj Zj = Z1 holds. Abadie et. al. (2007) provide mathematically that in these cases, the biases created by time variant country characteristics tend towards zero, as long as the pre-intervention period is long enough, and the match between the pre-intervention trends are strong. We point the reader to Abadie et. al (2007) who prove this point mathematically in Appendix B to their paper. Now, the key is to �nd weights such that the distance between the treated country and the synthetic control unit in the pre-reform period in terms of both GDP per capita and predictors of GDP per capita is as close to zero as possible. We use the Stata program provided by Abadie et. al. to �nd the best possible pre-reform matches for the country in each case study. For the method to yield accurate results, there are certain necessary conditions, which we describe next. 3.3 Feasibility of Synthentic Control Case Studies In order for an empirically rigorous case study that uses the synthetic controls method to effec- tively isolate the impact of certain interventions, the following broad criteria have to hold. • Signi�cant Intervention - The intervention has to be signi�cant at the level of the unit of treatment, which, in this case, is the country itself. This might be an obvious point, but it is important. An anti-corruption program in a local municipality can be evaluated for its efficiency, but our goal with this research program is to identify and analyze the impact of various larger programs and policies. That can only be done if the intervention is done at the national level, or done at a local unit that makes up a large part of the national unit. • Sustained Intervention - The intervention has to be sustained through a signi�cant period of time. If the intervention is repealed after a year or two, then the possibility of it affecting variables of interest becomes lower, and the analysis cannot detect any effect. • Absence of Signi�cant Exclusive Shocks - The effect of the intervention can only be esti- mated accurately if the period of intervention was not characterized by signi�cant shocks that affected the treated unit exclusively. In this case, let’s say for example that country X underwent a signi�cant transparency reform program, while at the same time; a large civil war broke out inside the country. It is impossible to isolate the effects of the transparency program given that conflict affects production and allocation of resources. In a similar vein, global business cycle shocks are not problematic to this endeavor if they affect the treated unit and the donor pool similarly. This can be tested and the donor pool can be shrunk to meet this requirement as well. • Non-Interference Across Units - The effect of the intervention cannot have an effect on the untreated unit. In this research project, an intervention cannot have a large effect on the 12 economy of other countries that did not go through the intervention. If this does not hold, then isolating the effect of the intervention becomes much more difficult. • Signi�cant Pre-Treatment Period - The synthetic control approach uses the variation in the pre-treatment period across the donor pool to create a synthetic control unit that mimics the treated unit pre-intervention. The larger the pre-treatment period, the smaller are the possible biases accruing due to unobservable time-variant unit characteristics. • Convex Hull Criteria - For the approach to work, some weighted combination of untreated units must be close to mimicking the treated unit before the intervention. If a unit is very far from other units in terms of the observed variables of interest, then the synthetic control method cannot give accurate predictions. Given this, the countries at the extreme ends of the spectrum of observed characteristics are not ideal for this method. • Donor Pool Criteria - The factor model that is used to estimate the synthetic control is a linear model. Given that this model might not hold strongly over disparate regions with a wide variety of characteristics, the donor pool may have to be restricted to regions with some similarity in observable characteristics such that the linear factor model can be a good approximation. Also, the donor pool needs to be restricted to states that did not have very signi�cant changes in economic policy or institutional frameworks themselves. As long as these conditions hold strongly, endogeneity of the intervention is not a problem. An example of endogeneity is when the intervention happens because of the level of the variable of interest, such as when an anti-corruption drive happens because corruption levels are high. However, in such a case, by de�nition, the levels of corruption in the synthetic control unit match with the levels in the treated unit before the drive. If the level of corruption led to the drive, then it would have done so in the case of the synthetic control unit as well. If there exist countries that had similar corruption levels (and other important characteristics), and did not have a corruption drive, then this suggests the effects of endogeneity are not what caused the intervention. 3.4 Intervention Selection Methodology Given these criteria, we use the following methodological process to select interventions. First, we select cases through a simple mathematical formula. Here, we are using the CPIA ratings to denote the changes in policy inputs and governance regimes. Hence, in order to meet condition of signi�cant intervention, we select cases for whom at least one of the CPIA ratings changed by a value of more than 1 between 2004 and 2009. Given that average of the variance across all ratings is 0.45, we believe ratings changes greater than 1 are signi�cant. Secondly, in order to meet condition of sustained intervention, we restrict the pool of countries by selecting countries where ratings changes were sustained for a period of 4 years or more. Given that the cases need to meet the requirement of an absence of signi�cant exclusive shocks, we avoid looking at countries that experienced signi�cant exclusive shocks, including conflicts during the period. If an economy is signi�cantly reforming both its constitutional structure and macro-economic policy at the same time, it becomes difficult to isolate the effects of each. To be sure that we meet the convex hull criteria, we check where our country of interest ranks on various characteristics and omit countries that are on the highest and lowest 5% on various characteristics. Finally, in order to look at really signi�cant changes, we restrict the case study sample to countries that have large changes for cluster average ratings for the four CPIA clusters. Now, we are left with �ve countries, Georgia, Maldives, Nigeria, Pakistan and Chad. In Figure 3.1, we show the various ratings changes associated with each country’s experience in the past 7 years. 13 Figure 3.1: List of Countries with Signi�cant Change in CPIA Ratings Chad has been beset with a large conflict between 2004 and today, and hence its suitability for a case study of this nature is in doubt. Maldives was hit by the tsunami in 2005, and this affected its output signi�cantly, although there is contention that economic management after the disaster worsened much more than it should have been if it was merely reacting to disaster relief. Nigeria and Georgia are cases suitable for analysis. However, found from our analysis that the Georgia case study does not lead to conclusive results due to a lack of robustness of the synthetic controls method. Because of these issues, we have only included the Nigeria case in this paper. 3.5 Factor Model Once we select our cases, we use the synthetic controls procedure to estimate intervention impact. The synthetic control method we use requires us to set up a factor model that describes how the outcome variable of interest is determined. Here, we use income per capita as the outcome variable of interest. There is a large variety of models used in the literature to capture the important variables that are correlated with income per capita. Instead of wading through the complete literature, we base our factor model on Levine and Renelt’s (1992) survey on the sensitivity of cross country regressions. In their survey, the authors divide independent variables used in cross-country growth regressions into two groups, variables that are always (or close to always) included in the regressions, and variables that have been picked and chosen by some researchers while avoided by others. Since the choice of our factor model is important and yet not as critical to our results as in the case of traditional regression analysis, we only use the �rst group of variables in our factor model. Such variables identi�ed by the authors correspond very closely with the Human Capital-Augmented Solow Model. The model thus includes variables on population growth, investment, human capital accumulation, and an initial income variable. We do however add other variables to perform robustness checks later on. We understand that there are signi�cant weaknesses of this model in explaining economic growth outcomes. Most obviously, it misses variables that explain improvements in technological inno- 14 vation, including policy choices on taxation, property rights, research and development. Such weaknesses however are not critical to our research because we restrict our donor pool to coun- tries that are similar to the country in question in the variables that are deemed to be important in determining growth outcomes, and yet are not included in the regression. Firstly, the donor pool only includes countries that are either part of an easily designated peer group, or at some designated distance away from the country in question on policy/institutional framework as measured by the CPIA ratings. We do this so as to avoid interpolating from countries that have much stronger/weaker policy/institutional frameworks. Since we are not using the CPIA ratings in the regression itself, this is an important step in helping us create a strong counterfactual. We also take out countries with populations much larger or smaller than the country in question, in order to avoid interpolation from very disparate units. We believe in this approach because the variables that are included in the regression are used to create the synthetic control such that the control mimics the treated country well. By re- stricting the donor pool to countries with similar ratings on policy/institutional structure, we are controlling for the possibility of other omitted factors biasing our results. Now, assuming that we can match pre-treatment variables, including both the variables described below and the outcome variable for a signi�cant pre-treatment period, we ought to be fairly con�dent of attributing the impact on growth outcomes to the interventions themselves. In constructing the model itself, the �rst variable we use is the Private Investment to GDP ratio. Growth literature has focused heavily on the role of physical capital in adding to per capita income. The Solow model proscribes a strong role for economic convergence among nations. Although unconditional convergence is not borne out in reality, the idea that conditional convergence happens is an accepted one. We thus use initial GDP per capita (as of 1995) as a proxy for convergence effects. Recent work on human capital suggests that the role of human capital accumulation in growth is very signi�cant. Efforts to turn the abstract concept of human capital into an operational variable have been fraught with data limitations and much criticism. We opt for the best possible measure, which is the Barro-Lee indicator of the average years of secondary schooling for people above 15 years. We pick this indicator over the enrollment variable, gross % of secondary school enrollment among the population in the age range appropriate for such enrollment when it is available. This is because the growth process today stems from the abilities of the work force today, rather than from the enrollment of the labor force of tomorrow. We take into account the lines of criticism that there are nonlinearities in the returns to schooling, based on the year of schooling and the quality of the education. Even so, the lack of measures that take into account these issues forces us to use the admittedly imperfect human capital measure. Finally, we include the population growth rate as well. Now, we are ready to run empirical case studies in order to assess Nigeria and Georgia’s reform performance. One �nal caveat is in order however. Even if the results of this analysis show that a country’s performance is not much stronger than the counterfactual unit, the takeaway is not that these reforms were unwarranted, or inadvisable. These reforms might be more important towards sustaining income growth than accelerating them, in which case, we might see no effect of reforms in the short post-intervention period. The analysis does however help us assess the performance of reforms to date in a clearer manner. 15 Chapter 4 Nigeria’s March towards Stronger Economic Management 4.1 Nigeria before the Reforms Nigeria has long been famous for its large development potential, rich natural resources and an ultimately tragic set of outcomes due to mis-governance and conflicts. With a population of more than 100 million, and proven oil reserves of 36.2 billion barrels, it appears to have the strongest economic potential of all African States. However, in 2000, Nigeria had a GNP per capita of US$300, lower than the per capita income during 1960, the year of independence. In 2000, 66% of the population lived in below the dollar-a-day poverty line. It is in this context that General Abacha died in 1998 and a democratically elected government assumed power in May 1999. During the post-independence period, Nigeria had suffered through chronic economic misman- agement and corruption. The discovery of petroleum had manifested itself as a tragic resource curse by obliterating non-oil exports as well as creating rent-seeking governance behavior on the part of the political elite. In the 1990s, the World Bank lamented that stronger price of oil had continuously led to wasteful spending by the government. When the oil price fell in 1998-99, lack of oil receipts led to a threat of current account crisis. As late as 2002, the IMF was stating that 2001-02 were marked by large macro imbalances, with a large �scal and current account de�cit. Large government spending in the face of oil price booms was seen as the culprit for such �scal mismanagement. Monetary policy was tightened in 2002, which stemmed the tide of inflation. The 2002 IMF consultation document states that the government had started controlling the budget in 2002, and yet further improvement was warranted. By September 2002, foreign exchange reserves had fallen to around $7 billion. During this period, the IMF noted that progress on structural reform, including on trade barriers had not moved ahead. It is also notable the CPIA ratings for debt policy in Nigeria during the period of 1999-2001 were the weakest, with Nigeria getting a 1, the lowest rating score for external debt management. Poverty in Nigeria had always been a major concern, and the 2003/04 living standards mea- surement survey showed that 54% of the population was living below the 1 dollar a day poverty line. 16 4.2 Nigeria’s Reforms since 2003 Nigeria consistently improved its ratings for the economic management and structural policies clusters between 2003 and 2009. Nigeria’s debt policy rating rose by 2 points from 2003 to 2009. Similarly, its trade policy rating rose by 1.5 points as well. The average rating for its economic management cluster went up by more than a point, while the average rating for structural policies rose by a point as well. The social inclusion and public governance clusters, however, have not seen any signi�cant ratings changes. Nigeria came up with the NEEDS (the National Economic Empowerment and Development Strategy) in 2004. This included reforms such as passage of a �scal responsibility bill. Also introduced was the idea of a reference oil price, which was to guide �scal decisions. The IMF saw NEEDS as a major �rst step towards a credible agenda of growth and poverty reduction. In the 2005 report, the IMF and IDA staff saw implantation of the strategy in 2004-05 as "impressive". In the 2007 progress report, the IMF staff state that macroeconomic policies improved, while there was still worry regarding �scal expansion and quality of budget spending. On other fronts, the NEEDS strategy included reforms on �nancial sector, transport system, accountability and transparency. The passing of the bill on Commitment to Extractive Industries Transparency Initiative, and the start of civil society involvement through the "Publish what you Pay" initiative meant that oil revenue data would be collected and published. Further promises on health, education and poverty eradication were made as well. That said however, the CPIA ratings show that WB analysts did not believe signi�cant progress in these sectors were made, unlike in economic management and structural policies cluster. The �nancial sector reforms in 2004 also forced consolidation in the banking sector by raising the minimum capital requirements by a ten-fold. This led to signi�cant banking mergers and consolidation. The government also replaced a pay-as-you-go system with a fully funded pension system. After these reforms, domestic credit to GDP ratios have risen from around 13% in 2002- 2006 to above 25% in 2007 and later. This large rise in �nancial deepening is impressive. 4.3 Nigeria’s Growth Performance post-reforms Nigeria’s GDP per capita (PPP at constant prices) in 2004 was 1687, which rose by 15.7% to 2000.78 in �ve years. Generally, this would be attributed to the effect of oil. Worldwide oil prices, as measured weekly weighted spot price, have risen from 20 dollars per barrel in early 2003 to 71.75 dollars per barrel at the end of 2009. This has led to a sharp rise in Nigeria’s state expenditures and these expenditures have been distributed, albeit imperfectly to parts of the population, thus raising mean income. This narrative however does not tell Nigeria’s story completely. While oil prices surged between 2003 and 2009, Nigeria’s Real non-oil GDP actually rose by 13.2% in 2004, by 8.6% in 2005, and by an average of 9.3% during 2003-2007. Whether the additional spending created by revenue windfall through oil spurred the rest of the economy is difficult to answer, but the data reveals that government expenditure to GDP ratio has gone down since 2001. In 2001, the ratio was 51%. As oil prices jumped in 2003 and kept on rising for the rest of the decade, the ratio declined to around 30% between 2003 and 2007. The �scal rule has been applied well, as the excess crude account that houses excess oil revenue amounted to $18 billion in late 2008. Given that the majority of the oil revenue goes to the government and government expenditures have been tightened, it is hard to suggest that most of the economic growth indirectly from oil revenue. If oil is not the only driver of growth, we need to investigate what is. Analysts believe that much of this growth came from accumulation rather than productivity increases. Interestingly, agriculture contributed around 40-45% of the growth during the 2003-2008 period. The impact of private remittances has been signi�cant in the second half the decade. Remittances have jumped from $1.06 billion in 2003 to $9.98 billion in 2008, which amounts to around 12% of real 17 GDP in 2008. While remittances are rarely seen as manifestations of successful policy actions (often they are ways in which people in countries with large conflicts or weak policy �ght for the economic survival of their families), the impressive performance in FDI �gures tell a different story. FDI rose from $2.85 billion in 2003 to $6.42 billion in 2007. These �gures suggest that foreign investment into Nigeria had increased signi�cantly, thus strengthening the notion that the business environment, at least for foreign investors, has become friendlier. Although this growth performance looks strong at face value, we need to create a suitable comparator unit in order to really understand whether Nigeria’s growth record is extraordinary or not. The critical question at hand is; had the reforms during 2001-2004 not happened, how would Nigeria have grown till date? We attempt this exercise next. 4.4 Running Synthetic Controls The synthetic controls procedure for Nigeria creates a synthetic control where countries with a positive weight on the control are Bolivia, Tajikistan and Zambia. Table 4.1 shows the nations with their weights. Table 4.1: Speci�c Country Weights in Nigeria’s Synthetic Control Country Weight Bolivia 0.18 Tajikistan 0.055 Zambia 0.764 The method works well in creating a synthetic unit that matches Nigeria in its pre-GDP trends before between 1995 and 2004. The RMSPE is 12.53, and as Table 4.2 shows, the characteristics of the synthetic unit are very similar to Nigeria between 1995 and 2003. There is one caveat. The investment rate differs between the synthetic and the real unit. When we look qualitatively at the two countries that have signi�cant weights in the synthetic control unit, we can see that both Zambia and Bolivia are economies with strong natural resource bases. Zambia is well know for its copper exports, and the price of copper has increased in the post-intervention period, similar to the price of oil. Similarly, Bolivia is well known for natural gas, and energy prices have risen all around. As a further check, none of the two countries have had conflicts. While this period has coincided with Evo Morales’ rule in Bolivia, the economy has hummed along due to the rising price of commodities in the international market. Finally neither of the two countries are Nigeria’s largest trading partners, which helps us believe in the non-interference assumption. Table 4.2: Predictor Balance Variables Treated Synthetic Population Growth Rate 2.44 2.50 Secondary School Enrollment Ratio 26.15 36.62 Investment Rate 2.79 11.41 Initial GDP per capita 1414.9 1404.9 Figure 4.1 below shows the GDP per capita trends for Nigeria and the Synthetic control unit. The synthetic control unit trends are the counterfactuals which show us how Nigeria would have performed if not for the reforms in 2003-04. 18 Figure 4.1: Nigeria against its Synthetic Control Unit As we can see from Figure 4.1, Nigeria’s economic growth performance has been stronger than that of its synthetic control between 2003 and 2009. This analysis suggests that had the reforms not happened in Nigeria during the 2003-2004 period, the country would have a GDP per capita of 1821 instead of 2000.77 in 2009. While this does not seem to be a huge gap, it is an increase of 10%, which is a large difference given the short post-reform period. 4.5 Placebo Tests The synthetic controls method uses aggregate data, and is not making statements about the general relationship between the variables of interest among the unit (country) space. That said however, there is uncertainty about the ability of the control unit to truly mimic the coun- terfactual economic path of Nigeria. Abadie et. al. operationalize this concept of uncertainty by running placebo tests, where similar synthetic control procedures are run for all the other countries in the donor pool. This allows the researcher to see whether the effect seen in Nigeria is large relative to the effect seen for randomly chosen countries in the donor pool. (Abadie et al, 2007) We perform such tests for Nigeria. The results of the placebo tests are shown in Figure 4.2, which shows the real-synthetic gap between the Income per capita trends for Nigeria against the rest of the donor pool group. We also exclude countries with a RMSPE greater than 5 times the RMSPE of Nigeria. 19 Figure 4.2: Placebo Test Gap for Nigeria and Other Countries This shows that among countries with strong synthetic controls in the donor pool, Nigeria’s growth performance is one of the strongest. There are only two countries with a much stronger performance than Nigeria in the donor pool. This de�nitely suggests that the reforms have heralded a strong performance in average income per capita in Nigeria between 2003 and 2009. 4.6 Robustness Checks A large criticism against the cross-country regression literature is the fact that when the variables used in the regression are changed, the estimates of the effects change as well. Given that there is not one guideline on the choice of variables, we attempt to test whether the synthetic controls procedure we are using is more robust to similar criticism in the particular case of Nigeria. We run robustness checks by including �ve other variables to the factor model. We then compare our results across each permutation with our initial result. If the results of various permutations are very different from the initial model, then the model does not display strong robustness. The �ve variables we choose are the following; sectoral shares of GDP for agriculture and industry, M2 deposits to GDP as a proxy for �nancial deepening, ratio of trade to GDP as a proxy for openness, and oil rents to GDP ratio. Table 4.3 shows the permutation of our robustness tests in more detail. Table 4.3: Permutations for Robustness Check Permutations Variables Included Initial Population Growth, Initial GDP per capita, Human Capital Proxy, Investment to GDP Ratio 2 Initial Variables + Agriculture Share of GDP + Industry Share of GDP 3 Permutation 2 + Oil Rents to GDP Ratio 4 Permutation 3 + M2 to GDP Ratio 5 Permutation 4 + Trade to GDP Ratio When we run these checks for Nigeria, we �nd that in all permutations, the results are very similar to results with the initial model. Permutation 5 achieves a slightly lower RMSPE of 11.89, and yet both the weights on comparator units and post-intervention income growth trend 20 are very similar. This suggests that the results we got are not sensitive to our choice of factor models, which is a sign of robustness of the methodology in this particular case. Overall, we see that Nigeria has performed better than its best comparison unit since 2003. 4.7 Beyond Average Income Per Capita There are strong limitations with conflating development as synonymous to growth in average income per capita. Firstly, average incomes can rise alongside a large rise in inequality, where the median citizen is no better off, while the country earns plaudits for its performance. To avoid such intellectual traps, we attempt to look at changes in median household income, but are beset by data problems. The lack of Gini coefficient values also hinders us in this regard. The lack of data on inequality and median income makes it hard to analyze whether the mean income data is dominated by outliers. We also attempt to delve into other development outcomes to �nd out if the increase in average income per capita is an indicator of broad based development or a statistical anomaly. Nigeria has achieved some success in the past decade, as the adult literacy rate, and the enrollment into education ratio have both gone up signi�cantly. The country has seen a rise in its HDI value from 0.4 in 2005 to 0.419 in 2009, which is again not a large, yet a positive change. The life expectancy has crept up slowly between 2000 and 2010. This means that ultimately, the rise in economic growth may have not had impact on other important development outcomes and yet we do not have a simple way of testing such a story. Even with such caveats, we can safely say that our methods have helped us attribute the recent rise in average income growth to the reforms of 2003-04. 21 Chapter 5 Conclusions In cases where robustness is strong, the synthetic control method helps us clearly assess the im- pact of reforms, such as the strong positive impact of Nigeria’s reforms on its growth performance since 2003. In such cases, this method has a stronger claim to causal inference than traditional regression methods. The use of synthetic control methods in evaluation of large scale reforms would thus be a welcome step towards adding more rigor in the reform evaluation process. The case of Nigeria shows there are instances where strong economic performance can be at- tributed with much certainty to reforms on macroeconomic, debt and �nancial policy. While the Georgia case is not included in the paper, it illuminated the natural advantage of the framework we are using. The fact that the method itself shows its suitability for certain cases means it can be used only for cases where it shows robust results. Unlike a cross-country regression frame- work, the researcher knows when the method is working well or not, hence avoiding a false sense of accuracy. Finally, this still leaves questions about the suitability of this method to understand impact of governance reforms, as these variables are slower moving. Overall, this method shows that in the speci�c case of Nigeria, the country bene�ted largely from its reforms in the mid-2000s. Nigeria’s reforms have marshalled increased incomes and investment compared to the best comparison unit. Critically, our qualitative research also agrees with this �nding. We however, still do not know much about and how exactly the reforms worked. Learning more about the success of Nigeria’s reforms should now be done in conjunction with micro-level studies that utilize stronger identi�cation techniques and more rigorous evaluation design that are only possible with data on a micro-scale. 22 Bibliography [1] Alberto Abadie Harvard University and NBER Alexis Diamond IFC, World Bank Group Jens Hainmueller MIT, Synthetic Control Methods for Comparative Case Studies: Estimat- ing the Effect of California’s Tobacco Control Program, Journal of the American Statistical Association, August 2009 [2] Fischer, Sahay, The Transition Economies after Ten Years, NBER Working Paper, April 2000 [3] Ricardo Hausmann, Dani Rodrik, Andres Velasco, Growth Diagnostics, 2005 [4] Daron Acemoglu, Simon Johnson, David Robinson, Colonial Origins of Comparative De- velopment, American Economic Review, 2001 [5] Douglas North, Institutions, The Journal of Economic Perspectives, Vol. 5, No. 1, pp. 97- 112, Winter 1991 [6] Daron Acemoglu, Institutions as a fundamental cause of long-run growth, Handbook of Economic Growth Volume 1A, Ed. Phillipe Aghion and Steven N. Durlauf, 2005 [7] Stephen Bond, Anke Hoeffler, Jonathan Temple, GMM Estimation of Empirical Growth Models, CEPR Discussion Paper No. 3048, September 12, 2001 [8] T.N. Srinivasan, Jagdish Bhagwati, Outward-Orientation and Development: Are revision- ists right?, Yale University Economic Growth Center Discussion Paper NO. 806, September 1999 [9] Ross Levine and David Renelt, A Sensitivity Analysis of Cross-Country Growth Regressions, The American Economic Review, Vol. 82, No. 4 (Sept 1992) Pp. 942-963 [10] David Card and Alan B. Krueger, Mininum Wages and Employment: A Case study of the fast-food industry in New Jersey and Pennsylvania, American Economic Review, Vol. 84 No. 4, Sep 1994, pp. 772-793 [11] Alberto Abadie and Javier Gardeazabal, The Economic Costs of Conflict: A Case Study of the Basque Country, American Economic Review, March 2003 [12] Marianne Bertrand, Ester Duflo and Sendhil Mullainathan, How much should we trust differences-in-differences estimates?, The Quarterly Journal of Economics, February 2004 [13] Dani Rodrik, Why we learn nothing from regressing economic growth on policies, Harvard University, March 25, 2005 [14] Daniel Kauffmann, Aart Kray; Governance Indicators: Where we are and where we need to go, World Bank Policy Research Working Paper No. 4370 [15] IDA DECVP at World Bank Group, Selectivity and Performance: IDA country assessment and development effectiveness, February 2007 [16] Ludger Wo Bmann, Specifying Human Capital, Journal of Economic Surveys, Volume 17, Issue 3, pages 239–270, July 2003 [17] Country Assistance Strategy for Nigeria, World Bank, 2000 23 [18] Nigeria: 2002 Article IV Consultation - Staff Report, IMF, 2002 [19] Energy Information Administration (EIA), infoctr@eia.gov [20] Volker Treichel Ed., Putting Nigeria to Work, World Bank Group, 2010 [21] World Bank Group, Georgia: A Blueprint for Reforms, 1993 [22] Country Assistance Strategy of World Bank Group for Georgia, September 1997 [23] Country Assistance Strategy of World Bank Group for Georgia, November 2003 [24] Vladimier Papava, The Political Economy of Georgia’s Rose Revolution, East European Democratization, 2006 [25] Independent Evaluation Group at World Bank Group, Georgia: Fast Track Brief, 2008 [26] World Bank Group, Country Strategy Georgia, 2005 [27] CEE Bankwatch Network, After the Rose Revolution - Trends of economic development and its impact on Georgia, March 2007 [28] World bank Group, Doing Business Report, 2008 [29] http://unstats.un.org/unsd/methods/m49/m49regin.htm/transition 24 List of Figures 2.1 CPIA Ratings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.1 List of Countries with Signi�cant Change in CPIA Ratings . . . . . . . . . . . . 14 4.1 Nigeria against its Synthetic Control Unit . . . . . . . . . . . . . . . . . . . . . . 19 4.2 Placebo Test Gap for Nigeria and Other Countries . . . . . . . . . . . . . . . . . 20 25 List of Tables 4.1 Speci�c Country Weights in Nigeria’s Synthetic Control . . . . . . . . . . . . . . 18 4.2 Predictor Balance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.3 Permutations for Robustness Check . . . . . . . . . . . . . . . . . . . . . . . . . . 20 26