Policy Research Working Paper 10449 Weighting Justice Reform Costs and Benefits Using Machine Learning and Modern Data Science Chris Mahony Matthew Manning Gabriel Wong Social Sustainability and Inclusion Global Practice May 2023 Policy Research Working Paper 10449 Abstract Can the impact of justice processes be enhanced with the paper identifies how subgroups in the intervention dis- inclusion of a heterogeneous component into an existing play different behavioral adjustments across the reference cost-benefit analysis app that demonstrates how benefactors period, revealing the heterogeneous distribution of costs and beneficiaries are affected? Such a component requires and benefits. Finally, the paper discusses the next version (i) moving beyond the traditional cost-benefit conceptual of the cost-benefit analysis app, which incorporates an framework of utilizing averages, (ii) identification of social artificial intelligence-driven component that reintegrates group or population-specific variation, (iii) identification of individual cost-benefit analysis projects using machine how justice processes differ across groups/populations, (iv) learning and other modern data science techniques. The distribution of costs and benefits according to the identi- paper argues that the app enhances cost-benefit analysis, fied variations, and (v) utilization of empirically informed development outcomes, and policy making efficiency for statistical techniques to gain new insights from data and optimal prioritization of criminal justice resources. Further, maximize the impact for beneficiaries. This paper outlines the app advances the policy accessibility of enhanced, social a method for capturing heterogeneity. The paper tests the group-specific data, illuminating optimal policy orientation method and the cost-benefit analysis online app that was for more inclusive, just, and resilient societal outcomes—an developed using primary data collected from a develop- approach with potential across broader public policy. mental crime prevention intervention in Australia. The This paper is a product of the Social Sustainability and Inclusion Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at cmahony@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Weighting Justice Reform Costs and Benefits Using Machine Learning and Modern Data Science Chris Mahony1*, Matthew Manning2*, Gabriel Wong3* Keywords: justice reform; cost-benefit analysis; machine learning; data science; justice processes; heterogeneity * This paper exclusively reflects the views of the authors and does not necessarily represent those of the World Bank nor its Board of Directors. The authors thank Varalakshmi Vemuru, Jose Cuesta, Rohini Srihari, James Brumby, Barry Maher, Patrick Barron, and Aly Rahim for useful feedback and suggestions. All remaining errors are solely the responsibility of the authors. 1 World Bank Group; Peloria P.B.C; 2 Department of Social and Behavioral Sciences, City University of Hong Kong; 3 ANU Centre for Social Research and Methods, The Australian National University Introduction The joint United Nations-World Bank flagship report, 'Pathways for Peace' (United Nations and World Bank, 2018), cites grievances surrounding social group-specific exclusion from access to justice and security as one of four arenas of social contest that inform the risk of violence. 1 The World Bank has developed analytical tools for identifying the political economy and efficacy of governance and justice reform including the relationship between justice sector policy and programmatic approaches and the World Bank’s twin goals of reducing extreme poverty and driving shared prosperity. 2 To achieve the above goals, a comprehensive economic framework is required for tracing how public policy processes or their substantive consequences vary across individuals and communities. Currently in this space, however, cost-benefit analysis (CBA) relies on the average treatment effect (ATE), which does not adequately unpack the contextual variations that moderate economic outcomes across affected communities or sub-groups within communities. This paper considers the policy question specific to the justice sector: Can the impact of justice processes be enhanced with the inclusion of a heterogenous component into an existing CBA APP that demonstrates how benefactors and beneficiaries are affected? The development of such a framework requires: (i) moving beyond the traditional cost-benefit conceptual framework of utilizing averages (i.e. ATE); (ii) identification of variation or differences between groups or populations (e.g. individual actor attributes, political structures, institutions); (iii) identification of how justice processes differ across groups/populations; (iv) distribution of costs and benefits according to the identified variations (i.e. heterogenous impact specific to the nature of justice policy/intervention or the context in which the policy/intervention occurs); and (v) utilization of empirically informed statistical techniques to gain new insights from data and maximize impact to beneficiaries. 1 The three other ‘arenas’ of societal contestation associated with social group specific grievances surrounding exclusion include access to political power, access to land and resources, and access to public services. See United Nations & World Bank. (2018). Pathways for Peace: Inclusive Approaches to Preventing Violent Conflict. World Bank. Washington DC. 2 See analytical approaches employed by the World Bank: https://www.worldbank.org/en/topic/governance/brief/justice-rights-and-public-safety. And preceding approaches: Justice diagnostics (https://documents1.worldbank.org/curated/en/803711468338346161/pdf/437070WP0Box3210March0200701P UBLIC1.pdf), Governance diagnostics (https://web.worldbank.org/archive/website00818/WEB/DIAGNOST.HTM). 2 Individually, these points are not novel, but it is their integration that provides new empirical opportunities. The Pathways for Peace report takes important steps in identifying the delivery of education, health care, water, sanitation, justice and security as “the glue” binding state and society (Milliken and Krause, 2002) via the basic minimum citizens expect in order to accept state authority (Gilley, 2009). However, the report acknowledges the complexity of the relationship between service delivery and legitimacy (Brinkerhoff et al., 2012; Sacks and Larizza, 2012; Stel and Abate, 2014; Mcloughlin, 2015; Fisk and Cherney, 2017). State legitimacy depends on public expectations, which are informed by prior experiences (Nixon et al., 2017), geography, identity, and culture (Sturge et al., 2017). The report acknowledges the Sustainable Development Goals (SDG 16.3) to promote the rule of law and ensure equal access to justice for all (United Nations and World Bank, 2018). The report emphasizes that development approaches must identify how, not only why, justice processes and outcomes discriminate, or are perceived to discriminate, against certain groups (United Nations and World Bank, 2018). Beyond establishment of frameworks identifying how processes treat groups differently, development actors must grapple with how to most equitably deploy finite resources to change that unequal treatment. For example, decisions regarding the equitable deployment of finite resources may be influenced by static (e.g. ethnicity) and dynamic (e.g. socio-economic status) indicators that recognize group differences within a community. Incorporating this data into CBA enables identification of how justice processes treat different groups and the consequences (economic and social) for those groups. Identifying and communicating the dynamic interactions of factors associated with the individual and net costs and benefits (including their inclusivity) of policy change increases scope for informed citizen participation – enhancing social inclusion and process legitimacy. Social inclusion and process legitimacy constitute two of four (social resilience and social cohesion being the others) clusters of factors comprising the concept of ‘social sustainability’, which is associated with increased per capita income and reduced poverty (Cuesta Leiva et al (2022). 3 3 We note the absence of consensus on the concept or definition of social sustainability. However, we acknowledge the four Cuesta Leiva et al (2022)-identified clusters of identified social sustainability factors. One of those factors is ‘process legitimacy’, described as the extent to which policy and implementation processes’ engagement with local norms and values, and their reconciliation of opposing viewpoints drive perceptions of 3 To take forward a framework that also appropriately prioritizes the use of finite resources, the costs and benefits of policy options must be properly weighed within the context of the “dynamic interaction of actors, institutions, and structural factors over time” (United Nations and World Bank, 2018). Tools necessary to achieve this outcome must drive optimal justice sector decisions in a systematic, data-informed, and resource-efficient way, including drawing on World Bank analytical products. These products include legal needs surveys, Functional Reviews, Institutional and Expenditure Reviews, Justice for the Poor dispute resolution analytics, the Doing Business survey (which includes a Quality of Judicial Process Index), The Women, Business and the Law index, and the Data and Evidence for Justice Reform platform. By drawing together data from all analytical processes, scope for comparison of various justice-related datasets is enabled. 4 Most importantly, the analytical approaches assist in identifying the various elements of justice processes whose costs and benefits may then be traced. Early CBA tools, such as the Washington State Institute of Public Policy's (WSIPP) Benefit-Cost Tool (Aos and Drake, 2010) and the Manning Cost Benefit Tool (MCBT) (College of Policing, 2022), were developed to support an evidence-based policing approach. More recent developments have been undertaken by Manning and Wong, which represents an extension of the above-mentioned MCBT (now adopted by the College of Policing UK), are beginning to incopporate machine learning and artificial intelligence, including the development of CBT APP (Manning, 2018) and ‘Smart’ CBTs (Manning et al., 2018), take important steps towards robust and time-sensitive analytical methods. These tools (currently in various stages of development), which have been validated using a range of crime data, 5 provide a framework with systematic data management capacity that enables user input decisions as sufficiently fair, credible, and acceptable. A second is ‘social inclusion’ which is the extent to which access to basic services and markets as well as political, social, and cultural spaces are enabled in order for participation in society with agency and dignity (Cuesta Leiva 2022). To measure the four (social inclusion, social resilience, social cohesion, and process legitimacy) components of ‘Social Sustainability’ they cite, (1) for ‘social inclusion’, labor participation, financial inclusion, public service accessibility, and political participation; (2) for ‘social resilience’, the ability to save, income diversity, shock-responsive assets, food security and migratory mobility; (3) for ‘social cohesion’, interpersonal and institutional trust, violence levels, and democratic and civic participation, and; (4) for ‘process legitimacy’, rule of law measurements, corruption levels, separation of power, objectivity and equality (including accessibility) of law/regulation application, and physical security. 4 See analytical approaches employed by the World Bank: https://www.worldbank.org/en/topic/governance/brief/justice-rights-and-public-safety. 5 The current version of MCBT, with crime-related data as examples, can be found at: https://dmm.anu.edu.au/CD-AV/. New examples are regulalry uploaded to demonstrate the capability of the tool to be adopted in different contexts. For access, please contact matthew.manning@anu.edu.au. 4 support and economic analysis. The tools can capture heterogeneity across social groups informing justice reform investment decisions that best manage and mitigate social group specific grievances while maximizing economic consequence. The ‘Smart CBT’, once fully developed, will move beyond orthodox techniques by developing a database system that securely stores and de-identifies project data. That data is then redeployed using a range of machine learning and data science techniques. In the context of justice reform investment, the Smart CBT, coupled with the proposed addition to the tool outlined in this paper, will be enhanced with the inclusion of new features: (i) analysis identifying methods of reform; (ii) policy adjustment; and/or (iii) capacity enhancement that optimizes the economic benefits to a society as a whole. The first challenge is to what extent the enhanced tool discussed above selects the most efficient option/s available while accounting for heterogenous treatment effects? Traditionally, CBA focuses on average benefits (e.g. average treatment effects (ATE)) and average costs. These metrics are not always adequately disaggregated among groups/communities, undermining scope for evaluation of interventions’ effects on horizontal inequalities (social inclusion). Heterogeneous treatment effects and/or costs are most often present but unaccounted for, potentially diminishing legitimacy, particularly for social groups that experience disproportionaly negative effects. The retention of big data enables greater capacity to undertake more ‘comprehensive’, rather than ‘narrow’ CBA. This means that efficiency potential is hidden in both the efficacy and implementation of public and private programs, if policies can be targeted at those who, net of costs, benefit the most and/or are most vulnerable. Here, we speak in terms of disproportionate social group access to public goods such as health, electricity, justice or other services (United Nations and World Bank, 2018). However, the primary contribution enabled by this work is the potential for advanced CBA to support ex-ante policy and programmatic analysis alongside orthodox Development Impact Evaluations and/or Poverty and Social Impact Analyses. This paper explores the utility of the CBA APP for a justice sector case. However, contextual factors identified in this or other smart CBAs could be more readily drawn upon to test other interventions across the policy spectrum, including for heterogenous impact. The development of such capacity, it is hypothesized, would significantly reduce human and material cost obstacles associated with repeated individual CBAs for different justice processes or other public policy areas. Re-using the CBA APP would re-vet contextual data representativieness alongside enabling more efficient CBA across the public policy spectrum. 5 In the following sections we consider the state of the literature and practice of cost- benefit analysis before introducing a systemtic method for weighing costs and benefits, including with disaggregation to different social groups. We then introduce the case and method for testing this approach, including in identifying heterogeneous distribution of costs and benefits. We then introduce an AI-driven component that reintegrates individual CBA projects, enhancing, we argue, CBA, development outcomes, and policy making efficiency for optimal prioritization of criminal justice resources. We discuss our results and the potential for public policy application to better illuminate social group specific costs and benefits, and identify optimally inclusive and just policy settings. 1. Cost-Benefit Analysis Literature The literature outlining existing knowledge and data gaps that have historically constrained reliable cost-benefit analysis in the justice field is scattered. We recognize that the literature on justice sector-specific CBA is developing, with much scope for advance as the data illuminating human behavior becomes more representative, reliable, and accessible. Amartya Sen notes the historically narrow nature of analysis solely through market values, which omit social choices that enable freedom of valuation and increased informational inputs (Sen, 2000). Harley et. al define narrow CBAs, which they observe as more commonly employed, as comprising “direct tangible benefits and costs” resulting from the programmatic or policy change (Harley et al., 2019). 6 Harley et. al define ‘comprehensive’ CBA as accounting for direct tangible benefits and costs plus “a more extensive accounting of the indirect economic benefits to all those affected”, including “benefits to individuals, the justice system, the economy and society” (Harley et al., 2019). Diverse datasets that capture heterogeneity, when drawn upon via modern artificial intelligence techniques, empower data management systems and development practitioners to deliver more comprehensive CBAs. Development actors play an important role in identifying optimal methods for justifying and prioritizing scarce justice sector resources that benefit society at large while accounting for disproportionate costs and benefits (both objective and subjective in nature). Importantly, this role must also fully consider social implications including intangible costs and benefits. The World Bank is enhancing and upholding a high level of transparency and integrity with regards to safeguarding investment that is efficient and transparent. CBA tools 6 Their consideration of CBA is specific to Legal Aid. 6 have an important role to play in advancing the prevention of integrity crimes. Approaches, reforms, and enforcement capacity relating to improper conduct such as improprieties surrounding tendering processes (particularly public sector contracts), bribery, and financial sector integrity (Financial Action Task Force (on Money Laundering) (FATF) compliance), for example, may best be prioritized via objective observation of costs and benefits. Engaging the economics of access to justice Like policy settings relating to FATF, a CBT may also be employed to identify broader justice sector policy cost-benefit implications. Evidence regarding human decision making in developing country contexts is scarce (World Bank, 2017). The World Bank observes three principals informing understanding of human cognition in these contexts. Firstly, that most judgments and choices are made automatically, not deliberatively. Second, people often “think socially – they act and think dependent on what others around them do and think. Third, individuals share a common perspective on making sense of the world based on persons in the society around them” (World Bank, 2017). Banerjee and Duflo (2011) identify the constrained capacity of the poor to take a longer view of their lives because of the extra mental energy used just to address the most basic of needs, like shelter, within a heirachy of needs (McLeod, 2007). Looking beyond a heirachy of needs and relying on randomized control trials in 18 countries, Banerjee and Duflo note that the constrained mental energy of the poor limits their ability to even envisage a future. Failure in criminal or civil justice processes exaggerates exclusion by increasing obstacles to low-level needs like shelter. The opportunity cost for disproportionately time-poor households of pursuing justice outcomes exaggerates barriers to accessing justice processes and just outcomes (Vickery, 1977). Inability to access justice processes also exaggerates other inequalities that render the poor time-poor. The wealthy enjoy chlorinated water, regular and directly bank-deposited salaries often with automatically deducted pension and health care contributions. The poor expend time and scarce resources to organize and engage justice processes to obtain these items (Banerjee and Duflo, 2011). A CBT needs to understand and respond to the logic of how poor people cope individually and collectively in specific contexts. Existing measures of access to justice (e.g. the World Justice Project Rule of Law Index) (The World Justice Project, 2019) alongside data collected by governments and development actors, may be incorporated into a CBT. Using CBA tools that incorporate heterogenous outcomes is important in order to determine how resources directed to eliminate barriers to justice can be efficiently and effectively allocated. 7 Data in fragile and conflict-affected states is scarce. However, legal aid programs enjoy particular success in situations where political space for other forms of justice sector support is low. A review of 199 legal aid studies found only ten that considered legal aid programs in ‘not free’ countries, indicating the scarcity of data and understanding of legal aid programs under repressive regimes (Goodwin and Maru, 2017). Scarce data demands further research in relation to legal aid programs’ medium and long-term effects, their sustainability, their development effect beyond the immediate legal need (including changes in agency), and the use of accountability mechanisms (including human rights commissions, ombuds offices, and public interest litigation) (Goodwin and Maru, 2017). We know little about legal empowerment, despite the multitude of forms it might take. Scarce knowledge surrounding the environments where legal aid programs are successful (at national or community level) demands research that codes according to demography, social norms, power differentials and other political circumstances (Goodwin and Maru, 2017). It also requires consideration of programmatically specific stakeholders and risks across the many forms of legal aid programming relating to many interactions between citizens and formal justice systems, informal justice systems, and formal and informal administrative processes. The Commission on Legal Empowerment for the Poor found that “…four billion people around the world are robbed of the chance to better their lives and climb out of poverty, because they are excluded from the rule of law” (Albright and de Soto, 2008). We know that there is a link between the strength of the Rule of Law and levels of violence. The cost of violence, is rarely provided as justification for Rule of Law programming. Similarly, the specific economic benefits of empowerment programs are rarely calculated, let alone cited as justification for government interest in their support. Governments (and non-state actors), particularly those that have fought to capture the state, may perceive bottom-up Rule of Law programming as undermining their control over resource distribution, revenue collection and other rule of law-related levers of power (Hackmann, 2012). Persuading governments that bottom-up Rule of Law programming drives growth for all, elevates normative pressure while also appealing to realist self-interest. Increases in income among the bottom 40 percent alongside increases in equality of opportunity has a greater positive impact on growth than that of other income quintiles (Narayan et al., 2013). Inequality drives lower growth and diminished market efficiency, to a significant extent as a result of diminished equality of opportunity for its most productive asset – its people (Stiglitz, 2012). 8 For development, modeling informs us that extreme poverty will lower to 3 percent by 2030 rather than between 6.7 and 4.7 percent if the bottom 40 percent’s income increases 2 percent faster than a country’s mean (Lakner et al., 2014). For development practitioners and for governments, these findings are critical to our approach to policy planning, and our selection, design and implementation of interventions. We know that the ‘bottom 40’ disproportionately spend their income, and thereby increase the velocity of money in an economy causing an increased multiplier of secondary effects, including increases in government’s tax revenue (Kahn, 1931; Rivas, 2003). We know that growth, disproportionately experienced by the top quintile, conversely, also correlates to capital flight constraining growth’s multiplier effect and deepening inequality (Collier, 2006). Increased spending by the ‘bottom 40’ also increases aggregate demand in the economy for goods and services provided by the upper income quintiles. Identifying multiplier effects, because of their secondary nature, are less specific to the nature of the supported justice reform. Additional economic activity, more income, and more spending in line with a community’s size and spending patterns, is driven, in a self-reinforcing cycle by surplus income or savings, particularly at the individual and household level (Cavallari et al., 2014). Legal aid CBAs more commonly focus on tangible benefits and costs directly from legal aid provision. However, specific legal aid intervention CBAs more commonly incorporate external (health and quality of life) effects (benefits and costs), despite the difficulty in allocating monetary value to those effects. Intangible benefits such as sense of empowerment, enhanced social clout or increased trust in (and engagement of) government are rarely incorporated. Development of robust Information Management Systems and utility of artificial intelligence techniques better enable their incorporation. An important question to consider in the context of the the aforementioned social and economic patterns regarding justice reinvestment is, for example, “Can we determine or estimate the average cost per-day to an economy of a social harm or good, such as pre-trial detention, including on a government’s tax base?” Making such an estimate requires considering the cost to the person detained, to their employer, to their dependents, and/or to the persons on whom they depend (who may have to undertake unproductive activity such as time off from work to visit the place of detention). Incarceration holds societal costs in that it lowers rates of educational attainment, increases unemployment, weakens community bonds, and increases adult criminality (Thornberry, 1987; Sampson and Laub, 1997; Thornberry and Krohn, 2001). 9 Estimating an average cost per day of pre-trial detention would allow us to use the total number of days of pretrial detention to estimate the total loss of income and total loss of tax revenue. The first step is to collect data on pre-trial detention and its consequences (Sarkin, 2009). We might also hypothesize that this loss is of greater effect because of the increased likelihood of expenditure by those disproportionately held in pre-trial detention – those in the ‘bottom 40’. This approach would exclude consideration of inequality-informing variables that pre-trial detention affects: the skills, training and education of the workforce to which detained persons are less exposed (Piketty, 2014). It also excludes the potential relationship to violence of the discontent that is often associated with detention (Thurston and Lebovich, 2013; Dufka, 2016). The logic of economics, however, does not always inform political decisions about Rule of Law programming, including efforts to drive down average time spent in pre-trial detention. Political actors experience a diversity of pressures, including incentives to harm the economy of a particular territory, for example a region where perceived adversary groups reside (Toft, 2005; Carment et al., 2009). In attempting to persuade governments of the merits of economic interdependence among arguably competing domestic groups, international development actors, employing analytical approaches, should identify and sensitize key commercial stakeholders linked to policy makers. An important part of the economic rationale for enhancing criminal justice efficacy is to build a nation’s wealth, as well as to stimulate/increase its GDP growth rate. The World Bank has quantitatively identified that equitable advancement in state justice systems also improves economic development. To drive economic development, a 1 percent increase in rule of law institutions’ value will increase intangible capital by 83 percent (Hamilton, 2006). Education and justice system investments, therefore, “…are the most important means of increasing the intangible components of total wealth” (Hamilton, 2006). For low-income countries, therefore, identifying optimally sustainable growth includes identification of policy orientation that maximizes deployment of rents from land and natural resources into physical and human capital (Hartwick, 1977). As violence tends to affect population subgroups disproportionately, it is important for an economic justification to be informed by how people interact under conditions of socio-economic hardship. 7 Scarce econometric explanations are available of social 7 See the following sections on interventions and their political economy. 10 interactions, especially amongst persons in low socio-economic environments; in particular in identifying the level of territorial control exercised by state and non-state actors (Gangopadhyay et al., 2014). For the purposes of determining the normative and economic utility of criminal justice interventions, an inadequate understanding of social conformity and deviance within peer groups diminishes our capacity to identify environments most receptive to criminal justice interventions and where those interventions will have the greatest impact (on violence and socio-economic development) (Gangopadhyay et al., 2014). Weighing risk of intervention capture or loss of community support Development actors aspire to comprehensive data regarding the direct and indirect functional and developmental effect of programs. To consider law enforcement effectiveness, we must turn to our knowledge of what form of deterrence is most effective. Young males, for example, respond to certainty of punishment rather than severity (Cooter and Ulen, 2008). Cooter and Ulen cite Lee and McCrary’s (Lee and McCrary, 2005) observation that the certainty of punishment, not severity of sentence, may act as a greater deterrence mechanism; therefore, informing greater investment in police presence and detection capacity than prisons. The risk that support for criminal justice and security sector reform is captured by a government or non-government actors is informed by the inclusivity of stakeholder engagement and our capacity to foster inclusivity in reforms themselves (Hellman et al., 2000). Where the presence of security actors is increased, trust between the police and the ‘policed’ – including perceptions of police legitimacy – is critical. Ensuring demographic inclusivity and demographic reflection of policed communities in deployed personnel, as well as connection between the police, community leadership and trading communities has also been found to enhance community support (Baker and Scheye, 2007). Conversely, low public perceptions of police integrity and legitimacy drive poor civic engagement with the police and increased criminality (Cooter and Ulen, 2008). Deterrence, therefore, is not only dependent upon police presence but also upon “…the support of good people to help the police [i.e third party involvement] and other legal officials” (Cooter and Ulen, 2008). The personal sacrifice of time, effort, convenience or safety by ‘cooperating people’ increases police effectiveness and certainty of deterrence (Cooter and Ulen, 2008). The cost per person illuminates the circular causal chain of an increasing number of people diminishing each individual’s cost of assisting, and in turn increasing willingness to assist among the community (because of group pressure and lower perceived cost) (Cooter and Ulen, 2008). These issues focus our attention on the dynamic of inter-personal trust, requiring 11 that we draw on not just psychology, but also from sociology and anthropology to determine a situation’s civic trust in authority (Hoff and Stiglitz, 2016). Alternatively, diminished trust in the legitimacy or integrity of the enforcing entity may dissuade public assistance, deepening both that entity’s inability to confront violence as well as public reluctance to assist (Rossmo, 2018) . The emergence of online language metadata enables scope to measure sentiment about related topics, the circumstances that affect them, and how sentiment about topics affects both sentiment about law enforcement and incidents of violence. Measuring the relationship of online language sentiment about violence-relevant topics has already been shown to not only relate to change in violence, but to also enable significant accuracy in predicting its change (Mahony et al., 2019). Contexts where state control of territory is contested In the future, capacity may emerge to forecast when, for example, criminal networks or the state are consolidating or ceding control over communities to one another and how strong a shift in control over territory might be. 8 Emerging data sources, such as geolocated online language data may assist in indicating change in phenomena such as which entity exercises effective authority over specific territory. Once we have identified the mechanisms associated with decline in criminal conduct, we can use available data to identify the most efficient use of resources to advance that decline. For example, because more certain (not longer) punishment is more heavily associated with decline in youth violent conduct than severity of sentence, redirecting money away from incarceration and prisons and toward (evidentially-informed) community policing constitutes a more efficient allocation of scarce resources to reduce youth violence (Lee and McCrary, 2005). This is known in the criminological literature as justice reinvestment. Using data procured from existing World Bank and other justice sector analytical products may assist in maximizing identification of the specific elements of community policing capacity that demand prioritized support. Identifying armed groups’ deterrence or incentives when designing justice interventions may be even more complex. To consider prioritization of policy in such circumstances is to consider that, firstly, criminal organizations’ control of 8 Our understanding of local context informs the degree of control over local civic life (criminal consolidation) and the nature of relations between non-state actors and the state (proximity between armed actors and the state). Where civic groups play important roles in mediating relations with the state, an opening to employ those groups as sensitizing agents exists. Where closer relations between the state and the armed actors exist, the state should take the lead in sensitizing the armed actors as to the intended focus of punitive state action. See: Enrique Desmond Arias. (2013). The impacts of differential armed dominance of politics in Rio de Janeiro, Brazil. Studies in Comparative International Development. 48:3, 263-284, p. 265. 12 urban spaces reflects rule of law’s breakdown as well as variant emerging micro- and local- level political and security orders and dispute resolution processes. Secondly, localized orders are embedded in, stem from, and affect wider political and economic institutions and practices central to development. Thirdly, variance in local armed dominance enables and constrains reforms and stakeholder mobilization behind them (Lee and McCrary, 2005). Prioritizing, sequentially, the economic security and peace building incentives of vulnerable armed actors, dilutes the risk of their remobilization and enhances prospects for consolidating sustainable peace and the accompanying benefits (rather than costs). Considering the knowledge and gaps outlined above, we: (i) outline a method for capturing heterogeneity based on earlier empirical studies undertaken on this issue; and (ii) demonstrate the method and enhanced APP using primary data from a study conducted in 2006 on the costs and outcomes associated with a developmental crime prevention intervention undertaken in Australia; (iii) apply our method to develop CBA outcomes that incorporate the heterogenous distribution across intervention subgroups; (iv) discuss how the method we developed in this paper is incorporated into the enhanced APP; (v) discuss future developments of the enhanced APP that will incorporate modern data science techniques. Overall, the method we describe, test, and apply to the enhanced APP can capture the complexity of prevention and will assist policy makers in producing evidence sensitive to contextual social group-specific variation. 2. A Systematic method for weighing costs and benefits Accounting for heterogeneity The main purpose of CBA is to inform policy makers on the economic viability of publicly funded programs. As briefly highlighted above, the benefits identified and measured are based on parameter estimates of the average effect of a given program or intervention, which are then compared to the associated average costs (Boardman et al., 2017). However, substantial heterogeneity often exists in the benefits and/or costs across individuals and groups. As such, the traditional CBA approach may hide valuable information about which social groups benefit the most net of costs. The Kaldor–Hicks efficiency criterion is commonly applied in CBA and states that a publicly funded program can be justified when the overall benefits outweigh the costs. This criterion is in contrary to the earlier Pareto criterion. It requires that no one will suffer from any change under the proposed intervention conditions. Theoretically, a program under the 13 Kaldor-Hicks efficiency criterion can be defended even if it produces undesirable or negative outcomes for some individuals or groups, so long as the overall benefits are greater than the overall costs to society (Manning, 2008) . However, we propose that the conceptual foundation of CBA (Kaldor-Hicks Criterion) does not adequately address the issue of heterogeneity in program impacts. We argue that an important consequence of accounting for heterogeneity is that international development assistance-driven justice sector gains, net of costs, disproportionately target those in extreme poverty and those in the bottom 40% of the population. Average and quartile treatment effects The availability of data now makes obtaining average treatment effect (ATE) easier, particularly when random control trials (RCT) are employed. In short, the average treatment of the treated group (ATT) equals the ATE (for the population). The RCT lends itself to a simple nonparametric analysis of the treatment effect, given by �0 �1 − ATT = ATE = �1 is, for example, the average number of episodes of political violence (using the where, Armed Conflict Location and Event Database (ACLED)) for the experimental group, while �0 is the average number of episodes of political violence for the control group. The average effect of the intervention of interest is simply the difference between the two averages. As stated above, however, the shortcoming of ATE is our lack of understanding of the distribution of effects. In addition, the mechanisms that make the prevention of conflict (economically) successful. It is inherently difficult to measure the policy impact for each individual, since, by definition, the counterfactual for each individual under the policy may be unobserved depending on the possibility for appropriate matching of experimental and control groups. Often, even when these groups are matched, the heterogeneity of policy impact is buried in the average effect. We can, however, disaggregate the population based on certain characteristics or key factors and compute the distribution of outcomes within the population (Heckman et al., 1997). A sign of heterogeneity is revealed by an uneven distance across individuals. Although we cannot fully appreciate the distribution of effects, we can nevertheless gain useful knowledge about whether there exist heterogeneous effects across groups in the population. 14 It is well understood, as discussed above, that group differences within a community can be observed among a range of static and dynamic factors. But unless we can observe these differences, we have to rely on the ATE of our policy, which does not fully appreciate the unique costs and benefits felt by each group. For example, a defensive policy aimed at reducing domestic terrorism and violence may have significant distributional costs and benefits across the population and, therefore, relying on the ATE could overestimate the effects on some in the population but underestimate the effects on others. So, if one of our variables of interest was social dominance orientation (SDO) (intellectualized as an individual’s preference for inter-group hierarchies within a social system or group-based discrimination), we must capture the levels of SDO that exist across our population. We note here that individuals with high SDO are more likely to support public policies that promote or recreate social hierarchies, while people with low SDO favor more equality-based policies (Pratto et al., 1994). Of course, there are many other variables that we can use to study the heterogenous effects on such a policy, which allow us to separate individuals and groups within the population that are affected by the policy and thus allowing for a comprehensive understanding of the economic implications of our policy decisions. The question now is, how can we disaggregate the population according to the observed and sometimes unobserved differences? Quantile treatment effects There exists a number of methods to capture heterogeneous effects. To capture such variation we outline, in what is a standard approach, the method used by Kristensen et al (Kristensen et al., 2017). The authors apply a method that, like us, attempts to monetize the economic benefits associated with an intervention. In addition, the authors clearly outline how they overcome the shortfalls of traditional CBAs, which focus exclusively on ATEs. In this study the authors employ a randomized control-trial experiment to examine the heterogenous impact of the Danish return-to-work program. The authors successfully demonstrate: (i) the benefits of estimating quantile treatment effects (QTE) (a well-established technique used widely in the social sciences and also included in a range of statistical programs 9); and (ii) the use of the efficiency potential (what we call potential efficiency gain) to assess the net benefits resulting from the reallocation of resources sensitive to the QTE. Kristensen and colleagues, based on their results, propose a screening system to triage participants in order to 9 See for example, Frölich, M. and Melly, B. (2010). Estimation of quantile treatment effects with Stata. Stata Journal. 3:10, 423-357. 15 create effeciencies and ultimately maximize overall program effects by taking into account subgroup differences. The aforementioned static and dynamic factors, which affect treatment outcome, can be measured either categorically or continuously. Categorical divisions of the population could be based on some naturally formed social groups (e.g. ethnicity and gender). The impact of a policy may be moderated by these factors where the policy may unevenly affect different groups within the population. For continuous latent variables (e.g. level of SDO), which are measured along a continuum, we could use confirmatory factor analysis (CFA) to study and describe the relationships between a set of observed variables (e.g. based on Likert scale items) and the specific set of continuous latent variables of concern (Fox, 2010; van der Linden, 2016). This is especially helpful when no pre-established indexes are available. This allows us to create a combined measure capturing the level or degree of an unobserved latent characteristic or factor. For example, when there are no reliable indicators that accurately classify individuals in the community with respect to level of SDO, an evaluation team may decide to incorporate certain assessment criteria (measured continuously) and assess the uniformity of these criteria in measuring SDO levels (the unobserved latent variable). Following Bitler et al. (2006), consider two distributions where we have our experimental and control groups, respectively, F1 and F0, and define quantile treatment effects (QTE) as ∆ = (1) − (0), where () is the qth quantile of distribution Ft. Accounting for heterogeneity Policy outcome differences within the distribution of the continuous latent variable will indicate whether the outcome effects are stronger for some individuals than for others. Figure 1 shows the cumulative distribution for hypothetical treatment and control groups in an RCT of a given policy. 16 Figure 1. Distribution of number of violent incidents, treatment and control groups 60 50 Reduced number of violent incidents 40 30 20 10 0 0.5 0.8 1.1 1.4 1.8 2.2 2.5 3.0 3.3 3.7 Average daily income (USD) Treated Control The horizontal difference in the distributions indicates the impact of the policy, and this difference corresponds to the QTE shown in Figure 2. A horizontal line for the ATE is presented in Figure 2, which indicates no difference across the continuum. Figure 2 clearly indicates that the hypothetical policy has varying impacts across the distribution. Given the positive relationship between the policy impact and the latent continuous variable, positive effects are found from about the 10th percentile onward. The comparison between ATE and QTE, shown in Figure 2, reveals substantial heterogeneity in the treatment effect. Specifically, violence prevention programs for domestic violence (our hypothetical example) are found to be more effective among households with higher income. 17 Figure 2. Average Quantile Treatment Effects 50 Reduced number of violent incidents 40 30 20 10 0 0 10 20 30 40 50 60 70 80 90 -10 Percentile (Average daily income) QTE Lower limit Upper limit ATE If we were able to perfectly screen individuals and only allocate treatment to individuals with a treatment effect above the average cost level, (in the absence of individualized cost data), that is individuals above the break-even line shown in Figure 3, then, we would be able to improve the overall net benefit of the policy. In this example, there is room to improve the net benefit of the policy for the population concerned. We define this as the potential efficiency gain (PEG). That is, the increased benefits net of costs that would accrue to society, if it is possible to perfectly screen individuals to those who benefit at least as much as the costs involved. PEG is given by: ( − ) = where ‘BC maximized’ refers to the scenario where the benefit-cost ratio is maximized due to a selective application of treatment/policy (i.e. treatment only applied to those individuals with a positive net benefit), and ‘BC baseline’ refers to the baseline where the treatment/policy has been adopted to everyone as a universal treatment option. The reader should note that there may be heterogeneity in benefits across individuals not captured by the quantile methods employed – this may be a result of variation of policy impact based on other unobserved moderators in the population. Hence, the PEG is not a maximum benefit 18 that might be obtained via controlling application of treatment/policy to certain groups based on a limited number of observed moderators. Figure 3. QTE and break-even analysis Moderators that expose heterogeneity Heterogeneity can be observed or unobserved. The PEG can only be calculated if evaluators can detect and identify the key factors, whether they are static or dynamic, categorical or continuous, that moderate the policy impact. As such, one needs to identify which covariates drive the heterogeneous effect. One could, for example, interact the intervention with a selection of theoretically relevant covariates (e.g. SES, level of education). If we find that the interaction terms are insignificant, then the heterogeneity observed earlier might not be due to these factors/potential moderators. Of course, the method employed will be dependent on the quality of the data and the hypotheses being tested, but here we only provide one of many methods to capture heterogeneity and estimation of PEG. 19 Given that perceptions of exclusion from access to justice are associated with increased risk of violence, ‘exclusion’ may constitute a covariate that moderates policy impact. The World Bank, for example, recently concluded the establishment of the WEI, which may be employed as data indicating selectivity of the provisional benefits (and costs). The WEI represent, across 181 countries from 1900-2018, denial of individuals’ access to services or participation in governed spaces based on their identity or belonging to a particular group. The indicators measure exclusion across four dimensions: exclusion from civil liberties; exclusion from access to public services; exclusion from access to state jobs; and exclusion from access to state business opportunities. Exclusion in these dimensions is grouped by five indices: socio-economic group; by gender; by rural/urban location; by political group; and by social group (including caste, ethnicity, language, race, region, religion, migration status, or some combination thereof). The WEI, therefore allow for exclusion, for example by gender, within a country or across a region, for example the Middle East across the four dimensions (Pillai, 2019). Of particular relevance to justice processes are the dimensions of exclusion of social groups in their access to civil liberties and the state service of justice. However, the efficacy of equal access to other state services, to state jobs, and state business opportunities may inform, in particular, identification of social group-specific exclusion in CBA modeling of justice processes to resolve equal access in those dimensions. A further covariate example that may moderate policy impact may be ‘governance’. The World Bank’s WGI reports aggregate and individual governance indicators for over 200 countries and territories from 1996, for six governance dimensions: Voice and Accountability; Political Stability and Absence of Violence; Government Effectiveness; Regulatory Quality; Rule of Law; and Control of Corruption. These aggregate indicators, based on over 30 individual data sources, combine the views of a large number of enterprise, citizen, and expert survey respondents. As data emerges specific to how different justice process exclude social groups, frameworks can be developed and vetted to inform what constitutes optimal social (such as inclusion) and economic net benefits. The United Nations and World Bank, on this issue specific to processes of accountability for atrocity crimes, note: “Weighing the equality of accountability processes against the imperative to bring perpetrators to book is critical to the challenge of advancing stabilization and justice in conflict-affected environments under SDG 16. Accountability processes may exacerbate 20 grievances related to specific social groups if they are perceived to discriminate between groups. How and why the real or perceived unequal treatment of social groups actually occurs varies from one process to another. Frameworks to identify how accountability processes treat groups differently can help to identify ways in which to preempt spoilers and mitigate risks of conflict.” (Mahony, 2016; United Nations and World Bank, 2018) 10 3. Method In this paper we use the enhanced CBA APP to process the cost and benefit data. We adopt the QTE method as described above. The enhanced APP proposes to overcome the shortcomings of existing cost-benefit tools (e.g. the Manning Cost-Benefit tool versions 1 and 2 (Manning et al., 2016; Manning and Wong, 2016a; Manning and Wong, 2016b)), with the inclusion of the heterogeneity component which allows us to: (i) identify variation or differences between groups or populations (e.g. individual actor attributes, political structures, institutions); (ii) identify how justice processes differ across groups/populations; and (iii) distribute costs and benefits according to heterogenous impact. Figure 4 illustrates the enhanced CBA APP. In this study, we demonstrate three of the six interacting modules (Modules 1, 2 and 3). A full discussion of the six modules included in the enhanced CBA APP is provided by Manning et al. (2018). 10 The UN and World Bank cite a framework that breaks down the jurisdictional and functional elements of criminal justice processes that advance or undermine equality before the law. 21 Figure 4. Updated cost benefit tool In terms of module 1, costs can be disaggregated by bearer (e.g. criminal justice system, local government, business, and society) and the outcomes (either positive or negative) can be, as described above, affected by identified moderators. Here, the user of the enhanced CBA APP specifies the identified and observed moderators (i.e. continuous or categorical). If the variable is continuous, the evaluator can conduct correlational analyses to justify the use of specific characteristics or factors of the population as a moderator. If the variable is categorical, then the evaluator can conduct ANOVA to identify the association between the moderator and the outcome of interest. Once the relationship between the moderator/s and the outcome is established, the benefits side of the CBA can now account for the heterogenous effect. This takes us to module 2 where CBA calculations are performed. Here we include steps such as accounting for: (i) economic assumptions (i.e. inflation and discount rates); (ii) confidence intervals (i.e. worst and best-case scenario); (iii) optimism/confirmation bias correction; (iv) percentage of total cost borne and spent each year; (v) attributable fraction (i.e. percentage or proportion of costs attributed to the intervention of concern); and (vi) heterogenous effects on outcome. Module 2 allows the evaluator to separate outcomes across 22 affected social groups (e.g. beneficiaries) if the moderator is categorical in nature (e.g. SES). If the moderator is continuous in nature, then the evaluator can employ a range of methods, for example QTE as described above, to generate patterns of benefits across the distribution of individuals or groups along the continuum. In module 3, the enhanced CBA APP provides the outputs of the economic analysis. In summary, the APP allows for cost-feasibility analysis (i.e. comparing the overall costs of the project against the budget), cost-savings analysis (i.e. comparing the costs of the project against the savings generated from avoided crimes), cost-effectiveness analysis (i.e. comparing the costs of the project against the number of units of output such as the number of crimes prevented) and CBA (i.e. comparing the costs of the project against the overall benefits – avoided crimes and other benefits such as enhanced safety). Tables and plots are also included to display, for example, net costs, net benefits, cost-benefit ratio, net benefits by bearers and potential efficiency gain (PEG - as described above). Data employed to test modules 1 to 3 using the enhanced CBA APP We employ data from the Pathways to Prevention Project (Homel et al., 2006) to test modules 1 to 3 of the enhanced CBA APP. The Pathways to Prevention project was established over twenty years ago as an early intervention, developmental crime prevention initiative focussed on the transition to school in one of the most disadvantaged urban areas in Queensland, Australia (Homel et al., 2001). Here, the goal was to provide positive pathways to individuals at-risk of later learning and behavioral problems that may eventually lead to crime and deviance. The project comprised two components: (i) a preschool intervention program (PIP), which was a child-focused and school-based set of activities; and (ii) a family independence program (FIP), which was a family-focused and community based set of activities run by Mission Australia (Manning et al., 2006). In this paper, we focus exclusively on the PIP component. PIP aimed to enhance the participant’s readiness to succeed at school. Through a sequence of structured small-group interactions with either specialist teachers or program staff, PIP aimed to develop a participants: (i) communication skills by introducing more abstract language, complex vocabulary and appropriate grammar formats as part of the preschool experience, and (ii) social skills to improve a participants ability to better interpret social interactions, overcome emotions that are unproductive (e.g. anger and anxiety) and develop strategies for dealing with problems that often occur during exchanges with peers (Manning et al., 2006). 23 To undertake our CBA, we use cost data derived from Manning et al. (2006) and outcome data collected from the chief investigator of the project, Professor Ross Homel. Project costs were estimated separately for three distinct stages: development, implementation, and evaluation. The cost analysis method employed and cost estimates are described in the study undertaken by Manning and colleagues (2006). Part of our CBA also requires us to estimate potential costs of a negative outcome – in this case the cost of dealing with behavioral problems – or the avoided costs if the intervention produces outcomes that reduce the probability of a child needing future behavioral management intervention. Manning and colleagues (2006) estimated the costs of four behavioral management alternatives (grouped into two categories) that may be required if a child presents with behavioral problems in the early years of education. Category 1 consists of programs developed to help improve the behavior of children with borderline or less challenging behavioral problems, while Category 2 consists of programs aimed at helping those children with more severe or extreme behavioral problems (see Table 1). Homel and colleagues (2015) measured difficult and challenging behavior using the Rowe Behavior Rating Inventory (RBRI). The RBRI is a validated teacher checklist used to assess the level of children’s difficult behavior (Rowe and Rowe, 1995). In this paper we classify children who received a score of 20 to 29 as displaying a low level of poor behavior and thus only requiring little assistance (i.e. Alternative 1). We coded children receiving a score greater than 30 but less than 40 as requiring more assistance (i.e. Alternative 2). Children who received a score greater than 40 were identified as having more severe or extreme behavioral problems and fell into Category 2 type programs (RBRI 40 - 49 – Alternative 3; RBRI ≥50 – Alternative 4). 24 Table 1: Behavioral management programs Category 1 Alternative 1 Pathways Communication Program (RBRI score 20-29) Alternative 2 School district behavioral management teams (Inala Cluster)- known as Behavior Support Team, Corinda District (RBRI 30- 39) Category 2 Alternative 3 Pathways Social Skills Program (RBRI 40- 49) Alternative 4 Behavioral School (Tennyson Special School) (RBRI 50+) The costs of the alternative programs estimated by Manning and colleagues (2006) are presented in Table 2. Table 2. Cost of alternative behavioral management alternatives Intervention Budget Cost No. of Participants Per participant cost Alternative 1 $47,861.41 125 $382.89 Alternative 2 $236,312.93 145 $1,629.74 Alternative 3 $13,999.93 100 $139.99 Alternative 4 $417,460.32 21 $19,879.06 Measuring heterogenous impact of PIP In terms of our analysis of potential heterogenous outcomes, we use scores derived from the same data used by Homel and colleagues (2015). As described in brief above, RBRI forms a total measure of behavioral adjustment ranging from 12 (positive adjustment) to 60 (poor adjustment). In the longitudinal study undertaken by Homel and colleagues (2015) there were five measures of behavioral adjustment across the reference period, one for each academic year. Our model specifies behavioral adjustment for the fifth assessment period as the dependent variable, behavioral adjustment for the first baseline period as a control, and the 25 exposure to PIP as an independent variable. The model also analyzes the heterogenous impact of PIP on behavioral adjustment through a series of regression analyses according to the number of siblings of the program participants (i.e. our diversity variable). We chose number of siblings as a diversity variable as it has been shown that children that have many siblings tend to display an increased odds of adverse developmental outcomes (de la Rochebrochard and Joshi, 2013). In practice, then, compliance at Time 1 has been partialled out, leaving only the difference between Time 1 and Time 5 to be explained by the independent variable in the model (Cohen and Cohen, 1983). Using the rnorm() function of the R Studio (R.Studio Team, 2020), the coefficients drawn from these analyses were applied to simulate nine datasets, each with 1,000 samples, with a child having no siblings to 8 siblings. For more details of the simulation please refer to Peng (2022). 4. Results Our regression results show that PIP led to statistically significant poor behavioral adjusted outcomes for participants with no siblings (β = 10.45, p < .001), 5 siblings (β = 9.10, p < .001), 7 siblings (β = 12.27, p < .001) and 8 siblings (β = 10.83, p < .001) as revealed by the change in RBRI score post intervention. The PIP, however, was estimated to be beneficial to children with 1 (β = -7.00, p < .001) or 2 siblings (β = -2.09, p < .001), but had no statistically significant effect on children with 3 (β = 0.57, p > .1), 4 (β = 2.01, p > .1) or 6 siblings (β = - 0.15, p > .1). Based on these results, we further estimate the number of individual children who potentially may be triaged into the aforementioned alternative treatment and behavioral management programs (as described in Table 1). The difference in proportion of children with an RBRI score of 20-29 (category 1), 30-39 (category 2), 40-49 (category 3) and 50-60 (category 4) were observed between the intervention group (with PIP) and control group (without PIP). Such a difference in proportion was applied to the actual data collected by Homel et al. (2015), suggesting either a potential reduction or increase in the number of children who may require future behavioral management interventions. The estimated number of children who require additional behavioral management interventions is based on the percentage of children sorted into one of the four RBRI score categories before and after the intervention. This is calculated by applying the percentages derived from the simulated dataset to the actual number of participants in the Homel dataset. We present these results in 26 Table 3 where we show, for example, that the estimated number of children requiring Alternative 1 (i.e. additional behavioral management intervention) with one sibling is 1.92, Alternative 2 (-2.44 (negative sign signifying a reduction in children requiring intervention)), Alternative 3 (-1.44), and Alternative 4 (-1.44). We then applied the outcome data with the cost data described above into the enhanced CBA APP. Since the provision of PIP did not affect the RBRI score of some of the subgroups (non-statistically significant effect), no benefits could be attributed to those groups. As such, the economic benefits for these groups were not estimated (see Table 3 column 3, row 5). For other groups, however, we found positive and significant benefits (resulting in a positive avoided cost), and also negative and significant benefits (resulting in additional future treatment cost). The results presented in Table 3 show the number of children under each category (i.e. by number of siblings) multiplied by the corresponding per participant costs of the management programs to generate an estimate of the costs, benefits, and net benefits of PIP (i.e. the avoided costs of behavioral management). As discussed above, PIP was most beneficial for those PIP participants that had 1 or 2 siblings. Table 3 reveals the net benefits to PIP participants with 1 or 2 siblings of $ $18,234.39 and $74,049.29, respectively. Table 3. Estimated CBA results according to the no. of siblings Estimated Estimated Estimated Estimated no. of no. of no. of no. of children children children children requiring requiring requiring requiring No. of Costs Benefits Net benefit Alternative Alternative Alternative Alternative siblings (AUD) (AUD) (AUD) 1 2 3 4 0 $4,611.92 -$20,928.98 -$25,540.90 -2.00 1.20 -1.00 1.00 1 $13,835.76 $32,070.15 $18,234.39 1.92 -2.44 -1.44 -1.44 2 $2,8824.5 $102,873.79 $74,049.29 -7.42 -9.42 0.74 -4.27 3 $15,565.23 $- -$15,565.23 - - - - 4 $10,376.82 $- -$10,376.82 - - - - 5 $6,917.88 -$27,133.43 -$34,051.31 -1.80 5.00 -1.40 1.00 6 $6,341.39 $- -$6,341.39 - - - - 7 $1,729.47 -$2,876.68 -$4,606.15 -1.00 2.00 0.00 0.00 8 $4,611.92 -$4,646.32 -$9,258.24 -1.00 3.00 1.00 0.00 The above economic estimates (derived from the enhanced CBA APP) are consistent with the findings of our regression analyses, both in strength and direction, where there were positive estimated benefits of the PIP intervention for children with 1 or 2 siblings and 27 negative estimated benefits for children with no siblings or 5 or 7 or 8 siblings. Figure 5 below, which is a figure derived from the enhanced CBA APP, illustrates this relationship. The figure suggests that the PIP should only be strategically provided to children with 1 or 2 siblings to maximize its benefits and avoid any unintended adverse effect of the PIP towards children with different numbers of siblings. Figure 5. Comparison of net benefit according to the no. of siblings No. of siblings We note that the example we use in this paper to illustrate to method and APP uses real data but simulates to adjust for small sample size and the potential outcomes of children requiring some form of future behavioral management intervention. In the absence of the above-described method that estimates the heterogenous distribution of outcomes across the PIP intervention group and enhanced CBA APP (modules 1 to 3 of the APP), we would have found it difficult to fully appreciate: (i) how the benefits of the intervention are distributed across the intervention group participants; (ii) the avoided costs of PIP in terms of future behavioral management interventions; and (iii) the best targeting of resources among the intervention group (in this case children with 1 or 2 siblings) to enhance the economic efficiency of the intervention, which arguably may lead to potential improved economic returns on investment. 5. Discussion and Conclusions Presented above was a clear outline and test of Modules 1 to 3 of the enhanced CBA APP. The data-driven capacity within the current version of the enhanced CBA APP can identify which justice processes and societal factors are most significant for the costs and benefits of processes specific to context. The current APP, therefore, is capable of accounting for macro variables like inflation, provision of best and worst-case scenarios, identification and accounting of data bias, proportion of costs borne per year, and effects on outcome, including 28 outcomes specific to social groups, to context and to specific intervention elements. Manning et al. (2016) provide a detailed discussion on these elements. However, the current version of the enhanced CBA APP is capable of more than what we have presented here. Below we describe three additional modules that are currently in various stages of development, testing and implementation. Module 4 (as shown in Figure 4) improves upon earlier economic tools, such as the MCBT, by drawing upon and exploiting machine learning (ML) techniques. Here, we incorporate a ‘User Input Database’ module, where all data entered by users into the APP is stored as one set of records per project. The enhanced APP will allow the user to store their input data on their server, providing a single data resource for the ML module (i.e. Module 6). Here, we intend to provide source code to allow users to manage their data on their own server, utilizing their own security protocols. Specifically, since Module 1 identified and established a relationship between moderators and outcome, module 4 then captures those relationships allowing the system to learn from these relationships and identify moderators that are relevant to outcomes concerned. This allows for the creation of a more comprehensive database and the possibility of subsequent ML, as described below in Module 6 (e.g. imputation of missing data). Linked to Module 4, Module 5 will provide code for users to build a database to store the calculated results after analysis. Storing the calculated results is crucial because it provides the ability to model the relationships between all input-relevant cost and benefit data (Module 1) and the output of CBA (Module 3). Therefore, the database in this module of the workflow diagram stores the benefits (that could also be weighted using a harm index) and analysis data, enabling the system to map the relations between input and output, and exploit this to learn and improve CBA over time. Finally, Module 6 integrates ML techniques to achieve two main goals, namely: (i) to provide input support to the user by imputing missing values (currently in testing stage), identifying potentially erroneous values, and make suggestions about relevant contextual factors; and (ii) to improve the analytical capabilities of CBA by usefully reducing the number and types of variables to minimize user effort (e.g. time-consuming data entry) and develop better estimates (e.g. cost savings; crimes avoided), based on what the system learns from earlier projects. 29 As with all policy and programmatic interventions, the moral and commonly aspirational arguments for justice interventions must be enhanced by collection of data representing how different intervention designs enhance or undermine the social groups’ equality before and after access to justice. Sucha a granular measure of the equality of social groups under specific justice policy settings could enhance measurement of this component of process legitimacy. Further, the legitimacy of justice interventions can be enhanced by better quantifying and comparing (and communicating) their economic and societal benefits, particularly where this can be done in such a way to demonstrate disproportionate impact for excluded social groups. Justice data collection and analysis, which employs broader societal data, as well as data generated from existing justice sector analytical methods, is a prerequisite for a ‘business case for justice’. An enhanced CBA APP, as described above, serves these objectives by improving the accuracy of justice sector resource allocation for maximum economic and societal outcomes while targeting society’s most vulnerable and excluded social groups. Traditional CBAs commonly seek to weigh anticipated policy or intervention benefits against a policy reform or an intervention’s cost to determine overall societal benefit. CBAs demand resources and time due to the many challenges of quantifying anticipated costs and benefits, idiosyncratic to the nature of policies and interventions in the diverse complexity of contexts in which they occur (Prescott, 2010). Drawing on multiple sources of societal and justice process data in a systematic way, allows governments and stakeholders more exhaustive identification of interventions’ cost-benefits. It also enables systematic intervention comparison across a range of metrics beyond monetary benefits (such as crimes avoided and level of safety improvement) and to identify which interventions have greatest benefit for the most vulnerable social groups. Future utilization of an enhanced CBA APP, which includes the tested heterogenous impact component and future machine learning capacity, will enable significant contextual factors identified in previous CBAs to be tested in future intervention CBAs. The enhanced CBA APP confronts the human and material resource problem of repeated individual CBAs across different justice processes. Broader societal data (e.g. geographic, economic, demographic, climate, security) as well as data specific to civil, criminal, and informal justice sector capacity can be retained and employed by the APP across (where appropriate) justice processes. Identified intervention costs are disaggregated by cost bearer (e.g. criminal justice system, local government, business, and society) and 30 intervention outcomes (either positive or negative) may be qualified by moderators, including, via the use of Governance and Exclusion indicators, the social groups that benefit and the societal governance issues perceived as most demanding attention. The enahnced CBA APP retention and AI-driven vetting of data integrity component will enable re-vetting of data’s representativeness over time and alert researchers/users of the APP to potentially erroneous values (making suggestions about relevant contextual factors), and significant factors for data input narrowing. For development actors considering budget support or programmatic operations, enhanced capacity to determine the costs and benefits of policy and programming, particularly for the most vulnerable, enhances operational credibility, legitimacy, government process ownership, and development outcomes. We know that a 1 percent increase in rule of law institutions’ value increases intangible capital by 83 percent – the most significant long-term development investment societies can make (Hamilton et al., 2005). We know that since education and justice system investments constitute the most significant components of intangible capital, identifying optimal approaches to enhancing justice processes is critical for economic development (Hamilton et al., 2005). Since rule of law and institutional confidence also constitute critical components of ‘social cohesion’ and ‘process legitimacy’, which are critical to social sustainability (Cuesta et al., 2022), identifying justice policy settings that enhance justice accessibility and associated outcomes also advance social sustainability. The enhanced CBA APP takes the first steps towards enabling government and stakeholder-led use of artificial intelligence to advance our analytical capacity to drive these inclusive justice objectives. Further, the potential of this methodological approach is more significant when considered across broader public policy and when drawing on emergent data sources and methods. Application of this approach when drawing more fully on data representing components of social sustainability (social inclusion, social resilience, social cohesion, and process legitimacy) may further enhance, via cost-benefit identification, context-specific exivdence of social sustainability. Development and dissemination of evidence identifying optimal policy settings based on weighting of costs and benefits in turn enhances informed public understanding, civic participation, and civic inclination to access justice processes (Mansuri and Rao, 2022; Innes and Booher, 2004). The results demonstrate significant efficacy of this approach and suggest exploration of its application to other justice and broader public policy processes. 31 References Albright, M., and de Soto, H. (2008). "Making the law work for everyone". (New York: United Nations Development Programme). Aos, S., and Drake, E. (2010). "WSIPP's Benefit-Cost Tool for States: Examining Policy Options in Sentencing and Corrections". (Olympia: Washington State Institute for Public Policy). Baker, B., and Scheye, E. (2007). Multi-layered justice and security delivery in post-conflict and fragile states: Analysis. Conflict, Security & Development 7(4), 503-528. Banerjee, A., and Duflo, E. (2011). Poor economics: A radical rethinking of the way to fight global poverty. New York: Public Affairs. Bitler, M., Gelbach, J., and Hoynes, H. (2006). What mean impacts miss: Distributional effects of welfare reform experiments. American Economic Review 96(4), 988–1012. Boardman, A., Greenberg, D., Vining, A., and Weimer, D. (2017). Cost-benefit analysis: concepts and practice. Cambridge: Cambridge University Press. Brinkerhoff, D., Wetterberg, A., and Dunn, S. (2012). Service delivery and legitimacy in fragile and conflict-affected states: Evidence from water services in Iraq. Public Management Review 14(2), 273-293. Carment, D., Prest, S., and Samy, Y. (2009). Security, Development and the Fragile State: Bridging the Gap between Theory and Policy. Taylor & Francis. Cavallari, P., Devlin, M.D., and Tucci, R.A. (2014). "Justice Measured: An Assessment of the Economic Impact of Civil Legal Aid in Arkansas". Arkansas Access to Justice Commission and University of Arkansas Clinton School of Public Service ). Cohen, J., and Cohen, P. (1983). Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. Hillsdale, NJ: Erlbaum. College of Policing (2022). Practical evaluation tools [Online]. Available: https://www.college.police.uk/research/practical-evaluation-tools. [Accessed]. Collier, P. (2006). African growth: why a ‘big push’? Journal of African Economies (15), 188-211. Cooter, R., and Ulen, T. (2008). "Law and economics". (New York: Pearson Addison Wesley). Cuesta Leiva, J., and Madrigal Correa, A., and Pecorari, N (2022). Social Sustainability, Poverty, and Income : An Empirical Exploration. Policy Research Working Paper no. WPS 10085 Washington, D.C.: World Bank Group. 32 http://documents.worldbank.org/curated/en/099915206132218553/IDU05fa103b0046 6704a130bfec06158420e23ee de la Rochebrochard, E., and Joshi, H. (2013). Siblings and child development. Longitudinal and Life Course Studies 4(3), 276-287. Dufka, C. (2016). Confronting Mali's new jihadist threat. The New York Times, p.Newspaper Article. Fisk, K., and Cherney, A. (2017). Pathways to institutional legitimacy in postconflict societies: perceptions of process and performance in Nepal. Governance 30(2), 263- 281. Fox, J.-P. (2010). Bayesian item response modeling: Theory and applications. New York: Springer Science & Business Media. Gangopadhyay, P., Rahman, M., and Bhattacharya, B.N. (2014). Are there any roles for social conformity and deviance in poverty? Insights from a field study on working poverty and educational investment in Bangladesh. Journal of the Asia Pacific Economy 19(4), 539-557. Gilley, B. (2009). The right to rule: how states win and lose legitimacy. New York: Columbia University Press. Goodwin, L., and Maru, V. (2017). What do we know about legal empowerment: Mapping the evidence. Hague Journal on the Rule of Law 9(1), 157-194. Hackmann, C. (2012). "Legal empowerment of the poor and its relation to pro-poor growth". (Paris: Organisation for Economic Co-operation and Development). Hamilton, K. (2006). Where is the wealth of nations?: measuring capital for the 21st century. Washington DC: World Bank Publications. Hamilton, K., Ruta, G., Bolt, K., Markandya, A., Pedroso-Galinato, S., Silva, P., et al. (2005). Where Is the Wealth of Nations?: Measuring Capital for the 21st Century. Herndon: World Bank Publications. Harley, G., Capita, I., Markovic, M., Panter, E., and Scott-Moncrieff, L. (2019). "A Tool for Justice: The Cost-Benefit Analysis of Legal Aid". (Washington, DC: World Bank). Hartwick, J. (1977). Intergenerational Equity and the Investing of Rents from Exhaustible Resources. American Economic Review 66(5), 972-974. Heckman, J.J., Smith, J., and Clements, N. (1997). Making the most out of programme evaluations and social experiments: Accounting for heterogeneity in programme impacts. The Review of Economic Studies 64(4), 487-535. 33 Hellman, J., Jones, G., and Kaufmann, D. (2000). Seize the state, seize the day: State capture, corruption, and influence in transition. Washington DC: The World Bank. Hoff, K., and Stiglitz, J.E. (2016). Striving for balance in economics: Towards a theory of the social determination of behavior. Journal of economic behavior & organization (126), 25-57. doi: 10.1016/j.jebo.2016.01.005. Homel, R., Elias, G., and Hay, I. (2001). "Developmental prevention in a disadvantaged community," in The social origins of health and well-being, eds. R. Eckersley, J. Dixon & B. Douglas. (Cambridge: Cambridge University Press), 269-278. Homel, R., Freiberg, K., Branch, S., and Le, H. (2015). Preventing the onset of youth offending: The impact of the Pathways to Prevention Project on child behavior and wellbeing. Trends & Issues in Crime and Criminal Justice (481), 1-10. Homel, R., Freiberg, K., Lamb, C., Leech, M., Carr, A., Hampshire, A., et al. (2006). "The Pathways to Prevention Project: The first five years 1999-2004". (Brisbane: Griffith University and Mission Australia). Innes, Judith E., and David E. Booher. "Reframing public participation: strategies for the 21st century." Planning theory & practice 5.4 (2004): 419-436 Kahn, R. (1931). The relation of home investment to unemployment. The Economic Journal 41(162), 173-198. Kristensen, N., Kolodziejczyk, C., Bolvig, I., and Houlberg, K. (2017). The efficiency potential in social programs: accounting for heterogeneity in benefit-cost analysis. Journal of Benefit-Cost Analysis 8(1), 1-23. Lakner, C., Negre, M., and Prydz, E.B. (2014). Twinning the goals: how can promoting shared prosperity help to reduce global poverty? Washington D.C.: The World Bank. Lee, D.S., and McCrary, J. (2005). "Crime, Punishment, and Myopia". National Bureau of Economic Research). Mahony, C. 2016. International Criminal Justice Case Selection Independence: An ICJ barometer. FICHL Policy Brief Series [Online], 58. Mahony, C., Albrecht, E., and Sensoy, M. 2019. The Relationship between Influential Actors’ Language and Violence: A Kenyan Case Study Using Artificial Intelligence. Available: https://www.theigc.org/wp-content/uploads/2019/02/Language-and- violence-in-Kenya_Final.pdf. Manning, M. (2008). Economic Evaluation of the Effects of Early Childhood Intervention Programs on Adolescent Outcomes. PhD, Griffith University. 34 Manning, M. (2018). "Manning Cost Benefit Tool APP (MCBT APP)". (Canberra: Australian National University). Manning, M., Homel, R., and Smith, C. (2006). Economic evaluation of a community-based early intervention program implemented in a disadvantaged urban area of Queensland. Economic Analysis and Policy 36(1 & 2), 99-120. Manning, M., Johnson, S., Tilley, N., Wong, G., and Vorsina, M. (2016). Economic analysis and efficiency in policing, criminal justice and crime reduction: What works? London: Palgrave. Manning, M., and Wong, G. (2016a). "Manning Cost Benefit Tool 1". (London: College of Policing UK). Manning, M., and Wong, G. (2016b). "Manning Cost Benefit Tool 2". (London: College of Policing UK). Manning, M., Wong, G., Graham, T., Ranbaduge, T., Christen, P., Taylor, K., et al. (2018). Towards a ‘smart’ cost-benefit tool: Using machine learning to predict the costs of criminal justice policy interventions. Crime Science 7(1), 1-12. Mansuri, Ghazala, and Vijayendra Rao. (2012). Localizing development: Does participation work?. Washington D.C.: The World Bank. McLeod, S. (2007). Maslow's hierarchy of needs [Online]. Available: http://www.simplypsychology.org/maslow.html [Accessed 28 August 2019]. Mcloughlin, C. (2015). When does service delivery improve the legitimacy of a fragile or conflict‐affected state? Governance 28(3), 341-356. Milliken, J., and Krause, K. (2002). State Failure, State Collapse, and State Reconstruction: Concepts, Lessons and Strategies. Development and change 33(5), 753-774. doi: 10.1111/1467-7660.t01-1-00247. Narayan, A., Saavedra-Chanduvi, J., and Tiwari, S. (2013). Shared prosperity: links to growth, inequality and inequality of opportunity. Washington D.C.: The World Bank. Nixon, H., Mallett, R., and McCullough, A. (2017). "Are Public Services the Building Blocks of State Legitimacy? Input to the World Banks 2017 World Development Report". (London: Secure Livelihoods Research Consortium). Peng, R. (2022). Generating random numbers [Online]. Leanpub. Available: https://bookdown.org/rdpeng/rprogdatascience/simulation.html [Accessed 1 November 2022]. Piketty, T. (2014). Capital in the Twenty-First Century. Harvard University Press. Pillai, S. (2019). Exclusion by Gender in the MENA Region. 15 April 2019. 35 Pratto, F., Sidanius, J., Stallworth, L., and Malle, B. (1994). Social dominance orientation: A personality variable predicting social and political attitudes. Journal of Personality and Social Psychology 67(4), 741-763. doi: 10.1037/0022-3514.67.4.741. Prescott, J.J. (2010). The challenges of calculating the benefits of providing access to legal services. The Fordham urban law journal 37(1), 303. R.Studio Team (2020). RStudio: Integrated Development for R. RStudio [Online]. Boston, MA: RStudio, PBC. Available: http://www.rstudio.com/ [Accessed 1 November 2022]. Rivas, L. (2003). Income taxes, spending composition and long-run growth. European Economic Review 47(3), 477-503. Rossmo, K. (2018). Criminal Investigative Failures. London: CRC Press. Rowe, K.J., and Rowe, K.S. (1995). "RBRI profile user’s guide". (Melbourne: Centre for Applied Educational Research). Sacks, A., and Larizza, M. (2012). Why quality matters: rebuilding trustworthy local government in post-conflict Sierra Leone. Washington D.C.: The World Bank. Sampson, R., and Laub, J. (1997). "A life-course theory of cumulative disadvantage and the stability of delinquency," in Developmental theories of crime and delinquency, ed. T. Thornberry. (New Brunswick: Transaction Publishers), 133-161. Sarkin, J. (2009). Prisons in Africa: An evaluation from a human rights perspective. Sur International Human Rights Journal 9(1), 22-49. Sen, A. (2000). The Discipline of Cost‐Benefit Analysis. The Journal of legal studies 29(S2), 931-952. doi: 10.1086/468100. Stel, N., and Abate, F.M. (2014). Between Control and Cooperation: Multi-Stakeholder Service Provision and the Legitimacy of State Institutions in Ethiopia’s Amhara National Regional State. The European Journal of Development Research 26(5), 743- 760. Stiglitz, J.E. (2012). The price of inequality. New York: W.W. Norton & Co. Sturge, G., Mallett, R., Hagen-Zanker, J., and Slater, R. (2017). "Tracking livelihoods, services and governance: Panel survey findings from the Secure Livelihoods Research Consortium". (London: Secure Livelihoods Research Consortium). The World Justice Project (2019). "The World Justice Project Rule of Law Index 2019". (Washington DC: The World Justice Project). Thornberry, T. (1987). Toward an interactional theory of delinquency. Criminology 25(4), 863-892. 36 Thornberry, T., and Krohn, M.D. (2001). "The development of delinquency," in Handbook of youth and justice, ed. S. White. (Boston: Springer), 289-305. Thurston, A., and Lebovich, A. (2013). "A Handbook on Mali's 2012-2013 Crisis". (Evanston: Northwestern University). Toft, M.D. (2005). The Geography of Ethnic Violence: Identity, Interests, and the Indivisibility of Territory. Princeton University Press. United Nations, and World Bank (2018). "Pathways for Peace: Inclusive Approaches to Preventing Violent Conflict". (Washington DC: World Bank). van der Linden, W. (2016). Handbook of item response theory. London: CRC Press. Vickery, C. (1977). The time-poor: A new look at poverty. Journal of Human Resources 12(1), 27-48. World Bank (2017). "Governance and the law". (Washington D.C: World Bank Publications). 37