CATALYSTS FOR CHANGE: PARENT- TEACHER ASSOCIATIONS IN MONGOLIAN SCHOOLS Final Project Evaluation of the Transparency and Accountability in Mongolian Education Project, P150842 Saha Meyanathan May 2021 Project Parent Teacher Associations in action. Credit: National School Parent Teacher Association 1 Acknowledgements The writer would like to thank and acknowledge the support of the Ulaanbaatar-based team that coordinated the evaluation work during the COVID-19 pandemic period. The team included Jayamaran Govindaraju, Nurbyek Raruzkhan, Tsolmon Bat-Ochir, Gansetseg Ganaa, Batzaya Purevsuren, Namsrai Tsoghbaatar, Bataa Batbayar, Oyundari Ganbaatar and Enkhmaa Baatar. The team would like to thank the Globe International Center and All For Education for facilitating the evaluation and their directors Naran and Tungalag who spent a considerable amount of time helping the team with background information and project materials. Andrei Mikhnev, Country Manager of the World Bank in Mongolia, Pagma, Ogtonbayar, and Amarbayasgalan all assisted at various stages of this evaluation. Khailuna of the Education Quality Reform Project provided detailed information and views on the project. A number of other CSOs in Mongolia and elsewhere also provided background inputs for the evaluation. They include Open Society Forum, Partnership For Transparency Fund, Mongolia Education Alliance, the Asia Foundation, and the Independent Research Institute Mongolia. The Steering Committee officials at the Ministry of Education (T. Nyam-Ochir), Ministry of Finance (M. Sanjaadorj) and Independent Authority Against Corruption (J. Sarangerel) provided extensive views on the project during the interviews. The team would also like to thank the over 200 stakeholders ranging from aimag governors, soum and bhag mayors to local officials, parents, teachers and students, for their valuable feedback on the project. The support from the GPSA team is gratefully acknowledged throughout the evaluation period. 2 Executive Summary Background This report constitutes the final evaluation of the Transparency and Accountability in Mongolian Education (TAME) project supported by a $650,000 four-year grant from the Global Partnership for Social Accountability (GPSA). The project’s goal was to “ strengthen civic engagement in the education sector to improve transparency and accountability of budget and procurement processes for the delivery of better-quality educational services in central and western Mongolia.” The aim was to jump-start civil society engagement in selected schools to improve the transparency of school budgets and procurement practices in order to improve service delivery. The eight provinces chosen (Uvurkhanghai, Khuvskul, Bulgan, Bayan-Ulgii, Khovd, Uvs, Arkhangai, and Sukhbaatar) are home to a number of minorities such as Kazakhs, Tuvas, Tsaatan, Buryats and others. The challenge of delivering educational services to these regions and minority groups is compounded by their remoteness. The TAME project was designed around two main components: the first to promote civic engagement at the school and district level through Parent Teacher Associations (PTAs), and the second focused on budget and procurement transparency at the whole provincial level, implemented through CSOs. Goals of the Evaluation The goals of this evaluation are to 1) show the results and social returns on investment made by the project, 2) generate learning and knowledge about the conditions in which the project achieved and may sustain and expand its results, and 3) demonstrate how and to what extent the GPSA’s Theory of Action was verified through the experiences of the TAME project. The success of the project measured against the GPSA’s Theory of Action indicates substantial benefits both in financial and social terms despite difficult entry conditions. Methodology This evaluation adopted a mixed-method approach to assess the evaluation objectives. Project results (outputs and impacts) are based on quantitative project data and qualitative observations and feedback from groups of project stakeholders. The quantitative data were obtained from project updates submitted to the GPSA by the lead grantee (Globe International Center) and the co-lead grantee (All for Education), World Bank project supervision reports, and the final project Implementation Completion and Results report. The qualitative findings were derived from Zoom- enabled focus group discussions from all project schools and selected officials in the eight provinces. Further, a reasonable number of face-to-face key informant interviews were also conducted with the Project Steering Committee and national CSOs in the capital city. To validate the findings from the reports and interviews, a field visit was undertaken to get first-hand knowledge of the changes in all project schools in the Bulgan aimag (province) and to interview key stakeholders. The scope of this evaluation also includes examining the linkage between the 3 TAME project and the World Bank’s Education Quality Reform Project’s (EQRP) school governance component. Additionally, reference is made to the schools-component of the Mainstreaming Social Accountability in Mongolia (MASAM) project, which was also implemented by the World Bank around the same time as the TAME project. Results Using the various implementation modalities and making the necessary mid-term corrections, the project achieved the two key project outcomes (i) improved stakeholder engagement and oversight in target schools and (ii) increased transparency of budgeting and procurement in the education sector in target aimags. Significant contributions were made to improving stakeholder engagement and oversight in target schools. By managing the engagement process carefully and diffusing the initial concerns of the schools and regional authorities, trust was gained to deepen the process. The PTAs, once established, demonstrated their value by making necessary school repairs and aiding teachers through the preparation of teaching materials, before addressing the psychological and governance needs of the schools. The formal linkage between component one of the TAME project and the EQRP was pivotal in building the credibility of the PTA process with the authorities. This increased support for the TAME project which in turn aided the EQRP in achieving one of its objectives: improving the governance of schools. The quantitative and qualitative evidence gathered confirms the achievement of the project objectives of using collaborative social accountability processes to achieve better school governance, budget and procurement transparency, and thereby better service delivery to schools. Based on stakeholders’ feedback from the regions, much was achieved by changing the system from a top-down vertical system of information and financial flows and reporting to a more open and accountable system. A major impact on the system was the change in the physical environment of the schools brought about through the use of the Good School Support Tool (GSST). A gamut of services was provided by the state, local officials, donors, and PTAs to upgrade physical infrastructure and utilities in the schools, including building a new school. To many parents and students, improvements in facilities meant an upgrade in the quality of education. Through the Budget Trust Tool (BTT) process, details on budgets became clearer to parents and teachers, more transparent, and administrators were made accountable for their use. Budget bulletin boards have carried over into the post-project period at all the project schools. Sustainability and Scalability By most accounts examined in the report, the TAME project results are seen to be sustainable over the medium term. The 31 PTAs and the National School Parent Teacher Association (NSPTA) are outcomes of the project and are still functioning post-project. This is the Most Significant Change (MSC) and impact brought about by the project for improving school governance, delivery of school services and student well-being. During the COVID-19 period, PTAs have been actively working with schools and teachers to help students with distance learning. Parents have 4 volunteered their time, provided money and in-kind contributions to make positive changes in the school environment. PTAs have expanded efforts to mobilize funds for school projects through the Local Development Fund (LDF) of the World Bank’s Sustainable Livelihoods Project (SLP). The Ministry of Education and Science (MECSS) and the Ministry of Finance (MOF) have both noted the contribution of the PTAs and indicated support for collaborative social accountability principles to be used for better service delivery of education services. The best way would be for MECSS to use the TAME lessons to scale up the establishment of PTAs in the non-project soums of the eight aimags and non-project aimags using the EQRP as the driving force. All the aimags in the Zoom focus group discussions and key informant interviews advocated for this approach. A version of the GSST has been used by MECSS to determine gaps in school infrastructure. 5 Contents Acknowledgements ....................................................................................................................... 2 Executive Summary ...................................................................................................................... 3 Acronyms and Abbreviations ...................................................................................................... 7 1. Introduction ............................................................................................................................... 8 2. Methodology .............................................................................................................................. 8 3. Conditions at Project Entry ................................................................................................... 10 4. Project Strategy....................................................................................................................... 13 5. Project Results ......................................................................................................................... 23 6. Sustainability of Project Results and Process ...................................................................... 45 7. Lessons and Recommendations ............................................................................................. 49 References .................................................................................................................................... 58 6 Acronyms and Abbreviations AFE All for Education (National Civil Society Coalition) Aimag Province Bhag District BTT Budget Trust Tool CPS Country Partnership Strategy CSA collaborative social accountability CSOs civil society organizations EQRP Education Quality Reform Project FGD focus group discussion GIC Globe International Center GOM Government of Mongolia GPSA Global Partnership for Social Accountability GSST Good School Support Tool IAAC Independent Authority Against Corruption ICR implementation completion and results report ISR implementation status and results report KII Key Informant Interview LDF Local Development Fund MASAM Mainstreaming Social Accountability in Mongolia MECSS Ministry of Education and Science MNT Mongolian Tugrik MoF Ministry of Finance MoU Memorandum of Understanding MTR mid-term review NSPTA National Student Parent Teacher Association NGOs non-governmental organizations OSF Open Society Forum (Mongolia) PSC Project Steering Committee PTA Parent Teacher Association SAcc social accountability SLP Sustainable Livelihoods Project Soum County ToA Theory of Action 7 1. Introduction Social accountability can be understood as “an approach to governance that involves citizens and civil society organizations in public decision making and in holding government accountable for its actions especially in the management of public resources” (Bousquet et al. 2012). The Global Partnership for Social Accountability1 (GPSA) supports approaches to social accountability that are collaborative and problem-driven: “when civil society actors adopt non-confrontational constructive strategies aimed at collectively solving problems and delivering specific results.” In late 2014 the GPSA approved the Transparency and Accountability in Mongolia Education (TAME) project through a $650,000 four-year grant. The project’s goal was to “strengthen civic engagement in the education sector to improve transparency and accountability of budget and procurement processes for the delivery of better-quality educational services in central and western Mongolia.” It was designed around two main pillars: the first to promote civic engagement at the school and district level and the other, that focused on budget and procurement transparency at the whole provincial level. A third component dealt with the knowledge and learning generated from the project. This report constitutes the final evaluation of the TAME project. The report is outlined as follows. The next section describes the methodology used. The section after that deals with the project background and conditions at project entry. This is followed by a focus on the project’s strategy, components, and processes employed to achieve project objectives. The section also explores how the GPSA’s Theory of Action (ToA) was adapted and applied to the governance of the Mongolian schools in the provinces. The section after that presents the quantitative and qualitative evaluation results, and this is followed by a discussion on sustainability of the project results. The final section offers an analysis on lessons learned and recommendations from the project. The success of the project measured against the GPSA’s ToA indicates substantial benefits both in financial and social terms despite difficult entry conditions. 2. Methodology 2.1 Evaluation Objectives The objectives and goals of this final evaluation of the TAME project are the following: 1. Assessing project strategy in relation to its intended outcomes and impacts, including how the PTAs were used for better service delivery and as catalysts for school governance 1 The GPSA is a multi-donor trust fund established in 2012 and contributes to the citizen engagement agenda by providing strategic and sustained support to CSO’s to reflect the voice of the beneficiaries, promote greater transparency and accountability, and achieve stronger development results. 8 reforms. Appraising the Program Theory of Action and its linkage to GPSA’s Theory of Action 2020 while paying attention to context and process.2 2. Evaluating the project’s effectiveness and results, including social return on investment and spillover effects. Examining how capacities were developed and how the project strengthened service delivery and analyzing variations across target areas. 3. Appraising the sustainability of the project, including risks to project results. 4. Drawing out lessons and learning from the project for the GPSA, the World Bank’s Education Quality Reform Project (EQRP) and other agencies on school reform efforts using PTAs as catalysts for improving the delivery of educational services and school governance. 2.2 Scope and Methodology of Evaluation This evaluation adopts a mixed-method approach to assess the four evaluation objectives above. Project results (outputs and impacts) are based on quantitative project data and qualitative observations and feedback from groups of project stakeholders. The quantitative data were obtained from project updates submitted to the GPSA by the lead grantee (Globe International Center), from the co-lead grantee (All for Education), World Bank project supervision reports and the final project Implementation Completion and Results report. The qualitative findings were derived from Zoom-enabled focus group discussions (FGDs) from all project schools and selected officials in the eight provinces.3 All interviews were based on semi-structured questionnaires. Further, a reasonable number of face-to-face key informant interviews (KIIs) were also conducted with the Project Steering Committee (PSC) and national CSOs in the capital city. To validate the findings from the reports and interviews, a field visit was arranged to get first-hand knowledge of the changes in all project schools and interview key stakeholders in the Bulgan aimag (province). Bulgan aimag was recommended to the evaluation team by the PSC as project results from this province had caught their attention. The scope of this evaluation includes examining the linkage between the TAME project and the World Bank Education Quality Reform Project’s (EQRP) school governance component. Additionally, reference is made to the school component of the Mainstreaming Social Accountability in Mongolia (MASAM) project, which was also implemented by the World Bank. The TAME and MASAM projects were initiated around the same time using social accountability tools to improve school conditions. Although largely similar in terms of objectives, the Theory of Action and the delivery models of both projects differed.4 2 According to the Terms of Reference the evaluation was encouraged to use causal analysis in reaching conclusions. 3 In total, there were 198 respondents in the Zoom interviews from all the thirty-one project schools. Another thirty-eight interviews were conducted face-to- face in Bulgan by the evaluation team during the validation visit. 4 While the TAME project had worked on specific delivery models linked to an articulated TOA, the MASAM project was designed on “aspirational goals” of community participation using a few social accountability instruments (participatory assessment, stake holder satisfaction survey and participatory school model) (Mainstreaming Social Accountability in Mongolia Project, 2020). 9 3. Conditions at Project Entry Mongolia is a unitary state with one central and three local level governments. It has 21 administrative provinces (aimags) and 330 counties (soums), and a capital city with its own status. Although large in terms of territory (about three times the size of France) it has the lowest population density in the world, making service delivery difficult and costly. The TAME project was designed to improve collaborative social accountability (CSA) practices in the delivery of education services in the central and western provinces5 in Mongolia. The aim was to jump-start civil society engagement in selected schools to improve the transparency of school budgets and procurement practices in order to improve service delivery. The aimag chosen (Uvurkhanghai, Khuvskul, Bulgan, Bayan-Ulgii, Khovd, Uvs, Arkhangai and Sukhbaatar) are home to a number of minorities such as Kazakhs, Tuvas, Tsaatan and Buryats.6 The challenge of delivering educational services to these regions and minority groups is compounded by their remoteness, with distances ranging from 500km (Arkhangai) to 1,700km (Bayan-Ulgii) from the capital city. Assessments done by the World Bank in 2017 (World Bank 2018a) revealed learning gaps between children from ethnic minority backgrounds and their Mongolian peers. The overall underperformance of students in Bayan-Ulgii aimag was even starker. Kazakh students performed worse than their non-Kazakh peers. Figure 1 shows the spread and location of the TAME project schools, as well as schools targeted by the MASAM project. In rural regions and herding communities, parents must hand over guardianship of their children to the school authorities; they entrust schools with the learning and well-being of their children, most of whom are housed in dormitories. There was little interaction or engagement between school authorities and parents in these regions prior to the project. School councils (with a tenuous legal status) only have a consultative function in schools with limited participation from parents and teachers. Most parents had no mechanism for channeling their views to school authorities or teachers and feared doing so would single out their children for adverse treatment. Teachers could not feedback on their principals, since principals had the discretion to influence teachers’ salaries (World Bank 2006). 5 One province, Sukhbaatar, is in the eastern part of Mongolia. This is home to minorities that belong to the Uzemchi, and Dariganga groups. 6 Other groups include the Ulud and Durvud. 10 Figure 1: Location of TAME Project and MASAM Project Schools Mothers and fathers of schoolchildren had little knowledge about school budgets, how they were allocated, managed and where the monies were spent. Few parents and teachers understood the complex per capita school budget formula7 used to allocate funds to schools. Teachers knew little about how they were evaluated and even less about how their salaries were derived. The top-down education system was teacher-centered and heavy on rote learning8. Students’ welfare and psychological needs were given lower priority. Teachers and parents had little training in discipline or listening to the emotional needs of students. Schools and dormitories were in disrepair, with most of them needing new buildings, repair of facilities and renovation. There was also little parent and societal interaction with district mayors, aimag governors and education departments on the funding of school services. The exception was their engagement in the LDF under the World Bank’s Sustainable Development Project. The local governance framework combines state management and local governance principles. Local assemblies (hurals) are elected directly, and governors appointed by local assemblies. This 7 The school budget formula is derived as follows: teachers’ salary + social contribu tion of employer + teacher's salary of primary class students + (primary class students * 4% only for Bayan-Ulgii province because of distance or uniqueness) + social contribution of employer per primary class student + (primary class students * 4% for only Bayan-Ulgii province because of distance or uniqueness) + other variable costs + (other variable costs * 3% only Bayan-Ulgii province because of distance or uniqueness) + (lunch cost student per-day 600 Tugrig *255 days) + (dormitory cost per student including food per day 2315 Tugrig * 255 days). Source: MECSS 8 The term refers to learning or memorization by repetition, often without an understanding of the reasoning or relationships involved in the material that is learned. 11 is in consultation with the Prime Minister in the case of aimags, and in consultation with aimag governors in the case of soums. This power structure also involves the appointment of school principals by the governors at both aimag and soum levels9 and often creates the need for school management staff to be on good terms with regional authorities during the budget preparation and approval process. Local civil society has had a hard time understanding the intricate cycles of top- down and bottom-up budget matching processes. As the TAME project was being designed in 2014, the Glass Account Law was passed, providing the legal space for civil society to examine budgets at all levels of government and engage with authorities on the use of public funds.10 While space existed for CSOs to function, enabled by legislation, the capacities of these organizations varied. At the capital city level, there has been a vibrant group of CSOs albeit with varying interests and capacities. At the aimag and soum levels, the numbers and capacities of the CSOs diminish, especially those that relate to public policy review and accountability. Mongolia’s Open Budget Index (OBI) was 51 out of 100 in 2015 at the TAME project start-up, below the cut-off point of 61 which indicates a country is publishing sufficient material to support informed public debate on the budget.11 The public participation component was much lower at 19. The accountability measures produced by V-Dem place Mongolia in the lower third of the scale for vertical, horizontal and diagonal accountability12 with diagonal accountability being above the other two related measures. Based on detailed research done by the Open Society Forum Mongolia (OSF), the local budget transparency index around the project start-up period indicated that none of the provinces produced citizens budgets. At the national level, there have been recent moves to produce such reports, however they have not been produced consistently or at the relevant times for civil society feedback. 9 Whenever there is a change in the government at the central level, personnel changes accompany that change from the relevant ministry all the way to the level of school principals in the case of education. 10 The key feature of the Glass Account Law is that it obliges all government agencies and legal entities with state involvement to make information on budgets and financial matters, including the utilization of finances, available to the public. Many of the reporting requirements imposed on government agencies were already in existence under the Law of Mongolia on Transparency of Information and the Right to Receive Information and the Law on the State Budget. 11 By 2019, the country had made some changes to move in the right direction and the overall OBI reached fifty-six. 12 Vertical accountability allows citizens to hold governments and politicians directly accountable. Horizontal accountability is ensured by oversight, checks and balances between different state institutions preventing abuse of power. Diagonal accountability on the other hand implies that media and civil society need to hold the government accountable through means such as, spread of information, publicity and other forms of engagement. 12 100 91 90 80 70 62 60 58.5 55 53 50 50 46.3 45.9 45 44 40 38.1 37 34.5 34.6 35.1 31 30 22 22 22 20.8 20 17 17 17 17 16 15 15 10 8 8 4 4 5 0 0 0 0 0 0 0 0 0 Arhanghai Uvurkhangai Sukhbaatar Khuvskul Bulgan Bayan-Ulgii Khovd Uvs Budget proposal Citizen's hural meetings Citizen's budget Local development funds Overall assessment rating Data source: Open Society Forum Figure 2: Local Budget Transparency of TAME Project Aimags by Percentage (2016) As illustrated in Figure 2, there were significant variations across the project aimags in terms of the openness of local assemblies and budget discussion to CSOs and citizens. Khuvskul was ahead of the other seven aimags with an index of ninety-one, followed by Arkhangai and Khovd. Bayan- Ulgii had the lowest rating of five (Open Society Forum Mongolia 2016). When all sub-indices of budget transparency (CSO involvement in budget proposals, Citizen Hural meetings, Local Development Fund, and production of Citizens Budgets) were taken into consideration, the overall index still placed Khuvskul top, followed this time by Uvs and Sukhbaatar, but with Bayan-Ulgii remaining last. If CSOs' engagement in budgets and budget monitoring in aimags has been weak, procurement oversight has been weaker. Although the Public Procurement Law allows the participation of representatives of civil society and PTAs in bid-evaluation committees, this practice has not made much headway. 4. Project Strategy It is instructive to note that the TAME project had articulated its Program Theory of Action which envisioned that “if stakeholders—government, school administrators, parents, teachers, and students—were more constructively engaged in education services and oversight, then providers of those services would improve their performanc e and be more accountable.” The assumption was that relevant stakeholders would be constructively engaged in educational services and 13 oversight of those services to improve the delivery of school services. This certainly was not the case when the project started, where there were minimal interactions among the stakeholders. Through the project, new forms of engagement were initiated, fostered, and incrementally expanded among the stakeholders at the school level and beyond. As noted in section 3, the pre- project context illustrated the disconnect and gaps in the educational delivery chain, especially to the rural regions of Mongolia. Furthermore, the Project ToA has to be seen in the broader context of the realities and drivers of the political economy and the dynamics of interactions between structural factors, institutions and stakeholders in the education sector of Mongolia 13. These country-specific drivers can either facilitate or hinder better education service delivery and oversight. They can also condition how the project results are sustained. Figure 3 illustrates how these drivers can interact to affect the oversight and delivery of educational services in the rural schools of Mongolia. Quality of Educational Services at PTAs Engagement MOES MOF Figure 3: Political Economy Analysis: TAME Project 13 The political economy analysis that is used here follows the broad framework developed by Fritz and Levy (2009; 2014) and is adapted to the particular context of the education sector in rural Mongolia. This type of “systems thinking” and analysis of the political economy environment beyond the project level is also advocated by the Global Delivery Initiative. (See Global Delivery Initiative n.d) It also stresses the need for adaptive approaches to projects. In addition, the World Development Report of 2018 emphasizes that power asymmetries can undermine policy effectiveness; and the unequal distribution of power in the policy area can lead to exclusion, capture and clientelism. (See World Bank 2018b). 14 By and large, the structural factors as described in section one have long conditioned the pre- project power dynamics and educational performance of minorities in the far-flung rural areas. Many factors affect the low performance of ethnic minorities at schools. Some of these factors include the need to master two languages at an early age, isolation from other families and groups in the same soum, and poverty.14 The quality of educational services, from buildings and facilities to teaching materials tends to be lower. The aimags rely on the local hierarchical power structures for the delivery of social services. This top-down paternalistic approach of the pre-transition era has been embedded into the institutional policies and structures right down to the soum level. The governors of aimags, mayors of soums, and heads of districts have considerable authority to influence school activities. Weak stakeholder capacity of parents, local CSOs and even the local media in understanding educational issues coupled with the informal norms of not questioning those in authority, results in the low quality of service delivery. Within the school environment, principals, managers and teachers wield considerable power. Parents have little knowledge about school budgets, school governance and what happens within the school environment. Rural citizens, parents, and students accept whatever services are given. Key institutions affecting the functioning of schools are the Ministry of Education and Science (MECSS) and Ministry of Finance (MoF). These two ministries coordinate closely to prepare annual budgets, oversee their spending and receive annual reports on results. All school-related policies are within the purview of MECSS and its branches in each aimag. Since there is little autonomy in school budgets (due to the fixed per capita budget formula) seeking additional financing beyond the soum seems to be a necessity. Very small school expenditures for events, fixing furniture, etc. are provided by soum mayors. In some cases, they advocate for the community to generate proposals to fix school facilities through the LDF.15 Financing larger amounts requires approaching the aimag branch of MECSS and the governor’s office and, as initiated by the TAME project, advocating school financing at the provincial assemblies (hurals) or even the national parliament. Oversight of budget spending and procurement is the purview of MoF, the Independent Agency Against Corruption (IAAC) and the National Audit Office (NAO). Although there are regulations enabling citizens to submit feedback on budget spending and procurement to the authorities, through Citizen’s Oversight Councils, not much has happened either by citizens or the regional CSOs. There are however initiatives by some national CSOs such as Open Society Foundation and Transparency International Mongolia (TIM) to scrutinize sectoral budgets and expose budget and procurement shortcomings.16 14 The poverty rate in Mongolia at the start of the TAME project was close to 30%. Poverty is highly associated with children, and the incidence of poverty is higher in the rural areas. Poverty affects children’s social and emotional stability . Malnutrition impacts their cognitive abilities. In the Zoom interviews described later in this report, some children mentioned that they faced discrimination from peers, since they lacked proper clothing or other school related materials. See https://www.worldbank.org/en/country/mongolia/publication/mongolia-poverty-update. 15 These are funds allocated to each county based on the community projects, proposed and prioritized by the community each year. The World Bank pioneered this idea through its Sustainable Livelihoods Project in 2001, which is now in its third phase of implementation. 16 Over the past decade there have been other CSOs formed to pursue social accountability initiatives and budget oversight. Examples are the Partnership for Social Accountability and Citizen Oversight of Budget Networks. (See World Bank 2013) 15 Thus, the engagement of stakeholders (as outlined by the Project ToA) was seen to be crucial in overcoming the barriers imposed by structural and institutional factors on the one hand, and within the local school environment on the other, to overcome the poor quality of educational services in rural schools. To enable engagement at the school level, the project strategy was to create PTAs as an intermediary step for responsible collaboration with school and relevant local institutions. Since parents have a direct interest in the well-being of their children, the strategy of giving them the voice and a platform paid dividends in terms of results achieved while minimizing the influence of structural factors. While it was a critical step, it was not sufficient to ensure better educational services in the aimags. The engagement had to reach out beyond the school to the aimag level. For this purpose, the project mustered the support of CSOs to engage with aimag-level authorities and hurals on educational budget issues. The design of the TAME project followed the GPSA’s principles of supporting processes at the “intermediate governance and managerial levels (counties and provinces) through which policies are implemented and services delivered” (GPSA 2020). As will be shown in the results section, this is where the results and impacts were visibly demonstrated and taken note of by MECSS and MoF. 4.1 Project Details The four-year TAME project, supported by the GPSA grant, was implemented by the CSO Globe International Center (GIC) in co-partnership with the CSO All for Education (AFE). GIC17 is well known in Mongolia for its work on democracy, governance, media freedom, transparency, and fair elections. It maintains global links with democracy research institutions and does frequent electoral updates locally. It proposed the specific topic of the project and the regions for interventions after having worked with the minority communities there on social and governance issues. Its partner, AFE, has considerable expertise on educational issues and is headed by former MECSS experts.18 GIC undertook the overall management of the project, liaised with ministries, and formed the Project Steering Committee (PSC) comprising of MECSS, MoF, and IAAC. Having obtained project support from MECSS and MoF, GIC prepared 41 MOUs with schools and agencies as entry points for engagement on the project. It also implemented component two on budget transparency, jointly with 21 national CSOs. AFE managed and implemented all the PTA-related activities (with 17 core CSOs) in the eight project aimags under component one. The project’s aim was to “strengthen civic engagement in the education sector to improve transparency and accountability of budgeting and procurement processes for the delivery of better-quality education services in western and central Mongolia” (Ali 2019). The project had three components. The first component focused on forming PTAs in 28 public schools in eight targeted aimags and apply social accountability tools to monitor school performance at the soum level. Since the objective was to strengthen civic engagement through PTAs, three indicators were 17 See https://www.gic.mn/en/ for details on GIC’s range of activities on governance. 18 See https://www.facebook.com/AFEMNG/ for details on the AFE coalition of CSOs that aims to redress “education inequality” issues . 16 formulated: percentage of stakeholders at project schools engaged in planning and oversight of school-level processes; the number of PTAs established, and the number of recommendations developed to address priority policy impediments. The second component was aimed at strengthening the capacity of CSOs and the media to monitor budgets and procurement processes in the education sector at the aimag level. In the case of component two, the objective was to improve the transparency and procurement processes beyond the school level. The indicators were the percentage of stakeholders participating in CSO initiated budget monitoring activities for education, the number of meetings of the local assemblies observed by CSOs, and the number of recommendations and policy actions to address priority policy impediments. The third component was designed to capture lessons and knowledge from implementation processes to support of the first two components. For component three the indicators were the number of case stories, tools other knowledge products developed (Ali 2019). 4.2 Project Implementation Strategy A six-step process was employed in component one to establish PTAs as vehicles for better school level governance. A core implementation team was constituted by AFE with instructions from GIC that internalized the objectives of the project. This team prepared for the advocacy meetings and charted out the implementation plans. Following this, a series of regional workshops were carried out for the engagement of broad stakeholders (local CSOs, parents, local communities) on project objectives and education issues. The next phase was the training of local facilitators (trainers) to encourage the establishment of the 28 PTAs in the selected provinces. Teams were formed for each aimag and the targeted schools. They carried out regional advocacy workshops on the benefits of the PTAs and how they could be established; built the capacity of each school team to form a PTA and formulate the necessary regulations (by-laws); trained them on their board functions; built the capacity of the newly elected boards; and facilitated peer learning and regional learning events for soums to learn from each other. The project website19 disseminated the lessons of all the activities. At the school level, initial meetings were held separately with school principals, teachers, and parents. This was then followed up with joint meetings between these groups, introducing the PTA concept. Next a core group of parents and teachers were trained on the PTA establishment process, holding elections, and preparing by-laws. The core group organized themselves as sub-groups to address these three tasks. When a PTA was set up another sub-group began the membership drive by reaching out to parents and teachers by various means of communications. Once the PTAs were established, they began assessing the school with the Good School Support Tool (GSST). This self-assessment tool was carried out by a multi-stakeholder group (parents, teachers, non- teaching staff, students, and even local CSOs and businesses). The GSST was essentially a 19 See http://www.sainsurguuli.mn/newsletter. 17 Community Score Card (CSC) which assessed three aspects of the school environment: physical, psychological and governance, identifying problems, issues, shortcomings in each area. Cases were identified with an analysis of their causes, followed by recommendations for change. The conclusions of the assessment were then disseminated to all stakeholders, including those from the authorities and media. The PTAs produced 18 recommendations (a required project indicator) focusing on policy reforms required at schools. These were reviewed by MECSS and MoF. Of critical importance for the momentum of PTA establishment was the timing of the EQRP’s formal support for the PTAs, including grants for their initial functioning. GIC and project staff proposed that component two should be implemented through CSOs with the use of the Budget Trust Tool (BTT). A five-step process was followed. A core implementation team of 21 Ulaanbaatar-based CSOs was constituted. This team fleshed out the broad engagement objectives and identified the target stakeholders to engage. MoUs were signed with regional authorities and used as entry points for the engagement on budget transparency and procurement in the education sector. Advocacy efforts disseminated information on sector budgets. Transparency issues were highlighted by making Glass Account, Budget and Information Transparency laws available. Next a group was selected to receive capacity building training on the BTT, how to use it to identify issues, cases and propose recommendations for changes. The team prepared the BTT and then tested and revised it for wider use. The BTT was rolled out in 29 districts by teams led by GIC and CSOs and was used to highlight issues with the education budget and procurement at both the school and regional levels. The training was repeated in most districts as the concepts were initially difficult to grasp. This was reinforced with peer learning sessions, budget open-hour (public hearing) sessions, outreach to administrators and elected representatives. Provinces that completed the exercise produced thirteen recommendations for policymakers which were discussed with the central ministries and regional authorities. The BTT contained a set of 12 sub-tools to examine the annual budget and procurement execution processes. Checklists were prepared to assess if transparency provisions in the laws and regulations were adhered to. Where transparency was lacking, disclosure of information was requested from the relevant regional authorities and schools according to the provisions of the specific laws. The CSOs together with local stakeholders also attended the aimag hural sessions when the aimag budget was tabled and discussed. Having followed the annual budget cycle that begins in April and ends in December, the CSOs prepared their analysis and recommendations for dissemination at a budget open hour, attended by the media. It is instructive to note that the BTT opened up the awareness of many parents, principals, and local administrators to the significant gaps in the school infrastructure financing gaps. The exercise also emphasized the opportunity for parents to advocate for school financing needs through the LDF allocated to each district. 18 4.3 Project Adaptation Strategy to Match Implementation Capacity It became apparent to the GPSA that the strategy of simultaneous implementation of components one and two in all eight provinces overwhelmed the capacity of the lead grantee and its partners during the first year of project implementation. Given the capacity issue, two adaptations were made. The original rollout plan for project activities was not deemed feasible, given the remoteness and geographical spread of the project areas. In late 2015, the first phase of the restructuring took place. A phased-in implementation schedule was introduced to enable roll-out of component one and two project activities during the second, third, and fourth years of the project life (2016, 2017, 2018). Other adjustments during this time included streamlining the results framework indicators and their targets. The second restructuring was done in late 2017 after the project’s mid -term review to increase the likelihood of achieving the Project Development Objective (PDO). This involved scaling up the PTA component by building synergies with the EQRP. Two components of the EQRP were relevant to the PTAs. One was the school grants component, and the other was the objective of improving school-level planning. This was a significant project adaptation since it brought benefits to both projects. By linking the PTAs to the EQRP’s objective of improving school-level planning, the EQRP could show how social accountability mechanisms under the TAME project could be mainstreamed to improve the performance of schools. To formalize this linkage, the EQRP made available grants to the PTAs under its school grants component. This effort emboldened the PTAs and gave them more credibility in the eyes of the officials. The second restructuring also encouraged PTAs to explore alternative avenues for engaging in the education budget process beyond the school level into local governance structures (Ali 2019) 20 4.4 Linking Project Implementation and Adaptation Strategy to GPSA’s Theory of Action The GPSA’s ToA envisioned that “if stakeholders—government, school administrators, parents, teachers, and students—were more constructively engaged in education services and oversight, then providers of those services would improve their performance and be more accountable.” Thus, it was assumed the appropriate set of stakeholders would be constructively engaged for better educational services. This would in turn lead to improved oversight, accountability and service delivery. As noted in section 2 of this report, one of the goals of this evaluation is to also use the insights from this project to verify the GPSA’s ToA21 and philosophy (GPSA 2020). The GPSA’s ToA envisages that “if both government and civil society had the experience and capacity to problem solve together, then government processes in many countries would lead to meaningful engagement between citizens and governments for more effective policy reforms, improved service 20 Also recommended was the scaling down the budget monitoring exercise and discontinuing the community radios. As it turned out, the budget monitoring exercise was rolled out in all the eight aimags but could not be completed in time in two provinces (Uvurkhanghai and Sukhbaatar). 21 The overall problem statement of the GPSA TOA emphasizes the following: government processes in many countries lack meaningful engagement between citizens and governments that lead to more effective policy reforms and improved service delivery, and that both governments and civil societies lack the experience and capacity to problem solve together. 19 delivery and ripple effects beyond the grant .”22 This part of the evaluation assesses how and to what extent the outcomes of the TAME project follow the predictions of GPSA’s T oA. Collaborative Social Accountability In recent years the GPSA has been supporting a new generation of CSA projects for better services, sector governance and accountability of governments. This is done by engaging communities, citizens and civil society groups to collectively solve problems of service delivery. The strategy complements country systems, the work of agencies that manage the public sector, and service delivery chains by using community driven actions. New information, ideas and resources are obtained through the synergy created by the efforts of civil society and coalitions in the public sector. Effectiveness The aim of GPSA grants is to contribute to country-level governance reforms and improved service delivery through more capable CSOs that can support CSA initiatives from the front line, to overcome implementation gaps. With GPSA support the grantees (civil society coalitions) develop capacities to engage collaboratively and meaningfully in service delivery processes, policy implementation and policy reforms. The GPSA leverages the World Bank’ s country and sector teams who assist in facilitating engagement with governments by identifying concrete opportunities for community inputs in programs, policy and service delivery processes. Adaptive Learning What will happen at project implementation is unknown in advance of the grant, but as more is learned from a project’s results and how they are measured, indicators may have to change, and components be adjusted. Adaptive learning and politically informed action by all stakeholders, including the GPSA, during the lifetime of a particular intervention and, critically, beyond the lifetime of a project, are important for the effectiveness of collaborative social accountability. Sustainability The GPSA provides grants as small experimental investments that have the potential for scaling up and become sustainable. Success for the GPSA at its program level is when elements and lessons from GPSA projects are taken up for action by the public sector beyond the individual GPSA project. Over time, with trust and joint experience, it is hoped that civil society, the public sector and development partners will seek to adapt insights from these collaborative processes into their activities. Governments and development partners could sustain the efforts by applying the lessons in additional localities, sectors, or in their policies well beyond the life of the GPSA project. 22 This formulation of the conditional statement is the author’s own interpretation of GPSA’s overall problem statement. 20 Figure 4: TAME Program Theory of Action The overall project design, implementation and adjustments made to achieve intended results mirror the GPSA’s ToA. Figure 4 depicts the stages of project preparation, implementation, and adaptation to achieve the intended results, and validates the GPSA’s ToA. It shows that stages one, two, three and four focused on the initial project strategy and preparation having identified the conditions at project entry of rural distance, disparity of educational outcomes especially of minority communities, and associating this with educational services delivered. Using the GPSA’s principle of CSA, the project targeted the issue of low stakeholder engagement in budget issues (transparency, accountability) and procurement as entry points. The political economy context, and the power favoring the authorities, was recognized. Given the central concern was schools with poor educational services, the schools were chosen as the entities to work with and PTAs formed as vehicles to institute change and deal with service delivery issues. This was one pillar of the project. The other was to engage with the local and regional institutions that affected educational policies and budgets. The foundational activities proved to be crucial in honing the appropriate social accountability tools to employ. As it turned out, both tools (GSST and BTT) and their adaptation to the local environment proved to be valuable in organizing, raising and addressing issues at the school level and regional level. This was done in a non-confrontational collaborative way by raising awareness of the issues, identifying and prioritizing the issues, and finding joint solutions to the issues with the proper institutions. Entry into the local schools and institutional system needs some form of legitimacy (that has been practiced for decades); this was 21 aided by the reputation of the World Bank and the MoU signed with MECSS. This MoU then formed the basis for the MoUs between schools and local authorities and the project. Building implementation capacity was certainly an area of concern. The original rollout plan was for concurrent implementation in all provinces. This approach had to be later modified given the remoteness and geographical spread of project areas and constraints on implementation capacities of the GIC’s partners and the overall grant size. Project restructuring (stages five, six, and seven) thus took place a year into the project and a phased approach to implementation was adopted. The results framework indicators were brought in line with the appropriate data to measure outcomes resulting from the interventions. The number of results indicators was rationalized and streamlined for easier monitoring. Instruments such as community scorecards and community radios were dropped from the design. The mid-term review was pivotal, allowing clarity of project objectives and results, alongside an adjustment of budgets. Two key outcomes here were the establishment of the PTAs and the formal linkage of the PTAs with the school governance component of the EQRP. The final stages (eight and nine) indicate the achievement of outcomes and impacts after the PTAs’ engagement levels were enhanced, CSOs’ collaboration with local assemblies improved, budget transparency augmented, and school facilities and services upgraded. These outcomes and impacts also contributed to higher-level objectives including the country’s objective of improving the quality of education, and the World Bank’s Country Partnership Strategy of engaging with marginalized and vulnerable groups. Thus, the GPSA’s principles of constructive engagement and adaptive learning were fully incorporated into the various stages of project implementation. Through constructive engagement, new collaborative models at the schools were established and significant resources were mobilized. As will be discussed in later sections, on top of being effective in achieving the project objectives, the project built CSO capacities, particularly engagement and implementation capacities. It also contributed to filling service delivery gaps at the front line. The partnership forged by the project with the World Bank teams on education, brought mutual benefits to the sector. The establishment and effective functioning of the PTAs and the NSPTA is the Most Significant Change (MSC) brought about by the project. These structures are seen to be sustainable over the medium term. The ripple effects of the TAME and MASAM projects and the SLP have been internalized by the government and are now being reflected in initiatives such as e-Mongolia and Smart Government that solicit citizen feedback. A number of donors are getting involved in supporting CSA initiatives, including the Swiss Development Corporation, Mercy Corps, Save the Children, and World Vision. Some of these agencies are now collaborating on the MASAM II project on the health sector. 22 Thus, the GPSA’s ToA envisaged at the beginning of the project and GPSA’s principles of using CSA and adaptive programming were all applied to bring success to the project. In the GPSA’s ToA success is determined by the extent the project helped project partners solve the intended problems. By the end of the TAME project, it had solved a number of problems. 5. Project Results 5.1 Achieving Quantitative Project Targets To achieve the overall project objective of engagement by stakeholders at the school and beyond to solve service delivery issues, the project had set intermediate quantitative targets to be achieved by components. For component one, the targets were stakeholder engagement levels at schools and districts; number of PTAs established; number of Good School Support Tool (GSST) training sessions delivered; recommendations adopted at the school level through the use of the GSST; and proposals to overcome education policy constraints. A large number of stakeholders were brought into the engagement process at the school and district levels. On average, over 130 stakeholders were involved in each of the 31 schools where the project was rolled out. The project exceeded the target of 28 PTAs, by establishing 31 PTAs. The three additional PTAs were established as a result of positive demonstrated benefits being noticed by neighboring schools in the same districts. In addition, the umbrella organization, the National School Parent Teacher Association (NSPTA) was established under the project to support the PTAs. Training on the GSST was done twice at each site so that the audience could absorb the contents. Through the use of the three modules of the GSST (physical, psychological, and governance evaluation) 84 recommendations were formulated and then adopted by the schools. The PTAs put forward 18 school policy recommendations (with most of them focusing on amending rules and regulations so that PTAs could get involved in the governance aspects of schools in the districts and beyond). Issues of school budgets and the dissemination of transparency laws to school authorities were also advocated. These were discussed with MECSS and MoF as part of this evaluation. While some of the recommendations were repeated (coming from different PTAs), it showed that the PTAs knew some of the policy issues, especially those on the budget formula to remote schools. Before the establishment of PTAs, parents were unaware of school budget allocations and how the budget was spent. With the PTAs, many of these issues even those at the granular school level were identified as discussed with relevant authorities. Component two intended to foster engagement by stakeholders with the authorities in power at the provincial level to advocate for the school budget and procurement transparency. To achieve this, the project intermediate targets were engagement rate of stakeholders, number of occasions project CSOs attended local assembly open hearings on education, number of school budgets monitored by project CSOs in the eight provinces, and number of PTAs that adopted action plans on 23 improving budget transparency and school services. In addition, there was: number of budget and procurement-related policy recommendations made using the Budget Trust Tool (BTT). Based on stakeholder attendance files submitted by GIC, around 80 stakeholders participated in the overall BTT exercise at each of the 31 schools. Variations across the aimags were to be expected given their relative distances and institutional capacities. Bulgan aimags engagement numbers stood out, followed by Khuvskul and Khovd. While the BTT was rolled out in all eight aimags, two of them (Uvurkhanghai and Sukhbaatar) had capacity issues23 in completing them on time as project closure approached. The CSOs that worked with the project made progressive inroads engaging with institutions responsible for education budgets. This was an incremental effort, with meetings initially hostile between the CSOs and institutions but becoming more conciliatory and collaborative.24 Meetings were held with aimag governors, soum mayors, Education Department heads, and most significantly with heads of regional assemblies where regional budgets are debated and approved. Thirteen sessions of the local assemblies on school budgets were observed by the project CSOs. The CSOs were allowed to attend and make their views known jointly with other stakeholders.25 The MoUs with the authorities initiated at the beginning of the project assisted in opening the doors to these officials and institutions. The CSOs carried out the school budget monitoring exercise in forty-eight project and non-project schools to gain insights on how budgets were allocated and divulged. Twenty-eight PTAs carried through action plans on improving budget transparency and school services. Finally, 13 recommendations on overcoming policy impediments to school budgets and procurement were drafted by stakeholders from the six participating aimags. Most of the suggested reforms centered around the need to enforce provisions in the Budget Law, Law on Information Transparency and Right to Information while making budget information more user friendly and disseminating Citizens Budgets in a timelier manner. Recommendations also argued for better definitions and adjustments to the per capita school budget formula that would enable schools to receive more funds. The BTT also had a module on the linkage of the school budgets to the LDF, and stakeholders advocated for revision of the regulations in the LDF so that the lead representatives are not given undue powers to re-rank citizens’ prior ranking of projects for funding. These were discussed with MECSS, MoF and the IAAC as part of this evaluation. The ministries, while appreciating the recommendations, felt that some changes were already being made to allocate more funding to remote schools by adjusting the per capita school budget formula. On the suggestion that citizens be part of the procurement evaluation committee, MoF advised that citizens should pass the necessary tests to sit on this committee. 23 The main CSOs appointed to co-deliver on the BTT with GIC in Uvurkhanghai and Sukhbaatar had time and organizational constraints. 24 As noted from the Zoom interviews. 25 These included parents and teachers. 24 Capturing learning and dissemination of project lessons is integral to GPSA-financed projects. For this purpose, the target for component three (knowledge and learning) was the number of case stories, tools, and other knowledge products developed and shared among the PTAs. Information dissemination was done by AFE, who updated the project website and Facebook page with project developments. During the project period, close to 210 updates were made on the project’s Facebook page. More targeted updates took place after the mid-term review of the project which emphasized the need to disseminate more demand-driven knowledge products such as: templates for writing to officials; lessons from peer learning activities; and ways to diffuse the power imbalance with principals and officials (GPSA 2017). Twenty-eight case stories, tools, and other knowledge products were developed and shared among the PTAs and 25 of those were videos.26 Information on the need for PTAs and how they were being formed in the regions were broadcast to citizens through local TV stations, newspapers, radio stations, and news websites. It is significant to note that AFE has continued to maintain the PTA Facebook and website even after project completion. 5.2 Spillover Benefits In addition to the four extra PTAs (including the NSPTA) that were established, there were spillover gains from peer learning activities that targeted the PTAs. As reported in the project ICR, “these activities were intended to build capacity to improve educa tional services by exploring and accessing different forms of local fundraising beyond the school level. These activities also empowered PTA members to engage in informed discussions about the planning and allocation of LDFs and how to monitor their use in meeting school needs” (Ali 2019). Project stakeholders have used the tools from the project to advocate for more funding through the LDF and recommended better procedures for the allocation of these funds through the established community voting process. During the project period, close to $185,000 was allocated to schools in the project districts from the LDF.27 Having become more aware of this source of school funding, PTAs have continued to lobby for school projects during the annual allocation of the LDF.28 Given the issue of low autonomy over school budgets, PTAs have engaged with local authorities to explore funding for capital expenditure through a number of other means. The ICR highlighted the case in Bulgan where the PTA, working closely with school and regional officials and lobbying for funding of their school, even reached the parliament. “In Bugat soum of the Bulgan aimag, a joint-school wide effort by the project school was organized to raise funds for the demolition of the old school buildings and the construction of new ones. Through media campaigning and extensive discussion and negotiation with public authorities, including Aimag Citizens’ Representatives, the case was ultimately brought to parliament and MNT 2.1 billion 26 See for example video of PTA activities. https://www.facebook.com/gantsetseg.ganaa.9693/videos/2629826360662178 . 27 Data from MoF website. The LDF guidelines place emphasis on households in each bhag participating in public meetings that determine projects to be financed. 28 The AFE reported that as of late 2020, PTAs in Bulgan and Bayan-Ulgii continued to lobby for LDF funding after the TAME project ended. 25 (approximately $796,000) funding was secured” (Ali 2019). Due to the interest and momentum that was generated by the project, local authorities worked with various agencies to fund school facilities from the central budget, local state budgets, the LDFs and donors.29 For the project period, schools in Bulgan soum managed to secure $2.72 million. Even more significant was Khuvskul aimag where the schools obtained funding of about $3.85 million. The third most significant funding was secured by Bayan-Ulgii aimag where the project schools gained about $1.92 million. This aimag was also noted for its extensive use of the LDF to fund small investments in project schools. Overall, the financing focused on the physical environment: new school buildings; school extensions; renovations; new/renovated dormitories; gymnasiums; art halls; fuel storage and sewage investments. Another spillover benefit of improved student well-being was derived from FGDs and will be elaborated in later sections. 4,500,000 4,284,640 4,000,000 3,500,000 3,000,000 2,720,000 Amount in USD 2,500,000 2,000,000 1,922,712 1,500,000 1,400,000 1,303,120 1,000,000 500,000 397,800 272,312 153,200 0 Arkhangai Bayan-Ulgii Bulgan Khovd Khuvsgul Sukhbaatar Uvs Uvurkhangai Aimag Figure 5: Aimag Investments in TAME Project Schools During TAME Project Sources: MECSS, MoF 29 These sources of funding are also aggregated into the aimag totals in Figure 5 26 5.3 Project Outcomes Using the various implementation modalities and making the necessary mid-term corrections, the project achieved the two key project outcomes (i) improved stakeholder engagement and oversight in target schools and (ii) increased transparency of budgeting and procurement in the education sector in target aimags (Ali 2019). As noted from the quantitative results, the project contributed significantly to improved stakeholder engagement and oversight in target schools. The project ToA called for the engagement of relevant stakeholders to achieve the outcomes. By managing the engagement process carefully and diffusing the initial concerns of the schools and regional authorities, trust was gained to deepen the process. The use of specific tools for engagement and the establishment of the PTAs also aided the engagement process. The PTAs once established, initially demonstrated their value making necessary school repairs and aiding teachers with the preparation of teaching materials, before addressing the psychological and governance needs of the schools. The formal linkage of component one with the EQRP was pivotal in building the credibility of the process with the authorities. This procedure added support for TAME and the project in turn aided the EQRP in achieving one of its objectives of improving governance of schools.30 The ICR confirmed that the project has contributed to increasing transparency in budgets and procurement processes in the education sector and that project schools have disclosed budget and procurement details to the schools and parents. “At the aimag level, with high levels of CSO commitment and incentive, the project facilitated spaces for dialogue between citizens and public sector decision makers to share information on education budgets and procurement” (Ali 2019). 5.4 Supporting Qualitative Evidence The qualitative evidence presented below is organized as follows. The Zoom FGD stakeholder results are discussed, followed by the results of the validation visit to Bulgan soum, and finally face-to-face interviews with the Project Steering Committee. The responses from the stakeholders were translated and edited for clarity and brevity. 30 The broad objective of the EQRP is to improve the quality of education for primary schools with particular emphasis on improving native language and math skills. The project has three components-- improving learning outcomes, pre-and in-service professional development of teachers and the implementation of school support program. A number of linkages between the EQRP’s school support program and the eventual tasks done by the PTAs can be readily identified. These include promoting social, emotional skills of students; encouraging individualized approaches for lagging students and for students living in dormitories; introducing enrichment programs for students; increasing instructional time for learning; improving school level planning functions and co-curricular activities. (See World Bank 2014). 27 ZOOM Interviews Election Process of a PTA Power Balancing Training Figure 6: Images from the Project and Evaluation Credits: Project Evaluation Team and National School Parent Teacher Association Given COVID-19 restrictions in Mongolia, qualitative evidence was gathered by the local evaluation team from project stakeholders including non-project schools that established PTAs with assistance from the project CSOs. Zoom-enabled FGDs were held with a large number of project stakeholders. Semi-structured questionnaires were sent to all participants with some details of the project to enable feedback. One hundred and ninety-eight respondents were interviewed from all project aimags, soums and schools. Despite some technical glitches, the Zoom FGDs were implemented successfully with the assistance of a local coordinator in each soum to organize meeting logistics. Mindful of the power imbalance between stakeholders, the interviews were staggered, with a manageable number of respondents at each session from the same cluster (PTAs, principals, officials etc.). In general, the feedback confirmed the quantitative outcomes and results obtained from the project documents and reports. Since the feedback was from clusters of stakeholders, the results are also organized by stakeholder groups to capture their viewpoints according to the three key evaluation objectives: engaging in social accountability, project results and benefits, and recommendations on the project. Stakeholders # % Aimag Governors 6 3% Aimag Education Department Officials 11 5% Soum Mayors 21 11% Students 62 31% CSOs / NGOs 14 7% School Principals 23 12% PTAs (Teachers) 31 16% PTAs (Parents) 30 15% TOTAL 198 100% Table 1. List of Participants in Zoom FGDs, May – July 2020 28 5.5 PTA Members, Students, School Principals One segment of FGDs focused on stakeholders from all the project schools. Sixty-one teachers and parents from all 31 PTAs joined the discussions which were carried out two aimags at a time. Twenty-three school principals and 62 students were interviewed separately so that they could talk freely. These three groups were all the ultimate beneficiaries of the project and formed the largest group in the FGDs (146 respondents). Their responses are summarized below. Engaging in Social Accountability Once the PTAs were formed, they held advocacy meetings with school principals, local officials and engaged with the media to publicize their work. They organized capacity-building initiatives for parents to improve their knowledge of the school curriculum and the learning process of students. PTA members engaged with the students to hear their concerns and even taught students how to use online learning methods. The engagement process between PTAs and the school and local authorities was by no means an easy one. Although the MoUs signed by the schools and officials provided entry points, the mutual trust had to be earned by both sides. Many school principals and local officials were worried that they were being checked and audited. The use of both GSST and BTT tools facilitated the engagement process. By using these tools, concerns at the schools were identified that had to be addressed jointly by all concerned parties. The focus on the physical learning environment and the efforts expended by the PTA members to repair school facilities won plaudits from the school authorities. With tangible results observed in the physical environment, the path was paved for the PTAs to engage in school governance and the psychological aspects of the learning environment. Through the use of the GSST’s school psychological environment module, PTAs and the schools brought to the fore issues that were not sufficiently engaged in the schools. Attitudes of teachers and parents have changed for the better in project schools; both parties cooperate on school issues and advocacy efforts with officials. Using the governance module of the GSST, PTAs identified issues and sent recommendations to the school management for inclusion in the annual school plans, particularly regarding school rules and budget transparency. Frequent interaction between the PTAs and school management teams was facilitated by the provision of dedicated PTA rooms in schools. PTA representatives are included in school councils and vice versa. “When faced with some major decision-making issues, it is most valuable to involve all school parents in large-scale advocacy work, such as sending formal requests, letters, and comments to the MECSS, MoF, the Government of Mongolia, and the State Great Hural (parliament). Teachers, students, and staff have become motivated to improve the overall learning environment.”31 With the use of the BTT, PTAs from the aimags engaged with the schools to adopt action plans to improve transparency and accountability of school services. The PTAs, using the BTT, have engaged with a number of state and non-state agencies to mobilize significant funds for their schools. 31 Head of Policy Unit, Local Education Department, Bulgan Soum. 29 Responses from 62 students from eight project aimags focused on the benefits they witnessed after the formation of PTAs. Most centered on having their voices heard, experiencing better emotional and social well-being, and facilities improvements enhancing their schooling and learning experience. Students talked positively about the formation of student councils which represented their views and provided new ways of communicating their concerns to the school authorities, PTAs and even to the larger regional audiences. Many were positive about the creation of better extracurricular activities such as eco-clubs, girls’ clubs, and journalist student clubs, as well as talent contests, essay writing contests and painting contests. Students expressed that these activities improved their emotional well-being. “The PTAs together with the police and student councils organized events to overcome peer bullying, prejudice and to discontinue playing computer games during school hours and this worked, with positive results.”32 Students saw marked improvements in school facilities, including improved boarding facilities in dormitories and drinking water placed outside classrooms. Some students however mentioned, “we still have difficult conditions at home, need winter clothing, have to deal with unethical behavior of parents and teachers, and the atmosphere at school is very cold.”33 Project Results and Benefits With the help of PTA members, the physical and learning facilities of schools were renovated, transport was provided to schools and utilities upgraded. PTAs began by changing the attitudes of teachers and students towards learning. They spent time explaining the Law on Violations and Child Protection to teachers so that teachers would be aware of the laws when disciplining students. Teachers’ methods of communicating with students improved once teachers were trained to focus on the needs of students. Parents began to pay more attention to the learning needs of their children. Trust in teachers by parents gradually increased. Schools drafted new dormitory rules in consultation with PTAs. The school management teams organized weekly meetings with students. It has become customary for school principals to provide performance reports to parents and officials twice a year. In the project schools, parents, teachers and students vote on school action plans and PTA activities are included in the plans. The PTAs, using the BTT, have engaged with a number of state and non-state agencies to mobilize significant funds for their schools. PTAs working with schools, have constructed noticeboards and posted school budgets for all school stakeholders to examine. PTA representatives are given a role in the preparation of school budgets. These results-oriented activities have empowered stakeholders to actively engage with officials at all levels on procurement and construction problems. Parents who have been concerned about the quality of the school lunch program started monitoring this aspect of school services. Many students expressed that the introduction of better extracurricular activities and contests, improved their emotional well-being. 32 PTA of school number one of Bulgan soum, Bulgan Aimag. 33 Students at Tuva Primary School Tsengel soum, Bayan-Ulgii Aimag. 30 Recommendations A school principal asserted that “if the World Bank supports the further development of this engagement, the quality of education would improve when there is a cooperation between the school administration and the continuous provision of open space for public participation in school affairs. This could become the best practice in all schools in Mongolia.”34 A PTA member was of the view that “transparency and accountability in the education sector will improve if PTAs are established in other schools in project aimags.”35 There were also suggestions that there is a need for PTAs to keep engaging with various institutions at the soum, aimag, ministries, the media and CSOs. Since the PTAs do not have enough financial resources to sustain themselves, they need to find ways of generating funds from World Vision, Mercy Corps, and international agencies. 5.6 Local Education Department Heads and Officials Engaging in Social Accountability Eleven directors and experts from the regional Education Departments also participated in the FGDs. These agencies had also signed the MoUs to participate in the project. The officials were of the opinion that with the project, a system has been created to increase the active participation of key stakeholders such as school administrators, teachers, parents, and trainers to improve the learning environment and increase the responsibilities of all parties. “ Cooperation between government and non-government organizations has improved”36 they reflected. The local Education Department officials recalled participating in regional and soum meetings on the project and discussing a range of education issues such as teaching materials, budgets and the learning environment. Project Results and Benefits Most of the officials acknowledged that the schools and the PTAs had success in improving the school physical environment and dormitory conditions. The officials mentioned that parents now have more access to schools and understand education issues including school budgets. While stressing their concern for education quality, they remarked that “parents and teachers are working together to monitor the quality of education. ”37 One official commented “the school physical, psychological, financial and governance environment from now on will be evaluated and reported to school administrators as well as to the soum and aimag administration. The results of this method have been very good. With the establishment of the PTAs, the project has been successfully implemented in the aimags.”38 “With the use of the BTT citizens are now more informed on the budget and realize that they can also participate in the budget process and with the Glass Account and Fiscal Transparency Law, budget and financial activities are made more open; purchases are becoming more transparent, and parents are beginning to monitor them.” 39 34 Principal of School Number Three of Bulgan soum, Bulgan aimag. 35 PTA of school, Ulaan-Uul soum, Khuvsqul aimag. 36 Head of the Education Department of Khovd aimag. 37 Official from Department of Education, Bulgan aimag. 38 Official from Department of Education, Bayan-Ulgii aimag. 39 Head of the Education Department of Khovd aimag. 31 The official from Khovd aimag asserted “even after project completion, we will implement the BTT methodology to ensure transparency in the education sector within the framework of social responsibility. Most importantly, cooperation between governmental and non-governmental organizations has been the most important benefit.”40 Recommendations Officials in the group maintained that PTAs in future should continue to cooperate with the government, NGOs and business organizations. “The TAME project, the Education Quality Reform Project, projects by World Vision are aimed at improving the quality of teaching and the learning environment. TAME is an important project that needs to be continued.”41 One official highlighted a need to educate citizens on the budget through the Lifelong Learning Center. 5.7 Governors and Mayors of Aimags and Soums The evaluation team managed to elicit the support of six aimag governors and 21 soum mayors to join the Zoom FGDs. Four sessions were held with this group to enable frank views to be exchanged. Their views and feedback on the project are also summarized under the themes of engaging in social accountability, project results and their recommendations on the project. Engaging in Social Accountability While the governors were involved at the higher decision-making level, it was the soum mayors who had frequent contact with project teams and PTAs. Following the establishment of PTAs, many mayors had participated in the project-sponsored bhag (district) meetings with local governments to advocate for PTA activities and to encourage parents to join the PTAs. Project Results and Benefits A mayor asserted that the project established non-governmental PTAs, these are very useful especially in the soums to bring teachers and parents together; there are many good examples of increased parental involvement in schools.”42 In Bayan-Ulgii aimag “the PTAs in Ulgii soum are very active in raising school issues. These PTAs were successful in raising school funding issues at the soum and aimag assemblies and had secured 205 million MNT (US$ 74,000) funding for school renovation and a gymnasium.”43 The mayors confirmed that parents who are skilled carpenters were encouraged to repair broken school desks, chairs and cabinets. Through the joint action of the PTAs and mayors, more schools are now connected to the central heating system and clean water and cameras are now installed in all soum offices and schools for school safety. “With the establishment of the PTAs, there have been many positive changes especially in the relationship between teachers and students.”44 “Although school councils are still in existence, they have 40 Official from Department of Education Khovd aimag. 41 Head of the Education Department of Khovd aimag. 42 The Mayor of Bulgan soum, Bulgan aimag. 43 The Mayor of Ulgii soum, Bayan-Ulgii aimag. 44 The Mayor of Ulaan-Khus soum, Bayan-Ulgii aimag. 32 included the representatives of the PTAs, and school information has been shared with them.” 45 Another key engagement result was the better understanding by these officials and their communities about their own education budgets, leading to their participation in LDF projects. “Citizen views are heard in the planning of the Local Development Funds and their voices are becoming more critical on access and quality of education and the need for budget transparency.”46 Mayors remarked that the project trained the NGOs and CSOs on how to prepare and use the BTT to improve the transparency of education budgets and procurement according to the legal provisions for budget transparency (Glass Account Law). “With public participation, citizens are more knowledgeable regarding budgeting, they know how to lobby for school funding and raise money. The BTT helped budgets to be spent more efficiently.”47 Recommendations The mayors stressed that “it would be beneficial to implement this project in other soums and remote schools.”48 According to the Governor of Bulgan aimag “Through this project, we realized that the involvement of professional organizations, NGOs, CSOs, parents and communities is crucial in addressing school issues”.49 5.8 Civil Society Organizations With the assistance of AFE, Zoom interviews were organized with 14 CSOs that implemented the project, most of them implementing component two.50 From the feedback received, the CSOs seem to have gained analytical capacity especially technical knowledge on GSST and BTT. They saw the value of the tools while engaging in the social accountability process. The CSOs “are getting more experienced in drawing results (from the tools) and planning next steps based on information received from officials and reporting these to regional assemblies.” 51 It does seem that the CSOs gained more experience in civic engagement especially on the methodology of engagement with officials, and experienced initial difficulties as expected. In the end, mutual trust was forged following the non-confrontational process of raising and solving budget issues. Engaging in Social Accountability Since most of the CSOs interviewed were involved with component two, they had engaged with the local assemblies for the first time in the regions. They encouraged public hearings and organized school stakeholders and others to attend and speak during the hearings, drawing issues from the BTT. Training was delivered on both tools and tailored to the absorption capacity of the audience. For most groups the training was repeated which also helped in forging cooperation 45 The Mayor of Ulgii soum, Bayan-Ulgii aimag. 46 The Mayor of Bugat soum, Bulgan aimag. 47 The Mayor of Bulgan soum, Bulgan aimag. 48 The Mayor of Tes soum of Bayam-Ulgii province remarked “it will be very effective to implement this project in other remote local schools in the future. I thank the World Bank.” 49 Mayor of Tes soum of Bayan-Ulgii. 50 In total there were seventeen core implementing CSOs that worked with AFE, and twenty-four CSOs with GIC on component two. These included branches of CSOs from the capital city. 51 Branch of “Liberal Women’s Brain Pool” (LEOS) NGO, Bayan-Ulgii. 33 between parents and teachers. To get the most of their efforts, they also engaged with media outlets to raise the citizens’ awareness on school budget issues. Project Results and Benefits The CSOs expressed that the benefits of the PTAs to the parents were apparent since they were given a voice to improve school facilities for their children. They maintained that “we raised the awareness of citizens to engage in the budget process. Together with the PTAs and the school authorities, we managed to erect bulletin boards disclosing school budgets and procurement details. We assisted the schools to raise funds through the use of the GSST and BTT exercises and oversaw the preparation of recommendations to overcome policy impediments to the transparency of school budgets.”52 The most active CSO in Bulgan (Women’s Association) was of the view that “the PTAs will continue with their activities even after the project is over, with the constant support and assistance from the soum mayor, school and the parents.” Recommendations The CSOs asserted that the PTAs should be established in Ulaanbaatar and not only in rural schools since the PTAs were a success in their opinion. For the next stage, they suggested selecting PTA board members who are younger and who can be more active in PTA activities. They professed that CSOs needed support to continue with the type of activities they had conducted. They remarked that there is still a need for further dissemination of the Glass Account Law and Transparency Law to open up budgets to citizens. 5.9 Bulgan Results from Validation Visit Bugat Soum School Meeting With the PTAs at Budget Transparency School Number One, Bulgan Noticeboard (School Soum Number Three, Bulgan) Figure 7: Images from the Bulgan Field Visit Credits: TAME Evaluation Team Mongolia imposed travel restrictions due to COVID-19 in March 2020 but relaxed travel to aimags during the national election period in June 2020. The local evaluation team took the advice of MoF (during the face-to-face interviews with them) to travel to Bulgan aimag during this period. The 52 The “Mongolian Trade Unions of Education and Science” CSO. 34 MoF officials were of the opinion that Bulgan had some of the best project results. KIIs were held with a sample of stakeholders involved in the project. Table 2 depicts the range of stakeholders interviewed. Thirty-eight stakeholders were interviewed, 30 of whom were also in the Zoom FGDs (held a month earlier). The team visited all project soums, all five project schools, and interviewed officials to validate the results of the project as found in project reports and from the Zoom FGDs. The visit was highly rewarding, making face-to-face contact with key players, champions and other stakeholders who had played a significant role in making this one of the most talked-about aimags in terms of impact and links with ministries created through the TAME project. Three project schools were in Bulgan soum, one in Bugat soum, and one in Orkhon soum. Two of these schools represent spillover benefits as they were not initially targeted by the project. The schools interviewed had a combined student population of 3,268 with 202 teachers. The responses are once again organized into three categories: engaging in social accountability; results and benefits of the project; and recommendations on the project. Since this was a manageable number of respondents, their responses are merged into the three categories. Stakeholders Number Teachers 2 Students 6 Soum Mayors 3 School Training Managers 4 School Principals 4 PTA Members (PTA Parent Coordinators, Teachers) 14 Aimag NGOs 3 Head of the Education Policy, Planning and 1 Administration Department of the Aimag Education Department School Librarian 1 TOTAL 38 *Number of stakeholders in Bulgan who were in 30 Zoom FGDs and also interviewed during visit. Table 2. TAME Project: List of Respondents Interviewed During the Field Visit to Bulgan Aimag, June 2020 Engaging in Social Accountability Members from the five PTAs were interviewed, mostly PTA board members. Parents were found to dominate the boards. Out of the 14 PTA members interviewed, 12 were women. However, the project has noted that more fathers are getting involved in their children’s education, attending meetings and contributing to in-kind efforts and fixing facilities at schools. Breaking the ice between the PTAs and the school authorities was not an easy process according to the PTAs. The MoUs between the project and the participating schools provided an entry point for discussion and collaboration, largely due to the reputation of the World Bank and the formal 35 support from the ministries. The objectives of the project were viewed with suspicion both by the schools and regional officials initially. As recounted by the principal of School Number Two in Bulgan soum: “In the beginning it was very challenging due to the top -down system of overall administration. The director of the school did not respect and did not work with the PTA. The parents of the school consist of government servants, the private sector, herders and low-income families. They were used to the top-down system and had no influence on the school.” Gradually the nature of the engagement changed through various training sessions for the PTA members, schools and officials. The use of the GSST helped the process by bringing together the various parties for self-assessment of the outstanding school issues. The relationship between the CSOs and the officials on the budget issues was no different. Officials, due to the top-down system, had little time for CSOs. It was only by using the MoUs and the various budget and transparency laws, that the CSOs convinced officials at various levels to engage with the project. Having demonstrated their tangible contributions to the physical improvement of the schools, the PTAs found that the school authorities and the local Education Departments became more trusting of their efforts as they began “changing the attitudes of those in the education system.”53 The teachers began communicating with students and their parents on school activities through their Facebook page with help from the PTAs. “There are now thousands of herder families involved in school activities, improving dormitories, and collaborating with bhag chiefs. Soum leaders have accepted working with PTAs and there is trust.”54 The PTAs have engaged beyond their soums and extended their reach to the aimag assembly, having taken up the funding issues for the schools through the use of the BTT. PTAs have also advocated for the rights of children with disabilities in public hearings. To extend their reach, they have used the local Herder TV to publicize project information, the GSST, BTT and local education issues. “Parental co-operation was poor, there was lack of engagement with parents” complained a PTA board member of School Number One. “I wrote to the Education Quality Reform Project for a grant to implement cooperation between the school and parents. Since then, attitudes have changed in making this collaboration work.”55 Parental attendance at school meetings means that children receive the same information from parents and teachers, reducing conflicts at home when parents are not aware of the school’s expectations of their children. Awards are given by the PTAs at the aimag level to parents who have done outstanding work at the school. The five PTAs in the aimag have formed a region-wide group with five board members (one from each PTA) and are working towards realizing their one goal and one vision objective. Through Facebook, the group communicates with parents and shares children’s achievements and activities. They have also forged a three-way agreement between the PTAs, the local education departments and the schools, which has produced positive working relationships. Working together, the group has raised its voice at the aimag level on issues such as the need for school buses. 53 PTA of Orhon soum. 54 PTA of Orhon soum. 55 PTA board member of School Number One, Bulgan soum. 36 Three of the project school principals who were interviewed were very familiar with the project and the EQRP. When teachers in School Number One (who had seen the effects of establishing PTAs in that school) were appointed as Principals in School Number Two and Three, they requested assistance from the project and EQRP to establish PTAs in their schools. Since then, they have benefitted through collaborating with the PTAs. One example quoted was the school bus for the Bugat School with 693 students to transport students to their distant homes. “Attitudes among students and teachers change when they see parents engaged in making teaching aids and delivering school supplies using their own funds” the principal of School Number Two recounted. PTAs have reached out to students and engaged with them on their schooling and learning needs. Acting on their feedback, they have organized training for teachers and parents on ways of communicating with students in school and at home to prevent possible abuse. By working with the PTAs, the mayors of the soums and the aimag governor have invited students to voice their opinions at provincial level meetings and participate in budget discussions. This has been a major collaborative engagement event. “Students have put forward their needs for debating platforms, sports and other facilities.”56 With the assistance of the PTA, the school librarian engages with all the students in the soum including those with disabilities. “I am very happy that my parents are going to the PTA activities,”57 said one student. “It helps me change my attitude towards them and teachers.” “Teachers these days routinely plan the curriculum every quarter and package the lessons for the semester to give parents an overview of the curriculum. When parents help by making school supplies, teachers’ attitudes change, and the workload of the teachers have been reduced.”58 Children’s suggestions are taken from the student suggestion boxes and reflected in the school plan. “At first teachers and staff were reluctant to point out their shortcomings, but with the implementation of the GSST, they tend to be present in school and improve their work. Parents are involved in the school construction work out of their own initiative.”59 The local education department of Bulgan soum asserted that they are under pressure from the CSOs to engage on school issues. “We welcome this, and we work together with them on issues such as the heating in School Number Two and building gymnasiums in School Number One and Three, and the construction of dormitories. We work with the schools and PTAs, and it is good that school problems are assessed by the PTAs, and they prioritize the issues and deal with them.”60 Project Results and Benefits The PTAs in Bulgan have managed to demonstrate that they are capable of filling gaps in the service delivery chain to schools. It is not only the hard infrastructure gaps that need to be filled in 56 Principal of school Bugat soum. 57 Student at school Bugat soum. 58 Principal of School Number Two Bulgan soum. 59 PTA of School Number One Bulgan soum. 60 Head of Policy Unit, Local Education Department, Bulgan soum. 37 rural schools but the soft aspects such as the academic and emotional needs of students, and the attitudes, behavior and concerns of teachers. There were many examples of PTAs engaged in improving hard infrastructure shown to the evaluation team. These ranged from fixing lights in the school and toilets, to erecting school fences and building sports facilities with outside grants. Parents fixed the interiors of schools and dormitories and built cabinets for teachers and students, while constructing and managing the school noticeboards. Laboratories were constructed, computers, printers, and a school bus were purchased with outside funding, and repairs of the sewage, water and heating systems were implemented. “ Getting the school bus was the result of the PTA working with the soum mayor to obtain finance from the soum budget and the Local Development Fund.”61 In the celebrated case of Bugat soum, reported in the ICR (Ali 2019) and talked about at MOF and MECSS, joint school-wide efforts including the PTA were mobilized to raise funds for the demolition of old school buildings and the construction of new ones. “Through media campaigning and extensive discussion and negotiation with public authorities, including Aimag Citizens’ Representatives, the case was ultimately brought to parliament, and MNT 2.1 billion (approximately US$ 796,000) in funding was secured” (Ali 2019). This was confirmed by the mayor of Bulgan who mentioned that the amount was even higher MNT 2.4 billion (US$ 900,000). “For the same school MNT 6 million (US$ 2,300) was secured from the aimag governor and a donation was received from the Member of Parliament to build fences around the school.”62 PTAs have worked on improving the overall quality of learning, psychological aspects and the learning environment. “Parents now worry about the quality of education. Before parents just gave money to the schools.”63 Teachers were reported to have improved their teaching methods and communications with students, moving away from rote learning64 and a preoccupation with achieving the best results on the national Olympiad competition. Student-centered learning methods are now being practiced to develop student understanding of the curriculum, using various aids prepared by parents to help students in their learning. There is additional emphasis on extra- curricular activities, and PTA mothers sewed sporting attire for sporting events. During the COVID-19 period, PTAs have been even more engaged: visiting students to provide guidance on distance learning and recording CDs with explanatory exercises. Mobile libraries are another innovation by the PTAs to encourage students to maintain an interest in reading. PTAs have organized youth empowerment training and, together with the health agency, carried out health inspections and vaccinations for students. Working with the aimag Family, Children and Youth Department, PTAs have jointly obtained financing for child protection initiatives by engaging with the local education, health and police departments while visiting families and advising them on child abuse and depression issues. The COVID-19 period has added to the pressure on schools to hire psychologists to deal with depressed children. One PTA member lamented, “ PTA capacity is small to work with depressed children.”65 61 Principal of School Number One. 62 Principal of School Bugat soum. 63 Mayor of Bugat soum. 64 This method emphasizes learning or memorization by repetition, often without an understanding of the reasoning or relationships involved. 65 PTA of School Number One, Bulgan soum. 38 Recommendations Three soum mayors were interviewed during the visit. They confirmed the positive role played by the PTAs and had a number of recommendations for the PTAs and the project. Their advice is to continue with the GSST annually to improve the school and to publicize the results on the information boards. “There is a need for more training of the PTAs on responsibility, accountability, and budgets. New members should be elected to the boards so that they can bring professional experiences into school boards. The issue of changing coordinators of PTAs is a challenge and needs to be addressed.”66 It was suggested that having one teacher and one parent on the administration team of the PTA, would make them more effective in working with the schools. On needing support to expand the project results, one mayor remarked “for children’s development, capacity building training, and school accountability, PTAs need support from donor agencies.”67 To scale up the PTA initiative, the mayor of Bulgan soum mentioned “Currently there are no PTAs in sixteen schools in our aimag. Other schools need to vote on an action plan for the establishment of the PTAs. It is possible to raise awareness of the importance of PTAs and to include funding in the local budget.” Box 3: Success Factors of PTAs in Bulgan Aimag When reviewing the key success factors of the PTAs in Bulgan, the evaluation team identified five aspects that deserve mention. These are listed below. Engaging and establishing good stakeholder working relationships. The PTAs established good working relationships in a number of ways by sharing information and ideas between the PTAs and school principals, teachers, bhag directors, soum mayors, local education departments, parents, students, local media, and local NGOs. The PTAs forged a three-way agreement between the PTAs, schools, and the local Education Departments to facilitate the PTA activities in the soums. Direct communication was continuously maintained and working relationships built between the PTA and these officials and agencies. The local TV outlet was used to broadcast the success stories, achievements while showcasing parents being active in school matters. The PTAs engaged with the aimag Independent Authority Against Corruption (IAAC) to implement the laws on budget and procurement transparency. Expanding the PTA process to non-project schools and building a wider team. The principals of the non-project schools were actively involved in the PTA of their previous school before being appointed as principals of School Number Two and School Number Three. They knew the benefits of the PTAs and lobbied for assistance from the project to establish their PTAs – a clear demonstration of the change PTAs brought to the schools. The PTAs of Bulgan, Orhon, and Bugat soums established a Facebook group with parents and share children’s achievements and activities. 66 PTAs of School Number One, Two and Three, Bulgan soum. 67 Mayor of Orhon soum. 39 All five PTAs work as a team with a common vision and assist each other with their activities. They share information and visit each other. Changing the attitudes among stakeholders. Changes in the attitude of parents, teachers, and students were noticed especially after the impacts of the PTAs on the school physical environment were visible. The soum mayors began supporting activities of the PTA for the common benefit of the community. A member of the PTA was included in the advisory committee of the soum mayor, affirming the value of the PTAs. Students have made known their views to the PTAs. Parents have become more aware of their children’s schooling needs. Children are invited to participate in aimag level budget discussions to have their voices heard. Championing the larger cause. Together with the PTAs, soum mayors have engaged and lobbied aimag level officials, the ministries, and even parliament for funding support to improve the physical conditions of the schools. A number of LDF grants were mobilized in this way to improve school facilities. PTAs work together with the local education, health, and police departments as a team when they visit families and children around the bhags during the COVID-19 period. Sustaining results beyond project life. PTAs have found a number of ways of sustaining project results. Schools have incorporated PTA activities into their school plans. PTAs receive continuing assistance from the soum mayors and other local authorities. These officials are keen to retain their residents by improving their well-being. The NSPTA provides technical and organizational support to all PTAs. The PTAs are sustaining themselves financially through various means; including in-kind donations of livestock from the herder parents which are then sold. 5.10 In-Person Key Informant Interviews with Government Officials of the Project Steering Committee The evaluation also conducted KIIs with three members of the Project Steering Committee from government agencies: MECSS, MoF and IAAC. Their feedback is summarized below. All three officials welcomed the project and noted its contribution to social accountability in education. As noted in the Zoom interviews and the Bulgan validation visit, these officials mentioned the change in attitudes, mindsets and collective efforts at schools to get support from aimags and even the ministry. They remarked that PTAs have increased the responsibilities of parents and that a collective culture is being established between parents, teachers, students, local authorities, and aimag governments. It was asserted that the ministries are in the process of moving towards this type of participatory approach and opening up for social audit68 of their sector work. This trend has also been aided by the Governments e-Mongolia initiative. It was reported to the evaluation 68 “Social audits” can be understood to be citizen-led processes that facilitate public participation in the monitoring of government service delivery and expenditures. See https://www.internationalbudget.org/publications/guide-to-conducting-social-audits-in-south-africa/. 40 team that a version of GSST is being used by the MECSS to determine gaps in the physical environment of schools that need financing. They acknowledged that transparency in school budgets has increased and that there is a need to spread the lessons and success of the PTAs to other aimags and soums. Mr. Nyam Ochir Mr. Sanjaadorj Ms. Sarangerel Director of the General Education Head of Budget Expenditure Officer and Commissioner, IAAC MECSS Division, MOF Figure 8: Project Steering Committee Mr. Nyam-Ochir, Ministry of Education and Science69 • I had a chance to visit Khovd aimag (Myangat and Mankhan soums) where I was made familiar with the results and outcomes of the TAME project in person. There is a change in the attitude and mindset of the stakeholders. Now they are saying “all issues can be solved as long as the personal attitude is changed”. This has become very popular. The issues now are resolved by collective efforts to obtain aimag support, and even from the ministry in resolving the school-related problems. The parents and the school would collectively write a letter requesting support, and it works. • There has been an improvement in terms of the parents’ responsibility with the establishment of PTAs, that is the establishment of a collective culture based on their participation. A lively and real relationship has been established between the parents, students, teachers, and the local authorities (even the aimag Governor meets with the parents on the school and their livelihood issues). All the dormitories there now have a special room where the parents can meet with their kids, and even spend a night there with them, which has led to improvement in learning by the students. • Transparency has increased, especially in granting the teachers’ awards. The ministry has issued the Regulation on Remuneration/Salary Increase of Teachers that now depends on making sure that teachers encourage the involvement of all students in learning (before, the teachers would only pay attention to the exceptional kids). This has led to improved teachers’ initiatives which in turn led to the improvement in teaching and learning by all students. The schools now evaluate the teachers’ performance collectively, based on the positive changes noticed in each child, and pay more attention to the quality of teaching, resulting in positive learning results. 69 The feedback from the officials were translated, edited for clarity and brevity. 41 • The teachers now are very interested in using the schools’ specialized classrooms (such as labs) and have realized that such classrooms are for the use of students. Before, such classrooms were locked up most of the time. • The school plan is now based on the inputs from the parents (for instance 300 parents have provided their inputs). Now the school principals, teachers and parents “look through one window.” Even the ministry’s decisions often recognize their inputs. The ministry’s attitude is also changing towards the participatory approach. • The PTAs are being set up in different ways. Some PTAs are merged into a soum-level PTA which made an agreement with the soum mayor.70 • There is the need for well-designed training on good PTA practices to disseminate to other aimags and soums where some people still resist to change their minds and attitudes. There is a need to ensure the sustainability of such initiatives in the long run. • On school financing, there is a need to find a mechanism to determine the school needs (should this be based on the PTAs proposals? the ranking of needs? who will be engaged in this?). Mr. Sanjaadorj,71 Head of Budget Expenditure Division, Ministry of Finance • Monitoring of the school budget and procurement is a very important issue, therefore PTAs monitoring the school expenses become examples of social auditing . There is a view that monitoring of the school expenses has improved by 11 percent, however, what methods have been used for such monitoring are not known. • It is believed that the introduction of the e-transparency project72 will improve school budget allocation, and this is being implemented. The school budget now is allocated based on the economic classification (so much on payroll, so much on books, etc.) which is difficult for ordinary citizens to understand. Now the tendency is to make it simpler - moving toward the output-based system – so much will be spent on xx to get yy results; whether the allocated money is sufficient if it is not- then why? • Citizen’s budgets have been published for the last three years. An electronic database is available online for people to see the expenditures. The dissemination of the basic information will be more innovative. A more automatic and systemized approach will be used. It will help in setting the standards and norms in school expenditures, especially in the allocation of the current expenditures. Using the Human Resource System, the payroll module of the teachers (which is the largest operating expenditure) is now being piloted. • From 2019 MECSS had disclosed the use of the allocated budget. All 330 soums can see their respective budgets on the map-based dashboard. Now the soum citizens can see and 70 Some of these PTAs were formed under the MASAM project. However, they are not formal entities as those established under the TAME project. The PTAs formed under the TAME project are registered entities as NGO capable of receiving funds and accountable for reporting their accounts to the authorities. CSOs on the other hand are informal entities formed for a particular cause. 71 Mr. Sanjaadorj participated in the meeting briefly after which Mr. Batsukh, policy advisor of MOF, took over the feedback with detailed views. 72 This is also known as “e-Mongolia” an integrated digital government service system which would operate through public service centers. Citizens can also express their opinions, complaints on government institutions. The first center began operation in May 2021 in Ulaanbaatar. 42 monitor the spending, how much has been spent on improvement of the school environment and so on. It is much better and easier for them to see and monitor using the application compared to the announcement boards. • With regard to the community participation-based planning, where the aimags conclude performance contracts with the soums and the soums with the schools on expenditures. We are not there. There is a top-down approach, no bottom-up approach as yet. For the time being the decentralization is slowly starting again. Several years ago, the decentralization was widespread, and most of the authority was given to the aimags and soums, but due to misuse of the budget funds (some people would buy the cars instead of spending on the necessary items) curbing of the decentralization effort followed. • Before, the school buildings were financed by the state budget whereas the aimags used to own them. Since about 2018 the authority to finance such investment was given to the aimags. The LDFs under the SLP, MASAM and the TAME project are interconnected in this regard and are used for the monitoring of expenditures. And here the social auditing is very important, and MoF is going to start it. The MoF also discussed and agreed with the Asian Development Bank to cooperate in this area. • Starting in 2020, MoF jointly with MECSS started collecting data on the school environment of all schools to find out which schools and dormitories have no proper toilets, and so on. Here, there is also a gender-based approach incorporating gender-sensitive policies. All data will be systemized and based on this, to decide where to channel the additional financing. • With regard to local procurement—this is a difficult issue—because citizens want their local companies, but those companies often lack capacity and capability. Many locals say that the bid evaluation should take place as far as possible from the construction site. Civil society and local citizens can monitor the fairness of the bidding, but not the technical side of the bids as they often lack the technical knowledge. Another issue is that while many locals attend the training on bid evaluation, most fail the exams and do not get the necessary certificate. Ms. Sarangerel, Independent Authority Against Corruption • In my view, the project was very useful as the transparency of information in the education sector generally was not satisfactory. There is a standard procedure for making information on schools and kindergartens more transparent, however the work on this is not progressing well. • Regarding parents’ donations to the schools - the law allows donations, however sometimes it is being used improperly. There is a need to look at how donations are used in the education sector, whether donations and their use follow the legislation. The IAAC could support the participation of the parents/PTAs in the bid evaluations of the school procurements as there is already a legal provision ensuring such participation. Today the situation with the participation of civil society in the bid evaluation is much better as the Public Procurement Law ensures the participation of the representatives of the civil society in the bid evaluation 43 committees. The IAAC also requires such participation and insists on it. Since there is a legal ground for the participation of the representatives of civil society in the bid evaluation committees, the IAAC can specifically issue the recommendation or send the official letters to the schools recommending the participation of the PTAs or parents in the bid evaluation. This will ensure the fulfilment of the project objectives. • The training of the parents/PTAs in public procurement and budget issues is very important. The IAAC can also make a proposal on it. Such training would be helpful for citizens capacity-building. The IAAC approved its policy document in 2016 (No.51). Under this National Program, there is no need for any MoU with us. The legal basis for the information, financing and procurement transparency is in place, therefore if there would be a request for support and cooperation in this area, we will be ready to cooperate. • The IAAC management would like to receive and study the project evaluation report. 5.11 Concluding Remarks on Results Drawing on the quantitative outcomes from the project reports, this section has triangulated those results with qualitative evidence from the larger Zoom-based FGDs, the interviews from the validation visit to Bulgan province, and the face-to-face interviews with the key officials at the Ministries.73 The end result does indeed confirm the achievement of the project objectives of using collaborative social accountability processes to achieve better school governance, budget and procurement transparency, and thereby better service delivery to schools. As noted in this section and the narratives from stakeholders’ feedback, much was achieved by moving from a top-down vertical system of information, financial flows and reporting to a more open and accountable system. A major impact on the system was the state of change in the physical environment of the schools. Rural school infrastructure was in disrepair, in need of renovation and upgrading. A gamut of services was provided by the state, local officials, donors, and PTAs to upgrade physical infrastructure and utilities in the schools (including building a new school). Funding for these were collectively sought by project stakeholders from various sources and the amounts were quite substantial. To many parents and students, improvements in facilities meant an upgrade in the quality74 of education. This was indeed a visible demonstration of collective action by key stakeholders which bought more buy-in from schools and other local officials. Building on such gains the PTAs and school teams began to focus on the softer aspects of quality, the learning environment: teachers and learning materials, teaching aids, teacher-student interactions, student meals; teachers’ training, salaries and incenti ves; and a focus on student-centered learning. Improvements and upgrading of dormitories were equally visible around the project sites. This 73 Triangulation of the evidence was also a requirement of the TOR for this evaluation. 74 No single definition adequately covers what constitutes quality in education systems as the term is still evolving. The conventional definition covers literacy, numeracy and life skills linked to teachers, content, methodologies; curriculum, examination systems; enabling policy, planning, management and administration. See https://educateachild.org/explore/barriers-to-education/quality. The World Development Report 2018 also includes enabling home and community environment, and school climate into the quality discussion. The EQRP makes greater reference to “quality of skills” through better cognitive and non-cognitive skills that will help citizens deal with complex and unexpected tasks and continue to learn throughout their lives. The project also noted that the quality of teaching in schools and the teacher training system were concerns in Mongolia (see World Bank 2014). For rural parents and students, the term denotes the more basic hardware and soft aspects of learning. 44 was complemented with the setting of rules of behavior jointly by students and administrators, and training teachers on the essentials of positive guardianship and behavior. School plans, which were the under the purview of the school management teams, became a collaborative exercise with inputs from parents, teachers and students.75 The disparate system of service delivery to care for the welfare of the children became connected when the PTAs, welfare workers, nurses, and police began organizing joint visits to families, especially for abused or disabled children. The school, soum and aimag officials and local assemblies responsible for school governance, use of resources and accountabilities, were also previously unlinked. It became a much more connected system under the project. PTAs, schools and the local education departments agreed school goals, plans and objectives and disseminating them to parents, students and the community. Parents had poor understanding of the school budgeting and procurement systems, which were opaque and cumbersome to understand. Through the BTT process, details became clearer to parents and teachers, budgets were made transparent, and the administrators were made accountable for their use. Budget bulletin boards have continued post-project period at all the project schools. Despite laws regarding budget transparency and accountability, and information made available to the public, administrators and officials were still unclear what the laws required. They had to be trained by the project on these legal obligations and documents in order to pry open the doors to information and school finances. A marked improvement has been seen as a result of the project, especially in horizontal accountability with gains being made in vertical and diagonal accountability.76 The project’s effectiveness went beyond accomplishing the project objectives and contributed to spillover effects including financial and social benefits. The TAME project stretched the use of its resources ($650,000) in terms of project efficiency. When compared with the MASAM project’s education component, the resources (all-inclusive costs) used to create social accountability processes at each school, was on average around $21,000 for TAME and $40,000 for MASAM. 77 6. Sustainability of Project Results and Process Enough time has elapsed since the close of the project at the end of 2018 to evaluate the sustainability of project results. In addressing the sustainability of the TAME project, it is useful to look at 1) the sustainability of structures supported by the project, and whatever risks that might 75 While it is too early to gather information on the effects of such changes on student performance, the CSO that worked with schools in Bayan- Ulgii (LEOS) asserted in late 2020 that the improved school-parent-teacher collaboration had an effect on students getting high scores in university entrance examinations. The CSO also maintained that the number of student complaints had dropped significantly. 76 Vertical accountability allows citizens to hold governments and politicians directly accountable. Horizontal accountability is ensured by oversight, checks and balances between different state institutions preventing abuse of power. Diagonal accountability on the other hand implies that media and civil society need to hold the government accountable through means such as, spread of information, publicity and other forms of engagement. 77 In the MASAM project, one-third of the total $3.2 m budget was allocated for the education component. Twenty-five schools were involved in a similar SAcc process to the TAME project but using different delivery mechanisms (experimental tools, participatory school assessment models, bottom-up approaches). In two provinces the MASAM project and TAME project overlapped in education but chose separate intervention schools. The thirty-one PTA schools were used in the TAME calculation. Source: Interview with MASAM project coordinator, Ulaanbaatar, 2020. 45 be faced and 2) other supporting factors to keep the momentum going. The issue of mainstreaming the social accountability process employed in the project is another requirement of the evaluation and is also addressed in this section. 6.1 Sustainability of Project Structures By most accounts examined in the report, the TAME project results are seen to be sustainable over the medium term. Key project outcomes: the 31 PTAs and the NSPTA, are still functioning post- project. This is the Most Significant Change and impact brought about by the project. The NSPTA is still in regular contact with all its members through Facebook updates and the project website. The project website is being updated with relevant information from MECSS and developments from PTAs themselves. During the COVID-19 period, the PTAs have been actively working with schools and teachers to help students with distance learning, visiting families and checking on their well-being with social workers. Parents have volunteered their time, provided money and in-kind contributions to make positive changes in the school environment. PTA Field Visit to Ulziit A PTA in Zuungovi Soum, Renovation of Dormitory Soum, Arkhangai Aimag Uvs Aimag Built a COVID- by Parents, Baruun-Urt 19 Awareness Board Soum, Sukhbaatar Aimag Figure 9: Images of PTA Activities Credits: National School Parent Teacher Association Finding sources of funding for the PTAs to keep their activities going has turned out to be crucial. Many have found creative ways of sourcing funding including membership fees; donations from businesses; income-generating activities such as selling PTA-made gifts; and organizing English classes for residents. Herder parents donate livestock to PTAs which are then sold to fund PTA activities. Soum mayors are also contributing to PTA activities from their budgets. Some PTAs have banked the grant received from the EQRP and used the interest for PTA expenses. PTAs are now able to independently seek funding from external sources since they are legal entities, and in some instances have been supported by agencies such as Mercy Corps, World Vision, Embassy of Japan, UNICEF and GIZ.78 These agencies have seen the value PTAs bring to the community. As well as the funding issue, PTAs face concerns over the turnover of board members. Here, the 78 The GIZ has more recently been involved in restructuring school heating issues. This is a major issue in the remote schools, especially in the Bayan-Ulgii province as noted by students interviewed. 46 guidance and training from the NSPTA to the boards continues to be valuable. As suggested by soum mayors, PTA boards would benefit from younger members and members from professional bodies. A soum mayor in Bulgan remarked that the administration teams of the PTAs need a teacher and a parent to make the PTAs more effective in their daily work. The value of having a well-functioning NSPTA acting as the secretariat for the PTAs cannot be underestimated. Support for this function by the AFE alliance would be crucial, mostly in terms of income generated from its other activities and grants. The NSPTA has been successful in obtaining two small grants from the EQRP. There is a risk that the gains made by the project could be compromised if AFE’s support to the NSPTA diminishes. Having seen the tangible results and contributions of the PTAs, schools show their support in many ways, including incorporating PTA activities into school plans, providing of rooms for PTA activities and meetings and office supplies. School plans are also based on inputs from parents as acknowledged by the MECSS. What seems crucial is for the PTAs to keep engaging with the new officials appointed recently after the local elections in October 2020 and the change in government in January 2021. Rebuilding trust and collaboration with the new officials at all local levels would go a long way to sustain project results. The PTAs in Bulgan have already started this process and in November 2020 proposed to the new aimag governor that PTAs are established, using the TAME methodology, in all soums that are without PTAs.79 In all likelihood, this could materialize as Bulgan aimag has demonstrated that it has the core group of champions in place to make this happen. For other aimags, PTAs need additional capacity-enhancing training to maintain engagement with all their stakeholders besides the officials. Here again the role of the NSPTA seems crucial along with assistance from other international agencies. 6.2 Other Supporting Factors PTAs have incrementally expanded efforts in mobilizing support for school projects through the LDF of the World Bank’s Sustainable Livelihoods Project (SLP), with some success as noted in the results section. Some soums have been quite successful (Khovd, Uvs, Arkhangai, Uvurkhanghai) while the others (Sukhbaatar, Khuvskul, Bulgan, and Bayan-Ulgii) could increase their efforts to obtain LDF funds (Figure 5). One concern for both PTAs and CSOs (reflected in their recommendations for component two) is the re-ranking of community projects proposed by citizens through community social accountability methods as required by the LDF guidelines: “When planning projects, programs and activities to be imp lemented by the aimag Local Development Fund, the citizens' suggestions from the soums were collected and ranked. The citizens' discussions were organized through the citizens' halls, and the Citizens' Lead Representatives seem to have the right to re-rank the polling.”80 79 Communication with the PTAs in Bulgan, November 2020 by the evaluation team. 80 Extract from recommendations made by BTT groups on policy impediments. 47 Several aimag and soum governors in the TAME project areas have expressed willingness to formalize and integrate citizen participation in decision making especially on the use of the LDF. NSPTA and CSOs such as OSF could still take this matter up as with the local authorities and the National Audit Office (NAO). OSF is monitoring budgeting and procurement issues in education and has shown interest in collaborating with AFE on this matter. 81 Another CSO to collaborate with is Transparency International Mongolia. NSPTA should be working with these CSOs, IAAC and NAO on the procurement issues. While procurement information is made available at the school level on noticeboards, the process of contract awards seems to be an area of concern for PTAs. Although local sourcing is preferred by schools, MoF maintains that the quality of local materials and contract management are issues that the central agencies should be responsible for. MoF has advised that more local citizens should be trained to pass the tests administered by MoF so that they could sit on the appropriate committees. It seems necessary that assistance is provided to NSPTA to deepen its work in this area.82 At the broader level, MECSS and MoF have both indicated support for CSA principles to be used for better service delivery of education services (as summarized in the results section). MECSS and MoF have both alluded to this collective culture, participatory approach and social audit. While inroads have been made into both ministries on the concepts and practices of CSA, the best way would be for MECSS to use the TAME lessons to scale up the establishment of PTAs in the non-project soums of the eight aimags and non-project aimags using the EQRP as the driving force. All the aimags in the Zoom FGDs and KIIs have advocated for this approach. It is relevant to note that a version of the GSST has been used by MECSS to determine gaps in school infrastructure. Both MoF and MECSS have noted the contributions made by the PTAs. They have expressed the need to disseminate the lessons of establishing PTAs in schools. PTAs have served as the change engine in the TAME project and catalysts for school-level reforms. Given that PTAs have also been trained on the BTT, they could eventually implement both the GSST and BTT and thus serve as the accountability mechanism in schools and with local governments. “The BTT could be the more valuable tool to be employed in allocating and monitoring the use of scarce financial resources in a systematic manner towards priorities established with citizens participation. The BTT offers a wider scope for sustaining PTA activities. It is also important that the PTA movement does not become confused with any school-based management system that the government might put in place. The two types of organizations usually have different functions. To be clear, ownership of the GSST and BTT, as social accountability tools, should not be owned by MECSS although MECSS can mainstream the practice of social accountability and use of the tools by the school system.” 83 This distinction is important to keep. “When these tools are converted to the ‘community score card’ process in the context of social 81 This was confirmed through a Zoom meeting with the OSF in Ulaanbaatar in June 2020 with the evaluation team. 82 During the interviews, concerns over procurement were highlighted especially in the Bayan-Ulgii province and this confirms the findings of OSFs study on this province, which received the lowest scores on the transparency of budgets. See section two of this report. 83 Feedback from WB consultants who worked with GIC and AFE during the implementation of the project. 48 accountability, it is inappropriate that they are viewed by stakeholders as owned and administered directly by MECSS staff.”84 Beyond the education sector, social accountability mechanisms have been used in the health sector by the MASAM project and some form of community feedback is practiced by the Ministry of Agriculture.85 Knowledge of such approaches has been embedded at the Cabinet Secretariat and MoF by the MASAM project. Almost 20 years since the idea of community participation was introduced to Mongolia through the World Bank’s SLP, other projects such as TAME, MASAM and small grants from the World Bank, the idea of community participation and social accountability has taken root. The World Bank’s Smart Government project and e-Health project also emphasize civic engagement and citizens feedback mechanisms to expedite service delivery. Thanks to these projects the World Bank is in a unique position to mainstream the social accountability process further. In May 2021, the MASAM II project was initiated involving the World Bank, the Swiss Development Agency, the GPSA and other donors. The project will focus on deepening the social accountability processes in the health sector. It is hoped that these ideas will permeate other sectors through the e-Mongolia transparency initiative of the authorities. The methodology of community participation is embedded in all soums through the SLP. Enabling community groups to become accountability mechanisms for each sector remains a challenge, due to specific political economy factors in each sector, combined with the need to identify groups of champions who care about sector issues, such as parents in schools. Once identified they would need capacity building through appropriate vehicles, such as the PTAs.86 It is useful to note that through two decades of projects and reports (mostly contributed by the World Bank) the term social accountability has now gained credence among a broad range of societal stakeholders. 87 7. Lessons and Recommendations Success as implied by the GPSA’s ToA is to what extent the project helped project partners to solve problems. Taking a holistic view of the TAME project, this final evaluation has identified a robust set of results that the project partners managed to produce over a four-year period. The most significant success was the setting up of the PTAs as vehicles for improving school governance, delivery of school services and student well-being. This final section identifies the lessons from the project and recommendations for deepening the impacts of the project. 84 Feedback from WB consultants who worked with GIC and AFE during the implementation of the project. 85 Through another WB grant, the social accountability process was tested in infrastructure and mining sectors in 2013 (See World Bank 2013) 86 In agriculture, herders care about their livelihoods and Pasture User Groups established by the Swiss Development Agency could play a role. 87 A large study on social accountability was carried out by the World Bank in 2007 which popularized the term (See World Bank 2007) 49 7.1 Lessons Conditions at Entry, Project Design, CSO Capacities The project’s aspirations at start-up were somewhat ambitious in scale and location. Working in the remotest areas of the country is always challenging for project teams, especially in the harsh Mongolian winters and with long travel times. However, by the end of the project, the implementing teams managed to produce significant results under difficult circumstances. Implementation improved especially after the mid-term review with key adjustments to the school governance and budget transparency components. The resulting overall results framework and indicators were better aligned with the objectives of the project. The restructuring was carried out with the assistance of the GPSA Secretariat and the World Bank’s Education Team. Implementing CSOs were on a learning curve, gaining most from their civic engagement, and building their operational capacities to carry out the various engagement processes required under the two main components. Modest gains were made by the CSOs in terms of analytical, technical and adaptive capacities.88 The PTAs and the CSOs increased their knowledge of policies and laws related to education, budgets and transparency as noted in their recommendations to the authorities. GIC, AFE and the 41 core CSOs that were involved in the implementation process, gained substantial implementation capacity on the GSST and BTT. From the perspective of learning from the GPSA’s global practices, GIC examined a number of GPSA’s country experiences in the education sector on social accountability and chose the Moldovan model of PTAs for the purposes of the project. The political economy of power structures and relations between officials, teachers and parents at both aimag and soum levels added another layer of the challenge at implementation. Although by the end of the project these power relations had become diffused, with hindsight, some aspects of the design could have been downsized from the beginning. “ Although GIC has local governance and communication expertise, it was not an expert in education policy and reform. AFE has that expertise and both CSOs were meant to complemen t each other’s expertise during project implementation.”89 By project completion, GIC had gained technical expertise and capacity in education and education budgets. “A lesson to be learned from the redesign of TAME at project inception is for CSOs to be more modest, to go for realistic goals based on their capacities rather than for aspirational goals based on the opportunities for change that existed in Mongolia.” 90 Another lesson worth considering is the timing of the provision of just-in-time support to GIC especially at the start-up stage of such projects that are to be implemented by CSOs (Ali 2019). Results Framework and Adaptability, Mid-term Corrections The first adaptation happened in 2015 with a revised project design to adopt a phased-in approach and improve effectiveness given the capacity constraints of the lead CSO and the geographical 88 See Poli M., & Guerzovich F. 2020 for a discussion on the capacity development issues surrounding GPSA sponsored projects. 89 Feedback from WB consultants who worked with the GIC and AFE during the implementation of the project. Setting up monitoring, data collection and management systems throughout the project is a critical function. During this evaluation, gaps in data recording and analysis of participants at events were found, especially in relation to baselines established at the start of the project. 90 Feedback from WB consultants who worked with the GIC and AFE during the implementation of the project. 50 spread of the projects. The second major adaptation happened as a result of the mid-term project assessment in 2017 and the subsequent mid-course corrections proved to be crucial for the project’s success. The GPSA and World Bank staff provided valuable advice, methods and support to improve the fit of the indicator system (and baseline) with an improved understanding of how project objectives and results should be assessed and measured by GIC. The indicators were reduced from 13 to eight to further rationalize project expectations and focus on what was critical to measure and how to measure them. The community radio component was dropped, the PTA component was scaled-up and the budget monitoring component scaled-down. The need for the schools to engage beyond their districts and extend into the aimag level authorities was made apparent and this proved to be critical for project success. These corrections put the project was on a better implementation trajectory. TAME Project Theory of Action Having a project ToA clearly articulated at the beginning of the project was a big plus for the project. The project ToA envisioned that “if stakeholders—government, school administrators, parents, teachers, and students—were more constructively engaged in education services and oversight, then providers of those services would improve their performance and be more accountable.” Thus, the assumption was that the appropriate set of stakeholders being constructively engaged would result in better educational services. This would in turn lead to improved oversight and accountability in service delivery at both the schools and the eight chosen aimags. The necessary actions were delivered through two institutional settings: PTAs in component one and a cluster of CSOs in component two. The engagement was fostered at schools, districts and provincial levels between the concerned stakeholders. The anticipated role of PTAs was not easy to realize owing to the lack of precedent in Mongolia. The project had to demonstrate the value of PTAs, an unfamiliar concept to many. School councils were already in existence, and the role of the PTAs was viewed with suspicion by school authorities. This evaluation has provided evidence that through the TAME project, new forms of engagement were fostered and expanded incrementally among the stakeholders at both the school and regional levels. This effort even reached the national parliament. Project stakeholders mobilized substantial school financing that was (and is) needed to improve the physical environment of rural schools. Civic engagement of herding and minority groups to solve school issues in remote locations is possible when appropriate models and support tools are made available. The MoUs signed with various agencies and schools created the entry points for the engagement process, aided by the reputation of the World Bank. Attitude and behavior change between parents, teachers and school administrators materialized only after the trust was built. A relevant lesson from the GPSA’s ToA is that interventions that are focused on the intermediate governance and managerial levels actually work, as was the case in the TAME project.91 91 Three assumptions of the GPSA’s Global Adaptable Theory of Action (2020) were borne out by this evaluation and they are the following: 1) GPSA grantees use adaptive management techniques to manage the project 2) governments have commitment to integrate inputs from civil society into policy choices and 3) coalitions within government and bank sector teams recognize legitimacy and value conferred by social accountability procedures and find opportunities to scale up to more programs and country systems. 51 Establishing PTAs as the Change Engines The Most Significant Change of the TAME project has been the formation and dynamic functioning of the PTAs supported by the AFE's NSPTA and the World Bank’s EQRP. The PTAs evaluated in this exercise can certainly become catalysts for better school governance reforms. All the 31 PTAs established under the project have demonstrated constructive participation in school governance and the transparency of budget funds. When funds are lacking for school facilities, they have mobilized external funds for school infrastructure. They have gained experience incrementally in collaborative school governance and have made their voices heard at school meetings, school annual plans and at regional meetings. PTA plans are incorporated into school plans by principals who initially were wary that PTAs were set up to “check and audit the work of school management and local education department officials.”92 Schools have recognized the financial and non-financial contributions of PTAs. In schools with PTAs there have been significant changes in physical and psychological learning environments. The PTAs undertook a process of gaining the trust of school administrators and other local officials, while building their own confidence by improving their management skills, experience and learning by doing. Their tangible contributions in changing the physical environments of the schools made the school authorities begin to realize the benefits of collaborative efforts with PTAs. Having Appropriate Collaborative Social Accountability Tools to Engage The organization and mobilization of stakeholders in both components of the project was aided immensely by having available, appropriate tools and adapting them to the local context and needs. Both the GSST and the BTT were used in the project to engage with stakeholders before turning them into Community Score Cards to address school and education budget issues. The issues were then prioritized for collective action. Both tools turned out to be crucial in obtaining positive project results. The tools went through a number of changes to adapt them to local needs with appropriate language. The GSST was particularly relevant, as it explored the three critical aspects of the school: physical environment, psychological environment and governance. All three aspects were useful to the stakeholders, especially the parents. Parents regarded student’s physical learning environment and psychological environment as the priority for achieving education quality. Whereas students rated the psychological and learning environment at both the school and their homes as critical. In project aimags during the COVID-19 pandemic, groups of parents, teachers, health officials and police visited and counselled parents and children to reduce instances of depression and abuse. It is clear that there are greater needs in this area that need to be met. Findings from the tools were used to modify all three school-related aspects. PTAs have also been trained on the BTT, and could eventually implement both the GSST and BTT, and thus serve as the accountability mechanism in schools and local governments. This is already being demonstrated in Bulgan aimag and is being encouraged in other aimags by the NSPTA. It would be important in this effort to use the same methodologies that TAME adopted. It is relevant to ensure “that the PTA movement does not become confused with any school -based management 92 PTA in Bulgan. 52 system that the authorities might put in place. When these tools are converted for the ‘community scorecard’ process in the context of social accountability, it would be inappropriate that they are viewed by stakeholders as owned and be administered directly by MECSS staff.”93 Having laws that open budgets and procurement to public scrutiny is one thing, making officials aware of these laws and making them engage with stakeholders is another, requiring further efforts as noted by this project. Synergies and Support from World Bank Teams The close working relationships between the GPSA and the supervision teams from the World Bank were key in adapting the project to the path of GPSA’s ToA principles. This synergy and a common vision between both teams during the restructuring phase of the project brought success to the project by the time the project ended. Both teams spent considerable amounts of time analyzing ways of restructuring the project, visiting project sites and engaging with concerned officials at MECSS who were changed in tandem with ministers. Two outcomes from the restructuring were critical for the subsequent success of the TAME project: the linking of the EQRP’s school planning component with the PTAs that were set up to pursue this agenda, and the need for the PTAs to engage beyond the district level to the provincial levels to secure school budget financing. The GPSA’s strategy of targeting better service delivery through CSA processes by linking with other World Bank upstream projects is valid - as borne out by the collaboration between TAME and EQRP. In this case, there have been synergies for both projects. There have also been synergies with the MASAM I project. Through the MASAM I project the main implementing CSO, the Mongolian Education Alliance (MEA) helped draft regulations for parents to be involved in schools. This regulation was approved by MECSS in March 2019. MASAM I used health messenger clubs to support child development with the participation of school social workers, doctors and class teachers in Sukhbaatar aimag, similar to the efforts of the PTAs in TAME project schools. In both projects, parents and communities started contributing to school supplies and fathers began to participate in school affairs. In Khovd and Uvurkhanghai aimags, MEA worked with AFE in both projects, using the TAME delivery models for the TAME project schools and MASAM I’s bottom-up approaches for the MASAM I schools. The TAME project aimags of Sukhbaatar and Uvurkhanghai, benefited from the citizens broadened understanding of social accountability concepts through the MASAM I sub-projects. Parents learned how to organize themselves to obtain funds for schools from the LDF and local assemblies (Mainstreaming Social Accountability in Mongolia Project 2020). Scalability, Sustainability and Links with EQRP PTAs have certainly contributed to better accountability and service delivery during the life of the project and beyond with the aid of the NPSTA, which was supported by the TAME project. The links established between TAME and EQRP benefited both projects and was a strategic move that gave credibility to the PTAs in the eyes of school administrators. FGDs and KIIs conducted during 93 Feedback from WB consultants who worked with the GIC and AFE during the implementation of the project. 53 this evaluation suggest that the PTA system is proving to be even more useful in the context of COVID-19. PTAs have engaged with all households during this difficult period to ensure continuity of students’ learning. In all project aimags there is demand for replication in non-project soums, as well as other aimags, using the same methodology and process. This is an issue for the GPSA, EQRP and MECSS to consider, and the opportunity for scale-up exists at least until the EQRP ends (expected in 2022). The 31 PTAs already supported by the TAME project and EQRP have found creative ways of generating income for their operations. These have included: membership fees; donations from businesses and the local authorities; self-generated income from the sale of gifts made by PTA members; organizing English classes; and even donations of livestock by parents for sale at the appropriate time. The PTAs have also received support from agencies like Mercy Corps, World Vision and the Embassy of Japan. The NSPTA continues to update the PTAs on education policies and the budget cycle for them to engage with their appropriate officials. The NSPTA continues to engage on a regular basis with the PTAs on developments in schools through its Facebook platform and project website. Some form of support from the GPSA and other agencies to the NSPTA to continue with such efforts seems necessary. On process sustainability and the ripple effects of the social accountability practices, recent developments within the government give hope for optimism. For the TAME project, it is clear that PTAs are key to embed the process within schools and become local-level accountability mechanisms. The evaluation found there is demand for this process to be replicated in schools not included in the target provinces and non-target provinces. MECSS and MoF are positively predisposed to the idea of social audits and the results of projects such as TAME, SLP and MASAM. Through the MASAM project there is also the beginnings of buy-in from the Ministry of Health, and a small engagement by the Ministry of Agriculture on its annual policy and results review process. At the broader governmental level, capacity for this approach was embedded at the Cabinet Secretariat and MoF by the MASAM project. The Government e-Mongolia initiative offers further ripple effects through citizen feedback at the newly established Public Service Centers. Many such initiatives in Mongolia are driven by external funding and when the projects end the initiatives also end, an effect compounded by the frequent changes in government and officials. Sustained efforts are needed to promote such collaborative social accountability processes. In agriculture, the process was already tested in the early 2000s through the community development funds of the SLP. Projects that serve as external catalysts (e.g., TAME project, SLP, MASAM project) are needed, but equally important would be the organization of groups of stakeholders who care about sector issues; like parents in education, herders in agriculture to sustain the process and act as accountability structures. It is advisable that the MASAM II project builds synergies with the on-going e-Health project, the COVID-19 Emergency Response Health System Preparedness Project, and the SLP to explore how social accountability mechanisms can be made beneficial to all the projects. One lesson from the MASAM I project was the need to find 54 structures to sustain the project outcomes (Mainstreaming Social Accountability in Mongolia Project 2020). Engaging with Relevant Ministries throughout Project Life The higher-level officials interviewed indicated that more effort had to be exerted by the lead grantee to keep officials and the Project Steering Committee informed of the progress of the TAME project. This lesson is also contained in the project ICR report. Despite frequent changes in personnel and governments, efforts should be made to update the relevant ministries of the project’s progress. The feedback received from officials during this evaluation was that the MASAM I project kept its Steering Committee officials (in the Cabinet Secretariat, MoF and MECSS) regularly informed of the project’s progress despite changes in personnel. This lesson could have been replicated by GIC. Lessons for the GPSA ToA One objective of this evaluation was to use project insights to verify the GPSA’s ToA at the program-wide level. At the global level the GPSA’s T oA assumes that if governments and civil society had the experience and capacity to problem solve together using collaborative social accountability processes, this would lead to more effective policy reforms, improved service delivery and ripple effects beyond its grant. Drawing on the results of the TAME project, many of the GPSA’s principles were validated. The appropriate set of stakeholders were identified, and they constructively engaged on the issues, aided by the use of specific tools that diffused power imbalances. Capacity was incrementally built for the engagement process across stakeholders. The implementing CSOs gained the most from their civic engagement followed by operational capacities. Modest gains were made by them in terms of technical and adaptive capacities. The principle of adaptive programming and course correction was also demonstrated by the project. By working closely with the World Bank’s Education Team, the GPSA managed to institute the necessary project adjustments to achieve project goals. Results from the project are being sustained by the PTAs and the NSPTA. Ripple effects of the TAME and MASAM projects have been taken into account by MECSS and MoF, along with the community involvement experience in the LDF. Reflecting on the results of such approaches, the government is favorably disposed to the concept of social auditing and has begun to engage with its citizens through the e- Mongolia initiative which also has a citizen’s feedback component. Another GPSA principle that is validated through the TAME project is the call by the GPSA to focus CSA interventions at the intermediate governance and managerial levels for better service delivery. This worked in this case by centering the project at the front line: the schools and the soums; and then extending the intervention beyond the front line to the aimags. 55 7.2 Recommendations The TAME project has demonstrated that civic engagement of herding communities and minority groups to solve their school issues is possible, even in remote locations, when appropriate engagement mechanisms and support tools are made available. Based on the experience of the TAME project and reflections on similar projects implemented in Mongolia, a summary of the recommendations is given below. Matching CSO Capacity with Scale of Project The project’s aspirations at start -up were somewhat ambitious in scale and location. It is recommended that just-in-time support is provided (by the GPSA and World Bank teams) to implementing CSOs especially at project start-up, to set up realistic systems and structures needed for effective project implementation. As was appreciated by the TAME project, sequencing implementation by localities offers CSOs the opportunity to learn and adapt the implementation process. Formulating Realistic and Measurable Results Framework CSOs should choose a manageable number of indicators focused on what is critical to measure and how to measure them. This is another support area that is needed at project initiation to be provided by the GPSA and World Bank teams. Crafting Clear Project Objectives and Project Theory of Action Having a well-crafted project objective and ToA assisted the TAME project implementation in a major way. The project ToA assisted this effort by making the clear assumption that if the relevant school stakeholders were more constructively engaged in education services and oversight then providers of those services would improve their performance and be more accountable. It is recommended that this approach be embedded into similar projects including the MASAM II project. Establishing Structures (PTAs) as the Change Engines The 31 PTAs evaluated in this exercise demonstrated that they can become catalysts for better school governance reforms. Regional officials and mayors have called for the mainstreaming of the PTAs to all soums. Mayors have recommended that the current PTA boards be trained to become more professional, and with more professionals elected as board members. Support to the NSPTA by the GPSA and other donors is recommended to enable this. Only through in-kind or financial support can the NSPTA prepare and deliver a series of year-long capacity building activities that would address these and other challenges. It is necessary for the PTAs to continue engaging with their stakeholders, including the newly appointed regional officials. Using Social Accountability Tools to Engage Given that PTAs have also been trained on the BTT, they could implement both the GSST and BTT annually and thus serve as the accountability mechanism in schools and at the local 56 governments. It is advised that support for this initiative be provided to the NSPTA by the GPSA and other donors such as the Asia Foundation, Mercy Corps, Save the Children and Open Society Foundation. In addition, the NSPTA should liaise with Transparency International Mongolia, the IAAC, MoF and the NAO on school procurement related issues. Having laws that open budgets and procurement to public scrutiny is one thing, making officials aware of these laws and making them engage with stakeholders at the front line and beyond is another. The NSPTA could also manage this task, and the Local Development Fund issue of re-ranking of community projects by community heads, when supported by agencies. Scaling-Up, Sustaining Results and Links with World Bank Projects PTAs have certainly contributed to better accountability and service delivery during the life of the project and beyond. All 31 PTAs and the NSPTA are being sustained post-project. In all project aimags there is demand for replication in non-project soums, as well as other aimags, using the same methodology and process. This is an issue for the GPSA, EQRP and MECSS to consider, and the opportunity for scale-up exists at least until the EQRP ends (expected in 2022). The robust results of the project could help to garner support for this effort from other agencies. 57 References Ali, Rabia. 2019. Implementation Completion and Results Report (ICR) Document - Transparency and Accountability in Mongolian Education (TAME) - P150842 (English). Washington, D.C.: World Bank Group. Bousquet et al. 2012. GPSA Adaptive Principles in the Global Partnership for Social Accountability’s Collaborative Social Accountability Model Fritz, Verena; Levy, Brian; Ort, Rachel. 2014. Problem-Driven Political Economy Analysis: The World Bank's Experience. Directions in Development--Public Sector Governance; Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/16389 Fritz, V.; Kaiser, K.; Levy, B. 2009. Problem-Driven Governance and Political Economy Analysis: Good Practice Framework. World Bank, Washington, DC. https://openknowledge.worldbank.org/handle/10986/16777 Gonzalez Asis, M.; Woolcock, M. 2015. Operationalizing the Science of Delivery Agenda to Enhance Development Results. Washington, DC: World Bank.. https://openknowledge.worldbank.org/handle/10986/23226 GPSA. 2020. The Global Partnership for Social Accountability: Theory of Action . Version September 2, 2020. Washington, DC. https://documents1.worldbank.org/curated/en/425301607358292998/pdf/The-Global- Partnership-for-Social-Accountability-Theory-of-Action.pdf GPSA. 2017. Mid-Term Review of TAME Project Report (unpublished). Mainstreaming Social Accountability in Mongolia Project. 2020. Impact Assessment. Ulaanbaatar. Open Society Forum Mongolia. 2016. Local Budget Transparency Index (In Mongolian). Poli M. & Guerzovich F. 2020. Capacity and Implementation Support Area: Portfolio Performance Review. GPSA Note 11. World Bank. 2018a. “Using Early Grade Learning Assessments to close Learning Gaps in Mongolia.” Feature story. October 1, 2018. https://www.worldbank.org/en/news/feature/2018/10/01/using-early-grade-learning-assessments- to-close-learning-gaps-in-mongolia. World Bank. 2018b. “World Development Report 2018: Learning to Realize Education's Promise.” Washington. D.C. https://openknowledge.worldbank.org/handle/10986/28340 World Bank. 2014. “Mongolia-Education Quality Reform Project” Washington, DC. 58 https://www.worldbank.org/en/news/loans-credits/2014/06/06/mongolia-education-quality- reform-project. World Bank. 2013. “Mongolia: Promoting Social Accountability.” Washington, DC. https://www.worldbank.org/en/news/feature/2013/02/21/mongolia-promoting-social- accountability.print World Bank. 2007 “The Enabling Environment for Social Accountability in Mongolia.” Washington, DC. https://openknowledge.worldbank.org/handle/10986/7752 59