99892 COMMUNITY ENGAGEMENT MECHANISMS FIELD EXPERIMENT IN PAKISTAN SALMAN ASIM • ALI ABBAS • MARIAM ADIL Table of Contents Dedication......................................................................................................................................... 2 Acknowledgments.............................................................................................................................. 2 Glossary............................................................................................................................................. 3 Executive Summary............................................................................................................................ 4 Introduction....................................................................................................................................... 9 Section 1: Design Phase................................................................................................................... 15 Chapter 1: Context and Project Design............................................................................................ 16 Chapter 2: Meeting Materials, Testing and Field Pilots..................................................................... 20 Section 2: Implementation Phase..................................................................................................... 31 Chapter 3: Community Mobilization and Village Meetings............................................................. 32 Chapter 4: Post-Meeting Engagement.............................................................................................. 37 Chapter 5: Elections and Capacity-Building Support....................................................................... 47 Section 3: Analysis............................................................................................................................ 55 Chapter 6: Treatment Receipt Analysis............................................................................................. 56 Chapter 7: Analysis of Text Messages................................................................................................ 61 Chapter 8: Analysis of Capacity-Building Support........................................................................... 66 Conclusion....................................................................................................................................... 71 Bibliography..................................................................................................................................... 72 Annexure.......................................................................................................................................... 74 Annex 1: Snapshots of Select Project Tools................................................................................... 75 Annex 2: Technical Design of the Community Dialogue Platform............................................... 76 Annex 3: Content of Informational Messages and Interactive Voice Response Calls..................... 81 Annex 3a: Informational SMS Messages – Round 1 (Translated).................................................. 81 Annex 3b: Informational SMS Messages – Round 2 (Translated)................................................. 83 Annex 3c: Interactive Voice Response Calls – Round 1 (Translated)............................................. 84 Annex 3d: Interactive Voice Response Calls – Round 2 (Translated)............................................. 86 Annex 4a: Workflow Diagram – Intervention 1 (School Management Committee)...................... 88 Annex 4b: Workflow Diagram – Intervention 2 (Community Dialogue Platform)....................... 89 Annex 5: Timeline of the Intervention......................................................................................... 90 Annex 6: Material from Executive Body Meetings........................................................................ 91 Annex 7: Sorting Algorithm......................................................................................................... 93 Annex 8: Project Cost Estimates................................................................................................... 95 Annex 9: Data on Interactive Voice Response Calls...................................................................... 98 Annex 10: List of Databases....................................................................................................... 105 2 DEDICATION Dedication This project is dedicated to the memory of Mr. Muhammad Nouman Bashir, who died after contracting the Hepatitis E virus during implementation of the project in Sindh province, Pakistan. Mr. Bashir was the project lead for the Weitek Group, a private firm contracted by the World Bank to help with this project. Mr. Bashir will be remembered for his honesty, integrity, professional commitment, and spontaneity, as well as for the warmth with which he in- teracted with his colleagues. Mr. Bashir was a source of inspiration for every member of the project team. He faced challenges with courage, responded to disappointments with patience, and injected immense energy into the project at critical stages. He stood for, and deeply cared about the cause of disadvantaged children in Sindh. The project team plans to raise funds to support the higher education of Mr. Bashir’s children. Those who would like to contribute should contact the corresponding author at: sasim@worldbank.org. Acknowledgments We thank the Government of Sindh’s Education and Literacy Department and the Reform Support Unit, in particular, Saba Mushtaq, Dr. Fazlullah Pechoho, Faisal Ahmed Muhammad, Dr. Hamzo Khan Tagar, Muhammad Nasim Qureshi and Sania Khursheed, for extensive collaboration and assis- tance; the World Bank’s intervention design team, in particular Priyanka Pandey, Lorenza De Icaza, Dr. Fazlullah Pechoho, and Faisal Ahmed; the Weitek Group, Dr. Hamzo Khan Tagar for field imple- mentation; Amar Fateen and Saindad Joyo for implementation support; Rachid Benmessaoud, Amit Dar, Dhushyanth Raju, Margo Hoftijzer and Umbreen Arif for their encouragement and support for this project. We thank our partners, M3 Technologies Private Ltd. for the design and customization of the Communi- ty Dialogue Platform; the Weitek Group for the design, testing and implementation of village-level meet­ ings and training of School Management Committees’ Executive Body members; J. Walter Thompson for the development of audio clips for village meetings; and the Social Policy and Development Centre for third-party monitoring of the intervention. In particular, we would like to thank Mr. Inam Fareed; Mr. Imran H. Naqvi of Weitek Group; Mr. Adnan Lotia; Ms. Farnaz Shama and Ms. Sana Nasim of M3 Technologies; Mr. Omar Murad of JWT; and Mr. Nadeem Ahmed and Mr. Manzoor Memon of the Social Policy and Development Centre for their commitment, dedication and hard work. We gratefully acknowledge the financial support provided for this project by the Strategic Impact Eval- uation Fund, the Education Sector Development Program, and the World Bank. September, 2015 GLOSSARY 3 Glossary CDP Community Dialogue Platform CO Country Office EB Executive Body ELD Education and Literacy Department FMP Field Mobilization Plan GoSindh Government of Sindh HIES Household Integrated Economic Survey HQ Headquarters (World Bank) ICT Information and Communications Technology IFS International Financial Statistics IVR Interactive Voice Response JWT J. Walter Thompson M3Tech M3 Technologies PKR Pakistani Rupee PSLM Pakistan Social and Living Standards Measurement Survey RSU Reform Support Unit SBM School-based Management Reforms SBP State Bank of Pakistan SERP Sindh Education Sector Reform Program SIP School Improvement Plan SMC School Management Committee SPDC Social Policy and Development Centre UC Union Council USD U.S. Dollar WDR World Development Report WB World Bank WG Weitek Group 4 DEDICATION Executive Summary EXECUTIVE SUMMARY 5 Community engagement and direct beneficiary feedback can improve project outcomes under the right conditions. Community-focused projects in developing countries in Asia, Africa and South America have shown that such projects can create both successes and failures. Successes include in- tensive participation by the community, such as in Ghana, where the Community Secondary Schools Construction Project led to district assemblies effectively handling procurement for civil works. In Indonesia, development of links between school committees and local governing bodies, such as village councils, empowered school committees to hold school management accountable, leading to the estab- lishment of village study hours and increased parental satisfaction with the committees’ performance. On the other hand, the domination of school development and monitoring committees by the political elite, and misuse of power by elected representatives in Karnataka, India, led to favoritism in the hiring of teachers, and adversely affected the move towards greater community participation in public education. The World Development Report (2004) highlighted the role of client power and “voice” to in- crease accountability and improve delivery of service. Over the years, the World Bank has supported interventions to improve citizen participation and social accountability in education. In 2013, the Pres- ident’s Delivery Unit1 agreed to incorporate beneficiary feedback in all operational World Bank projects within the next five years. The objective of this report is to contribute to the ongoing dialogue on how to harness citizen engagement to have an impact. The report documents the design, implementation and results of a pilot project aimed at building closer relations between schools and their local com- munities. The project was implemented in three districts in rural Sindh, Pakistan, as part of the Sindh Education Sector Reform Program (SERP-I & II). The Sindh Education Sector Reform Program supported the provision of annual grants to schools, with the aim of increasing the involvement of parents and communities in local schools. This engage- ment was expected to lead to a diffusion of school management powers to parents and other members of the community over time, resulting in better management of schools and improved educational outcomes. At the end of the first phase of the program, the Government of Sindh successfully ensured timely transfers of these grants to more than 80% of functional government schools. However, these grants did not result in sustained community participation in school management. Funds remained underutilized and, in certain cases, were misappropriated. To address this problem, two interventions were designed to empower parents and communities to actively and directly participate in the improvement of education at the local level. The issuance of school grants alone generally failed to bring about greater community involve- ment in the school improvement process. This was primarily due to two reasons. First, only a select group of individuals in the community had access to information about the amount and delivery date of the grant. There was no mechanism in place to inform or rally parents, so that they could monitor the use of the funds. Second, even if community members were aware of the grant, they had limited knowledge about how to engage with the school to improve utilization of these grants. Essentially, the program lacked an interface to connect the community with schools. Such an interface could provide the government with a channel to issue information on school inputs, while giving parents a platform to lobby for school improvement and hold service providers accountable. The interventions were designed, taking into account the theoretical, practical and contextual understanding of rural Sindh. First, villagers were provided with the opportunity to interface with other community members through an externally administered and moderated community meeting, 1 President’s Delivery Unit.” “Established in January 2014, the President’s Delivery Unit (PDU) monitors the World Bank Group’s performance and provides a forum to identify planning or implementation issues related to institutional priorities. 6 EXECUTIVE SUMMARY with parents and other community members convening at the community’s main school. They learned about the activities of the school, about grants made available to the school, and about ways to engage with government schools through school committees. These meetings provided community members with a space to discuss issues related to school performance. The design limitations of this intervention are straightforward. In a feudal society like rural Sindh, socio-economic power tends to be concentrated among the community’s elite, so that many communi- ty members may not feel comfortable engaging in an open dialogue with school administrators during a public meeting. This is particularly true in communities where teachers are patronized by village leaders and politicians, and where community members generally acquiesce to the existing power elite in the village. In addition, villagers might be less willing to attend meetings that take place during working hours and that could cut into their earnings. Another potential obstacle is the direct cost of organizing public meetings. The costs associated with helping village residents organize an initial community-level meeting could be expected to drop with subsequent meetings, when villagers have more experience, but there could still be some intervention costs. The project addressed some of these limitations by leveraging information communication tech- nology. Taking advantage of high mobile phone penetration rates in rural Sindh,2 we developed a sup- plementary, cell phone-based interface, called the Community Dialogue Platform, to link community members to schools. The Platform opened an anonymous, two-way communication channel between the two sides, allowing for an open exchange of information within the community on school-related issues at low cost. The project introduced multiple ‘nudges’ through the Community Dialogue Platform to maintain community focus on local education. Comments, opinions and complaints received from community members were summarized on a weekly basis and sent to all participants registered on the database. The campaign also provided to registered users the names and phone numbers of members of the Executive Body, an elected group of representatives leading the School Management Committee. The Executive Body is an elected group of representatives comprising a subset of the School Management Committee. The Executive Body is tasked with leading the effort to develop the School Improvement Plan and to implement it. Midway through the campaign, the project team obtained feedback from the community through a round of automated calls requesting keypad responses. The team transferred airtime credits to all registered users, to allow those who did not have pre-paid balances on their phones to participate in the conversation. It followed up by sending a series of informational text messages to reinvigorate the dialogue, refocus it on the role of school committees, and prevent drift from the core messages of the project. Twenty eight percent of those who registered for the Platform were active users. But even those who did not send messages directly to the portal found the campaign to be useful. The dialogue generated by the Community Dialogue Platform gives us micro-level insights into the nature, scope and extent of education-related problems faced by local communities in rural Sindh. One of the common themes in the dialogue was the need for regular maintenance of buildings, classrooms and toilets. In addition, communities expressed concern over the shortage of teachers and non-teaching staff, as well as the state of basic facilities, including drinking water supply, boundary walls, furniture, fans and electricity. Numerous villages expressed concern about low-quality teaching and sought better teach- ers for English, Sindhi and Islamic studies. Users also raised concerns about a lack of textbooks at schools; schools’ practice of charging students for textbooks that were supposed to be provided by the government for free; and the failure by district and sub-district officials to regularly monitor schools. 2 Project census data from 2012 indicates that on average, 74 percent of households at the village-level and 78% of households at the main settlement-level have access to a mobile phone. EXECUTIVE SUMMARY 7 The project encouraged the community to use school grants for school maintenance and repairs. Some communities said that school funds from the government were not reaching the schools. Oth- ers said that resources provided were not sufficient to pay for school improvement projects. In some villages, communities complained that appointed individuals misused or misappropriated funds. On the positive side, numerous villages reported that since the initiation of the Community Dialogue Platform, school committee meetings were being held regularly at schools, with some villages also indi- cating that teacher absenteeism had fallen with proactive monitoring by the community. The design of both intervention arms – community meetings and the Community Dialogue Platform – depended on a functioning, appropriately constituted School Management Committee that had the capacity to introduce improvements to the school. One potential scenario would have been to focus solely on bolstering citizen voice. The hope would be that over time, villagers’ increased knowledge of the Committee’s procedural rules and mandate, and their monitoring of school fund usage would lead to the dismissal of weak Committee members and the election of more informed community leaders – strengthening the School Management Committee as an institution. Another possible scenario would be to intervene and accelerate this process by conducting fresh elections, inducting new members and building their capacity to undertake school improvement activities. Both of these approaches have their merits and demerits. We tested both separately, as well as together through a crossover design. Almost half the villages in each intervention were crossed over or complemented by elections and capacity support for the Executive Body members of the school committees. Democratic elections and special training can help to strengthen the capacity of School Management Committees and their Executive Bodies. Facilitators in the field worked with crossover villages to conduct democratic elections of school committees following official guidelines. The newly elected members received hands-on training through three highly structured Executive Body meetings. These meetings familiarized the elected Executive Body members with the functions of the school committee and provided training on how to develop a School Improvement Plan. The members received assistance on how to finalize the Plan and secure ratification at a village-level meeting. Critically, they also received training on the key functions of the Executive Body – withdrawing funds, navigating the procurement process, bookkeeping, and monitoring the implementation of the School Improvement Plan. The development of the School Improvement Plan through community-wide discussions brought interesting facts to the fore, such as the underutilization of existing funds. Improvement Plans are supposed to be prepared by the executive body members in line with the funds available to the school committees. Though a typical school in these districts had approximately PKR 22,000 in funds avail- able at the time of the meeting, a large proportion of schools reported they had accumulated more funds over a number of years. For instance, the schools that fell in the 90th percentile in terms of avail- able funds in district Mirpurkhas and district Mitiari had PKR 77,500 and PKR 62,000, respectively, each equivalent to at least three years of unutilized funds. School Improvement Plans highlighted schools’ most urgent needs. Analysis of data on various line items appearing in these Plans showed that common expenditures across villages included spending on the repairs of ceilings, boundary walls, gates, and toilets, and on the repair and purchase of furniture and other classroom facilities for students. The ability of field teams to canvass and mobilize a sufficiently large section of the target com- munities was critical for the success of the project. On average, participation rates for village-level meetings exceeded the initial threshold of 60 percent across the two treatments. However, there were variations across communities, with participation rates ranging from 21 percent to 100 percent, for 10th 8 EXECUTIVE SUMMARY and 90th percentile villages, respectively. Community meetings were reorganized in 28 villages where participation rates in the first attempt ended up to be less than 20% of the households. A large majority of participants in these meetings had access to at least one cell phone number at the household level. Overall, the average registration rate of 63 percent on the Community Dialogue Platform was consis- tent with estimates of mobile phone penetration rates reported for rural Sindh. Sustaining the community’s interest in the project through a series of activities was key to kin- dling and sustaining the interest of the community to participate in the Community Dialogue Platform initiative. Equally important were the hands-on support and guidance provided by an inter- mediary firm to introduce the community to the Community Dialogue Platform; the identification of local champions (community volunteers and School Management Committee members) to support, spread and encourage the use of this Platform; and the availability of multilingual text support that allowed for the inclusion of the dominant local languages spoken in rural Sindh. The successful implementation of the project interventions entailed two key elements. First, the facilitators in the field must have the capacity to implement the designed intervention. Second, a large number of target participants must be exposed to the intervention as intended. To ensure the former, the project team hired two specialized private firms with demonstrated prior experience to support the design of materials and community mobilization efforts. Multiple safeguards were put in place to discourage the firms from shirking their responsibilities in an effort to reduce costs. For the village-level meeting to be considered successful, at least 40% of the households in the target community had to attend; otherwise, another meeting had to be scheduled.3 In addition, the team introduced measures to ensure that the field facilitators could execute the treatment and convey the message to the community as intended. These measures included standardized intervention protocols and delivery methods, along with intensive in-class and field-based training of facilitators. Periodic feedback from a third-party monitoring firm and regular supervision by the project team played a critical role in keeping implementation on track. The interventions discussed in this report sought to repair the broken feedback loop between local communities and schools via School Management Committees. The novel element of the project was to link citizen-voice mechanisms to grassroots institutions such as School Management Committees, as well as to empower and enable citizens to effect positive change in service delivery. We followed a 360-degree design approach for this project, targeting the issue of a lack of community par- ticipation from different angles. Mobilization efforts, village-level meetings, and dialogue initiated on the Community Dialogue Platform all sought to raise awareness and process knowledge among com- munity members, so that they could engage effectively with the school. The dialogue relayed commu- nity-identified needs to the elected representatives of school committees. The project also encouraged the community to proactively monitor the use of school funds. We deliberately designed interventions to create a self-sustaining, community-managed loop to identify, manage and monitor school improve- ment activities with little dependence on government departments. Further, since the design of the Platform created a feedback mechanism enabling communities to receive information and engage with schools via school committees, this intervention fell within the broader class of citizen-feedback models that combine information technology with grassroots institutions (school committees) for improved service delivery. During the intervention rollout, the project team collected process data, which it sub- sequently analyzed to produce the results reported in this study. 3 This benchmark was later relaxed to 20%, due to the high average participation rates achieved by the firms. DEDICATION 9 Introduction 10 INTRODUCTION Most governments around the world have been successful in getting children into schools, but not all of them have been able to impart education to those enrolled (Pritchett, 2014). This large-scale government failure to provide quality education at the point of delivery is the result of weak public sector management, governance and capacity (World Bank, Project Document SERP-II). Providing high-quality teacher-child experiences in schools daily calls for an active role by communities and parents in their schools, such as promoted through school-based management reforms. Such reforms primarily seek to shift administra- tive responsibilities and authority to the level of schools. The hope is that such a move would increase accountability and transparency in the education system, and make education services more responsive to the need of end users. The development of a strong local “voice” is thought to have the power to reshape education outcomes by affecting the level and mix of education inputs that reach the school, as well as by improving the efficiency with which these resources are used (Bruns et al. 2011). Giving a voice to local stakeholders, should therefore lead to better learning outcomes for children enrolled in school. Projects aimed at enhancing the voice of the community in developing countries have revealed new op- portunities for these communities. In Ghana, the World Bank collaborated with the Government to provide matching grants for 140 local communities that were ready to implement development projects. The communities received two-thirds of funding for local projects if they could cover the remaining cost in cash, building material or labor. Under this program, called the Community Secondary Schools Con- struction Project, district assemblies effectively handled procurement for civil works. It was later found that local leadership played a key role in social mobilization, though initial field surveys did not capture data on the degree of leadership availability in a particular community.4 Other projects aimed at boosting community voice produced mixed results. In Indonesia, a phased-in field experiment provided grants and training, along with the holding of elections and the establish- ment of links with local government bodies. The provision of grants and training appeared to have little or no effect on education indicators. But the elections and the creation of linkages with local political stakeholders positively affected education outcomes in the target area, which comprised 520 rural pub- lic schools in six districts in Central Java and Yogyakarta, Indonesia.5 Structural constraints have also affected the ability of school bodies to actualize their potential. After the creation of School Development and Monitoring Committees in Karnataka, India, members of the Leg- islative Assembly pressured the government to give significant powers to the assembly so that they could affect the workings of the school committees. The legislators secured authority to nominate the president of the School Development and Monitoring Committees, and the nine parent members of the body.6 Such political interference in community-based organizations is prevalent across the border in Pakistan as well, where feudal tendencies have adversely affected the power dynamic across communities. A key manifestation is the use of influence by local power holders to get their favorites hired as head teachers, reducing the “voice” of the community and politicizing the school hiring process.7 4 Uemura, Mitsue. 1999. Community participation in education: what do we know?. Washington, DC: World Bank. . 5 Pradhan, M., Suryadarma, D., Beatty A., Wong, M., Alishjabana, A., Gaduh, A., & Artha, R.. Improving Educational Quality through Enhancing Community Participation: Results from a Randomized Field Experiment in Indonesia. Policy Research Working Papers. The World Bank. September 2011. 6 Oxfam India. (2014). Community Participation and Institutional Experiences in School Education: School Development and Monitoring Committees in Karnataka. New Delhi. . 7 Ali, S. (2006). Why does policy fail? Understanding the problems of policy implementation in Pakistan - a neuro-cognitive perspective. International Studies in Educational Administration, 34(1). < http://ecommons.aku.edu/cgi/viewcontent.cgi? article=1009&context=pakistan_ied_pdck>. INTRODUCTION 11 More recently, development agencies have encouraged and supported governments to implement re- forms aimed at empowering local stakeholders in school-based management. This has led to the emer- gence – or in some instances, revitalization – of School Management Committees comprised of par- ents, community members and teachers. However, emerging evidence from developing countries has not been very promising; even after several years of de jure existence, school committees have remained largely non-functional. In some cases, Committee members were not even aware of the existence of such Committees, let alone understanding their expected participation in these bodies. In other in- stances, when these Committees have been found to be functional, community participation in these organizations has remained low (Pandey et al., 2009, and Banerjee et al., 2010). The situation is not much different in rural Sindh, Pakistan. Under the Sindh Education Sector Reform Program undertaken by the Government of Sindh, School Management Committees were re-acti- vated as a formal channel through which local communities could engage with government schools. Each Committee receives an annual grant from the Government worth PKR 22,000 to pay for school improvement projects. The Committees have total control over these funds and may withdraw them as needed without seeking authorization from line authorities. With more than 80% of functional schools in the province receiving grants, the Committees are institutionally established and receive funds from the Government every year. However, there is little evidence to suggest that these funds are properly utilized – if they are used at all. To find a cost-effective solution to this problem and to strengthen the link between communities and schools, we designed and implemented two complementary community-engagement inter- ventions. The first intervention sought to mobilize the community through an externally admin- istered, large village-level meeting to provide information on the rights, roles and responsibilities of parents and the community. It also provided these stakeholders with options for engaging with Government-run primary schools via School Management Committees. The second intervention consisted of a mobile phone-based interface, called the Community Dialogue Platform, that har- nessed text messaging services to build an anonymous, two-way communication channel between villagers and their schools. Both approaches provided communities with a platform to receive school-related information, to ex- change views on education issues and to ensure that chosen representatives on the School Management Committee followed up on the community’s recommendations for school improvement. To address any capacity constraints at the Committee level, we used a crossover design to layer these two interven- tions with transparent elections of Committee members and participatory training to help members develop a School Improvement Plan. These interventions are currently being evaluated. This project built upon a new wave of progress that tapped information and communications tech- nology solutions to empower citizens, strengthen accountability and improve service delivery. It is widely acknowledged that information provision and effective communication with beneficiaries play an instrumental role in improving governance (Coffey, 2007). In this project, we extended this thinking by designing mechanisms in which “voice” and accountability were central; we created an inclusive public space for dialogue and debate between communities and schools. In doing so, we facilitated a freer flow of information among the Government, schools and communities, and opened up opportunities for local stakeholders to participate directly in decisions that affect them. The creation of this feedback mechanism among key stakeholders in the education sector places this intervention in the larger category of citizen-feedback models that combine elements of technology solutions with local institutions – in this case, School Management Committees – to improve the provision of social services. 12 INTRODUCTION We expect to see improvements in underlying accountability relationships resulting from stronger link- ages between communities and schools. The teacher is the main instrument for learning in any school, so any changes in underlying accountability relationships have the power to affect teachers’ effort in schools, as measured by attendance, time devoted to preparing lesson plans and other teacher on-task measures. This, in turn, can impact student achievement as measured by performance on standard cognitive tests. These interventions are of particular importance in the context of rural Sindh. Community members find it difficult to alter their relationship with power elites and the Government in the context of the public education system, leaving them with the sole option of voicing their discontent, in the hope that the Government will consequently improve its performance.8 Several factors have traditionally limited community members’ ability to implement change. Private schools are virtually non-existent, making it harder, if not impossible, for parents to withdraw their children from public schools and send them to private schools. In the absence of market-based ac- countability, the only option for discontented parents is to directly raise any concerns they may have with educators. The interventions sought to create an interface that connected school administrators with parents, opening up opportunities for parents and other community members to take part in the school improvement process. This was done through a traditional, face-to-face meeting between the community and teachers. However, we were cognizant of the limitations of such meetings in a feudal society such as Sindh. Citi- zens here can easily feel intimidated and may find it easier to acquiesce to the status quo.To address this issue, the project team created a Community Dialogue Platform to positively disrupt the system. The Platform virtually connected community members with each other and with school administrators. It provided them with a medium to receive education-related information, anonymously exchange views, and ensure that their chosen representatives in school committees worked on their suggestions for school improvement. The overall objective was to elicit and sustain meaningful participation by the community in the man- agement of schools. This could only be possible if the interface was integrated with local institutions. For this reason, School Management Committees in approximately half of the sample villages were strengthened through elections and capacity-building support to enable Committee members to effec- tively respond to community-identified needs. The primary purpose of this report is to record the rationale and motivations behind the decisions taken by the project team during the project’s design and implementation, and to catalyze a candid discussion about the challenges faced during implementation. The report is divided into three sections. The first section deals with the design phase of the project. There are two chapters in this section. The first lays out the framework and context for project design; the second discusses the design and testing of instruments, the portal, manuals and training of field facilitators. Section Two focuses on the imple- mentation phase of the project. The first chapter in this section reviews village mobilization efforts and the convening of village-level meetings. The second chapter documents the post-meeting engagement process through the Community Dialogue Platform, while the third chapter reviews capacity-building support for newly constituted School Management Committees. 8 Hirschman, Albert O. Exit, Voice and Loyalty: Responses to the Decline in Firms, Organizations and State. Cambridge: Harvard UP, 1978. Print. INTRODUCTION 13 The third and the final section reports the key findings from collation, synthesis and analysis of the process data collected for all interventions in this project. The first chapter in this section reports sta- tistics on measures of participation and other indicators to measure treatment fidelity – strategies that measure accuracy and consistency of interventions. The second chapter analyzes text messaging traffic generated on the portal for the duration of the campaign, while the final chapter gives detailed expen- ditures analysis of School Improvement Plans developed by the School Management Committees. Section 1: Design Phase Were the interventions designed to include ingredients from theory and practice? 16 CONTEXT AND PROJECT DESIGN Chapter 1: Context and Project Design This field experiment was designed and implemented to accelerate the revitalization of School Management Com- mittees in rural Sindh – an important sub-program for improving the quality and performance of government schools supported under the Sindh Education Sector Reform Program (SERP-I and SERP-II). The experiment is currently undergoing rigorous evaluation. Lack of awareness among local communities about their roles and respon- sibilities in managing School Management Committees, as well as communities’ lack of capacity in executing these responsibilities, have contributed to less-than-optimal utilization of Committee funds. To address these gaps, the project team designed two interventions to inform, enable and empower communities to become more engaged in the school improvement process. The first treatment arm used community-level meetings to provide informa- tion on roles, responsibilities, rights, and options for recourse in engaging with government schools via School Management Committees. The second arm provided similar information and a forum for community discussion through a mobile phone-based Community Dialogue Platform. An additional crossover element introduced elections and capacity support into the School Management Committee’s leadership group, called the Executive Body, to ensure its members had the capability to carry out their responsibilities. Encouraged by this field ex- periment, the Government of Sindh has started developing similar citizen-engagement mechanisms under the Information and Communications subcomponent of the Sindh Global Partnership for Education project. Sindh Education Sector Sindh province has a population of approximately 42.4 million,9 accounting for roughly one-quarter of Pakistan’s total population. The province has a net enrollment rate at the primary level of 53%, with large differentials between urban areas (63%) and rural areas (45%). This is lower than the national average of 57%.10 There are 46,724 government schools in Sindh, 40,517 of which are functional. Among the total, 42,342 are primary schools, 2,336 are middle/elementary schools,11 1,752 are sec- ondary schools and 294 are higher secondary schools.12 In total, the schools have approximately 4.08 million students and 145,438 working teachers, including 100,322 male teachers and 45,116 female teachers. 141,320 of these are government teachers.13 As is the case with other countries in South Asia, the public education system in Sindh, Pakistan, is rid- dled with problems of teacher absenteeism, high dropout rates and poor maintenance of school facilities. Under the Sindh Education Sector Reform Program undertaken by the Government, School Manage- ment Committees were reactivated as a formal channel for local communities to engage with government schools and to address some of these problems. School Management Committees in primary schools re- ceive annual grants from the Government worth PKR 22,000 each to finance school improvements. The administrative performance and financial management of the Committees are subject to review by parents and the wider local community. Communities vet and approve the School Improvement Plan prepared by the Committees in a village-level meeting, and monitor the plan’s implementation. 9 Population Welfare Department, Sindh < http://www.pwdsindh.gov.pk/>. 10 Pakistan Bureau of Statistics. Pakistan Social and Living Standards Measurement Survey (PSLM) 2012-13 Provincial / District. Government of Pakistan. 2013. 11 Schools covering grades 6, 7 and 8 are known as middle/elementary in Pakistan. 12 Secondary schools cover grades 9 and 10, while higher secondary schools cover grades 11 and 12. 13 Reform Support Unit. Annual Schools Census 2013-14. Education and Literacy Department, Government of Sindh. 2014. . CONTEXT AND PROJECT DESIGN 17 Design of Interventions This pilot project is part of an ongoing World Bank program to provide technical and advisory support to the Government of Sindh and is aimed at improving the quality and performance of government pri- mary schools under the multi-pronged Sindh Education Sector Reform Program (SERP-II). The Bank worked with the Reform Support Unit, the implementation arm of the Sindh Government’s Education and Literacy Department, to design the interventions, which were piloted in select districts of rural Sindh. The aim of these interventions was to explore concrete ways to elicit meaningful and sustained local community engagement to improve education outcomes. More importantly, the interventions were to serve as a first experimental evaluation of a class of redressal systems, promoted by the World Bank to incorporate beneficiary feedback in operational projects. The World Bank hired consultancy firms J. Walter Thompson (JWT), Weitek Group and M3 Tech- nologies (M3Tech) to assist with the design and implementation of the project. An intervention team comprised of international and local consultants closely supervised the effort, working closely with the Reform Support Unit to develop materials for village-level mobilization and large-scale meetings at the village level. JWT, an advertising, marketing and communications firm, converted the content of the meetings into standardized audio clips – distinct for each intervention – that were played at all commu- nity meetings. Weitek Group’s social mobilization and media wing created posters, banners, flip charts and mosque announcements, as well as the field strategy for mobilizing the community to attend these meetings. M3 Technologies, a technology firm that specializes in text messaging services, developed and provided “back-end” support for implementing the Community Dialogue Platform that virtually connected community members in each village. Fidelity to Theory and Practice The pilot interventions were designed to elicit meaningful and sustained participation by communities in helping to manage local schools. The Sindh Education Sector Reform Program (SERP-I) stream- lined the process for issuing annual school grants to schools in a timely manner. However, substantive weaknesses in school management systems at the community level resulted in the underutilization and misappropriation of these funds. These interventions were designed to address this challenge by enabling and empowering the community to actively participate in the school improvement process. Communities had to be informed of their rights, roles and responsibilities, and they were provided with options for engaging with government schools. The Government of Sindh had prepared and dis- tributed School Management Guidelines to communities to facilitate their involvement, but its effort had limited success in rural Sindh, where a large majority of villagers are illiterate. Further, mechanisms to strengthen linkages between communities and schools were not functional on the ground. Take School Management Committees, which are supposed to serve as an official channel of communication between communities and local schools. Such committees are comprised of a General Body, made up of parents, and a five-member Executive Body, made up of the head teacher and community represen- tatives. In theory, with the exception of the head teacher, the members of the Executive Body should be democratically elected and meet regularly to oversee school improvements, while the General Body should meet at least biannually. In practice, at a large majority of schools, head teachers and community elites tended to choose the Executive Body members and the Committees rarely held General Body meetings. When School Management Committees were functional, their Executive Body members often lacked training and capacity to act on community-identified needs for school improvement. 18 CONTEXT AND PROJECT DESIGN Selection of Study Districts Three districts in rural Sindh were selected for the implementation of these interventions: Mitiari, Mir- purkhas, and Sanghar. We used the Pakistan Social and Living Standards Measurement Survey to rank the districts based on two basic education indicators: the proportion of adults who have ever attended school, and school participation rates of primary age children (5-12 years). Using administrative school census data, we ranked districts according to size, measured by the number of schools and villages in each district. Out of a total 28 districts in the province, Mitiari was ranked the third-smallest, Mir- purkhas was ranked 12th and Sanghar was ranked 18th. In terms of education indicators, Mirpurkhas had one of the lowest levels of education outcomes, followed by Mitiari (close to the median), while Sanghar ranked among the highest. Overall, these three districts are a relatively representative sample of rural Sindh and were deemed relatively safe for the field teams to operate in. In these three districts, a total of 287 sample villages were targeted for the interventions. Impact Evaluation Design The project team randomly assigned the 287 villages to one of five groups that were part of the project: T1 – Control: No information or capacity-building intervention was undertaken. T2 – INFO-MEET: A face-to-face meeting was organized for all stakeholders (parents, teachers, village leaders and council members). Participants received information packages, including scripts for recording audio clips, posters and pamphlets that had been designed to explain the roles and responsibilities of parents and community members, and to encourage participation in school-level management and decision-making. The participants were encouraged and given time to discuss school-related issues during these meetings. T3 – INFO-SMS: A face-to-face meeting was organized for all stakeholders (parents, teachers, vil- lage leaders and council members) to introduce the community to an information and communi- cations technology-enabled text messaging platform. This platform creates a virtual network of all stakeholders in the community. It allowed for the sharing of key messages derived from the content of the information package that was disseminated during the face-to-face meeting, and framed similarly to tweets on Twitter. This reinforced key messages shared during the initial meeting. Participants were encouraged to provide comments and feedback in response to messages shared through the virtual platform. The platform also sent periodic summary messages compiled from the comments received from all registered users. Automated Interactive Voice Response calls were used to gather feedback from the community regarding the dialogue platform. T4 – INFO-MEET-SUPPORT:14 In addition to information exchange (T2), under this inter- vention, school committees were reconstituted through elections according to official guidelines. Once elected, the newly constituted Executive Body had three meetings, totaling three hours each. During these meetings, newly elected members received capacity-building support and resources that they needed to perform their expected roles and responsibilities. 14 INFO-MEET is interchangeably used with “SMC Intervention” in the report. INFO-MEET-SUPPORT is also referred to as SMC Intervention with Elections and Capacity Support. CONTEXT AND PROJECT DESIGN 19 T5 – INFO-SMS-SUPPORT15: In addition to information exchange (T3), under this interven- tion, school committees were reconstituted through elections according to official guidelines. Once elected, the new Executive Body held three meetings, totaling three hours each. During these meetings, newly elected members received capacity-building support and resources that they needed to perform their expected roles and responsibilities. 15 INFO-SMS is interchangeably used with “SMS Intervention” or “CDP Intervention” in the report. INFO-SMS-SUPPORT is also referred to as SMS Intervention with Elections and Capacity Support. 20 MEETING MATERIALS, TESTING AND FIELD PILOTS Chapter 2: Meeting Materials, Testing and Field Pilots The interventions were designed meticulously so that different arms of the project not only included relevant “active ingredients” based on theory and practice, but also applied international best practices to rural Sindh. This was essential for ensuring successful project implementation. The design was aligned with multiple international best practices that attempt to improve transparency and accountability at the local level. These included projects such as Check My School Program, a citizen-feedback platform in the Philippines; On Track, a citizen-feedback system that allows residents in marginalized communities in Bolivia to report problems related to public services; Daraja, a citizen-monitoring and -feedback initiative on water services in rural Tanzania; and Map Tandale, an interactive map used to provide detailed information on education, health, water, accessibility and security needs to commu- nities in Tanzania. Using this rich knowledge base and an innovative design, the project team attempted to address limitations that had derailed earlier attempts to affect change in educational governance in rural Sindh. Introduction to Project Design Designing all elements of the project, calibrating its different components and finalizing all instruments for project rollout took a significant amount of time. Initial design activities began in July 2011, con- tinuing until the end of December 2012. This 18-month period allowed for different stakeholders to come together, contribute towards the development of project tools, and refine the project process to ensure successful implementation. The task that took the longest was the development of the Com- munity Dialogue Platform, the key technology-based solution for initiating two-way communication among community members, raising their awareness and strengthening accountability at the local level. The design attempted to tackle multiple constraints faced by villagers, including skewed power dynam- ics at the local level, lack of transparency and accountability, low informational awareness and high op- portunity costs in terms of time spent away from earning livelihoods. The project team generally – and the facilitators, specifically – sought to ensure that the village meetings held as part of the project were inclusive, with every individual given an equal opportunity to participate, irrespective of gender, caste or creed. Project tools such as the Community Dialogue Platform were designed to be inexpensive for villagers to use, to encourage high participation rates. The project team also sought to minimize the time commitment of participants to encourage greater engagement with schools. This chapter documents the design of project instruments and details the progressive development of project design. The first part of the chapter discusses the design, content and testing of the instruments. The second section provides insights into the process for developing the Community Dialogue Plat- form and its testing. The third section delves into how the project team trained field facilitators. The chapter closes with a brief discussion about the costs of different elements of the design phase. MEETING MATERIALS, TESTING AND FIELD PILOTS 21 Design, Content and Testing of Instruments for Village Mobilization and General Body Meetings Design and Content The intervention team worked with the Reform Support Unit of the Sindh Government, implement- ing partner Weitek Group, and JWT to develop materials used during village meetings. Standardized meeting materials designed for the intervention included: 1) an information package; 2) audio clips; 3) flip charts; 4) facilitator scripts; and 5) a field mobilization plan. These materials were produced through a process which ensured gradual design improvements, undergoing multiple rounds of testing and refinement. The final version incorporated changes based on feedback received from partner firms and the World Bank project team. The project team used a range of meeting materials in hopes that the messages could be delivered consistently across villages in an interactive way, with reinforcement of critical information. Information Package: At this stage, the design team, in consultation with local stakeholders, devel- oped a set of information messages that were to be delivered to communities through all the interven- tions. This information package had three key goals: o Inform the community about School Management Committee grants and give suggestions on how to monitor use of these funds; o Introduce the idea of the Community Dialogue Platform by explaining its purpose and uses; and o Explain to community members their roles and responsibilities in terms of maintaining the transparent functioning of the School Management Committee. and how they could help to improve education outcomes for the community. Audio Clips: The three components of the information package were converted into audio clips. These were: 1) basic information on School Management Committees; 2) demonstration of the Community Dialogue Platform; and 3) detailed information on the school committees. A main goal was to control the quality of delivery of these messages and to ensure that the key messages were de- livered consistently across treatment villages. Two audio clips were produced. The first provided in- formation on the structure, funding sources, rules and regulations, as well as benefits, of a functional School Management Committee. The second focused exclusively on the Community Dialogue Plat- form, explaining its purpose, process and uses. Based on a feedback cycle, the audio clips were re- vised multiple times, from the original, poem-based format, and finally to an audio drama. The two principles for the development of the audio clips were that they had to provide relevant information succinctly, and they had to be catchy and appealing to the audience. An international marketing and communication firm, JWT Worldwide, was engaged to develop the story plot in drama format. The design team also sought to make the language of the audio clips easy to understand, and to adapt the clip to the context of rural Sindh. Flip Charts: The design team used flip charts to recap the content of the audio clips during vil- lage meetings, to make sure that the audience didn’t miss any important points. The information in the flip charts was precise and easy to understand, covering all the major ideas discussed in the audio clips. 22 MEETING MATERIALS, TESTING AND FIELD PILOTS Facilitator Script: The key motivation behind developing a detailed facilitator script was to standardize the structure of the village-level meeting. The village meeting was divided into eight, timed parts. For each part, the facilitator used a prepared script to engage with the audience. The design team sought to keep the scripts clear and concise, with the aim of lending more structure to the meeting and expediting the facilitator’s job. Since most of the target audience spoke Sindhi, the script was translated into Sindhi to make it easier to understand. The team tailored the content of the scripts for the School Management Committee and the SMS interventions so that the scripts were aligned with each intervention’s objectives. Field Mobilization Plan: The project team developed an exhaustive guide for how to hold village meetings, to serve as the go-to document for project implementers in the field. While tool development and training were necessary to prepare implementation teams for project rollout, organizers felt there was also a need for a comprehensive manual that field teams could consult when rolling out different aspects of the project. The Field Mobilization Plan incorporated feedback from all the stakeholders, including the World Bank team and partner organizations. Once the Plan was finalized, it was given to field facilitators and their supervisory staff. This document guided all field activities over the course of the project, and was critical to its successful implementation. The Plan reduced the team’s reliance on training refreshers, decreasing potential project costs. The Field Mobilization Plan also provided instructions on activities that field teams were expected to implement to mobilize communities for village-level meetings. It provided specific details on when field facilitators should arrive in the treatment villages, how they should undertake the final round of mobilization prior to the meeting, and what precise tools and instruments (such as audio equipment and flip charts) they were expected to check and set up before the start of the meeting. The Plan also provided suggestions on the sequence of actions that were to be taken from the beginning of the meeting until the end. These included do’s and don’ts for the meeting; for example, field facilitators should ensure that the scripts were in their hands during the meeting. For example, the field facilitators should not say more than they were expected to, or accept gifts from participants. They should prevent disruption and drift during the meeting. The Plan provided details on how to take attendance, how to transfer data to back-end Community Dialogue Platform operators,16 and how to moderate the post-meeting discussion. Annex 1 provides photos of select instruments that were designed for the project. Testing of Instruments Central to the success of the implementation of this intervention was the ability of the field teams to mobilize the community to attend the meetings. The teams made significant efforts to identify and test the most effective materials and activities for attracting community members to the village-level meet- ing. Implementation partner Weitek Group helped to design banners, flyers and balloons. All of these materials had the name of the campaign, “Bachon Sai Pyaar Ilm ki pukaar” (literally translated as “The call to education is love for children”), as well as the venue and time of the meeting printed on them. The team also tested four softer catalysts, or “nudges”, aimed at increasing community participation: 1) mosque announcements; 2) student performances; 3) a campaign to rally village elders and youth; and 4) a pre-recorded musical jingle that was broadcast via megaphone during door-to-door mobilization. Tech support was provided to the portal at the backend. These project team members summarized the messages received from 16 the field and blasted them back to the community. They were also responsible for sending informational messages and transferring mobile phone credit to communities. MEETING MATERIALS, TESTING AND FIELD PILOTS 23 These materials and activities were tested in three villages in rural Sindh. These three villages, however, were not part of the overall sample of villages selected for the rollout of the interventions. Three banners were displayed in each village: one at a key location in the village and the other two in front of schools. Children in the schools received flyers, with an average of 40 flyers distributed per school. Ten flyers were to be fixed to the perimeter of each banner (at least four per banner). Ten flyers were displayed with balloons in strategic locations, such as at the school gate, and in local schools and community gathering areas, known as autaaq. Different distribution mechanisms were tested across pilot villages. In one village, flyers were distributed to at least 25 randomly chosen households per settlement in the village, instead of in schools, for a maximum of 50 flyers. Thirty balloons were strategically placed in the village, with ten put together with flyers in three strategic locations; ten placed in the main settlement; and ten in the peripheral settlement. One day before the meeting, mosque announcements were made before each of the two evening calls to prayer. On the day of the meeting, two announcements were made: one, two hours before the meeting; the second, just before the meeting. The head teacher was requested to work with a group of students to prepare a performance at the village meeting. Finally, the field teams contacted village elders, influential community members and youth to ask them to rally parents and community members to attend these meetings. The field teams conducted focus group discussions and filled observation questionnaires to get a sense from villagers about which combination of materials and activities was seen as most effective in ensuring maximum village participation. The lessons and insights drawn from the focus group discussions were used to fix the parameters of the strategy for field mobilization. The most effective strategies included rallying of community members by the head teacher and the students’ performance in the school. Mosque announcements, door-to-door campaigning with a megaphone, and flyers were perceived to be more effective by the focus group participants compared to other catalysts. The megaphone announcements by the field team on the day of the meeting were particularly effective in getting more female villagers to attend the meeting. Design and Testing of Community Dialogue Platform Learning from Best Practices The project team’s first goal was to document best practices in the sector, especially in the context of in- formation technology supported development initiatives. Innovative international grassroots programs are leveraging the power of mobile phones and text messages to make the development process more trans- parent, effective and inclusive. Such programs include On Track, a citizen-feedback system that allows residents in marginalized communities in Bolivia to report problems related to public services; Daraja, an innovative platform in Tanzania that seeks to bring water problems to the attention of local governments and media organizations; Check My Schools, a participatory monitoring platform in the Philippines that allows users to upload information on schools through a number of social media platforms; and Map Tandale, an interactive map used to provide detailed information on education, health, water, accessibility and security needs to communities in Dar es Salaam, Tanzania. The design of the Community Dialogue Platform builds upon the experiences and lessons drawn from these interventions and was carefully vetted and reduced to feasible elements that were most relevant and amenable to the technology infrastructure and socio-economic context of rural Sindh. The team also consulted academics and practitioners who had experience in implementing technology-based interventions locally.17 17 Many of these discussions and ideas generated from them are provided at the following blog: http://edupolicydialogue. wordpress.com/. 24 MEETING MATERIALS, TESTING AND FIELD PILOTS Following this process, the project team decided that the design of the virtual portal should provide information to local community members; engage them and foster dialogue among them; be easy and relatively inexpensive to use; provide multilingual support; and more broadly, assist in improving edu- cational outcomes at the school level in treatment villages. The development of the text messaging-based Platform began in January 2012. Over the next year, the portal went through multiple developmental iterations. During this period, field testing assessed the portal’s response to user-generated text messages at the village level. The next section provides details on the origins of the portal, the multiple iterations that it went through and the formal pilot tests that were conducted. The Community Dialogue Platform’s Origins and Pre-Pilot Recalibration The portal’s most basic form borrowed from a previous World Bank project being undertaken under the Sindh Education Sector Reform Program (SERP-I).18 After testing began, the project team realized that the existing portal had two fundamental weaknesses in terms of the objectives of the text messaging intervention. First, it was designed to primarily serve as a mechanism to redress complaints, whereby participants used the portal to inform the project team about issues involving the state of education at the local level. Second, the portal collected contact information and other identification details only for teachers, not for parents, who play the primary role in making decisions about their children’s education. These weaknesses made it necessary to overhaul the platform. A local firm, M3 Technologies, provided technical support for this revamp. The portal was reprogrammed to create an outgoing channel for messages, in addition to accepting incoming messages that contained participant feedback. This gave the project team the capability to blast text messages to participants, summarizing weekly discussions at the village level, as well as to send informational messages about School Management Committees and other education-related issues. The idea was to give community members a platform for discussing and finding solutions for education-related problems, and not just for registering complaints. The project team recognized that it needed to ensure that participants understood the portal’s main purpose: that it was to be used by the local community both to discuss problems and to collectively come up with solutions. The informational messages were generally extracted from information in the audiotapes. The project team broke down the script of the audiotapes into shorter sentences, which were sent to villages in the form of individual text messages. Each text message contained a sentence or two; combined, they summed up the information provided in the audiotapes. The project team tested and calibrated the final set of text messages to ensure clarity. Chapter 4 provides further details on these informational text messages. The use of mass social communication platforms such as Facebook, Whatsapp and Twitter is common in urban centers in Pakistan, and urban citizens are quite familiar with how technology-based social interaction mechanisms work. However, this is not the case for local communities in rural areas of Sindh, where the use of social media is highly limited. An average villager does not understand how chatting 18 Complaints Management System was established under the World Bank’s partnership with GoSindh. MEETING MATERIALS, TESTING AND FIELD PILOTS 25 works, or how a dialogue about social issues can be initiated on a communal, technology-based communication platform. Further, villagers are not always comfortable sharing their opinions with the entire community on a public forum. To make the system user-friendly, communication on the portal was restricted solely to text messaging. Given the high rate of mobile penetration in Pakistan (approximately 70%; Evans 2013), the use of test messaging-based communication was the best way to ensure that participants felt technologically com- fortable while interacting on the portal. To give villagers the confidence to publicly state their opinions and share feedback, the project team made interaction on the portal anonymous. While the villagers received incoming messages on their individual phones, the core ideas in these messages were shared with all community members in a weekly summary sent to all community members. This summary included key ideas discussed during the weekly conversation, but protected the identity of villagers. The scope of the portal’s database was also expanded to include other stakeholders besides teachers, including parents, School Management Committee members, other community members and village volunteers.19 Thus, the project team made an effort to take the conversation about education from in- side the school to the community. While teachers are important stakeholders at the school level, parents have a direct stake in the quality of education that their children receive. Piloting the Portal This section provides insights into two major testing phases during the portal’s development. The first major pilot exercise took place between June 6, 2012 and June 14, 2012. Field-testing of the portal was tied in with the piloting phase of the entire project, which was rolled out in four villages. In terms of the portal, the primary aim of the piloting process was to assess its feasibility, as well as its adaptability to local conditions. During the General Body meeting, the field facilitator introduced villagers to the Community Dialogue Platform, conducted a demonstration exercise whereby villagers were invited to send messages to a dedicated mobile phone number – also known as a long code20 – of the Platform and receive summary messages identifying the primary reasons behind lagging edu- cational outcomes in the village. Finally, they were able to register on the portal. Ensuring ease of access (through the use of text messaging) and anonymity of users proved key to get- ting community buy-in for using the portal. Villagers expressed concern during General Body meetings over the possibility of being identified with their opinions, as they did not wish to antagonize power holders in the government or in the informal feudal economy. The project team eased these concerns by emphasizing that the portal did not give any identifying information about participants who had sent messages, and as a result, participants appeared more enthusiastic about using the Platform. The team found that villagers were at times more comfortable sending messages in Sindhi instead of English or Urdu. Thus, adjustments were made for the Community Dialogue Platform to accommodate Sindhi script, together with English, Urdu and Roman Urdu. Technology partner firm M3Tech worked with Motorola to develop a script for the portal so that it could identify, store and display all four languages. Since a significant portion of village members were illiterate, the project team was concerned that many participants would not be able to use even their mobile phones to participate on the portal. To address 19 These included both parent and non-parent School Management Committee members. 20 A long code is simply a phone number which people can text or call. Long codes are just like a mobile phone number. The only reason why it is called ‘long’ is because it is longer than a short code which is a dedicated number and usually four digits. 26 MEETING MATERIALS, TESTING AND FIELD PILOTS this concern, each village selected two literate community volunteers to assist villagers in sending mes- sages to the portal. The Platform was tweaked so that it would take into account the identifier code of these volunteers, indicating that they were sending a message on behalf of another participant using the participant’s mobile phone. A lack of airtime minutes in their phones21 could hinder participants from sending text messages and engaging with the portal. To address this issue, the project team added a feature to the Platform so that team members could directly transfer credit to participants. Transferring credit in real time during Gen- eral Body meetings proved challenging; when villagers were asked to give their contact information for credit transfer, people rushed towards the moderator, leading to chaos. This resulted in mistakes, such as credit being transferred to one number multiple times and not at all to other numbers. The project team ultimately decided to transfer credit after the meeting, using the database of participants created during General Body meetings. The Community Dialogue Platform pilot also tested the response time of the meeting-portal interface. Demonstration of the portal required accurate information – a long code number – be given to villag- ers; a quick synthesis of poll responses received by back-end Platform operators; and real-time blasting of the summary text message to villagers. Further, depending on the quality of the mobile phone sig- nals, there was sometimes a time lag in the blasting of the summary text and its reception by meeting participants. Fillers were needed to sustain the interest of participants until they received a message back from the portal. Overall, the results of the pilot phases were highly positive. Villagers were enthusiastic about using the portal, and field facilitators and back-end operators were able to build sufficient confidence around its use. Once the team had made key changes to the portal in response to the feedback received, all systems were cleared for rollout, field mobilization tools were developed and tested, field facilitators trained, and field activities rolled out. Inclusion through Innovation Unlike other technology-based platforms that tie together multiple forms of social media with text messaging, the Community Dialogue Platform focused on the exclusive use of text messaging for devel- oping momentum in grassroots, education-related governance. For example, in the Check My School program in the Philippines, text messages were one of many forms of data received from schools. Other data were delivered via Facebook, Twitter and webpages. While this allows for a larger range of elec- tronic options to be utilized, it also runs the risk of diluting the intensity in traffic on any one particular platform. Given low awareness of other platforms in rural Sindh, their use would have resulted in lower overall activity in this project. The Community Dialogue Platform sent a clear message to participants that text messaging technology is a powerful tool that they could use to interact with each other. This minimalistic approach to commu- nication simplified engagement and enabled participants to take part in the project more conveniently. An important challenge that the project team had to overcome was the issue of accommodating multi- ple languages on the portal. Given the high levels of illiteracy in rural Sindh, the use of English as the portal’s sole language would have excluded a sizeable portion of the population. On the other hand, the diversity of languages spoken in Sindh made it too difficult to cater to all languages. M3 Technologies, 21 Airtime minutes or phone balance is commonly referred to as “credit” in Pakistan. MEETING MATERIALS, TESTING AND FIELD PILOTS 27 in consultation with the World Bank team, struck a balance by increasing the languages used to include Urdu, Roman Urdu and Sindhi. This approach accommodated a larger group of participants and made it easier for community members to utilize the Platform. The portal was innovative in its ability to send out bulk messages to users. The project team could reach a large number of participants in a limited period of time. Because information went to users in different households of the same village simultaneously, all the users in a particular village received the same informa- tion, allowing them to participate in formal and informal discussions on education in an equitable manner. The Community Dialogue Platform is designed to be user-friendly for participants, back-end oper- ators, and data managers. The final product that was used during the project rollout could be easily adapted to evolving ground realities. Annex 2 describes various technical design features of the Platform that gave it this level of flexibility. Tool Development and Testing for Elections and Capacity-Building An elections script was developed for Assistant District officers to ensure that they had a precise un- derstanding of their role during the General Body meeting when elections were conducted. This script provided specific prompts to the officials and, critically, provided contingency planning on how to re- spond if there were insufficient interest in the community for elections. The project team held a special workshop in Hyderbad, Sindh, to train the district education officials. This not only helped prepare them for the first meeting, but also got them onboard for supervising elections and record-keeping during the crossover intervention. District officers received an honorarium to cover the travel cost of participating in election meetings. As part of the capacity-building measures, the team prepared scripts for the three Executive Body meet- ings for sample villages in the interventions. For the first Executive Body meeting, a handout was also developed to complement the script. This was essential for reinforcing the messages given during this meeting, which would provide a foundation for the future functioning of the group. Scripts for the second and third Executive Body meetings were translated into Sindhi so that field facilitators could make the meetings more inclusive and ensure that all members completely understood what was being discussed. Besides the broader Field Mobilization Plan which was developed for the entire project, a specialized plan was created for School Management Committee Executive Body elections. The Plan underlined the re- sponsibilities of Weitek Group, the implementing partner that facilitated the Executive Body meetings, as well as the role of district officials in conducting the elections during the first, village-level General Body meeting. The Field Mobilization Plan also provided detailed information on how to introduce the project to village participants, as well as information about the Executive Body and election procedures. Training of Field Facilitators Field facilitators played a critical part of the project. They mobilized the community to attend the vil- lage meeting, facilitated General Body meetings, facilitated elections and moderated Executive Body meetings. Given that they were the face of the project for local communities, it was imperative that they had a strong command over the project’s modalities, language and objectives, as well as experience in dealing with shifting field scenarios. Further, it was essential for facilitators moderating Executive Body 28 MEETING MATERIALS, TESTING AND FIELD PILOTS meetings in the crossover intervention to possess the skills and the right attitude to moderate discus- sions for participatory development of School Improvement Plans. The project team tried to ensure that community members understood that the project was not just about redressing grievances. Rather, it was about sparking dialogue among community members for the better- ment of local education. Thus, villagers were requested to focus on issues that could be addressed by the community itself, rather than on problems that couldn’t be fixed solely through community action. For example, low-quality teachers is a broader issue related to formal governance; villagers can lodge a com- plaint against such teachers, but they can’t easily fix this situation on their own. On the other hand, if a school requires repairs, villagers could initiate a consultative process, include the larger community in the discussion, come up with suggestions on what should be repaired and which vendors to use, and use the School Improvement Plan to tap School Management Committee funds and pay for the work. Since field facilitators were in direct contact with the local communities in the target villages, it was critical that this point was first impressed upon these facilitators. Thus, every aspect of field facilitator training included discussions on how villagers could work together to achieve local solutions to ed- ucation-related problems. This was essential to ensure appropriate use of the portal, which also sent periodic nudges in the form of text messages aimed at preventing any drift in the conversation from what was achievable at the local level. The World Bank team made intensive efforts to train field facilitators so that the project could achieve what it had set out to achieve: greater intra-community dialogue about education, increased account- ability of the School Management Committee, and enhanced awareness and concern about the state of education at the village level. The team of facilitators was initially trained on the materials that were prepared for General Body meetings, including flip charts and audiotapes. The training included field and classroom components. In the field, the facilitators received instruc- tions on how to implement the Field Mobilization Plan and the script devised for General Body meet- ings. The script included details on where participants should sit, where flip charts should be placed, who should be turning them, and how best to maintain cultural sensitivity and propriety during the meeting. Field facilitators needed to understand exactly what their role was during the meeting and what the expectations for them were. Classroom trainings had two objectives: 1) to train field facilitators on the specific use and pur- pose of meeting tools; and 2) to test the ability of field facilitators to implement the project. The project-tool training involved a mix of lecture and video. Implementing partner Weitek Group prepared the video, which showed how field facilitators should conduct a General Body meeting. Master trainers were cognizant of the fact that the group of field facilitators included individuals from diverse backgrounds. Thus, trainings had to be adjusted to ensure that all field facilitators were on the same page. Once the lecture and the video had been completed, field facilitators, one at a time, conducted a mock village meeting for the World Bank supervisors, using the different instruments provided to them. The Bank team played the role of community participants, raising challenging questions for the facilitators to answer. Any facilitators who lacked a clear understand- ing of the process were retrained. While General Body meetings can be difficult to run because of the large number of participants, Ex- ecutive Body meetings may prove challenging for other reasons. They are more intensive, with a select group of community members undertaking the development of the School Improvement Plan and therefore, facilitators should have a greater command over the subject matter. Facilitators had to ensure that power was evenly balanced among all five members of the body and to prevent any one member MEETING MATERIALS, TESTING AND FIELD PILOTS 29 – such as the head teacher or the chairperson – from exerting too much influence over the agenda or decisions. Facilitators did this by encouraging silent members to join in the conversation. At the beginning of the project’s rollout, more project staff were deployed to target villages. This helped ensure that field facilitators got the support they needed, while also providing a way to supervise them during General Body meetings. Observers accompanied the team to assess the performance of field fa- cilitators. A key benefit of this strategy was that the staff could quickly detect any initial problems in the project design as well as any facilitator mistakes. The team held a debriefing with the field facilitators after every meeting, providing feedback on their performance and encouraging them to seek guidance on challenges they faced in the field. The World Bank project team invited field facilitators to give suggestions and opinions on the project. If the feedback applied to the entire suite of interventions, then relevant adjustments were made in project design. The establishment of such a feedback loop allowed the team to address any residual design weaknesses at the start of the project rollout. The presence of additional staff helped field facili- tators learn from their mistakes and improve their performance early on. Design Costs for each of the Interventions Project costs associated with the design of instruments for each of the three interventions are shown as “Total Fixed Development Costs” in Annex 8. The three primary cost items within the design phase were development of audiotapes, printing of pamphlets and posters, and costs associated with staff time allocated to each of the three interventions. Partner organization JWT was responsible for the produc- 30 MEETING MATERIALS, TESTING AND FIELD PILOTS tion of audiotapes; implementing partner Weitek Group was responsible for the printing of pamphlets and posters; and the World Bank managed and paid all project staff and consultants. Consequent staff costs included fees for consultants and staff, both in the World Bank headquarters in Washington, D.C., and in the Pakistan country office. Since development of project tools for the three interventions happened simultaneously, overall staff time costs were divided across the three interventions equally. This came out to USD 2,803 per inter- vention for the Pakistan country office, and USD 22,088 per intervention for headquarters. Development of audiotapes cost USD $1,037, with two-thirds of the total going towards development of audiotapes for Intervention 1 and the rest for Intervention 2. The printing of pamphlets and posters cost USD $1,500 each for the School Management Committee and the text messaging interventions. Overall, design activities cost USD $27,083 for Intervention 1, USD $26,737 for Intervention 2, and USD $24,891 for the crossover intervention. Chapter 3 moves beyond designing and pilot testing and explores project rollout in the context of the first set of activities: community mobilization and General Body meetings. Section 2: Implementation Phase Were the treatments implemented as intended? 32 COMMUNITY MOBILIZATION AND VILLAGE MEETINGS Chapter 3: Community Mobilization and Village Meetings The project employed a three-pronged approach, implementing three interventions that built off each other to try to improve local educational outcomes in rural Sindh. The first intervention brought the community together in an open dialogue, and the second created a virtual network to sustain this dialogue. The third intervention sought to increase the capacity of local communities to successfully manage School Management Committees and to channel this dialogue to achieve positive results. The initial rollout activity across the three interventions were village-level General Body meetings. Meticulous project design and piloting ensured that few unexpected roadblocks appeared during the rollout. This assured the evaluation team that implementation pains – charac- teristic of numerous pilot projects – did not drive results for the impact evaluation tied to these interventions. This section provides an overview of the implementation of village meetings across the three interventions, while also highlighting key differences among them. Description of Interventions The core intervention being evaluated is community engagement to revitalize School Management Committees under two distinct mechanisms: 1) a community-level meeting to engage the community in a dialogue for school improvement via the Committees; and 2) a virtual network of community members to engage in a similar dialogue supported through text messages on mobile phones. The first intervention arm used an existing social platform, enabling community members to participate in traditional meetings to acquire information and engage in a public dialogue on school-related issues. The second arm created an innovative virtual platform through which registered community members received school-related information, anonymously sent text messages about these issues, and received a summary of key observations and issues twice every month. By crosscutting these two treatment arms with elections and capacity-building support (training on how to prepare a School Improvement Plan) for the School Management Committee’s Executive Body, we can uncover complementarities that might exist between community-level dialogue and the capacity of the elected Committee members to respond to community-identified needs. These interventions were implemented within rural communities in treatment villages, following a design that facilitated rigorous evaluation of interventions of different kinds. Community Mobilization Parents of children enrolled in school, teachers and representatives of school committees were the pri- mary stakeholders targeted in the two interventions. To mobilize a maximum number of community members, mobilization teams visited target schools and villages two days before the meeting. They used census data to acquire contact information of head teachers in target schools. The field teams contacted these head teachers and told them about the objective of the visit in advance. They also sought the opinion of influential community-level stakeholders on the best day, time and venue for the meeting. COMMUNITY MOBILIZATION AND VILLAGE MEETINGS 33 Field teams were instructed to mobilize at least one member from each household in the community, and to try to ensure maxi- mum participation by households with chil- dren enrolled in government-run primary schools. Social mobilization efforts aimed at getting the community to participate in these meetings were the same across the two interventions. Two days before the meet- ing, village mobilizers put up posters in key spots in the village to announce the meet- ing time and venue. They also went around the village playing a catchy jingle and an- nouncing the time and venue of the meet- Figure 1: Community Mobilization for Village-Level Meeting ing using a megaphone. The mobilization teams made announcements at the mosque using a script from the intervention design team. The head teacher was informed five days in advance and asked to tell children to bring their parents to the meeting. A group of students in each school was tasked with preparing a short performance for the village meeting. Village-Level Meetings (INFO-MEET) Participation rates in village-level meetings Figure 2: Performance by student group were largely satisfactory, with attendance rates upwards of 60% for most of these meetings. In 17 villages, where attendance rates were very low or mobilization efforts were not successful in the first attempt, a second meeting was scheduled. A team of two individuals facilitated each meeting. Participants first filled out an at- tendance sheet, including household details and their mobile phone number, if they had one. Once this process had been completed, a group of students from the school deliv- ered a short performance that encouraged parents to attend these meetings. Figure 3: Participants Listening to an Audio Clip Field facilitators then gave a scripted introduction to explain the purpose of the meeting. The script aimed for a friendly, crisp tone to keep the atmosphere amicable while being sensitive to the time con- straints faced by participants. The two field facilitators introduced themselves and spoke briefly about 34 COMMUNITY MOBILIZATION AND VILLAGE MEETINGS the purpose of the meeting. They then played a 10-minute pre-recorded audio clip featuring a story that highlighted the importance of education. The clip also introduced meeting participants to School Management Committees, their structure, purpose, and membership, as well as the amount of funds available to the Committees. A flip chart presentation reinforced and recapped these themes. This in- troduction was the same across both treatment arms. The second part of the meeting focused on how the community could engage with and leverage School Management Committees to improve the local school. For the first intervention, a second audio clip highlighted in detail specific actions that participants could take to improve educational outcomes. These included regular participation in the Committee’s General Body meeting to vote on the School Improvement Plan; involvement in reviewing the performance of teachers and school committee mem- bers; and participation in community efforts to enroll out-of-school children in school. The audio clip emphasized the need for parents and the broader community to take an active interest in the schooling of their children. The clip encouraged them to make regular visits to the school to see whether teach- ers were present and engaged in teaching. They were also asked to take stock of any facilities needing repairs and to ensure that their findings would be reflected in the School Improvement Plan. The audio clip encour- aged parents and the wider community to actively track the progress of activities listed in the school plan. Further, the clip suggested ways in which community members could contact representatives of the School Management Committee. Meeting facilitators introduced Committee members who were present, and a take-home brochure provided members’ contact numbers to participants. A flip chart presentation re- inforced the messages in the audio clip. A facilitated discussion among community members followed. At the end of the General Body meeting, the facilitators encouraged community members to organize a second General Body meeting to take the discussion forward. The staff emphasized that a second meeting, which was to be conducted by villagers independently, would be crucial for sustaining the conversation about education at the local level, and for teaching villagers how to conduct broad-based, transparent and inclusive meetings on their own. Village-level Meetings (INFO-CDP) For the second treatment arm, once the first audio clip ended and the flip charts had recapped its con- tent, the field facilitators introduced the participants to the text messaging-based Community Dialogue Platform. A second, short audio clip introduced the concept and purpose of the Platform, and a field facilitator then illustrated the concept using a flip chart. Participants also received a hands-on demonstration of the virtual platform. For demonstration purposes, five to 10 participants at the meeting who had a credit balance in their mobile phone were selected to re- ceive a multiple-choice format question from the portal: “What is the main issue facing your school?” This question was translated into the local language. The participants were asked to respond in real time from a menu of options including: lack of facilities at the school, teacher attendance, and quality of learning. The platform soon after sent a summary message back to the participants, noting the option chosen by the majority of participants. This message noted the option chosen by the majority of the participants. (See Annex 2 for details on the technical design of the Platform.) One of the participants read aloud the message received from the portal, providing attendees with a concrete example of how the Platform worked and its key features: 1) customized summary messages captured key discussions; 2) summary messages preserved the anonymity of senders; and 3) participant feedback was summarized and quickly turned around. Reg- COMMUNITY MOBILIZATION AND VILLAGE MEETINGS 35 istration of participants on the platform fol- lowed an opt-out design. Meeting facilitators requested participants to provide phone num- bers when recording attendance. After the demonstration, they had the option to have their numbers removed, in case they did not want to register on the portal. Hardly any of the participants chose to unsubscribe. The project design included a number of mea- sures aimed at making sure all villagers had the ability to use the Platform, even if they did not have mobile phone airtime credit or if they had a low-level of mobile phone literacy. A Figure 4: Audio clip summarized with the help of flip charts nominal credit balance was transferred to par- ticipants at the start of the project and again, midway through the campaign. Participants selected two literate village-level representa- tives, called “community volunteers,” during the meeting to help less mobile phone-literate participants. These volunteers received hands- on training from the field team so they could help community members type and send mes- sages. The field team also encouraged these volunteers to actively contribute to the discus- sion on the platform. Community volunteers were expected to help community members send messages Figure 5: Demonstration of CDP from the members’ own phones. Each vol- unteer received a unique code that he should include at the beginning of a facilitated message, to signal his involvement. This al- lowed the project team to track the number of messages expedited by volunteers, and the number of active and passive volunteers in each village. Chapter 7 provides an analysis of these numbers. To incentivize communi- ty volunteers to perform their duties, they received mobile phone airtime credit as a reward in proportion to the number of text messages they helped generate. Figure 6: Selection of Community Volunteers After the selection of two volunteers, the two project field facilitators conducted an open discussion with the participants. The aim was to elicit questions and garner feedback about the meeting. The discussion allowed villagers to hold a facilitated, open-ended conversation about the state of schools in the area, and to discuss how community mem- bers could contribute to their betterment. 36 COMMUNITY MOBILIZATION AND VILLAGE MEETINGS The entire workflow of Intervention 2 (INFO-CDP) is illustrated in “Annex 4b: Workflow Diagram – Intervention 2 (Community Development Platform). Capacity-Building: Elections and Support Exploiting large sample sizes, which were a strength of the study, both treatments were crossed over with capacity support for School Management Committees. A subset of villages received field-level facilitation to conduct democratic elections of Committee members, following the protocols stated in official guidelines. The respective subdistrict Sindh Government official received an honorarium for at- tending the meeting and for submitting a bank statement indicating the transfer of authority over funds to the newly elected chairperson – a parent member of the School Management Committee. Most of the villages in the treatment sample were able to conduct elections. However, in two villages where subdistrict officials did not show up or where field teams were met with serious political resistance, the Committee elections were cancelled. The newly elected members, along with the head teacher who co-chairs the Committee, received hands-on training on the core principles of the School Management Committee through three struc- tured meetings conducted over three weeks. The first meeting essentially recapped all the information that was provided in the village-level Committee meeting, along with familiarizing the elected members with the functions of the Executive Body. The second meeting provided training on how to develop a School Improvement Plan, with partici- pants filling out a school improvement chart. The facilitators guided the discussion by illustrating how to fill in different fields with the help of a model Plan chart, featuring a representative primary school in Sindh that was shared with us by the Sindh Government’s School Management Committee team at the Reform Support Unit. In addition, the elected School Management Committee members received an overview of their roles and responsibilities. The third and last meeting focused on finalizing the School Improvement Plan drafted in the second meeting. This meeting was also used to explain the process of ratifying the Plan in a village-level Gen- eral Body meeting. Finally, the elected members received training on withdrawing funds, bookkeep- ing and ways to ensure that implementation of activities outlined in the School Improvement Plan remained on track. Chapter 5 takes a deeper look at the implementation of the crossover intervention and its costs. Annex 6 provides snapshots of materials used in the Executive Body meetings. Project Costs Associated with Community Mobilization and Village Meetings General Body meetings for the two interventions (INFO-MEET & INFO-CDP) cost a total of USD 40,000. Basic per-unit meeting costs were approximately equal across the two interventions. This was the actual implementation cost of the General Body meetings, excluding design, monitoring and World Bank staff costs. These costs included expenses incurred for arranging the logistics of these meetings, including travel and housing, as well as costs associated with testing and piloting material, and actual implementation in the form of General Body meetings. Implementing partner Weitek Group was responsible for the General Body meeting rollout, after the World Bank trained and prepared its staff. Chapter 2 reviews the training of staff for these meetings. CHAPTER 4: POST-MEETING ENGAGEMENT 37 POST-MEETING Chapter 4: Post-Meeting Engagement ENGAGEMENT Post-meeting engagement refers to project design features that kicked in after the initial general body meeting. These design features ensured that a) the community-level dialogue that was Post-meeting engagement refers to project design features that kicked in after the initial General Body meet- initiated at the general body meeting was sustained, and b) intra-community conversation was These design features ing.continuously ensured that: reinvigorated 1) the community-level through external nudges, dialogue that also which was initiated prevented at the General drift within the dialogue. Body meeting was sustained; and 2) intra-community conversation was Each of these external catalysts were part of a larger feedback mechanism, which continuously reinvigorated throughincorporated external nudges, citizens’ which also feedback and prevented induced in the dialogue. driftfurther community Each input. external catalystsengagement of thesePost-meeting was part of a created a larger mechanism, feedback channel two-way for which incorporated villagers, whereby,citizens’ they couldand feedback induced provide further their inputcommunity input. to the community-level Post-meeting engagement created a two-way channel for villagers to provide input dialogue, receive information on what other community members were saying and be reminded of as part of a communi- ty-level dialogue; to receive fundamental processinformation knowledge on what other was which community members necessary were for saying; andthe navigating to get reminders institutional structures on how to tap institutional structures such as School Management Committees to address (such as SMCs) that villagers could utilize for solving education-related problems locally. This education-related problems chapterlocally. This chapter describes the how describes rstly, these catalysts process through were applied, which as well these as the content instruments of each were of these used and secondly, the catalytic nudges. content of It each with ends of a brief these discussionnudges. catalytic on the costsIt associated ends with with these methods. a brief discussion on costs associated with these external nudges. Activating the Feedback Mechanism deployed further for four key nudges in the Activating the Feedback Mechanism Follow-up design features which ensured post-meeting engagement were speci c to the post-village meeting duration of the project: 1) credit transfer; 2) informational SMS messages; Follow-up CDP and design that sought features only therefore, applied ensure to to post-meeting engagement the second 3) summary were SMS specific to the Com- messages; and 4) follow-up munity Platform and therefore, Dialogue (INFO-SMS) intervention only applied and the crossover to the second of intervention Interactive Voice (INFO-SMS) Response and (IVR) calls. crossover of the elections withelections the CDPwith the Platform For intervention. intervention. the For the School Management Committee SMC intervention, intervention, follow-up wasfollow up was ensured ensured through A timeline the second, non-facilitated General of Body these external nudges is provided meeting. through the second, non-facilitated general below. Since general body meetings were Once village meeting concluded in INFO-SMS and Community the meeting. body phased acrossPlatform Dialogue treatment crossover villages, while some of (INFO-SMS-SUPPORT) villages, organizers used the contact information the nudges participants of were who had employed at speci c dates registered Once the on the Platform village to create meeting a virtualin concluded database of community acrossmembers, parents all villages andas (such School for credit transfer), Management INFO-SMS and CDPmembers Committee crossover for each village. The Platformdi was deployed erent for four clusters key nudges of villages followed slightly after facilitators conducted the village (INFO-SMS-SUPPORT) villages, contact meetings: 1) credit di erent timelines. The table 3) transfer; 2) informational text messages; below highlights summary text messages; and 4) follow-up information of participants who registered on Interactive Voice Response calls. average points in time when these external the CDP was used to create a virtual database A timeline of these external nudges is provided below. Since Generalof catalysts were activated Body meetings in treatment villages were not conduct- ed in all villages at the same time, the table below highlights the average points in time when theseduration. community members, parents and SMC over the course of the project members for each village. The CDP was external catalysts were activated in treatment villages over the course of the project. Table 1: Timeline for External Nudges Table 1: Timeline for External Nudges Weeks Weeks Weeks Weeks Weeks 1-2 3-8 9-12 13-16 17-20 Credit Transfer Informational SMS Summary SMS IVR Calls Implementation Phase 34 38 POST-MEETING ENGAGEMENT Each of the nudges was tied to the others, collectively contributing towards the final goal of ensuring sustained community engagement. The initial credit transfer incentivized participation and removed the hurdle of low mobile credit, allowing villagers to participate in the conversation about education at the local level. This was followed by a string of informational messages, which provided a context for the conversation in the aftermath of the General Body meeting and encouraged the initiation of a dialogue. Once the Community Dialogue Platform began to receive messages from community mem- bers, the project team started blasting a sustained chain of summary text messages, synthesizing the conversation at the village level while protecting the anonymity of participants. This created a two-way channel whereby participants were not only providing feedback and comments, but also conversing with each other. Further, Interactive Voice Response calls were also made at the start of the project to provide participants with another mode of providing feedback on the project. This was followed by a second round of credit transfer, informational messages and Interactive Voice Response calls to reinvigorate the dialogue and to serve as a refresher on process knowledge. This feedback mechanism is illustrated in Figure 7. Figure 7: Feedback Mechanism for Sustaining Community Dialogue on Education ROUND 1: INITIAL PUSH INFORMATIONAL SUMMARY SMS SYNTHESIZING KEY ELEMENTS ROUND 2: REFRESHER SMS MESSAGES MESSAGES OF WEEKLY COMMUNITY INPUT FOR OTHER VILLAGERS SUSTAINED COMMUNITY DIALOGUE ROUND 1: GATHERING FEEDBACK ON MULTIPLE ROUNDS: NEGATING LOW CREDIT DIALOGUE PARTICIPATION IVR CALLS CREDIT AS REASON FOR LOW TRANSFER PARTICIPATION ROUND 2: UNDERSTANDING CONSTRAINTS TO PARTICIPATING IN DIALOGUE Combining all of these nudges, an iterative feedback mechanism was created and maintained to sustain the community-level dialogue initiated through the project. The following subsections provide details on the implementation of each of these external catalysts. 1) Credit Transfer The project attempted to use innovative ways to increase inclusiveness and participation. The project team felt that a lack of credit might adversely affect the ability of users to participate in the Community Dialogue Platform. To address this issue, it transferred free credit to registered participants, with the aim of encouraging more active, individual participation on the portal. It also issued free credit to two community volunteers per village who were elected by villagers to assist in usage of the portal, as a way to incentivize the volunteers. Village participants received a flat PKR 30 each, while the volunteers were awarded credit based on performance. Volunteers got PKR 50 in credit for helping generate 10 to 20 text messages, PKR 100 for 21 to 50 text messages, and PKR 200 for 51 or more texts. POST-MEETING ENGAGEMENT 39 2) Informational Text Messages The information contained in the second audio clip regarding the Community Dialogue Platform was broken down into a set of short, comprehensible text messages. Each message was carefully crafted to contain one complete piece of information. The informational text messages were sent in two rounds through the Community Dialogue Platform to all registered participants. The first round took place over two weeks following the village meeting, with one to two messages being sent every day. Registered participants received a total of 20 short messages. These addressed different aspects of the School Management Committee, including funding and members’ authority over financial resources; and inputs, such as books provided to schools by the Sindh Government. Other topics included the structure of the Committee, including details about the election process; action planning; rules and purpose of the General Body meetings; and ways in which the Committee’s Executive Body could be contacted and held accountable. The second round of texts built on the first. Sixteen messages went to community members in total, building on the information provided in the first round. The second round of text messages focused on three central themes: (1) school improvements through the use of the portal; (2) information regarding the School Management Committee and the School Improvement Plan; and (3) information on the functionality of the text portal. For further details regarding the content of the text messages in Rounds 1 and 2, please see Annexes 3a and 3b. 3) Summary Text Messages The primary purpose of the Community Dialogue Platform was to give community members a “voice” and to facilitate their participation in the decision-making process in the local education sector. Therefore, it was important that text messages sent by individual community members were shared with the community at large. However, pilot testing showed that villagers had a strong de- sire for their postings to remain anonymous. To balance these two factors, the design team blasted weekly messages to registered participants in a particular village, summarizing comments, opinions and complaints received from community members in that village – but without identifying who had made the comments. This weekly blast of summary text messages began in January 2013 and continued through June 2013. Summary messages went to participants who were registered on the portal. The messages included information on how many text messages were received from community members over the last week; what community members thought was the most critical problem at the local school; what worked in the school; and how the community members could approach their School Management Committee to help with school improvements. These summary messages helped to maintain the momentum of the Community Dialogue Platform, and allowed villagers to stay connected. Thus, summary messages formed the backbone of the Platform by keeping the community engaged over the period of the project, strengthening the links between in- dividuals and the school, and advancing the community’s understanding of, and interest in, the School Management Committee. 40 POST-MEETING ENGAGEMENT 4) Interactive Voice Response Calls Midway through the campaign, the design team obtained feedback from the community through In- teractive Voice Response calls. The purpose of the calls was to complete the feedback loop that had been started with the initial, informational text messages. The calls not only complemented these messages, but were also used by the project team to collect data. Similar to the informational messages, the Interactive Voice Response Calls took place in two rounds. In the first round, four calls were made to registered participants, seeking information on participation in the village meeting, preferred language for text messages, village members’ activity on the Commu- nity Dialogue Platform, and potential increase in village members’ participation in education-related issues as a result of engagement through the Platform. These four calls were sent on days 1, 2, 9 and 14 following the village-level meeting in every participating village. Initially, the project team had intended to conduct a solo round of calls. However, analysis of feedback received from the first round indicated that a second round was required to ensure sustainability of post-meeting engagement. Round 2 of the follow-up calls comprised four calls made to community members and two calls made to community volunteers who were elected during the village meeting to assist villagers in their use of the Community Dialogue Platform. The four calls made to commu- nity members sought information on their perception of the effectiveness of community volunteers, their opinion of the most significant change that had been brought about because of the project, any constraints to participation, and their opinion on the usefulness of informational text messages. The two calls made to community volunteers gave them information about their roles and the incentives structure that existed for them. The Interactive Voice Response calls provided key information to the project team in the days follow- ing the initial village-level meetings. The specific content of the calls is covered on the following pages, while Chapter 7 discusses statistics on the calls. Content for Communication Informational messages, summary messages and Interactive Voice Response calls from the Community Dialogue Platform to the villagers promoted interaction among community members, and helped build trust in the platform. This section deals with the content of these three critical nudges. 7.1 Informational Text Messages: Round 1 The first round took place over 14 days following the village meeting. One to two texts went out to par- ticipants daily, with 20 informational messages sent in total. These messages sought to break down the information provided in audio clips played during village meetings. They served both as a refresher, and as a means to keep participants engaged in the project. Unlike with the audio clips, village participants could read these messages whenever they wanted, as long as they did not delete them from their mobile phones. Therefore, the participants could refer to the texts whenever they had any questions about the School Management Committee. POST-MEETING ENGAGEMENT 41 Information on the Project The informational text messages provided key information on the project, including how participants could express their opinions and how these would be blasted back in the form of summary messages that protected their anonymity. School Management Committee Structure and Elections Information was provided on the structure of the School Management Committee, ensuring that partic- ipants were aware of the existence of both the organization’s General Body and its Executive Body. It was important that villagers knew that they were automatically a part of the General Body, so that they did not feel that they had to be part of the process to be a member. By virtue of being parents or teachers, they had a vested interest in the functioning of the school and were thus as much a part of the Committee’s General Body as anyone else in the community. Details were also provided on the constitution of the five-member Executive Body. The message emphasized that since the chairperson was always one of the parents in the community, village members had a strong role to play in the administrative success of the Committee. The messages also provided information on the electoral process of the Executive Body. The communi- ty’s participation in this process is key for making the functioning of the Committee more transparent. The texts gave details on the Committee’s different positions and respective responsibilities, and also on when and how elections should take place. School Management Committee Resources Participants received information about mandated annual funds that are provided to the Committees by the Sindh Government. The texts covered the annual funding amount (PKR 22,000), names of Executive Body members who are authorized to operate the account (teacher or chairperson), and the specific areas in which this funding could be used. These messages were crucial for increasing transpar- ency and accountability of the Committee among community members. Armed with this knowledge, members could keep track of how resources received by the Committee were being utilized, or in cases where the government failed to provide the funds, members could seek recourse. General Body Meeting The General Body meeting is of critical importance to a functioning School Management Committee. This is where community members express their opinions regarding various expenditures and plans for school improvement. Most importantly, once the Executive Body develops a School Improvement Plan based on the suggestions received in the General Body meeting, it must secure ratification for the Plan from the General Body. This is the reason why informational text messages covered in detail topics related to the function, process and benefits of the General Body meeting. Participants received information about how many of these meetings are conducted annually, how they should be publicized and what rate of attendance is necessary to reach binding decisions. Participants were reminded about their role in these meetings and in key areas of discussion. These included issues related to school facilities, teachers’ involvement, and strategies for enrolling more children at the school. Participants were encouraged to actively engage in the decision-making process during General Body meetings, as such meetings are the main venue through which the School Management Commit- tee can take shape as a robust grassroots institution. The Committee then can provide support to the school while making the school administration accountable for service delivery. 42 POST-MEETING ENGAGEMENT Staying Engaged with the School Management Committee The messages advised community members on how they could remain engaged with the Committee year-round. One suggestion was for participants to visit the school regularly, to ascertain its condition, teacher presence, and quality of instruction. Further, they should actively contribute to the Commit- tee’s planning process – the primary strategizing tool for identifying any problems or shortages facing the school. The texts encouraged participants to contact Executive Body members without hesitating, as a way to hold the body accountable for its actions, as well as to provide its members with valuable input. Annex 3a provides a translation of the informational text messages sent in Round 1. Informational Text Messages: Round 2 The second wave of informational messages was sent over the remaining duration of the project. Mul- tiple informational messages went out, falling under three main themes: SMS Portal – School Improve- ment; School Management Committee Information; and SMS Portal – Functioning. SMS Portal – School Improvement A cluster of messages was dedicated to explaining the purpose of the portal, benefits of its use, and ways in which it could be leveraged to connect the community to schools. Once again, an emphasis was placed on the need for participants to engage the Executive Body and teachers at the school level. Further, participants were reminded that the School Improvement Plan was a critical component of the functioning of the School Management Committee. Therefore, community members should monitor its preparation and implementation. It was reiterated that the project team would summarize messages sent by participants to the portal and blast them back to the community. The system could ensure the participants’ anonymity, allowing them to express their opinions freely without fear of any negative repercussions. School Management Committee Information This round summarized the messages sent earlier that dealt with the structure, purpose and function- ing of the School Management Committee. These included messages such as “A primary school’s School Management Committee is provided with PKR 22,000 annually”, or “School Management Committee elections and School Improvement Plan approvals are done through voting among Committee members.” Such messages sought to remind community members about the purpose of the Committee and their role in its functioning. SMS Portal – Functioning A small number of Round 2 informational text messages dealt with the specifics of the text message portal’s functioning. Participants were reminded of the contact number to which they should send any messages. Another text provided the name and contact information of their community volunteer, who could assist them in sending and reading text messages. Annex 3b provides a translation of the informational text messages sent in Round 2. POST-MEETING ENGAGEMENT 43 7.2 Summary Messages Each summary text message sent to a particular village reiterated issues, opinions and suggestions that were received from members of that village. This section provides a brief overview of recurring issues and suggestions made by village-level participants. Infrastructure, Facilities and Amenities Lack of adequate infrastructure was a major problem facing many schools. The most prevalent among the many infrastructure-related issues was a lack of maintenance of buildings, classrooms and toilets, which at times rendered them unusable. A few schools needed major repairs to resume school activities. Also, a majority of schools suffered from a shortage of basic facilities, such as clean drinking water, fans, electricity and furniture. Some schools that had access to clean water had no water fountains to drink from. At times, schools did not have the hand pumps needed for drawing underground water. Thus, a recurring sugges- tion was for more funds to go to schools for installing hand pumps. Some villages complained about a shortage of support staff for schools, such as cleaning staff and guards (patewalas in Sindhi). Many of the villagers expressed dissatisfaction with the number of rooms in their school and the furniture provided. Villages at times also lacked proper roads or other access to schools, causing a major inconvenience to students. There were multiple reports of schools not having boundary walls or of the walls being in need of urgent repair. Respondents frequently cited a lack of main gates as a significant problem. Some felt that students needed playgrounds to feel motivated to go to school. Schools often did not have such recreational facilities, and villagers requested funds for their construction. Finally, certain villages complained about a lack of hygiene and cleanliness in their schools. Teaching Staff The primary concern with regards to teachers and headmasters was absenteeism, and many villagers said they believed that a decrease in absenteeism could significantly improve their village’s educational outcomes. Another major concern was the low quality of instruction, with villagers specifically de- manding better English, Sindhi and religious (“deeni”) teachers. Multiple villagers complained that the number of teachers at their schools was not sufficient with respect to the number of children enrolled. For example, villagers who were sending their children to the government-run middle school Mah- mood Thaheem reported that there was one teacher for 200 students, making it difficult for students to receive an appropriate level of attention and support. Given the relatively conservative culture of rural Sindh, a few villagers sought female teachers for their schools so that female students could attend. Villagers cited punctuality of teachers as a major concern, as teaching staff frequently arrived at school late and left early. Some villages said teachers had an unfriendly and unenthusiastic attitude, and that even following complaints, schools showed little improvement. However, certain villages were happy with the way their schools were functioning and satisfied with the quality of education. Some villagers applauded the Community Dialogue Platform, saying it had brought positive change to their school by exerting community pressure on teaching staff to show up for work regularly and improve performance. 44 POST-MEETING ENGAGEMENT Government Engagement Many villagers viewed the government either as a hindrance to better schooling or as being indifferent to the needs of the schools. Many villages said free textbooks had not been provided to their schools, as promised. At a few schools, teachers had even charged students for the textbooks provided free by the Sindh Government. Quite a few villages felt that government officials did not make a sufficient number of visits to schools to monitor their performance. Lastly, a small number of villagers in the treatment villages raised allegations of corruption by government officials, with the result that School Manage- ment Committee funds failed to reach schools or were embezzled along the way. Finances Scarcity of funds seemed to be a major issue, with participants saying grants provided by the Sindh Government were insufficient for undertaking required maintenance at schools. Some villages reported that individuals in the Executive Body had misused the funds, while others said the funds were not reaching the schools at all. School Management Committees According to feedback received from participants, the functioning of the School Management Com- mittees was substantially affected by the Community Dialogue Program in the treatment villages. However, a certain number of villages requested an increase in the number of Committee meetings to discuss issues such as finances, enrollment and quality of education. It was reported that after the be- ginning of the project, School Management Committee meetings were being held, with some villagers crediting the portal for this change. Feedback also suggested that community members’ perception of the rate of teacher absenteeism had changed since the inception of the project, with villagers now more likely to take an active interest in whether a teacher was present at the local school. Certain villages failed to conduct Committee elections for the Executive Body, while in others, the schools were reported as failing to hold any Committee meetings. These reports were usually accompa- nied by requests from villagers for these meetings to be held for their schools. Miscellaneous Some community members reported that schools were being regularly used for community meetings (autaqs) and for hosting weddings, rather than for educational purposes. Also certain villages reported that school build- ings had not been used at all for a protracted period. In both cases, villagers wanted the schools to become operational for educational purposes again. A small fraction of villagers said community volunteers were not fulfilling their duties of informing people about the project. This issue is tackled further in the next section. 7.3 Interactive Voice Response Calls Informational text messages and summary messages sent through the portal created a strong feedback channel among villagers, and between the villagers and the intervention team. The team used another tool – Interactive Voice Response calls – to get direct feedback from the community about their expe- rience with the portal. The benefit of these calls was that they also reached individuals who could not read or write, and whom community volunteers might otherwise not have reached. POST-MEETING ENGAGEMENT 45 There were two key constraints that emerged in the use of the interactive call platform: First, the time duration of the pre-recorded voice message was short, and villagers sometimes did not understand either the question or the listed response options. Second, many villagers did not have a basic understanding of the calls and hung up after a few seconds, assuming that the calls were not intended for them. However, since the calls were a complementary tool, they were only expected to plug gaps in the feedback chain. There were two rounds of outbound Interactive Voice Response calls. Details are provided below. Follow-up Interactive Voice Response: Round 1 In Round 1, voice calls were scheduled on days 1, 2, 9 and 14 following the initial village meeting. An introductory call went out to participants, to alert them that they would receive such automated calls from the project in the future. The introductory call also collected information on the status of households’ attendance at the village meeting. This allowed for the project team to fill gaps in their understanding of the project in a particular village. The purpose of the second call was to learn about participants’ preferences for the language used in the text messages sent to them. One of the underlying goals of the project was to enhance inclusivity in local governance. This is because catalyzing a conversation regarding schooling at the village level requires the engagement of a wide range of villagers. This call, therefore, targeted those participants who might have been struggling to participate in the project because of a language barrier. Following this call, participants received future text messages in the language that they selected. The third follow-up call sought to gather information about the participation of villagers registered on the Community Dialogue Platform, and to understand why they might be avoiding being actively involved. The last call in the first round sought to gather information on whether the Platform was generating intra-community engagement, and specifically, between which stakeholders. This call asked villagers whether they thought the project was achieving its short-term objective of catalyzing a conversation on education within the community. Annex 3c provides a translation of the Interactive Voice Response calls made in Round 1. Follow-Up Interactive Voice Response: Round 2 A second round of calls took place in the Sindhi language. Once again, the project team made four follow-up calls to participants. However, it also made two calls exclusively to village volunteers. The calls to village participants were brief, primarily repeating the questions that were posed to them during the first round of calls. This was done with the dual aim of maintaining their engagement in the project and collecting data to check if responses to certain questions had changed over time. Par- ticipants were asked whether they thought the project had brought any changes to the village – a key question gauging perceptions of the project’s impact. In cases where participants thought that the Com- munity Dialogue Platform had positively affected the community, the call then asked them to identify specific areas where they had seen changes. These might include school facilities, teacher absenteeism or intra-community interaction. The call also asked participants whether they found useful the summary messages that had been blasted out over the past few months. Their responses provided the project team with an important measure of the effectiveness of these messages. 46 POST-MEETING ENGAGEMENT The project team also wanted to know why some villagers declined to participate in the project, es- pecially at later stages. Participants were again asked about the primary cause of non-participation. Perceived reasons cited included lack of airtime credit, inability to read and write, and inability to understand summary messages. Two brief calls went to community volunteers who had been selected at the village meetings to assist participants in accessing the Community Dialogue Platform. Unlike the calls made to participants, these did not seek to obtain – but rather, to provide – information. The first call to community volun- teers provided procedural information on how to undertake their responsibilities under the portal in- tervention. The second call provided them with information on the incentive structure for volunteers, in terms of free credit balance. Annex 3d provides a translation of calls made in Round 2. The impact of these external nudges on the traffic generated on the Community Dialogue Platform is analyzed in Chapter 7. Post-Meeting Engagement Costs Cumulative costs of post-meeting engagement amounted to USD $22,565. These costs show up in “Annex 8: Project Costs” under the head Operational Costs of Portal. Major costs associated with post-meeting engagement were text messaging charges and consultant fees for portal management. These two categories accounted for approximately 90% of all post-meeting engagement costs related to the portal. The remaining 10% was divided between credit transfer (7%) and Interactive Voice Re- sponse calls (3%). Text messaging costs resulted from the use of long code, details on which are provided in “Annex 2: Technical Design – Community Dialogue Platform.” The main cost of text messaging was for sending weekly summary text messages and periodic informational text messages to all treatment villages which were part of Intervention 2 (INFO-SMS) and its crossover variant (INFO-SMS-SUPPORT). Consul- tant fees include remuneration for back-end portal operators and managers. The next chapter takes a closer look at the crossover intervention, which combined elections with ca- pacity support, with the aim of strengthening the Executive Body. The villages selected for treatment were already part of Interventions 1 and 2. However, not all villages in these two interventions received capacity-building support. ELECTIONS AND CAPACITY-BUILDING SUPPORT 47 Chapter 5: Elections and Capacity-Building Support The elections and capacity-building intervention were rolled out with the aim of ensuring that the School Management Committee was run in a spirit of democracy and inclusiveness. The crossover intervention demonstrated the entire process cycle of democratically electing members of the Executive Body, and then training them so that they had the requisite skills to execute their official duties. Targeting the capabilities of the Executive Body helped to strengthen the School Management Committee institutionally. This chapter provides an overview of the processes undertaken in the rollout of this intervention, which included elections and structured meetings for Executive Body members. The interventions climaxed with the creation and community ratification of a School Improvement Plan. This completed the 360-degree design approach that the project team undertook, combining elements of social mobilization, a generation of intra-community dialogue, and skills transfer for the effective running of School Management Committees. Rationale The first two interventions attempted to initiate a deliberative process that sought to give the local com- munity a “voice”, and community members a platform to engage each other to come up with local solu- tions to education-related problems. Thus, the School Management Committee campaign immersed community members into the Committee mechanism that was available to them, but which they had until then, not utilized to its full potential. By bringing community members on to the Community Dialogue Platform, the text campaign went further than the School Management Committee cam- paign, providing a platform for community members to discuss education issues in a sustained manner. To stop the project activities at this point, the project team would be assuming that the community’s process knowledge, awareness, and monitoring of funds would, over time, lead to the dismissal of weak members and the induction of proactive community leaders. This, in turn, would result in the incremental strengthening of the School Management Committee as an institution. Another possibility was to accelerate the process by conducting fresh Committee elections, inducting new members and building their capacity to effectively implement the demands of the community. The field experiment gave us an opportunity to empirically test the relative effectiveness of these approaches in rural Sindh through the crossover design. The aim of the elections and capacity-building intervention was to strengthen the School Management Committee as an institution through participatory training of elected Committee members. For this, it was necessary that the Executive Body – the core team leading the Committee – be democratically elected, inclusive and capable of carrying out its responsibilities. The School Management Committee and text messaging campaigns attempted to make the Committee inclusive. Using the crossover inter- vention, the project team tried to ensure that the Executive Body was democratically elected and able to meet its responsibilities. 48 ELECTIONS AND CAPACITY-BUILDING SUPPORT Description of Rollout This capacity-building exercise was a crossover intervention in the sense that it piggybacked on the School Management Committee and text messaging campaigns, and was not a standalone intervention. Of the 119 treatment villages in the sample chosen for the Committee campaign, 57 were crossed over with the elections and capacity-building intervention. Of the 111 treatment villages that formed the sample for the text messaging campaign, 56 villages were crossed over with the elections and capacity-building intervention. Elections In the crossover villages, the General Body meeting was followed by elections, which were facilitated by implementing partner Weitek Group and conducted by district officials. The law requires that a district official be present during the election, or the elections could not take place. This occurred in two target villages, and the issue is further discussed in Chapter 8. The School Management Committee’s Executive Body includes the head teacher, two elected par- ents (a Chairperson and a parent representative), and two elected non-parent community mem- bers. While the head teacher is automatically selected to the Executive Body as General Secretary, elections must be held for the other four positions. Under the intervention, elections were con- ducted in two phases. In the first phase, participants who were parents of children enrolled at the local school could nominate themselves to run for the Executive Body positions. After they addressed the other participants, a round of voting followed. The parent with the highest number of votes became chairperson, and that with the second-highest number of votes became the parent representative. Voting was primarily based on a show of hands, with the project team ensuring that participants did not vote more than once. During the second phase, non-parent community members nominated themselves and addressed the other participants in a bid to generate support. The community voted for two non-parent villagers in this phase, with the two who won the high- est number of votes joining the Executive Body. Recurring elections not only can help build confidence in individuals to nominate themselves, but also can help legitimize the general community’s right to exercise its powers and responsibilities. Given that many of these villages had not previously held School Management Committee elections, the project’s field facilitators had to guide the participants through every step of the process. Aspects of the elections that might appear relatively simple, such as nominating a community member for the Executive Body, proved at times difficult to carry out in the field. Community members required constant, personalized encouragement from the field facilitators. More importantly, community members needed to see the benefits of participating in the electoral process. Making them understand that only they could change the state of education locally was crucial to the successful implementation of the intervention. Field facilitators were trained to play the role of guides during this phase of the crossover intervention. The process for transferring School Management Committee responsibilities to the communities includes elections being conducted under the supervision of the Assistant District Officer (sub-dis- trict official). Further, a legal document is produced by the officer verifying that the election has been conducted. This document includes names and contact information for new members of the Executive Body. Since it was imperative that the elections be transparent, all steps in this process were strictly followed. This also helped to deter potential pushback from incumbent Executive Body members who might have resisted elections for fear of being displaced, and to ensure that the newly elected members had the legitimacy to execute their responsibilities. ELECTIONS AND CAPACITY-BUILDING SUPPORT 49 Once elections were successfully held and the new Executive Body had been formed, the Assistant District Officer drafted an official letter signaling that responsibility for the functioning of the School Management Committee was being transferred to the new Executive Body members. This legal doc- ument was primarily used to update the Committee’s bank account details. Three copies of this docu- ment were prepared, with one given to either the head teacher or the chairperson; the second kept by implementing partner Weitek Group; and the third retained by the district officer for governmental record-keeping. Capacity Building While elections of the new Executive Body gave its new members technical control of the School Man- agement Committee, de facto power might still rest with previous members who were not willing to give up their authority. Further, with no formal transition mechanisms and with a weak capacity to im- plement their duties, newly elected members would have struggled to lead the Committee effectively. In the context of these institutional weaknesses, the crossover intervention conducted three intensive Executive Body meetings to provide newly elected members with the capacity, confidence and legit- imacy that they needed to run the Committee. These Executive Body meetings were conducted in a classroom, with one field facilitator moderating. The first Executive Body meeting was used to educate the new members on the Body’s function- ing. Members received a handout and two audio clips that provided information on their roles and responsibilities. Members were also reminded of what was expected of them: engagement with the community; development of the School Improvement Plan; fulfillment of all procedural require- ments of the Executive Body; maintenance of organization documentation; and implementation of the School Improvement Plan. Running the Executive Body is a complex process, so it was import- ant to give newly elected members an overview of its functions before diving into the specifics. The meeting stressed the need for members to monitor multiple aspects of the local school’s operations. These included teacher attendance and quality of instruction, student attendance and learning, and the school’s physical infrastructure. The second Executive Body meeting introduced new members to the School Improvement Plan. The Plan is the most visible aspect of the School Management Committee. This document provides specific information on what needs to be fixed at the school in a particular year, and what resources will be required for the repairs. More importantly, it should incorporate feedback received from the commu- nity, as the product of an inclusive process which seeks to find local solutions to local problems. The Committee is authorized to include three types of cost items in the improvement plan: school repairs, labor wages for school cleaning, and transportation for children to and from school. Executive Body members studied a sample School Improvement Plan, which had been completed cor- rectly. Before they developed their own Plan, members had to find out how much the Committee had in its bank account. The cumulative cost estimates provided in the Plan had to be less than the funds available in the bank account, or if there were none, then less than the annual grant provided by the Sindh Government, totaling PKR 22,000. For this project, members had to ascertain the Committee’s bank balance on their own. In the future, the project design could be improved by having field facilita- tors accompany the members to the bank to get the information. The sample School Improvement Plan served as a model when Executive Body members developed their own Plan. The newly elected Executive Body was expected to emulate the sample plan in terms of format, but fill in requirements specific to the local school. The sample School Improvement Plan appears on the next page. 50 ELECTIONS AND CAPACITY-BUILDING SUPPORT Figure 8: Sample School Improvement Plan Field facilitators discussed each field in the School Improvement Plan with the Executive Body mem- bers, and then opened up the floor to questions. Once every member understood the purpose and format of the Plan, they began the task of developing their own Plan. The field facilitators provided guidance by steering the conversation towards potential requirements of the local school. More impor- tantly, they sought to ensure that the power dynamic within the organization was not skewed, and that no one member had more influence over the proceedings than any other. Head teachers were prone to dominating discussions, given their traditional status as the power center in schools. Facilitators peri- odically steered the conversation away from them and to newly elected members, especially parents of students enrolled at the school. The development of the School Improvement Plan was a highly participatory process, in line with the design and objectives of the project. The field facilitators were present during the development of the Plan, to help guide Executive Body members. They also answered any questions members might have, such as whether their cost estimates were in line with available funds. They assisted the members in fill- ing out the template, including making sure that they filled in the right boxes. It must be emphasized, however, that the facilitators adopted a hands-off approach during this phase and encouraged members to take ownership of all substantive elements of the plan development process. This included debating what the needs of the school were, what the expected cost estimates were, and whether the goals of the School Improvement Plan were realistic. Thus, facilitators ensured that this process provided members with intensive, hands-on training so that the latter could lead the process independently during the next round of plan development. Finalizing the School Improvement Plan and securing approval for it from the community can be a protracted process, with implementation potentially proving even more challenging. The Executive Body is expected to remain as transparent to the community as possible. Documentation of its activities provides a record for community members to refer to if needed. However, the Executive Body is ac- countable not just to the community, but also to the Sindh Government, which provides annual fund- ing. Therefore, all of its activities, especially those involving the use of funds, must be fully documented and publicized. Local community members who serve on the Executive Body tend to have the least ELECTIONS AND CAPACITY-BUILDING SUPPORT 51 experience in this area, so it is critical that they receive adequate training to build up their capability to document the group’s activities. Procedurally, once the body finalizes the School Improvement Plan, it is introduced on the floor during a General Body meeting. At this meeting, parents give their feedback on the Plan and introduce amendments. Once participants have voted on and approved the Plan, it is posted in the school where the community can see it. Specific rules also govern implementation. If a project costs more than PKR 5,000, the Executive Body must display a tender notice on the outside wall of the school for potential vendors to see. Based on vendor interest, the Executive Body collects at least three bids and compares them. It then selects the cheapest vendor providing goods or services of appropriate quality, and draws up a purchase order. The order goes on public display for 15 days and is sent to the office of the Exec- utive District Officer (Education). Acquisition of goods and services can begin during this time. Any payments made to vendors thereafter are established through a payment voucher, with details on each recorded in a cash book. The cash book provides cumulative information on School Management Committee funds that have been utilized and the balance that is available. Any goods acquired under the School Improvement Plan should be recorded in the stock register, a book which contains details of all inventory and which serves as the Executive Body’s primary inventory management tool. Damage or theft of goods is also recorded in the stock register. In this context, the third Executive Body meeting was organized with the aim of: 1) assisting members in finalizing the School Improvement Plan, and 2) providing them with the capability to implement the Plan. Members received a sample tender notice, which highlighted examples of goods that could be purchased. Figure 9: Sample Tender Notice 52 ELECTIONS AND CAPACITY-BUILDING SUPPORT Executive Board members received instructions on how to fill out the tender notices, providing details on the quantity, type and estimated cost of the item that was required (in the case of the sample, the item was chairs for students). The members also studied a sample bid comparison form. The reason for getting quotations from several vendors was to create competition among the suppliers and to ensure a fair selection process that was not based on personal connections, thus theoretically increasing the accountability of Executive Board members. It also established a system for School Management Com- mittees to attract the best bidder, as vendors might try to underprice each other. Practically speaking, however, at the village level, many of the vendors might be villagers who worked as masons, carpenters or the like, with limited understanding of the competitive bidding process. While the Executive Body meetings increased the capacity of members to invite and evaluate competitive bids, the problem re- mained that local vendors might not have the capability to participate appropriately. This is another example of how processes that appear sound in theory can be difficult to implement on the ground. The sample quotation comparison form that was used to guide the Executive Body members is pro- vided below: Figure 10: Sample Quotation Comparison Form A similar exercise sought to familiarize Executive Body members with payment vouchers and purchase orders. Members also got practice filling out the cash book and the stock register. The facilitators em- phasized the need to maintain these documents and they made a strong push to ensure that the mem- bers had the capability to complete them independently. However, certain caveats should be noted. The parents in the Executive Body who wished to improve the poor educational outcomes of the community had graduated from the same, weak educational sys- tem. For many of them, reading, writing and even simple math, such as addition, subtraction and mul- tiplication, could be challenging. Thus, it was difficult to ensure that they would follow all the require- ELECTIONS AND CAPACITY-BUILDING SUPPORT 53 ments for maintaining these documents. To implement the School Improvement Plan, new Executive Body members must not only be able to fulfill their commitment to the community and complete any required documentation, but also be able to balance issues that are endemic to a feudal economy. For example, some members expressed concern that if they started spending Committee resources, they might attract the attention of local feudal lords who might want to acquire control of these resources. Also, spending funds meant more record-keeping. Given the challenging process that implementation of the School Improvement Plan entailed, there was a risk that Executive Board members might spend less so as to minimize the amount of formal documentation they would need to complete. These are some of the issues that local communities must surmount to play an effective role in im- proving grassroots governance in the education sector in rural Sindh. Designers and implementers of projects related to improving local educational governance should also be cognizant of these challenges, so that they can give rural communities the appropriate tools to organize themselves, fulfill official procedures and deflect local pressures. Annex 6 provides snapshots of the different sample instruments used during the three Executive Body meetings. Elections and Capacity Support Costs Overall, elections and capacity-building efforts cost USD $49,185. The costs were more or less evenly divided between total fixed development costs (USD $24,891) and total variable costs (USD $24,294) of the crossover intervention. Annex 8 provides a breakdown of these costs and other details. The largest fraction of the cost was attributable to time charged by staff and consultants at the World Bank headquarters and the Pakistan country office. This amounted to approximately 92% of the cost of the crossover intervention. Designing the crossover intervention was a time-intensive process, and this phase of the project cost the most in terms of staff and consultant fees. Of the remaining USD $3,833, or 8%, USD $513 was spent on organizing a half-day workshop for Assistant District officers. This workshop was arranged to familiarize district officials with the objec- tives of the project, and to emphasize the importance of their participation in the crossover interven- tion. This was especially important given that School Management Committees are legally bound to conduct elections under the supervision of district officials such as Assistant District officers. An addi- tional USD $3,320 went to honorariums for the officers, to compensate them for traveling to villages to supervise and monitor the elections process. Section 3: Analysis A Were the treatments received by the participants and acted upon? 56 TREATMENT RECEIPT ANALYSIS Chapter 6: Treatment Receipt Analysis Community mobilization and General Body meetings served to involve local community members in the project. Once initial participation of villagers in the target districts had been achieved, the field team rolled out progressive pieces of the project with the aim of ensuring that this engagement was sustained. This chapter fleshes out specifics of this rollout, and provides information on the scope of two of the three interventions: School Management Committee mobilization and the deployment of the Community Dialogue Platform. While the first two sections of the chapter answer questions such as “how many?” for both intervention arms by looking at sample size, participation rates and registration rates, the rest of the chapter focuses on the platform and how the project team used it to sustain participation through nudges such as credit transfer, community volunteers and Interactive Voice Response calls. Annex 10 describes the major data sets used for analysis in this section. Villages by Treatment Status: General Body Meetings The final sample of villages for Intervention 1 comprised 119 villages. A total of 15 tehsils (sub-district administrative unit) and 67 union councils22 were included in the treatment group for this intervention. Among the 119 villages, cross-cutting capacity-building support (INFO-MEET-SUPPORT) took place in 57 villages. Intervention 2 included 111 villages in the Community Dialogue Platform intervention. A total of 15 unique tehsils and 66 union councils were part of this treatment. Among these 111 villages, cross-cutting elections and capacity-building activities (INFO-SMS-SUPPORT) were undertaken in 56 villages. In the other 55 villages, only the Community Dialogue Platform was rolled out. For Intervention 1, out of the 119 treatment villages, 47 were in district Mirpurkhas, 17 were in district Matiari and 55 were in district Sanghar. For Intervention 2, out of the 111 treatment villages, 43 were in district Mirpurkhas, 20 were in district Matiari and 48 were in district Sanghar. Table 2: Distribution of Sampled Villages Intervention 1 Intervention 2 Total Villages 119 111 Election 57 56 Non-Election 62 55 Mirpurkhas 47 43 Mitiari 17 20 Sanghar 55 48 22 Tehsils and union councils are administrative divisions. A province is divided into districts, which is further divided into tehsils. Multiple union councils form a tehsil. TREATMENT RECEIPT ANALYSIS 57 For Intervention 1, three villages were added to the sample for non-election villages, against the planned sample of 59 villages, leading to the actual sample comprising 62 villages.23 Conversely, In- tervention 2 saw an attrition of two villages (one election village and one non-election village).24 The following table compares planned and actual sample sizes across the four types of interventions, plus the control group. Table 3: Planned versus Actual Sample Size COMMUNITY ENGAGEMENT T1: INFO-MEET T2: INFO-SMS (59 villages planned sample, 62 villages (57 villages planned sample, 55 villages actual sample) actual sample) CONTROL COMMUNITY ENGAGEMENT PLUS SUPPORT T3: INFO-MEET-SUPPORT T4: INFO-SMS-SUPPORT (57 villages) (57 villages planned sample, 57 villages (57 villages planned sample, 56 villages actual sample) actual sample) Table 4: Participation Rates by Treatment and District 50th 10th 90th   Mean percentile percentile percentile (Median) Overall 63% 21% 63% 100% SMC         Overall 62% 17% 60% 100% Election 58% 17% 57% 100% Non-Election 65% 19% 73% 100% SMS         Overall 63% 27% 63% 100% Election 58% 22% 54% 100% Non-Election 69% 27% 69% 100% Mirpurkhas 75% 41% 83% 100% Mitiar 49% 18% 37% 100% Sanghar 57% 17% 57% 100% While the initial threshold for participation rates was set at 60%, this threshold was relaxed to 20% af- ter all the meetings were held, given the difficulty of matching census data with households in the field and given average participation rates across treatment villages of almost 60%. The project team agreed 23 Oversampling occurred in this group because the implementing partner visited additional villages which were situated close to certain GPS points in order to prevent the relevant ones from being missed. 24 The project team faced stiff political resistance in these two villages. After efforts to overcome this problem failed, the intervention was called off in these villages. 58 TREATMENT RECEIPT ANALYSIS to physically revisit any villages for which the participation rate was less than 20%. It is encouraging to note that participation rate on the 10th percentile was 22%, which was higher than the revised threshold of participation rates. Thus, participation rates were mostly satisfactory across the two treatment arms. Registration Rates for Community Dialogue Platform An estimated 4,981 individuals registered on the Community Dialogue Platform. On average, 45 in- dividuals per treatment village registered on the portal. However, a caveat is in order. Registration on the portal was a multistage, multimodal process. The field team primarily signed up contacts during the initial village meeting. However, in certain villages, the meeting participation rate was below the threshold of 20% of the number of households in the census. In such cases, the General Body meeting had to be rescheduled, and the team had to re-collect contact information from these villages and add them to the portal’s contacts list. Of the total number of participants registered with the portal, 2,405 were from villages receiving the election crossover, while 2,576 came from non-election villages. Adjusting for the total number of villages, averages differed only slightly across treatment types, with an average of 43 contacts per village received from election villages and an average of 47 contacts per village received from non-election villages. The overall average registration rate (fraction of number of unique, registered contacts to the number of unique households represented at village meetings) stood at 62.8%. However, there was significant variation across villages in terms of registration rates. The 10th percentile of registration rates was 33%, compared to 91% for the 90th. Within districts, Sanghar had the highest average registration rate (68.4%), while Mitiari had the low- est (53.2%). Mirpurkhas had an average registration rate of 60.8%. This variance was lower when we consider villages by treatment type. The average registration rate in non-election villages (64.9%) was only slightly higher than that in election villages (60.6%). Table 5: Registration Rates on CDP 10th Median (50th)   Mean 90th Percentile Percentile Percentile Mirpurkhas 61% 32% 63% 88% Mitiari 53% 34% 51% 80% Sanghar 68% 37% 72% 96%           Election 61% 30% 62% 93% Non-Election 65% 35% 69% 90% Overall 63% 33% 65% 91% Given that the project sought to use the Community Dialogue Platform as a way of generating internal conversations around education, these are significant numbers since each participant on the portal was encouraged and prodded to reach out to other community members to discuss local school-related TREATMENT RECEIPT ANALYSIS 59 issues. This multiplier effect – whereby registered users interacted with non-users – was expected to help boost the “voice” of the community. The compounding of the community’s voice should increase upward pressure on the government for greater accountability and transparency in the education sector. Credit Transfer to Participants To account for the staggered initiation of the project in the 111 treatment villages (INFO-SMS & IN- FO-SMS-SUPPORT), the project team transferred airtime credit to participants in different villages at different times. Credit was transferred on the following dates: Table 6: Dates for Credit Transfer 19 25 28 19 18 No. of Villages Credit Transfer Date 01/03/13 1 1/03/13 21/03/13 0 2/04/13 05/04/13 The project team staggered credit transfer based on the gradual, cumulative incorporation of different treatment villages to the portal. However, only individuals who verified that they were registered on the portal at the time of the credit transfer, received credit. Since registration was phased and the credit transfer was staggered, some of the participants who were registered on the portal did not receive credit. Of a total 4,981 unique registered users, 4,788 got free credit and 131 community volunteers received incentives in the form of free credit. Participation Rates of Community Volunteers The role of community volunteers was to expedite local participation in the use of the Community Di- alogue Platform. Volunteers were charged with generating intra-community discussion around educa- tion, and to assist villagers in interfacing with the portal, including assisting them in sending messages to the portal, and sending messages on their behalf. A total of 212 unique community volunteers were part of the intervention. A fraction of these were active volunteers, carrying out their responsibilities appropriately, while the others, known as passive volunteers, did not. Of the total 59 active volunteers, 30 came from election villages, and 29 from non-election villages. Conversely, there were 153 passive volunteers. These individuals were seen as hav- ing participated in the broader discussion on education at the village-level, and in many instances, they sent messages as regular users. But they did not directly assist other participants in sending messages to the portal. This implies that they did not send messages at the behest of others with the correct village and community volunteer codes. It is important to gauge how many villages had a volunteer who was actively engaged in the project. Of the 111 treatment villages, approximately 48 had active volunteers. Of these, 25 were election villages and 23 were non-election villages. Therefore, 43% of the treatment villages had at least one active volunteer. 60 TREATMENT RECEIPT ANALYSIS Table 7: Villages with Active Community Volunteers ELECTION NON ELECTION TOTAL VILLAGES VILLAGES VILLAGES 25 23 48 Activity Generated by Interactive Voice Response Calls Field teams made Interactive Voice Response calls in 96 villages, to 4,427 participants in total. That resulted in 17,708 calls altogether, or an average of 185 calls per village. Of these, an average of 131 calls per village were answered overall. Disaggregation of this information reveals that an average of 60 calls per village were answered during the first round, while an average of 71 calls per village were answered during the second round. Thus, the success rate of the calls in Round 2 was higher than in Round 1. Chapter 7 presents further details and an analysis of responses to the calls. ANALYSIS OF TEXT MESSAGES 61 Chapter 7: Analysis of Text Messages The Community Dialogue Platform created an open dialogue among community members regarding educa- tion at the local level. More importantly, it allowed users to seek local solutions to local problems facing schools at the village level. This chapter delves into activity on the portal, looking at how many individuals actively used the portal, the extent to which they used it across geographic sub-units and time, and how well par- ticipants responded to external catalysts, such as community volunteers and Interactive Voice Response calls. Active and Passive Users The project defined active users as those registered users who sent at least one non-junk text message over the course of the project. Passive users referred to individuals who did not engage directly with the portal, but shared their opinions through active users. In the communal culture of rural Sindh, certain members of the community are at times entrusted with representing the larger community. As a result, discussions among a group of individuals might not lead to all group members sending messages indi- vidually. Instead, a few people in the gathering would be entrusted with synthesizing the information generated and sending it to the portal. Of a total 4,981 unique registered users on the portal, 1,399, or 28%, were active users. Treatment villages had an average of 12 active users per village. While the number of registered users reflected the general interest that was generated by the portal and the success of the project team in getting villagers to register with it, the number of active users is a more accurate reflection of sustained interest in the portal. It is important to understand whether the relatively low ratio of active users to registered users was an indication of a general lack of interest in the portal, or whether, as suggested earlier, the opinions of both active and passive users were being synthesized by active users. We define “active villages” as villages where at least one non-junk text message was sent to the portal. All 56 of the election villages were active, while 54 of the 55 non-election villages were active. This is an encour- aging statistic, and shows that there were no systematic issues in terms of a general lack of interest in the portal across villages. Analysis of SMS Traffic – Over Time and by District This section provides an analysis of activity on the portal over time, aggregated at the treatment, district and tehsil levels. Project rollout was phased in. The first village meeting was not held in all villages on the same day. Instead, it took almost 50 days (February 7, 2013 – March 27, 2013) for all of the treatment villages to hold their first meeting. This part of the analysis puts data in 10-day buckets, with Days 1-10 representing the first 10 days after the initial village meeting in a particular village, and Days 110-120 representing the last 10 days of formal activity on the portal in the same village. After the 10-day buckets were created for each village, these buckets were simply aggregated. 62 ANALYSIS OF TEXT MESSAGES The project cycle saw total traffic of 5,336 messages, with approximately one-third of the messages (1,836) generated in the first 10 days following the village meeting. An average of 16 messages per village were sent during the first 10 days, compared to an average of two messages per village during the last 10 days. The initial surge occurred during a period when communities were being mobilized and informational text messages were being sent to them. This indicates that these efforts had a significant impact on generating initial activity on the portal. However, following the initial round of informational messages, traffic on the portal dropped as expected, stabilizing at an average of two to four messages per village per 10 days in the months following the project initiation. District-wise analysis showed that the overall trend in text messaging activity on the portal was by and large similar to individual trends in activity in the treatment districts. This is illustrated in Figure 11. There were no large differentials across districts in terms of activity on the portal at the launch of the portal. However, towards the end of the project, there were higher levels of activity in district Sanghar and lower levels of activity in district Mirpurkhas, as compared to the overall mean. Activity in district Mitiari matched the overall mean (two messages per village). These trends are illustrated in the graph below: Figure 11: SMS Traffic per Village: District-wise Credit Transfer Analysis The project team transferred airtime credit in two rounds, once initially and then midway through the project cycle. Credit transfer, together with informational and text messages, acted as catalysts for increas- ing activity on the portal. While the initial surge could potentially be partially attributed to credit transfer, there were no systematic jumps in activity in the target districts once the initial surge had eased. Thus, the second round of credit transfers does not appear to have had an impact on portal activity. One reason for this could be that access to mobile credit was not a critical determinant of engagement with the portal. Between February 1, 2013 and June 30, 2013, a total of 74% of the messages sent to ANALYSIS OF TEXT MESSAGES 63 the portal was junk – i.e., messages that were either incomplete or irrelevant. This strengthens the ar- gument that lack of credit was not a major constraint for a majority of the participants. Intuitively, free credit should have positive spillovers by acting as an incentive to participate in the project; however, these effects are not reflected in the existing data. Activity of Community Volunteers Adjusting for outliers and community volunteers who sent zero messages to the Community Dialogue Platform, there were 59 active volunteers out of 212 unique volunteers. In effect, approximately a quarter of the volunteers who had signed up for the project eventually sent messages to the portal on behalf of other community members. Volunteers were periodically encouraged through informational messages to generate intra-community interest in the project. The 59 active volunteers sent a total of 287 messages, or an average of five messages per volunteer. In election villages, 30 active volunteers sent a total of 132 messages to the portal, an average of 4.4 mes- sages per volunteer. In non-election villages, 29 active volunteers sent a total of 155 messages to the portal, averaging 5.3 messages per volunteer. The data shows that volunteers in non-election villages as- sisted other villagers more actively than their counterparts in election villages. However, volunteers were only credited with assisting villagers if the messages received on the portal could be traced back to them. The messages sent from other villagers’ phones had to contain the identifying code of the volunteer. The average number of volunteer messages sent for villages that fell in the 10th percentile in this cat- egory was one, while that for villages in the 90th percentile was 11. Thus, the scope of activity across community volunteers varied significantly. These numbers are tabulated below: Table 8: Community Volunteers Activity   Total # of Avg # 10th 90th Messages Volunteers of Msgs Percentile Percentile Election 132 30 4.4 1 7.5 Non-Election 155 29 5.3 1 11 Overall 287 59 4.9 1 11 Analysis of Responses from Interactive Voice Response Calls Chapter 4 discussed the specific content of the Interactive Voice Response calls, the idea behind each, and the options and questions that were part of its interactive design. Annex 9 reassesses the rationale behind these calls, discusses the process for executing them, and highlights the period over which these calls were made. This subsection deals with the specific number of calls made, the number answered and the number answered that generated valid responses. These statistics assist us in understanding the success rate of Interactive Voice Response calls in the context of the intervention. Specifically, the statistics in this subsection include calls made during the first round (four calls made to village participants) and the second round (four calls made to participants and two made to community volunteers). All components of the analysis in this subsection are accompanied by tree diagrams, which can be found in Annex 9. These illustrate the statistics for a particular call. Table 9 provides detailed descriptive statistics on Interactive Voice Response calls. 64 ANALYSIS OF TEXT MESSAGES Table 9: Descriptive Statistics for IVR Calls It can be difficult to achieve a high response rate for Interactive Voice Response calls, because these calls might not catch respondents at a convenient time; respondents might not be willing to answer calls from unknown numbers; or respondents might not understand the automated instructions in the calls. We can check if the first two factors affected the project by looking at the percentage of calls that were answered. For Round 1, a total 33% of all calls were answered. For Round 2, the percentage rose to 38%. Looking at the fraction of answered calls that generated valid responses provides key insights into the automated message’s ability to elicit a valid response from the user. Of the calls that were answered during Round 1, some 87% elicited valid responses. This percentage dropped in Round 2, with 55% of all calls answered eliciting valid responses. Key Messages Received on Community Dialogue Platform The Community Dialogue Platform tagged incoming text messages according to their content, giving the intervention team an accurate idea of the most significant topics of discussion on the portal. For this purpose, 18 non-junk, high-occurring tags were created. Annex 7 takes a look at these categories in detail. A word cloud helps to visualize the issues that were most frequently discussed by users on the Community Dialogue Platform. ANALYSIS OF TEXT MESSAGES 65 Figure 12: Word Cloud of Key Words The size of each of the words in the word cloud is proportional to the frequency of its appearance in complaints, opinions and suggestions sent by villagers.25 The diagram shows that a large number of participants talked about infrastructure and facilities at schools, including the number of classrooms, boundary walls, gates and roofs. Villagers also frequently discussed issues related to teachers, and to School Management Committee funds, specifically the allocation of finances and the fund’s manage- ment. These findings can provide critical feedback to any efforts to scale up the intervention. 25 The word cloud merges some of the categories originally used for sorting. “SMC” here includes “SMC,” “SMC Fund,” “Executive Committee” and “Budget.” “SMS Campaign” includes “SMS Campaign” and “Thanks.” Finally, the category “Others” pulls together “Parent,” “Flood,” “Stipend” and “S.I.P.” 66 ANALYSIS OF CAPACITY-BUILDING SUPPORT Chapter 8: Analysis of Capacity-Building Support The suite of interventions introduced by the project included village mobilization, the provision of a “voice” to local community members, and institutional strengthening. The analysis up to now has largely focused on the first two interventions. While mobilization and provision of a “voice” are crucial for improving educa- tional accountability at the local level, villagers can affect further change by utilizing existing institutions. For this, community members need concrete process knowledge of these institutions to be able to interface with them, and the capacity to harness their potential. Elections and capacity support (also termed the “crossover intervention”) provided the community with precisely these capabilities. This section builds on the structure of the crossover intervention that was introduced in Chapter 5, and provides information on its key elements: the buy-in of the community and the success of the project team in mobilizing villagers for this intervention. These were gauged through participation rates in Executive Body meetings and the successful development of School Improvement Plans, which are a critical tool used by local participants to spur change through School Management Committees. Elections The crossover intervention was rolled out in 57 Intervention 1 (School Management Committee) vil- lages and 56 Intervention 2 (SMS) villages. Elections took place in at least one school in all crossover villages which were part of Intervention 1.26 For crossover villages within the treatment sample for In- tervention 2, elections were not held in two villages: village Dharoro in district Mirpurkhas and village Khipro in district Sanghar.27 After the Executive Body elections, the project team provided training and capacity-building to the new members. The engine of these capacity-building measures were three Executive Body meetings, during which members received training on drafting School Improvement Plans, record-keeping, engaging vendors and maintaining the School Management Committee bank account. Participation Rates in Executive Body Meetings A high attendance rate for Executive Body meetings was critical for the successful implementation of the project, as well as for the future sustainability of the School Management Committees. While community feedback and oversight is essential for a Committee’s effective functioning, the Executive Body members are the ones who get things done – fulfilling legal requirements, such as maintaining documents related to the bank account, record-keeping for the Committee, drafting and finalizing the School Improvement Plan, and procuring goods and services for the school. 26 Attendance sheets for the three Executive Body meetings were used as an indicator that elections were held in a particular village. 27 It was necessary for elections to be conducted in the presence of governmental district officials. In these two villages however, district education staff did not show up. ANALYSIS OF CAPACITY-BUILDING SUPPORT 67 Analysis of attendance sheets data received from crossover intervention villages shows that attendance for Executive Body meetings was high across districts as well as treatment arms (School Management Com- mittee and SMS). As explained earlier, an Executive Body has five members. Attendance for Executive Body meetings was never less than four in any of the villages across the three meetings. In fact, a large fraction of Executive Body meetings were attended by all five members. The high level of attendance indi- cates that mobilization efforts made by project staff in the crossover intervention villages were successful. The mean attendance level across treatment types (SMS and School Management Committee) was approximately 4.8 out of 5. There was almost no variation in mean attendance across the two types of crossover interventions. These results are reproduced in Table 10: Table 10: Mean Attendance Rates for SMC & SMS Crossover Interventions   1st EB Meeting 2nd EB Meeting 3rd EB Meeting SMC + Elections 4.76 4.80 4.83 SMS + Elections 4.80 4.79 4.80 Overall 4.78 4.80 4.81 Mirpurkhas 4.7 4.9 4.9 Mitiari 4.4 4.3 4.3 Sanghar 5.0 5.0 5.0 We find that for Executive Body meetings in Mirpurkhas and Sanghar, the mean attendance level was approximately 4.9 and 5, respectively. For district Mitiari, the mean attendance level was closer to 4. There was relatively greater potential for a lack of a tiebreaking vote, which could cause problems in this district. However, it must be emphasized that this is only one factor in ascertaining the ability of Executive Bodies to successfully oversee the School Management Committee. Statistics related to Executive Body meetings also have a temporal element: The first meeting was fol- lowed by the second, which was followed by the third and final meeting. Any increase or decrease in mean attendance levels from Executive Body Meeting 1 to Meeting 2 and Meeting 3 could indicate an increase or decrease in the level of interest in the meeting among members over time. However, as illustrated in Table 10, participation rates were sustained across the three meetings for both variants of the crossover treatment, as well as for all three districts. This indicates that the quality of training being imparted in Executive Body meetings was successful in maintaining members’ interest. There are multiple non-monetary benefits associated with a successful School Management Commit- tee. These include parental involvement in school-related activities, greater accountability of Sindh Government officials, and heightened awareness among the community about the advantages of ed- ucation. However, non-monetary benefits may take time to manifest. On the other hand, finance-re- lated benefits can be realized in the short term. This is because increased use of School Management Committee resources translates into better facilities, potentially better teachers and an overall improved learning environment for students. The School Improvement Plan contains estimates of costs for making various physical improvements to schools. In the following section, we analyze these cost estimates prepared by Executive Bodies. This 68 ANALYSIS OF CAPACITY-BUILDING SUPPORT analysis is made both across districts (comparing average cost estimates between districts) and treatment arms (comparing average estimates made between villages in the different treatment samples). Thus, we train a lens on the level of finances that communities require for the better functioning of their schools through an analysis of approved School Improvement Plans, which are also an important indicator for need at the local level. Average Cost Estimates in School Improvement Plans Executive Body members received training on how to develop a School Improvement Plan for their School Management Committee. First, however, they learned how to check the availability of funds in the Committee’s bank account. Once they ascertained the availability of funds – or lack thereof – they could start developing the improvement plan, based on the amount of funds available and the needs of the school. Aggregate cost estimates were recorded for each village. These were extracted from School Improvement Plans prepared by the Executive Bodies and finalized during the third and final Executive Body meeting held under the auspices of the crossover intervention. As with attendance rates, we averaged cost esti- mates across schools at the village level, and across villages at the district level (or at the treatment level). The Sindh Government is legally bound to provide PKR 22,000 (approximately USD 214) annually to each School Management Committee. However, the individual Executive Bodies did not limit their cost estimates to this funding level. The cost estimates in many Plans were greater than PKR 22,000. This is primarily due to two reasons: 1) many of the School Management Committees had not un- dertaken repairs over multiple years, leading to significant investments for repairing depreciated assets; and 2) some Committees had been receiving annual funding which they had not spent, and therefore, had accumulated in their bank accounts. These Committees could afford to develop plan cost estimates greater than the annual funding limit. Looking at the distribution of mean cost estimates, it is clear that large differentials existed across dis- tricts. The overall mean cost estimates for Mirpurkhas (PKR 40,365 or USD 392) were almost twice as large as those for Sanghar (PKR 23,214 or USD 225). We can therefore infer that, by and large, villages in Mirpurkhas either required a larger overhaul of their schools as compared to schools in Sanghar, and/ or have been underspending their funds more than those in Sanghar. Table 11: Distribution of Cost Estimates (Statistics) Mean Mean Standard 10th 50th percentile 90th   (Current (PKR) Deviation percentile (Median) percentile USD) Mirpurkhas 40,365 392 30,039 21,000 25,525 77,500 Mitiari 34,047 331 17,780 21,025 21,950 62,000 Sanghar 23,214 225 9,749 16,000 21,500 28,600 While differentials in cost estimates existed across districts, there were also large variations across vil- lages within the same districts. Looking at the 10th and 90th percentiles, we see that the 10th percentile cost estimate for Mirpurkhas was PKR 21,000, compared to PKR 77,500 for the 90th percentile. If ANALYSIS OF CAPACITY-BUILDING SUPPORT 69 we use cost estimates as a proxy for available funds, we can assert that villages at the 90th percentile in Mirpurkhas had funds that accumulated for an average of 3.5 years (PKR 77,500, when annual fund- ing is PKR 22,000). Variation in Mitiari revealed a similar pattern as in Mirpurkhas. However, in Sanghar, we find a nar- rower range: The 10th percentile cost estimate was PKR 16,000, compared to the 90th percentile cost estimate of PKR 28,600. Figure 13 illustrates the distribution of cost estimates for the three districts. Figure 13: Distribution of Cost Estimates Across treatment arms, there was relative consistency in terms of average cost estimates. District Mitiari showed higher mean cost estimates as compared to Sanghar, but lower mean cost estimates as compared to Mirpurkhas. Expenditure Analysis of School Improvement Plans The earlier word cloud analysis presented in Chapter 7 highlighted multiple issues related to education, including lack of quality teachers, poor facilities, insufficient finances, lack of support from the education department; and the community’s perception of the effectiveness of the SMS campaign. The School Improvement Plan, in contrast, is supposed to focus on the specifics of running a functioning school: ceilings, boundary walls, gates, fans, bathrooms, blackboards, and desks. It is clear, therefore, that the Executive Body’s primary responsibility is to ensure that the school is physically running smoothly. 70 ANALYSIS OF CAPACITY-BUILDING SUPPORT Figure 14: Word Cloud of Frequently Stated Budget Items in School Improvement Plans This analysis relies on the frequency of common budget line items appearing in School Improvement Plans as a measure of the basic needs of schools at the local level.28 The size of each word in the word cloud in Figure 14 is proportional to the frequency with which it appeared in the Plans. The word cloud reveals that there was a dire need to implement structural repairs in schools. Furniture and fixtures also featured frequently across School Improvement Plans. Next in line were water, sanitation and hygiene needs. Clean drinking water, access to bathrooms, and good hygiene ensured by regular cleaning are necessary for attracting students to schools and for providing them with a healthy environment for learning. Many of the schools function without electricity, which is why electricity installation and repairs were also frequently cited. Lastly, schools appeared to be in urgent need of stationery for students.29 It is obvious that generally, the first step that School Management Committee Executive Bodies wanted to take was to improve the physical condition of the schools in order to make them more attractive for stu- dents to attend. Generally, a community driven to improve the state of education at the local level would also be incentivized to take care of school facilities that it has helped to repair and renovate. The word cloud reflects the dire physical state of schools at the local level. It must be kept in mind that Exec- utive Body members have a low capability for requesting bids for jobs, filling purchase orders, record-keeping and maintaining budgets. Given that procurement appears to be a major next step in implementing School Improvement Plans, designers of future interventions should keep in mind the capacity constraints of Exec- utive Body members, when it comes to navigating the process for procuring goods and services. 28 Cost estimates for budget heads are not used to construct the word cloud. This is because certain cost items, especially those related to construction are expensive and would therefore, shift the distribution of funds in their favor, and skew their perceived importance. Frequency of cost items is a better indicator to identify common issues faced by schools across the three target districts. 29 Infrastructure here encapsulates cost items such as ceiling and floor construction, as well as the school’s gate, walls and fixtures such as doors cabinets, blackboards, fans and windows. Facilities include furniture such as chairs and tables; stationary for students; as well as amenities related to hygiene, toilets and clean drinking water. Repairs include both repairing of infrastructure as well as painting of walls and ceiling. CONCLUSION 71 Conclusion The community engagement mechanisms tested in this field experiment attempt to advance our under- standing of challenges faced in the design, implementation and take-up of innovative technology-based solutions. This knowledge can help to inform the use of similar strategies in other World Bank projects. The design of a project can help to shape not only the level of participation among community mem- bers, but also the degree of responsiveness of, and collaboration among, local community members. In this project, the key design parameters that contributed to the success of village-level meetings were: 1) reaching out to the community through channels that they trusted and that they were familiar with, such as village elders, youth, and mosques; 2) selecting community volunteers who understood the local context and who could facilitate participation of community members; and 3) providing support and constantly monitoring the implementing partner companies to ensure that they did their job effectively in the field. Likewise, the salient design features of the Community Dialogue Platform intervention were: 1) a viable technology architecture for two-way communication to take place in rural Sindh; 2) customization of the Platform to take into consideration the local language as well as mobile devices used in these villages; and 3) a combination of nudges in the form of airtime credit and community volunteers to support the take-up of this initiative. Lastly, the key design elements of capac- ity-support intervention were: 1) a transfer of authority through the district officials to newly elected members in village meetings; 2) participatory training where the members were expected to draft a School Improvement Plan by the end of the third meeting; and 3) availability of financial resources to implement those plans. This field experiment illustrates the level of complexity involved in a small-scale project targeting 230 communities in rural Sindh. We recommend conducting similar field experiments before scaling up such mechanisms for sustained citizen engagement. Lessons learned from the design and implementa- tion of interventions could prove useful for developing a refined design to be implemented by the Sindh Government. The pilot stage could address the fundamental issue of what strategies most effectively engage local communities. Once a successful design has been established, the dialogue and efforts of the Sindh Government could focus exclusively on implementation. Otherwise, we might commit an error in concluding that mechanisms to support community engagement are not effective when, in fact, the quality of supervision and capacity might have been lacking on the government side. 72 BIBLIOGRAPHY Bibliography Agarwal, S., Asim, S., Khan, A. (2012). Implementation Completion and Results Report: Sindh Education Sector Project, Pakistan. Report Number: ICR2481. Washington DC: World Bank. Ali, S. (2006). Why does policy fail? Understanding the problems of policy implementation in Pakistan - a Neuro-Cognitive Perspective. International Studies in Educational Administration, 34(1). < http:// ecommons.aku.edu/cgi/viewcontent.cgi?article=1009&context=pakistan_ied_pdck>. Asim, S. “About.” Weblog post. Education Policy Dialogue. n.d. Web. 17 Aug. 2014. . Asim, S., Raju, D., Adil, M., Pandey, P. and De Icaza, L.. (2013). Impact of Community Engagement Mechanisms on Education Outcomes in Sindh, Pakistan, Baseline Survey: Unpublished Data. Washington DC. Banerjee, A. V., Banerji, R., Duflo, E., Glennerster, R., & Khemani, S. (2010). Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India. American Economic Journal: Economic Policy, 1-30. Bruns, B., Filmer, D., & Patrinos, H. A. (2011). Making Schools Work: New Evidence on Accountability Reforms. Washington DC: World Bank. Carneiro, P., Das, J., & Reis, H. (2010). Estimating the Demand for School Attributes in Pakistan. University College London, Institute for Fiscal Studies and Centre for Microdata Methods and Practice. (Preliminary Version). Coffey International Development. (2007). The Role of Communication in Governance: Detailed Analysis. . Evans, Peter. (2014). Pakistan - Key Statistics, Telecom Market and Infrastructure, Regulatory Overview, and Forecasts. Budde Comm. . Hirschman, Albert O. Exit, Voice and Loyalty: Responses to the Decline in Firms, Organizations and State. Cambridge: Harvard UP, 1978. Print. Oxfam India. (2014). Community Participation and Institutional Experiences in School Education: School Development and Monitoring Committees in Karnataka. New Delhi. . Pakistan Bureau of Statistics. (2013). Pakistan Social and Living Standards Measurement Survey (PSLM) 2012-13 Provincial / District. Government of Pakistan. Pandey, P., Goyal, S., & Sundararaman, V. (2009). Community Participation in Public Schools: Impact of Information Campaigns in Three Indian States. Education Economics, 17(3), 355-375. Pradhan, M., Suryadarma, D., Beatty A., Wong, M., Alishjabana, A., Gaduh, A., & Artha, R. (2011). BIBLIOGRAPHY 73 Improving Educational Quality through Enhancing Community Participation: Results from a Randomized Field Experiment in Indonesia. Policy Research Working Papers. The World Bank. Pritchett, L. (2013). The Rebirth of Education: Schooling Ain’t Learning. Washington, DC: Center for Global Development. Reform Support Unit. (2014). Annual Schools Census 2013-14. Education and Literacy Department, Government of Sindh. Uemura, Mitsue. (1999). Community Participation in Education: What Do We Know?. Washington, DC: World Bank. . World Bank. (2003). World Development Report 2004: Making Services Work for Poor People. World Bank. https://openknowledge.worldbank.org/handle/10986/5986 License: CC BY 3.0 IGO.” World Bank. (2009). Sindh Education Sector Project (SEP) Project Appraisal Document. (Report No: 47642-PK). Washington DC. World Bank. (2013). Sindh Education Sector Project (SEP-II) Project Appraisal Document. (Report No: 74552-PK). Washington DC. 74 ANNEXURE Annexure 1. Snapshots of Select Project Tools 2. Technical Design of the Community Dialogue Platform 3. Content of Informational Messages and Interactive Voice Response Calls a. Informational SMS Messages – Round 1 (Translated) b. Informational SMS Messages – Round 2 (Translated) c. Interactive Voice Response Calls – Round 1 (Translated) d. Interactive Voice Response Calls – Round 2 (Translated) 4. Workflow Diagrams a. Intervention 1 (School Management Committee) b. Intervention 2 (Community Dialogue Platform) 5. Timeline of the Intervention 6. Material from Executive Body Meetings 7. Sorting Algorithm 8. Project Cost Estimates 9. Data on Interactive Voice Response Calls 10. List of Databases ANNEXURE 75 Annex 1: Snapshots of Select Project Tools Instruments such as handouts, flyers and scripts played a critical role in the successful implementation of the project. Handouts and flyers helped to spread the word about village meetings while easy-to-understand scripts provided guidelines on how to conduct the various kinds of meetings. This annex provides snapshots of these project tools. 76 ANNEXURE Annex 2: Technical Design of the Community Dialogue Platform While Chapters 3 and 4 provide an overview of how the Community Dialogue Platform was designed, pi- loted and implemented, the significance of the portal, as well as its innovative structure, necessitate a detailed discussion of its design features. This annex lays out the fundamental concepts upon which the platform was built, and provides details about its structure. This annex is especially relevant for readers who wish to under- stand the nuts and bolts of the portal. The Community Design Platform was programmed using ASP.Net on the Windows 8.0 platform. ASP. Net is a faster and easier alternative to other Web application frameworks such as Perl or C. Further, it can accommodate a host of languages, including VBScript or Jscript. It also allows for separation between the database and the web design aspects of the platform. Such features allowed for ease of maintenance. This is important given that programmers of the portal are not necessarily the same peo- ple who maintain it. Anyone with a working knowledge of the platform could, therefore, make course corrections to avoid potential problems that might emerge during project rollout. This feature thus protected the portal from potential staffing changes on the project team. An external Modulator/Demodulator, or modem, was used for sending and receiving messages. The modem was connected to the server, allowing for messages to be directed to specific sections of the portal’s database. Before any activity was initiated, the administrator of the Platform checked the con- nectivity status of the modem by sending a test message and waiting for a response, and by checking the modem test report for its connectivity status. In case the modem was not responding, the administrator had to contact the M3 Technologies support team to restart it. The portal has its own dedicated URL with server space reserved for storing large amounts of data. It can be accessed at http://www.aknservices.com/worldbanksms/. A username and password are re- quired to log in; this protects the database from unauthorized users and secures the confidentiality of the participants. A snapshot of the main page of the portal is provided in Figure 1530. The page has a minimalist design, with an emphasis on functionality and ease-of-use rather than aesthetics. The Community Dialogue Platform was supported by three principles: 1) a smooth flow of infor- mation from the users to the platform; 2) ease of data management for back-end data managers; and 3) versatile access to existing information for data analysis. This subsection discusses these principles and provides a background on the decisions that the project team had to make to balance them with feasibility. Streamlined Participant-Portal Information Flow The first decision that M3 Technologies – the lead technology partner on the project – and the World Bank team had to make involved the channel to be used to connect with village participants. The cost of sending messages to participants would depend on the traffic generated on the portal. Given that the traffic flow was relatively unpredictable, it was difficult to do a cost-benefit analysis of the method for sending messages to participants. The two primary methods for sending these text messages were the long code and the short code. Short codes are reliable, and faster than long codes, and would enable users such as the project team to track receipt of messages. Short code charges are not tied to the level of traffic generated, and therefore, 30 All snapshots are provided in the “notes” section at the end of this chapter. ANNEXURE 77 are ideal for massive informational and marketing campaigns which generate hundreds of thousands of text messages. However, they have a fixed price level and are much more expensive in terms of fees paid to telecom carriers. If short code were used, each message received from village participants would have cost PKR 1 per participant, unless an overall traffic threshold were achieved. Bear in mind that this was a discounted rate for nonprofit organizations. For corporate campaigns, such messages cost twice as much. The Bank’s project partner M3 Technologies held discussions with two of Pakistan’s big- gest telecommunications companies, Mobilink and Telenor, on whether they could reduce these rates further. The two companies said villagers could use short code at no cost to them, if the project could pay a fixed cost per month (approximately PKR 100,000 or USD 980) on behalf of the participants. Long codes are less reliable compared to short codes, and do not provide administrators with the abil- ity to confirm receipt of messages sent. However, they do not have a fixed price and the cost for both sending and receiving messages is relatively inexpensive. Long codes are not regulated like short codes, reducing the ability of users to control spam. But that also means that administrators have greater flex- ibility in sending out messages, with no wait time required between sending different text messages. In contrast, in the use of short code, telecom companies will vet any messages before they are received by project participants. Long codes also have a dedicated number that can send and receive messages, and which could be easily saved and recalled by project participants. Analyzing these pros and cons, the project team decided to use long codes so that: 1) the cost of the project to both the project team and the participants was minimal; and 2) the flow of information was uninhibited. The long code used for the project was +92 315 3002182. The project team at M3 Technologies was responsible for consistent debugging of the portal in case of any issues. Debugging was especially necessary at the beginning of the project when unexpected issues were likeliest to arise. The team carefully assessed whether text messages being sent by village partici- pants were actually being received on the portal, and whether texts being sent by the project team were being received by village participants. As mentioned earlier, long codes do not allow for receipt of mes- sages to be verified. However, the team on the ground informed the M3 Technologies team whether the summary messages had been received by the community during the initial field visits. This sustained evaluation process was necessary to ensure that there were no stoppages in the flow of information between participants and the portal. Versatile Data Management The initial village meeting played a significant role in generating new traffic on the portal. But other catalysts were needed to maintain participant interest. These included individual messages sent to par- ticipants in two rounds over the course of the project, and periodic summary text messages that were sent to participants on a weekly basis. Further, maintenance of village-level information, such as data related to village meetings, polling, tagging of messages and contact details of participants, was time sensitive and required a robust system of data management. Data in the form of messages was automatically stored in the database when these messages were re- ceived using long code. The larger database consisted of multiple filters, such as date, contact number of the message sender, village code and type of text message content. With the help of these filters, ex- tracting specific parts of the database became easier. Otherwise, this would have been a Herculean task, requiring managers to sift through large amounts of unwanted data to get what they needed. 78 ANNEXURE Attendance data in the form of contact information of participants could be conveniently uploaded on the portal. This was done by accessing the “Meeting” tab and clicking on “Upload Meeting Contacts.” The relevant tabs for this process are highlighted in Figure 16 in the notes below. For ease of use, attendance details received from meetings did not have to be entered one participant at a time. Instead, the project team could fill out a pre-designed Excel worksheet and upload it to the portal. It was then automatically appended to the existing database maintained by the portal. Wherever bulk information had to be uploaded, a pre-designed Excel sample was provided so data managers were always aware of the correct format of the worksheet that was to be uploaded. This precluded errors and made uploading information more convenient. Figure 17 provides a snapshot of a sample worksheet, specifically the one used to upload attendance information. The portal could also be used to update contact information after the initial meeting. Once the meeting ended, the portal could send text mes- sages to participants. These could be in the form of single text messages sent to individual numbers, or bulk texts sent to a large number of participants in specific villages. The bulk texts were the primary tool used to send summary messages as well as informational messages to participants. Using two col- umns in a simple Excel spreadsheet, portal users could construct a list of messages corresponding with contact numbers to which these messages should be sent. Once this was done, the file was uploaded on the portal. The portal would then automatically go down the list and blast text messages to the corresponding participants. Cataloguing received messages was just as important, if not more so, as managing the process of send- ing messages. This was especially true since a large number of messages received from participants were junk messages. Junk messages – those that were incomplete or irrelevant – would have clogged up the Community Dialogue Platform if they were not filtered. It was important to sort non-junk messages into different buckets related to different aspects of school management and educational governance at the local level. This allowed for a large, nebulous dataset to be turned into smaller fractions of information focused on different topics. Sorting was accomplished by tagging messages based on the message’s topic of discussion. One of the weaknesses of the portal design was that tagging had to be undertaken manually. The primary reason for this was because the design could not undertake tagging in multiple languages, and participants used several languages for texting: English, Urdu, Roman Urdu and Sindhi. Therefore, tagging required significant manpower and was bound to induce some errors. However, the magnitude of errors was assumed to be low, since a single message could be tagged with multiple labels and therefore, each non-junk message was expected to fall into at least one relevant category. Annex 7 provides further details about the categorization of messages. Ease of Access to Data for Analysis The portal was designed to accommodate simple data extraction and recall. The interface provides highly intuitive categorization of data, making it easy for analysts to query and retrieve information from the database. This was a critical design feature, since the analysts were assumed to be different from the data managers. It was essential that the design of the portal allow for data retrieval without an exhaustive understanding of the data entry and management process. Figure 15 provides a snapshot of the portal’s main page. Designers placed a strong emphasis on making it visually effective. The tabs at the top allow for easy navigation through the data. The “Intervention” tab gives options for filtering and retrieving information that was collected during the intervention. This tab is primarily used to access text messages that were received from project participants. Such ANNEXURE 79 messages could be sorted based on village type, individual contact number, community volunteer code, and their tags. Similarly, the “Meeting” tab – as the name suggests – provides access to information collected during the first village meeting. The “Sent SMS” tab provides detailed information on messages sent from the portal to participants by the project team. Once again, the key feature of the portal is its ability to filter information. Logs of sent messages could be viewed either by SMS type (single or bulk) or by contact numbers that the messages were sent to. This feature allows data analysts to hone in on different parts of the sent text messaging data. A key feature of summary statistics on portal activity that could be generated by the portal was that these statistics could be filtered by date, mobile number, SEMIS (Sindh Education Management Infor- mation System) code (government-assigned identification codes for public schools), GPS coordinates, district, tehsil, village, settlement ID (project-defined), contact person name, and type of contact person (village representative or participant). Therefore, analysis of activity on the Community Dialogue Plat- form can be disaggregated using such filters. This disaggregation allows analysts to examine heterogene- ity in activity on the portal. For example, the project team might seek to analyze activity during specific periods for a particular district, and for a specific group of stakeholders, such as village participants. The disaggregation of data on the portal allows the team to do this, saving significant amounts of time. One of the most important outcomes of real-time analyses was that the project team could retain situational awareness of the project, and understand which areas had a higher fraction of active, versus passive, users. Active users were defined as participants who sent non-junk messages to the portal, while passive users did not directly engage with the portal, but who were likely to inform the opinion of active users. 80 ANNEXURE Notes The following snapshots illustrate different aspects of the Community Dialogue Platform: Figure 15: Main Page of the Community Dialogue Platform Figure 16: Uploading Attendance Contacts on the Community Dialogue Platform Figure 17: Example of Sample Worksheets on the Community Dialogue Platform ANNEXURE 81 Annex 3: Content of Informational Messages and Interactive Voice Response Calls External nudges that aimed to catalyze activity on the Community Dialogue Platform and to prevent drift in community-level discussions were meticulously crafted to be highly effective in the context of rural Sindh. An- nex 3 deals with two such nudges – Informational SMS messages and Interactive Voice Response calls – and sheds light on their content. The first two sub-annexes provide English translations of informational messages that were sent in Urdu across two rounds. The last two sub-annexes provide information on Interactive Voice Response calls, including their purpose and the translation of their content. Annex 3a: Informational SMS Messages – Round 1 (Translated) The table below provides details on specific informational text messages sent to community members during Round 1, categorized by themes and sorted chronologically. Table 12: Translation of Informational Messages - Round 1 Day Thematic Area Informational SMS Message The Government of Sindh welcomes you to the SMS project. Teachers, School Management Committee members and parents will be connected via 1 Welcome Message this project, in which all of us, together, will be able to provide your child with a better education. You will be provided free of charge information regarding the school through Information on the SMS project. You can text your opinion about the given information to 2 Process 0315-3002182. A summary of all messages received from the village shall be sent back to you. Every year the School Management Committee of each primary school receives PKR 22,000, which goes into the Committee’s account. This account can be used with the signatures of the headmaster or the chairperson. 3 SMC Funds A School Management Committee’s budget can be spent on three things: 1) facilities and maintenance of the school; 2) wages for workers to clean the school; and 3) transportation of students to and from school Every year, the Government of Sindh provides all books free of charge to 4 School Inputs each student in primary school. These do not include notebooks. Each School Management Committee has two components: the General Body and Executive Body. The General Body includes teachers and parents of all the schoolchildren enrolled in the respective school. The Executive Body consists of five members: the headmaster, who is the general secretary; two representatives from among the parents, one of who is 5 Structure the chairperson; and two respected members of the village. The responsibility of managing the School Management Committee’s administrative processes lies with the head teacher and chairperson. Since the chairperson is from among you, you have a huge role to play in the workings of the Committee. 82 ANNEXURE Elections for the Executive Body in a proper manner are necessary for the success of the School Management Committee. Members are elected every 6 two years. Elections are held during a meeting of the General Body. The headmaster is responsible for conducting these elections. A general announcement regarding the elections should be made at least 10 days before the elections. Four members of the Executive Body, excluding 7 the headmaster, are elected. People who want to become members of the Elections Executive Body come forward on Election Day. Elections for these seats are held one by one. The Government of Sindh would like to inform you that there are two ways to vote in School Management Committee elections: by ballot or by a show of hands. All members of the General Body will vote for their preferred can- 8 didate. The candidate with the most votes for a seat will be elected to that seat. No member can be a close relative of another member. Keep in mind that in a girls school, at least three of the members should be female. The Government of Sindh would like to inform you that the biggest respon- sibility of the School Management Committee is the smooth running and betterment of the school. 10 Action Planning If there is a dearth of anything at the school, then it should be addressed through the following ways: take part in School Management Committee meetings; visit schools; and stay in regular contact with the members of the Executive Body. The General Body must meet at least twice a year. A general announcement for these meetings should be made at least three days prior to the meeting. The day and timing of the meeting shall be posted outside the school as well as communicated to children. At least one-third of the parents must be present at the meeting. It is the responsibility of the headmaster to maintain minutes of the meeting. 11 Each parent, especially mothers, should actively participate in the General Body meetings. How the money allocated to the School Management Com- mittee is to be utilized should be decided in the meeting. Voice your opinion on the Executive Body’s plans regarding the betterment of the school and approve them. If a certain expenditure is causing differing opinions, then the General Body one agreed upon by the majority shall be accepted. Meeting Teachers’ attendance, teaching techniques and students’ academic perfor- mances should be discussed in the General Body meeting and decisions about them executed. If there is a complaint about a teacher or a student is lacking in his or her studies, discuss the issue at the meeting or at school with the teacher. If this is not enough, then talk to the head teacher or officer 12 superior to the head teacher. The Executive Body makes a list of the village’s children who are not enrolled in school. Discuss ways in which all such children can be enrolled in school. It is essential for such decisions that you give your opinion and a unanimous decision is reached. If a certain decision is causing differing opinions, then the one agreed upon by the majority shall be accepted. ANNEXURE 83 The Government of Sindh would like to inform you that you should visit schools and check the following: Do the teachers arrive at school punctually and teach properly? Do the students know how to read and write according School Visit to their grade level? Are the school facilities usable or not? Are approved plans for improving the school being implemented properly? If you feel anything is lacking, then discuss it in the General Body meeting or contact 13 the Executive Body. The betterment of the school will happen through the Executive Body because it comprises members from among you. You should talk to these members without hesitation and give them your opinions and improve the Contact with school together. Check if the Executive Body meets on a monthly basis and Executive Body carries out its work properly. You have been provided names and mobile numbers of all committee 14 members. If you cannot meet them, then talk to them on the phone. Stay in regular contact with these members. Annex 3b: Informational SMS Messages – Round 2 (Translated) The informational text messages sent out in this round are categorized and presented in the table below: Table 13: Translation of Informational Messages - Round 2 Thematic Area Informational SMS Message • The SMS project is a method for improving the school. Love of children, a call to knowledge. • Make the Executive Body, head teacher and teachers hear your views. • These steps can lead to an improvement in the school: • Exchange views via the SMS project. • Get involved in discussions with the Executive Body and teachers. SMS Portal – School • Take part in the planning of improvements in the School Improvement Improvement Plan or school. • Make sure that the expenditure of the School Management Committee’s budget is according to approved School Improvement Plan. • A summary of the villagers’ opinion is sent on a weekly basis to the villagers. The summary does not include anyone’s name and your anonymity is guaran- teed. • You should send your opinions and recommendations via text message to the villagers. 84 ANNEXURE • Each primary school has its own School Management Committee. • Primary school’s School Management Committee receives annual funding of PKR 22,000. • SMC’s Executive Body has five members. Four members are selected via elections. The Executive Body has a monthly meeting. • SMC’s General Body consists of all the parents. General Body meetings are held School Management at least twice a year. Committee • School Improvement Plan consists of the details of the expenses incurred by the Information School Management Committee. • Villagers should include all such expenses in the School Improvement Plan that they consider important for the school via discussions through the SMS project. • The School Management Committee’s elections and approval of School Improvement Plan are decided by a vote among the General Body. • Name and phone numbers of members of the Executive Body are provided in the brochure given to you during the meeting. If you cannot find the names, then ask the head teacher. • Express your desire to participate in the SMS project by messaging 0315- 3002182. • To send messages via the portal, you received mobile credit worth PKR 30 after the meeting. Use this credit and express your opinion via text message. You will SMS Portal - receive credit in the upcoming months as well. Functioning • If you do not know how to read or write text messages, then meet with your village representatives [representatives’ names]. These are their contact numbers: [representatives’ contact information]. If these representatives do not help you, let us know. Annex 3c: Interactive Voice Response Calls – Round 1 (Translated) The table below provides information on the purpose, content and options provided in these Interactive Voice Response calls: Table 14: Translation of Interactive Voice Response Calls - Round 1 Call Purpose Content Options # Audience: Community Participants This was an introductory message, The Government of Sindh wel- 1. For “Yes,” press 1. checking on the status of households’ comes you to the SMS project. A attendance at the village meeting. few days ago, a meeting was held 2. For “No,” press 2. 1 in your village for the initiation 3. For “I don’t know,” of the SMS project. Did anyone press 3. from your household attend the meeting? ANNEXURE 85 The purpose of this voice message Through the SMS project, 1. For “Sindhi,” press 1. was to acquire participants’ prefer- important information about the ences for the language used in the school will be sent to you in the 2. For “Urdu,” press 2. text messages sent to their mobile form of a text message. These text 2 phones. Participants received text messages can be sent in Urdu or messages in the preferred languages Sindhi, depending on your pref- after this Interactive Voice Response erence. Which language do you call. want to read the text message in? The third Interactive Voice Response The SMS project has been initiat- 1. For “Yes,” press 1. follow-up call aimed to gather ed to bring improvements to the information on the participation of school. Your participation in this 2. For “No,” press 2. registered villagers in the project, and project is very important. Have to understand the reasons for them you sent any text message(s) to staying away from project activities. the project? If “No,”, then why did you not 1. For “lack of credit participate in the project? balance,” press 1. 3 2. For “I don’t know how to text,” press 2. 3. For “I don’t want to participate in the project,” press 3. If “I don’t know how to text,” did 1. For “Yes,” press 1. the village representatives help you in sending a text message? 2. For “No,” press 2. The last Interactive Voice Response Informational messages about the 1. For “Yes,” press 1. call of the first round sought to School Management Committee gather information on whether have been sent to you over the 2. For “No,” press 2. intra-community engagement was last few weeks. Based on these being generated by the Community text messages, did you have a Dialogue Platform project and spe- conversation with anyone? cifically, between which stakeholders. If “Yes,” with whom did you have 1. For “Members of the 4 this conversation? Executive Committee,” press 1. 2. For “Teacher or head teacher,” press 2. 3. For “Other parents, friends and relatives,” press 3. 86 ANNEXURE Annex 3d: Interactive Voice Response Calls – Round 2 (Translated) The table below provides information on the purpose, content and options provided in these Interac- tive Voice Response calls: Table 15: Translation of Interactive Voice Response Calls - Round 2 Call # Purpose Content Options Audience: Community Participants The first Interactive Voice Response Hello. This call is being made by 1. For “Improvements call of the second round asked the Government of Sindh. What in the school’s facilities,” participants about changes brought is the biggest change that has been press 1. about by the Community Dialogue brought about in your village Platform intervention. because of the SMS project? 2. For “Lower teacher absenteeism,” press 2. 1 3. For “More interaction among community members,” press 3. 4. For “No improvement,” press 4. The purpose of this call was to Hello. This call is being made by 1. For “Lack of credit understand the reasons for the Government of Sindh. What balance,” press 1. potential non-participation of is the biggest problem for you in registered participants in the terms of participating in the SMS 2. For “I don’t know how Community Dialogue Platform project? to read and write text intervention. messages,” press 2. 2 3. For “I don’t understand the summary messages,” press 3. 4. For “I haven’t faced a problem,” press 4. This important call was made to Hello. This call is being made 1. For “Very useful,” understand the effectiveness of by the Government of Sindh. In press 1. summary text messages that were your opinion, how helpful are the blasted to the local community. summary text messages that are 2. For “Useful to a 3 sent by the SMS project? certain extent,” press 2. 3. For “Not useful at all,” press 3. The last call, similar to the first one, Hello. This call is being made by 1. For “Improvements asked participants about changes the Government of Sindh. What in the school’s facilities,” brought about by the Community is the biggest change that has been press 1. Dialogue Platform intervention. brought about in your village be- cause of the SMS project? 2. For “Lower teacher absenteeism,” press 2. 4 3. For “More interac- tion among community members,” press 3. 4. For “No improvement,” press 4. ANNEXURE 87 Audience: Village Representatives The first Interactive Voice Response Respected representative, this call call made to village representatives is being made by the Government provided procedural information to of Sindh. It is your responsibility the representatives on how to to strengthen the SMS project. undertake their responsibilities Help villagers in composing and under the Community Dialogue sending text messages about Platform intervention. the school, and win free credit 1 balance as a prize. Keep in mind that: 1. The text message should only be sent from villagers’ mobile phone numbers. 2. The text message should in- clude your code. The second Interactive Voice Respected representative, this call Response call made to village is being made by the Government representatives provided them of Sindh. From May 1, it will with information on the incentive be even easier to win free credit structure that they faced, in terms balance as a prize. After May 1, of free credit balance. you will receive free credit balance based on the following: 2 1. PKR 50 for getting 10-20 text messages sent. 2. PKR 100 for getting 20-50 text messages sent. 3. PKR 200 for getting 50 or more text messages sent. based on the following: 1. PKR 50 for getting 10-20 SMS messages sent 2. PKR 100 for getting 20-50 88 ANNEXURE SMS messages sent 3. PKR 200 for getting 50 or more SMS messages sent Annex 4a: Workflow Diagram – Intervention 1 ANNEX 4A: WORKFLOW (School DIAGRAM – INTERVENTION 1 Management Committee) This annex focuses This annex speci focuses cally on specifically the SMC onthe intervention School Managementand illustrates Committee each step intervention and taken in its illustrates rollout each step with key details. taken in its rollout with key details. SOCIAL Field teams utilized posters, a musical At least one member from each household mobilized by eld teams MOBILIZATION jingle and public announcements via CAMPAIGN megaphones and mosques Attendance rates > 40% for majority of VILLAGE Meeting facilitated by two team meetings. Very low rates were used as MEETING members who received prior classroom indicator to remobilize community and training and utilized prepared script reorganize meeting ORGANIZED SMC ELECTIONS FACILITATED First audio tape/CD played, detailing Second audio tape/CD played, purpose, structure, membership and CAPACITY- recommending actions for improving funding of SMCs, and recapped using BUILDING OF education outcomes and recapped ip charts PARTICIPANTS using ip charts A discussion was facilitated between community members, based on the FACILITATED model introduced in audio tapes/CDs DISCUSSION and ip charts Training included recap of SMC Three hands-on trainings provided functions; developing, nalizing and over a period of three weeks to elected CAPACITY-BUILDING OF ratifying SIP; and withdrawing funds, members and head teacher who is the ELECTED MEMBERS bookkeeping and tracking progress of co-chair of SMC SIP activities Core Activity Cross-cutting Activity Description Annexure 75 ANNEXURE 89 Annex 4b: Workflow Diagram – Intervention 2 (Community Dialogue Platform) ANNEX 4B: WORKFLOW DIAGRAM – INTERVENTION 2 (CDP) This annex focuses specifically on the Community Dialogue Platform and illustrates each step taken in its This annex focuses speci cally on the CDP and illustrates each step taken in its design, testing and design, testing rollout with keyand rollout, including key details. details. DEVELOPMENT Building on past experiences Multilingual, economical, easy-to-use OF SMS PORTAL Systems pilot-tested in sample villages FIELD TESTING and feedback used to re ne nal portal OF PORTAL ROLLOUT IN Villagers mobilized through posters TREATMENT and announcements via megaphones VILLAGES and mosques Attendance of participants used to VILLAGE Audio tape/CD played, detailing the build database of participants via MEETING uses of the portal and recapped with GPRS submissions ORGANIZED the help of ipcharts SMC ELECTIONS FACILITATED Participants respond to an MCQ by Interest in portal built on 3 pillars: SMS summary text of responses DEMONSTRATION anonymity; awareness of community’s blasted in real-time to participants. OF PORTAL viewpoints; cheap, easy to use tech Two facilitators selected to assist those SELECTION OF Interest in portal built on 3 pillars: participants in using the portal who are COMMUNITY anonymity; awareness of community’s not mobile literate VOLUNTEERS viewpoints; cheap, easy to use tech Disincentives to using portal because Registration on the Portal REGISTRATION of cost removed through provision of free credit to participants COMMUNITY DIALOGUE PLATFORM ESTABLISHED Training included recap of SMC Three hands-on trainings provided functions; developing, nalizing and over a period of three weeks to elected CAPACITY-BUILDING OF ratifying SIP; and withdrawing funds, members and head teacher who is the ELECTED MEMBERS bookkeeping and tracking progress of co-chair of SMC SIP activities Core Activity Cross-cutting Activity Description Annexure 76 90 ANNEXURE ANNEX 5: TIMELINE OF THE INTERVENTION Annex 5: Timeline of the Intervention Design of the intervention began in April of 2011, with signi cant amounts of time taken to develop and of thethe ne tune Design design ofbegan intervention the project in April and to bring 2011, together multiple with significant amounts ofstakeholders time taken tosuch as partner develop and fine-tune organizations. While training of facilitators and pilot activities went through multiple iterations, the nal the design of the project and to bring together multiple stakeholders, such as partner organizations. While train- round of piloting, eld testing and trainings occurred in January 2013. The project was rolled out in ing of facilitators February 2013 and andpilot tillwent activities continued through the end multiple of June 2013. iterations, the final The following round Gantt chartof provides piloting, field-testing an overview of and trainings these occurred key project in January 2013. The project was rolled out in February 2013 and continued until the activities. end of June 2013. The following Gantt chart Tableprovides an overview 16: Timeline of these key project activities. of Intervention 2011-12 Table 16: Timeline of Intervention 2013 April - Jan Feb Mar Apr May Jun July Aug ACTIVITIES Dec Coding of Community Development Portal (CDP) Pilot testing of CDP portal Development of Community Engagement Material Training of eld facilitators Social Mobilization Village Meeting Demonstration of CDP Revisits to Villages Informational SMS messages round 1 IVR calls round 1 Summary SMS messages Informational SMS messages round 2 IVR calls round 2 Wrap up and analytics generation from CDP Annexure 77 ANNEXURE 91 Annex 6: Material from Executive Body Meetings The project aimed to strengthen the institution of the School Management Committee by providing its lead- ers – the Executive Body members – with capacity-building support. This support focused specifically on strengthening process knowledge on how to manage the Executive Body. Lack of such knowledge was seen as the missing link in the smooth functioning of the School Management Committee, following -community mobilization. This annex provides snapshots of the materials that were used for this capacity-building support during the three Executive Body meetings and procedural instruments that Executive Body members were expected to use on a routine basis when managing the affairs of the School Management Committee. Figure 18: Extract from Executive Body Meeting Script 92 ANNEXURE Figure 19: Sample Tender Notice, Quotation Comparison Form, Purchase Order and Payment Voucher Figure 20: Sample Cash Book Figure 21: Sample Stock Register for Executive Body Participants ANNEXURE 93 Annex 7: Sorting Algorithm The Community Dialogue Platform was regularly receiving large amounts of data in the form of text mes- sages sent to the portal by village-level participants. It was crucial that the project team was aware of what this data meant: What were issues of common interest at the local level? How were participants reacting to the portal? What was the percentage of junk to non-junk messages being received by the portal? To streamline the process of analyzing these messages, they were diverted into different data buckets, based on their content. This annex explains the algorithm that was used to sort the data. Messages received on the portal were tagged manually. This had to be done because of the multiple languages being used on the portal. Tagging allowed for real-time and future analysis of the data that was being generated in the form of accumulating text messages received from village participants. For example, analysts could identify what villagers felt were the key issues in the education sector simply by looking at the number of times a keyword – such as “School Management Committee” – appeared. The Community Dialogue Platform had the capability to tag a single message with multiple tags. However, multiple messages could not be tagged in one go. If multiple messages could be tagged simul- taneously, then the process of tagging messages would have taken less time; this is something that could be improved in the design of similar portals in the future. As discussed in Section 8, specific keywords were used to tag all incoming messages. The list of these keywords with their descriptions follows below: Table 17: Sorting Categories Sr. # Keyword Description 1 Junk Incomplete or irrelevant messages. 2 Teacher There was an issue with the teachers in the school, including teacher absentee- ism and teacher quality. 3 Infrastructure The structural integrity of the school was in question, or the amenities provid- ed within the structure were insufficient. 4 Facilities Messages that mentioned facilities, such as bathrooms and drinking water, or provision of desks and chairs. 5 Learning Issues such as poor quality of classroom instruments or instruction or the like, which affected students’ learning at schools. 6 SMC Fund Queries or suggestions about the annual fund provided to School Management Committees. 7 Textbooks Complaints or suggestions related to the books used at school. Issues included a lack of books, incomplete or torn texts, or no books at all. 8 School Management Suggestions and queries about School Management Committees in local Committee schools. 9 Thanks Messages containing notes of appreciation for the project or for community champions. 10 Education Used for messages which covered broad suggestions or queries regarding the Department state of education at schools, such as whether a school was good or poor, or if more attention should be given to education by the government. 11 Parent Messages pertaining to the role of parents in education within the local com- munity. 12 Flood Mention of floods as a reason for poor educational performance in schools and suggestions about how the impact of floods could be mitigated. 94 ANNEXURE 13 Questions and suggestions about the functioning of the Executive Body of the Executive Body School Management Committee. 14 Queries or suggestions which directly mentioned the SMS campaign being SMS Campaign rolled out by the World Bank. 15 Stipend Discussion about the stipend being provided at the school level. 16 School Improvement Plan Conversations around School Improvement Plans. 17 Questions about the budgeting practices at the school, both in terms of School Budget Management Committees and overall budgetary allocations from the government 18 Miscellaneous All other items of discussion If it was not clear whether a message belonged to any of the first 17 categories, then it was tagged as “Miscellaneous.” While most categories are self-explanatory, it is important to describe the types of messages that were marked as junk. The portal identified messages at times as Junk, Junk2 and Junk3, based on categories formed by the team for internal use after the number of junk messages shot up. However, there is no differentiation between the three categories and all three primarily deal with sim- ilar junk messages. A significant number of messages that were received on the portal were blank messages. Since these did not add any value to the discussion on the portal, they were tagged as junk. Certain messages were incomplete, meaning that they might have provided identifiers of individuals, schools or communities in the form of village codes and SEMIS codes and were being received from registered participants, but that they were either incomplete or contained irrelevant information. An example of an incomplete message would be: “JDM our school has”. An extreme example of an irrelevant message would be: “The economy of Punjab province is performing better this year”. A subcategory of irrelevant messages was one in which the sender appeared to be attempting to have a casual conversation on the portal, or was forwarding content that was not relevant to the project. Such messages sent during the project cycle included poetry, local and national news and comments about local politics. For incomplete messages, a caveat is in order. At times, a message was too long and was, thus, broken into two messages, causing the first to be incomplete and the second to be nonsensical. However, if the two were put together, one complete, relevant message could be formed. Messages were split if they were more than 160 characters long if sent in English, or more than 68 characters long if sent in Urdu or Sindhi. M3 Technologies, the primary IT partner on the project, merged all messages that appeared to be split if they were received within a very short duration (one or two minutes). Most of the merging was undertaken manually. ANNEXURE 95 Annex 8: Project Cost Estimates Cost-effectiveness analysis of the project provides additional understanding of the project’s impact. This section lays the foundations for such an analysis, by highlighting the assumptions made in distributing costs across different project components, and specifying the magnitude of these costs. Assumptions for Cost Distribution: 1. School Management Committee villages totaled 119 in number (12,258 participants). SMS villag- es totaled 111 in number (13,384 participants). 2. Two-thirds of the cost of “Development of Audiotapes” was attributed to Intervention 1 (School Management Committee) and one-third of the cost was attributed to Intervention 2 (SMS). 3. For “Executive Body Meetings,” the cost was averaged over multiple schools in each village. 4. For “Honorarium for Assistant District Officer,” the total cost was divided across the four months when honorariums were given: February to May 2013. Monthly exchange rates were used. 5. For “Half-day Workshop for District Officials,” the exchange rate for the month of January 2013 was used. 6. For “Consultants’ Fees for Portal Management,” “SMS Charges”, “Credit Transfer to Participants”, “Credit Transfers to Community Volunteers” and “Interactive Voice Response Calls”, monthly PKR-to-USD exchange rates were used. 7. For all “Monitoring Costs,” average PKR-to-USD exchange rate was used for the year 2013 (dura- tion of contract for Social Policy Development Centre). 8. For “Staff Time” in Total Fixed Development Cost and “Additional Staff Time” in Total Variable Cost, average PKR-to-USD exchange rate was used for the year 2012 in the case of design costs, and for the year 2013 in the case of implementation costs. All costs for “Staff Time were split equal- ly across the three interventions. All costs for “Additional Staff Time” were split equally across the three interventions, with the exception of additional monitoring costs for the portal. These were built into the cost estimate provided for Country Office implementation costs for Intervention 2. 9. “Opportunity Cost for Participants” was calculated as: (Daily Wage / 2) x Number of Participants. For Intervention 1, villagers were advised to conduct a second, non-facilitated meeting after the first General Body meeting had been conducted by the project team. The assumption was made that in all Intervention 1 villages, the second, non-facilitated meeting was conducted with the same participation rate as the first. Therefore, for Intervention 1, the overall opportunity cost was multiplied by two. 10. The daily wage was approximated to be PKR 333, as per the Household Integrated Economic Survey (HIES) 2010-11. Conversion to USD was done using average PKR-to-USD exchange rates for 2012 and 2013. 11. he source for PKR-to-USD exchange rate was “Monthly Statistical Bulletin, Annual Report of SBP and International Financial Statistics (IFS).” 12. All costs are given in USD. 96 ANNEXURE Table 18: Project Costs Contracting Capacity Cost Item Partner SMC SMS Support     Unit Cost Total Unit Cost Total Unit Total (Village) Cost (Village) Cost Cost Cost Fixed Development Cost               Development of         Audiotapes JWT 691 346 Printing of Pamphlets/ Weitek     13 1,500 13 1,500 Posters Group Staff Time Staff/Consultants (HQ) World Bank 22,088 22,088 22,088 Staff/Consultants 2,803 2,803 2,803 (Country Office) Total Fixed Development Cost     27,083   26,737   24,891 Variable Cost               Implementation               General Body Meetings 174 20,000 174 20,000     Weitek Executive Body     Group 87 10,348 87 9,652 Meetings Honorarium for ADOs         29 3,320 Half-day workshop for Reform             District Officials Support Unit Logistics           205 Refreshments           308 Operational Costs of Portal               ANNEXURE 97 Consultants’ Fees for           9,286 Portal Management SMS Charges       10,593     Credit Transfer to         M3Tech   1,398 Participants Credit Transfers to           86 Community Volunteers Interactive Voice           1,202 Response Calls Monitoring Cost               Staff Fee             Nadeem Ahmed   2,457   2,457     Manzoor H. Memon   1,422   1,422     SPDC Reimbursable Expenses             Hotel, subsistence   1,970   1,970     Local transportation   2,463   2,463     Intercity transportation   985   985     Additional Staff Time             Staff/Consultants (HQ) World Bank 14,911 14,911 14,911 Staff/Consultants 5,550 12,347 5,550 (Country Office) Opportunity Cost for     51,235   22,564     Participants Total Variable Cost 101,976   111,632   24,294 Total Project Cost   129,058   138,368   49,185 98 ANNEXURE Annex 9: Data on Interactive Voice Response Calls Annex 3 provided details on the content of Interactive Voice Response calls. Chapters 7 and 8 provided the rationale for these external nudges and descriptive statistics on responses to these calls. Annex 9 brings these three parts of the report together to illustrate call responses in the form of tree diagrams. Interactive Voice Response Calls in Round 1: During the first round, four calls were made to village participants. The first call aimed to reach 4,427 participants, with 1,249 successfully responding to a question about meeting participation. Most participants had attended the first village meeting. Figure 22: Responses to Message 1 (IVR Calls - Round 1) Message 1 Total Number of Calls 4427 Response - No Response - Valid Response - Answer 1896 1249 1282 Yes No Don’t Know 801 171 277 The second call sought to reach 4,427 participants, with 1,370 successfully responding to a question about preferred language for text messages. Most participants preferred Sindhi as the language used for informational and summary text messages sent to them. Figure 23: Responses to Message 2 (Interactive Voice Response Calls - Round 1) Message 2 Total Number of Calls 4427 Response - No Response - Valid Response - Answer 1503 1370 1554 Sindhi Urdu 977 393 ANNEXURE 99 The third call reached out to 4,427 participants, with 1,255 successfully responding to a question about participants’ engagement in the Community Dialogue Platform. More than half of the participants had sent text messages to the portal. Of those who had not, a majority said it was because of a lack of credit balance. A significant portion was inactive on the portal because of mobile illiteracy. Of these, a majority found the village representatives helpful. Figure 24: Responses to Message 3 (Interactive Voice Response Calls - Round 1) Message 3 Total Number of Calls 4427 Response - No Response - Valid Response - Answer 1724 (Participation) 77 1255 Yes No 737 518 Message 3 If No - Reason for not participating 518 Response - Valid Response - Answer Response - No 464 54 Doesn’t Doesn’t No Credit know how want to Balance to use participate 243 Mobile 50 171 100 ANNEXURE Figure 25: Yes/No Responses to Message 3 (Interactive Voice Response Calls - Round 1) Message 3 Doesn’t know how to use Mobile 171 Response - Valid Response - Answer Response - No 94 77 Facilitators Facilitators were not were helpful helpful 39 55 The fourth and final call sought to reach 4,427 participants, with 1,124 successfully responding to a question about villagers’ interaction at the community level due to the Community Dialogue Platform project. Most participants had attended the first village meeting. A majority of respondents indicated they had had a conversation about the project with someone else in the village. Of these, a large per- centage had interacted with Executive Body members of the School Management Committee. Figure 26: Responses to Message 4 (Interactive Voice Response Calls - Round 1) Message 4 Total Number of Calls 4427 Response - Valid Response - Answer Response - No 1124 1488 Regulatory Teachers Family Body 709 474 491 ANNEXURE 101 Interactive Voice Response Calls in Round 2: During the second round, four calls went out to village participants and two calls to village representatives. Participants The first call sought to reach 4,427 participants, with 876 successfully responding to the question about changes brought about by the Community Development Platform project. Most participants believed that school facilities had undergone the biggest changes, followed by teacher absenteeism and village-level coordination in the education sector. Figure 27: Responses to Message 1 (Interactive Voice Response Calls - Round 2) Message 1 Total Number of Calls 4427 Response - No Response - Valid Response - Answer 1789 876 1762 School Teachers Village No Facilities Absence Coordination Improvement 389 197 91 199 The second call sought to reach 4,427 participants, with 996 successfully responding to a question about problems faced by participants during engagement with the portal. A large majority of respon- dents found it difficult to participate due to a lack of credit balance. However, a significant number also said they faced no problems during participation in the project. 102 ANNEXURE Figure 28: Responses to Message 2 (Interactive Voice Response Calls - Round 2) Message 2 Total Number of Calls 4427 Response - No Response - Valid Response - Answer 1871 996 1560 Unable to Message Don’t have Low Credit read/write Summary issues 612 msg 85 198 101 The third call sought to reach 4,427 participants, with 932 participants successfully responding to a question about the utility of summary text messages sent to them. Most participants found these messages to be beneficial. Figure 29: Responses to Message 3 (Interactive Voice Response Calls - Round 2) Message 3 Total Number of Calls 4427 Response - No Response - Valid Response - Answer 1625 932 1870 Beneficial Partial Not 499 Beneficial Beneficial 204 229 ANNEXURE 103 The fourth call sought to reach 4,427 participants, with 917 successfully responding to a question about changes brought about by the Community Development Platform project. Most participants had attend- ed the first village meeting. This was the same question that was posed during the first call of this round (Round 2). However, this time the responses were less positive. This could indicate a dip in the project’s activities at the tail-end of the intervention. A majority of respondents thought that the project brought about improvements in school facilities, reduction in teacher absenteeism and increased village coordina- tion. The remaining respondents indicated no improvements in the suggested categories. Figure 30: Responses to Message 4 (Interactive Voice Response Calls - Round 2) Message 4 Total Number of Calls 4427 Response - No Response - Valid Response - Answer 1904 917 1606 No Increase in Reduce teacher Village Improvement school facilities Absence Coordination 314 277 188 138 Worth noting here is that the success rate of the Interactive Voice Response calls (number of calls responded to successfully) dropped significantly when we moved from Round 1 to Round 2. This is illustrated by a drop of 30% and 18% for calls 1 and 4, respectively. This suggests that interest in the project might have started to lag at the tail-end of the intervention. Future efforts must be cognizant of this issue, and should aim to sustain the level of interest among participants, including, perhaps, by making the calls more user-friendly. Village Representatives Two calls were made to village representatives. Unlike the calls made to participants, these calls did not pose questions. Rather, they provided basic information on the responsibilities of, and incentives for, the representatives. The first call sought to reach 203 village representatives, with 136 representatives successfully respond- ing to the call. The second call sought to reach 203 village representatives, of which 140 successfully responded. 104 ANNEXURE Figure 31: Responses to Messages 1 & 2 (IVR Calls - Round 2 – Community Volunteers) Message 1 Total Calls 203 Response - No Response - Valid Response - Answer 67 136 136 Message 2 Total Calls 203 Response - No Response - Valid Response - Answer 63 140 140 Table 19 provides a consolidated overview of all relevant statistics related to Interactive Voice Response calls made during Rounds 1 and 2. Table 19: Detailed Descriptive IVR Statistics Avg # # of valid Avg #of # of # of calls % of calls #of valid responses valid calls answered answered answered/ responses for calls responses/ village answered village Round 1 Msg 1 4,427 1,282 29% 13 1,249 97% 13 Msg 2 4,427 1,554 35% 16 1,370 88% 14 Msg 3 4,427 1,448 33% 15 1,255 87% 13 Msg 4 4,427 1,448 34% 16 1,124 76% 12 Total Round 1 17,708 5,772 33% 60 4,998 87% 52 Round 2 Msg 1 4,427 7,624 40% 18 876 50% 9 Msg 2 4,427 1,560 35% 16 996 64% 10 Msg 3 4,427 1,870 42% 19 932 50% 10 Msg 4 4,427 1,606 36% 17 917 57% 10 Total Round 2 17,708 6,798 38% 71 3,721 55% 39 Overall 35,116 12,570 3.5% 131 8,719 69% 91 ANNEXURE 105 Annex 10: List of Databases Over the course of the project, multiple databases were compiled. The biggest was the dataset on information received on the Community Dialogue Platform, followed by those on attendance, par- ticipation and registration data, among others. Annex 10 lists each of these databases and provides brief descriptions on them. Table 20: Primary Project Data Sources Sr. # Database Description and Purpose Source Listed villages that were part of the treatment, Project design 1. Treatment Villages Dataset with details on district, tehsil, union council, documents treatment type and village name. All attendance sheets were appended and the Attendance Sheets Dataset consolidated data cleaned for analysis on total General 2. (Village Meetings) – Inter- number of participants, registration rates and the Body meeting vention 1 like. These attendance sheets were prepared for attendance sheets Intervention 1. All attendance sheets were appended and the Attendance Sheets Dataset consolidated data cleaned for analysis on total General 3. (Village Meetings) – Inter- number of participants, registration rates and the Body meeting vention 2 like. These attendance sheets were prepared for attendance sheets Intervention 2. A dataset was prepared containing number of Executive Attendance Sheets Dataset attendees in each of the three Executive Body 4. Body meeting (Executive Body Meetings) meetings for every School Management Committee attendance sheets in the crossover intervention. All School Improvement Plans finalized during School Dataset for School Improve- the third Executive Body meeting were digitized 5. Improvement ment Plans for analyzing plan cost estimates and key focus Plans areas for the School Management Committees. This dataset was provided by M3 Technologies and listed credit transfer dates and number of 6. Credit Transfer dataset M3 Technologies participants per village to whom credit was transferred. This raw database was an unfiltered collection of Community 7. Word Bank SMS database all text messages received on the portal. Dialogue Platform This database provided counts of text messages Portal Database – Unique Community 8. received for 10-day buckets (starting from Day 1 Messages Received Dialogue Platform to Day 120) for each of Intervention 2 villages. This dataset provided identifying and contact Community Volunteer 9. information on all community volunteers, as well as M3 Technologies Database data on their activity on the portal as volunteers. Contacts uploaded by M3 Technologies were pro- Uploaded Contact Informa- 10. vided in this dataset. It aided in triangulating the M3 Technologies tion Database count for the number of contacts registered. This dataset listed the number of participants who Active and Passive Users 11. were active on the Community Dialogue M3 Technologies Dataset Platform, and also listed “active villages.” This dataset listed schools that were selected to be part of the treatment in the intervention villages. School Level Randomization Project design 12. It provided details, such as district, tehsil, UC, Dataset documents GPS coordinates, address, head teacher contact information and treatment type. 106 ANNEXURE This database provided information on the Project Executive Body Schools schools for which Executive Board meetings were 13. implementation Participation Database held, as well as dates when these meetings were documents conducted. General Body meeting dates for all the treatment Project Meeting Dates and Details 14. villages were provided in this dataset. implementation Dataset documents This dataset was constructed using Interactive Interactive Voice Response Voice Response data provided by M3Tech fol- 15. M3 Technologies Calls Dataset lowing the two rounds of calls that were made to participants and community volunteers. The Keywords dataset aided in analyzing key Community 16. SMS Keywords Dataset areas of concern for participants through the Dialogue Platform construction of a word cloud. Information on funds available in School Manage- Project 17. Available Funds Dataset ment Committee bank accounts was taken from implementation this dataset. documents This dataset provided information on the total number of households in every village. The data 18. Household Census Dataset Pre-project census was used in measuring participation rates across treatment villages.