from EVIDENCE to POLICY Learning what works for better programs and policies January 2020 MEXICO: Can mobile tutors improve learning in remote schools? Achieving inclusive and quality education for all is a global pri- remote areas is a major challenge, and many low and middle- ority, and policymakers are still grappling with the best ways income countries rely on members of the community to teach to ensure the poorest and most marginalized children are in local schools. school and learning. About 120 million children –– many from In Mexico, the government runs a mobile tutor program rural areas –– are still out of school, despite a recent global to send recent university graduates to work in schools in the country’s most rural and marginalized communities, where teachers are typically members of the community with limited education and training. The World Bank’s Strategic Impact Evaluation Fund (SIEF) supported an evaluation to measure EDUCATION the impacts of the mobile tutoring program in the Chiapas re- gion, where a high fraction of the population is poor and indig- enous. The evaluation found that the program improved the rate at which students transitioned from primary to secondary school and parents’ reports of their children’s social-emotional skills. Test scores increased as well. A variant of the program that extended the amount of training that tutors received both before deployment and during their tenure demonstrated slightly higher improvements. These results convinced the gov- ernment, which had considered shutting down the program, to push for universal access to education. Even when they stay in instead continue sending mobile tutors to remote communities school, children from poor, rural areas often show the lowest and to adopt some of the changes that made the program more gains in learning. Recruiting and retaining teachers to work in effective. Context Mexico’s Chiapas region, where the study took place, is the graduates who do not have formal teaching qualifications and poorest region in the country, according to 2016 state statistics. receive only very basic training. They receive a monthly allow- Schools in marginalized communities are run by Consejo Na- ance and can eventually benefit from a scholarship for univer- cional para el Fomento de la Educacion, known as CONAFE, sity studies. a semi-autonomous government agency responsible for provid- Students in CONAFE schools generally perform worse ing education services in remote communities with fewer than than those in the regular system. In 2013, 13 percent of 2,500 people. CONAFE schools typically have a single multi- CONAFE students scored Good or Excellent in the national grade classroom, with an average of 10- 15 students taught by standardized examination, compared with 42 percent in the one instructor. The instructors are usually secondary school ministry-run schools. This brief summarizes the results of a 2018 Endline Report by Francesco Agostinelli, Ciro Avitabile, Matteo Bobba and Alonso Sanchez, “The short-term effects of the Mobile Pedagogical Tutors: Evidence from a Randomized Control Trial in rural Mexico.” In 2009, CONAFE started a mobile tutoring program aimed ents had complained about infrequent visits from the APIs, the at improving the quality of education in these remote and dis- researchers also provided the supervisors of the APIs an increase advantaged schools. The tutors or Asesores Pedagogicos Itiner- in salary in exchange for a mandatory increase in the number of antes (Mobile Pedagogical Advisors), known as APIs, are recent times they visited the communities. university graduates assigned to two schools. They are expected In another variant of the program, researchers increased the to spend about two weeks per month in each school providing training for APIs to two weeks from one week and organized one-on-one tutoring to the students who require the most help, bi-monthly peer meetings so tutors could discuss challenges making home visits to encourage parental involvement, and and learn what their peers were doing to solve problems. The providing pedagogical support to the local non-professional contents (and the costs) of the augmented training were de- teachers. But the API program, as structured, didn’t lead to im- signed jointly with CONAFE in order to guarantee scalability, provements in student learning, as assessed in an earlier evalua- in case the program proved to be effective. tion, and the progression rate from primary to lower secondary The interventions started in September 2014 and contin- school remained low. ued through a second school year ending in the spring of 2016. In 2014, as part of the World Bank project to support CONAFE schools, researchers introduced some changes to the Population at a glance… program as it was implemented nationally to improve the ef- • Communities in the Chiapas pilot were small and fectiveness of the tutors and set up a randomized control trial remote, with 100 inhabitants on average. to test the impact of the changed model on students’ learning • Sixteen percent had no road and 64 percent lacked a EDUCATION and grade progression. First, they used an API’s ability to speak paved road. • Eleven percent of communities had political conflicts the main indigenous language in the community as the most (primarily due to the presence of Zapatista rebels). important criterion for assigning APIs to communities. In the • Test scores were substantially lower in community previous model, 2/3 of tutors assigned to these communities schools than the national average. did not speak any indigenous language. Second, because par- Evaluation This study used a randomized controlled trial to test the im- Commission for the Evaluation of Social Policy. pacts of the strengthened mobile tutoring program and the Researchers conducted follow-up data collection in in the variant with augmented training. The evaluation took place in spring of 2016. To measure students’ reading and basic math a sample of 230 schools that had never received the mobile ability, two tests were administered: the Early Grade Reading tutoring program before. Of these 230 schools, 70 schools were Assessment (EGRA) and the Early Grade Math Assessment assigned to receive the APIs, 60 were assigned to receive the (EGMA), which had been adapted to the Mexican context. Al- APIs with augmented training, and 100 schools were assigned though these tests are meant for students in the early primary to the control group with no mobile tutoring at the time of grades, children in the community schools often lag behind the study. their urban counterparts by a couple of standard deviations, Researchers collected no baseline data and instead used and thus require an easier test. Surveyors interviewed 1,930 administrative data to assess balance among the experimental children in 3rd to 6th grade and caregivers in 1,050 households groups. Specifically, they used the 2013 results of a national about their expectations for their children’s education and their exam (the ENLACE), as well as information about school in- involvement in their children’s schooling. Caregivers were also puts from a census of schools carried out by the Secretariat of asked about their children’s behavior. Surveyors also did ob- Public Education and information about communities, includ- servations of classrooms to document teachers’ pedagogical ing poverty rates, from the national census and the National practices. Findings Mobile tutors helped improve students’ literacy at home, however, did appear to benefit from the program skills…. with augmented training, making gains of 0.192 standard deviations on the EGMA exam. Compared to the control group, students in the schools re- ceiving the strengthened mobile tutoring program scored an In communities where the tutors received the ex- average of 0.135 standard deviations higher on the EGRA tra training and the bimonthly meetings, children’s test, while students in schools receiving mobile tutors with social-emotional skills also improved relative to augmented training scored an average of 0.227 standard de- the control group, based on what their parents re- viations higher. Despite this large difference in the average ported. treatment effects, these impacts can- The tutors were supposed to help students and parents not be statistically identify the emotional issues that could affect their learning distinguished from outcomes and deal with them. Although the initial training each other. When didn’t have any specific content related to social-emotional the researchers issues, supervisors and coordinators encouraged tutors to investigated test use the peer meetings to also discuss how to deal with chil- scores further, they dren’s emotions. Parents of children in schools served by tu- found that these tors in the enhanced training program reported significant gains were primar- improvements in social-emotional wellbeing of their chil- ily in the areas that dren, while parents’ reports in the group that received the students were al- tutors without the augmented training could not be statis- ready doing well tically distinguished from reports from the control group. in, like reading fa- miliar words, dic- In schools served by mobile tutors who had re- tation, and reading ceived the augmented training, primary school stu- comprehension, as opposed to the areas where students were dents were more likely to continue to secondary initially weakest, like the recognition of initial sounds and school. letter sounds. The progression rate between primary and secondary school …but math skills didn’t improve as much. is low in CONAFE schools, with about 60 percent of pri- mary school students continuing to secondary school. But The math performance of students with mobile tutors could in communities where the mobile tutors went through the not be statistically distinguished from scores of their peers more intensive training and support program, the rate rose in control schools. Students who had the mobile tutors by 14 percentage points over the control group. Children with augmented training had higher scores than students who received the program without the augmented training in the control group, though this gain was only marginally saw a smaller increase that was not statistically significant. statistically significant. A breakdown of scores by topic area As with the findings for math, these gains in the transition suggests that statistically significant gains only came in the to primary school were concentrated among students who most basic task – number recognition – with limited to did not speak an indigenous language at home, suggesting no gains in more difficult tasks like addition and subtrac- that the program may have benefited children who were not tion. Children who did not speak an indigenous language the most at risk for dropping out before secondary school. The positive changes may have stemmed from im- bile tutors with the standard training could not be distin- provements in teachers’ pedagogy and greater pa- guished from their peers in the control group. Similarly, rental involvement. an index capturing the extent to which parents invested in their children’s education (as proxied by their aspirations to Surveyors observed teachers in their classrooms and docu- see their children graduate from high school, their helping mented their ability to adjust to students’ learning speed, with homework, and meeting with teachers, among oth- how much time they spent on teaching activities, and how er activities) significantly improved only for schools that engaged their students were. The teachers in schools that received tutors with the augmented training. The study received mobile tutors with augmented training received didn’t find any changes in how much kids studied or how significantly higher classroom practice scores than teach- often they attended school. ers in the control group, while teachers who received mo- Conclusion Ensuring that children in remote regions have access to investment in education. high quality services is a global challenge. This evaluation Because both interventions tested during the pilot were EDUCATION provides evidence on the impact of a mobile tutor program designed within the restrictions of resources available to implemented in highly marginalized communities in rural the government agency responsible for remote community Mexico, and the results suggest that mobile tutors can be ef- schools, the tested models were also scalable. As a result, fective in improving learning and children’s transitions from the government of Mexico scaled up the program variant primary to secondary school, particularly when the tutors with the augmented training for the mobile tutors in the receive adequate training and opportunities to troubleshoot fall of 2017, a promising step toward creating tangible im- issues with their peers. The tutors helped teachers improve provements in learning for disadvantaged children in rural their classroom practices and encouraged more parental Mexico. The Strategic Impact Evaluation Fund, part of the World Bank Group, supports and disseminates research evaluating the impact of development projects to help alleviate poverty. The goal is to collect and build empirical evidence that can help governments and development organizations design and implement the most appropriate and effective policies for better educational, health, and job opportunities for people in low and middle income countries. For more information about who we are and what we do, go to: http://www.worldbank.org/sief. The Evidence to Policy note series is produced by SIEF with generous support from the British government’s Department for International Development and the London-based Children’s Investment Fund Foundation (CIFF). THE WORLD BANK, STRATEGIC IMPACT EVALUATION FUND 1818 H STREET, NW, WASHINGTON, DC 20433