Policy Research Working Paper 11146 Accelerating Learning in Ecuador’s Technical Institutes The Impact of Using Mixed Reality to Teach Auto-Mechanics Diego F. Angel-Urdinola Marjorie Chinen Education Global Department June 2025 Policy Research Working Paper 11146 Abstract This study evaluates the impact of incorporating mixed real- usability, motivation, and engagement. This comprehen- ity—including both augmented and virtual reality—into sive approach enabled the study to quantify the effects of auto-mechanics training for students enrolled in selected the training on student learning and identify mechanisms public technical technological institutes in Ecuador. The through which learning occurred. The results indicate that intervention aims to enhance students’ understanding of students exposed to mixed reality–based instruction scored, automotive mechanics by teaching the fundamental princi- on average, 0.37 standard deviation higher on post-tests ples of internal combustion engine operation through nine than those in the control group—a statistically significant competency-based learning modules delivered over one aca- effect at the 1 percent level. These findings are consistent demic semester. The study employed a stratified randomized with effect sizes observed in training programs aimed at controlled trial at the class level, assigning auto-mechanic college students in high-income countries. The evidence classes within each institute to either the mixed reality– also points to increased engagement and motivation as key enhanced training laboratory (treatment) or the standard channels through which mixed reality enhanced learning, curriculum (control). To measure learning outcomes, stu- underscoring the potential of immersive technologies to dents completed pre- and post-intervention cognitive tests, improve vocational training outcomes in low- and mid- complemented by student surveys assessing technology dle-income settings. This paper is a product of the Education Global Department. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at dangelurdinola@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Accelerating Learning in Ecuador's Technical Institutes: The Impact of Using Mixed Reality to Teach Auto-Mechanics Diego F. Angel-Urdinola and Marjorie Chinen 1 JEL Classification: I20, I21, I23, J24 Keywords: Mixed Reality, Virtual Reality, Augmented Reality, Education, EdTech, Learning, Skills Development, Virtual Laboratories 1 The authors would like to express their sincere gratitude to all the teachers who participated in the various phases of the study, as well as to the principals of the technical institutes for their invaluable collaboration. Special thanks to Robert Dean, Jimmy Vainstein, Javier Taipe, Abraham Moran, Catalina Castillo for their support in the design and implementation of this program and for ensuring the success of the project. The team also acknowledges the contributions of students and staff of Namseoul University in the process of curricula and software development, as well as the World Bank’s Global Corporate Solutions (GCS) for their support with laboratory design, procurement, and deployment. This study was financed by a grant from the Korea-World Bank Partnership Facility. I. Introduction Education technology has the potential to transform the delivery of technical education globally (World Bank, 2021). Many developing countries are actively seeking to expand and modernize their technical education and training systems, which are vital for developing a skilled workforce, enhancing employment opportunities, and driving economic growth (Dill & Van Vught, 2010; Levin et al., 2023). A common challenge for the effective delivery of technical education in developing countries is the limited availability of hands-on pedagogical experiences necessary to develop practical skills—particularly in programs that require the use of laboratories. In this context, a critical question arises: what policy alternatives can provide an affordable and effective mechanism to equip students with practical training? With the rise of education technology, and in response to the inherent challenges of setting up laboratories for technical education (cost, maintenance, update), educators worldwide are exploring the possibility of accelerating the development of virtual laboratories as pedagogical support for imparting content in selected technical programs. A virtual laboratory is an interactive computer simulation of a laboratory. The purpose of virtual laboratories is to give students experiential interaction that will lead to learning. While virtual laboratories offer many advantages, it is essential to systematically assess their effectiveness to ensure they adequately contribute to student learning and skills development. This study presents findings from an experimental evaluation conducted in selected technical institutes within Ecuador's public technical education system. The evaluation assesses the extent to which a virtual laboratory that uses mixed reality (virtual and augmented realities) can contribute to improving student learning in auto-mechanics. The findings aim to provide valuable insights into the effectiveness of virtual labs in enhancing technical skills and knowledge among students in this field. This auto-mechanics virtual laboratory is the first of its kind in Ecuador’s network of State Technical and Technological Institutes (TTIs). This virtual lab seeks to improve student learning in the domain of auto-mechanics of students who are being introduced to the basic principles of the operation of internal combustion engines for the first time. The curriculum and software for this virtual laboratory were developed by a multidisciplinary group of professors, education and multimedia experts and programmers from the Cotopaxi TTI in Ecuador, the World Bank, and Namseoul University, a leader in VR development, research, and education in the Republic of Korea. The results of this evaluation aim to broaden the limited evidence on the use of EdTech, particularly virtual laboratories for 2 technical education, in developing countries. They also seek to assess the lessons learned from implementing EdTech in systems that are still developing their capacity and to inform the ongoing higher education policy in Ecuador. The paper is organized into six sections. Section II reviews existing literature, examining how virtual labs and immersive training contribute to skill development in students. Section III describes the intervention. Section IV outlines the evaluation framework, detailing the methodology and instruments used. Section V presents the evaluation results, highlighting key findings. Finally, Section VI concludes by summarizing the study's insights and implications. II. Literature Review In some educational fields, the development of student skills remains a challenge for trainees and their tutors, partly because of the limited availability of hands-on training or access to proper content and learning situations. As a response, educators are starting to rely on Virtual Laboratories, many of which use Mixed Reality (Augmented and Virtual Reality), to develop learning experiences that would otherwise not be easily accessible to students (Johnson et al., 2016). 2 Virtual laboratories offer students the chance to engage in practical training in a safe and pressure-free setting, with the added benefit of allowing for repeated practice. Training simulators vary in their functionalities and features, with some advanced virtual labs offering a completely immersive experience for users. With the advent of new technologies, modern virtual laboratories now offer high-fidelity imaging and sensing capabilities that facilitate realistic manipulation of physical tools and provide real-time feedback (McLaurin & Stone, 2012). Additionally, virtual laboratories can grant students access to scenarios and learning environments that would be challenging or impossible to experience otherwise. These opportunities can significantly enhance students' learning curves by simulating real-life conditions and situations without the constraints of time or space, and with considerably reduced risks. Moreover, by developing certain skills in a virtual setting, it is possible to reduce the costs associated with materials, time, and expert availability that are typically required for traditional training methods (Angel-Urdinola et al., 2019). Virtual laboratories have been 2 The term Virtual Reality (VR) applies to computer-simulated environments that can imitate physical presence in places in the real world, as well as in imaginary worlds, and simulate the illusion of participation in a synthetic environment with an external observation of such surroundings. VR simulations can be constructed employing 3D graphics using a desktop computer (non-immersive) or using a head-mounted display (immersive). In non-immersive VR, the simulated environment is displayed on a conventional computer with sound and graphics coming through the computer’s speaker and monitor, and the interaction is controlled through a regular computer mouse. Augmented reality (AR) is an interactive experience that combines the real world and computer-generated content. 3 widely applied in technical education fields such as aviation, design, auto-mechanics, industrial risk prevention, and robotics (Angel-Urdinola et al., 2019; Buiu & Gansari, 2014; Wei et al., 2013). Active learning and constructivism are frequently recognized as key mechanisms that enhance the educational benefits of using simulated environments. Constructivism posits that students acquire knowledge by constructing their own understanding and integrating it with their pre-existing knowledge base. This approach enhances student engagement, motivation, and personalized learning experiences (Madathil et al., 2017). Advocates of virtual laboratories contend that immersive simulations can significantly boost motivation and engagement, which are critical factors influencing student learning (Dafli et al., 2017; Klopfer & Squire, 2008; Mikropoulos & Natsis, 2011). According to Krokos et al. (2019), students tend to retain more information and are better able to apply their knowledge after engaging in immersive learning experiences. In a recent study, Makransky and Peterson (2021) introduced the Cognitive Affective Model for Immersive Learning (CAMIL) to describe the learning process in immersive 3D environments. The authors concluded that immersive technologies positively impact knowledge acquisition. Hamilton et al. (2020) showed that the skills acquired through Mixed Reality technology can better prepare trainees to tackle real-world challenges and scenarios. Some of the existing literature suggests that virtual laboratories can effectively enhance students' technical skills in education. For instance, Román-Ibáñez et al. (2018) assessed the extent to which virtual laboratories can enhance technical skills acquisition. Their study demonstrates that an immersive simulator for industrial robotic arms can offer engineering students a more realistic experience, effectively reducing robotic design and performance issues like stuttering (jerky movements) and lag (response delays). Similarly, Lampi (2013) found that employing virtual laboratories for teaching computer network support can enhance students' accuracy and speed in network configuration compared to conventional training methods. Zhou et al. (2018) implemented an educational application for computer assembly using Mixed Reality. The authors found that the application provided excellent usability and user experience. While there was no observed difference in performance between learners using the application and those using traditional methods, there was a significant decrease in task completion time for learners using the application. Rupasinghe et al. (2011) found that students who trained in virtual laboratories developed better technical skills in using a borescope to assess aircraft corrosion. A recent meta-analysis by Howard et al. (2021), which evaluated the effects of virtual reality training programs across more than 185 experimental 4 studies, found that virtual reality training programs are generally more effective than their comparison groups. On average, the mean performance in VR training was 0.541 standard deviation higher than that of the comparison groups. Specifically, for programs assessing learning outcomes, the treatment group outperformed the control group by an average of 0.491 standard deviation. For programs specifically targeting college students, the effect size was 0.432 standard deviation. Finally, several authors suggest that training using MR can be cost-effective. Dela Cruz and Mendoza (2018) examined the cost implications of using a virtual lab to teach students about pneumatic systems and concluded that virtual labs offer a cost-effective alternative to traditional setups. Simulations reduce expenses related to equipment maintenance and repair. Similarly, Stone, Watts, Zhong, and Wei (2011), along with McLaurin and Stone (2012), found that virtual laboratories can develop welding skills as effectively as traditional labs, but at a lower cost. However, most of the existing evidence on the effectiveness of virtual laboratories in promoting learning and skill development originates from high-income countries. Furthermore, many of these studies rely on small sample sizes—often involving fewer than 50 participants— and are concentrated in a narrow set of fields, such as health care, vehicle operations, and welding (Angel-Urdinola et al., 2019). This study addresses key limitations in the literature by implementing a large-scale randomized controlled trial—larger than those in most prior studies—in a developing country context. It makes a significant contribution by providing evidence on the effectiveness of immersive technologies in vocational education and offering policy-relevant insights for resource-constrained settings. III. Description of the Intervention Ecuador’s technical and technological education system includes 58 public TTIs and three conservatories nationwide. The system offers a total of 172 academic programs in diverse fields such as automotive mechanics, electronics, logistics, accounting, early childhood development, gastronomy, business administration, floriculture, hotel trading and tourism, visual arts, and marketing, among others. At the time of the intervention, enrollment in state TTIs reached 45,000 students, of which 80 percent were enrolled in the presence-based modality, 8,000 students (17.8 percent) in dual programs, and 2.2 percent (1,000 students) in the hybrid/online modality. The system of TTIs employs 6,958 teachers, of which 56 percent 5 work full-time. About 60 percent of all teachers have an undergraduate degree, 32 percent have a graduate degree, and 6.7 percent hold a vocational one. Within the system, there are a total of 14 public TTIs that are considered centers of excellence as they have their own infrastructure and recently benefited from investments in workshops, connectivity, and laboratories. Compared to the remaining public TTIs, these 14 institutes have strong management, teaching staff, and IT capacity. Through a competitive call for proposals, these elite institutes were invited to submit applications to implement the Auto- Mechanics Mixed Reality Laboratory within their institutions. Applicants were required to demonstrate how the laboratory would be integrated into their academic offerings, provide adequate space for its installation, and show commitment from both management and teaching staff to utilize the laboratory in accordance with its implementation standards and pedagogy model. Selected TTIs would also agree to comply with the research protocols designed to assess the impact of the intervention. Of the 14 TTIs, 12 submitted proposals to participate in the program. Following a rigorous review process, five TTIs were selected to participate in the intervention: Cotopaxi, Loja, Tungurahua, Luis Arboleda Martinez, and Luis Rogerio Gonzalez. Each selected TTI received laboratory equipment (hardware and software to accommodate up to 35 students), teacher training, laboratory installation, furniture, maintenance and technical support for five years, as well as technical assistance to carry out the program’s impact evaluation. Prior to this selection process, the laboratory was tested and adjusted over two consecutive academic semesters at the Cotopaxi TTI. This phase allowed the research team to test the laboratory setup, make necessary adjustments, and establish implementation protocols to ensure it was used as intended and properly integrated into the academic curriculum and lesson plans. A more detailed account of the laboratory’s piloting, calibration, and adjustment process is documented in Angel-Urdinola et al. (2023). 3.1 Intervention The intervention employed a laboratory equipped with Mixed Reality technology to deliver specialized training for students enrolled in two courses: Internal Combustion Engines and Maintenance and Repair of Engines. The training program was implemented across the five beneficiary TTIs from May 2023 to February 2024. The first cohort of students participated from May to August 2023, while the second cohort began in October 2023 and completed the program in February 2024. 6 The laboratory provides students with an immersive learning environment through a desktop system called zSpace, which enables 3D interaction with engine components. Using 3D glasses and a stylus pointer (held-like pen), learners can naturally move their heads and hands to pick up, rotate, and explore virtual objects with full 360-degree freedom movement. The zSpace software also includes three operational modes within each learning module: a learning mode that offers guided lessons, a practice mode for independent exploration without instructional scaffolding, and an assessment mode that allows students to self-evaluate their understanding. Each workstation is also equipped with a tablet that allows the student to access 2D Augmented Reality (AR) content via QR codes markets printed in an accompanying guide available in every workstation. Each tablet includes two applications: the first (touch screen) identifies engine parts and their functions, and the second (augmented reality) presents students with problem-solving activities that promote collaborative learning. These experiences provoke discussion among students by presenting problems and offering immediate feedback and explanations to guide learning. In addition, a second tablet at each station features a touch- based flash card application designed to help students learn the names, functions and uses of engine parts and tools. Working in pairs, students quiz each other to master the target content. Text can be displayed and hidden easily to facilitate collaborative study. Each laboratory includes six work workstations, designed to accommodate groups of up to five students, promoting both independent and collaborative learning. Each group rotates roles every 15–20 minutes—one or two students work independently on the zSpace system, while the others use the tablets in pairs for AR and flash card activities. This setup fosters engagement, peer interaction, and teamwork, enhancing the overall learning experience. The overall training course consists of a nine-module curriculum, with each module designed to be delivered in a 90-minute class. The objectives of Module 1, Engine Overview, are to (i) identify the major components of an internal combustion engine and (ii) describe how the internal combustion process works. Modules 2 through 7 build foundational knowledge of engine components, including the functioning of intake systems, valve-trains, crankshaft and flywheel, fuel and injection systems, and pistons and connecting rods. The objectives of these modules are to (i) identify the parts in each component assembly, (ii) describe the function of each component, and (iii) describe the operation of each component. Module 8 focuses on Disassembly; by the end of the module, students are expected to (i) describe the steps to disassemble an internal combustion engine and (ii) identify the appropriate tools used in the 7 process. Module 9 covers Assembly and aims to (i) describe the steps taken to assemble an internal combustion engine and (ii) identify the appropriate tools used in the process. 3.2 Pedagogical Approach The traditional class rollout follows the approved course curriculum for the Auto- mechanics Program. Technical institutes revise this curriculum every three years and submit it to the Higher Education Council for approval. The curriculum consists of three components: (i) in-class instruction (54 hours per semester), (ii) hands-on practice with engines in the laboratory (42 hours), and (iii) independent group work (48 hours) to reinforce class assignments, practice with available equipment, etc. The Mixed Reality laboratory curriculum is fully embedded within the “in-class” instruction component of the course. The pedagogical approach underlying the curriculum is based on blended learning models, combining Mixed Reality simulations with learner-centered, collaborative activities to actively engage students in the learning process. Rather than delivering direct instruction, teachers act as facilitators, personalizing learning based on student performance, preferences and goals. As the responsibility of the learning process shifts to the students, higher-order skills—such as complex problem solving, social skills (e.g., coordinating with others), process skills (e.g., critical thinking, self-monitoring), systems skills (e.g., analysis, judgement, decision making) and cognitive abilities (e.g., problem solving, creativity)—are reinforced. This shift in pedagogy involves a redefinition of the teacher’s role: instead of being the primary source of information, teachers guide students to acquire the information provided by the laboratory resources, encourage the proper use of the laboratory, foster discussion and analysis, and provide clarification when needed. Finally, the curriculum's main objective is to improve students' general understanding of basic motor operations. The intervention did not involve curricular changes. All content in the Mixed Reality curriculum is a subset of the original in- class instruction curriculum. Each module includes approximately 90 minutes of instruction. Prior to the beginning of the training, students received 60 minutes of orientation to familiarize themselves with the technology. IV. Methodology 4.1 Theory of Change and Randomization Based on the available literature, we identify two main channels through which the use of the auto-mechanics MR Laboratory could influence student learning outcomes: First, the immersive features of the laboratory environment may enhance student engagement by creating 8 interactive, visually rich learning experiences that capture attention and sustain interest. Second, these features may also increase students’ intrinsic motivation to learn by making the content more accessible, relevant, and enjoyable. Together, these channels are expected to strengthen cognitive engagement and deepen learning (Mikropoulos & Natsis, 2011; Kavanagh et al., 2017; Madathil et al., 2017) (Figure 1). Randomization was conducted at the class level using a stratified design, ensuring that within each technical and technological institute, all classes focused on auto-mechanics were assigned either to use the Mixed-Reality laboratory (treatment group) or to continue with the standard curriculum (control or business-as-usual group). To maintain the integrity of the randomization, control group classes were explicitly instructed not to use the MR laboratory. Of the 20 class groups spanning two courses and two semesters, 3 11 were randomly assigned to use the MR laboratory, comprising a total of 223 students. The remaining 9 class groups, with 193 students, served as the control group, following the standard business-as-usual instructional approach. Figure 2 illustrates the data collection timeline for each of the two cohorts. Although spillover effects are possible due to within-institution randomization, we argue that these are low for several reasons. First, the intervention’s impact is primarily dependent on direct access to and use of the lab facilities, with control teachers instructed to avoid using the lab entirely. Second, the pedagogical features of the intervention were specifically designed to be effective only within the lab’s context. Third, academic courses are offered in both day and night shifts, and interactions between students in these two shifts are limited. However, as with any within-school design, some degree of spillover cannot be entirely ruled out. Control teachers may have used the lab despite recommendations or adapted elements of the intervention’s pedagogical approach in ways that enhanced learning outcomes. As such, the estimates presented in this study likely represent a lower-bound effect of the Mixed-Reality laboratory’s potential to improve student learning outcomes. 4.2 Outcome Variable The primary outcome of the study is student learning, as measured by performance on standardized tests. The instrument used to assess student learning was an academic test designed to evaluate cognitive outcome in the domain of auto-mechanics. The test covered content from the 9 training modules, aligned with the course curriculum. Questions were 3 The sample includes students from two distinct semesters (cohorts). In the first semester (Cohort 1), 197 students participated in the study. In the second semester (Cohort 2), 219 students took part. 9 developed to address three levels of Bloom’s Taxonomy: remembering (or recalling or recognizing facts, terms, basic concepts), understanding (testing ability to understand and interpreting information), and applying (capturing testers ability to apply learned information in new scenarios). The test comprised 48 multiple-choice items 4 (Table 1). Since the cognitive outcome was developed specifically for this study and used for the first time in Ecuador, we conducted a pilot study with an independent contractor specializing in the design and implementation of psychometric tests to assess academic knowledge proficiency in post-secondary education (see Angel-Urdinola et al., 2023 for more details). The exam questions were developed by the independent contractor in close coordination with auto- mechanics curricula experts from Namseoul University and two Ecuadorian subject matter experts in motor mechanics. The final version of the test satisfied all criteria for internal consistency and reliability to ensure its suitability for the study. 5 The assessment was administered by the independent contractor via an online platform, with on-site support from an exam proctoring specialist. It was conducted at two points in time: at the beginning of the semester (baseline) and again at the end of the semester (endline), after students had completed all nine modules and before the start of final exam weeks at the institutions. Each student was given a unique, personalized link to access the test. The firm also graded the responses. Finally, the independent contractor computed the outcome variable using a two-parameter Item Response Theory model, accounting for both the difficulty and discrimination of the questions across different levels of student ability. The resulting score has a mean of 600 and a standard deviation of 150 points. The research team subsequently standardized the score relative to the mean and standard deviation of the control group. 4.3 Baseline Covariates and Balance Results After the cognitive assessment, a brief survey was administered to collect basic demographic information (e.g., gender, age 6), as well as academic and professional details (e.g., whether the student has repeated any subjects or completed a pre-professional internship). The survey also gathered easy-to-collect proxies for socioeconomic status, including the 4 The 48 items were distributed across the different levels of Bloom's Taxonomy as follows: 22 items for remembering, 15 for understanding, 8 for application, and 3 for analysis. 5 A copy of the cognitive test can be provided upon request. Please contact the authors for more information. 6 Since females represent only 3 percent of the sample, the variable is not included in the balance and impact analysis. 10 student's employment status (part-time, full-time, or only studying), maternal education level, and receipt of cash transfers (Bono Solidario). 7 Table 2 presents the student-level characteristics. The sample is well-balanced between the treatment and control groups, with an average student age of 21 years. Approximately 40 percent of students in both the treatment and control groups combine study and work, while 34 percent have complemented their academic training with a practical internship opportunity. About 13.5 percent of participant in both groups have repeated at least one academic subject during their studies, 15 percent receive the Bono Solidario cash transfer (a proxy for economic vulnerability), and 46 percent have mothers whose highest level of education is primary school or less. Additionally, about 35 percent attend the morning shift, while the majority attend the night or evening shift as most students in TTI work and study simultaneously. None of the baseline characteristics shows statistically significant differences between the treatment and control groups. It is also worth noting that the sample primarily consisted of male students (96 percent). Consequently, gender is not included in the subsequent analysis, as the field of auto- mechanics in Ecuador remains predominantly male. 4.4 Mechanisms of Change In line with the theory of change, the study gathers indicators to explore the potential mechanisms through which the intervention may have enhanced learning outcomes. Student motivation was evaluated using an adapted version of the Science Motivation Questionnaire (Glynn, Brickman, Armstrong, & Taasoobshirazi, 2011), tailored to focus on auto-mechanics learning. This 15-item instrument assesses three critical components of motivation: intrinsic motivation, self-efficacy, and self-determination, each rated on a five-point Likert scale. Engagement was measured using a modified version of the Student Engagement Questionnaire developed by Kember & Leung (2009), which assesses students' perceptions of the teaching and learning environment. The adapted engagement tool comprised a 16-item scale, rated on a five-point Likert scale. Both scales were constructed by summing the items and calculating the percentage change. Table 2 presents the average scores from the two baseline assessments, while Annexes 1 and 2 provide the full set of questions used in the motivation and student engagement questionnaires. 7 Students' employment status can positively affect learning if the work they are engaged in pertains to auto- mechanics. However, it could also affect learning negatively if students who work have time constraints for independent study. 11 4.5 Usability Additionally, for the treatment group at endline, we gathered information to evaluate the usability of the laboratory’s hardware (tablets and z-space desktop). We used an adapted version of the System Usability Scale (Brooke, 1996), which included 9 items to assess the usability of the z-space and another 9 items for the tablets. The 1–5 scale used for evaluation is presented in the Annex 3. All scales in the study were constructed by summing the item values, with negatively worded items reversed (to ensure higher scores are associated with better usability). 4.6 Per-Student Cost The rollout of the laboratory required investments in various areas, including curriculum design, software development, hardware acquisition, software licensing, a four- year pre-paid technical support plan, laboratory adaptation, teacher training, and other related expenses to enhance educational infrastructure. The total cost of the intervention was estimated at US$769,000 (refer to Annex 4 for detailed breakdown). On average, each participating Technical and Technological Institution would offer two to three classes per semester, with approximately 60 students enrolled per term. Assuming a five-year utilization period—aligned with the duration of the technical support contract— the intervention is expected to benefit at least 3,750 students. Consequently, the intervention incurred a cost of approximately US$205 per student. It is important to note that the cost of equipping a virtual laboratory—estimated at US$80,000 per lab for hardware and software licenses—is comparable to that of setting up a traditional laboratory with physical training engines. 8 However, virtual labs offer distinct advantage, including the ability to update software as technology evolves, and to support a broader range of technical training programs, such as welding. This flexibility allows the laboratory infrastructure to be repurposed for multiple vocational courses, potentially reducing overall costs and improving long-term sustainability. 4.7 Empirical Strategy We estimate the average impact of being eligible to use the Mixed Reality laboratory— referred to as the Intention-to-Treat (ITT) effect—among students enrolled in higher education auto-mechanics courses, using the following model: 8 A single training engine typically ranges between USD 10,000 and USD 15,000. A laboratory with capacity for 35 students would require five training engines, bringing the total cost to approximately USD 50,000 to USD 75,000. 12 = 0 + 1 + +2 ℎ + ′ + ′ + (1) where denotes the cognitive outcome for student i in class s, is an indicator variable for whether class s was randomly assigned to use the Mixes Reality Laboratory for first- semester students; Cohort is a dummy variable indicating whether the student is from cohort 2 (i.e., the second semester of implementation), as compared to Cohort 1 (the first semester). represents the stratification dummies accounting for differences in student characteristics across technological institutes; controls for baseline characteristics of student i, including the pretest cognitive score, 9 age, household receipt of social assistance (i.e., Bono Solidario), student employment status, repetition of subjects, and prior completion of a pre-professional internship. These are included to improve efficiency and to correct any baseline imbalances. is the residual term. Standard errors are clustered at the class level, representing the treatment unit. The main parameter of interest is 1 . We estimate the ITT of the Mixed Reality Laboratory on student learning (cognitive outcome). Attrition from the beginning to the end of the semester was low, at only 8 percent. This minimal attrition was comparable between the treatment group (10 percent) and the control group (7 percent). Consequently, the impact results are based on the listwise sample of 381 students. V. Results 5.1 Main Impacts We start by assessing the average impact of eligibility to use the Mixed Reality Lab on the cognitive outcome of interest. The main results are shown in Table 3. Column 1 reports findings from a specification that controls only for institution and cohort fixed effects. Column 2 presents results for the baseline specification, which additionally controls for the pretest cognitive score and other baseline characteristics as previously outlined. On average, students in the treatment group scored 0.37 standard deviation higher than those in the control group, with statistical significance at the 1 percent level. 9 Thus, the model estimates the treatment effect using an Analysis of Covariance (ANCOVA), which improves statistical power by controlling for baseline outcome (McKenzie, 2012). 13 5.2 Mechanisms of Change The theory of change suggests two mechanisms through which the use of the Mixed Reality laboratory could enhance learning outcomes. First, by improving student motivation, the laboratories may make learning more interactive and personalized, offering self-paced learning and immediate feedback—factors that collectively make education more engaging and enjoyable. In addition to increasing motivation, these aspects may also boost student engagement. Data collected through the online survey supports these hypotheses. Results presented in columns 1 and 2 of Tables 4 and 5 indicate that students in the treatment group are, on average, more motivated and engaged. These findings imply that the learning experience facilitated by the Mixed Reality laboratory helped motivate and engage students. 5.3 Usability Results from the usability survey conducted at the end of the semester indicate that over 80 percent of students in the treatment group found both the zSpace technology and laboratory tablets easy to use. They also reported that the functions were well integrated, they quickly learned to use the technology, and they felt confident using it. However, a notable proportion of students (approximately 27 percent for zSpace and 34 percent for tablets) found these devices uncomfortable to use (Figures 3 and 4). VI. Conclusions Educators are increasingly utilizing Virtual Laboratories to create learning experiences that would otherwise be challenging or costly for students to access. These laboratories provide students with opportunities for practical training without the associated pressure or danger, and they allow for repeated practice. Virtual laboratories that incorporate immersive simulations can significantly enhance motivation and engagement, which are crucial factors influencing student learning. Although these laboratories have shown promising potential to improve learning and skills development in high-income countries and on a small scale, evidence of their effectiveness in developing countries and on a larger scale remains limited. This study evaluates the impact of incorporating Mixed Reality (Augmented and Virtual Reality) to deliver training in auto-mechanics to two cohorts of students enrolled in the Technology in Automotive Mechanics Program at selected Public Technical and Technological Institutes in Ecuador, using a randomized controlled trial experiment. This virtual laboratory is the first of its kind within Ecuador's State Technical and Technological Institutes network. The training program aims to enhance students' understanding of auto-mechanics after being 14 introduced to the basic principles of internal combustion engine operation. The curriculum comprises nine competency-based learning modules delivered over one academic semester. To measure the training's impact on learning outcomes, students completed a cognitive test both before and after participating in the course. Results show that, on average, students in the treatment group scored 0.37 standard deviation higher than those in the control group—a statistically significant difference at the 1 percent level. This effect size is consistent with findings from training programs targeting college students in high-income countries, which typically report gains of approximately 0.43 standard deviation. Increased student engagement and motivation associated with the use of the Mixed Reality Laboratory are identified as key drivers of the observed learning improvements. While these results are promising, they should be interpreted with caution. Effective use of virtual laboratories requires training institutions to meet minimum thresholds in management, instructional, and IT capacity. Given these criteria, only 14 public Technical and Technological Institutes were deemed sufficiently prepared to implement this technology effectively. As a result, the external validity of the findings depends heavily on the capacity and readiness of institutions to adopt and sustain educational technologies and integrate them meaningfully into curricula and instructional plans. Such conditions are not always present in developing countries and require investment in institutional capacity building. Moreover, the design and implementation of virtual labs require upfront investment. Nonetheless, they may represent a cost-effective solution over time. Virtual labs enable students to interact with high-cost equipment without the need for physical purchases, thereby reducing expenditures on consumables and minimizing wear and tear. They also support frequent software updates as technology evolves, ensuring that students are trained using up- to-date tools and techniques. In many cases, the same virtual lab hardware can be repurposed across multiple courses by simply installing software packages, thereby maximizing the return on the investment. As the cost of equipment increases or the risk associated with its use by novices rises—such as heavy machinery, vehicles, or medical equipment—simulation-based training becomes an increasingly attractive and cost-effective alternative. Additionally, the successful implementation of virtual laboratories requires providing technical and pedagogical support to teachers. The research team conducting this study dedicated significant time to training teachers on effectively using the technology and integrating the virtual laboratory content into the course curriculum and lesson plans. Before the impact evaluation rollout, the research team piloted the virtual laboratory training for two 15 semesters, making several adjustments that facilitated the adequate scale-up and use of the laboratory, as documented in Angel-Urdinola et al., 2023. Furthermore, results indicate that not all students feel comfortable using the hardware and software of the virtual laboratory, suggesting that teachers must provide ongoing support and enhance the digital skills of students who use EdTech tools for education. 16 References Angel-Urdinola, D., Castillo-Castro, C., & Hoyos, A. (2021). Meta-Analysis Assessing the Effects of Virtual Reality Training on Student Learning and Skills Development. Policy Research Working Paper; No. 9587. World Bank, Washington, DC. Angel-Urdinola, D., Castillo-Castro, C., Chinen, M., Dean, R., Taipe-Yugcha, L.J. (2023) Value Added Evaluation for a Laboratory that Uses Augmented and Virtual Reality to Improve Student Learning in Auto-mechanics. Washington, D.C. World Bank Group. http://documents.worldbank.org/curated/en/099000002142331490 Bloom, B. S., & Krathwohl, D. R. (1956). Taxonomy of educational objectives; the classification of educational goals by a committee of college and university examiners. Handbook I: Cognitive Domain. New York, NY; Longmans, Green. Brooke, J. (1986). SUS—A Quick and Dirty Usability Scale. Usability Evaluation in Industry, 189-194. Buiu, C., & Gânsari, M. (2014). Designing robotic avatars in Second Life - A tool to complement robotics education. In IEEE Global Engineering Education Conference, EDUCON, 1016–1018. IEEE Computer Society. Dafli, A., Arfaras, G., Bamidis, P.D. (2016). Versatile Mixed Reality Educational Spaces - A Medical Education Implementation Case, 15th International Conference on Ubiquitous Computing and Communications and 2016 International Symposium on Cyberspace and Security (IUCC-CSS),132-137. De la Cruz, D. R., & Mendoza, D. M. (2018). Design and development of virtual laboratory: A solution to the problem of laboratory setup and management of pneumatic courses in bulacan state university college of engineering. In 2018 IEEE games, entertainment, media conference (pp. 1–23). IEEE. Dill, D. D., & van Vught, F. A. (2010). National Innovation and the Academic Research Enterprise: Public Policy in Global Perspective. Johns Hopkins University Press. Ferreyra, M.M., Avitabile, C., Botero, J., Haimovich, F., & Urzúa, S. (2017). At a Crossroads: Higher Education in Latin America and the Caribbean. Directions in Development— Human Development. World Bank, Washington, DC. Glewwe, P., and M. Kremer. (2006). “Schools, Teachers, and Education Outcomes in Developing Countries,” in E. Hanushek and F. Welch, eds., Handbook of the Economics of Education: Vol 1. North Holland Glynn, S. M., Brickman, P., Armstrong, N., & Taasoobshirazi, G. (2011). Science motivation questionnaire II: Validation with science majors and nonscience majors. Journal of Research in Science Teaching, 48(10), 1159–1176. Hamilton, D., McKechnie, J., Edgerton, E., & Wilson, C. (2021). Immersive virtual reality as a pedagogical tool in education: a systematic literature review of quantitative learning outcomes and experimental design. Journal of Computers in Education, 8, 1–32. Howard. M., Gutworth, M.B., Jacobs, R,R. (2021). A Meta-Analysis of Virtual Reality Training Programs. Computers in Human Behavior, 121, 106868 17 Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Hall, C. (2016). NMC Horizon Report: 2016 Higher Education Edition. Austin, Texas: The New Media Consortium. Kavanagh, S., Luxton-Reilly, A., Wuensche, B., & Plimmer, B. (2017). A systematic review of Virtual Reality in education. Themes in Science and Technology Education, 10(2), 85- 119. Kay, R. (2011). Evaluating learning, design, and engagement in web-based learning tools (WBLTs): The WBLT Evaluation Scale. Computers in Human Behavior, 27(5), 1849- 1856. Kember, D., & Leung, D. (2009). Development of a questionnaire for assessing students’ perceptions of the teaching and learning environment and its use in quality assurance. Learning Environments Research, 12, 15-29. DOI:10.1007/s10984-008-9050-7 Klopfer, E., & Squire, K. (2008). Environmental detectives-the development of an augmented reality platform for environmental simulations. Educational Technology Research and Development, 56(2), 203–228. Krokos, E., Plaisant, C., & Varshney, A. (2019). Virtual memory palaces: immersion aids recall. Virtual Reality, 23(1), 1–15. Lampi, E. (2013). The effectiveness of using virtual laboratories to teach computer networking skills in Zambia (Doctoral dissertation, Virginia Tech). Levin,Victoria; Santos,I.V; Weber,M. l; Iqbal,S; Ashwani, A; Comyn,P.J.; Katayama,H; Hoftijzer,MA (2023). Building Better Formal TVET Systems: Principles and Practice in Low- and Middle-Income Countries. Washington, D.C. World Bank Group. http://documents.worldbank.org/curated/en/099071123130516870 McKenzie, D. (2012). Beyond baseline and follow-up: The case for more T in experiments. Journal of Development Economics, 99(2), 210–221. https://doi.org/10.1016/j.jdeveco.2012.01.002 Madathil, K. C., Frady, K., Hartley, R., Bertrand, J., Alfred, M., & Gramopadhye, A. (2017). An empirical study investigating the effectiveness of integrating virtual reality-based case studies into an online asynchronous learning environment. Computers in Education Journal, 8(3), 1-10. Makransky, G., Petersen, G.B. (2021). The Cognitive Affective Model of Immersive Learning (CAMIL): a Theoretical Research-Based Model of Learning in Immersive Virtual Reality. Educational Psychology Review 33 (3), 937–958. Mayer, R. (2009). Multimedia Learning. Cambridge University Press. Mayer, R. (2014). Cognitive Theory of Multimedia Learning. In R. Mayer (Ed.), The Cambridge Handbook of Multimedia Learning (Cambridge Handbooks in Psychology, pp. 43-71). Cambridge: Cambridge University Press. McLaurin, E. J., & Stone, R. T. (2012, September). Comparison of virtual reality training vs. integrated training in the development of physical skills. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 56, No. 1, pp. 2532-2536). Sage CA: Los Angeles, CA: SAGE Publications. McMahan, A. (2003). Immersion, Engagement, and Presence. In Wolf and Paren (Ed.). The 18 Video Game Theory Reader. New York: Routledge. 67-86. Mikropoulos, T. A., & Natsis, A. (2011). Educational virtual environments: A ten-year review of empirical research (1999–2009). Computers & Education, 56(3), 769-780. Mount, N. J., Chambers, C., Weaver, D., & Priestnall, G. (2009). Learner immersion engagement in the 3D virtual world: principles emerging from the DELVE project. Innovation in Teaching and Learning in Information and Computer Sciences. Román-Ibáñez, V., Pujol-López, A. F., Mora-Mora, H., Pertegal-Felices, L. M., & Jimeno- Morenilla, A. (2018). A low-cost immersive virtual reality system for teaching robotic manipulators programming. Sustainability, 10(4) Rupasinghe, T. D., Kurz, M. E., Washburn, C., & Gramopadhye, A. K. (2011). Virtual reality training integrated curriculum: An aircraft maintenance technology (AMT) education perspective. International Journal of Engineering Education, 27(4), 778. Schubert, T., Friedmann, F. & Regenbrecht, H. (2001). The experience of presence: Factor analytic insights. Presence: Teleoperators and virtual environments, 10(3), 266-281. Stone, R. T., Watts, K. P., & Zhong, P. (2011). Virtual reality integrated welder training. Welding Journal, 90(7), 136s Wei, W., Dongsheng, L., & Chun, L. (2013, September). Fixed-wing aircraft interactive flight simulation and training system based on XNA. In 2013 International Conference on Virtual Reality and Visualization (pp. 191-198). IEEE. World Bank (2021). Unleashing the Power of Educational Technology in TVET System. Washington D.C. The World Bank Group. Zhou, Y., Ji, S., Xu, T., & Wang, Z. (2018) Promoting knowledge construction: a model for using virtual reality interaction to enhance learning. Procedia computer science, 130 (2018), 239-246 19 Figure 1. Theory of Change of the Mixed-Reality Laboratory Program Source: Authors’ elaboration. Figure 2. Timeline of Data Collection Endline Endline Baseline Baseline May August October February 2023 2024 Cohort 1 Cohort 2 Source: Authors’ elaboration. Table 1. Description of the Cognitive Outcome Number of items: 48 Type: Multiple-choice items distributed across the different levels of Bloom's Taxonomy: Remembering: 22 items; Understanding: 15 items; Application: 8 items; Analysis: 3 items Topics: Items cover content from all 9 training modules. Module 1: General Description of the Engine Module 2: Crankshaft and Flywheel Module 3: Pistons and Connecting Rods Module 4: Valve Systems Module 5: Lubrication and Cooling Systems Module 6: Intake and Exhaust Systems Module 7: Ignition and Fuel Injection Systems Module 8: Disassembly Module 9: Assembly Source: Authors’ elaboration. 20 Table 2. Characteristics of the Sample at the Baseline (N = 416) (1) (2) (3) (4) (5) P-val. Mean SD Mean SD Variable control vs control control treat treat treat Age 20.880 4.463 20.764 3.201 0.895 Work and study (Y/N) 0.599 0.491 0.600 0.491 0.994 Have done an internship (Y/N) 0.344 0.476 0.350 0.478 0.962 Repeated at least one subject (Y/N) 0.135 0.343 0.155 0.362 0.684 Received cash transfer (Y/N) 0.151 0.359 0.159 0.367 0.845 Mother edu: Primary 0.464 0.500 0.486 0.501 0.734 Mother edu: Secondary 0.391 0.489 0.405 0.492 0.760 Mother edu: Higher 0.146 0.354 0.109 0.312 0.384 Morning Shift (Y/N) 0.347 0.477 0.471 0.500 0.646 Cognitive pretest 546.163 123.677 572.542 123.083 0.262 Motivation scale 62.380 9.765 64.018 8.760 0.223 Engagement scale 67.396 10.329 68.177 9.405 0.613 Table 3. Impact on Cognitive Outcome (1) (2) Cognitive Outcome Cognitive Outcome Mixed Reality Laboratory 0.515*** 0.371*** (0.126) (0.097) Institution FE & Cluster Yes Yes Controls No Yes Mean Pure Control 0.000 0.000 SD Pure Control 0.997 0.997 N 381 377 R-squared 0.107 0.499 Note: The cognitive score is standardized based on the mean and standard deviation of the control group. The controls include the pretest cognitive score, a dummy variable indicating whether the student’s household receives the Bono Solidario, a dummy variable for whether the student both works and studies or only studies, whether the student has repeated any subjects, whether the student has completed a pre-professional internship, and a dummy variable capturing whether the student attends the morning shift versus the night shift. Standard errors (in parentheses) are clustered at the class level. * p < 0.10, ** p < 0.05, *** p < 0.01 21 Table 4. Intermediate Mechanism, Motivation (1) (2) Motivation Motivation Mixed Reality Laboratory 2.357*** 2.499*** (0.667) (0.635) Institution FE & Cluster Yes Yes Controls No Yes Mean Pure Control 61.354 61.354 SD Pure Control 11.463 11.463 N 380 380 R-squared 0.171 0.179 Note: The controls include age, a dummy variable indicating whether the student’s household receives the Bono Solidario, a dummy variable for whether the student works and studies or only studies, whether the student has repeated any subjects, whether the student has completed a pre-professional internship, two dummy variables for the mother’s education level (primary education or less, and secondary education), and a dummy variable capturing whether the student attends the morning or night shift. Standard errors (in parentheses) are clustered at the class level. * p < 0.10, ** p < 0.05, *** p < 0.01 Table 5. Intermediate Mechanism, Engagement (1) (2) Engagement Engagement Mixed Reality Laboratory 3.845*** 4.057*** (1.066) (1.006) Institution FE & Cluster Yes Yes Controls No Yes Mean Pure Control 64.309 64.309 SD Pure Control 12.820 12.820 N 380 380 R-squared 0.167 0.180 Note: The controls include age, a dummy variable indicating whether the student’s household receives the Bono Solidario, a dummy variable for whether the student works and studies or only studies, whether the student has repeated any subjects, whether the student has completed a pre-professional internship, two dummy variables for the mother’s education level (primary education or less, and secondary education), and a dummy variable capturing whether the student attends the morning or night shift. Standard errors (in parentheses) are clustered at the class level. * p < 0.10, ** p < 0.05, *** p < 0.01 22 Figure 3. Usability of zSpace Technology: User Experience and Feedback Usability of zSpace Technology I felt confident using it. 84% It was easy to use. 84% I think most people learn it quickly. 85% The functions were well integrated. 87% I’d like to use it frequently. 90% It was uncomfortable to use. 54% 27% I had to learn a lot before using it. 63% It seemed unnecessarily complex to me. 66% I need technical support to use it. 71% 0% 20% 40% 60% 80% 100% 120% Strongly disagree/Disagree Neutral Agree/Strongly agree Source: Authors’ elaboration. Figure 4. Usability of Tablets: User Experience and Feedback Usability of the Tablets I felt confident using it. 84% It was easy to use. 87% I think most people learn it quickly. 80% The functions were well integrated. 87% I’d like to use it frequently. 89% It was uncomfortable to use. 47% 34% I had to learn a lot before using it. 64% It seemed unnecessarily complex to me. 57% I need technical support to use it. 58% 0% 20% 40% 60% 80% 100% 120% Strongly disagree/Disagree Neutral Agree/Strongly agree Source: Authors’ elaboration. 23 ANNEX 1: Motivation Questionnaire To better understand what you think and how you feel about the auto-mechanics course, please respond to each of the following statements from the perspective of “When I am in an auto-mechanics course…” Statements Never Rarely Sometimes Often Always (0) (1) (2) (3) (4) Intrinsic Motivation The auto-mechanics I learn is relevant to my life Learning auto-mechanics is interesting Learning auto-mechanics makes my life more meaningful I am curious about discoveries in auto-mechanics I enjoy learning auto-mechanics Self-Efficacy I am confident I will do well on auto-mechanics tests I am confident I will do well on auto-mechanics labs and projects I believe I can master auto-mechanics knowledge and skills I believe I can earn a grade of “A” in auto-mechanics I am sure I can understand auto-mechanics Self-Determination I put enough effort into learning auto-mechanics I use strategies to learn auto-mechanics well I spend a lot of time learning auto-mechanics I prepare well for auto-mechanics tests and labs I study hard to learn auto-mechanics ANNEX 2: Student Engagement Please indicate whether you agree with the following statements. Statements Strongly Disagree Neutral Agree Strongly Disagree (2) (3) (4) Agree (1) (5) I have become more confident in my ability to keep learning At the institute, I have learned to be more adaptable The instructor uses a variety of teaching methods Students are given the opportunity to participate in class The instructor makes an effort to help us understand the course material The course program helps students understand its contents When I struggle with course content, the explanations provided by the instructor are helpful There is enough feedback on activities and assignments to make sure we learn from the work we do A variety of assessment methods are used in the course To do well in this course, good analytical skills are required The assessments evaluate our understanding of key concepts in this course Communication between the instructor and students is good The instructor is helpful when asked The amount of work expected from us is fairly reasonable I feel a strong sense of belonging to my class group The curriculum for my program is well integrated 24 ANNEX 3: System Usability Scale Please indicate whether you agree with the following statements about the zSpace technology used in the laboratory. Statement Strongly Disagree Neutral Agree (4) Strongly Disagree (2) (3) Agree (5) (1) I would like to use the zSpace technology frequently I found the zSpace technology unnecessarily complex The zSpace technology was easy to use I need technical support to use the zSpace technology I found that the various functions in the zSpace technology were well integrated I imagine that most people would learn to use the zSpace technology very quickly I found the zSpace technology very awkward to use I felt very confident using the zSpace technology I had to learn many things before I could use the zSpace technology Note. The same scale was used to assess the usability of the tablets included as part of the Mixed Reality laboratory. 25 ANNEX 4: Cost Associated with the Intervention Cost Ingredient Description Cost (USD) Collaboration between professors in Korea and 1. Curricula Design 50,000 Ecuador to design the curriculum Development of software for zSpace and tablet- 2. Software Development 150,000 based technologies Remote support services (5 years) 53,000 3. Technical support and laboratory deployment Equipment import / transport 26,000 Teacher training and implementation guidance 30,000 Sub-Total 98,000 Laboratory hardware (zSpace, tablets) 350,000 4. Laboratory hardware Accessories (eyewear, clips, pointers, etc.) 10,000 and licenses (5 labs, 30 students each) Software licenses and warranties 50,000 Sub-Total 410,000 5. Furniture and lab Tables, chairs, secured storage, curtains, electrical adaptation 50,000 installations (5 Labs) TOTAL COST 769,000 26