Governance of the Service Delivery Chain for Youth Mental Health in Lithuania Key findings from a public sector employee survey June 2022 Funded by the European Union Acronyms ALA Association of Local Authorities BL Bureaucracy Lab EC European Commission HBSC Health Behavior in School-aged Children KPI Key Performance Indicator LSU Lithuanian Students' Union MoESS Ministry of Education, Science and Sports MoH Ministry of Health MoSSL Ministry of Social Security and Labor NAE National Agency for Education NGO Non-governmental Organization OECD Organization for Economic Cooperation and Development PHB Public Health Bureau PMO Prime Minister's Office RCT Randomized Controlled Trial STRATA The Government's Strategic Analysis Center ŠVIS Švietimo Valdymo Informacinė Sistema (Education Management Information System) WB World Bank WDR World Development Report WMS World Management Survey Contents Acknowledgements........................................................................................................................................ i Executive summary ....................................................................................................................................... ii Chapter 1: The youth mental health crisis in Lithuania ................................................................................ 1 1.1 The context ......................................................................................................................................... 1 1.2 The objectives of the report ............................................................................................................... 3 Chapter 2: Conceptual framework and empirical approach ........................................................................ 5 2.1. Conceptual Framework ...................................................................................................................... 5 2.2 Survey design ................................................................................................................................ 8 Chapter 3: Main findings............................................................................................................................. 11 3.2 Inputs ................................................................................................................................................ 15 3.3 Organizational and personnel drivers of the youth mental health service delivery chain ............... 17 3.3.1. Attitudes: Stigma and motivation ............................................................................................. 17 3.3.2 Work environment: Collaboration and protocols ...................................................................... 33 3.3.3 Knowledge.................................................................................................................................. 25 3.3.4 Capacity building ........................................................................................................................ 27 3.3.5 Management quality .................................................................................................................. 35 Chapter 4: Policy recommendations ........................................................................................................... 39 4.1 Improving Measurement and Evaluation in the Service Delivery Chain .................................... 48 4.1.1 Improve primary data collection................................................................................................ 48 4.1.2 Embed evaluations in policy making .......................................................................................... 49 4.1.3 Facilitate the use of data in decision making by municipal authorities ..................................... 50 4.2 Optimizing capacity building .......................................................... Error! Bookmark not defined. 4.2.1 Update capacity building content to address stigma ................................................................ 39 4.2.2 Establish standards for training and incentivize staff to participate ......................................... 39 4.2.3 Incorporate culture shift as part of capacity building ................................................................ 40 4.3 Enhancing Work Environment .......................................................................................................... 41 4.3.1 Strengthen teamwork in addressing mental health by schools ................................................ 41 4.3.2 Strengthen collaboration among subnational level institutions ............................................... 43 4.3.3 Expand, standardize, and universally apply protocols addressing mental health in schools .... 43 4.4 Strengthening Management ............................................................................................................. 44 4.4.1 Improve strategic and managerial competencies of school leadership .................................... 45 4.4.2 Combine capacity building with incentives for principals and administrators .......................... 46 Annexes ....................................................................................................................................................... 51 Annex 1: Timeline of the project ........................................................................................................ 51 Annex 2: The service delivery chain of youth mental health in Lithuania .......................................... 52 Annex 3: Sampling strategy ................................................................................................................ 53 Annex 4: Municipalities covered in the survey ................................................................................... 56 Annex 5: Survey design and implementation ..................................................................................... 57 Annex 6: RCT ....................................................................................................................................... 58 List of figures, boxes and tables Figure 1: The production function for mental health services for the youth ............................................... 6 Figure 2: The survey covered 43 out of the 60 municipalities in Lithuania (municipalities surveyed depicted in blue) .......................................................................................................................................................... 9 Figure 3: There is municipality level variation in the share of students that experienced some form of mental health challenges in the fall term of 2020 (school staff estimates) ............................................... 12 Figure 4: There is school level variation in the share of students that experienced some form of mental health challenges in the fall term of 2020 (school staff estimates)............................................................ 13 Figure 5: There is municipality level variation in the share of students that were formally diagnosed with mental health disorders in the fall term of 2020 (school staff estimates) ................................................. 13 Figure 6: There is school level variation in the share of students that were formally diagnosed with mental health disorders in the fall term of 2020 (school staff estimates) ............................................................. 14 Figure 7: There is a significant school level variation in the number of times teachers referred students to the school psychologist in the past 12 months. ......................................................................................... 14 Figure 8: School staff face major challenges while working from home, with the major one being poor quality of collaboration while working remotely........................................................................................ 16 Figure 9: A majority of respondents do not disagree with the statement - Individuals with mental health challenges are more dangerous to other people than the average person (school staff estimates) ........ 19 Figure 10: Older school staff have more mental health stigma ................................................................. 20 Figure 11: Stigma is negatively associated with identifying youth mental health challenges.................... 21 Figure 12: Stigma is negatively associated with the perceived school response to student mental health challenges ................................................................................................................................................... 21 Figure 13: Variation exists in the stigma index for municipalities in Lithuania (schools and public administration) ........................................................................................................................................... 22 Figure 14: Different tiers of public administration have notably different responses to the statement - It's best not to talk about mental health challenges ........................................................................................ 23 Figure 15: Older public administration staff have more stigma ................................................................. 23 Figure 16: A majority of public administrators do not seek to leave their organization in the near future .................................................................................................................................................................... 25 Figure 17: School staff have low knowledge of the HBSC study and report a high variation in responses to HBSC specific questions .............................................................................................................................. 26 Figure 18: Lower knowledge of youth mental health issues amongst school staff is associated with stigma .................................................................................................................................................................... 26 Figure 19: There is a variation in the number of reported hours of training for school staff .................... 28 Figure 20: There is regional variation in the uptake of MoH youth mental health training program amongst schools ........................................................................................................................................................ 29 Figure 21: School staff who spend more time in training have lower levels of stigma .............................. 30 Figure 22: School staff who spend more time in training perceive better mental health service delivery 31 Figure 23: School staff who spend more time in training report higher prevalence of student mental health issues ........................................................................................................................................................... 31 Figure 24: School staff rarely collaborate with municipality staff to create policies and plans for youth mental health .............................................................................................................................................. 34 Figure 25: There is diversity in overall school management quality scores across Lithuania .................... 37 Figure 26: The management quality in schools affects the mental health support offered to students ... 37 Figure 27: There is diversity in overall public administration management scores across Lithuania ........ 38 Figure 28: Public procurement dashboard- An example of applied data analytics for decision making ... 50 Figure 29: The service delivery chain of youth mental health in Lithuania ................................................ 52 Figure 30: Sampling strategy for public administration institutions (ministry example) ........................... 55 Figure 31: Sampling strategy for schools .................................................................................................... 56 Figure 32: The MoH branded version of the e-mail school administrators received to encourage them to advertise the training (text in Lithuanian) .................................................................................................. 58 Figure 33: Number of times treatment emails were opened by the T1 and T2 groups through time ....... 60 Figure 34: Treatment and control group regressions on number of staff per module .............................. 60 Box 1: Measuring stigma............................................................................................................................. 18 Box 2: School staff strongly respond to communicated training opportunities: findings from an impact evaluation ................................................................................................................................................... 32 Box 3: Canadian experience: High impact school mental health interventions ......................................... 40 Box 4: Teaming for students‘ mental health: resources in the USA ........................................................... 42 Box 5: Examples of guidance documents for schools from Canada and the USA ...................................... 43 Box 6: Why is Shanghai the best in student learning? ................................................................................ 47 Box 7: Research tools in policy making ....................................................................................................... 49 Table 1: Summary of recommendations to improve governance of youth mental health service delivery chain vi Table 2: Major milestones of the project 51 Acknowledgements This report is the product of the Bureaucracy Lab, a partnership between the Governance Global Practice and the Development Impact Evaluation Department (DIME) of the Research Group. It is part of a multi- country collaboration with the European Commission (EC), funded via the Part II Europe 2020 Programmatic Single-Donor Trust Fund with the EC (TF073353), the objective of which is to empirically understand the personnel determinants of, and mechanisms influencing, productivity in public administration and service delivery units in EU Member States. The report was prepared by a WB team led by Zahid Hasnain (Lead Governance Specialist) and Daniel Rogger (Senior Economist), and consisting of Gailius Praninskas (Consultant), Jolanta Blazaite (Consultant) Mantasha Husain (Consultant), Patricia Rose Paskov (Consultant) and Ravi Somani (Consultant). The team also benefited from valuable support on the survey design from Alessandra Fenizia (Assistant Professor, George Washington University). The team would like to thank Roby Senderowitsch (Practice Manager, Governance Global Practice, Europe and Central Asia), Pedro Arizti (Senior Public Sector Specialist), and the Governance ECA Ops Team for their overall advice, guidance, and support throughout the development of this report. We also want to thank the peer reviewers, Marco Larizza (Senior Public Sector Specialist), Shomikho Raha (Senior Governance Specialist), and Reena Badiani-Magnusson (Program Leader), for their very helpful comments and suggestions. And we are grateful to Marcus Heinz (Country Manager) for his guidance on how to maximize the impact of the report. The team would like to express its gratitude to government officials of the Government’s Strategic Analysis Center (STRATA) — Dalia Bagdžiūnaitė, Dovilė Gaižauskienė, and Viktoras Urbis, the Ministry of Health (MoH)—Laura Masiulienė, Ignas Rubikas, and Simona Bieliūnė, the Ministry of Education, Science and Sports (MoESS)—Gražina Šeibokienė, Laima Rutkauskienė, Jolanta Navickaitė and Bagdonaitė Jurgita, and the Ministry of Social Security and Labor (MoSSL)—Jolanta Sakalauskienė and Tania Gurova. Special thanks goes to officials from all the public institutions who supported the implementation of the survey and who took the time to respond to the questionnaire. Finally, we are grateful for the close collaboration and continuous support of officials from DG REGIO and DG REFORM of the EC, particularly Lewis Dijkstra, Philippe Monfort, and Mina Shoylekova. i Executive summary 1. There is a youth mental health crisis in Lithuania. In 2018, 24 percent of students in grade 1 to 9 thought about suicide, 40 percent experienced low levels of psychological well-being, and 24 percent experienced bullying. The COVID-19 pandemic and resulting school closures has worsened the mental health of adolescents, with the share of students experiencing severe psychosocial difficulties increasing from 14 to 20 percent after the start of the pandemic. Mental health challenges have several delirious consequences for the youth, including poor performance in school and negative outcomes later in life such as substance abuse, poor health, unemployment, and incarceration. Addressing this crisis is a priority for the Government of Lithuania, as documented in several strategic plans and budgetary programs. 2. At the request of the Government, the World Bank’s Bureaucracy Lab undertook an empirical diagnostic of the youth mental health service delivery chain, focusing on the education sector. The Government recognizes that improving service delivery requires addressing a variety of governance challenges in the several organizations in the education and public health system—central ministries, municipalities, Public Health Bureaus (PHBs), schools, and clinics—that are responsible for the delivery of youth mental health services. The immediate priority, however, is to improve youth mental health services offered by schools, as these are the first and primary locus of contact between public sector workers and youth and are responsible for prevention and early detection of mental health problems. The Government also recognizes that school services are influenced by the public administrators who set policy, provide funding, and supervise and monitor school activities. The Lab’s diagnostic, therefore, did a focused analysis of the education service delivery chain, examining factors in schools and the public administration that impact the delivery of youth mental health services. This study is part of a broader collaboration between the European Commission (EC) and the World Bank on measuring the organizational and personnel determinants of public administration productivity. 3. The study was based on a conceptual framework of a production function of youth mental health service delivery. The framework is motivated by the “system level” approach of World Development report 2018: Learning to Realize Education’s Promise, examining both school level factors and bureaucratic capacity, but does a deep dive on the organizational and personnel drivers of service delivery in schools and the public administration. These drivers include attitudes, behaviors, and knowledge of public sector workers; extent of capacity building; the work environment of the organization in which the public sector workers operate; and the quality of management. The choice of these factors is informed both by the academic literature which has underlined their importance, and by pragmatism so as to identify areas of reform that can be relatively easily acted upon in the short- (approximately 1 year) and medium-term (3 years) within available resources and by the relevant authorities. 4. The Lab conducted a survey of approximately two thousand public sector workers to measure these drivers, the quality of youth mental health services provided by schools, and the correlations between the drivers and services. The survey design followed a stratified sample of first, selecting the appropriate public sector organizations and second, selecting a representative sample of personnel within those organizations. The survey was implemented in 83 public administration organizations and 230 ii schools, with interviews conducted of 956 public administrators and 1026 school personnel. When presenting the results however, the report limits the analysis to the public administration respondents who are more involved in financing, supervising, and monitoring schools in mental health service delivery (around 47 percent of the public administration sample). The survey was conducted from March 2021 to October 2021 via secure, encrypted video calls between the enumerator and respondent (on account of COVID-19), and each interview lasted approximately one hour. Main findings 5. Most students do not receive help for the mental health challenges that they face. School staff report that only 12 percent of students received any help from their school, 64 percent received education on mental health, and 19 percent visited the school psychologist in the past 12 months. There is considerable variation across schools in the quality of these services, pointing to the importance of school- level factors in determining the quality of mental health services. 6. Public sector personnel have high levels of stigma towards mental health which influences both their perceptions of mental health as a problem and the actions that they take to help students. The survey used several questions to measure stigma, drawing on the academic literature. For example, both school staff and public administrators report that about 8 out 10 of their colleagues would want it to remain a secret if they were diagnosed with mental health problems. More than a quarter of school staff believe that it is best not to talk about mental health challenges, and 17 percent state that mental health problems are a sign of personal failure. A majority of respondents in schools and the public administration agree or do not disagree that individuals with mental health problems are dangerous to other people. Male and older employees demonstrated higher stigma in both schools and the public administration. Stigma is correlated with a lower likelihood of identifying mental health challenges and lower quality of school services. 7. School staff and administrators have little knowledge of the extent of youth mental health problems in Lithuania. The survey measures knowledge with reference to staff awareness of the findings of a major national health study, the Health Behavior in School-aged Children (HBSC) study, that is conducted in Lithuania every four years and is used by the MoH in policymaking and monitoring. Ninety two percent of school staff, 89 percent of school psychologists, and 85 percent of public administrators were not aware of the HBSC study. Even those staff who were aware of the study were widely inaccurate in their estimates of mental health statistics, such as the share of students not satisfied with life or experiencing bullying in school. School staff who have low knowledge of the study display higher mental health stigma. 8. This high level of stigma and low level of knowledge about youth mental health underlines the importance of training to change attitudes and improve the capacity to respond, but training is underprioritized and underutilized. Seventy-one percent of school staff are unaware of the youth mental health training offered regularly by the MoH, and 61 percent of school managers’ report that their school did not participate in the training in the past 12 months. There is also a wide variation in the number of iii self-reported hours that school staff spent in training in the past 12 months, ranging from 0 to a 100 hours, pointing to a lack of any established standard or enforcement of the amount expected of school staff. Those who do participate agree that training sessions on youth mental health were useful in imparting practical knowledge and skills and improved collaboration amongst staff members in the school. Indeed, staff who spend a higher amount of time in mental health training also have lower levels of stigma. 9. Prevention programs implemented by schools have a positive self-reported impact on school staff and students, but schools lack strategy while implementing these programs. More than 80 percent school staff agree or strongly agree that prevention programs in their school have made it easier to talk openly about prevention themes within the school and have strengthened their skills and knowledge to effectively deal with issues in the prevention area, and about 51 percent of respondents perceive a significant impact of the programs on students with challenges in the areas addressed by prevention programs. However, the study found that the choice of prevention programs was not influenced by the recommendations of public health offices and municipal administrations and under 2 percent of all implemented prevention programs covered parents and class teachers, who play an important role in the mental health of young adults. 10. An impact evaluation conducted as part of the study showed that branding and advertising training from relevant authorities improved its uptake. In 2021, the MoH started an online, self-paced mental health training module to improve competencies of school staff. In the randomized control trial (RCT), some staff were randomly sent reminder emails encouraging participation in training, with one treatment group getting emails from the MoH, and another from the Lithuanian Student Union (a non- governmental organization). The control group received no such emails. The main finding of the evaluation was that these email reminders were effective in increasing participation in the training, and that the email from the MoH was more impactful in terms of the number of staff starting and completing the training. 11. There are low levels of collaboration between school staff, between staff and parents, and between school staff and public administrators. Most school staff do not even know about the presence of a school psychologist, with only 32 percent of staff correctly identifying whether the school has a psychologist. Only 34 percent of school staff engaged with parents a few times regarding the mental health of their children in the past 12 months. A quarter of school staff reported never having worked with their municipality to create educational measures for youth mental health. More than 40 percent of school respondents do not agree that over the last 12 months, budgeting decisions regarding schools and youth made by the municipality were done in close collaboration with their school. These problems of collaboration extend across the public administration, with around 12 percent of the public administrators reporting that they never worked closely with officials from other important and relevant tiers of government and government ministries to create policies and plans for youth mental health in the past 12 months. 12. Management quality is an important determinant of service delivery and varies amongst schools, and amongst different public administration organizations and units. This study assessed iv management quality using an widely used measure (the World Management Survey) which includes data on indicators like organizational goal setting and monitoring, the use of performance incentives, extent of innovation and problem solving by staff, and flexibility and autonomy for staff. There is high variance in the overall management quality across schools and public administration, implying that the organizational environment in which youth mental health is being addressed varies substantially. The quality of management is also correlated with student mental health services, with better managed schools also scoring higher on staff assessments of service quality. An example of this impact is that schools with above average managers are more likely to employ psychologists than schools with worse management (54 percent compared to 41 percent) and this difference is also reflected in school staff perceptions—in better managed schools, 61 percent of staff agree or strongly agree that there is sufficient mental health staff, compared to 53 percent in worse managed schools. Recommendations 13. These findings, combined with the existing research conducted by STRATA, suggest implementing a multifaceted approach to improving the school mental health service delivery chain. The recommendations, outlined below, aim to provide partners with informed insights on what needs to be achieved, followed by how to institute the change, and who (which agencies at which administrative level) may be best suited to lead the process. They deploy multiple and mutually reinforcing activities like improvement of attitudes, better capacity building, improving work environment and strengthening management along with the use of evidence in policy making. Table 1 summarizes short- (within 1 year) and medium-term (within 3 years) recommendations, which can be implemented without substantial additional funding. v Table 1: Summary of recommendations to improve governance of youth mental health service delivery chain Recommendations Action to be taken Timeline Leading institutions Address stigma through training, incentives, and communication Improve training content to address stigma Draw on the expertise of local and international specialists in youth mental Short-term MoH, Association of health to improve the curricula of the MoH training and prevention programs Local Authorities (ALA), to address the problem of stigma municipal administrations, schools Establish standards for training and Conduct a need-based assessment, for both staff and students, before Short-term MoH, MoESS, schools, incentivize staff to participate implementing a training program or a prevention program municipal administration Establish a standard for the number of mental health training hours Medium-term MoH, MoESS, municipal undertaken by school staff and include this standard in the annual administrations performance assessment of staff Ensure effective branding and communication of training programs and Short-term MoH, MoESS prevention programs to maximize staff participation Incorporate culture change in the design of Improve the culture around capacity building programs so that trainees are Short-term Schools training programs encouraged to apply the learning of training programs and not view it as a “box ticking” exercise Fostering collaboration in the work environment Strengthen teamwork in addressing mental Use dedicated interventions to foster teamwork and broader discussions around Short-term MoESS, schools health by schools mental health across the public service Encourage closer relations among teachers, parents, and students by involving Short-term MoESS, schools students' self-government groups in mental health literacy activities Strengthen collaboration among Use the modality of a participatory budget to foster collaboration Short-term Municipal subnational level institutions administrations, schools Mobilize inter-agency and youth coordinators to promote co-production and Short-term Municipal collaboration in the area of youth mental health administrations Expand, standardize, and universally apply Develop national level guidelines for schools to serve as a reference in Medium-term MoESS, MoH protocols addressing mental health in developing protocols to address mental health in schools schools vi Strengthening management Improve strategic and managerial Establish a competency framework for leaders that includes of strategic Medium-term MoESS, MoH, NAE competencies of school and administration thinking, organizational goal setting, change management, human resource leadership management, and community engagement as required competencies Implement a mix of different types of capacity building approaches for Short-term MoESS, NAE, managers, including traditional curriculum-based training and peer-based Association of School exchanges Leaders, Schools Combine capacity building with incentives Use the task-setting component of performance evaluation to incentivize Short-term MoESS, NAE, municipal for principals and administrators acquisition of strategic and managerial competencies administrations Strengthen interaction between different levels of administration in Medium-term MoESS, NAE, municipal performance evaluation administrations, schools Introduce behavioral nudges to remind principals to follow protocols on school Short-term MoESS, NAE mental health, and to disseminate good practices on fostering collaboration or teacher management Improving measurement and evaluation in the service delivery chain Improve primary data collection Include granular, individual data in ŠVIS and other administrative datasets Medium-term Prime Minister’s Office (PMO) Replicate the WB survey on organizational and personnel drivers on an annual Short-term PMO, MoESS, MoH, basis to monitor a core set of performance indicators MoSSL Embed monitoring and evaluation in policy Selectively use randomized controlled trials (RCT) to gauge effect of different Medium-term Government’s making policies ministries, STRATA Facilitate use of data in decision making by Provide easily accessible and user-friendly data management tools Medium-term PMO municipal authorities Use municipal level data for establishing KPIs in the areas of public health and Short-term Association of Local educational support and repeat measurement to benchmark progress towards Authorities (ALA) these KPIs. vii Chapter 1: The youth mental health crisis in Lithuania 1.1 The context 14. There is a youth mental health crisis in Lithuania. Over one in ten students in schools was formally diagnosed with a mental health malady in the 2019-2020 school year. In 2018, 24 percent of students in grade 1 to 9 thought about suicide, 40 percent experienced low levels of psychological well- being, and 24 percent experienced bullying, the highest levels of any country in the Organization for Economic Cooperation and Development (OECD) countries.1 The COVID-19 pandemic has worsened the mental health of adolescents. Schools in Lithuania were closed for much longer than the OECD average during the pandemic, with middle schools closed for 137 days (the OECD average was 93 days) and high schools for 132 days (the OECD average was 101 days)2, which contributed to the share of students experiencing severe psychosocial difficulties increasing from 14 to 20 percent after the start of the pandemic.3 15. Poor youth mental health has several deleterious consequences. Mental health challenges are strongly correlated with inadequate socio-emotional development, poor student performance, and negative outcomes later in life such as substance abuse, poor health, unemployment, and incarceration.4 Mental health also has economic impacts, undermining the development of human capital and hurting the economic growth and prosperity of countries, especially through productivity losses at both individual and societal levels.5 Half of all mental health conditions start by age 14, but most cases are undetected and untreated.6 1 Data from the World Health Organization (WHO) Health Behavior in School-aged Children (HBSC) study (2018), which is a collaborative cross-national study of adolescent health and well-being, and includes a survey undertaken every four years (WHO Europe). 2 OECD. (2021, May). OECD Policy Responses to Coronavirus (COVID-19): Supporting Young People’s Mental Health Through the COVID-19 Crisis. OECD. https://www.oecd.org/coronavirus/policy-responses/supporting-young- people-s-mental-health-through-the-covid-19-crisis-84e143e5/ 3 Vilnius University. (2021, January). VU Researchers Found How the Beginning of the COVID-19 Pandemic Affected Lithuanians' Psychological Well-Being. https://www.vu.lt/en/news-events/news/8752-vu-researchers-found-how- the-beginning-of-the-covid-19-pandemic-affected-lithuanians-psychological-well-being. 4 Rossen, E., Cowan, K. C. (2014). Improving Mental Health in Schools. SAGE Journals, 96(4), 8-13. https://doi.org/10.1177/0031721714561438 WHO. (2013). Investing in Mental Health: Evidence for Action . Department of Mental Health and Substance Abuse, WHO, Geneva. 5 World Bank Group. (2018). Global Mental Health: Some Perspectives on Challenges and Options for Scaling Up Response. World Bank Group: Global Mental Health Initiative 6 WHO. (2021, November). Adolescent Mental Health. WHO. https://www.who.int/news-room/fact- sheets/detail/adolescent-mental-health 1 16. Addressing this challenge of poor and worsening youth mental health is a policy priority of the Government of Lithuania. Mental health is a key component of several government strategic documents, including the National Health Strategy for 2014–20257, and the National Public Healthcare Development Program for 2016–2023.8 Mental health promotion and prevention activities are being implemented through several programs, such as the National Plan on Suicide Prevention for 2020- 20249 and the National Drug, Tobacco and Alcohol Control and Use Prevention Program 2018-2028.10 Lithuania’s current government, which started its term in 2020, has further prioritized mental health by setting targets and assigning mental health promotion tasks to the government’s ministries. Special programs, like Strengthening of Mental Health and Emotional Literacy, funded by the state budget, are being developed for the implementation of these tasks. The policy area of youth mental health currently holds heightened importance as the Government of Lithuania responds to and builds back from the COVID-19 crisis. The Government’s programs align with the UN 2030 Agenda and the Sustainable Development Goals, which have listed promotion of mental health and well-being among global priority areas (Goal 3). 17. Meeting these policy priorities requires improving the governance of the service delivery chain for youth mental health. The government can, and should, spend more on public health. Lithuania’s latest assessment by the OECD notes that the country’s health spending at 6.5 percent of GDP remains below that of countries with a similar income per capita.11 There are, however, limits to how much more the government can spend given the likely fiscal constraints over the next few years as Lithuania recovers from the pandemic. The priority, therefore, is to deliver better services with limited resources, which requires addressing governance problems in the youth mental health service delivery system. One of the directions of Lithuania’s health care reforms, as highlighted by the Government’s Program 2020, is to improve and streamline processes, enhance collaboration between institutions, and improve the capacity, accountability, and attitudes towards mental health of public sector personnel in schools, municipal administration, and ministries. 18. There is little data or rigorous empirical analysis on what the key governance weaknesses are to inform evidence-based reforms. Several studies report coordination challenges between the healthcare, education, and social security sectors that all have a mandate in delivering youth mental health services, a lack of shared understanding of the problem, lack of engagement with the youth, and even a lack of agreement on mental health terms and definitions.12 While these point to a variety of 7 Lithuanian Health Strategy (2014-2025), Government of Lithuania. 8 National Public Healthcare Development Program (2016-2023), Government of Lithuania. 9 National Plan on Suicide Prevention (2020-2024), Government of Lithuania. 10 National Drug, Tobacco and Alcohol Control and Use Prevention Program (2018-2024), Government of Lithuania. 11 OECD. (2018). OECD Reviews of Health Systems: Lithuania 2018. Assessment and Recommendations . OECD. https://www.oecd.org/health/health-systems/OECD-Reviews-of-Health-Systems-Lithuania-2018-Assessment-and- Recommendations.pdf 12 Coppens, E., Vermet, I., Knaeps, J., De Clerck, M., De Schrijver, I., Matot, J. P., Van Audenhove, C. (2015). Adolescent Mental Health Care in Europe: State of the Art, Recommendations, and Guidelines by the ADOCARE network. ADOCARE. http://www.adocare.eu/wp-content/uploads/2015/12/ADOCARE-UK-2015.pdf Lithuanian Youth Council and the European Youth Forum. (n.d.). Joint Submission of the Lithuanian Youth Council (LiJOT) and the European Youth Forum (YFJ) Submission to the UN Office of the High Commissioner for Human Rights. 2 governance weaknesses, little evidence exists on the quality of implementation of mental health services to youth, either on the efficacy of such programs, or on the characteristics of the public servants who support, design, and implement these programs. OECD recommends that in Lithuania “in all spheres of health policy, a more decisive implementation of reforms needs to be accompanied with systematic evaluations to understand what may or may not work, why and what course-adjustments might be required to achieve better results faster.”13 1.2 The objectives of the report 19. Considering the above, the Government, specifically the MoH and STRATA invited the WB’s Bureaucracy Lab (BL) to undertake an empirically based diagnosis of the personnel dimension of the youth mental health service delivery chain. While the governance challenges are broad, this activity does a deep dive on organizational and human resource management, and the attitudes, skills, and behaviors of the public sector workers responsible for delivering these services across all tiers of government, from schools up to the municipal and central administration. The reasons for this narrower but deeper analysis are two-fold. First, youth mental health is a labor-intensive service and improving the productivity of the workforce can yield large improvements in service delivery. Second, this activity is part of a broader collaboration between the EC and the WB’s BL, funded by the Part II Europe 2020 Programmatic Single- Donor Trust Fund with the EC, on measuring organizational and personnel determinants of public administration productivity. The activity seeks to work with five European Union member states (Lithuania being one of them) to generate micro-level data to support evidence-based organizational and personnel management reforms in priority areas. 20. This study focuses on the education sector service delivery chain for youth mental health . Lithuania has two main channels of youth mental health support: the public health system and the education system (Annex 2). The primary health providers and Mental Health Centers under the MoH are responsible for the diagnosis and treatment of mental health disorders, and the National Public Health Center and municipal Public Health Bureaus (PBS) promote public health, including mental health. Schools under the management of municipalities and the Ministry of Education, Science, and Sports (MoESS) are responsible for youth mental health promotion, prevention, and early intervention. There is also a smaller role of Ministry of Social Security and Labor (MoSSL) which manages youth centers and supports NGOs providing mental health services, such as helplines. This study focuses on schools and the public administration that overseas and supports them, as international experience underlines the importance of early detection of mental health challenges. Research shows that the average age of onset across all mental disorders is 14.5 years, that early detection and help is key, and that schools are the most important locales for providing support given that adolescents spend such a large proportion of their time Universal Periodic Review, 26th Session: Lithuania. https://uprdoc.ohchr.org/uprweb/downloadfile.aspx?filename=2759&file=EnglishTranslation 13 OECD. (2018). OECD Reviews of Health Systems: Lithuania 2018. Assessment and Recommendations . OECD. https://www.oecd.org/health/health-systems/OECD-Reviews-of-Health-Systems-Lithuania-2018-Assessment-and- Recommendations.pdf 3 in them. Lithuania’s Law on Education14 highlights the role of schools in youth mental health service delivery, including communicating with students experiencing mental health challenges, determining whether the child's condition requires specialized and tailored education, and directing students to receive appropriate assistance. 21. For this diagnostic, the WB undertook an original and representative survey of approximately two thousand staff in the public administration and in schools. The objective of the survey is to: • Measure key organizational and personnel drivers in the public administration and schools that impact the quality of service delivery. • Measure the correlations between these organizational and personnel drivers on youth mental health services and outcomes. • Identify actionable recommendations that can be implemented in the short and medium term. • Demonstrate the value of a survey as an empirical tool to provide regular, robust data on key indicators with which to track progress with reforms. • Demonstrate the value of this approach to improve the delivery of mental health services in other contexts given the global mental health challenge posed by COVID-19. 22. The report is structured to give a comprehensive picture of the governance factors in Lithuania’s mental health services delivery system. Chapter 2 sets forth the conceptual framework used to study the challenge and to design the survey. Chapter 3 charts the results from the study, closely following the designed conceptual framework. Policy recommendations constitute Chapter 4, with actionable proposals given for each tier of the government. 14 Government of Lithuania. (1991). Law on Education of the Republic of Lithuania . Republic of Lithuania. https://e- seimas.lrs.lt/portal/legalAct/lt/TAD/TAIS.1480/asr; STRATA. (2020). The Service Delivery Chain of Youth Mental Health in Lithuania- Diagnostic I: School Context. STRATA, Government of Lithuania. https://strata.gov.lt/images/tyrimai/2021-metai/20210406-lietuvos-jaunimo-psichikos- sveikatos-paslaug-teikimo-grandins-diagnostika.pdf 4 Chapter 2: Conceptual framework and empirical approach 2.1. Conceptual Framework 23. This report undertakes a “systems approach” to understanding education and health outcomes that is gaining prominence in policy and academic literature. As the World Development Report 201815 shows, learning outcomes are conditional on both school level factors, like the quality of teachers, school management, and infrastructure; and bureaucratic capacity and the broader political economy.16 Similarly, the WB Health Strategy17 underscores the importance of addressing multiple governance challenges at the administrative and facility levels to achieve health outcomes. There is a growing academic literature on the organizational and personnel economics of state capability that demonstrates, using randomized control trials and statistical analysis, the importance of several factors—collaboration, norms and culture, mission orientation, quality of management, access to training, and monetary and non-monetary incentives—that impact the productivity of public sector workers and thereby government outputs and performance.18 24. Drawing on this literature, the survey design is based on a conceptual framework of a production function of youth mental health service delivery (Figure 1). A production function is the process by which “inputs,” or the human and financial resources available to a public sector organization (such as infrastructure, number of staff, and goods and materials), are converted to “outputs” and ultimately outcomes that citizens care about. As the scope of this report is on the human resource factors in service delivery, the conceptual framework emphasizes individual-level and organizational-level drivers that impact the performance of government workers in the entire youth mental health service delivery system, from administrators down to school personnel, which in turn impacts the quality of youth mental health services delivered. While different administrative levels have different functions, roles, responsibilities and interests, there are a set of common individual and organizational aspects that influence the actions and behaviors of the personnel in these units. 15 World Bank Group. (2018). The World Development Report 2018—Learning to Realize Education’s Promise. World Bank Group. https://www.worldbank.org/en/publication/wdr2018 16 The service delivery chain depicting this is given in Annex 2 17 World Bank Group. (2020). Protecting, Promoting and Accelerating Health Gains for Women, Children and Adolescents: Global Financing Facility 2021-2025 Strategy. World Bank Group. https://www.globalfinancingfacility.org/sites/gff_new/files/documents/GFF-Strategy-2021-2025.pdf 18 Besley, T.J., Burgess, R., Khan, A., & Xu, G. (2021). Bureaucracy and Development. NBER Working Paper 29163. DOI 10.3386/w29163. Finan, F., Olken, B.A., & Pande, R. (2017). The Personnel Economics of the Developing State. Handbook of Economic Field Experiments- Elsevier, 2(6), 467-514. https://doi.org/10.1016/bs.hefe.2016.08.001 5 Figure 1: The production function for mental health services for the youth Source: Authors 25. The conceptual framework emphasizes five personnel drivers across these administrative levels: • The attitudes of public sector workers. A large economics, psychology and public administration literature has shown that it is the combination of intrinsic and pro-social motivation—the desire to work hard for the enjoyment of the task itself and the desire to exert effort for the benefit of others and not for oneself—that drives the performance of public sector workers.19 As such, positive attitudes and behaviors—motivation, job satisfaction, engagement—are a good proxy measure of staff effort given the unique difficulties of monitoring the daily activities of public sector employees. In the context of mental health, stigma and bias towards those exhibiting these challenges are an especially important attitude that impacts the effort of public sector workers and the likelihood of them assisting youth. • The knowledge and skills of public sector workers are obvious determinants of their productivity, particularly for a complex challenge like mental health. • Relatedly, the extent of capacity building activities to both upgrade knowledge and skills, and also remove mental health stigma, will impact the quality of the service delivery. • The work environment of an organization, recognizing that service delivery is a group output that requires collaboration within schools, and between schools and the public administration. It is well known that investments in individual training in the absence of a conducive work environment that incentivizes and enables staff to apply their knowledge to collective work are likely to be wasted.20 Therefore, teamwork, operating protocols, and trust are all important aspects of organizational culture that impact individual knowledge and attitudes, and government outputs, especially for mental health services given their cross-institutional nature. 19 Perry, J. L. (2020). Managing Organizations to Sustain Passion for Public Service . Cambridge University Press. 20 Ibid. 6 • Management quality, like goal setting, regular monitoring of the achievement of targets, and incentives, both monetary and non-monetary, for staff to be performance-oriented impact the productivity of a variety of public sector organizations such as schools, hospitals, and social security offices.21 26. There are several factors that are excluded from the framework for pragmatic reasons. Notably absent is public sector pay as that is a complex reform with significant fiscal implications; staffing levels, for similar resource availability reasons; and organizational structure and formal inter-institutional coordination mechanisms which were not within the scope of this study. The framework also excludes a host of socioeconomic factors in the home environment of youth that impact mental health, or the broader political context that impact accountability relationships between public officials. The logic of the framework, and the survey, is to identify a narrower set of factors that can be relatively easily acted upon within available resources; and which are within the locus of control of administrative units thereby potentially yielding service delivery improvements in the short and medium term. 27. Following this conceptual framework, the survey collects data from all three administrative levels in the education service delivery chain that are responsible for mental health service delivery. Schools are the frontline layer of the service delivery chain, with teachers, school psychologists, and principals supporting students in identifying and helping respond to mental health challenges. As noted earlier, Lithuania’s legislation mandates that schools have the main responsibility for awareness raising, prevention, early detection, and referral for mental health problems of students, and qualitative studies by STRATA reinforce the importance of school staff in delivering mental health services.22 The academic and policy literature also underscores the importance of school staff in identifying early stages of mental illness and referring children to appropriate providers.23 School staff can also provide feedback on whether a mental health program or treatment is working or not.24 Barriers to accessing mental health 21 Bloom, N. & Reenen, J. V. (2007). Measuring and Explaining Management Practices Across Firms and Countries. The Quarterly Journal of Economics, 122(4), 1351–1408. 22 STRATA, 2020 23 Rothì, D. M., Leavey, G., & Best, R. (2008). On the Front-line: Teachers as Active Observers of Pupils’ Mental Health. Teaching and Teacher Education, 24(5), 1217. https://doi.org/10.1016/j.tate.2007.09.011 Reinke, W. M., Stormont, M., Herman, K. C., Puri, R., & Goel, N. (2011). Supporting children's mental health in schools: Teacher perceptions of needs, roles, and barriers. School Psychology Quarterly, 26(1), 1–13. https://doi.org/10.1037/a0022714Powers, J. D., Wegmann, K. M., Blackman, K. F., & Swick, D. C. (2014). Increasing Awareness of Child Mental Health Issues among Elementary School Staff. Families in Society: The Journal of Contemporary Human Services, 95(1): 43. DOI:10.1606/1044-3894.2014.95.6 24 Hoover, S. A., & Mayworm, A. M. (2017). The Benefits of School Mental Health. In K. D. Michael & J. P. Jameson (Eds.), Handbook of Rural School Mental Health . Springer International Publishing/Springer Nature, 3-16. https://doi.org/10.1007/978-3-319-64735-7_1 7 services can be removed by effective partnerships between schools and mental health agencies, leading to improved standards and range of mental health services for young people.25 28. Public administrators regulate, finance, and monitor schools. Most of Lithuania’s schools are municipal, and municipal administrations are responsible for managing schools, including hiring of school principals, setting performance targets for schools and principals, and for monitoring achievement of those targets, including through the individual performance evaluations of principals. Municipal public administrators may also target financial and human resources to support schools in their provision of youth mental health services. MoESS monitors education and coordinates the activities of the education units of municipal administrations, including approving regulations for the training, qualification raising, certification and performance evaluation of school principals, teachers, and education support specialists. The mental health support to schools also extends to the health and social security administrations. The MoH and PHBs have responsibilities in programming of mental health activities in schools and in providing capacity building to school staff, and the MoSSL deals indirectly with youth mental health through managing auxiliary institutions. By including school staff and public administrators in the research design, the two most directly influential layers of government for direct provision of services are included in the analysis.26 2.2 Survey design 29. The survey design followed a stratified sample of first, selecting the appropriate public sector organizations and second, selecting a representative sample of personnel within those organizations (Annex 3 provides the details). The survey was implemented in 83 public administration organizations and 230 schools. The public administration organizations included the MoH, the MoESS, the MoSSL at the national level; and municipal administrations at the subnational level. Forty three out of the 60 municipalities of Lithuania were selected—Vilnius City Municipality was selected directly, considering its size, and the remaining 42 municipalities were selected at random and are distributed across the country (Figure 2)27. The organizations interviewed in each municipal administration included the PHBs, which are the municipal arm of the MoH responsible for a variety of public health services; and the education, finance and social services and support administrations which all play an important role in youth mental health service delivery. Within these 43 municipalities, 243 schools28 were randomly selected from the subset of schools teaching at least one of grades 9–12 (I – IV gymnasiums). For representativeness, 5 to 7 25 Fazel, M., Hoagwood, K., Stephan, S., Ford, T. (2014). Mental Health Interventions in Schools 1: Mental Health Interventions in Schools in High-income Countries. Lancet Psychiatry, 1(5), 377-387. doi: 10.1016/S2215- 0366(14)70312-8. PMID: 26114092; PMCID: PMC4477835 Stephan, S. H., Sugai, G., Lever, N., Connors, E. (2015). Strategies for Integrating Mental Health into Schools via a Multitiered System of Support. Child and Adolescent Psychiatric Clinics of North America, 24(2), 211-231. https://doi.org/10.1016/j.chc.2014.12.002 26 STRATA (2020) details how these layers interact with each other in further detail 27 A list of the municipalities covered in the survey is given in Annex 4. 28 Even though 243 schools were sample, the survey was implemented in 230 schools as mentioned earlier in the paragraph. 8 schools were selected from each municipality according to municipality size. In municipalities where there were less than five schools, all schools were selected. Figure 2: The survey covered 43 out of the 60 municipalities in Lithuania (municipalities surveyed depicted in blue) Source: WB survey 30. Interviews were conducted for 1982 government employees across these organizations, of which 956 were in the public administration, and 1026 in schools. In each selected ministry and agency, 40 employees were sampled to interview, and in each PHB all employees were sampled to participate in the survey. Between 12 and 22 employees in each municipality were sampled, based on its size. In each school, a principal, one support specialist (where possible, a psychologist, if not a social pedagogue or another special education specialist, in that order of preference) and three randomly selected teachers were sampled. In general, there was a distribution of managers and non-managers in the survey, with 171 managers in the public administration survey and 209 principals in the school survey. 31. The report focuses on public administration staff who directly service the youth mental health policies and programs. To describe public administration survey results, this report focuses on employees whose work is directly related to mental health service delivery in schools. At the ministry level, these are the MoESS and the MoH, as well as the daughter agency of the MoESS, the National Agency for Education (NAE), which has responsibilities in field of educational data, monitoring and research. 29 There is a similar focus on health and education employees within municipalities and PHBs. Taken together, around 47 percent of the public administration respondents surveyed are directly related to the issues of mental health challenges in schools, as these employees work in the health or education sectors, and the survey results on public administrators presented in the next chapter are limited to these core staff. 32. The data collection was adapted in response to the challenges posed by COVID-19. The survey was conducted from March 2021 to October 2021 via secure, encrypted video calls between the 29 STRATA, 2020 9 enumerator and respondent, and each interview lasted approximately one hour. The WB used Microsoft Teams to ensure that these interviews remained end-to-end encrypted. Contacts for the target population were resourced from the internet, as such information is freely available online. Institutional approvals were obtained for these surveys and only after the approvals were received did the survey team reach out to respondents individually via email with an introduction and scheduling options. Upon confirmation, the survey team sent respondents a call link, consent form, and more information about the study. The survey received a very high response rate of 82.6 percent. Digital literacy, as well as access to technology and the internet by the respondents are the prime factors which led to the success of this mode of survey. 33. The survey was designed through a collaborative and iterative process between the WB and Government counterparts, particularly STRATA, MoH, MoESS, and MoSSL. The WB participated in several workshops, organized by STRATA, with government partners and non-governmental organizations, which helped develop the conceptual framework and identify survey modules and questions. Government counterparts also provided help in implementing the survey by providing the necessary approvals and encouraging participation. 34. A mixed methods approach served a guide to the analysis. A research approach using qualitative information to complement quantitative information makes diagnosis holistic and helps in understanding nuances of softer topics. The analysis outlined in this report was undertaken in close collaboration with STRATA. The STRATA and WB teams jointly advised one another on the undertaking of complementary qualitative process work (STRATA) and quantitative survey work (the WB). STRATA’s report and recommendations broadly focus on recognizing the challenges and finding solutions to the youth mental health delivery system30, a theme that broadly resonates in this report. 35. The report also outlines the results of an impact evaluation on mental health training. Prior to and during baseline survey data collection, an RCT was conducted which aimed to improve uptake for an existing MoH online mental health training program and measure the effects of information and messenger identity on uptake and completion of the MoH training. 30 STRATA, 2020 10 Chapter 3: Main findings 36. This chapter lays out the main findings from the WB study and follows the conceptual framework detailed in Chapter 2. The first section documents mental health service delivery outputs and outcomes—whether schools are adequately providing mental health services, the output of the production function conceptual framework, and the prevalence of mental health challenges amongst the youth. The second section lays out the resources available to the surveyed respondents. The third and final section then details each of the personnel and organizational drivers of the service delivery chain— attitudes, knowledge, capacity building, work environment, and the quality of management. 37. To summarize, the main findings are: • School staff report that a high share of adolescents experience mental health challenges in Lithuania, but very few students receive any help for these problems. • Mental health stigma is prevalent among school staff and public administrators. Males and older employees exhibit higher levels of stigma. Stigma is associated with a lower likelihood of identifying mental health challenges and lower quality of school services. • Both school staff and public administrators are motivated to work on tasks that benefit others, suggesting that it is stigma, not general motivation, which is the main attitudinal challenge. Notably though, general motivation declines with age and a significant minority of public administrators are searching for other jobs. • Knowledge about mental health challenges is low amongst school staff and public administrators. Knowledge is also correlated with stigma. • Most school staff are unaware of the youth mental health training offered by the MoH, but those who are aware evaluate these training programs positively. Similarly, a majority of school principals report that in the past 12 months, their school did not participate in the MoH youth mental health training program. School staff who spend more time in training have lower levels of stigma, report a higher prevalence of student mental health issues, and perceive higher levels of mental health action taken by their school. Prevention programs implemented by schools also have a positive impact on school staff and students. • There is a low level of collaboration on mental health policies between staff within schools, between schools and municipalities, and between public administrators and their counterparts in other government departments. • Management quality varies across schools, and amongst different public administration organizations and units. Management quality in schools is associated with the mental health education and support that schools offer to their students. 11 38. For several measures, indices were constructed from several survey questions to provide a standardized measure of a driver, output, or outcome. The procedure for generating these indices is fairly standard. Each index is constructed by first standardizing each individual response to either the school or the overall public administration sample mean by subtracting the mean from each individual response and dividing by the standard deviation. This implies that the responses of school personnel are compared to other school personnel, and public administrators are compared to other public administrators. A composite index is then constructed by summing over a few questions that measure that belief or practice and is divided by that same number. A higher level of any index indicates a stronger belief or practice. For example, a higher stigma index indicates that the individual holds higher levels of stigma than the average in their subpopulation. These indices help measure more complex relationships and better map out the service delivery framework. 3.1 Student mental health outcomes and outputs 39. Mental health challenges are pervasive amongst students in Lithuania, with significant variation across municipalities and schools. In the survey, school staff estimate that an average of 20 percent of all the students in grades 9-12 (or 9-10, depending on the school) experienced some form of mental health challenge in the fall term of 2020. They also estimated that, on average, 4 percent of the same population of students were formally diagnosed with mental health disorders. These numbers are slightly below those reported in the administrative data collected from the Švietimo Valdymo Informacinė Sistema (ŠVIS)—the official education data portal—which estimates that 7 percent of students were formally diagnosed with mental health challenges in 2020. There is some municipal, and significant school level, variation in these responses, as seen in Figures 3, 4, 5, and 6. For example, the share of students experiencing mental health challenges ranges from a high of over 60 percent in some schools to a low of less than 5 percent, with similar variation in the percentages with formal diagnosis. Figure 3: There is municipality level variation in the share of students that experienced some form of mental health challenges in the fall term of 2020 (school staff estimates) Source: WB survey 12 Figure 4: There is school level variation in the share of students that experienced some form of mental health challenges in the fall term of 2020 (school staff estimates) Source: WB survey Figure 5: There is municipality level variation in the share of students that were formally diagnosed with mental health disorders in the fall term of 2020 (school staff estimates) Source: WB survey 13 Figure 6: There is school level variation in the share of students that were formally diagnosed with mental health disorders in the fall term of 2020 (school staff estimates) Source: WB survey 40. Most students do not receive help to address these considerable mental health challenges. School staff report that only 12 percent of students received any help from their school, 64 percent received education on mental health challenges and mental health, and 19 percent of these students visited the school psychologist in the past 12 months. While students may not have received help, they were being regularly referred to school psychologists. Over the same period, 65 percent of teachers report referring a student to a school psychologist for an overall average of 5 students being referred per teacher. When these referrals are adjusted for the number of students in each school, there is significant variation in them, from a high referral of 30 times per teacher to no referrals in the past 12 months. (Figure 7). Figure 7: There is a significant school level variation in the number of times teachers referred students to the school psychologist in the past 12 months. Source: WB survey 14 41. These survey results were used to produce mental health outcome and output indices to provide summary measures and to explore correlations between drivers and outputs and outcomes, as detailed in the next section. A Mental Health Prevalence Index was constructed from the two outcome variables of the share of students that experienced some form of mental health challenges, and the share that were formally diagnosed with mental health challenges. A higher level of this index means that school staff estimate a greater incidence of mental challenges amongst students. This index varies considerably across schools, with a minimum score of −0.68 and a maximum of 3.70. Similarly, a Mental Health Action Index was constructed from the survey questions on the share of students receiving help from their school, receiving education on mental health, and visiting the school psychologist. A higher level of this index indicates a stronger perceived school response to mental health challenges, that is, better quality of service delivery. This index also varies significantly across schools, with some schools displaying a high of 2.14 to others scoring as low as −0.97. 42. The report uses these survey-based indices for analysis instead of administrative data, for two reasons. For most of the variables of interest, administrative data only exists at the school level, whereas the survey data is more granular and at the individual level, which enables more robust correlations as there are a larger number of observations. Secondly, where comparisons are possible, the survey estimates are similar to those derived from the administrative data, which provides confidence in this approach. For instance, the average age of the school survey sample is 50.5 years old, compared to 51 in the administrative data, and the average tenure for school education staff is 24 years in the survey, and 25 in the administrative data. At the school-level, the relationship between age and the proportion of students estimated to be diagnosed with mental health issues estimated by the within-survey and the administrative data relationship is slightly positive and close to 0. A similar pattern is observed with tenure. Moreover, the survey-based estimates of mental health outcomes are correlated with those from the administrative data. 3.2 Inputs 43. The staffing profile of schools has some implications for mental health service delivery. While this study focuses on the organizational and personnel practices, and attitudes and behaviors of public sector personnel, rather than resources and personnel inputs, some aspects of school staffing do stand out. In general school staff are well qualified, with 46 percent of them having an undergraduate degree and 52 percent a graduate degree. Staff are also stable in their careers, as 80 percent of the respondents are permanent employees who have been at their current school for the past 17 years. There is, however, a significant shortage of school psychologists. According to the ŠVIS data, 47 percent of schools in Lithuania have psychologists; however, the study surveyed a psychologist in only 30 percent of the schools which suggests that most schools lack a psychologist. Since psychologists are the primary mental health providers in schools this can have noteworthy impacts on youth mental health31. The average age of school personnel, at approximately 51 years, is relatively high, which has implications for mental health stigma, as discussed in the next section. Finally, men are overly represented in managerial positions— 31 STRATA, 2020 15 roughly 16 percent of all school respondents are men, but 30 percent of all managers are men—and this gender distribution also has implications for stigma. 44. School staff do not believe that there are resource constraints in effectively delivering mental health services. About 49 percent of the school staff agree, and 35 percent neither agree nor disagree, with the statement “I have sufficient tools, resources, and knowledge to effectively respond to mental health challenges of students”. A majority of respondents also state that there are a sufficient number of staff to meet the mental health needs of students in their school, which is somewhat surprising given the shortage of psychologists. These self-assessments of capacity though may reflect an under-estimation of mental health challenges, and mental health stigma, and weaknesses in within-school collaboration, as detailed below. 45. Administrators and school staff report several challenges in working remotely because of COVID-19. Both school staff and public administration staff report that a significant proportion of their work cannot be done remotely—school staff state that 28 percent of their workplace tasks cannot be done remotely, as compared to 22 percent for administrators. The major challenges for remote work for school staff include poor teamwork and collaboration, lack of management support, and security concerns related to working online (Figure 8). For public administrators, a major challenge in undertaking core tasks remotely is that the nature of their work requires regular face-to-face interaction. Figure 8: School staff face major challenges while working from home, with the major one being poor quality of collaboration while working remotely Source: WB survey 16 3.3 Organizational and personnel drivers of the youth mental health service delivery chain 3.3.1. Attitudes: Stigma and motivation 46. Mental health conditions can often be stigmatized and trivialized. 32 Mental health stigma is a significant barrier to seeking and obtaining treatment and impacts the trajectory and aftermath of mental illnesses.33 Stigma and prejudice can also have severe economic impacts, hurting income, employment opportunities, public perceptions of resource allocation, and healthcare expenses.34 Stigma can prevent adolescents from seeking help and create many barriers which may lead to social exclusion and retardation of growth. In a study, adolescents reported that the dread of being stigmatized was a major barrier in seeking mental health assistance at school.35 Young adults who divulge information about their mental health report stigmatizing behavior from school staff and peers.36 Moreover, significant stigma exists amongst mental health providers.37 Because young children and adolescents spend so much of their time in school, school employees invariably become the first point of contact for students in need of advice and aid for their mental health.38 Hence, stigma among school staff may lead to negative effects on students’ mental health. 47. Given that stigma is difficult to measure, the survey used several questions, drawing on academic literature, to generate reliable estimates (Box 1). Some questions were asked in the third 32 Robinson, P., Turk, D., Jilka, S. & Cella, M. (2019). Measuring Attitudes Towards Mental Health Using Social Media: Investigating Stigma and Trivialization. Social Psychiatry and Psychiatric Epidemiology, 54, 51–58. https://doi.org/10.1007/s00127-018-1571-5 33 Ahmedani B. K. (2011). Mental Health Stigma: Society, Individuals, and the Profession. Journal of social work values and ethics, 8(2), 41–416. Sickel, A. E., Seacat, J. D. & Nabors, N. A. (2015). Mental Health Stigma Update: A Review of Consequences. Advances in Mental Health: Promotion, Prevention and Early Intervention , 12(3), 202-215. https://doi.org/10.1080/18374905.2014.11081898 Corrigan, P. W. (2000). Mental Health Stigma as Social Attribution: Implications for Research Methods and Attitude Change. Clinical Psychology: Science and Practice, 7(1), 48–67. https://doi.org/10.1093/clipsy.7.1.48 34 Sharac, J., Mccrone, P., Clement, S., & Thornicroft, G. (2010). The economic impact of mental health stigma and discrimination: A systematic review. Epidemiologia E Psichiatria Sociale, 19(3), 223-232. doi:10.1017/S1121189X00001159 35 Bowers, H., Manion, I., Papadopoulos, D. & Gauvreau, E. (2012). Stigma in School-based Mental Health: Perceptions of Young People and Service Providers. Child and Adolescent Mental Health, 18(3), 165-170. https://doi.org/10.1111/j.1475-3588.2012.00673.x 36 Moses, T. (2010). Being Treated Differently: Stigma Experiences with Family, Peers, and School Staff Among Adolescents with Mental Health Disorders. Social Science Medicine, 70(7), 985-993. doi: 10.1016/j.socscimed.2009.12.022 37 Star, L., Mulgrew, L., Akroyd, S., Sinai, H., Goodman, K. & Wyllie, A. (2005). ”Like Minds Like Mine”: Research with Mental Health Service Providers. Ministry of Health, New Zealand. https://www.likeminds.org.nz/assets/Uploads/research-with-mental-health-serviceproviders.pdf 38 Abdinasir, K. (2019, October 22). How Education Shapes Young People‘s Mental Health. Centre for Mental Health. https://www.centreformentalhealth.org.uk/publications/making-grade 17 person to avoid social desirability bias.39 Multiple questions measuring various standpoints on stigma were asked to negate the effect of answers which could have been affected due to interpersonal factors. For example, questions referring to close friends or colleagues may be received with hesitation. Hence, other questions were asked to address such hesitation (if any, at all). A composite index of all such stigma questions (explained later in this section) was then used to study relationships of stigma with age, prevalence of mental health issues in adolescents, and others. Box 1: Measuring stigma The WB survey draws on the academic literature to identify questions with which to measure stigma. Studies have used a wide spectrum of mental illnesses, like schizophrenia, depression, bipolar disorder, anxiety disorder, and other emotional and behavioral disorders, to construct mental illness stigma scales. 40 Some studies investigate stigma of frontline psychiatric and psychological staff as well and were particularly relevant for the survey design.41 In these studies, measures of mental illness stigma include questions like “ people with mental illness living in the community would endanger local residents”, “children with emotional and behavior disorders would hurt themselves or other children”, “I worry about telling people I receive psychological treatment”, and “some people with mental health problems are dangerous.”42 The questions that the WB survey measure stigma by association, help seeking behavior, and social impairment, all of which have been included in studies for both adults and children. Drawing on this literature, the questions on stigma in the survey are: 1. If a close friend experienced mental health challenges, would they want it to remain a secret or not? 2. Now consider your colleagues in your institution. If ten of these colleagues were chosen at random, how many out of these ten do you think would want it to remain a secret if they experienced mental health challenges? 3. Please indicate the degree to which you agree or disagree with the following statements. a. It's best not to talk about mental health challenges. b. Having any indication of mental health challenges, such as inability to rest, worry, depression, or anxiety is a sign of personal failure. c. Individuals with mental health challenges are more dangerous to other people than the average person 39 Development Impact Evaluation Wiki. (n.d.). Sensitive Topics. World Bank Group. https://dimewiki.worldbank.org/Sensitive_Topics 40 King, M., Sokratis, D., Shaw, J., Watson, R., Stevens, S., Passetti, F., Weich, S., & Serfaty, M. (2007). The Stigma Scale: Development of a Standardized Measure of the Stigma of Mental Illness. The British Journal of Psychiatry, 190, 248–254. DOI: 10.1192/bjp.bp.106.024638 41 Chen, M.-D., & Chang, Y.-C. (2016). Personnel Attitudes toward People with Mental Illness at a Psychiatric Hospital in Taiwan. International Journal of Social Psychiatry, 62(4), 361–368. https://doi.org/10.1177/0020764016634385 42 Ma, H.-I., & Hsieh, C.-E. (2020). Questionnaires on Stigmatizing Attitudes among Healthcare Students in Taiwan: Development and Validation. BMC Medical Education, 20(1). https://doi.org/10.1186/s12909-020-1976-1 McKeague, L., Hennessy, E., O'Driscoll, C., & Heary, C. (2015). Peer Mental Health Stigmatization Scale: Psychometric Properties of a Questionnaire for Children and Adolescents. Child and Adolescent Mental Health, 20(3), 163–170. https://doi.org/10.1111/camh.12088 18 48. The survey finds that there is considerable mental health stigma amongst school staff in Lithuania. An overwhelming 83 percent of school personnel report that if one of their close friends experienced mental health challenges, they would want it to remain a secret. When asked “If ten of your colleagues were chosen at random, how many out of these ten do you think would want it to remain a secret if they experienced mental health challenges?”, respondents report that, on an average, 7.5 out of 10 colleagues would want it to remain a secret.43 More than a quarter (26 percent) of school staff agree with the statement “It's best not to talk about mental health challenges.” Around 17 percent of the school staff agrees that having any indication of mental health challenges, such as inability to rest, worrying, depression, or anxiety is a sign of personal failure. Alarmingly, as seen in Figure 9, the majority does not disagree with the statement “individuals with mental health challenges are more dangerous to other people than the average person”. Figure 9: A majority of respondents do not disagree with the statement - Individuals with mental health challenges are more dangerous to other people than the average person (school staff estimates) Source: WB survey 49. The prevalence of stigma is higher among males and older staff. Given that there are several ways of measuring stigma, the five survey questions (detailed in Box 1) were combined to construct an overall Stigma Index to capture average beliefs. Male school staff had higher scores on the stigma index with an average of 0.16 compared to female staff who scored –0.03. This is especially problematic since males are disproportionately represented in managerial level positions, with males comprising 16 percent of the overall school sample but 30 percent of all school managers. The extent of stigma amongst school 43 Academic literature has validated the use of such questions to measure stigma rather than, for example, privacy. For example, “I am scared of how other people will react if they find out about my mental health problems” in King et al 2007; and “I won’t let people know if there is a person with mental illness in my family” in Ma and Hsieh 2020 19 staff is positively correlated with age, with older individuals having a higher stigma index (Figure 10). There was no difference in stigma between permanent staff and contractual staff. Figure 10: Older school staff have more mental health stigma Source: WB survey 50. Stigma is associated with a lower likelihood of identifying mental health challenges and lower quality of school services. There is a negative correlation between the Mental Health Prevalence Index and the Stigma Index (Figure 11) implying that school staff that hold more stigmatized beliefs indicate less prevalence of student mental health issues. Similarly, there is a negative correlation between the Stigma Index and the Mental Health Action Index, with respondents who have higher stigma perceiving their schools as acting less on mental health issues of students, though the association is less robust and more suggestive (Figure 12). The correlation may also mean that that schools where staff perceive less school action on mental health issues have staff who have more stigma. 20 Figure 11: Stigma is negatively associated with identifying youth mental health challenges Source: WB survey Figure 12: Stigma is negatively associated with the perceived school response to student mental health challenges Source: WB survey 21 51. The prevalence of stigma varies across schools and municipalities. The minimum value of stigma index constructed for schools (calculated as an average of individual school staff stigma index for each school) is –1.04 and the maximum is 1.09, centered at 0. For municipalities, accounting for both public administration staff and school staff, the minimum stigma index is –0.46 and the maximum stigma index is 0.19, centered at 0. This variation is depicted in Figure 13, revealing that stigma is higher in some schools and municipalities than in others. Figure 13: Variation exists in the stigma index for municipalities in Lithuania (schools and public administration) Source: WB survey 52. Public administration employees also exhibit high levels of stigma, though slightly lower than school staff. Respondents report that, on average, 8 out of 10 of their colleagues would want their mental health challenges to remain a secret and 78 percent of respondents report that if one of their close friends experienced mental health challenges, they would want it to remain a secret. Thirty-one percent of the respondents agree or strongly agree that it is best not to talk about mental health challenges, with the agreement strongest in municipalities (Figure 14). Fifteen percent respondents agree that having any indication of mental health challenges, such as inability to rest, worrying, depression, or anxiety is a sign of personal failure. Some 16 percent of the respondents agree, and 30 percent do not disagree with the statement “Individuals with mental health challenges are more dangerous to other people than the average person”. Similar to school staff, male and older public administration employees have higher levels of stigma (see Figure 15 on the correlation of the stigma index and age). 22 Figure 14: Different tiers of public administration have notably different responses to the statement - It's best not to talk about mental health challenges Source: WB survey Figure 15: Older public administration staff have more stigma Source: WB survey 53. School staff report high motivation to assist students. Given the difficulties in monitoring effort and output in public sector jobs, public service motivation, seeing the purpose and value of work and commitment to remain in the public sector are key determinants of individual and organizational 23 performance44. Beyond compensation, civil servant’s day-to-day decisions and choices, and hence performance, are also shaped by feelings such as professional pride and societal responsibility45. While stigma is high, general motivation is also high, which suggests that addressing stigma is the main attitudinal change that is needed among school staff. Eighty percent agree or strongly agree that they feel motivated to effectively respond to mental health challenges of students. Forty percent agree and 55 percent strongly agree that they get motivated to work on tasks that have the potential to benefit others. 54. Motivation for public administrators decreases with experience. Although 97 percent of public administrators agree or strongly agree that they are motivated to work on tasks that benefit others, 57 percent report that their motivation was lower relative to what it was when they started in the public administration. This is due to their wages, limited promotion opportunities, lack of quality management, limited opportunity to contribute to impactful decisions, limited responsibility and working environment. Only 19 percent of the respondents have higher motivation than what it was when they started in public administration, and this is because they find their work interesting and challenging, have good relationships with their team and have skills which help them to thrive in their present roles. Twenty-four percent of the employees have the same motivation as when they joined their role. 55. Expectedly, some public administration employees seek to leave their organization. Ten percent of the respondents actively sought to leave their organization in the near future at the time of the survey. Interestingly, this figure was highest at the ministry level, where 16 percent of the respondents reported that they were actively seeking to leave (Figure 16). 44 Cantarelli, P., Belardinelli, P., & Belle, N. (2016). A Meta-Analysis of Job Satisfaction Correlates in the Public Administration Literature. Review of Public Personnel Administration, 36(2), 115–144. https://doi.org/10.1177/0734371X15578534 Dixit, A. (2002). Incentives and Organizations in the Public Sector: An Interpretative Review. The Journal of Human Resources, 37(4), 696–727. https://doi.org/10.2307/3069614 Ewenstein, B., Hancock, B., & Komm, A. (2016). Ahead of the Curve: The Future of Performance Management . McKinsey Quarterly. https://www.mckinsey.com/business-functions/people-and-organizational-performance/our- insights/ahead-of-the-curve-the-future-of-performance-management 45 Guszcza, J. (2016). HR for Humans: How Behavioral Economics Can Reinvent HR . Deloitte Review Issue 18. https://www2.deloitte.com/us/en/insights/deloitte-review/issue-18/behavioral-economics-evidence-based-hr- management.html 24 Figure 16: A majority of public administrators do not seek to leave their organization in the near future Source: WB survey 3.3.2 Knowledge 56. School staff have little knowledge of mental health issues nationally. The survey measures knowledge with reference to staff awareness of the findings of a major national health study that is conducted in Lithuania every four years, the HBSC study. The HBSC study was chosen as a reference as it is used by the MoH to track children’s physical and mental health, with the last study completed in 2018. Some 92 percent of school staff, and 89 percent of school psychologists, were not even aware of the HBSC study. Moreover, being aware about the study does not make school staff more accurate in their estimates of mental health challenges. School staff who knew about the study estimated that 11 percent of students had thought about suicide compared to the actual figure of 24 percent reported in the study. School staff were also widely inaccurate in their answers to questions like ‘what share of these (HBSC surveyed) students did not feel satisfied by life?’, ‘what share of these (HBSC surveyed) students experienced bullying at school?’ and ‘what share of these (HBSC surveyed) students did not feel sufficiently supported by teachers?’ Figure 18 shows graphs for these responses, with the actual statistics from the HBSC depicted as a vertical red line. 25 Figure 17: School staff have low knowledge of the HBSC study and report a high variation in responses to HBSC specific questions Source: WB survey 57. Low knowledge of national mental health issues is positively correlated with stigmatized beliefs. The research team constructed an indicator on knowledge of mental health which is measured as the proportion of answers within 5 percentage points of the actual statistic from the HBSC for the questions regarding national mental health discussed above. Figure 19 shows that this indicator is negatively correlated with the stigma index, that is, school staff with low knowledge of national mental health issues report more stigmatized beliefs. Figure 18: Lower knowledge of youth mental health issues amongst school staff is associated with stigma Source: WB survey 26 58. Public administration staff also have little knowledge of national mental health issues. About 85 percent of administrators were not aware of the HBSC study. There is some variation in this finding within the tiers of government, where more than a quarter of the workers in PHBs being aware of the study. These staff are also inaccurate in their estimates of the findings of the study, with respondents stating that, on average, 13 percent of students surveyed in the HBSC study contemplated suicide. A very small percentage of administers correctly estimated the answers to the other survey questions. 3.3.3 Capacity building 59. Given this low level of knowledge, mental health training is important for school staff to improve their awareness of mental health challenges and to reduce stigma. Since teachers and other school staff play a significant role in prevention and identification of mental health challenges amongst school children, it is critical to ensure that teachers have the necessary knowledge, competence, and abilities to reduce the effects of mental illness on education and to execute their job adequately46. Studies show that high school teachers’ training improves their knowledge, reduces stigma, and shifts their attitude toward treatment to more closely resemble those of mental health experts. Students also reported obtaining more mental health information from the school47. Other studies also show that mental health training amongst schoolteachers can improve mental health literacy, knowledge, and attitudes48 Training can assist school staff in identifying students with mental illnesses and referring them to suitable assistance49. 60. The MoH provides training to school staff on mental health. To facilitate the process of implementation of the National Public Health Care Development Program for 2016–2023, the MoH approved the Procedure for Increasing the Competence of School Staff in the Field of Mental Health. This legal act provides for training on the competence of school staff in the field of mental health. The aim of 46 Whitley, J., Smith, J. D., & Vaillancourt, T. (2013). Promoting Mental Health Literacy Among Educators: Critical in School-Based Prevention and Intervention. Canadian Journal of School Psychology, 28(1), 56–70. https://doi.org/10.1177/0829573512468852 47 Jorm, A.F., Kitchener, B.A., Sawyer, M.G., Scales, H., & Cevtkovski, S. (2010). Mental Health First Aid Training for High School Teachers: A Cluster Randomized Trial. BMC Psychiatry, 28(1), 56-70. https://doi.org/10.1186/1471- 244X-10-51 48 Carr, W., Wei, Y., Kutcher, S., & Heffernan, A. (2018). Preparing for the Classroom: Mental Health Knowledge Improvement, Stigma Reduction and Enhanced Help-Seeking Efficacy in Canadian Preservice Teachers. Canadian Journal of School Psychology, 33(4), 314–326. https://doi.org/10.1177/0829573516688596 Kutcher, S., Wei, Y., McLuckie, A. & Bullock, L. (2013). Educator Mental Health Literacy: A Program Evaluation of the Teacher Training Education on the Mental Health & High School Curriculum Guide. Advances in School Mental Health Promotion, 6(3), 83-93. https://doi.org/10.1080/1754730X.2013.784615 Wei, Y., Kutcher, S., Hines, H. & MacKay, A. (2014). Successfully Embedding Mental Health Literacy into Canadian Classroom Curriculum by Building on Existing Educator Competencies and School Structures: The Mental Health and High School Curriculum Guide for Secondary Schools in Nova Scotia. Literacy Information and Computer Education Journal, 5(3). 49 Wei, Y., & Kutcher, S. (2014). Innovations in Practice: ‘Go -to’ Educator Training on The Mental Health Competencies of Educators in the Secondary School Setting: A Program Evaluation. Child and Adolescent Mental Health, 19(3), 219-222. https://doi.org/10.1111/camh.12056 27 the training is to increase the knowledge of school staff about the mental health of children and adolescents and to strengthen their ability to respond to youth mental health challenges. This training occurs at various points throughout the year, and has an online component, as well as an in-person component to it. For some time during the COVID-19 pandemic, the training was conducted entirely online. 61. School staff have low awareness and uneven uptake of teacher training programs. The survey asked questions about intent to participate in future training (in 2021) and actual participation in training in the previous 12 months (in 2020). Some 71 percent of school workers are not familiar with the MoH youth mental health training program. For those that are familiar, 46 percent of the respondents planned to participate in only the online portion of the training, 25 percent in the full training (both the interactive portion as well as the online portion), while 29 percent did not plan to participate in the training at all. Eighty percent of the male respondents planned to participate in only the online portion of the training. Notably, there are negligible differences in these results when disaggregated by permanent staff and contractual staff members. There was also a wide variation in the number of reported hours that school staff spent in training in the past 12 months (Figure 20), ranging from 0 to almost a 100 hours, pointing to a lack of any established standard or enforcement of the amount expected of school staff. Figure 19: There is a variation in the number of reported hours of training for school staff Source: WB survey 62. School management also reports low utilization of such training. About 61 percent of school principals and vice-principals report that in the past 12 months, their school did not participate in the MoH youth mental health training program, further highlighting the need for more collaboration amongst the different arms of the government. There is some regional variation in the uptake of training which may be relevant for the MoH to gauge overall regional interest in their training programs (Figure 21). 28 Qualitative study by STRATA also found that there is some difference in how municipalities choose to implement central government directives, including training programs.50 Figure 20: There is regional variation in the uptake of MoH youth mental health training program amongst schools Source: WB survey 63. Staff report that training programs are useful to them. For those who planned to participate in the 2021 MoH youth mental health training, 85 percent of the respondents agree or strongly agree that the training would strengthen their ability to support student mental health needs and 81 percent of the respondents agree or strongly agree that their participation in this training was important to school administration. Overall a majority of the school staff agree that the past trainings provided them with relevant and useful knowledge and skills in the field of mental health, that they had been able to apply the concrete actions recommended by the training to solve problems relevant to their school community in the field of mental health, and as a result of the training, it had become easier to collaborate with other staff at their school in assisting students with mental health challenges. Some topics that were covered by the trainings for youth mental health that the respondents attended in the past 12 months (excluding prevention programs) were: how to teach socio-emotional skills to students, how to recognize early signs of mental health challenges in students, how to engage and work with families of students with mental health challenges, and how to maintain student mental well-being in the classroom. 64. School staff who spend more time in training have lower levels of stigma, report a higher prevalence of student mental health issues, and also perceive higher levels of mental health action taken by their school. Figure 22 shows that the stigma index is negatively correlated with the number of hours that school staff spent on youth mental health training in the past twelve months, with respondents that spent more time on youth mental health training scoring low on the mental health stigma index. Figure 23 and Figure 24 show that the hours of mental health training attended by school staff are positively correlated with the Mental Health Action Index and the Mental Health Prevalence Index. 50 STRATA, 2020 29 Though the precise mechanisms by which training impacts outputs is not clear, and given that training is voluntary there may also be selection effects as more knowledgeable and less stigmatized staff chose to participate in the training, it is still likely that training improves awareness of mental health challenges and reduces stigma, which in turn results in more action by school staff to help students. Figure 21: School staff who spend more time in training have lower levels of stigma Source: WB survey 30 Figure 22: School staff who spend more time in training perceive better mental health service delivery Source: WB survey Figure 23: School staff who spend more time in training report higher prevalence of student mental health issues Source: WB survey 31 65. An impact evaluation conducted as part of the study showed that branding and advertising training improved its uptake. Box 2 details these findings. Box 2: School staff strongly respond to communicated training opportunities: findings from an impact evaluation Through the WB’s partnership with the MoH, the team designed a field experiment (RCT) to test whether school staff react positively to reminders for mental health training in the absence of additional financial incentives, and to understand how such messages can be communicated most effectively. In 2021, the MoH started an online mental health training to improve competencies of school staff across Lithuania. The 8-hour, 8-module, online training is made available publicly, is self-paced, free and can be done at home. The WB partnered with the MoH to advertise this training to school administration and staff. In particular, 1014 schools in Lithuania were randomized into different treatment groups. The training was directly advertised to the treatment groups by sending the administrators of these schools two institutionally branded emails with otherwise identical content encouraging participation in the training and the sharing of the email to their school staff. The first group received a MoH branded email, the second group received a Lithuanian Student Union (LSU) branded email, and the third group received no emails. The main finding of the experiment is that both school administrators and school staff respond differentially to emails by the MoH and the LSU, and that both these emails are effective in increasing initial uptake, although the email by the MoH causes a higher number of staff to start training. Significantly, only the email by the MoH is strong enough to persist for training completion. Further details on the experiment are given in Annex 6. 66. Schools also implement several prevention programs aimed at reducing mental health risks for students. A school prevention program is a set of planned and systematic measures that embeds knowledge, life skills, attitudes and values in school communities which help to protect young people’s psycho-social health. Prevention programs in Lithuania have been institutionalized by the Lithuanian Law on Education 51 and have been implemented in Lithuania’s schools for more than a decade. During 2016- 2021, the European Social Fund funded implementation of prevention programs in 839 (out of the total of about 1,000) Lithuania’s schools.52 Schools were able to choose from 18 prevention programs recommended by the MoESS. Besides these mainstream prevention programs, schools implement various other prevention activities, funded from municipal budgets. According to school principals and vice- principals, the schools have implemented prevention programs a total of 633 times, averaging 3 programs per school at the time of the survey. 67. Prevention programs significantly increase the knowledge of school staff in prevention areas. School teachers, psychologists, social pedagogues, social workers and other administrative employees have a positive outlook on the prevention programs in their schools. There is significant agreement amongst school workers that prevention programs in their school have made it easier to talk openly about 51 Government of Lithuania. (1991). Law on Education of the Republic of Lithuania . Republic of Lithuania. https://e- seimas.lrs.lt/portal/legalAct/lt/TAD/TAIS.1480/asr 52 Government of Lithuania. (2021). Creating the Future of Lithuania: European Union Funds Investment Action Program for 2014-2020. Government of Lithuania. https://www.esinvesticijos.lt/lt/paraiskos_ir_projektai/saugios- aplinkos-mokykloje-kurimas-ii?fbclid=IwAR2etFXDenOvdlgZLrslynSPndtZLzzUBdc2_DAeeyuzphUqZIWoZWYW1jw 32 prevention themes within the school (81 percent agree or strongly agree) and have strengthened their skills and knowledge to effectively deal with issues in prevention areas (84 percent agree or strongly agree). 68. Staff only partially apply their knowledge and skills in practice, as evidenced by outcomes of prevention programs for students. About 51 percent of respondents perceive a significant impact of the programs on students with challenges in the areas addressed by prevention programs. Sixty-three percent of respondents agree or strongly agree that prevention programs have fulfilled their goals at their school. The most common reasons why prevention programs did not achieve their goals were: uninterested students, poor quality of programs, and insufficient cooperation amongst the school staff. 69. Prevention programs unevenly cover municipalities, pupils’ age groups and school community segments. The number of prevention programs amongst municipalities varied from an average of 1 to 8 per school. Primary school students make up about 36 percent of the total number of pupils in grades 1- 1253, but 58 percent of all implemented prevention programs aim at this age group54. The survey finds that parents and class teachers play a vital role in responding to youth mental health needs, however, prevention programs specifically targeting these segments of school community constitute under 2 percent of all implemented prevention programs. 70. Schools lack a strategic approach to prevention programs. In their choice of prevention programs, schools mostly rely on their own judgement of the program’s adequacy. The most frequently mentioned motives for choosing a particular prevention program were: suitability of the program for a particular age group, convergence with the school's priorities, selection of the program by the school community, and past effectiveness of the program. The choice of prevention programs was least influenced by the recommendations of public health offices, municipal administrations, and other schools. 71. The content of prevention programs does not sufficiently and efficiently cover the key components of mental health literacy. In the opinion of school principals and vice-principals, prevention programs do cover mental health topics, but to a varying degree. (In this respect, there is a consensus that 10 out of the 18 programs cover mental health topics). 3.3.4 Work environment: Collaboration and protocols 72. There are low levels of collaboration amongst school staff and between staff and parents. Collaboration is a salient element of an effective youth mental health delivery system55. Fifteen percent of the staff have never engaged with the school psychologist on matters of youth mental health in the past 12 months (this figure is for schools that have a psychologist). Only 34 percent of the school staff engaged with parents a few times regarding the mental health of their children in the past 12 months, 53 Statistics Lithuania: The Official Statistics Portal. (2020-2021). https://www.stat.gov.lt/en 54 The NAE gives guidelines on implementing prevention program s in schools according to the students’ age. More details here. 55 STRATA, 2020 33 and 16 percent never did so. Men are less collaborative, with 29 percent of males never engaging with a school psychologist in the past 12 months, and 28 percent not reaching out to parents regarding the mental health of their children. One striking finding indicating a lack of collaboration between school staff is the unawareness of whether the school even has a psychologist. Only in 36 percent of surveyed schools, all respondents uniformly agree whether the school does or does not have a psychologist. Furthermore, when this figure is compared with the ŠVIS data statistics from the previous year, only 32 percent of staff correctly identified whether there is a psychologist at their school. 73. There is also scope for more collaboration between schools and municipalities. For mental health services to be effectively planned and executed, it is important that there is coordination between the frontline providers (schools) and government authorities. This ensures that policies and plans are made with ample diffusion of knowledge between the two tiers. Although 65 percent of the respondents agree or strongly agree that municipalities carry out their work effectively, and 77 percent agree or strongly agree that leadership in their municipality is competent, almost half of the respondents (49 percent) report having worked with municipalities only a few times or even less in the past 12 months. Only 43 percent of school staff report that they worked a few times over the past year to create education measures and plans for youth mental health, 20 percent only once and about a quarter reported never having worked with their municipality to create educational measures for youth mental health (Figure 17). More than 40 percent respondents do not agree that over the last 12 months, budgeting decisions regarding schools and youth made by the municipality were made in close collaboration with their school. Figure 24: School staff rarely collaborate with municipality staff to create policies and plans for youth mental health Source: WB survey 74. The survey of public administrators also indicates significant gaps in collaboration. Very few respondents indicated a high degree of collaboration with officials from other public sector organizations 34 in their sector56 to create public sector policies and plans or to create policies and plans for youth mental health. Around 12 percent of the respondents report that they never worked closely with officials from other tiers of government and government ministries to create policies and plans for youth mental health, 19 percent worked only once, and 37 percent only did it a few times in the past 12 months. Twenty-six percent of respondents do not have contact information for an official from another tier with whom they could get in touch to discuss public sector programming and policy making. These findings are consistent with focus group interviews conducted by STRATA which investigated youth participation in decision making, and highlighted lack of coproduction among institutions on subnational level. 75. The presence of a psychologist influences whether schools have protocols to address mental health challenges. About 49 percent of school staff agree or strongly agree that their school has clear protocols for identifying mental health challenges in students, but 21 percent staff disagree or strongly disagree. Schools that have psychologists are more systematic with their mental health plans—in these schools, 55 percent of staff members agree or strongly agree with the statement, compared to 42 percent of staff in schools with no psychologist. 3.3.5 Management quality 76. Management quality plays an important role in the performance of personnel. In the OECD countries, better management practices are correlated with improvements in school staff performance, as measured in multiple rounds of the Teaching and Learning International Survey.57 Teachers also tend to feel more valued and energized when management practices are conducive.58 The quality of supervisors, incentives, monitoring mechanisms and a culture of innovation are some salient components of management that can affect teacher performance. Public administration management practices also impact service delivery, primarily through monitoring and incentivizing school principals, and capacity building.59 Generally, management practices which support employees are often reciprocated by 56 For example, PHB respondents were asked about their municipality; municipal respondents in Education, Culture, Tourism and Sport were asked about the MoESS; and municipal respondents in Health were asked about the MoH. 57 Bryson, A., Stokes, L., & Wilkinson, D. (2018). Can HRM Improve Sc hools’ Performance? IZA Discussion Paper No. 11348. http://dx.doi.org/10.2139/ssrn.3129292 OECD. (2018). TALIS - The OECD Teaching and Learning International Survey . OECD. https://www.oecd.org/education/talis/ 58 Waeyenber, T. V., Peccei, R., & Decramer, A. (2020). Performance Management and Teacher Performance: The Role of Affective Organizational Commitment and Exhaustion. The International Journal of Human Resource Management, 33(4), 623-646. https://doi.org/10.1080/09585192.2020.1754881 59 Cilliers, J., Dunford, E., & Habyarimana, J. (2022) What Do Local Government Education Managers Do to Boost Learning Outcomes?, The World Bank Economic Review, lhac001, https://doi.org/10.1093/wber/lhac001 35 employees in the form of higher commitment to work and improved trust. This in turn leads to better organizational performance.60 77. The survey measured several dimensions of management quality using a widely used approach. The study adopted a modified version of the World Management Survey (WMS), which is a survey-based measure produced by academics and used to measure the quality of management in many private and public sector organizations globally.61 The modified WMS for schools included modules on setting and monitoring of incentives, monitoring of progress, targeting for goals and objectives, standardization of instructional processes, data driven planning, innovation and new practices, continuous improvement in practices, and staffing decisions. Following the practice of the WMS, these questions were standardized to construct a composite measure called the Management Quality Index. An index for each feature of management was constructed by comparing the principal’s response to the average response from other principals in the survey sample. A composite management quality index was further created from these category indices that measures the overall management practice. 78. There is considerable variation in the overall quality of management across schools (Figure 25) Some aspects of management are also consistently weaker across schools. On a scale of 1 to 5, the module on incentives monitoring (which covers questions on performance indicators for school, review of school performance and frequency of such reviews) scores the highest at 3.64. The module on staffing (which covers questions on recruitment, training and retention of skilled teachers) scored the lowest at 2.87, along with the module on performance incentives (which covers questions on management of poor performance, reward for good performance and promotion criteria) at 2.88. 79. Management quality is associated with mental health services offered to students . Qualitative studies by STRATA showed that as administrators, school principals have a great power to set the agenda for youth mental health in schools.62 As Figure 26 shows, schools that score higher on the Management Quality Index also score higher on the School Mental Health Action index. This suggests that given a uniform framework for mental health service delivery at a central level, the variance in individual management quality in schools affects the mental health education and support that schools offer to their students. An example of this impact is that schools with above average managers are more likely to employ psychologists than schools with worse management (54 percent compared to 41 percent) and this 60 Whitener, E. M. (2001). Do ‘High Commitment’ Human Resource Practices Affect Employee Commitment? A Cross - Level Analysis Using Hierarchical Linear Modeling. Journal of Management, 27(5), 515-535. DOI:10.1177/014920630102700502 Appelbaum, E., Bailey, T., Berg, P., & Kalleberg, A.L. (2000). Manufacturing Advantage: Why High-Performance Work Systems Pay Off. Economic Policy Institute, Cornell University Press. Paauwe, J. (2009). HRM and Performance: Achievements, Methodological Issues and Prospects. Journal of Management Studies, 46(1), 129-142. https://doi.org/10.1111/j.1467-6486.2008.00809.x Boxall, P., & Purcell, J. (2011). Strategy and Human Resource Management (3rd ed.). Palgrave Macmillan. 61 The WMS is a cross-industry, cross country survey to gauge the quality of management across organizations. It is a systematic and standardized measure of management performance across establishments. More information on: https://worldmanagementsurvey.org/ 62 STRATA, 2020 36 difference is also reflected in school staff perceptions—in better managed schools, 61 percent of staff agree or strongly agree that there is sufficient mental health staff, compared to 53 percent in worse managed schools. Figure 25: There is diversity in overall school management quality scores across Lithuania Source: WB survey Figure 26: The management quality in schools affects the mental health support offered to students Source: WB survey 37 80. The quality of management varies across public administration organizations and units, but this variation is less than the variation for schools. The adapted WMS for public administration covered modules on organizational goal and target setting, performance evaluations and incentives, autonomy in decisions and flexibility of procedures, degree of staff involvement and staffing practices. As explained for schools, these WMS questions were standardized to form a management quality index. This index varies considerably across not only organizations, but also within units in these organizations (Figure 27). Note that the ranking of each organization is standardized to all organizations interviewed, irrespective of their function. This is because the point of the WMS is to compare management quality overall, and not just in health and education units. Hence, the graph shows health and education units along with the overall quality of institutions. Figure 27: There is diversity in overall public administration management scores across Lithuania Source: WB survey 81. Specific management quality indicators in public administration show dispersion. At the public administration level, questions which discuss how the organization monitors progress on its objectives present the highest composite score of 3.80, and questions which gauge effective organizational targeting like setting of goals and targets, convergence of individual staff goals with those of the organization and communication of such goals score 3.79 on a scale from 1 to 5. Similar to the school results, questions on staffing, like recruitment of talented individuals and their promotion criteria, scored the lowest at 3.13. 38 Chapter 4: Policy recommendations 82. This chapter presents recommendations to improve the individual and organizational drivers that the survey revealed as associated with the delivery of youth mental health services in Lithuania’s public high schools. In addition, the report also emphasizes improving measurement and using impact evaluations to inform policies. All the recommendations are short- and medium-term and can be implemented without substantial additional funding. 4.1 Address stigma through training, incentives, and communication 83. Addressing the high prevalence of mental health stigma is a priority reform and requires a suite of measures to change attitudes and behaviors towards mental health and improve knowledge of and capacity to respond the mental health problems of students. The survey revealed that while the MoH training as well as training within the framework of school prevention programs builds staff capacity, more needs to be done. The content of training programs can be improved to address the problem of stigma more effectively and improve mental health knowledge. Staff also need to be incentivized to avail of these training opportunities, through strong communication from authorities, as revealed by the findings of the RCT, and incorporating completion of training in staff performance evaluations. A good example of this multi-pronged approach is Canada’s Go-to educator training program which aids students suffering from mental health challenges (Box 3). 4.2.1 Improve training content to address stigma 84. The curricula of current capacity building programs can be improved to address stigma drawing on local and international experts. The WB study finds that prevention programs and the MoH youth mental health training programs are effective in providing school staff with knowledge and skills related to youth mental health. However, the content of these programs should be updated to include new components on mental health literacy in general and mental health stigmatization in particular. Expert opinion and knowledge can be useful in these cases. For example, local and international NGOs offer various stigma removal initiatives which may be attractive to school communities and helpful in curating content for capacity building. These may include mindfulness practice, stigma recognition tests, identification of stigmatizing language, a Stigma-Free School Manifesto, and other interventions reducing stigma levels in school communities. More broadly, the Life Skills Development Program in General Education, the Teacher’s Qualification Raising Program, the Millennial School Program, the Mental Health and Emotional Literacy Program, as well as the Plan for Implementation of the Amendment to the Law on Education on Integrating Children with Special Educational Needs into General Education Schools in Lithuania, may serve as a strategic framework for stigma reduction measures in the curricula. 4.2.2 Establish standards for training and incentivize staff to participate 85. There need to be clear standards for mental health training and incentives for staff to undertake the training. These can be done in the following ways: 39 a. The survey revealed considerable variation in the number of reported hours of training on youth mental health that school staff undertake. There should be a standard for the number of mental health training hours expected for school staff, and the completion of such training should be assessed in the annual performance evaluations of these staff. b. The RCT conducted as part of this study revealed that school staff strongly respond to communicated training opportunities, which clearly suggests that public administrators, particularly those that have management responsibilities over school staff, should have a strong branding and communication plan that accompanies training. Social media formats may be employed to broadcast this information to schools and staff, such as a special address by the MoESS, the MoH, and a mayor or head of municipal administration . c. Schools’ reasoning of choice of prevention programs revealed lack of a strategic approach. Therefore, a needs assessment should be conducted before implementing training and prevention programs. For prevention programs, this also means assessing which age group to target, which mental health challenges to target, and which stakeholder to involve in capacity building. This approach will aid in recognizing the differences amongst school staff and amongst schools and help identify specific training requirements. 4.2.3 Incorporate culture change in the design of training programs 86. Besides the capacity building activities itself, there is a need to improve the culture around training so that it is not viewed merely as a “box ticking” exercise and the the learning from the training is incorporated in everyday work of school staff. A WB RCT in Ghana showed that the design of training programs can make them more impactful in a job.63 In one treatment, trainees collaborated to identify challenges within their units, and discussed how to overcome barriers to reforms through an action plan. The evaluation found that this action-oriented training module helped improved productivity of civil servants. Therefore, improvements in collaboration (elaborated in the next section), and tailoring capacity building initiatives might help to improve the positive impacts of the training and prevention programs. Box 3: Canadian experience: High impact school mental health interventions School mental health interventions in Canada have three components which are essential and are systematically and frugally applied with success. These are: mental health literacy for both students and educators, in-service and pre-service training for teachers, and school-site provision of integrated mental health care to youth. Canada’s Mental Health and High School Curriculum is an evidence-based and freely accessible mental health literacy resource. The resource features classroom-based modules for 12-18- year-old students that are easily embedded in the school curriculum. This resource has gained world-wide recognition and is proven to be highly impactful. 63 Azulai, M., Fornasari, M., van Acker, W., Rasul, M., Rogger, D., & William, M.J. (2021). Building State Capability through Impact Evaluation in Ghana. World Bank Group. 40 “Go-To” Educator Professional Learning trains educators (subject or classroom teachers, pedagogical support staff, principals, or other staff members) in the early identification, support, triage and referral of students who may be exhibiting mental health disorders. Go-To Educators learn how to complement and collaborate with student service providers in school settings, enhancing the development of a whole-school support network that can better meet student needs. An on-line learning platform www.teachmentalhealth.org addresses practical classroom strategies and how to link students in with appropriate services within and outside the school community. The platform is informed by inputs from more than 30 faculties in education and used across Canada and globally. The resource can be applied in both undergraduate and postgraduate teacher educations as well as for self-study professional development. School-based health centers address students’ mental health care needs, while concurrently supporting their own health care needs and social service requirements. Some of their advantages are: a) they provide the greatest ease of access for the largest number of young people; b) they are designed to be youth friendly; c) they can provide a full range of interventions (from promotion to prevention to care); d) they can be seamlessly linked to primary health providers; and e) they are relatively inexpensive to establish. When properly implemented, such centers can provide both site-based integration of services and horizontal integration into primary health care and social services. Sources: Kutcher & Wei, 202064, Kutcher, Wei & IWK Health Center, 201765, Mental Health Literacy. Org66 4.2 Improving the Work Environment 4.2.1 Strengthen teamwork in addressing mental health by schools 87. Measures to improve collaboration in schools should be planned and implemented. In Lithuania, there is no obligation for schools to employ psychologists or other educational support specialists. If a school has 400 or more students, funds are provided to set up a psychologist position. According to the Ratings education analytics magazine, nearly one third of Lithuanian general education schools do not have psychologists and dozens of schools do not have a single student support specialist, while some schools employ 4-6 psychologists and social workers.67 A quarter of the country's nearly 1,000 schools have less than 100 students, and these small schools are typically covered by psychologists based at municipal Pedagogical Psychological Services. While the WB survey did not assess the actual availability 64 Kutcher, S. & Wei, Y. (2020). School Mental Health: A Necessary Component of Youth Mental Health Policy and Plans. World Psychiatry, 19(2), 174-175. https://doi.org/10.1002/wps.20732 65 Kutcher, S., Wei, Y. & IWK Health Center. (2017). Mental Health and High School Curriculum Guide: Understanding Mental Health and Mental Illness, Version 3. Teen Mental Health. Org. http://mentalhealthliteracy.org/schoolmhl/wp-content/uploads/2015/09/Mental-Health-High-School-Curriculum- Guide.pdf 66 Mental Health Literacy. (n.d.). Go-to Educator Professional Learning. Mental Health Literacy.Org . https://mhlcurriculum.org/go-educator-professional-learning/ 67 Grigaliūnaitė, V. (2012, December 08). The Head of a School who has Been Criticized for Not Having a Psychologist: “That is Simply Not True”. 15min. https://www.15min.lt/naujiena/aktualu/lietuva/kritikos-del-psichologo- nebuvimo-sulaukusios-mokyklos-vadovas-tai-paprasciausia-netiesa-56-1610684 41 of psychologists, the participation of psychologists in the schools’ handling of youth mental health challenges was low. 88. The survey revealed that most respondents were not even aware of whether the school has a psychologist, which points to lack of collaboration in addressing students’ needs that will not be addressed by only increasing the number of psychologists. The school leadership should be made aware of the situation and take measures to promote teamwork in strengthening students’ mental health. In Canada and the USA for example, school systems use interventions to foster teamwork around mental health. Canada provides professional training for educators, preparing school staff to complement and collaborate with student service providers; and school-based mental health coordinators in the USA coordinate mental health services and prevention efforts, as well as oversee comprehensive supports for children with mental health challenges, stress, and trauma (Box 4)68. 89. Greater collaboration between school staff, parents, and teachers also needs to be fostered. STRATA’s report provides several recommendations to this effect, including involving students' self- government groups in mental health literacy activities.69 Box 4: Teaming for students’ mental health: resources in the USA The National Center for School Mental Health at the University of Maryland School of Medicine has developed a School Mental Health Quality Guide, focused on teaming. The guide contains background information on teaming, best practices, possible action steps, examples from the field, and resources. The Guide promotes school mental health teams—groups of school and community stakeholders at a school or district level that meet regularly, use data-based decision making, and rely on action planning to support student mental health. School mental health teams facilitate communication, collaboration, and mutual support among individual team members who might otherwise work in isolation. These teams develop a vision and prioritize improvement efforts to ensure that the school mental health system is meeting the individual needs of all students and the larger school community. School mental health teams work to maximize resources available to address the mental health needs of students and their families. Schools may have one team devoted to the full continuum of mental health supports (mental health promotion to selective and indicated intervention) or multiple teams that address different parts of the continuum. Sources: National Center for School Mental Health, 202070, Center for School Mental Health, 201871 68 Website of Coeur d'Alene School District, Idaho. 69 STRATA, 2021. 70 National Center for School Mental Health. (2020). School Mental Health Quality Guide: Teaming . NCSMH, University of Maryland School of Medicine. https://www.schoolmentalhealth.org/media/SOM/Microsites/NCSMH/Documents/Quality-Guides/Teaming- 7.16.21.pdf 71 Center for School Mental Health. (2018). School Mental Health Teaming Playbook: Best Practices and Tips from the Field. http://csmh.umaryland.edu/media/SOM/Microsites/CSMH/docs/Reports/School-Mental-Health- Teaming-Playbook.pdf 42 4.2.2 Strengthen collaboration among subnational level institutions 90. Innovative ways of budgeting can be one of the ways to encourage collaboration. The survey identified lack of co-production between administrative levels and among subnational level institutions. Budgeting is one of the areas where co-production may be encouraged. The modality of a participatory budget, which is a way to decide together with the community on the use of public funds, and where community members themselves propose ideas to a municipality, school or organization and vote to choose the ones they like best, may be used to promote students’ mental health. According to Transparency International, about 30 schools and 22 municipalities in Lithuania have implemented participatory budget at least once by October 2021.72. Students’ self-governments may be involved in budgeting for mental health and decide about mental health initiatives to be implemented in schools. The principle of Youth4Youth has proven effective in various initiatives and is in line with the findings of qualitative research done by STRATA. 91. Municipalities have resources which may be directed to promote co-production and collaboration in the area of youth mental health. Municipalities employ inter-institutional coordinators. The mandate of these employees is to bring different institutions together to solve complex multi-faceted problems. Inter-institutional coordinators may cover social services, children's rights, probation, NGOs, public health, education, youth centers – a variety of topics and activities performed by various organizations which fall under different government’s ministries. Municipalities also employ youth coordinators, who may highlight youth mental health in their annual work plans and address the topic in an integrated way involving all schools in the municipality. 4.2.3 Expand, standardize, and universally apply protocols addressing mental health in schools 92. National protocols on mental health should be drafted to steer protocols at the school level . The survey found out that about 50 percent of Lithuania’s schools have clear protocols for identifying mental health challenges in students. Establishing detailed protocols, which cover all aspects of mental health promotion in schools – from preventive activities to addressing specific mental health situations may facilitate better response and improved services to youth. STRATA also recommend’s agile legislation at the national level for implementation at the local level73. The WB team recommends developing national level guidelines for schools, based on which schools may be required to develop inclusive protocols addressing students’ mental health. Box 5 gives some examples of such guidance document in Canada and the USA. Box 5: Examples of guidance documents for schools from Canada and the USA The Canadian guidance document Key Principles and Strategies for K-12 Mental Health Promotion in Schools discusses the key principles that underlie the elements of the MoESS Mental Health in Schools Strategy and 72 Transparency International. (n.d.). Participating Budget. Transparency International. https://www.transparency.lt/dalyvaujamasisbiudzetas/ 73 STRATA, 2020 43 follows with sub-strategies, recommended actions, and resources to support mental health and well-being with a school-based focus. Schools are invited to adapt strategies to their needs and develop protocols which best suit their communities and classrooms and best cater for the needs of students and adult members of the school community. The Guidance to States and School Systems on Addressing Mental Health and Substance Use Issues in Schools is a joint informational bulletin by the Substance Abuse and Mental Health Services Administration and the Centers for Medicare & Medicaid Services of the USA, which provides the public schools and school systems with information about addressing mental health and substance use issues in schools. The bulletin informs about various resources and strategies which may be used in developing protocols to address students’ mental health challenges by public schools. Sources: Ministry of Education, Canada, 202174, McCance-Katz & Lynch, 201975 4.3 Strengthening Management 93. The quality of management in schools and the public administration can have a significant impact on mental health services for students. The survey has revealed considerable variation in management practices across organizations in the mental health service delivery chain in Lithuania and established a correlation between management quality in Lithuania’s schools and the mental health support that schools offer to their students. There is increasingly robust cross-national evidence on the importance of school principals in delivering learning; irrespective of country context, better managed schools have higher student learning outcomes.76 Principals have a key role in supporting teachers and monitoring their performance, creating a conducive, collaborative work environment in schools, and in engaging with parents. While academic literature is largely on the relationship between management quality and learning outcomes, the findings can be extended to student mental health as a critical aspect of learning. 74 Ministry of Education, Canada. (2021). Key Principles and Strategies for K-12 Mental Health Promotion in Schools. British Columbia, Ministry of Education, Canada. https://www2.gov.bc.ca/assets/gov/education/administration/kindergarten-to-grade-12/key-principles-and- strategies-for-k-12-mental-health-promotion.pdf 75 McCance-Katz, E. & Lynch, C. (2019). Guidance to States and School Systems on Addressing Mental Health and Substance Use Issues in Schools. Substance Abuse and Mental Health Services Administration, Centers for Medicare and Medicaid Services, Center for Medicaid and Chip Services. https://store.samhsa.gov/sites/default/files/d7/priv/pep19-school-guide.pdf 76 Bloom, N., Lemos, R., Sadun, R. & Van Reenen, J. (2015). Does Management Matter in schools? The Economic Journal, 125(584) 647–674. https://doi.org/10.1111/ecoj.12267 Fryer, Roland G. (2017) .Management and Student Achievement: Evidence from a Randomized Field Experiment. NBER Working Paper No. w23437.| Waters, T. Marzano, R., & McNulty, B. (2003). Balanced Leadership: What 30 years of research tells us about the effect of leadership on student achievement. Denver, Colorado: McREL International. 44 4.3.1 Improve strategic and managerial competencies of school leadership 94. Strengthening management requires establishing a competency framework for leaders, and a combination of capacity building and improving the incentives of principals and administrators to take organizational and people management seriously. Most OECD countries use competency frameworks for senior managers to guide human resource management, including workforce planning, recruitment, training, and performance evaluation.77 The Government’s Decree on the Implementation of the Law on Civil Service78, enacted in January 2021, introduced the competence model in civil service in Lithuania. The required qualifications for school principals and the procedure of their evaluation are laid out in dedicated orders by the MoESS79. In 2020, the MoESS initiated the development of a model of competencies for heads of educational institutions. The annual self-assessments and monitoring evaluations of heads of Lithuanian educational institutions show consistently insufficient level of skills in the areas of strategic thinking and change management, human resource management and management of educational process80. The WB recommends interventions in these areas to help managers develop adequate competences. A roadmap on boosting these competences may be helpful. The roadmap may be shared with principals in follow-up of their performance evaluation, as part of recommendations for competence development. 95. There is a large literature that shows that improving the capacity of managers through different types of trainings can increase organizational productivity.81 On content, these trainings should aim to strengthen the core attributes of management—goal setting and monitoring, goal alignment, performance feedback, and supporting staff to enable them to deliver on their goals. Organizational performance management, with goals and targets identified in strategic plans or budget documents, is necessary for creating a common vision for all staff and creating line-of-sight on how an individual’s work contributes to corporate goals. Moreover, a training program that helps managers develop these goals, if done in a participatory way by involving staff, can improve both the quality of organizational plans and staff understanding and commitment to them. To be effective coaches, managers will also need targeted training in assessing employee skills, mastering difficult conversations, and giving constructive feedback. For example, in Ireland, managers need to undergo 30 hours of training specifically on performance 77 OECD. (2017). Skills for a High Performing Civil Service. OECD. https://www.oecd.org/gov/skills-for-a-high- performing-civil-service-9789264280724-en.htm 78 Regarding the implementation of the Law on Civil Service of the Republic of Lithuania. (2018). Government of Lithuania. 79 Regarding the Approval of the Description of Qualification Requirements for the Heads of State and Municipal Educational Institutions (Excluding Higher Education Institutions). (2011). MoESS, Government of Lithuania. On the Minister of Education and Science 2018 March 27 order no. V-279 “On the Approval of the Regulations on the Evaluation of the Activities of the Heads of State and Municipal Educational Institutions (except for Higher Education Institutions), their Deputies for Education, and the Heads of the Departments Organizing Education”. (2021). MoESS, Government of Lithuania. 80 Šileikytė, D. (2021, March). On the Competencies of Heads of Educational Institutions. Švietimo Naujienos. https://www.svietimonaujienos.lt/apie-svietimo-istaigu-vadovu-kompetencijas/ 81 McKenzie, D., Woodruff, C., Bjorvatn, K., Bruhn, M., Cai, J., GonzalezUribe, J., Quinn, S., Sonobe, T. & Valdivia, M. (2021). Training Entrepreneurs. VoxDevLit, 1(2). https://voxdev.org/sites/default/files/Training_Entrepreneurs_Issue_2.pdf 45 appraisal and, in Canada, managers receive special training on performance management from the Canada School of Public Service. Lithuania’s academic institutions offer capacity development opportunities for school managers. For instance, the Institute for Professional Development at Vytautas Magnus University offers in-service dedicated training for principals, such as Principal’s Tribune and Leadership School. The WB team recommends the review and assessment of the training offering to be able to recommend effective training programs and identify potential gaps in training provision. 96. A mix of different types of capacity building approaches should be implemented, including traditional curriculum-based training and peer-based exchanges of ideas. Traditional, class-room style management training programs have been successful at improving organizational practices and outcomes and are cost-effective.82 In addition, components that aim to improve attitudes, such as mindset training and aspiration nudges, can enhance the learning impacts of traditional training approaches. Consulting programs, providing intensive and tailored support to organizations and managers to improve their operational practices have also been very effective, although are relatively expensive to implement.83 Group-based consulting programs, by leveraging peer-learning incentives, have been found to have positive effects on learning and implementing best practices, and on organizational efficiency, while exhibiting substantially lower costs per participant.84 Mentoring and peer-based learning programs have also been found to improve short-term organizational outcomes, and are most impactful when lower- performing managers get matched with better-performing ones who are not seen as competitors.85 Since there are limits to such matching, mentoring programs should complement but not replace traditional training programs. In 2021, the NAE announced a mentorship program86. Forty-three mentors completed special training and mentor heads of educational institutions in their first year of service. The results of this initiative may be assessed; specific areas of mentorship may be singled out and the institute of mentorship may be expanded to cover the corps of school managers more widely (Lithuania has approximately one thousand public schools). Peer-based learning may be promoted in collaboration with the Lithuanian ASL which identifies sharing management knowledge among its aims87. 4.3.2 Combine capacity building with incentives for principals and administrators 97. Capacity building should be combined with improved incentives for principals and administrators to take management seriously. These include managers’ own performance evaluations through the annual performance appraisal system and, more importantly, through regular performance 82 McKenzie et al. (2021) 83 Bloom, N., Eifert, B., Mahajan, A., McKenzie, D., Roberts, J. (2013). Does Management Matter? Evidence from India , The Quarterly Journal of Economics, 128(1), 1–51, https://doi.org/10.1093/qje/qjs044 84 Iacovone, L., Maloney, W. & McKenzie, D. (forthcoming). Improving Management with Individual and Group-Based Consulting: Results from a Randomized Experiment in Colombia. https://drive.google.com/file/d/1cWBEnAqGGF2tfU20VrTjblxvVfLV_k7i/view 85 Cai, J. & Szeidi, A. (2018). Interfirm Relationships and Business Performance. The Quarterly Journal of Economics, 133(3), 1229–1282. 86 NAE. (n.d.). For Less Experienced New School Leaders, the Help of Mentors. NAE. https://www.nsa.smm.lt/2021/04/29/maziau-patyrusiems-naujiems-mokyklu-vadovams-mentoriu-pagalba/ 87 The Lithuanian ASL 46 conversations. Increasingly, managers in OECD countries have a different performance appraisal system than the rest of the civil service, with an emphasis on their achievement of strategic organizational goals as well as their managerial and leadership skills.88 This approach was introduced in Lithuania in 2020. A new competency assessment procedure has entered into force, by which municipal authorities set annual activity tasks for school managers; managers self-assess performance and NAE draws conclusions on and issues a certificate of managers’ competency assessment. The WB team recommends using the task- setting component of performance evaluation to incentivize acquisition of strategic and managerial competencies. Reaching out to municipal administrations with a clear proposition and recommended KPIs will be necessary to ensure this incentive. Capacity building incentives can be combined with behavioral nudges. For example, SMS messages have been used to send reminders and could potentially be used to remind principals to follow protocols on school mental health, and to disseminate good practices on fostering collaboration or teacher management. It is important that both capacity building and incentives are grounded in a competency framework for leaders. These aspects are a major reason why Shanghai schools have been consistently ranked the highest globally in delivering learning (Box 6). Box 6: Why is Shanghai the best in student learning? Students in Shanghai have consistently achieved the highest scores in the areas of reading, science, and mathematics on the Program for International Student Assessment (PISA), a highly respected and widely used global assessment of 15-year-olds’ educational abilities. While there are a variety of reasons for Shanghai’s high performance, effective school principals have been key. Shanghai has a rigorous system for training principals and holding them accountable for results based on a core set of competencies that include organizational management, teacher management, school culture, and community engagement. The Shanghai municipality collaborated with Shanghai Normal University to develop a one-year training program on these competencies. The program takes place once a week and includes group lectures, individual research projects, field visits, and mentoring by experienced principals to new principals. Shanghai also uses the concept of professional communities to foster collaboration and peer-to-peer learning among teachers and principals. Administrators evaluate the performance of principals along these competencies, and this performance evaluation impacts salaries and promotions. For example, Principals are expected to observe classes regularly and provide feedback to teachers on instructional improvement. They are also expected to use various modalities to regularly interact with students, teachers, and parents, including through face-to- face meetings and on-line fora. Source: Liang, Kidwai & Zhang, 201689 98. Interaction between different levels of administration in performance evaluation should be strengthened. Annual performance reviews must be part of a bigger performance conversation strategy. Successful contemporary organizations branch beyond the traditional, annual performance 88 Kuperus, H. & and Rode, A. (2016). Top Public Managers in Europe: Management and Employment in Central Public Administrations. Ministry of the Interior and Kingdom Relations, The Hague. https://www.eupan.eu/wp- content/uploads/2019/02/2016_1_NL_Top_Public_Managers_in_Europe.pdf 89 Liang, X., Kidwai, H. & Zhang, M. (2016). How Shanghai does it: Insights and Lessons from the Highest-ranking Education System in the World. The World Bank. https://openknowledge.worldbank.org/handle/10986/24000 47 review. NAE and education units of municipal administrations may organize regular performance conversations between the annual reviews. The key principles of modern performance reviews are: reviews should happen frequently (quarterly or monthly); they should be a two-way conversation; be future focused - review recent performance and coach to impact, development, and growth; modern performance conversations are open and collaborative; based on rich and real-time employee data; and conclude with next steps and follow-up90. 4.4 Improving Measurement and Evaluation in the Service Delivery Chain 99. Monitoring and evaluation should be mainstreamed to inform policies. The OECD has also recommended that Lithuania establish a whole-of-government policy framework for evaluation.91 There are three aspects to improving monitoring and evaluation. First, improve primary data collection by improving the quality of administrative data and deploying regular surveys to monitor key drivers. Second, use impact evaluations to measure the effect of policies. Third, facilitate the use of data in decision- making. 4.4.1 Improve primary data collection 100. The current administrative data should be expanded to include a granular picture at the individual level. Data is essential for making informed policy decisions as it provides a baseline from which to monitor progress and key performance indicators (KPIs). The WB study found that existing administrative data are limited and only provide basic demographic, human resource, and outcome variables. ŠVIS, the information system for the education sector, covers the demographics of school staff (age, gender, tenure, qualification, and education level) and some school-level characteristics (the number of students and classes, the number of students with special needs, the number of implemented prevention programs, and 10th and 12th grade exam grades). The system lacks broader spectrum of data on outcomes, thus limiting the ability to gauge the effects of education policy and reforms. Most of the ŠVIS data is school-level data, and does not provide information on individual students or staff. Granular data, such as data on students, would allow for more detailed analysis on downstream effects of policies and treatments. For example, data on students’ demographic characteristics, grade averages, and number of disciplinary incidents allows to see the effects of interventions on students from different sections of the socio-economical distribution. Even less data is available for public administration, with nearly no de- identified public data available for use in research. 101. The WB survey can serve as a baseline for establishing performance indicators for the key drivers of mental health services for the youth and can be replicated annually in a streamlined format 90 Stange, J. (2021, January). 13 Employee Performance Review Tips That Actually Improve Performance. Quantum Workplace. https://www.quantumworkplace.com/future-of-work/13-performance-review-tips-that-actually- improve-employee-performance 91 OECD Directorate for Public Governance. (2021). Mobilizing Evidence at the Centre of Government in Lithuania: Strengthening Decision Making and Policy Evaluation for Long-term Development. OECD. https://doi.org/10.1787/323e3500-en 48 to measure the impact of reforms. These drivers include the response of schools to mental health challenges, stigma, the amount and quality of training undertaken, teamwork and collaboration around mental health, and the quality of management. While the government has undertaken several recent surveys on topics of youth mental health in Lithuania, which were funded from the national budget, these were conducted on small samples of respondents and did not fall under a unified framework in terms of methodology and surveyed groups. Government’s agencies may consider consolidating and streamlining funds to support studies that produce sound evidence and impact across fields under different tiers of government, especially in complex areas like mental health. Large-scale surveys that measure a consistent set of features through the service delivery chain can provide unique insights into the steps required to strengthen those services. 4.4.2 Embed evaluations in policy making 102. RCTs should also be used to causally identify the effects of different policies (Box 7). The impact evaluation used in this study for example, showed that school staff respond strongly to messages from the MoH, and not as strongly to messages that come from the LSU. This may inform the go vernment’s agencies on communication strategies as well as attitudes and motivation of school staff. Similar experiments may be used to test other treatments and policies. The expertise for developing an optimal monitoring and evaluation plan may be found within the ministries’ strategic planning units, other institutions subordinate to the ministries, which carry out monitoring and evaluation, and STRATA which has an extensive mandate to promote evidence-based policy making. Box 7: Research tools in policy making Decision makers need evidence- such as data and the results of studies – to determine if policies are working as intended and identify potential improvements. Surveys allow policy makers to better map out the overall situation, as well as the actual causal chain and mechanisms behind the success or failure of certain policies. Repeating a survey can enhance its value and introduce new features, such as evaluation of changes. Monitoring and detecting important changes is usually a key reason for sampling in time. There are two types of recurring surveys. The first type — repeated cross-sectional surveys — consists of surveys that interview individual respondents only once, but field similar surveys (with the same basic sample design, a core set of questions administered each time, and the same method of data collection) on some periodic time schedule. The second type of over-time survey — longitudinal or panel surveys — involves interviewing members of the same sample repeatedly. Both types of survey make it possible to monitor time trends, and various hybrid designs (such as rotation group designs) are possible that combine features of repeated cross-section and panel designs. RCT is an evaluation technique that draws from experimental design in order to measure the impact of a development project. Pilots and randomized experiments are useful when introducing policies aimed at bigger populations when no baseline data is available to measure the efficiency of the treatment or intervention. The strength of such designs lies in their simplicity. Experiments in which at least two groups are artificially created: one that receives treatment, and one that does not, allow for comparisons and enable to determine the effectiveness of the intervention. 49 Sources: Tourangeu, 200392, Webber & Prouse, 201893 4.4.3 Facilitate the use of data in decision making by municipal authorities 103. Municipalities perform essential functions and should be assisted with knowledge infrastructure to ensure the use of data. The Law on Local Self-Government of the Republic of Lithuania provides municipal authorities with important functions94, such as educational support and strengthening public health. These functions are funded through budget appropriations, managed by municipal administrations. Informed administrative decisions, including budgeting, at the municipal level are essential for success of policies as well as efficient public spending. The Ministry of Economy and Innovation of Lithuania promotes impact procurement95, as a mechanism enabling sound public investment and spending. Municipalities hesitate to undertake impact procurement primarily because they find it difficult to establish KPIs in the social area. The WB team recommends that easily accessible and user-friendly data management tools are developed to support municipalities in decision making. The Government may consider development of a dashboard to provide municipalities with information such as KPIs and help them monitor the implementation of delegated responsibilities and set estimates and targets for future work. Figure 28 gives an example of an applied data analytics dashboard for decision makers. Figure 28: Public procurement dashboard- An example of applied data analytics for decision making 92 Tourangeu, R. (2003). Recurring Surveys: Issues and Opportunities. A Report to the National Science Foundation Based on a Workshop Held on March 28-29, 2003. 93 Webber, S. & Prouse, C. (2018). The New Gold Standard: The Rise of Randomized Control Trials and Experimental Development. Economic Geography, Taylor and Francis Journals, 94(2), 166-187. DOI: 10.1080/00130095.2017.1392235 94 Government of Lithuania. (2010). Law on Local Self-Government of the Republic of Lithuania. Republic of Lithuania. https://www.infolex.lt/teise/DocumentSinglePart.aspx?AktoId=65125&StrNr=7# 95 Versli Lietuva. (n.d.). Impact Procurement: New Opportunities for Social Businesses and NGOs to Develop Quality Public Services and Collaborate with Municipalities. https://www.enterpriselithuania.com/verslauk/ijunk- poveiki/poveikio-pirkimai/ 50 Source: WB Annexes Annex 1: Timeline of the project Table 2: Major milestones of the project Phase Timeline Establish client relationships in Lithuanian May 2020 - August 2020 government Assist STRATA in designing and analyzing co- July 2020 - August 2020 creative sessions for a Lithuanian School Forum Assist STRATA in qualitative field work with NGOs, July 2020 - November 2020 experts, stakeholders Procure survey firm for baseline survey November 2020 - December 2020 Prepare first survey instrument draft November 2020 - December 2020 Assist STRATA in co-creative stakeholder session December 2020 Assist STRATA in first Diagnostic delivery December 2020 (final version published by STRATA in February 2021) 51 Design impact evaluation with MoH inputs December 2020 - March 2021 Prepare second survey instrument draft January 2021 - March 2021 Assist STRATA with co-creative workshops in 5 January 2021 - March 2021 selected schools Program, pilot and adjust survey instrument February 2021 - April 2021 Implement School Email Impact Evaluation March 2021 - April 2021 Train enumerators March 2021 - April 2021 Sign data agreement and transfer administrative April 2021 - August 2021 data Survey data collection May 2021 - October 2021 Data cleaning and analysis June 2021 – January 2022 Annex 2: The service delivery chain of youth mental health in Lithuania Figure 29: The service delivery chain of youth mental health in Lithuania 52 Source: STRATA, 202096 Annex 3: Sampling strategy Selection of ministries and national agencies The WB directly selected 4 ministries for the survey - the MoESS, the MoH, and the MoSSL. They have been selected based on research topics as ministries whose work is closely related to the provision of mental health services to young people. The research team also selected the NAE, anticipating the importance of this agency in the field of educational assistance. Selection of staff for ministries and national agencies The selection was done using Stata. In each selected ministry and agency, the WB selected 40 employees to sample. - First, 5 units were randomly selected. Then a manager and up to 10 employees from each unit were picked. If this overfilled the sample, the research team dropped the corresponding number of employees from the largest unit. In case of a tie, the research team dropped from lower rank units (i.e., if unit 3 and unit 4 are tied, an employee was dropped from unit 4). 96 STRATA, 2020 53 - If this did not fill the intended sample, random employees from other departments were selected to fill the sample. - Any left-over employees in picked units were assigned to the back-up sample. If this did not fill the back-up sample, back-ups from other units were picked, until at least 5 managerial level employees and 35 regular employees were included in the back up sample. Selection of municipalities The selection was done in Excel to ensure transparency with survey firm. The WB selected 43 out of 60 municipalities from Lithuania for the survey. Vilnius City Municipality was selected directly, considering its importance for the representativeness of the sample. The remaining 42 municipalities were selected at random, first by randomly selecting 42 of the 47 remaining public health offices and then again randomly selecting the municipality to be administered from each public health office. This sampling method ensures that each municipality selected for the survey corresponds to one public health office and is representative of municipalities at the national level. Selection of municipal employees The selection was done using Stata. According to municipality size, the research team sampled 12 to 22 employees. - First, one random education unit in each municipality was picked to sample its manager and up to 7 employees. - Then, one random non-education unit in each municipality was picked to sample its manager and up to 7 employees. If this overfilled the sample, the corresponding number of employees from the largest unit were dropped. In case of a tie, employees from the non-education unit were dropped. - If there were at least three employees remaining to sample, a second non-education random unit, its’ manager and up to 7 employees were sampled. If this overfilled the sample, employees were again dropped from the largest unit. In the case of a tie, the WB prioritized dropping lower rank units (if units 2 and 3 are tied, drop 3, if units 1 2 are tied, drop 2, etc.). - If this did not fill the intended sample, the WB selected random employees from other departments. - Any left-over employees in picked units were assigned to the back-up sample. If this did not fill the back-up sample, back-ups from other units were picked, until at least 3 managerial level employees and 10 regular employees were present in the back up sample. Selection of public health offices The selection was done in Excel. The WB selected 40 out of 48 public health offices from all over Lithuania for the survey. The Vilnius City PHB was selected directly, considering its importance for the representativeness of the sample. The remaining 39 of the 47 public health offices were selected at random. 54 Selection of public health office staff In each public health office, all employees were selected to participate in the survey, except for specialists who directly provide public health services in schools. Selection of schools The selection was done in Microsoft Excel. 243 schools were randomly selected from the subset of schools teaching at least one of grades 9–12 (I – IV gymnasiums). For representativeness, 5 to 7 schools according to municipality size were selected. In municipalities where there are less than five schools, all schools were selected. Ten schools were replaced due to them refusing to participate in the survey. Where possible, the research team replaced with other schools from the same municipality. For municipalities that do not have enough schools, all remaining schools across Lithuania were ranked randomly and a replacement was picked Selection of school staff In each school, the WB randomly selected a principal, three teachers, and one support specialist (The research team tried to sample psychologists where possible, but not all schools employ them. In other schools, the roles of psychologists are at least partly taken over by a social pedagogue or another social worker). Figure 30: Sampling strategy for public administration institutions (ministry example) Source: WB calculations 55 Figure 31: Sampling strategy for schools Source: WB calculations Annex 4: Municipalities covered in the survey 1. Akmenės r. sav. 2. Alytaus m. sav. 3. Alytaus r. sav. 4. Anykščių r. sav. 5. Biržų r. sav. 6. Druskininkų sav. 7. Ignalinos r. sav. 8. Jonavos r. sav. 9. Joniškio r. sav. 10. Jurbarko r. sav. 11. Kauno m. sav. 56 12. Kazlų Rūdos sav. 13. Kėdainių r. sav. 14. Klaipėdos m. sav. 15. Klaipėdos r. sav. 16. Kretingos r. sav. 17. Lazdijų r. sav. 18. Mažeikių r. sav. 19. Neringos sav. 20. Pagėgių sav. 21. Pakruojo r. sav. 22. Panevėžio m. sav. 23. Panevėžio r. sav. 24. Pasvalio r. sav. 25. Plungės r. sav. 26. Prienų r. sav. 27. Radviliškio r. sav. 28. Skuodo r. sav. 29. Šakių r. sav. 30. Šalčininkų r. sav. 31. Šiaulių m. sav. 32. Šiaulių r. sav. 33. Šilalės r. sav. 34. Šilutės r. sav. 35. Širvintų r. sav. 36. Tauragės r. sav. 37. Telšių r. sav. 38. Trakų r. sav. 39. Ukmergės r. sav. 40. Utenos r. sav. 41. Varėnos r. sav. 42. Vilkaviškio r. sav. 43. Vilniaus m. sav. Annex 5: Survey design and implementation The survey included two questionnaires: one for public administrators and one for school staff. A few survey modules were common to both the public administrators and the school staff, whereas a few were distinct and especially crafted to gauge nuances in public administration and school administration separately. The modules covered recruitment, promotion, performance management, compensation, as well as employees’ attitudes, knowledge and perception about mental health challenges. The different thematic areas covered by the survey are: • The quality of management practices in offices, agencies, health bureaus, and schools • The motivation, engagement, attitude, and personality traits of personnel • The selection, performance management, advancement, rewards • Incentives, teamwork, bottlenecks, and adaptation to COVID-19 57 • Inter-sectoral coordination and collaboration on youth mental health and other policies • Experiences with and perceptions of school prevention programs and teacher training programs • Awareness of mental health issues, services, and demand in schools • Attitudes and perceptions of mental health The survey employed a novel way of interviewing respondents, using Microsoft Teams. Enumerators used Microsoft Teams to conduct online interviews. This conveyed a sense of being personally interviewed, but also enabled remote survey work to ensure safety of the enumerators and the respondents in light of the COVID-19 safety protocols. The survey was conducted by Civitta on behalf of and overviewed by the WB. Annex 6: RCT For the RCT, 1014 schools in Lithuania were randomized into three email treatment groups: the top-down, MoH treatment group (T1), the bottom-up (client), LSU treatment group (T2) and the control (C) group. The training was directly advertised to the treatment groups by sending the administrators of these schools two institutionally branded emails with otherwise identical content encouraging participation in the training and the sharing of said email to their school staff. The T1 group received a MoH branded email, the T2 group received an LSU branded email, and the C group received no emails. An example email, as sent by the MoH, is shown in Figure 32. Figure 32: The MoH branded version of the e-mail school administrators received to encourage them to advertise the training (text in Lithuanian) 58 Source: MoH, Government of Lithuania The experiment allows for tracking several metrics across these different treatment groups. The main finding of the experiment is that both school administrators and school staff do respond differentially to treatments, and that both treatments are effective in increasing initial uptake, although only the T1 treatment is strong enough to persist for training completion. Interestingly, the initial response to the email, as well as down-stream effects on training participation can both be tracked. For example, Figure 33 shows that that the MoH treatment is more effective in generating a response through email metrics – over the course of the experiment, the MoH email is opened … times, and the LSU email is opened … times. This trend seems to hold throughout the duration of the experiment. This is also reflected in further down-stream effects – although both treatments seem to be bringing in school staff to the training, the top-down treatment is more effective. Compared to the control group, the top-down treatment causes roughly 2.8 times the staff to start the training, and 2 times to finish the training. Meanwhile, the bottom- up treatment causes 1.7 times the staff to start the training and 1.3 times to finish the training. Finally – consider Figure 34 for an illustration of the overall effects of the treatments. The beta coefficient here estimates the average number of staff per school participating in each training module by treatment 59 groups. In effect, this graph shows that although both treatments are statistically significant at the 95percent level for training sign-up, only the top-down treatment is strong enough to persist throughout to the end of the training. The WB team is still conducting additional study of underlying mechanisms that lead to the increased effectiveness of the top-down email, but this seems to indicate that the top-level institutions have significant pull in potentially increasing school staff engagement and participation in training. There are no significant effects on training performance, i.e., through test scores. Figure 33: Number of times treatment emails were opened by the T1 and T2 groups through time Source: WB survey Figure 34: Treatment and control group regressions on number of staff per module 60 Source: WB survey 61