State of Maryland SABER Country Report EDUCATION MANAGEMENT INFORMATION SYSTEMS 2015 Key Policy Areas Status 1. Enabling Environment EMIS in the State of Maryland is supported by strong policies, people, and processes at state and local levels. The Code of Maryland Regulations (COMAR) effectively positions the state-level EMIS managed by the Maryland State Department of Education (MSDE) as the point of reference system that collects, processes, and disseminates education data on a regular basis. With a decentralized state education system, in which counties are autonomous and independent in many areas of decision making, COMAR effectively regulates and empowers counties to build systems and innovate in relation to reporting, managing, and using data. In this way, COMAR guides the systematic flow of data from schools to counties and finally to MSDE. Each county has an EMIS, selected and managed independently by the county and integrated with the overarching state-level EMIS. The strong legal framework lays the foundation for key ingredients that contribute to Maryland’s advanced enabling environment, including talented human resources, a dedicated budget augmented by federal and state partial funding, and a statewide data- driven culture. Powered by clear vision from high-level decision makers and solid buy-in from leaders across the education system, EMIS has evolved from a focus on compliance to a more innovative, learning-focused approach to data management and utilization. New institutionalization strategies and incentive structures put Maryland at the forefront of this national trend in the United States. 2. System Soundness Each county has an established EMIS, although they vary widely in terms of design and degree of advancement. Each county has a data warehouse. Some systems have been built in-house and others purchased off the shelf. The state-level EMIS was built in-house and comprises three primary systems: (1) the Educational Data Warehouse (EDW), (2) the Longitudinal Data System (LDS), and (3) the Web Data Collection System (WDCS). The EDW is the hub for statewide K12 education data. Data coverage in the EDW includes administrative and learning outcomes data. Established in 2010, the LDS integrates K12 with higher education and workforce data and receives additional flat file transfers for early childhood, career and technology education, and special education data. The LDS infrastructure is established, although all data have not yet been fully loaded. MSDE tries to establish clear data-sharing processes, but some counties report that at times MSDE makes changes to data requirements, requests data outside of designated times with short notice, and requests already collected data, indicating gaps in coordination and communication. The lack of fully integrated financial and human resources data at county and state levels creates data siloes, limiting comprehensive EMIS functionality. 3. Quality Data Quality data are effectively mandated by federal and state policies and implemented through a variety of systematic processes. First, “The Maryland Student Records System Manual” documents procedures for maintenance of student data across a decentralized education system by sharing requirements and maintenance directions on topics such as data definitions, sample forms, relevant policies, and data coding requirements. An array of other manuals are regularly updated at the state level and circulated to counties describing processes and requirements for reporting, such as the “Attendance Procedures and Web Data Collection System User Manual” and the “High School Assessment Status and Completers Reporting and Procedures Manual.” Second, with regard to flow of data, schools report data to counties through a variety of different digital systems that are unique to each county. Counties then input student data into the EDW by uploading flat text files through the WDCS. The WDCS is a common, data-inputting interface across different county systems. At the state level, data flow is supported by a Data Governance Plan that guides the response to data requests and informs the collection, reporting, and use of data. Third, multiple points of validation exist to ensure accurate reporting. Data are validated first when they are transferred from schools to counties, and again through the WDCS when data move from counties to the state-level EDW. Finally, access to data is restricted to relevant staff and data privacy measures are strictly enforced at all levels. 4. Utilization for Decision Making Data utilization in Maryland is embedded in the policy framework and enhanced through a common culture that recognizes the value of data. With this foundation, data are utilized in decision making by different stakeholders at all levels of the education system (e.g., policy makers, principals, teachers, administrators, parents, students). Parents and students have real-time access to student learning data. Teachers use data to track progress toward Student Learning Objectives. Principals and school administrators actively use data to evaluate teachers, monitor school progress, and manage school plans. Policy makers use data to monitor education quality and equity, improve accountability, and gauge effectiveness of policies and programs. In addition, data are accessible to the general public, media, and researchers for academic pursuits. Utilization of data in the classroom for instructional gains varies between counties, with some counties stronger in this area than others. Further, digital learning programs used in the classroom are often not recorded in the county EMIS, missing an opportunity to collect another aspect of student learning data. Utilization would be stronger if more collaboration occurred between counties. Minimal collaboration between counties limits opportunities to learn from peers, share resources, and strengthen systems. These opportunities for sharing of resources would be especially advantageous to counties operating on lower budgets. If constructed effectively under an overarching EMIS strategy, counties could benefit from collective bargaining power and more efficient use of resources, while MSDE could leverage statewide experience in EMIS implementation to ultimately achieve better education results. MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Introduction examples, such as how to use collaboration and professional development to improve data utilization. In 2011 the World Bank Group commenced a multiyear Lessons from Maryland’s EMIS can be applied to systems program designed to support countries in at any stage of development, from latent to the most systematically examining and strengthening the advanced systems. performance of their education systems. Part of the World Bank’s new Education Sector Strategy, this Overview of SABER-EMIS evidence-based initiative, called SABER (Systems Approach for Better Education Results), uses diagnostic Information is a key ingredient in an effective education tools for examining education systems and their system. SABER–Education Management Information component policy domains against global standards and Systems (EMIS) aims to help countries improve data best practices and in comparison with the policies and collection, data and system management, and data use practices of countries around the world. By leveraging in decision making. SABER-EMIS assesses the this global knowledge, the SABER tools fill a gap in the effectiveness of a country’s EMIS, with the aim of availability of data and evidence on what matters most informing policy dialogue and helping countries better to improve the quality of education and achievement of manage education inputs and processes to achieve better results. This report discusses the results of overall efficiency and strong learning outcomes. applying the SABER Education Management Information Systems (EMIS) tool in the State of Maryland in the A successful EMIS is credible and operational in United States. planning and policy dialogue as well as teaching and learning. It produces and monitors education statistics The objectives of this paper are twofold. First, it within an education system and has a multifaceted examines the system according to key policy areas, structure, comprising the technological and institutional identifies successes and challenges in the system, and arrangements for collecting, processing, and provides recommendations to support the continued disseminating data (Abdul-Hamid 2014). It is crucial for advancement of EMIS in Maryland. tracking changes, ensuring data quality and timely reporting of information, and facilitating the utilization Second, throughout the assessment, examples are of information in decision making. shared that illustrate good practices in the degree of intent and implementation of relevant policies, providing The SABER-EMIS assessment methodology is built on learning opportunities to stakeholders within Maryland, four key policy areas that are essential to EMIS and as well as those outside of Maryland and the United must be assessed to understand and ultimately States. Within Maryland, this report may be of interest strengthen the system. Each policy area is defined by a to counties, which can learn about what their peers and set of policy levers (actions that help governments reach neighbors are doing with regard to EMIS. Globally a the policy goal) and indicators (measuring the extent to variety of lessons can be drawn, such as the following: which the policy levers are achieved) (figure 1). x How to establish the key enabling conditions for Figure 1: SABER-EMIS Policy Areas and Levers EMIS Policy Areas x The value of transitioning an EMIS from a Policy Levers: legal framework, organizational structure Enabling and institutionalized processes, human resources, compliance focus to a learning focus Environment infrastructural capacity, budget, data-driven culture x Key takeaways from building an integrated EMIS System Policy Levers: data architecture, data coverage, data x The importance of leadership vision and buy-in Soundness analytics, dynamic system, serviceability Policy Levers: methodological soundness, accuracy and Given that Maryland falls between an established and an Quality Data reliability, integrity, periodicity, and timeliness advanced EMIS, these good practices are relevant to education stakeholders around the world. Readers are Utilization for Policy Levers: openness to EMIS users, operational use, Decision Making accessibility, effectiveness in disseminating findings encouraged to explore and learn from EMIS in Action Source: Abdul-Hamid 2014. 1 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 A strong enabling environment lays the foundation for Using the EMIS data collection instrument, policy levers an effective EMIS. Enabling environment refers to the are scored on a four-level scale (latent, emerging, laws, policies, structure, resources, and culture established, and advanced) to assess the extent to which surrounding an EMIS that make data collection, both policy intent and implementation are achieved management, utilization, and access possible. (figure 2). System soundness ensures key processes, structures, Approach and integration capabilities in an effective EMIS. Education data are sourced from different institutions, Intent and Implementation but all data feed into and make up EMIS. Databases The EMIS assessment examines policy intent and the within an EMIS are not viewed as separate databases but degree to which intended policies are effectively as part of the whole EMIS. Key aspects of system implemented on the ground (figure 3). Intent refers to soundness include what data are covered in EMIS and the way in which EMIS and its overarching purpose are how they come together in the overarching system. articulated by decision makers and documented in policies and legislation, as well as standards and strategy Quality data establish the mechanisms required to documents. Assessing intent alone reveals only part of collect, save, produce, and utilize information in an the picture. As such, this EMIS assessment also evaluates accurate, secure, and timely manner. Data quality is a policy execution. Implementation refers to the degree to multidimensional concept that encompasses more than which intentions take place during the day to day just the underlying accuracy of the statistics produced. It activities of stakeholders (e.g., policy makers, county means that not only are the data accurate, but that the administrators, principals, teachers, students, etc.). data address specific needs in a timely fashion. Quality Implementation can be observed through utilization of data lay the groundwork for utilization. EMIS by stakeholders, budget allocation, distribution of human resources, availability of professional An effective EMIS is utilized in decision making by all development activities, communication and users (parents, students, teachers, principals, and dissemination of information, as well as the extent of policy makers) across the education system. An EMIS needs to be used so that measures can be taken to Figure 3: Policy Intent, Implementation, and Outcomes Cycle improve educational quality. Accurate information on education sector performance enables more informed 9 Policies 9 Utilization policies and programs. To assess utilization, it is 9 Vision and Buy-in 9 Processes imperative to understand where decision making occurs, 9 Standards 9 Institutionalization if the capacity to analyze and interpret education data 9 Strategy 9 Budget exists, and if specific data are available to inform Documents 9 Human Resources 9 Communication decisions. Intent Implementation Figure 2: SABER Scoring and EMIS Development 4 Advanced 3 Comprehensive Established enabling 2 Enabling environment, Emerging environment, processes, 1 processes, structure, data Outcomes Basic enabling Latent structure, data management, environment, management, utilization, and Limited enabling processes, 9 Teaching and Learning structure, data utilization in place integration in environment, with some place, with 9 Management processes, management, utilization integration intelligent 9 Planning and Monitoring structure, data analytics management, 9 Transparency utilization 9 Governance Source: Abdul-Hamid 2014. Source: Authors. 2 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 institutionalization across the system. Once policy intent interviews and meetings took place with the following and implementation are analyzed, the EMIS assessment entities: explores the results of these two key components, with a focus on system effectiveness and efficiency, in 1. Maryland State Department of Education addition to teaching and learning, and management and 2. Purposeful sample of public school systems, planning. Strong education systems will ultimately use including Anne Arundel County, Cecil these outcomes to inform the effectiveness of policies County, Charles County, Harford County, and education strategies and make adjustments as Howard County, Kent County, and necessary, creating the cyclical process illustrated in Montgomery County figure 3. In Maryland, EMIS intent and implementation 3. Sample of schools were assessed through desk research, analysis of system 4. Maryland Longitudinal Data System Center applications and utilization, as well as interviews with a variety of stakeholders (table 1). Counties were selected using purposeful sampling based on the stage of EMIS development, as well as population Table 1: Examining Policy Intent and Implementation and budget characteristics. Of Maryland’s 24 counties, Policy intent Implementation seven counties were assessed totaling more than 25 x Multiple meetings with x Interviews with percent of the total number of school systems in the MSDE stakeholders at state, state. x Extensive review of county, and school levels relevant federal, state, x Classroom observations It is important to note that the primary focus of the and county policies x EMIS vendor report is Maryland’s state-level EMIS, not individual x Researched relevant demonstrations (county county systems. The report examines each county-level standards and state level) EMIS to assess implementation and to gauge the planning documents x Review of county systems x Found and compared (e.g., data warehouse, effectiveness of the overarching state policies and federal and state grant student information practices. That said, county-level implementation applications and follow- system, learning examples illustrate good practices, providing learning up reports on utilization management system, opportunities for readers. of grant funding etc.) x Analysis of data quality and comprehensiveness Country Overview x Thorough utilization assessment In the United States, states have direct oversight over x Examination of most aspects of the public education system, performing professional development political, administrative, and fiscal functions that are activities and outcomes often the work of ministries of education in countries Source: Authors. with centralized education systems (U.S. Department of Methodology Education 2008). Policies at the state level define the education system within each state, including critical The EMIS assessment methodology consists of a review factors such as curriculum, assessments, teacher of written policies and technical documents as well as qualifications, and resource distribution as well as what interviews with key stakeholders across the education data are collected and when data must be reported. system to ensure proper implementation. Local education agencies (LEAs) at the county (or district) level implement and enforce these requirements. In Research and investigation for the Maryland EMIS operating local school systems, they also develop and assessment took place between August and December implement their own policies, hire and supervise 2014. The authors conducted a comprehensive review of teaching staff, and raise money. LEA structures vary by federal, state, and county policies, as well as technical state and region but are generally managed by a documents and other background materials. To further governing body referred to as the school board. EMIS examine intent and implementation, a series of 3 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 data are often a central part of school board decision collect, collate, analyze, and report complete statistics making. on the condition of American education; conduct and publish reports; and review and report on education School system revenue is generated by federal, state, activities internationally. and local sources, with the majority of funding coming from state and local entities. The federal government is According to NCES, roughly 10 percent of all students intended to supplement funding, with a focus on attend private schools (table 2). With regard to private increasing equity by providing additional funding to school data, NCES has conducted the Private School underserved school systems. Most federal funding is set Universe Survey every two years since 1989. The survey annually through the congressional appropriations generates biennial data on the total number of private process. State funding comes primarily from income and schools, teachers, and students and provides a list of sales taxes, and local funding comes from property taxes. private schools to serve as a sampling frame for additional NCES analysis. Given this decentralized education system, EMISs vary significantly from state to state, and in some states they An interesting approach to education reform in the U.S. vary from county to county. All state EMISs report data is the formation of charter schools. The concept of to the federal government based on predetermined charter schools emerged in the 1970s and 80s. Initially schedules, and they serve the same education structure. designed as legally and financially autonomous public Public school systems provide education services for schools, the model has gained steady momentum over children in pre-kindergarten through 12th grade. the last several decades, with roughly 5 percent of public Elementary school (primary) reaches students from school students nationwide enrolled in charter schools kindergarten to grade 5, middle school serves grades 6 to (NCES 2015). According to the United States Department 8, and high school (secondary) serves grades 9 to 12 of Education (2015), a public charter school is a publicly (figure 4). funded school that is typically governed by a group or organization under a legislative contract (or charter) with The National Center for Education Statistics (NCES) is the the state or jurisdiction. The charter exempts the school primary federal entity for collecting and analyzing data from certain state or local rules and regulations. In return related to education in the United States and other for flexibility and autonomy, the charter school must nations (NCES 2015). It is located within the U.S. meet the accountability standards outlined in its charter. Department of Education and the Institute of Education Sciences. NCES fulfills a Congressional mandate to Figure 4: U.S. Education Structure, through High School Source: U.S. Department of Education, National Center for Education Statistics, Annual Reports Program. http://nces.ed.gov/programs/digest/d13/figures/fig_01.asp. Accessed January 4, 2016. 4 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Table 2: U.S. Education Indicators, at a Glance Maryland Overview Schools and enrollment Maryland is located in the Mid-Atlantic region of the Institution Schools Enrollment United States, bordering Delaware, Pennsylvania, Public pre-kindergarten 35.0 million Virginia, Washington, D.C., and West Virginia. It through grade 8 (2012–13) comprises 24 school districts (or counties) that serve 92,375 Public grades 9 through 12 14.8 million more than 866,000 students (table 3). The state is (2012–13) governed by the State Board of Education, and the Public charter schools (2012– 6,100 2.3 million Maryland State Department of Education (MSDE) is led 13) by the State Superintendent. The department has three Private pre-kindergarten 4.0 million key offices: the Office of the Deputy for School through grade 8 (2011–12) Effectiveness, the Office of the Deputy for Teaching and Private grades 9 through 12 1.3 million Learning, and the Office of the Deputy for Finance and (2011–12) Administration (appendix A: Maryland State Department Percent of students in private schools 9.6% of Education Organizational Chart). The state-level EMIS (2011–12) is managed by the Office of Teaching and Learning in the Attainment Division of Curriculum, Assessment and Accountability 2013 2014 (DCAA). High school completion 90% 91% Bachelor’s or higher degree 34% 34% Maryland’s highly decentralized education system Master’s or higher degree 7% 8% creates a significant amount of variation in EMIS across Performance the state. MSDE is the central collector of education data Proficient grade 8 reading 36% and is also responsible for sharing county-level data with (2013) the federal government for compliance purposes (figure Proficient grade 8 35% 5). Each county selects and manages their own mathematics (2013) Mathematics literacy of 15- 481 (international avg. information system that collects data from schools. As year-olds (PISA 2012) is 494) described in the methodology section, this report refers Graduation/dropout to both individual county-level EMISs as well as the Public high school graduation ratea 81% overarching state-level EMIS, with the latter being the (2011–12) primary focus of the assessment. MSDE collects minimal Percentage of 16- to 24-year-olds not 7% data from private schools. Provision of data from private enrolled in school who have not schools is voluntary, although most provide aggregate- completed high school (2013) level data. Funding provides some incentive, because Poverty (2013) student enrollment data are required to receive state Percentage of 5- to 17-year-old children 20.7% funding. Private schools are also required to provide in families living in poverty information when they absorb students who were Pupil-teacher ratio previously in the public school system. Public schools 16.0 Figure 5: EMIS in Decentralized Education Systems Private schools 12.5 Public school revenue and expenditure (2011–12) Total revenues $620 billion Total expenditures $621 billion evel E State-level EMIS Current expenditures per studentb $11,014 County-level y-level E County EMIS EMIS County-level EM Source: NCES 2015, http://nces.ed.gov/programs/coe/ataglance.asp. a. Graduation rate is based on the Averaged Freshman Graduation Rate (AFGR), which is the number of high school diplomas awarded expressed as a percentage of the estimated freshman class four years earlier. Schools County-level EMIS Schools b. Current expenditures refers to school operations and does not include capital outlay (expenditures for property and for buildings and alterations completed by school district staff or contractors) or interest on school debt. Expenditures are reported in constant 2013–14 dollars, based on Schools Source: Authors the Consumer Price Index. 5 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Table 3: Maryland Education Indicators, at a Glance c. 4-Year Adjusted Dropout Rate calculated by dividing total Dropouts by 4-Year Adjusted Cohort. Students who drop out of high school General information remain in the adjusted cohort—denominator of calculation. Number of counties in the state 24 Total public schools (2013–14) 1,448 Maryland in Context Total nonpublic schools (2013–14) 1,425 Estimated state population (2013) 5,928,814 Compared to state education systems across the United Enrollment States, Maryland is among the highest performers. Stage 2013 2014 Maryland’s public schools ranked among the top three in Elementary (K–5) 322,048 327,994 the nation for the past eight years in Education Week’s Middle (6–8) 184,187 187,227 “Quality Counts” report (Education Week 2015). In 2015, High (9–12) 256,836 253,589 U.S. News & World Report rated Maryland number 1 for Total (Pre-K–12) 859,638 866,169 best high schools nationwide. Additionally, for the last Attendance nine years Maryland has ranked first in the percentage of 2013 2014 public school students scoring a 3 or better on at least Elementary 95.5% 95.7% one Advanced Placement exam (Baltimore Sun 2015b). Middle 95.1% 95.4% High 92.5% 92.7% Additionally, the National Assessment of Educational Performance Progress (NAEP), a nationally representative and 2013 2014 continuing assessment of student performance across Grade 8 MSA, 67% 58.7% the United States, also shows strong results for mathematics Maryland. NAEP is administered by the U.S. Department Grade 8 MSA, reading 81% 76.9% of Education and disseminated via The Nation’s Report SAT (composite mean of 1,456 1,439 Card (nationsreportcard.gov). The 2013 NAEP 2,400 total points) achievement-level percentages show Maryland Graduation/dropout generally performing better than U.S. public school 2013 2014 averages (figure 6). Graduation (4-year 85% 86.4% adjusted)a Dropout (annual)b 3 3% Dropout (4-year 9.4% 8.4% Figure 6: NAEP Achievement-Level Results adjusted) c Teacher qualifications (2013–14) Achievement Level Percentages, Less than Bachelor’s degree 0.5% Maryland and U.S. Bachelor’s degree 42.6% Below Basic Basic Proficient Advanced Master’s or Master’s equivalent 37.6% Master’s degree + 30 hours or more 19.3% U.S. 23 42 31 4 Expenditure (2013–14) Reading Local operating budget from federal, $12.2 billion Maryland 18 40 36 7 Grade 8 state, and local sources (includes U.S. 33 33 26 8 state-paid retirement) Reading Percentage of state budget spent on 28.8% Maryland 23 32 30 14 Grade 4 education U.S. 27 39 26 8 Average spending per student $13,572 Math Sources: MD Report Card 2014; MSDE Fact Book 2014; U.S. Census Grade 8 Maryland 26 37 26 12 2013. a. 4-Year Adjusted Graduation Rate is calculated by dividing total U.S. 18 41 34 8 Math Diplomas Earned by 4-Year Adjusted Cohort. Students who drop out of high school remain in adjusted cohort—denominator of calculation. Maryland Grade 4 18 36 33 13 b. Annual dropout rate computed by dividing number of dropouts by total number of students in grades 9–12 served by the school. Source: nationsreportcard.gov. 6 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Maryland consistently performs better in the advanced This is much higher than the U.S. average, where only category and places well above U.S. averages in 32.2 percent of the students had a score of 530 and 7 proficient reading for grades 4 and 8. Maryland also percent were in the advanced category. Countries whose outperforms the United States, with fewer students performance levels are similar to that of Maryland scoring in the below basic category. include Austria, the Czech Republic, France, Hungary, Poland, Slovakia, and Slovenia. Maryland’s performance NAEP is also used to compare U.S. results with those is also comparable to the top performing U.S. states of from international assessments, including Progress in Minnesota, New Jersey, and Vermont (Peterson et al. International Reading Literacy Study, Program for 2011). International Student Assessment (PISA), and Trends in International Mathematics and Science Study (TIMSS). Maryland EMIS Results Maryland fairs well in international assessments, reflecting a quality education system. For example, the This section presents the main results of EMIS NAEP-TIMSS Linking Study predicted an average TIMSS diagnostics described in the previous sections. Results score of 514 in Maryland for eighth-grade mathematics and scores for each policy goal are presented, along with and 528 for eighth-grade science, which is higher than supporting evidence. the average score of Australia, Chile, Finland, Hungary, Norway, Slovenia, and OECD countries (OECD 2014) Policy Area 1: Enabling Environment (figure 7). Advanced zzzz Maryland’s enabling environment was assessed in the According to a 2011 Harvard Kennedy School Report that following areas: (1) Legal Framework; (2) Organizational used NAEP to map PISA scores, 36.5 percent of the Structure and Institutionalized Processes; (3) Human students in Maryland had an average score in math well Resources; (4) Infrastructural Capacity; (5) Budget; and above 530 (proficient in NAEP), and 10.1 percent of the (6) Data-driven Culture. students had a score of 623 (advanced level in NAEP). Figure 7: Maryland in International Context, Using TIMSS EMIS in the United States has been especially influenced by three policies: (1) the Elementary and TIMSS Math Achievement, 2011 (Grade 8) Secondary Education Act (ESEA) of 1965, reauthorized in 2001 as the No Child Left Behind (NCLB) Act; (2) the Korea 613 Family Educational Rights and Privacy Act (FERPA); and (3) the Children's Online Privacy Protection Act (COPPA) Maryland 514 (table 4). Finland 514 Table 4: Federal Legislation Influencing EMIS Federal Australia 505 Summary descriptions legislation Provides funding and promotes equal Hungary 505 access to education, established standards, and accountability. Also provides an Slovenia 505 ESEA/NCLB opportunity to move from data for compliance to data utilization for student OECD 504 learning outcomes. Protects the privacy of student educational Norway 475 records and applies to all schools that FERPA receive funding from the U.S. Department Chile 416 of Education. Governs the collection of information that COPPA Source: Authors’ calculation based on TIMSS 2011 database; NCES 2013. is gathered online from children under the 7 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 age of 13 and applies to the operators of data targets. For example, as part of the American websites and online services directed at Recovery and Reinvestment Act (ARRA) of 2009, Race to children. COPPA is enforced by the Federal the Top (RTTT) is a $4.35 billion initiative built on the Trade Commission. framework of comprehensive reform in four core areas: Sources: U.S. Department of Education; Federal Trade Commission. adopting rigorous standards and assessments that prepare students for success in college and the At the federal level, ESEA/NCLB outlines accountability workplace; recruiting, developing, retaining, and steps that have critical implications for EMIS. Key rewarding effective teachers and principals; building legislation is included in Improving Basic Programs data systems that measure student success and inform Operated by Local Education Agencies (ESEA/NCLB Part teachers and principals how they can improve their A, Section 1111), which specifically states the following: practices; and turning around the lowest-performing schools. x Each state shall have a statewide accountability system that ensures that all local educational Additionally, the Education Sciences Reform Act (ESRA) agencies, public elementary schools, and public of 2002 initiated the Statewide Longitudinal Data secondary schools make adequate yearly Systems (SLDS) Grant Program to support the progress toward the state’s student academic development and implementation of states' longitudinal achievement standards data systems, as well as the expansion of K –12 systems x That adequately yearly progress be defined in a to include data from pre-school to the workforce (P- manner that is statistically valid and reliable 20W). Additionally, the SLDS program seeks to help x Each state shall establish statewide annual states, districts, schools, and educators make data- measurable objectives and intermediate goals to informed decisions to improve student learning. As of meet objectives 2012, SLDS awarded grants totaling $514 million. x Each state shall develop a uniform averaging Maryland received three SLDS grants: $5,690,718 (2006), procedure to track the progress of schools $5,990,186 (2009), and $3,963,473 (2012). To drive toward reaching adequate yearly progress state-level implementation of the longitudinal data x Each state shall establish a set of high-quality, system, Maryland enacted the Maryland Longitudinal yearly student academic assessments that Data System Act (Chapter 190, Senate Bill 275) in 2010 to include, at a minimum, academic assessments in establish the Maryland Longitudinal Data System as a mathematics, reading or language arts, and statewide data system containing certain student data science that will be used as the primary means of from all levels of education and into the state’s determining the yearly performance of the state workforce. toward meeting the state's student academic achievement standards At the state level, the Code of Maryland Regulations x Each state educational agency may incorporate (COMAR) ensures compliance with and implementation the data from the assessments under this of state law and is the key policy document that defines paragraph into a state-developed longitudinal and guides Maryland’s EMIS. COMAR effectively data system that links student test scores, length establishes state-level compliance with federal of enrollment, and graduation records over time legislation. Specifically, COMAR Title 13A mandates data x Each state that receives assistance under this supply by requiring all schools to maintain a system of part shall prepare and disseminate an annual information on enrollment, attendance, and promotion state report card in a concise, understandable of students to track annual measurable objectives, the and uniform format and annual performance targets established by the State x Each state shall collect and disseminate Board. COMAR also includes timelines for reporting, as information in a manner that protects the well as penalties if data are not reported or if privacy of individuals. intentionally misreported. Several other key federal policies and grant initiatives The Division of Curriculum, Assessment and have supported states (including Maryland) in reaching Accountability is responsible for the state-level EMIS. 8 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 The division’s responsibilities include (MSDE 2013) the Table 5: Compliance vs. Learning-Driven EMISsa following: Compliance-driven Learning-driven x Narrow focus on x Using data to inform x Administration of the Maryland School reporting data to comply instruction, especially at Performance Program's annual Report Card. This with state and federal classroom level, high annually collected data provides accountability policies levels of teacher on the state, school system, and school levels. x Lack of integration utilization x Facilitation of several divisions within the x Limited utilization, x Using data to inform Department and local school systems for the especially in the management at all levels development, administration, and scoring and classroom (school, county, state, reporting of all assessments, as well as providing x Focus on summative federal) assessments (to evaluate x Using data to predict at- support in monitoring adherence to test security learning) such as state risk youth and intervene requirements. early and national assessments, x Maintenance of the Education Data Warehouse not formative x Highly integrated data and the collection of data from local school assessments (to monitor comprising systems and other entities, and the validation, learning and growth) such administrative, learning definition, and maintenance of multiyear data in as quizzes (formative and accordance with Department and Division summative assessments), policies and procedures to ensure data quality human resources, and and accessibility. financial data to fully understand how inputs impact outcomes Since ESEA passed in 1965, data supply and utilization Source: Authors. in Maryland (and across the United States) have a. Summary list, not exhaustive. evolved from a compliance-driven approach to a In 1972 Maryland passed an educational accountability learning-driven approach (table 5). Several early policies law that mandated statewide goal setting and testing positioned Maryland as a pioneer in using education (Michaels and Ferrara 1999). According to Dr. Smith, the data, not just for compliance with federal policies, but innovative approach was ahead of the curve. School also to inform instruction at the school level (Dr. Jack systems reported school performance through nationally Smith, Chief Academic Officer, Office of Teaching and normed tests and MSDE published an assessment Learning, MSDE, interview with authors, August 2014) accountability testing program report for every school (figure 8). and school system. Michaels and Ferrara (1999) underscore Dr. Smith’s statement: “Although school Figure 8: Policies Drive Evolution from Compliance to Learning report cards are fairly common today, in the 1970s it was innovative and even revolutionary to provide a public accounting for the performance of individual schools.” In 1977, Project Basic was implemented to account for gaps in the Maryland Accountability Testing Program report such as the lack of attendance information and disaggregated performance data. Project Basic had two primary components: (1) a basic skills framework of 165 competencies that all schools were required to cover by the end of grade 8 and (2) minimum competency tests in reading, writing, math, and citizenship that students Source: Authors. must pass to receive a high school diploma. In May 1991, the Maryland School Performance Assessment Program (MSPAP) was first administered to 9 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 approximately 150,000 students in grades 3, 5, and 8. Figure 9: EMIS Tracks Decline in State Assessment Results Performance on the MSPAP was used to evaluate schools during Curriculum Reform and to provide information to guide school improvement Common Core efforts. Curriculum Begins In 2001, these early efforts were further supported at the federal level with the reauthorization of ESEA as No Child Left Behind, which mandated annual testing, reporting of individual student results, and disaggregation of results by race and special services groups. Soon after, between 2003 and 2015, a series of curriculum and assessment reforms were adopted in Maryland, which used assessment data not just for compliance with federal policies, but also to guide instructional improvement. In 2003 the Maryland School Assessment (MSA) Program launched. Results were used to evaluate school Source: Washington Post 2014. performance, guide school improvement, and inform Maryland’s Anne Arundel County has been at the instruction. In 2010 the Maryland General Assembly forefront of implementing the Common Core Standards passed the Education Reform Act, and the State Board of and the PARCC assessment (see Anne Arundel County Education adopted the Common Core State Standards, a Public Schools 2012). Data shared via the county’s EMIS more rigorous and thinking-based set of content have been a central part of this implementation process standards. Dr. Smith explained that the Common Core– because the data have provided constant feedback on based assessments mark a critical shift to measure the ability of teachers to teach the new curriculum, as student learning growth for both school accountability well as the extent to which students are learning the new and educator evaluation. The Common Core curriculum curriculum. Following the decline in MSA results, introduced the need for a new assessment; thus in 2015, Superintendent of Anne Arundel County, George Arlotto, Maryland administered the Partnership for Assessment remarked, “There remains in these results data that is of Readiness for College and Careers (PARCC) assessment useful to administrators, principals and teachers as we for the first time. continue to move forward.” Maryland’s recent curriculum and assessment reform Comprehensive and quality data are established process underscores the importance of a sound EMIS through policy and reinforced with regularly updated that accurately collects, analyzes, and distributes manuals. The Maryland Student Records System assessment data. In this case, EMIS was a useful tool in Manual, updated and approved for publication in May tracking the alignment of curriculum and assessment 2015, provides instructions and sample forms to assist in results. In 2013 EMIS data on the MSA revealed a decline the maintenance of required information on enrollment, of 5 percentage points for elementary students and attendance, and promotion. In a decentralized system, nearly 7 points for middle school students (figure 9), a this type of instructional manual is absolutely essential in result of the shift to new academic standards that were ensuring that county-level data, stored in a myriad of not yet fully aligned with assessments. State and county different information systems, often using different educators anticipated the alignment gap and EMIS data software, can integrate with the state-level EMIS. were able to verify it, while continuing to provide important information about the achievement of specific The processes for sharing and reporting data from student groups, classrooms, and schools. Maryland county to state and state to federal government are responded to the publically available data immediately, documented in the legal framework, including timelines conducting outreach to schools and parents to explain and penalties for failure to comply. Additionally, the the reform process and the reasons for the decline. Maryland Longitudinal Data System Act positioned the Maryland Longitudinal Data System Center to serve as a 10 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 central repository of student and workforce data, At MSDE, the Division of Curriculum, Assessment and including data sets provided by the following: Accountability maintains the Education Data Warehouse. It is responsible for the collection of data x State Department of Education from local school systems and other entities, as well as x Local education agencies the validation, definition, and maintenance of multiyear x Maryland Higher Education Commission data in accordance with documented policies and x Institutions of higher education and procedures to ensure data quality and accessibility x Department of Labor, Licensing, and Regulation. (MSDE 2003). The Office of Accountability consists of an Analysis and Data Systems branch, an Accountability The legal framework does not specify or encourage data Support Services branch, and a Research and Evaluation sharing between local education agencies, a factor that branch (appendix B: Division of Curriculum, contributes to high levels of variation in the quality and Accountability and Assessment Organizational Chart). scope of EMISs between counties. Differences in budgets Each branch is staffed with technical experts. also contribute to this variation. In some cases, opportunities exist for counties with fewer resources to At the county level, organizational structures vary due to collaborate with neighboring counties to share resources high levels of autonomy. In some cases EMIS is managed and good practices. Currently the legal framework does by a Chief Accountability Officer; in other cases it may be not explicitly support this type of collaboration. At the a collaboration between the technology and business state level, the budget for EMIS is spread out and departments. Most counties have educational strategies separated by vendor, generally comprising roughly 15 to that are tracked using education data. Additionally, 20 line items (MSDE in interview with authors 2014). processes for collecting, managing, and reporting data are documented, and responsibilities for associated Maryland’s legal framework emphasizes data tasks are assigned. At both state and local levels staff utilization, transparency, and data-driven decision working on EMIS are qualified. Often EMIS teams will making. Martin O’Malley who served as Governor of consist of staff with various advanced degrees and Maryland from 2007 to 2015, championed numerous technical skills such as doctoral and master’s degrees, as policies and programs that supported data-driven policy well as statisticians, data analysts, and database decisions, goal setting, and tracking. Among these architecture experts. At times, technical experts hired as initiatives are the Longitudinal Data System Act, the contractors to perform specific, highly technical, short- Open Data Policy, as well as programs such as StateStat, term tasks. a data-based management approach to make Maryland's government more effective and accountable. According Professional development opportunities for EMIS staff to the Data Quality Campaign, a nonprofit organization vary between state and local levels and across counties. committed to improving the availability and use of high- At the state level, opportunities exist for EMIS staff to quality education data to improve student achievement, attend and participate in conferences, as well as to Maryland has 8 of 10 Actions to Ensure Effective Data pursue additional training opportunities. Additionally, Use, compared with four in 2011. Maryland has Teacher Professional Development Standards, which are intended to guide efforts to The legal framework ensures confidentiality of improve professional development for all teachers. They respondents’ data through a variety of federal and state are derived from the National Staff Development policies. At the federal level, key policies include the Council's Standards for Staff Development. Standard 7 is Family Educational Rights and Privacy Act and the titled Data-driven and states that effective teacher Children's Online Privacy Protection Act. Various state professional development relies on rigorous analysis of policies mandate further requirements in management data. Indicators for this standard include (MSDE 2014a) of and access to student data, especially COMAR’s the following: Maryland Student Records Regulations (COMAR 13A.08.02). Privacy statements exist on both the MSDE x Access to high-quality student data from various website as well as the Maryland Report Card website. sources, organized in user-friendly formats 11 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 x Knowledge and skills necessary to use reporting student information and student assessment disaggregated student data for planning, data. To train teachers on the new system, the county implementation, and evaluation of professional provided the option of either an online Moodle Course development and instructional programs or an in-person trainer based on teacher preference. The x Schools and districts make time for teachers and training lasted six to eight hours and finished with a quiz others to examine student data as the starting and an option to retake the quiz if necessary. The county point for planning professional development also trained roughly 55 Technology Liaisons, who were x Disaggregated student data are analyzed to full-time teachers trained to provide technology support. identify gaps between student learning and Chris Wilkinson, former Instructional Data Specialist with standards for proficiency to inform the choice of Harford County’s Curriculum, Instruction, and the content of professional development and Assessment Department remarked that the technology x As appropriate to school and district needs, data support provided by liaisons was a critical part of analysis focuses on results from approved successful user adoption. national, state, and local assessments, as well as student work samples and portfolios and In addition to the liaisons, various professional behavioral indicators such as attendance and development materials such as brochures, simple step- disciplinary referrals. by-step directions, and automated help buttons in the system (designed by Performance Matters) helped new Professional development for EMIS staff at the local users learn how to navigate the new EMIS. Wilkinson level depends on EMIS team size and available budget. explained that instead of a lengthy manual, the county Counties are resourceful, designing programs often with provided a series of one-page sheets to tailor trainings to limited resources. For example, when Harford County teachers’ busy schedules (figure 10). implemented a new EMIS, the county worked with the Teachers Association and used the “train the trainer” Figure 10: Harford County Training model to effectively reach the necessary school staff (box 1). Harford worked with vendor, Performance Matters, to build an integrated system for recording, tracking, and Box 1: Train the Trainer Model Scales EMIS across Schools Training a group of trainers who then take EMIS skills back to their schools is an efficient way to scale EMIS Student Information System training. In Harford County teachers were trained to work as Technology Liaisons in their respective schools and supported by a suite of learning tools. Harford’s EMIS training checklist includes the following: x Highly interactive and visual courses to make the learning experience informative and fun x Motivate participants with engaging material and certificates upon completion x Flexible learning options offer either online Moodle courses or in-person training x Ongoing and easy to use support—such as Source: Harford County 2014. one-to-two-minute video lessons and short pamphlets—recognizes teachers’ busy schedules and need to find answers fast Source: Harford County in interview with authors 2014 12 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 “Looking at data as a tool to inform teaching was a new Figure 11: MDK12 Homepage skill for many of our teachers,” explained Wilkinson. “They generally looked at it as a result, but simple professional development tools customized to their needs helped them to see data as a tool to better themselves as teachers.” Maryland’s data-driven culture is part of a national trend. Across the United States, a movement around open data is well underway with platforms such as data.gov, cities.data.gov, counties.data.gov, and states.data.gov. The private and social sectors are also Source: mdk12.msde.maryland.gov. seasoned supporters of this movement. The Data Quality MDK12 was initially funded as part of a research grant Campaign is a strong example of the social sector taking from the U.S. Department of Education’s Office of a key role in promoting effective utilization of education Educational Research and Improvement, later replaced data. National movements around the value of data and by the Institute of Education Sciences. When the grant data utilization have the potential to positively influence finished, MSDE adopted the project, maintaining and state-level education systems. expanding its objectives. MSDE’s ability to maintain the initiative is a positive outcome. Such initiatives may be Maryland’s enabling environment is supported by a even more impactful if launched under the umbrella of strong data-driven culture that prioritizes data as a MSDE and as part of the MSDE website. Not only does fundamental element of operations and decision this make it easier for stakeholders to find the website, making. An example of a tool that promotes data-driven but it also shows alignment with the larger MSDE culture is MDK12.org (figure 11). The site was created to education strategy. help schools analyze their state assessment data and guide them in making data-driven instructional decisions that would support improved performance for all Policy Area 2: System Soundness students. Contributing to the content and resources on Established zzz| the site were talented educators from across Maryland. Maryland’s EMIS soundness was assessed in five critical Specifically, the site was designed to serve as a practical areas: (1) Data Architecture; (2) Data Coverage; (3) Data tool usable by a variety of stakeholders to assist in Analytics; (4) Dynamic System; and (5) Serviceability. understanding, analyzing, and making use of student achievement data (MSDE 2014b). According to MSDE, Of the seven counties examined during the assessment, feedback from principals and teachers indicates that none of them had the same EMIS structure, and all were many of them have found the website useful in in different stages of upgrade (table 6; see appendix C improving student performance. for extended table). All of the counties had a central data warehouse or database that integrates with The site includes a Toolkit to promote data-driven administrative data and learning outcomes data. This culture, consisting of presentations, background structure is critical for compliance purposes. The materials, and assessment tools that help schools use counties do not have EMISs that integrate administrative data to reach goals. The site also provides various user and learning outcomes data with financial and human guides for different stakeholder segments and data resources data. Some of the county systems incorporate analysis tutorials on topics such as the following: some human resources data, such as teacher evaluations; however, extended human resources data x How is the School Progress Index Calculated? are generally captured in other systems. Likewise, some x How MSDE Calculates Graduation Rate? financial data are captured in these systems, but x How Dropout Rates are Calculated? extended data, especially on salaries or professional development expenses, are captured outside of EMIS. 13 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Table 6: Examples of Counties and Their Data Warehouses education system (figure 12). Currently Maryland is County Data Warehouse Vendors transitioning from a siloed system, into a more Anne Arundel PowerSchool™ integrated system. Likewise, many of the county EMISs Cecil County Pearson Inform™ are also pursuing greater integration of data. Charles County Amazon Web Services™ Harford County Performance Matters™ (Assessment Figure 12: Moving toward an Integrated Approach and Data Management System) Howard County edVantage™ Kent County SchoolNet™ Admin Learning Montgomery Built in-house with MicroStrategies™ data data County Finance Data Warehouse Source: Authors. data A comprehensive EMIS should include administrative Data A L F H.R. data, financial data, human resources data, and Warehouse H.R. learning outcomes data (table 7). This information data should be available at both the individual and aggregate level. The type of data entered into the system needs to Source: Adapted from Abdul-Hamid 2014. follow logic, fixed methodology, and have a well-defined purpose (Abdul-Hamid 2014). Maryland’s state-level EMIS was built in-house using an Oracle platform and comprises the Educational Data Table 7: Data Coverage, Best Practice, and Maryland Warehouse (EDW), Longitudinal Data System Type Best practice Maryland Operational Data Store, Web Data Collection System, x Demographic x Demographic and Report Card Data Mart. Established in 1999, the x Health x Health EDW is the hub for statewide K12 education data and Admin. x Student progression x Progression includes staff and school information. For students the Data (enrollment, repeat, x School-level data following information is captured: attendance, dropout, etc.) enrollment, dropouts, retention and progression, x School-level data student discipline, High School Assessment data, MSA x Budgets and x Some budgets data, national standardized assessment data (SAT, A.P., revenues and revenues Financial ACT). Additionally, graduation rates, special education, x Spending x Cost per student data high school completion, and grade 12 documented x Subsidies x Cost per student decisions are also recorded. Staff assignments, x General x Some general certifications, and Highly Qualified Teachers are also demographics demographics recorded in the warehouse, although staff salaries are Human x Salaries x Some not included. Resources x Performance performance data evaluation evaluation Additional databases that do not integrate with the EDW x Professional include the Employee Information System, Early development Childhood, Career and Technology Education Access, and x Classroom x National and the Special Education Database. The Report Card Data Learning assessments state Mart is used to annually disseminate information via the outcomes x National assessments assessments Maryland Report Card. data x International x International assessments assessments Source: Adapted from Abdul-Hamid 2014. The EDW system is a distributed, multitier system, with several Oracle databases hosted in a multiserver Dell Further, data stored in integrated as opposed to siloed environment. Educational data are inserted into the systems provide more opportunities for advanced Oracle databases via Informatica ELT programs. Reports analytics, ultimately enabling greater insight into an and data analysis are performed using COGNOS C8 BI, 14 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 SAS 10, and OBIEE 11g (MSDE 2013–14). All systems are Information System, a Learning Management System, defined based on a wireframe that illustrates integration and a Data Warehouse. capabilities and movement of data (figure 13; see appendix D for larger version). The cost structures for each system were negotiated and ultimately were well within budget requirements for the Figure 13: MSDE EMIS Wireframe county (table 8). According to Howard County, the budget for the integrated system totaled $5.8 million, which is just under 1 percent of the FY 2015 operating budget of $758.8 million. Table 8: System Cost per Student System Cost per student ($) Student information system 9.33 Learning management system 2.68 Data warehouse 7.62 Source: Howard County 2015. The student information system is a comprehensive, web-based system that includes student demographics, d d enrollment, registration, scheduling, attendance, grading, discipline, transcript, and master schedule Between 2012 and 2015, Howard County implemented builder. It equips teachers, administrators, and parents a cutting edge EMIS. The success of the new system with the power to obtain up-to-the-minute access to stemmed from the county’s ability to develop key student information. enabling conditions for EMIS, most notably, vision, strategy, and human resources. The process started in The learning management system is a cloud-based 2012, when the county hired a new superintendent. With learning platform that teachers, students, and parents the support of the new superintendent, Howard County can access on any device at any time. It integrates with created the Office of Accountability and hired a chief the student information system to seamlessly populate accountability officer to steer the process of identifying class information and pass critical student data. and implementing an effective EMIS and a strong accountability strategy. The data warehouse is a centralized repository of data that empowers administrators, principals, and school Following an extensive assessment, including internal staff with timely and accurate longitudinal data to inform dialogue and focus groups, as well as several external instructional practices and student performance and audits conducted by the Center for Education Policy provide an opportunity to increase student achievement. Research at Harvard University, IMPAQ International, and Pearson, the chief accountability officer and her Reflecting on the implementation of the integrated team determined that existing data systems did not EMIS, Howard County identified key advantages that the provide adequate data to support the county’s strategic new system introduced as well as implementation plan. The team needed multiple data points to effectively challenges (table 9). A key outcome in addition the inform decision making, and they needed data to actively advantages below is that the new system provides a monitor progress toward goals. Constructing a system meaningful foundation for data-driven target setting. that could integrate the data and make them readily Before the integrated system was established, available and easy to analyze was key. Guided by a performance management targets were not aligned to strategic timeline (appendix E: Howard County Strategic the overarching strategy, nor were they consistently Plan for New EMIS), the decision was made to build a tracking and facilitating growth (Hitch and Fullerton comprehensive, integrated system including a Student 2013). 15 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Howard County also provided a list of what they Table 9: Howard County Reflections on Implementation of identified as key success factors. These are the essential Integrated EMIS elements that made implementation possible and Advantages Challenges helped overcome obstacles: x Ability to obtain accurate, • Communication outside real time data division x Leadership support and vision x Single point of entry • Aggressive x Funding x Longitudinal analysis implementation timeline x Parallel approach to project implementation x Predictive analysis • Training of 8,000+ staff on x Skilled technical staff x Promotes equity three products x Project timeline and milestones • End user buy-in Source: Howard County 2015. x Monitoring (daily, weekly) x Communication within division and with vendor education, along with student social security numbers, x Collaboration, internally and externally creating a bridge over a potential “identity gap,” so that x Detailed training plan the longitudinal data system is able to follow the x Managing and understanding the functional progression of students across all four stages (Early details (specifications) of systems Childhood, K-12, Postsecondary, to Workforce) (figure x Ability to influence change in culture 14). x Equip staff with the right tools and technology Figure 14: SASID Helps Link to Workforce x Flexibility Early PK-12 Postsecondary Workforce All Maryland public school students are assigned a State Childhood Assigned Student Identifier (SASID), a unique number that is assigned to and remains with a student throughout his or her career in Maryland public schools. The SASID enables Maryland to track students as they SSN SASID move from grade to grade and across campuses and/or SASID SSN SSN districts within the state. It also enables MSDE to Source: Maryland Longitudinal Data System Center 2014. longitudinally track student performance. Maryland is also participating in efforts to link student The SASID is defined and assigned by the Maryland State learning with teacher inputs; however, the extent of Department of Education through the Unique Student implementation is unclear. The state will participate in Identifier System (USIS). The system has been phase II of the Teacher-Student Data Link (TSDL) Project, operational since 2008 and tracks students as far back as an initiative that brings states and school districts 1986. County education offices are responsible for together to work on a common approach to linking requesting and maintaining SASIDs for all students teachers’ inputs and students' learning outcomes. Across enrolled in their county using the USIS system and must the nation, states are looking to use this type of notify MSDE via USIS of any changes to student accountability data to increase student learning. In information and provide required supporting Maryland, the TSDL collection is referred to as the documentation. Updates include changes in student Student Course Grade Teacher (SCGT) collection. On a grade, boundary school, and name changes. All SASIDs semiannual basis it captures a student and his or her reported to MSDE are validated via USIS (MSDE 2015). demographics with a teacher and his or her The SASIDs are archived after students reach the end of demographics, as well as course and grade the academic year of their 22nd birthday. (performance) information. With support from federal grants, the architecture and framework for the SCGT The SASID is also a central component that enables collection are in place and trainings are underway; integration with other systems. It provides a key element however, it does not yet appear to be used in regular in structuring identifiers for the longitudinal data system. practice at county and school levels. It is also used as an identifier in postsecondary 16 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 The Maryland Longitudinal Data System (MLDS) is a Ddd2: MLDS Guiding Policy Questions Box statewide data system that aims to integrate individual- level student data and workforce data to inform Postsecondary Readiness and Access decision making and ultimately improve the state’s 1. Are students academically prepared to enter education system. The MLDS is complex, consisting of postsecondary institutions and complete programs multiple different data owners, interagency data in a timely manner? stewards, robust data architecture, and a myriad of data 2. What percentage of high school exiters go on to security, quality, and analytical tools and processes enroll in postsecondary education? (appendix F: Maryland Longitudinal Data System Center, 3. What percentage of high school exiters entering Data Process Flow). college are assessed to need to take developmental courses, in what content areas? The Maryland Longitudinal Data System Center is a state 4. Which financial aid programs are most effective in improving access and success for students? agency responsible for overseeing and maintaining the MLDS and mandated to effectively organize, manage, Postsecondary Completion disaggregate, and analyze individual student data and to 5. How likely are students placed in developmental examine student progress and outcomes over time, courses to persist in postsecondary education and including preparation for postsecondary education and transfer and/or graduate? the workforce. The MLDS Center has a clearly defined set 6. Are community college students able to transfer of responsibilities, which include (MLDS Center 2015) the within the state to four-year institutions following: successfully and without loss of credit 7. What are the differences in performance, x Serving as a central repository of student and retention, and graduation, including time to workforce data degree, of students across various postsecondary x Ensuring compliance with the federal Family programs? Educational Rights and Privacy Act (FERPA) and 8. What are the characteristics of two-year other relevant privacy laws and policies institutions that are allowing students to persist x Designing, implementing, and maintaining strict most effectively and either graduate or transfer? system security procedures 9. Which four-year institutions are graduating x Conducting research pursuant to the Governing students most effectively and timely? Board’s research agenda x Maintaining a public facing website and data Workforce Outcomes portals and 10. What happens to students who start at community colleges and do not go on to four-year institutions? x Fulfilling public information requests. 11. What are the educational and labor market outcomes for individuals who use federal and state To fulfill these responsibilities, the center works in resources to obtain training at community colleges partnership with the Maryland Higher Education or other postsecondary institutions? Commission (MHEC), the Maryland State Department of 12. What economic value do noncredit community Education, the Maryland Department of Labor, Licensing, college credentials have in the workplace? and Regulation (DLLR), and the University of Maryland 13. Are college graduates successful in the workforce? School of Social Work and College of Education 14. What are the workforce outcomes for students (appendix G: Maryland Longitudinal Data System). who earn a high school diploma but do not transition to postsecondary education? In 2010 former governor O’Malley passed Senate Bill 15. What are the workforce outcomes of Maryland 275, Chapter 190, which established the Maryland high school noncompleters? Longitudinal Data System. The bill authorized the MLDS Governing Board to oversee the project and mandated Source: Maryland Longitudinal Data System Center, interview with that the board comprise high-level officials from across authors, 2014. the participating agencies. This leadership support and collaboration was an essential element to the success of 17 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 the project. As part of the design phase, the board standards for a key set of education data elements to identified 15 key policy questions that each participating streamline the exchange, comparison, and agency needed answered, and from there they worked understanding of data within and across P-20W backwards to build a system capable of answering those institutions and sectors. questions (box 2). The board also has oversight of the MLDS Center research agenda, which is guided by the 15 The Data Quality Campaign also supports this effort by policy questions. The research agenda also has basic tracking the progress of states toward achieving 10 guidelines; for example, all research analyses and Essential Elements of statewide longitudinal data research reports intended to inform policy and systems and the 10 State Actions to Ensure Effective Data programming will utilize data from at least two of the Use. In 2014 Maryland scored 10 out of 10 Essential three partner agencies (MSDE, MHEC, DLLR). Elements of statewide longitudinal data systems and eight out of 10 State Actions to Ensure Effective Data The foundation of the MLDS is established, including Use. The two-point loss was a result of the lack of a fully consensus across stakeholders, governance structures, operational and utilized LDS. The Data Quality management, and infrastructure; however, the system is Campaign’s (DQC’s) State Actions assessment consists of not yet fully populated with data and operational. The 24 questions. Maryland answered “Yes” to all but three last Data Inventory shows that most data from questions (appendix H: DQC’s 10 State Actions to Ensure participating entities have been approved, although data Effective Data Use, Maryland’s 2014 Score): are in various stages of preparation and loading into the system (MLDS Center 2014). Fully loaded and utilized x Parents, teachers, and appropriate stakeholders data in the MLDS will mark a major milestone for have access to student-level longitudinal data. Maryland’s education system as a whole. x State policy ensures that teachers and parents have access to their students’ longitudinal data. Maryland’s process of establishing the foundation and x Teachers and principals are trained to use infrastructure for the MLDS can guide other institutions. longitudinal data to tailor instruction and inform In general longitudinal data systems do not need to be schoolwide policies and practices. extremely robust, they simply need the following: These questions point to the existence of a complete and 1. An identifier system fully operational and utilized LDS. As described above, 2. Common code sets that track information over Maryland has established a sound infrastructure for the time and system but is yet to reach this level of complete 3. Systems and processes to keep data secure. implementation. In addition to these technical elements, key ingredients The State of Florida provides an example of a fully are ongoing participation and support from high-level operational and utilized longitudinal data system (box 3). decision makers as well as thorough consensus building In spite of a turbulent start due to political restructuring, and communication efforts across stakeholder groups. Florida’s statewide longitudinal data system helped to institutionalize EMIS in the education system, while MLDS is part of a national movement for states to cultivating a data-driven culture. State educational implement longitudinal data systems. In 2005, as part of accountability policies date back nearly 50 years, the Educational Technical Assistance Act of 2002, the positioning Florida at the forefront of efforts to track and SLDS Grant Program started awarding grants and learn from education data. technical assistance to states to catalyze the successful design, development, implementation, and expansion of Although Florida was certainly a pioneer in longitudinal K-12 and P-20W (data from early learning to workforce) data systems, the centralized data warehouse longitudinal data systems. Additional national efforts architecture was set up in 2003, and more than 10 years supported this movement such as the Common later it is ready for upgrades and enhancements. Florida Education Data Standards project, a national pursued the same federal grant funds for upgrades that collaborative effort to develop voluntary, common data Maryland pursued (ARRA/RTTT, ESRA/SLDS). Several key 18 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 upgrades include (McQuiggan and Sapp 2014) the Box 3: Florida State’s Pioneering Longitudinal Data System following: The Florida Department of Education (FDOE) in the x Improving data quality through a feedback loop U.S. serves nearly 2.7 million students, 4,200 public that reports data quality issues back to the schools, 28 colleges, 192,000 teachers, 47,000 college agency where the quality issue originated. professors and administrators, and 321,000 full-time x A more efficient unique identifier (UID) system staff throughout the state. It is one of the country’s that uses a common, statewide UID as opposed pioneers in collecting and tracking student-level data to a local UID which has a cumbersome and with the oldest longitudinal data system in the inefficient process for tracking student country dating back to 1995. Florida’s Statewide movement. Longitudinal Data System tracks 2.7 million students x A better process for researcher access to data across multiple agencies via a centralized data with greater automation and use of data marts warehouse. Utilization examples include (DQC 2006): and cubes, as well as business intelligence; while Florida had good open data policies for x Accountability and reporting across the researchers, the initial process to provide education system researchers with access to data was highly x Reporting almost immediately after the 2- to 3- manual and time consuming. week submission period x Organization of a data governance board, x Districts provided files containing data on their which was surprisingly not established with the own students who are included in calculations initial SLDS; in retrospect this governing body for accountability purposes by the state would have benefited the state, especially x High use of data by teachers and administrators management of metadata. x FDOE staff work with legislative staff to ensure or strengthen understanding of the data used These upgrades also reveal lessons learned. An by legislators additional key lesson from Florida is the importance of x Florida Office of Program Policy Analysis and establishing funding mechanisms that will maintain the Government Accountability uses student level system after the initial federal grants expire. data to examine performance in various areas Commitments from state legislature, either in the form in the context of costs of education of matching grants or ongoing funding, ensure the long- term viability of the system (McQuiggan and Sapp 2014). Policy Area 3: Quality Data Advanced zzzz The quality of data captured by Maryland’s EMIS was assessed in four areas: (1) Methodological Soundness; (2) Accuracy and Reliability; (3) Integrity; and (4) Periodicity and Timeliness. In Maryland the quality of data is bolstered by a strong enabling environment. COMAR consistently prioritizes and mandates quality data. For example, COMAR 13A.08.02.07 states, “To ensure that student records maintained under this title are relevant and accurate, a Source: Adapted from Florida Department of Education 2012 local school system and educational institution regulated by the State Board of Education shall provide for review 19 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 and updating of student records.” It goes on to detail x Detailed directions on how to digitalize student what data need to be updated and in what timeframe. record cards and Finally, the law includes validation measures, stating x Appropriate coding (definition and instructions) that, “Each principal of an educational institution shall for electronic student records cards. certify annually in writing to the local superintendent the accuracy of student data maintained in the student An array of other manuals are regularly updated at the records at the educational institution.” state level and circulated to local agencies describing processes and requirements for reporting, such as the To ensure that all schools and counties are following the “Attendance Procedures and Web Data Collection same processes, COMAR mandates use of a guiding System User Manual” and the “High School Assessment manual for student data referred to as the Maryland Status and Completers Reporting and Procedures Student Records System Manual. For example, COMAR Manual.” Each manual is an example of bridging intent 13A.08.02.04 states that procedures for administration and implementation. of student records are contained in the manual: “Records shall be kept for individual students in accordance with Local authorities input student data into the state-level the regulations of the State Board of Education and the data warehouse by uploading flat text files through the Maryland Student Records System Manual.” Web Data Collection System. A data collection system that reaches individual schools is an essential part of an Additionally, COMAR 13A.08.02.06 references the advanced EMIS. The data collection technology must also manual with regard to retention and disposition of data: be combined with a comprehensive data collection “Guidelines and standards for the retention and process. In Maryland, the data collection process is disposition of student records maintained under this title driven by reporting schedules that identify deadlines by shall be those adopted in the Maryland Student Records which specific data must be submitted. For example, System Manual.” This is a good example of how the enrollment data are reported by September 30th of each intention of quality data is communicated through year. This allows counties to build coded data extraction policies and then supported through to implementation packages that don’t change, which increases efficiency. by the existence of a manual to guide the process for handling student data. Although processes exist to guide data reporting and collection, challenges still arise. Some counties explained The Maryland Student Records System Manual is an that it is difficult to respond to data requests from MSDE essential implementation tool that ensures concise and when requirements and file specifications change, or comprehensive maintenance of student data across a when requests are made outside of the agreed upon highly decentralized education system. The first manual schedule and with short notice. Additionally, there were was produced in 2011, and an updated version was reports that MSDE sometimes requests data that it has released in 2015. The manual provides directions on already received, pointing to a lack of sound procedures and actions that administrators must follow coordination and communication processes within as they track students moving through the school system MSDE. and provide necessary reports. The forms provided in the manual are designed to incorporate all requirements In addition to improving internal processes, MSDE could necessary for a student to earn a Maryland diploma. The consider strengthening how information and updates are manual contains (MSDE 2015) the following: communicated. The Ohio Department of Education provides a good example of a website that clearly x List of relevant policies communicates priority information through intuitive x Delegation of county and school responsibility user pathways and calls to action (box 4). x Complete list of all data elements and their descriptions x Detailed directions on how to update and manage student record cards 20 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Box 4: Ohio State’s Innovation in EMIS Communications system automatically validates data submitted by local The State of Ohio’s Department of Education website authorities. Counties are required to develop and presents a comprehensive section on data. The implement procedures to ensure that data are collected website has an intuitive user interface, with clear calls and records maintained accurately in accordance with to action and resources, marked in the box titled the guidelines set forth in the most recent Student “How Do I …?” Additionally, the column marked Records System Manual. “Latest News” shares the most recent EMIS Newsflashes. These brief newsflashes share pertinent Network and data security, including privacy, are information on data collection, management, and essential priorities for Maryland, evidenced by policies utilization. They act as a transparent and easy to and comprehensive implementation procedures. With access update for key deadlines, changes to data regard to the enabling environment, the core piece of requests, important forms, system upgrades, etc. federal legislation that establishes the foundation for student data privacy is FERPA. This federal law protects Under Topics, a subsection on EMIS takes users to an the privacy of student education records and applies to EMIS-specific page that presents a new set of “How all schools that receive funds under an applicable Do I …?” questions, this time specific to EMIS. The first program of the U.S. Department of Education, which question is “How do I get help with EMIS reporting?” essentially covers all public schools. FERPA legislates that Once again, the user is provided with an easy to use, parents and guardians have rights to children’s intuitive set of options. education data, until the student reaches 18 years of age (U.S. Department of Education 2015). In Maryland, COMAR supports compliance with FERPA and adds additional privacy measures around student data. From an implementation perspective, the Maryland Student Records System Manual provides guidance on how schools should manage student records in compliance with FERPA and COMAR. MSDE follows Standard Operating Procedures and compensating controls designed to protect privacy and security of data, throughout the entire data lifecycle. Additionally, the launch of the Maryland Longitudinal Data System, which contains personally identifiable information (PII), required specific security enhancements. A sample of these security and privacy measures includes (Maryland Longitudinal Data System Center 2014) the following: Source: Ohio Department of Education, http://education.ohio.gov/Topics/Data. x Separation of PII data. Located in Production Environment only in separate subnet, VPN, and behind dedicated firewall. Data validation measures exist at each level of the x Access to the Production Environment is education system, beginning with schools, then local restricted. Hands Off and Fully Automated education agencies and at the state level. Each county- processing by support and development staff. level EMIS has a set of automated validation procedures x Use of Industry Standards such as COBIT 5 to that check data when entered at the school level. incorporate quality assurance and audit Principals are required to be familiar with the student standards into the management process. No PII records system at his or her school and to provide data captured in any of the logs. leadership to their staff on its implementation, while also x Centralized management of metadata with periodically checking the system's operation (MSDE Oracle Metadata Repository, 2015). At the county level, the web data collection x Encryption in-flight and at-rest. 21 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 x Privacy Compliant Release of information. Only Figure 15: MSDE Data Governance Plan compliant data may be released to ensure that the data meet the requirements of FERPA, Privacy Act, and other relevant privacy laws and policies. x De-identified data used for Analysis and Research. Aggregation used when releasing data. x Yearly security training. x Federal and state background checks for all employees. Network and data security is increasingly important given the growing threat of cyber-attacks. In January 2014, the Maryland’s Howard County Public School System experienced a series of power outages that may Source: Provided to authors by MSDE 2014. have been related to a cyber-attack (Toth 2014). Later that year, Prince George’s County Public School System from research institutions, the general public, media, reported a security breach involving employees’ policy makers, and others. personal data, including social security numbers (Wiggins 2014). Across the nation, similar attacks are reported on The data governance plan shows that a request for data a regular basis. Schools and counties have access to a will first go through the Data Review Committee. If wealth of student and employee data, from attendance approved, another layer of approval takes place with the records to medical issues, and experts emphasize the Data Governance Committee, consisting of the Chief need for education leaders to be prepared for cyber- Academic Officer, Assistant Superintendent of DCAA, threats (Lestch 2015). other Assistant Superintendents and the Branch Chief of DCAA. If the request is approved again, it moves on to a EMISs at both state and local levels are regularly Data Steward/Analyst, who will prepare the file with updated, and quality checks and audits are conducted FERPA compliance. Finally, in addition to responding to internally and externally. MSDE conducts gap analyses requests, the Data Governance Plan also illustrates key and regularly compares data structures to national data aspects of data flow, from collection, to reporting and models to identify areas for improvement. In accordance finally utilization. with the Data Quality Campaign’s 10 Essential Elements of Statewide Longitudinal Data Systems, Maryland established Element 10, a state data audit system for assessing data quality, validity, and reliability. MSDE follows strict data governance procedures. The MSDE Data Governance Plan (figure 15; see appendix I for larger version) outlines the people and processes responsible for establishing and enforcing policies, as well as guidance with regard to collection, reporting, and use of data. This type of plan is important considering that many different types of institutions make data requests from MSDE on a regular basis. Such data requests come from within the Division of Curriculum, Assessment and Accountability itself or from other divisions within MSDE, in addition to external requests 22 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Box 5: The Oregon Data Project Shows the Power of Data Policy Area 4: Utilization for Decision to Improve Student Learning Outcomes Making The Oregon Direct Access to Achievement (DATA) Established zzz| Project was a three-year $4.7 million initiative The utilization of Maryland’s EMIS was assessed by designed to increase data utilization in the classroom. examining four areas: (1) openness, (2) operational use, The fundamental idea was that data will not drive (3) accessibility, and (4) effectiveness in disseminating student achievement without effective training findings. strategies for educators. In addition to having the right policies, practices, and systems in place for data utilization, effective education systems also provide At the school level, teachers and principals use EMIS in training on why data are important and how to use a variety of ways, from management to classroom data. instruction. An effective EMIS empowers teachers to better use data, which in turn has the potential to In 2007 the Oregon DATA Project set out to train directly improve student learning outcomes (box 5). In teachers on the value and use of data to enhance Maryland teachers use data to track progress toward classroom practice and improve student learning by Student Learning Objectives (SLOs) and to better giving teachers the resources to collect, analyze, and understand student needs in general. Teachers use SLOs use longitudinal data (Data Quality Campaign 2012). to design and evaluate their instruction. SLOs are The project led to a behavior change in the way measurable instructional goals established for a specific teachers valued and used data, which in turn had group of students over a set period. Additionally, they positive outcomes for student achievement. serve as one of the measures of student growth for the State Teacher Evaluation model and may represent 20– A project evaluation report (Dunn 2011) confirmed 35 percent of a teacher’s evaluation. EMIS is a critical that at the onset of the project, schools that partner for teachers throughout the SLO process, which participated in the Oregon DATA Project had lower begins with planning the SLOs and moves on to capturing achievement numbers than nonparticipating schools. a baseline of student knowledge, then design and After just two years, the percentage of students at or delivery of curriculum, followed by another assessment, above proficient on the state test grew significantly and finally analysis and utilization of student learning more than students in those schools whose teachers outcomes data. EMIS is used throughout this process. did not receive training. Additionally, the Through this process, and with the help of an effective achievement gap between the two groups of schools EMIS, teachers can answer questions such as the decreased in reading and closed in math. following: Professional collaboration may contribute to a data- x What do students know and not know before driven culture. Teachers participating in the Oregon and after the curriculum is delivered? DATA Project were eager to collaborate with one x Are their gaps in the curriculum that prevent another in interpretation and utilization of data in the students from learning? classroom. This outcome is supported by a growing x Do certain students have special needs and body of evidence that suggests that collaborative require further intervention? work with data may accelerate interest in and use of x Are such interventions working? data among educators (U.S. Department of Education 2011; Wayman and Stringfield 2006; Chen, Heritage, x Is a student’s poor performance due to absence and Lee 2005). or inability to understand the content? Source: Oregon Data Project 2012 Cecil County uses a sophisticated EMIS that combines employs an Academic Index that pulls data from demographic student data with instructional data to academic factors that may impact success in school such track student learning by various disaggregated as Absence, Discipline, and Assessment Scores. In demographics (e.g., gender or ethnicity). The system also 23 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 addition to tracking general class performance, this information in a more dynamic way to address student report is a predictive tool that identifies students at risk needs. Teachers—and principals as well—have access to of dropping out of school. The Academic Index is information that significantly shifts the way we do calculated based on cutoff points, with four or more business, making it more efficient, reliable, and fast.” points displayed in red (indicating need for intervention), two or three points displayed in yellow, and zero or one EMIS can also help cultivate a culture of data utilization points displayed in green (figure 16). among teachers (figure 17). In Maryland’s Kent County, the school system uses SchoolNet, a product under the Teachers gain greater insight into the classroom, and Pearson-owned PowerSchool student information supervisors benefit from aggregate data. Reflecting on system. SchoolNet combines student information with the Academic Index, Regina Roberts, a principal from instructional data and allows teachers to create, store, Cecil County, commented, “This is a system that builds and share their own mini-assessments (also referred to off of what teachers already know individually and as formative or short-cycle assessments). While creates incredible value by aggregating that knowledge SchoolNet was launched, the school was also rolling out in an efficient, easy to use format. We can interact with the professional learning community (PLC) model: a workplace strategy to foster collaborative learning Figure 16: Cecil County Academic Index among colleagues. Assessments The SchoolNet EMIS combined with the collaborative PLC Grades (numbers of D’s or F’s) 0 0 Grades 1 1 framework was catalytic for teachers, who immediately Grades 2 2 started collaborating around assessment data. Karen Grades 3+ 3 Couch, the superintendent of Kent County, described the Discipline (suspensions per school year) benefit of the combined tools: “We have become more Suspensions 0 0 sophisticated in understanding the value of a benchmark Suspensions 1 1 test and the resulting data.” Dr. Couch continued, Suspensions 2–3 2 explaining the collective benefits, “Independently, it Suspensions 4+ 3 changed how teachers view assessments and how they Attendance (excused and unexcused) use data in planning their SLOs. Collaboratively, it Absences 91–100% 0 transformed the dialogue and collaboration between Absences 85–90% 1 teachers to be more data-driven and more energetic. Absences 80–84% 2 They are required to meet twice a month, but many now Absences 0–79% 3 meet once a week to collaborate and discuss data. Walking together as a collaborative culture ensures the Academic Index: Marking Period 1 effective use of data is a true partnership.” School name Figure 17: Collaboration Drives Teacher Utilization of EMIS High Risk 12 students (3.5%) Current grade: 6 Medium Risk 35 students (10.3%) Total enrollment: 340 Low Risk 286 students (84.1%) *Not Available 7 students (2.1%) Source: Education Week 2014, teachers in Kent County collaboratively Source: Provided to authors by Cecil County, 2014. examine progression data. 24 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Across Maryland, principals use EMIS to manage staff evaluation process. This user-driven design process and plan more effectively (figure 18). They rely on data helped to create a product that truly responds to the to understand and make decisions about their schools. needs of principals. Specific user requests included Principals look at data to answer many questions, such spellcheck, automatic saving every 30 seconds, and color as the following: codes to differentiate automatic and manual correlations. x Are the teachers in my school effective, and are students learning? Asset management, or the Asset Hub, allows teachers to x Am I on track to meet district and state student- share, vet, and collaborate around instructional assets. teacher ratio goals? Teachers also have the opportunity to rate assets. Assets x Is learning at my school equitable; is it fair across with higher ratings float to the top of the system, making demographic groups such as gender and them more visible to users, while weaker assets drop to ethnicity? the bottom of the list. Asset management makes x Is professional development effective, and what teaching more efficient and collaborative. is the return on investment in professional development? The Charles County TEAMS product includes an x What is my school budget, and is it managed automated, intuitive evaluation system that improves efficiently? the rate and quality of teacher evaluations (figure 19). Before establishment of the system, a single teacher Charles County worked with vendor Insystech to evaluation took a principal between four and five hours. customize The Evaluation & Assets Management That process included scheduling the class observation, System (TEAMS). While TEAMS works best on the cloud conducting the observation, writing up notes, analyzing in an environment with the internet, it can also be notes, scheduling the follow-up, and finally meeting with customized for regions without the internet or with the teacher to discuss results. Maryland state law limited internet access. Because Charles County has requires two evaluations per teacher per year. The consistent internet access, the full capabilities of the inefficient process led to principals putting off TEAMS product are available and include Staff evaluations until they would stack up at the end of each Evaluation, Admin Management, Student Assessment semester, forcing principals to rush through the build-up and Student Information Systems, Asset Management, of evaluations to meet deadlines. The purpose of the and Search. evaluation, as an instrument to monitor and strengthen the skills of teachers, was entirely lost. The incorporation Principals were brought in to help design the evaluation of EMIS through the TEAMS product transformed the system with TEAMS, and they now use it throughout the evaluation process. Figure 18: Principals Gain Greater Insight Using EMIS TEAMS helps principals schedule the observation, send calendar invites, collect and manage notes, and share documents. When conducting an observation, principals use TEAMS to schedule the initial observation with the teacher. During the observation they take notes directly in the TEAMS system, on a tablet or laptop. Following the observation, the principal will review notes and then click “Correlate,” which automatically selects phrases from the notes that are relevant to the state evaluation methodology (box 6) and correlates the notes with the relevant domain, saving principals the extra time of rewriting notes into the evaluation framework. After the automated transfer, principals have the opportunity to Source: Taken by authors, principal at Glendale Elementary School, Anne edit domains in case anything was missed in the Arundel County, 2014. automated correlation process. During training, 25 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Figure 19: Innovation in Data Collection for Teacher Evaluation Kim Hill, the superintendent of Charles County, emphasized the importance of the dialogue between principal and teacher: “Throughout development and design of this product, principals were at the table. They were demanding, as they should have been, because they were taking away obstacles and barriers. Their insights focused on what is important, which is the conversation.” Amy Hollstein, Assistant Superintendent of Instruction, also supported this point stating, “The best part of the whole process is the dialogue between the principal and the teacher. We call it the courageous conversation.” With the TEAMS system in place and integrating with student assessment data, shifts in the work culture are taking place. A focus on true professional development— genuinely improving teaching and management expertise—has taken hold. All parties are happy with the system, which is quick, easy to use, and accurate. “It sounds so simple, but it is so incredibly profound,” stated Dr. Smith, as he reflected on Charles County’s success. “To help your staff develop as professionals from the day they walk in the door is a huge step in the profession, a fundamental shift that raises the level of professionalism for teachers and principals.” Box 6: Teacher Evaluation Model: An EMIS Opportunity The state evaluation model examines a 50/50 split between qualitative professional practice measures Source: Provided to authors by Insystech, 2015. (inputs) and quantitative student growth measures principals were taught about the terminology to (outcomes). Based on the Charlotte Danielson maximize correlations, and the correlation taxonomy in Framework, four practice domains are evaluated that the system can also be added to and revised. Next, the comprise qualitative professional practice: (1) correlated domains are transferred into the framework planning and preparation, (2) instructional delivery, for scoring. The principal scores on a scale of 1 to 4, with (3) classroom management and environment, and (4) an option to add notes next to each score. Scores and professional responsibilities. Performance in each notes are aggregated over time to show changes in domain is worth 12.5 percentage points. The TEAMS professional development. teacher evaluation product includes both qualitative and quantitative measures, effectively comparing Perhaps the most critical aspect of the tool is that it teacher inputs with student outcomes. Integrating cultivates conversation and transparency (figure 20). this model with EMIS provides an opportunity to After each observation, the teacher has access to the effectively track teacher inputs and student learning scores and write-ups before they meet with the principal outcomes over time. to discuss results, reducing surprises and adding to a dialogue that is truly focused on teacher professional Source: MSDE Teacher Professional Practice, http://www.marylandpublicschools.org/msde/programs/tpe/tpp.html. development. 26 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Figure 20: EMIS Drives Collaboration between County and The success of these efforts to share information hinges School Administrators on the effectiveness of an EMIS. An effective EMIS can reliably share information in a timely manner to improve school accountability and increase parent engagement. In the long run, such practices have the potential to improve student learning outcomes and system efficiency. Policy makers use data to monitor education quality and equity and for planning. A key tool that policy makers in Maryland use to hold schools accountable is the School Progress Index (SPI), which reports EMIS data and analysis through a transparent platform. The SPI came about as a result of federal reforms to NCLB, which Source: Taken by authors, principal of C. Paul Barnhart Elementary and initially tracked accountability through Adequate Yearly Assistant Superintendent of Instruction, Charles County, 2014. Progress (AYP). In 2011 the U.S. Department of Education gave states the opportunity to develop a new system for Across Maryland, parents and students are actively measuring and reporting school performance. Maryland using EMIS to access assignments, grades, even redesigned its accountability system, focusing on the homework that helps parents better support their progress schools are making towards improving student children’s learning. Figure 21 provides an example from achievement, closing achievement gaps, and enabling Performance Matters of a parent-student interface. The students to move towards readiness for college and platform includes grades and attendance, grade history, career by mastering grade-level and course-level teacher comments, and a variety of other information curriculum goals each year (MSDE 2012b). that creates transparency and fosters communication and collaboration between parents, students, and The change is further evidence of Maryland’s transition teachers. With advancements in technology, especially from a compliance-focused system to one that targets mobile technology, students and parents interact student growth and learning. The SPI provides more in- directly with EMISs more frequently. EMIS vendors are depth, student-level information than AYP, which simply increasingly focused on building mobile apps for tracked how a school’s scores would change from year to education stakeholders and EMIS users to interact with year. EMIS on the go. Figure 21: Performance Matters Student and Parent Dashboard Source: Performance Matters, sample dashboard 2014. 27 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 The SPI evaluates schools on a continuous scale based on Ddd7: Where's My School Pilot Program Box the variables of Achievement, Growth, Gap Reduction, and College- and Career-Readiness and makes results of Australia’s My School platform (myschool.edu.au), each school available publically via the annual Maryland which launched in January 2010, is a critical Report Card. The index identifies schools for component within a set of national education reforms intervention, support, and recognition depending on that transformed data and reporting processes to their progress. increase accountability and improve education outcomes. EMIS plays a fundamental role throughout The SPI is based on high multiple measures that include the reforms and creation of My School. student achievement data in English/Language Arts, Mathematics, and Science; growth data in Before these reforms took place, the Australian English/Language Arts and Mathematics; and gaps, government struggled with unreliable data. based on the gap score between highest-achieving and Communities, especially parents, did not have access lowest-achieving subgroup in mathematics, reading, to data and therefore had little ability to understand science, cohort graduation, and cohort dropout rates. how their schools fit into the larger picture and to exert any influence over instruction. Further, there The index guides interventions from policy makers by was no nationally comparable, single source of data. categorizing schools into one of five strands that determine the district and state support schools receive. Under the management of ACARA, My School has The state affords top-performing schools greater grown to include information about each school’s flexibility while lower-performing schools receive student population, the average achievement of progressively more prescriptive technical assistance, students in the National Assessment Program expectations, and monitoring. (NAPLAN), indication of student progress over time, and school financial data, comparable across all During the transition from AYP to the SPI, MSDE was Australian schools, as well as other useful data, such careful to provide ongoing communication about the as attendance rates and staff numbers at each school reform and what it mean for schools, staff, students and (ACARA 2014). Appendix K presents a sample school parents. Maryland Classroom (Vol. 18, No. 3) is a good profile and demonstrates the comprehensive data example of this communication effort, providing available in a simple user interface. frequently asked questions, as well as detailed information on methodology and utilization of the index. In addition to parent and community use of transparent data, policy makers use the data to inform Appendix J shows the SPI results for Beall Elementary in decision making. A number of key policy reforms have Maryland’s Allegany County. Various tools are built into been guided by My School data, including a review of the webpage to educate the user about how to Australian government funding and the Making Every understand and use the index. The School Progress Index School a Great School reform program. Policy lessons is reported through the Maryland Report Card, which learned are widely relevant, especially for countries also contains demographic data, enrollment and with a federal-state system: attendance rates, absentee rates, student mobility, teacher qualifications, and data about students receiving x Buy-in from leadership was essential special services. A similar utilization of EMIS was x Commitment was made to lengthy consensus tremendously successful in Australia (box 7). building across nine government departments x Additional funds from the Australian Accountability systems evolve over time, usually in Government supported the project accordance with changes to assessment frameworks, x Communication to the community, including curriculum standards, and other shifts in measurement explaining their rights to information, and policies. The transition from AYP to the SPI marked maintained momentum an initial accountability improvement effort, moving Source: OECD 2012. from a binary (yes or no) framework to one that is more 28 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 multidimensional. With the launch of the Common Core NAEP and state-specific test scores to build a nationally standards and the PARCC assessment, the SPI will need comparable school quality index. This report does not to continue its evolution. comment on the validity of the Location Inc. school quality index; however, its existence points to an MSDE is currently in the process of determining a new interesting trend in education data that underscores the accountability framework, which will be submitted for importance of a well-functioning EMIS, at every level of approval to the state Board of Education in the spring of the education system. 2016. One area under consideration is the addition of a new indicator to complement the existing set of As data are increasingly valued and used across different Achievement, Growth, Gap Reduction, and College- and sectors and industries, it is important that the Career-Readiness, by providing local context. institutions that collect and manage these data ensure the highest levels of professionalism and commitment to A growing trend in Maryland, and across the United producing quality, timely information. Zillow is another States, is the use of EMIS data by noneducation example of a real estate company that integrates with stakeholders. The private sector, including financial student data, using GreatSchools.org, an independent institutions and the real estate industry, provides nonprofit organization that shares school information examples of noneducation stakeholders using education (figure 23). Performance on standardized tests within data. Figure 22 shows a map of Maryland cities, with each state makes up the majority of the GreatSchools colors indicating the quality of public schools in each city. Rating; however, in a growing number of states where The website, Neighborhood Scout, touts its ability to data are available, additional information is included provide enterprise data for every neighborhood and city such as information on how much students are learning in the country. in a given year and how prepared they are for college (GreatSchools.org 2015). Education data on Neighborhood Scout comes from another company called Location Inc., which developed an algorithm that uses the student passing rate on the Figure 23: Real Estate Companies Use EMIS Data Figure 22: Neighborhood Scout, Enterprise-Grade Data Source: Zillow.com. Source: NeighborhoodScout.com. 29 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Minimal collaboration between local education which stem in part from the lack of an overarching EMIS agencies limits utilization, especially in counties with strategy to guide and coordinate implementation. limited resources. Counties do not appear to collaborate around lessons learned or implementation. Although Strengths MSDE makes some attempts to cultivate collaboration, more could be done to incentivize partnerships, EMIS in Maryland is powered by a combination of vision collaboration, and peer-to-peer learning. at the highest levels of state and county leadership, as well as full support and buy-in from decision makers and Utilization of data in the classroom for instructional stakeholders across the education system. Strong gains is expanding; however, often the programs being policies position EMIS for continued growth and success. used are not linked to the local-level EMIS, missing an Teachers are increasingly using data in the classroom, in opportunity to collect a deeper layer of student learning part a result of a strong data-driven culture. The MLDS is data. Learning programs such as DIBELS, Fountas & built on a strong foundation and integration strategy. Pinnell, Scholastic Reading & Math Inventory, and many others are commonly used in the classroom, but data are Opportunities not fed into the county EMIS. This is more of a technical integration gap that vendors should work to improve. MLDS is well poised to drive statewide data-driven That said, MSDE could monitor for, identify, and suggest decision making to the next level. There is high potential to counties any instructional programs that have good for expansion of instructional program data to add more integration capabilities. relevance and meaning to EMIS. Similarly, quality financial and human resources data are available to be fully integrated in EMIS, carrying the potential to boost Conclusion and Policy Recommendations systemwide efficiency. Full implementation and utilization of the Student Course Grade Teacher This section presents a set of recommendations based on collection will return new insights on what influences and the assessment of EMIS in Maryland (table 10). The drives student learning outcomes. Strengths, Weaknesses, Opportunities, Threats (SWOT) profile (table 11) summarizes key points from the Weaknesses assessment and informs recommendations. County-level EMISs vary significantly, and unequal Table 10: Maryland EMIS Rankingsa resource distribution, including counties that fall Advanced between state and local funding opportunities, is 1. Enabling Environment zzzz evident. The highly decentralized and autonomous Established education system reduces potential for efficiency gains 2. System Soundness zzz| (e.g., resource sharing, learning from successes and Advanced challenges of others) and blocks opportunities for 3. Quality Data zzzz collaboration between counties. Established 4. Utilization for Decision Making zzz| Threats a. Assessment results are further detailed in appendix L: Summary of Policy Lever Benchmarking, and appendix M: Extended Rubric. MSDE does not have a statewide EMIS strategy to guide progress and incrementally support counties as they Overall, EMIS in Maryland is very strong and provides an each develop their systems. Changes in leadership could array of good practices across key policy areas from reduce momentum around data management and which others can learn. EMIS in Maryland derives utilization. Costly systems, often supported by grants, strength from its advanced enabling environment and can be challenging to maintain. quality data. System soundness and utilization for decision making are established, but limitations exist, 30 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Table 11: Maryland EMIS SWOT Profile Strengths Weaknesses x Advanced enabling environment and quality data x Slow progress integrating human resources and x Vision from decision makers, buy-in from implementers financial data with EMIS x Utilization in the classroom for teaching and learning x Unequal resource distribution x Strong data-driven culture x Lack of collaboration between counties x MLDS has strong integration strategy x Highly decentralized system blocks efficiency Opportunities Threats x Continued growth of longitudinal data system x Lack of a statewide EMIS strategy x National and state-level grant programs x Changes in leadership x Expansion of assessment and instructional program x No systematic strategy for statewide data x Availability of strong financial and human resource data utilization training x Completion and utilization of SCGT collection x Costly systems Recommendations for Maryland In many aspects, Maryland is at the forefront of It is clear that EMIS implementation varies significantly innovations in EMIS; however, the state would benefit across counties. MSDE should prioritize support to from an overarching EMIS strategy that guides and counties with fewer resources and help them to achieve encourages counties to reach the next level in EMIS realistic milestones. implementation. With a decentralized education system and highly autonomous counties, it is important for A final consideration for the proposed statewide EMIS MSDE to continue to lead counties forward, as opposed strategy is inclusion of an analytics agenda to continue to being led by them. A statewide EMIS strategy that Maryland’s momentum around a learning-driven conveys state goals and objectives, as well as customized approach to data management and utilization. The MLDS milestones for each county, would help MSDE in is certainly evidence of progress in this area. MSDE delivering on its leadership role. With a strategy in place, should continue driving these efforts by communicating it becomes easier to identify opportunities to offer long-term goals and helping counties to identify and strategic support, cultivate learning between counties, reach their own objectives around analytics. and collectively hold one another accountable. Table 12: Summary of Federal and State Grants That Could be Aligned under a State EMIS Strategy A statewide EMIS strategy would also bring cohesion and Project Grant type Amount alignment to the numerous state and federally funded Maryland Federal grant $5.6 million grant programs that are relevant to EMIS (table 12). In Longitudinal Data awarded to MD (2006) some cases, such as the MDK12 initiative funded by the System (2006, 2009, $5.9 million Office of Educational Research and Improvement, MSDE 2012) (2009) maintained the program, although its connection to and $3.9 million alignment with MSDE are not entirely clear. Improved (2012) Race to the Top Federal grant $250 million coordination and alignment of these initiatives under an (2010) awarded to MD overarching strategy could make them more effective, State Fiscal Federal grant $2 million accessible, and sustainable. Stabilization Fund awarded to MD (2010) An important component of a statewide EMIS strategy EMIS-specific grants State grant $700,000 should target equitable resource distribution and for new system from MSDE to (2014) support mechanisms across counties. (2014, 2015) Howard County $65,000 (2015) Source: Authors. 31 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Maryland has made steady progress in evolving from an EMIS is not being fully leveraged to increase efficiency EMIS that is dominated by a compliance focus to one across the education system. At both state and county that is driven by learning and innovation; however, levels, a variety of siloed databases manage and track additional improvements can still be made. To human resources and financial data. Efforts to summarize, a compliance-focused EMIS is characterized incorporate these data into EMIS would improve the by an extreme focus on reporting to comply with state ability to track inputs and measure value for money and federal policies. An EMIS that is driven by learning across the system. Bringing these data together is an will be fully compliant but will also use data in more essential part of tracking efficiency and effectiveness innovative ways to cultivate student learning and across the education system. Nationally, Maryland is provide insights into the education system that allow usually among the highest per pupil expenditures (figure decision makers at all levels to understand what drives 24). Although this can certainly be a positive indicator in student learning. Maryland could accelerate this terms of strong investment in education, it can also evolution by doing the following: reveal inefficiencies in spending. Without fully integrated data, decision makers cannot defend efficiency across x Providing more tools and resources that assist the system. educators in using data during classroom instruction; the Oregon Data Project is a good Figure 24: Current Per-Pupil Expenditures for U.S. Public example of a statewide effort to provide training Elementary and Secondary Education, 2008–09 and support for data-driven decision making In Dollars National Average: $10,591 x Hosting meetings, conferences, or workshops on Median: $10,189 the topic of data utilization for learning and x Creating positive incentives for teachers and schools that are leaders in using data for learning. A variety of digital instructional programs (e.g., DIBELS, Dreambox, or Conceptua Math) are being used in classrooms, but data from these programs are not consistently reaching EMIS. Although this speaks to a larger technology challenge, some of which depends on private sector solutions, finding ways to continue adding Source: U.S. Department of Education 2009. learning outcomes data to the school-level EMIS is key. MSDE can play a key role by incentivizing collection of Several cutting-edge EMIS initiatives are underway in formative assessment data and identifying and Maryland but have not yet reached completion, suggesting instructional programs that have greater statewide institutionalization, and utilization. For integration capabilities with EMIS. For example, schools example, the SCGT collection is an exciting program that in Anne Arundel County integrate Fountas & Pinnell data links teachers and courses with student learning into their county-level EMIS, Performance Matters. outcomes, but it does not appear to be complete. When Effectively collecting summative assessment data is a finished, this innovation will provide principals and strong first step, the next step is finding ways to collect, administrators with tremendous insight into the efficacy manage, and track formative assessments and real-time of their teaching staff, as well as additional information student learning data. This results in a much more related to student learning outcomes. Similarly, efforts comprehensive and holistic view of what influences to operationalize and fully utilize the Maryland student learning. It also evolves the value of instructional Longitudinal Data System will also mark an exciting programs from benefiting a single teacher or classroom milestone for the state. The Florida LDS provides a to advancing the entire system. concrete example of how a state was able to institutionalize a LDS and drive utilization across different levels of the education system. Efforts to 32 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 complete these initiatives and provide training on how to strategy also provides counties with collective bargaining use this data in management are highly recommended. power. Aggregated content made available through the strategy could include items such as the following: There appear to be some gaps in communication and coordination around data sharing at MSDE. Although State level MSDE tries to establish clear processes for when and how to share what data, some counties report that at x EMIS strategy—vision and milestones times MSDE makes changes to data requirements, x Repository of EMIS policies, standards, manuals, requests data outside of designated times with short and strategic documents notice, and requests already collected data, indicating x Schedule of data-reporting deadlines gaps in coordination and communication. Better x Upcoming grant opportunities relevant to EMIS communication within MSDE as well as transparent x Upcoming EMIS collaboration activities (e.g., platforms that communicate requirements and webinars, convenings) deadlines (such as the Ohio Department of Education x EMIS hub with state and county resources (e.g., website) could ease some these obstacles. training materials, presentations, system documents such as wireframes) A missed opportunity exists around collaboration between counties. Maryland’s counties are each doing County level innovative and exciting work with regard to EMIS implementation, but they are not sharing ideas and x Summary EMIS profile—a simple overview of lessons learned across county borders. This type of each county’s system, and key EMIS contact(s) sharing would be especially useful for counties with x Technology plans and EMIS strategic documents smaller budgets and fewer resources to commit to EMIS. x Reviews of vendors, technology systems, etc. One area that continually came up during interviews was the need for more formative assessments and exemplary The EMIS strategy is a tool that supports MSDE in lessons. Counties with larger budgets, or those that encouraging diverse and innovative counties that work received grants, had these learning tools professionally together to achieve statewide education goals. Further, developed. Counties with limited resources would the EMIS strategy helps MSDE in capturing and sharing benefit from a mechanism that opens these tools and the rich clusters of knowledge that each county has makes them accessible across the state. Programs that developed through their own experiences implementing integrate with EMIS, such as Insystech’s TEAMS product, EMIS. can build “asset hubs” that allow instructional materials to be aggregated, organized, shared, and even rated by From the perspective of counties, the EMIS strategy users. should provide a centralized menu of options where stakeholders can find tools and resources relevant to their needs. The aim is that regardless of the unique Building a Statewide EMIS Strategy needs of each county—whether they are upgrading an Many of the recommendations refer to the potential old system or starting an entirely new system, whether benefits of a statewide EMIS strategy in Maryland. This they are launching a data utilization training program or section shares ideas about how such a strategy could be simply searching for tips on how to use data in classroom developed and delivered. MSDE carries the important instruction—they can find relevant information through responsibility of steering statewide education results the statewide EMIS strategy and online platform. The toward collective success. The EMIS strategy can support reality is that this knowledge already exists across MSDE in this mission by leveraging policies, standards, Maryland, but it is not harnessed in an effective manner. and incentives to cultivate collaboration, cost-sharing The result is inefficiency, with counties often reinventing opportunities, and learning networks. Communicated the wheel, or falling into the same gaps that their effectively, the EMIS strategy would serve as a one-stop colleagues and neighbors could have helped them to shop for all stakeholders, including a hub for tools, avoid. resources, reporting schedules, and milestones. The 33 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 A successful EMIS strategy should strive to meet the county websites, social media, e-mail, and other needs of both state and county stakeholders. This will communication channels is worth exploring. Perhaps the encourage stakeholders to engage with the strategy steering committee could circulate a questionnaire that often because it provides direct value to their work. asks stakeholders how they prefer to receive MSDE can cultivate such value through incentives (e.g., information. grants or award and recognition programs) and cost- sharing or collaborative activities. An initial effort should Part of Maryland’s strength rests in its decentralized be made to reach out to EMIS stakeholders in each structure, a characteristic that is embraced and county to identify where they are with EMIS defended at the county level. That said, it is important implementation and what needs they may have. Based that decentralization and autonomy do not create siloes on that feedback MSDE could arrange webinars, cross- and walls that ultimately come at the expense of learning county coaching and learning networks, repositories of and efficiency. The purpose of the EMIS strategy is to relevant documents, etc. prevent this from happening by supporting MSDE in cultivating collaboration, shared goals, and alignment, In addition to providing direct value to each county, the while also celebrating Maryland’s innovative and diverse EMIS strategy also outlines Maryland’s overarching counties. vision and objectives for EMIS, bringing cohesion and alignment to statewide EMIS activities. Innovative interactive tools could even show how each county is Lessons Learned and Global Relevance contributing to the statewide EMIS goals, similar to the way that the DQC tracks state progress toward ensuring Maryland’s EMIS falls between an established and an effective data utilization. advanced system, offering a variety of good practices that are relevant to education stakeholders within the state and around the world that are working to Establishing Leadership and a implement their own systems. This section summarizes Communication Mechanism some of the key good practices taken from Maryland’s EMIS experience. The success of the EMIS strategy depends in part on leadership and communication. With regard to A sustainable and well-functioning EMIS requires the leadership, it is important that state and county decision knowledge that EMIS is more than an IT system. At both makers support the strategy, and that a committed state and county levels, a common trend that was group or steering committee is identified to carry the observed during interviews was that each EMIS project forward. The steering committee should have comprises policies, people, and processes, in addition to stakeholders from across the education system, the underlying IT architecture. A common problem with consisting of both state and county officials and including EMIS implementation is the belief that it is primarily teachers and principals. technology. For EMIS to be successful, a holistic approach should be pursued that takes into account the Communication plays a key part in developing buy-in. many factors, technology and beyond, that contribute to How can stakeholders support the strategy if they do not long-term success and sustainability. know about it, know what it intends to accomplish, and, most importantly, understand how it will benefit their The journey from a compliance-focused EMIS to a day-to-day work? A simple communications campaign learning-focused EMIS is complicated, but it is also a with steady ongoing updates (e.g., the EMIS Newsflash) game changer. At the core of this transition is a behavior helps make all stakeholders feel part of the process. As change in the way that data are valued. The shift moves such, developing an online platform for the EMIS from seeing data as a tool to monitor progress, to strategy is important. Carving out a space on an existing understanding that data, as well as the larger EMIS, can platform such as marylandpublicschools.org or be used to promote teaching and learning as well as mdk12.org would be useful. Most importantly, it should management and planning. This report documents be an online space that stakeholders are aware of and important factors that helped EMIS in Maryland frequently visit. Finding ways to share updates through 34 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 transition from a focus on compliance to a focus on Commitment to ongoing professional development and learning. capacity building yields significant returns. There is never an end to professional development, and this is Vision for EMIS from the highest levels of leadership is especially true of EMIS given the ever-changing world of essential. Both at MSDE and across Maryland’s counties, technology advancements. Ensuring adequate budget significant progress is made with EMIS when the highest for professional development, as well as strategies to levels of leadership support and promote the vision for scale professional development, ultimately strengthens EMIS. This was the case with Governor O’Malley in his EMIS as a whole. Harford County effectively scaled EMIS support of the Maryland Longitudinal Data System. It was professional development through the train-the-trainer also the case with Renee Foose, superintendent of model. Because EMIS reaches across the education Howard County, who drove the implementation of an system, it is important to think about capacity building at integrated county EMIS. Effective leaders catalyze all levels (from teachers to policy makers), not just the change and adoption, bring different stakeholder groups core team responsible for maintaining EMIS. together, define the roadmap, ensure effective funding and staffing, and lead by example. Data utilization at the school level is crucial. As the frontline of data collection, schools are essential to the Support from the highest levels of leadership is overall success of an EMIS. As a result, efforts to ensure essential, but buy-in must continue down the that teachers and principals are not only simple inputters organizational structure and throughout the education of data, but also thoughtful users of those data will system. Each of the leaders mentioned above were strengthen the overarching EMIS. One way to do this is supported by directors and managers that carried their to establish collaborative networks around data vision through to implementation. Without this buy-in at utilization. Kent County’s Professional Learning multiple points across the education system (with county Community model is an example of using collaboration administrators, principals, teachers, etc.), advancements to drive utilization. in EMIS are challenging, if not impossible. The federal government plays an important role in When designing a new system, start with the policy catalyzing an ecosystem for data utilization, which questions that key stakeholders want the system to reaches state and local levels and streams across answer. By answering this question early in the process, private and social sectors. Across the United States data the risk of ending up with a fragmented and limited utilization and open data are gaining momentum, with system is reduced. Sometimes this requires multiple platforms such as data.gov, cities.data.gov, efforts to build consensus across a variety of counties.data.gov, and states.data.gov. Additionally, departments and stakeholder groups. government offices such as the U.S. Department of Education, Office of Planning, Evaluation and Policy To achieve an effective EMIS, policy makers should Development routinely publish informative content on a focus on translating what is written in policies into a variety of topics including education technology, such as culture of data utilization. This report contains a variety “Teachers' Ability to Use Data to Inform Instruction: of good practices with regard to institutionalizing EMIS Challenges and Supports” (2011) and “Use of Education by bridging intent and implementation. It is important Data at the Local Level: From Accountability to that policy makers not only develop legislation and Instructional Improvement” (2010). The private and strategic plans, but also think through to social sectors are also essential parts of this ecosystem. implementation. This includes supporting and The Data Quality Campaign, a non-profit organization, is advocating for tools, resources, incentives, a strong example of the social sector playing a key role in institutionalization practices, and communication promoting effective utilization of education data. efforts, ensuring that high-level intent meets on-the- ground implementation. 35 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Acknowledgments SABER Systems Approach for Better Education Results SASID State Assigned Student Identifier This report was prepared by SABER-EMIS team members SCGT Student Course Grade Teacher Husein Abdul-Hamid (Task Team Leader), Sarah Mintz SLDS State Longitudinal Data System and Namrata Saraogi. Special thanks to colleagues who SLO Student Learning Objective provided support and feedback to the report: Dingyong SPI School Progress Index Hou and Noah Yarrow (peer reviewers), Manar El- TEAMS The Evaluation & Assets Management System TIMSS Trends in International Mathematics and Science Iriqsousi, Cassia Miranda, and Michael Melamed. The Study report benefited immensely from the guidance and TSDL Teacher Student Data Link support of Dr. Jack Smith, Chief Academic Officer, Office UID Unique Identifier of Teaching and Learning, Maryland State Department of USIS Unique Student Identifier System Education, as well as the entire team at MSDE. Finally, WDCS Web Data Collection System the SABER EMIS team is especially grateful for the insights provided by principals, teachers and administrators in Anne Arundel County, Cecil County, Charles County, Harford County, Howard County, Kent County, and Montgomery County. Acronyms ARRA American Recovery and Reinvestment Act AYP Adequate Yearly Progress COMAR Code of Maryland Regulations COPPA Children’s Online Privacy Protection Act DCAA Division of Curriculum, Assessment, and Accountability DLLR Department of Labor, Licensing, and Regulation DQC Data Quality Campaign EDW Education Data Warehouse EMIS Education Management Information System ESEA Elementary and Secondary Education Act ESRA Education Sciences Reform Act FDOE Florida Department of Education FERPA Family Educational Rights and Privacy Act HSA High School Assessment LDS Longitudinal Data System LEA Local Education Agency MHEC Maryland Higher Education Commission MSA Maryland School Assessment MLDS Maryland Longitudinal Data System MSDE Maryland State Department of Education MSPAP Maryland School Performance Assessment Program NAEP National Assessment of Educational Progress NCLB No Child Left Behind P-20W Pre-school to the workforce PARCC Partnership for Assessment of Readiness for College and Careers PII Personally Identifiable Information PISA Program for International Student Assessment PLC Professional Learning Community RTTT Race to the Top 36 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 References B7-E2FD-42E5-9F09- E4DDCC02F4EE/29219/StudentRecordsManualUpdated Abdul-Hamid, H. 2014. “SABER EMIS Framework Paper.” 6202011.pdf. World Bank, Washington, DC. ———. 2012a. “Frequently Asked Questions: School Progress ACARA (Australian Curriculum, Assessment and Reporting Index.” Authority). 2014. “About My School.” http://www.marylandpublicschools.org/w/SPI_Q_and_ http://www.acara.edu.au/verve/_resources/About_My A.pdf _School_2014.pdf. ———. 2012b. “Maryland Classroom.” Vol. 18, no. 3. Anne Arundel County Public Schools. 2012. “The Journey to http://www.marylandpublicschools.org/mdclassroom/V Greatness.” Strategic Plan for Anne Arundel County ol18_No3_122012.pdf. Public Schools. ———. 2013–14. “The Fact Book.” Armstrong, J., and K. Anthes. 2001. “How Data Can Help: http://www.marylandpublicschools.org/msde/divisions/ Putting Information to Work to Raise Student bus_svcs/docs/Fact_Book_2013-2014.pdf. Achievement.” American School Board Journal. ———. 2014. “Maryland Report Card.” MDReportCard.org. November 2001. ———. 2014a. “Professional Learning Standards.” Baltimore Sun. 2015a. “Maryland Slips from First to Third in http://mdk12.msde.maryland.gov/instruction/professio National Education Ranking.” January 8, 2015. nal_development/standards/index.html. http://www.baltimoresun.com/news/maryland/educati on/bs-md-maryland-number-three-20150108-story.html ———. 2014b. “Support and Maintenance for Mdk12 and Report Card Websites.” Consulting and Technical ———. 2015b. “Maryland First in Nation in Advanced Services Task Order Request for Proposals. Placement.” March 11, 2015. http://doit.maryland.gov/contracts/Documents/catsPlu http://www.baltimoresun.com/news/maryland/educati s_torfp_status/R00B4400075_MDK12.pdf. on/blog/bs-md--maryland-first-in-ap-20150311- story.html ———. 2015. “Maryland Student Records System Manual.” http://marylandpublicschools.org/MSDE/newsroom/pu Chen, E., M. Heritage, and J. Lee. 2005. “Identifying and blications/docs/MDStudentRecordsSystemManual2016. Monitoring Students’ Learning Needs with Technology.” pdf. Journal of Education for Students Placed at Risk 10 (3): 309–32. ———. 2003. “Divisions: Curriculum, Assessment & Accountability.” Dunn, K. 2011. “Oregon DATA Project Final Evaluation Report.” http://www.marylandpublicschools.org/MsDE/divisions Next Level Evaluation, Fayetteville, AR. /instruction/index.html. Education Week. 2014. “Under Common Core, Teachers Band Maryland Longitudinal Data System Center. 2014. “Building a Together to Build Math ‘Coherence’.” System to Protect Identities.” Presentation to MIS ———. 2015. “Quality Counts.” Conference by Chandra Haislet. http://www.edweek.org/ew/qc/. ———. 2015. “Data Inventory.” GreatSchools.org. 2015. “GreatSchools Ratings: Methodology https://mldscenter.maryland.gov/egov/Publications/ML Report.” DS_DataInventory_DEC_2014_V41_FINAL.pdf. Hitch, R., and J. Fullerton. 2013. “SDP Review: Strategic Use of McQuiggan, J., and A. W. Sapp. 2014. Implement, Improve and Data Howard County Public School System.” Harvard Expand Your Statewide Longitudinal Data System: Strategic Data Project, Harvard University, Cambridge, Creating a Culture of Data in Education. New York: Wiley. MA. Michaels, H., and S. Ferrara. 1999. “Evolution of Educational Howard County. 2015. “Integrated Systems for HCPSS.” Reform in Maryland: Using Data to Drive State and Policy Presentation to World Bank by Grace Chesney and Justin Local Reform.” In Handbook of Educational Policy, ed. G. Benedict, April. J. Cizek. San Diego, CA: Academic Press. Lestch, C. 2015. “Cybersecurity in K-12 Education: Schools Face National Center for Education Statistics (NCES). 2015. “About Increased Risk of Cyber Attacks.” Fedscoop. Us.” https://nces.ed.gov/about. Accessed December 30, http://fedscoop.com/cybersecurity-in-k-12-education- 2015. schools-around-the-country-face-risk-of-cyber-attacks. ———. 2015. “The Condition of Education 2015.” U.S. State MSDE (Maryland State Department of Education). 2011. Department of Education, Washington, DC. “Maryland Student Records Manual.” http://nces.ed.gov/programs/coe/ataglance.asp. http://marylandpublicschools.org/NR/rdonlyres/3B33B5 37 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 ———. 2013. “U.S. States in a Global Context: Results from Improvement.” American Journal of Education 112 (4): the 2011 NAEP-TIMSS Linking Study.” Institute of 549–71. Education Sciences, U.S. Department of Education, Wiggins, O. 2014. “Security Breach Reveals Personal Data on Washington, DC. Prince George’s School Employees.” Washington Post. OECD (Organisation for Economic Co-operation and https://www.washingtonpost.com/local/education/sec Development). 2012. “Delivering School Transparency in urity-breach-reveals-personal-data-on-prince-georges- Australia: National Reporting through My School.” school-employees/2014/11/21/fdd3de9c-71e2-11e4- http://www.oecd- ad12-3734c461eab6_story.html. ilibrary.org/docserver/download/9812071e.pdf. ———. 2014. “Education at a Glance 2014: OECD Indicators.” http://dx.doi.org/10.1787/eag-2014-en. Peterson, P., L. Woessmann, e. Hanushek, and C. Lastra- Anadón. 2011. “Globally Challenged: Are U.S. Students Ready to Compete?” PEPG Report no. 11-03, Harvard’s Program on Education Policy and Governance, Education Next, Taubman Center for State and Local Government, Harvard Kennedy School, Cambridge, MA. Toth, S. 2014. “Howard Schools Recover from Possible Cyber Attack.” Baltimore Sun. http://www.baltimoresun.com/news/maryland/howard /ellicott-city/ph-ho-cf-hcpss-internet-0123-20140121- story.html. U.S. Census Bureau. 2014. “Public Education Finances: 2012.” G12-CG-ASPEF. US Government Printing Office, Washington, DC. U.S. Department of Education, National Center for Education Statistics. 2015. “The Condition of Education 2015” (NCES 2015–144), Charter School Enrollment. ———. 2015. “FERPA” http://www2.ed.gov/policy/gen/guid/fpco/ferpa/index. html. ———. 2011. “Teachers’ Ability to Use Data to Inform Instruction: Challenges and Supports.” Office of Planning, Evaluation and Policy Development. DOE, Washington, DC. ———. 2009. "National Public Education Financial Survey (NPEFS)." Fiscal year 2009, Version 1a. National Center for Education Statistics, Common Core of Data (CCD), DOE, Washington, DC. U.S. News and World Report. 2015. “How States Compare in the 2015 Best High Schools Rankings.” http://www.usnews.com/education/best-high- schools/articles/how-states-compare Washington Post. 2014. “Md. Student Test Scores Drop Significantly as State Shifts to Common Core.” July 11, 2014. https://www.washingtonpost.com/local/education/md- student-test-scores-drop-significantly-as-state-shifts-to- common-core/2014/07/11/7c0f960a-08f9-11e4-a0dd- f2b22a257353_story.html Wayman, J. C., and S. Stringfield. 2006. “Technology- Supported Involvement of Entire Faculties in Examination of Student Data for Instructional 38 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix A: Maryland State Department of Education Organizational Chart 39 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix B: Division of Curriculum, Accountability and Assessment Organizational Chart 40 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix C: Sample of Different Systems Used across Counties Additional programs Additional programs County Data warehouse Student information integrated not integrated PowerSchool Performance Matters TIENET (special DIBELS, Fountas & Baseball Card, needs), Achievement Pinnell Schools Series, TOADS Anne Arundel Administrative Student Information System (SASI) Cecil County Pearson Inform iTracker PowerSchool DIBELS, AppliTrack Data warehouse iSeries connects to TEAMS DIBELS, Rigby PM using Amazon Web Edline and Books, AppliTrack, Charles County Services Gradebook, now Substitute Employee Blackboard products Management System (SEMS) Performance Matters Performance Matters Performance Matters Assessment and Data Baseball Card Harford County Management System Built with edVantage Built with Synergy LMS built with Howard County Canvas by Instructure SchoolNet PowerSchool Scholastic Reading and Math Inventory, Kent County DIBELS, Dreambox, Conceptua Math Built data warehouse Online Online Achievement Substitute Employee with MicroStrategies Administrative and Reporting Management System Student System (OARS), (SEMS), Montgomery Information System Human Resource County (OASIS) Information System (HRIS), Financial Management System (FMS) 41 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix D: Maryland State Department of Education Enterprise Picture Data Sources and Processes for the P12 EDW & MLDS-ODS, P20 MLDS, and MHEC-ODS 42 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix E: Howard County Strategic Plan for New EMIS 43 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix F: Maryland Longitudinal Data System Center, Data Process Flow 44 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix G: Maryland Longitudinal Data System 45 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix H: DQC’s 10 State Actions to Ensure Effective Data Use, Maryland’s 2014 Score 46 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix I: Maryland State Department of Education Data Governance 47 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix J: Maryland School Progress Index, Beall Elementary 48 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix K: My School Australia, Profile for Gordon East Public School 49 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix L: Summary of Policy Lever Benchmarking Policy Goal Policy Lever Score Weight Benchmark Legal framework 3.66 15% Advanced Organizational structure and 4.00 15 Advanced institutionalized processes Enabling Environment Human resources 3.83 15 Advanced Infrastructural capacity 3.83 15 Advanced Budget 4.00 15 Advanced Data-driven culture 3.43 10 Advanced Data architecture 3.69 20 Advanced Data coverage 2.45 30 Established System Soundness Data analytics 2.67 15 Established Dynamic system 2.84 15 Established Serviceability 2.73 20 Established Methodological soundness 3.78 25 Advanced Accuracy and reliability 3.55 25 Advanced Quality Data Integrity 3.66 25 Advanced Periodicity and timeliness 4.00 25 Advanced Openness 2.62 15 Established Utilization in Decision Operational use 3.03 50 Advanced Making Accessibility 3.52 20 Advanced Effectiveness in disseminating findings 2.20 15 Established a. 0–0.9 = Latent; 1–1.9 = Emerging; 2–2.9 = Established; 3–4 = Advanced. 50 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Appendix M: Extended Rubric, Maryland Scores Highlighted Red Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced The system contains crucial components of a comprehensive The system enabling The system lacks The system contains contains basic The system contains environment, which major components of most components of components of a crucial components of POLICY AREA 1: ENABLING ENVIRONMENT addresses related a comprehensive a comprehensive comprehensive a comprehensive policy elements and enabling enabling enabling enabling environment enables the environment environment environment functioning of an effective and dynamic system Institutionalization of system: EMIS is institutionalized as an integral part of the education system and the government Responsibility: responsibility for collecting, processing, and disseminating education statistics is given to a clearly designated institution Basic components or agency An existing legal of a legal Most elements of a An existing legal Dynamic framework: the legal framework supports A legal framework is framework or 1.1 Legal framework legal framework are framework supports a framework is dynamic and elastic so a fully functioning not in place informal in place fully functioning EMIS that it can adapt to advancements in EMIS mechanisms are in technology place Data supply: the legal framework mandates that schools participate in EMIS by providing education data Comprehensive, quality data: the requirement for comprehensive, quality data are clearly specified in the EMIS legal framework 51 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Data sharing and coordination: the legal framework allows for adequate data sharing and coordination between the Ministry of Education and agencies and/or institutions that require education data Utilization: the legal framework emphasizes data-driven education policy Budget: the education system budget includes a line item for EMIS Confidentiality: the legal framework guarantees that respondents' data are confidential and used for the sole purpose of statistics The system is The institutional institutionalized The institutional The system is structure of the within the The system is not structure of the institutionalized system is defined government, has specified in policies, system is not within the Organizational within the well-defined and what exists does clearly specified in government, has well- structure and Organizational structure and government, and it 1.2 organizational not have well-defined policies, it has some defined organizational institutionalized institutionalized processes has defined processes, and has organizational organizational processes, and has processes organizational several processes; EMIS has processes, and its several functionalities processes, but its functionalities limited functionalities functionalities are beyond statistical functionalities are beyond statistical limited reporting limited reporting Qualified staff Minimum standards Some staff are The majority of staff All staff are qualified operate the system, of qualification are qualified to operate are qualified to to operate the system, Personnel: the core tasks of EMIS are and opportunities not met for the the system, and operate the system, and well-established 1.3 Human resources identified, and EMIS is staffed with are available to majority of staff that limited and frequent opportunities are qualified people improve their operate the system, opportunities are opportunities are constantly available to performance and and opportunities are available to available to improve improve staff retention not available to improve staff 52 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced improve their performance and staff performance performance and performance and retention and retention retention Professional development: professional retention training is available for EMIS staff Data collection: tools for data collection are available Database(s): databases exist under the The system has a The system has a well- umbrella of the data warehouse and well-defined The system has an defined infrastructure have both hardware and software infrastructure to infrastructure that to fully perform its means perform data The system lacks a The system has a Infrastructural allows it to perform data collection, 1.4 Data management system: a system is collection, well-defined basic or incomplete capacity some of its functions management, and in place that manages data collection, management, and infrastructure infrastructure in an integral dissemination processing, and reporting dissemination manner functions in an functions in an Data dissemination: data dissemination integral manner integral manner tools are available and maintained by the agency producing education statistics Personnel and professional development: the EMIS budget contains a specific budget for EMIS personnel and their professional development Maintenance: the EMIS budget The system budget contains a specific budget for system contains the majority The system budget is The system budget is maintenance and recurrent costs of required comprehensive, The system suffers The system has a comprehensive, categories to ensure 1.5 Budget ensuring that the from serious basic or incomplete ensuring that the Reporting: the EMIS budget contains a that most parts of system is sustainable budgetary issues budget system is sustainable specific budget for reporting costs the system are and efficient and efficient sustainable and Physical infrastructure: the EMIS efficient budget contains a specific budget for physical infrastructure costs Efficient use of resources: processes and procedures are in place to ensure that resources are used efficiently 53 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced A data-driven culture A data-driven culture exists that prioritizes The system has a prioritizes data as a The system suffers data management and data-driven culture A data-driven culture fundamental because there is not a utilization within and that demonstrates exists that prioritizes element of data-driven culture beyond the education Data-driven a basic appreciation data management Data-driven culture operations and that prioritizes data system, and evidence culture of data and interest and utilization within decision making, management and of that culture is in developing and beyond the both inside and data utilization in present in daily better data education system outside of the decision making interaction and utilization practices education system decision making at all levels The system has The system has The processes and The processes and basic processes some processes and structure of the structure of the The system lacks and a structure a structure, but they EMIS are sound and system are sound and POLICY AREA 2: SYSTEM SOUNDNESS processes and that do not support do not fully support support the support the structure the components of the components of components of an components of an an integrated an integrated integrated system integrated system system system The system's data The system's data structure has most The data The system's data architecture elements of the data The data architecture architecture is well structure does not includes some architecture; is well defined to 2.1 Data architecture Data architecture defined to ensure have a well-defined components; however, it has some ensure full system full system data architecture however, it is deficiencies that functionality functionality incomplete affect the system's functionality Administrative data: EMIS contains administrative data The data in the system are The data in the Financial data: EMIS contains financial comprehensive and The data in the The data in the The data in the system system are far from data cover administrative, system include system include most are comprehensive 2.2 Data coverage being comprehensive, financial, human some of the data but not all of the and cover all data Human resources data: EMIS contains and coverage is resources, and areas data areas areas human resources data limited learning outcomes Learning outcomes data: EMIS contains data learning outcomes data 54 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Basic tools and processes are Tools and processes available, but the Tools and processes Tools and processes are available to Tools and processes system is not are available; are available to perform data are available to capable of 2.3 Data analytics Data analytics however, data perform data analytics analytics at different perform limited conducting analytics are not at different levels on a levels on a regular tabulations advanced analytical performed regularly regular basis basis steps (e.g., predictive models, projections) Quality assurance measures: the system is dynamic and maintains The system in place The system in place is quality assurance measures The system in place is not easily The system in place is not easily adaptable is elastic and easily adaptable and The system in place elastic and easily Data requirements and considerations: to changes and adaptable to allow requires significant is easily adaptable, adaptable to allow for 2.4 Dynamic system mechanisms are in place for addressing advancements in data for changes and time and resources but it remains changes and new and emerging data requirements needs, because no advancements in to accommodate reasonably complex advancements in data System adaptability: EMIS is elastic and quality assurance data needs changes and/or needs easily adaptable to allow for changes standards are used advancements and/or advancements in data needs Validity across data sources: information brought together from different data and/or statistical Services provided by frameworks in EMIS is placed within Services provided by the system are valid the data warehouse using structural the system are valid across data sources, and consistency measures across data sources, integrate integrate Integration of noneducation databases noneducation noneducation into EMIS: data from sources collected databases into the Serious issues exist Inconsistencies The data are databases into EMIS, by agencies outside of the EMIS are EMIS, and archive related to data exist related to consistent and valid; and archive data at 2.5 Serviceability integrated into the EMIS data data at the service of validity and data validity and however, some the service of EMIS warehouse EMIS clients by consistency consistency concerns still exist clients by ensuring the Archiving data: multiple years of data ensuring the relevance, are archived, including source data, relevance, consistency, metadata, and statistical results consistency, usefulness, and usefulness, and Services to EMIS clients: services timeliness of its timeliness of its provided by the system to EMIS clients statistics statistics include ensuring the relevance, consistency, usefulness, and timeliness of its statistics 55 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced The system has most mechanisms in place The system has the needed to collect, The system has the mechanisms The system has save, and produce required mechanisms required to collect, basic mechanisms timely, high-quality in place to collect, The system lacks save, produce, and to collect, save, information for use save, produce, and mechanisms to utilize information, and produce in decision making; utilize information, collect, save, or POLICY AREA 3: QUALITY DATA which ensures timely, quality however, some which ensures produce timely, high- accuracy, security, information; additional measures accuracy, security, quality information and timely, high- however, its are needed to and timely, high- for decision making quality information accuracy might be ensure accuracy, quality information for use in decision questionable security, and/ or for use in decision making timely information making that can be used for decision making Concepts and definitions: data fields, records, concepts, indicators, and metadata are defined and documented in official operations manuals along with other national datasets and The methodological endorsed by the government The methodological The methodological The methodological basis for producing The methodological basis for producing basis for producing Classification: there are defined basis for producing educational basis for producing educational statistics educational statistics education system classifications based educational statistics statistics follows educational statistics from raw data follows most Methodological on technical guidelines and manuals does not follow the basics of from raw data follows 3.1 follows required soundness Scope: the scope of education statistics internationally internationally internationally internationally internationally is broader than and not limited to a accepted standards, accepted accepted standards, accepted standards, accepted standards, small number of indicators (e.g., guidelines, or good standards, guidelines, and good guidelines, and good guidelines, and good measurements of enrollment, class practices guidelines, and practices practices practices size, and completion) good practices Basis for recording: data recording systems follow internationally accepted standards, guidelines, and good practices Source data: available source data Source data and Source data and Source data and Source data and Accuracy and Source data and 3.2 provide an adequate basis for statistical techniques statistical statistical techniques statistical techniques reliability statistical techniques compiling statistics are sound and techniques have follow most required are sound and 56 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Validation of source data: source data reliable, and lack soundness and basic soundness elements to be reliable, and statistical are consistent with the definition, statistical outputs reliability and reliability, but sound and reliable, outputs sufficiently scope, and classification as well as time sufficiently portray statistical outputs but statistical portray reality of recording, reference periods, and reality do not portray outputs do not valuation of education statistics reality portray reality Statistical techniques: statistical techniques are used to calculate accurate rates and derived indicators Professionalism: EMIS staff exercise their profession with technical Education statistics Education statistics independence and without outside contained within Education statistics contained within the interference that could result in the the system are contained within the Education statistics system are mostly violation of the public trust in EMIS Education statistics guided by limited system are guided by contained within guided by principles statistics and EMIS itself contained within the principles of all three principles of 3.3 Integrity system are not of integrity (two of system are guided by integrity (one of the integrity: Transparency: statistical policies and guided by principles the three principles principles of integrity three principles of professionalism, practices are transparent of integrity of professionalism, professionalism, transparency, and Ethical standards: policies and practices transparency, and transparency, and ethical standards in education statistics are guided by ethical standards) ethical standards) ethical standards Periodicity: the production of reports and other outputs from the data The system produces The system The system produces The system produces The system produces warehouse occur in accordance with data and statistics produces some most data and Periodicity and data and statistics all data and statistics 3.4 cycles in the education system neither periodically data and statistics statistics periodically timeliness periodically in a periodically and in a Timeliness: final statistics and financial nor in a timely periodically and in a and in a timely timely manner timely manner statistics are both disseminated in a manner timely manner manner timely manner The system is wholly There are no signs The system is used The system is used The system is wholly utilized by different that the EMIS is by some education by most education utilized by different users for decision utilized in decision stakeholders, but stakeholders but is users for decision POLICY AREA 4: UTILIZATION FOR DECISION MAKING making at different making by the not for major not fully operational making at different levels of the majority of education policy decision in governmental levels of the education system stakeholders making decision making education system EMIS stakeholders: EMIS primary The system is open The system lacks The system is open The system is open The system is open to stakeholders are identified and use the to education openness to to some education to the majority of all education 4.1 Openness system in accordance with the legal stakeholders in education stakeholders in education stakeholders in terms framework terms of their stakeholders in terms terms of their stakeholders in of their awareness 57 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced User awareness: current and potential awareness and of their awareness awareness and terms of their and capacity to utilize EMIS users are aware of EMIS and its capacity to utilize and capacity to utilize capacity to utilize awareness and the system outputs the system the system the system capacity to utilize the system User capacity: EMIS users have the skills to interpret, manipulate, and utilize the data produced by the system to ultimately disseminate findings Utilization in evaluation: data produced by EMIS are used to assess the education system Utilization in governance: data produced by EMIS are used for governance purposes Data produced by Data produced by Data produced by the Data produced by Data produced by the Utilization by schools: data produced the system are used the system are used system are not used the system are system are used in by EMIS are used by schools in practice by the 4.2 Operational use in practice by the in practice by used in practice by practice by the main majority of Utilization by clients: data produced by main education education some education education education EMIS are used by clients (including stakeholders stakeholders stakeholders stakeholders stakeholders parents, communities, and other actors) Utilization by government: the system is able to produce summative indicators (derived variables) to monitor education system Understandable data: data are presented in a manner that is easily digestible Education statistics Education statistics Widely disseminated data: education are presented in an are presented in an statistics are disseminated beyond the understandable understandable Ministry of Education and/or the manner, are widely The system suffers The system has The system has manner and are education statistics-producing agency 4.3 Accessibility disseminated using from serious major accessibility minor accessibility widely disseminated to other EMIS stakeholders clear platforms for accessibility issues issues issues using a clear platform Platforms for utilization: platforms are utilization, for utilization, standardized across EMIS and are complemented by complemented by customizable to user needs user support user support User support: assistance is provided to EMIS users upon request to help them access the data 58 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 Scoring Description of Policy levers Indicators best practices Latent Emerging Established Advanced Dissemination strategy: national A dissemination plan governments have an information has been dissemination strategy in place Dissemination of implemented; Dissemination is The dissemination of Effectiveness in education statistics Dissemination is however, room reasonably education statistics via 4.4 disseminating via an EMIS is neither strategic nor exists for Dissemination effectiveness: strategic, but an EMIS is strategic findings strategic and effective improvement (for dissemination of EMIS statistics is ineffective and effective effective full effectiveness in effective relation to strategic engagement) 59 MARYLAND, U.S. ǀ EMIS SABER COUNTRY REPORT |2015 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative collects data on the policies and institutions of education systems around the world and benchmarks them against practices associated with student learning. SABER aims to give all parties with a stake in educational results—from students, administrators, teachers, and parents to policy makers and business people—an accessible, detailed, objective snapshot of how well the policies of their country's education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of Education Management Information Systems. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. 60