Education Benchmarks

The Fourth Annual Report
Written by Dr. Robert D. Gaudet, Senior Research Analyst
Sponsored by The University of Massachusetts Donahue Institute - January 2002

Overview: School District Effectiveness on the 2001 MCAS

The first analysis of school district effectiveness came out in February of 1999 and evaluated the 1998 MCAS in terms of district demography.Subsequent reports considered the 1999 and 2000 MCAS results.The central tool of these analyses is the Effectiveness Index methodology that examines the relationship between selected community demographic characteristics and educational outcomes.These characteristics include:average education level,average income,poverty rate, single-parent status,language spoken,and percentage of school-age population enrolled in private schools.These variables were chosen because they correlate with achievement and because the education literature identifies them as connected to academic performance.(See Appendix F for information on the Effectiveness Index.)

Researchers ranging from James Coleman in the 1960s to James Comer in the 1990s have demonstrated that community demographics play a major role in how well children do in school.Today the media and researchers often reference the role that demography plays in student achievement.The Effectiveness Index model provides a means of isolating the role played by community characteristics on student performance on statewide educational assessments.With a community ’s achievement context factored into its test results,it is possible to know how much value a school system adds to demographic expectations.In the absence of a methodology to control for the demographic diversity of Massachusetts,listing MCAS scores primarily demonstrates the relative advantage or disadvantage that community characteristics bring to student performance.Any raw ranking order of MCAS scores reflects district demography much more than it represents anything else.A sorting of MCAS results would tell us more about local real estate values or the percentage of luxury car ownership in a community than it would about school quality.

This fourth edition analyzes data from the first MCAS administration that counts as a graduation requirement for the class of 2003 -the 2001 tests.For the first time, scores improved dramatically across the Commonwealth.Students in schools in very different communities – cities and suburbs – scored significantly higher than those peers in previous classes.

There are several details of note about this report:

• Much of the analysis focuses on Grade 10 results.This is the first year the tests count as a graduation requirement and analyzing the progress of the Class of 2003 can help identify issues and perhaps help shape future policy.

• Small districts (those with fewer than 45 students taking MCAS exams in one grade)are not included in this report.This is not a reflection of a bias against small systems.Rather,small sample size can add significant error to any statistical analysis.

• Only Grades 4,8,and 10 ELA and Math scores are evaluated in the study.

• This fourth edition covers 98%of the state's public school population in Grades 4 and 8 and 88%of Grade 10 students.(Regional vocational-technical schools, which are not included in this report,educate 10%of the Grade 10 students.) ¹

This report identifies school districts that add value to the learning readiness of their students as indicated by higher-than-demographically-predicted test scores. Identifying such systems is a first step to determining if they are indeed providing more effective educational services to their students.Identifying best practices in higher-scoring systems that are demographically similar to lower-scoring systems is the first step in helping other systems implement productive policies and practices to help all children learn.

While this report lists districts that over-performed expected scores,it also evaluates progress in different kinds of communities after four administrations of MCAS.We know that students all over the state made significant gains on the 2000 MCAS compared to the 2000 assessment.We also know that some Massachusetts school districts,despite having made solid score gains,still have many students who have not yet demonstrated mastery of the critical basic skills assessed by MCAS.Performance Progress by Different Kinds In addition to assessing the 2001 MCAS scores of individual systems in terms of demography,this report evaluates the performance improvement of 2000-2001 in terms of district demography.

Summary Observations

Year 4 of the MCAS has produced the first solid gains from one year to the next. While scores were generally up,the greatest gains were in Grade 10 . While this report,like its predecessors,identifies those systems that over-perform based on demography,the dramatic improvement in scores provides an opportunity to expand the research to begin to make some observations about the overall efficacy of the 1993 education reform legislation.

Primary Observations: Solid Improvement but Demography Still Matters

After four years of data analysis,one point emerges clearly:Districts that over- perform their demography tend to be middle-class or demographically advantaged communities.As was the case in previous years,upper-demography communities are about two times more likely to over-perform than are communities of lower demography. ²

The relative lack of capacity of lower demography communities to exceed their demographic characteristics on standards-based assessments remains a persistent point of concern.Even with much higher average scores on the 2001 MCAS compared to previous years,many demographically disadvantaged communities still have 40%to 50 %of their students not passing MCAS.It is likely that schools in these districts will need to provide more robust interventions to help all students achieve the basic skills measured by the ELA and Math MCAS.

Based on the results of the 2001 MCAS,most of the students in middle and upper- demography districts perform well enough now to pass (achieve one point above Fail)the MCAS graduation requirement.Demographically disadvantaged districts have much more work to do to lift more of their students into success in school and on MCAS.

The greatly improved results of the 2001 MCAS confirm a basic fact of life for educational achievement:Without substantial changes in urban and other disadvantaged schools,a student's educational success will continue to be a function of zip code.The goal of education reform should be to change the finding of the Willis-Harrington Commission of 1965 that found that,concerning educational quality,"It matters vitally to every individual where the accident of birth and home locates him."Thirty-five years after the most comprehensive education study in state history spoke of the impact of demography and geography on achievement,we still face the challenge of neutralizing the impact of demographics on educational outcomes. ³

2001 MCAS Performance by Demography

After eight years of the current education reform effort,at least two points are clear:

• Most Massachusetts schools districts are in good shape concerning teaching their students the basic skills measured by MCAS.

• On the Grade 10 MCAS,121 districts had fewer than 10%of their students failing ELA,another 50 districts had fail rates between 11%and 15%which means that 171 districts out of 223 in the data set are in very good shape in terms of passing MCAS ELA.

• Concerning Math,69 districts had fewer than 10%of their students failing,with another 50 districts with fail rates of 11%to 15%.Another 49 districts had Fail rates between 16%and 20%which means that 168 districts out of the 223 in the data set used for this report are in good shape in terms of passing MCAS Math.

• Demographically challenged districts are having a hard time implementing the dramatic school reforms needed to help their teachers help students master the skills needed to be successful in life.Virtually all of the districts with higher percentages of students in Fail are demographically challenged.

• As was the case last year and the year before,there are many repeat performers in the Effective Districts list.Generally,about half of the sixty systems identified in each grade and subject in this report as adding value to the demographic characteristics of their students are repeaters from last year.This is not surprising;a system that had organized itself to enhance student achievement in 2000 is likely to have kept that up in 2001.(Grades 4,8,and 10 English Language Arts and Math are included.The Science and Technology MCAS is no longer given in Grade 4 and 10,and the Grade 8 results from the 2001 MCAS are not reported yet.)

Even with the dramatic score improvement,the over-performers include many repeaters.Communities that are organized to deliver effective educational services continue to over-perform demographically even when the scores go up.

The MCAS: The Massachusetts Comprehensive Assesment System

The chief objective of the state's education reform initiative is to enable public school students to achieve a certain level of knowledge and skill.The Massachusetts Department of Education has established the level by setting out what students are expected to learn in each basic subject.School districts are supposed to see to it that their students learn what they are expected to learn.

MCAS has three broad purposes:Performance Assessment;Diagnostics;and Accountability.

• The Performance aspect of the MCAS – the scores – has been the most publicized part of education reform.Newspapers and electronic media regularly report scores of districts.

• The Diagnostics built into MCAS give educators the capacity to identify specific learning weaknesses in individual students as well as target problems in curriculum and teaching that may impede achievement.School systems should be able to improve student learning through the use of these diagnostic tools.

• Finally,MCAS gives citizens an Accountability tool that measures how well schools are doing in moving their students towards high achievement.With MCAS data over time as a guide,we should be able to increase the accountability of individuals,schools,and systems.

MCAS has created some controversy.Citizens and students in some communities are not supportive of MCAS for various reasons.The Massachusetts Teachers Association has mounted several major media campaigns against MCAS as it now stands.Generally speaking,the opposition has come from the education industry and from people in very wealthy communities.On the other hand,many business leaders and several prominent newspapers support MCAS education reform. Public sentiment has been generally favorable.

The most comprehensive analysis of the MCAS is probably a recent study done by Achieve,Inc.,a national independent non-profit education reform organization. The report,Measuring Up:A Report on Educational Standards and Assessments for Massachusetts ,finds:

• Overall,Massachusetts'high standards and high school tests are of high quality and are aligned,providing a solid foundation on which to build state education policy.

• The grade 10 tests are rigorous yet reasonable – and are in fact the most challenging of the exit-level tests Achieve has reviewed.

• The mathematics standards generally are well organized,jargon-free,clear and precise. 4

Student academic performance,including how students do on MCAS tests,is affected by two broad sets of influences:school factors and non-school factors.The first entail what happens in school,and thus what is within the control of the school district itself.The second entails conditions outside the schools,such as the demographic profile of the students and the community.As we look at a given district's average score on an MCAS test,we have to be able to discern how much of the score is tied to school factors,and how much of the score is explained by non- school factors.School factors include sufficiency of resources (usually defined as how much money is spent on education);effectiveness of teachers;quality of curriculum and such.Unfortunately,it is very difficult to measure any of these things except per-pupil spending.

In the research reported in this paper,non-school factors consist largely of the overlapping demographic conditions of family life and community life.This study uses six such conditions in a given school district:its median level of educational attainment,its median income level,its percentage of households above the poverty line,its percentage of single-parent families,its percentage of non-English-speaking households,and its level of private school enrollment.Statistical analysis shows that these factors form much of the non-school influence on how the state's students do on such standardized tests as the MCAS. 5

Identifying Over-Performing Systems

Identifying systems that over-perform their demography is important in that such systems may have valuable lessons to offer similar systems in their efforts to boost student achievement.In fact,the state Department of Education and MassInsight Education,a well-respected non-profit education reform organization,are working to identify exemplary schools to determine which specific initiatives have contributed to the outstanding performance of a school or district.(See the MassInsight web site for more information on their Building Blocks initiative - www.massinsight.com.

What the Effectiveness Index Tells Us: Statewide Results

The Effectiveness Index (EI)provides a measure of the school district's contribution to its student performance.The Effectiveness Index supplies a piece of crucial insight as to which school districts are more effective.

For a given district,the Effectiveness Index gauges the impact that school factors have on the average MCAS score.The greater the positive impact of the school factors,the higher the district's Effectiveness Index will be.

The Index is calculated in the following manner:For a given district,the six demographic factors are used as the basis for projecting a likely average score on the MCAS.The demographically likely score is then compared to the average score that the students in the district actually received.The Effectiveness Index is the number that represents the difference between the likely score and the actual score.

If the number is negative – if the actual score is lower than the likely score – then this suggests that what is happening in the schools in the district is not enabling its students to perform beyond the demographic expectations for them.If the number is a positive number – if the actual score is higher than the likely score – then this suggests that what is happening in the schools is helping the district's students to surpass the demographic expectations for them.(For a fuller account of the development of the Effectiveness Index,please see Appendix C.) This research applies the Effectiveness Index methodology to the MCAS scores of school districts in the state.One of the consistent findings of this analysis is that demography explains most of the variation in test scores from district to district.

It is important to understand that demography is not necessarily destiny.Where a person is born or raised does not dictate how successful or unsuccessful that person will be in life.Similarly,demography does not determine educational achievement in all cases.There are examples of successful urban schools where disadvantaged students do very well by any measure.However,demography does create the context in which schools operate and in which learning occurs.Other factors being equal,it is less likely that students from a disadvantaged environment will be as successful in educational achievement as will be students from advantaged situations.Demography is about tendency, not destiny.