Education Benchmarks

The Third Annual Report
Written by Robert D. Gaudet, Senior Research Analyst
University of Massachusetts Donahue Institute - February 2001

This research is designed to identify Massachusetts school districts whose student test scores exceed the scores predicted by their demographic characteristics. The work is not intended to rank districts' performances but rather to highlight the efforts of districts whose students are exceeding what they would be expected to achieve on statewide standardized tests. The goal is to enable other districts to study and learn from the efforts of systems identified as effective in this work.

"EDUCATION, THEN, BEYOND ALL OTHER DEVICES OF HUMAN ORIGIN, IS THE GREAT EQUALIZER OF THE CONDITIONS OF MEN .…"

"GOOD EDUCATION RELIEVES PRESSURE ON OTHER SOCIAL SUPPORT SYSTEMS; A WEAK EDUCATION SYSTEM STRESSES ALL OTHER SYSTEMS AND SERVICES, INCLUDING INDUSTRY. IN SHORT, PUBLIC EDUCATION IS CENTRAL TO AMERICA'S QUALITY OF LIFE AND ECONOMIC FUTURE."

The first analysis of school district effectiveness came out in February of 1999 and evaluated the 1998 MCAS in terms of district demography. The second report was released in February 2000 and considered the 1999 MCAS. The central tool of these analyses is the Effectiveness Index methodology that examines the relationship between selected community demographic characteristics and educational outcomes. These characteristics include: average education level, average income, poverty rate, single-parent status, language spoken, and percentage of school-age population enrolled in private schools. These variables were chosen because they correlate with achievement and because the education literature identifies them as connected to academic performance. (See Appendix F for information on the Effectiveness Index.)

Researchers ranging from James Coleman in the 1960s to James Comer in the 1990s have demonstrated that community demographics play a major role in how well children do in school. The Effectiveness Index model provides a means of isolating the role played by community characteristics concerning student performance on statewide educational assessments. With a community's achievement context factored into its test results, it is possible to know how much value school systems add to demographic expectations. In the absence of a methodology to control for the demographic diversity of Massachusetts, listing MCAS scores primarily demonstrates the relative advantage or disadvantage that community characteristics bring to students. Any raw ranking order of MCAS scores reflects district demography much more than it represents anything else. A sorting of MCAS results would tell us more about local real estate values or the percentage of SUV ownership in a community than it would about school quality.

The Effectiveness Index identifies school districts that add value to the learning readiness of their students as indicated by higher-than-demographically-predicted test scores. Identifying such systems is a first step to determining if they are indeed providing more effective educational services to their students. Identifying best practices in effective systems that are demographically similar to less effective systems may help those systems improve their school services.

There are several details of note about this report:

• Small districts (those with fewer than 45-50 students taking MCAS exams in one grade) are not included in this report. This is not a reflection of a bias against small systems. Rather, small sample size can add significant error to any statistical analysis.

• Grade 8 History is not evaluated in the study.

• This third edition covers 98% of the state's public school population in Grades 4 and 8 and 88% of Grade 10 students. (Regional vocational-technical schools, which are not included in this report, educate 10% of the Grade 10 students.)

Massachusetts students attend school in district systems and in regional systems. Most students attend district systems while regionals educate 9% of the Grade 4 students and 15% of the Grade 8 and Grade 10 students in Massachusetts. District school districts (e. g., Boston, Woburn, South Hadley) serve students from only one community. Regional systems serve students from several communities. For example, Nashoba Regional educates children from the towns of Bolton, Lancaster and Stow. In the case of evaluating the effectiveness of regional school systems, community demographics have been factored to reflect the regional school district characteristics.

• As was the case last year, there are many repeat performers in the Effective Districts list. Generally, about half of the sixty systems identified in each grade and subject area (Grades 4, 8, and 10 English Language Arts, Math, Science and Technology) in this report as adding value to the demographic characteristics of their students are repeaters from last year. This is not surprising; a system that had organized itself to enhance student achievement in 1999 is likely to have kept that up in 2000. (See Appendix B for a listing of systems that were identified as over-performing in 1999 and 2000.)

• Districts that over-perform their demography tend to be middle-class or demographically advantaged communities. Generally, upper-demography communities are about two times more likely to over-perform than are communities of lower demography. I define upper demography communities as being in the top 25% of Massachusetts communities based on my demographic methodology; I define lower demography communities as those in the bottom 25% of the state in terms of such demography. (See page 22 for more information on community demography.)

The relative lack of capacity of lower demography communities to exceed their demographic characteristics on standards-based assessments is a persistent point of concern. Students from middle and upper-middle class communities generally do well on any assessments, including MCAS. For such districts, demography is not a major barrier to success. Indeed, for many districts, demography is a guarantor of success. Reforming various elements of their educational offerings will probably be enough to bring virtually all students up to state standards.

For demographically challenged districts, demography is a consistent drain on achievement. Our cities and distressed towns need to over-perform their demography for their students to develop the basic skills needed to live and work in Massachusetts. Based on three years of MCAS, most of the students in middle and upper-demography districts perform well enough now to pass (achieve one point above Fail) the MCAS graduation requirements. Districts that are disadvantaged have much more work to do to overcome their demography and lift more of their students into success in school. After three MCAS administrations, it is clear that disadvantaged districts are having a hard time implementing the dramatic school reforms needed to help their teachers and students outperform their community characteristics.

• Some districts have compiled a solid three-year record of over-performing their demography. It might be helpful to study those districts that have consistently over-performed in specific subject areas. The following lists identify the districts in the state that had the greatest over-performance in ELA and Math, combining results from the 1998, 1999, and 2000 MCAS.

(Appendix C lists districts that have demonstrated the most over-performance over the three MCAS administrations.)

• Many districts that over-performed their demography did so without any apparent benefit from high per-pupil school spending or high levels of new state education reform aid. Generally, over-performers spent at or below state average and were not the recipients of generous amounts of Chapter 70 education reform aid. Braintree and Stoneham, two of the most effective systems in this analysis, receive $336 and $319 respectively per-pupil under education reform. The average amount of education reform aid per-pupil in Massachusetts under education reform is $1,263.

It is important to note that this does not mean that reform money does not matter. This observation illustrates two points:

1) The education reform funding formula is designed to equalize funding. A district spending at or near its formula-determined per-pupil amount when the law was passed would not receive substantial additional funding. Braintree and Stoneham both spent above state average in 1994 when the reform law became functional. These districts did not need to have their spending supplemented by significant state aid.

2) It is easier to implement reform in school districts that do not face major demographic challenges as they work to improve pedagogy. Money matters in demographically challenged districts, but until their educational delivery systems are adapted to meet student needs more effectively, more money will not guarantee success.

Under the state education reform funding formula, demographically disadvantaged systems generally receive much more additional school funding from the state than do middle-class or advantaged systems. For example, under education reform, Lawrence, the most demographically disadvantaged system in the state, receives $3,643 per student per year in reform aid. Many upper-demography communities receive $300 or less per student per year in such aid.

Even with additional state funding, demographically-challenged communities like Lawrence face formidable obstacles in reforming their schools to the extent necessary to educate their students to meet high standards. (See Footnote 11 for information about calculating the amount of new education reform aid. Appendix C, which lists those districts that have had the greatest over-performance on three years of MCAS assessments, also lists the amount of new money districts receive under education reform.)

The good news here is that some of the districts that substantially over-perform their demography did receive substantially more state aid as a result of education reform. These districts tend to be part of Middle Massachusetts, the 140 or so communities that are neither demographically advantaged or disadvantaged. For example, Carver, Leicester, and Uxbridge, all Middle Massachusetts communities, solidly over-performed their demography on the 2000 MCAS in terms of total score (English Language Arts, Math, Science) and also received substantial amounts of new state funding under education reform.

This may suggest that, in the case of reform-savvy middle-class communities that were underfunded before the Education Reform Act of 1993, additional funding has been spent in ways that have boosted student achievement. Again, improving the way educational services are delivered is easier in districts that do not face formidable demographic challenges in terms of the learning readiness of their students.

• One sobering observation from the 1998 and 1999 reports is still true. So far, after seven years of increased reform funding to many systems, there is little evidence that many of our schools have changed in any fundamental ways. MCAS scores were relatively flat from 1998 to 1999 to 2000. There is certainly good news in the scores of some districts, but in the main, the pace of progress has not matched the hopes and expectations of many citizens.

• Demography matters less. When this study was first done on the 1998 MCAS, community demography accounted for 86% of the variation in total MCAS test scores across the Commonwealth. In the 2000 MCAS overall total score, demography accounts for slightly less (83%). Here, however, is one situation where focusing on the overall picture may mask important detail. Demography matters most when looking at total MCAS scores; demography matters relatively less when considering individual grade and subject areas.

Of particular note is the fact that for ELA4 (Grade 4 reading), the amount of variation in test score accounted for by demography dropped from 72% in 1999 to 64% in 2000. Similarly, the amount of score variation accounted for by demography dropped from 68% to 60% for ELA10, Grade 10 reading.

This is encouraging. As the impact of demography on reading outcomes shrinks, the impact of what happens in the classroom grows. The reduced role of demographics in accounting for reading scores is evidence that education reform has made the school a more powerful force in shaping reading outcomes over the past three years. This is good news in that developing reading skills is fundamental to any successful school reform effort. Results on the 2000 MCAS indicate that, while still the dominant factor in achievement, demography matters less than it used to for some subjects and grades.

Testing plays an important role in most of the contemporary school reform efforts in the United States. The Massachusetts education reform effort is no exception. Its testing vehicle is the Massachusetts Comprehensive Assessment System or, as it is commonly known, the MCAS.

MCAS tests have been part of the education landscape in Massachusetts since Spring of 1998. The assessments were developed under the 1993 Education Reform Act that substantially increased state aid for education. The legislation created MCAS to give educators information about how to improve teaching and learning and to determine how much improvement was actually taking place in the schools as a result of the new money and the reform package. The tests are designed to assess student performance across the state.

The MCAS is a battery of tests that have been given each year to students in Grades 4, 8, and 10 in each school district in 1998, 1999, and 2000. The MCAS is aligned with a series of curriculum frameworks developed by the state Department of Education. MCAS covers such academic subjects as math, science, history, and literacy skills. The test scores are broken down by individual student, school, and district. The scores for individual students are available to their parents, teachers, principals, and superintendents. The scores for entire schools and districts are available to the general public. Students are expected to pass (achieve a score of one point above Fail) the Math and English Language Arts 10th Grade MCAS test as a high school graduation requirement.

The chief objective of the state's education reform initiative is to enable public school students to achieve a certain level of knowledge and skill. The Massachusetts Department of Education has established this level by setting out what students are expected to learn in each basic subject. School districts are supposed to see to it that their students learn what they are expected to learn.

MCAS has three broad purposes: Performance Assessment; Diagnostics; and Accountability.

• The Performance aspect of the MCAS - the scores - has been the most publicized part of education reform. Newspapers and electronic media regularly report scores of districts.

• The Diagnostics built into MCAS give educators the capacity to identify specific learning weaknesses in individual students as well as target problems in curriculum and teaching that may impede achievement. School systems should be able to improve teaching and learning through utilization of these diagnostics.

• Finally, MCAS gives citizens an Accountability tool that measures how well schools are doing in moving their students towards high achievement. With MCAS data over time as a guide, we should be able to increase the accountability of individuals, schools, and systems.

With the MCAS, the state has, for the first time in its history, an evaluation mechanism that measures how much progress students are making towards meeting well-defined goals. At the same time, individual school districts are urged to complement the MCAS by developing their own parallel methods of assessing how their students are doing. The Massachusetts education reform effort uses assessment as a way to help all students move towards a high level of academic achievement.

Just as this education improvement effort views higher student achievement as its end, it views the improvement of the public schools as its chief means to achieve this end. What happens in school is by no means the only or even the leading influence on how pupils currently perform on standardized academic tests. However, what happens in school obviously is the only means that is currently within the control of the schools and school systems themselves. School change is the only means of reform that is at the disposal of the education improvement effort as it now exists.

MCAS: Fair of Foul?

Some have criticized MCAS for various reasons: the tests stifle educational creativity; standardized tests are inherently flawed; one test should not be the determinant of being given a high school diploma. There certainly is much room to debate the specifics of any major educational policy, but it may be unwise to blame the test for its findings. When one is diagnosed with high blood pressure and high cholesterol, one can always ignore the diagnosis and criticize the doctor. That type of behavior only ensures substantial problems down the road.

Much of the anti-MCAS activity focuses on the eventual high-stakes nature of MCAS which calls for students to demonstrate a basic mastery of English Language Arts and Math as a condition of receiving a high school diploma. Paul Reville, an architect of contemporary education reform, has noted that the focus on the score aspect of MCAS has clouded the fact that MCAS is designed "to be a tool to help educators establish a data-driven system of education." Because of MCAS, educators have solid information around which to organize their teaching.

The bulk of the criticism has come from advantaged communities, places where the schools are thought to function reasonably well. As Thomas Fortmann, a citizen with successful careers in high technology and urban education, points out:

By any standard - SAT scores, district-wide standardized tests, employment assessments, community college student assessments and MCAS - there is a vast range of performance demonstrated by students in Massachusetts. Literally, a town where the overwhelming majority of students pass MCAS can be situated next to a city where most students do not pass. Andover and Lawrence, Brookline and Boston, Shrewsbury and Worcester all provide graphic evidence of such achievement disparity. On the 2000 MCAS:

Rather that criticizing the messenger, it would be better to use the MCAS as a catalyst for the radical changes needed in our urban systems to make them able to educate their young people. While this may be inconvenient or politically offensive to some people who live in our more advantaged communities, using MCAS to prod effective, reflective change would make for a much more equitable Commonwealth for the new century.

Without substantial changes in urban and other disadvantaged schools, a student's educational success will continue to be a function of zip code. The goal of education reform should be to change the finding of the Willis-Harrington Commission of 1965 that found that, concerning educational quality, "It matters vitally to every individual where the accident of birth and home locates him." Thirty-five years after the most comprehensive education study in state history spoke of the impact of demography and geography on achievement, we still face the challenge of neutralizing the impact of demographics on educational outcomes.