Sponsored by

the University of Massachusetts

Donahue Institute

by

Dr. Robert D. Gaudet,

Senior Research Analyst

February, 2001

Observations 5

The MCAS 10

Improving Our Schools 14

Effective and Noteworthy School Districts 20

Basic Kinds of Community in Massachusetts 22

The Lessons of Middle Massachusetts 28

Student Achievement in Demographically-Challenged Districts 33

Education Reform in Massachusetts in 2001 38

Appendix A:

Effective and Noteworthy School Districts on the 2000 MCAS 43

Appendix B: Repeaters in 1999 and 2000 58

Appendix C:

Over-performing School Districts on the 1998, 1999, and 2000 MCAS 65

Appendix D:

School Districts that Most Over-performed

their Demography on Total 2000 MCAS Scores 78

Appendix E:

Demographically-Challenged School Districts that

Over-performed on the 1998, 1999, and 2000 MCAS 80

Appendix F: Deriving the Effectiveness Index 86

"Education, then, beyond all other devices of human origin,

is the great equalizer of the conditions of men .…"

– Horace Mann

• Jack Rennie, prime mover behind the Education Reform Act of 1993

The first analysis of school district effectiveness came out in February of 1999 and evaluated the 1998 MCAS in terms of district demography. The second report was released in February 2000 and considered the 1999 MCAS. The central tool of these analyses is the Effectiveness Index methodology that examines the relationship between selected community demographic characteristics and educational outcomes. These characteristics include: average education level, average income, poverty rate, single-parent status, language spoken, and percentage of school-age population enrolled in private schools. These variables were chosen because they correlate with achievement and because the education literature identifies them as connected to academic performance. (See Appendix F for information on the Effectiveness Index.)

Researchers ranging from James Coleman in the 1960s to James Comer in the 1990s have demonstrated that community demographics play a major role in how well children do in school. The Effectiveness Index model provides a means of isolating the role played by community characteristics concerning student performance on statewide educational assessments. With a community’s achievement context factored into its test results, it is possible to know how much value school systems add to demographic expectations. In the absence of a methodology to control for the demographic diversity of Massachusetts, listing MCAS scores primarily demonstrates the relative advantage or disadvantage that community characteristics bring to students. Any raw ranking order of MCAS scores reflects district demography much more than it represents anything else. A sorting of MCAS results would tell us more about local real estate values or the percentage of SUV ownership in a community than it would about school quality.

The Effectiveness Index identifies school districts that add value to the learning readiness of their students as indicated by higher-than-demographically-predicted test scores. Identifying such systems is a first step to determining if they are indeed providing more effective educational services to their students. Identifying best practices in effective systems that are demographically similar to less effective systems may help those systems improve their school services.

There are several details of note about this report:

• As was the case last year, there are many repeat performers in the Effective Districts list. Generally, about half of the sixty systems identified in each grade and subject area (Grades 4, 8, and 10 English Language Arts, Math, Science and Technology) in this report as adding value to the demographic characteristics of their students are repeaters from last year. This is not surprising; a system that had organized itself to enhance student achievement in 1999 is likely to have kept that up in 2000. (See Appendix B for a listing of systems that were identified as over-performing in 1999 and 2000.)

The relative lack of capacity of lower demography communities to exceed their demographic characteristics on standards-based assessments is a persistent point of concern. Students from middle and upper-middle class communities generally do well on any assessments, including MCAS. For such districts, demography is not a major barrier to success. Indeed, for many districts, demography is a guarantor of success. Reforming various elements of their educational offerings will probably be enough to bring virtually all students up to state standards.

For demographically challenged districts, demography is a consistent drain on achievement. Our cities and distressed towns need to over-perform their demography for their students to develop the basic skills needed to live and work in Massachusetts. Based on three years of MCAS, most of the students in middle and upper-demography districts perform well enough now to pass (achieve one point above Fail) the MCAS graduation requirements. Districts that are disadvantaged have much more work to do to overcome their demography and lift more of their students into success in school. After three MCAS administrations, it is clear that disadvantaged districts are having a hard time implementing the dramatic school reforms needed to help their teachers and students outperform their community characteristics.

• Some districts have compiled a solid three-year record of over-performing their demography. It might be helpful to study those districts that have consistently over-performed in specific subject areas. The following lists identify the districts in the state that had the greatest over-performance in ELA and Math, combining results from the 1998, 1999, and 2000 MCAS.

points above each district’s predicted performance level.

• In Grade 4 Math, Orleans scored 29 points above predicted score, with Harvard scoring 25 points above its predicted result.
• In Grade 8 ELA, Stoneham scored 17 points above predicted score with Harvard scoring 14 points over prediction.
• In Grade 8 Math, Harvard scored 31 points and Ipswich scored 24 points over predicted score.
• In Grade 10 ELA, Stoneham’s actual score was 24 points over predicted score, and Hatfield’s students scored 23 above predicted performance.
• In Grade 10 Math, Harvard once again topped the over-scorers with 42 points over predicted score, and Lenox scored 34 over predicted score.

(Appendix C lists districts that have demonstrated the most over-performance over the three MCAS administrations.)

• Many districts that over-performed their demography did so without any apparent benefit from high per-pupil school spending or high levels of new state education reform aid. Generally, over-performers spent at or below state average and were not the recipients of generous amounts of Chapter 70 education reform aid. Braintree and Stoneham, two of the most effective systems in this analysis, receive $336 and $319 respectively per-pupil under education reform. The average amount of education reform aid per-pupil in Massachusetts under education reform is $1,263.

It is important to note that this does not mean that reform money does not matter. This observation illustrates two points:

• The education reform funding formula is designed to equalize funding. A district spending at or near its formula-determined per-pupil amount when the law was passed would not receive substantial additional funding. Braintree and Stoneham both spent above state average in 1994 when the reform law became functional. These districts did not need to have their spending supplemented by significant state aid.

• It is easier to implement reform in school districts that do not face major demographic challenges as they work to improve pedagogy. Money matters in demographically challenged districts, but until their educational delivery systems are adapted to meet student needs more effectively, more money will not guarantee success.

Under the state education reform funding formula, demographically disadvantaged systems generally receive much more additional school funding from the state than do middle-class or advantaged systems. For example, under education reform, Lawrence, the most demographically disadvantaged system in the state, receives $3,643 per student per year in reform aid. Many upper-demography communities receive $300 or less per student per year in such aid.

Even with additional state funding, demographically-challenged communities like Lawrence face formidable obstacles in reforming their schools to the extent necessary to educate their students to meet high standards. (See Footnote 11 for information about calculating the amount of new education reform aid. Appendix C, which lists those districts that have had the greatest over-performance on three years of MCAS assessments, also lists the amount of new money districts receive under education reform.)

The good news here is that some of the districts that substantially over-perform their demography did receive substantially more state aid as a result of education reform. These districts tend to be part of Middle Massachusetts, the 140 or so communities that are neither demographically advantaged or disadvantaged. For example, Carver, Leicester, and Uxbridge, all Middle Massachusetts communities, solidly over-performed their demography on the 2000 MCAS in terms of total score (English Language Arts, Math, Science) and also received substantial amounts of new state funding under education reform.

This may suggest that, in the case of reform-savvy middle-class communities that were underfunded before the Education Reform Act of 1993, additional funding has been spent in ways that have boosted student achievement. Again, improving the way educational services are delivered is easier in districts that do not face formidable demographic challenges in terms of the learning readiness of their students.

• One sobering observation from the 1998 and 1999 reports is still true. So far, after seven years of increased reform funding to many systems, there is little evidence that many of our schools have changed in any fundamental ways. MCAS scores were relatively flat from 1998 to 1999 to 2000. There is certainly good news in the scores of some districts, but in the main, the pace of progress has not matched the hopes and expectations of many citizens.

• Demography matters less. When this study was first done on the 1998 MCAS, community demography accounted for 86% of the variation in total MCAS test scores across the Commonwealth. In the 2000 MCAS overall total score, demography accounts for slightly less (83%). Here, however, is one situation where focusing on the overall picture may mask important detail. Demography matters most when looking at total MCAS scores; demography matters relatively less when considering individual grade and subject areas.

Of particular note is the fact that for ELA4 (Grade 4 reading), the amount of variation in test score accounted for by demography dropped from 72% in 1999 to 64% in 2000. Similarly, the amount of score variation accounted for by demography dropped from 68% to 60% for ELA10, Grade 10 reading.

This is encouraging. As the impact of demography on reading outcomes shrinks, the impact of what happens in the classroom grows. The reduced role of demographics in accounting for reading scores is evidence that education reform has made the school a more powerful force in shaping reading outcomes over the past three years. This is good news in that developing reading skills is fundamental to any successful school reform effort. Results on the 2000 MCAS indicate that, while still the dominant factor in achievement, demography matters less than it used to for some subjects and grades.

Testing plays an important role in most of the contemporary school reform efforts in the United States. The Massachusetts education reform effort is no exception. Its testing vehicle is the Massachusetts Comprehensive Assessment System or, as it is commonly known, the MCAS.

MCAS tests have been part of the education landscape in Massachusetts since Spring of 1998. The assessments were developed under the 1993 Education Reform Act that substantially increased state aid for education. The legislation created MCAS to give educators information about how to improve teaching and learning and to determine how much improvement was actually taking place in the schools as a result of the new money and the reform package. The tests are designed to assess student performance across the state.

The MCAS is a battery of tests that have been given each year to students in Grades 4, 8, and 10 in each school district in 1998, 1999, and 2000. The MCAS is aligned with a series of curriculum frameworks developed by the state Department of Education. MCAS covers such academic subjects as math, science, history, and literacy skills. The test scores are broken down by individual student, school, and district. The scores for individual students are available to their parents, teachers, principals, and superintendents. The scores for entire schools and districts are available to the general public. Students are expected to pass (achieve a score of one point above Fail) the Math and English Language Arts 10th Grade MCAS test as a high school graduation requirement.

The chief objective of the state's education reform initiative is to enable public school students to achieve a certain level of knowledge and skill. The Massachusetts Department of Education has established this level by setting out what students are expected to learn in each basic subject. School districts are supposed to see to it that their students learn what they are expected to learn.

MCAS has three broad purposes: Performance Assessment; Diagnostics; and Accountability.

• The Performance aspect of the MCAS – the scores – has been the most publicized part of education reform. Newspapers and electronic media regularly report scores of districts.
• The Diagnostics built into MCAS give educators the capacity to identify specific learning weaknesses in individual students as well as target problems in curriculum and teaching that may impede achievement. School systems should be able to improve teaching and learning through utilization of these diagnostics.
• Finally, MCAS gives citizens an Accountability tool that measures how well schools are doing in moving their students towards high achievement. With MCAS data over time as a guide, we should be able to increase the accountability of individuals, schools, and systems.

With the MCAS, the state has, for the first time in its history, an evaluation mechanism that measures how much progress students are making towards meeting well-defined goals. At the same time, individual school districts are urged to complement the MCAS by developing their own parallel methods of assessing how their students are doing. The Massachusetts education reform effort uses assessment as a way to help all students move towards a high level of academic achievement.

Just as this education improvement effort views higher student achievement as its end, it views the improvement of the public schools as its chief means to achieve this end. What happens in school is by no means the only or even the leading influence on how pupils currently perform on standardized academic tests. However, what happens in school obviously is the only means that is currently within the control of the schools and school systems themselves. School change is the only means of reform that is at the disposal of the education improvement effort as it now exists.

Some have criticized MCAS for various reasons: the tests stifle educational creativity; standardized tests are inherently flawed; one test should not be the determinant of being given a high school diploma. There certainly is much room to debate the specifics of any major educational policy, but it may be unwise to blame the test for its findings. When one is diagnosed with high blood pressure and high cholesterol, one can always ignore the diagnosis and criticize the doctor. That type of behavior only ensures substantial problems down the road.

The bulk of the criticism has come from advantaged communities, places where the schools are thought to function reasonably well. As Thomas Fortmann, a citizen with successful careers in high technology and urban education, points out:

By any standard – SAT scores, district-wide standardized tests, employment assessments, community college student assessments and MCAS – there is a vast range of performance demonstrated by students in Massachusetts. Literally, a town where the overwhelming majority of students pass MCAS can be situated next to a city where most students do not pass. Andover and Lawrence, Brookline and Boston, Shrewsbury and Worcester all provide graphic evidence of such achievement disparity. On the 2000 MCAS:

• In Andover, 87% passed the Grade 10 ELA; 81% passed the Grade 10 Math. In Lawrence, 33% passed the ELA and 23% passed Math.

• In Brookline, 81% passed the Grade 10 ELA; 78% passed the Math. In Boston, 44% passed ELA and 34% passed Math.

• In Shrewsbury, 92% passed the Grade 10 ELA; 82% passed the Grade 10 Math. In Worcester, 49% passed ELA and 38% passed Math.

Rather that criticizing the messenger, it would be better to use the MCAS as a catalyst for the radical changes needed in our urban systems to make them able to educate their young people. While this may be inconvenient or politically offensive to some people who live in our more advantaged communities, using MCAS to prod effective, reflective change would make for a much more equitable Commonwealth for the new century.

Without substantial changes in urban and other disadvantaged schools, a student's educational success will continue to be a function of zip code. The goal of education reform should be to change the finding of the Willis-Harrington Commission of 1965 that found that, concerning educational quality, "It matters vitally to every individual where the accident of birth and home locates him." Thirty-five years after the most comprehensive education study in state history spoke of the impact of demography and geography on achievement, we still face the challenge of neutralizing the impact of demographics on educational outcomes.

The more test scores can be used to inform decisions about how to alter what happens in school, the better the chances to make the schools more effective in helping their students learn more and improve their performance on standardized academic tests like the MCAS. Properly used, the results can pinpoint which approaches to teaching and learning are working and which are not. The essence of education reform in Massachusetts can be summed up in a few words: Better student performance through more effective schools.

However, for the MCAS to fulfill its intended role in the current education reform effort, there at least two important conditions that have to be met.

To meet the second condition, we must be able to use the MCAS scores as one tool to discern the effectiveness of our schools. We must be able to establish how effective they are today, and to track the rise or fall of their effectiveness in the future. Thus, finding ways to measure school effectiveness is essential to education reform.

Student academic performance, including how students do on MCAS tests, is

influenced by two broad sets of factors: school factors and non-school factors. The first entail what happens in school, and thus what is within the control of the school district itself. The second entails conditions outside the schools, such as the demographic profile of the students and the community. As we look at a given district's average score on an MCAS test, we have to be able to discern how much of the score is tied to school factors, and how much of the score is explained by non-school factors.

How well does the school design and the curriculum promote learning for all? Are teachers top-notch professionals who have both the skills and commitment to teach all students? Are professional development activities rigorously aligned with efforts to increase student achievement? Is there strong, solid leadership in the school? Are there high expectations for all? Are parents full partners in their children's education? Are there adequate resources to do the job? These are all questions about school factors.

In the research reported in this paper, non-school factors consist largely of the overlapping demographic conditions of family life and community life. This study uses six such conditions in a given school district: its median level of educational attainment, its median income level, its percentage of households above the poverty line, its percentage of single-parent families, its percentage of non-English-speaking households, and its level of private school enrollment. Statistical analysis shows that these factors form much of the non-school influence on how the state's students do on such standardized tests as the MCAS.

As we all know, students in advantaged districts tend to get higher standardized test scores than students in disadvantaged districts. Thus, if a district's students get a high average score on an MCAS or other standardized test, the test score by itself does not tell us how much of the score is explained by school factors and how much is explained by non-school factors. A high score might be tied more to advantaged demography than to what actually happens in the district's schools. The score by itself is not a sound guide to how effective is the school district.

We cannot begin to zero in on just how effective the school district itself is unless we can distinguish between the respective influences of the two types of factors. Only then can we discern how effectively the district itself performs, and how much it contributes to its students' average performance on the MCAS.

Identifying systems that over-perform their demography is important in that such systems may have valuable lessons to offer similar systems in their efforts to boost student achievement. In fact, the state Department of Education and MassInsight Education, a well-respected non-profit education reform organization, are working together to identify Exemplary Schools and determine which specific initiatives have contributed to the outstanding performance of a school or district. (See the MassInsight web site for more information - www.massinsight.com.)

It is especially important to identify over-performing demographically-challenged systems. After three years of MCAS, we know that many of our less advantaged districts still have far to go to meet new state standards. Helping these systems move ahead is critically important to ultimate success. When we identify low demography/over-performing systems, we need to study them carefully and learn the lessons they offer on how to reach and teach all children.

The Effectiveness Index (EI) provides a measure of the school district's contribution to its student performance. The Effective Index supplies a piece of crucial insight as to which school districts are more effective.

For a given district, the Effectiveness Index gauges the impact that school factors have on the average MCAS score. The greater the positive impact of the school factors, the higher the district's Effectiveness Index will be.

The Index is calculated in the following manner: For a given district, the six demographic factors are used as the basis for projecting a likely average score on the MCAS. The demographically-likely score is then compared to the average score that the students in the district actually received. The Effectiveness Index is the number that represents the difference between the likely score and the actual score.

If the number is negative – if the actual score is lower than the likely score – then this suggests that what is happening in the schools in the district is not enabling its students to perform beyond the demographic expectations for them. If the number is a positive number – if the actual score is higher than the likely score – then this suggests that what is happening in the schools is helping the district's students to surpass the demographic expectations for them. (For a fuller account of the development of the Effectiveness Index, please see Appendix F.)

This research applies the Effectiveness Index methodology to the MCAS scores of school districts in the state. One of the consistent findings of this analysis is that demography explains most of the variation in test scores from district to district. Demography is less important in explaining 4th grade reading which is perhaps the linchpin of successful education reform efforts (see page 8). That is good, but demography still accounts for most of how much students achieve academically.

It is important to understand that demography is not necessarily destiny. Where a person is born or raised does not dictate how successful or unsuccessful that person will be in life. Similarly, demography does not determine educational achievement in all cases. There are examples of successful urban schools where disadvantaged students do very well by any measure. However, demography does create the context in which schools operate and in which learning occurs. It is less likely that students from a disadvantaged environment will be as successful in educational achievement as will be students from advantaged situations. Demography is about tendency, not destiny.

Results from this year's research are similar to results from last year's work: about 83% of the variation in total MCAS test scores (scores for all test-taking students for the nine MCAS tests combined) is explained by demography. Last year the figure was 86%. In general, demography explains a bit less of individual grade level subject area scores. For example, demography explains 69% of Grade 4 Math scores, 64% of Grade 4 English Language Arts scores, 79% of the Grade 8 Math results, and 68% of Grade 10 Math outcomes. That demography is a powerful predictor of assessment scores is why Weston and Wayland have high MCAS scores and why Holyoke and Brockton have low MCAS scores. Thus, though demography is not destiny, in this case it sets a strong tendency, a tendency that has been confirmed by three applications of MCAS.

A simple way to depict the respective contributions that demography and the schools make to the average level of student performance on the MCAS is this:

Nonetheless, a number of districts achieved test scores that are significantly higher than their demography predicts. The goal of education reform is to flip the relative weights of the two elements in the achievement equation. If education reform is successful, over time we will see that:

becomes the norm; what happens in school will be more powerful than whatever learning readiness or lack thereof the children bring to school.

The Effectiveness Index lets us identify two types of school districts that are interesting in terms of education reform: Effective Districts and Noteworthy Districts.

An EFFECTIVE district meets two specifications:

• Its Effectiveness Index is a positive number - that is, its actual

• Its actual score is equal to or higher than the average MCAS

score for the state as a whole.

Thus, a district that meets both of these specifications invites further scrutiny to determine whether its practices provide a worthwhile model for other districts. Not all districts that meet the two effectiveness specifications will prove to have lessons to teach other systems.

A NOTEWORTHY district fits the first specification but does not fit the second. Since its performance helps its students to go beyond their demography, it is still worthy of note. What such a district is doing educationally can hold useful lessons for districts that are demographically similar, but do not outscore their demography. Such a district is more likely to deliver a return on future public investment than an ineffective district is. Here Ludlow. Leominster, and Everett are outstanding examples.

• Ludlow is a demographically-challenged community near Springfield in Western Massachusetts. Its total MCAS scores are much higher (23 ranks) than its demography would predict.
• Leominster is a small city on Route 2 in Central Massachusetts. Its overall score on all nine MCAS tests is 22 ranks higher than its demography would predict.
• For the third year in a row, Everett's overall score on all nine tests combined is higher than its demography predicts (by 11 ranks). Further, Everett's Grade 4 scores were substantially better than demography would suggest. This is especially heartening in that Everett is a very disadvantaged city that faces many challenges in moving its students up to high standards.

It is very difficult to understand how well we are doing in terms of implementing education reform unless the various characteristics of Massachusetts communities are factored into the evaluation. The Effectiveness Index research is based on a methodology that statistically analyzes demographic data so that individual school districts can be compared to their demographic peers. For purposes of this report, I will place the Commonwealth’s school systems into four categories: Advantaged Massachusetts; Middle Massachusetts; Challenged Massachusetts; and Fifteen Cities, a subset of Challenged Massachusetts. Each of these groupings is based on the demography of their communities.

Middle Massachusetts consists of about 140 communities (the number varies with grade due to the fluctuating number of regional systems as grade level changes) clustered around the demographic center of the state. These are typical Massachusetts hometowns, where average is the norm. Middle Massachusetts is home to communities like Norwood, Beverly, Abington, and South Hadley. These districts average a $6,015 per-pupil expenditure. PPE’s range from $10,814 in Cambridge to $4,391 in Winthrop. Typically, each receives $882 per year per pupil in new education reform aid, compared to the state average of $1,263. The range of per-pupil reform aid is $3,067 (Mohawk Trail Regional) to $63 (Somerset). There are relatively few students (12%) who qualify for free or reduced lunch, a measure of poverty. The statewide average for Free/Reduced Lunch is 26%.

Challenged Massachusetts is made up of the 50 communities at the lower end of the state’s demography. Districts in this category include communities such as Hull, Palmer, and Holbrook, as well as many cities. These districts average a $6,100 per-pupil expenditure. Typically, each receives $1,626 per year per pupil in new education reform aid, compared to the state average of $1,263. The range of state education reform aid is $3,643 (Lawrence) to $233 (Ayer). Many students (33%) qualify for free or reduced lunch, a measure of poverty. The statewide average for Free/Reduced Lunch is 26%.

The Fifteen Cities are the 15 urban communities that are the most demographically disadvantaged of the 50 communities that comprise Challenged Massachusetts. This cluster includes cities such as Worcester, Boston, Fitchburg, and Lawrence. While these communities are included in Challenged Massachusetts, there are three good reasons to focus on them as part of the Fifteen Cities.

The districts in the Fifteen Cities average a $6,503 per-pupil expenditure. The range is $8,118 (Boston) to $5,876 (Fitchburg). On average, each receives $2,447 per year per pupil in new education reform aid, compared to the $1,263 state average. The range of state aid is $3,643 (Lawrence) to $1,559 (Somerville). This is the only segment of the state’s demography where a majority of students (57%) qualify for free or reduced lunch, a measure of poverty. The statewide average for Free/Reduced lunch is 26%.

PPE 98 = Fiscal Year 1998 Per-pupil expenditure

Education Reform Aid = Per-pupil education reform (Chapter 70) aid (See Footnote 11)

Free Lunch Eligibility = Percentage of students eligible for Free or Reduced Lunch

Percentage of M8 FAIL = Percentage of failing students statewide failing M8 MCAS

While a district’s demographics have much to do with the educational performance of its students, some districts have chosen to organize themselves to move beyond their demography. Several systems have demonstrated that community and student characteristics need not be the determining factor in student achievement. (See Appendix A for a listing of over-performers on the 2000 MCAS.)

• On Grade 4 MCAS, Ayer, and Everett over-performed in ELA 4; Ayer, Chelsea, and Orange over-performed in Mathematics; and Ayer, Everett, and Orange over-performed in Science and Technology.

• On Grade 8 MCAS, demographically-challenged overachievers include Gardner, Lawrence, and Berkshire Hills (Great Barrington, Stockbridge, West Stockbridge) in ELA; Chelsea, Lawrence, Leominster, Ludlow, and Gardner in Math; and Gardner, Lawrence, Leominster, and Chelsea in Science and Technology.

• On the Grade 10 MCAS ELA, systems that scored beyond their demography included Southbridge, Clinton, Chelsea, Fitchburg, Ayer, and Hull. On Grade 10 Math, overachievers included Ware, Chelsea, Ayer, Greenfield, Webster, and Southbridge. On the Science and Technology tests, overachievers included Chelsea, Clinton, Southbridge, Hull, Ayer, North Adams, and Fitchburg.

All of the above substantially over-achieved their demography (3 scaled score points or better) despite facing significant demographic challenges.

MCAS and the Class of 2004 Students who took last Spring's MCAS and are members of the Class of 2004 will have to pass MCAS – score Needs Improvement or better on both the Math and English Language Arts (ELA) tests – in order to graduate. Although students take the History MCAS and Science and Technology MCAS tests, members of the Class of 2004 will not have to pass them to meet state requirements.

Let us imagine that the scores obtained on last year's Grade 8 MCAS by students in the Class of 2004 counted for graduation.

• For students who attend school in Advantaged Massachusetts, the 50 communities at the top of the state's demography, 98% of the students would have passed the ELA test and 84% would have passed the Math. Since students must pass both to graduate, at least 16% of the students in demographically advantaged communities – those who did not pass Math – would have failed if passing last year's Grade 8 MCAS were a graduation requirement.

• Concerning students who live in Middle Massachusetts, the 140 or so communities around the center of the state's demography, 94% passed last year's Grade 8 ELA MCAS. In Math, 69% passed. Since students must pass both ELA and Math, at least 31% of students in Middle Massachusetts – those who did not pass Math – would have failed if passing last year's Grade 8 MCAS were a graduation requirement.

• For students who attend school in Challenged Massachusetts, the 50 communities in the lower part of the state's demography, 86% of the students would have passed the ELA test and 51% would have passed the Math. Since students must pass both to graduate, at least 49% of students in demographically disadvantaged communities – those who did not pass Math – would have failed. Children in the schools in the lower 25% of the state's demography account for 49% of all of the public school students in Massachusetts.

• For students who attend school in the Fifteen Cities, the most demographically challenged districts in the state, 74% passed last year's Grade 8 ELA MCAS. On the Grade 8 Math test, 32% of students in the Fifteen Cities passed. Since students must pass both to graduate, at least 68% of students the Fifteen Cities – those who did not pass Math – would have failed. Students in the Fifteen Cites comprise 26% of all of the public school students in Massachusetts.

The following chart illustrates the range of achievement between advantaged and disadvantaged communities in Massachusetts as reflected on the 2000 Grade 8 ELA and Math MCAS.

The basic message of this chart is largely unchanged from last year. Not surprisingly, a district's success in placing its students in the Pass category is very dependent upon its demography. The challenge facing education reformers is clear as we assess the first eight years of the contemporary school improvement effort: Boost the achievement of students in our less advantaged communities.

The 50 most demographically enriched communities of Massachusetts – Advantaged Massachusetts – are fine places to live and attend school. On the other hand, these districts may not have much to teach us about making public education more effective. Students in these districts, by dint of parental involvement and positive demographics, are well prepared academically. Just about all of them do well on just about any test.

Communities in that part of the state known as Middle Massachusetts are much more interesting from an education reform perspective. In the demographic ranking of Massachusetts school districts, about 140 districts are concentrated in the demographic center of the state. These districts, with over two-and-a-half million people, make up what might be called Middle Massachusetts. They are the every-towns that comprise the "typical" Massachusetts community – Norwood, Beverly, Abington, South Hadley. These systems, which are essentially demographically neutral, may be well suited to play a crucial role in education reform.

For the state as a whole, demographic differences among the districts explain 83% of the variation in the districts' average overall test scores. All or much of the other 17% of the variation is probably explained by the differences in how the school districts themselves organize their educational delivery systems.

When looking at either end of the demographic ladder (Advantaged Massachusetts with Weston, Wellesley, Medfield; Challenged Massachusetts with Lawrence, Lynn, New Bedford), we notice that scores tend to be very high or very low. While there is some variation, the solid suburbs score well. Conversely, the cities have low scores, again with some variation. In short, the overwhelming majority of students in advantaged districts pass MCAS easily; most students in disadvantaged districts are very far away from passing.

The pattern in Middle Massachusetts is different. Its districts exhibit a wide range of test scores – although their demography is relatively similar.

This variation can be seen in the following bar graphs that show total MCAS scores for the 2000 MCAS Grade 8 tests. (I am using the Grade 8 tests because these students are members of the second class that will have to pass MCAS in order to qualify for a high school diploma.)

For each of the 14 districts in the demographic middle of the state – the seven districts above the demographic mean and the seven below the mean – the tip of the bar represents its average test score on the 2000 MCAS. While demography for this group is a constant, scores range widely. The top scaled score of these 14 districts in Grade 8 ELA is 247 (Braintree); the lowest score is 238. In Grade 8 Math, the top scaled score is 239; the lowest score is 225. A 9-point difference in ELA scores and a 14-point difference in Math scores in communities that are very similar is significant in that top-scoring systems may well have lessons to teach systems that have not done as well.

The range in the percentage of students who failed the 2000 ELA MCAS varied from 1% (Tyngsborough) and 3% (Braintree) to 14%. Similarly, the range of Fail in Grade 8 Math was 14% (Tyngsborough) to 33%, again a wide performance range in communities that are very similar demographically. Middle Massachusetts school systems looking to improve their achievement might look to Braintree, Woburn, Tyngsborough, and Beverly for guidance.

As you can see, there is wide variation in actual test scores and fail rates in towns that are very similar demographically.

Since the demographic variation is slight, but the variation in test scores is great, this pattern suggests that much of the variation is explained less by demography than by differences in what the schools of Middle Massachusetts are doing.

Further, the test scores of Middle Massachusetts districts with high positive numbers on the Effectiveness Index are just as high as the scores of many of the advantaged districts. For example, for Grade 4, a class that started school after the Education Reform Act of 1993 was enacted, the total MCAS Grade 4 scores of Woburn, a Middle Massachusetts district, are equal to the scores of Newton, a very advantaged community. Similarly, fourth graders in Arlington, another Middle Massachusetts district, scored one point better than students in Franklin and Milton, two demographically advantaged districts. It is important to note that Newton, Franklin, and Milton all made the Effective School Districts list for 2000; these systems are doing a good job adding value to the educational performance of their students. The story here is how well schools in Woburn and Arlington did in moving their students beyond their demographics.

If more Middle Massachusetts districts become as effective as Woburn and Arlington, then more Middle Massachusetts districts will see improved student achievement. Because Middle Massachusetts districts are demographically similar, what makes for effective schools in these districts is more likely to be replicable for other Middle Massachusetts districts looking to improve. Thus, Middle Massachusetts can be an especially fruitful place to seek, and expect to find districts whose schools are well positioned to help all of their students achieve high standards.

Three years of evaluating school district effectiveness with respect to demography has demonstrated that it is much more likely that an upper- or middle-demography district will over-perform its demographically-predicted score. Thus, it is not helpful to compare cities and disadvantaged towns to more middle- and upper-class communities when assessing performance. Lower demography communities should be placed together and compared against each other to identify systems that are doing an exemplary job.

This does not mean that standards should be lowered for some districts. It does mean that developing the kinds of schools where all children in fact do learn will require more creativity, commitment, and time when dealing with demographically disadvantaged communities.

Developing educational strategies that can overcome the impact of disadvantageous demography is a major challenge for many districts. This reality tends to make it much more difficult for an urban district or a disadvantaged town to score above predicted levels. Achieving even a few points above the demographic prediction may indicate positive reform efforts in a lower-demography system. A disadvantaged system that scores at state average will have had to do much more work than a middle-class system with the same score.

The following reflects an analysis of the MCAS performance of the lower 50 school districts as sorted by demography. This research combines results from the 1998, 1999, and 2000 MCAS. The scaled scores listed below reflect total performance on all three MCAS administrations. (See Appendix E for more detail.)

• In Grade 4 ELA, Ayer and Everett scored 9 scaled score points above each district’s predicted performance level. They also scored above the state average.

• In Grade 4 Math, Ayer and Everett scored 15 points above predicted score, and both districts scored at or above state average. Orange and Clinton were also strong performers.

• In Grade 4 Science and Technology, Orange scored 14 points above predicted level. Everett scored 12 over prediction, and Ayer scored 11 points above the demographically predicted score. All three systems were above state average.

• In Grade 8 ELA, Clinton scored 10 points above predicted score, Berkshire Hills scored 9 points over predicted level, and Gardner scored 8 points over prediction. All of these systems scored above state average.

• In Grade 8 Math, Ludlow scored 14 points and Methuen scored 8 points over predicted score, with both systems above state average.

• In Grade 8 Science and Technology, Gardner and Berkshire Hills scored 10 points above prediction with both systems scoring above state average.

• In Grade 10 ELA, Southbridge’s actual score was 12 points over predicted score, Fitchburg’s students scored 9 above prediction, and Hull scored 6 points over predicted performance. All three districts were above state average.

• In Grade 10 Math, Ayer topped the demographically-challenged over-scorers at 18 points over predicted score. Southbridge and Ware scored 16 over predicted level. All three were above the state average.

• In Grade 10 Science and Technology, Webster, North Adams, and Clinton scored 9 points over predicted level with all three scoring above the state average.

Conspicuous in their absence from the lists of schools that over-perform demography are most large urban districts. Lawrence and Chelsea, two very disadvantaged systems that have developed collaborations with the state Department of Education and Boston University respectively, are recognized in this research for producing scores that exceed demography (see following section). There are few other major urban districts that are identified for noteworthy performance.

This is not a surprise in that a basic finding of this research is that demographically challenged systems have a much more difficult time in exceeding demographic expectations than do middle-class or advantaged districts. (See page 5). A superintendent in a district like Lowell or Worcester faces a much more difficult challenge in moving students up to meeting high standards than does a superintendent in a district like Beverly or Dedham. Despite this, there are some large urban systems that have demonstrated a capacity to move beyond demography.

The most encouraging work has been done in Grade 4. These children started school after education reform had been enacted. They are young and susceptible to improved classroom organization and more effective pedagogy. Unfortunately, urban students in Grade 8 and especially in Grade 10, have spent much of their academic careers in classrooms where meeting high standards was not a priority. Therefore, Grade 4 and earlier is where many savvy superintendents have concentrated their reform resources.

In ELA 4 on the 2000 MCAS, Springfield, Chelsea, Holyoke, Worcester, and New Bedford have shown the most over-performance of major urban systems. In Math 4, Chelsea, Lawrence, Worcester, and Springfield have exceeded demography. And in Grade 4 Science and Technology, Chelsea, Lawrence. Fall River, Worcester, and Springfield have moved student performance beyond their community’s characteristics. The overscores for these systems are relatively modest when compared to those of middle-class and advantaged systems, but, since the challenge is so much greater in urban districts, the gains are important. (See Appendix E.)

Lawrence, Chelsea and Holyoke are the three most demographically disadvantaged communities in the Commonwealth. They are in a class by themselves when it comes to the challenges they face each day in overcoming demographic disadvantage. Chelsea schools have been in partnership with Boston University for many years, and the Massachusetts Department of Education has taken a special interest in the Lawrence public schools.

Lawrence and Chelsea tend to be identified by this research as over-performing in terms of their demography. These systems are not designated Effective because they do not score at or above state average, a requirement for an Effective School District in this report. This year, as was the case in the past two reports, Chelsea and Lawrence are among those challenged systems that over-perform on MCAS. The fact that, even with a strong over-performance over their demographically-predicted scores, they are still so far below even state average, is a measure of the enormity of the challenge that faces urban educators.

Holyoke, with similar demography, does not fare as well as Lawrence and Chelsea in its quest to add value to the learning readiness of its children. Holyoke has not had special attention paid to it by the Commonwealth, nor has it been able to collaborate with an outside higher education institution to help improve education. While neither Lawrence nor Chelsea is even close to creating schools that are successful in helping their children reach basic standards, they do fare better than Holyoke. The extra help received by Lawrence and Chelsea has helped their situations, although each district has miles to go before it even begins to meet the goal of ensuring that all children learn.

While students need to develop a broad range of skills and competencies, reading and writing are the foundations for academic success. E. D. Hirsch, Jr., author of The Schools We Need and the developer of the Core Knowledge Curriculum, observes that "... good competency tests in reading turn out to be powerfully indicative of achieved abilities that go far beyond reading." (Hirsch, Education Week, Feb 2, 2000; p, 40 in "The Tests We Need," p, 64) Veteran educators understand that reading is the linchpin of academic success.

Having solid reading and writing skills are necessary conditions for doing well on the MCAS tests. This is true even of the tests in mathematics. Many of the problems on the mathematics tests, particularly in grade eight and ten, are word problems. You cannot understand these problems if you cannot understand the words. In all subjects, moreover, many questions call for a written answer, as short as a sentence or two or as long as an essay of several paragraphs.

The tables in Appendix A lists Effective and Noteworthy districts in terms of student reading performance by grade level on MCAS 2000 (See ELA scores by grade). Several interesting districts that are effective in teaching reading (ELA) in Grade 4 are Eastham, Woburn and Ayer. Each of these districts had MCAS scores substantially above their predicted value and above state average. In Grade 8, Bedford, Nauset regional, Brookline, and Gardner solidly over-scored their demography in ELA and scored above state average.

Grade 10 presents an interesting situation. Some people say that today's high school students understand that MCAS does not count towards graduation so they do not invest significant personal resources in doing their best on the test. Time will tell if that is correct, but it is still important to recognize those systems that do perform beyond demographic expectations. Grade 10 over-performers in English Language Arts include Hatfield, Stoneham, Harvard, Shrewsbury, and Southbridge.

The Education Reform Act of 1993 provides an opportunity to transform our

schools. The Act can be understood in terms of two basic changes it brought about:

• Sharply increasing the amount of state aid targeted at disadvantaged, low-spending communities; and

• The establishment of statewide academic performance standards (and curriculum frameworks) and an assessment device to measure progress towards meeting those standards (the MCAS).

The desired outcome of all of this is improved student achievement. The annual MCAS statewide assessments are one way to gauge changes in student performance over time. MCAS is only given to a few grades (currently 4, 8 and 10 with changes on the way), and it cannot replace local assessments. Districts concerned about helping all children learn should use other tests to supplement MCAS results. For example, a district that employs the Stanford 9 for its local assessments should set up a process to track student progress year-to-year.

Seven years into the endeavor, there has been relatively little research done on the impact of the current education reform effort. We do have some information most of which is from First Findings (cited below):

• We know that billions of dollars in new state school aid has been distributed. Some cite a figure of $6 billion; others cite a lower number, generally around $3 billion. (The explanation for the different numbers reflects how one calculates the increased aid. There has been over $6 billion in new state aid since 1993, but about half of that would have been distributed anyway. The additional $3 billion or so is new funding that was mandated by the Education Reform Act of 1993.)

• School districts have attained the target of 100% of their foundation budgets on schedule. The new funding was intended to give each district a basic foundation level of resources. That has happened. (See First Findings: The Summative Report of the Educational Management Audit Board, Massachusetts Department of Revenue, Division of Local Services, March 2000.)

• Most districts have hired additional teachers, and teacher salaries are up. First Findings found that for the subset of districts considered, teacher salaries rose about 6% faster than inflation. The increases for veteran teachers was more, with real salaries growing by roughly 12%. (See An Update on School Reform in Massachusetts by Thomas J. Kane, Associate Professor of Public Policy, John F. Kennedy School of Government, Harvard University, which was done in conjunction with First Findings.)

On the other side of the ledger:

• Costs related to special education services absorb a disproportionate amount of the new financial resources.

• Most districts have not utilized governance changes establishing performance-based contracts for principals.

• While nearly all school districts have created school improvement plans, many of these plans do not address student performance or test results.

• There is so far little correlation between the increased investment in the classroom as measured by per-pupil net school spending and achievement.

The message of First Findings was that, while some changes have occurred, most school cultures and school systems have not fundamentally reorganized themselves to boost student achievement. The post-reform education delivery system in Massachusetts is essentially the same as it was pre-reform. Since March 2000, when First Findings came out, we have had another MCAS administration that shows relatively little change from the first two administrations. This past summer, Boston and Malden made an effort to change teacher contracts to make schools more effective in teaching students. In both cases, the teacher unions essentially rebuffed the reform efforts. Indeed, the rallying cry at contract negotiation time was that proposed changes were "anti-teacher" and showed "a lack of respect." Thus, as the new century begins with the challenge of doing a better job of educating all children, the work rules, which govern virtually every aspect of how schools are operated in Malden and Boston and other districts, are basically unchanged from what they have been for many years.

Public policy is written in bold strokes. Of necessity, a law or a program must be drawn broadly enough to cover all contingencies. Once a public initiative is in operation, however, the game may change. Seven years after passage of the Education Reform Act of 1993, I would suggest that several points have emerged.

• Education reform is not a singular enterprise. Given the vast diversity of the Massachusetts educational community, a school improvement plan in a town like Natick must be quite different from a plan in a city such as Taunton. Most of our students in middle-class and advantaged districts are already within relatively easy distance of passing MCAS. Most of our students in demographically challenged urban communities are not. The remedy for the cities will be quite different from the prescription for the towns.

It is clear that the cities need a more robust education reform effort. Seven years and three billion dollars into this effort, the general reform tools provided by the Education Reform Act of 1993 do not appear to be strong enough to make a difference to the educational success of urban youth.

The challenge in Middle Massachusetts and Advantaged Massachusetts is to move students up to Proficient and Advanced. That is a far different quest than providing a young person in Boston or Worcester with the basic literacy and math tools needed to have a good chance for success in life.

• We do not know how well we are doing. There is a remarkable lack of evaluation of education reform in Massachusetts. Less than one-quarter of one percent of state education reform funding has been used to evaluate how well we are spending the other 99.75% of the money. The state Department of Education recently developed School Performance Rating Reports. The Massachusetts Education Reform Review Commission has done some thoughtful work. The Department of Revenue's Education Management Accountability Board has done the most extensive reports. The best single effort is First Findings (cited above), but that enterprise was never intended to be a definitive evaluation of the current school improvement effort. It is not productive to point fingers about which agency should have done this or that. What is needed is to jump-start a careful and sustained analysis of education reform in Massachusetts.

• We need to improve the schools. With a booming business sector for most of the late 1990s, some of the structural shortcomings of the Massachusetts economy may have been hidden. Living in Massachusetts is very expensive; the climate is very cold. The Commonwealth does not attract many people from out-of-state into the work force. For Massachusetts to sustain its economic viability, we will have to find a way to find productive employees in-state. Thousands of these potential workers now attend urban schools that, as of today, are not preparing them for success in life or in the job market. If we are to continue to set the standard for economic success, then we must reorganize urban schools to be effective in educating urban youth.

After three years of MCAS, we do see slow progress. A pace of incremental change that may be acceptable in a middle-class district, where just about everyone passes MCAS now, is not good enough for a system with high percentages of students who do not possess basic skills and who fail MCAS.

• MCAS did not create the reality of thousands of students who cannot read, write, or do math at a basic level. MCAS has only identified the underlying reality that many students in many districts are awarded a high school diploma without possessing the basic skills expected of a high school graduate. MCAS is unique in Massachusetts educational history in that it is the first program that tests all systems every year. Pre-MCAS, each school system could choose its own assessment tool and declare victory based on the results. As a Boston public school parent in the 1980s, I was routinely assured that students were learning. As a Massachusetts citizen of the 1990s with access to MCAS data, I know that those statements may have been a tad inaccurate. MCAS has leveled the playing field in terms of assessments. We now know how well our cities are doing relative to our middle class towns relative to our advantaged communities.

MCAS is a truth-teller of sorts in that it assesses all systems every year using the same scoreboard and rules. That is a unique contribution to our understanding of how well our schools are doing. People of good faith and sincere heart can debate various aspects of MCAS, but in the absence of a means to evaluate what our students actually know, schools and districts can declare themselves successful in educating students regardless of the objective reality. It may be that, had MCAS been in place ten or twenty years ago, we would not have seen the routine social promotion that has placed thousands of high school students in situations where it will be very difficult for them to demonstrate any mastery of skills.

This study captures the role that demography plays in student performance on the MCAS. While demography is not destiny, it does establish a tendency. If we overlook the tendency of disadvantaged districts to produce low scores, then we will continue to consign the children of those districts to a future of unfulfilled potential. After seven years of education reform, demography still plays too large a role in the school performance of our children. In Massachusetts in 2001, where you happen to live has more to do with your success in school than any other factor.