Education Benchmarks

We applied the Effectiveness Index to the MCAS scores ofthe 200 largest school districts in the state. These districtscomprise 93 percent of the total population.

The demographic differences between the 200 largest districtsexplain 86% of the variation in the districts's overall averagetest scores - that is, their scores for all of their test-takingstudents for the nine MCAS tests combined. Thus, though demographyisn't destiny in this case, it sets a strong tendency.

A simple way to depict the respective contributions thatdemography and the schools make to the average level of studentperformance on the MCAS is this:

Nonetheless, a number of districts achieved test scoresthat are significantly higher than their demography predicts.

Four Types of School Districts

The Effectiveness Index lets us identify three types of schooldistricts: effective, noteworthy, and ineffective.

An EFFECTIVE district meets two specifications:

1) Its Effectiveness Index is a positive number - that is,its actual score on the test is higher than its demographically likely score.

2) Its actual score is equal to or higher than the averageMCAS score for the state as a whole.

Thus, Stoneham, whose demography places it in the middleof the state's demographic ladder, is an example of an effectivedistrict. Its actual score is substantially higher than its likelyscore. And its actual score is higher than the statewide averagescore. Indeed, its actual score ranks it 46th among the state's200 largest school districts. Its demography would predict thatits score would rank 111.

A NOTEWORTHY district fits the first specification but doesn'tfit the second. Since its performance helps its students to gobeyond their demography, it is still worthy of note. For whatsuch a district is doing can hold useful lessons for districtsthat are demographically similar, but do not outscore their demography.And such a district is more likely to deliver a return on futurepublic investment than an ineffective district is.

Here, Everett and Worcester are outstanding examples.Everett's overall score on all nine tests combined is much higherthan its demography predicts. Worcester's scores on the gradefour tests surpass its demographic prediction.

An INEFFECTIVE district has a negative index number,and its actual score that is less than the average MCAS scorefor the state as a whole.

For each of these three types of districts, the EffectivenessIndex sets a baseline for improvement. If a district is ineffective,then its short-term goal should be to become efective. If a districtis noteworthy, its short-term goal should be to get its test scoreshigh enough to exceed the statewide average. If a district iseffective, its short-term goal should be to raise its actual testsscores further, so that it will become a fourth type of district-- a SUCCESSFUL district.

A SUCCESSFUL district, as presently defined, gets a 75% passrate on each of the tested subjects in each of the three grades.

Currently, no successful districts exist, because none of thestate's 200 largest districts meet the department's definitionof a successful district. Presumably, some districts will improvetheir test scores by enough in the near future to do so.

However, a number of districts did come close to achieving success. On five of the nine MCAS tests, 75% of Harvard's students earnedpassing scores. Harvard got an overall score of 2208 on all nineof the MCAS tests combined, higher than any of the other 199 districts.In Medfield and Wellesley, 75% of the students earned a passingscore on several of the tests.

None of these districts is among the 50 most demographicallydisadvantaged in the state.

This study uses the Effectiveness Index to identify, and thento rank, on each of several fronts, the 50 most effective districtsand 10 most noteworthy districts. Thus, on pages TK thorugh TK,there are four sets of rankings:

The Importance of Reading and Writing

For each of the three grades, this study also ranks thedistricts that are effective or noteworthy in language arts --essentially, reading and writing.

This is because reading and writing are necessary conditionsfor doing well on the MCAS tests. This is true even of the testsin mathematics. Many of the problems on the mathematics's tests,particularly in grade eight and ten, are word problems. You cannotunderstand these problems if you cannot understand the words.In all subjects, moreover, many questions call for a written answer, as short as a sentence or two or as long as an essay ofseveralparagraphs.

Finally, the children who took the grade four MCAS testsin May of 1998 entered first grade in 1994, the first full yearthat the Education Reform Act was in force. Thus, they are thechildren of this long-term education reform initiative. The entirespan of their K-12 experience will be shaped by the requirementsestablished by the reform act. How well these children do intheir school careers will be the first full measure of the impactof the act.

The three sets of language arts rankings appear on pagesTK through TK.

This study highlights these districts because they mighthave lessons to offer to other districts to help them to enhance their contributionto their students's future performance of the MCAS.

In time, we might depict the contributions that demographyand school make to the average level of student performance on the MCAS tests in this fashion:

Middle Massachusetts

In the demographic ranking of the 200 largest school districts,100 districts are concentrated in the demographic middle of thestate. These districts, with 2 million people, make up what mightbe called Middle Massachusetts. They may be well-suited to playa crucial role in the short-term future of education reform.

For the state as a whole, as we've seen, demographic differencesbetween the 200 largest districts explain 86% of the variationin the districts's average overall test scores. All or much ofthe other 14% of the variation is probably explained by the differencesin how the school districts themselves behave.

However, this 14% isn't spread out evenly across all200 districts. Little of this variation is found in the mostadvantaged districts, where the relationship between demographyand test scores is generally strong. Thus, these advantaged districtsgenerally get high test scores. The same holds for the most disadvantageddistricts. Here, the relationship between demography and testscores is also generally strong. The actual test scores of allthese districts are well below the statewide average.

The pattern in Middle Massachusetts is different. Itsdistricts exhibit a wide range of test scores -- even though theirdemography is relatively similar. Thus, much of the 14 pointsof variation is concentrated in the 100 MiddleMass districts.

This variation can be seen in these bar graphs. For eachof the 20 districts closest to the demographic middle of the state-- thus, the districts that form the middle of Middle Massachusettsitself -- the tip of one bar represents its demography, and thetip of the other bar represents its MCAS test score..

Since the demographic variation is slight, but the variationin test scores is great, this pattern suggests that much of thevariation is explained less by demography than by differencesin what the schools of MiddleMass are doing.

Further, the test scores of MiddleMass districts with high positive numbers on the Effectiveness Index are just as high as the scores of many of the advantaged districts. Thus, the scoresof a MiddleMass district like East Longmeadow are very nearly equal to the scores of Longmeadow, one of the more demographicallyadvantaged districts in the state.

Some MiddleMass districts did particularly well on thegrade four tests. The scores of Shrewsbury, Pembroke, and EastLongmeadow were equal to or higher than the scores of such advantageddistricts as Norwell, Cohasset, and Duxbury. To be sure, thesedisticts had high scores, as one would expect. What's less expectedis that districts farther down the demographic ladder did justas well.

So, if more MiddleMass districts become as effective asEast Longmeadow, Shrewsbury, and Pembroke, then more MiddleMassdistricts will get test scores as high as the test scores of the advantaged districts.

Moreover, insofar as MiddleMass districts are demographicallysimilar, what makes for effective schools in an effective districts in MiddleMassis more likely to make for effective schools in an ineffectivedistrict MiddleMassachusetts.

Thus, in the short run, Middle Massachusetts can be anespecially fruitful place to seek, and expect to find, a relativelyswift rise in MCAS test scores.

Education Reform in Massachusetts

The Education Reform Act of 1993 provides an opportunityto transform our schools. The MCAS can be the backbone of our effort to do so.It can assess the performance of districts, schools, and individualstudents, and it can inform the public about its schools. Moreimportantly, MCAS's built-in diagnostics can help teachersto help all children learn better. Under the act, increased statefunds provide substantial amounts of new money for districts touse for reform.

Massachusetts stands at a critical crossroads. The elementsare in place for exciting statewide reform, but the barriers tochange are substantial. This study captures the role that demographyplays in student performance. Again, though demography is notdestiny, it does establish a tendency. If we overlook the tendencyof disadvantaged districts to produce low scores, then we willcontinue to consign the children of those districts to a futureof unfulfilled potential.

Without a broad long-term effort to do what is needed toenhance performance by all, we can expect more of the same: apolarization of academic performance that troubles even thosewhose children are fortunate enough to have been born into a situationthat makes a powerful contribution to their academic success.