Education Benchmarks

This work applied the Effectiveness Index methodology to the 1999 MCAS scores of school districts in the state. A few districts were too small for results to be statistically reliable so they are not included. The districts considered for this report comprise 97% percent of the total student population of those taking the MCAS in 1999.

One of the consistent findings of this research is that demography explains most of the variation in test scores from district to district. Results from this year's research are similar to results from last year's work: about 84% of the variation in test results (scores for all of the test-taking students for the nine MCAS tests combined) is explained by demography. That is why Weston and Wayland have high MCAS scores and why Holyoke and Brockton have low MCAS scores. Thus, though demography is not destiny, it sets a strong tendency.

A simple way to depict the respective contributions that demography and the schools make to the average level of student performance on the MCAS is this:

Despite the heavy weight of demography in today's learning equation, a number of districts achieved test scores that are significantly higher than their demography predicts.

Effective and Noteworthy School Districts

The Effectiveness Index lets us identify two types of school districts that are interesting in terms of education reform: Effective Districts and Noteworthy Districts.

An EFFECTIVE district meets two specifications:

1) Its Effectiveness Index is a positive number - that is, its actual score on the test is higher than its demographically likely score.

2) Its actual score is equal to or higher than the average MCAS score for the state as a whole.

Thus, a district that meets both of these specifications invites further and closer scrutiny to determine whether its practices provide a worthwhile model for other districts. Not all districts that meet the two effectiveness specifications will prove to have lessons to teach other systems.

Stoneham, Norwood, and Millbury are examples of effective school districts. Actual total MCAS scores in those districts are substantially higher than their demographically likely score. Stoneham's MCAS scores are 58 ranks higher than its demography; Norwood's scores are 42 ranks higher; and Millbury's scores are 34 ranks above its demographic position. Additionally, each of these districts' total 1999 MCAS score is higher than the statewide average score.

A NOTEWORTHY district fits the first specification but doesn't fit the second. Since its performance helps its students to go beyond their demography, it is still worthy of note. What such a district is doing educationally can hold useful lessons for districts that are demographically similar, but do not outscore their demography. And such a district is more likely to deliver a return on future public investment than an ineffective district.

Here, Everett and Gardner are outstanding examples.

• For the second year in a row, Everett's overall score on all nine tests combined is much higher than its demography predicts. Further, Everett's 10th grade scores were substantially better than demography would suggest. This is interesting because urban high schools so far have proven to be very problematic in terms of implementing effective reform.

• Gardner's total MCAS scores on all three MCAS tests surpass its demographic prediction. What is especially encouraging is that Gardner middle and high school students greatly over-performed their demography, again an unusual circumstance for demographically challenged communities.

Making the Grade

MCAS and the Class of 2003 For the Class of 2003, success on the MCAS is currently defined as scoring Needs Improvement or better on both the Math and English Language Arts (ELA) tests. Although students take the Science and Technology MCAS tests, members of the Class of 2003 will not have to pass it to meet state requirements.

Let us imagine that the scores obtained on last year's 8th grade MCAS by students in the Class of 2003 counted for graduation. For students who attend school in the 25 most advantaged communities , 3% of the students would have failed the ELA test and 12% would have failed the Math. Since students must pass both to graduate, 12% of students in demographically advantaged communities would have failed.

For students who attend school in the 25 most disadvantaged communities , 27% of the students would have failed the ELA test and 65% would have failed the Math. Since students must pass both to graduate, 65% of students in demographically disadvantaged communities would have failed. Children in the schools in the 25 most demographically challenged communities account for 31% of all of the public school students in Massachusetts.

The chart on the following page illustrates the range of achievement between advantaged and disadvantaged communities in Massachusetts. This presentation looks at the state in terms of demographic kind of community (KOC) with cities and towns being placed in one of six KOC clusters. The methodology for measuring demography was developed in a dissertation. (See Appendix B.)

The Importance of Reading and Writing

While students need to develop a broad range of skills and competencies, reading and writing are the foundations for academic success. E. D. Hirsch, Jr., author of The Schools We Need and the developer of the Core Knowledge Curriculum, observes that "... good competency tests in reading turn out to be powerfully indicative of achieved abilities that go far beyond reading." (E. D. Hirsch, Education Week, Feb 2, 2000; p, 40, in "The Tests We Need.") Veteran educators understand that reading is the linchpin of academic success.

Having solid reading and writing skills are necessary conditions for doing well on the MCAS tests. This is true even of the tests in Mathematics. Many of the problems on the Math tests, particularly in grade eight and ten, are word problems. You cannot understand these problems if you cannot understand the words. In all subjects, moreover, many questions call for a written answer, as short as a sentence or two or as long as an essay of several paragraphs.

The tables in Appendix A list effective districts in terms of student performance in various subjects including reading. Several interesting systems that are effective in teaching reading (ELA) in Grade 4 are Orange, Monson, and Stoughton. Each of these districts had MCAS scores substantially above their predicted value. In Grade 8, Braintree, Quabbin Regional, and Northampton solidly over-scored their demography in ELA.

This study highlights these districts because they might have lessons to offer to other districts to help them to enhance their contribution to their student's future performance of the MCAS. It is especially important to note and study the success of demographically challenged and demographically average communities in exceeding expectations. In particular, Orange, a very challenged system, deserves consideration for adding value to the reading skills of its students.

Middle Massachusetts

In the demographic ranking of Massachusetts school districts, about 140 districts are concentrated in the demographic middle of the state. These districts, with over two-and-a-half million people, make up what might be called Middle Massachusetts (MidMass). These systems may be well suited to play a crucial role in the short-term future of education reform.

For the state as a whole, as we have seen, demographic differences among the districts explain 84% of the variation in the districts' average overall test scores. All or much of the other 16% of the variation is probably explained by the differences in how the school districts themselves behave.

When looking at either end of the demographic ladder, we notice that scores tend to be very high or very low. While there is some variation, the solid suburbs score well. Conversely, the cities have low scores, again with some variation. In short, the overwhelming majority of students in advantaged districts pass MCAS easily; most students in disadvantaged districts are very far away from passing.

The pattern in Middle Massachusetts is different. Its districts exhibit a wide range of test scores -- although their demography is relatively similar.

This variation can be seen in the bar graph on the next page that shows total MCAS scores for the 1999 MCAS Grade 8 tests. (This utilizes the Grade 8 tests because these students are members of the first class that will have to pass MCAS in order to qualify for a high school diploma. Results for other grade levels are similar.)

For each of the 25 districts in the demographic middle of the state - the districts that form the middle of Middle Massachusetts itself - the tip of the bar represents its actual test score on the 1999 Grade 8 MCAS. The centerline represents the score each community would be expected to achieve based on its demographics. The numbers on the bottom of the chart represent MCAS scaled score numbers. Please Note: The communities are listed to the left of their MCAS performance; only over-performers are identified.

As you can see, there is wide variation in actual test scores in towns that are very similar demographically. Since the demographic variation is slight, but the variation in test scores is great, this pattern suggests that much of the variation is explained less by demography than by differences in what the schools of MidMass are doing.

This variation has practical implications. Woburn is a district that over-performs its demography. Marlboro is a demographically similar system that is literally right next to Woburn in the demographic methodology. For Marlborough, 66% of the students passed the Grade 8 Math MCAS; for Woburn the pass rate was 82%. (Passing is defined as Needs Improvement or better in Math and ELA.) It is also interesting to note that Woburn's per-pupil expenditure was $5991; Marlborough's was $6534.

Further, the test scores of MidMass districts with high positive numbers on the Effectiveness Index are just as high as the scores of many of the advantaged districts. For example, for Grade 4, a class that started school as the Education Reform Act of 1993 was enacted, the total MCAS Grade 4 scores of Norwood, a MidMass district, are equal to the scores of Westwood, a very advantaged community. Similarly, fourth graders in Pembroke, another MidMass district, scored the same as students in Hamilton-Wenham, an advantaged district. It is important to note that both Westwood and Hamilton-Wenham students performed very well on the MCAS; the story here is how well youngsters in Norwood and Pembroke did.

So, if more MidMass districts become as effective as Norwood, Pembroke, and Woburn, then more MidMass districts will get test scores as high as the test scores of the advantaged districts.

Moreover, insofar as MidMass districts are demographically similar, what makes for effective schools in an effective districts in MidMass is more likely to make for effective schools in an ineffective district in MidMass.

Thus, in the short run, MidMass can be an especially fruitful place to seek, and expect to find, a relatively swift rise in MCAS test scores.

Education Reform in Massachusetts

The Education Reform Act of 1993 provides an opportunity to transform our schools. The Act can be understood in terms of two basic changes it brought about:

1) Sharply increasing the amount of state aid targeted at disadvantaged, low-spending communities; and

2) The establishment of statewide academic performance standards (curriculum frameworks) and an assessment device to measure progress towards meeting those standards (the MCAS).

The MCAS can be the backbone of our effort to make the schools work for all students, regardless of where they happened to be born and raised. It can assess the performance of districts, schools, and individual students, and it can inform the public about its schools. More importantly, MCAS's built-in diagnostics can help teachers to help all children learn better. Under the act, increased state funds have provided substantial amounts of new money for many districts to use for reform.

This study captures the role that demography plays in student performance on the 1999 MCAS. While demography is not destiny, it does establish a tendency. If we overlook the tendency of disadvantaged districts to produce low scores and under-educated students, then we will continue to consign the children of those districts to a future of unfulfilled potential. After seven years of education reform in Massachusetts, demography still plays too large a role in the school performance of our children.

Without a broad long-term effort to do what is needed to enhance performance by all, especially students in our disadvantaged systems, we can expect more of the same: A polarization of academic performance that troubles even those whose children are fortunate enough to have been born into a situation that makes a powerful contribution to their academic success.

After several years of education reform, not everything is clear. What is indisputable, however, is that we know that children in advantaged school districts are equipped to succeed in meeting state MCAS graduation requirements. We also know that children from disadvantaged districts, after billions of dollars in new spending, are still woefully under-prepared for success. "All children can learn" certainly sounds nice; the evidence is clear that we are far away from developing the kinds of effective schools where all children in fact do learn.

Massachusetts stands at a critical crossroads. The elements are in place for exciting statewide reform, but the barriers to change are substantial. The Education Reform Act of 1993 is being considered for reauthorization in the spring of 2000. While much has been accomplished, much remains to be done.