Sponsored by

the University of Massachusetts

Donahue Institute

by

Dr. Robert D. Gaudet,

Senior Research Analyst

March, 2000

"Education, then, beyond all other devices of human origin,

is the great equalizer of the conditions of men..."

-- Horace Mann

The first analysis of school district effectiveness came out in February of 1999 and evaluated the 1998 MCAS in terms of district demography. The central tool of that analysis was the Effectiveness Index methodology that examines the relationship between selected demographic characteristics and educational outcomes. These characteristics include: average education level, average income, poverty rate, single-parent status, language spoken, and percentage of school-age population enrolled in private schools. These variables were chosen because they correlate with achievement and because the education literature identifies them as connected to academic performance.

Researchers ranging from James Coleman in the 1960s to James Comer in the 1990s have demonstrated that community demographics play a major role in how well children do in school. The Effectiveness Index provides a means of isolating the role played by community characteristics concerning student performance on statewide educational assessments. With a community’s achievement context factored into its test results, it is possible to know how much value school systems add to demographic expectations.

The Effectiveness Index identifies school districts that add value to the learning readiness of their students as indicated by higher-than-predicted test scores. Identifying such systems is a first step to determining if they are indeed providing more effective educational services to their students. Identifying best practices in effective systems that are demographically similar to less effective systems may help those systems improve their school services.

This second edition incorporates several new elements into the process:

1) More districts are included. Last year's analysis was confined to communities of over 6,000 population. This year's evaluation considers any school district where at least 50 students took the MCAS exam in a grade. Last year's work covered about 93% of the students in the Commonwealth. This year's extends to about 97% of the student population taking MCAS.

2) Regional school districts are included in this year's work. Last year, only communities, not regional districts which are comprised of several communities, were considered. That meant that individual communities were evaluated outside of their regional school identity. For example, last year Pembroke was evaluated as Pembroke in 10th grade even though Pembroke is part of the Silver Lake Regional District for high school. This year's analysis looks at school district performance on a regional district basis where appropriate. Community demographics have been factored to reflect the regional school district characteristics.

• We see many repeat performers. Many districts that outperformed their demography last year did the same this year. These include Woburn, Harvard, North Reading, and Norwood. This is not surprising; a system that had organized itself to enhance student achievement in 1998 is likely to have kept that up in 1999.

• Some of the new over-performers made the list this year because they were not large enough to have been considered last year. Orleans, a strong over-performer this year, was too small to be included in last year's work. With smaller districts included this year, Orleans is in the mix.

• Districts that over-perform their demography tend to be middle-class or demographically advantaged communities. Generally, upper-demography communities are two to three times more likely to over-perform than communities that are of lower demography. This is unsettling in that middle and upper-middle class communities do not need to over-perform their demography to meet state achievement standards. Based on two years of MCAS, most of their students perform well enough now to pass the MCAS graduation requirements. Districts that are disadvantaged need to overcome their demography in order to lift more of their students into success on MCAS, but so far these districts are having a hard time outperforming their community characteristics.

• Districts that over-performed their demography did so without any apparent benefit from high per-pupil school spending or high levels of new state education reform aid. Generally, over-performers spent at or below state average and were not the recipients of generous amounts of Chapter 70 (education assistance and reform) aid. This fact is interesting in that providing additional funding is a major reform tool of the Education Reform Act of 1993. Based on this analysis, that extra funding has not necessarily purchased the systemic change that enables systems to help their students perform above their demographic expectations.

• So far after seven years of reform funding, there is little evidence that the schools have changed in any fundamental ways. MCAS scores were relatively flat from 1998 to 1999. This is of concern because EducationWeek pointed out that "Of the states in the early stages of testing, only Massachusetts failed to post significant gains in the second year of its new assessment." (David J. Hoff, EducationWeek, Jan 26, 2000; p. 12; "Testing's Ups And Downs Predictable.") With flat early results, the challenge of improving results, especially for demographically disadvantaged systems, is daunting.

Identifying systems that over-perform their demography is important in that such systems may have valuable lessons to offer similar systems in their efforts to boost student achievement. Most of the over-performers were not among the demographically disadvantaged systems. This means that when we identify low demography/over-performing systems, we need to study them carefully and see if they indeed do have lessons to teach their peers across the Commonwealth. After two years of MCAS, we know that many of our less advantaged districts have far to go to meet new state standards, so helping them move ahead is critically important to ultimate success.

The real battle in education reform is to change the way disadvantaged systems teach their children. Approximately 60% of students in very disadvantaged systems are not now in a position to meet state graduation standards. Those systems are home to 266,000 students, 31% of the state's student population. These are future citizens who are struggling to learn how to read, write, and do basic math at a level sufficient to meet state standards and to succeed in life. Identifying peer systems that have figured out how to help their students over-perform their demography is essential to making education reform work in Massachusetts.

Testing plays an important role in most of the contemporary school reform efforts in the United States. The Massachusetts education reform effort is no exception. Its testing vehicle is the Massachusetts Comprehensive Assessment System or, as it is commonly known, the MCAS.

The MCAS is a battery of tests that is given each year to students in Grades 4, 8, and 10 in each school district. The MCAS is aligned with a series of curriculum frameworks that are being developed by the state Department of Education. MCAS covers such academic subjects as math, science, and literacy skills, with more subjects to be added later. The test scores are broken down by individual student, school, and district. The scores for individual students are available to their parents, teachers, principals, and superintendents. The scores for entire schools and districts are available to the public.

The chief objective of the state's education reform initiative is to enable public school students to achieve a certain level of knowledge and skill. The Massachusetts Department of Education has established this level by setting out what students are expected to learn in each basic subject. School districts are supposed to see to it that their students learn what they are expected to learn. The purpose of the MCAS is to gauge periodically how students are doing as they try to achieve this level of knowledge and skill.

With the MCAS, the state has, for the first time in its history, an evaluation mechanism that measures how much progress students are making towards meeting established goals. At the same time, individual schools districts are urged to anticipate and complement the MCAS by developing their own parallel methods of assessing how their students are doing. Thus, the education reform effort uses assessment as a way to help all students move toward a high level of academic achievement.

Just as this overall effort views higher student achievement as its end, it views the improvement of the public schools as its chief means to achieve this end. What happens in school is by no means the only or even the leading influence on how pupils currently perform on standardized academic tests. However, what happens in school obviously is the only means that is currently within the control of the schools themselves. So it is the only means of reform that is at the disposal of the education improvement effort as it now exists.

The more the test scores can be used to inform decisions about how to alter what happens in school, the better the chances to make the schools more effective in helping their students learn more. Properly used, the results can pinpoint which approaches to teaching and learning are working and which are not. The MCAS also includes an array of diagnostic tools that let teachers and administrators spot areas where students perform poorly, so that the staff can work with the students to mend the weaknesses.

Consequently, the essence of education reform in Massachusetts can be

summed up in a few words: Better student performance, through more effective schools.

However, for the MCAS to fulfill its intended role in the current education reform effort, there at least two important conditions that have to be met.

FIRST, the tests, and other assessments, must be fair and accurate. They must measure what children have learned, rather than just their social or economic background. They must not be biased for, or against, any group of students.

SECOND, the tests must be used to make the public schools more effective. Thus, the scores should drive an ongoing analysis of what makes the school experience effective. They must provide teachers with a critical piece of information about the potential learning problems and possibilities of individual students. And the information must be used as a basis for helping all students to do better.

To meet the second condition, we must be able to use the MCAS scores as one tool to discern the effectiveness of our schools. We must be able to establish how effective they are today, and to track the rise or fall of their effectiveness in the future. Thus, finding ways to measure school effectiveness is essential to education reform.

Student academic performance, including how students do on MCAS tests, is

influenced by two broad sets of factors: school factors and non-school factors. The first entail what happens in school, and thus what is within the control of the school district itself. The second entails conditions outside the schools, such as the demographic profile of the students and the community. As we look at a given district's average score on an MCAS test, we have to be able to discern how much of the score is tied to school factors, and how much of the score is explained by non-school factors.

How well do the school design and the curriculum promote learning for all?

Are teachers top-notch professionals who have both the skills and commitment to teach all students? Are professional development activities rigorously aligned with efforts to increase student achievement? Is there strong, solid leadership in the school? Are there high expectations for all? Are parents full partners in their children's education? Are there adequate resources to do the job? These are all questions about school factors.

In the research reported in this paper, non-school factors consist largely of the overlapping demographic conditions of family life and community life. This study utilizes six such conditions in a given school district: its median level of educational attainment; its median income level; its percentage of households above the poverty line; its percentage of single-parent families; its percentage of non-English-speaking households; and its level of private school enrollment. Statistical analysis shows that these factors form much of the non-school influence on how the state's students do on such standardized tests as the MCAS.

We cannot begin to zero in on just how effective the school district itself is unless we can distinguish between the respective influences of the two types of factors. Only then can we discern how effectively the district itself performs, and how much it contributes to its students' average performance on the MCAS.

The Effectiveness Index (EI) provides insight into this distinction, and consequently provides some measure of the school district's contribution to its student performance. Thus, it supplies a piece of crucial insight as to which schools are more effective.

For a given district, the Effectiveness Index gauges the impact that school factors have on the average MCAS score. The greater the positive impact of the school factors, the higher the district's Effectiveness Index will be.

The Effectiveness Index is calculated in the following manner: For a given district, the six demographic factors are used as the basis for projecting a likely average score on the MCAS. The demographically-likely score is then compared to the average score that the students in the district actually received. The Effectiveness Index is the number that represents the difference between the likely score and the actual score.

If the number is negative – if the actual score is lower than the likely score – then this suggests that what is happening in the schools in the district is not enabling its students to perform beyond the demographic expectations for them. If the number is a positive number – if the actual score is higher than the likely

score – then this suggests that what is happening in the schools is helping the district's students to surpass the demographic expectations for them. (For a fuller account of the development of the Effectiveness Index, please see Appendix B.)

This work applied the Effectiveness Index methodology to the 1999 MCAS scores of school districts in the state. A few districts were too small for results to be statistically reliable so they are not included. The districts considered for this report comprise 97% percent of the total student population of those taking the MCAS in 1999.

One of the consistent findings of this research is that demography explains most of the variation in test scores from district to district. Results from this year's research are similar to results from last year's work: about 84% of the variation in test results (scores for all of the test-taking students for the nine MCAS tests combined) is explained by demography. That is why Weston and Wayland have high MCAS scores and why Holyoke and Brockton have low MCAS scores. Thus, though demography is not destiny, it sets a strong tendency.

A simple way to depict the respective contributions that demography and the schools make to the average level of student performance on the MCAS is this:

Despite the heavy weight of demography in today’s learning equation, a number of districts achieved test scores that are significantly higher than their demography predicts.

The Effectiveness Index lets us identify two types of school districts that are interesting in terms of education reform: Effective Districts and Noteworthy Districts.

An EFFECTIVE district meets two specifications:

1) Its Effectiveness Index is a positive number - that is, its actual

score on the test is higher than its demographically likely score.

2) Its actual score is equal to or higher than the average MCAS

score for the state as a whole.

Thus, a district that meets both of these specifications invites further and closer scrutiny to determine whether its practices provide a worthwhile model for other districts. Not all districts that meet the two effectiveness specifications will prove to have lessons to teach other systems.

Stoneham, Norwood, and Millbury are examples of effective school districts. Actual total MCAS scores in those districts are substantially higher than their demographically likely score. Stoneham's MCAS scores are 58 ranks higher than its demography; Norwood's scores are 42 ranks higher; and Millbury's scores are 34 ranks above its demographic position. Additionally, each of these districts' total 1999 MCAS score is higher than the statewide average score.

A NOTEWORTHY district fits the first specification but doesn't fit the second. Since its performance helps its students to go beyond their demography, it is still worthy of note. What such a district is doing educationally can hold useful lessons for districts that are demographically similar, but do not outscore their demography. And such a district is more likely to deliver a return on future public investment than an ineffective district.

Here, Everett and Gardner are outstanding examples.

• For the second year in a row, Everett's overall score on all nine tests combined is much higher than its demography predicts. Further, Everett's 10th grade scores were substantially better than demography would suggest. This is interesting because urban high schools so far have proven to be very problematic in terms of implementing effective reform.

• Gardner's total MCAS scores on all three MCAS tests surpass its demographic prediction. What is especially encouraging is that Gardner middle and high school students greatly over-performed their demography, again an unusual circumstance for demographically challenged communities.

MCAS and the Class of 2003 For the Class of 2003, success on the MCAS is currently defined as scoring Needs Improvement or better on both the Math and English Language Arts (ELA) tests. Although students take the Science and Technology MCAS tests, members of the Class of 2003 will not have to pass it to meet state requirements.

Let us imagine that the scores obtained on last year's 8th grade MCAS by students in the Class of 2003 counted for graduation. For students who attend school in the 25 most advantaged communities, 3% of the students would have failed the ELA test and 12% would have failed the Math. Since students must pass both to graduate, 12% of students in demographically advantaged communities would have failed.

For students who attend school in the 25 most disadvantaged communities, 27% of the students would have failed the ELA test and 65% would have failed the Math. Since students must pass both to graduate, 65% of students in demographically disadvantaged communities would have failed. Children in the schools in the 25 most demographically challenged communities account for 31% of all of the public school students in Massachusetts.

The chart on the following page illustrates the range of achievement between advantaged and disadvantaged communities in Massachusetts. This presentation looks at the state in terms of demographic kind of community (KOC) with cities and towns being placed in one of six KOC clusters. The methodology for measuring

Not surprisingly, a district's success in placing its students in the pass category is very dependent upon its demography. The dimension of the challenge facing education reformers is clear as we end the first phase of the school improvement effort: Find ways to boost achievement of students in our less advantaged communities.

Advantaged communities include Franklin, Marblehead, Natick, and Newton.

Upper Mid-Mass communities include Walpole, Ipswich, Brookline, and Chatham.

Lower Mid-Mass communities include Beverly, East Bridgewater, Stoneham, and Agawam.

Challenged communities include Dartmouth, Quincy, Peabody, and Methuen.

Very Challenged communities include Everett, Ware, Fall River, and Boston.

While students need to develop a broad range of skills and competencies, reading and writing are the foundations for academic success. E. D. Hirsch, Jr., author of The Schools We Need and the developer of the Core Knowledge Curriculum, observes that "... good competency tests in reading turn out to be powerfully indicative of achieved abilities that go far beyond reading." (E. D. Hirsch, Education Week, Feb 2, 2000; p, 40, in "The Tests We Need.") Veteran educators understand that reading is the linchpin of academic success.

Having solid reading and writing skills are necessary conditions for doing well on the MCAS tests. This is true even of the tests in Mathematics. Many of the problems on the Math tests, particularly in grade eight and ten, are word problems. You cannot understand these problems if you cannot understand the words. In all subjects, moreover, many questions call for a written answer, as short as a sentence or two or as long as an essay of several paragraphs.

The tables in Appendix A list effective districts in terms of student performance in various subjects including reading. Several interesting systems that are effective in teaching reading (ELA) in Grade 4 are Orange, Monson, and Stoughton. Each of these districts had MCAS scores substantially above their predicted value. In Grade 8, Braintree, Quabbin Regional, and Northampton solidly over-scored their demography in ELA.

This study highlights these districts because they might have lessons to offer

to other districts to help them to enhance their contribution to their student's future performance of the MCAS. It is especially important to note and study the success of demographically challenged and demographically average communities in exceeding expectations. In particular, Orange, a very challenged system, deserves consideration for adding value to the reading skills of its students.

In the demographic ranking of Massachusetts school districts, about 140 districts are concentrated in the demographic middle of the state. These districts, with over two-and-a-half million people, make up what might be called Middle Massachusetts (MidMass). These systems may be well suited to play a crucial role in the short-term future of education reform.

For the state as a whole, as we have seen, demographic differences among the districts explain 84% of the variation in the districts' average overall test scores. All or much of the other 16% of the variation is probably explained by the differences in how the school districts themselves behave.

When looking at either end of the demographic ladder, we notice that scores tend to be very high or very low. While there is some variation, the solid suburbs score well. Conversely, the cities have low scores, again with some variation. In short, the overwhelming majority of students in advantaged districts pass MCAS easily; most students in disadvantaged districts are very far away from passing.

The pattern in Middle Massachusetts is different. Its districts exhibit a wide range of test scores -- although their demography is relatively similar.

This variation can be seen in the bar graph on the next page that shows total MCAS scores for the 1999 MCAS Grade 8 tests. (This utilizes the Grade 8 tests because these students are members of the first class that will have to pass MCAS in order to qualify for a high school diploma. Results for other grade levels are similar.)

For each of the 25 districts in the demographic middle of the state – the districts that form the middle of Middle Massachusetts itself – the tip of the bar represents its actual test score on the 1999 Grade 8 MCAS. The centerline represents the score each community would be expected to achieve based on its demographics. The numbers on the bottom of the chart represent MCAS scaled score numbers. Please Note: The communities are listed to the left of their MCAS performance;

As you can see, there is wide variation in actual test scores in towns that are very similar demographically. Since the demographic variation is slight, but the variation in test scores is great, this pattern suggests that much of the variation is explained less by demography than by differences in what the schools of MidMass are doing.

This variation has practical implications. Woburn is a district that over-performs its demography. Marlboro is a demographically similar system that is literally right next to Woburn in the demographic methodology. For Marlborough, 66% of the students passed the Grade 8 Math MCAS; for Woburn the pass rate was 82%. (Passing is defined as Needs Improvement or better in Math and ELA.) It is also interesting to note that Woburn’s per-pupil expenditure was $5991; Marlborough’s was $6534.

Further, the test scores of MidMass districts with high positive numbers on the Effectiveness Index are just as high as the scores of many of the advantaged districts. For example, for Grade 4, a class that started school as the Education Reform Act of 1993 was enacted, the total MCAS Grade 4 scores of Norwood, a MidMass district, are equal to the scores of Westwood, a very advantaged community. Similarly, fourth graders in Pembroke, another MidMass district, scored the same as students in Hamilton-Wenham, an advantaged district. It is important to note that both Westwood and Hamilton-Wenham students performed very well on the MCAS; the story here is how well youngsters in Norwood and Pembroke did.

So, if more MidMass districts become as effective as Norwood, Pembroke, and Woburn, then more MidMass districts will get test scores as high as the test scores of the advantaged districts.

Moreover, insofar as MidMass districts are demographically similar, what

makes for effective schools in an effective districts in MidMass is more likely to make for effective schools in an ineffective district in MidMass.

Thus, in the short run, MidMass can be an especially fruitful place to seek, and expect to find, a relatively swift rise in MCAS test scores.

The Education Reform Act of 1993 provides an opportunity to transform our

schools. The Act can be understood in terms of two basic changes it brought about:

1) Sharply increasing the amount of state aid targeted at disadvantaged, low-spending communities; and

2) The establishment of statewide academic performance standards (curriculum frameworks) and an assessment device to measure progress towards meeting those standards (the MCAS).

The MCAS can be the backbone of our effort to make the schools work for all students, regardless of where they happened to be born and raised. It can assess the performance of districts, schools, and individual students, and it can inform the public about its schools. More importantly, MCAS’s built-in diagnostics can help teachers to help all children learn better. Under the act, increased state funds have provided substantial amounts of new money for many districts to use for reform.

This study captures the role that demography plays in student performance on the 1999 MCAS. While demography is not destiny, it does establish a tendency. If we overlook the tendency of disadvantaged districts to produce low scores and under-educated students, then we will continue to consign the children of those districts to a future of unfulfilled potential. After seven years of education reform in Massachusetts, demography still plays too large a role in the school performance of our children.

Without a broad long-term effort to do what is needed to enhance performance by all, especially students in our disadvantaged systems, we can expect more of the same: A polarization of academic performance that troubles even those whose children are fortunate enough to have been born into a situation that makes a powerful contribution to their academic success.

After several years of education reform, not everything is clear. What is indisputable, however, is that we know that children in advantaged school districts are equipped to succeed in meeting state MCAS graduation requirements. We also know that children from disadvantaged districts, after billions of dollars in new spending, are still woefully under-prepared for success. "All children can learn" certainly sounds nice; the evidence is clear that we are far away from developing the kinds of effective schools where all children in fact do learn.

Massachusetts stands at a critical crossroads. The elements are in place for exciting statewide reform, but the barriers to change are substantial. The Education Reform Act of 1993 is being considered for reauthorization in the spring of 2000. While much has been accomplished, much remains to be done.

------------------------------------------------------------

The following charts list the top sixty districts in terms of their score on the Effectiveness Index. The Effectiveness Index is the difference between the likely score based on a district's demography and the district's actual score on the MCAS. Districts are listed from higher to lower Effectiveness Indexes.

When analyzing anything about student performance, it is important to understand that many Massachusetts communities are part of regional school districts and that, depending on grade level, a community's school district may change. Thus, the Grade 4 results reflect evaluating a different set of school districts than do the Grade 8 results and do the Grade 10 results. The following charts represent 97% of the state's students. Due to small sample size, those districts where fewer than 50 students took the MCAS are not included.

There are 351 communities in Massachusetts. There are 204 school districts whose children took MCAS tests in all three grades. The other communities are part of various regional school districts. Thus, the only way to compare total MCAS score is to limit that analysis to the 204 systems whose children in fact took MCAS tests at all three grade levels, Grades 4, 8, and 10.

Listings are provided for the following categories:

Grade four performance: all three subjects combined (English Language Arts, Math, Science)

Grade four performance: English Language Arts

Grade four performance: Math

Grade four performance: Science

Grade eight performance: All three subjects combined (English Language Arts, Math, Science. (NOTE: The History test results are not included in this analysis.)

Grade eight performance: English Language Arts

Grade eight performance: Math

Grade eight performance: Science

Grade ten performance: All three subjects combined (English Language Arts, Math, Science)

Grade ten performance: English Language Arts

Grade ten performance: Math

Grade ten performance: Science

This is also a listing of the 60 most Effective Districts of the 204 systems whose students participated in all three grades of MCAS in-district. This listing is derived from the same basic methodology as are the lists for Effective Districts in individual grades and subjects. It is based on looking at the difference between a district’s demographic rank and its MCAS score rank. For example, if a district was ranked the 100th highest demographically and its students achieved a combined MCAS score that was the 80th highest in the state, that district would be a +20 in terms of overall effectiveness. If that district that was ranked #100 demographically and achieved a combined score that was the 120th highest, it would be a -20 in terms of overall effectiveness.

The Effectiveness Index (EI) is derived by comparing actual scores on standardized tests with scores as predicted by a model which factors in the role community characteristics play in educational outcomes.

The Community Effects Factor (CEF) model, the statistical basis for the Effectiveness Index, was developed in a doctoral dissertation (EducationAchievement Communities: A New Model for "Kind of Community" in Massachusetts Based on an Analysis of Community Characteristics Affecting Educational Outcomes, May 1998, University of Massachusetts, Amherst). That work is the basis for determining school effectiveness. The model examines the relationship between selected demographic characteristics and educational outcomes. These characteristics include: average education level, average income, poverty rate, single-parent status, language spoken, and percentage of school-age population enrolled in private schools. These variables were chosen because they correlate with achievement and because the education literature identifies them as connected to academic performance.

In order to refine a better CEF model, it is first necessary to factor the impact of these demographic variables on each other. This can be done through a technique known as principal component analysis that is a statistical mechanism that reduces many variables to a few salient ones that have the most impact on an outcome. Once the factors have been identified, a regression analysis produces the equations that can be used to either build a kind-of-community model or to predict expected district

performance on achievement tests. The degree to which a community's characteristics lifts or lowers test scores is reflected in a Community Effects Factor (CEF), a measure of demography.

The CEF, which is a measure of the demographic lift or drag of each community concerning educational achievement, is a good point of departure for analyzing school and school district effectiveness. The CEF identifies expected levels of performance based on community characteristics which, for better or worse, are very powerful indicators of educational achievement in Massachusetts. In this analysis, Weston is the most demographically advantaged community in the state in terms of educational outcomes (CEF = + 2.8), and Lawrence is the least advantaged (CEF = – 4.8). The CEF has a strong relationship, or correlation, to test scores. In practical application, the CEF establishes the methodology for predicting likely standardized test scores based on district demography.

o Utilize the CEF model to predict a score for each district. This predicted score is based solely on community characteristic as they affect educational outcomes.

o Compare the actual to the predicted score. Systems whose actual scores are significantly higher than predicted scores and whose absolute scores are at or above state average are identified as effective. Systems with positive effectiveness indexes but scores below state average are identified as noteworthy.

For more information, please contact the UMass Donahue

Institute, (617) 287-7055 or contact Robert Gaudet,