Education Benchmarks

The more test scores can be used to inform decisions about how to alter what happens in school, the better the chances to make the schools more effective in helping their students learn more and improve their performance on standardized academic tests like the MCAS. Properly used, the results can pinpoint which approaches to teaching and learning are working and which are not. The essence of education reform in Massachusetts can be summed up in a few words: Better student performance through more effective schools.

However, for the MCAS to fulfill its intended role in the current education reform effort, there at least two important conditions that have to be met.

FIRST, the tests, and other assessments, must be fair and accurate. They must measure what children have learned, rather than just their social or economic background. They must not be biased for, or against, any group of students.

SECOND, assessments must be used to make the public schools more effective. Thus, the scores should drive an ongoing analysis of what makes the school experience effective. They must provide teachers with a critical piece of information about the potential learning problems and possibilities of individual students. Ultimately, the information should be used as a basis for helping all students to do better.

To meet the second condition, we must be able to use the MCAS scores as one tool to discern the effectiveness of our schools. We must be able to establish how effective they are today, and to track the rise or fall of their effectiveness in the future. Thus, finding ways to measure school effectiveness is essential to education reform.

Measuring Educational Effectiveness

Student academic performance, including how students do on MCAS tests, is influenced by two broad sets of factors: school factors and non-school factors. The first entail what happens in school, and thus what is within the control of the school district itself. The second entails conditions outside the schools, such as the demographic profile of the students and the community. As we look at a given district's average score on an MCAS test, we have to be able to discern how much of the score is tied to school factors, and how much of the score is explained by non-school factors.

How well does the school design and the curriculum promote learning for all? Are teachers top-notch professionals who have both the skills and commitment to teach all students? Are professional development activities rigorously aligned with efforts to increase student achievement? Is there strong, solid leadership in the school? Are there high expectations for all? Are parents full partners in their children's education? Are there adequate resources to do the job? These are all questions about school factors.

In the research reported in this paper, non-school factors consist largely of the overlapping demographic conditions of family life and community life. This study uses six such conditions in a given school district: its median level of educational attainment, its median income level, its percentage of households above the poverty line, its percentage of single-parent families, its percentage of non-English-speaking households, and its level of private school enrollment. Statistical analysis shows that these factors form much of the non-school influence on how the state's students do on such standardized tests as the MCAS.

As we all know, students in advantaged districts tend to get higher standardized test scores than students in disadvantaged districts. Thus, if a district's students get a high average score on an MCAS or other standardized test, the test score by itself does not tell us how much of the score is explained by school factors and how much is explained by non-school factors. A high score might be tied more to advantaged demography than to what actually happens in the district's schools. The score by itself is not a sound guide to how effective is the school district.

We cannot begin to zero in on just how effective the school district itself is unless we can distinguish between the respective influences of the two types of factors. Only then can we discern how effectively the district itself performs, and how much it contributes to its students' average performance on the MCAS.

Identifying Over-performing Systems

Identifying systems that over-perform their demography is important in that such systems may have valuable lessons to offer similar systems in their efforts to boost student achievement. In fact, the state Department of Education and MassInsight Education, a well-respected non-profit education reform organization, are working together to identify Exemplary Schools and determine which specific initiatives have contributed to the outstanding performance of a school or district. (See the MassInsight web site for more information - www.massinsight.com.)

It is especially important to identify over-performing demographically-challenged systems. After three years of MCAS, we know that many of our less advantaged districts still have far to go to meet new state standards. Helping these systems move ahead is critically important to ultimate success. When we identify low demography/over-performing systems, we need to study them carefully and learn the lessons they offer on how to reach and teach all children.

The Effectiveness Index

The Effectiveness Index (EI) provides a measure of the school district's contribution to its student performance. The Effective Index supplies a piece of crucial insight as to which school districts are more effective.

For a given district, the Effectiveness Index gauges the impact that school factors have on the average MCAS score. The greater the positive impact of the school factors, the higher the district's Effectiveness Index will be.

The Index is calculated in the following manner: For a given district, the six demographic factors are used as the basis for projecting a likely average score on the MCAS. The demographically-likely score is then compared to the average score that the students in the district actually received. The Effectiveness Index is the number that represents the difference between the likely score and the actual score.

If the number is negative - if the actual score is lower than the likely score - then this suggests that what is happening in the schools in the district is not enabling its students to perform beyond the demographic expectations for them. If the number is a positive number - if the actual score is higher than the likely score - then this suggests that what is happening in the schools is helping the district's students to surpass the demographic expectations for them. (For a fuller account of the development of the Effectiveness Index, please see Appendix F.)

What the Effectiveness Index Tells Us: Statewide Results

This research applies the Effectiveness Index methodology to the MCAS scores of school districts in the state. One of the consistent findings of this analysis is that demography explains most of the variation in test scores from district to district. Demography is less important in explaining 4th grade reading which is perhaps the linchpin of successful education reform efforts (see page 8). That is good, but demography still accounts for most of how much students achieve academically.

It is important to understand that demography is not necessarily destiny. Where a person is born or raised does not dictate how successful or unsuccessful that person will be in life. Similarly, demography does not determine educational achievement in all cases. There are examples of successful urban schools where disadvantaged students do very well by any measure. However, demography does create the context in which schools operate and in which learning occurs. It is less likely that students from a disadvantaged environment will be as successful in educational achievement as will be students from advantaged situations. Demography is about tendency, not destiny.

Results from this year's research are similar to results from last year's work: about 83% of the variation in total MCAS test scores (scores for all test-taking students for the nine MCAS tests combined) is explained by demography. Last year the figure was 86%. In general, demography explains a bit less of individual grade level subject area scores. For example, demography explains 69% of Grade 4 Math scores, 64% of Grade 4 English Language Arts scores, 79% of the Grade 8 Math results, and 68% of Grade 10 Math outcomes. That demography is a powerful predictor of assessment scores is why Weston and Wayland have high MCAS scores and why Holyoke and Brockton have low MCAS scores. Thus, though demography is not destiny, in this case it sets a strong tendency, a tendency that has been confirmed by three applications of MCAS.

A simple way to depict the respective contributions that demography and the schools make to the average level of student performance on the MCAS is this:

Nonetheless, a number of districts achieved test scores that are significantly higher than their demography predicts. The goal of education reform is to flip the relative weights of the two elements in the achievement equation. If education reform is successful, over time we will see that:

becomes the norm; what happens in school will be more powerful than whatever learning readiness or lack thereof the children bring to school.

Effective and Noteworthy School Districts

The Effectiveness Index lets us identify two types of school districts that are interesting in terms of education reform: Effective Districts and Noteworthy Districts.

An EFFECTIVE district meets two specifications:

1) Its Effectiveness Index is a positive number - that is, its actual score on the test is substantially higher than its demographically likely score.

2) Its actual score is equal to or higher than the average MCAS score for the state as a whole.

Thus, a district that meets both of these specifications invites further scrutiny to determine whether its practices provide a worthwhile model for other districts. Not all districts that meet the two effectiveness specifications will prove to have lessons to teach other systems.

Braintree, Stoneham, Norwood, and Millbury are examples of effective school districts. Actual total MCAS scores in those districts are substantially higher than their demographically likely scores. Braintree's demographic rank places it in the middle (#91) of the 202 systems whose students took all three grade level MCAS tests, but its total MCAS score places in #32. Stoneham's total MCAS scores are 48 ranks higher than its demography; Norwood's scores are 24 ranks higher; and Millbury's scores are 37 ranks above its demographic position. Additionally, each of these district's total 2000 MCAS score is higher than the statewide average score.

Of special note this year is the performance of students in Clinton, Ayer, and Gardner. These three communities are demographically challenged yet scored above state average while significantly adding to the demographically-predicted scores of their students. Clinton's total score was 52 ranks above its demographic rank; Ayer's total MCAS score was 43 ranks above its demography; and Gardner was 24 ranks above its demography. Please note that, despite challenging demography that would predict lower-than-state-average scores, each of these communities had to substantially exceed its demography to score above state average.

A NOTEWORTHY district fits the first specification but does not fit the second. Since its performance helps its students to go beyond their demography, it is still worthy of note. What such a district is doing educationally can hold useful lessons for districts that are demographically similar, but do not outscore their demography. Such a district is more likely to deliver a return on future public investment than an ineffective district is. Here Ludlow. Leominster, and Everett are outstanding examples.

• Ludlow is a demographically-challenged community near Springfield in Western Massachusetts. Its total MCAS scores are much higher (23 ranks) than its demography would predict.

• Leominster is a small city on Route 2 in Central Massachusetts. Its overall score on all nine MCAS tests is 22 ranks higher than its demography would predict.

• For the third year in a row, Everett's overall score on all nine tests combined is higher than its demography predicts (by 11 ranks). Further, Everett's Grade 4 scores were substantially better than demography would suggest. This is especially heartening in that Everett is a very disadvantaged city that faces many challenges in moving its students up to high standards.