Written by Robert D. Gaudet, Senior Research Analyst
University of Massachusetts Donahue Institute - January 1999

Testing plays an important role in most of the contemporaryschool reform efforts in the United States. The Massachusettseducation reform effort is no exception. Its testing vehicle isthe Massachusetts Comprehensive Assessment System or, as it'scommonly known, the MCAS.

The chief objective of the state's education reform initiativeis to enable public school students to achieve a certain levelof knowledge and skill. The Massachusetts Department of Educationhas established this level by setting out what students are expectedto learn in each basic subject. School districts are supposedto see to it that their students learn what they're expected tolearn. The purpose of the MCAS is to gauge periodically how studentsare doing as they try to achieve this level of knowledge and skill.

Each year. in every district in the state, the MCAS testsare given to public school students in grades four, eight, andten. They cover such academic subjects as math, science, andliteracy skills. The test scores are broken down by individualstudent, school, and district. The scores for individual studentsare available to their parents, teachers, principals, and superintendents.The scores for entire schools and districts are available to thegeneral public.

With the MCAS, the state has, for the first time in itshistory, an evaluation mechanism that measures how much progressstudents are making toward well-defined goals. At the same time,individual schools districts are urged to anticipate and complementthe MCAS by developing their own parallel methods of assessinghow their students are doing. Thus, the education reform effortuses assessment as a way to help all students move toward a high level of academic achievement.

Just as this effort views higher student achievement asits end, it views the improvement of the public schools as itschief means to achieve this end. What happens in school is byno means the only or even the leading influence on how pupilscurrently perform on standardized academic tests. However, whathappens in school obviously is the only means that is currentlywithin the control of the schools themselves. So it's the onlymeans of reform that is at the disposal of the education improvementeffort as it now exists.

Improving Our Schools

Thus, the more the test scores can be used to inform decisionsabout how to alter what happens in school, the better the chancesto make the schools more effective in helping their students toimprove their performance on standardized academic tests likethe MCAS. Properly used, the results can pinpoint which approachesto teaching and learning are working and which are not. The MCASalso includes an array of diagnostic tools that let teachers andadministrators spot areas where students perform poorly, so thatthey can work with the students to mend the weaknesses.

Consequently, the essence of education reform in Massachusettscan be summed up in a few words: Better student performance, throughmore effective schools.

However, for the MCAS to fulfill its intended role in thecurrent education reform effort, there at least two importantconditions that have to be met.

FIRST, the tests, and other assessments, must be fair andaccurate. They must measure what children have learned, ratherthan just their social or economic background. They must notbe biased for, or against, any group of students.

SECOND, they must be used to make the public schools moreeffective. Thus, the scores should drive an ongoing analysis ofwhat makes the school experience effective. They must provideteachers with a critical piece of information about the potentiallearning problems and possibilities of individual students. Andthe information must be used as a basis for helping all studentsto do better.

To meet the second condition, we must be able to use theMCAS scores as one tool to discern the effectiveness of our schools.We must be able to establish how effective they are today, andto track the rise or fall of their effectiveness in the future.Thus, finding ways to measure school effectiveness is essentialto education reform.

Measuring Effectiveness

Student academic performance, including how students doon MCAS tests, is influenced by two broad sets of factors: school factors and non-schoolfactors. The first entail what happens in school, and thus whatis within the control of the school district itself. The secondentails conditions outside the schools, such as the demographicprofile of the students and the community. As we look at a givendistrict's average score on an MCAS test, we have to be able todiscern how much of the score is tied to school factors, and howmuch of the score is explained by non-school factors.

How well do the school design and the curriculum promotelearning for all? Are teachers top-notch professionals who have both the skillsand commitment to teach all students? Are professional developmentactivities rigorously aligned with efforts to increase studentachievement? Is there strong, solid leadership in the school?Are there high expectations for all? Are parents full partnersin their children's education? Are there adequate resources todo the job? These are all questions about school factors.1

In the reserach reported in this paper, non-school factorsconsist largely of the overlapping demographic conditions of familylife and community life. We use six such conditions in a givenschool district: its median level of educational attainment, itsmedian income level, its percentage of households above the povertyline, its percentage of single-parent families, its percentageof non-English-speaking households, and its level of private schoolenrollment. Statistical analysis shows that these factors formmuch of the non-school influence on how the state's studentsdo on such standardized tests as the MCAS.2

As we all know, students in advantaged districtstend to get higher standardized test scores than students in disadvantageddistricts. Thus, if a district's students get a high average scoreon an MCAS or other standardized tests, the test score by itselfdoesn't tell us how much of the score is explained by school factorsand how much is explained by non-school factors. A high scoremight be tied more to advantaged demography than to what actuallyhappens in the district's schools. The score by itself isn't asound guide to how effective the school district is.

We cannot begin to zero in on just how effective the schooldistrict itself is unless we can distinguish between the respectiveinfluences of the two types of factors. Only then can we discernhow effectively the district itself performs, and how much itcontributes to its students' average performance on the MCAS.

The Effectiveness Index provides insight into this distinction,and consequently provides some measure of the school district'scontribution to its students's performance. Thus, it suppliesa piece of crucial insight as to which schools are more effective.

For a given district, the Effectiveness Index (EI) gaugesthe impact that school factors have on the average MCAS score. The greater the positive impact of the school factors, the higherthe district's Effectiveness Index will be.

The Index is calculated in the following manner: For agiven district, the six demographic factors are used as the basisfor projecting a likely average score on the MCAS. The demographically-likelyscore is then compared to the average score that the studentsin the district actually received. The Effectiveness Index isthe number that represents the difference between the likely scoreand the actual score.

If the number is negative - if the actual score islower than the likely score - then this suggests that whatis happening in the schools in the district is not enabling itsstudents to perform beyond the demographic expectations for them. If the number is a positive number - if the actual scoreis higher than the likely score - then this suggests thatwhat is happening in the schools is helping the district's studentsto surpass the demographic expectations for them. (For a fulleraccount of of the development of the Effectiveness Index, pleasesee Appendix B.)

¹Per pupil expenditure [PPE] is a school factor. but our measuresof it are not always reliable. There is no standard accountingprocedure for establishing PPE. For example, some systems mightinclude teacher retirement costs, capital costs, federal funds,and long-term disability obligations in their per-pupil spendingfigure. Others might not. So comparisons across districts aredifficult to make.

²Other family an community conditions are crucial to students success, but are hard to observe and measure. One would have to monitor many families and communities closely over time to discern how family and community behavior affect school outcomes. How many books are read in the family? How much time is taken up by TV-watching? How do the community's adult's treat children other than their own? Does the community mentor its young people? It is hard to get reliable answers to such questions. But we know that the children of advantaged families and communities are more likely on average to have such resources and support, and children of less advantaged situations are less likely to have such. Therefore, we use gross measures of such support as a proxy for answers to the more specific questions that are so hard to pursue.