Education Benchmarks

The Effectiveness Index (EI) is derived by comparing actual scores on standardized tests with scores as predicted by a model which factors in the role community characteristics play in educational outcomes.

The Community Effects Factor (CEF) model, the statistical basis for the Effectiveness Index, was developed in a doctoral dissertation (EducationAchievement Communities: A New Model for "Kind of Community" in Massachusetts Based on an Analysis of Community Characteristics Affecting Educational Outcomes, May 1998, University of Massachusetts, Amherst). That work is the basis for determining school effectiveness. The model examines the relationship between selected demographic characteristics and educational outcomes. These characteristics include: average education level, average income, poverty rate, single-parent status, language spoken, and percentage of school-age population enrolled in private schools. These variables were chosen because they correlate with achievement and because the education literature identifies them as connected to academic performance.

In order to refine a better CEF model, it is first necessary to factor the impact of these demographic variables on each other. This can be done through a technique known as principal component analysis that is a statistical mechanism that reduces many variables to a few salient ones that have the most impact on an outcome. Once the factors have been identified, a regression analysis produces the equations that can be used to either build a kind-of-community model or to predict expected district performance on achievement tests. The degree to which a community's characteristics lifts or lowers test scores is reflected in a Community Effects Factor (CEF), a measure of demography.

The CEF, which is a measure of the demographic lift or drag of each community concerning educational achievement, is a good point of departure for analyzing school and school district effectiveness. The CEF identifies expected levels of performance based on community characteristics which, for better or worse, are very powerful indicators of educational achievement in Massachusetts. In this analysis, Weston is the most demographically advantaged community in the state in terms of educational outcomes (CEF = + 2.8), and Lawrence is the least advantaged (CEF = - 4.8). The CEF has a strong relationship, or correlation, to test scores. In practical application, the CEF establishes the methodology for predicting likely standardized test scores based on district demography.

Correlation is a process that identifies the interdependence of one variable with another. Correlation simply shows "the extent to which two things typically run together." [The Economist, 6 Dec. 1997, p. 82]. Correlation is not equivalent to causation; it can only reveal tendencies between variables, not identify causes. Correlations simply demonstrate relationships. A perfect correlation would be 1.0. For example, the correlation between inches and feet is 1.0 because it is a perfect linear fit; 12 inches always equals one foot. Correlations in real world situations involving human behavior are never 1.0.

The dissertation research found that the correlation, or the connection, between spending (Per-Pupil Expenditure or PPE) and achievement in Massachusetts was .28, which is relatively low. While spending clearly matters, merely increasing spending levels has a relatively weak impact on results. Increasingly, many people are coming to the realization that how a system spends money is more important than how much money it spends. The achievement outcome accounted for by the community effects factor (CEF) is much stronger; that relationship is .85. This is not to say the community context, the CEF, is the most important determinant of school success, but it is a significant element that must be a major consideration in any plan to improve education in disadvantaged areas.

The Effectiveness Index was generated in the following manner:

• Utilize the 1999 MCAS results as an outcome indicator for achievement in each of the state's most populous 200 communities. (NOTE: This model does not evaluate results in the smallest communities of the state which comprise about 3% of the population.)

• Utilize the CEF model to predict a score for each district. This predicted score is based solely on community characteristic as they affect educational outcomes.

• Compare the actual to the predicted score. Systems whose actual scores are significantly higher than predicted scores and whose absolute scores are at or above state average are identified as effective. Systems with positive effectiveness indexes but scores below state average are identified as noteworthy.

For more information, please contact the UMass Donahue Institute, (617) 287-7055 or contact Robert Gaudet, (617) 469-6843; rgaudet@rcn.com.