Education Benchmarks

The Effectiveness Index (EI) is derived by comparing actual scores on standardized tests with scores as predicted by a model which factors in the role community characteristics play in educational outcomes.

The Community Effects Factor (CEF) model was developed in a doctoral
dissertation (Education Achievement Communities: A New Model for "Kind of Community" in Massachusetts Based on an Analysis of Community Characteristics Affecting Educational Outcomes, May 1998, University of Massachusetts, Amherst). That work is the basis for determining school effectiveness. The model examines the relationship between selected demographic characteristics and educational outcomes. These characteristics include: average education level, average income, poverty rate, single-parent status, language spoken, and percentage of school-age population enrolled in private schools. These variables were chosen because they correlate with achievement and because the education literature identifies them as connected to academic performance.

In order to refine a better CEF model, it is first necessary to factor the impact of these demographic variables on each other. This can be done through a technique known as principal components analysis that is a statistical mechanism that reduces many variables to a few salient ones that have the most impact on an outcome. Once the factors have been identified, a regression analysis produces the equations that can be used to either build a kind-of-community model or to predict expected district
performance on achievement tests. The degree to which a community's characteristics lifts or lowers test scores is reflected in a Community Effects Factor (CEF), a measure of demography.

The CEF, which is a measure of the demographic lift or drag of each community concerning educational achievement, is a good point of departure for analyzing school and school district effectiveness. The CEF identifies expected levels of performance based on community characteristics which, for better or worse, are very powerful indicators of educational achievement in Massachusetts. In this analysis, Weston is the most demographically advantaged community in the state in terms of educational outcomes (CEF = + 2.8), and Lawrence is the least advantaged (CEF = - 4.8). The CEF has a strong relationship, or correlation, to test scores.

Correlation is a process that identifies the interdependence of one variable with another. Correlation simply shows "the extent to which two things typically run together." [The Economist, 6 Dec. 1997, p. 82]. Correlation is not equivalent to causation; it can only reveal tendencies between variables, not identify causes. Correlations simply demonstrate relationships. A perfect correlation would be 1.0. For example, the correlation between inches and feet is 1.0 because it is a perfect linear fit; 12 inches always equals one foot. Correlations in real world situations involving human behavior are never 1.0.

The correlation, or the connection, between spending (Per-Pupil Expenditure or PPE) and achievement in Massachusetts is relatively low. While spending clearly matters, merely increasing spending levels has a relatively weak impact on results. Increasingly, many people are coming to the realization that how a system spends money is more important than how much money it spends. The achievement outcome accounted for by the community effects factor (CEF) is much stronger; that relationship is .83. This is not to say the community context, the CEF, is the most important determinant of school success, but it is a significant element that must be a major consideration in any plan to improve education in disadvantaged areas.

The Effectiveness Index (EI) is generated in the following manner:

• · Utilize the 1998 MCAS results as an outcome indicator for achievement in
each of the state's most populous 240 or so districts (the exact number depends on grade level). (NOTE: This model does not evaluate results in the smallest communities of the state which comprise about 2% of the population.)

• · Utilize the EI model to predict a score for each district. This
predicted score is based solely on community characteristic as they affect
educational outcomes.

• · Compare the actual to the predicted score. Systems whose actual scores
are significantly higher than predicted scores and whose absolute scores are at or above state average are identified as Effective. Systems with positive effectiveness indexes but scores below state average are identified as Noteworthy.

The basis for this model was developed as a doctoral dissertation. The following provides detail on the statistics behind the model which is used to predict expected scores based on demographics.