Education Benchmarks

The Effectiveness Index (EI) is derived by comparing actual scoreson standardized tests with scores as predicted by a model whichfactors in the role community characteristics play in educationaloutcomes.

The Community Effects Factor (CEF) model was developed in a doctoral dissertation (Education Achievement Communities: A New Modelfor "Kind of Community" in Massachusetts Based on anAnalysis of Community Characteristics Affecting Educational Outcomes, May 1999, University of Massachusaetts,Amherst). That work is the basis for determining school effectiveness. The model examines the relationship between selected demographiccharacteristics and educational outcomes. These characteristicsinclude: average education level, average income, poverty rate,single-parent status, language spoken, and percentage of school-agepopulation enrolled in private schools. These variables werechosen because they correlate with achievement and because theeducation literature identifies them as connected to academicperformance.

In order to refine a better CEF model, it is first necessary tofactor the impact of these demographic variables on each other. This can be done through a technique known as principal componentanalysis that is a statistical mechanism that reduces many variablesto a few salient ones that have the most impact on an outcome. Once the factors have been identified, a regression analysisproduces the equations that can be used to either build a kind-of-communitymodel or to predict expected district performance on achievement tests. The degree to which a community'scharacteristics lifts or lowers test scores is reflected in aCommunity Effects Factor (CEF). a measure of demography.

The CEF, which is a measure of the demographic lift or drag ofeach community concerning educational achievement, is a good pointof departure for analyzing school and school district effectiveness. The CEF identifies expected levels of performance based on communitycharacteristics which, for better or worse, are very powerfulindicators of educational achievement in Massachusetts. In thisanalysis, Weston is the most demographically advantaged communityin the state in terms of educational outcomes (CEF = + 2.8),and Lawrence is the least advantaged (CEF = - 4.8). The CEF has a strong relationship, or correlation,to test scores.

Correlation is a process that identifies the interdependence ofone variable with another. Correlation simply shows "the extentto which two things typically run together." [The Economist, 6 Dec. 1997, p. 82]. Correlation is not equivalent to causation;it can only reveal tendencies between variables, not identifycauses. Correlations simply demonstrate relationships. A perfectcorrelation would be 1.0. For example, the correlation betweeninches and feet is 1.0 because it is a perfect linear fit; 12inches always equals one foot. Correlations in real world situationsinvolving human behavior are never 1.0.

The correlation, or the connection, between spending (Per-PupilExpenditure or PPE) and achievement in Massachusetts is .28, whichis relatively low. While spending clearly matters, merely increasingspending levels has a relatively weak impact on results. Increasingly,many people are coming to the realization that how a system spendsmoney is more important than how much money it spends. The achievementoutcome accounted for by the community effects factor (CEF) ismuch stronger; that relationship is .86. This is not to say thecommunity context, the CEF, is the most important determinantof school success, but it is a significant element that must bea major consideration in any plan to improve education in disadvantagedareas.

The Effectiveness Index was generated in the following manner:

Utilize the 1998 MCAS results as an outcome indicator forachievement in each of the state's most populous 200 communities. (NOTE: Thismodel does not evaluate results in the 151 smallest communities of the statewhich comprise about 7% of the population.)

Utilize the CEF model to predict a score for each district. This predicted score is based solely on community characteristic asthey affect educational outcomes.

Compare the actual to the predicted score. Systems whoseactual scores are significantly higher than predicted scores and whose absolutescores are at or above state average are identified as effective. systems with positive effectiveness indexes but scores belowstate average are identified as noteworthy.

The basis for this model was developed as a doctoral dissertation. The following provides detail on the statistics behind the modelwhich is used to predict expected scores based on demographics.