Misconceptions About The Value Added Model's Reliability - Is It Fair?

Is the Value Added Model really fair? Do those who rate teachers (principals and other observers) rate teachers demonstrating the same proficiency at the same levels? Well, to keep the playing field level principals and other observers rated the same lesson. This relationship is referred to as the “reliability” of an observation. To measure this reliability relationship the ratings made by principals were compared to ratings made by other observers.

The principal and observer ratings were compared for all components of the Framework and all levels of the rating scale. If principals rate higher or lower than observers the district can provide principals with training and support on how to rate teachers. What we are finding out is some principals may be utilizing the results to their advantage and manipulating opportunities for teachers.

Here were some of the results:

Principals rated a total of 4,747 teachers for the four

Framework components:

·         Distinguished: 803 (17%) of teachers rated. (High-End)

·         Proficient: 2,530 (53%) of teachers rated. (Mid-Scale)

·         Basic: 1,291 (27%) of teachers rated. (Low-End)

·         Unsatisfactory: 123 (3%) of teachers rated. (Low-End)

External observers rated 4,852 teachers for the four Framework components.

·         Distinguished: 157 (3%) of teachers rated.

·         Proficient: 3,259 (67%) of teachers rated.

·         Basic: 1,342 (28%) of teachers rated.

·         Unsatisfactory: 93 (2%) of teachers rated.

Observers and principals gave similar ratings for Unsatisfactory and Basic. There is a proportional difference in ratings by principals for Distinguished than the ratings given by observers. Principals seemed to favor rating a teacher as distinguished rather than proficient. As it turned out, principals rated “at Distinguished” at higher value-added measures than teachers at the Proficient level.

Examples

Mid-scale rating ratios of Principal & Observer rating Proficient or Basic are 1 to 1

Scale high end – Principal rating Distinguished rather than Proficient, compared to Observer is 6 to 1.

Principal rating Basic rather than Unsatisfactory

Compared to observers, 3 to 4.

The statistics were a result of Principals and Observers observing 257 teachers simultaneously.

★      Principals gave 52% of teachers a Distinguished rating.

★      While Observers gave 24% of the teachers a Distinguished rating.

 Principals Tend To Differ From Observers

Principals were more likely to call a teaching practice Distinguished while Observers were more likely to call the same practice Proficient. Many principals believe it was better to rate at the Distinguished level in order to preserve the relationship between them and the teacher.

Another set of circumstances in the observation was how different principals rated the teaching practice. Principals who were known to be lenient rain in teaching practices higher than the Observers. Principals that were a little harder to please rated the teaching practice lower than did the Observers.

 The study reveals that 11% of the principals rated lower than the Observers, 72% of the principals rated the teachers on par with the observers. Leaving 17% of the principals rating higher than the Observers. There has been some resistance in school administrators complying with the rating Framework. There were some principals who had some difficulty in understanding the concepts laid out by the Framework.

Others, however, thought that teachers put on a “show” when the principal visited the classroom. Principals and administrators need to fully understand the Framework because if they cannot conduct proficient ratings, the evaluation system for teachers is at risk.

 Sources

Authors: Lauren Sartain, Sara Ray Stoelinga, and Eric R. Brown; with: Stuart Luppescu, Kavita Kapadia Matsko, Frances K. Miller, Claire E. Durwood, Jennie Y. Jiang, and Danielle Glazer. November 2011. Rethinking Teacher Evaluation In Chicago. Lessons learned from classroom observation Principal-Teacher conferences and District implementation. Consortium On Chicago School Research At The University Of Chicago Urban Education Institute.

Charlotte Danielson. 2013. The framework for teaching Evaluation Instrument.