An empirical analysis of rating effectiveness for a state productivity and quality award

This study measured the rating effectiveness (interrater reliability, halo error, leniency or severity, range restriction) for each sector (Private Manufacturing Sector, PrivateService Sector, Public Local Sector, and Public State and Federal Sector) of a StateQuality Award using examiner scores from the 1999 award process. Results were compared with the rating effectiveness of the 1998 State Quality Award examiner scores.Interrater reliability for the Private Manufacturing Sector and range restriction for all sectors were recalculated for the 1998 State Quality Award examiner scores.The Private Manufacturing Sector's scores exhibited,low to moderate interrater reliability in both 1998 and 1999. Both the Private Service Sectors and the Public LocalSector's scores exhibited low interrater reliability in 1999, which is lower the the 1998 slow to moderate interrater reliability. The Public State and Federal Sector's scores exhibited low interrater reliability in both 1998 and 1999. Halo error was present for all examiners (both years) whose halo error could be measured by the determinant of the correlation coefficient matrix, Scores from most of the dimensions in each sector showed severity in the 1999 process, while scores from most of the dimensions in each sector showed leniency in the 1998 scoring process. Scores from only a few of the dimensions in each sector showed range restriction for both the 1998 and 1999 scoring processes.Due to the small sample size, caution should be taken in generalizing the results of this study. The small sample is a result of using field data from an actual Award rather than a controlled laboratory setting.