Reliability of a performance evaluation process and the effect of confidence ratings on reliability

The purpose of this study was to examine the reliability of a performance evaluation form in an on-the-job setting. Also this study was used to determine if the use of a confidence rating scale, in addition to the performance rating scale, could be used as an indicator of the opportunity of the rater to observe performance. A 54 item rating form using the summated rating format was developed through a series of interactions between a professional consultant and the users of the form in a manner similar to the retranslation method of scale development. The rating form was used to rate the performance of 143 job incumbents in the circulation department of a newspaper. Two sets of ratings were obtained, one by the first line supervisor and a second by the department head.

Interrater reliability coefficients were computed for items separately, for subgroups of items, for "Overall" subgroup items taken together, and for the total overall form. In all but a few exceptions the reliability coefficients were higher than expected in the light of other field studies, especially those studies using two different levels of management as in this study. The total overall interrater reliability coefficient for the performance rating was .65. Alpha coefficients were also computed for the subgroups of items and total overall form for two levels of raters. These were very high indicating a high internal consistency for the rating form.

To examine the use of the confidence rating the interrater reliability coefficients for items with high rater confidence scores were compared with the reliability coefficients for items with no restriction on the confidence rating. This was also repeated where the interrater reliability coefficients for items with a low department head confidence rating were used instead of a high confidence rating. in each case confidence rating had no relationship to higher or lower reliability coefficients respectively.

The high interrater reliability coefficients in this study would indicate that a relatively high degree of reliability can be obtained in an on-the-job setting if reasonable care is taken with the development of a performance rating form and in training and instructing the raters who use it. The results of this study indicated that the confidence ratings had no apparent relationship to reliability indices in this case and are of little value.