Analyses of interrelationships among three scoring models, three performance evaluation scales, and the raters' supervisory styles
The purposes of this study were (1) to investigate and compare the applicability of three scoring models to three different types of performance evaluation scales and specific modifications thereof, and (2) to investigate the relationship of the raters' supervisory style to the performance evaluations made.
The scoring models included (1) the averaging model (Y = a + bX), (2) the psychophysical, power function (Log Y = Log k + n Log (E/I)), and (3) the psychophysical, power regression (Log Y = Log k + neLog E – niLog I).
The three performance evaluation scales were an anchored, a graphic, and a performance specimen checklist. Scale comparability was appraised through estimates of reliability and content similarity. After scoring the original scales by the three scoring models, the scales were modified to ascertain the effects on scale sufficiency and the ability of the scoring models to explain the stimulus-response relationship. The checklist scale modifications included (1) respondent-determined favorableness (extreme and moderate) of performance specimens and (2) length changes which shortened the scale, first, to 27 specimens, and then to 54 specimens prematched to the items in the graphic scale. All three original scales and the respondent-determined favorableness checklist scales were further modified by eliminating all items that failed to significantly discriminate with regard to the overall evaluation ratings.
Two dimensions of the raters' supervisory style--consideration and initiating structure--were compared to the ratings made on each evaluation scale item, and to the indices of unitary element overlap for the two ratees per rater. These comparisons were repeated for each original scale and analyzed separately for effective (positive) and ineffective (negative) items in the evaluation scales.
The power regression scoring model was superior in explaining the stimulus-response relationship in 14 of the 16 evaluation scale variations. The averaging model produced the highest sufficiency coefficients in the two anchored scales, but the lowest coefficients in the other 14 scale variations.
The anchored scales demonstrated the highest levels of sufficiency under all three scoring models. The checklist scale modifications of extreme-favorableness scoring and 27 matched specimens caused the greatest decreased in sufficiency coefficients. The moderate-favorableness scoring and 54 matched specimens caused the sufficiency coefficients to decrease with the averaging scoring model and to increase with the two psychophysical scoring models. The elimination of nondiscriminating items caused increases in sufficiency coefficients for all evaluation scale variations and with all three scoring models.
The mean score for the raters' supervisory style dimension of consideration was higher than the mean of the total group when above-average ratings or effective specimens were reported and tended to be lower when below-average ratings or ineffective specimens were reported. The initiating structure mean score tended to be lower for high ratings and effective-specimens but did not show a specific for low ratings and ineffective specimens. pattern
Thesis77b.E494.pdf
8.62 MB
Adobe PDF
8ba3293c0b88e2f335086866ec143dff