Different behavior versus differential opportunity to observe : factors associated with inter-rater agreement within a 360-degree appraisal process

The explanations for performance appraisal rating disagreement between supervisors and subordinates center on one of the following four perspectives; raters have different opportunities to observe performance, raters have different expectations, raters use different scales, and raters have different constructs of performance appraisal dimensions. An alternative, and much simpler, explanation for rating disagreement is that the ratee behaves differently when interacting with different raters; therefore, the raters rate the ratee differently. The purpose of this study was to explore and better understand rating differences in a 360-degree performance appraisal context. Specifically, this study examined rating differences between supervisors and subordinates as well as among subordinates to identify potential explanations for rating differences. Using actual 360-degree performance appraisal data, influence tactic ratings, Leader-Member Exchange (LMX) ratings, and self-monitoring scores, this study examined whether the relationship (i.e., LMX) between the rater and ratee might be related to rating differences and rater agreement. Self-monitoring was proposed as an individual difference variable that could be related to rater agreement. Three-hundred sixty-two managers, 40 senior managers, and 1086 subordinates participated in the study. Data was collected through surveys sent to each of the three sources approximately one month prior to completing the company's annual 360-degree performance appraisal process. The surveys included measures of influence tactics, Leader-Member Exchange, and self-monitoring. Results provided some support for the study's hypotheses. In-group subordinates did rate their manager differently than out-group subordinates on both influence tactic ratings and on performance appraisal ratings. Further, LMX similarity moderated rater agreement, measured through rwg, both between senior managers and subordinates as well as among subordinate raters. The results indicate that those with similar relations with the target rated the target similarly. None of the relationships hypothesized for self-monitoring received support. The findings pertaining to LMX rating differences and agreement moderation suggest that rater disagreement may be a function of ratee behavior as opposed to error or misperception as hypothesized in much of the performance appraisal literature. Direction for future research, study limitations and implications of these findings are discussed.