The equivalence of job analysis agents : a methodological reappraisal

The purpose of this research was to conceptually and empirically investigate the equivalence of ratings provided by different types of job analysis agents. Specifically, ratings of the job of power-plant operator given by job incumbents, their supervisors, students with access to the job description, and students who were only told the job title were examined for equivalence.

Several existing methods for assessing similarity of ratings were reviewed before the empirical investigation was undertaken. Because the correlational approaches appeared deficient for this purpose, a definition of rater equivalence was developed which involves the use of distance measurements in multivariate space. By this definition, the ratings from different types of raters are considered equivalent if and only if the centroids and the dispersions of their multivariate distributions are the same. Equivalence of centroids assures that significant differences do not exist between types of raters in their mean ratings of a job on different dimensions. Equivalence of dispersions (measured by median squared distance from the group centroid) guarantees that interrater agreements (within groups of different types of analysts) do not differ significantly.

Four types of raters completed the Position Analysis Questionnaire (PAQ). One power-plant operator and his or her supervisor from each of 94 plants in 30 companies across the country rated the incumbent's job. Undergraduate management students also completed the PAQ for the job of

vi power-plant operator. Half were provided with a description of the job (n = 94) and half were told only the job title (n = 94). The dependent variables were the l3 overall factor scores obtained from the individuals' responses to the PAQ items.

Both the correlational approach for assessing similarity and the set of multivariate techniques developed from the definition of rater equivalence were applied to the same set of sample data. . Significant correlations among different types of raters over the 13 PAQ dimensions suggested that the source of job analysis information makes little difference. However, an examination of the equivalence of dispersions and the equivalence of centroids produced very different results than that provided by the correlational analyses concerning the equivalence of different job analysis agents. Significantly more agreement (measured by means of dispersion) existed among incumbents than among either group of students. Supervisors displayed greater interrater agreement than did students given only the job title. Moreover, the distance between each pair of group centroids was statistically significant (measured by means of multivariate F ratio). No two groups of analysts met the requirements for their ratings to be considered equivalent.

The findings have implications for procedures used in collecting job analysis information and for methods used to assess the equivalence of raters. The sensitivity of the PAQ was demonstrated by the very large differences in ratings between those provided by students and those given by individuals more familiar with the job (viz., incumbents and their supervisors). Also, the findings suggested that practitioners should expect modest differences in job analysis information depending on whether the information is provided by incumbents or by their supervisors. , Perhaps most importantly, the results further demonstrated that the choice of methods employed to determine the equivalence or similarity of different types of raters can substantially affect a researcher's conclusions. The definition of equivalence which was developed and applied to actual data is offered as an improvement over the more traditional correlational approaches.