Doctoral Dissertations

Larry N. Long

Date of Award

5-1990

Degree Type

Dissertation

Degree Name

Doctor of Philosophy

Major

Psychology

Major Professor

William H. Calhoun

Committee Members

Stephen J. Handel, Siegfried C. Dietz, John W. Lounsbury

Abstract

The most widely accepted model for evaluating the outcome of training programs is Kirkpatrick's four level model. The four levels in the model are measures of trainee reactions to the training, measures of the amount of trainee learning which occurred in the training, measures of trainee on-the-job behavior which improved after the training, and measures of organization results which improved after the training. According to Kirkpatrick's model the four levels are arranged in a hierarchy such that results level data are proposed to be the most useful to the organization followed in order of decreasing usefulness by behavior level data, learning level data, and, finally, reactions level data which is proposed to be the least useful of the four levels to the organization. In the thirty years since Kirkpatrick first proposed his model only one study was located which used data at each of the four levels to test the hierarchical arrangement of the levels. The results of this one study provided only partial support for the model. The present study tested the hierarchical arrangement of Kirkpatrick's four levels by using data collected at all four levels during an actual program evaluation of a project management training course in a large organization. The training program was designed to measure trainee reactions, trainee learning, trainee behavior, and organization results. The present study tested the four levels of evaluation data in terms of their reported utility for decisionmaking. The basis for the design of the current study was a literative review which revealed that the purpose of evaluation research is to provide information for decisionmaking, and that the results of evaluation research are not generally used. Participants in this investigation were 152 management employees in three training organizations of a large utility. Participants were randomly assigned to five groups. Participants in each of the first four groups were asked to examine only one level of data, either reactions, learning, behavior or results level data. These respondents then indicated their willingness to make decisions about the training, based on the data they examined, by completing a questionnaire containing 18 Likert-type response items. The fifth group of participants was asked to examine data at all four levels, and to then rank order the levels of data in terms of importance to decisionmaking about the training by completing a second questionnaire containing six items. Each questionnaire also contained items to collect demographic data, and items designed to assess the validity of the labeling of the levels for each data set examined. Data analyses revealed that overall the five groups did not differ significantly in terms of demographic makeup. Further analyses of the validity check items revealed that all the data sets appeared to be validly labeled with the one exception of the reactions level data set used in conjunction with the first questionnaire which contained the Likert-type response items. Parametric and nonparametric analyses of variance were initially used to analyze the data. These analyses revealed one significant result in each questionnaire. Further post-hoc analyses on the two items showing significant results were conducted. The results of these post-hoc analyses supported the hierarchical arrangement of the levels in Kirkpatrick's model; namely that results level data had the greatest reported utility for decisionmaking followed in order of decreasing utility by behavior level data, learning level data, and, finally, reactions level data which had the least reported utility for decisionmaking. This outcome was much stronger for the data from the second questionnaire which involved the examination and ranking of all four of the evaluation data levels. A major unanticipated result of the study was the finding that when all responses were combined within each questionnaire, the behavior level evaluation data had the greatest reported utility for decisionmaking. This outcome was the same for both questionnaires. The practical implication of these findings is that training program evaluations should be designed to include, as a minimum, the measurement of on-the-job behavior of the trainees. This level of evaluation data has been shown to possess the greatest utility to the organization in terms of subsequent decisionmaking about the value of the training program. The study concludes with a detailed discussion of the potential methodological and conceptual limitations of the research design. Suggestions for future research are also discussed.

Download
Files over 3MB may be slow to open. For best results, right-click and select "save as..."

Share

COinS