The development and validation of an instrument for evaluating consumer and homemaking programs

1. How well does the instrument accurately and comprehensively measure the valid criteria established by Buero and Harrison?

2. Does the evaluation instrument elicit similar responses among different population groups (the teacher of a consumer and homemaking program, a home economics teacher educator, a regional supervisor of home economics, and another consumer and homemaking teacher)?

An evaluation instrument consisting of 25 categories with 145 criterion statements was constructed based on the criteria verified as essential for evaluating consumer and homemaking programs by Buero and Harrison. Each criterion statement was to be rated on a scale of 5-Superior, 4-Very Good, 3-Good, 2-Fair, and 1-Weak. An "Unable to Judge" rating was also included for situations in which not enough information was available or the criterion statement was not applicable to the program being evaluated. The instrument was checked for content coverage and clarity by a panel of experts. Vocational-Technical Education graduate students in a seminar at The University of Tennessee, Knoxville, read the instrument and made suggestions to improve its clarity and usefulness.

A pilot study was conducted to determine the usability and reliability of the evaluation instrument through the test-retest method. The evaluation instrument was administered twice to ten consumer and homemaking teachers in East Tennessee with a two- to three-week interval between each administration. Total scores for the tests and total scores for the retests were compared using Spearman Rho Rank Order Correlation.

For 29 programs in five southeastern states, the teacher of the consumer and homemaking program, another consumer and homemaking teacher, the home economics regional supervisor, and a home economics teacher educator were asked to rate the consumer and homemaking program using the evaluation instrument. The ratings for each criterion statement for each consumer and homemaking program from the four raters were compared using Friedman's Analysis of Variance. Programs in which the raters did not show a significant difference were classified as showing agreement among the raters. A binomial test was used to determine if the number of programs showing agreement among the raters was significant.

Selected findings of the study were as follows:

1. The panel of experts concurred that the evaluation instrument accurately and comprehensively measured the Harrison and Buero valid criteria for evaluating consumer and homemaking programs.

2. The Spearman Rho Rank Order Correlation for the test-retest reliability check for the ten programs included in the pilot study was 0.9030. This correlation was significant at the .05 level.

3. Of the 145 criterion statements used in the evaluation instrument, 110 criterion statements had z values for the number of programs showing agreement among the evaluators that were significant at the .05 level. Thirty criterion statements had more than 50 per cent of the programs showing agreement among the evaluators. Only five of the 145 criterion statements had less than 50 per cent of the programs showing agreement among the evaluators.

Based on the findings of this study, the following conclusion was made:

The evaluation instrument developed in this study is basically a valid and reliable instrument for evaluating consumer and homemaking programs.

The primary recommendation derived from this study is that the evaluation instrument should be considered for use by states in evaluating their consumer and homemaking programs through self-evaluation by the teachers, and on-site evaluation by other consumer and homemaking teachers, home economics teacher educators, and home economics regional supervisors.