Inter-rater Reliability and Related Variables of a Newly Developed Measure of Quality of Student Participation

Some researchers suggest that quality is preserved when students are rewarded for quantity of class participation (Boniecki & Moore, 2003; Bruss, 2009; Mainkar, 2007; Zaremba & Dunn, 2004); however, few studies have targeted the systematic assessment of participation quality. The primary purpose of the study was to develop a reliable system for rating quality of student participation, investigate whether quality of participation is preserved when students are given credit for the amount of class participation, and examine the relationship between participation quality and important course variables.

The researcher in the current study developed a rating system to evaluate the quality of student participation in 2 small sections of an undergraduate class at a large Southeastern university. The primary observer rated the quality of each student comment and recorded the number of comments each student contributed each discussion day. In order to assess the reliability of the coding scheme, a secondary observer rated the participation quality and quantity on the third day in each unit. As outlined in the syllabus, instructors awarded credit for the amount of participation on randomly drawn days at the conclusion of select units.

The average inter-observer agreement was 90% for the number of productive comments contributed by each student and 49% for the number of non-productive comments contributed by each student. The percent of productive comments that each student contributed each day was the primary dependent variable. Visual inspection and proportion analyses of the percent of productive comments revealed that quality of participation was generally preserved during units in which credit was awarded for the frequency of participation. On average, students who participated frequently were significantly more productive than those who participated infrequently. Additionally, a student’s frequency of course participation and critical thinking at the onset of the course significantly predicted classification into high- and low-quality responders.

Future research suggestions include the following: expanding the definition of the quality of student participation into 3 overall qualitative categories rather than 2, increasing the criteria for participation credit in small course sections, and providing credit for the quality of participation.