College Environment Behavioral and Perceptions Survey (CEBPS): Using In-Depth Interviews as a Validation Method for Survey Development

Background: Although there are some validation procedures based in qualitative approaches (e.g. asking what someone thinks about the structure of a survey question during face, content validation, or cognitive interviews), there are no approaches to validate a subject’s responses by a “gold standard assessment”. The purpose of this study was to use a mix-methodology approach to validate subjects’ perceptions, which used indepth interviews as a criterion-based validation method.

Methods: 30 participants (15 male and 15 female) completed the CEBPS survey and an indepth interview with questions mirroring those in the survey. A priori coding was used to develop the same codebook for the surveys and interviews. Each question (n=27) had three corresponding codes (first code= agree; second code=disagree; third code=neither), with a total of 81 possible codes. Surveys and interviews were checked to see if codes were used in both (recoded as 1), interview only (recoded as 2), or survey only (recoded as 3). Recoding was done by adding the recoding from the original three codes for each question together (total of 1=agree, >1=disagree). Frequencies were calculated to determine agreement rate between surveys and interviews.

Results: Out of 27 total questions, 21 were found to have >80% agreement in responses in both the survey and the interviews with an average of 92% agreement. For the remaining 6 questions, an average overall accuracy of 71% was calculated, with responses found to be missing more often in the interviews than the surveys (25% vs 11%). An average of 7% of participants agreed with the statement in the survey, whereas an average of 15% agreed in the interview on the same question.

Conclusions and Implications: Participants were more likely to agree in interviews than surveys, which could be due to social desirability. Participants may have had a desire to seem more healthful with researcher interviewing them. Results showed a weakness in survey only approach in that participants were more likely to choose “neither agree nor disagree” option on the survey than in the interview. This could be due to the participant taking time to think about the answer during the interview.