Development and Validation of the Statistics Assessment of Graduate Students

This study developed the Statistics Assessment of Graduate Students (SAGS) instrument, and established its preliminary item characteristics, reliability, and validity evidence. Even though there are limited number of assessments available for measuring different aspects of statistical cognition, these previously available assessments have numerous limitations. The SAGS instrument was developed using Rasch modeling approach to create a new measure of statistical research methodology knowledge of graduate students in education and other behavioral and social sciences. Thirty-five multiple-choice questions were written with stems representing applied research situations and response options distinguishing between appropriate use of various statistical tests or procedures. A focus group meeting with upper level graduate students was held in order to revise the initial instrument. Then, a six-person expert panel reviewed the revised items for content validity and to improve the quality of the instrument. The finalized SAGS instrument with 25 cognitive questions and demographic questionnaire was administered online, and 132 participants fully completed the instrument. Results showed that, one SAGS item was not consistent with the Rasch model. This item and distractors of two other items were flagged to be modified during future administrations. Reliability indices, separation indices, constructs maps, and known group comparisons provided the supportive evidence for reliability and validity. Preliminary simulation study conducted with higher order IRT models rejected three parameter logistic (3 PL) model and indicated no impact of guessing parameter when describing the observed data. A simulation study further provided positive evidence towards using ICOMP type model selection criteria that guard against correlations of parameter estimates when choosing the best model among a portfolio of IRT models. Sample independent parameter estimates obtained using Rasch and IRT approaches in this study open an avenue to develop customizable yet psychometrically sound statistical research methodology assessments.