Man vs. Machine: Evaluating Open Access LIS Journals

In the fall of 2025, five librarians at the University of Tennessee Knoxville Libraries gathered to conduct research on how ChatGPT can be used to evaluate open access journals. 15 diamond open access Library and Information Science journals were chosen using filters from the Directory of Open Access Journals. We developed a set of journal quality criteria based on the OASPA’s “Principles of Transparency and Best Practice in Scholarly Publishing.” We then created two main rubrics: one rubric for the human researchers and ChatGPT to evaluate journal quality based on the established set of criteria, and one rubric for the human researchers to measure how ChatGPT performed in its evaluation. A poster depicting our study results was accepted to the 2025 Charleston Conference.