Name: Scaling OER Peer Review with Artificial Intelligence: A MERLOT Pilot Study
Start: 2026-10-09T11:05:00-0400
End: 2026-10-09T11:35:00-0400

All sessions are available online except round tables, special activities, and workshops.

Scaling OER Peer Review with Artificial Intelligence: A MERLOT Pilot Study

Friday October 9, 2026 11:05am - 11:35am EDT

7 DR5

ID: 33987

Open educational resources (OER) are expanding rapidly across disciplines and repositories, yet the peer review systems designed to evaluate them have not scaled at the same pace. As a result, many high-quality OER remain unreviewed, limiting their discoverability, credibility, and adoption. Because peer review typically relies on volunteer disciplinary experts, communities such as the Biology Editorial Board of MERLOT face persistent backlogs of materials awaiting evaluation. In addition, OER are sometimes perceived as lower quality simply because they are inexpensive and relatively easy to produce, despite the rigor of many existing resources. This project explores whether artificial intelligence (AI) can meaningfully support OER peer review workflows while maintaining the rigor and transparency expected in scholarly evaluation. In partnership with industry collaborators, we are piloting an AI-assisted review system that applies the MERLOT Peer Review rubric to OER. The system uses structured prompts to guide AI in generating rubric-aligned draft reviews addressing key evaluation dimensions: quality of content, potential effectiveness as a teaching tool, ease of use, and accessibility.Importantly, the goal of this work is not to replace expert reviewers but to investigate how AI might augment human review processes. The AI generates structured preliminary evaluations that can assist with summarizing materials, rubric alignment, and draft review generation. Human reviewers then assess the AI-generated reviews using the same rubric criteria to determine whether the AI evaluation is coherent, accurate, and useful for disciplinary review boards. A composite review containing both AI and human review would be submitted as the final review. The study design compares AI-generated reviews with expert human peer reviews across a sample of OER drawn from established repositories such as MERLOT, OpenStax, and the Open Textbook Library. Pilot testing begins with a small set of materials to refine workflows and prompt design, followed by a larger evaluation set allowing comparison of scoring alignment between AI and expert reviewers. Key metrics include agreement between AI and expert ratings across rubric dimensions, reproducibility of AI scores across repeated evaluations, and rubric-based assessments of the clarity and completeness of AI-generated reviews.Additional system capabilities include automated citation verification through open databases such as PubMed and the Directory of Open Access Journals, link validation to identify outdated or broken resources, and analysis of visual elements. These tools allow AI to assist with time-consuming review tasks while preserving the need for disciplinary expertise in evaluating scientific accuracy and pedagogical appropriateness.This presentation will describe the design of the AI-assisted review workflow, the process of translating a human peer review rubric into structured AI prompts, and preliminary findings from early pilot testing. We will also discuss limitations and ethical considerations, including where AI evaluation is reliable, where it requires human oversight, and how AI-supported review might responsibly scale peer review capacity within open education ecosystems.By examining how AI can support, but not replace, expert peer review, this work contributes to broader conversations about the future of open knowledge infrastructures and the responsible integration of emerging technologies into open education systems.

Speakers

Michael Plotkin

Associate Professor, Department Chair. Co-Editor MERLOT Biology Editorial Board, Mt. San Jacinto College

Michael Plotkin is associate professor and department chair of biological sciences at Mt. San Jacinto College in California. He is an active member of the college’s honors enrichment program and has held roles in several OER initiatives, including serving as a reviewer for the California... Read More →

Medora Huseby

Associate Professor, Colorado State University

Medora Huseby is a member of the teaching faculty in the Department of Microbiology, Immunology, and Pathology at Colorado State University, where she focuses on open educational practices and student engagement in open education. She chairs the Open Educational Resources (OER) Committee... Read More →

Friday October 9, 2026 11:05am - 11:35am EDT
7 DR5 MIT Samberg Conference Center, 50 Memorial Drive, Cambridge MA 02139 USA

Presentation

Speaker Modality Hybrid
Track Exploring Emergent Technologies and the Future of Openness

Tickets

Success you have shared X tickets.

OEGlobal 2026

Michael Plotkin

Medora Huseby

Get help with the event

Success you have shared X tickets.

OEGlobal 2026

Michael Plotkin

Medora Huseby

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Get help with the event