One Model to Score Them All: Unified Scoring of Learning Strategies with LLMs
Saved in:
| Title: | One Model to Score Them All: Unified Scoring of Learning Strategies with LLMs |
|---|---|
| Language: | English |
| Authors: | Andreea Dutulescu, Stefan Ruseti, Mihai Dascalu, Danielle McNamara |
| Source: | International Educational Data Mining Society. 2025. |
| Availability: | International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/ |
| Peer Reviewed: | Y |
| Page Count: | 7 |
| Publication Date: | 2025 |
| Sponsoring Agency: | Institute of Education Sciences (ED) |
| Contract Number: | R305T240035 |
| Document Type: | Speeches/Meeting Papers Reports - Research |
| Descriptors: | Scoring, Computational Linguistics, Computer Software, Artificial Intelligence, Generalization, Scoring Rubrics, Student Evaluation, Prediction, Evaluation Criteria, Learning Strategies, Feedback (Response), Evaluation Methods, Writing Evaluation |
| Abstract: | The assessment of student responses to learning-strategy prompts, such as self-explanation, summarization, and paraphrasing, is essential for evaluating cognitive engagement and comprehension. However, manual scoring is resource-intensive, limiting its scalability in educational settings. This study investigates the use of Large Language Models for automating the evaluation of student responses based on expert-defined rubrics. We fine-tune open-source LLMs on annotated datasets to predict expert ratings across multiple scoring rubrics, ensuring consistency and efficiency in assessment. Our findings indicate that multi-task fine-tuning, which involves training a single model across multiple scoring tasks, consistently outperforms single-task training by enhancing generalization and mitigating overfitting. This advantage is particularly noticeable in recent architectures, where multi-task training enables robust performance across diverse evaluation criteria. Notably, our Llama 3.2 3B model achieved high performance, outperforming a 20x larger zero-shot model while maintaining feasibility for deployment on consumer-grade hardware, emphasizing the potential for scalable AI-driven assessment solutions. This research contributes to open education by fine-tuning open-source models and publicly releasing trained models, training scripts, and evaluation frameworks. The proposed approach supports automated, reproducible, and scalable assessment of learning strategies, facilitating timely feedback for students and reducing the burden on educators. [For the complete proceedings, see ED675583.] |
| Abstractor: | As Provided |
| Notes: | https://github.com/upb-nlp/EDM-LLM-Scoring |
| IES Funded: | Yes |
| Entry Date: | 2025 |
| Accession Number: | ED675648 |
| Database: | ERIC |
Be the first to leave a comment!
