TY - GEN
T1 - Using Large Language Models to Provide Formative Feedback in Intelligent Textbooks
AU - Morris, Wesley
AU - Crossley, Scott
AU - Holmes, Langdon
AU - Ou, Chaohua
AU - McNamara, Danielle
AU - Dascalu, Mihai
N1 - Publisher Copyright:
© 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.
PY - 2023
Y1 - 2023
N2 - As intelligent textbooks become more ubiquitous in classrooms and educational settings, the need arises to automatically provide formative feedback to written responses provided by students in response to readings. This study develops models to automatically provide feedback to student summaries written at the end of intelligent textbook sections. The study builds on Botarleanu et al. (2022), who used the Longformer Large Language Model, a transformer Neural Network, to build a summary grading model. Their model explains around 55% of holistic summary score variance when compared to scores assigned by human raters on an analytic rubric. This study uses a principal component analysis to distill scores from the analytic rubric into two principal components – content and wording. When training the models on the summaries and the sources using these principal components, we explained 79% and 66% of the score variance for content and wording, respectively. The developed models are freely available on HuggingFace and will allow formative feedback to users of intelligent textbooks to assess reading comprehension through summarization in real-time. The models can also be used for other summarization applications in learning systems.
AB - As intelligent textbooks become more ubiquitous in classrooms and educational settings, the need arises to automatically provide formative feedback to written responses provided by students in response to readings. This study develops models to automatically provide feedback to student summaries written at the end of intelligent textbook sections. The study builds on Botarleanu et al. (2022), who used the Longformer Large Language Model, a transformer Neural Network, to build a summary grading model. Their model explains around 55% of holistic summary score variance when compared to scores assigned by human raters on an analytic rubric. This study uses a principal component analysis to distill scores from the analytic rubric into two principal components – content and wording. When training the models on the summaries and the sources using these principal components, we explained 79% and 66% of the score variance for content and wording, respectively. The developed models are freely available on HuggingFace and will allow formative feedback to users of intelligent textbooks to assess reading comprehension through summarization in real-time. The models can also be used for other summarization applications in learning systems.
KW - automated summary scoring
KW - intelligent textbooks
KW - large language models
KW - transformers
UR - http://www.scopus.com/inward/record.url?scp=85164905321&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85164905321&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-36336-8_75
DO - 10.1007/978-3-031-36336-8_75
M3 - Conference contribution
AN - SCOPUS:85164905321
SN - 9783031363351
T3 - Communications in Computer and Information Science
SP - 484
EP - 489
BT - Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings
A2 - Wang, Ning
A2 - Rebolledo-Mendez, Genaro
A2 - Dimitrova, Vania
A2 - Matsuda, Noboru
A2 - Santos, Olga C.
PB - Springer Science and Business Media Deutschland GmbH
T2 - 24th International Conference on Artificial Intelligence in Education , AIED 2023
Y2 - 3 July 2023 through 7 July 2023
ER -