A hybrid embedding approach for short-answer grading using BERT and universal sentence encoder

Abstract

Evaluating learner responses is vital in assessment, especially for free-text answers, which are most challenging due to their demand for reasoning and expression. These responses reflect the learner’s understanding of the subject matter. However, manual evaluation is time-consuming and also prone to human error, highlighting the need for automation. Automatic Short-Answer Grading (ASAG) is a rapidly advancing area within the domain of educational technology, focusing on automating the evaluation process of student responses. Although numerous approaches have been proposed in recent years, the challenge of identifying the most accurate and reliable method for short-answer grading still remains an open research question. This work investigates the use of advanced natural language processing techniques, specifically text embeddings generated by Bidirectional Encoder Representations from Transformers (BERT) and the Universal Sentence Encoder (USE), for automatic grading of short answers. Furthermore, the work explores the possibilities for identifying the dependency of the method on data and proposes a novel hybrid model that appropriately evaluates student responses. A sample of 500 student responses was extracted from the 1,672 set-1 records of the publicly available HP: Short Answer Scoring (SAS) dataset. By integrating BERT and USE models to measure semantic and contextual similarity against reference answers, the study automated the scoring process. This hybrid approach achieves an accuracy of 0.554 and a quadratic weighted Kappa score of 0.999. The high coefficient reflects categorical agreement success through threshold-based binning, rather than granular decimal precision. The results are also compared with existing benchmarks established using the HP: SAS dataset.