TY - GEN
T1 - Assessing Student Programming Process Using Automated Reasoning
AU - Acuña, Ruben
AU - Bansal, Ajay
N1 - Publisher Copyright:
© 2023 IEEE.
PY - 2023
Y1 - 2023
N2 - This Innovative Practice Full Paper presents an automated reasoning approach to the assessment of a student's programming process as captured by an autograder. When students are assessed in computing courses, they are typically assessed on their completed work, such as a programming assignment. Although such an assessment measures a student's general ability, it provides only an indirect measure of the effectiveness of their programming (or problem-solving) process. Without the ability to examine the process by which a student completes an assignment, an instructor may have difficulty teaching problem-solving skills since they cannot measure how the student's skill develops. Some computing courses make use of an autograder which automatically assesses student homework. The trace captured by an autograder is a unique source of information about a student's problem-solving ability, which can be analyzed to measure properties of that learner's process. This provides a way to assess the process-based aspect of student learning, enabling instructors to not only teach problem-solving skills but to determine how well students acquire these skills. This enables teachers to evaluate the methodology they employ and to select instructional material based on student need. Automatically analyzing the trace captured by an autograder has two major challenges: the scale of the data makes traditional approaches computationally expensive, and the definition of a problem-solving process is often expressed in a commonsense way based on an expert's intuition. These challenges can be addressed by using an Answer Set Programming (ASP) approach. ASP is a declarative language that is intended for solving computationally hard problems, and through various extensions, supports human-style commonsense reasoning. Assessing a student is computationally hard due to the combinatorial nature of the different ways an assignment may be solved. In this paper, we discuss the design of an ASP-based system for automatically assessing a student's problem-solving process using the trace captured by an autograder, and present a preliminary evaluation of the system applied within a university-level course on data structures & algorithms.
AB - This Innovative Practice Full Paper presents an automated reasoning approach to the assessment of a student's programming process as captured by an autograder. When students are assessed in computing courses, they are typically assessed on their completed work, such as a programming assignment. Although such an assessment measures a student's general ability, it provides only an indirect measure of the effectiveness of their programming (or problem-solving) process. Without the ability to examine the process by which a student completes an assignment, an instructor may have difficulty teaching problem-solving skills since they cannot measure how the student's skill develops. Some computing courses make use of an autograder which automatically assesses student homework. The trace captured by an autograder is a unique source of information about a student's problem-solving ability, which can be analyzed to measure properties of that learner's process. This provides a way to assess the process-based aspect of student learning, enabling instructors to not only teach problem-solving skills but to determine how well students acquire these skills. This enables teachers to evaluate the methodology they employ and to select instructional material based on student need. Automatically analyzing the trace captured by an autograder has two major challenges: the scale of the data makes traditional approaches computationally expensive, and the definition of a problem-solving process is often expressed in a commonsense way based on an expert's intuition. These challenges can be addressed by using an Answer Set Programming (ASP) approach. ASP is a declarative language that is intended for solving computationally hard problems, and through various extensions, supports human-style commonsense reasoning. Assessing a student is computationally hard due to the combinatorial nature of the different ways an assignment may be solved. In this paper, we discuss the design of an ASP-based system for automatically assessing a student's problem-solving process using the trace captured by an autograder, and present a preliminary evaluation of the system applied within a university-level course on data structures & algorithms.
KW - answer set programming
KW - automated assessment
KW - computing education
KW - process assessment
UR - http://www.scopus.com/inward/record.url?scp=85182978293&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85182978293&partnerID=8YFLogxK
U2 - 10.1109/FIE58773.2023.10342936
DO - 10.1109/FIE58773.2023.10342936
M3 - Conference contribution
AN - SCOPUS:85182978293
T3 - Proceedings - Frontiers in Education Conference, FIE
BT - 2023 IEEE Frontiers in Education Conference, FIE 2023 - Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 53rd IEEE ASEE Frontiers in Education International Conference, FIE 2023
Y2 - 18 October 2023 through 21 October 2023
ER -