Autograder Impact on Software Design Outcomes

This Innovative Practice Full Paper presents an analysis of autograder usage on software engineering design outcomes. Autograders have found increasing use in software engineering and computer science courses due to a variety of reasons. They are often introduced to support large class sizes, increase the reproducibility of grades, and provide formative feedback to students. Many of the classes that apply autograders are introductory in nature, such as CS1 and CS2 courses, or an introduction to algorithms course. These courses typically focus on assessing student programming ability on specific aspects of a programming language (e.g., Java inheritance) or algorithms (e.g., sorting, searching). One common drawback to automated assessment with programming is the need to impose additional structure on assignments. For example, students asked to implement a game might be given a detailed list of functions to implement, skipping analysis and design to determine what functions and algorithms are relevant. In this work, we discuss the experience of introducing an autograder into a second-year course that teaches software engineering topics such as personal software process, object-oriented design, and UML. Although we have the traditional motivations to use an autograder, it is challenging in a software engineering course since the additional structure required by an autograder is potentially detrimental to the engineering design outcomes. To address this, we developed an autograder in such a way as to minimize the design choices that it imposed on the students. The partial structure that is imposed together with the formative feedback produced by the autograder helps to scaffold student learning as they transition to more complex programs. The autograder was used during a spring 2023 offering of our software engineering course. We analyzed the data gathered in this course to evaluate the course change by asking: how does introducing an autograder impact course design outcomes? And, how does introducing an autograder impact personal process data that is collected? Answering these questions helps to inform the software engineering community about the suitability of introducing automated assessment tools into advanced computing and software engineering courses.