TY - JOUR
T1 - Before and during COVID-19
T2 - A Cohesion Network Analysis of students’ online participation in moodle courses
AU - Dascalu, Maria Dorinela
AU - Ruseti, Stefan
AU - Dascalu, Mihai
AU - McNamara, Danielle S.
AU - Carabas, Mihai
AU - Rebedea, Traian
AU - Trausan-Matu, Stefan
N1 - Funding Information:
Our overarching objective is to evaluate students' behaviors and interaction patterns and predict student grades based on their activity in online forum discussions and click-stream data. In ReaderBench, Cohesion Network Analysis (CNA; M. Dascalu, Trausan-Matu, McNamara, & Dessus, 2015) is used to assess semantic cohesion among students' posts. Weekly CNA sociograms are generated to examine how the interactions between peers and with tutors evolve from one week to the next (see e.g., Sirbu, Dascalu, Crossley, McNamara, & Trausan-Matu, 2019; Sirbu et al., 2018). These visualizations also provide insights into students' behaviors in association with course events, such as deadlines, assignments, tests, and exams. The visualizations are designed for teachers to follow the evolution of students in term of interactions, interactivity, and online participation, enabling them to intervene when they notice a decrease in participation or inactivity, thus increasing students' chances of passing or obtaining a better grade. Various sources of information from CNA and students’ behaviors (e.g., from clickstream and log data) are combined to predict student grades. We also provide a qualitative analysis of the CNA visualizations based on observations of one of the authors, who has over 25 years of experience in teaching the Algorithm Design course, combined with extensive experience in conducting research on topics related to Computer-Supported Collaborative Learning (CSCL) and Intelligent Tutoring Systems (ITS). In contrast to the previous studies performed by M.-D. Dascalu et al. (2020), M. Dascalu, McNamara, et al. (2018) and Crossley, Paquette, Dascalu, McNamara, and Baker (2016), the entire processing flow is integrated and performed in Python. In this current version, we introduce an integrated pipeline that accounts for all types of indices (CNA, time series, and textual complexity), and data derived from clickstream logs are also integrated in the final predictions; although used separately in previous analyses, participation and initiation indices are combined for the first time. Moreover, a neural network for predicting course grades was introduced, while the visualizations were updated.Within the next academic year (2019–2020), 117 students wrote posts on the Moodle platform, divided as in the previous year into three student-cohorts, guided by the same 3 lecturers and 15 teaching assistants. In total, 135 discussion threads and 535 contributions were generated. The partial grades with points gathered throughout the semester were taken into consideration for this experiment. Students achieved partial grades that ranged from 0 to 7.61 (M = 4.53, SD = 1.55) out of 6. Additional bonus points were also awarded and the final grades considered caped values (M = 4.62, SD = 1.42; see Fig. 1b). This normalization was performed because multiple bonus points (e.g., participation to contests, such as ACM International Collegiate Programming Contest) were awarded on different criteria in the two academic years, and a comparative scoring baseline was required to build transferable models across the two years. The mean and standard deviation values are comparable between the two years. The course lasted for 14 weeks (i.e., between February 17, 2020 and May 22, 2020) and it was also held in Romanian.The work was funded by a grant of the Romanian National Authority for Scientific Research and Innovation, CNCS – UEFISCDI, project number TE 70 PN-III-P1-1.1-TE-2019-2209, ATES – “Automated Text Evaluation and Simplification”. This research was also supported in part by the Institute of Education Sciences, U.S. Department of Education, through Grants R305A180261 and R305A180144 to Arizona State University and by the Office of Naval Research (Grants: N00014-17-1-2300 and N00014-19-1-2424). Opinions, conclusions, or recommendations do not necessarily reflect the view of the Institute or the U.S. Department of Education, or the Office of Naval Research.
Funding Information:
The work was funded by a grant of the Romanian National Authority for Scientific Research and Innovation, CNCS – UEFISCDI , project number TE 70 PN-III-P1-1.1-TE-2019-2209, ATES – “Automated Text Evaluation and Simplification” . This research was also supported in part by the Institute of Education Sciences, U.S. Department of Education , through Grants R305A180261 and R305A180144 to Arizona State University and by the Office of Naval Research (Grants: N00014-17-1-2300 and N00014-19-1-2424 ). Opinions, conclusions, or recommendations do not necessarily reflect the view of the Institute or the U.S. Department of Education, or the Office of Naval Research.
Publisher Copyright:
© 2021 Elsevier Ltd
PY - 2021/8
Y1 - 2021/8
N2 - The COVID-19 pandemic has changed the entire world, while the impact and usage of online learning environments has greatly increased. This paper presents a new version of the ReaderBench framework, grounded in Cohesion Network Analysis, which can be used to evaluate the online activity of students as a plug-in feature to Moodle. A Recurrent Neural Network with LSTM cells that combines global features, including participation and initiation indices, with a time series analysis on timeframes is used to predict student grades, while multiple sociograms are generated to observe interaction patterns. Students’ behaviors and interactions are compared before and during COVID-19 using two consecutive yearly instances of an undergraduate course in Algorithm Design, conducted in Romanian using Moodle. The COVID-19 outbreak generated an off-balance, a drastic increase in participation, followed by a decrease towards the end of the semester, compared to the academic year 2018–2019 when lower fluctuations in participation were observed. The prediction model for the 2018–2019 academic year obtained an R2 of 0.27, while the model for the second year obtained a better R2 of 0.34, a value arguably attributable to an increased volume of online activity. Moreover, the best model from the first academic year is partially generalizable to the second year, but explains a considerably lower variance (R2 = 0.13). In addition to the quantitative analysis, a qualitative analysis of changes in student behaviors using comparative sociograms further supported conclusions that there were drastic changes in student behaviors observed as a function of the COVID-19 pandemic.
AB - The COVID-19 pandemic has changed the entire world, while the impact and usage of online learning environments has greatly increased. This paper presents a new version of the ReaderBench framework, grounded in Cohesion Network Analysis, which can be used to evaluate the online activity of students as a plug-in feature to Moodle. A Recurrent Neural Network with LSTM cells that combines global features, including participation and initiation indices, with a time series analysis on timeframes is used to predict student grades, while multiple sociograms are generated to observe interaction patterns. Students’ behaviors and interactions are compared before and during COVID-19 using two consecutive yearly instances of an undergraduate course in Algorithm Design, conducted in Romanian using Moodle. The COVID-19 outbreak generated an off-balance, a drastic increase in participation, followed by a decrease towards the end of the semester, compared to the academic year 2018–2019 when lower fluctuations in participation were observed. The prediction model for the 2018–2019 academic year obtained an R2 of 0.27, while the model for the second year obtained a better R2 of 0.34, a value arguably attributable to an increased volume of online activity. Moreover, the best model from the first academic year is partially generalizable to the second year, but explains a considerably lower variance (R2 = 0.13). In addition to the quantitative analysis, a qualitative analysis of changes in student behaviors using comparative sociograms further supported conclusions that there were drastic changes in student behaviors observed as a function of the COVID-19 pandemic.
KW - Click-stream data
KW - Cohesion Network Analysis
KW - Learner interactions
KW - Learning patterns
KW - Moodle
KW - Sociograms
KW - Student behavior
UR - http://www.scopus.com/inward/record.url?scp=85104297662&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85104297662&partnerID=8YFLogxK
U2 - 10.1016/j.chb.2021.106780
DO - 10.1016/j.chb.2021.106780
M3 - Article
AN - SCOPUS:85104297662
SN - 0747-5632
VL - 121
JO - Computers in Human Behavior
JF - Computers in Human Behavior
M1 - 106780
ER -