30 Million Canvas Grading Records Reveal Widespread Sequential Bias and System-Induced Surname Initial Disparity
In "Seminars and talks"

Speakers

Helen Wang
Helen Wang

University of Michigan

Helen Wang is a final-year PhD candidate at Ross School of Business, University of Michigan, where she is advised by Prof. Jun Li and Prof. Damian Beil. Her research is dedicated to advancing equitable opportunity for the younger generation and underserved populations, with a focus on uncovering and addressing disparities that have long been overlooked by industry practitioners and policymakers in contexts such as Education Technology (EdTech), AI of Education, and childcare. She employs a variety of empirical and experimental methods such as econometrics, field experiments, and natural language processing. Her research has been published in MSOM and acknowledged by multiple best paper competitions of flagship academic conferences such as INFORMS, POMS, and ACM EAAMO. Before she came to UM, she earned her Bachelor’s degree in Economics with a minor in Data Science from Fudan University, China.


Date:
Friday, 6 December 2024
Time:
10:00 am - 11:30 am
Venue:
NUS Business School
Mochtar Riady Building BIZ1-0302
15 Kent Ridge Drive
Singapore 119245 (Map)

Abstract

The widespread adoption of learning management systems in educational institutions has yielded numerous benefits for teaching staff but also introduced the risk of unequal treatment towards students. We present an analysis of over 30 million Canvas grading records from a large public university, revealing a significant bias in sequential grading tasks. We find that assignments graded later in the sequence tend to (1) receive lower grades, (2) receive comments that are notably more negative and less polite, and (3) exhibit lower grading quality measured by post-grade complaints from students.

Furthermore, we show that the system design of Canvas, which pre-orders submissions by student surnames, transforms the sequential bias into a significant disadvantage for students with alphabetically lower-ranked surname initials. These students consistently receive lower grades, more negative and impolite comments, and raise more post-grade complaints as a result of their disadvantaged position in the grading sequence. This surname initial disparity is observed across a wide range of subjects, and is more prominent in social science and humanities as compared to engineering, science and medicine. The assignment-level surname disparity aggregates to a course-level surname disparity of students’ GPA and can potentially lead to inequitable job opportunities. For platforms and education institutions, the system-induced surname grading disparity can be mitigated by randomizing student submissions in grading tasks. Education institutions should keep the workload of graders at a reasonable level to reduce fatigue and/or have multiple graders as a cross validation to enhance grading quality.