Fairness Audits of Institutional Risk Models in Deployed ML Pipelines
Source: arXiv:2604.19468v1
Type: Cross
Abstract
Fairness audits of institutional risk models are critical for understanding how deployed machine learning pipelines allocate resources. Drawing on a multi-year collaboration with Centennial College, where our prior ethnographic work introduced the ASP-HEI Cycle, we present a replica-based audit of a deployed Early Warning System (EWS), replicating its model using institutional training data and design specifications.
Key Findings
Our audit evaluates disparities by gender, age, and residency status across the full pipeline, including training data, model predictions, and post-processing, utilizing standard fairness metrics. The results reveal significant systematic misallocation of resources:
- Younger, male, and international students: These groups are disproportionately flagged for support, even when many of them ultimately succeed in their studies.
- Older and female students: They face under-identification despite having comparable dropout risk, which raises concerns about equity in the support provided.
Impact of Post-Processing
The analysis also highlights how post-processing techniques amplify these disparities. The method of collapsing heterogeneous probabilities into percentile-based risk tiers contributes to the inequity, as it obscures the nuances of individual risk profiles. This misallocation not only affects the students’ access to necessary support but also perpetuates existing inequalities within the educational system.
Methodology
This work provides a replicable methodology for auditing institutional machine learning systems. By applying the ASP-HEI Cycle, we are able to assess the impact of algorithmic decisions on student outcomes, enabling institutions to better understand their operational dynamics. Our approach demonstrates the importance of evaluating construct validity alongside statistical fairness, ensuring that the models used reflect the realities of the student population.
Broader Implications
This audit contributes to a larger discourse on algorithms, student data, and power dynamics in higher education. It emphasizes the need for maintaining fairness in machine learning applications, especially in sensitive areas such as educational support systems. By shedding light on the disparities that emerge and compound across various stages of the pipeline, we aim to foster a more equitable approach to resource allocation in educational institutions.
Conclusion
The findings of this fairness audit serve as a critical reminder of the complexities involved in deploying machine learning technologies in social contexts. Institutions must be vigilant in assessing the implications of their models and ensure that their applications do not inadvertently perpetuate biases. Through continued research and collaboration, we can work towards developing systems that advocate for fairness and equity in higher education.