Auditing Gender and Geographic Disparities in Machine Learning Models of KIP Kuliah Award Decisions: A Two-Cohort Temporal Evaluation

Muhammad Thoriq; Sakinah Ali Pitchay; Farida Ridzuan

doi:10.33102/mjosht.597

Authors

Muhammad Thoriq Department of Informatics, Universitas Adzkia, Padang 25175, West Sumatra, Indonesia.
Sakinah Ali Pitchay Cybersecurity and Systems Research Unit, Universiti Sains Islam Malaysia, Bandar Baru Nilai, 71800 Nilai, Negeri Sembilan, Malaysia.
Farida Ridzuan Faculty of Science and Technology, Universiti Sains Islam Malaysia, Bandar Baru Nilai, 71800 Nilai, Negeri Sembilan, Malaysia.

DOI:

https://doi.org/10.33102/mjosht.597

Keywords:

Machine Learning, algorithmic fairnes, decision support, educational financial assistance, temporal validation, KIP Kuliah

Abstract

Universities increasingly consider machine learning (ML) to help screen applications for need-based scholarships. Such models learn from past award decisions and may carry forward the group differences they contain, yet they are usually tested on the same intake used to build them. In practice, a model built on one year is applied to the next. This study examines whether a fairness audit of one intake still holds for the following intake. All applications for the Kartu Indonesia Pintar Kuliah (KIP Kuliah) scholarship at a private university in Indonesia in 2023 (n = 708) and 2024 (n = 829) were analysed. Logistic regression, random forest, and XGBoost models were trained on the 2023 cohort and tested once on the 2024 cohort; gender and geographic origin were withheld from the models and used only to compare their selections with past decisions. The models predicted past decisions only slightly better than chance, and their apparent loss of accuracy in 2024 was largely due to a lower acceptance rate rather than to poorer applicant rankings. No consistent gender gap was found. When the models filled a fixed quota, applicants from municipal (city) schools were selected 19–23 percentage points more often than those from regency schools, about twice the gap in the 2024 decisions themselves, because parental occupation acted as an indirect signal of school location. This gap depended on the selection rule and did not hold under every setting. The proposed contribution is a two-intake audit protocol that separates disparities inherited from past decisions from those added by a model. Institutions using such models should re-audit them before every intake.

Downloads

Download data is not yet available.

References

[1] World Bank, The Promise of Education in Indonesia. Washington, DC, USA: World Bank, 2020, doi: 10.1596/34807.

[2] Pusat Layanan Pembiayaan Pendidikan, Pedoman Pendaftaran Kartu Indonesia Pintar Kuliah (KIP Kuliah) Merdeka Tahun 2023, ver. 3.0. Jakarta, Indonesia: Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi, 2023. [Online]. Available: https://kip-kuliah.kemdiktisaintek.go.id/uploads/Pedoman-Pendaftaran-KIP-Kuliah-2023-update-15Mei2023-_e79a89.pdf

[3] Pusat Layanan Pembiayaan Pendidikan, Pedoman Pendaftaran KIP Kuliah Merdeka Tahun 2024, ver. 1.0. Jakarta, Indonesia: Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi, 2024. [Online]. Available: https://kip-kuliah.kemdikbud.go.id/uploads/20240207-Pedoman-Pendaftaran-KIP-Kuliah-2024_dfb2c1.pdf

[4] V. Alatas, A. Banerjee, R. Hanna, B. A. Olken, R. Purnamasari, and M. Wai-Poi, “Self-targeting: Evidence from a field experiment in Indonesia,” J. Polit. Econ., vol. 124, no. 2, pp. 371–427, 2016, doi: 10.1086/685299.

[5] P. Ninghardjanti, W. Murtini, A. Hindrayani, and K. B. Sangka, “Evaluation of the Smart Indonesia Card (KIP) program for the vocational schools in Surakarta City,” in Proc. 1st UPY Int. Conf. Educ. Soc. Sci. (UPINCESS 2022), Atlantis Press, 2023, pp. 217–226, doi: 10.2991/978-2-494069-39-8_20.

[6] W. D. Ahmad and A. A. Bakar, “Ensemble machine learning model for higher learning scholarship award decisions,” Int. J. Adv. Comput. Sci. Appl., vol. 11, no. 5, pp. 303–312, 2020, doi: 10.14569/IJACSA.2020.0110540.

[7] H. Hendri et al., “A hybrid data mining for predicting scholarship recipient students by combining K-means and C4.5 methods,” Indones. J. Electr. Eng. Comput. Sci., vol. 33, no. 3, pp. 1726–1735, 2024, doi: 10.11591/ijeecs.v33.i3.pp1726-1735.

[8] M. H. Ismail, T. R. Razak, N. M. Noor, and A. A. Aziz, “Evaluating machine learning algorithms for predicting financial aid eligibility: A comparative study of random forest, gradient boosting and neural network,” in Proc. 18th Int. Conf. Ubiquitous Inf. Manage. Commun. (IMCOM), 2024, pp. 1–6, doi: 10.1109/IMCOM60618.2024.10418450.

[9] A. T. Remolado, R. V. Perocho, and C. C. Hormillada, “Predicting scholarship eligibility using machine learning: A data-driven approach,” in Proc. 9th Int. Conf. Inf. Technol., Inf. Syst. Electr. Eng. (ICITISEE), 2025, pp. 421–426, doi: 10.1109/ICITISEE68184.2025.11355053.

[10] T. Suprapti, B. Nurhakim, B. W. A. Hermina, and V. A. S. Simbolon, “A decision tree model with grid search optimization for scholarship recipient classification,” J. Tek. Inform. (JUTIF), vol. 6, no. 5, pp. 3800–3813, 2025, doi: 10.52436/1.jutif.2025.6.5.5235.

[11] B. Isnanto and R. Sulaiman, “An empirical comparison of C4.5, naive Bayes, and KNN for scholarship selection,” J. Inf. Syst. Informatics, vol. 8, no. 3, pp. 3252–3268, 2026, doi: 10.63158/journalisi.v8i3.1617.

[12] J. G. Moreno-Torres, T. Raeder, R. Alaiz-Rodríguez, N. V. Chawla, and F. Herrera, “A unifying view on dataset shift in classification,” Pattern Recognit., vol. 45, no. 1, pp. 521–530, 2012, doi: 10.1016/j.patcog.2011.06.019.

[13] R. S. Baker and A. Hawn, “Algorithmic bias in education,” Int. J. Artif. Intell. Educ., vol. 32, no. 4, pp. 1052–1092, 2022, doi: 10.1007/s40593-021-00285-9.

[14] R. F. Kizilcec and H. Lee, “Algorithmic fairness in education,” in The Ethics of Artificial Intelligence in Education. New York, NY, USA: Routledge, 2022, pp. 174–202, doi: 10.4324/9780429329067-10.

[15] M. Hardt, E. Price, and N. Srebro, “Equality of opportunity in supervised learning,” in Adv. Neural Inf. Process. Syst., vol. 29, 2016, pp. 3315–3323.

[16] A. Chouldechova, “Fair prediction with disparate impact: A study of bias in recidivism prediction instruments,” Big Data, vol. 5, no. 2, pp. 153–163, 2017, doi: 10.1089/big.2016.0047.

[17] A. Castelnovo, R. Crupi, G. Greco, D. Regoli, I. G. Penco, and A. C. Cosentini, “A clarification of the nuances in the fairness metrics landscape,” Sci. Rep., vol. 12, Art. no. 4209, 2022, doi: 10.1038/s41598-022-07939-1.

[18] S. Caton and C. Haas, “Fairness in machine learning: A survey,” ACM Comput. Surv., vol. 56, no. 7, pp. 1–38, 2024, doi: 10.1145/3616865.

[19] J. Gardner, C. Brooks, and R. Baker, “Evaluating the fairness of predictive student models through slicing analysis,” in Proc. 9th Int. Conf. Learn. Anal. Knowl. (LAK), 2019, pp. 225–234, doi: 10.1145/3303772.3303791.

[20] R. Yu, H. Lee, and R. F. Kizilcec, “Should college dropout prediction models include protected attributes?,” in Proc. 8th ACM Conf. Learn. @ Scale, 2021, pp. 91–100, doi: 10.1145/3430895.3460139.

[21] O. B. Deho, S. Joksimovic, J. Li, C. Zhan, J. Liu, and L. Liu, “Should learning analytics models include sensitive attributes? Explaining the why,” IEEE Trans. Learn. Technol., vol. 16, no. 4, pp. 560–572, 2023, doi: 10.1109/TLT.2022.3226474.

[22] G. Raftopoulos, G. Davrazos, and S. Kotsiantis, “Fair and transparent student admission prediction using machine learning models,” Algorithms, vol. 17, no. 12, Art. no. 572, 2024, doi: 10.3390/a17120572.

[23] R. A. Opoku, B. Pei, and W. Xing, “Unveiling accuracy-fairness trade-offs: Investigating machine learning models in student performance prediction,” J. Learn. Anal., vol. 12, no. 2, pp. 125–139, 2025, doi: 10.18608/jla.2025.8543.

[24] K. Kesgin, S. Kiraz, S. Koşunalp, and B. Stoycheva, “Beyond performance: Explaining and ensuring fairness in student academic performance prediction with machine learning,” Appl. Sci., vol. 15, no. 15, Art. no. 8409, 2025, doi: 10.3390/app15158409.

[25] N. Pham, H. Pham Ngoc, and A. Nguyen-Duc, “Fairness for machine learning software in education: A systematic mapping study,” J. Syst. Softw., vol. 219, Art. no. 112244, 2025, doi: 10.1016/j.jss.2024.112244.

[26] Bernama, “PTPTN BAeK scholarship expanded to benefit 10,000 students,” Bernama, Jun. 14, 2026. [Online]. Available: https://www.bernama.com/en/news.php?id=2568298

[27] Ministry of Education Malaysia, Malaysia Education Blueprint 2015–2025 (Higher Education). Putrajaya, Malaysia: Ministry of Education Malaysia, 2015.

[28] Association of Southeast Asian Nations, ASEAN Guide on AI Governance and Ethics, Feb. 2024. [Online]. Available: https://asean.org/wp-content/uploads/2024/02/ASEAN-Guide-on-AI-Governance-and-Ethics_beautified_201223_v2.pdf

[29] Ministry of Science, Technology and Innovation, National Guidelines on Artificial Intelligence Governance and Ethics (AIGE). Putrajaya, Malaysia: MOSTI, Sep. 2024. [Online]. Available: https://www.malaysia.gov.my/en/my-initiative/whole-government-digital-services/tadbir-urus/national-guidelines-on-artificial-intelligence-governance-and-ethics-aige

[30] F. Ding, M. Hardt, J. Miller, and L. Schmidt, “Retiring adult: New datasets for fair machine learning,” in Adv. Neural Inf. Process. Syst., vol. 34, 2021, pp. 6478–6490.

[31] J. Schrouff et al., “Diagnosing failures of fairness transfer across distribution shift in real-world medical settings,” in Adv. Neural Inf. Process. Syst., vol. 35, 2022, pp. 19304–19318.

[32] Y. Yang, H. Zhang, J. W. Gichoya, D. Katabi, and M. Ghassemi, “The limits of fair medical imaging AI in real-world generalization,” Nat. Med., vol. 30, no. 10, pp. 2838–2848, 2024, doi: 10.1038/s41591-024-03113-4.

[33] B. Memarian and T. Doleck, “Fairness, accountability, transparency, and ethics (FATE) in artificial intelligence (AI) and higher education: A systematic review,” Comput. Educ. Artif. Intell., vol. 5, Art. no. 100152, 2023, doi: 10.1016/j.caeai.2023.100152.

[34] M. Zajko, “Automated government benefits and welfare surveillance,” Surveill. Soc., vol. 21, no. 3, pp. 246–258, 2023, doi: 10.24908/ss.v21i3.16107.

[35] M. Thoriq, F. Maulana, Y. S. Eirlangga, N. Hayati, and M. A. Madani, “Implementasi algoritma naïve Bayes dalam prediksi penerimaan mahasiswa penerima beasiswa KIP di Universitas Adzkia,” J. Fasilkom, vol. 15, no. 1, pp. 108–114, 2025, doi: 10.37859/jf.v15i1.8959.

[36] M. Thoriq, F. Maulana, and A. Q. Ayun, “Perbandingan naïve Bayes dan random forest pada seleksi KIP di Universitas Adzkia,” J. Pustaka AI, vol. 5, no. 3, pp. 602–610, 2025, doi: 10.55382/jurnalpustakaai.v5i3.1385.

[37] M. Thoriq, F. Maulana, S. A. Pitchay, F. H. M. Ridzuan, and W. Ramanda, “Explainable AI pada seleksi beasiswa KIP berbasis machine learning di Universitas Adzkia,” J. Pustaka AI, vol. 6, no. 1, pp. 80–91, 2026, doi: 10.55382/jurnalpustakaai.v6i1.1767.

[38] L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001, doi: 10.1023/A:1010933404324.

[39] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., 2016, pp. 785–794, doi: 10.1145/2939672.2939785.

[40] T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,” PLoS ONE, vol. 10, no. 3, Art. no. e0118432, 2015, doi: 10.1371/journal.pone.0118432.

[41] B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. New York, NY, USA: Chapman & Hall, 1993.

[42] F. Kamiran and T. Calders, “Data preprocessing techniques for classification without discrimination,” Knowl. Inf. Syst., vol. 33, no. 1, pp. 1–33, 2012, doi: 10.1007/s10115-011-0463-8.

[43] F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.

[44] H. Weerts, M. Dudík, R. Edgar, A. Jalali, R. Lutz, and M. Madaio, “Fairlearn: Assessing and improving fairness of AI systems,” J. Mach. Learn. Res., vol. 24, no. 257, pp. 1–8, 2023.