Causal Machine Learning to Estimate the Impact of Using Generative AI on Students' GPA And Skill Retention

Angelina Hadriani, Ade Sumaedi, Layli Ana, Sugiyanti Sugiyanti, Yuni Yulistiani

Abstract


The rapid adoption of Generative Artificial Intelligence (GenAI) in higher education raises questions about its effects on academic performance and durable skills. This study evaluates the adjusted relationship between GenAI usage, GPA Change, and Skill Retention Score using 50,000 student records. Weekly GenAI Hours is treated as the exposure, while academic, behavioral, institutional, and psychological characteristics are included as adjustment covariates. A cross-fitted Double Machine Learning workflow is supported by overlap diagnostics, continuous-treatment sensitivity analysis, quartile comparison, and exploratory heterogeneous-effect analysis. High GenAI usage is associated with higher GPA Change (ATE = 0.0328; 95% CI: 0.0297–0.0358) and Skill Retention Score (ATE = 0.8098; 95% CI: 0.5733–1.0464). The quartile comparison shows the same direction. However, each additional weekly GenAI hour is associated with lower skill retention (ATE = −0.1096; 95% CI: −0.1298 to −0.0895). Thus, greater usage intensity does not necessarily indicate deeper learning. Because the data are observational, the estimates remain conditional on measured covariates. Universities should promote GenAI as a structured learning aid while preserving independent reasoning and sustained skill development.


Keywords


Generative AI, Causal Machine Learning, GPA Skill Retention

Full Text:

PDF

References


E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh,S. Günnemann, E. Hüllermeier, et al., “ChatGPT for good? On opportunities and challenges of large language models for education,” Learning and Individual Differences, vol. 103, Art. no. 102274, 2023, doi: 10.1016/j.lindif.2023.102274.

C. K. Lo, “What is the impact of ChatGPT on education? A rapid review of the literature,” Education Sciences, vol. 13, no. 4, Art. no. 410, 2023, doi: 10.3390/educsci13040410.

A. Tlili, B. Shehata, M. A. Adarkwah, A. Bozkurt, D. T. Hickey, R. Huang, and B. Agyemang, “What if the devil is my guardian angel? ChatGPT as a case study of using chatbots in education,” Smart Learning Environments, vol. 10, Art. no. 15, 2023, doi: 10.1186/s40561-023-00237-x.

C. Zastudil, M. Rogalska, C. Kapp, J. Vaughn, and S. MacNeil, “Generative AI in computing education: Perspectives of students and instructors,” in Proc. 2023 IEEE Frontiers in Education Conference (FIE), College Station, TX, USA, 2023, pp. 1–9, doi: 10.1109/FIE58773.2023.10343467.

UNESCO, Guidance for Generative AI in Education and Research. Paris, France: UNESCO, 2023.

OECD, OECD Digital Education Outlook 2026: Exploring Effective Uses of Generative AI in Education. Paris, France: OECD Publishing, 2026, doi: 10.1787/062a7394-en.

S. S. Kusumawardani, D. Wulandari, P. Pannen, F. A. Ekadiyanto, I. M. Wiryana, A. Purwarianti, and S. A. I. Alfarozi, Panduan Penggunaan Generative Artificial Intelligence pada Pembelajaran di Perguruan Tinggi, ver. 1.1. Jakarta, Indonesia: Direktorat Pembelajaran dan Kemahasiswaan, 2024.

S. Chen and A. C. K. Cheung, “Effect of generative artificial intelligence on university students’ learning outcomes: A systematic review and meta-analysis,” Educational Research Review, vol. 49, Art. no. 100737, 2025, doi: 10.1016/j.edurev.2025.100737.

Y. Bai and S. Wang, “Impact of generative AI interaction and output quality on university students’ learning outcomes: A technology-mediated and motivation-driven approach,” Scientific Reports, vol. 15, Art. no. 24054, 2025, doi: 10.1038/s41598-025-08697-6.

J. O. Wecks, J. Voshaar, B. J. Plate, and J. Zimmermann, “Generative AI usage and exam performance,” arXiv preprint arXiv:2404.19699, 2024, revised Nov. 2024.

B. J. Zimmerman, “Self-regulated learning and academic achievement: An overview,” Educational Psychologist, vol. 25, no. 1, pp. 3–17, 1990, doi: 10.1207/s15326985ep2501_2.

P. R. Pintrich, “The role of goal orientation in self-regulated learning,” in Handbook of Self-Regulation, M. Boekaerts, P. R. Pintrich, and M. Zeidner, Eds. San Diego, CA, USA: Academic Press, 2000, pp. 451–502.

J. Dunlosky, K. A. Rawson, E. J. Marsh, M. J. Nathan, and D. T. Willingham, “Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology,” Psychological Science in the Public Interest, vol. 14, no. 1, pp. 4–58, 2013, doi: 10.1177/1529100612453266.

R. A. Bjork and E. L. Bjork, “Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning,” in Psychology and the Real World: Essays Illustrating Fundamental Contributions to Society, M. A. Gernsbacher, R. W. Pew, L. M. Hough, and J. R. Pomerantz, Eds. New York, NY, USA: Worth Publishers, 2011, pp. 56–64.

H. L. Roediger III and J. D. Karpicke, “Test-enhanced learning: Taking memory tests improves long-term retention,” Psychological Science, vol. 17, no. 3, pp. 249–255, 2006, doi: 10.1111/j.1467-9280.2006.01693.x.

A. A. Alimin and H. Hariyadi, “Pola pemanfaatan generative AI dalam penulisan akademik mahasiswa Kalimantan Barat,” VOX EDUKASI: Jurnal Ilmiah Ilmu Pendidikan, vol. 16, no. 2, pp. 455–466, 2025, doi: 10.31932/ve.v16i2.5516.

B. A. Dewantara and L. K. Dewi, “Generative AI dalam pembelajaran mahasiswa: Antara inovasi pendidikan dan integritas akademik,” JIIP: Jurnal Ilmiah Ilmu Pendidikan, vol. 8, no. 7, pp. 8209–8217, 2025, doi: 10.54371/jiip.v8i7.8680.

Nursalam, J. Ambarita, and F. Resha, “Pemanfaatan ChatGPT dalam penulisan karya ilmiah mahasiswa perguruan tinggi di Maluku: Studi metode campuran,” Jurnal Pendidikan dan Kebudayaan, vol. 10, no. 2, pp. 252–274, 2025, doi: 10.24832/jpnk.v10i2.6387.

T. Mairisiska and N. Qadariah, “Persepsi mahasiswa FTIK IAIN Kerinci terhadap penggunaan ChatGPT untuk mendukung pembelajaran di era digital,” Jurnal Teknologi Pembelajaran Indonesia, vol. 13, no. 2, pp. 107–124, 2023, doi: 10.23887/jurnal_tp.v13i2.2653.

G. Siemens and R. S. J. d. Baker, “Learning analytics and educational data mining: Towards communication and collaboration,” in Proc. 2nd International Conference on Learning Analytics and Knowledge (LAK ’12), Vancouver, BC, Canada, 2012, pp. 252–254, doi: 10.1145/2330601.2330661.

R. Ferguson, “Learning analytics: Drivers, developments and challenges,” International Journal of Technology Enhanced Learning, vol. 4, nos. 5–6, pp. 304–317, 2012, doi: 10.1504/IJTEL.2012.051816.

V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins, “Double/debiased machine learning for treatment and structural parameters,” The Econometrics Journal, vol. 21, no. 1, pp. C1–C68, 2018, doi: 10.1111/ectj.12097.

S. Wager and S. Athey, “Estimation and inference of heterogeneous treatment effects using random forests,” Journal of the American Statistical Association, vol. 113, no. 523, pp. 1228–1242, 2018, doi: 10.1080/01621459.2017.1319839.

S. Athey, J. Tibshirani, and S. Wager, “Generalized random forests,” The Annals of Statistics, vol. 47, no. 2, pp. 1148–1178, 2019, doi: 10.1214/18-AOS1709.

S. Athey and G. W. Imbens, “The state of applied econometrics: Causality and policy evaluation,” Journal of Economic Perspectives, vol. 31, no. 2, pp. 3–32, 2017, doi: 10.1257/jep.31.2.3.

S. R. Künzel, J. S. Sekhon, P. J. Bickel, and B. Yu, “Metalearners for estimating heterogeneous treatment effects using machine learning,” Proceedings of the National Academy of Sciences, vol. 116, no. 10, pp. 4156–4165, 2019, doi: 10.1073/pnas.1804597116.

E. H. Kennedy, “Towards optimal doubly robust estimation of heterogeneous causal effects,” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 85, no. 3, pp. 908–940, 2023, doi: 10.1093/jrsssb/qkad017.

O. Zawacki-Richter, V. I. Marín, M. Bond, and F. Gouverneur, “Systematic review of research on artificial intelligence applications in higher education—Where are the educators?” International Journal of Educational Technology in Higher Education, vol. 16, Art. no. 39, 2019, doi: 10.1186/s41239-019-0171-0




DOI: https://doi.org/10.55311/aiocsit.v7i2.411

Refbacks

  • There are currently no refbacks.