Validity and Reliability of the Artificial Intelligence Literacy Instrument for Prospective Islamic Education Teachers: A Rasch Analysis

Herlini Puspika Sari; devi arisanti; Aldeva ilhami

doi:10.24014/potensia.v11i2.38308

Validity and Reliability of the Artificial Intelligence Literacy Instrument for Prospective Islamic Education Teachers: A Rasch Analysis

Herlini Puspika Sari, devi arisanti, Aldeva ilhami

Abstract

This study aims to validate an Artificial Intelligence (AI) literacy instrument for Islamic Education (PAI) students using Rasch analysis to evaluate its reliability, validity, item-fit statistics, and unidimensionality. The instrument was developed to measure the extent of AI literacy among prospective Islamic Education teachers. A cross-sectional survey design was employed, involving 64 students from a university in Riau, Indonesia. Participants completed an AI literacy instrument consisting of 85 items. The sampling technique used was quota sampling. Data were analyzed using the Rasch model with the aid of Winsteps software version 3.73. The results of the Rasch analysis indicated that the instrument demonstrated good reliability (α = 0.85), excellent item quality (0.94), and consistent respondent reliability (0.84). In terms of validity, the three dimensions of AI literacy were confirmed to possess unidimensional properties. The practical implication of this study lies in providing higher education institutions with a validated AI literacy instrument that can be used to assess the knowledge of future Islamic Education teachers. By employing Rasch analysis, this study contributes to enhancing the psychometric robustness of measurement tools, particularly in the context of assessing artificial intelligence literacy in Indonesia.

Keywords

validity; reliability; Rasch model; AI literacy; Islamic Education

References

Abbas, Tokeer. 2023. “Ethical Implications of AI in Modern Education: Balancing Innovation and Responsibility.” Social Sciences Spectrum 2(1): 51–57.

Biagini, G, S Cuomo, and M Ranieri. 2023. “Developing and Validating a Multidimensional AI Literacy Questionnaire: Operationalizing AI Literacy for Higher Education.” In CEUR Workshop Proceedings, eds. Schicchi D., Istitute for Education Technology National Research Council Via Ugo la Malfa 152, Palermo, Taibi D., Istitute for Education Technology National Research Council Via Ugo la Malfa 152, Palermo, Temperini M., and Ingegneria Informatica Universita Roma Sapienza Automatica, e Gestionale “Antonio Ruberti”, Via Ariosto, 25, Roma. University of Florence, Florence, Italy: CEUR-WS.

Bond, Trevor G., Zi Yan, and Moritz Heene. 2021. Applying the Rasch Model Applying the Rasch Model:Fundamental Measurement in The Human Science. Fouth Edit. New York: Routledge. doi:10.4324/9781410614575.

Boone, William J., Melissa S. Yale, and John R. Staver. 2014. Rasch Analysis in the Human Sciences Rasch Analysis in the Human Sciences. doi:10.1007/978-94-007-6857-4.

Boone, William J. 2016. “Rasch Analysis for Instrument Development: Why, When, and How?” CBE—Life Sciences Education 15(4): rm4.

Fadlelmula, Fatma Kayan, and Saba Mansoon Qadhi. 2024. “A Systematic Review of Research on Artificial Intelligence in Higher Education: Practice, Gaps, and Future Directions in the GCC.” Journal of University Teaching and Learning Practice 21(6): 146–73.

Faisal Tariq Hasan, and Rafiq Rahman. 2024. “Artificial Intelligence in Personalized Learning: A New Era of Education.” Proceeding of The International Conference of Inovation, Science, Technology, Education, Children, and Health 4(2): 306–8. doi:10.62951/icistech.v4i2.127.

Grassini, S. 2024. “A Psychometric Validation of the PAILQ-6: Perceived Artificial Intelligence Literacy Questionnaire.” In ACM International Conference Proceeding Series, Department of Psychosocial Science, University of Bergen, Norway: Association for Computing Machinery. doi:10.1145/3679318.3685359.

Kesuma, Dorie P, and Lisa Amelia Fransen. 2025. “Implementasi Instrumen MAILS (Meta AI Literacy Scale) Untuk Pengukuran Tingkat Literasi AI Pada Mahasiswa Ilmu Komputer.” Jurnal Sistem Informasi, Manajemen dan Teknologi Informasi 3(1): 25–36.

Metsämuuronen, Jari. 2023. “Seeking the Real Item Difficulty: Bias-Corrected Item Difficulty and Some Consequences in Rasch and IRT Modeling.” Behaviormetrika 50(1): 121–54.

Müller, Marianne. 2020. “Item Fit Statistics for Rasch Analysis: Can We Trust Them?” Journal of Statistical Distributions and Applications 7(1): 5.

Ng, D T K, W Wu, J K L Leung, T K F Chiu, and S K W Chu. 2024. “Design and Validation of the AI Literacy Questionnaire: The Affective, Behavioural, Cognitive and Ethical Approach.” British Journal of Educational Technology 55(3): 1082–1104. doi:10.1111/bjet.13411.

OECD. 2025. Empowering Learners for the Age of AI. Paris: OECD Publishing.

Prasetya, Wanda Agus, and Anggi Tias Pratama. 2023. “Item Quality Analysis Using the Rasch Model to Measure Critical Thinking Ability in the Material of the Human Digestive System of Biology Subject in High School.” Jurnal Penelitian Dan Evaluasi Pendidikan 27(1): 76–91.

Rahayu, Sri. 2023. “The Impact of Artificial Intelligence on Education: Opportunities and Challenges.” Jurnal Educatio FKIP UNMA 9(4): 2132–40.

Shi, Lehong. 2025. “Assessing Teachers’ Generative Artificial Intelligence Competencies: Instrument Development and Validation.” Education and Information Technologies. doi:10.1007/s10639-025-13684-5.

Sumintono, Bambang, and Wahyu Widhiarso. 2015. Aplikasi Pemodelan Rasch Pada Assessment Pendidikan. 1st ed. Bandung: Trim Komunikata Publishing House.

UNESCO. 2024. AI Competency Framework for Students. Paris: United Nations Educational Scientific and Cultural Organization. doi:10.54675/jkjb9835.

Wilton, Lesley, Stephen Ip, Meera Sharma, and Frank Fan. 2022. “Where Is the AI? AI Literacy for Educators.” In International Conference on Artificial Intelligence in Education, Springer, 180–88.

Zhang, Jia, and Zhuo Zhang. 2024. “AI in Teacher Education: Unlocking New Dimensions in Teaching Support, Inclusive Learning, and Digital Literacy.” Journal of Computer Assisted Learning 40(4): 1871–85.

Zootzky, Gregor, and Alexander Pfeiffer. 2024. “Educational Transformation through AI: Preparing for a New Era of Learning.” In INTED2024 Proceedings, IATED, 1202–7.

DOI: http://dx.doi.org/10.24014/potensia.v11i2.38308