Construction and Validation of the Artificial Intelligence Disclaimer Literacy Scale Instrument (AI-DLS)

Abdul Rahman, Azlena Vira Safitri, Lazaro Kumala Dewi

Abstract


The use of Generative Artificial Intelligence (GenAI) in higher education poses epistemic risks, including information hallucinations, limited accuracy, and content bias. Although AI developers include disclaimers as a risk mitigation measure, the effectiveness of this mechanism is often hampered by the phenomenon of warning fatigue, so it does not always encourage user evaluative behavior. This study aims to develop and validate the Artificial Intelligence Disclaimer Literacy Scale as a psychometric instrument to measure students' awareness, understanding, attitudes, and critical evaluation of AI disclaimers. The study employed a quantitative psychometric development design divided into five stages: literature review, indicator formulation, instrument item development, empirical testing, and statistical validation. The instrument was tested on 165 students at the Faculty of Teacher Training and Education. Data analysis included item validity testing using corrected item–total correlation, reliability testing using Cronbach's Alpha, and Exploratory Factor Analysis (EFA) using the Principal Component Analysis (PCA) method with Varimax rotation to test construct validity. The results showed that the AI-DLS instrument, consisting of 30 statement items, was proven valid and reliable. All items had a correlation value ≥ 0.30, with excellent reliability across all four dimensions (Cronbach's Alpha = 0.828 - 0.937). The EFA results showed a Kaiser–Meyer–Olkin (KMO) value of 0.932 and a factor structure capable of explaining 66.89% of the total variance. These findings indicate that the AI-DLS has strong psychometric qualities and is suitable for use as a diagnostic instrument to map students' epistemic readiness to interact with Artificial Intelligence systems critically and responsibly.

Keywords


Artificial Intelligence; Generative AI; AI Disclaimer; Instrument Validation

Full Text:

PDF

References


Bittle, K., & El-Gayar, O. (2025). Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda. Information, 16 (4), 296. https://doi.org/10.3390/info16040296

Carroll, F. (2024). Human-Browser Interaction: Investigating Whether the Current Browser Application's Design Actually Makes Sense for Its Users? International Journal of Human–Computer Interaction, 40 (22), 7562–7573. https://doi.org/10.1080/10447318.2023.2266789

Costello, A. B., & Osborne, J. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical assessment, research, and evaluation, 10(1).

Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61 (2), 228–239. https://doi.org/10.1080/14703297.2023.2190148

Dwivedi, YK, Kshetri, N., Hughes, L., Slade, EL, Jeyaraj, A., Kar, AK, Baabdullah, AM, Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, MA, Al- Busaidi, AS, Balakrishnan, J., Barlette, Y., Basu, S., Bose, I., Brooks, L., Buhalis, D., … Wright, R. (2023). Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges, and implications of generative conversational AI for research, practice, and policy. International Journal of Information Management, 71, 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642

Gimpel, H., Hall, K., Decker, S., Lämmermann , L., Mädche , A., Röglinger , M., Ruiner, C., Schoch, M., Schoop, M., Urbach, N., & Vandirk , S. (2023). Unlocking the Power of Generative AI Models and Systems such as GPT-4 and ChatGPT for Higher Education: A Guide for Students and Lecturers. DOI:10.13140/RG.2.2.20710.09287/2

Gümüş, M. M., & Kara, M. (2025). Development and validation of the Generative AI Literacy for Learning Scale (GenAI-LLs). Australasian Journal of Educational Technology. https://doi.org/10.14742/ajet.10236

Hair, J.F., Hult, GT M., Ringle, C. M., & Sarstedt, Marko. (2017). A primer on partial least squares structural equation modeling (PLS-SEM). Sage.

Ji, Z., Lee, N., Frieske , R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto , A., & Fung, P. (2023). Survey of Hallucinations in Natural Language Generation. ACM Computing Surveys, 55 (12), 1–38. https://doi.org/10.1145/3571730

Miao, Fengchun, Giannini, Stefania, & Holmes, Wayne. (2023). Guidance for generative AI in education and research. UNESCO.

Mollick, E. (2024). Co-intelligence: Living and Working with AI. Ebury Publishing. https://books.google.co.id/books?id=o8Qm0AEACAAJ

Ng, D.T.K., Wu, W., Leung, J.K.L., Chiu, TKF, & Chu, SKW (2024). Design and validation of the AI literacy questionnaire: The affective, behavioral, cognitive and ethical approaches. British Journal of Educational Technology, 55 (3), 1082–1104. https://doi.org/10.1111/bjet.13411

Nong, Y., Cui, J., He, Y., Zhang, P., & Zhang, T. (2024). Development and Validation of an AI Literacy Scale. Journal of Artificial Intelligence Research, 1 (1), 17–26. https://doi.org/10.70891/jair.2024.100029

Parasuraman, R., & Manzey, D.H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors: The Journal of the Human Factors and Ergonomics Society, 52 (3), 381–410. https://doi.org/10.1177/0018720810376055

Rahman, A., & Dewantara, B. A. (2025). The Role of Lecturers as Correctors in the Human-in-the-Loop (HITL) Model to Improve the Accuracy of Artificial Intelligence-Based Learning Evaluation. Advances In Education Journal, 2(1), 429-436. https://journal.al-afif.org/index.php/aej/article/view/208

Rahman, A., Nurmahmudah, F., Putra, E. C. S., Harsono, A. M. B., Dewantara, B. A., Arsyad, M. Z. T., & Alfarisa, F. (2025). Artificial Intelligence (AI) as the Reflective Partner: Empowering Teachers for Deep Learning Pedagogy. RIGGS: Journal of Artificial Intelligence and Digital Business, 4(3), 3877–3889. https://doi.org/10.31004/riggs.v4i3.2426

Silvia Agustina, & Ahman. (2024). Development and Measurement Psychometrics Inventory Student Career Planning School Intermediate Vocational School (SMK) Based Rasch Model Analysis. G-Couns: Journal Guidance and Counseling, 9 (1), 131–142. https://doi.org/10.31316/gcouns.v9i1.6232

Wogalter , M. (2018). Communication-Human Information Processing (C-HIP) Model (pp. 33–49). https://doi.org/10.1201/9780429462269-3

Zhai, C., Wibowo, S., & Li, L.D. (2024). The effects of over-reliance on AI dialogue systems on students' cognitive abilities: a systematic review. Smart Learning Environments, 11 (1). https://doi.org/10.1186/s40561-024-00316-7




DOI: https://doi.org/10.31764/justek.v9i2.37803

Refbacks

  • There are currently no refbacks.


JUSTEK : Jurnal Sains dan Teknologi sudah terindeks

    

EDITORIAL OFFICE: