Analyzing Student Cognitive Engagement in AI-Based Learning using Prompting Techniques

Muhammad Rifqy Ramadana; Saida Ulfa; Yerry Soepriyanto

doi:10.24014/ijaidm.v8i2.36160

Analyzing Student Cognitive Engagement in AI-Based Learning using Prompting Techniques

Muhammad Rifqy Ramadana, Saida Ulfa, Yerry Soepriyanto

Abstract

With the increasing integration of AI in education, understanding how students engage cognitively in AI-assisted learning is crucial. Cognitive engagement in AI-assisted learning is important because it helps students interact meaningfully with AI tools, process information critically, and enhance their learning outcomes through effective AI-driven feedback and responses. To improve response quality in AI, one effective method is utilizing prompting techniques, which guide AI to generate more accurate, relevant, and structured responses, enhancing student learning experiences. This research investigates students' cognitive engagement when learning with AI-based tools using different prompting techniques, including Zero-Shot, Chain of Thought, Interactive Prompting, and Elaborate Prompting. A total of 54 students participated, and their engagement was assessed using a cognitive engagement questionnaire. The results, analyzed through a One-Sample T-test, reveal that students demonstrate significantly positive in cognitive engagement when using prompting techniques in AI-based learning. Furthermore, the findings suggest that effective prompting enhances the quality of AI-generated responses, positioning AI Chatbots as valuable learning assistants. This study provides important insights into optimizing AI-based learning strategies, highlighting the role of prompting in fostering deeper student interaction and engagement with AI tools.

Keywords

AI-Based Learning; Analyzing Student; Cognitive Engagement; Prompting Technique

References

L. Chen, P. Chen, and Z. Lin, “Artificial Intelligence in Education: A Review,” IEEE Access, vol. 8, pp. 75264–75278, 2020, doi: 10.1109/ACCESS.2020.2988510.

X. Chen, D. Zou, H. Xie, G. Cheng, and C. Liu, “Two Decades of Artificial Intelligence in Education,” Educational Technology & Society, vol. 25, no. 1, Jan. 2022.

E. Adamopoulou and L. Moussiades, “An Overview of Chatbot Technology,” 2020, pp. 373–383. doi: 10.1007/978-3-030-49186-4_31.

F. Clarizia, F. Colace, M. Lombardi, F. Pascale, and D. Santaniello, “Chatbot: An Education Support System for Student,” 2018, pp. 291–302. doi: 10.1007/978-3-030-01689-0_23.

M. Shumanov and L. Johnson, “Making conversations with chatbots more personalized,” Comput Human Behav, vol. 117, p. 106627, Apr. 2021, doi: 10.1016/j.chb.2020.106627.

Z. Ma, Z. Dou, Y. Zhu, H. Zhong, and J.-R. Wen, “One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, USA: ACM, Jul. 2021, pp. 555–564. doi: 10.1145/3404835.3462828.

W. Kaiss, K. Mansouri, and F. Poirier, “Effectiveness of an Adaptive Learning Chatbot on Students’ Learning Outcomes Based on Learning Styles,” International Journal of Emerging Technologies in Learning (iJET), vol. 18, no. 13, pp. 250–261, Jul. 2023, doi: 10.3991/ijet.v18i13.39329.

M. P.-C. Lin and D. Chang, “CHAT-ACTS: A pedagogical framework for personalized chatbot to enhance active learning and self-regulated learning,” Computers and Education: Artificial Intelligence, vol. 5, p. 100167, 2023, doi: 10.1016/j.caeai.2023.100167.

P. M. Papadopoulos, S. N. Demetriadis, I. G. Stamelos, and I. A. Tsoukalas, “Prompting students’ context-generating cognitive activity in ill-structured domains: does the prompting mode affect learning?,” Educational Technology Research and Development, vol. 57, no. 2, pp. 193–210, Apr. 2009, doi: 10.1007/s11423-008-9105-6.

T. Wu et al., “A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development,” IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 5, pp. 1122–1136, May 2023, doi: 10.1109/JAS.2023.123618.

O. Dewi and K. Wijayanti, “Mathematical Critical Thinking Ability in Terms of Students Learning Motivation in Probing Prompting Learning,” Unnes Journal of Mathematics Education, vol. 11, no. 2, 2022.

I. Nyoman Sudyana, D. Sudrajat, S. Sudadi, H. Fitriyah, and S. Adymas Pranajaya, “The Influence of The Probing Prompting Learning Model on The Development of Students’ Critical Thinking Ability,” Journal on Education, vol. 6, no. 1, pp. 3571–3577, Jun. 2023, doi: 10.31004/joe.v6i1.3454.

Y. Soepriyanto and D. Kuswandi, “Gamification Activities for Learning Visual Object-Oriented Programming,” in 2021 7th International Conference on Education and Technology (ICET), IEEE, Sep. 2021, pp. 209–213. doi: 10.1109/ICET53279.2021.9575076.

S. Ulfa, E. Surahman, I. Fatawi, and T. Hirashima, “Development Adaptive Instructional Scaffolding on Online Forum Discussion to Improve Personalization in MOOCs Learning Environments,” Journal of Educators Online, vol. 21, no. 3, May 2024, doi: 10.9743/JEO.2024.21.3.20.

X. Xu, X. Wang, Y. Zhang, H. Zhang, and Y. Wu, “Applying ChatGPT to Tackle the Side Effects of Personal Learning Environments in Higher Education: A Teacher and Teaching Perspective,” 2023. doi: 10.3233/FAIA230769.

E. Theophilou et al., “Learning to Prompt in the Classroom to Understand AI Limits: A Pilot Study,” 2023, pp. 481–496. doi: 10.1007/978-3-031-47546-7_33.

G. Sawalha, I. Taj, and A. Shoufan, “Analyzing student prompts and their effect on ChatGPT’s performance,” Cogent Education, vol. 11, no. 1, Dec. 2024, doi: 10.1080/2331186X.2024.2397200.

S. Li and S. P. Lajoie, “Cognitive engagement in self-regulated learning: an integrative model,” European Journal of Psychology of Education, vol. 37, no. 3, pp. 833–852, Sep. 2022, doi: 10.1007/s10212-021-00565-x.

M. Wallace-Spurgin, “Implementing Technology: Measuring Student Cognitive Engagement,” International Journal of Technology in Education, vol. 3, no. 1, p. 24, Nov. 2019, doi: 10.46328/ijte.v3i1.13.

S. Li, “Measuring Cognitive Engagement: An Overview of Measurement Instruments and Techniques,” International Journal of Psychology and Educational Studies, vol. 8, no. 3, pp. 63–76, Oct. 2022, doi: 10.52380/ijpes.2021.8.3.239.

S. Ekin, “Prompt Engineering For ChatGPT: A Quick Guide To Techniques, Tips, And Best Practices,” May 04, 2023. doi: 10.36227/techrxiv.22683919.v2.

P. Korzyński, G. Mazurek, P. Krzypkowska, and A. Kurasinski, “Artificial intelligence prompt engineering as a new digital competence: Analysis of generative AI technologies such as ChatGPT,” Entrepreneurial Business and Economics Review, no. 3, pp. 25–38, Nov. 2023.

H. L. Walker et al., “Reliability of Medical Information Provided by ChatGPT: Assessment Against Clinical Guidelines and Patient Information Quality Instrument,” J Med Internet Res, vol. 25, p. e47479, Jun. 2023, doi: 10.2196/47479.

J. D. Zamfirescu-Pereira, R. Y. Wong, B. Hartmann, and Q. Yang, “Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, New York, NY, USA: ACM, Apr. 2023, pp. 1–21. doi: 10.1145/3544548.3581388.

J. Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” in Advances in Neural Information Processing Systems 35 (NeurIPS 2022) , 2022.

J. A. Fredricks, P. C. Blumenfeld, and A. H. Paris, “School Engagement: Potential of the Concept, State of the Evidence,” Rev Educ Res, vol. 74, no. 1, pp. 59–109, Mar. 2004, doi: 10.3102/00346543074001059.

P. C. Blumenfield, T. M. Kempler, and J. S. Krajcik, “Motivation and Cognitive Engagement in Learning Environments,” in The Cambridge Handbook of The Learning Sciences, New York: Cambridge University Press, 2006, ch. 28.

A. W. Prananta, R. R. P. Megahati S, N. Susanto, and J. H. Raule, “Transforming Education and Learning through Chat GPT: A Systematic Literature Review,” Jurnal Penelitian Pendidikan IPA, vol. 9, no. 11, pp. 1031–1037, Nov. 2023, doi: 10.29303/jppipa.v9i11.5468.

Y. Guo and D. Lee, “Leveraging ChatGPT for Enhancing Critical Thinking Skills,” J Chem Educ, vol. 100, no. 12, pp. 4876–4883, Dec. 2023, doi: 10.1021/acs.jchemed.3c00505.

J. Sweller, “Cognitive load theory,” in International Encyclopedia of Education(Fourth Edition), Elsevier, 2023, pp. 127–134. doi: 10.1016/B978-0-12-818630-5.14020-5.

M. Wallace-Spurgin, “Implementing Technology: Measuring Student Cognitive Engagement,” International Journal of Technology in Education, vol. 3, no. 1, pp. 24–38, Nov. 2019, doi: 10.46328/ijte.v3i1.13.

H.-C. Yeh, “The synergy of generative AI and inquiry-based learning: transforming the landscape of English teaching and learning,” Interactive Learning Environments, pp. 1–15, Apr. 2024, doi: 10.1080/10494820.2024.2335491.

Y. Walter, “Embracing the future of Artificial Intelligence in the classroom: the relevance of AI literacy, prompt engineering, and critical thinking in modern education,” International Journal of Educational Technology in Higher Education, vol. 21, no. 1, p. 15, Feb. 2024, doi: 10.1186/s41239-024-00448-3.

DOI: http://dx.doi.org/10.24014/ijaidm.v8i2.36160

Refbacks

There are currently no refbacks.

Office and Secretariat:

Big Data Research Centre
Puzzle Research Data Technology (Predatech)
Laboratory Building 1st Floor of Faculty of Science and Technology
UIN Sultan Syarif Kasim Riau

Jl. HR. Soebrantas KM. 18.5 No. 155 Pekanbaru Riau – 28293
Website: http://predatech.uin-suska.ac.id/ijaidm
Email: ijaidm@uin-suska.ac.id
e-Journal: http://ejournal.uin-suska.ac.id/index.php/ijaidm
Phone: 085275359942

Journal Indexing:

IJAIDM Stats