A RAG-Based Academic Information Chatbot Using Lightweight LLaMA and Indo-Sencence-BERT

Abstract

In the current digital era, Institut Teknologi Kalimantan (ITK) encounters challenges in delivering academic information that is fast, accurate, and easily accessible to students, lecturers, and academic staff. Access to important information such as administrative procedures, report writing guidelines, and academic policies remains largely reliant on manual systems and static handbooks. To address this issue, this study investigates a chatbot system utilizing the Retrieval-Augmented Generation (RAG) framework through LLaMA model. The chatbot combines semantic retrieval and natural language generation to provide relevant and accurate answers based on existing academic documents. Evaluation was conducted on two lightweight LlaMA models: 1.5 and 3B parameters. Furthermore, different embedding vector also evaluated along with Indo-Sentence-BERT as well as the chunking size. The most optimal configuration was achieved using LLaMA 3B as the generative model and Indo-Sentence-BERT as the retriever, with a chunk size of 200 tokens and an overlap of 10 tokens. This setup achieved a RAGAS score of approximately 0.9, a competitive MRR of 0.5, and response latency under 1 second. Although LLaMA 1B recorded a higher MRR (0.6), its low RAGAS score made it less favorable. Overall, the LLaMA 3B and Indo-Sentence-BERT configuration is recommended to enhance the efficiency of academic information retrieval at ITK.