Comparative Evaluation of AIML and IndoBERT on Chatbots School Information Services: Performance Analysis and User Experience
DOI:
https://doi.org/10.32493/jiup.v11i1.56235Keywords:
Chatbot, Natural Language Proccesing, School Information Services, Comparative Evaluation, AIMLAbstract
School information services that rely on manual staff frequently face delays and inconsistent responses. Chatbots offer an automated solution; however, selecting the appropriate technological approach for educational contexts still requires sufficient empirical evidence. This study compares two fundamentally different approaches: AIML (Artificial Intelligence Markup Language), which operates through explicit rule-based text pattern matching, and IndoBERT, a Transformer-based model fine-tuned on a labeled dataset for intent classification and combined with a vector similarity retrieval mechanism to extract answers from the school knowledge base. This comparison is essential given that the fundamental differences in how these approaches work directly affect their ability to handle language variation and their feasibility for resource-constrained school environments. Evaluation was conducted through technical testing on a labeled dataset (n ≈ 225) and a user study involving 60 participants comprising 20 students, 20 parents, and 20 teachers. Results show that the NLP approach outperformed AIML in intent classification (91.3% vs. 72.9%), macro F1 (0.89 vs. 0.72), coverage (97% vs. 84%), resolution rate (93% vs. 78%), and user satisfaction based on SUS scores (82.4 vs. 68.1). AIML, however, achieved lower response latency (127 ms vs. 213 ms). All differences were statistically significant (p < 0.01). The primary contribution of this study is a measurable comparative empirical evidence between AIML and IndoBERT+retrieval for secondary school information services, accompanied by a hybrid deployment recommendation that balances answer quality, responsiveness, and institutional resource constraints.
References
M. Amin, K. Nazik, and A. Salwa, Interacting with educational chatbots : A systematic review. Springer US, 2023. doi: https://doi.org/10.1007/s10639-022-11177-3.
E. Mursidah, L. Ambarwati, and F. A. Karima, “Implementasi Chatbot Layanan Informasi Pendaftaran Mahasiswa Baru Program Pascasarjana Departemen eknik Informatika ITS,” J. Teknol. Inf. dan Ilmu Komput., vol. 9, no. 2, pp. 1–8, 2022, doi: https://doi.org/10.21107/nero.v7i1.276.
J. F. Urquiza-Yllescas, S. Mendoza, L. M. Sánchez-Adame, J. Rodríguez, and D. Decouchant, “Educational Chatbots: A Systematic Mapping Study,” J. Intell. & Fuzzy Syst., vol. 49, no. 4, pp. 960–990, 2025, doi: https://doi.org/10.1177/18758967251356870.
G. Caldarini, S. Jaf, and K. McGarry, “A Literature Survey of Recent Advances in Chatbots,” Information, vol. 13, no. 1. p. 41, 2022. doi: https://doi.org/10.3390/info13010041.
T. Debets, S. K. Banihashem, D. Joosten-Ten Brinke, T. E. J. Vos, G. Maillette de Buy Wenniger, and G. Camp, “Chatbots in education: A systematic review of objectives, underlying technology and theory, evaluation criteria, and impacts,” Comput. Educ., vol. 234, p. 105323, 2025, doi: https://doi.org/10.1016/j.compedu.2025.105323.
U. S. A. Dewi Arumsaria, Kharismab, “Sistem ChatbotLayanan Informasi Mahasiswa Menggunakan Algoritma Long Short-Term Memory,” JUSTIN (Jurnal Sist. dan Teknol. Informasi), vol. 2, no. 2, pp. 77–86, 2024. doi: https://doi.org/10.30989/ijds.v2i2.1489.
A. F. Zulhilmi, R. S. Perdana, U. Brawijaya, and P. Korespondensi, “Pengenalan Entitas Bernama Menggunakan Bi-LSTM pada Chatbot Bahasa Indonesia,” vol. 11, no. 5, pp. 965–970, 2024, doi: https://doi.org/10.25126/jtiik.1077968.
F. Baharuddin and M. Naufal, “Fine-Tuning IndoBERT for Indonesian Exam Question Classification Based on Bloom’s Taxonomy,” J. Inf. Syst. Eng. Bus. Intell., vol. 9, pp. 253–263, Nov. 2023, doi: https://doi.org/10.20473/jisebi.9.2.253-263.
A. Dwiyono, A. Abdiansah, and M. Fachrurrozi, “Analisis Perbandingan Klasifikasi Intent Chatbot Menggunakan Deep Learning BERT, RoBERTa, dan IndoBERT,” J. Inf. Syst. Res., vol. 6, no. 1, pp. 595–606, 2024, doi: https://doi.org/10.47065/josh.v6i1.6051.
N. R. P. Syallya, A. A. Pravitasari, and A. Helen, “NLP-Based Intent Classification Model for Academic Curriculum Chatbots in Universities Study Programs,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 9, no. 1 SE-Artificial Intelligence, Feb. 2025, doi: https://doi.org/10.29207/resti.v9i1.6276.
N. Shahin and L. Ismail, “From rule-based models to deep learning transformers architectures for natural language processing and sign language translation systems: survey, taxonomy and performance evaluation,” Artif. Intell. Rev., vol. 57, no. 10, p. 271, 2024, doi: https://doi.org/10.1007/s10462-024-10895-z.
W. Hadinata and L. Stianingsih, “Implementasi Natural Language Processing pada Chatbot Untuk Helpdesk Informasi Wisata ( Studi kasus : Tangerang Raya ),” J. Ilm. Inform., vol. 12, pp. 175–181, Sep. 2024, doi: https://doi.org/10.33884/jif.v12i02.9279.
R. Jullfikar, R. Megasari, and R. A. Sukamto, “Pengembangan Chatbot Menggunakan BERT untuk Pengelolaan Komunikasi Beasiswa Ikatan Alumni Universitas Pendidikan Indonesia,” Digit. Transform. Technol., vol. 5, no. 1 SE-Articles, pp. 391–400, Jul. 2025, doi: https://doi.org/10.47709/digitech.v5i1.6367.
A. S. Al-Ayyubi, A. Hadi, D. Novaliendry, and K. Budayawan, “Integrasi Chatbot Akademik Berbasis Natural Language Processing Pada Sistem Informasi Akademik di STEI Ar Risalah,” J. Pendidik. Tambusai, vol. 9, no. 2 SE-Articles of Research, pp. 27134–27141, Aug. 2025, doi: https://doi.org/10.31004/jptam.v9i2.31214.
Gilbert, S. Beatricia, A. Reinaldo, and M. Hasani, Text Augmentation for Indonesian Intent Classification: Comparative Study. 2024. doi: https://doi.org/10.1109/ICITACEE62763.2024.10762810.
B. V. Kartika, M. J. Alfredo, and G. P. Kusuma, “Revue d ’ Intelligence Artificielle Fine-Tuned IndoBERT Based Model and Data Augmentation for Indonesian Language Paraphrase Identification,” vol. 37, no. 3, pp. 733–743, 2023. doi: https://doi.org/10.18280/ria.370322
R. Santosa, A. B. Nusantara, and S. Imron, “Comparative Analysis of SVM and IndoBERT for Intent Classification in Indonesian Overtime Chatbots,” J. Syst. Comput. Eng., vol. 6, no. 3, pp. 258–270, 2025. doi: https://doi.org/10.61628/jsce.v6i3.2058
Z. A. R. S. P. P. P. A. Annisa, “Kombinasi Intent Classification dan Named Entity Recognition pada Data Berbahasa Indonesia dengan Metode Dual Intent and Entity Transformer,” J. Teknol. Inf. dan Ilmu Komput., vol. 11, no. 5 SE-Ilmu Komputer, pp. 1017–1024, Oct. 2024, doi: https://doi.org/10.25126/jtiik.2024117985.
I. A. L. H. S. Rahma;, “Penerapan Text Augmentation untuk Mengatasi Data yang Tidak Seimbang pada Klasifikasi Teks Berbahasa Indonesia,” J. Teknol. Inf. dan Ilmu Komput., vol. 10, no. 6 SE-Ilmu Komputer, pp. 1329–1340, Dec. 2023, doi: https://doi.org/10.25126/jtiik.2023107325.
K. Peyton and S. Unnikrishnan, “A comparison of chatbot platforms with the state-of-the-art sentence BERT for answering online student FAQs,” Results Eng., vol. 17, p. 100856, 2023, doi: https://doi.org/10.1016/j.rineng.2022.100856.
R. M. Holis, P. E. P. Utomo, and B. F. Hutabarat, “Semantic FAQ Chatbot Using SBERT (Sentence-BERT) and Cosine Similarity for Academic Services,” Brill. Res. Artif. Intell., vol. 5, no. 2 SE-Articles, pp. 915–922, Oct. 2025, doi: https://doi.org/10.47709/brilliance.v5i2.7027.
Downloads
Published
Issue
Section
Categories
License
Copyright (c) 2026 Adam Herlambang, Farida Ardiani
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
Jurnal Informatika Universitas Pamulang have CC-BY-NC or an equivalent license as the optimal license for the publication, distribution, use, and reuse of scholarly work.
In developing strategy and setting priorities, Jurnal Informatika Universitas Pamulang recognize that free access is better than priced access, libre access is better than free access, and libre under CC-BY-NC or the equivalent is better than libre under more restrictive open licenses. We should achieve what we can when we can. We should not delay achieving free in order to achieve libre, and we should not stop with free when we can achieve libre.
Jurnal Informatika Universitas Pamulang is licensed under a Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)
YOU ARE FREE TO:
- Share : copy and redistribute the material in any medium or format
- Adapt : remix, transform, and build upon the material for any purpose, even commercially.
- The licensor cannot revoke these freedoms as long as you follow the license terms
This work is licensed under a 