NLP-Based Intent Classification Model for Academic Curriculum Chatbots in Universities Study Programs

Najma Rafifah Putri Syallya; Anindya Apriliyanti Pravitasari; Afrida Helen

doi:10.29207/resti.v9i1.6276

Najma Rafifah Putri Syallya Universitas Padjadjaran
Anindya Apriliyanti Pravitasari Universitas Padjadjaran
Afrida Helen Universitas Padjadjaran

DOI: https://doi.org/10.29207/resti.v9i1.6276

Keywords: Chatbot, IndoBERT, Intent Classification

Abstract

Chatbots are increasingly prevalent in various fields, including academic fields. Universities often rely on lecturers and staff for information access, which can lead to delays, limited availability outside working hours, and the risk of missed questions. This study aims to develop a chatbot model capable of addressing questions about the curriculum through intent classification, reducing reliance on manual responses, and providing a solution that ensures quick, accurate information retrieval. The research focuses on optimizing the IndoBERT model for intent classification and addresses challenges that arose due to imbalance data, which could have impacted model performance. Data was collected through an open poll on common curriculum-related questions asked by students. To address data imbalance, we tried oversampling techniques, such as SMOTE, B-SMOTE, ADASYN, and Data Augmentation. Data augmentation was chosen and successfully addressed the imbalance problem while maintaining data semantics effectively. We achieved the best model with hyperparameters batch size of 8, learning rate of 0.00001, 15 epochs, and 64 neurons in the hidden layer, resulting in 98.7% accuracy on the test data. Evaluation metrics further demonstrate the model's robustness across multiple intents. This research demonstrates the advantages of the IndoBERT model in intent classification for academic chatbots, achieving excellent performance.

Downloads

Download data is not yet available.

References

[1] C. W. Okonkwo and A. Ade-Ibijola, “Chatbots applications
in education: A systematic review,” Computers and
Education: Artificial Intelligence, vol. 2, p. 100033, 2021,
doi: 10.1016/j.caeai.2021.100033.
[2] S. Mohamad Suhaili, N. Salim, and M. N. Jambli, “Service
chatbots: A systematic review,” Expert Systems with
Applications, vol. 184, p. 115461, Dec. 2021, doi:
10.1016/j.eswa.2021.115461.
[3] J. Weizenbaum, “ELIZA—a computer program for the study
of natural language communication between man and
machine,” Commun. ACM, vol. 9, no. 1, pp. 36–45, Jan. 1966,
doi: 10.1145/365153.365168.
[4] G. Caldarini, S. Jaf, and K. McGarry, “A Literature Survey of
Recent Advances in Chatbots,” 2022, doi:
10.48550/ARXIV.2201.06657.
[5] J. Schuurmans and F. Frasincar, “Intent Classification for
Dialogue Utterances,” IEEE Intell. Syst., vol. 35, no. 1, pp.
82–88, Jan. 2020, doi: 10.1109/MIS.2019.2954966.
[6] M. Y. Helmi Setyawan, R. M. Awangga, and S. R. Efendi,
“Comparison Of Multinomial Naive Bayes Algorithm And
Logistic Regression For Intent Classification In Chatbot,” in
2018 International Conference on Applied Engineering
(ICAE), Batam: IEEE, Oct. 2018, pp. 1–5. doi:
10.1109/INCAE.2018.8579372.
[7] S. Larson et al., “An Evaluation Dataset for Intent
Classification and Out-of-Scope Prediction,” in Proceedings
of the 2019 Conference on Empirical Methods in Natural
Language Processing and the 9th International Joint
Conference on Natural Language Processing (EMNLPIJCNLP), Hong Kong, China: Association for Computational
Linguistics, 2019, pp. 1311–1316. doi: 10.18653/v1/D19-
1131.
[8] A. Cohan, W. Ammar, M. Van Zuylen, and F. Cady,
“Structural Scaffolds for Citation Intent Classification in
Scientific Publications,” in Proceedings of the 2019
Conference of the North, Minneapolis, Minnesota:
Association for Computational Linguistics, 2019, pp. 3586–
3596. doi: 10.18653/v1/N19-1361.
[9] A. Vaswani et al., “Attention Is All You Need,” Aug. 01,
2023, arXiv: arXiv:1706.03762. Accessed: Jul. 14, 2024.
[Online]. Available: http://arxiv.org/abs/1706.03762
[10] C. Raffel et al., “Exploring the Limits of Transfer Learning
with a Unified Text-to-Text Transformer,” Sep. 19, 2023,
arXiv: arXiv:1910.10683. Accessed: May 02, 2024. [Online].
Available: http://arxiv.org/abs/1910.10683
[11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT:
Pre-training of Deep Bidirectional Transformers for
Language Understanding,” May 24, 2019, arXiv:
arXiv:1810.04805. Accessed: May 02, 2024. [Online].
Available: http://arxiv.org/abs/1810.04805
[12] T. B. Brown et al., “Language Models are Few-Shot
Learners,” Jul. 22, 2020, arXiv: arXiv:2005.14165. Accessed:
May 02, 2024. [Online]. Available:
http://arxiv.org/abs/2005.14165
[13] F. Koto, “IndoLEM and IndoBERT: A Benchmark Dataset
and Pre-trained Language Model for Indonesian NLP,”
COLING 2020 - 28th International Conference on
Computational Linguistics, Proceedings of the Conference.
pp. 757–770, 2020.
[14] K. S. Nugroho, A. Y. Sukmadewa, H. W. DW, F. A. Bachtiar,
and N. Yudistira, “BERT Fine-Tuning for Sentiment Analysis
on Indonesian Mobile Apps Reviews,” in 6th International
Conference on Sustainable Information Engineering and
Technology 2021, Sep. 2021, pp. 258–264. doi:
10.1145/3479645.3479679.
[15] A. Fadhlurohman, “Development of Indonesian Language
Intelligent Chatbot for Public Services in JAKI Application,”
Proceedings of 2023 IEEE International Smart Cities
Conference, ISC2 2023. 2023. doi:
10.1109/ISC257844.2023.10293514.
[16] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P.
Kegelmeyer, “SMOTE: Synthetic Minority Over-sampling
Technique,” 2011, doi: 10.48550/ARXIV.1106.1813.
[17] H. Han, W. Wang, and B. Mao, “Borderline-SMOTE: A New
Over-Sampling Method in Imbalanced Data Sets Learning,”
in International Conference on Intelligent Computing, 2005.
[Online]. Available:
https://api.semanticscholar.org/CorpusID:12126950
[18] J. Wei and K. Zou, “EDA: Easy Data Augmentation
Techniques for Boosting Performance on Text Classification
Tasks,” 2019, doi: 10.48550/ARXIV.1901.11196.