Enhancing Usability Testing Through Sentiment Analysis: A Comparative Study Using SVM, Naive Bayes, Decision Trees and Random Forest

Abstract

In the digital age, mobile applications have become an integral part of everyday life, making usability testing an essential factor in ensuring a seamless user experience. Traditional usability testing methods often demand considerable resources, including time and cost, which calls for more efficient and automated alternatives. This study explores the use of sentiment analysis as an innovative approach to evaluate the usability of mobile applications. By analyzing user reviews from the Google Play Store, the research compares the effectiveness of four machine learning algorithms—Support Vector Machine (SVM), Naive Bayes, Decision Tree, and Random Forest—in classifying sentiment and evaluating usability. A dataset consisting of 2,000 reviews from a banking app was collected and labeled based on usability criteria, such as efficiency, user satisfaction, learnability, memorability, and error rates. The feature extraction process utilized Term Frequency-Inverse Document Frequency (TF-IDF) to enhance the relevance of the review texts for sentiment analysis. The findings reveal that Random Forest achieved the highest accuracy (68.15%) and demonstrated the best performance in terms of F1 Score, precision, and recall, although it had the longest processing time. In contrast, Naive Bayes, while the fastest, showed lower accuracy and F1 Score, making it suitable for applications with large datasets or limited processing time. Decision Tree and SVM offered a balanced trade-off between speed and accuracy. The study concludes that Random Forest is the preferred choice when high accuracy and prediction performance are crucial, despite its longer processing time. Meanwhile, Naive Bayes is more appropriate for scenarios demanding rapid data processing, and SVM and Decision Tree are recommended when a balance between speed and accuracy is needed.