Evaluating the Quality of Academic Center for Education, Culture and Research (ACECR) of Isfahan’s E-learning System Services Using the WebQual Method and based on the Integrated Approach of the Kano Model and the Importance-Performance Analysis Matrix

Document Type : Original Article

Authors

• Mitra Pashootanizadeh ¹
• Elahe Reisi ²
• Ali Mansoori ¹

¹ Associate Professor, Department of Knowledge and Information Science, University of Isfahan, Isfahan, Iran

² MSc in Knowledge and Information Science, University of Isfahan, Isfahan, Iran.

10.22067/infosci.2024.83678.1172

Abstract

Introduction: Numerous universities and educational institutions, particularly following the onset of the COVID-19 pandemic, have implemented e-learning systems. The Academic Center for Education, Culture and Research (ACECR) of Isfahan serves as an exemplar of an institution that consistently utilizes e-learning services. This research endeavors to assess the quality of the Isfahan ACECR e-learning platform by employing the WebQual instrument and integrating the methodologies of the Kano model and the Importance-Performance Analysis matrix.
Methodology: This research was applied in nature and conducted using a survey method. The statistical population comprised 231 students aged 18 and above, enrolled in the ACECR of Isfahan e-learning platform, across the educational fields of medicine, humanities, English language, and non-English language disciplines. The sampling method employed was stratified random sampling, where learners were selected from classes based on their educational groups. Cochran's formula was utilized to determine the sample size, which was estimated to be 144 individuals. The data collection tool employed in this research was the WebQual questionnaire, which comprises four dimensions: usability, information quality, interaction quality, and overall impact, with a total of 23 items, based on the WebQual4 questionnaire by Barnes and Vidgen (2002). To incorporate the Kano model and the importance-performance analysis matrix, the items of the WebQual tool were replicated in two separate questionnaires. These questionnaires aimed to assess the importance-performance relationship and identify the items that significantly influence learner satisfaction. The validity of the questionnaire was assessed using formal validity. To achieve this, the questionnaire was provided to faculty members of the University of Isfahan, who were asked to review each item and provide feedback. Based on their input, modifications were made to ensure the validity of the instrument. To determine the reliability coefficient of the questionnaire, Cronbach's alpha was calculated. This was done by administering an online questionnaire to a sample of 30 students from the research population, resulting in a Cronbach's alpha coefficient of 0.935. The Kolmogorov-Smirnov test was employed to assess the normality of the data. Descriptive and analytical statistical methods, including mean, frequency percentage, and geometric mean, were used to summarize and analyze the data.
Findings: The analysis of the Kano model, based on the WebQual items, revealed that most of the features related to the ACECR system were classified under the category of one-dimensional needs. The following items were identified as part of the one-dimensional needs group: "providing information with appropriate details by the system," "creating a sense of security for users accessing the system," "enabling communication between professors, learners, experts, and other stakeholders through the system," "facilitating easy communication with professors, learners, experts, and other stakeholders through the system," "instilling confidence in receiving the desired service as promised by the system," "ease of learning to operate the system," "ease of using the system," and "providing accurate information by the system." Additionally, the following items from the essential needs group of the Kano model were identified as critical: "providing valid information by the system," "providing appropriate and relevant information by the system," "providing understandable information by the system," and "ensuring the security of the user's personal information by the system." According to the results of the Kano model, failure to meet these essential needs leads to dissatisfaction among learners. In the analysis of the Importance-Performance Matrix, 7 out of 23 WebQual items were placed in the first quadrant, indicating that these items require further attention and are therefore considered essential. Additionally, 7 items from the WebQual questionnaire were positioned in the second quadrant of the Importance-Performance Matrix. This placement indicates that both the importance and performance of these items were acknowledged by the learners, and they were satisfied with these features in the system. Four items were located in the third quadrant, suggesting they have low priority. Finally, 5 items were positioned in the fourth quadrant. While these items are of low importance, they exhibit high performance from the learners' perspective, yet they contribute to a waste of resources.
Discussion & conclusion: Both models identified the "overall effect" dimension as a side and attractive need. The results of each WebQual dimension, analyzed through the Kano model, further revealed that the "usability" and "interaction quality" dimensions fall under the category of one-dimensional needs for learners. This means that failure to meet the items within these dimensions leads to dissatisfaction, while their complete and appropriate fulfillment results in learner satisfaction with the system's activities. According to the findings, both of these dimensions are placed in the second quadrant of the Importance-Performance Matrix. This positioning indicates that, in addition to being highly important to learners, the ACECR system also performed well in relation to these WebQual dimensions. On the other hand, the "quality of information" dimension, categorized as an essential requirement, plays a critical role in learner satisfaction. If the items related to this dimension are not fulfilled, dissatisfaction with the system's services increases. This dimension is placed in the first quadrant of the Importance-Performance Matrix, indicating that, despite its high importance to learners, it does not currently meet their expectations. Therefore, immediate improvements in performance are necessary. Finally, the "overall effect" dimension, which is categorized as an attractive need in the Kano model, is situated in the third quadrant of the Importance-Performance Matrix. While it is not considered highly important from the learners' perspective, the results suggest that enhancing the performance of this dimension can significantly increase learner satisfaction with the quality of service provided by the ACECR system.
Originality: E-learning in Iran, particularly within ACECR, has a relatively short history. As such, there is an ongoing need for continuous evaluation of these systems to enhance service quality, improve productivity, and retain as well as attract users. Among the various methods used to assess users' perceptions of a website, the WebQual model is widely recognized. Additionally, to better understand users' demands and needs, the Importance-Performance Analysis Matrix and the Kano model are useful tools for measuring customer satisfaction with services. In this research, these three models have been applied in combination to evaluate different aspects of the ACECR system.

Keywords

• Quality of Services
• Electronic Learning
• WebQual
• Kano model
• Matrix Performance - importance analysis
• Academic Center for Education
• Culture and Research (ACECR)

Main Subjects

• Services evaluation

---

References

Academic Center for Education, C. a. R. I. b. (2021). ACECR: about us. http://acecr.ir/en/page/1830/about-acecr

Al-Fraihat, D., Joy, M., & Sinclair, J. (2020). Evaluating E-learning systems success: An empirical study. Computers in human behavior, 102, 67-86.

Arnes, F., & Hanum, Y. (2019). Website Quality Evaluation of Visitor Satisfaction (Case Study: SBMPTN website). International Research Journal of Advanced Engineering and Science, 4(3), 241-245. https://doi.org/10.5281/zenodo.3365595

Barnes, S. J., & Vidgen, R. T. (2001). Assessing the quality of auction web sites. Proceedings of the 34th annual Hawaii international conference on system sciences,

Esmaeili Givi, M., Keshavarz, H., & Kargar Azad, Z. (2023). Quality assessment of E-learning website using asymmetric impact–performance analysis and Kano’s customer satisfaction model: a case study based on WebQual 4.0. Information Discovery and Delivery, 51(1), 35-46. https://doi.org/10.1108/IDD-08-2021-0083.

Fiena Rindani, F. R., & Sulistyo Puspitodjati, S. P. (2020). Integration of Webqual Method to Importance Performance Analysis and Kano Model to Analyze System Quality of E-Government: Case Study LAPOR! Jurnal Sistem Informasi (Journal of Information System), 16(2), 1-17. https://doi.org/10.21609/jsi.v16i2.937

Forghani, M. A., Rashidinezhad, M., Gholshan, S., & Darbandi, S. (2020). Evaluating the Quality of Educational Services of Bahonar University of Kerman Using the Combined Approach of Kano Model and QFD. Higher Education Letter, 13(49), 37-64.

Ghasemi, A., & Shahriarifard, A. (2016). Identifying & ranking of effecting factors on E-learning qualities. Technology of Education Journal (TEJ), 10(3), 207-218. https://doi.org/10.22061/tej.2016.539. [In Persian]

Ghasemi, M., Fardanesh, H., Hatani, J., & Ahmady, S. (2018). Evaluation of the electronic learning system of medical education (Case study of Shahid Beheshti Medical School). Education Strategies in Medical Sciences, 11(4). https://doi.org/10.29252/edcbmj.11.04.06. [In Persian]

Jundillah, M. L., Suseno, J. E., & Surarso, B. (2019). Evaluation of e-learning websites using the Webqual method and importance performance analysis. E3S Web of Conferences,

Kano, N. (1984). Attractive quality and must-be quality. Journal of the Japanese society for quality control, 31(4), 147-156.

Kazemi, M., & Ebadi Koohsare, F. (2020). New methods of education. Behvarz, 31(106), 60-65. https://doi.org/10.22038/BEHV.2020.50046.1139. [In Persian]

Lestarini, D., & Surendro, K. (2017). Designing kano-based e-service quality model to improve user satisfaction. 2017 2nd International conferences on Information Technology, Information Systems and Electrical Engineering (ICITISEE),

Lu, D.-N., Le, H.-Q., & Vu, T.-H. (2020). The factors affecting acceptance of e-learning: A machine learning algorithm approach. Education Sciences, 10(10), 270. https://doi.org/10.3390/educsci10100270

Madanian, M., & Shekarchizadeh, A. (2019). Reviewing and analyzing the concepts and dimensions of service quality management. Standard management and Quality, 9(2), 90-109.

Martilla, J. A., & James, J. C. (1977). Importance-performance analysis. Journal of marketing, 41(1), 77-79. https://doi.org/10.2307/1250495

Naghizade, H. (2015). Investigating the impact of electronic service quality on customer satisfaction in Tehran Municipality (case study of Hadian Shahr Institute) Islamic Azad university

Padlee, S. F., Reimers, V., Mokhlis, S., Anuar, M. M., & Ahmad, A. (2020). Keep up the good work in research universities: An importance-performance analysis. Australasian marketing journal, 28(2), 128-138.

Phadermrod, B., Crowder, R. M., & Wills, G. B. (2019). Importance-performance analysis based SWOT analysis. International journal of information management, 44, 194-203. https://doi.org/10.1016/j.ijinfomgt.2016.03.009

Rupere, T., & Jakovljevic, M. (2021). Usability and User Evaluation of an Integrated Multimedia E-Learning Management System. Knowledge Management & E-Learning, 13(3), 334-366. https://doi.org/10.34105/j.kmel.2021.13.018

Shia, B. C., Chen, M., & Ramdansyah, A. D. (2016). Measuring customer satisfaction toward localization website by WebQual and importance performance analysis (case study on AliexPress Site in Indonesia). American Journal of Industrial and Business Management, 6(2), 117-128. https://doi.org/10.4236/ajibm.2016.62012