Analysis of Students' Satisfaction Level on iLearn Quality during COVID-19 Pandemic with WebQual 4.0 and PLS-SEM

ABSTRACT


A. INTRODUCTION
The COVID-19 pandemic that hit the world has an impact on various sectors of life, such as the economy (Fernandes, 2020;Ozili & Arun, 2020), environment (Chakraborty & Maity, 2020;Zambrano-Monserrate et al., 2020), and education (Akat & Karataş, 2020;König et al., 2020). The impact of the COVID-19 pandemic on the education sector includes the learning process that cannot be conducted directly (offline). In order to deal with the problem, educational institutions changed the learning system from offline to online and implemented a system to support online learning called Learning Management System (LMS) (Findik-Coşkunçay et al., 2018). An LMS is a web-based application capable of transforming face-toface sessions by offering students a space for online learning (Wichadee, 2015). There are several LMSs available to institutions for online courseware management, whether the open source (e.g., Moodle, Sakai, Google Classroom) or commercial (e.g., Blackboard, Brightspace D2L) (Mohd Kasim & Khalid, 2016).
The learning process at Andalas University is also affected by the COVID-19 pandemic. An LMS called iLearn (https://ilearn.unand.ac.id) was implemented by Andalas University during the pandemic. iLearn is distributed to several faculties and postgraduate programs for localization management and other related purposes, which can be accessed by Andalas University students registered in the courses taken. Some studies, such as in (Fearnley & Amora, 2020;Koh & Kan, 2020), found that students' satisfaction level in accessing an LMS could affect the learning result. Thus, it is important for Andalas University as an educational institution to provide good LMS facilities to meet student satisfaction so that students can maintain and even improve learning outcomes in the midst of sudden changes in the learning process due to the COVID-19 pandemic. Therefore, in this research, we will analyze how iLearn quality affects students' satisfaction levels. Website quality is measured by the WebQual 4.0 instrument, which consists of three variables, i.e., Usability, Information Quality, and Service Interactions Quality (Barnes & Vidgen, 2003). Each variable is compiled on the appropriate indicators. The variables in WebQual 4.0 are latent variables measured through the indicators.
WebQual 4.0 can be used as an instrument to measure the level of user satisfaction in various types of websites, such as e-commerce websites (Andry et al., 2019;Sutisna et al., 2019), information websites (Firdaus et al., 2019;Rahmat et al., 2021), and government websites (Donie et al., 2019;Titiani et al., 2020). In this research, the relationship between WebQual 4.0 variables and the influence of its indicators are analyzed. The Structural Equation Modeling (SEM) method can be used to analyze the relationship between latent variables and their indicators simultaneously (Hwang et al., 2020). There are two approaches in SEM, i.e., Covariance-Based SEM (CB-SEM) and Variance-Based SEM (PLS-SEM). Based on (F. Hair Jr et al., 2014), researchers tend to use PLS-SEM because of its minimal assumption, such as non-normal distribution data, small samples, and formative indicators. This assumption is a relief for researchers to use PLS-SEM because some data tends to be difficult to obtain, resulting in a small number of samples and not normally distributed. This strength of PLS-SEM and the advanced technology causes the usage of PLS-SEM to increase exponentially (Blanco-Encomienda & Rosillo-Díaz, 2021;Chin et al., 2020;Khan et al., 2019). Among the uses of PLS-SEM are in the fields of economy (Buitrago R. et al., 2021;Palos-Sanchez et al., 2021), health (Mardianto et al., 2021), environment (Wang et al., 2021, and education (Arthur, 2019a(Arthur, , 2019bTee et al., 2021). Several studies have examined the level of student satisfaction on LMS, such as in (Alkhateeb & Abdalla, 2021;Alturise, 2020;Lee & Jeon, 2020). However, most of them use a descriptive statistical approach or first generation regression models, so it cannot be identified which indicators and variables significantly affect student satisfaction. In this study, student satisfaction will be analyzed using an inferential statistical approach which is PLS-SEM as a second generation regression model that can identify which indicators and variables significantly affect the level of student satisfaction simultaneously. Inferential statistics is known for its strict assumptions, including the data distribution and large data sizes. With the PLS-SEM approach, these problems can be overcome due to PLS-SEM's ability to handle data with a small sample. The PLS-SEM method is used in this research because the number of samples used is small (100 samples) with an ordinal scale and non-normal distribution data. Research on the analysis of user satisfaction on website quality using the WebQual 4.0 instrument certainly shows different results for different methods and types of websites (Nugraha et al., 2020;Shiau et al., 2019). This research aims to analyze the influence and significance of the WebQual 4.0 variables on students' satisfaction level toward iLearn using the PLS-SEM method.

B. METHODS
This study uses quantitative methods because it aims to determine the causal relationship by using statistical analysis, namely PLS-SEM. Then, from the perspective of this research, it uses an ethical approach in the sense that the researchers collected the data by firstly determining the variables from the existing theory, namely WebQual 4.0. Then this study uses primary data obtained by questionnaires which are then distributed to respondents as research samples. The data is the results of questionnaires distributed to Andalas University Mathematics Department students enrolled in the iLearn courses for the odd semester of 2020/2021. Researchers took a sample of 100 people randomly with four groups categories based on the entry year. Group 1 is the class of 2017 and below, group 2 is the class of 2018, group 3 is the class of 2019, and group 4 is the class of 2020. Each group is randomly sampled as 25 students per group, so a total of 100 students are involved in this research.
The questionnaire in this study uses questions on the WebQual 4.0 instrument. These questions are the validated standard indicators set to form the three WebQual 4.0 variables, which is Usability, Information Quality, and Service Interactions Quality (Barnes & Vidgen, 2003). Therefore, the questionnaire used is valid to measure the level of student satisfaction on iLearn. The research variables used are presented in Table 1. iLearn is easy to learn to operate U1 The interaction with iLearn is clear dan understandable U2 iLearn is easy to navigate U3 iLearn is easy to use U4 iLearn has an attractive appearance U5 iLearn design is appropriate to the type of the website U6 iLearn conveys a sense of competency U7 iLearn creates a positive experience U8 iLearn provides accurate information IQ1 iLearn provides timely information IQ2 iLearn provides believable information IQ3 iLearn provides relevant information IQ4 iLearn provides easy-to-understand information IQ5 iLearn provides detail information at the right level IQ6 iLearn present information in an appropriate format IQ7 iLearn has a good reputation SIQ1 iLearn secured personal information SIQ2 iLearn creates a sense of personalization SIQ3 iLearn conveys a sense of community SIQ4 iLearn make it easy to communicate with the organization SIQ5 The service in iLearn is delivered as promised SIQ6 Satisfaction Level (Y4) Overall, iLearn is satisfying to use SL1 Based on the WebQual 4.0 variable, a hypothetical path diagram of the measurement and structural model will be formed. The measurement model equation, in general, is written as in equations (1) and (2) (Hancock & Mueller, 2013). Then the parameter estimation for the measurement and structural equations will be made using the PLS-SEM algorithm through the SmartPLS 3.0 software. The PLS-SEM algorithm includes two stages as follows  : Stage 1: Estimating the latent variable value.
The latent variable value will be estimated by iteration. One iteration consists of four steps. Iteration is carried out until it reaches the convergence limit.
Step 1.1: Outer Approximation : initial estimation value vector of latent variable Yi that related with latent variable Yj eij : inner weight value vector, which a correlation of related Yi and Yj m : the number of Yi that related to Yj Step 1.3: Updating Outer Weight Outer weight (̃jh) estimation consists of two, outer weight estimation for reflective models (6) and outer weight estimation for formative models (7).
Step (8) ̃jh s and ̃jh s-1 is the outer weight estimation of h-th indicator on the j-th latent variable at the s-th and (s-1)-th iteration. If the outer weight value has met the convergence limit (8), the iteration is stopped. Furthermore, the estimated value of each latent variable is obtained as in equation (9).
(9) If it does not meet the convergence limit, the iteration is repeated from Step 1.1 to Step 1.4 until it meets the convergence limit.
Stage 2: Estimating the outer loading value and the path coefficient. Estimating the outer loading value of the h-indicator on the j-latent variable is done by looking for the correlation of latent variables and indicators as in (10).
(10) In the structural model, the path coefficient value is estimated using Ordinary Least Squares (OLS) by minimizing the sum squares of the residuals. The general equation of the structural model with the endogenous latent variable Yj and the exogenous latent variable Yi is written as in (11).
where β is the path coefficient vector, and ζ is the residual vector. Path coefficient estimation for Yj is presented in (12).
Significant indicators and variables will be determined by the evaluation criteria for the measurement and structural models from the estimation results. The reflective measurement model was evaluated using indicator reliability, internal consistency reliability (composite reliability value ≥ 0.7), convergent validity (AVE ≥ 0.5), and discriminant validity (based on cross loadings). In indicator reliability evaluation, an indicator with outer loading above 0.70 can be maintained in the model. Outer loadings between 0.40 and 0.70 can be considered to be removed from the model if it could increase the composite reliability (ρc) and AVE above the threshold (ρc ≥ 0.7 and AVE ≥ 0.5). Outer loadings value below 0.40 can be directly removed from the model .
The evaluation of the structural model begins by paying attention to the VIF value to see the existence of collinearity between latent predictor variables in the structural model. If the VIF value is greater than 5, then the latent predictor variable can be considered deletion from the model. Furthermore, the structural model is evaluated using the path coefficient criteria to find out the strength of the relationship and the level of significance between paths in the structural model structural. The significance level can be seen through the P-value. The smaller the P-value, the higher the significance level of a path on the structural model. On the other hand, the higher the P-value, the lower the significance level of a path in the structural model. Then, evaluate the value of the coefficient of determination (R 2 ) to determine the percentage of the variance of the endogenous latent variable that can be explained by the latent variable exogenous. Then, evaluate the criteria for effect size (f 2 ) to determine how much influence the exogenous latent variable has against endogenous latent variables in the model. f 2 < 0.02 states a very small effect of relationship, 0.02 ≤ f 2 < 0.15 states a small effect, 0.15 ≤ f 2 < 0.35 states a moderate effect, and f 2 ≥ 0.35 states a large effect .

C. RESULT AND DISCUSSION 1. Hypothetical Model
Model estimation and evaluation will be carried out simultaneously based on the hypothetical model path diagram, which can be seen in Figure 1. Evaluation of the hypothetical model is divided into two, measurement model evaluation and structural model evaluation. Based on Figure 1, there are three exogenous latent variables, Usability (with indicators U1, U2, U3, U4, U5 U6, U7, and U8), Information Quality (with indicators IQ1, IQ2, IQ3, IQ4, IQ5, IQ6, and IQ7) and the Service Interaction Quality (with indicators SIQ1, SIQ2, SIQ3, SIQ4, SIQ5, and SIQ6). Each exogenous latent variable is related to its indicator reflectively. One endogenous latent variable, namely the Satisfaction Level variable (with indicator SL1) connected directly to the exogenous latent variables Usability, Information Quality, and Service Interaction Quality, as shown in Figure 1.

Measurement Model Evaluation
The following is measurement model evaluation as shown in Table 2. loading of U5, IQ2, SIQ3, and SIQ4 are still above 0.40. These indicators can be considered for elimination if they can increase the internal consistency reliability calculated from the composite reliability (ρc) and convergent validity (AVE) value above the threshold (ρc ≥ 0.7 and AVE ≥ 0.5). If the value of composite reliability and AVE obtained has reached the threshold, then the indicators U5, IQ2, SIQ3, and SIQ4 can be maintained in the model.
Based on Table 2, the composite reliability of all latent variables is above 0.70. Thus, it can be concluded that latent variables have good internal consistency reliability, and the indicators can be used to measure latent variables. It also can be seen that the AVE value of all latent variables is above 0.50. Thus, it can be concluded that the latent variable is able to explain half or more of the variance of the indicators. The composite reliability and AVE values of all latent variables are above the threshold (ρc ≥ 0.7 and AVE ≥ 0.5). Therefore, the indicators U5, IQ2, SIQ3 and SIQ4 can be maintained in the model, as shown in Table 3.
The measurement model equation can be presented in a path diagram, the following is Path diagram of the analysis result as shown in Figure 2.

Structural Model Evaluation
The following is Structural Model Evaluation as shown in Table 4. From Table 4, it can be seen that the VIF value of all latent variables is below 5. Thus all latent variables can be maintained in the model. The path coefficient obtained from the relationship between Usability, Information Quality, Service Interaction Quality and Satisfaction Levels is positive, so it can be concluded that the three variables positively affect the Satisfaction Level. The P-value of Usability and Service Interaction Quality on the Satisfaction Level is 0.001. Thus it can be concluded that the variables of Usability and Service Interaction Quality have a high significance level on Satisfaction Level. The P-value of Information Quality on the Satisfaction Level is 0.420. Thus, it can be concluded that the Information Quality and satisfaction level have a low level of significance.
The effect size (f 2 ) of the Usability and Service Interaction Quality is in the interval of 0.02 ≤ f 2 < 0.15. It can be concluded that Usability and Service Interaction Quality has a small effect on the Satisfaction Level. The effect size (f 2 ) of the Information Quality is smaller than 0.02, so it can be concluded that the Information Quality has a very small effect on the Satisfaction Level, as shown in Table 5. Based on Table 5, the R 2 value obtained is 0.607. This indicates that 60.7% of the variance of the satisfaction level is explained by the Usability, Information Quality, and Service Interaction Quality, and the remaining 39.3% is affected by other variables outside the research model. In other words, the accuracy of the structural model in estimating the level of students' satisfaction with iLearn quality is 60.7%. Based on the analysis results, the structural equation model is obtained as in (35).
The three variables in the model have a positive influence on the level of satisfaction. It means, the higher the quality of Usability, Information Quality, and Service Interactions Quality at iLearn, the higher students' satisfaction level in accessing the iLearn website. iLearn as an LMS in Andalas University, plays an important role in the learning process in the era of the COVID-19 pandemic. However, even after the pandemic ends, iLearn will still be used to support the learning process. Therefore, it is important to evaluate the iLearn quality on student satisfaction. This study obtained the variable's significance in influencing student satisfaction towards iLearn. The analysis resulted that the Information Quality variable has a low significance on Student Satisfaction Level. This is in line with (Back et al., 2016), where the performance of Information Quality does not require much attention to improve the quality of an LMS in order to fulfil student satisfaction levels. Subsequently, this study also found that Usability and Service Interaction Quality significantly affected student satisfaction in accessing iLearn. As in (Effendy et al., 2021;Rismayani & Soetikno, 2020), it is important to improve and evaluate Usability and Service Interaction Quality indicators because these variables play an essential role in influencing student satisfaction with LMS and online learning.
Thus, Andalas University as an educational service provider, can prioritize improving the quality of iLearn services in the aspects of Usability and Service Interaction Quality. This improvement can be made by paying attention to the indicators that measure these variables. Thus, it is expected that the improvement of services for these two aspects can enhance the quality of iLearn in supporting the learning process.

D. CONCLUSION AND SUGGESTIONS
Based on data analysis using the PLS-SEM method, the structural equation model for students' satisfaction level obtained is: Y4 = 0.391Y1 + 0.100Y2 + 0.347Y3 + ζ From this model, it can be seen that the variables that significantly affect the students' satisfaction level (Y4) are Usability (Y1) and Service Interaction Quality (Y3). In contrast, the Information Quality variable (Y2) has a low significance in influencing students' satisfaction on iLearn quality. Thus, reviewing the indicators that measure the iLearn quality could increase students' satisfaction in accessing iLearn. This will lead to optimal online learning in the current era of the COVID-19 pandemic. In addition, from the R 2 , it can be concluded that 60.7% of the variance of the Satisfaction Level is explained by the Usability, Information Quality, and Service Interaction Quality variables. The remaining 39.3% is explained by other variables outside the research. For further research, other indicators or variables that affect the quality of iLearn can be added to obtain a more accurate model.