Validation of depression, anxiety and stress scales (DASS-21): Immediate psychological responses of students in the e-learning environment

The COVID-19 epidemic has caused higher education institutions in Vietnam to immediately transfer from traditional classrooms to e-learning environments. This interacts with high expectations and habits of learning and training can adversely affect the mental health of students. The purpose of this study is to validate the DASS-21 scale for use in the mental health screening in Vietnamese students when they suffer an immediate psychological reaction in the e-learning environment. Strict statistical analyzes (including Cronbach's alpha, Exploratory Factor Analysis, Confirmatory Factor Analysis, Average Variance Extracted, Average Shared Variance) have led to a well-fitting model of DASS-18 with a three-factor structure to measure the mental health of Vietnamese students in an e-learning environment. Results DASS-18 reported the rates of depression, anxiety, and stress in levels of moderate severity or above in Vietnamese students at 50%, 19.7%, and 37.3%, respectively. However, a rate of anxiety up to 43.1% by using DASS-21 indicating that many students may be misdiagnosed for the level of anxiety. Finally, linear regression analyses are used to examine the influence of socio-demographic factors on the immediate psychological responses of students to an e-learning environment in the context of the COVID-19 epidemic.


Introduction
The 2019 coronavirus disease (COVID-19) is a public health emergency of international concern (Wang et al., 2020). The COVID-19 epidemic has greatly affected every aspect of social life and business production. In Vietnam, in an unprecedented move to prevent the spread of the virus, the government closed all schools in the country from February 3, 2020 to May 3, 2020. With the policy of the Ministry of Education and Training is "pausing to school, but not stopping learning", e-learning, television-based learning becomes the optimal solution when COVID-19 epidemic can be prolonged and complicated. At the primary and secondary education level, television-based learning has been used to maintain students' knowledge, while waiting to return to school. At the higher education level, e-learning is applied to the maximum extent possible in the formal university courses. On March 23, 2020, the Ministry of Education and Training recognized all results of online teaching and learning by higher education institutions, while previously only allowed 30% of the training program. However, the instant transition of higher education from traditional classrooms to e-learning, they interact with high expectations and habits of learning and training that can adversely affect the mental health of students (Kulsoom & Afsar, 2015). Besides, the academic context in the COVID-19 epidemic could further exacerbate the mental health of students.
struggles that can further exacerbate depression, anxiety and stress (Andrews & Wilding, 2004). Therefore, these factors are used by us to explain the depression, anxiety and stress of Vietnamese university students in an e-learning environment. Currently, to the knowledge of researchers, there is no information on mental health of Vietnamese university students when they suffer an immediate psychological reaction in the e-learning environment with the context of COVID-19.
One of the commonly used scales for the detection of mental health problems is the Depression, Anxiety, and Stress Scale -DASS (P. Lovibond & S. Lovibond, 1995). The reliability and validity of DASS and its short-form-the DASS-21 have been widely recognized to assess depression, anxiety, and stress among adults (Le et al., 2017). Because DASS and DASS-21 are developed in Australia and applied to different socio-cultural contexts and types of populations, they need to be carefully considered (Le et al., 2017). Much evidences have been found showing that DASS-21 may not be valid when applied in several different contexts (González-Rivera, Pagá n-Torres & Pé rez-Torres, 2020). Currently, to the knowledge of researchers, no research paper has examined the reliability, validity, and factorial structure of DASS-21 for university students in an e-learning environment. In the Vietnamese context, DASS-21 has been confirmed to be reliable and suitable for use to assess symptoms of common mental health problems, especially depression and anxiety among Vietnamese adolescents, but its ability in detecting stress may be limited (Le et al., 2017). However, university students do not appear in previous research samples in Vietnam (Le et al., 2017;T.D. Tran, T. Tran & Fisher, 2013). This increased doubts about the validity of the DASS-21 among university students, at least in Vietnam.
The main purpose of this study is to validate the DASS-21 scale for use in the mental health screening in Vietnamese students when they suffer an immediate psychological reaction in the e-learning environment of the COVID-19 outbreak. Specifically, this study had three main objectives: -To assess whether the original DASS-21 is a good model for measuring the mental health of Vietnamese students in an e-learning environment.
-To investigate the mental health of Vietnamese students in the first two weeks, they are immediately transferred from the traditional classroom to the e-learning environment of the COVID-19 outbreak.
-To determine how socio-demographic factors affect the mental health of Vietnamese students in the e-learning environment.

Materials Information
This research was conducted at Hanoi University of Science and Technology (HUST), Vietnam. HUST is one of the most prestigious universities in Vietnam, top 1.000 rankings of Times Higher Education 2019. In 2010, HUST became a member of the ACU (ASEAN Cyber University) project to transform from traditional learning to e-learning. In 2012, HUST started implementing the first e-learning courses through the link "http://lms.hust.edu.vn". By the first half of 2019, only 25 e-learning courses are developed, and many of them are elective. Although e-learning exists for a relatively long time, but it still seems to be in its infancy. However, when the government closed all schools due to the COVID-19 epidemic, e-learning actually played a prominent role at HUST. All traditional classroom-based activities at HUST are instantly transformed into an e-learning environment based on the "Microsoft Teams" platform. Although HUST has strived to support its faculty and students by providing instructional documents, online videos, and online training courses on e-learning lectures. In the initial stage, both faculty and students feel pressured to teach and learn in an e-learning environment. Four weeks later, a midterm exam in the traditional classroom was also transformed into an e-learning environment. This further exacerbates the level of depression, anxiety, and stress among students.

Design
This was a study of instrument design. For 2 weeks from March 6 to March 20, 2020, we conducted an online survey using convenience sampling techniques. The data collection was conducted using online questionnaires through the Microsoft Forms platform. A total of 15 online classes (about 2000 students) was selected to collect survey information.

Measurement
Participants' socio-demographic characteristics, including gender, residential area, grade point average (GPA), and residence status were assessed using study-specific questions. A yes / no question was used to test students' attendance to learning in an e-learning environment. If the students' response is "Yes", they will continue to answer with DASS-21. While other students will not answer anything more. In other words, this helps us eliminate online truancy students in the collected data.
The DASS-21 is the shortened version of the DASS developed by P. Lovibond and S. Lovibond (1995) which was used. The three-dimensional self-reporting scales assess the presence and intensity of affective states of depression, anxiety, and stress in the last week on a 4-point Likert response scale. Each subscale has seven items and its total score ranges from 0 to 21 points. A higher score indicates higher symptomatology of depression, anxiety, and stress. We used the Vietnamese version of DASS-21 which was validated by Tran et al. (2013). Three subscales of DASS-21 are named DASS-21-Depression (DASS-21-D), DASS-21-Anxiety (DASS-21-A), and Stress (DASS-21-S).

Data Analysis
The task of data analysis was performed in SPSS. In the first step, Cronbach alpha statistics were conducted to assess reliability in terms of internal consistency. A cut-off point of 0.7 of Cronbach's alpha value actually reflects internal consistency (Cortina, 1993). The Exploratory Factor Analysis (EFA) was conducted for the 21 items of the DASS-21 to analyze the factorial structure. We used the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy and Bartlett's test of sphericity. A cut-off point of 0.45 of KMO values was used to select items for a factor (Cohen, 2013). Then, Cronbach alpha statistics were conducted to assess reliability in terms of internal consistency in each factorial structure.
When EFA and Cronbach alpha statistics were statistically significant, the Confirmatory Factor Analysis (CFA) in AMOS version 20 software was conducted to examine the fit of the DASS-21. In the Chi-Square test (χ2), the model fit is acceptable if p> 0.05. However, a p-value is usually less than 0.05 due to the χ2 value being sensitive to sample size. This would lead to a model fit being rejected if p < 0.05 (Anderson & Gerbing, 1988). Thus, model fit was assessed using a guideline for determining model fit is developed by Hooper, Coughlan and Mullen (2008), including: a ratio of χ2/df as high as 5.0; RMSEA value ranging from 0.05 to 0.07; a higher cut-off of 0.9 for GFI, NFI and AGFI; SRMR values as high as 0.08; a higher cut-off of 0.95 for CFI and TLI. The convergent and discriminatory validity of DASS-21 were examined through the Average Variance Extracted (AVE). The AVE must be equal to or greater than 0.50 for determining the convergent validity. In order to determine the discriminatory validity of each dimension, the Maximum Shared Variance (MSV) and the Average Shared Variance (ASV) must be less than the value obtained from the AVE of each factor. In addition, the correlation between the instrument factors was calculated using Pearson's coefficient (r). Values less than 0.35 were considered to be low correlations, values between 0.36 and 0.67 were considered moderate correlations, values between 0.68 and 0.89 were considered high correlations and values from 0.90 onwards were considered to be very high correlations (Taylor, 1990).
Finally, linear regression analyses were used to examine the influence of socio-demographic factors on the immediate psychological responses of students to an e-learning environment in the context of the COVID-19 epidemic.

Exploratory Factor Analysis
After two weeks of online surveys, we received 304 questionnaires with complete data, and there were no questions left blank. The Cronbach alpha test results in a DASS-21 value of 0.909, a DASS-21-D value of 0.882, a DASS-21-A value of 0.807, a DASS-21-S value of 0.842. Cronbach alpha values were greater than 0.7 in all cases indicating that all collected data achieved internal consistency within a large group and subgroups.
An EFA was conducted to examine the factor loading of DASS-21 with a sample size of 304 university students in Northern Vietnam (Table 1). They achieved a KMO value of 0.913 (greater than 0.45), a p-value of 0.00 (less than 0.05) so the rotated factor loading matrix was statistically significant. The results of Total Variance Explained result an Eigenvalues values in the "Total" column of 1.062 (greater than 1) and a Cumulative value of 59.16% (greater than 50% of total variance). In other words, 21 original items explained to 59.16% of the total variance of the new factors. The result of the rotated factor loading matrix indicates that the 21 original items loaded for the four new factors. In all cases, factor loading values were greater than 0.5 and no items loaded more than two new factors. The original items loaded correctly to DASS-21-D and DASS-21-S by P. Lovibond and S. Lovibond (1995). However, the items on the DASS-21-A scale loaded for two new factors, which are formed from a group of physical items (A2, A4, A7) and a group of mental items (A9, A15, A19, A20). The results of Cronbach alpha for four new factors show that items A2, A4 and A7 of the DASS-21-A scale were deleted, because their alpha value is less than 0.7. The result of Cronbach's alpha for the remaining 18 items results in an alpha value of 0.908 (greater than 0.7) showing that deleted items do not really contribute to the reliability of the DASS scale.

Confirmatory Factor Analysis
In AMOS, the Confirmatory Factor Analysis (CFA) was conducted to examine the fit of a new DASS-18 model. But a good model fit of DASS-18 was achieved when covariance links were added between error variances of D10 and D17, D10 and D21, D17 and D21, S6 and S18, and S8 and S12 (Figure 1) In all cases, the Pearson's correlation coefficients between DASS-18-D, DASS-18-A, and DASS-18-S was a moderate level (lower than 0.68) indicating that the scales of stress, depression, and anxiety were moderately discriminatory. AVE calculations show that the depression, and anxiety scales achieved convergence effect (with the AVE of anxiety is 0.550; the AVE of depression was 0.504, greater than 0.5). But the AVE of the stress scale was 0.424 (less than 0.5), showing that its ability to detect stress is limited in Vietnamese university students.   Table 2 describes the depression, anxiety, and stress status of Vietnamese university students in the early stages of learning in an e-learning environment taking place in the context of COVID-19. More specifically, 13.5% of students have depression from severe or above; 13.1% of students have anxiety from severe or above, and 17.1% of students have stress from severe or above. If using the anxiety scale of DASS-21, 26.3% of students could have anxiety from severe or above. In other words, some students may not have anxiety, they may be diagnosed with anxiety if we use DASS-21. High rates of depression, anxiety, and stress in levels of moderate severity or above at students were 50%, 19.7%, and 37.3%, respectively.

Socio-demographic Factors and Psychological Impact
The relationship between socio-demographic factors and psychological impact was shown in Table 3. Of the total respondents, 71.4% of the students were male, 28.6% were female, and they were mostly from rural areas (69.7%). During the implementation of the social distancing to prevent COVID-19, the majority of students stayed in their family home to study online (76%). The student's GPA was mostly distributed at the "Good" level.
Other sociodemographic factors were not associated with DASS-18 subscale scores.

Discussion
Some previous studies show that the linguistic and cultural factors of each country have a significant effect on the validity of the DASS 21 scale as they are translated and applied (Gonzá lez-Rivera et al., 2020). This led us to doubt about the validity of the DASS-21 scale when Vietnamese students faced e-learning culture in the context of the COVID-19 epidemic. The findings show that the DASS 21 scale is ineffective in measuring the mental health of Vietnamese students in the e-learning environment. Strict statistics have led to a well-fitting model of DASS-18 with a three-factor structure to measure the mental health of Vietnamese students. This resulted in the removal of 3 items (A2, A4, A7) from the anxiety scale of the original DASS-21 by P. Lovibond and S. Lovibond (1995). This result is different from the studies of Le et al. (2017) and Tran et al. (2013) when they confirmed that the DASS-21 scale is valid for adolescents, and pregnant or postpartum women in Vietnam. From there, we question why the DASS-21 scale is not valid for Vietnamese students, at least in an E-learning environment? Similar to the findings of this study, several previous studies also showed many other versions of the original DASS-21 scale, such as DASS-20, DASS-18, DASS-17, DASS-12, DASS-9 (Table 4). Most of these versions result in a three-factor structure, but there is no similarity between items removed from the original DASS-21.  (2013) Three-factor Medical students in Malaysia D3, D13, D17, D21, S1, S8, S18, S12; A2, A4, A19, A20 a Order of items on DASS-21 scale by P. Lovibond & S. Lovibond (1995) Data from a cross-sectional survey presented a picture of anxiety, depression, and stress among students who were immediately confronted with an e-learning environment in the context of the COVID epidemic. The findings showed that the anxiety factor has the lowest rate (19.7%) among students in an e-learning environment, which is in contrast to the highest rate (47%) of anxiety among general university students reported by Bayram and Bilgel (2008). Similarly, the anxiety factor among students in an e-learning environment was also strongly contrasted with a very high anxiety rate (up to 54.5%) of prospective medical students by Yusoff et al. (2013), and up to 55% of medical students by Teh, Ngo, Binti Zulkifli, Vellasamy and Suresh (2015). The highest rate of depression (up to 50%) among students in e-learning environments exceeds the average rate of 30.6% (Ibrahim, Kelly, Adams & Glazebrook, 2013). All of these rates are calculated from moderate severity or above. In general, the immediate psychological responses of students to an e-learning environment in the context of the COVID-19 epidemic indicate that a rate of anxiety among students was low, however, the rate of depression and stress is very high. The findings of this study also showed that there was a significant difference in the rate of anxiety among students through the use of the DASS-18 and DASS-21 scales. We were surprised that an anxiety rate among students was significantly increased from 19.7% of using DASS-18 to 43.1% of using DASS-21. In other words, many students may be misdiagnosed for the level of anxiety when using DASS-21. This implies that the validity of the DASS-21 scale needs to be clearly verified before they can be applied in each specific context. The analysis of the relationship between socio-demographic factors and psychological impact shows that female students are significantly associated with a higher level of anxiety. This result only partially supports the results of Dyrbye et al. (2006) when they suggested that female students have a higher level of all three DASS subscale. Residential factors are shown to be unrelated to DASS-18 scores of students with an e-learning environment in the context of the COVID-19 epidemic. They differ from those reported by Bayram and Bilgel (2008) when they assert that students from rural areas are associated with higher levels of depression, anxiety, and stress. Besides, living with relatives and friends is associated with a lower level of stress for students with an e-learning environment, which have not been found in previous studies. GPA is also reported to be unrelated to the mental health of students in an e-learning environment.
In general, this study contributes to the existing DASS literature a new DASS-18 model with three-factor structures for measuring the mental health of Vietnamese students in an e-learning environment. The findings also show that there are serious shortcomings if using the DASS-21 scale to test students' anxiety for an e-learning environment in the context of the COVID-19 epidemic.