Reliability and validity of a post-traumatic checklist-5 (PCL-5) among fire and rescue officers in Selangor, Malaysia Open Access

Firefighters are emergency workers who are repeatedly exposed to traumatic events during the course of their services. Their roles are characterized by high levels of work demands and routine exposure to both physical and psychological stressors [1]. A recently published meta-analysis of the worldwide current prevalence of post-traumatic stress disorder (PTSD) in rescue workers yielded a prevalence of approximately 10% for PTSD, suggesting about 150,310 first responders may meet criteria for current PTSD, and of that, 7% of current firefighters suffer from PTSD [2]. A different study undertaken in Australia among current firefighters and retired firefighters showed a PTSD prevalence of 8% and 18% respectively [3].

In the Malaysian setting, firefighters are involved in various tasks, some of which may include traumatic events or critical incidents. They are at the front line for emergency cases and various rescue activities such as Emergency Medical Rescue Services (EMRS), Hazardous Material Unit Teams (HAZMAT), road traffics accidents, high-rise rescues, cliff rescues and water rescues. These high-risk tasks, for the most part, involve experiencing or witnessing critical incidents or traumatic events, which will carry the risk of developing psychiatric illnesses, particularly PTSD.

The Clinician-Administered PTSD Scale for DSM-V (CAPS-5) is currently the gold standard in PTSD assessment by psychiatrists; however, it is labor-intensive and the assessor requires rigorous training. Nevertheless, there are many self-reported tools available for screening purposes, which adopt fewer items and take up less time. Recently, the American Psychiatric Association (APA) developed PCL-5, which was adapted from the previous PCL-C and -M (DSM-IV) in 2013. There is a growing number in validation studies for this tool, yet, to date, there are limited reliability and validity studies across different populations. Therefore, this study aimed to establish reliability, construct validity, and concurrent validity for the Malay version of the PCL-5 (MPCL-5) among fire and rescue officers in the state of Selangor, Malaysia.

Five observations were needed for each item to be applied before the sample size could be determined [4]. Therefore, 100 fire and rescue officers in Selangor, who were from operational teams with at least 1-month duration of service, were conveniently selected.

All demographic variables were collected, including age, gender, ethnicity, marital status and household income. Additional questions regarding the duration of service were also surveyed.

The self-administered Malay language version of the PCL-5 (MPCL-5) was used to measure the current status of probable PTSD, which was translated and validated by a previous researcher in a different study [5]. The PCL-5 consists of 20 items corresponding to the DSM-V for post-traumatic stress disorder (PTSD) symptoms. Each item has a five-point rating scale ranging from 0 (not at all) to 4 (extremely). Thus, the scale yields a cumulative score of 0–80. A score of 33 and above is taken as the cut-off point for positive signs of PTSD [6]. The PCL-5 consists of four clusters of symptoms, represented by item questions 1–5 (A: Intrusion), item questions 6–7 (B: Avoidance), item questions 8–14 (C: Negative states), and item questions 15–20 (D: Arousal).

The Trauma Screen Questionnaires (TSQ) is a 10-item symptom screen that was designed for screening PTSD, particularly among survivors of all types of traumatic stress. The TSQ is derived from the PTSD Symptom Scale–Self Report (PSS-SR), based on DSM-IV [7], and has five re-experiencing items and five arousal items. The cut-off score of TSQ-M used is any 5 out of 10 symptoms is taken as positive for PTSD, as it has optimum sensitivity and specificity [8].

This study was conducted under the ethical approval of the Research and Ethics Committee, Medical Faculty of National University of Malaysia (FF-495-2017).

Reliability was evaluated by testing the overall Cronbach's alpha and the individual cluster of the four clusters of PCL-5. Construct validity was examined through principal component factor analysis with varimax rotation. Kaiser-Meyer-Olkin (KMO) and Bartlett's test of sphericity were used to test for the appropriateness of factor analysis. In this study, the sample was adequate, as the KMO value was 0.91. The eigenvalue greater than 1 was used to determine the number of factors to be extracted while loading factor values of 0.4 and below were suppressed. Concurrent validity was established by analyzing the PCL-5 Malay version with a validated TSQ-M to produce the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). In this study, a translated Malay version of TSQ (TSQ-M) was used as the gold standard, where it had been validated in a local sample population and other studies with good reliability and validity [9–13] and yielded high levels of sensitivity and specificity compared to those of the CAPS diagnostic interview [14]. Convergent validity was tested using the Pearson correlation between the total scores of the two instruments. All data were analyzed using the Statistical Package for Social Sciences (SPSS) version 20.

Table 1 shows the characteristics of participants. The mean age of participants was 33.3 (±7.9) and 10.8 (±7.8) for years of service. The majority of them were male 96 (96%), of Malay ethnicity 93 (93%), who were single 74 (74%) and with a household income between 2001 and 3000 RM, or Ringgit Malaysia, the national currency.

The frequency of PTSD diagnosis made using the two instruments was lower in TSQ-M. It was found that the diagnosis of PTSD was 42% (n = 42) of the sample when using MPCL-5 compared to only about 16% (n = 16) when using TSQ-M.

Generally, MPCL-5 showed a strong internal consistency, with a Cronbach's alpha value of 0.96. Itemized analysis of the different clusters' symptoms also revealed that the Cronbach's Alpha values in all domains ranged from 0.83 to 0.93 (Table 2).

Table 3 showed total items statistics. Overall, the items indicated a satisfactory correlation with the corrected items' total score. Lowest values were seen in items A5 and D16. Cronbach's alpha values of deleted items range from 0.956 to 0.959.

Factor analysis showed four-factor components as shown in Table 4. The loading factors were acceptable, ranging from 0.462 to 0.875, and 76.9% of the variance was explained. The scree plot showed that four components had eigenvalue of at least one or more (Figure 1).

Receiver operating characteristic (ROC) (Figure 2) analyses were applied to compare the screening performance of the MPCL-5 by using TSQ-M as the validated test for PTSD. The cut-off score of 5 was used for TSQ-M for positive PTSD symptoms. The area under the curve was 0.84, p < 0.001 with a 95% confidence interval between 0.74 and 0.95, substantially above the random ROC.

Convergent validity was assessed using Pearson's correlation where it had indicated a relatively strong positive correlation between MPCL-5 and TSQ-M scores (r = 0.635).

Table 5 shows that the cut-off score of 33 was found to be sound with sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), which were 0.81, 0.65, 0.31, and 0.94 respectively. However, it was observed that for the cut-off score of 35, the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) slightly improved with values of 0.81, 0.71, 0.35 and 0.95 respectively. A chi-square test was performed to assess the association between individual items of PCL5 and PTSD diagnosis. All items were statistically significant.

In this study, the Malay version of post-traumatic stress checklist-5 (MPCL-5) generally showed good reliability with the evidence of high internal consistency for items and its components respectively. The construct validity effectually illustrated four-factor components with acceptable loading factors. However, concurrent validity reported an acceptable value of high sensitivity and relatively low specificity. Correlation between the total score between the two instruments showed good convergent validity.

The MPCL-5 showed a high internal consistency as reported with previous studies [6, 15, 16]. A recent validation study for post-traumatic stress checklist-5 (PCL-5) among adults was completed in Malaysia showing a high internal consistency. The study involved adult patients in orthopedic wards and adolescents, with Cronbach's Alpha values of 0.889 and 0.91, respectively [5, 17]. The Exploratory Factor Analysis (EFA) showed four factors with the original PCL-5, which correspond to the 4 clusters of symptoms. Several validation studies in different populations have also reported the four-factors model [5, 6, 16–18].

The MPCL-5 showed a relatively high value of sensitivity (81.25) with a lower value for specificity (65.48) which makes it appropriate for use as a screening tool for those who have post-traumatic stress disorder (PTSD) symptoms. Further testing should be carried out amongst those with probable PTSD by using the Clinician-Administered PTSD Scale (CAPS) via a structured interview by a trained clinician or psychiatrist to properly diagnose PTSD. A highly sensitive test is most helpful to the clinician as it will have fewer false negatives results. When the test result is negative with relatively low specificity, which means it will be falsely positive for a number of those who don't have PTSD symptoms. Conversely, a highly specific test will have fewer false positives and will be most helpful to the clinician when the test result is positive as the low sensitivity might have led to a higher false-negative result.

The sensitivity and specificity for the MPCL-5 obtained from this study are acceptable with a cut-off score of 33. However, with a higher cut-off score of 35, the sensitivity and specificity would have been slightly improved and is the optimum objective. This is very close to a study completed among veterans in the UK, where the optimum cut-off score was 34 [19]. There have been many studies that showed good cut-off points of 33 [5, 6, 15] but there are also studies with different cut-off points. For example, a study that evaluated the psychometric properties of the PCL-5 among undergraduate students who had experienced a very stressful life event found that the PCL-5 achieved a sound sensitivity with a cut-off score of 37 [6]. Several studies also recommended a wide range of cut-off scores from 30 to 60 [20, 21]. Recently, a cut-off score of 31 to 39 was recommended to predict DSM-5 PTSD diagnosis among veterans [15, 16].

Low positive predicted value (PPV) was noted possibly as a result of the relatively low prevalence [11] of PTSD by the TSQ-M. A similar trend was reported in other concurrent validations using CAPS-5, where the prevalence for PTSD was lower than that of the MPCL-5 [17]. Concurrent validation from this study is limited by using validated TSQ-M, and not the Clinician-Administered PTSD Scale-5 (CAPS-5), which is regarded as the gold standard. However, previous studies have shown an acceptable level of validity and ratability, almost as good as that of CAPS-5 [14].

This is the first validation study of the Malay version of a post-traumatic stress checklist-5 (MPCL-5) for Malaysian fire and rescue officers. This study demonstrates that MPCL-5 is a valid and reliable scale for screening probable PTSD diagnosis among those who experienced traumatic events.

The authors are grateful to the Fire and Rescue Department of Selangor, Malaysia for their assistance and support.Conflicts of Interest: None declared.