Comparison of Early Warning Scores in Determining the Prognosis of COVID-19 Patients

By Mustafa Akarca¹, Tuba Cimilli Ozturk², Tevfik Patan², Fatma Sari Dogan², Ebru Unal Akoglu²

Affiliations

1. Emergency Medicine Department, Erbaa State Hospital, Tokat, Turkiye
2. Emergency Medicine Department, Fatih Sultan Mehmet Education and Research Hospital, Istanbul, Turkiye

doi: 10.29271/jcpsp.2024.02.166

ABSTRACT
Objective: To compare the effectiveness of early warning score systems in predicting 30-day poor outcomes in Coronavirus Disease (COVID-19) patients admitted to the emergency department.
Study Design: Descriptive study.
Place and and Duration of the Study: Fatih Sultan Mehmet Education and Research Hospital, Istanbul, Turkiye, from March 2020 to March 2021.
Methodology: The patients who presented to the emergency department, diagnosed with COVID-19 and tested positive for polymerase chain reaction were analysed. The study included the calculation of the rapid emergency medicine score, risk stratification in the emergency department in acutely ill older patients score, 4C mortality score, and modified early warning score for the patients. These scores were then compared in terms of their ability to predict adverse outcomes, defined as intensive care admission and/or mortality.
Results: During the study period, 10,281 COVID-19 patients were admitted to the emergency department. Out of them, 1,826 patients were included in the study. There were 159 (8.7%) cases with poor outcomes. The risk stratification in the emergency department in acutely ill older patients Score was the most successful in poor prognosis.
Conclusion: Based on the findings of this study, the risk stratification in the emergency department in acutely ill older patients score demonstrated greater efficacy compared to other early warning scores in identifying patients diagnosed with COVID-19 who had an early indication of a poor prognosis.

Key Words: Early warning score, 4C mortality score, REMS, Rise-up score, MEWS, Emergency department, COVID-19.

INTRODUCTION

Provision of 24-hour uninterrupted healthcare is one of the essential roles of emergency departments worldwide. However, one of the biggest challenges facing these departments is the increasing number of non-emergency cases that are admitted for treatment alongside real emergency cases. This problem can be attributed to a lack of adequate primary care and outpatient clinics, which leads to patients seeking treatment in emergency departments, as well as the lack of health literacy among patients.¹

In many countries, including Turkiye, hospital admissions have been on the rise in recent years, with emergency department admissions accounting for a significant proportion of these admissions.² 

The Coronavirus Disease (COVID-19) pandemic has further highlighted the importance of managing emergency departments effectively. The fear and obscurity created by the pandemic have led to increased pressure on emergency departments as patients flock to these facilities.³ As a result, it has become even more critical to have efficient systems in place, such as early warning scores, to help identify and prioritise urgent cases. Early warning scores can be a valuable tool in identifying and prioritising truly urgent cases, helping emergency departments to manage their resources effectively and provide optimal care to patients.

Early warning scores are scoring systems based on the quantitative and rapid evaluation of changes in vital signs.⁴ Initially, they were developed to predict the need for intensive care units (ICU) among patients currently under follow-up in inpatient units, as well as to identify and monitor unstable patients to detect preventable cardiac arrests.^5,6 The rapid emergency medicine score (REMS) is a scoring system developed in 2004 to predict in-hospital mortality in non-surgical patients.⁷ The REMS, which was redesigned by adding peripheral oxygen saturation and patient age to the rapid acute physiology score (RAPS), was found to be superior to the RAPS in predicting in-hospital mortality. The modified early warning score (MEWS), on the other hand, is an estimation tool that has been used for years. Scores of five or more are associated with increased mortality and hospitalisation in the ICU.⁸  The risk stratification in the emergency department in acutely ill older patients (RISE-UP) score was tested in the Netherlands in 2019, involving 603 geriatric patients, a significant portion of the emergency department population; later publications have indicated that the RISE-UP can also be used in non-geriatric patients.⁹ The 4C mortality score was designed in 2020 following a study conducted with approximately 35,000 patients diagnosed with COVID-19 in 260 hospitals in Scotland.¹⁰

This study aimed to compare the effectiveness of early warning score systems in predicting 30-day poor outcomes (intensive care hospitalisation and mortality) in COVID-19 patients admitted to the emergency department and to examine the potential value of laboratory tests in this context.

METHODOLOGY

It was a descriptive study conducted at Fatih Sultan Mehmet Education and Research Hospital, Turkiye, between March 2020 and March 2021. The data of patients over the age of 18 years who were admitted to the emergency department, whose international classification of disease (ICD) code was entered as “U07.3 2019-nCov (Novel Coronavirus) Disease” in the preliminary or final diagnosis, and whose polymerase chain reaction (PCR) result on the relevant date was positive, were analysed. Patient data were obtained from the hospital information management system. Patients with negative test results, <18 years old, who were pregnant, or patients whose medical records lacked the necessary data to calculate scores were excluded from the study.

Demographic data, examination findings at the time of admission to the emergency department, vital parameters, history, and some laboratory values were accepted as prognostic indicators at that time (C-reactive protein, albumin, lactate dehydrogenase (LDH), total bilirubin, urea, and blood urea nitrogen) were recorded. The clinical course of patients was examined, and patients with poor outcomes (intensive care hospitalisation and mortality) were identified within 30 days. Patients who received outpatient treatment and/or were discharged after inpatient hospitalisation were considered to have a good outcome. Among the patients whose initial treatment plan or afterwards hospitalisation at the ward was outpatient, those who needed intensive care or died within 30 days were also included in the poor outcome group. The REMS, RISE-UP, 4C mortality score, and MEWS were calculated for the patients included in the study, and these scores were compared in terms of their ability to predict poor outcomes.

All statistical analyses were performed using the Statistical Package for the Social Sciences for Windows 20.0 (SPSS, Chicago, IL, USA). The conformity of the continuous variable data to normal distribution was evaluated with the Shapiro–Wilk test. When comparing the differences of continuous variables between the two groups, the student’s t-test or the Mann–Whitney U test was used based on their conformity to normal distribution. The percentage and numerical values for categorical variables of patients, the mean (standard deviation), and the median (25^th quartile to 75^th quartile) values were specified for continuous variables. Pearson’s chi-squared test was used to compare categorical variables. Receiver operating characteristic (ROC) curve analysis was performed to evaluate the effectiveness of variables derived from the scoring systems in predicting ICU requirements and mortality outcomes. The closer the area under the curve (AUC) value is to one as a result of this analysis, the more valuable the test is. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) calculations were performed for scoring systems based on the predictive value obtained from the ROC curve analysis. All data were expressed based on a 95% confidence interval (CI) and a significance level of p<0.05.

RESULTS

A total of 10,281 COVID-19 patients who were admitted to the emergency department during the study period and whose diagnosis was confirmed by polymerase chain reaction (PCR) test were detected. Since this study was retrospective, patients who did not have all the data required to calculate all four scores were excluded from the study. In addition, five patients who were brought to the emergency department with cardiopulmonary arrest and were considered dead were excluded from the study. A total of 1,826 patients were included in the study. The patient flow chart is presented in Figure 1.

To calculate the power of the study, the results of the current data and the results of the post-hoc analysis based on the number of patients were evaluated. The authors calculated the strength of the study using the mean and SD data of the 4C mortality score from two independent patient groups (patients ending in mortality and other patients). The number of patients who ended in mortality was 110, while the number of other patients was 1.716, and the mean 4C mortality score of these groups was 12.16 (4.36) and 4.84 (3.61), respectively. As a result of the post-hoc analysis performed on these data, the calculated power of the study was 100%, while the type I patient (alpha) was 0.01. When the same calculation was made considering the mean MEWS and mean REMS of these two patient groups, the power analysis results did not change.

Figure  1:  Patient  flow  chart  for  the  study.

Table I: Distribution of data by poor and good outcomes.

	Poor outcome (n=159)	Good outcome (n=1667)	p-values**
Age	70.87±13.81	49.79±18.09	<0.001
Gender
Female	79 (49.7%)	858 (51.5%)	0.667
Male	80 (50.3%)	809 (48.5%)
Comorbidity	126 (79.2%)	652 (39.1%)	<0.001
Vital parameters
SBP (mmHg)	133.15±26.14	128.66±18.83	0.006
DBP (mmHg)	73.02±12.71	74.93±11.59	0.049
Fever (°C)	36.70 [36.40–37.00]	36.50 [36.2–36.70]	<0.001
Pulse (beats/min)	96.36±18.92	92.66±17.03	0.010
SpO2 (%)	93 [88–96]	97 [96–98]	<0.001
RR (breaths/minute)	18 [15–24]	15 [13–17]	<0.001
MAP (mmHg)	92.39±16.01	92.82±12.44	0.683
Laboratory values
Albumin (g/L; normal range 35–52)	37.04±5.48	43.30±4.29	<0.001
CRP (mg/L; normal range 0–3)	7.86 [1.63–17.43]	0.65 [0.00–3.11]	<0.001
LDH (U/L; normal range 135–225)	307 [230–439]	200[171–248]	<0.001
BUN (mmol/L; normal range 2,1–7,1)	0.83[0.62–1.58]	0.63[0.57–0.76]	<0.001
BUN = Blood urea nitrogen; CRP = C-reactive protein; GCS = Glasgow coma score; DBP = Diastolic blood pressure; LDH = Lactate dehydrogenase; MAP = Mean arterial pressure; SBP = Systolic blood pressure; SpO2 = Oxygen saturation; RR = Respiratory rate. *Data are expressed as n (percent), mean±standard deviation, median [25th quantile-75th quantile]. **When giving p-values, the student test was used for continuous variables with normal distribution, and the Mann–Whitney U test was used for non-normal distribution. Pearson’s chi-squared test was used to compare categorical data.

Table II: Sensitivity, specificity, positive and negative predictive values of 4C mortality score, RISE-UP, MEWS, and REMS in predicting an ICU admission and/or mortality outcome.

Scoring	ROC AUC	Cut-off value	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	p-value
4C Mortality Score	0.876 (0.848-0.903)	≥8	82	77	25	98	p<0.001
RISE-UP	0.881 (0.856-0.906)	≥8	72	83	29	97	p<0.001
MEWS	0.715 (0.669-0.761)	≥2	64	72	18	95	p<0.001
REMS	0.850 (0.824-0.875)	≥5	83	69	20	98	p<0.001
4C = Coronavirus clinical characterisation Consortium; AUC = Area under curve; MEWS = Modified early warning score; NPV = Negative predictive value; PPV = Positive predictive value; REMS = Rapid emergency medicine score; RISE-UP = Risk stratification in the emergency department in acutely ill older patients; ROC = Receiver operating characteristic.

Figure 2. ROC curve of 4C mortality score, RISE-UP, MEWS, and REMS to predict intensive care admission and/or mortality status of COVID-19 patients.

Of the patients included in the study, 937 (51.3%) were females and 889 (48.7%) were males. The ages of all patients were between 18 and 95, with a mean age of 51.62 (18.71) years. A total of 778 patients (42.6%) had at least one comorbid disease (liver disease, diabetes, chronic renal disease (estimated GFR <30), chronic heart disease, dementia, chronic neurological diseases, connective tissue diseases, HIV, malignancy, and chronic pulmonary disease except for asthma) were present. The distribution of demographic data, vital parameters, and laboratory data of the patients according to their outcomes are presented in Table I.

The cut-off values of the 4C mortality score, RISE-UP, MEWS, and REMS scoring systems were calculated to predict the clinical poor outcome of patients, and their sensitivity, specificity, PPV, and NPV values were calculated.

The total number of patients with poor outcomes (intensive care admission and/or mortality) in this study was 159 (8.7%). In Table II, the cut-off values of the 4C mortality score, RISE-UP, MEWS, and REMS scoring systems in predicting a clinically poor outcome of these patients and their sensitivity, specificity, PPV, and NPV values are described. The ROC curve analysis used to determine these values is illustrated in Figure 2.

When the data were evaluated separately for intensive care hospitalisation and mortality, the RISE-UP score was the most successful in predicting both intensive care admission and mortality with AUC: 0.874 (95% CI: 0.847–0.901) and AUC: 0.907 (95% CI: 0.884–0.930) respectively. The 4C mortality score was calculated as AUC: 0.873 (95% CI: 0.845-0.902) and AUC: 0.889 (95% CI: 0.856-0.923), respectively, in terms of ICU admission and/or mortality outcome, making it the second-best performing scoring system.

DISCUSSION

In this study, the performances of the 4C mortality score, RISE-UP, MEWS, and REMS scoring systems were compared in predicting the clinical poor outcome in patients diagnosed with COVID-19. The RISE-UP score was the most successful score in predicting ICU admission and/or mortality, with AUC: 0.881 (95% CI: 0.856-0.906). The 4C mortality score was calculated as 0.876 (95% CI: 0.848-0.903), making it the second-best performing scoring system.

When the demographic data of this study group were examined, the authors found that the mean age of all patients was 51.62 (18.71) years (min:18–max:95), while the mean age of 110 patients with mortality was 72.66 years. While the mean age was 70.87 (13.81) years in all patients in the poor outcome group, it was 49.79 (18.09) years in the good outcome group, and there was a significant correlation between increasing age and poor outcome. Similar to the results of this study, in Pepe et al.'s study of 5.746 patients, it was found that the rate of unstable clinical presentation increased with increasing age. In that study, the mortality rate in the group below 65 years of age was 6.8%, while the in-hospital mortality rate above the age of 65 was found to be 32%.¹¹ Again, in another retrospective study, it was reported that worsening outcomes are more common with age, and mortality and admission to the ICU are high.¹²

Although it was not statistically significant, in this study, the male gender indicated a worse outcome. An article published in August 2020 revealed that the higher COVID-19 case fatality rate and increased disease severity in men compared to women are likely due to a combination of behavioural/lifestyle risk factors, the prevalence of comorbidities, and ageing.¹³

A meta-analysis of 1,786 patients found that patients with comorbidities had a higher risk of poor outcomes. In the study, when the comorbidities of 1,786 patients were compared, it was observed that hypertension (15.8%) was the most common, followed by cardiovascular and cerebrovascular diseases (11.7%), which increased the risk.¹⁴ In this study, it was found that 79.2% of the patients in the poor outcome group had comorbidities, in line with the literature. There was a significant correlation between the two groups in terms of poor outcomes and comorbidity. There was also a significant difference in oxygen saturation and respiratory rate between the poor and good outcome groups. In a multi-centric retrospective study including 4.800 patients, it was reported that oxygen saturation was below 95% in 72% of 101 patients who ended in mortality, while only 11% of the remaining 4.705 patients had oxygen saturation below 95% and hypoxia increased mortality. In the same study, when patients were compared in terms of the respiratory rate per minute, it was reported that the respiratory rate was 23.00 (20.00–27.00) in the mortality group, whereas it was 20.00 (19.00–20.00) in the other group, and there was a significant difference between the two groups.¹⁵

When the poor and good outcome groups in terms of laboratory parameters were compared, a decrease in albumin and an increase in CRP, BUN, and LDH values were found to be significant in the group with poor outcomes compared to the group with good outcomes. In a meta-analysis that included 21 studies in which the data of 3.377 patients were examined, it was found that patients who ended in mortality had lower albumin levels and higher CRP, LDH, and BUN values compared to the other group.¹⁶ This study’s data were found to be compatible with the literature.

Looking at the literature, many studies have compared the scoring systems used in COVID-19 and other diseases. The RISE-UP and 4C mortality score were found to be the most successful scoring systems in a study that included 403 patients from the Netherlands and compared the predictability of intensive care hospitalisation and mortality rates of 11 different scoring systems.¹⁷ In a study of 693 patients published in Japan, the RISE-UP, 4C mortality score, REMS, and A-DROP scores were compared. In predicting in-hospital death, the RISE-UP score was 0.82. The authors stated that they found the difference between these two significant.¹⁸ In a retrospective single-centric study of 5.127 patients conducted in Poland, it was found that a MEWS score above 5 was a warning sign for poor outcomes.¹⁹

A total of 20.891 suspected COVID-19 patients from 70 emergency departments were included in an observational cohort study that examined the pandemic respiratory infection emergency system triage (PRIEST) score, which was created by adding age, gender, and patient performance status to the national early warning score 2 (NEWS2) score. It has been reported that a PRIEST score of ≥5 predicts 30-day mortality with 98% precision.²⁰

Heydari et al. compared the qSOFA 4C mortality score, NEWS2, and PRIEST scores in 921 COVID-19 patients, to predict mortality. It was found that the PRIEST and NEWS2 scores outperformed the others. However, no significant superiority was found between the PRIEST and NEWS2 scores. The PRIEST score was found to be the score with the highest sensitivity and negative predictive value for predicting mortality. For this reason, it was stated that it can be used in the emergency department to recognise and discharge non-critical patients earlier.²¹ Again, in the same study, it was found that the other scores were more suitable for triage because they included laboratory parameters in the 4C mortality score.

In a prospective validation analysis that included 101 patients, the 4C mortality score, CURB-65, COVID-GRAM, and NEWS-2 scores were compared. In terms of predicting mortality, the 4C Mortality Score was found to be the best predictor with AUC: 0.80, and when compared in terms of hospitalisation to the ICU, it was found to be the second-performing score after NEWS-2.²²

On December 2022, when the Omicron variant was dominant in China, a new scoring system was studied by Zhang et al, to detect critical illness early in patients with only the Omicron variant. The score, called critical illness risk scoring (CIRS) was compared with CURB-65, SOFA, and 4C mortality score. A total of 2.459 patients were divided into two groups as derivation and validation cohorts. A significant difference was found when the CIRS was compared with the CURB-65, SOFA, and 4C mortality score.²³

In another study conducted by Hager et al. in Germany, a total of 347 patients with COVID-19 PCR positivity were divided into three separate groups, and cohort analysis was performed. The NEWS, qSOFA, COVID-GRAM, CURB-65, and 4C mortality score were compared. While the NEWS score indicated the best performance in terms of admission to the ICU, the 4C mortality score indicated the best performance in terms of in-hospital mortality estimation.²⁴

An important limitation of this study is that it was conducted in a single-centre and tertiary education and research hospital. In this sense, the application of the results of the study to the general population is limited. During the COVID-19 pandemic, different clinical courses were observed with different variants of the virus at different times. Although the alpha variant was dominant in Turkiye at the time of this study, the patients included in the study may still have been infected with different variants. While conducting the study, the effect of the vaccination process in Turkiye was not known, since vaccination of those over the age of 18 started on 25 June 2021. Since it was attemp-ted to include patients with all the necessary parameters to calculate each of the scores retrospectively, a significant amount of data was lost.

Patient treatment guidelines in Turkiye were constantly updated based on scientific evidence. Although the criteria for hospitalisation in the ICU did not change much, there may have been different applications at different times, especially in the indications for hospitalisation. In addition, the timing of intubation was also one of the areas that were discussed at that time. Its contribution to mortality remains unclear. In this retrospective study, possible different approaches to intubation timing may have affected the results.

CONCLUSION

An optimal scoring system is still being investigated. In conclusion, according to the results of this study, the RISE-UP score was found to be more successful than other early warning scores in recognising patients with a diagnosis of COVID-19 with a poor prognosis earlier. The use of the RISE-UP score in the management of COVID-19 patients in emergency departments may contribute to the correct and effective use of resources and also it might be an effective tool in guiding clinicians in estimating mortality and intensive care hospitalisations.

ETHICAL APPROVAL
This study was approved by the institutional local ethics committee (REC no: 2021/0572).

PATIENTS’ CONSENT:
There was no informed consent due to the retrospective nature of the study.

COMPETING INTEREST:
The authors declared no competing interests.

AUTHORS’ CONTRIBUTION:
MA, TCO, FSD, TP: Drafting the manuscript, acquisition, analysis, and interpretation of data.
MA, TCO, EUA, FSD: Supervision, review, writing, and editing.
TCO, MA, FSD, EÜA, TP: Result analysis.
TCO, FSD, EÜA, TP: Critical revision of content.
MA, TCO, FSD: Formal Analysis, investigation, and resources.
All authors approved the final version of the manuscript to be published.

REFERENCES

1. Shahid R, Shoker M, Chu LM, Frehlick R, Ward H, Pahwa P. Impact of low health literacy on patients' health outcomes: a multicenter cohort study. BMC Health Serv Res 2022; 22(1): 1148. doi:10.1186/s12913-022-08527-9
2. Bestemir A, Aydin H. Yillik 300 milyon Hasta Muayenesi; Turkiye'de 2. ve 3. Basamak Kamu Saglik Tesisleri Acil Servis ve Poliklinik Hizmetlerinin Degerlendirilmesi. Sakarya Tip Dergisi 2022; 12(3):496-502. doi: 10.31832/smj.11 28439
3. Khan G, Sheek-Hussein M, Al Suwaidi AR, Idris K, Abu-Zidan FM. Novel coronavirus pandemic: A global health threat. Turk J Emerg Med 2020; 20(2):55-62. doi:10.4103/2452- 2473.285016
4. Lowe KE, Zein J, Hatipoğlu U, Attaway A. Association of smoking and cumulative pack-year exposure with COVID-19 outcomes in the Cleveland Clinic COVID-19 Registry. JAMA Inter Med 2021; 181(5):709-11. doi:10. 1001/jamainternmed.2020.8360
5. Liang W, Guan W, Chen R, Wang W, Li J, Xu K, et al. Cancer patients in SARS-CoV-2 infection: a nationwide analysis in China. Lancet Oncol 2020; 21(3):335-7. doi:10.1016/ S1470-2045(20)30096-6.
6. Wu Z, McGoogan JM. Characteristics of and important lessons from the coronavirus disease 2019 (COVID-19) outbreak in China: summary of a report of 72 314 cases from the Chinese Center for Disease Control and Prevention. JAMA 2020; 323(13):1239-42. doi:10.1001/jama.2020.2648.
7. Olsson T, Terént A, Lind L. Rapid Emergency Medicine Score: a new prognostic tool for in‐hospital mortality in nonsurgical emergency department patients. J Inter Med 2004; 255(5): 579-87. doi:10.1111/j.1365-2796.2004.01321.x.
8. Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified Early Warning Score in medical admissions. QJM 2001; 94(10):521-6. doi:10.1093/qjmed/94.10.521.
9. Van Dam PM, Zelis N, Stassen P, Van Twist DJ, De Leeuw PW, Van Kuijk S, et al. Validating the RISE UP score for predicting prognosis in patients with COVID-19 in the emergency department: a retrospective study. BMJ Open 2021; 11(2):e045141. doi:10.1136/bmjopen-2020-045141.
10. Knight SR, Ho A, Pius R, Buchan I, Carson G, Drake TM, et al. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ 2020; 370(3339). doi:10.1136/bmj.m3339.
11. Pepe M, Maroun-Eid C, Romero R, Arroyo-Espliguero R, Fernàndez-Rozas I, Aparisi A, et al. Clinical presentation, therapeutic approach, and outcome of young patients admitted for COVID-19, concerning the elderly counterpart. Clin Exp Med 2021; 21(2):249-68. doi: 10.1007/ s10238- 021-00684-1.
12. Liu K, Chen Y, Lin R, Han K. Clinical features of COVID-19 in elderly patients: A comparison with young and middle-aged patients. J Infect 2020; 80(6):e14-e8. doi: 10.1016/j.jinf.2020.03.005.
13. Haitao T, Vermunt JV, Abeykoon J, Ghamrawi R, Gunaratne M, Jayachandran M, et al. COVID-19 and sex differences: mechanisms and biomarkers. Mayo Clin Proc 2020; 95(10):2189-203. doi: 10.1016/j.mayocp.2020.07.024.
14. Paudel SS. A meta-analysis of 2019 novel coronavirus patient clinical characteristics and comorbidities. Preprint from Research Square, 08 Apr 2020. doi: 10.21203/ rs.3.rs- 21831/v1).
15. Liu W, Yang C, Liao Y-G, Wan F, Lin L, Huang X, et al. Risk factors for COVID-19 progression and mortality in hospitalized patients without pre-existing comorbidities. J Infect Public Health 2022; 15(1):13-20. doi: 10.1016/j.jiph.2021.11.012.
16. Henry BM, De Oliveira MHS, Benoit S, Plebani M, Lippi G. Hematologic, biochemical and immune biomarker abnormalities associated with severe illness and mortality in coronavirus disease 2019 (COVID-19): A meta-analysis. Clin Chem Lab Med 2020; 58(7):1021-8. doi: 10.1515/cclm- 2020-0369.
    
     
17. van Dam PM, Zelis N, van Kuijk SM, Linkens AE, Brüggemann RA, Spaetgens B, et al. Performance of prediction models for short-term outcome in COVID-19 patients in the emergency department: a retrospective study. Ann Med 2021; 53(1):402-9. doi: 10.1080/07853890.2021.1891453.
18. Kuroda S, Matsumoto S, Sano T, Kitai T, Yonetsu T, Kohsaka S, et al. External validation of the 4C Mortality Score for patients with COVID-19 and pre-existing cardiovascular diseases/risk factors. BMJ Open 2021; 11(9):e052708. doi: 10.1136/bmjopen-2021-052708.
19. Butkiewicz S, Zaczyński A, Pańkowski I, Tomaka P, Rzońca E, Slęzak D, et al. Retrospective study to identify risk factors for severe disease and mortality using the modified early warning score in 5127 patients with COVID-19 admitted to an emergency department in poland between March 2020 and April 2021. Med Sci Mon 2022; 28:e 938647. doi: 10.12659/MSM.938647.
20. Goodacre S, Thomas B, Sutton L, Burnsall M, Lee E, Bradburn M, et al. Derivation and validation of a clinical severity score for acutely ill adults with suspected COVID-19: the Priest observational cohort study. PloS One 2021; 16(1): e0245840. doi: 10.1371/journal.pone.0245840.
21. Heydari F, Zamani M, Masoumi B, Majidinejad S, Nasr-Esfahani M, Abbasi S, et al. Physiologic Scoring Systems in Predicting the COVID-19 Patients’ One-month Mortality; a Prognostic Accuracy Study. Arch Acad Emerg Med 2022; 10(1):e83. doi: 10.22037/aaem.v10i1.1728. eCollection 2022.
22. Yildiz H, Castanares-Zapatero D, Hannesse C, Vander-meersch D, Pothen L, Yombi JC. Prospective validation and comparison of COVID-GRAM, NEWS2, 4C mortality score, CURB-65 for the prediction of critical illness in COVID-19 patients. Infect Dis (Lond) 2021; 53(8):640-2. doi: 10.1080/23744235.2021.1896777.
23. Zhang Y, Han J, Sun F, Guo Y, Guo Y, Zhu H, et al. A practical scoring model to predict the occurrence of critical illness in hospitalized patients with SARS-CoV-2 omicron infection. Front Microbiol 2022; 19(13):1031231. doi: 10.3389/fmicb.2022.1031231. eCollection 2022.
24. Häger L, Wendland P, Biergans S, Lederer S, de Arruda Botelho Herr M, Erhardt C, et al. External Validation of COVID-19 Risk Scores during Three Waves of Pandemic in a German Cohort—A Retrospective Study. J Pers Med 2022; 12(11):1775. doi: 10.3390/jpm12111775.