This study used data from a retrospective cohort of the CICU of the Samsung Medical Center, an ongoing registry to evaluate the clinical characteristics, management, and outcomes of critically ill cardiac patients. All consecutive patients admitted to the CICU between September 1, 2012, and December 31, 2018, were included in the study. Patients were included if they were aged 18 years or older and excluded if they were admitted for less than 24hours or if data on delirium were unavailable. The study was approved by the Institutional Review Board of Samsung Medical Center (IRB No. 2020-10-102), and the requirement for informed consent was waived due to the observational nature of the study.

The CICU is a 12-bed ICU with a 1:2 nurse-to-patient ratio. Critically ill cardiac patients receive level 1 care managed by a dedicated cardiac intensivist.12 Details of the comprehensive critical care of the Samsung Medical Center CICU have been described in previous reports.13,14 Briefly, the CICU provides noninvasive and invasive devices for monitoring hemodynamic and cardiovascular support, including mechanical ventilation and extracorporeal membrane oxygenation. Cardiac surgery support is readily available and multidisciplinary care is provided through consultation with a dietitian, pharmacist, and respiratory care practitioner.

The clinical practice guidelines for general intensive care published by The Society of Critical Care Medicine were applied. To monitor delirium, the Confusion Assessment Method for the intensive care unit (CAM-ICU) assessment was performed by nurses three times a day in patients with a Richmond Agitation-Sedation Scale score of -3 (indicating movement or eye-opening to voice but no eye contact) or higher.15 The validated Korean version of the CAM-ICU is routinely used in the CICU.16 A senior nurse rechecked the recorded CAM-ICU results on a daily basis.

Delirium was the primary outcome of this study. Delirium was defined as a positive CAM-ICU within 7 days of CICU admission. Clinical characteristics, primary diagnoses, vital signs, laboratory test results, and clinical presentations at CICU admission were included as candidate variables. To select the optimal variables for creating a delirium prediction model for CICU patients, we used several machine learning methods to assess the importance of the candidate variables, including random forest, extreme gradient boosting, partial least squares, and Plmnet-elastic.net. We assessed the importance of the variables and enhanced the appropriateness of their selection with machine learning methods. We selected 4 commonly used models for regression analysis from a range of various machine learning algorithms. Among the variables identified as highly important, we selected those that repeatedly exhibited high importance or were deemed clinically significant based on previous studies and expert consensus (J.H. Yang and R.E. Ko), were selected to construct the model.

Baseline patient characteristics are summarized as numbers and proportions for categorical variables and medians with interquartile ranges [IQR, 25–75th percentiles] for continuous variables. The clinical characteristics of the delirium and nondelirium groups were compared using the chi-square or Fisher exact test for categorical variables and the Student t-test for continuous variables. Univariate logistic regression analysis was performed to estimate the odds ratios (ORs) for each variable. The ORs of each variable are reported with 95% confidence intervals (95%CIs). Coefficients, standard errors, and Z-values were also calculated.

To develop a delirium prediction scoring model, logistic regression analysis was performed using the selected variables to develop a formula for the delirium prediction model for CICU patients. For virtual internal validation, 100-repeated hold-out validation was used to further evaluate the performance of the model. To perform internal validation, the patients were divided into a 7:3 ratio. Logistic regression was conducted using the enter method, using a variable beta coefficient value to achieve more accurate predictions with small variables. In the construction of prediction models, logistic regression provides estimated regression coefficients that are of interest. These coefficients represent the log-odds of the outcome variable. To facilitate meaningful comparisons across variables, we standardized these coefficients. Standardization ensured that we evaluated the effect of each variable on a common scale. Specifically, standardization was based on the ranges of the variables, identifying the scenario in which the range of each variable was largest range. The prediction scoring model for delirium assigns scores reflecting the likelihood of delirium onset. Higher scores indicate higher risk.

We conducted a calibration plot to demonstrate that the information loss was negligible when converting the continuous values of the predictors to categorical values. The calibration plot compares the observed and predicted probabilities of delirium based on 2 models: one using the continuous values of the clinical factors as predictors, and the other using the categorical values of the clinical factors as predictors. A receiver operating characteristic curve was used to demonstrate the performance of the prediction model using the area under the curve (AUROC) and 95%CIs. All tests were 2-sided, and a P value ≤ .05 was considered to indicate statistical significance. All analyses were performed using R Statistical Software (version 3.2.5; R Foundation for Statistical Computing, Austria).

During the study period, 4261 patients aged 18 years or older were admitted to the CICU. We excluded patients who were not admitted for more than 24hours (n=1473) and those without available data on delirium (n=14). Finally, 2774 patients were included in the analysis. Of the 2774 eligible patients, 677 (24.4%) experienced delirium during CICU admission. The baseline characteristics of the patients are shown in table 1. Overall, patients with delirium were high-risk participants. More patients in the delirium group were treated with extracorporeal membrane oxygenators (12.4% vs 2.0%, P<.001) and received mechanical ventilation support (38.7% vs 5.4%, P<.001) than those in the nondelirium group. Regarding clinical outcomes, CICU death (11.4% vs 1.8%, P<.001), hospital death (18.2% vs 3.1%) and CICU length of stay (4.7 [IQR 2.4–9.0] days vs 1.9 [IQR 1.3–3.0] days, P<.001) were significantly higher in patients in the delirium group than in those in the nondelirium group.

A flow diagram of the study is shown in figure 1. Using the clinical features available at CICU admission, we applied machine learning methods to assess the relevance of potential predictors. Figure 2 shows the relative importance of each variable and the performance of the model among different machine learning algorithms. Detailed ranking of the top 40 variables by their importance scores is shown in table 1 of the supplementary data. The machine learning models demonstrated good predictive performance for delirium occurrence: the random forest had an AUROC of 0.8692 (95%CI, 0.8426-0.8958), extreme gradient boosting had an AUROC of 0.8377 (95%CI, 0.8066-0.8688), partial least squares had an AUROC of 0.8662 (95%CI, 0.8400-0.8924), and plmnet-elastic.net had an AUROC of 0.8662 (95%CI 0.8400-0.8924).

Figure 1.

Study flowchart. BUN, blood urea nitrogen; CICU, cardiac intensive care unit; INR, international normalized ratio; SMC, Samsung Medical Center; WBC, white blood cells; XGBoost, extreme gradient boosting.

(0.37MB).

Figure 2.

Selection of features by machine learning method and score calculation formula. The figure displays the 30 most influential variables of each machine learning method. 95%CI, 95% confidence interval; ALT, aspartate aminotransferase; AST, alanine aminotransferase; AUROC, area under the receiver operating characteristic curve; BMI, body mass index; BT, body temperature; BUN, blood urea nitrogen; CABG, coronary artery bypass graft; CNS, central nervous system; CRP, C-reactive protein; DBP, diastolic blood pressure; Hb, hemoglobin; Hct, hematocrit; HR, heart rate; INR, international normalized ratio; MAP, mean arterial pressure; MCS, mechanical circulatory support; SBP, systolic blood pressure; SPO2, oxygen saturation by pulse oximetry; RR, respiratory rate; WBC, white blood cells; XGBoost, extreme gradient boosting.

(0.68MB).

Among the potential predictors, we selected those that were clinically significant and frequently ranked as highly important for developing a delirium prediction model for CICU patients. Using machine learning methods, we selected 11 variables that shared the feature of high ranking, such as mechanical ventilation, albumin, blood urea nitrogen (BUN), international normalized ratio (INR), white blood cells, C-reactive protein, age, platelet count, hemoglobin, heart rate, and creatinine. We used a minimal set of clinically meaningful and easily applicable variables selected from the repeated variables. The final model included albumin level, INR, BUN level, white blood cell count, C-reactive protein level, age, heart rate, and mechanical ventilation as predictors.

A logistic regression model was constructed using selected variables. The predictive performance of the logistic regression model was good (AUROC, 0.860; 95%CI, 0.850-0.890, table 2). Figure 1 of the supplementary data shows a restricted cubic spline plot used to examine the association between each continuous variable and the occurrence of delirium. Age, INR, and heart rate were converted into categorical variables for ease of use in clinical settings. Age was transformed into a categorical variable using a spline curve. To facilitate clinical acceptance, we chose the cut points for heart rate and INR based on the clinical definitions. For heart rate, we used 40 bpm and 100 bpm as the cut points according to the definitions of bradycardia and tachycardia. We adopted 40 bpm as the cut point for considering the implantation of a temporary pacemaker for patients with third-degree heart block (or atrioventricular block) who had a heart rate below 40 bpm when awake.17 For INR, we used 1.2 and 1.7 as the cut points, based on the normal value of 1.2 and the application of the Model for End-Stage Liver Disease Score,18 which is a score that calculates the severity of liver disease. Finally, the delirium prediction scoring model was developed and the score calculation formula is presented in the blue box of figure 2 (AUROC, 0.861; 95%CI, 0.843-0.879). The cutoff value of 0.2 was selected based on the Youden index value. When a cutoff score of 0.2 was used, a sensitivity of 0.83 (95%CI, 0.77-0.88), specificity of 0.71 (95%CI, 0.67-0.74), positive predictive value of 0.47 (95%CI,0.42-0.53), and negative predictive value of 0.93 (95%CI, 0.90-0.95) were obtained. The results of the goodness-of-fit test between the observed probability calculated using the model with the continuous values of predictors and the predicted probability calculated using the model with the categorical values of predictors indicated no statistical difference between the 2 models, as evidenced by P=.585 (figure 2 of the supplementary data). An Excel file that enables automatic calculation of the formula with user input is provided in table 2 of the supplementary data.

A 100-repeated cross-validation was performed using all eligible patients. In this simulation, 70% of the patients were used for training, and 30% for testing. The average cross-validated AUROC was 0.8549 (95%CI, 0.8266-0.8833; figure 3).

Figure 3.

The area under the receiver operating characteristic curve (AUROC) of 100 repeats of hold-out validation. 95%CI, 95% confidence interval.

(0.15MB).

This study has several limitations. First, the significance of our findings may have been influenced by the inherent biases of the nonrandomized registry data. However, we used an extensive database of all consecutive patients admitted to the CICU of the Samsung Medical Center. Second, this study was performed retrospectively and internally validated. Prospective interventional studies are required to verify the performance of the model and confirm its clinical usefulness. However, a 100-repeated cross-validation was conducted for the prospective simulation. Third, this study was conducted in a single level 1 care CICU. Therefore, the generalizability of the findings requires further validation. Despite these limitations, a strength of our study was that the CAM-ICU assessments were conducted by nurses 3 times daily for all consecutively admitted CICU patients. Therefore, we analyzed precise information on delirium in CICU patients.