With an increasing clerical burden and challenges to the usability of electronic health care records, AI provides opportunities to standardize and improve efficiency in data collection and the quality of communication (figure 3A), with the aim of reducing menial and time-consuming tasks and allowing a focus on patient-physician interactions. Analytical methods of speech and text analysis could improve information transfer by guiding communication (providing feedback on delivery of information and clarity of content), standardizing medical history taking, and informing patients in accessible, easy to understand language.10 Patients’ verbal responses, facial expressions, and tone of voice could be analyzed to guide interaction, whether on-site or using telemedicine.11 Additionally, communication can now take place over electronic patient portals, where AI-based tools, such as natural language processing (NLP) and machine learning (ML), can categorize patient free-text messages, with the aim of organizing triage and automating responses for urgent cardiovascular medical issues.12

Figure 3.

A pipeline showing various parts of the cardiac imaging workflow. Artificial intelligence (AI) can be integrated throughout the pipeline with to goal of achieving automation, standardization and data integration, as well as improved efficiency and accuracy.

(0.44MB).

In the ambulatory setting, risk assessment and triaging are conventionally achieved through decision support systems (DSS) where AI-based feature extraction from images or clinical reports could be integrated to automate input and increase efficiency.13 Examples are the possibility of automatically creating an accurate list of patient diagnoses from clinical notes,14 or identifying risk factors from electronic health records, such as the risk associated with sudden cardiac death in patients with hypertrophic cardiomyopathy.15 Furthermore, ML integration can advance risk assessment in the ambulatory setting of specific patient groups. A boosted decision-tree algorithm was used in a cohort of 5822 heart failure patients to assess mortality risk.16 The model was externally validated, yielding a similarly satisfying performance and thus providing proof of generalizability in the heart failure disease spectrum. An efficient, widely applicable risk model could help prompt identification and triaging toward further investigation and advanced care for appropriately selected patients.

An integrated AI/DSS algorithm could also help in choosing the appropriate imaging modality to provide the greatest diagnostic and prognostic information in the least amount of time, while maximizing patient safety and minimizing additional testing costs.17 In a recent multicenter study, an automated point-of-order DSS, based on decision tree algorithms, was demonstrated to rapidly determine the appropriateness of cardiac imaging tests for the assessment of coronary artery disease (CAD).18 Another study demonstrated an NLP-based approach for computed tomography (CT) and cardiac magnetic resonance (CMR) protocol assignment.19 As shown by these studies, point-of-order AI/DSS could be an efficient tool to guide patient imaging decisions and maximize efficiency for both patients and health care providers.

Once the cardiac imaging modality has been chosen, AI can improve the process of image acquisition (figure 3B). In echocardiography, convolutional neural networks (CNNs)—a type of deep learning (DL) well-suited for image-oriented tasks—can automate echocardiographic image acquisition by guiding the motion of the probe toward optimal positions, such as the 4-chamber view, enabling capture of diagnostic echo images after minimal training.20 The US Food and Drug Administration recently approved the first cardiac software using AI to guide ultrasound image acquisition.21 The implications include improved staff education, wider applicability of imaging in low-expertise centers, fewer imaging artefacts, and higher reproducibility of exams.

Novel technology can also improve CMR and CT acquisition—CMR imaging suffers from long acquisition times due to high temporal and spatial resolution. Because of their ability to use large data sets to learn key reconstruction parameters, ML approaches, particularly DL, have been recently proposed to accelerate CMR scan times.22 Likewise, generative adversarial networks have been applied to synthesize cine-like CMR images from real-time CMR sequences (ie, sequences used as an alternative when patients are unable to hold their breath or have arrhythmias during scanning), resulting in improved image quality, with clearer images and sharper anatomical distinction.23 CT has also benefited from promising advances from ML aimed at speeding up reconstruction times and reducing radiation dose. DL methods have been applied to learning with low-dose CT images and to reconstruct them to routine-dose CT images, synthesize contrast CT images from noncontrast images, reduce noise in low-dose CT scans, enable lower-dose CT and sparse-sampling CT, and reduce metal artefacts.24 These examples of AI contributions to image acquisition will prove valuable in medical education, time-efficiency, cost-reduction, and patient safety.

After acquisition, raw image data intended for analysis is commonly heterogeneous—with variable image quality, acquired with different settings and with machines from different vendors. Therefore, preprocessing (including anonymization, normalization, and handling of missing data) is a crucial part of the imaging pipeline (figure 3C). Characteristics such as image contrast, brightness and resolution affect the robustness and accuracy of image analysis. To name a few examples, normalization can include adjustment and standardization of image size,25–29 applying filters to images to remove noise,25 and enhancing contrast to help in the process of feature extraction.30 To prepare data for DL, echocardiographic videos can be converted to multidimensional numeric arrays of pixel intensities, with dimensions representing time, coordinates in space, and encoding of color information.31 Normalization of CT images can also be achieved through a DL, eg, U-Net based, strategy.32 Furthermore, DL has recently addressed more challenging tasks, with CNNs screening for motion artefacts or erroneous orientation of the 4-chamber view in the first step of the image analysis pipeline.33,34 ML can also provide solutions for missing data, with innovative imputation methods using cardiac imaging data.35

Overall, preprocessing approaches accomplish uniformization of imaging data, enabling improved postprocessing and feature extraction, in which various feature selection strategies can be employed to determine the most interesting features of the dataset. However, regardless of imaging modality, data processing to extract clinical variables (ie, volumes, wall thickness, deformation, valve morphology) is notoriously time-consuming, meticulous, and prone to interoperator differences and error. Image interpretation is highly experience- and observer-dependent, resulting in unsatisfactory measurement variability. This can prove problematic for longitudinal patient follow-up. This is the case, for example, in cardio-oncology or heart failure, where reproducible assessment of cardiac function is essential, or for multicenter, multiexpert datasets, where variability in measurements biases the analysis. Automation of image analysis through ML can therefore be beneficial by saving time and by increasing accuracy, reproducibility, and standardization33,36–38 (figure 3D).

Identifying cardiac views is the crucial first step in image analysis. When a cardiologist recognizes the cardiac view, view-specific cardiac structures can be segmented, measured and quantified to assess cardiac function and remodelling. View classification has lately been approached mainly through DL methods such as CNNs, incorporating both spatial and temporal information contained in the echo loops, and achieving a high accuracy rate of 92.1%.26 More advanced CNN models can classify full standard transthoracic echocardiograms (B-mode, M-mode, Doppler, both still images and videos) from patients with a range of pathologies, technical settings, and image quality.27 By avoiding idealized training subsets, such models show their potential applicability to the clinical setting, with high accuracy of 97.8%.

Following view recognition, image segmentation is the process of partitioning an image into its constituent parts (eg, each part corresponding to a distinct cardiac structure) to quantify structure and function. In 2-dimensional echocardiography, ML-based fully automatized left ventricular (LV) volume, ejection fraction and longitudinal strain measurements have been shown to be rapid, reproducible, and in good agreement with manual measurements.39 Furthermore, an automated image interpretation pipeline has been shown to be feasible, incorporating view identification, image segmentation, and quantification of structure and function.31 Three-dimensional (3D) echocardiography could offer a more reproducible quantification of cardiac chambers but requires cumbersome manual processing, heavily burdening workflow and applicability. An automated adaptive analytics algorithm for simultaneous quantification of ventricular and atrial volumes proved to be reproducible and comparable with manual segmentations and CMR values, but the technology is applicable only in patients with sufficient image quality.40 Feasible solutions for automated 3D right ventricle assessment have also been demonstrated.36 Additionally, when considering valve assessment, automated 3D transesophageal echocardiography mitral valve geometry quantification showed a good correlation with manual measurements and a significant reduction in measurement time, although, again, there was a need for high-quality images.41 While challenges in automating 3D dataset analysis are still prominent, current studies demonstrate an incremental shift, showing potential to improve workflow limitations in the clinical setting. A comprehensive review of automated quantification in echocardiography is available.5

Similar development in automation has been seen in CMR. Fully automated quantification of LV mass, biventricular volumes and ejection fraction has been demonstrated in a heterogeneous CMR dataset, showing feasible segmentations and similar results to manual quantification, albeit demonstrating lower agreement in severely altered anatomy and reduced image quality.38 Automated CNN-based CMR-image segmentation and quantification has been shown to be faster, than the most precise human techniques and to have similar precision, even when challenged with real-world, multicenter, multidisease scan-rescan data to assess measuring precision.42 Furthermore, integration of quality control algorithms, detecting erroneous outputs, has been demonstrated in a cohort of healthy volunteers and patients with a wide variety of heart diseases.33 Cardiac function and tissue characterization can also be addressed; fully automated phase-contrast CMR aortic flow quantification showed a more rapid, feasible alternative for large CMR dataset segmentations and analysis,37 while semiautomated quantification43 and synthetic data approaches44 have been suggested to automate late gadolinium enhancement segmentation. Synthetic data have also been used to generate CMR images based on a reference biomechanical model of the LV to create a “ground truth” for testing the robustness of segmentation and registration methods.45

Following the image analysis and processing, the ensuing steps include integration of the derived measures with other data sources, resulting a comprehensive representation of the patient in focus. Traditional methods of phenotyping are challenged by unstandardized quantification, geometric assumptions, high observer variability, limited set of parameters, and a tendency to discretize continuous phenogroups. As shown in figure 4, the combination of information-rich imaging (ie, deformation imaging, 3D datasets, tissue characterization, 4-dimensional flow etc.) and ML facilitated the shift from one dimensional descriptors of cardiac function and structure to high-resolution, multi-parametric phenotyping.4 A relevant message learned through such advanced approaches is that cardiac diseases commonly represent a disease spectrum, where a binary classification into diseased or healthy does not reflect the underlying complexity. A recent unsupervised ML approach, using LV longitudinal myocardial velocity patterns, categorized hypertensive and breathless patients into a transition zone of the heart failure with preserved ejection fraction spectrum, demonstrating potential pitfalls of conventional clinical diagnostic algorithms, as well as the broad spectrum of the heterogeneous heart failure with preserved ejection fraction syndrome.46 Moreover, it is precisely the heterogeneity of heart failure, and the nonlinearity of diastolic function, that can present an appropriate challenge for ML, especially unsupervised approaches, which can extract hidden patterns in data and cluster patients regardless of a priori knowledge or known clinical labels.47,48 Data from the same patient, both from rest and exercise echocardiography, can be integrated using ML to create a spatiotemporal-rest-exercise representations of LV function to determine heart failure with preserved ejection fraction.48 Finally, the strong potential of data integration and phenotyping through AI is best illustrated through novel approaches combining knowledge from imaging, genomics and proteomics through the combination of high-throughput DNA sequencing combined with ML methods to tackle challenges of scalability and high-dimensionality of data.4 As a notable example, polygenic risk scores of LV phenotypes have been shown predictive of heart failure independently of clinical risk factors, and CMR derived phenotypes highly heritable, showing that LV image-derived phenotypes and remodelling are related to the underlying genetic basis.49 Moreover, the combination of high-resolution phenotyping and machine-based data analysis showed that titin truncating variants, previously thought of as irrelevant in the general population, are associated with higher LV volumes in CMR analysis and eccentric remodelling.50 Proteomics have also been used to identify CAD risk –predicting both high-risk plaques and the absence of CAD on coronary CT angiography in patients with suspected disease.51

Figure 4.

A scheme showing the process of artificial intelligence (AI) mediated data integration leading to improved patient phenotyping, personalized treatment, and clinical research.

(0.4MB).

The advances in data integration and phenotyping are inherently linked to significant improvement in diagnostics and outcome prediction (figure 3E), though most studies using AI are still observational, single-center, and can only be considered hypothesis generating. Nevertheless, AI has been used on imaging data to diagnose myocardial infarction, heart failure, CAD, atherosclerosis, cardiomyopathies, and valvular heart disease among others. DL models demonstrated comparable results to cardiologists in detecting the presence and location of ischemic regional wall motion abnormalities,52 or high accuracy of detection (AUC, 0.94) of chronic myocardial infarction in noncontrast enhanced cine CMR images, compared with late gadolinium enhancement CMR.53 Results have also been seen in CT studies, where texture analysis was more objective and reproducible in diagnosing chronic myocardial infarction when compared with visual assessment.54 In heart failure, unsupervised ML has been used on a large trial dataset to identify phenogroups with distinct clinical characteristics related to outcome,55 or to identify patients with heart failure with preserved ejection fraction through spatiotemporal variations of LV strain rate during rest and exercise echocardiographic data.48 The ability of cardiac imaging to avoid unnecessary invasive procedures presents a clear target in CAD. DL applications have shown successful identification of obstructive CAD from single-photon emission CT perfusion imaging.56 In recent multicentre studies, a gradient boosting classifier showed that the addition of resting CT perfusion to CT angiography can improve prediction of significant ischemia in coronary stenosis,57 while an on-site, ML-based, CT fractional flow reserve algorithm improved the performance of CT angiography by reclassifying hemodynamically nonsignificant stenosis, performing as well as computational fluid dynamic approaches.58 Calcification quantification and plaque characterization have also been increasingly explored, demonstrating that radiomic features may be able to discriminate napkin-ring sign plaques,59 a challenging task due to its qualitative nature. In valve disease, support vector machines and linear discriminant analysis have been used to separate patients according to the severity of mitral regurgitation, quantified based on textural features extracted from three echocardiographic B-mode views, with > 99% accuracy for each of the qualitative levels of regurgitation.25 Likewise, discriminating different cardiomyopathies has been a sensible target for the advanced phenotyping capabilities of AI. Radiomic texture analysis on CMR T1 images, and a support vector machine classifier, have been applied to discriminate between hypertensive heart disease and hypertrophic cardiomyopathy.29 An ensemble ML model used speckle-tracking echocardiographic data for automated discrimination of pathological and physiological patterns of remodelling seen in hypertrophic cardiomyopathy and athlete's hearts (AUC, 0.80).60 A detailed overview of diagnostic applications of AI in cardiac imaging is found in Al’Aref et al.6 and Martin-Isla et al.8

After full data analysis is performed, an AI-fuelled, rapid, precise, and reproducible reporting of findings is beneficial for efficient patient management (figure 3F). Here, automatic recognition and translation of voice into text speech recognition– was one of the first examples of AI integration into the imaging workflow. Despite unresolved challenges, application in radiology departments has already shown benefits in reducing reporting time and costs, as well as increasing productivity.61 Furthermore, AI could serve as a “fail-proof” for study reporting echocardiography reports can be analysed by artificial neural networks to predict patient mortality and hospital readmissions for heart failure patients.62 NLP can help clinical interpretation of reports and report drafting by assessing posttest risk after myocardial perfusion imaging,63 where underestimation of ischemia in reporting has been previously noted. Another crucial challenge is the failure to follow-up imaging recommendations, potentially leading to patient health deterioration, failed advanced treatment and rise of costs. Scalable and automated follow-up detection NLP algorithms can therefore be useful to determine adherence rates for follow-up imaging and define patients who may benefit most from potential engagement, with the aim of mitigating risk.64

AI fields, including ML, NLP, computer vision and robotics, have generated high interest to address challenges in the field of interventional cardiology with the aim of improving real-time decision making, streamlining workflows in the catheterization laboratory and standardizing catheter based procedures through advanced robotics.65 A clear example can be found in the imaging solutions for transcatheter aortic valve replacement procedure planning and valve choice. In transcatheter aortic valve replacement, CT is the current standard for the determination of prothesis sizing, however, automated 3D transoesophageal echocardiography software can allow modelling and reproducible quantification of aortic annular and root dimensions with high correlation of measurements with CT.66 Furthermore, post transcatheter aortic valve replacement, there is a lack of a solid reference method to assess paravalvular regurgitation which can be addressed using an ultrasound simulation-based pipeline.67 In the field of heart failure, unsupervised ML has been applied to integrate whole-cycle echocardiographic data and heterogeneous clinical data to predict response to cardiac resynchronization therapy.68 DSS and risk assessment after hospital discharge have also seen advances a boosted ensemble algorithm showed greater prognostic value of predicting CAD than current integrated coronary CT angiography risk scores by maximizing usage of stenosis and plaque composition information.69 Moreover, a ML integration of clinical and CT angiography data predicted 5-year all-cause mortality (AUC, 0.79), performing significantly better than existing metrics. Similar approaches have been proposed in prediction of 3-year major adverse cardiac events in patients undergoing single-photon emission CT myocardial perfusion imaging, where ML integration of clinical, stress and imaging variables was found to have a superior predictive accuracy compared with visual or automated perfusion assessment in isolation.70

The described AI integration and automation of analysis can advance patient care in various clinical settings; however, integration also bears relevant implications for clinical research (figure 1, figure 4). The quality and size of the available datasets determines the quality of the ML-derived results. A major obstacle here is the need for high-quality expert annotations in imaging datasets, as labelling carries inherent uncertainties, biases and assumptions. Recently, self-supervised methods have been suggested to tackle this problem due to their ability to explore data that has not been labelled.71 Additionally, synthetic data, with realistic properties and imaging modality specific noise textures, can help bypass the problem by being used in the training procedure of the ML algorithms in addition to the real clinical data. As an example, a generative adversarial network has been used for the synthesis of realistic CT images based on body phantoms with the goal of increasing the dataset size to improve training and performance of vessel segmentation networks.72 Furthermore, many ML algorithms are limited by a scarcity of large and heterogeneous datasets most available studies are confined to single-center cohorts or cohorts from specific populations, however, ML has shown capacity to integrate data from different datasets to achieve a more robust analysis.73 In addition, there is increasing initiative to expand the availability of imaging databases, biobanks, bioresources and registries for ML training an example would be the UK Biobank initiative or the REFINE-SPECT (REgistry of Fast Myocardial Perfusion Imaging with NExt generation SPECT) registry.8 Nevertheless, data sharing and a lack of efficient methodologies to satisfy all involved stakeholders remains a common challenge. Next-generation methods for federated, decentralized ML have been proposed to replace the current paradigm of data sharing and centralized storage –algorithms could be distributed to sites or devices where data is kept to perform tasks locally, returning results to the central repository to update the main algorithm.74 Ideas to integrate solutions from the financial sector, such as blockchain technology, have also been proposed to decentralized databases, secure traceable and scalable data exchange, and integrate AI tools that are blockchain-enabled, distributed, and tied to a system of incentives that flow to the owner of each data on the basis of its value.75

Finally, when datasets are available, high-quality analysis needs to be performed. Recent reports envision a fully automated analysis - a CNN network, using echocardiographic images, was used to identify local cardiac structures, perform automated measurements of structure and function, and predict phenotypes that modify cardiovascular risk - based solely on imaging data.28 Another recent study performed a retrospective analysis on a 10-year echo database from diverse vendors, showing that CNN algorithms can support scalable, low-cost analysis within the health care system in a reasonable timeframe.31 Importantly, the application of these automated processing tools warrants quality control. Manual inspection of each segmentation is not feasible in larger cohorts, therefore approaches such as reverse classification accuracy show potential for accurate and fully automatic quality control, as shown on a large number of CMR cases from the UK Biobank study.76 Moreover, parameters calculated from the automated segmentations, such as stroke volume of the left and right ventricle, can be tracked, and exams with disproportionate stroke volumes flagged. Additional ML algorithms, such as support vector machines, can be used to classify outputs of ML algorithms as abnormal or normal.33 Lastly, research data is not only stored in images. Management of cardiac patients produces an immense stream of clinical data, most commonly in nonstandardized, unstructured reports, not feasible for immediate analysis. As discussed earlier, NLP algorithms can play a crucial role, extracting cardiac concepts from multiple center-derived free-text and semi-structured reports. The promise of such technologies represents an important link in automation of database analysis for clinical research.