We developed a 3D-image-based unsupervised prediction model, called vox2pred, for predicting the progression of pulmonary diseases based on a conditional generative adversarial network (cGAN). The architecture of the vox2pred model includes a time generator that consists of an encoding convolutional network and a fully connected prediction network, and a discriminator network. The time generator is trained to generate the progression time from the chest 3D CT image volumes of each patient. The discriminator is a patch-based 3D-convolutional network that is trained to differentiate between “predicted pairs” of a chest CT image volume and a predicted progression time from “true pairs” of the chest CT image volume and the corresponding observed progression time of the patient. For a pilot evaluation, we retrospectively collected high-resolution chest CT images of 141 patients with the coronavirus disease 2019 (COVID-19). The progression predictions of the vox2pred model on these patients were compared with those of existing clinical prognostic biomarkers by use of a two-sided t-test with bootstrapping. Concordance index (C-index) and relative absolute error (RAE) were used as measures of the prediction performance. The bootstrap evaluation yielded C-index and RAE values of 87.4% and 18.5% for the vox2pred model, whereas those for the visual assessment of the CT images in terms of a total severity score were 62.4% and 51.8%, and for the total severity score for crazy paving and consolidation (CPC), they were 64.7% and 51.3%, respectively. The increase in the accuracy of the progression prediction by the vox2pred model was statistically significant (p < 0.0001), indicating the high effectiveness of vox2pred as a prediction model for pulmonary disease progression in chest CT.
We developed and evaluated the effect of deep radiomic features, called U-radiomics, on the prediction of the overall survival of patients with the coronavirus disease 2019 (COVID-19). A U-net was trained on chest CT images of patients with interstitial lung diseases to classify lung regions of interest into five characteristic lung tissue patterns. The trained Unet was applied to the chest CT images of patients with COVID-19, and a U-radiomics vector for each patient was identified from the bottleneck layer of the U-net across all the axial CT images of the patient. The U-radiomics vector was subjected to a Cox proportional hazards model with an elastic-net penalty for predicting the survival of the patient. The evaluation was performed by use of bootstrapping, where the concordance index (C-index) was used as the comparative performance metric. Our preliminary comparative evaluation of existing prognostic biomarkers and the proposed U-survival model yielded the C-index values of (a) extent of well-aerated lung parenchyma: 51%, (b) combination of blood tests of lactic dehydrogenase, lymphocyte, and C-reactive protein: 63%, and (c) U-survival: 87%. Thus, the U-survival significantly outperformed clinical biomarkers in predicting the survival of COVID-19 patients, indicating that the U-radiomics vector of the U-survival model may provide a highly accurate prognostic biomarker for patients with COVID-19.
We developed an image-based unsupervised survival prediction model, called pix2surv, based on a conditional generative adversarial network (cGAN), and evaluated its performance based on chest CT images of patients with the coronavirus disease 2019 (COVID-19). The architecture of the pix2surv model includes a time generator that consists of an encoding convolutional network and a fully connected prediction network, and a discriminator network. The time generator is trained to generate survival-time images from chest CT images of each patient. The discriminator is a patch-based convolutional network that is trained to differentiate between “fake pairs” of a chest CT image and a generated survival-time image from “true pairs” of the chest CT image and the corresponding observed survival-time image of the patient. For evaluation, we retrospectively collected high-resolution chest CT images of COVID-19 patients. The survival predictions of the pix2surv model on these patients were compared with those of existing clinical prognostic biomarkers by use of a two-sided t-test with bootstrapping. Concordance index (C-index) and relative absolute error (RAE) were used as measures of the prediction performance. The bootstrap evaluation yielded C-index and RAE values of 80.4% and 15.6% for the pix2surv model, whereas those for the extent of the well-aerated lung parenchyma were 49.8% and 33.6%, and for a combination of blood tests of lactic dehydrogenase, lymphocyte, and C-reactive protein were 69.8% and 25.5%, respectively. The increase in survival prediction by the pix2surv model was statistically significant (p < 0.0001), indicating high effectiveness of the pix2surv model as a prognostic biomarker for the survival of patients with COVID-19.
Serrated polyps were historically believed to be benign lesions that have no cancer potential. However, recent studies have revealed a molecular pathway where serrated polyps can develop into colorectal cancers. Because serrated polyps tend to be flat and pale lesions, they are challenging to detect in colonoscopy, whereas CT colonography can detect serrated polyps based on a phenomenon called contrast coating. However, the differentiation of contrast coating from tagged feces requires great skill from the reader. The purpose of this pilot study was to explore the performance of 3D deep learning in the detection of serrated polyps. The materials included 94 CT colonography cases with biopsy-confirmed serrated polyps. We explored how to adapt the architecture of our baseline 3D DenseNet into the limited dataset by modification of the architectural parameters. The detection performance of the different 3D DenseNets and a reference 3D ResNet and a 3D AlexNet were compared by use of 10-fold cross-validation in terms of their sensitivity and false-positive rate within a clinically meaningful performance range by use of the free-response operating characteristic analysis. Our preliminary results indicate that the optimized 3D DenseNet can yield a high detection performance for serrated polyps that is comparable to those of state-of-the-art conventional CADe systems for traditional polyps in CT colonography.
We developed a novel survival prediction model for images, called pix2surv, based on a conditional generative adversarial network (cGAN), and evaluated its performance based on chest CT images of patients with idiopathic pulmonary fibrosis (IPF). The architecture of the pix2surv model has a time-generator network that consists of an encoding convolutional network, a fully connected prediction network, and a discriminator network. The fully connected prediction network is trained to generate survival-time images from the chest CT images of each patient. The discriminator network is a patchbased convolutional network that is trained to differentiate the “fake pair” of a chest CT image and a generated survivaltime image from the “true pair” of an input CT image and the observed survival-time image of a patient. For evaluation, we retrospectively collected 75 IPF patients with high-resolution chest CT and pulmonary function tests. The survival predictions of the pix2surv model on these patients were compared with those of an established clinical prognostic biomarker known as the gender, age, and physiology (GAP) index by use of a two-sided t-test with bootstrapping. Concordance index (C-index) and relative absolute error (RAE) were used as measures of the prediction performance. Preliminary results showed that the survival prediction by the pix2surv model yielded more than 15% higher C-index value and more than 10% lower RAE values than those of the GAP index. The improvement in survival prediction by the pix2surv model was statistically significant (P < 0.0001). Also, the separation between the survival curves for the low- and high-risk groups was larger with pix2surv than that of the GAP index. These results show that the pix2surv model outperforms the GAP index in the prediction of the survival time and risk stratification of patients with IPF, indicating that the pix2surv model can be an effective predictor of the overall survival of patients with IPF.
We developed and evaluated the effect of U-Net-based radiomic features, called U-radiomics, on the prediction of the overall survival of patients with idiopathic pulmonary fibrosis (IPF). To generate the U-radiomics, we retrospectively collected lung CT images of 72 patients with interstitial lung diseases. An experienced observer delineated regions of interest (ROIs) from the lung regions on the CT images, and labeled them into one of four interstitial lung disease patterns (ground-glass opacity, reticulation, consolidation, and honeycombing) or a normal pattern. A U-Net was trained on these images for classifying the ROIs into one of the above five lung tissue patterns. The trained U-Net was applied to the lung CT images of an independent test set of 75 patients with IPF, and a U-radiomics vector for each patient was identified as the average of the bottleneck layer of the U-Net across all the CT images of the patient. The U-radiomics vector was subjected to a Cox proportional hazards model with elastic-net penalty for predicting the survival of the patient. The evaluation was performed by using bootstrapping with 500 replications, where concordance index (C-index) was used as the comparative performance metric. The preliminary results showed the following C-index values for two clinical biomarkers and the U-radiomics: (a) composite physiologic index (CPI): 64.6%, (b) gender, age, and physiology (GAP) index: 65.5%, and (c) U-radiomics: 86.0%. The U-radiomics significantly outperformed the clinical biomarkers in predicting the survival of IPF patients, indicating that the U-radiomics provides a highly accurate prognostic biomarker for patients with IPF.
Colorectal cancer is the second leading cause of cancer deaths worldwide. Computed tomographic colonography (CTC) can detect large colorectal polyps and cancers at a high sensitivity, whereas it can miss some of the smaller but still clinically significant 6 – 9 mm polyps. Dual-energy CTC (DE-CTC) can be used to provide more detailed information about scanned materials than does conventional single-energy CTC. We compared the classification performance of a 3D convolutional neural network (DenseNet) with those of four traditional 3D machine-learning models (AdaBoost, support vector machine, random forest, Bayesian neural network) and their cascade and ensemble classifier variants in the detection of small polyps in DE-CTC. Twenty patients with colonoscopy-confirmed polyps were examined by DE-CTC with a reduced one-day bowel preparation. The traditional machine-learning models were designed to identify polyps based on native radiomic dual-energy features of the DE-CTC image volumes. The performance of the machine-learning models was evaluated by use of the leave-one-patient-out method. The DenseNet was trained with a large independent external dataset of single-energy CTC cases and tested on blended image volumes of the DE-CTC cases. Although the DenseNet yielded the highest detection accuracy for typical polyps, AdaBoost and its cascade classifier variant yielded the highest overall polyp detection performance.
KEYWORDS: 3D modeling, Computer aided diagnosis and therapy, 3D image processing, Medical imaging, Performance modeling, Computed tomography, Virtual colonoscopy, Visualization, Data modeling, Solid modeling
Three-dimensional (3D) convolutional neural networks (CNNs) can process volumetric medical imaging data in their native volumetric input form. However, there is little information about the comparative performance of such models in medical imaging in general and in CT colonography (CTC) in particular. We compared the performance of a 3D densely connected CNN (3D-DenseNet) with those of the popular 3D residual CNN (3D-ResNet) and 3D Visual Geometry Group CNN (3D-VGG) in the reduction of false-positive detections (FPs) in computer-aided detection (CADe) of polyps in CTC. VGG is the earliest CNN design of these three models. ResNet has been used widely as a de-facto standard model for constructing deep CNNs for image classification in medical imaging. DenseNet is the most recent of these models and improves the flow of information and reduces the number of network parameters as compared to those of ResNet and VGG. For the evaluation, we used 403 CTC datasets from 203 patients. The classification performance of the CNNs was evaluated by use of 5-fold cross-validation, where the area under the receiver operating characteristic curve (AUC) was used as the figure of merit. Each training fold was balanced by use of data augmentation of the samples of real polyps. Our preliminary results showed that the AUC value of the 3D-DenseNet (0.951) was statistically significantly higher than those of the reference models (P < 0.005), indicating that the 3D-DenseNet has the potential of substantially outperforming the other models in reducing FPs in CADe for CTC. This improvement was highest for the smallest polyps.
Imbalanced training data introduce important challenge into medical image analysis where a majority of the data belongs to a normal class and only few samples belong to abnormal classes. We propose to mitigate the class imbalance problem by introducing two generative adversarial network (GAN) architectures for class minority oversampling. Here, we explore balancing data distribution 1) by generating new sample from unsupervised GAN or 2) synthesize missing image modalities from semi-supervised GAN. We evaluated the effect of the synthetic unsupervised and semi-supervised GAN methods by use of 1,500 MR images for brain disease diagnosis, where the classification performance of a residual network was compared between unbalanced datasets, classic data augmentation, and the proposed new GAN-based methods.The evaluation results showed that the synthesized minority samples generated by GAN improved classification accuracy up to 18% in term of Dice score.
We developed a novel survival analysis model for images, called pix2surv, based on a conditional generative adversarial network (cGAN). The performance of the model was evaluated in the prediction of the overall survival of patients with rheumatoid arthritis-associated interstitial lung disease (RA-ILD) based on the radiomic 4D-curvature of lung CT images. The architecture of the pix2surv model is based on that of a pix2pix cGAN, in which a generator is configured to generate an estimated survival time image from an input radiomic image of a patient, and a discriminator attempts to differentiate the “fake pair” of the input radiomic image and a generated survival-time image from a “true pair” of the input radiomic image and the observed survival-time image of the patient. For evaluation, we retrospectively identified 71 RA-ILD patients with lung CT images and pulmonary function tests. The 4D-curvature images computed from the CT images were subjected to the pix2surv model for evaluation of their predictive performance with that of an established clinical prognostic biomarker known as the GAP index. Also, principal-curvature images and average principal curvatures were individually subjected, in place of the 4D-curvature images, to the pix2surv model for performance comparison. The evaluation was performed by use of bootstrapping with concordance index (C-index) and relative absolute error (RAE) as metrics of prediction performance. Preliminary result showed that the use of 4D-curvature images yielded C-index and RAE values that statistically significantly outperformed the use of the clinical biomarker as well as the other radiomic images and features, indicating the effectiveness of 4D-curvature images with pix2surv as a prognostic imaging biomarker for the survival of patients with RA-ILD.
We developed a novel ensemble three-dimensional residual network (E3D-ResNet) for the reduction of false positives (FPs) in computer-aided detection (CADe) of polyps on CT colonography (CTC). To capture the volumetric multiscale information of CTC images, each polyp candidate was represented with three different sizes of volumes of interest (VOIs), which were enlarged to a common size and were individually subjected to three 3D-ResNets. These 3D-ResNets were trained to calculate three polyp-likelihood probabilities, p1, p2 and p3, corresponding to each input VOI. The final polyp likelihood, p, was obtained as the maximum of p1, p2 and p3. We compared the classification performance of the E3D-ResNet with that of a non-ensemble 3D-ResNet, ensemble 2D-ResNet, and ensemble of 2D- and 3D-convolutional neural network (CNN) models. All models were trained and evaluated with 21,021 VOIs of polyps and 19,557 VOIs of FPs that were sampled with data augmentation from the CADe detections on the CTC data of 20 patients. We evaluated the classification performance of the models with receiver operating characteristics (ROC) analysis using cross-validation, where the area under the ROC curve (AUC) was used as the figure of merit. Preliminary results showed that AUC value (0.98) of the E3D-ResNet was significantly higher than that of the reference models (P < 0.001), indicating that the E3D-ResNet has the potential of substantially reducing the FPs in CADe of polyps on CTC.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.