This PDF file contains the front matter associated with SPIE Proceedings Volume 12931, including the Title Page, Copyright information, Table of Contents, and Conference Committee information.

A wealth of medical knowledge is used to make clinical decisions, yet treatment or disease outcomes are challenging to assess without clinical trials. However, clinical trials take time, are expensive, and are impossible to perform for every decision. One approach to systematically assess treatment outcomes involves the retrospective analysis of clinical notes, e.g., radiology and pathology reports. While such studies are often performed by clinicians who manually review the notes and other information, such retrospective analysis can benefit from the automated parsing of radiology and pathology reports to provide systematic framework to extract outcome information. In this study, we used a large language model, i.e., ChatGPT (GPT-3.5), to parse 267 radiology and pathology reports and extract information related to response to neoadjuvant chemotherapy in patients with breast cancer. Our study includes a heterogeneous group of 89 women who underwent neoadjuvant therapy and underwent two MRI exams, pre- and post-therapy, followed by surgery (lumpectomy or mastectomy). We assessed the treatment response based on clinical reports from the post-therapy surgical excision. From the reports, we extracted the number of lesions, their anatomic location, and size. Our study provides insight into neoadjuvant chemotherapy response, indicating that even cases with complete MRI response can still have residual invasive breast carcinoma (1/3 of subjects), and, on the other hand, even those with reduced MRI response (⪅30% reduction in tumor size) can have no residual tumor at surgery, indicating that when cancer responds to treatment, it may not be captured by the MRI. The large language model achieved sensitivities of 84-94% in extracting the information from radiology reports, but had lower performance in the pathology reports, 72-87%, where more information is provided in free format. While this study is preliminary and performed in a small cohort, it illustrates the complexity of outcome prediction using radiology images.

Blinded Independent Central Review (BICR) is recommended by the US FDA for registration of oncology trials as image assessment bias is avoided and no chance of unblinding of patient data. Double read with adjudication is the method used to reduce endpoint assessment variability. In cases of disagreement between the readers, a third reader called an adjudicator, reviews the assessment by the two radiologists and decides which assessment is most accurate. Adjudication Rate (AR) and Adjudicator Agreement Rate (AAR) are the two indicators used to evaluate reviewer performance and overall trial variability and quality. Sentiment Analysis (SA) is based on natural language processing and can tag the data as ‘positive’, ‘negative’ or ‘neutral’ although current technologies can provide a more complex analysis of emotions in the written text. Medical SA can analyze patients’ and doctors’ opinions, sentiments, attitudes, and emotions in the clinical background. Python, the most frequently used programming language for deep learning worldwide and ChatGPT, an AI-based chatbot can be used for assessing adjudicator comment quality based on sentiment analysis. If successful, this analysis can open another novel implementation for Large Language Models (LLMs) or ChatGPT in clinical research and medical imaging. This prospective study involved the review of cases for 100 subjects by board-certified radiologists using the Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria. The study employed a double read with adjudication paradigm in a central imaging review setup. The agreement of adjudication was assessed and compared with the overall response, agreed reader, and medical text. The medical text entered by the adjudicator is usually a free text field that typically lacks standardization and control over its content, which may affect its correlation with reviewer selection for agreement. Although uncommon, errors by the adjudicator can occur due to ambiguous text, mis-clicks, or application delay errors. To analyze the adjudicator’s comments, sentiment analysis was conducted using a Python plug-in with ChatGPT as a large language model. Based on this analysis, the subjects were categorized as either having “Potential Error” or “No Error”. The algorithm supported by ChatGPT was evaluated against a Gold Standard, determined by a board-certified radiologist with over 20 years of experience in the BICR process. A comparison was made to assess accuracy and reproducibility, revealing that only four out of 100 subjects had different outcomes. The sensitivity was calculated as 0.857, specificity as 1.0, and accuracy as 0.96. The remarkable Natural Language Processing (NLP) capabilities of ChatGPT are evident in its ability to classify the sentiment as positive, negative, or neutral based on the free-text adjudicator comments provided during the review process. This classification enables a comparison with the actual assessment, adjudicator agreement, and overall patient outcome, highlighting the impressive performance of ChatGPT in this regard.

Percutaneous liver ablation is a minimally invasive procedure to treat liver tumors. Postablation images are highly significant as they distinguish normal post-procedure changes from abnormalities, preventing unnecessary retreatment and confirming procedural quality. However, the cancer surveillance imaging reports after the procedure can be numerous and challenging to read. Moreover, annotated data is limited in this setting. In this study we used the cutting-edge large language model Llama 2 to automatically extract critical findings from real-world diagnostic imaging reports without the need of training a new information extraction model. This could potentially automate part of the outcome research and registry construction process, as well as decrease the number of studies needed to review for research purposes. A dataset of 87 full-text reports from 13 patients who underwent percutaneous thermal ablation for pancreatic liver metastases were used to benchmark the capability of Llama 2 for cancer progression finding extraction and classification. We asked Llama 2 to determine whether there is cancer progression within the given report and then classify progression findings into Local Tumor Progression (LTP), Intrahepatic Progression (IHP) and Extrahepatic Progression (EHP). Llama 2 achieved decent performance for detecting progression at study level. The precision is 0.91 and recall is 0.96, with specificity 0.84. However, the classification of progression into LTP, IHP and EHP still needs to be improved.

Primary Angle Closure Disease (PACD) is the most common cause of vision impairment worldwide. Early treatment is crucial in preventing vision loss. Anterior Segment Optical Coherence Tomography (AS-OCT) is an imaging modality that produces images of anterior structures such as the Anterior Chamber Angle (ACA) and the scleral spur. However, adoption of the AS-OCT modality has been gradual due to AS-OCT analysis not being standardized and inefficient. medical professionals typically must annotate each image by hand using proprietary software and use expert knowledge to diagnose PACD based on the key features annotated. Using an imaging-informatics-based approach on a dataset of almost 1200 images, we have developed a DICOM-compatible system to streamline and standardize AS-OCT analysis, utilizing a HIPAA-compliant database requiring a secure login to protect patient privacy. Previously, we developed a streamlined approach towards annotating key features in AS-OCT images which will be used to validate the results produced by SimpleMind – an open-source software framework supporting deep neural networks with machine learning and automatic parameter tuning. SimpleMind is integrated into the system to increase the efficiency of analyzing AS-OCT images and eliminate the need to annotate images for clinical diagnosis. The goal is to develop a comprehensive and robust hybrid system combining traditional and deep learning image processing methods to detect the scleral spur and estimate a measure of the anterior chamber angle’s degree of openness from AS-OCT images. This paper presents a hybrid method of determining the ACA boundary region to produce an angle measurement that can help indicate PACD.

The “STAT” designation for imaging studies is often overused and misused, obscuring the actual urgency of an imaging order. Not all STAT imaging orders are equal in terms of urgency, so we create semi-supervised machine learning models to classify actual urgency of the STAT imaging studies with more than 20,000 studies, even though only a small subset of data in the training set was manually labeled by the experts.

Radiation Therapy seeks to treat cancers through the dosage of destructive radiation to target volumes. The treatment plans, detailing the application of radiation dosage, are currently created to adhere to formal guidelines and target dose levels based on physician experience and trial-and-error rather than standard quantitative methods. We propose a web-based informatics application to introduce data driven methods and uniformity into radiation therapy treatment plan creation. We use a quantitative comparison of tumor position and structural anatomy between retrospective cases and a current case undergoing treatment planning to identify useful and relevant retrospective treatment plans for use as templates and reference during current treatment plan creation. The system is based on a database of 403 retrospective DICOM RT objects from University of California Los Angeles and State University of New York Buffalo; Roswell Park as well as the quantitative features we extract from each case. The quantitative identifiers we develop and use in our feature extraction process are the Overlap Value Histogram (OVH) and the Spatial Target Similarity (STS) calculated between the tumor volume and each Organ At Risk (OAR) of irradiation. The similarity between each retrospective case and the current case is the gower’s distance sum of all the earth mover’s distance values calculated between the OVHs and STSs for each OAR in common between the two cases. By calculating the similarity between the current case and each retrospective case we construct a similarity index from which clinicians can select cases they deem useful in their current treatment planning process. Case outcomes will be stored in our database allowing the discovery of correlations between the structural anatomy, tumor position, treatment plans, and outcome, enabling treatment plan benchmarking. These methods allow our informatics system to increase usage of data driven methodologies and standardized practices in radiation therapy treatment planning.

A critical, often overlooked barrier to implementing automated analysis tools in the clinical setting is identifying the subset of acquired in a scanning session DICOM objects appropriate for automated analysis. Although the DICOM standard has rich metadata with specific fields describing the collected sequence, the text input description fields are often unreliable due to the lack of rigorous constraints. Automating clinical and research applications requires better identification and selection processes for the increasing utilization of image processing applications, CAD systems, and the need for huge multi-site datasets with data from multiple source devices and manufacturers. The medical imaging field urgently needs a tool for automated image-type classification. In this work, we developed a robust, easily extensible classification framework that extracts key features from well-characterized DICOM header fields to identify image modality and acquisition plane. Utilizing classical machine learning paradigms and a heterogeneous dataset of over 250 thousands scan volumes collected over 50 sites, using 77 scanners models, we achieved 98.9% accuracy during the K-Fold Cross-Validation for classifying 12 image modalities and 99.96% accuracy on image acquisition plane classification. Furthermore, we demonstrated model generalizability by achieving 95.7% accuracy on out-of-sample animal data. Our proposed framework can be crucial in eliminating error-prone human interaction, allowing automatization, and increasing imaging applications’ reliability and efficiency. The proposed framework has been released as an open-source project and is readily accessible as a Python pip package under the name dcm-classifier.

The persistent need for more qualified personnel in operating theatres exacerbates the remaining staff’s workload. This increased burden can result in substantial complications during surgical procedures. To address this issue, this research project works on a comprehensive operating theatre system. The system offers real-time monitoring of all surgical instruments in the operating theatre, aiming to alleviate the problem. The foundation of this endeavor involves a neural network trained to classify and identify eight distinct instruments belonging to four distinct surgical instrument groups. A novel aspect of this study lies in the approach taken to select and generate the training and validation data sets. The data sets used in this study consist of synthetically generated image data rather than real image data. Additionally, three virtual scenes were designed to serve as the background for a generation algorithm. This algorithm randomly positions the instruments within these scenes, producing annotated rendered RGB images of the generated scenes. To assess the efficacy of this approach, a separate real data set was also created for testing the neural network. Surprisingly, it was discovered that neural networks trained solely on synthetic data performed well when applied to real data. This research paper shows that it is possible to train neural networks with purely synthetically generated data and use them to recognize surgical instruments in real images.

The difference between chronological age and predicted biological brain age, the so-called “brain age gap”, is a promising biomarker for assessment of overall brain health. It has also been suggested as a biomarker for early detection of neurological and cardiovascular conditions. The aim of this work is to identify group-level variability in the brain age gap between healthy subjects and patients with neurological and cardiovascular diseases. Therefore, a deep convolutional neural network was trained on UK Biobank T1-weighted-MRI datasets of healthy subjects (n=6860) to predict brain age. After training, the model was used to determine the brain age gap for healthy hold-out test subjects (n=344), and subjects with neurological (n=2327) or cardiovascular (n=6467) diseases. Next, saliency maps were analyzed to identify brain regions used by the model to render decisions. Linear bias correction was implemented to correct for the bias of age predictions made by the model. The trained model after bias correction achieved an average brain age gap of 0.05 years for the healthy test cohort while the neurological disease test cohort had an average brain age gap of 0.7 years, and the cardiovascular disease test cohort had an average brain age gap of 0.25 years. The average saliency maps appear similar for the three test group, suggesting that the model mostly uses brain areas associated with general brain aging patterns. This works results indicate potential in the brain age gap for differentiation of neurologic and cardiac patients from healthy aging patterns supporting its use as a novel biomarker.

SF-CT-PD is a single-file derivative of the DICOM-CT-PD file format for CT projection data that stores projections within a single DICOM file, stores pixel data detector-row-major, and stores projection-specific parameters as ordered tables within the DICOM header. We compared the performance of SF-CT-PD against DICOM-CT-PD in read speed, disk usage, and network transfer. Cases were sampled from TCIA’s “LDCT-and-Projection-data" dataset and encoded into DICOM-CT-PD and SF-CT-PD representations. Read tests were conducted for four programming languages on hard-disk and solid-state drives. Rsync-based network transfer analysis measured the Ethernet throughput for each format. Accuracy of the implementation was confirmed by analyzing reconstructions and transfer file-checksums for each format. SF-CT-PD was generally more performant in read operations and disk usage. Network throughput was equivalent between the formats, with file-checksums indicating file integrity. Reconstruction accuracy was supported by difference image agreements. SF-CT-PD represents a viable extension of DICOM-CT-PD where a single file is preferred.

Data shift, also known as dataset shift, is a prevalent concern in the field of machine learning. It occurs when the distribution of the data used for training a machine learning model is different from the distribution of the data the model will encounter in a real-world, operational environment (i.e., test set). This issue becomes even more significant in the field of medical imaging due to the multitude of factors that can contribute to data shifts. It is crucial for medical machine learning systems to identify and address these issues. In this paper, we present an automated pipeline designed to identify and alleviate certain types of data shift issues in medical imaging datasets. We intentionally introduce data shift into our dataset to assess and address it within our workflow. More specifically, we employ Principal Components Analysis (PCA) and Maximum Mean Discrepancy (MMD) algorithms to detect data shift between the training and test datasets. We utilize image processing techniques, including data augmentation and image registration methods, to individually and collectively mitigate data shift issues and assess their impacts. In the experiments we use a head CT image dataset of 537 patients with severe traumatic brain injury (sTBI) for patient outcome prediction. Results show that our proposed method is effective in detecting and significantly improving model performance.

Medical image auto-segmentation techniques are basic and critical for numerous image-based analysis applications that play an important role in developing advanced and personalized medicine. Compared with manual segmentations, auto-segmentations are expected to contribute to a more efficient clinical routine and workflow by requiring fewer human interventions or revisions to auto-segmentations. However, current auto-segmentation methods are usually developed with the help of some popular segmentation metrics that do not directly consider human correction behavior. Dice Coefficient (DC) focuses on the truly segmented areas, while Hausdorff Distance (HD) only measures the maximal distance between the auto-segmentation boundary with the ground truth boundary. Boundary length-based metrics such as surface DC (surDC) and Added Path Length (APL) try to distinguish truly predicted boundary pixels and wrong ones. It is uncertain if these metrics can reliably indicate the required manual mending effort for application in segmentation research. Therefore, in this paper, the potential use of the above four metrics, as well as a novel metric called Mendability Index (MI), to predict the human correction effort is studied with linear and support vector regression models. 265 3D Computed Tomography (CT) samples for three objects of interest from three institutions with corresponding auto-segmentations and ground truth segmentations are utilized to train and test the prediction models. The five-fold cross-validation experiments demonstrate that meaningful human effort prediction can be achieved using segmentation metrics with varying prediction errors for different objects. The improved variant of MI, called MIhd, generally shows the best prediction performance, suggesting its potential to indicate reliably the clinical value of auto-segmentations.

For deep learning-based machine learning, not only are large and sufficiently diverse data crucial but their good qualities are equally important. However, in real-world applications, it is very common that raw source data may contain incorrect, noisy, inconsistent, improperly formatted and sometimes missing elements, particularly, when the datasets are large and sourced from many sites. In this paper, we present our work towards preparing and making image data ready for the development of AI-driven approaches for studying various aspects of the natural history of oral cancer. Specifically, we focus on two aspects: 1) cleaning the image data; and 2) extracting the annotation information. Data cleaning includes removing duplicates, identifying missing data, correcting errors, standardizing data sets, and removing personal sensitive information, toward combining data sourced from different study sites. These steps are often collectively referred to as data harmonization. Annotation information extraction includes identifying crucial or valuable texts that are manually entered by clinical providers related to the image paths/names and standardizing of the texts of labels. Both are important for the successful deep learning algorithm development and data analyses. Specifically, we provide details on the data under consideration, describe the challenges and issues we observed that motivated our work, present specific approaches and methods that we used to clean and standardize the image data and extract labelling information. Further, we discuss the ways to increase efficiency of the process and the lessons learned. Research ideas on automating the process with ML-driven techniques are also presented and discussed. Our intent in reporting and discussing such work in detail is to help provide insights in automating or, minimally, increasing the efficiency of these critical yet often under-reported processes.

Federated Learning (FL) has attracted increasing attention in medical imaging as an alternative to centralized data sharing that can leverage a large amount of data from different hospitals to improve the generalization of machine learning models. However, while FL can provide certain protection for patient privacy by retaining data in local client hospitals, data privacy could still be compromised when exchanging model parameters between local clients and servers. Meanwhile, although efficient training strategies are actively investigated, significant communication overhead remains a major challenge in FL as it requires substantial model updates between clients and servers. This becomes more prominent when more complex models, such as transformers, are introduced in medical imaging and when geographically distinct collaborators are involved in FL studies for global health problems. To this end, we proposed FeSEC, a secure and efficient FL framework, to address these two challenges. In particular, we firstly consider a sparse compression algorithm for efficient communication among the distributed hospitals, and then we ingrate the homomorphic encryption with differential privacy to secure data privacy during model exchanges. Experiments on the task of COVID-19 detection show the proposed FeSEC substantially improves the accuracy and privacy preservation of FL models compared to FedAvg with less than 10% of communication cost.

Fully automated organ segmentation on Computed Tomography (CT) images is an important first step in many medical applications. Many different Deep Learning (DL) based approaches are being actively developed for this task. However, often it is hard to make a direct comparison between two segmentation methods. We tested the performance of two deep learning-based CT on an independent dataset of CT scans. Algorithm-1 performed much better on the segmentation of the kidney. In contrast, the performance of the two algorithms was similar for the segmentation of the liver. For both algorithms, a number of outliers (Dice <= 0.5) were observed. With limited scan acquisition parameters, it was not possible to diagnose the root cause for the outliers. This work highlights the urgent need for complete DICOM header curation. The DICOM header information could help to pin-point the scanning parameters that lead to segmentation errors by Deep Learning algorithms.

In our previous study, we developed a Conditional GAN (CGAN) model, pix2pix, to simulate one side of a mammogram by using the other side as a condition image. Despite generating plausible mammograms, various artifacts appeared in some generated mammograms. As our model uses each woman’s breast mammogram as condition, patient-specific breast tissue characteristics can affect resulting GAN simulated mammograms, as well as artifacts. This study therefore analyzed the potential relationship of GAN generated mammographic artifacts with patient variables closely related to breast tissue characteristics, which are age and breast density (mammographic percent density). We trained our CGAN using Craniocaudal (CC) views of 1366 normal/healthy women. Using trained CGAN, we synthesized mammograms of 333 women with dense breasts, where 97 had unilateral mammographically-occult breast cancer. We found four artifact types, checkerboard, breast boundary, nipple-areola, and black spots around calcifications (black spots) – with an overall incidence rate of 69%. We then evaluated if there was a systematic difference in age and breast density of GAN simulated mammograms with and without artifacts. The results show no meaningful difference in age and breast density between the simulated mammograms with and without checkerboard artifact (p⪆0.05). For breast boundary and nipple areola artifacts, we found that those artifacts appeared more on denser breasts (p⪅0.001). For black spot artifacts, it appeared more on older women (p⪅0.003) and on less dense breasts (p⪅0.001). In summary, this study found that there is possible correlation between patient variables and the chance of having certain artifacts on CGAN simulated mammograms.

The generation of valid and realistic dental crown bottoms plays a central role in dentistry, as dental crown bottoms are the first point of contact between a tooth preparation and its crown. Every tooth is different, and the retention of the crown bottom heavily depends on how well it fits the preparation while conserving essential properties for ceramic adhesion and smoothness. From this, the generation of the crown bottom becomes a difficult task that only qualified individuals such as dental technicians can complete. Standard geometric modelling techniques such as Computer-Aided Design (CAD) software programs have since been used for this purpose, providing a reliable basis for the generation of dental crown bottoms. Conversely, recent improvements in deep learning have presented new avenues in shape generation tasks that allow for personalized shapes to be created in a short period of time based on learned context. Starting from a set of preparation shapes, this project seeks to compare the efficacy of automatic geometric techniques to deep learning methods in the framework of dental crown bottom shape generation. Results show that deep learning methods such as GANs demand no human manipulation and provide similar visual results to the geometric model on unseen cases in an unsupervised manner. Our code is available at https://github.com/ImaneChafi/C.B.GEN and https://github.com/ImaneChafi/Prep-GAN

We developed a novel 3D generative Artificial Intelligence (AI) method for performing Electronic Cleansing (EC) in CT Colonography (CTC). In the method, a 3D transformer based UNet is used as a generator to map an uncleansed CTC image volume directly into a virtually cleansed CTC image volume. A 3D-PatchGAN is used as a discriminator to provide feedback to the generator to improve the quality of the EC images generated by the 3D transformer-based UNet. The EC method was trained by use of the CTC image volumes of an anthropomorphic phantom that was filled partially with a mixture of foodstuff and an iodinated contrast agent. The CTC image volume of the corresponding empty phantom was used as the reference standard. The quality of the EC images was tested visually with six clinical CTC test cases and quantitatively based on a phantom test set of 100 unseen sample image volumes. The image quality of EC was compared with that of a previous 3D GAN-based EC method. Our preliminary results indicate that the 3D generative AI-based EC method outperforms our previous 3D GAN-based EC method and thus can provide an effective EC method for CTC.

Deep learning techniques for medical image analysis have reached comparable performance to medical experts, but the lack of reliable explainability leads to limited adoption in clinical routine. Explainable AI has emerged to address this issue, with causal generative techniques standing out by incorporating a causal perspective into deep learning models. However, their use cases have been limited to 2D images and tabulated data. To overcome this, we propose a novel method to expand a causal generative framework to handle volumetric 3D images, which was validated through analyzing the effect of brain aging using 40196 MRI datasets from the UK Biobank study. Our proposed technique paves the way for future 3D causal generative models in medical image analysis.

Data scarcity and data imbalance are two major challenges in training deep learning models on medical images, such as brain tumor MRI data. The recent advancements in generative artificial intelligence have opened new possibilities for synthetically generating MRI data, including brain tumor MRI scans. This approach can be a potential solution to mitigate the data scarcity problem and enhance training data availability. This work focused on adapting the 2D latent diffusion models to generate 3D multi-contrast brain tumor MRI data with a tumor mask as the condition. The framework comprises two components: a 3D autoencoder model for perceptual compression and a conditional 3D Diffusion Probabilistic Model (DPM) for generating high-quality and diverse multi-contrast brain tumor MRI samples, guided by a conditional tumor mask. Unlike existing works that focused on generating either 2D multi-contrast or 3D single-contrast MRI samples, our models generate multi-contrast 3D MRI samples. We also integrated a conditional module within the UNet backbone of the DPM to capture the semantic class-dependent data distribution driven by the provided tumor mask to generate MRI brain tumor samples based on a specific brain tumor mask. We trained our models using two brain tumor datasets: The Cancer Genome Atlas (TCGA) public dataset and an internal dataset from the University of Texas Southwestern Medical Center (UTSW). The models were able to generate high-quality 3D multi-contrast brain tumor MRI samples with the tumor location aligned by the input condition mask. The quality of the generated images was evaluated using the Fréchet Inception Distance (FID) score. This work has the potential to mitigate the scarcity of brain tumor data and improve the performance of deep learning models involving brain tumor MRI data.

Towards the goal of developing an informed, intuitive, and generalized artificial intelligence model for the early-stage diagnosis of Colorectal Cancer (CRC), in this work, we present a generative model-based technique to improve the training and generalization performance of machine learning classification algorithms. Through this approach, we address the challenge of acquiring sizable and well-balanced datasets within the clinical domain. Our methodology involves training generative models on already available medical data, learning the latent representations, and finally generating new synthetic samples to be used for downstream tasks. We train dedicated UNet2D-based Denoising Diffusion Probabilistic Models (DDPMs) using our custom dataset, which consists of textural images captured by our novel Vision-based Tactile Sensor (VS-TS), called Hysense. These UNet2D DDPMs are employed to generate synthetic images for each potential class. To thoroughly study the effectiveness of using synthetic images during training, we compared the performance of multiple classification models, ranging from simple to state-of-the-art approaches, with our evaluation focusing solely on real images. Specifically for our dataset, we also extend the use of dedicated UNet2D DDPMs to generate synthetic images of not just possible classes, but also other features that may be present in the image, such as whole or partial contact of sensor with polyp phantoms. Through our experimental analyses, we demonstrated that the utilization of generative models to enrich existing datasets with synthetic images leads to improved classification performance and a reduction in model biases.

Lesion segmentation in medical images, particularly for Bone Marrow Edema-like Lesions (BMEL) in the knee, faces challenges due to imbalanced data and unreliable annotations. This study proposes an unsupervised deep learning method with the use of conditional diffusion models coupled with inpainting tasks for anomaly detection. This innovative approach facilitates the detection and segmentation of BMEL without human intervention, achieving a DICE testing score of 0.2223. BMEL has been shown to correlate and predict disease progression in several musculoskeletal disorders, such as osteoarthritis. With further development, our method has great potential for fully automated analysis of BMEL to improve early diagnosis and prognosis for musculoskeletal disorders. The framework can be extended to other lesion detection as well.

This study proposes an innovative 3D diffusion-based model called the Cycle-consistency Geometric-integrated X-ray to Computed Tomography Denoising Diffusion Probabilistic Model (X-CBCT-DDPM). The X-CBCT-DDPM is designed to effectively reconstruct volumetric Cone-Beam CBCTs (CBCTs) from a single X-ray projection from any angle, reducing the number of required projections and minimizing patient radiation exposure in acquiring volumetric images. In contrast to the traditional DDPMs, the X-CBCT-DDPM utilizes dual DDPMs: one for generating full-view x-ray projections and another for volumetric CBCT reconstruction. These dual networks synergistically enhance each other's learning capabilities, leading to improved reconstructed CBCT quality with high anatomical accuracy. The proposed patient-specific X-CBCT-DDPM was tested using 4DCBCT data from ten patients, with each patient's dataset comprising ten phases of 3D CBCTs to simulate CBCTs and Cone-Beam X-ray projections. For model training, eight phases of 3D CBCTs from each patient were utilized, with one for validation purposes and the remaining one reserved for final testing. The X-CBCT-DDPM exhibits superior performance to DDPM, conditional Generative Adversarial Networks (GAN), and Vnet, in terms of various metrics, including a Mean Absolute Error (MAE) of 36.36±4.04, Peak Signal-to-Noise Ratio (PSNR) of 32.83±0.98, Structural Similarity Index (SSIM) of 0.91±0.01, and Fréchet Inception Distance (FID) of 0.32±0.02. These results highlight the model's potential for ultra-sparse projection-based CBCT reconstruction.

While many academic projects develop software, methods, and/or products that may be of broad interest, few are licensed for use outside the institution of origin. Within an academic setting, there are key challenges to building and sustaining healthcare technologies and translating them into widely available tools with a national or global community and user base. Hurdles include identifying broad and significant gaps and needs, acquiring funding for developers, project management, user support, implementing commercial grade development processes and user experience design, and choosing a sustainable financial model and licensing plan. In addition, moving beyond the academic sphere into the commercial realm requires an investment in business processes and skills, including the need for branding, marketing, sales, business development, operating, infrastructure, regulatory/compliance, legal, and fundraising expertise. This report will share experiences and insights based on imaging informatics platform licensing, illustrated with the following examples: First, a clinical trials imaging informatics platform will be discussed, developed initially to manage all the clinical trials imaging assessments within a premier Comprehensive Cancer Center. It was then licensed, initially through multi-center academic licensing, and now licensed commercially for use in over 4,200 active clinical trials at 22 organizations, including 12 NCI-designated cancer centers. Second, a web-based medical imaging framework and its underlying libraries will be covered, an open-source software platform that has become the standard for over 1,000 academic and industry software projects. Providing a road map for translational licensing from academia may help guide other projects to enable use beyond the institution of origin.

Diet, lifestyle and an aging population have led to many diseases, some of which can be seen well in the eyes and analyzed by simple means, such as OCT (Optical Coherence Tomography) scans. This article presents a comparative study examining transfer learning methods for classifying retinal OCT scans. The study focuses on the classification of several retina alterations such as Age-related Macular Degeneration (AMD), Choroidal Neovascularization (CNV), Diabetic Macular Edema (DME) and normal cases. The approach was evaluated on a large dataset of labeled OCT scans. In this work we use CNN architectures such as VGG16, VGG19, ResNet50, MobileNet, InceptionV3 and Xception with the weights pre-trained on ImageNet and then fine-tuned on the domain-specific data. The results indicate that the proposed transfer learning is a powerful tool for classifying multi-class retinal OCT scans.

Radiofrequency ablation (RFA) with continuous ultrasonography (US) monitoring is a non-surgical alternative to traditional thyroid surgery for treating benign symptomatic thyroid nodules. Monitoring nodules over time through US imaging is used to determine procedural success, primarily indicated by measured volume reduction. These images also capture other rich clinical characteristics that we believed could be systematically interrogated across patients to better understand and stratify nodule response to RFA. We performed radiomic texture analysis on 56 preoperative and postoperative US thyroid nodule images from patients treated with RFA that generated 767 radiomic feature (RFs) measurements. Using dimensionality reduction and clustering of thyroid nodules by their US image texture features, unique populations of nodules were discovered that suggest these methods combined with radiomics texture analysis as a useful system for stratifying thyroid nodules. Additionally, individual texture features were found to be different between nodules with successful and unsuccessful outcomes, further supporting radiomics features as potential biomarkers for RFA-treated thyroid nodules.

This study presents a lightweight pipeline for skin lesion detection, addressing the challenges posed by imbalanced class distribution and subtle or atypical appearances of some lesions. The pipeline is built around a lightweight model that leverages ghosted features and the DFC attention mechanism to reduce computational complexity while maintaining high performance. The model was trained on the HAM10000 dataset, which includes various types of skin lesions. To address the class imbalance in the dataset, the synthetic minority over-sampling technique and various image augmentation techniques were used. The model also incorporates a knowledge-based loss weighting technique, which assigns different weights to the loss function at the class level and the instance level, helping the model focus on minority classes and challenging samples. This technique involves assigning different weights to the loss function on two levels - the class level and the instance level. By applying appropriate loss weights, the model pays more attention to the minority classes and challenging samples, thus improving its ability to correctly detect and classify different skin lesions. The model achieved an accuracy of 92.4%, a precision of 84.2%, a recall of 86.9%, a f1-score of 85.4% with particularly strong performance in identifying Benign Keratosis-like Lesions (BKL) and Nevus (NV). Despite its superior performance, the model's computational cost is considerably lower than some models with less accuracy, making it an optimal solution for real-world applications where both accuracy and efficiency are essential.

Worldwide, breast cancer presents a significant health challenge, necessitating innovative techniques for early detection and prognosis. Although mammography is the established screening method, it has drawbacks, including radiation exposure and high costs. Recent studies have explored the application of machine learning to frontal infrared images for breast cancer detection. However, the potential of infrared imaging from angular views has not been thoroughly explored. In this paper, we investigate, develop, and evaluate classification models for breast cancer diagnosis using lateral and oblique infrared images. Our approach incorporates radiomic features and convolutional neural networks along with various feature fusion techniques to train deep neural networks. The primary objective is to determine the suitability of angular views for breast cancer detection, identify the most effective view, and assess its impact on classification accuracy. Utilizing the publicly available Database for Mastology Research with Infrared Images (DMR-IR), we apply an image processing pipeline for image improvement and segmentation. Additionally, we extract features using two strategies: radiomic features and convolutional neural network features. Subsequently, we conduct a series of k-fold cross-validation experiments to determine whether the features and feature fusion techniques are effective. Our findings indicate that oblique images, particularly when combined with DenseNet features, demonstrate superior performance. We achieved an average accuracy of 97.74%, specificity of 95.25%, and an F1 score of 98.24%. This study contributes to the advancement of machine learning in early breast cancer detection and underscores the significant potential of angular views in thermal infrared imaging, leading to improved diagnostic outcomes for patients worldwide.

The advancement of multi-slice CT technology has enabled exact analysis, diagnosis, and treatment of renal and renal tumors. Using contrast agents during CT imaging allows for more accurate acquisition of information regarding blood vessels, organs, and lesions. In this study, we employed three-phase contrast-enhanced CT images of the abdominal region to analyze small-sized renal tumors, explicitly focusing on challenging-to-distinguish types such as papillary Renal Cell Carcinoma (pRCC), chromophobe Renal Cell Carcinoma (RCC), and oncocytoma. The goal is to quantitatively characterize the features of renal tumors and strive for a high-precision discriminatory classification.

Radiomics, a burgeoning field within medical imaging, has gained momentum for its potential to provide nuanced insights into lesion characteristics. This study introduces a pioneering approach to benchmarking radiomic features, delving into their correlations with ground truth measurements and subsequent clustering patterns. By analyzing 59 simulated lesions mined from the publicly available QIDW Liver II Hybrid Dataset, a vast array of radiomic features were extracted and evaluated using Pyradiomics. A total of 2060 features are correlated with ground truth volume and contrast measurements, revealing intricate relationships. Novel co-clustering patterns emerge, underscoring the versatility and complexity of radiomic features. These findings not only contribute to lesion characterization precision but also advanced our understanding of radiomic intricacies. The presented research holds potential to refine radiomics-driven medical insights, paving the way for more informed clinical decision-making and improved patient care.

The world health organization’s global Tuberculosis (TB) report for 2022 identifies TB, with an estimated 1.6 million, as a leading cause of death. The number of new cases has risen since 2020, particularly the number of new drug-resistant cases, estimated at 450,000 in 2021. This is concerning, as treatment of patients with drug resistant TB is complex and may not always be successful. The NIAID TB Portals program is an international consortium with a primary focus on patient centric data collection and analysis for drug resistant TB. The data includes images, their associated radiological findings, clinical records, and socioeconomic information. This work describes a TB Portals’ chest x-ray-based image retrieval system which enables precision medicine. An input image is used to retrieve similar images and the associated patient specific information, thus facilitating inspection of outcomes and treatment regimens from comparable patients. Image similarity is defined using clinically relevant biomarkers: gender, age, Body Mass Index (BMI), and the percentage of lung affected per sextant. The biomarkers are predicted using variations of the DenseNet169 convolutional neural network. A multi-task approach is used to predict gender, age and BMI incorporating transfer learning from an initial training on the NIH Clinical Center CXR dataset to the TB portals dataset. The resulting gender AUC, age and BMI mean absolute errors were 0.9854, 4.03years and 1.67 kg|m². For the percentage of sextant affected by lesions the mean absolute errors ranged between 7% to 12% with higher error values in the middle and upper sextants which exhibit more variability than the lower sextants. The retrieval system is currently available from https://rap.tbportals.niaid.nih.gov/find similar cxr.

The incidence rate for Type 2 Diabetes Mellitus (T2DM) has been increasing over the years. T2DM is a common lifestyle-related disease and predicting its occurrence before five years could help patients to alter their lifestyle ahead and hence prevent T2DM. We intend to investigate the feasibility of radiomics features in predicting the occurrence of T2DM using screening mammography images which could benefit us in terms of the preventability of the disease. This study has examined the prevalence of T2DM using 110 positive samples (developed T2DM after 5 years) and 202 negative samples (did not develop T2DM after five years). The whole breast region was selected as the Region Of Interest (ROI), from which radiomics features were to be extracted. The mask was created from every image using a modified threshold value (by Otsu's binarization method) to obtain a binary image of the breast. 668 radiomics features were then extracted and analyzed using different machine learning algorithms built in the Python programming language such as Random Forest (RF), Gradient Boosting Classifier (GBC), and Light-Gradient Boosting Model (LGBM) as they could give excellent classification and prediction results. A five-fold cross-validation method was carried out; the accuracy, sensitivity, specificity and AUC were calculated when implementing each of the algorithms, and hyperparameter tuning was carried out to tune the models for better performance. The RF and GBC produced good accuracy results (⪆ 70%), but low sensitivity values. LGBM’s accuracy is almost 70% but it has the highest sensitivity (43.9%) and decent specificity (74.4%).

Lung cancer, in a majority of cases associated with smoking, ranks as the second leading type of cancer globally. The predominant forms are non-small cell carcinoma and small cell carcinoma. Lung cancer is diagnosed based on biopsies, surgical resection specimens, or cytology. Standard work-up of histopathological lung cancer samples includes Immunohistochemistry (IHC) staining, which allows the visualization of specific proteins expressed on cellular structures in the sample. The present use case to focus on tumor/stroma and immune cell evaluation in non-small cell lung cancer is clinically relevant. As computational methods become increasingly adopted in clinical settings, they are frequently employed, for instance, to quantify tumor cell content prior to processing lung cancer biopsies for molecular pathology analyses. Moreover, computational methods play a key role in evaluating immune cells and detecting immune checkpoint markers in distinct tissue sections. These analyses are essential for designing targeted immuno-oncology treatments. Current pathological analysis of these samples is both time-intensive and challenging, often hinging on the expertise of a few highly skilled pathologists. This reliance can introduce variability in diagnoses, possibly leading to inconsistent patient outcomes. An automated solution using computer vision, however, has the potential to assist pathologists in achieving a more accurate and consistent diagnosis. Our paper introduces a novel approach that leverages deep unsupervised learning techniques to autonomously label regions within IHC-stained samples. By extracting radiomic features from small patches in whole slide images and utilizing Self-Organizing Maps, we developed a robust clustering model. Additionally, we introduced a novel database of IHC-stained lung cancer pathological images. Our findings indicate that unsupervised clustering is a promising approach to meet the increasing demand for high-quality annotations in the emerging field of computational pathology.

Pneumoconiosis, an occupational respiratory illness triggered by inhaling mineral dust with increasing prevalence and severity worldwide. Chest radiograph plays a vital role in the screening of pneumoconiosis. The pneumoconiosis staging mainly depends on the small opacity in the lung fields, and early-stage pneumoconiosis staging has been a challenging task, necessitating quantitative diagnostics. Thoracic Computed Tomography (CT) images represent the gold-standard modality for evaluating local abnormalities and understanding structure-function relationships in an organ. The thoracic CT images have the potential to provide an essential feature for distinguishing subtle morphological patterns of micro-nodule distributions in the lungs through quantitative analyses. This study investigates whether quantitative representations based on topological data analysis can capture particulate shadows' three-dimensional (3D) distribution properties in pneumoconiosis in volumetric CT images.

In pediatric patients with respiratory abnormalities, it is important to understand the alterations in regional dynamics of the lungs and other thoracoabdominal components, which in turn requires a quantitative understanding of what is considered as normal in healthy children. Currently, such a normative database of regional respiratory structure and function in healthy children does not exist. The purpose of this study is to introduce a large open-source normative database from our ongoing Virtual Growing Child (VGC) project, which includes measurements of volumes, architecture, and regional dynamics in healthy children (six to 20 years) derived via dynamic Magnetic Resonance Imaging (dMRI) images. The database provides four categories of regional respiratory measurement parameters including morphological, architectural, dynamic, and developmental. The database has 3,820 3D segmentations (around 100,000 2D slices with segmentations), which to our knowledge is the largest dMRI dataset of healthy children. The database is unique and provides dMRI images, object segmentations, and quantitative regional respiratory measurement parameters for healthy children. The database can serve as a reference standard to quantify regional respiratory abnormalities on dMRI in young patients with various respiratory conditions and facilitate treatment planning and response assessment. The database can be useful to advance future AI-based research on MRI-based object segmentation and analysis.

Magnetic Resonance Imaging (MRI) plays a pivotal role in diagnosing and predicting the course of Multiple Sclerosis (MS). A distinctive biomarker, Paramagnetic Rim Lesions (PRL), offers promise but poses challenges in manual assessment. To address this, we introduce a direct PRL segmentation approach and extensively evaluate various methods, with a focus on preprocessing and input modalities. Our study emphasizes instance segmentation metrics tailored for sparse lesions. Single-modal inputs show limitations, except for FLAIR and Magnitude, exhibiting potential in PRL detection. Integrating Phase and/or MPRAGE with FLAIR enhances the detection capacity. Notably, applying white matter masks yields mixed results, while lesion masks improve overall performance. Despite the complexities of PRL segmentation, our optimal model, FLAIR+Phase, attains a F1 score of 0.443, a Dice score coefficient per True Positive of 0.68 and a deceiving Dice score of 0.191 on the test set. This highlights the intricate nature of the PRL segmentation task. Our work pioneers an automated approach to PRL analysis, offering valuable insights and paving the way for future advancements in this critical field.

Virtual Reality (VR) enables new perspectives and approaches for interaction with a computer-generated environment, thus enhancing data science and a variety of applications. An Electrocardiogram (ECG) records the heart’s electrical activity, allowing insights into its behavior and identifying potential problems. Novel textile shirts with integrated ECG sensors offer convenient and continuous ECG recording. We investigate the feasibility of using such a shirt with textile sensors for real-time monitoring of an ECG signal in VR. We developed an application that wirelessly records and analyzes the ECG from a subject in real-time, visualizing both the ECG itself and the heartbeat in VR. The heartbeat is visualized on an animated, three-dimensional heart model, while the ECG signal is displayed as a time series graph. For analysis, we employ a real-time heartbeat detection algorithm. The recorded signal can be monitored live in the VR environment with a delay of less than 10 ms. Thus, the combination of smart wearables and VR demonstrates how immersive analytics facilitates real-time heart monitoring. Eventually, similar approaches can open new possibilities for training medical personnel as well as educating a broader interested audience.

Pathologic diagnosis is the "gold standard" for diagnosing breast cancer and is increasingly used to assess the response to Neoadjuvant Chemotherapy (NACT). Despite its high accuracy and sensitivity, pathology is invasive and requires biopsy of the patient's breast tissue. Dynamic Contrast-Enhanced Magnetic Resonance Imaging (DCE-MRI) is the standard of care in breast cancer management and is critical for noninvasive prediction of pathological response to NACT. To this end, we propose a transformer model based on DCE-MRI that is guided by histopathological image data to predict responses to NACT. A cross-attention mechanism was developed to facilitate information interaction between histopathological images and DCE-MRI. Specifically, we designed a modality information transfer module to synthesize histopathological image features from DCE-MRI features. During the training stage, we propose to stochastically use synthesize histopathological image features rather than the real features as network inputs. This strategy enables us to predict the response to NACT by using DCE-MRI alone, regardless of the availability of histopathological images. In this study, 239 patients with paired DCE-MRI and histologic images were included; 32 patients (13.4%) achieved a pathological Complete Response (pCR), while 207 patients (13.4%) had nonpCR. A total of 146 samples were used as the training set, and 93 samples were used as the testing set. The experimental results showed that the proposed histopathological information-guided model using DCE-MRI and histopathological images had a greater predictive performance (AUC=0.824) than either the traditional DCE-MRI (AUC=0.687) or histopathological image-based model (AUC=0.765) in predicting the response to NACT in patients with breast cancer.

Integration of sports medicine and human performance is a developing field with the potential to aid people of various abilities. Previous work done has shown the normalization of sports medicine data as well as the building of a framework for the storage and retrieval of sports medicine data. The result of this is the Integrated Biomechanics Informatics System (IBIS). IBIS has the capacity to store, view, and retrieve data and can also be expanded to include different sports medicine research related applications. One such application is a data processing application to create force vector overlays for decision support. Users were able to log into IBIS using secure personal logins and access the data processing application to create force vector overlays. The efficacy of the data processing application was tested by having users process data to create force vector overlays using the IBIS application and then comparing it to the traditional workflow. The new workflow using the IBIS application was found to be quicker and easier to use than the traditional workflow while generating identical results. The deployment of this data processing tool for decision support shows the capability of IBIS to be expanded to include more tools such as automatic foot contact detection. In the future we hope to apply these developed tools in a clinical setting with our collaborators at Rancho Los Amigos National Rehabilitation Center, and therefore show the efficacy of this tool and IBIS in a clinical environment with a variety of patients.

ShapeAXI represents a cutting-edge framework for shape analysis that leverages a multi-view approach, capturing 3D objects from diverse viewpoints and subsequently analyzing them via 2D Convolutional Neural Networks (CNNs). We implement an automatic N-fold cross-validation process and aggregate the results across all folds. This ensures insightful explainability heat-maps for each class across every shape, enhancing interpretability and contributing to a more nuanced understanding of the underlying phenomena. We demonstrate the versatility of ShapeAXI through two targeted classification experiments. The first experiment categorizes condyles into healthy and degenerative states. The second, more intricate experiment, engages with shapes extracted from CBCT scans of cleft patients, efficiently classifying them into four severity classes. This innovative application not only aligns with existing medical research but also opens new avenues for specialized cleft patient analysis, holding considerable promise for both scientific exploration and clinical practice. The rich insights derived from ShapeAXI’s explainability images reinforce existing knowledge and provide a platform for fresh discovery in the fields of condyle assessment and cleft patient severity classification. As a versatile and interpretative tool, ShapeAXI sets a new benchmark in 3D object interpretation and classification, and its groundbreaking approach hopes to make significant contributions to research and practical applications across various domains. ShapeAXI is available in our GitHub repository https://github.com/DCBIA-OrthoLab/ShapeAXI.

Blinded Independent Central Review (BICR) is pivotal in maintaining unbiased assessment in oncology clinical trials employing various assessment criteria like Response Evaluation Criteria In Solid Tumors (RECIST) in clinical trials framework. This paper emphasizes the potential of Large Language Models (LLMs) such as OpenAI’s GPT-4 and ChatGPT trained on clinical trials’ documents and other reader training materials, to significantly improve interpretation assistance and real-time query resolution. During central review process these documents are easily accessible to readers but sometimes, given that most readers read on multiple ongoing clinical trials on regular basis, it is a daunting task to search details like trial design, endpoints and specific reader rules. Through various pre-trained frameworks using ChatGPT Application Programming Interface (API), an AI-based chatbot can be used for helping readers saving time by providing accurate study design related questions based on uploaded training documents. If successful, this analysis can open another novel implementation for LLMs (or ChatGPT) in clinical research and medical imaging. This prospective study involved the review of study design and protocol available from clinicaltrials.gov database maintained by The National Library of Medicine (NLM) at the National Institutes of Health (NIH). ClinicalTrials.gov is a registry of clinical trials that contains information on clinical studies funded by the NIH, other federal agencies, and private industry. The database includes over 444,000 trials from 221 countries. The NLM works with the US Food and Drug Administration (FDA) to develop and maintain the database. The ChatGPT based Chatbot was trained on clinical trial design data from respective studies to grasp the intricacies of assessment criteria, patient population, inclusion / exclusion criteria and other assessment nuances, as applicable. Fine-tuning with prompt engineering ensured Chatbot to understand the language and context specific to BICR. The resulting models serve as intelligent assistants to provide a user-friendly interface for reviewers. Reviewers can engage with the chatbot in natural language, obtaining clarifications on assessment guidelines, terminology, and complex cases. To train the Chatbot, we searched for lung cancer studies with following specifications: “Completed Studies | Studies with Results | Interventional Studies | Lung Cancer | Phase 3 | Study Protocols | Statistical Analysis Plans (SAPs)” from the clinicaltrials.gov website. Without any bias, we selected the first three studies in search results for our prospective study and seven questions to ask, representative of commonly encountered queries for readers. The algorithm supported by ChatGPT was evaluated against a Gold Standard medical opinion, determined by a board-certified radiologist with over 20 years of experience in the BICR process. The Chatbot provided immediate, contextually accurate insights for all the questions across three studies. The trick questions which did not have the answers, data or references in the provided text were rightly called out by the Chatbot, suggesting the user to check with document or study team. Though sometimes, in addition to referencing the study team or documents, it did provide a clinical practice or general criteria related feedback. Real-time query resolution reduces response time, preventing delays in assessment and decision-making. LLMs can streamline training by offering on-the-spot explanations and references, enhancing reviewer proficiency and efficiency. By acting as interactive chatbots, LLMs have immense potential to improve quality and efficiency by offering contextual guidance and expedited responses, ultimately enhancing decision-making and study efficiency.

Breast cancer is a genetically heterogeneous disease with distinct gene expression patterns within a tumor. However, the invasive and expensive nature of genomic examination impedes its extensive use in clinical practice. Magnetic Resonance Imaging (MRI) is noninvasive and widely used for cancer diagnosis and treatment. In view of this, we developed a contrast learning-based framework to synthesize genomic characteristics from MRI. Specifically, we extracted image features using the 3D-ResNet18 architecture, while cell subpopulation features were obtained through a Multilayer Perceptron (MLP). The contrastive learning network aligns the image features and genomic features in the representation space using a contrastive loss. We saved the weights of the image feature extractor from the contrastive learning stage as pretraining weights for the generator in the generative model and used the discriminator to distinguish between the generated immune cell subpopulations and the real immune cell subpopulations. Further survival analysis of the generated immune cell subpopulations was conducted using the log-rank test. The dataset consisted of 135 patients, with 81 samples allocated to the training set and 54 samples assigned to the testing set. Based on the univariate Cox hazard model, ten immune cell subpopulations significantly associated with overall survival were identified. Immune cell subpopulations were generated using the model proposed in this work, and the risk score was calculated using multivariate Cox regression. The generated risk score of immune cells achieved the R square of 0.48 and 0.43 in the validation and the test cohort, respectively. Significant differences in prognosis were observed after grouping the patients according to risk score, with p values of 0.033 and 0.011 in the validation and test sets, respectively.

We propose an unsupervised method to detect lung lesions on FDG-PET/CT images based on deep image anomaly detection using 2.5-dimensional (2.5D) image processing. This 2.5D processing is applied to preprocessed FDG-PET/CT images without image patterns other than lung fields. It enhances lung lesions by parallel analysis of axial, coronal, and sagittal FDG-PET/CT slice images using multiple 2D U-Net. All the U-Nets are pretrained by 95 cases of normal FDG-PET/CT images having no lung lesions and used to transform CT slice images to normal FDG-PET slice images without any lesion-like SUV patterns. A lesion-enhanced image is obtained by merging subtractions of the transformed three normal FDG-PET images from the input FDG-PET image. Lesion detection is performed by simple binarization of the lesion-enhanced images. The threshold value varies from the case and is the 30-percentile voxel value of the target lesion-enhanced image. In each extracted region, the average of the intra-regional voxel values of the enhanced image is computed and assigned as a lesion-like score. We evaluated the proposed method by 27 patients FDG-PET/CT images with 41 lung lesions. The proposed method achieved 82.9 % of lesion detection sensitivity with five false positives per case. The result was significantly superior to the detection performance of FDG-PET image thresholding and indicates that the proposed method may be helpful for effective lung lesion detection. Future works include expanding the detectable range of lesions to outside lungs, such as mediastinum and axillae.

We developed a novel 3D transformer-based UNet method for performing Electronic Cleansing (EC) in CT Colonography (CTC). The method is designed to map an uncleansed CTC image volume directly into the corresponding virtually cleansed CTC image volume. In the method, the layers of a 3D transformer-based encoder are connected via skip connections to the decoder layers of a 3D UNet to enhance the ability of the UNet to use long-distance image information for resolving EC image artifacts. The EC method was trained by use of the CTC image volumes of an anthropomorphic phantom that was filled partially with a mixture of foodstuff and an iodinated contrast agent. The CTC image volume of the corresponding empty phantom was used as the reference standard. The quality of the EC images was tested visually with six clinical CTC test cases and quantitatively based on a phantom test set of 100 unseen samples. The image quality of EC was compared with that of a conventional 3D UNet-based EC method. Our preliminary results indicate that the 3D transformer-based UNet EC method is a potentially effective approach for optimizing the performance of EC in CTC.

Colorectal Cancer (CRC) is the third most common cancer and second most common cause of cancer deaths. Most CRCs develop from large colorectal polyps, but most polyps remain smaller than 6 mm and will never develop into cancer. Therefore, conservative selective polypectomy based on polyp size would be a much more effective colorectal screening strategy than the current practice of removing all polyps. For this purpose, automated polyp measurement would be more reproducible and perhaps more precise than manual polyp measurement in CT colonography. However, for an accurate and explainable image-based measurement, it is first necessary to determine the 3D region of the polyp. We investigated the polyp segmentation performance of a traditional 3D U-Net, transformer-based U-Net, and denoising diffusion-based U-Net on a photon-counting CT (PCCT) colonography dataset. The networks were trained on 946 polyp volumes of interest (VOIs) collected from conventional clinical CT colonography datasets, and they were tested on 17 polyp VOIs extracted from a PCCT colonography dataset of an anthropomorphic colon phantom. All three segmentation networks yielded satisfactory segmentation accuracies with average Dice scores ranging between 0.73-0.75. These preliminary results and experiences are expected to be useful in guiding the development of a deep-learning tool for reliable estimation of the polyp size for the diagnosis and management of patients in CRC screening.

Deep Learning is advancing medical imaging Research and Development (R&D), leading to the frequent clinical use of Artificial Intelligence/Machine Learning (AI/ML)-based medical devices. However, to advance AI R&D, two challenges arise: 1) significant data imbalance, with most data from Europe/America and under 10% from Asia, despite its 60% global population share; and 2) hefty time and investment needed to curate proprietary datasets for commercial use. In response, we established the first commercial medical imaging platform, encompassing steps like: 1) data collection, 2) data selection, 3) annotation, and 4) pre-processing. Moreover, we focus on harnessing under-represented data from Japan and broader Asia. We are preparing/providing ready-to-use datasets for medical AI R&D by 1) offering these datasets to companies and 2) using them as additional training data to develop tailored AI solutions. We also aim to merge Blockchain for data security and plan to synthesize rare disease data via generative AI.

This study presents new methods to utilize bioprinting technology to recreate hard and soft tissue based on medical imaging. Using the proposed methods, medical DICOM files are converted into STL files and bioprinted into PCL:HA scaffolds. The high model precision provides potential for their use as image-based, implantable biological scaffolds for musculoskeletal applications. Initially, femur segments were recreated for the hard tissue model and spinal discs for the soft tissue model. All models were bioprinted utilizing PCL:HA ratios of 80:20 for ease of manufacturing. Printing methodology circumvents the need for solvents in PCL printing, allowing for scaffolds to be created in a safer, faster, and more cost-effective manner. For the hard tissue models, DICOM files are converted directly into STL files utilizing the software 3D Slicer. The software directly segments the imaging to print a specified section of hard tissue. For soft tissue models, a protocol called negative imaging was created to generate soft tissue models based on hard tissue imaging. This method is employed due to the difficulty of isolating soft tissue, such as spinal discs, in DICOM files. The protocol utilizes CAD to generate a mold of the soft tissue from the surrounding hard tissue. The mold is then cleaned in Meshmixer to fix imperfections. To gauge the strength of various PCL:HA ratios for future tissue models, PCL:HA ratios from 90:10 to 65:35 were printed into standardized cubes and compression tested to determine stiffness. Various infill geometries were also tested to see their effect on the scaffold stiffness.

The Global Cancer Observatory estimates approximately 900,000 head and neck cancer cases annually. Accurate segmentation of tumors and lymph nodes from medical images is essential for cancer treatment planning. Manual segmentation by experts slows the clinical workflow and is subject to inter-observer variability. Deep learning-based segmentation methods can address these issues, improving efficiency and accuracy. The study compares Squeeze-and-Excitation (SE) U-Net and SegResNet architectures for multi-class tumor segmentation. They were trained and tested with the HECKTOR 2022 dataset to evaluate model architecture and hyperparameters in both efficiency and accuracy. The models delineated gross primary tumors and lymph node tumors in the head and neck region, with a ground truth manual segmentation. Trained on NVIDIA Tesla V100-PCIE GPU with 32GB memory, up to eight core CPU with 16GB memory each, the models had mean DSC of 0.79 and 0.61 for SegResNet and SE U-Net, respectively over 300 epochs for a test set of 100 image pairs. Gross tumor accuracy was 0.73 and 0.65, respectively. Nodal tumor accuracy was lower with DSC of 0.70 and 0.45, respectively. The SegResNet model performed comparably with grand challenge submissions, likely due to more layers and filters. The SegResNet model was more scalable to higher resolutions and had a higher computational efficiency. Deep learning model architectures show promising performance that could be considered for integration into clinical workflows for multi-tumor segmentation in soft tissues.

Our previous study showed that GAN-generated artifacts frequently appear in GAN-generated mammograms. As GAN artifacts could affect the performance of downstream tasks (e.g., augmentation), it may be beneficial to develop algorithms for detecting GAN-generated mammographic artifacts. Thus, we developed classification and segmentation algorithms for GAN-generated mammographic artifacts to classify the case with artifacts and then segment those artifacts in the GAN-generated mammograms. Using our two internal screening datasets, we trained and tested a Conditional GAN (CGAN) algorithm to simulate breast mammograms. For CGAN training, we used 1366 normal (without cancer) right/left breast mammograms. We then tested our CGAN model on an independent dataset of mammograms from 333 women with dense breasts for possible CGAN artifacts. An experienced radiologist evaluated the CGAN-generated mammograms, identified cases with artifacts, and segmented them. For the development of the classification models for artifact detection, we split the second dataset into training and testing sets using an 8:2 ratio. Using the radiologist’s annotation as ground truth, we trained a classifier (DenseNet121) to identify the two most common artifacts in CGAN-simulated mammograms, checkerboard and non-smooth breast boundary artifacts. We then trained a Unet to segment artifacts precisely. Our classifier achieved an AUC of 0.67 for the checkerboard artifacts and an AUC of 0.78 for breast boundaries. Our segmentation algorithm achieved a dice score of 0.64 for the checkerboard artifact and 0.57 for the breast boundary artifact. We showed that it is possible to identify the mammograms with CGAN artifacts. More investigation is needed to improve the segmentation and classification of artifacts.