The generation of realistically simulated photoacoustic (PA) images with ground truth labels for optical and acoustic properties has become a critical method for training and validating neural networks for PA imaging. As state-of-the-art model-based simulations often suffer from various inaccuracies, unsupervised domain transfer methods have been recently proposed to enhance the quality of model-based simulations. The validation of these methods, however, is challenging as there are no reliable labels for absorption or oxygen saturation in vivo. In this work, we examine various domain shifts between simulations and real images such as simulating the wrong noise model, inaccuracies in modeling the digital device twin or erroneous assumptions on tissue composition. We show in silico how a Cycle GAN, unsupervised image-to-image translation networks (UNIT) and a conditional invertible neural network handle these domain shifts and what their consequences are for blood oxygen saturation estimation.
This study delves into the largely uncharted domain of biases in photoacoustic imaging, spotlighting potential shortcut learning as a key issue in reliable machine learning. Our focus is on hardware variation biases. We identify device-specific traits that create detectable fingerprints in photoacoustic images, demonstrate machine learning's capability to use these discrepancies to determine the device that acquired the image, and highlight their potential impact on machine learning model predictions in downstream tasks, such as disease classification.
Data-driven approaches to the quantification problem in photoacoustic imaging have shown great potential in silico, but the inherent lack of labelled ground truth data in vivo currently restricts their application and translation into clinics. In this study we leverage Fourier Neural Operator networks as surrogate models to synthesize multispectral photoacoustic human forearm images in order to replace time-consuming and not inherently differentiable state-of-the-art Monte Carlo and k-Wave simulations. We investigate the accuracy and efficiency of these surrogate models for the optical and acoustic simulation step.
Optical and acoustic imaging techniques enable noninvasive visualization of structural and functional tissue properties. Data-driven approaches for quantification of these properties are promising, but they rely on highly accurate simulations due to the lack of ground truth knowledge. We recently introduced the open-source simulation and image processing for photonics and acoustics (SIMPA) Python toolkit that has quickly been adopted by the community in the context of the IPASC consortium for standardized reconstruction. We present new developments in the toolkit including e.g. improved tissue and device modeling and provide an outlook on future directions aiming at improving the realism of simulations.
Peripheral artery disease (PAD) is widespread among the elderly population where narrowing arteries in lower limbs are causing a lack of perfusion. This work explores the benefit of volumetric photoacoustic imaging (v-PAI) over conventional 2D PAI for PAD diagnosis and monitoring. To this end, we leverage the recently proposed approach of Tattoo tomography, which generates a v-PAI representation from a set of 2D PAI slices. Preliminary results of the ongoing study indicate that v-PAI can increase the sensitivity of early-stage PAD detection. Conclusively our Tattoo approach has the potential to become a valuable tool in PAD diagnostics.
KEYWORDS: Monte Carlo methods, Diffuse reflectance spectroscopy, In vivo imaging, Tissues, Optical imaging, Machine learning, Hyperspectral imaging, Functional imaging, Evolutionary algorithms, Data analysis
Simulations are indispensable in the field of biomedical optical imaging, particularly in functional imaging. Given the recent rise of artificial intelligence and the lack of labeled in vivo data, synthetic data is not only important for the validation of algorithms but also crucial for training machine learning methods. To support research based on synthetic data, we present a new framework for assessing the quality of synthetic spectral data. Experiments with more than 10,000 hyperspectral in vivo images obtained from multiple species and various organ classes indicate that our framework could become an important tool for researchers working with simulations.
Significance: Optical and acoustic imaging techniques enable noninvasive visualisation of structural and functional properties of tissue. The quantification of measurements, however, remains challenging due to the inverse problems that must be solved. Emerging data-driven approaches are promising, but they rely heavily on the presence of high-quality simulations across a range of wavelengths due to the lack of ground truth knowledge of tissue acoustical and optical properties in realistic settings.
Aim: To facilitate this process, we present the open-source simulation and image processing for photonics and acoustics (SIMPA) Python toolkit. SIMPA is being developed according to modern software design standards.
Approach: SIMPA enables the use of computational forward models, data processing algorithms, and digital device twins to simulate realistic images within a single pipeline. SIMPA’s module implementations can be seamlessly exchanged as SIMPA abstracts from the concrete implementation of each forward model and builds the simulation pipeline in a modular fashion. Furthermore, SIMPA provides comprehensive libraries of biological structures, such as vessels, as well as optical and acoustic properties and other functionalities for the generation of realistic tissue models.
Results: To showcase the capabilities of SIMPA, we show examples in the context of photoacoustic imaging: the diversity of creatable tissue models, the customisability of a simulation pipeline, and the degree of realism of the simulations.
Conclusions: SIMPA is an open-source toolkit that can be used to simulate optical and acoustic imaging modalities. The code is available at: https://github.com/IMSY-DKFZ/simpa, and all of the examples and experiments in this paper can be reproduced using the code available at: https://github.com/IMSY-DKFZ/simpa_paper_experiments.
KEYWORDS: Data modeling, Monte Carlo methods, Scattering, Photoacoustic imaging, Optical simulations, Neural networks, Machine learning, Light scattering, In vivo imaging, Imaging spectroscopy
Photoacoustic imaging (PAI) has the potential to revolutionize healthcare due to the valuable information on tissue physiology that is contained in multispectral signals. Clinical translation of the technology requires conversion of the high-dimensional acquired data into clinically relevant and interpretable information. In this work, we present a deep learning-based approach to semantic segmentation of multispectral PA images to facilitate interpretability of recorded images. Based on a validation study with experimentally acquired data of healthy human volunteers, we show that a combination of tissue segmentation, sO2 estimation, and uncertainty quantification can create powerful analyses and visualizations of multispectral photoacoustic images.
Previous work on 3D freehand photoacoustic imaging has focused on the development of specialized hardware or the use of tracking devices. In this work, we present a novel approach towards 3D volume compounding using an optical pattern attached to the skin. By design, the pattern allows context-aware calculation of the PA image pose in a pattern reference frame, enabling 3D reconstruction while also making the method robust against patient motion. Due to its easy handling optical pattern-enabled context-aware PA imaging could be a promising approach for 3D PA in a clinical environment.
Photoacoustic imaging (PAI) is an emerging medical imaging modality that provides high contrast and spatial resolution. A core unsolved problem to effectively support interventional healthcare is the accurate quantification of the optical tissue properties, such as the absorption and scattering coefficients. The contribution of this work is two-fold. We demonstrate the strong dependence of deep learning-based approaches on the chosen training data and we present a novel approach to generating simulated training data. According to initial in silico results, our method could serve as an important first step related to generating adequate training data for PAI applications.
In this work, we present the open source “Simulation and Image Processing for Photoacoustic Imaging (SIMPA)” toolkit that facilitates simulation of multispectral photoacoustic images by streamlining the use of state-of-the-art frameworks that numerically approximate the respective forward models. SIMPA provides modules for all the relevant steps for photoacoustic forward simulation: tissue modelling, optical forward modelling, acoustic modelling, noise modelling, as well as image reconstruction. We demonstrate the capabilities of SIMPA by performing image simulation using MCX and k-Wave for the optical and acoustic forward modelling, as well as an experimentally determined noise model and a custom tissue model.
Eric Heim, Tobias Roß, Alexander Seitel, Keno März, Bram Stieltjes, Matthias Eisenmann, Johannes Lebert, Jasmin Metzger, Gregor Sommer, Alexander W. Sauter, Fides Regina Schwartz, Andreas Termer, Felix Wagner, Hannes Götz Kenngott, Lena Maier-Hein
Accurate segmentations in medical images are the foundations for various clinical applications. Advances in machine learning-based techniques show great potential for automatic image segmentation, but these techniques usually require a huge amount of accurately annotated reference segmentations for training. The guiding hypothesis of this paper was that crowd-algorithm collaboration could evolve as a key technique in large-scale medical data annotation. As an initial step toward this goal, we evaluated the performance of untrained individuals to detect and correct errors made by three-dimensional (3-D) medical segmentation algorithms. To this end, we developed a multistage segmentation pipeline incorporating a hybrid crowd-algorithm 3-D segmentation algorithm integrated into a medical imaging platform. In a pilot study of liver segmentation using a publicly available dataset of computed tomography scans, we show that the crowd is able to detect and refine inaccurate organ contours with a quality similar to that of experts (engineers with domain knowledge, medical students, and radiologists). Although the crowds need significantly more time for the annotation of a slice, the annotation rate is extremely high. This could render crowdsourcing a key tool for cost-effective large-scale medical image annotation.
A. Franz, A. Seitel, M. Servatius, C. Zöllner, I. Gergel, I. Wegner, J. Neuhaus, S. Zelzer, M. Nolden, J. Gaa, P. Mercea, K. Yung, C. Sommer, B. Radeleff, H.-P. Schlemmer, H.-U. Kauczor, H.-P. Meinzer, L. Maier-Hein
Due to rapid developments in the research areas of medical imaging, medical image processing and robotics,
computer assistance is no longer restricted to diagnostics and surgical planning but has been expanded to surgical
and radiological interventions. From a software engineering point of view, the systems for image-guided therapy
(IGT) are highly complex. To address this issue, we presented an open source extension to the well-known
Medical Imaging Interaction Toolkit (MITK) for developing IGT systems, called MITK-IGT. The contribution
of this paper is two-fold: Firstly, we extended MITK-IGT such that it (1) facilitates the handling of navigation
tools, (2) provides reusable graphical user interface (UI) components, and (3) features standardized exception
handling. Secondly, we developed a software prototype for computer-assisted needle insertions, using the new
features, and tested it with a new Tabletop field generator (FG) for the electromagnetic tracking system NDI
Aurora ®. To our knowledge, we are the first to have integrated this new FG into a complete navigation system
and have conducted tests under clinical conditions. In conclusion, we enabled simplified development of imageguided
therapy software and demonstrated the utilizability of applications developed with MITK-IGT in the
clinical workflow.
Visualization of anatomical data for disease diagnosis, surgical planning, or orientation during interventional
therapy is an integral part of modern health care. However, as anatomical information is typically shown on
monitors provided by a radiological work station, the physician has to mentally transfer internal structures
shown on the screen to the patient. To address this issue, we recently presented a new approach to on-patient
visualization of 3D medical images, which combines the concept of augmented reality (AR) with an intuitive
interaction scheme. Our method requires mounting a range imaging device, such as a Time-of-Flight (ToF)
camera, to a portable display (e.g. a tablet PC). During the visualization process, the pose of the camera and
thus the viewing direction of the user is continuously determined with a surface matching algorithm. By moving
the device along the body of the patient, the physician is given the impression of looking directly into the human
body. In this paper, we present and evaluate a new method for camera pose estimation based on an anisotropic
trimmed variant of the well-known iterative closest point (ICP) algorithm. According to in-silico and in-vivo
experiments performed with computed tomography (CT) and ToF data of human faces, knees and abdomens,
our new method is better suited for surface registration with ToF data than the established trimmed variant of
the ICP, reducing the target registration error (TRE) by more than 60%. The TRE obtained (approx. 4-5 mm)
is promising for AR visualization, but clinical applications require maximization of robustness and run-time.
KEYWORDS: Medical imaging, Algorithm development, Image processing, Current controlled current source, Cameras, 3D acquisition, 3D image processing, 3D-TOF imaging, Image registration, In vitro testing
Time-of-flight (ToF) cameras are a novel, fast, and robust means for intra-operative 3D surface acquisition. They acquire surface information (range images) in real-time. In the intra-operative registration context, these surfaces must be matched to pre-operative CT or MR surfaces, using so called descriptors, which represent surface characteristics. We present a framework for local and global multi-modal comparison of surface descriptors and characterize the differences between ToF and CT data in an in vitro experiment. The framework takes into account various aspects related to the surface characteristics and does not require high resolution input data in order to establish appropriate correspondences. We show that the presentation of local and global comparison data allows for an accurate assessment of ToF-CT discrepancies. The information gained from our study may be used for developing ToF pre-processing and matching algorithms, or for improving calibration procedures for compensating systematic distance errors. The framework is available in the open-source platform Medical Imaging Interaction Toolkit (MITK).
Augmented reality (AR) for enhancement of intra-operative images is gaining increasing interest in the field of
navigated medical interventions. In this context, various imaging modalities such as ultrasound (US), C-Arm
computed tomography (CT) and endoscopic images have been applied to acquire intra-operative information
about the patient's anatomy. The aim of this paper was to evaluate the potential of the novel Time-of-Flight
(ToF) camera technique as means for markerless intra-operative registration. For this purpose, ToF range data
and corresponding CT images were acquired from a set of explanted non-transplantable human and porcine
organs equipped with a set of marker that served as targets. Based on a rigid matching of the surfaces generated
from the ToF images with the organ surfaces generated from the CT data, the targets extracted from the
planning images were superimposed on the 2D ToF intensity images, and the target visualization error (TVE)
was computed as quality measure. Color video data of the same organs were further used to assess the TVE of a
previously proposed marker-based registration method. The ToF-based registration showed promising accuracy
yielding a mean TVE of 2.5±1.1 mm compared to 0.7±0.4 mm with the marker-based approach. Furthermore,
the target registration error (TRE) was assessed to determine the anisotropy in the localization error of ToF
image data. The TRE was 8.9± 4.7 mm on average indicating a high localization error in the viewing direction
of the camera. Nevertheless, the young ToF technique may become a valuable means for intra-operative surface
acquisition. Future work should focus on the calibration of systematic distance errors.
One of the main challenges related to computer-assisted laparoscopic surgery is the accurate registration of
pre-operative planning images with patient's anatomy. One popular approach for achieving this involves intraoperative
3D reconstruction of the target organ's surface with methods based on multiple view geometry. The
latter, however, require robust and fast algorithms for establishing correspondences between multiple images of
the same scene. Recently, the first endoscope based on Time-of-Flight (ToF) camera technique was introduced.
It generates dense range images with high update rates by continuously measuring the run-time of intensity
modulated light. While this approach yielded promising results in initial experiments, the endoscopic ToF
camera has not yet been evaluated in the context of related work. The aim of this paper was therefore to
compare its performance with different state-of-the-art surface reconstruction methods on identical objects. For
this purpose, surface data from a set of porcine organs as well as organ phantoms was acquired with four
different cameras: a novel Time-of-Flight (ToF) endoscope, a standard ToF camera, a stereoscope, and a High
Definition Television (HDTV) endoscope. The resulting reconstructed partial organ surfaces were then compared
to corresponding ground truth shapes extracted from computed tomography (CT) data using a set of local and
global distance metrics. The evaluation suggests that the ToF technique has high potential as means for intraoperative
endoscopic surface registration.
Image-guided therapy systems generally require registration of pre-operative planning data with the patient's anatomy. One common approach to achieve this is to acquire intra-operative surface data and match it to surfaces extracted from the planning image. Although increasingly popular for surface generation in general, the novel Time-of-Flight (ToF) technology has not yet been applied in this context. This may be attributed to the fact that the ToF range images are subject to considerable noise. The contribution of this study is two-fold. Firstly, we present an adaption of the well-known bilateral filter for denoising ToF range images based on the noise characteristics of the camera. Secondly, we assess the quality of organ surfaces generated from ToF range data with and without bilateral smoothing using corresponding high resolution CT data as ground truth. According to an evaluation on five porcine organs, the root mean squared (RMS) distance between the denoised ToF data points and the reference computed tomography (CT) surfaces ranged from 3.0 mm (lung) to 9.0 mm (kidney). This corresponds to an error-reduction of up to 36% compared to the error of the original ToF surfaces.
The main challenges of Computed Tomography (CT)-guided organ puncture are the mental registration of the
medical imaging data with the patient anatomy, required when planning a trajectory, and the subsequent precise
insertion of a needle along it. An interventional telerobotic system, such as Robopsy, enables precise needle
insertion, however, in order to minimize procedure time and number of CT scans, this system should be driven
by an interface that is directly integrated with the medical imaging data. In this study we have developed and
evaluated such an interface that provides the user with a point-and-click functionality for specifying the desired
trajectory, segmenting the needle and automatically calculating the insertion parameters (angles and depth).
In order to highlight the advantages of such an interface, we compared robotic-assisted targeting using the old
interface (non-image-based) where the path planning was performed on the CT console and transferred manually
to the interface with the targeting procedure using the new interface (image-based). We found that the mean
procedure time (n=5) was 22±5 min (non-image-based) and 19±1 min (image-based) with a mean number of CT
scans of 6±1 (non-image-based) and 5±1 (image-based). Although the targeting experiments were performed
in gelatin with homogenous properties our results indicate that an image-based interface can reduce procedure
time as well as number of CT scans for percutaneous needle biopsies.
Lena Maier-Hein, Conor Walsh, Alexander Seitel, Nevan Hanumara, Jo-Anne Shepard, A. Franz, F. Pianka, Sascha Müller, Bruno Schmied, Alexander Slocum, Rajiv Gupta, Hans-Peter Meinzer
Computed tomography (CT) guided percutaneous punctures of the liver for cancer diagnosis and therapy (e.g.
tumor biopsy, radiofrequency ablation) are well-established procedures in clinical routine. One of the main
challenges related to these interventions is the accurate placement of the needle within the lesion. Several
navigation concepts have been introduced to compensate for organ shift and deformation in real-time, yet, the
operator error remains an important factor influencing the overall accuracy of the developed systems. The aim
of this study was to investigate whether the operator error and, thus, the overall insertion error of an existing
navigation system could be further reduced by replacing the user with the medical robot Robopsy. For this
purpose, we performed navigated needle insertions in a static abdominal phantom as well as in a respiratory
liver motion simulator and compared the human operator error with the targeting error performed by the robot.
According to the results, the Robopsy driven needle insertion system is able to more accurately align the needle
and insert it along its axis compared to a human operator. Integration of the robot into the current navigation
system could thus improve targeting accuracy in clinical use.
In this paper, we evaluate the target position estimation accuracy of a novel soft tissue navigation system with a
custom-designed respiratory liver motion simulator. The system uses a real-time deformation model to estimate
the position of the target (e.g. a tumor) during a minimally invasive intervention from the location of a set of
optically tracked needle-shaped navigation aids which are placed in the vicinity of the target.
A respiratory liver motion simulator was developed to evaluate the performance of the system in-vitro. It
allows the mounting of an explanted liver which can be moved along the longitudinal axis of a corpus model to
simulate breathing motion. In order to assess the accuracy of our system we utilized an optically trackable tool
as target and estimated its position continuously from the current position of the navigation aids. Four different
transformation types were compared as base for the real-time deformation model: Rigid transformations, thinplate
splines, volume splines, and elastic body splines. The respective root-mean-square target position estimation
errors are 2.15 mm, 1.60 mm, 1.88 mm, and 1.92 mm averaged over a set of experiments obtained from a total
of six navigation aid configurations in two pig livers. The error is reduced by 76.3%, 82.4%, 79.3%, and 78.8%,
respectively, compared to the case when no deformation model is applied, i.e., a constant organ position is
assumed throughout the breathing cycle.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.