2.1.

System Structure

A frequency-domain US-guided DOT system designed by our group was utilized to collect phantom and patient data.¹⁶ The system employed a hand-held probe that integrated nine source fibers and 14 detection fibers. The light was delivered sequentially at four wavelengths (730, 785, 808, and 830 nm) to each of the nine fibers, and 14 parallel photomultiplier tube detectors detected the light from each source position simultaneously. The laser diodes were modulated at 140.02 MHz, and the system utilized heterodyne detection to mix the detected signals with a 140-MHz reference signal to generate 20-kHz signals. Following this, the output of the mixer for each channel underwent amplification and filtering at 20 kHz before being sampled by a 16-channel analog-to-digital converter.

For real-time US image guidance, we positioned a commercial linear US probe at the center of the DOT probe to obtain US and DOT measurements of the targeted lesion beneath, as detailed in Ref. 34. The system is depicted in Fig. 2. The probe position was fixed during DOT data acquisition, with a data acquisition time of $\sim 3$ to 4 s for each data set. US recording was paused during DOT data acquisition and resumed once the DOT data collection was complete.

Fig. 2

Sketch of the DOT system. The probe is placed on the compressed breast.

2.2.

Simulation and Phantom Configuration

To mimic the clinical dataset as much as possible and make our model more generalizable to the clinical dataset, we employed the finite element method (FEM) in COMSOL software and conducted Monte Carlo (MC) simulations using the Virtual Imaging Clinical Trial for Regulatory Evaluation (VICTRE)³⁵ breast phantom to generate forward measurements. The FEM simulation can replicate the DOT reconstruction with artifacts in shallow targets. Meanwhile, the MC simulation sought to reproduce DOT reconstruction with artifacts related to tissue heterogeneity by varying the fat fraction within the digital breast phantom.

For the FEM approach, we approximated complex clinical scenarios as a single-target breast-shaped model. In the setting of geometry, a hemisphere with a radius of 10 cm was employed. Various inclusions, such as a ball, ellipsoid, cube, a ball with two combined hemispheres, a star, and letters, were considered with varying absorption coefficients to improve the complexity of the dataset. In the simulation, we positioned nine sources and 14 detectors based on the geometry of our clinical DOT probe. Further details of this setting can be found in Ref. 36.

In the MC approach, the digital phantom generated by VICTRE, with a radius of 7 cm and height of 5 cm, served as a heterogeneous condition by incorporating various tissue types. Simulations were performed by translating and rotating the probe on the phantom, with fat fractions varying from 20% to 80%. Additional details of the MC simulation can be found in Ref. 37.

For the phantom study, we utilized high-contrast targets made of calibrated polyester resin (${\mu }_a=0.23{\mathrm{cm}}^{-1}$) and low-contrast targets made of silicon (${\mu }_a=0.11{\mathrm{cm}}^{-1}$). The targets were immersed in an intralipid solution (${\mu }_a=0.02{\mathrm{cm}}^{-1}$, ${\mu }_s^{\prime }=6-8{\mathrm{cm}}^{-1}$) and placed over a silicon plate whose ${\mu }_a$ was similar to that of the solution. We recorded the co-registered US images and the DOT measurements with different targets centralized underneath the probe.

In this study, we utilized 10,975 sets of simulation data and 360 sets of phantom data. Further dataset details are provided in Table 1.

Table 1

Range of parameters used in simulations and phantoms.

2.3.

DOT Patient Data

Our US-guided DOT system has been utilized in clinical studies with protocols that received approval from the appropriate Institutional Review Boards and complied with the Health Insurance Portability and Accountability Act.^38,39 All participants were fully informed about the study’s purpose, procedures, and potential risks before signing a written consent form. To maintain patient confidentiality, all data used in this study were de-identified. Table 2 lists the details of the histologic group, age, histologic diagnosis, and information on shallow cases.

Table 2

Patient information.

2.4.

Conjugate Gradient Descent (CGD) Reconstruction

We used the CGD reconstruction as the input for our model. Details about this reconstruction method can be found in Ref. 40. In summary, we modeled photon migration using a diffusion equation for the photon density wave and applied the Born approximation to relate the scattered field ($U_{\mathrm{sc}}$) to the changes in absorption coefficients ($\delta {\mu }_a$), as follows:

To solve for $\delta {\mu }_a$, we formulated the inverse problem as

The spatial grid used for reconstruction measures $9\mathrm{cm}\times 9\mathrm{cm}\times 3.5\mathrm{cm}$. This grid is divided into a fine-mesh grid centered at the lesion location and a coarse-mesh grid for the background. The resolutions for the fine and coarse meshes are $0.25\mathrm{cm}\times 0.25\mathrm{cm}\times 0.5\mathrm{cm}$ and $1.5\mathrm{cm}\times 1.5\mathrm{cm}\times 0.5\mathrm{cm}$, respectively. Lesions typically occupy one to three depth layers of our seven-layer reconstruction, indicating that most lesions have a vertical diameter of 0.5 to 2 cm.

3.1.

Overall Structure

The overall network structure, depicted in Fig. 3, comprises two main components: an encoder, which solves the forward diffusion equation, and a decoder, which addresses the inverse problem, mapping perturbations to spatial absorption distributions.

Fig. 3

Structure of the APU-Net model. The encoder, up to the perturbation, functions as the solver for the photon diffusion equation, while the remaining neural network components serve as the solver for the inverse problem.

In the encoder, the reconstruction DOT images and their corresponding fine-mesh regions serve as inputs, where the fine mesh delineates a refined grid in the spatial domain. By preserving the background as a coarser mesh grid, computational resources are concentrated on specified areas, enhancing the accuracy of the result. The concatenation of the reconstruction and fine mesh is then fed into an attention-convolution block, comprising a convolution layer followed by a CoT module, as elaborated in Sec. 3.3. Subsequently, a pooling layer compresses the features to a lower dimension. This module repeats four times, ultimately reshaping the output into a one-dimensional array.

In the bottleneck, multiple fully connected layers map the one-dimensional features to the perturbation, addressing the forward problem. The mean square error (MSE) loss between the bottleneck output and the perturbation is calculated as part of the final loss equation. The perturbation undergoes additional fully connected layers and is reshaped back to a three-dimensional form.

In the decoder, we begin by concatenating features from the encoder side as a skip connection, strategically employed to prevent overfitting. Then, four attention-conv blocks are utilized to decode these features into the reconstruction. We reinforce the spatial distribution emphasis by again concatenating the fine mesh with the features. Conclusively, two additional convolution layers are applied to acquire the enhanced reconstruction, solidifying the decoder’s role as the solver for the inverse problem in the diffusion equation.

3.2.

CoT Attention Block

A DOT reconstruction, being a functional image, often lacks lesion detail due to low resolution, posing challenges for traditional deep learning models to accurately recognize targets and achieve satisfactory performance. To address this issue and improve the model’s focus on target areas within the DOT image, we introduced the CoT block as a self-attention module. The CoT module, designed to aggregate contextual information among input keys for guiding the learning of a dynamic attention matrix, demonstrates substantial potential in visual recognition. By linking neighboring information in keys to queries, it enables adaptive focus on the lesion and surrounding areas. This addition aids the model in understanding features more effectively and assigning attention to relevant areas.

The CoT module’s structure, as outlined in Fig. 4(a), involves transforming a 3D feature map $X$ (with dimensions $H\times W\times C$) into keys $K=X$, queries $Q=X$, and values $V=X{W}_v$, where $W_v$ is the embedding matrix, achieved through a $1\times 1$ convolution. Departing from the traditional method, which uses $1\times 1$ convolution to encode the keys, the CoT module utilizes $k\times k$ convolution over the neighbor keys within the $k\times k$ spatial grid. The learned keys, denoted as the mined static context $K_1$, take the shape of $H\times W\times C$. The module then embeds attention by concatenating $K_1$ and the $Q$ then processing them with attention embedding $A_e$ which includes two consecutive $1\times 1$ convolutions with a rectified linear unit activation after the first convolution:

Fig. 4

(a) Structure of the attention-conv block, where the “$\times$” and “$+$” blocks denote the element-wise multiplication and addition operations, respectively. (b) Detailed architecture of the attention embedding module.

Here, unlike the isolated pairwise convolution, the CoT module combines the query with the key and the surrounding area at each location. Subsequently, the dynamic contextual representation of inputs $K_2$ is calculated as

3.3.

Loss Function and Training Schemes

To refine the model’s focus on lesions within the image, we employed a weighted MSE loss $L_i$ for reconstruction, expressed as

In addition, to measure the semantic difference between the enhanced reconstruction and the ground truth, we utilized a pre-trained VGG16⁴¹ model to extract feature domain loss $L_f$ as the perceptron loss⁴²

The MSE loss $L_p$ between the bottleneck output and the perturbation, as mentioned in Sec. 3.1, is calculated as

The overall loss combines perturbation and reconstruction aspects, defined as

Here, $\alpha$, $\beta$, and $\gamma$ denote the weights for each loss, optimized during training.

In accordance with Sec. 2.2, we initially trained the model exclusively on multiple-target simulations, employing a learning rate of 0.0001 over 200 epochs. To manage the learning rate decay, a threshold of 0.01 was established, triggering adjustment if substantial loss drops were not observed within a specified epoch range. Subsequently, we conducted fine-tuning using single-target simulations and phantom data to reflect typical clinic scenarios. This phase employed a learning rate of 0.00005, consistent with the previous weight decay, and spanned 200 epochs. Following the training and fine-tuning, we determined the coefficients for loss calculation to be $\alpha =5$, $\beta =1$, $\gamma =0.01$, $a=0.98$, and $b=0.02$, ensuring a balanced consideration of different components within the loss function from the grid search.

In addition, we applied various data augmentation techniques. These included adding random Gaussian noise to the original DOT reconstruction, rotating images randomly within a range of $-45$ to 45 deg, and applying random affine transformations. The affine transformations encompassed random variations in the rotation, translation, and scaling.

3.4.

Evaluation Metrics

We assessed the performance of our model by measuring its effectiveness in removing artifacts and improving target contrast in different depth layers. To evaluate artifact contrast, we introduced the metric $C_{\mathrm{arti}}$, defined as the ratio of the maximum hemoglobin concentration within the artifact region to the maximum hemoglobin concentration within the lesion region

A lower value of $C_{\mathrm{arti}}$ indicates better artifact removal.

For depth contrast, we calculated the hemoglobin contrast among different depth layers. Specifically, we defined $C_{12}$ as the ratio of the maximum hemoglobin concentrations between the second and first depth layers, and $C_{13}$ as the ratio between the third and first layers. Unlike the artifact contrast, for $C_{12}$ and $C_{13}$, a value closer to one suggests that the hemoglobin contrast is consistent across layers, which is desirable.

4.1.

Test Results on Simulation Dataset

We first evaluated the model’s performance using a separate simulation dataset. Figure 5 presents a boxplot comparing the input, output, and ground truth results, providing a detailed overview of the performance. In this dataset, which includes 1647 simulated cases, our APU-net successfully improved the depth contrast for both layers.

Fig. 5

Comparative depth contrasts among the input reconstruction, the output of the model, and the ground truth.

For the depth contrast between the first and second layers ($C_{12}$), our model increased the contrast from an initial average of $0.9926(\pm 0.1048)$ to $0.9989(\pm 0.0353)$. Similarly, for the depth contrast between the first and third depth layers ($C_{13}$), the model improved the contrast from $0.9836(\pm 0.1004)$ to $0.9997(\pm 0.0335)$. More importantly, the APU-Net significantly reduced the variance of the contrast.

4.2.

Artifact Removal on Clinical Dataset

We then assess the model’s generalization to clinical datasets. We leverage examples of clinical DOT hemoglobin maps to showcase the model’s efficacy in artifact removal. Notably, the hemoglobin maps are derived from the absorption distribution at four wavelengths.³⁴ Figure 6 presents three instances of low-quality reconstructions, accompanied by US images highlighting the targets on the left side, which include several dashed orange lines indicating the different depth layers of the lesion, along with one line representing the upper depth layer and one representing the lower depth layer.

Fig. 6

Examples of low-quality DOT reconstructions and corresponding corrected DOT images from the clinical dataset, with blue ovals indicating the locations of lesions. (a) A shallow malignancy with a source pattern obscuring the lesion’s top. (b) A deeper malignancy with shadow effects. (c) A benign lesion with artifacts attributed to tissue heterogeneity.

In Fig. 6(a), we observe a malignant case at a shallow depth of 0.6 cm, where the lesion’s upper portion is obscured by sources near the probe surface. The model’s output on the right presents a more refined target at the center, with all source artifacts effectively removed. Figure 6(b) illustrates another malignant case, characterized by shadow effects stemming from intense absorption in the top layer. Here, our APU-Net model successfully restores the target’s absorption, aligning closely with the first layer’s shape and value, indicative of a favorable depth profile. Figure 6(c) showcases a benign case with artifacts caused by the heterogeneous background tissue. Here, the model adeptly identifies the lesion at the center while eliminating surrounding artifact-like regions, albeit exhibiting a lower maximum hemoglobin. Next, we compare the artifact contrasts of the original reconstructions and the outputs from the model.

Figure 7(a), a boxplot of the inputs and outputs, shows the statistical details of the images. Based on 34 patients with artifacts caused by shallow targets or tissue heterogeneity, the model reduced the artifact contrast from $0.8448(\pm 0.2888)$ to $0.5580(\pm 0.1678)$. Figures 7(b) and 7(c) break down the artifact contrast by benign and malignant groups. For the benign group, the APU-Net decreased the artifact contrast from $0.8691(\pm 0.2901)$ to $0.5787(\pm 0.1755)$. In the malignant group, the artifact contrast was reduced from $0.7971(\pm 0.1785)$ to $0.4907(\pm 0.1263)$. These results clearly demonstrate the effectiveness of our model in reducing artifact contrast across different types of lesions.

Fig. 7

Comparison of artifact contrasts between the input reconstruction and the output of the model. (a) Artifact contrast for all cases. (b) Artifact contrast for the benign group. (c) Artifact contrast for the malignant group.

These examples underscore the model’s seamless transition from simulation and phantom studies to clinical scenarios, demonstrating its robust performance. Subsequently, we provide further statistical analysis of the model’s efficacy on clinical data.

4.3.

Statistics on Clinical Dataset

To elucidate the model’s efficacy in enhancing depth profiles within clinical settings, we conducted a comprehensive analysis based on data collected from 83 patients. Among these patients, 45 had DOT reconstructions with more than two layers, while 20 had DOT reconstructions with three layers. In Fig. 8, we depict the maximum hemoglobin contrast as introduced in Sec. 3.3. After excluding six cases in the second-layer calculation and three cases in the third-layer calculation due to exceptionally low hemoglobin levels, our APU-Net model demonstrates notable improvement in contrast for deeper layers. Specifically, the contrast for the second layer $C_{12}$ increased from $0.7273(\pm 0.1650)$ to $1.0688(\pm 0.1379)$, while for the third layer $C_{13}$, it improved from $0.3811(\pm 0.1941)$ to $1.0611(\pm 0.1045)$.

Fig. 8

Contrast of hemoglobin between the second and first layers, as well as between the third and first layers.

We then examined the depth contrast, separated into benign and malignant groups, as shown in Fig. 9. We observed significant differences in all subgroups (second-layer benign, second-layer malignant, and third-layer malignant), except for the third-layer contrast in the benign group, where only one patient’s data were available, making statistical analysis difficult. In the second layer, the depth contrast improved from $0.7634(\pm 0.1225)$ to $1.0724(\pm 0.1972)$ for the benign group and from $0.7020(\pm 0.1881)$ to $1.0663(\pm 0.0800)$ for the malignant group. For the third-layer contrast in the malignant group, the APU-Net increased the contrast from $0.3892(\pm 0.1987)$ to $1.0665(\pm 0.1063)$, indicating a significant improvement.

Fig. 9

Depth contrast subgroup study. (a) Second-layer contrast for the benign group. (b) Second-layer contrast for the malignant group. (c) Third-layer contrast for the malignant group.

We computed the maximum hemoglobin values, categorized by benign and malignant cases. For benign cases, the average output value is $64.24\pm 18.83\mu \mathrm{M}$, which is lower than the input average value of $69.84\pm 12.56\mu \mathrm{M}$. For the malignant cases, the averaged outputs are $82.70\pm 20.67\mu \mathrm{M}$, which is higher than the input hemoglobin of $75.73\pm 21.57\mu \mathrm{M}$, with a similar variance. There is no statistical significance between the input and output of the benign and malignant subgroups, respectively, which is expected since the goal of the study is to reduce image artifacts and improve the lesion depth profile. Furthermore, our analysis revealed that our APU-Net model enhanced the differentiation between benign and malignant groups, as illustrated in Fig. 10. This finding underscores the potential of our model to enhance diagnostic accuracy in future studies.

Fig. 10

Maximum hemoglobin levels in benign and malignant groups, categorized by input reconstruction and APU-Net output.

4.4.

Ablation Study

Ablation studies were conducted to evaluate the impact of key design components in APU-Net on synthesis performance. We focused on the effectiveness of the attention module, by removing it to measure the enhancements facilitated by the attention blocks. To assess the impact of employing attention modules in the neural net, we evaluated the artifact contrast along with the depth profiles of the second and third layers in the DOT reconstruction.

Figure 11 presents the results for artifact contrast. We observed a significant improvement when using APU-Net with the attention module than without it. APU-Net achieved an artifact contrast of $0.5580(\pm 0.1678)$, whereas the configuration without the attention module had an artifact contrast of $0.6530(\pm 0.1964)$.

Fig. 11

Artifact contrast for ablation study.

Figure 12 illustrates the contrasts of deeper layers versus the first lesion layer. Removing the attention modules from the model resulted in a second-layer contrast of $1.1169\pm 0.3998$ and a third-layer contrast of $0.8976\pm 0.1826$. While the model without the attention module showed a similar mean contrast compared with the current model, it also demonstrated larger variances either in the second or the third layer. In addition, the exclusion of the attention module during training led to $\sim 32\%$ and 23% reductions in hemoglobin value for benign and malignant cases, respectively, emphasizing the critical role of attention modules in enhancing the model’s spatial distribution awareness and, consequently, preserving accurate values in the enhanced images.

Fig. 12

Deeper layer contrast for ablation study. (a) Second-layer contrast. (b) Third-layer contrast.