1.

Introduction

The utilization of deep learning (DL) in pathology departments has the potential to significantly improve patient care by supporting physicians with tasks such as providing melanoma diagnosis,¹ detecting breast cancer metastases,^2–4 and grading prostate cancer.^5–8 Nevertheless, DL systems for medical imaging are known to encounter generalization issues due to their sensitivity to outliers and domain shift, i.e., a change in the underlying data distribution.^9,10 This can be encountered when an algorithm is deployed in a new medical center as well as in the same medical center over time.¹¹ Relying solely on the DL output score to assess the reliability of the predictions often does not work well due to the commonly observed over-confidence of such algorithms.¹² Therefore, developing and deploying new methods for assessing the dependability of the DL predictions is essential for wider clinical acceptance of the technology.¹³ Uncertainty estimation has been proposed as one of the solutions that could potentially improve the performance and robustness of DL systems.¹⁴ However, deriving the maximum benefit of uncertainty estimation in segmentation DL for digital pathology is a complex task requiring careful evaluations.

This work focuses on DL application to breast cancer metastasis segmentation as this type of cancer is one of the most prevalent cancers worldwide.¹⁵ To provide appropriate treatment, it is crucial to determine if cancerous cells have spread to other organs and tissues. Typically, the nearest lymph nodes are removed surgically, and tissue samples are fixed, sliced, and stained with hematoxylin and eosin (H & E) to create glass slides. This procedure is referred to as sentinel lymph node dissection. A pathologist then carefully examines the glass slides or digitized whole slide images (WSIs) to detect potential tumor metastases. This can be a labor-intensive and time-consuming process, particularly because multiple WSIs from each patient may require examination.² Therefore, assistance from DL systems could prove to be valuable.^16–18

We propose a spatial uncertainty aggregation (SUA) framework for improving the effectiveness of uncertainty estimation of a DL model trained to segment breast cancer metastases in lymph nodes. We hypothesize that the overall confidence of a neighborhood of pixels may contain additional information; hence we aggregate segmentation predictions into regions and compute an uncertainty measure for each of them. Our results show that utilizing spatial information works better than considering pixels independently.

We focus on clinical relevance; hence in the evaluation of the proposed method we (a) fix the segmentation threshold instead of relying on threshold-independent metrics and (b) work on false negatives (FN) detection as this task often requires more time from pathologists than false positive (FP) rejection. Importantly, the SUA framework is agnostic to the uncertainty estimation technique and requires no intervention from pathologists. In the study, we utilize the deep ensemble (DE) uncertainty estimation method¹⁹ due to the promising results exhibited in related work.^14,20 Our experiments reveal a strong correlation between aggregated uncertainty and incorrect segmentation, and we explore whether this information can enhance the performance of a DL diagnostic system.

2.

Related Work

Several techniques have been developed to estimate uncertainty in DL, such as test time augmentations,²¹ Monte Carlo dropout,²² and DEs.¹⁹ Previous studies have demonstrated the usefulness of estimating uncertainty in computational pathology, in which a common strategy involves identifying the most uncertain predictions for manual review by medical professionals, allowing them to focus on challenging cases.^23,24 Similarly, uncertainty heatmaps can be generated and superimposed on the original image for visual inspection.^25,26 The latter approach can be valuable during the algorithm development phase, but it is impractical for pathologists under time constraints in clinical production.

Alternatively, incorporating uncertainty estimates into the DL framework may not require intervention from a physician. For example, previous works have examined the correlation between estimated uncertainty and mispredictions^14,20 or outliers²⁷ in classification, which could be used to reduce the error in performance. In addition, combining uncertainty with the softmax output may enhance the generalizability and robustness of DL-based classifiers for histopathology applications.¹⁴ In segmentation, it has been shown that uncertainty can be used to filter out FP prediction areas on digital pathology data.²⁸ Our work falls under this direction of research as it is focused on determining how uncertainty estimation could be used to refine the segmentation predictions by detecting FNs.

3.

Method

3.2.

Negative Prediction Regions

The SUA framework targets the detection of FN regions. The analysis is performed for what we denote as negative prediction regions (NPRs). An NPR is defined as a cluster of adjacent pixels that have been classified as negative by the DL algorithm but are reasonable candidates for being FN pixels. Notably, NPRs do not need to be adjacent to the areas that are already predicted positive by the DL algorithm. NPRs are determined by locating groups of adjacent pixels that are assigned a softmax value output by the DL model within a chosen interval. The upper threshold is the cutoff value for a positive prediction. The lower softmax bound is used to avoid excessively large NPRs. Algorithm 1 provides a pseudo code for determining NPRs in a WSI. The function for labeling the connected regions, i.e., the $\mathrm{skimage}.\mathrm{measure}.\mathrm{label}$ function from the sklearn library,²⁹ is based on the work of Wu et al.³⁰ We explored the impact of different settings of the lower threshold: 0.55, 0.65, 0.75, and 0.85.

Algorithm 1

Determining NPRs in a given WSI.

Input: segmentation Heatmap, lowerThreshold, upperThreshold

$t1\leftarrow \mathrm{lowerThreshold}$

$t2\leftarrow \mathrm{upperThreshold}$

$\mathrm{binarySegmentation}=(\mathrm{segmentationHeatmap}>t1)\&(\mathrm{segmentationHeatmap}<t2)$

$\mathrm{nprIslands}=\mathrm{skimage}.\mathrm{measure}.\mathrm{label}(\mathrm{binarySegmentation},\mathrm{connectivity}=2)$

Output: nprIslands

The segmentation threshold is set to 0.95 in our study as this results in eight average FPs per WSIs on the validation data. This decision was based on the free-response receiver operating characteristic (FROC) metric,² which assesses the clinical relevance of metastases detection algorithms. Therefore, the evaluated NPRs were built considering pixels with softmax scores in the ranges of 0.55 to 0.95, 0.65 to 0.95, 0.75 to 0.95, and 0.85 to 0.95. We also experimented with smaller NPRs, i.e., with softmax scores in the 0.55 to 0.65 and 0.65 to 0.75 ranges, but both SUA and the baseline (see Sec. 3.4) showed substantially inferior performance; hence we excluded them from the results. Figure 1 visualizes an example of a WSI with its NPRs.

Fig. 1

Visualization of NPRs. The leftmost image shows a part of the original WSI, and the middle one shows the segmentation heatmap produced by a DL model. In the rightmost image, NPRs are visualized in different colors. They are determined by finding adjacent pixels that are assigned a segmentation softmax score in a predetermined range (0.55 to 0.95 in this example).

3.3.

Spatial Uncertainty Aggregation framework

Figure 2 shows the SUA framework. Given a segmentation algorithm and uncertainty estimation method, segmentation and uncertainty heatmaps are generated. As discussed above, two thresholds are employed to create NPRs: the upper threshold identifies pixels predicted to be of a positive class, whereas the lower threshold is utilized to restrict the NPRs’ areas. Within NPRs, the uncertainty values are aggregated and applied to distinguish between true negative (TN) and FN predictions. We evaluate two different aggregation functions: the average and the 90th percentile of the values. Note that any uncertainty estimation technique can be utilized in the SUA framework. If the resulting aggregated uncertainty score is above a set threshold, the NPR is marked as an FN region. This threshold is empirically determined on validation data.

Fig. 2

Outline of the SUA framework that utilizes spatially aggregated uncertainty to identify FNs.

4.

Implementation Details and Data

5.

Experiments

We conducted two experiments that are briefly described below. The goal was to evaluate the potential of the SUA framework to enhance the usefulness of the underlying uncertainty estimates and investigate its practical impact on boosting the performance of segmentation NNs in in-domain and domain-shift scenarios.

6.

Results

6.2.

Experiment 1

From the histograms in Figs. 3 and 4, we conclude that the separation between FNs and TNs achieved by the 90th percentile and the average uncertainty aggregation approaches are similarly good. $F_{\beta =0.5}$ scores computed on the validation data revealed that using average entropy resulted in a marginally better performance. Hence, we report the results using the average uncertainty aggregation approach in the remainder of the work. Figure 5 shows a corresponding histogram using the baseline method, i.e., processing pixels entropy without spatial aggregation. Less separation between FN and TN predictions for the considered softmax score ranges is achieved in this case.

Fig. 3

Histogram of the 90th percentile entropy of the NPRs built using varying softmax ranges. FNs are the islands that have at least 90% overlap with the ground truth tumor annotation. Camelyon validation data.

Fig. 4

Histogram of the average entropy of the NPRs built using varying softmax ranges. FNs are the islands that have at least 90% overlap with the ground truth tumor annotation. Camelyon validation data.

Fig. 5

Histogram of the baseline pixel entropy divided between TN and FN predictions. Pixels analyzed with softmax values in the ranges of 0.55 to 0.95, 0.65 to 0.95, 0.75 to 0.95, and 0.85 to 0.95. Camelyon validation data.

Table 2 shows the $F_{\beta =0.5}$ scores achieved by the SUA framework with different sizes of NPRs and the baseline with a matching constraint on the softmax score of considered pixels. It confirms the observation that aggregating uncertainty spatially results in a better separation between TN and FN predictions in almost all considered scenarios. Overall, the highest mean $F_{\beta =0.5}$ scores on the Camelyon and Sentinel datasets of 0.92 and 0.86, respectively, were achieved by the SUA framework.

Table 2

Fβ=0.5 performance of FN versus TN differentiation on the Camelyon and Sentinel test datasets. In the SUA method, an NPR is considered to be FN if it contains at least 90% tumor pixels. The mean and standard deviation values are computed over the 1000 bootstrapped iterations.

Dataset	Method	Fβ=0.5 with softmax scores in ranges
		0.55 to 0.95	0.65 to 0.95	0.75 to 0.95	0.85 to 0.95
Camelyon	Baseline	0.71 (0.05)	0.74 (0.06)	0.76 (0.06)	0.81 (0.06)
	SUA	0.92 (0.02)	0.91 (0.02)	0.89 (0.02)	0.73 (0.04)
Sentinel	Baseline	0.42 (0.06)	0.43 (0.06)	0.45 (0.06)	0.48 (0.06)
	SUA	0.86 (0.04)	0.85 (0.04)	0.83 (0.04)	0.64 (0.05)

6.3.

Experiment 2

Table 3 summarizes how the average Dice score is affected by applying SUA and the baseline for refinement of the segmentation heatmaps. We can see that updated heatmaps through the baseline method systematically result in lower average Dice scores than the original predictions. The SUA framework achieves higher average DICE scores than observed on the original predictions when NPRs were computed using 0.55 to 0.95 and 0.65 to 0.95 softmax ranges on both test sets. Figure 7 shows an example of a segmentation refined by applying the SUA framework.

Table 3

Average Dice score before and after segmentation refinement by the SUA and baseline methods. The results are reported per considered softmax ranges, i.e., 0.55 to 0.95, 0.65 to 0.95, 0.75 to 0.95, and 0.85 to 0.95. The mean and standard deviation values are computed over the 1000 bootstrapped samples.

Dataset	Method	Original Dice	Dice scores after update, softmax in ranges
			0.55 to 0.95	0.65 to 0.95	0.75 to 0.95	0.85 to 0.95
Camelyon	Baseline	0.706 (0.023)	0.676	0.628	0.654	0.680
			(0.025)	(0.028)	(0.026)	(0.024)
	SUA		0.709	0.709	0.706	0.700
			(0.024)	(0.024)	(0.024)	(0.024)
Sentinel	Baseline	0.700 (0.026)	0.672	0.672	0.676	0.681
			(0.028)	(0.028)	(0.027)	(0.027)
	SUA		0.704	0.703	0.702	0.695
			(0.026)	(0.026)	(0.026)	(0.026)

The box plots in Fig. 6 summarize the results of the FNCR metric computed over 1000 bootstrap iterations. SUA achieved substantially higher median values in all scenarios compared with the baseline. Domain shift has an impact on SUA effectiveness as the median values fall from being between 0.98 and 0.95 on the Camelyon data to 0.94 and 0.82 on the Sentinel data for the considered NPRs. However, the effectiveness of the baseline in FN detection suffers a much sharper drop due to the domain shift: from around 0.8 to around 0.4 on the Camelyon and Sentinel data, respectively.

Fig. 6

FNCR achieved by SUA and the baseline on the Camelyon and Sentinel datasets with 1000 bootstrap iterations. The red horizontal line in each box indicates the median value. The results are reported per considered softmax ranges, i.e., 0.55 to 0.95, 0.65 to 0.95, 0.75 to 0.95, and 0.85 to 0.95.

Table 4 shows the number of negative WSIs that had some FN areas introduced after the refinement using the SUA and baseline methods. SUA incorrectly updated significantly fewer such WSIs in all considered scenarios. In fact, the baseline method refined nearly all negative WSIs in the Camelyon and Sentinel datasets, which have 80 and 107 negative WSIs, respectively.

Table 4

Number of negative WSIs that had some FP areas introduced by the segmentation refinement.

Dataset	Method	# WSIs, softmax scores in ranges
		0.55 to 0.95	0.65 to 0.95	0.75 to 0.95	0.85 to 0.95
Camelyon	Baseline	77	80	79	77
	SUA	7	6	9	25
Sentinel	Baseline	104	104	104	104
	SUA	31	31	42	65

There was no change in FROC-AUC or ROC-AUC observed from applying either of the FN detection methods compared with the original segmentation results.

7.

Discussion

In this study, we aimed to determine whether spatially aggregating uncertainty could improve the ability to distinguish between TN and FN predictions in histopathology segmentation and bring clinical value by refining the segmentation heatmaps. First, we examined whether regions with incorrect negative predictions tended to have consistently different uncertainty values from the correct predictions. Our findings from experiment 1 indicate a significant correlation between the aggregated epistemic uncertainty over NPRs and FN predictions. Notably, we observed that broader NPRs, i.e., based on softmax scores in the 0.55 to 0.95, 0.65 to 0.95, and 0.75 to 0.95 ranges, yielded higher $F_{\beta =0.5}$ scores than narrower regions (with softmax scores in 0.85 to 0.95 range), suggesting that combining more pixels improved spatial uncertainty estimates. The baseline achieved substantially lower $F_{\beta =0.5}$ scores, confirming that it is advantageous to incorporate the spatial information within the uncertainty estimation. Given our definition of FN as containing at least 90% of tumor pixels, these results are highly promising, suggesting that spatial epistemic uncertainty could enhance the performance of a DL system for breast cancer metastases segmentation.

To evaluate the practical benefits of FN-detection-based heatmap refinement compared with utilizing the underlying uncertainty directly, we proposed the FNCR metric. The left plot in Fig. 6 indicates that the detected FN areas with the SUA framework enables a relatively successful refinement of in-domain segmentation heatmaps with most bootstrap runs achieving an FNCR of around 0.98 for NPRs with softmax ranges of 0.55 to 0.95 and 0.65 to 0.95. In all cases, the baseline exhibited a substantially worse performance measured in the median FNCR. Furthermore, the baseline also had a higher variability in its performance compared with the SUA framework, as indicated by the length of the whiskers of the box plots. We conclude that the SUA framework based on NPRs with softmax ranges of 0.55 to 0.95 and 0.65 to 0.95 performed best on the heatmap refinement task.

The lack of generalizability is a severe problem affecting DL for pathology applications; hence an important question is if the SUA framework can to some extent mitigate the observed negative effects on the performance from the domain shift. Based on relatively high $F_{\beta =0.5}$ scores in Table 2, it seems that there is a possibility of successfully detecting FNs in the Sentinel data. A drop in the median value of the FNCR is relatively small on the segmentation heatmap refinement task for the domain shift for the NPRs with softmax ranges of 0.55 to 0.95, 0.65 to 0.95, and 0.75 to 0.95. The improvement in the average Dice score is slightly lower on the domain shift data compared with the in-domain data (see Table 3). However, Table 4 reveals that a much higher number of negative WSIs had some incorrect refinement done on the Sentinel data compared with the Camelyon data. This is a worrying trend as increasing the number of false WSI predictions would severely compromise potential clinical benefits of uncertainty integration. We conclude that the utilized uncertainty estimation is sensitive to the domain shift and, hence, may not be suitable in domain shift scenarios. This is consistent with observations in previous studies.^14,41 Even if uncertainty estimation can bring tangible value under in-domain data assumption, it is essential to have approaches that are able to handle the domain shift scenarios. As our results indicate, unfortunately, the SUA framework is unable to address this problem. Ensuring that uncertainty estimation is robust to domain shifts and hence helps to alleviate the drop in performance due to poor generalization of the DL algorithms is a very important future research topic.

It is challenging to evaluate what tangible benefits the incorporation of uncertainty with a DL model may provide. The average Dice score seems to be insensitive to some changes in prediction heatmaps. This may be caused by difference in the size of metastases, i.e., macro metastases are very large and, hence, more easily detectable by DL algorithms and pathologists. The average Dice score appears to primarily reflect the accurate segmentation of this type of metastases. This is supported by the minimal difference in the average Dice score between unmodified segmentation for the Camelyon and Sentinel datasets, whereas FROC-AUC indicates a significant difference in segmentation performance between these two datasets. The observation is consistent with prior work.³⁸ It is difficult to say if the observed 0.003 Dice score improvement after refinement with the SUA framework could bring some clinical value because the decrease of 0.006 in the average Dice score on the original segmentation of the Sentinel data when comparing with the Camelyon data is also small.

Given the strong correlation between the SUA output and the regions with incorrect negative predictions observed in experiment 1, the uncertain clinical benefit brought by the uncertainty-based refinement in experiment 2 is a somewhat disappointing result, which highlights a potential need for improving the underlying uncertainty estimation approach. This is a surprising finding because several previous works reported promising performance of the DE uncertainty estimation method. However, their evaluation primarily focused on the correlation between incorrect predictions and the uncertainty values. Our study emphasizes the importance of evaluating the clinical value by acting upon the uncertainty information.

It seems that we are able to fill in some missed gaps in the segmentation (see Fig. 7), but ideally, we would aim to detect completely missed tumor metastases. However, defining what is a missed metastasis is not straightforward. For example, how far should a missed metastasis be from the correctly segmented one to have clinical relevance? If a patient has a macro metastasis, micro metastases become less important for the diagnostic decision, whereas isolated tumor cells usually do not have any impact.⁴² How to incorporate this knowledge in the evaluation of missed tumor metastases detection is an open question. Therefore, we believe that such a direction of work is not only very important but also requires careful considerations and experimentation. Hence, it could be the focus of future research.

Fig. 7

Example of a refined WSI from the Camelyon data via the SUA framework based on NPRs with the 0.55 to 0.95 softmax range. “Tumor mask” is the ground truth tumor annotation.

A general challenge in computational pathology is to determine which evaluation metrics represent well the clinical impact of an approach. The Dice score does not take into consideration the prevalence of FP predictions, which can cause severe problems in clinical practice. To avoid data leakage, we are not able to reliably use FROC-AUC either. It would require tuning the segmentation threshold for each test data and using them in the heatmap refinement steps. We consider such a process to be clinically unrealistic. Utilizing thresholds learned on validation data resulted in no change in FROC-AUC and ROC-AUC after the refinement. Future work could aim to propose metrics that would more closely represent performance from a clinical point of view.

As mentioned, our approach is appealing because it can generalize to different combinations of DL architectures, targeted tasks in digital pathology, and uncertainty estimation techniques. The results indicate that the chosen base uncertainty estimation method is not able to significantly boost the DL performance on breast cancer metastasis segmentation in lymph nodes. However, several other studies have shown the potential benefits of uncertainty estimation in other DL applications in digital pathology.^27,43 Hence, our observed improvement of utilizing SUA compared with uncertainty estimation directly on in-domain data indicates that the SUA approach could improve the obtained benefits from uncertainty estimation in the previously studied scenarios.

There are several limitations of this work. Only one uncertainty estimation method, i.e., the DE method, was tested with the SUA framework. Because the SUA framework is independent of the uncertainty estimation approach, investigating alternative uncertainty estimation techniques that are more suitable for computational pathology applications and that could be combined with the SUA framework could provide an unquestionable clinical impact. We believe that this direction could prove valuable. Moreover, in this study, we focused on the breast cancer metastasis segmentation task due to the clinical value of addressing this problem with DL assistance. It would be valuable to confirm that the observed benefits apply to other important segmentation tasks in digital pathology, such as nuclei segmentation.

8.

Conclusion

Our analysis indicates that the SUA framework proposed for FN detection exhibits encouraging outcomes on in-domain data, with a noticeable correlation between FN regions and the aggregated uncertainty. Moreover, utilizing SUA for refinement of segmentation heatmaps further confirms its potential benefits compared with utilizing the underlying uncertainty directly. However, utilization of the chosen uncertainty estimation technique did not result in substantial improvements on the segmentation outcome. This negatively affected the effectiveness of segmentation refinement via SUA. We recommend future work to focus on determining a more robust uncertainty estimation method to combine with the SUA framework as well as further improving the evaluation approaches to quantify the potential benefits in realistic clinical settings.