2.1

Multi-kernel model for non-local means

Non-local Means (NLM)⁶ is a classical filter that utilizes the dissimilarity measure between patches to operate in a non-local area (even the entire image). Actually, the mathematical model for non-local means is not unique. For example, in Reference⁹, GNLMKIM employs multi Gaussian kernels to define the measure and applies Shannon regularizer to balance the linear relationship between various kernels. The specific model can be defined by:

where, i and j denote the pixel positions from the definition domain Ω of image x; x_i and x_j are the two corresponding image patches;G_t(t = 1,…,k)is the Gaussian kernel used to measure the similarity between the two patches. Naturally, becomes the dissimilarity between patches. λ_t can be viewed as the importance of the single kernel G_t.p represents the regularization parameter that trades off the two terms of the model. As declared in Reference⁹, the outputs of NLM can be derived from the optimization model under the single-kernel case. Meanwhile, with multi-kernel methods, the filters derived from the above model usually get more powerful ability in edge detecting. It is the fact that motivates us to modify NLB with multi-kernel strategy.

2.2

Non-local block

Non-local block, i.e., NLB⁵, captures the long-range dependencies of pixels and thus becomes key to semantic segmentation. Specifically, the block is defined as:

where f (x_i, x_j) denotes the similarity between position i and j in the input and S(x) is the normalization factor. W_z and W_g are two 1 × 1 convolutions respectively.

For the similarity f, we take embedded Gaussian as an example to describe the definition in detail. Under this condition, , θ(x_i) = W_θx_i, φ(x_j) = W_φx_j, , and W_θ are two 1 × 1 convolutions. , H, W respectively indicate their channel number, input width and input height.

For an input (C indicates the input channel number), the standard NLB with embedded Gaussian is shown in Figure 1a. Here, we intend to design a multi-kernel version of NLB, compared with the original one, which can get more powerful performance when being used in semantic segmentation.

Figure 1.

(a): Architecture of a standard NLB; (b): Initial definition of MKNLB; (c): Architecture of MKNLB. N = H × W.

Note: : Matrix multiplication; : Element-wise sum.

3.1

Initial definition

As aforementioned, the standard NLB is based on the single Gaussian kernel and the existing work indicated that a multi-Gaussian kernel generally gets better behaviors in edges⁹. The fact motives us to extend NLB to its multi-kernel version (named MKNLB) to improve the ability in segmentation.

For simplicity, we take two kernels in our MKNLB. The definition is

where , are the two Gaussian kernels, σ(x_j = W_σx_j, W_σ is a 1 × 1 convolution, other symbols all have the same meanings as those in NLB formulated in equation (2).

For clarity, the definition of MKNLB is illustrated in figure 1b. Its effectiveness in semantic segmentation will be validated in the experiments part. Considering the multi-kernel strategy may increase the computational burden at first glance, we design an efficient implementation of MKNLB next, whose complexity is comparable to the standard NLB.

3.2

Efficient implementation

As shown in figure 1a, the similarity calculation with matrix multiplication, i.e., θ(x_i) × φ(x_j), is the main computational burden in non-local block. Similarly in Reference⁷, the operation can be simplified and expressed as:

where N = H × W. Therefore, this multiplication of matrices has a complexity of .

In our MKNLB defined in Figure 1b, due to the participation of the two kernels, the similarity part becomes θ(x_i) × φ(x_j) + θ(x_i) × σ(x_j). At first glance, the computational burden may increase. Fortunately, it can be reduced with the distributive law of matrix multiplication. The specific expression is changed as follows:

Equation (5) indicates that, MKNLB can be implemented by performing the two-matrix addition first and then followed by one time matrix multiplication. Considering the former complexity is distinctly lower than the latter. Therefore, this implementation of MKNLB can be approximated as . That is, the complexity of our MKNLB is comparable to that of the standard NLB. For clarity, the implementation is illustrated in Figure 1c.

4.1

Datasets and evaluation metrics

Cityscapes: It consists of 5000 images from 50 different cities belonging 19 categories. In order to facilitate training, validation, and testing, the images have been divided into 2975, 500, and 1525 segments, respectively

ADE20K: The dataset contains 20210 images in the training dataset with 150 semantic classes, 2000 images make up the validation set, while 3352 make up the test set. As well known, the dataset is particularly challenging in semantic segmentation datasets due to complex scenarios.

Metrics: The mean intersection over union (mIoU) is used to evaluate all datasets.

4.2

Training details

During training, our code follows a standard frame from the semantic segmentation open resource library MMSegmentation¹². Two Quadro Rtx 6000 GPUs are used for all experiments. We apply stochastic gradient descent (SGD) with the weight decay is 0.0005. The initial learning rate γ₀ = 0.01 is decayed following the poly learning rate policy, where γ₀ is multiplied by . For Cityscapes, we set the batch size is 4 and randomly crop the input images to 512×512. For ADE20K, we set the batch size is 8 and randomly crop the input images to 512×1024. For the two datasets, we choose random flip and scale these images within [0.5, 2]. For all experiments, we select the pre-trained ResNet-101 as backbone framework.

4.3

Comparisons with other methods

In this section, we will analyze the results of the two datasets (Cityscapes¹⁰ and ADE20K¹¹) for semantic segmentation.

On one hand, we compare the proposed multi-kernel non-local network with five other methods on the Cityscapes validation set. Table 1 is shown the experiment outcomes of mIoU numbers. We trained all methods for 8K iterations. Based on the same backbone, the multi-kernel non-local network attains 77.59% mIoU. It can be observed that 2.68% mIoU better than the original non-local network. We also find that our method performs better than the previous methods by more than 0.71% mIoU.

Table 1.

Comparisons with the state-of-the-arts on the Cityscapes validation set.

On the other hand, we compare the performance of our method on the ADE20K validation set. The outcomes of mIoU numbers are shown in Table 2. We trained 8K iterations in this dataset to compare the performance with other methods. As we all know, the dataset is challenging to train due to a variety of image sizes, complex semantic information, and the difference between training and validation sets. Despite under this condition, our method also achieves 41.35% mIoU. It can still be 1.03% better than the original non-local network and also defeat other listed methods.

Table 2.

Comparisons with the state-of-the-arts on the ADE20K validation set.