2.1.

Transformer

Swim-Transformer It is an attention mechanism-based neural network architecture designed for sequential data modeling. Rather than relying on recurrent neural networks (RNNs) which can be computationally expensive, Swim-Transformer uses Transformer blocks with relative positional embeddings to effectively capture long-term dependencies in sequential data. Compared to existing state-of-the-art models such as Faster R-CNN and Mask R-CNN, Swim-Transformer has shown better performance on several benchmark object detection datasets including COCO and PASCAL VOC. It offers a promising alternative approach for modeling visual data with sequential structures.Cross-Transformer It introduces a self-attention mechanism across modes, bringing the information from multiple modes together in the same model.The core idea of Cross-Transformer is to use self-attention mechanisms to capture relationships within sequence data as well as interactions across modalities. It computes the correlation within the mode by encoding each mode separately and then using the multi-mode attention mechanism. Next, in the cross-modal layer, the corresponding attention weights are used to fuse the information between the different modalities.Through self-attention computation of cross modes, Cross-Transformer can effectively mine the feature links between different modes to better understand and process multimodal data.

2.2.

Muti-scale feature fusion

Multi-scale feature fusion is a computer vision technique utilized to enhance the performance of models in object detection tasks, particularly for detecting objects of different sizes. This technique involves the integration of features from various layers or levels of a convolutional neural network(CNN) to obtain a more complete and informative representation of an image. Different levels in a CNN capture different levels of abstraction and spatial resolution, which makes them better suited for detecting objects of different sizes.

For example, features obtained from lower layers have higher spatial resolution than those from higher layers, which can be useful for detecting small objects. Conversely, features from higher layers are better suited for detecting larger and more complex objects. By fusing these features at multiple scales, the model can benefit from the strengths of each layer and achieve better overall object detection performance.

There are several ways to implement multi-scale feature fusion, including using skip connections to pass features between different layers, using feature pyramid networks (FPNs) to merge features across different resolutions, and using top-down and bottom-up pathways to aggregate information from different scales. These techniques have been proven successful in various computer vision tasks, such as object detection, semantic segmentation, and imageclassification.

3.1.

Overall Architecture

The proposed network architecture, called MSTR-YOLO(shown in Figure 1), is a hybrid model that combines convolution and self-attention. Firstly, we use STR-Darknet (Section3.2)as the backbone, which not only removes the Focus module but also integrates multi-head self-attention into the original CSP-Darknet to extract more Individualized features. Secondly, The Tini-BiFPN, which replaces PANet, is designed to aggregate features from different backbone levels. (Section 3.3) Lastly, we explore different fusion methods and select pixel-level fusion for high computational efficiency to fuse IR and RGB modes

Figure. 1:

The architecture of the MSTR-YOLOv5. a) MSTR-Darknet backbone with transformer encoder blocks at the end. b) The Neck use the structure Tini-BiFPN which combines the advantages of BiFPN and Transformer

3.2.

STR-Darknet

The purpose of the Focus module in the YOLOv5 backbone is to gather the pixel values from the input image and then reconstruct them into smaller complementary images. The size of the reconstructed image decreases as the number of channels increases. Therefore, it will lead to a decrease in resolution and loss of spatial information for small targets. Considering that the Focus module in the YOLOv5 backbone was replaced with the CBS module to improve the detection of small targets, which relies on higher resolution.

To improve the semantic discriminability and mitigate class confusion for RSI in large-scale and complex scenes, collecting and correlating scene information from a large neighbourhood can help learn relationships between objects. However, convolutional networks have limitations in capturing global context information due to locality constraints of convolution operations.In contrast, transformers can globally attend to dependency relationships between image feature patches while preserving sufficient spatial information, enabling multi-head self-attention based object detection. To enhance transferability of learned features and capture long-range contextual information, we propose the STR-Darknet backbone to extract features for detectors. The design of STR-Darknet is straightforward(as shown in Figure 3): we embed Swim-Transformer (STR) layers into the top CSPDark block to achieve global self-attention on 2D feature maps. It’s worth noting that when the network is relatively shal-low and the feature maps are relatively large, early use of transformer layers to enforce boundary regression can lead to the loss of meaningful contextual information. Therefore,in STR-Darknet, transformer layers are only applied to P5 instead of P3 and P4, to avoid this issue.

3.3.

Tini-BiFPN

The size differences of RSIs can be significant, and the representation ability of single-feature maps in conv-olutional neural networks is limited. Therefore, it is essential to effectively represent and process multi-scale features. The traditional top-down FPN [16][12] is essentially limited by one-way information flow. To address this, PANet [13] adds an additional bottom-up path aggregation network, as shown in Figure 4(b). Further studies on cross-scale connections were conducted in [14][15][16]. In those works, a simple and efficient Weighted Bidirectional Feature Pyramid Network (BiFPN), as shown in Figure 4(c), achieves two optimizations for cross-scale connections.In this paper, Inspired by BiFPN, we have designed a lightweight neck network which we call Tini-BiFPN, it not only implifies BiFPN to fit the P5 structure but also introduce the Swim-Transformer into it.as shown in Figure 4(d) and Figure 5.

3.4.

Multimodal Fusion

Multimodal fusion is an effective approach for integrating diverse information from multiple sensors.The more information is utilized to distinguish objects, the better performance can be achieved in object detection. there are three prominent fusion methods: decision-level fusion,feature-level fusion, and pixel-level fusion. However, decision-level fusion, due to its high computational requirements, is not considered in this paper. Instead, we focus on describing our proposed feature-level fusion and pixel-level fusion techniques, which enable the integration of information at different processing depths within the network.

Figure 1 and Figure 2 illustrate show the feature-level fusion of various blocks works. To ensure a fair comparison, the IR image is expanded to three bands. Each block’s fusion operation Cross-1,Cross-2,Cross-3, represent the fusion operation performed in the Low-Level, Mid-Level, High-Level, on the other hand, is considered to be a Feature-Level fusion operation.

Figure. 2:

The backbone structure of YOLOv5s. The low-level texture and high-level semantic features are extracted by stacked CSP, CBS, and SPP structures.

Figure. 3:

The backbone structure of YOLOv5s. The low-level texture and high-level semantic features are extracted by stacked CSP, CBS, and SPP structures.

When it comes to pixel-level fusion(RGB+IR), we normalize the input RGB and IR images to intervals of [0,1], then combine them with relatively low computational effort compared to the other fusion methods, which fuse the information during later procedures to speed up the inference. As Section will demonstrate, pixel-level fusion achieves better results than feature-level fusion when combining different types of complementary information.

when it comes to feature-level fusion,we try to fuse three scale feature by cross transformer. In multi-head cross- attention, we map the input patch sequence Xrgb to Q1,K1,V1 and Xir to Q2,K2,V2 in the headi (i = 1…h, and his the the num-ber of head), following the Q-K-V attention in transformer

As illustrated in Figure 4, the cross-attention layer plays a pivotal role in aggregating key information (K-V pairs) from two different branches to establish attention. Specifically, the K-V pairs from the two branches are concatenated to-gether. Similarly, in order to aggregate K-V pairs from the RGB branch to the IR branch, we also concatenate the K-V pairs from these two branches.

Figure. 4:

Feature network design (a) FPN (b) PANet adds an additional bottom-up pathway on top ofFPN. (c) BiFPN implements two optimizations for cross-scale connections. (d) Tini-BiFPN simplifies BiFPN to fit the P5 structure and introduce Swim-Transformer into it

Figure. 5:

Tini-BiFPN

Figure. 6:

The Cross-Attention in our cross-transformer feature backbone

In this context, the operator [·, ·] refers to the concatenation along the token dimension as the default operation. The cross-attention feature map can be computed as:

3.5.

Loss Function

The loss function of our network consists of two parts: detection lossLv and SR construction lossLs, which can be computed as :

Where λ1 and λ2 refer to the coefficients used to balance two training tasks. In this case, the L1 loss (rather than L2 loss) is utilized to calculate the SR construction loss, denoted as Ls, between the input image X and the SR result S. This can be expressed as follows:

The detection loss consists of three components: the loss for determining whether there is an object (Lobj), the loss for object localization (Lloc), and the loss for object classification (Lcls). These components are used to eval-uate the loss of prediction, which can be expressed as follows:

Here, in Equation 7, the variable j represents the layer of the output in the head. The weights aj, bj, and cj correspond to the weights assigned to different layers for the three loss functions. The weights λloc, λobj, and λcls are used to regulate the emphasis of errors among box coordinates, box dimensions, objectness, no-objectness, and classification.

4.1.

Datasets

The VEDAI dataset is used for multi-class vehicle detection in aerial images. It contains 3640 vehicle instances, including 9 categories such as ships, cars, campers, airplanes, shuttle buses, tractors, trucks, freight vehicles, and other categories. The dataset includes 1210 aerial images of size 1024x1024. with four uncompressed color channels, comprising three RGB color channels and one additional nearinfrared channel.

we use the VEDIA dataset to evaluate our model, and we report mAP (average of all 10 IoU thresholds, ranging from [0.5 : 0.95]) and AP50.

Table 1:

Distribution of Available Class Instances in the VEDAI Dataset Across 10 Folds.

4.2.

Training Environment and Details

Our proposed framework is implemented in PyTorch and is executed on a workstation equipped with an NVIDIA 3090 GPU. The VEDAI dataset is utilized to train our MSTRYOLO model.

For training, we employ the standard Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.937 and weight decay of 0.0005 for Nesterov accelerated gradients. The batch size is set to 2 (8). Initially, the learning rate is set to 0.01. The entire training process consists of 300 epochs and takes approximately 12 hours to complete.

4.3.

Model Evaluation

The accuracy assessment evaluates the agreement and discrepancies between the detection results and the reference mask. To evaluate the performance of the methods being compared, the accuracy metrics used are recall, precision, and mAP (mean Average Precision). The calculation of precision and recall metrics is defined as follows:

TP is the count of correctly classified positive samples, FP is the count of incorrectly classified positive samples, and FN is the count of incorrectly classified negative samples. These metrics are used to evaluate the accuracy and performance of detection algorithms.

The mAP (mean Average Precision) is a comprehensive indicator that averages AP values. It calculates the area under the Precision-Recall curve for all categories using an integral method. This metric quantifies the overall accuracy of an object detection model and considers the trade-off between precision and recall.

4.4.

Result Analysis and Ablation Experiments

We validate the effectiveness of our proposed method by conducting a series of ablation experiments on the first fold of the validation set. These experiments enable us to analyze the importance of each component in VDEIA (Visual Detection and Evaluation of Image Analysis).