With advancements in unmanned aerial vehicle (UAV) technology, UAV applications are rapidly growing, and their operations are becoming increasingly intelligent. Localization of UAVs commonly relies on global navigation satellite systems combined with inertial navigation systems through sensor fusion. However, this approach is vulnerable to significant risks, such as signal spoofing. In military conflicts, signal spoofing by hackers poses a severe security threat with potentially catastrophic outcomes. To address this issue, we propose a two-stage vision-based UAV localization method. This approach utilizes multi-category semantic segmentation and template matching to establish a connection between heterogeneous sensors. Experimental results demonstrate the method’s effectiveness in accurately identifying the UAV’s location within extensive geographical areas captured in remote sensing images. In addition, it achieves high precision in aligning UAV locations with Baidu maps, offering robust and accurate localization capabilities.