Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning

Elise Bishoff; Charles Godfrey; Myles McKay; Eleanor Byler

doi:10.1117/12.2663498

13 June 2023 Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning

Elise Bishoff, Charles Godfrey, Myles McKay, Eleanor Byler

Author Affiliations +

Proceedings Volume 12519, Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imaging XXIX ; 125190M (2023) https://doi.org/10.1117/12.2663498
Event: SPIE Defense + Commercial Sensing, 2023, Orlando, Florida, United States

Abstract

In overhead image segmentation tasks, including additional spectral bands beyond the traditional RGB channels can improve model performance. However, it is still unclear how incorporating this additional data impacts model robustness to adversarial attacks and natural perturbations. For adversarial robustness, the additional information could improve the model’s ability to distinguish malicious inputs, or simply provide new attack avenues and vulnerabilities. For natural perturbations, the additional information could better inform model decisions and weaken perturbation effects or have no significant influence at all. In this work, we seek to characterize the performance and robustness of a multispectral (RGB and near infrared) image segmentation model subjected to adversarial attacks and natural perturbations. While existing adversarial and natural robustness research has focused primarily on digital perturbations, we prioritize on creating realistic perturbations designed with physical world conditions in mind. For adversarial robustness, we focus on data poisoning attacks whereas for natural robustness, we focus on extending ImageNet-C common corruptions for fog and snow that coherently and self-consistently perturbs the input data. Overall, we find both RGB and multispectral models are vulnerable to data poisoning attacks regardless of input or fusion architectures and that while physically realizable natural perturbations still degrade model performance, the impact differs based on fusion architecture and input data.

Conference Presentation

Citation Download Citation

Elise Bishoff, Charles Godfrey, Myles McKay, and Eleanor Byler "Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning", Proc. SPIE 12519, Algorithms, Technologies, and Applications for Multispectral and Hyperspectral Imaging XXIX , 125190M (13 June 2023); https://doi.org/10.1117/12.2663498

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available