Current paradigms for collecting data to train fieldable computer vision (CV) algorithms are inefficient and expensive in terms of time and resources, and they have limited ability to adapt to changing target signatures. By leveraging opportunistic sensing of the operator’s natural behavior (e.g., firing a weapon, placing markers on a map), it is possible to triage the data for CV algorithms. When paired with an After Action Review (AAR) to confirm target presence and location, the resulting pipeline can efficiently update the CV for changing target signatures. Subjects (n = 10) participated in a simulated mission in which they were asked to mark video frames in which targets belonging to three classes first appeared, then perform a brief AAR in which they were asked to mark target locations in these frames. The CV (Detectron2) was initially trained on one target class, then retrained on instances of a novel class acquired by this pipeline. We report that retraining the CV on as few as 8 images resulted in good localization performance (AP50 85.6, AR 42.5) on unseen test images to this novel class. Ten minutes of retraining (600 iterations) was sufficient for good performance (AP50 58.3, AR 27.5). Data augmentation via random occlusions and apertures (‘bubbling’) boosted the training set 192-fold and improved the ratio of hits to false alarms and improved resilience to naturalistic occlusion and small sizes (AP 80-100 and AR 80). These results support our approach as an efficient method to adapt CV via partially labeled operational data.
Brent Lance, Gabriella Larkin, Jonathan Touryan, Joe Rexwinkle, Steven Gutstein, Stephen Gordon, Osben Toulson, John Choi, Ali Mahdi, Chou Hung, Vernon Lawhern
The application of Artificial Intelligence and Machine Learning (AI/ML) technologies to Aided Target Recognition (AiTR) systems will significantly improve target acquisition and engagement effectiveness. Although, the effectiveness of these AI/ML technologies is based on the quantity and quality of labeled training data, there is very limited labeled operational data available. Creating this data is both time-consuming and expensive, and AI/ML technologies can be brittle and unable to adapt to changing environmental conditions or adversary tactics that are not represented in the training data. As a result, continuous operational data collection and labeling are required to adapt and refine these algorithms, but collecting and labeling operational data carries potentially catastrophic risks if it requires Soldier interaction that degrades critical task performance. Addressing this problem to achieve robust, effective AI/ML for AiTR requires a multi-faceted approach integrating a variety of techniques such as generating synthetic data and using algorithms that learn on sparse and incomplete data. In particular, we argue that it is critical to leverage opportunistic sensing: obtaining operational data required to train and validate AI/ML algorithms from tasks the operator is already doing, without negatively affecting performance on those tasks or requiring any additional tasks to be performed. By leveraging the Soldier’s substantial skills, capabilities, and adaptability, it will be possible to develop effective and adaptive AI/ML technologies for AiTR in the future Multi- Domain Operations (MDO) battlefield.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.