Based on current capabilities, many Machine Learning techniques are often inscrutable and they can be hard for users to trust because they lack effective means of generating explanations for their outputs. There is much research and development investigating this area, with a wide variety of proposed explanation techniques for AI/ML across a variety of data modalities. In this paper we investigate which modality of explanation to choose for a particular user and task, taking into account relevant contextual information such as the time available to them, their level of skill, what level of access they have to the data and sensors in question, and the device that they are using. Additional environmental factors such as available bandwidth, currently usable sensors and services are also able to be accounted for. The explanation techniques that we are investigating range across transparent and post-hoc mechanisms and form part of a conversation with the user in which the explanation (and therefore human understanding of the AI decision) can be ascertained through dialogue with the system. Our research is exploring generic techniques that can be used to underpin useful explanations in a range of modalities in the context of AI/ML services that operate on multisensor data in a distributed, dynamic, contested and adversarial setting. We define a meta-model for representing this information and through a series of examples show how this approach can be used to support conversational explanation across a range of situations, datasets and modalities.
Machine learning systems can provide outstanding results, but their black-box nature means that it’s hard to understand how the conclusion has been reached. Understanding how the results are determined is especially important in military and security contexts due to the importance of the decisions that may be made as a result. In this work, the reliability of LIME (Local Interpretable Model Agnostic Explanations), a method of interpretability, was analyzed and developed. A simple Convolutional Neural Network (CNN) model was trained using two classes of images of “gun-wielder” and “non-wielder". The sensitivity of LIME improved when multiple output weights for individual images were averaged and visualized. The resultant averaged images were compared to the individual images to analyze the variability and reliability of the two LIME methods. Without techniques such as those explored in this paper, LIME appears to be unstable because of the simple binary coloring and the ease with which colored regions flip when comparing different analyses. A closer inspection reveals that the significantly weighted regions are consistent, and the lower weighted regions flip states due to inherent randomness of the method. This suggests that improving the weighting methods for explanation techniques, which can then be used in the visualization of the results, is important to improve perceived stability and therefore better enable human interpretation and trust.
Situational understanding requires an ability to assess the current situation and anticipate future situations, requiring both pattern recognition and inference. A coalition involves multiple agencies sharing information and analytics. This paper considers how to harness distributed information sources, including multimodal sensors, together with machine learning and reasoning services, to perform situational understanding in a coalition context. To exemplify the approach we focus on a technology integration experiment in which multimodal data — including video and still imagery, geospatial and weather data — is processed and fused in a service-oriented architecture by heterogeneous pattern recognition and inference components. We show how the architecture: (i) provides awareness of the current situation and prediction of future states, (ii) is robust to individual service failure, (iii) supports the generation of ‘why’ explanations for human analysts (including from components based on ‘black box’ deep neural networks which pose particular challenges to explanation generation), and (iv) allows for the imposition of information sharing constraints in a coalition context where there is varying levels of trust between partner agencies.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.