Image captioning involves generating a sentence that describes an image. More recently, it has been driven by encoderdecoder approaches where the encoder such as convolutional neural network (CNN) can extract the visual features of an image. The extracted visual features are passed to a decoder such as a long short-term memory (LSTM) network in order to generate a sentence that describes the image. One major challenge with this approach is to precisely include the scene of an image in the generated sentences. To resolve this challenge, visual scene features have been used with unidirectional LSTM decoders. However, for long sentences, this limits the precision of the generated text. This research proposes a novel approach to generate sentences using visual scene information with a bidirectional LSTM decoder. The encoder is based on Inception v3 to extract the object features and Places365 to extract the scene features. The decoder uses a bidirectional LSTM to generate a sentence. The encoder-decoder model is trained using the Flickr8k dataset. Results show improved performance for generating longer sentences with a 9% increase in BLEU-3 and a 12% increase in BLEU-4 scores compared to compared to other encoder-decoder methods that are limited to only using global image features. Visually impaired people that use screen readers would benefit from this research as they would get an enhanced description of an image that includes the background scene thereby creating a wholesome picture in the mind of the reader.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.