Improved convolutional neural network based scene classification using long short-term memory and label relations

Convolutional neural network (CNN) is more and more important in pattern recognition. In this work, we adopt label relations and long short-term memory (LSTM) to develop an accurate CNN-based scene classification algorithm. Traditional scene classification algorithms assume that labels are mutually exclusive. However, this is not reasonable when an image has a variety of objects and hence has multiple labels. In this work, we apply two label relations, which are exclusive and hierarchy relations, to improve the accuracy of multiple-label scene classification. For example, it is impossible that an image has both the labels of 'factory' and 'garden'. If the label 'factory' is assigned to an image, the probability that it has the label of 'garden' should be lowered. We also use image captioning to construct a scene classification model and propose an LSTM based method to further explore label relations and obtain more accurate results for scenic image labeling.