Modelling a spatial-motion deep learning framework to classify dynamic patterns of videos

Sandeli Priyanwada Kasthuri Arachchi, Timothy K. Shih, Noorkholis Luthfil Hakim

研究成果: 雜誌貢獻期刊論文同行評審

6 引文 斯高帕斯(Scopus)


Video classification is an essential process for analyzing the pervasive semantic information of video content in computer vision. Traditional hand-crafted features are insufficient when classifying complex video information due to the similarity of visual contents with different illumination conditions. Prior studies of video classifications focused on the relationship between the standalone streams themselves. In this paper, by leveraging the effects of deep learning methodologies, we propose a two-stream neural network concept, named state-exchanging long short-term memory (SE-LSTM).With the model of spatial motion state-exchanging, the SE-LSTM can classify dynamic patterns of videos using appearance and motion features. The SE-LSTM extends the general purpose of LSTM by exchanging the information with previous cell states of both appearance and motion stream. We propose a novel two-stream model Dual-CNNSELSTM utilizing the SE-LSTM concept combined with a Convolutional Neural Network, and use various video datasets to validate the proposed architecture. The experimental results demonstrate that the performance of the proposed two-stream Dual-CNNSELSTM architecture significantly outperforms other datasets, achieving accuracies of 81.62%, 79.87%, and 69.86% with hand gestures, fireworks displays, and HMDB51 datasets, respectively. Furthermore, the overall results signify that the proposed model is most suited to static background dynamic patterns classifications.

期刊Applied Sciences (Switzerland)
出版狀態已出版 - 1 2月 2020


深入研究「Modelling a spatial-motion deep learning framework to classify dynamic patterns of videos」主題。共同形成了獨特的指紋。