A Video Analytic In-Class Student Concentration Monitoring System

Mu Chun Su, Chun Ting Cheng, Ming Ching Chang, Yi Zeng Hsieh

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

Automatic learning feedback monitoring and analysis are becoming essential in modern education. We present a video analytic system capable of monitoring in-class student's learning behaviors and providing feedback to the instructor. It is a common practice nowadays for students to take electronic notes or browse online using laptops and cellphones in class. However the use of technology can also impact student concentration and affect learning behaviors, which can seriously hinder their learning progress if not controlled properly. In this pioneering study, we propose a non-intrusive deep-learning based computer vision system to monitor student concentration by extracting and inferring high-level visual behavior cues, including their facial expressions, gestures and activities. Our system can automatically assist instructors with situational awareness in real time. We assume only RGB color images as input and runable system on edge devices for easy deployment. We propose two video analytic components for student behavior analysis: (1) The facial analysis component operates based on Dlib face detection and facial landmark tracking to localize each student and analyze their face orientations, eye blinking, gazes, and facial expressions. (2) The activity detection and recognition component operates based on OpenPose and COCO object detection can identify eight types of in-class gestures and behaviors including raising-hand, typing, phone-answering, crooked-head, desk napping, etc. Experiments are performed on a newly collected real-world In-Class Student Activity Dataset (ICSAD), where we achieved nearly 80% activity detection rate. Our system is view-independent in handling facial and pose orientations with average angular error < 10°. The source code of this work is at: https://github.com/YiZengHsieh/ICSAD.

Original languageEnglish
Pages (from-to)294-304
Number of pages11
JournalIEEE Transactions on Consumer Electronics
Volume67
Issue number4
DOIs
StatePublished - 1 Nov 2021

Keywords

  • activity detection
  • behavior cues
  • deep learning
  • eye blinking
  • face detection
  • facial expression
  • facial orientation
  • landmark tracking
  • object detection
  • posture analysis
  • student concentration
  • Video analytics
  • yawning

Fingerprint

Dive into the research topics of 'A Video Analytic In-Class Student Concentration Monitoring System'. Together they form a unique fingerprint.

Cite this