This study aims to design and develop a game-based English interactive learning environment using video-capture virtual reality technology. The system is designed for English teaching course in elementary school by physical body interaction. Two detection methods are used in the system, one is RGB color values detection, the other is gestures detection. There are six stages of learning activities are designed by integrating into English curriculum with specific tasks and learning objectives. In addition to operate the system by physical movement, students can also use various properties, such as pistol, x-ray searchlight, magnet, spray paint can, and conical cap. This is designed to improve the accuracy of detection, as well as to increase the fun during the learning process. The system also provides an interface for teachers to edit the teaching materials when they have different requirement for teaching. A preliminary study was conducted at an elementary school of 30 second grade class students. The data collected from the pre- and post- tests. The result shows that there are significant differences between the pre-test and the post-test (p=.031<.05).