The aim of this study is to design and develop a Physically Interactive Learning Environment, the PILE system, by integrating video-capture virtual reality technology into a classroom. The system is designed for elementary school level English classes where students can interact with the system through physical movements. The system is designed to be easily established with a minimal amount of equipments that includes a personal computer, a webcam, and a projector. The learning activities comprise six stages, holding specific tasks and learning objectives. Each stage is designed with a distinct device. These devices, including a conical cap, a pistol, a searchlight, a magnet, and a spray paint can, are designed to improve the accuracy of detection as well as to increase student enjoyment during the learning process. Furthermore, the system consists of five functional modules, such as providing an interface for teachers to incorporate appropriate learning materials according to their specific teaching requirements. An empirical study is conducted to examine the effects of the use of the PILE system by comparing two different types of English learning methods with 60 second-grade students from two classes at an elementary school in Taiwan. Four different tests are used to assess the different aspects of the system: an English learning achievement test, a questionnaire assessing students' learning motivation, a Short Feedback Questionnaire (SFQ), and a teacher interview. The results of students' English learning achievement tests show that there was a significant difference between the pretest and the posttest in the experimental group, as well as between the two groups in the delayed test. These results demonstrate that the system had a significantly beneficial effect on students' long-term learning. The results from the questionnaires on students' learning motivation and the SFQ reveal that the system enhanced the students' learning motivation. The results gained from the teacher's interview illustrate that the teacher believed this system was beneficial in assisting English learning. All findings collectively demonstrate that the proposed PILE system effectively assist English learning in a classroom environment.