Human action recognition is one of the most important issues in computer vision. In this paper, the main idea is to design a general approach to recognize the human behavior. This approach is based on a pre-collected action database, which extracted by the depth images and forms the sequences of skeletons, and trained by the proposed Action Forests (AF) model. AF extends the random forest algorithm by using different decision functions to fit the skeletal features in 3D space. The system achieves the real-time classification result without the limitation of background and camera position. In the experiments, we collected several human behavior with single-character actions and two-character interactions to train the AF model. The skeleton features were retrieved from the depth sensor Kinect. We investigated the effect of several training parameters in AF. In conclusion, AF can learn the skeletal features efficiently and runs at 30 frames per second on action classification with high accuracy.