Because of the change of family structure and population ageing, elderly and children care is become a very important issue in modern society. When adults are busy working, they have no time to care elderly and children who standalone in the home. This paper proposes an elderly and children care system to solve this important problem. The proposed intelligent surveillance system is based on action recognition technique of image processing. In this paper, a three stream convolution neural network is proposed for recognize human actions such as fall floor and baby craw. If the system detect abnormal activities are occurred, it will raise alarm and notice family members. In the experiment, there are totally 21 categories activities are collected from HMDB-51 dataset, UCF-101 dataset and Internet. The proposed system achieves 93.42% recognition rate of selected actions.