This paper proposes a new structure for applying to real-time evolutionary face tracking of the streaming images. In the described method, first we use the features such as skin color models and facial proportions to extract the face region and complete the pre-processing, and then track the moving face location with particle swarm optimization algorithm. The experimental results show that the face detection method which we have developed has a higher detection rate and complex backgrounds tolerance than the traditional Viola-Jones detector. Compared to the method of searching the face region in the whole image with the streaming image sequences one by one, the execution effect of searching the face region in the key region images by using the method of evolutionary face tracking system based on particle swarm optimization algorithm has a fast execution speed and better accuracy. Furthermore, in order to achieve the goal of real-time processing, we also complete the software design and verify with the personal computer as well as Nios II embedded processor-based FPGA platform, and analyze the efficiencies of various modules to find the functional module of efficiency bottleneck for the further implementation of a hardware architecture, to improve the overall efficiency of the system through a hardware/software co-design. Experimental results demonstrate that the proposed structure is efficient and robust in face tracking under dynamic environments with real-time performance.