The digital surveillance system becomes more and more popular in recent years. The systems attempt to raise amount of high resolution cameras, consequently those systems stupendously increase the computational load on central server. As in the intelligent object recognition processing flow, the technique on tracking multiple targets, such as tracking group of people through occlusion, is still challenging. In this paper, we present a platform-based System-on-a-Programmable-Chip (SOPC) surveillance system. We discuss the behavior of the moving objects with adaptive search method. With this system, we can track the moving people in successive frame by object boundary box and velocity without color cues or appearance model. The proposed system can still solve the problem on the detection error even the foreground is similar to the background. The overall architectures of the intelligent surveillance system consist of three parts: multiple background maintenance (MBM) accelerator, object labeling and an SOPC processor.