In this paper, we present a hardware design for the intelligent surveillance system. We have developed a complete system-level solution on algorithm level, and then algorithm has been implemented in very-large-scale integration (VLSI). The behavior of the moving objects is evaluated using adaptive search method. We propose the method to track the moving people in successive frames by object boundary box and velocity without color cues or appearance model. Even though people are interacting with each other or the occlusion is occurring due to other foreground objects, still the proposed algorithm can perform well in such situations. Furthermore, we consider the distance of object from camera for an adaptive search range to deal with the people movement issue. Although foreground is similar to the background in some conditions, the proposed algorithm can still detect the object well. We also propose an embedded data compression technique, which does not only reduce the computational complexity, but also achieves high compression rate. The overall system is developed in a platform-based system-on-a-programmable-chip for demonstration of result. In VLSI implementation results, the logical gate count is 139.890 K and the throughput of foreground detection is 6403 K pixels/s.