The ability to detect and track specific people is considered a key prerequisite for serving mobile robots. This paper presents a tracker using a binocular camera capable of tracking a specific human in both indoor and outdoor environments. The proposed tracker detects the target with the combination of human detection, color histogram, and block matching algorithm (BMA). When the color of the object is similar, the position of the object is determined by the predictor of the Kalman filter. Finally, the mobile robot is controlled based on the depth information of the target object. The effectiveness of the system is verified by human experiments in indoor and outdoor environments.