In this work, a vehicle tracking system is developed to deal with daytime and nighttime traffic surveillance videos. For daytime videos, vehicles are detected via background modeling. For nighttime videos, headlights of vehicles need to be located and paired to initialize vehicles for the tracking purpose. An algorithm based on likelihood computation is developed to pair the headlights of vehicles. In addition, we apply a specialized system state transition model of the Kalman filter to adapt to common settings of traffic surveillance cameras. The experimental results have shown that the proposed method can effectively track vehicles in both daytime and nighttime surveillance videos. 2010 IEEE.