Robustness of visual tracking using mobile cameras plays a key role in success of emerging applications of handheld devices and mobile platforms. Therefore, this paper proposes a novel compensated motion model for a particle filter to achieve fast and accurate estimation. At each time instant, the particle filter indirectly predicts the object motion in the image by compensating the camera motion at the prediction stage, where camera motion projected onto the image are extracted with the aid of speed-up robust feature (SURF). The object position in the image is then corrected based on observations of the color feature. Experimental results show that our proposed tracking scheme with a small number of particles performs well on mobile platforms even if the rapid and irregular object motion exists.