This paper presents a novel template matching method to detect and track pedestrians for people counting in real-time. Firstly, a novel background subtraction method is proposed for extracting all foreground objects from background. Then, a shadow elimination method is used to remove unwanted shadow from the background. In order to identify pedestrians from non-pedestrian objects, this paper proposed a novel grid-based template matching scheme to robustly verify each pedestrian. Usually, a pedestrian will have different appearances at different positions. The grid-based approach can effectively reduce the perspective effects into a minimum since it uses different templates to record the appearance changes at each grid. When more templates are used, the detection process will become more inefficient. To speed up its efficiency, an integral image is used to filter out all impossible candidates in advance. Lastly, a tracking method is applied to tracking the direction of each moving pedestrian so that the real number of passing people per direction can be counted more accurately. Experimental results have proved that the proposed method is robust, accurate, and powerful in people counting.