Atomic force microscopy (AFM) is a powerful measurement instrument which can build 3-D topography image of conductive and nonconductive samples at nanoscale resolution. However, due to the scan method of conventional AFM, the induced mechanical resonance of the scanner and the scan in area of uninterest would strictly limit the scan speed. In this study, we improve these problems with our designed AFM system from three aspects. First, the sinusoidal trajectory is applied to lateral scanning of the AFM rather than the traditional raster trajectory, so the scan rate can be increased without inducing vibration of the lateral scanner. Second, with this promising scan trajectory, the internal model principle-based neural network complementary sliding-mode controller and adaptive complementary sliding-mode controller are designed to achieve high precision scanning and to cope with the system parameter uncertainties and external disturbance. Finally, with the aid of an auxiliary optical microscopy and the scanned information during the scanning process, scan path planning can be adopted to focus the scanning on samples such that the total scan time is further shortened. Extensive experimental results are provided to show the appealing performance of the proposed method.