The vibration modulated stitching interferometry acquires many subaperture phases timely forming the high overlapping density subapertures for asphere phase stitching. The large number of overlapping subapertures had been proven effective in suppressing the reference error. In this research, we propose a pixel-by-pixel reference calibration method by using the averaged difference between the stitched phase and compensated phase within the overlapping subapertures. The measurement for both tested optics and calibration of the reference optics are accomplished in a single phase stitching process. The requirement for a high-quality reference optics or dedicated reference calibration procedure for subaperture stitching interferometry is therefore significantly eliminated. Both the simulation and experimental results shows the feasibility of the proposed method for high frequency reference error and most of the form error in the third order Zernike aberrations.