We report the results of a pulse variation analysis for the fourth discovered accretion-powered millisecond pulsar, XTE J1807-294, during its 2003 outburst as observed by the RossiX-Ray Timing Explorer. Pulsation is significantly detected only in the first ∼90 out of ∼150 days' observations. The pulse phase variation is too complex to be described by a model with orbital motion plus a simple polynomial. Precise orbital parameters, with Porb = 40.073601(8) minutes and ax sin i = 4.823(5) lt-ms, were obtained after detrending the pulse phases in daily observed 150 s segments, folded with a constant spin frequency without including the Keplerian orbit. The binary barycenter-corrected pulse phases show a smooth evolution and clear negative phase shifts coincident with the flares seen in the light curve and with enhancements of fractional pulse amplitude. The nonflare pulse phases for the first ~60 days of data are well described by a fourth-degree polynomial implying that the neutron star was spun up during the first ∼60 days with a rate v̇ = (1.7 ± 0.3) × 10-13 Hz s-1 at the beginning of the outburst. Significant soft phase lags up to ∼500 μs (∼0.1 cycles) between 2 and 20 keV were detected for the nonflare pulse phases. We conclude that the anomalous phase shifts are unlikely to be due to the accretion torque but could result from movement of the "hot spot" on the surface of the neutron star.