SATD-based intra mode decision for H.264/AVC video coding

In this letter, a rate estimation technique using transform coefficient variance is proposed in the rate-distortion cost function for intra mode decision. The new cost function achieves better coding performance than the cost function using the p-domain model. The new cost function brings out an average speed-up factor of 13 when compared with the rate-distortion optimization (RDO) mode decision, but with considerable degradation in rate-distortion performance. To improve the coding performance, we propose a sum of the absolute transformed differences (SATD)-based mode decision algorithm, which incorporates both SATD and the variance information into the RDO mode decision method. The results show that the SATD-based mode decision scheme can achieve a significant improvement in computation (with 21% of total encoding time, or a speed-up factor of 5) with negligible peak signal-to-noise ratio loss. The proposed algorithm is also compared with two other distinct algorithms, and the results indicate that the proposed algorithm outperforms these two edge detection algorithms in coding performance, as well as time saving.