We propose a hybrid coding system that utilizes both Wavelet Packet (WP) and DCT techniques. To process each audio frame, the system selects either WP or DCT as the transform based on the frame flatness measures in the wavelet domain and the frequency domain. If DCT is chosen, all DCT coefficients are quantized by a non-uniform quantizer according to the frequency masking curve. On the other hand, frame data are segmented into 26 fixed subbands when WP is chosen. Then, the system selectively utilizes DCT to promote frequency resolution of each subband based on the subband flatness measure. By quoting optimal bit-allocation for non-ideal filter bank, the masking threshold from psychoacoustic model can be translated into specific criteria in the wavelet domain for quantization. Experiment results show that the proposed system is superior to MP3 and AAC LC profile at 64k bps.