This paper proposes a digital modulation /demodulation measurement system implemented by the Agilent advanced design system (ADS)-connected solution with vector signal analyzer (VSA) software, VSA instrument built into an analog digital converter (ADC), and an ESG vector signal generator. The ADS system software is suitable to design a digital modulation signal for error vector magnitude (EVM) analysis in RF transmitter (TX). The bit error rate (BER) analysis in RF receiver (RX) was estimated by the recording file from VSA software simultaneously. The proposed testing platform can measure the characteristics of RF transceivers, such as EVM, BER, constellation, and eye diagrams. The nonlinearity and distortion properties were evaluated through the EVM system parameter.