Multi-agent reinforcement learning (MARL) for vehicular communication management is an emerging topic attracting considerable research attention. In this paper, to enhance the system throughput and spectrum efficiency, the vehicular agents can select different transmission modes, power, and sub-channels to maximize the overall system throughput in a decentralized manner. We propose a novel MARL resource allocation algorithm capable of congestion avoidance for vehicular networks with a multi-agent extension of advantage actorcritic (A2C). The cooperative action and congestion avoidance are achieved by global rewards and a unique dump channel respectively. Moreover, comparison with landmark schemes is conducted on the realistic setup. The result shows that the agent achieves favorable performance with the proposed scheme to the environment.