A filterbank transmultiplexer (FB TMUX) system has been considered as a kind of multicarrier communication technology. With the property of low sidelobe spectrum, the FB TMUX system has better performance than the OFDM system when there is narrowband interference or synchronization imperfection. In this paper, the design of the prototype filter for a TMUX system is studied. We explore the influence of stopband attenuation and the transition bandwidth on FB TMUX reconstruction performance. From simulation results, we can find a connection between the optimum reconstruction performance and the transition bandwidth of the prototype filter for a given filter length.