We numerically investigate a new function modified from Hamming function to apply in weighting the waveguide structure of a mismatched optical coupler with the characteristics of low crosstalk, short length, and broad C+L-band. The full factorial design and beam propagation method are being used to seek the optimal structure parameters of coupling waveguide. This modified Hamming weighting function (M-HWF) is proved to have the superior performances to Hamming weighting function (HWF) in several ways. In theoretical computation, the M-HWF and HWF waveguides have the coupling lengths of 11 mm and 16 mm, respectively at crosstalk of -40 dB and operating wavelength of 1.57 μm. After the numerical design of waveguide structure parameters, the M-HWF also obtain a shorter coupling length of 4.50 mm than coupling length of 5.90 mm of HWF at crosstalk of -40 dB and within the C+L-band (1.53∼1.61 μm). The M-HWF waveguide with the shortest coupling length of 4.5 mm can even reach a broader bandwidth between 1.50 and 1.70 μm.