The 2018 Mw 6.4 Hualien earthquake generated a large peak-to-peak velocity of over 2 m/ s, with a period of 3 s at the south end of the Milun fault, which resulted in the collapse of five buildings. To investigate the shallow subsurface soil structure and evaluate possible effects on the ground motion and building damage, we performed microtremor measurements in the Hualien basin. Based on the velocity structure jointly inverted from both Rayleigh-wave dispersion curves and microtremor horizontal-to-vertical spectral ratio data, we found that the shallow subsurface structure generally deepens from west to east. Close to the Milun fault, the structure becomes shallower, which is consistent with faulting during the 2018 earthquake and the long-term tectonic displacement. There is no significant variation for the site conditions in the north–south direction that can explain the large peak ground velocity in the south. As a result of the dense measurements in the heavily damaged area, where three high-rise buildings totally collapsed, these locations have the average S-wave velocity of the upper 30 m (AVS30 ) values and are relatively high compared to the more distant area from the Meilun River. This is somewhat unusual, because lower AVS30 values indicating softer ground conditions are expected close to the river. We did not find any characteristic subsurface soil structure that may contribute to the building collapses. The large 3 s pulse was probably generated by source effects, rather than subsurface soil amplification.