Long-period pulse-like strong ground motions were recorded near the Milun fault and caused serious building damages in the Hualien City during the main shock of the 2018 Hualien, Taiwan, earthquake. In this paper, questionnaire survey on feeling of shaking, indoor damages, structural, and nonstructural damages to high-rise reinforced concrete residential buildings were analyzed by three building groups according to the positions to the Milun fault and different site conditions. Relationships between the results of questionnaire survey on building damages and characteristics of strong ground motions were studied. Furthermore, a characterized seismic source model was proposed to elucidate the generation mechanism of near-fault long-period strong ground motions. In this model, a shallow strong motion generation area (SMGA7) in the south part of Milun fault with rise-time of 2.6 s contributed to the long-period pulse-like strong ground motions. Besides, a deep SMGA (SMGA8) under the SMGA7 with rise-time of 1.2 s also contributed to the strong ground motions. Using this model, strong ground motions at the sites of buildings for questionnaire survey were synthesized. Because of the directivity effects, amplitudes of ground motions were increased from north to south along the Milun fault, which can explain the serious building damages at the buildings located at the southwest to the Milun fault. The reasons of building damages were also examined from the aspect of predominant periods of ground sites and fundamental periods of buildings, which were extracted from microtremor measurement. Damages at the buildings located on the central part of the hanging wall with relatively good site condition were the lightest for almost all the surveyed items. Pulses with period in the range of 1 s–2 s which were generated by the SMGA8 and amplified by the near surface layers are considered to be responsible for building damages.