As the smart city concept becomes increasingly popular and important in both industries and academia, a broad range of data collected from various sources is employed to assist the policy maker in making more informed decisions. Among these data, some are well-structured and stored in spreadsheets, such as the building and site information stored in archive documents, whereas the rest are unstructured, such as the images or videos taken by unmanned aerial vehicles and written texts extracted from most recent news reports. This paper proposes a multi-source data-driven framework that can rapidly estimate the seismic response of non-ductile reinforced concrete frame buildings by leveraging the images and well-tabulated data. This framework meticulously incorporates computer vision and well-structured data processing techniques. To demonstrate its efficacy, the proposed framework is applied to a comprehensive dataset, which includes 1400 non-ductile reinforced concrete frame designs, their nonlinear structural models, associated seismic responses, and the building exterior images. A thorough review of the application result reveals that the proposed framework is able to efficiently and reliably estimate the seismic drift demands in non-ductile reinforced concrete frames subjected to earthquake scenarios. Such a multi-source data-driven framework would become an essential component in constructing a smart city.