Geoelectric self-potential (SP) signals are sensitive to natural and anthropogenic factors. The SP spectral characteristics under the different factors in Taiwan were investigated, and the SP spectral scalings were correlated with urbanization level, seismicity, and crustal deformation. The ambient SP noise models were first established by estimating the probability density functions of the spectrograms at each frequency. The effects of the natural and anthropogenic factors on the SP signals are understood by comparing the SP noise models under various conditions, such as precipitation, urbanization, and electric trains. Results show that the SP signals in areas of high industrialization and human activity and areas close to train stations behave as white noises and exhibit a distinct spectral ripple at frequencies around 1 Hz. On the other hand, the SP spectral power law parameters, Gutenberg-Richter b values, and dilation strain rates were estimated by using the SP, earthquake catalog, and GPS data, respectively, during 2012–2017. By investigating the correlations of the SP spectral parameters with the Gutenberg-Richter b value, dilation strain rates, and urbanization level, the SP optimal frequency band is found between 0.006 and 1 Hz due to the high correlation between the SP and seismicity data and between the SP and dilation data and the low correlation between the SP and urbanization data. Hence, this study may help the filtering and screening of the SP data and facilitate the understanding of the mechano-electric behavior in the crust.