GeoWeb Crawler: An extensible and scalable web crawling framework for discovering geospatial web resources

Chih Yuan Huang, Hao Chang

Research output: Contribution to journalArticlepeer-review

18 Scopus citations


With the advance of the World-Wide Web (WWW) technology, people can easily share content on the Web, including geospatial data and web services. Thus, the "big geospatial data management" issues start attracting attention. Among the big geospatial data issues, this research focuses on discovering distributed geospatial resources. As resources are scattered on the WWW, users cannot find resources of their interests efficiently. While the WWW has Web search engines addressing web resource discovery issues, we envision that the geospatial Web (i.e., GeoWeb) also requires GeoWeb search engines. To realize a GeoWeb search engine, one of the first steps is to proactively discover GeoWeb resources on the WWW. Hence, in this study, we propose the GeoWeb Crawler, an extensible Web crawling framework that can find various types of GeoWeb resources, such as Open Geospatial Consortium (OGC) web services, Keyhole Markup Language (KML) and Environmental Systems Research Institute, Inc (ESRI) Shapefiles. In addition, we apply the distributed computing concept to promote the performance of the GeoWeb Crawler. The result shows that for 10 targeted resources types, the GeoWeb Crawler discovered 7351 geospatial services and 194,003 datasets. As a result, the proposed GeoWeb Crawler framework is proven to be extensible and scalable to provide a comprehensive index of GeoWeb.

Original languageEnglish
Article number136
JournalISPRS International Journal of Geo-Information
Issue number8
StatePublished - Aug 2016


  • GeospatialWeb
  • Open Geospatial Consortium
  • Resource discovery
  • Web crawler


Dive into the research topics of 'GeoWeb Crawler: An extensible and scalable web crawling framework for discovering geospatial web resources'. Together they form a unique fingerprint.

Cite this