Building a search engine for geospatial data

With the advance of computer network and the World-Wide Web technology, everyone can easily setup their own websites or post content on the Web. The concept of allowing users to contribute Web resources has been summarized as the second generation of the Web, which is the Web 2.0. This Web 2.0 concept continues to generate and attract rapid growth in Web resources. However, while resources are being generated faster than anytime in the past, the "big data" has become a critical data management issue. Among the resources generated and shared on the Web, geospatial data also fits into the 3Vs characteristics of big data which are velocity, volume and variety. While geospatial data are scattered on the World-Wide Web, data discovery becomes one of the critical issues, which makes users difficult to identify and discover the geospatial data they need. Therefore, the ultimate goal of this research is to build a search engine for users to search any geospatial data available on the Web. To realize this search engine, the first step is to design and implement geospatial web crawlers to discover different types of geospatial data. Web crawlers have been widely used to discover web pages on the Web. In this paper, we propose using web crawlers to collect geospatial web services that follow open standards such as the Open Geospatial Consortium (OGC) Sensor Observation Service (SOS), Web Feature Service (WFS), Web Map Service (WMS) and Web Coverage Service (WCS). These standards define service interfaces hosting sensor observations, features, maps and coverages respectively. The preliminary results show that the proposed solution is able to discover 31 SOSs, 22 WFSs, 280 WMSs, and 59 WCSs.