EFFICIENT MANAGEMENT OF GEON LIDAR DATASETS USING COMMODITY CLUSTERS

The GEON LiDAR Workflow (GLW) is an innovative and breakthrough approach for the processing and interpolation of LiDAR (Light Distance And Ranging) point cloud datasets [1], developed as part of the NSF Geosciences Network (GEON) Project. One of the many challenges of the current GLW architecture is the efficient management of the LiDAR datasets due to its sheer volume and density. A single dataset can have over a billion points. The current GLW architecture makes use of a spatially-enabled IBM DB2 database that stores the point cloud data in a spatial data type format and runs on an IBM terascale machine, which is a Teragrid resource. The IBM system is a high-end multiprocessor with 32 processors, 128 GB main memory that is especially suitable for data intensive tasks.

In this paper we propose a solution for hosting LiDAR datasets on affordable commodity clusters, without compromising on performance. We can achieve this by taking advantage of IBM DB2's partitioning feature. In this approach, the LIDAR datasets are partitioned across multiple machines that form a cluster. Each database partition has its own independent database manager, each with its own data, configuration files, indexes, and transaction logs, hence ensuring better scalability and more combined processing power. We will also implement an adaptive and flexible spatial indexing system based on user access patterns. If a particular spatial region is accessed more frequently, then we will create a larger number of spatial grid indexes (potentially of varying grid sizes) for the corresponding data table(s) in the LiDAR database. This will help speed up access to areas of greater interest and usage.

The science leads for GLW are Prof. Ramon Arrowsmith and Chris Crosby at the Arizona State University. GLW is available for public use via the GEON Portal [3].

[1] Efrat Jaeger-Frank, Christopher J. Crosby, Ashraf Memon, Viswanath Nandigam, J. Ramon Arrowsmith, Jeffery Conner, Ilkay Altintas, Chaitan Baru, A Three Tier Architecture for LiDAR Interpolation and Analysis, Lecture Notes in Computer Science, Volume 3993, Apr 2006, Pages 920-927

[2] Nandigam, V., Baru, C., Chandra, S., and Frank, E., LIDAR-IN-A-BOX: Serving LIDAR Datasets Via Commodity Clusters, Geoinformatics 2007 Conference, San Diego, CA, May 2007

[3] GEON LiDAR Workflow, http://portal.geongrid.org/lidar.