SEMANTIC SFM: ENHANCING GEOLOGICAL MAPPING WITH AUTOMATED 3D SEMANTIC ANALYSIS

Structure-from-Motion (SfM), particularly when combined with Unpiloted Aerial Vehicles (UAVs), has significantly enhanced geoscientific research by providing detailed 3D models, such as point clouds, of geological surface features. Extracting semantics—detecting and classifying features of interest—in point clouds is critical for many geoscience studies. However, this process is usually performed manually, which is time-consuming. To address this issue, we introduce Semantic SfM, an approach to automatically extract semantics in SfM-derived point clouds. This method diverges from conventional practices that apply deep learning directly to point clouds. Instead, we utilize 2D image instance segmentation techniques on input images, allowing for the rapid detection and classification of features of interest. Subsequently, Semantic SfM leverages the camera poses and point clouds derived from standard SfM processes. We project the points of the point cloud back onto the camera image planes, aligning them with the 2D segmentations. This process establishes a correlation between the semantics in the 2D images and the corresponding points in the 3D point cloud. A key challenge we address is the association of semantics across different images, as the semantics are independently generated from various images. Specifically, we have developed an object registration algorithm that efficiently merges semantic data from different images. This algorithm assesses the point overlap (3D intersection of union) between projected objects in different images, merging the semantics when the overlap exceeds a set threshold. Our Semantic SfM approach is designed to be compatible with existing SfM software, such as Agisoft and OpenDroneMap, and does not require additional data preprocessing. A notable advantage of our approach is the elimination of the laborious task of 3D point cloud annotation. Additionally, our method further eliminates the need for annotating 2D images by utilizing state-of-the-art zero-shot deep-learning models, such as Grounding DINO [1] and SAM [2]. This innovation marks a significant stride in the field of geoscience, offering a more efficient, accurate, and automated process for extracting valuable semantic information from 3D models.