WHAT MATTERS MOST? MEASURING FEATURE IMPORTANCE FOR GEOTHERMAL RESOURCES USING SUPERVISED LEARNING

Recent evaluation of strategies for conventional hydrothermal resource assessment in the United States has relied upon machine learning methods (i.e., logistic regression, SVMs, XGBoost, and multilayer perceptron neural networks [i.e., MLPs]) to predict resource favorability using features (i.e., heat flow, distance to faults, distance to magma bodies, maximum horizontal strain, and seismic event density) from the U.S. Geological Survey’s 2008 Geothermal Resource Assessment. Two of the machine learning algorithms (i.e., SVMs and MLPs) must rely on model-agnostic measures of feature importance (i.e., measures of feature importance that are applicable regardless of an algorithm’s conceptual framework; e.g., sensitivity analyses and SHapely Additive exPlanation [SHAP] values), while the other two machine learning algorithms also offer straightforward, model-gnostic (i.e., algorithm-specific) measures to interpret the relative contributions of features on favorability predictions (i.e., feature coefficients for logistic regression, weight, gain, cover, and F score for XGBoost). Relative feature importance is measured for all machine learning algorithms using the model-agnostic measures, and, when possible, model-gnostic measures are shown for comparison.

The various measures of feature importance, in general, are consistent with each other. Heat flow and distance to faults are commonly identified as the most important features and strain and seismicity as the least important features in the majority of the models produced by the machine learning algorithms. However, some measures of feature importance deviate from this behavior when used with some machine learning algorithms (e.g., sensitivity analysis by F1 score or SHAP values with ensembled SVMs). We discuss these differences between measures of feature importance across the machine learning algorithms and what these differences may represent in the context of predicting geothermal resources. In so doing, we caution that the rankings from some measures of feature importance are considerably unlike the rankings from other measures. Hence, we demonstrate that the evaluation of feature importance benefits from a multi-measure approach that permits higher confidence when the rankings are consistent across different measures.