IDENTIFYING IMPORTANT OBSERVATIONS USING CROSS VALIDATION AND COMPUTATIONALLY FRUGAL SENSITIVITY ANALYSIS METHODS

Sensitivity analysis, calibration, and uncertainty evaluation methods are critical to developing useful models of complex hydrologic systems for which important characteristics can not be measured accurately and(or) completely enough to define model input values fully (Saltelli et all, 2008; Hill and Tiedeman, 2007; both textbooks published by Wiley). These methods allow the modeller to explore the relations between different types of data and the processes represented in the model, including the testing of hypotheses about system structure (alternative models).

Model nonlinearity and its consequences for model calibration and sensitivity analysis are evaluated by Kavetski1 and Kuczera (2007, Water Resources Research - WRR). Experience suggests that many models of natural systems are linear enough for local sensitivity analysis methods to be useful (see examples of groundwater flow and advective transport, conservative and reactive groundwater transport, and streamflow and transport cited by Foglia et al., 2009, WRR). This suggests that the concern expressed by Saltelli et al. [2008, p. 11] that local methods are inefficient in terms of the analyst’s time is perhaps overstated, but clear comparison of nonlinear and linear methods are needed to better understand the opportunities and limitations of linear methods.

Local sensitivity analysis is based on first-order, second moment (FOSM) approximations. The linear statistics for which results are shown in this presentation are calculated using UCODE_2005 (Poeter et al., 2005, USGS report). They include fit-independent and fit-dependent statistics (Hill and Tiedeman, 2007). The fit-independent statistics are dimensionless scaled sensitivities (DSS), leverage, and observation-prediction (OPR). The fit-dependent statistics are DFBETAS and Cook’s D. The role of parameter interdependence as measured by parameter correlation coefficients (PCC) is discussed (only DSS does not account for parameter interdependence).

The alternative to local sensitivity analysis commonly is global sensitivity analysis (Saltelli et al., 2008). Global methods most commonly used in hydrology are GLUE (Generalised Likelihood Uncertainty Estimation) and MCMC (Markov Chain Monte Carlo). Here, we compare linear methods with results obtained through cross-validation. In cross validation, all observations are used to produce a calibrated model and associated predictions. Then one or more observations are removed, the regression repeated, and resulting changes in parameter values and predictions are evaluated. Large changes in parameter values and predictions indicate the associated observations are important.

In this work, comparisons of local sensitivity analysis and cross-validation are conducted using a groundwater model of the Maggia Valley, Southern Switzerland (Foglia et al., 2007, Ground Water); applicability to climate models is inferred using Torn and Hakim (2008, Monthly Weather Review). Results show that the frugal linear methods produced about 70% of the insight from about 2% of the model runs required by the computationally demanding methods. Linear methods were not always able to distinguish between moderately and unimportant observations. However, they consistently identified the most important observations. Importance both to estimated parameters and predictions of interest was readily identified.

Results suggest that it can be advantageous to consider local sensitivity analysis in model evaluation, possibly to provide insights used to improve the design of more demanding methods.