BETTER TOGETHER: ENRICHING EYE TRACKING DATA FROM A VIRTUAL VOLCANO SIMULATION WITH CONCURRENT USABILITY MEASURES AND CONCEPTUAL ASSESSMENTS

Advances in affordable physiological monitoring, especially eye tracking, have enabled geoscience educators to begin exploring issues of attention, engagement, and other geocognitive processes. These techniques are now being brought to bear on evaluation of Virtual Volcano, an interactive 3D computer simulation designed to address specific conceptual targets from survey-based preconceptions research. Primary learning goals include: 1.) what drives volcanic activity, 2.) how and why eruptive styles vary, and 3.) the prediction of volcanic hazards. Preliminary results using traditional pre-test vs. post-test assessments revealed modest improvements in the understanding of eruptive mechanics (learner normalized gains of +5% to +8%). However, a more comprehensive view of learner behavior is necessary to evaluate whether, and to what extent, instructional technologies like Virtual Volcano have met their design goals.

To that end, we present a triangulated research methodology wherein traditional education measures (pre-test and post-test scores), physiological monitoring techniques (eye tracking and facial affect measurement), and direct user experience measures (mouse activity logs, task performances, and think aloud protocols) combine to produce a much fuller picture of learner behavior(s) than could be achieved by using any one of these methodologies individually. Furthermore, this approach enables more fine-grained analyses of learner performance on a per-task basis rather than relying solely on per-learner aggregate data.

We also propose a best practices framework for the development of interactive simulations in the geosciences based on formative assessment, rapid prototyping and testing, and triangulated evaluation of outcomes at all stages of the design process.