APPLYING CLASSICAL TEST THEORY AND ITEM RESPONSE THEORY IN THE VALIDATION OF THE EARTH SYSTEMS THINKING CONCEPT INVENTORY

A key area in understanding the development of systems thinking skills in the geosciences relies on the development of a quantitative tool to assess systems thinking abilities in the context of the Earth sciences. This talk will overview how both classical test theory and item response theory were used to validate Earth systems thinking concept inventory (EST CI). The instrument was administered in multiple iterations using Amazon’s Mechanical Turk (MTURK) (n= 1004). By using these two approaches to validity and reliability in a complementary fashion, we were able to take an iterative approach to provide robust evidence for both validity and reliability. Additionally, the instrument is semi-customizable as language regarding feedbacks can be shifted between using ‘positive’ and ‘negative’ or ‘reinforcing’ or balancing’ terminology, with the latter resulting in better reliability among a largely novice audience. This talk will provide tools for practitioners to apply elements of both classical theory and item response theory to the development of their own instruments and overview the theoretical underpinnings of both approaches and how they can be integrated.