ANALYSIS OF CONCEPTEST QUESTION DATA: SUMMARY FINDINGS FROM LARGE CLASSES FOR NON-SCIENCE MAJORS AT AN OPEN-ENROLLMENT UNIVERSITY

Analyses of conceptual question responses collected electronically over the past four years from over 1200 students enrolled in Earth Science classes for non-majors illustrated this pedagogical technique can be used as an effective predictor of student success. Data were generally analyzed at the semester level rather than collectively due to instructional variations from semester-to-semester, demographic differences and other data analysis considerations that precluded merging of the data. Student correct response rates and participation (as measured by the number of received responses) correlated strongly with exam and course grades (n>120 per semester; p<0.005; δ=0.7). These trends (though not as strong) appeared prior to the first exam suggesting such data can be used to identify at risk students for early intervention. Similar relationships were found when student data were binned into groups simply based on how quickly they registered their class remote (low, medium and high ID number). Students with low ID numbers (earlier registration) had higher correct response rates (63% versus 57%; p<0.02), higher course grades (83% versus 72%; p<0.001) and better attendance than those with later registration dates (90% versus 74%, even after correcting for missing data). On average, students spent 43 +/- 4 seconds pondering a question before selecting an answer. Response time was moderately correlated to correct response rate, course performance and attendance (δ<0.3) and did not appear to vary by gender, word count or question type (text-only, image-based or mixed). Overall, there were poor correlations between student correct response rates in class compared to summative assessments of the same concepts when tested days or weeks later. There was no evidence this pedagogical technique changed overall course performance when comparing a class using this approach to a similar assessment-based class that did not use remote controls (p>0.5). As such, this method is recommended for classes presently lacking in significant student-student interaction or where it is not practicable to monitor attendance by other means.