SOME TYPES OF QUESTIONS ARE HARDER THAN OTHERS: ANALYZING STUDENT PERFORMANCE ON 1200 QUIZ QUESTIONS FOR AN INTRODUCTORY GEOSCIENCE COURSE

Geoscience instructors routinely assign a variety of forced-choice questions for students as part of either formative assessments in class (e.g., Conceptests) or summative quizzes and/or exams. Relatively little research has been conducted to compare and contrast student success between different question types. We analyzed thousands of student answers to approximately 1200 questions using a quizzing tool (CLASS) that was deployed in a Physical Geology course. CLASS (Confidence-based Learning Accuracy Support System) allows us to measure both student performance (% correct responses) and student confidence in their answer choice. The questions included in this study were reviewed by colleagues for content and construct validity and edited accordingly before use.

We examined five different types of questions that were used frequently across a dozen quizzes. The question formats included: 1) True/False; 2) Multiple choice; 3) Multiple choice with an image; 4) Double True/False; and, 5) Select All. The average number of correct answers per question declined in the same order that the question formats are listed. Student performance was highest for True/False (averaging ~80% correct) and lowest for Select All questions (~60% correct). Scores for all question types ranged from ~20% to 100%. In all cases, students’ average confidence for each question type was greatest for higher scoring questions. However, the average student confidence per question type showed little difference despite the significant gap in performance. Students became progressively more overconfident as the difficulty of the question format increased. Several low scoring questions in the “Select All” category were edited, often changing format to “Select X” where X represented the number of correct responses. These changes resulted in an average improvement in scores (~9%) but no change in student confidence for the modified questions in a subsequent iteration of the course.