UNINTENDED CUING IN TEST DESIGN: COLLEGE STUDENT DATA AND SPATIAL ANALYSIS OF EYE TRACKING

The design of a test itself may be just as important for performance as a student’s knowledge of the material being tested. In this study, a set of multiple choice questions containing common test designer idiosyncrasies was created to evaluate the extent to which design influences performance. Common idiosyncrasies included: longer answers, answers containing technical language, and answers that correspond to language in the stem of the question. Results from roughly 700 incoming college freshmen indicate that long or technical answers are chosen much more often than other answers. Eye tracking data from 15 students indicate that attention is unequal across all answer options; for example, participants spend much less time gazing at response option “d” than at other options. Differences in attention to salient features of long or technical answers provide evidence for mechanisms underlying the test taking observed in college freshmen. These results suggest that some students are using underlying signals to improve test performance. As a consequence, test scores are likely to be reflective of not only the student’s knowledge of the material, but their ability to pick up on cues in the questions themselves.