KNOWLEDGE SURVEYS: APPLICATIONS AND RESULTS

Knowledge surveys contain questions that reflect both the content, and the different kinds of thinking expected of students in a course. Surveys that cover an entire course may be 200 or more questions. Excerpted questions may be used to create mini-surveys that cover only one or a few topics. Scores on surveys or exams depend in large part on the specific items included. Exam scores are often of limited reliability because they typically consist of a small number of items. Knowledge surveys, containing many more items, have greater validity.

To guide studying, and to evaluate the validity of knowledge surveys, we had students take mini-surveys just before taking hour exams. Correlation between survey results and exam scores is moderate to high. Results from several exams, in several different courses, yield correlation coefficients of 0.54 to 0.91. The correlation for one class was only 0.54, but when data for one foreign student (who grossly underestimated his knowledge) are removed, the correlation improves to >0.85. Mini-surveys contain questions not on the actual exams. If we select only the questions that appeared on both surveys and exams, correlations improve to >0.80 in all cases. Student self-assessments are reliable predictors of exam performance.

For our classes, final grades depend on many things, traditional exams counting for <40%. At the beginning of a semester, students take an initial survey; at the end an identical final survey. There is no correlation between initial survey scores and final course grades. Final survey results correlate well with final course grades. Correlation, for five different classes, was 0.53, 0.72, 0.78, 0.79 and 0.88. These correlations are greater than correlation between final exam scores and final grades, suggesting that Knowledge Surveys are a superior way to measure learning.

We gave identical knowledge surveys to students who took Mineralogy between 2000 and 2005. Except for students who went on to work in mineralogy/petrology, retention of rote information and basic definitions declined rapidly and continuously after class completion. Analysis and application skills declined, but less precipitously. Ability to evaluate and synthesize information generally remained quite high. Course goals should be aligned toward developing higher order thinking skills because basic knowledge retention is, in large part, short lived.