USING CALIBRATED PEER REVIEW IN A LARGE GENERAL EDUCATION OCEANOGRAPHY CLASS

The calibrated peer review method (developed by UCLA chemists: http://cpr.molsci.ucla.edu/) was implemented in the EarthEd Online software system and used during a winter quarter UCSB oceanography class consisting of about 70 students. The EarthEd Online software (http://oceanography.geol.ucsb.edu) provides student access to earth data, assignments, and grades. The calibrated peer review method is a system where students write their paper and "hand it in" online. After the due date, they then review 3 papers that have been written by the instructor and scored according to a carefully developed rubric. The student scores the 3 instructor written papers and is prompted if his/her average score for each section of the paper is outside of preset limits. After the 3 "calibration" papers have been scored, the student scores 3 peer papers and last, their own paper, according to the same rubric. A student may request that the instructor or teaching assistant grade her/his paper, for any reason.

This method provides strong student motivation for evaluating example papers of various qualities, reading peer papers and noticing different writing styles, and evaluating their own work according to the guidelines set by the instructor.

The assignment results in 2 scores. The first is the score for the quality of the paper and is based on the 3 peer reviews, weighted by the reviewer's mean square error from the rubric item scores. Instructor assigned scores over-ride peer scores. The second score is for the "quality of the reviews." It is based on the student's rms error from the 3 calibration papers, the difference between the peer reviewed paper scores and its actual score, and the difference between the scoring of his/her own paper and its final score.

The report screens in EarthEd Online allow the instructor to quickly identify papers where the student's own score varies significantly from the peer assigned score. These are graded by the teaching assistants or instructor. For the 4 written assignments, 61% to 75% of the peer scores were within 90% of score assigned by the student to her/his own paper. 15 to 20% of the papers were scored by the teaching assistants, with most adjustments less than 20%. The final exam indicated increased student learning.