DO COURSE SYLLABI REFLECT OBSERVED TEACHING PRACTICES? USING REGRESSION ANALYSIS WITH K-FOLD CROSS VALIDATION TO BUILD A PREDICTIVE MODEL

The Reformed Teaching Observation Protocol (RTOP) was used to observe 345 undergraduate geosciences classes from a variety of institution types across the United States between 2009 and 2019. The RTOP provides a quantitative measure of the extent of reformed teaching practices used during a single class period, and is scored from 0-100 across five subscales. Starting in 2013, course syllabi were collected to accompany RTOP observations, leading to a data set of 155 unique courses with coupled observations and syllabi. The goal of this project was to develop a syllabus scoring rubric and build a statistical model that predicts RTOP score based on syllabus elements. The syllabus rubric was developed from literature-based best practices, previous syllabi evaluation studies, and elements of classroom observation protocols that align with information commonly found in syllabi. The final scoring rubric had four categories we hypothesized aligned with classroom practices: 1) Course Philosophy and Learning Outcomes (6 items); 2) Student Activities (6 items); 3) Student Support & Logistics (4 items); and 4) Assessment (1 item + course grade % breakdown). A process of iterative syllabi coding refined rubric items and established sufficient inter-rater reliability by three raters (0.95), ensuring that remaining syllabi could be reliably scored by a single rater. After scoring all syllabi, we found no significant correlation between total rubric score and RTOP score. To further assess the ability of specific syllabus rubric items to predict RTOP score, the full data set was randomly split into a training set for regression analysis and model building (80% of data), and a test set for model assessment (20% of data). We used k-fold cross validation with backwards stepwise regression to select the predictor variables included in the final regression model. Four syllabus criteria in the final regression model are most predictive of RTOP score: the number of high-level learning outcomes, presence of active learning elements, presence of metacognitive activities, and percentage of grade from exams. This work will be useful to instructors working to incorporate reformed teaching practices, who should be mindful that syllabi that accurately reflect course practices are an important form of communication with students.