ASSESSING UNDERGRADUATE RESEARCH EXPERIENCES USING KNOWLEDGE SURVEYS

Undergraduate research (UR) experiences are widely valued for recruiting and developing a vigorous and diverse STEM workforce. Program assessments commonly employ participant surveys due to their ease of use in UR settings. However, such survey results frequently rely on generalized descriptions of learning tasks, ambiguous or unequal Likert scales, and self-reports of learning gains. Here, we describe an adaptation of the well-studied knowledge survey instrument for assessing UR. This instrument is easy to administer and adapt for different purposes and is ideally suited for assessing UR experiences.

Our knowledge survey addresses twenty-two knowledge and skill outcomes central to UR. Each item begins with the root “I can” followed by a detailed description of a research-related task (e.g., maintaining field notebooks, reading scientific articles, preparing poster presentations). Students respond to the items using four Likert-scale options that describe their ability to complete the task and their understanding of professional level knowledge. The format and structure of the knowledge survey offers the context of an exam that promotes more informed and intentional self-assessment.

We used the knowledge survey as a pre/post measure in two different research experiences for undergraduates (REU). Prior to UR, rising sophomores (Gateway REU) reported lower mean confidence (1.9 on scale of 1-4) to complete the tasks in the research knowledge and skills sub-construct compared with Advanced REU rising seniors (2.6). After the summer field experience, mean responses rose to 2.8 and 3.2, respectively, with both cohorts reporting significant gains (Cohen’s d = 1.70 and 1.50, respectively). After attending a professional conference, the mean responses increased again (x = 3.1 and 3.3, respectively); the effect sizes for those in the Gateway (Cohen’s d = 1.96) were higher than the Advanced REU (Cohen’s d = 1.64). Control groups, consisting of Year 1-2 and Year 3-4 geoscience majors, had mean responses (2.0 and 2.6, respectively; Cohen’s d = 1.29) similar to the pre-REU Gateway and Advanced cohorts, respectively. These results confirm the value of UR experiences, document the powerful impacts these can have on younger students, and demonstrate the promise of knowledge surveys for assessing undergraduate research.