EVALUATING THE EFFECTIVENESS OF A MINORITY OUTREACH PROGRAM

The Rural Alaska Honors Institute (RAHI) is a summer bridging program that assists Alaska Natives in making the academic and social transition between high school and college. RAHI provides college-bound students from rural, predominantly Alaska Native, communities with full scholarships to attend a six week residential program at the University of Alaska Fairbanks. As part of the program, RAHI students can earn up to 11 college credits in a variety of different areas. Since 2003, RAHI has included a geoscience course, to both instruct RAHI students in the geosciences as well as to attract and retain these students into a geoscience career.

A multifaceted approach is being used to evaluate the impacts of this program on attracting students and retaining them in a geoscience career pipeline. One component involves the administration of surveys to participants prior to and following the program. The survey questions were specifically developed to address factors that indicate probable retention in a geoscience career pipeline. The questions were then cognitively tested in order to ensure that they were culturally appropriate for a rural Alaska population. Cognitive testing of survey questions is designed to ensure that a respondent: (1) interprets the questions as intended by the question writer, (2) has the information required to answer the question, (3) can use the information to come up with an accurate and valid response, and (4) can choose the response option that matches his or her answer. Cognitive testing is also used to develop survey items that perform equivalently in different languages and cultures (Levine et al., 2004)¹ .

This cognitive testing identified several questions that were not working as intended and could consequently give misleading results. For example, students' lack of familiarity with different science disciplines (such as physics, chemistry, and biology) required that these areas be briefly described. While these types of modifications to a survey appear minor, they may mean the difference between a meaningful evaluation instrument and one that is either uninformative or actually misleading.

¹ Levine, R., Gonzalez, R., Weidmer, B., & Gallagher, P. (2004). Cognitive Testing of English and Spanish Versions of Health Questionnaires. American Association of Public Opinion Researchers 59th Annual Conference, Phoenix, AZ.