A MIXED-METHODS APPROACH TO ASSESSMENT DESIGN THAT INCORPORATES CULTURE AND PLACE

The validity of assessment implemented within Native American and other cultural minority populations can be improved by incorporating elements of culture and place. We propose a new methodology for effective assessment across cultural boundaries. Effective cross-cultural assessment requires: (1) engagement of community members from the target community in the actual assessment design, and (2) a variety of data collection approaches (sequential surveys, focus groups, questionnaires, and think-aloud interviews). We present an example from a collaboration with the Blackfeet and Navajo (Diné) Nations to revise items from the Geoscience Concept Inventory (a standardized instrument) and to identify additional geoscience concepts for new items. We collected data through one-on-one semi-structured interviews with Tribal college faculty, science educators and Native students employed a think-aloud strategy to gather information on select items of the GCI. Interview transcripts were analyzed to identify emergent themes and coded to determine theme frequency. Next, through a combination of sequential surveys and focus group sessions, cultural experts provided feedback on the role of assessment and identified important geoscience concepts within their local communities. Lastly, Tribal college faculty and Native students were surveyed to determine how they related to geoscience, Native science, place and culture. The open-ended responses and focus group transcripts were analyzed to identify emergent themes and used to guide the format, language, and content of the new geoscience assessment items. New assessment items, authored collectively by the Native expert groups, were piloted with students. The student responses were analyzed collectively by the expert groups to identify language and content for use in the design of selected response items. We will discuss the rationale for the selection of a variety of data collection tools as well as the data analysis techniques that incorporate individual researchers, research pairs, and expert groups. This work demonstrates how collective efforts are essential to the validation of assessment instruments that are valid across different cultures and places.