Recommendations for undergraduate biology education include integration of research experiences into the curriculum, regardless of major. While non-biology majors and biology majors differ in affective characteristics, it is not clear if they differ in their incoming science process skills. We created a scenario-based assessment instrument – designed to gauge science process skills – that was accessible to nonmajors and majors. We evaluated nonmajors' and majors' open-ended responses using a rubric. We also assessed students' science identity, confidence, and attitudes with a pre-course survey. While affective differences between the populations are evident, we did not detect meaningful differences in science competency. These findings indicate that nonmajors and majors are skilled in the process of science and have the ability to engage in meaningful scientific inquiry, confirming our hypothesis that, in supporting a scientifically literate citizenry, educators must emphasize teaching strategies that target affective differences between nonmajors and majors.
Building on the foundational work of the “Science: A Process Approach” program (Livermore, 1964), recent recommendations for undergraduate biology education include the promotion of an inquiry-based curriculum, regardless of a student's major (AAAS, 2011). In response, science educators are incorporating research experiences into their courses that emphasize developing students' science process skills (SPS): hypothesis construction, experimental design, data interpretation, and communication (Coil et al., 2010; Luckie et al., 2013).
While recommendations for integrating research experiences into the curriculum are independent of major, notable differences between non-biology majors and biology majors (hereafter “nonmajors” and “majors”) may need consideration when designing course research experiences (Ballen et al., 2017). Nonmajors are less motivated by science (Knight & Smith, 2010; Glynn et al., 2011), less interested in it (Knight & Smith, 2010), less confident with respect to it (Cotner et al., 2017), and have less positive attitudes toward it (Sundberg et al., 1994; Cotner et al., 2017) than majors. While nonmajors and majors differ in affective characteristics, it is unclear whether nonmajors are less skilled in their ability to do science (e.g., developing testable questions and hypotheses, designing experiments, interpreting results, and communicating scientific information) than majors. Gauging differences in these core competencies will be beneficial to instructors designing curricula for these students.
Our SPS learning objectives for introductory biology students are as follows:
Ask a testable question.
Propose a testable hypothesis and prediction that aligns with the question.
Evaluate the strengths and weaknesses of an experiment or design a well-controlled experiment that aligns with the hypothesis.
Interpret data from tables and graphs.
Draw conclusions that are supported by the data.
Develop future research plans based on previous results.
Communicate research plans and findings in writing.
Use science in daily life.
In addition to being able to assess these learning objectives, we wanted an instrument that was accessible to nonmajors and majors with an open-ended question format. While each existing SPS assessment instrument (Shortlidge & Brownell, 2016) met some of our goals, no single instrument met them all. Thus, we developed a scenario-based SPS assessment instrument (hereafter “SPS scenario”) and rubric.
Our goal was to assess nonmajors' and majors' incoming science process skills. We evaluated students' responses from the SPS scenario using a rubric to meet this goal. We ensured that our sample populations of nonmajors and majors were representative of the larger populations at our institution in their affective characteristics (Cotner et al., 2017) by evaluating their responses on a pre-course science identity, confidence, and attitudes survey.
We define “majors” as undergraduate students enrolled in the College of Biological Sciences and “nonmajors” as undergraduate students enrolled in any other college at the University of Minnesota. The majors discussed here are incoming (first-year) students, and the nonmajors are students enrolled in an introductory biology course for nonmajors (described in Cotner & Gallup, 2011; Cotner et al., 2013; Cotner & Hebert, 2016). The data are students' SPS scenario or survey responses or institutional data. Our Institutional Review Board reviewed the research plans and granted an exempt status (Study 1405E50826). All participants gave informed consent to participate and were free to opt out of the study.
We developed the SPS scenario (available at https://cbs.umn.edu/cotner-lab/sps-scenario-whipworm) through an iterative process – during which the wording changed considerably – based on feedback from four biology experts. We began by generating our SPS learning objectives (see above), drawing from recommendations from national reports (AAAS, 2011), the biology education literature (Coil et al., 2010; Luckie et al., 2013), and discussions with biology faculty. While developing the instrument, we followed these guidelines: (1) the basis for the scenario should have valid research support (i.e., be peer-reviewed), (2) the topic should be relevant to nonscientists, (3) the language and content should be accessible to nonscientists (i.e., no jargon or specialized knowledge required), and (4) the instrument should follow a progressive format that displays a portion of the scenario and related questions at a time. The progressive format prevented students from modifying previous answers after receiving additional information. The SPS scenario consisted of open-ended questions that covered a range of skills. The corresponding learning objective(s) are listed at the end of each SPS scenario question (see https://cbs.umn.edu/cotner-lab/sps-scenario-whipworm).
The SPS scenario describes the use of parasites to mitigate the impact of inflammatory bowel disease (IBD). The scenario is based on the work of Weinstock and colleagues (Summers et al., 2003, 2005a, 2005b) and begins by explaining IBD, an autoimmune reaction, and the possible role of parasites in preventing immune system diseases. The SPS scenario continues with scientists hypothesizing that parasite infection could mitigate the impact of IBD. At this point, students design an experiment to test the scientists' hypothesis. Once the students' experiments are entered in the online instrument, the scientists' methods are provided and students list strengths and weaknesses of the scientists' experiment. Then the students interpret results from the scientists' data and generate follow-up questions, hypotheses, and predictions. Finally, they apply the scenario data to make a hypothetical decision about their own health.
The SPS scenario was validated with biology faculty and students. The scenario was developed and revised, in a year-long series of discussions, with four biology experts. The resulting version of the scenario was shared with faculty members and postdoctoral associates throughout the College of Biological Sciences for feedback. After incorporating this feedback, the instrument was validated through student interviews with nonmajors (n = 12) and majors (n = 12). Although some students' responses highlighted some scientific misconceptions, there was a clear understanding of the scenario's language and content. Therefore, no changes to the scenario assessment were made following student interviews. Students were compensated with an hourly wage or a gift card to the university bookstore.
Identity, Confidence & Attitudes Survey
The identity, confidence, and attitudes survey (Figure 1) consisted of published Likert-scale items that asked participants to report their science identity (Cole, 2012), science confidence (based on similar items in Seymour et al., 2004; Robnett et al., 2015), and attitudes toward science (Lopatto, 2004; Cole, 2012). The survey also asked if participants had taken advanced science classes (Cole, 2012).
We administered the SPS scenario and the identity, confidence, and attitudes survey before students took any biology classes at the university. Surveys were administered to incoming majors online one week before they attended a 2015 summer orientation program and were completed before they participated in that program. Surveys were distributed to nonmajors online one week before the start of the fall 2015 semester and were completed prior to the first day of class. The scenario and the survey took students ~30 minutes and ~10 minutes to complete, respectively. Nonmajors were awarded one point of assignment credit for completing the scenario and one point of extra credit for completing the survey. Completion of both instruments contributed two-thirds of one percentage point to an individual's final grade. Students could omit survey item(s) and still receive credit, provided they entered their ID. The structure of the majors' program prohibited awarding course points for completing the surveys. However, survey response rates were >90% for both nonmajors and majors.
Rubric Development & Scoring
We developed a rubric to collect data from students' responses to the open-ended SPS scenario questions (Figure 2). The rubric had a scaled design (Humphry & Heldsinger, 2014) and was aligned with our eight learning objectives (see above).
Because collecting data from open-ended responses is time intensive, we evaluated a random, small sample of nonmajors' (n = 30) and majors' (n = 25) SPS scenario responses. Two or three individuals (S.H. and upper-level undergraduate students) scored each de-identified student response using the rubric. The average percent agreement after individual coding was 93%, with the percent agreement for each item ranging from 81% to 100%. Student scorers were compensated with an hourly wage. Representative student responses and scoring examples are available at https://cbs.umn.edu/cotner-lab/rubric-scoring.
One-way ANOVAs were used to compare our sample population (n = 55) to the larger student population (n = 1450) that completed the survey, with respect to gender, ethnicity, and ACT score. Participant ethnicity was collapsed into two categories (white and nonwhite) due to low numbers of American Indian, Asian American, and African American participants.
Continuous data are reported as means ± standard deviation and were analyzed using a t-test. Categorical data are reported as counts and percentages (excluding missing responses) and were analyzed using a chi-square test. Statistical significance was defined as P < 0.05.
Samples Were Representative of the Larger Populations
The nonmajors (n = 30) and majors (n = 25) samples were similar in demographic and affective characteristics (P > 0.05) to the larger nonmajors and majors groups from the same semester described in Cotner et al. (2017).
Most participants were female (57% of nonmajors and 64% of majors) and white (90% of nonmajors and 72% of majors). The mean ACT score was significantly different (P < 0.001) between nonmajors and majors (26 ± 2.86 vs. 30 ± 2.20), a consistent distinction between these two groups. Significantly fewer nonmajors than majors reported taking advanced high school science classes (37% vs. 100%, P < 0.001).
Nonmajors Differ from Majors in Science Identity, Confidence & Attitudes Toward Science
Affective differences between the nonmajors and majors samples mirrored those observed by Cotner et al. (2017). Nonmajors were less likely to describe themselves as “a science person,” were less confident in their ability to “do” science, and had less interest in science than majors.
Nonmajors & Majors Demonstrated Science Competencies
When asking questions and proposing hypotheses, nonmajors and majors performed similarly. All nonmajors (100%) and majors (100%) asked testable questions and proposed testable hypotheses or predictions, and most nonmajors (97%) and majors (96%) proposed a hypothesis or prediction that was aligned with their question. Questions, hypotheses, or predictions were considered testable if they were about the natural world (i.e., data to test the hypothesis are available).
Nonmajors and majors performed similarly for most aspects of designing an experiment. All nonmajors' (100%) and most majors' (96%) experiments were aligned with the scientists' hypothesis. Most nonmajors (90%) and majors (96%) included a control group. Most nonmajors (97%) and majors (88%) included more than one subject per control or experimental group. Likewise, nonmajors performed as well as majors when identifying strengths and weaknesses of the scientists' experiment. For example, 17% of nonmajors and 16% of majors identified participant randomization as a strength, and 27% of nonmajors and 28% of majors recognized that using only patients' self-reported outcomes was a weakness. One difference between nonmajors and majors was the level of detail in their experiment, with nonmajors providing fewer details than majors (Figure 3). An experiment was categorized as “a detailed experiment” if it included at least seven of the eight detailed procedure items listed in Figure 2, “a few details missing” if it included four to six items, and “most details missing” if it included fewer than four items. None of the nonmajors (0%) provided a detailed experiment, while 16% of the majors did; 53% of nonmajors' experiments were missing a few details, compared to 80% of majors'; and 47% of nonmajors provided an experiment that was missing most details, compared to 4% of majors.
Nonmajors and majors interpreted results similarly. The scientists' interpretation of the results provided in the scenario was reasonable; 73% of nonmajors and 70% of majors agreed with the scientists' interpretation. The remaining nonmajors (27%) and majors (30%) either disagreed with the scientists' interpretation or did not definitively agree or disagree. Typically, the nonmajors and majors who disagreed with the scientists' claim did so because they didn't think the scientists' claim was valid given the small sample size of the study, not because they misinterpreted the data.
The other difference between nonmajors and majors was their use of data to make a hypothetical decision about their own health (Figure 4). Fewer nonmajors (30%) used data to support their decision than majors (48%). Among the nonmajors (37%) and majors (22%) who did not use data, responses ranged from a willingness to “try anything,” to taking an expert's advice (e.g., a doctor's or scientist's), to being disgusted by the idea of ingesting parasites. The remaining nonmajors (33%) and majors (30%) wanted additional studies performed or more information (e.g., about side effects or other risks); because we weren't sure whether they were using data or not, they were grouped into the “wanted more information” category.
Finally, a total SPS score was calculated for each student by adding the scores (top of rubric; Figure 2) for all nine SPS items. The maximum possible score was 20 and the minimum zero. Nonmajors' and majors' total SPS scores were not significantly different (17.03 ± 1.38 vs. 17.04 ± 3.03, P = 0.99). Excluding majors that did not complete all questions (n = 2), the average total SPS score of the majors was 17.87 (± 1.01). All nonmajors completed all questions. Comparing complete cases, nonmajors' total SPS scores (17.03 ± 1.38) were significantly different (P = 0.02) than majors' (17.87 ± 1.01); however, both groups scored high (>17/20), and the difference in their mean scores is less than one point.
Nonmajors and majors do not differ markedly in their abilities to “do” science, as evaluated by our scenario-based assessment. Nonmajors, like majors, can pose questions, analyze results, and draw conclusions based on data. The main difference between groups was the level of detail provided for the experimental design – a distinction that may be a proxy for differential motivation (Knight & Smith, 2010; Glynn et al., 2011) rather than competency. Nonmajors may have less patience for doing science during this exercise and may stop writing details before the majors do. Majors may be more motivated to appease or impress the survey distributors – staff and faculty, with whom the students will be engaged over the next several years. However, nonmajors receive a survey e-mail from their instructor, which may be motivating. Given that all majors reported taking advanced science courses compared to one-third of the nonmajors, differences in the level of experimental detail could also be due to majors being more comfortable with the language of science.
Interestingly, where the two populations differ is in their perceptions of their abilities to pose questions, analyze results, and draw conclusions based on data; nonmajors report less confidence in their abilities than majors. Nonmajors also report different perceptions of science and the utility of science than majors, but in general most students, in both groups, see science as relevant and important, a finding that complements those of studies by Miller (2004) and Allum et al. (2008) that investigated pro-science sentiments in the general public.
Our findings echo affective differences previously reported (Knight & Smith, 2010; Glynn et al., 2011; Cotner et al., 2017) and highlight a key message to science educators designing courses for nonmajors: Nonmajors have the skills to engage in the practice of science. Thus, nonmajors can probably be tasked with the same level of material and projects as majors. However, nonmajors may be less motivated, confident, and likely to see science as personally relevant. These affective differences likely suggest different approaches to teaching.
Several investigators have espoused using personal examples, in an applied context, when the goal is scientific literacy. For example, Forte and Guzdial (2005) found increased student interest and motivation after tailoring a nonmajors introductory computer science course to focus on programming to manipulate digital media – a skill students could immediately incorporate into their extracurricular interests. Similarly, transforming a nonmajors neuroscience course by using course materials at the intersection of the humanities and the sciences (e.g., books by Oliver Sacks and V. S. Ramachandran) led to increased student motivation to take additional science courses and, for some students, a decision to major in science (McFarlane & Richeimer, 2015). Another course, entitled “Issues in Biology” and taught in a context-based, inquiry-driven manner, shows promise for increasing positive perceptions of biology as a discipline; students saw biology as more relevant, worthwhile, and meaningful at the course's end than they had before it began (Gardner et al., 2016). Finally, the most popular lab activity in the nonmajors course described here studies human evolution by focusing on sperm competition in humans – an appealing topic by nature of being somewhat subversive and controversial (Cotner & Gallup, 2011).
We see a positive message in our findings – that nonmajors have the ability to do science, as recommended by AAAS (2011) and others (Nelson, 1999; Handelsman et al., 2007), a finding that reduces instructional barriers to implementing flipped classrooms (Herreid & Schiller, 2013), inquiry (Gormally et al., 2009), and course-based undergraduate research experiences (CUREs; Auchincloss et al., 2014) in courses that include nonmajors. The greater challenge is appealing to these students' motivation, interest, and confidence. Doing so, by learning from some of the examples cited above, may represent a shift in instructor perceptions, from strictly focusing on skills and content to considering students' affective characteristics. In conclusion, we encourage our peers in STEM to share successes in the domain of teaching to motivate, interest, and inspire confidence, with the goal of promoting a scientifically literate society.
We thank Melissa Kruesel, Christine Lian, Clay Mazur, Julia VandenBoom, and Anna Wojcicki for helpful comments on the rubric and assistance collecting data; Jessamina Blum, Catherine Kirkpatrick, and Robin Wright for helpful comments on the SPS scenario; and Jonathan Andicoechea, Cissy Ballen, Hillary Barron, Connor Neill, Lori Patrick, Jake Peterson, and Seth Thompson for helpful comments on the manuscript. This work was supported by a National Science Foundation grant (NSF 1432414) awarded to Sehoya Cotner and Catherine Kirkpatrick.