There have been multiple national calls for curricular reform in science, technology, engineering, and mathematics (STEM), including a need to instill democratic skills in students. Democratic skill building can be embedded in STEM classrooms through intentional “deliberative pedagogies” that include communication, collaboration, and application of information. We developed and implemented a deliberative pedagogy, Deliberative Democracy (DD), for introductory majors and nonmajors undergraduate biology courses and took a longitudinal, qualitative research approach to understand students' experiences and perceptions of DD. We asked students to respond to open-ended survey questions regarding DD at two time points and conducted semi-structured follow-up interviews. All data were iteratively open-coded using content analysis. Students' perceptions of DD were lasting and generally positive, including self-reported themes related to DD promoting their awareness of the “real-world applications of science,” and increased “scientific literacy.” Negative perceptions of DD were largely related to issues with “group dynamics.” We detected differences between majors' and nonmajors' perceptions of DD, including in regard to scientific literacy and class time use. DD is a replicable pedagogy that can assist in instilling democratic skills in biology students.

Introduction

As educators, we aim to improve science, technology, engineering, and mathematics (STEM) courses to engage and retain students, yet it is important to remember that we are also educating members of society (National Research Council, 2003; President's Council of Advisors on Science and Technology, 2012). Our students will likely participate in civic activities, such as attending community meetings and voting – activities that will affect our collective future. In Educating for Deliberative Democracy, Thomas (2010) summarizes three main democratic skills that undergraduates should obtain: (1) effective communication, (2) effective collaboration, and (3) competent application of information. Further, Vision and Change (AAAS, 2011) prescribes that understanding the relationship between science and society is a core competency for students to gain before graduating with an undergraduate degree in biology.

One way of working to integrate democratic skills into the undergraduate curriculum is through deliberative pedagogies, which are relatively new to postsecondary education (Shaffer et al., 2017). To date, a small number of investigations on the impacts of deliberative pedagogies for non-STEM majors courses have emerged in the literature. For example, Jones et al. (2014) found that participants in a nanotechnology deliberation gained increased knowledge in the specific science content and appreciated the opportunity to have discussions with their peers. In another example, Drury (2015) documented a climate change deliberation in which the students reflected positively on the opportunity to work on their communication and critical thinking skills.

The deliberative pedagogy implemented in this study, “Deliberative Democracy” (DD), was first developed at Portland State University (PSU) for a nonmajors biology course in which the bulk of course content was structured around DD activities (Weasel & Finkel, 2016). Researchers reported that the modules increased students' learning gains and engagement in the class. The success of DD in this nonmajors biology course led to an expanded initiative to implement DD in majors biology, chemistry, and physics classrooms at PSU. In this new setting, DD was implemented to facilitate an engaging learning environment, and provide students the opportunity to gain important democratic skills to be carried with them beyond the classroom. We first aimed to gain a coarse-grain understanding of students' perceptions of the DD pedagogy in the majors biology and chemistry courses at PSU, finding that student perceptions were generally positive (e.g., seeing real-world applications of the pedagogy), yet many had negative experiences as well (e.g., difficult group dynamics) (Komperda et al., 2018; Shortlidge et al., 2019). Here, we took a finer-grain approach to detect evidence of students gaining the aforementioned democratic skills as a result of the DD experiences. To identify if DD interventions could have lasting impacts, we asked our students directly about their perceptions of DD over time. We took a qualitative approach to conduct a two-year study of majors and nonmajors introductory biology courses implementing DD at PSU (see the DD activities list in the Supplemental Material, available with the online version of this article).

This study is guided by the following research questions:

  1. What were the similarities and/or differences in students' perceptions of DD immediately after their course and a year later? How were these self-reported perceptions aligned with expected DD outcomes/democratic skills?

  2. How do these perceptions differ by course type (majors vs. nonmajors)?

Characteristics of DD Modules

DD is a small-group active-learning strategy that includes a deliberation exercise. Students are introduced to a real-world problem that correlates with their course content, and through reading, deliberation, and research they are asked to come to consensus on a policy recommendation. They are assigned related readings, quizzes, and group worksheets to build a consensus statement (Table 1).

Table 1.

Components of DD modules with expected outcomes, democratic skills, and examples.

Assignment/ActivityExpected OutcomeDemocratic SkillExample
Peer-reviewed readings, literature search, quizzes, exams Students will search for, read, understand, and assess peer-reviewed literature. Application of Information http://dx.doi.org/10.1016/j.envpol.2015.07.001 
Media readings, online searches, quizzes Students will search for, read, and assess non-peer-reviewed articles. Application of Information https://www.nytimes.com/interactive/2018/02/23/opinion/columnists/poisons-in-our-bodies.html 
Group work, worksheets, engage with instructor/LAs Students will practice communication skills. Communication https://www.pdx.edu/stem/deliberative-democracy 
Group work, worksheets, engage with instructor/LAs Students will collaborate with peers. Collaboration 
Group work, worksheets, individual research Students will be able to apply course content to the real world. Application of Information 
Assignment/ActivityExpected OutcomeDemocratic SkillExample
Peer-reviewed readings, literature search, quizzes, exams Students will search for, read, understand, and assess peer-reviewed literature. Application of Information http://dx.doi.org/10.1016/j.envpol.2015.07.001 
Media readings, online searches, quizzes Students will search for, read, and assess non-peer-reviewed articles. Application of Information https://www.nytimes.com/interactive/2018/02/23/opinion/columnists/poisons-in-our-bodies.html 
Group work, worksheets, engage with instructor/LAs Students will practice communication skills. Communication https://www.pdx.edu/stem/deliberative-democracy 
Group work, worksheets, engage with instructor/LAs Students will collaborate with peers. Collaboration 
Group work, worksheets, individual research Students will be able to apply course content to the real world. Application of Information 

Real-World Problems

A key component of DD is connecting the course content to real-world problems. For example, while teaching the endocrine system in the majors-level biology course, we introduced evidence that potential endocrine disruptors were ingredients in common household products such as cosmetics. We developed a DD module in which students were asked to determine “How should cosmetic products that contain potential endocrine-disrupting chemicals be regulated?” These questions are meant to be broad, with no clear-cut answers. As a deliberative pedagogy, we aim for the question to inherently evoke multiple levels of discussion that cross disciplines from basic to cutting-edge science, to socioeconomics, to environmental safety and justice.

Readings

Students are generally assigned two readings focusing on the topic: a media article (e.g., from the New York Times) and a peer-reviewed article or review article from a disciplinary journal. The two articles may be directly linked, such that the media article was written in direct response to the primary article and includes a link to that article; however, this is not required. The media article demonstrates how science is presented through a media lens, oftentimes revealing possible cascading effects to the general public. The peer-reviewed article demonstrates how scientific research is conducted and communicated in writing. Online quizzes were assigned to assess students' understanding of the readings.

In-Class Activities

DD is designed to be conducted by small groups (typically three to six students) and may be done as a one- or two-day module. Graduate teaching and undergraduate learning assistants (TAs and LAs) can be used to help facilitate small-group work in large classrooms (Otero et al., 2006; Talbot et al., 2015). Student roles can be utilized to assign tasks and encourage equitable conversation (e.g., we use roles of leader, recorder, spokesperson, facilitator, summarizer, and devil's advocate). Typically, on day 1 of a two-day module, students complete a worksheet outlining their collective knowledge gaps and assign topics for further, out-of-class, research (see Table 1 for weblink to worksheet examples). Groups are asked to come to an initial consensus statement addressing the problem before they convene. On the second day, students return with the materials they found, discuss them in their group, and complete a second worksheet. This sheet asks them to document their evidence and tie each topic back to the course material. The last section of the worksheet is a refined consensus statement where students can incorporate new information and design follow-up experiments to test unanswered questions. During the module, instructors have students report out their thoughts and findings in a both a whole-class discussion and between small groups. Instructors pace class time and address possible misconceptions through clicker questions when needed. For a one-day module, students are guided through a similar worksheet based on previously assigned readings and research performed in class, and they form a consensus statement at the end of the period. Expected outcomes and example components of a DD from a majors-level biology course on endocrine-disrupting chemicals and their potential impacts are outlined in Table 1. In this study, nonmajors and fall-term majors courses implemented one-day modules. Winter- and spring-term majors courses implemented two-day modules. Further, LAs were used in the majors courses during the entire academic year.

Methods

Collecting & Coding Student Perceptions of DD after Their Course

We used an online survey (Qualtrics) to collect students' perceptions at the end of the term from six introductory biology courses, three for majors and three for nonmajors (see the course syllabi in the Supplemental Material). For each course, we asked the open-ended prompt: “How do you feel about the Deliberative Democracy strategy used in this class?” Two researchers performed an inductive content analysis of the open-ended student responses (Krippendorff, 2012). We first sorted responses into positive, negative, and neutral bins, then applied codes to responses. Codes were then sorted into overarching themes. Neutral/non-informative responses (i.e., “it [DD] was fine”) were not included in further analyses. To assess reliability and objectivity in the coding of responses, we performed inter-rater reliability via Cohen's kappa until we reached a value of 0.61 or greater (substantial agreement) (Cohen, 1960). This study was approved by the PSU Institutional Review Board (nos. 153524 and 184471).

Collecting & Coding Student Perceptions of DD One Year Later

We recruited participant responses from the same cohort of students one year later through an online survey. The survey asked students which DD they remembered participating in, and then asked (1) “Reflecting on these activities [course-specific DD activities], what was the most memorable aspect and why?” and (2) “What was the most frustrating aspect and why?” We omitted any responses where students did not accurately choose the activities in which their class participated.

We used the same process as described in the previous section to code student responses into salient themes within the “memorable” and “frustrating” categories. We continued iterative coding analysis until we achieved kappa ≥0.61.

Student Interviews

To gather more nuanced perceptions of DD, we conducted semi-structured interviews (Cohen & Crabtree, 2006). These were completed after the second survey with a subset of participants (n = 19), who were recruited from the previous survey administrations. We asked questions such as “Did participating in DD influence you outside of the classroom in any way?” We used content analysis to categorize responses into salient themes. (Example interview quotes can be found in Table 4.) We continued analysis until we achieved kappa ≥0.61.

Statistical Analysis

Pearson's chi-square tests were used when comparing two groups. All reported significance was determined by P-values ≤0.05.

Results

Student Perceptions of DD Are Generally Positive

Overall, 470 students responded to the survey (n = 290 majors; n = 180 nonmajors; 66% response rate). Of those, a total of 82% (n = 385) across courses offered positive perceptions of DD immediately after their course. A smaller proportion of students (44%; n = 206) offered negative perceptions. A subset (25%; n = 117) had both positive and negative perceptions in their responses. Lastly, 5% (n = 24) had neutral perceptions, which were removed from the analysis. Table 2 reports the salient themes of positive perceptions, and how the themes align with expected democratic skill outcomes. Table 3 reports the salient themes of negative perceptions. Themes are expressed as percentage of students who mentioned that specific theme. Two themes of the positive and three of the negative perceptions did not align explicitly with any of the democratic skills outcomes.

Table 2.

Aggregated “positive” and “memorable” perceptions of DD themes and the applicable democratic skill that aligns with each theme, as reported by students. Significant differences in the proportion of responses that fell into a particular category between the two time points are indicated (* P ≤ 0.05; ** P ≤ 0.001). Students often had responses that fell into more than one theme.

ThemeApplicable Democratic SkillResponses after Course
(Positive)
(n = 446)
Responses One Year Later
(Memorable)
(n = 95)
DescriptorsExample Quotes
Working with Peers** Communication and Collaboration 17% 32% DD helped students collaborate, discuss, and get to know their peers. “It [DD] helps to get to know fellow students especially in a larger class setting.” (Nonmajors)
“These activities really opened the opportunity to meet other students with similar interests and mindsets.” (Majors) 
Real-World Applications** Application of Information 20% 30% DD showed the applications of science (i.e., current events, society/policy, and other real-world scenarios). “This is a great way to connect what we are learning in biology to psychology, politics, capitalism, ethics etc.” (Nonmajors)
“The DDs show the real-world applicability of what we are learning.” (Majors) 
Educational Application of Information 8% 12% DD was informative, and/or tied well to the course materials. “Reflecting on the citric acid cycle was one of the most memorable. I think the process of releasing energy was pretty fascinating and expanded my understanding of organic energy storage and usage.” (Nonmajors)
“This is definitely an excellent way to learn the material.” (Majors) 
Scientific Literacy* Application of Information 8% 15% Students reflect on readings/searching for articles. “The articles were great reading. I enjoyed reading different kinds of topics, even when they're not my first choice.” (Nonmajors)
“[DD] helped with being able to dissect scientific papers and have a deeper understanding of how to formulate questions.” (Majors) 
Valuable** N/A 11% N/A Students had an understanding of the purpose/value of DD. “I like them [DD] and think they are very useful.” (Nonmajors)
“I can understand how this could improve interest in biology.” (Majors) 
New Perspectives** Application of Information N/A 18% DD helped the student gain/observe a new perspective. “[DD] helped me to see perspectives or consequences that I had not thought of.” (Nonmajors)
“The professor asked us to bring our favorite cosmetic product, I went home and searched [and] found out most of them contain those chemicals. That discussion we had in class totally changed my perspective on cosmetic and hygiene products.” (Majors) 
ThemeApplicable Democratic SkillResponses after Course
(Positive)
(n = 446)
Responses One Year Later
(Memorable)
(n = 95)
DescriptorsExample Quotes
Working with Peers** Communication and Collaboration 17% 32% DD helped students collaborate, discuss, and get to know their peers. “It [DD] helps to get to know fellow students especially in a larger class setting.” (Nonmajors)
“These activities really opened the opportunity to meet other students with similar interests and mindsets.” (Majors) 
Real-World Applications** Application of Information 20% 30% DD showed the applications of science (i.e., current events, society/policy, and other real-world scenarios). “This is a great way to connect what we are learning in biology to psychology, politics, capitalism, ethics etc.” (Nonmajors)
“The DDs show the real-world applicability of what we are learning.” (Majors) 
Educational Application of Information 8% 12% DD was informative, and/or tied well to the course materials. “Reflecting on the citric acid cycle was one of the most memorable. I think the process of releasing energy was pretty fascinating and expanded my understanding of organic energy storage and usage.” (Nonmajors)
“This is definitely an excellent way to learn the material.” (Majors) 
Scientific Literacy* Application of Information 8% 15% Students reflect on readings/searching for articles. “The articles were great reading. I enjoyed reading different kinds of topics, even when they're not my first choice.” (Nonmajors)
“[DD] helped with being able to dissect scientific papers and have a deeper understanding of how to formulate questions.” (Majors) 
Valuable** N/A 11% N/A Students had an understanding of the purpose/value of DD. “I like them [DD] and think they are very useful.” (Nonmajors)
“I can understand how this could improve interest in biology.” (Majors) 
New Perspectives** Application of Information N/A 18% DD helped the student gain/observe a new perspective. “[DD] helped me to see perspectives or consequences that I had not thought of.” (Nonmajors)
“The professor asked us to bring our favorite cosmetic product, I went home and searched [and] found out most of them contain those chemicals. That discussion we had in class totally changed my perspective on cosmetic and hygiene products.” (Majors) 
Table 3.

Aggregated “negative” and “frustrating” perceptions of DD themes and the applicable democratic skill that aligns with each theme, as reported by students. Significant differences in the proportion of responses that fell into a particular category between the two time points are indicated (* P ≤ 0.05; ** P ≤ 0.001). Students often had responses that fell into more than one theme.

ThemeApplicable Democratic SkillResponses after Course
(Negative)
(n = 446)
Responses One Year Later
(Frustrating)
(n = 95)
DescriptorsExample Quotes
Class Time Use N/A 15% 19% DD was impinging on class time (i.e., less time for lectures/exam prep, took too long, too many DDs). “The in-class deliberative aspect was a waste of time.” (Nonmajors)
“Most frustrating was not being confident with the material that was going to be on the exams and having to go through a DD instead of the class slides.” (Majors) 
Challenges of Group Dynamics** Communication and Collaboration 13% 54% Group members were not prepared, distracted, not engaged, got credit regardless of participating. “The most frustrating aspect is when your group is quiet and there isn't really a discussion or argument about the subject. Sometimes they just want to complete the worksheet for points and be quiet for the discussion parts or group talk.” (Nonmajors)
“Some students tend to do most of the work and others just sit and talk. If the students put more time and interest into the DD, the class and the work would benefit the students more.” (Majors) 
DD Topic* N/A 8% 19% Students did not enjoy the DD topic and/or found the materials not relevant to the course. “Spending the time to read and discuss when they [DDs] weren't that interesting.” (Nonmajors)
“I think some of the topics are not interesting since there are less real-life applications.” (Majors) 
Implementation* N/A 8% 19% Students found the activity either rushed or too long, and/or the worksheet's format was not helpful. “Sometimes it [DD] felt like busy work.” (Nonmajors)
“I feel that the worksheets we filled out did not work out as well in reality, even though it seems like they would work hypothetically.” (Majors) 
ThemeApplicable Democratic SkillResponses after Course
(Negative)
(n = 446)
Responses One Year Later
(Frustrating)
(n = 95)
DescriptorsExample Quotes
Class Time Use N/A 15% 19% DD was impinging on class time (i.e., less time for lectures/exam prep, took too long, too many DDs). “The in-class deliberative aspect was a waste of time.” (Nonmajors)
“Most frustrating was not being confident with the material that was going to be on the exams and having to go through a DD instead of the class slides.” (Majors) 
Challenges of Group Dynamics** Communication and Collaboration 13% 54% Group members were not prepared, distracted, not engaged, got credit regardless of participating. “The most frustrating aspect is when your group is quiet and there isn't really a discussion or argument about the subject. Sometimes they just want to complete the worksheet for points and be quiet for the discussion parts or group talk.” (Nonmajors)
“Some students tend to do most of the work and others just sit and talk. If the students put more time and interest into the DD, the class and the work would benefit the students more.” (Majors) 
DD Topic* N/A 8% 19% Students did not enjoy the DD topic and/or found the materials not relevant to the course. “Spending the time to read and discuss when they [DDs] weren't that interesting.” (Nonmajors)
“I think some of the topics are not interesting since there are less real-life applications.” (Majors) 
Implementation* N/A 8% 19% Students found the activity either rushed or too long, and/or the worksheet's format was not helpful. “Sometimes it [DD] felt like busy work.” (Nonmajors)
“I feel that the worksheets we filled out did not work out as well in reality, even though it seems like they would work hypothetically.” (Majors) 

Students' Perceptions of DD a Year Later

A year later, 95 students responded to our survey (n = 63 majors, n = 32 nonmajors, 14% response rate). Of those, a total of 98% (n = 93) answered the question regarding what was memorable about DD, and 96% (n = 91) answered the prompt regarding what they remembered as frustrating about DD. In total, 79 (83%) of respondents answered both questions. We report the themes as explained previously. Table 2 describes the salient themes of memorable perceptions. A new theme arose (“New Perspectives”), while the theme “Valuable” was dropped during open coding. Table 3 describes the salient themes of frustrating perceptions. Two themes of the memorable and three of the frustrating perceptions did not align explicitly with any of the democratic skills outcomes.

Positive & Memorable Perceptions by Course Type

Pearson's chi-square tests indicated significant differences between the majors and nonmajors courses in three of six themes for positive (memorable) perceptions (Figure 1). Descriptions of each theme can be seen in Table 2.

Figure 1.

Changes in students' positive and memorable perceptions by course type: significant differences among responses by course, chi-square, Pearson correlation test of significance (* P ≤ 0.05; ** P ≤ 0.001). Note: many students made statements that fell into more than one theme.

Figure 1.

Changes in students' positive and memorable perceptions by course type: significant differences among responses by course, chi-square, Pearson correlation test of significance (* P ≤ 0.05; ** P ≤ 0.001). Note: many students made statements that fell into more than one theme.

Negative & Frustrating Perceptions by Course Type

Pearson's chi-square tests indicated significant differences between majors and nonmajors courses in three of four themes for negative (frustrating) perceptions (Figure 2). Descriptions of each theme can be seen in Table 3.

Figure 2.

Changes in students' negative and frustrating perceptions by course type: significant differences among responses by course, chi-square, Pearson correlation test of significance (** P ≤ 0.001). Note: many students made statements that fell into more than one theme.

Figure 2.

Changes in students' negative and frustrating perceptions by course type: significant differences among responses by course, chi-square, Pearson correlation test of significance (** P ≤ 0.001). Note: many students made statements that fell into more than one theme.

Student Interviews

We conducted semi-structured interviews with students (n = 19 total; n = 13 majors; n = 6 nonmajors). Themes and example quotes can be found in Table 4.

Table 4.

Student interview (n = 19) responses reflected the salient themes from the survey responses of perceptions of DD (refer to Tables 2 and 3 for theme descriptors).

ThemeApplicable Democratic SkillExample Quotes
Working with Peers Communication and Collaboration “It [DD] was interesting. The classes were really large, so a lot of people had different thoughts about it. And we all came from different backgrounds which was also interesting because it was a nonmajors course. So, we definitely didn't have the same way of thinking about things.” (Nonmajors)
“There was one group where we all started, none of us understood much of anything. So, when we started going through the questions and then going back through the article, we were all just back and forth – everyone going around – and then we all finally figured it out. And it was just fantastic.” (Majors) 
Challenges of Group Dynamics Communication and Collaboration “A lot of the time people not being accountable for their own work and the discussions sometimes falling flat because people weren't engaging well … [two people] didn't do the work and now this [worksheet] becomes more of a burden on two people in the group rather than four.” (Nonmajors)
“There's always going to be some people who aren't contributing as much, and that may be frustrating … contributing as in they're just not really trying very hard. They bring back sources that aren't good for your second day when you're trying to craft your policy statement.” (Majors) 
Scientific Literacy Application of Information “DD has helped because it has introduced you to understanding, even if you don't understand, specifically all the language used in the research article – I can kind of get the overall idea and can use the abstract to help me navigate. I can always look up terms, so I can at least, if not fully, understand it – the main idea is so great!” (Nonmajors)
“[DD] totally showed me how to do it [literature search] and I didn't even know the format of scientific papers – so for us as we go through these sections on “how you write it and how it is read,” it makes a lot more sense. Now I can understand.” (Majors) 
New Perspectives Application of Information; Communication and Collaboration “People would like to express opinions that I had never heard before, that I didn't share with them. And so just having a discussion with someone that I didn't agree with is exposure to just an idea – even if it's not something I agreed with – it was an exchange of information.” (Nonmajors)
“It [DD] was good. It was definitely eye-opening because we had a problem, had to try to come up with our own solutions – which is how science works – so it was like a very real-real example … made the class less like a class and more of like an experience.” (Nonmajors)
“I think by the end of the class period, there was a short moment of ‘well, I can see how this added value for the world as a whole going forward,’ now there's 200 people who have experienced this [DD] and clearly thought that fertilizer had no point other than to create algae blooms or something.” (Majors)
“That one [DD on endocrine-disrupting chemicals] definitely made me far more aware: what we actually consume and how that affects us.” (Majors) 
ThemeApplicable Democratic SkillExample Quotes
Working with Peers Communication and Collaboration “It [DD] was interesting. The classes were really large, so a lot of people had different thoughts about it. And we all came from different backgrounds which was also interesting because it was a nonmajors course. So, we definitely didn't have the same way of thinking about things.” (Nonmajors)
“There was one group where we all started, none of us understood much of anything. So, when we started going through the questions and then going back through the article, we were all just back and forth – everyone going around – and then we all finally figured it out. And it was just fantastic.” (Majors) 
Challenges of Group Dynamics Communication and Collaboration “A lot of the time people not being accountable for their own work and the discussions sometimes falling flat because people weren't engaging well … [two people] didn't do the work and now this [worksheet] becomes more of a burden on two people in the group rather than four.” (Nonmajors)
“There's always going to be some people who aren't contributing as much, and that may be frustrating … contributing as in they're just not really trying very hard. They bring back sources that aren't good for your second day when you're trying to craft your policy statement.” (Majors) 
Scientific Literacy Application of Information “DD has helped because it has introduced you to understanding, even if you don't understand, specifically all the language used in the research article – I can kind of get the overall idea and can use the abstract to help me navigate. I can always look up terms, so I can at least, if not fully, understand it – the main idea is so great!” (Nonmajors)
“[DD] totally showed me how to do it [literature search] and I didn't even know the format of scientific papers – so for us as we go through these sections on “how you write it and how it is read,” it makes a lot more sense. Now I can understand.” (Majors) 
New Perspectives Application of Information; Communication and Collaboration “People would like to express opinions that I had never heard before, that I didn't share with them. And so just having a discussion with someone that I didn't agree with is exposure to just an idea – even if it's not something I agreed with – it was an exchange of information.” (Nonmajors)
“It [DD] was good. It was definitely eye-opening because we had a problem, had to try to come up with our own solutions – which is how science works – so it was like a very real-real example … made the class less like a class and more of like an experience.” (Nonmajors)
“I think by the end of the class period, there was a short moment of ‘well, I can see how this added value for the world as a whole going forward,’ now there's 200 people who have experienced this [DD] and clearly thought that fertilizer had no point other than to create algae blooms or something.” (Majors)
“That one [DD on endocrine-disrupting chemicals] definitely made me far more aware: what we actually consume and how that affects us.” (Majors) 

Discussion

We examined how students perceived a deliberative pedagogy and if student perceptions of the pedagogy persisted and/or shifted over time. The intention of DD modules was to facilitate students working on important democratic skills, including communication, collaboration, and application of information (Thomas, 2010). We found that students were, without specific prompting, reflecting on the intended outcomes of DD after the course, both immediately and one year later. With these data, we can understand the most salient aspects of students' experiences in a DD activity. Moreover, students self-reported that they had gained a new awareness of – or new perspectives on – big, real-world problems.

Many students will continue on to careers and/or engage in civic activities that will require them to work and interact in a collaborative setting. Our results show that students working with their peers is a key aspect of DD. Many students reflected positively on gaining the opportunity to discuss and collaborate with other students with various perspectives (see Tables 2 and 4). Jones et al. (2014) had a similar finding about the appreciation for working with peers. We were not surprised to find that a top theme was the challenges of group dynamics, with many responses describing how other students did not contribute to group efforts (see Tables 3 and 4). Equal contribution of group members has been found to be an important aspect of group learning (i.e., Livingstone & Lynch, 2000; Chang & Brickman, 2018). Although this is a reported negative aspect of DD, it is not necessarily due to the modules, but because of group dynamics itself – which could be an unavoidable aspect in some group-learning settings, and important for instructors to consider, especially as it relates to students having equitable conversations (Tanner et al., 2003).

Most undergraduates have little experience reading and using scientific literature, although it may be a crucial skill for their future endeavors (Janick-Buckner, 1997; Rehorek & Dafoe, 2018). Both majors and nonmajors found that the article and literature-search components could be positive aspects of DD. For example, students spoke of reading and dissecting peer-reviewed literature (see Tables 2 and 4). Although, a year later, only the majors students continued to mention this, there is evidence that DD has created a positive association of the scientific literature for some of our students. We believe this outcome could be a product of the one-day versus two-day DD module design, as the majors course did the two-day modules during winter and spring terms. Encouragingly, DD can present an avenue by which to engage students with peer-reviewed literature.

Additionally, we found that DD is a platform for students to collaborate and apply the course content to the real world, even in a large classroom. Evidence of this is clear when students describe gaining an awareness and/or new or changed perspectives on science and society. We did not explicitly expect this outcome, but it is salient to some students as described here:

I always thought it was absolutely absurd not to get vaccinated. But looking at that, getting more perspective on why people don't, I still do vaccinate but being able to understand the other side instead of just shooting it down was definitely huge.

Implementing DD in biology can help students see how course content applies to their own lives, as well as gain an appreciation of other people's perspectives. We imagine that this outcome extends beyond the classroom and will affect their future life choices.

Suggestions for Instructor Implementation

We found that closely aligning the DD topic to the course content may lead to a more successful module. Our study shows that students care about how their time is being spent in the classroom, especially the majors biology students. If instructors align course content and include DD material on exams, students may have an increased appreciation and use for DD modules.

Challenges of group work dynamics may be difficult to remedy but are part of the real world. Intentionally addressing the importance of collaboration in the classroom, especially as students enter the workforce (World Economic Forum, 2016), may help increase student buy-in. Further, while assigning students roles may encourage equitable participation, support from instructors, TAs, or LAs is important for these roles to be utilized to their fullest (Talbot et al., 2015; Chang & Brickman, 2018).

Limitations of the Study

It is important to note that these data are self-reported and thus can be subject to both inflation and/or understating of students' actual experiences (Bowman, 2011); however, our findings show repeated salient themes in our study. We would like to also note that course-specific differences may be due to different implementation styles (e.g., one- versus two-day module types). We did not control for this aspect in our study. Further, these data may not be representative of biology students at other institutions.

Conclusion

We found that students' perceptions of DD were generally positive, with room for improvements in implementation. The perceived outcomes largely aligned well with the intended goals of deliberative pedagogy, as well as with national calls for students to understand the relationship between science and society. The study design allowed us to identify that there are long-term perceptions of DD that students retain over the course of a year. Particularly salient were students' reports of how working in groups can have memorable positive and negative effects. Using brief, policy-oriented DD modules in introductory biology presents a mechanism to encourage active learning and increase communication, collaboration, and application skills.

Acknowledgments

We thank the instructors and students who participated in this study. This research was supported in part by an award under the Howard Hughes Medical Institute Science Education Program (award no. 52008105 to G.P.S.).

References

References
AAAS
(
2011
).
Vision and Change in Undergraduate Biology Education: A Call to Action
.
Washington, DC
:
American Association for the Advancement of Science
.
Bowman
,
N.A.
(
2011
).
Validity of college self-reported gains at diverse institutions
.
Educational Researcher
,
40
(
1
),
22
24
.
Chang
,
Y.
&
Brickman
,
P.
(
2018
).
When group work doesn't work: insights from students
.
CBE–Life Sciences Education
,
17
(
3
).
Cohen
,
D.
&
Crabtree
,
B.
(
2006
).
Semi-structured interviews. Robert Wood Johnson Foundation, Qualitative Research Guidelines Project
. http://www.qualres.org/HomeSemi-3629.html (accessed
January
31
,
2019
).
Cohen
,
J.
(
1960
).
A coefficient of agreement for nominal scales
.
Educational and Psychological Measurement
,
20
,
37
46
.
Drury
,
S.A.M.
(
2015
).
Deliberation as communication instruction: a study of a climate change deliberation in an introductory biology course
.
Journal on Excellence in College Teaching
,
26
(
4
),
51
71
.
Janick-Buckner
,
D.
(
1997
).
Getting undergraduates to critically read and discuss primary literature
.
Journal of College Science Teaching
,
27
,
29
32
.
Jones
,
A.R.
,
Anderson
,
A.A.
,
Yeo
,
S.K.
,
Greenberg
,
A.E.
,
Brossard
,
D.
&
Moore
,
J.W.
(
2014
).
Using a deliberative exercise to foster public engagement in nanotechnology
.
Journal of Chemical Education
,
91
,
179
187
.
Komperda
,
R.
,
Barbera
,
J.
,
Shortlidge
,
E.E.
&
Shusterman
,
G.P.
(
2018
). Connecting chemistry to community with deliberative democracy. In
Citizens First! Democracy, Social Responsibility and Chemistry
(vol.
1297
, pp.
81
98
).
Washington, DC
:
American Chemical Society
.
Krippendorff
,
K.
(
2012
).
Content Analysis: An Introduction to Its Methodology
, 3rd ed.
Thousand Oaks, CA
:
Sage
.
Livingstone
,
D.
&
Lynch
,
K.
(
2000
).
Group project work and student-centered active learning: two different experiences
.
Studies in Higher Education
,
25
,
325
345
.
National Research Council
(
2003
).
BIO2010: Transforming Undergraduate Education for Future Research Biologists
.
Washington, DC
:
National Academies Press
.
Otero
,
V.
,
Finkelstein
,
N.
,
McCray
,
R.
&
Pollock
,
S.J.
(
2006
).
Who is responsible for preparing science teachers?
Science
,
313
,
445
446
.
President's Council of Advisors on Science and Technology
(
2012
). Engage to excel: producing one million additional college graduates with degrees in science, technology, engineering, and mathematics.
Washington, DC
:
U.S. Government Office of Science and Technology
.
Rehorek
,
S.J.
&
Dafoe
,
N.J.
(
2018
).
The art of referencing as an often overlooked aspect of scientific literacy: study of a classroom intervention
.
American Biology Teacher
,
80
,
423
428
.
Shaffer
,
T.
,
Longo
,
N.V.
,
Manosevitch
,
I.
&
Thomas
,
M.S.
(Eds.) (
2017
).
Deliberative Pedagogy: Teaching and Learning for Democratic Engagement
.
East Lansing
:
Michigan State University Press
.
Shortlidge
,
E.E.
,
Rain-Griffith
,
L.
,
Shelby
,
C.
,
Barbera
,
J.
&
Shusterman
,
G.P.
(
2019
).
Despite similar perceptions and attitudes, postbaccalaureate students outperform in introductory biology and chemistry courses
.
CBE–Life Sciences Education
,
18
(
1
).
Talbot
,
R.M.
,
Hartley
,
L.M.
,
Marzetta
,
K.
&
Wee
,
B.S.
(
2015
).
Transforming undergraduate science education with learning assistants: student satisfaction in large-enrollment courses
.
Journal of College Science Teaching
,
44
(
5
),
24
30
.
Tanner
,
K.
,
Chatman
,
L.S.
&
Allen
,
D.
(
2003
).
Approaches to cell biology teaching: cooperative learning in the science classroom – beyond students working in groups
.
Cell Biology Education
,
2
,
1
5
.
Thomas
,
N.L.
(
2010
).
Educating for Deliberative Democracy
.
San Francisco, CA
:
Jossey-Bass
.
Weasel
,
L.H.
&
Finkel
,
L.
(
2016
).
Deliberative pedagogy in a nonmajors biology course: active learning that promotes student engagement with science policy and research
.
Journal of College Science Teaching
,
45
(
4
).
World Economic Forum
(
2016
).
The future of jobs: employment, skills and workforce strategy for the fourth industrial revolution
. http://www3.weforum.org/docs/WEF_FOJ_Executive_Summary_Jobs.pdf (accessed
July
11
,
2019
).

Supplementary data