Why shouldn't teachers teach to the test? If that seems problematic, then the test needs changing, not the teaching. In our new Age of Accountability, at least, testing is not going away soon (Shepard, 2010).
One challenge seems especially acute: testing beyond content. Climategate. Revised mammogram recommendations. Erroneous links between the measles vaccine and autism. Suspect communication with coma patients. An informed judgment in these cases requires understanding the nature of science (NOS), not just scientific concepts (Sacred Bovines, November, 2010). But how do we test for this type of understanding? In particular, how do we get past the multiple-choice format, with its inherent tendency to reduce knowledge to piecemeal facts? If a primary goal is to prepare scientifically literate citizens — able to participate in public and personal decision-making involving science — more sophisticated assessment formats are needed.
Nowadays, a teacher would seem to have plenty of options for assessing NOS understanding. Over 2 dozen instruments have appeared in the past half century. Choose your favorite acronym: VNOS, NOSS, NOST, COST, VOSTS, VASS, FAST, TOUS, TOES, TOSRA, WISP, at least (Lederman et al., 1998). Such tests carry risks, however. Most are based on a benchmark set of beliefs. Yet even experts disagree about many of the claims. The beliefs seem arbitrary, as critics have noted. Alternatively, one could adopt a more neutral, diagnostic posture, as other tests do. Here, the aim is merely to characterize student belief, or opinion. But this implies that there is no objective standard for measuring knowledge of NOS. The dilemma highlights a shared assumption — another sacred bovine? — that NOS is a set of beliefs and thus beyond fair, unbiased evaluation. Here, I profile an alternative perspective for evaluating NOS understanding (Allchin, 2011b).
A Well-Informed Analysis
In evolution education, at least, biology teachers have recognized that the goal is not to shape belief or attitudes. Rather, it is to inform. We aim to develop understanding of divergent lineages and natural selection, not beliefs in human nature. Just so for NOS. One aims to help students understand scientific practice. Not to indoctrinate them into a set of "scientific" values. Nor instill beliefs about the authority of Science writ large. Many NOS "tests" ask a student whether they agree or disagree with particular views about science. But personal belief is irrelevant. Rather, the student needs to develop a functional understanding of how science works in order to interpret the reliability of its claims.
Moreover, where the ultimate aim is interpreting science in society, NOS assessment should be concrete and case-based (Nott & Wellington, 1998; Phillips & Norris, 1999). Abstract generalities, such as "science is tentative" or "scientists are creative" — the focus of available tests — mean little without context. Rather, a student should be able to apply any relevant principle in a particular case. For example, consider the revised mammogram recommendations issued by a U.S. task force in late 2009: can a student compare the status of these specific claims to earlier ones? Are all the claims equally "tentative"? Are there good reasons for a change in consensus? Or has science been eclipsed by efforts to cut costs at the expense of women's health? Such an assessment complements an approach of teaching NOS through historical and contemporary case studies (Sacred Bovines, January 2011).
Cases in the news or advertising in mass media seem appropriate vehicles for assessment (Glynn & Muth, 1994; Norris & Phillips, 1994; Korpan et al., 1997; Murcia & Schibeci, 1999): MRI scans of smokers, causes of teen suicide, a new Climate Change Vulnerability Index. The understanding we seek in students is functional, not academic. Questions will thus ideally focus on analytical skills, not decontextualized concepts. What is the nature of investigation into the cause of the Gulf oil spill? Or of the science about long-term psychological effects of violent video games? Namely, can a student intrepret a case in current events, not merely echo abstract NOS principles rendered in other cases? Students must demonstrate thoughtful competence, not merely choose among preformed answers.
Working with cases in the news is not without potential pitfalls, however. It may seem simple enough to have each student adopt and "defend" a position on a particular issue. Yet assessing such views (students will be the first to note) is subject to teacher bias. A mitigating strategy for many teachers, I think, is to focus on the student's argument. One thus evaluates only how well they justify their judgments. Is the reasoning logical? Are the assumptions valid? Is the evidence sound? And so on. The teacher can thereby sidestep contentious judgment in grading.
Yet this approach (which I too once used) has a hidden cost. One assesses only an ability to assemble an argument. A well-formed argument does not necessarily indicate depth or breadth of NOS understanding. Worse, perhaps, one cannot differentiate between healthy reasoning and unfruitful rationalization. Do students reach their judgments based on the evidence, or do they compile the evidence based on preformed judgments? Cognitive science reminds us that our minds typically follow the second pattern. We tend to adopt a position first, often relying on emotion, then cherry-pick evidence to "justify" it (Sacred Bovines, August 2010). Effective education, however, should not foster the inherent tendency to short-circuit the more important lessons about thinking through evidence.
An effective alternative, then, seeks a well-informed analysis. That is, the student becomes responsible for interpreting the case fully and objectively for a friend or public official (who is to decide independently). Can the student recognize the NOS features and explain their relevance? Here, there is no position, no rationalizing. Instead, there is analysis. Notably, evidence in such a context must be complete. All the relevant information and perspectives are essential. All must be fully explained. Omissions matter. One can thereby measure breadth of functional NOS knowledge.
Another key feature of analysis is its degree of detail, or specificity. Are comparisons to similar cases coarse- or fine-grained? Are subtle but significant distinctions noticed? This measures depth of knowledge. Based on a student's analysis, one can thus evaluate — not merely assess — breadth and depth of NOS understanding. One thereby dissolves the dilemma of arbitariness and subjectivity, on the one hand, and lack of evaluative fulcrum, on the other.
Consider now a prototype question, based on the mammogram case (Figure 1). The student is asked to provide a well-informed analysis. For background and reference, there are several documents from the Internet (for links, see Allchin, 2010b). In this case, students are not asked to evaluate the clinical studies or evidence themselves. (Indeed, who among us would know enough to understand all its complexities? That was the reason for an expert panel.) Rather, the crux of the case is NOS. In addition, the student must defer their own judgment in informing an acquaintance. No "Well, I think that…" or "I recommend…." Hype or spin must yield to substantive and balanced analysis.
In scoring a student's written response, a teacher thus need not address the student's own view. Rather, the aim is to assess: how informed and complete is the analysis of the available information? Relevant (scorable) factors here include
• the role of systematic study versus anecdote
• the role of probability in inference
• sample size
• the nature of conceptual change
• the role of prior beliefs
• emotional bias in interpreting evidence and risk
• gender bias
• sources of funding
• credibility of sources
(see Sacred Bovines, November 2010; Allchin, 2010b). For each factor, the analysis may be ranked as simple or detailed, short or extended, minimal or elaborated. Perhaps add further credit for informative comparison to other cases (possibly encountered in class lessons). Optionally, one might also ask the student to identify important information that is missing — and where one would likely find such information. Here, one might say, is a sample test of scientific literacy, not unlike the familiar AP essay in format and scoring.
It's a sample for trial and discussion. Other prototypes, based on the cases noted in the introduction, are available online (Allchin, 2010b). Are these test questions we might proudly teach to?
Many state curricula, borrowing from national standards, now include nature of science as a learning objective or thematic thread. Clear explanations of what that means, however, are far less common. Even rarer are samples of how to teach NOS or how to evaluate student understanding of it. The prototype questions here provide a concrete, even if tentative, target. They reflect as plainly and as transparently as possible the ultimate aim of scientific literacy and the central role of understanding how science works (or doesn't work!). They also offer further context for identifying and articulating just what about NOS is important for every citizen and consumer to learn.
Science educators have been advocating some form of NOS learning for decades. At the same time, NOS has been virtually absent from conventional classroom practice and standardized tests. One may well imagine that progress has been thwarted by educational perspectives and practices so deeply entrenched as to pass unnoticed. As portrayed in this series of essays, effective NOS education seems to challenge numerous assumptions all at once: that content is utmost; that providing answers is more instructive than posing good problems; that contemporary cases are more relevant than historical ones; and that anything beyond content inevitably reduces to personal opinion and cannot be evaluated fairly. NOS remains an elusive, even if valued, teaching goal. Meditating on the reasons for the apparent deficits, one may find occasion yet again to imagine that some sacred bovines are, ultimately, quite profane.
DOUGLAS ALLCHIN has taught both high school and college biology and now teaches history and philosophy of science at the University of Minnesota, Minneapolis, MN 55455; e-mail: firstname.lastname@example.org.