Development of a Concept Inventory on Open and Transparent Research Practices

In recent years psychology researchers have begun espousing practices that promote openness and transparency, including sharing data, providing reproducible analysis code, and pre-registering hypotheses and analysis plans. These practices have emerged in response to concerns over failures to replicate previous findings, conventional practices that undermine the robustness of published research, and outright fraud (Nelson et al., 2018). In concert with these widespread methodological shifts, academics have sought to integrate the lessons of the replication crisis and open science movement into undergraduate psychology education (Blincoe & Buchert, 2020; Chopik et al., 2018; Frank & Saxe, 2012; Sarafoglou et al., 2020). Still, the extent to which open science research practices are reflected in the teaching of research methods in psychology remains unclear (Jarke et al., 2022). Moreover, little is known about how introducing undergraduates to open science practices affects their conceptual understanding of quantitative research methods and their broader attitudes toward psychological research.

To bridge this gap, we developed and evaluated a targeted assessment instrument, the Open Science Concept Inventory (OSCI), which can be used to evaluate the impact of teaching open science practices in research methods courses. In what follows, we first present a brief overview of emerging open and transparent research practices, addressing some of the key concepts that can be incorporated in a quantitative research methods curriculum and are assessed by the OSCI. We review some of the pedagogical efforts to date that have sought to integrate open and transparent research practices in undergraduate courses. Then, we describe the steps we took to develop the OSCI to evaluate students’ conceptual understanding of these topics and to assess the impact of new materials on open science in a research methods course.

Psychology has undergone a transformation as a field over the past decade. A confluence of events around 2010–2012, including revelations of replication failures and data fraud, forced researchers to reflect on the best practices for collecting, analyzing, and reporting data (Nelson et al., 2018). Concerns during this “replication crisis” centered on a number of questionable research practices (QRPs), including: failing to report all conditions, measures, or experiments in a study (Simmons et al., 2011), “fishing expeditions” during statistical analysis (e.g., conducting multiple tests and reporting only those that reach statistical significance, referred to as p-hacking; Simonsohn et al., 2014), reporting unpredicted results as if they had been hypothesized in advance (i.e., “hypothesizing after the results are known” or HARKing; Kerr, 1998), and other practices that exploit researcher degrees of freedom (Wicherts et al., 2016). In contrast to cases of data fabrication or fraud, which appear to be relatively rare (Fanelli, 2009), QRPs had been culturally accepted practices (John et al., 2012) stemming from pervasive incentive structures in science that reward the publication of positive, confirmatory, or novel findings.

In response to the replication crisis and the QRPs fueling it (attested also in other fields, such as biomedical research; Ioannidis, 2005; criminology, Pridemore et al., 2018; ecology and evolutionary biology, Kelly, 2019), researchers started embracing new methodological practices that promote transparency and openness. These practices include voluntarily posting data, sharing analysis code, pre-registering hypotheses, and documenting methods and results fully through open science tools (Gernsbacher, 2018a; Klein et al., 2018).

These practices have also begun to make their way into the psychology undergraduate curriculum. Understanding the replication crisis and open science is especially important for students who will pursue further training in psychology and will benefit from instruction on the practical skills and conceptual foundations for conducting robust research. However, regardless of their eventual career path, this instruction is also essential for all students to make reasoned judgments about the credibility of psychological research and scientific evidence more broadly (Sarafoglou et al., 2020). Pownall and colleagues (2022) review recent approaches for incorporating open and transparent practices in courses and find benefits in terms of practical skills for conducting research and student engagement, as well as some evidence for improved attitudes toward psychological research. For example, in a study by Chopik and colleagues (2018), after a one-hour lecture that covered issues surrounding replicability and open science, undergraduates reported being more skeptical about findings from specific studies but expressed increased confidence in psychology as a field (e.g., considering it more similar to the natural sciences).

Other educators have shifted the focus of student projects to replications of published findings (Frank & Saxe, 2012; Jekel et al., 2020; Wagge et al., 2019). Advocates of this approach argue that didactic replications provide a rich pedagogical context for undergraduates to learn the basic tools of science, while also contributing independent replications of existing findings. More recently there have also been educational efforts focused on large-scale pre-registered replications and extensions of classic findings in judgment and decision making and social psychology (Collaborative Open-science Research, 2022). Beyond replications, preregistrations of projects pursuing novel research questions have also been integrated in undergraduate courses; for example, in capstone research courses (Blincoe & Buchert, 2020). These efforts can help the field by making replication mainstream (Gernsbacher, 2018b), demystifying the process of preregistration, and highlighting the benefits of openness and transparency for the next generation of science producers and consumers.

However, in most cases, these educational efforts have not been accompanied by systematic evaluation of whether they in fact improve students’ conceptual foundations for conducting research (Pownall et al., 2022; see also Linn et al., 2015). After targeted instruction, researchers do not typically examine directly learning gains about open science concepts. For example, although Chopik et al. (2018) found that students endorsed open research practices after their instructional intervention, it’s unclear whether students could reason about the motivating problems with QRPs or apply the concepts from the lecture to new situations. Systematic evaluations are needed to identify approaches for undergraduate research methods instruction that both train practical skills and foster the conceptual understanding necessary to apply those skills in other circumstances outside the classroom (Linn et al., 2015). Toward that end, we developed and evaluated a tool which we then use to systematically assess the effectiveness of a targeted curriculum on open science integrated in a research methods course.

At this juncture, we should acknowledge that the scope and level of detail of a targeted curriculum on open science is difficult to prescribe. Some of the “best practices” emerging from the open science movement have been the subject of debate, highlighting that methodological reform requires nuance and rigor (Devezer et al., 2021). For example, preregistration is typically described as a step in the research pipeline that reduces researchers’ degrees of freedom and improves diagnosticity (in discriminating a particular hypothesis from its alternatives) by distinguishing exploratory and confirmatory predictions. However, not all researchers agree about the appropriateness of preregistration. Some argue that preregistration does not, in fact, improve the diagnosticity of statistical tests, because statistical models typically approximate the underlying theory poorly (Szollosi et al., 2020). Others make the point that distinguishing between justified and arbitrary choices of hypotheses and analyses (i.e., between discovery-oriented and theory-testing research) is more useful than distinguishing between exploratory and confirmatory predictions (Oberauer & Lewandowsky, 2019). Other scholars argue that, although reproducibility is desirable, the relationship between reproducibility and scientific discovery is not well understood (Devezer et al., 2019, 2021). For example, Devezer and colleagues (2019) showed through simulations that the scientific process may not converge on the truth even when results are reproducible. Related perspectives frame the replication crisis as a crisis in theory rather than a crisis in methods. Many researchers argue that scientific reform requires a stronger grounding in theory development, specification of formal models, and transparency in metamethods (Guest & Martin, 2021; Muthukrishna & Henrich, 2019; Smaldino, 2017; Szollosi et al., 2020; van Rooij & Baggio, 2020). Finally, the current scientific reform movement has been criticized for being exclusionary of epistemic traditions rooted in qualitative research (Field et al., 2021). For example, linking replicability with scientific legitimacy and trustworthiness is problematic (Praat et al., 2020; Field et al., 2021), as it makes assumptions about the dominant methods in science (including psychology). It also ignores the means by which qualitative researchers achieve rigor, such as maintaining detailed metadata, having multiple coders (Lorenz & Holland, 2020), member checking, and engaging in reflexivity (Field et al., 2021).

In sum, there are multiple pathways to improve the robustness and rigor of scientific research. A key challenge for instructors of open science (and research methods more generally) is to avoid prescribing a set of one-size-fits-all procedures, and instead support students in learning to reason about the research process and to identify potential limitations and opportunities for bias. In the present work, we focus on a set of widely acknowledged factors that have contributed to the reproducibility crisis and which are appropriate for undergraduate courses on quantitative research methods. However, we recognize that the epistemic roots of psychology and other scientific disciplines are diverse and draw on other methodological traditions such as qualitative and mixed methods approaches.

In this preregistered set of studies,¹ we developed and evaluated the Open Science Concept Inventory (OSCI), which we then used to assess the effectiveness of a targeted curriculum on open science in an undergraduate psychology research methods course. To develop the OSCI, we first created an initial set of vignettes describing dilemmas faced by a researcher or other stakeholder. In Study 1, we administered these vignettes to participants recruited from a psychology subject pool and elicited open-ended responses about how researchers should respond to each scenario. By qualitatively coding participants’ elicited responses, we generated multiple choice options which included a “best practice” and common misconceptions for 40 retained vignettes. In Study 2, these multiple-choice items were presented to a new group of participants, whose responses were analyzed with Item Response Theory (IRT) to select the final set of items for the OSCI. Finally, Study 3 was a preliminary “proof of concept” for using the OSCI as an assessment tool: we used a pre-/post-test design across two semesters (Fall 2019 and Spring 2020) to examine performance on the OSCI before and after students completed open science modules in a psychology research methods lab course. We evaluated whether the integration of open science instruction improves students’ conceptual understanding of these topics. Following previous studies (Chopik et al., 2018), we also assessed whether students’ attitudes toward psychological research and reported self-efficacy would improve by the end of the course. We predicted that students would experience domain-relevant learning gains after completing the curriculum on open and transparent research practices, as assessed by the OSCI. Moreover, we predicted that, by the end of the course, students would report increased self-efficacy in research and improved attitudes toward psychology as a field.

Sixty-four students (38 identified as female, 26 as male; age M = 21.4 years, SD = 3.8) from UNC Charlotte in the United States completed the survey.² We did not collect information about the participants’ race and ethnicity. Participants were compensated with course credit or a $10 payment. Six students (9%) were psychology majors; a further 38 students (59%) were majoring in another STEM discipline. Forty-one students (64%) had previously taken a college-level statistics course and 14 students (22%) had previously taken a college-level research methods course.

Forty-one vignettes and open-ended prompts were created for Study 1 using the following procedure. The vignettes described hypothetical scenarios faced by a researcher, science consumer, or other stakeholder. For example, a scenario could describe a researcher engaging in a specific questionable research practice in order to increase their likelihood of obtaining a significant result during statistical analysis.

Each vignette was followed by a question prompting respondents to express their opinion about what that person should do in the situation or if they agreed with the researcher’s actions. Prompts were designed to have a similar logical and syntactic structure, following the format: “Do you agree with how X […]? Why or why not?”, or “Would you advise X to […]?. Why or why not?”.

The full set of items used to prompt open-ended responses in Study 1 can be found at our OSF repository for the project (https://osf.io/vq26u/).

First, we identified a list of target concepts related to open science and questionable research practices (e.g., publication bias, p-hacking, HARKing). We did so by identifying highly-cited recent papers (e.g., John et al., 2012; Simmons et al., 2011; Simonsohn et al., 2014) related to the replicability crisis, and identified questionable research practices and solutions advocated by the open science movement. Although at the time we did not refer to community-generated efforts in this domain, such as the glossary generated by the Framework for Open and Reproducible Research Training (FORRT, Parsons et al., 2022), we did generate a representative subset of those concepts.³

Next, we created 41 vignettes addressing these concepts. Of these, 9 items addressed primarily p-hacking (optional stopping, outlier exclusion, dropping participants or observations, covariate inclusion, outcome switching, selective reporting of conditions, outcomes, and experiments), 6 items addressed the incentive structures of science (including publication bias), 5 items addressed replication or computational reproducibility, 5 items addressed misconduct, fraud, or data fabrication, 4 items addressed preregistration, 4 items addressed the use of open data, 2 items addressed HARKing, and 6 items addressed other concepts related to transparency (interrater reliability, participant bias, salami slicing, data peaking). Appendix A shows the distribution of these concepts across items.

Because we aimed to evaluate the understanding of these concepts, we created vignettes that described scenarios faced by individuals from different perspectives (see an example in Table 1). Perspectives included researchers at different career stages and situations (e.g., honors thesis student, graduate student, early career researcher, established researcher, co-author), as well as science consumers and evaluators (e.g., employee of pharmaceutical company, conference paper reviewer, editor), and other stakeholders (e.g., funding agency). Because our goal was to develop an assessment of how people reason about the application of concepts related to open science in diverse contexts (not just their factual, definitional knowledge of these concepts), it was necessary to describe the scenario with some level of detail across a few sentences in order to support a preferred path of action.

To improve the readability of our vignettes, during this early design phase, we sought feedback on an initial set of vignettes (N = 14) from two experts in academic assessment from the Office of Assessment and Accreditation (OAA) at our institution. Through our discussions, we identified ways to reduce the cognitive load associated with reading the vignettes and to minimize ambiguity in their interpretation. Changes in our vignettes and survey involved the following: (a) presenting a set of definitions of key terms (e.g., hypothesis, sample size, statistical significance, pre-registration, peer review) at the beginning of the survey, in order to provide necessary background while offloading the verbiage in individual vignettes, and (b) ensuring that the prompts following the vignettes had a similar syntactic and logical structure (namely, “Do you agree with how X […] ? / Would you advise X to […] ? Why or why not?)”. Toward this end, in some cases, we split a scenario into two (or more) decision dilemmas faced by the researcher, with separate prompts corresponding to each decision point. We also obtained feedback from the experts from the OAA at a second time point, on the entire set of 41 vignettes. This second round of feedback resulted in small edits to improve clarity.

The study took place in a laboratory and the survey was administered through Qualtrics. Upon giving consent, participants provided demographic information (age and gender) and information about their academic background, including their major field of study, and whether they had completed a college-level statistics and research methods courses.

At the beginning of the survey, participants were presented with a set of definitions of key terms which appeared in multiple vignettes (e.g., hypothesis, sample size, statistical significance). As noted, this was done to ensure that participants, regardless of their course of study, had the necessary background on these key concepts while minimizing the use of definitions within the vignettes’ text. Participants had unlimited time to study these terms.

Next, participants responded to the open-ended prompts of the vignettes. Because initial piloting indicated that the full survey with the 41 items took up to two hours to complete, items were presented in one of two lists (of 20 and 21 items respectively) to shorten the duration of the study and reduce participants’ fatigue. Items across the two lists were matched to the extent possible in terms of the topics covered, prompt complexity, and number of multi-part questions represented (where different questions are associated with the same scenario). Appendix A lists the distribution of topics across the items of the two lists. To mitigate fatigue and other order effects, within each list, the presentation of the items was randomized with the constraint that multi-part questions were presented in sequence. An equal number of participants completed each list of items (32 participants per list).

After responding to the vignettes, participants were debriefed about the purpose of the study and were compensated for their participation. Sessions took approximately one hour to complete.

The results of Study 1 were qualitative. The final product was a set of multiple-choice options for each question item, based on participants’ open-ended responses. To generate these multiple choice questions, we used the following procedure: (1) Before viewing participants’ responses for a given vignette, we generated a target response for each item corresponding to the best course of action. This was done to establish whether at least a subset of the participants could produce the target response spontaneously. (2) We grouped participants’ open-ended responses under consistent themes. When two or more responses made the same point, we grouped them under that common theme. If a participant’s response addressed more than one theme, we cross-listed the response under all relevant groups, with the pertinent phrases of the response highlighted in each group. If a single responder made a distinctive well-reasoned point that could serve as a distractor option, we listed it as a separate theme. We grouped generic responses together (e.g., “because it’s biased” or “this may skew the results”), along with other miscellaneous or uncodable responses that did not fit into any of the emerging themes. (3) We summarized each theme in a statement capturing the common thread of the responses in that group (e.g., “I would advise David to not abandon the project because non-significant findings are informative and should be published” in response to the scenario in Table 1). Whenever possible, we used language similar to that used by participants. (4) Finally, we referred to the target response we generated in step 1 to identify the correct response statement among the summary statements for the groups of responses. If needed, we revised the wording of the target response to capture more accurately the conceptual distinction that we wanted students to make when reasoning about the vignette

Once we had generated summary statements for the themes of responses in each vignette, we selected and refined a set of four multiple-choice options to be used in Study 2 as follows: (1) We included the correct / best response. (2) We included response themes that were clearly incorrect (e.g., “David should not try to publish these results because replicating someone else’s work is unethical”). If necessary, we modified the wording of these statements, which was based on participants’ open-ended responses, to minimize ambiguity in their interpretation. (3) We excluded response themes that were broadly accurate but did not represent the target response for the prompt (e.g., “I would advise David to not abandon the project because additional replications are needed to establish if there is a true effect.”). The rationale was to select multiple-choice options that clearly distinguished between correct and incorrect responses. (4) If, as the result of this process, there were fewer than 3 response options that could be included as distractors, we generated additional options to yield a total of 3 distractors. (5) Where necessary, we made revisions to the vignettes. For some vignettes, step 3 revealed that participants produced responses that were broadly correct due to ambiguity in the scenario or in the prompt. In those cases, we slightly modified those vignettes to minimize ambiguity about the preferred course of action in that scenario.

Three coders (the two authors and a research assistant) independently completed the qualitative coding of different subsets of items, grouping participants’ open-ended responses into themes and generating a preliminary set of multiple-choice options. Coders then jointly discussed all coded items and selected the final set of multiple-choice options for Study 2. One item was excluded because coders could not identify sufficiently distinctive correct and incorrect options.⁴ This yielded a final set of 40 vignettes along with their multiple-choice options to be used for Study 2. Nine items were part of multi-item groups based on a common scenario (items 7–9, 10–11, 12–13, and 14–15).

The goal of Study 2 was to evaluate the multiple-choice version of the OSCI in a larger sample of undergraduates. The full set of 40 multiple-choice items were presented to a new group of participants. Responses were then analyzed with Item Response Theory (IRT) to select the final set of items for the OSCI.

N = 262 undergraduate students (149 identified as female, 113 as male) at UNC Charlotte in the United States participated for course credit or for a $10 payment. A further N = 6 individuals were excluded from analysis due to incomplete responses. The mean age was 20.46 years (SD = 4.28). We did not collect information about the participants’ race and ethnicity. In our preregistration, we had planned to collect data from 200 participants, based on a prior study that developed a concept inventory for research methods (N = 208, Veilleux & Chapman, 2017). However, because of participant availability, we ended up running an additional 62 participants.⁵

Thirty-seven students (14%) were psychology majors, while a further 101 students (39%) were majoring in another STEM field. 84 students (32%) had previously taken a college-level statistics course, 12 students (5%) had previously taken a research methods course but no statistics course, and 44 students (17%) had taken both a statistics course and a research methods course. The remaining 122 students (47%) had not taken either statistics or research methods courses at the college level.

We used the set of 40 vignettes and their multiple-choice response generated and selected through the process described in Study 1. This set of items can be found at our OSF repository for the project (https://osf.io/mxv59/).

As with Study 1, the study took place in a laboratory. The survey was administered using Qualtrics and was set up similarly to Study 1. Participants first provided demographic information (age, gender) and information about their major field of study, and their statistics and research methods experience. Next, participants were presented with the same review of “Key Concepts” as in Study 1. Participants had unlimited time to study these concepts and were not tested on them directly before proceeding to the vignettes. Participants then responded to the 40 vignettes associated with multiple-choice response options. Items were presented in random order, except that multi-part questions were presented in sequence. At the end of the study, participants were debriefed and compensated for their participation. Sessions took approximately one hour to complete.

Responses were treated as dichotomous (correct/incorrect) according to the best responses to each vignette identified in Study 1. Participants selected the correct response on 22.34 items on average (SD = 5.91), an average proportion of 56% correct (SD = 15%). The proportion of correct responses (see Table 2) ranged from 16–87%, indicating that the items covered a range of difficulty levels.

IRT analysis. We used Item Response Theory (IRT; Toland, 2014) to model responses to the OSCI questions. IRT models were fit using the LTM R Package (Rizopoulos, 2006). IRT models responses to multi-item assessments as a function of item difficulty and individuals’ latent ability in the domain. In our preregistration we planned to estimate a 3-parameter model (3PL) which includes item-specific parameters for discrimination, difficulty, and the likelihood of guessing. Discrimination (a) indicates the extent to which an item distinguishes between people of varying ability levels, captured by the slope of the item characteristic curve (ICC). Difficulty (b) indicates the ability level associated with a 50% chance of responding correctly. Given the large number of parameters in the 3PL model relative to our sample size, we departed from the preregistration by comparing it to simpler IRT models: the 2-parameter model (2PL) which includes item-specific parameters for discrimination and difficulty; and a 1-parameter (Rasch) model with item-specific parameters for difficulty and a single discrimination parameter for all items. The 2PL model provided a better fit of the data compared to the 1-parameter (Rasch) model (likelihood ratio test, $χ2$(39) = 246.9, p < .001), while the 3PL model did not significantly improve performance beyond the 2PL model (⁠$χ2$(40) = 40.4, p = .45).

Table 2 lists the results from fitting the 2-parameter (2PL) model for the initial 40 items evaluated for inclusion in the final OSCI. As per our preregistration, we excluded: 1 item due to a negative point biserial correlation (item 12), since this indicates that students who answered this item correctly did poorly on the overall test; another item was excluded because it appeared in the same question group (item 13); 2 items due to a significant $χ2$ test indicating poor fit to responses (items 11 and 32); and 3 items due to low or negative discrimination parameters (items 26, 27, 34). Excluding these 7 items led to a final set of 33 items.

Relationship to prior experience. Following our preregistration, we conducted planned exploratory analyses to examine whether performance on the refined set of 33 items of the OSCI was related to prior experience, as measured by previous enrollment in statistics and research methods courses (Figure 1). In separate models, regression was used to model the raw scores (proportion of correct responses) and the predicted ability levels based on the 2PL model described in the previous section, using only those items that were retained after exclusions. Since similar results were obtained for proportion correct and estimated ability, we only report results for proportion of correct responses for ease of interpretation (for the results on estimated ability see supplementary material on OSF: https://osf.io/yrdng/). A logistic regression model of the proportion of correct responses indicated a negative effect of having taken research methods and a significant interaction between having taken both statistics and research methods (Table 3). We used linear contrasts to compare the predicted accuracy for each combination of the statistics and research methods factors (using the emmeans R library, Lenth et al., 2019, with the Tukey correction for multiple comparisons). Compared to students who had taken both statistics and research methods, scores were lower among students who had taken neither (Odds ratio (OR) = .83, 95% CI [.71, .98], z = –3.01, p = .01), students who had only taken research methods (OR = .60 [.44, .80], z = –4.54, p < .001), and students who had only taken statistics (OR = .77 [.65, .92], z = –3.88, p < .001). Among those students who had not taken research methods, there was no difference based on whether they had taken a statistics course or not (OR = 1.07 [.94, 1.22], z = 1.36, p = .52). Surprisingly, among those who had not taken statistics, scores were higher for those who had not taken research methods compared to those who had (OR = 1.39 [1.06, 1.82], z = 3.12, p = .01). Although this implies a negative effect of research methods coursework on performance, it should be noted that this group (who had taken research methods but not statistics) included only a dozen students and this finding should be interpreted with caution.⁶

The goal of Study 2 was to test the multiple-choice OSCI in a large group of undergraduates and evaluate its psychometric properties using an item response theory analysis. Upon evaluating the IRT parameters of the original set of 40 items, we refined the OSCI to a final set of 33 items. In addition, we obtained initial evidence that performance on the concept inventory was related to students’ prior training, as scores on the final set were highest among students who had previously taken both college-level statistics and research methods courses. These differences based on prior coursework could be due to a number of factors. For example, in these courses, students gain greater familiarity with research- and statistics-related concepts that could help them understand and reason about the scenarios. Another possibility is that students experienced direct exposure to open science practices in the context of these courses. Self-selection bias could have also played a role: students who had taken these courses could have more aptitude to reason about science. Having obtained this initial evidence for the OSCI’s sensitivity to prior coursework, we set out to evaluate whether it could capture learning gains associated with the introduction of a targeted curriculum in a psychology research methods course.

Study 3 was a preliminary “proof of concept” for using the OSCI as an assessment tool. The final set of 33 items selected for the OSCI were used to evaluate student learning across two implementation rounds in a research methods course in Fall 2019 and Spring 2020. Learning gains were evaluated using a longitudinal design with two timepoints: a pre-test prior to the introduction of the target curriculum in the course and a post-test at the end of the semester. Along with the OSCI, an additional questionnaire was administered to assess changes in students’ self-efficacy and broader attitudes toward psychological research.

Forty-seven undergraduate students majoring in Psychology enrolled in a research methods course at UNC Charlotte in the United States participated for extra credit for that course. Participants came from four different sections of this course taught across two semesters (Fall 2019 and Spring 2020). The first and second author each taught one of these sections per semester. The potential sample size was limited by the enrollment cap of the course (25 students per section). Of the 47 participants who enrolled in the study, 37 completed both the pre-test and post-test (19 in Fall 2019, 18 in Spring 2020).⁷ In the final sample there were 29 students identifying as female and 8 as male; their mean age was 21.59 years (SD = 4.15). We did not collect information about the participants’ race and ethnicity. Given the relatively small sample size we obtained, we deviated from our original preregistration (where we had proposed analyzing data from two semesters separately), and pooled the data of students from two semesters into one sample for analysis.

The entire set of 40 items of Study 2 was administered to students; however, the focus of Study 3 is on the retained 33 items of the OSCI selected in Study 2.⁸

The new curriculum on open and transparent research practices included three recorded video lectures.⁹ The student learning objectives of this curriculum were aligned with the targeted development of the OSCI. The objectives were that students could recognize the problems with the pervasive incentive structures in science, distinguish between research misconduct and QRPs, recognize the effects of common QRPs on robustness of scientific literature, identify open science solutions to address these problems, think critically about research practices, and be able to situate open science guidelines within the scientific ecosystem (including funding agencies and academic publishers). The video lectures introduced the credibility crisis in Psychology (part 1a), questionable research practices (part 1b) and open science solutions (part 2). The topics covered in the three videos are described in more detail in Appendix B. The video lectures and lecture slides can be accessed at our OSF repository.

We developed a questionnaire on self-efficacy and attitudes toward psychology research by taking questions from Kardash (2000) and Chopik et al. (2018) (see Appendix D for the full set of questions). Our analyses focused on items that were related to three broad constructs: self-efficacy (10 items), interest in research (4 items), and trust in psychological research (5 items). Self-efficacy was measured by having students rate their perceived ability to complete certain research activities (e.g., generating a hypothesis, statistical analysis, etc.). Interest in research involved questions about students’ own motivation to pursue research-related careers. Trust in psychological research was measured through students’ degree of agreement with statements about the results of psychological studies. Responses to all items were made on a 5-point scale (from 1 = “Strongly disagree” to 5 = “Strongly agree”). We also included items from Chopik et al. (2018) which assessed overall attitudes toward psychology (items 1–5), but did not analyze them further since there was no clear connection between the open science curriculum and broad attitudes about psychology or its relationship to other fields.

To minimize the likelihood that students enrolled in the course felt coerced to participate, each instructor made the announcement about this study at the beginning of the semester as an extra credit opportunity in the other instructor’s class. Students were informed that their participation was optional and that their instructor would not be informed about whether they decided to participate until after the course was complete. Participants received 1% extra credit toward their course grade for completing the pre-test and an additional 2% for completing the post-test.

The pre-test was administered after the instructors completed an overview of statistics in the course, which included core concepts such as null hypothesis testing, Type I and Type II error, and statistical power. This ensured that participants had the necessary background on these key constructs. Students who volunteered to take part in the study were sent a link to a Qualtrics survey and had one week to complete it.

After launching the survey, participants provided informed consent and responded to demographic questions (age and gender). The format of the survey was the same as Study 2: participants were presented with a list of “Key Concepts”, followed by the 40 final items of the OSCI with multiple choice options. After responding to the OSCI items, participants completed the questionnaire for self-efficacy, interest, and attitudes toward research.

The open science module was available on the learning management system for the courses between weeks 7 – 15 of the semester. Students typically had a week to view each of the three videos and answer the questions of a low-stakes quiz embedded within the video. These quiz questions were inserted to ensure that the students paid attention; scores counted towards the students’ assignments grade for the course and was independent from extra credit for participating in Study 3.

After the open science curriculum had been completed, students were sent a link to the Qualtrics survey for the post-test. Participants had one week to complete the post-test, which was identical to the pre-test.

Accuracy. Proportion correct was comparable at the Pre-test (Fall 2019: M = .60, SD = .12; Spring 2020: M = .49, SD = .19) and the Post-test (Fall 2019: M = .61, SD = .24; Spring 2020: M = .50, SD = .26). For proportion of correct responses, there was no effect of timepoint (⁠$β$ = .03, SE = .04, 95% CI [–.05, .12], z = .75, p = .46), semester (⁠$β$ = .24, SE = .12, 95% CI [–.04, .51], z = 1.73, p = .08) or instructor (⁠$β$ = .05, SE = .14, 95% CI [–.23, .33], z = .37, p = .71). Similar results were obtained for predicted accuracy scores derived from the IRT model (see Supplementary Materials: https://osf.io/yrdng).

During the analysis we observed some surprisingly low task durations given the number of items to be completed, suggesting low-effort responses from some participants (minimum duration: 3.5 minutes). We therefore conducted a non-preregistered, exploratory analysis to examine whether the amount of time taken to complete the task was related to performance. Total task durations for both the Pre- and Post-test were aggregated and binned to form equal-sized quintiles. Figure 2 shows proportion correct across task duration quintiles for the Pre- and Post-test. There was a marked difference in performance among the lowest quintile compared to the other groups, with participants who completed the task in less than 20 minutes performing near chance levels at both the Pre- and Post-test (and performing better at Pre-test, in fact). Performance was higher among participants with longer task durations and appeared to increase from Pre-test to Post-test. This was confirmed in a separate logistic regression model after excluding any participants who fell in the lowest quintile for either the Pre- or Post-test (N = 12, 4 in Fall 2019, 8 in Spring 2020). This indicated an effect of timepoint (Table 4), with proportion correct lower in the Pre-test (Fall 2019: M = .62, SD = .08; Spring 2020: M = .60, SD = .17) than the Post-test (Fall 2019: M = .71, SD = .16; Spring 2020: M = .66, SD = .22). There were again no effects of semester or instructor for this subset of participants. The findings were similar for the students’ estimated ability, for which we also documented a significant effect of time point (see Appendix E).

Given that students who expended more effort when completing the OSCI did improve in their performance, we qualitatively examined item-based performance to identify areas of the curriculum that were particularly effective. The items of the OSCI that showed the greatest improvement in accuracy (≥ 20%) concerned the following topics: the value and informativeness of null findings (e.g., when reviewing papers, item 2; when deciding to continue data collection after piloting, item 33), the value of exact replications and not just novel extensions (item 37), the weighing of evidence from preregistered versus non-preregistered studies (item 3), the appropriateness of adhering to one’s original analysis plan (e.g., to exclude outliers, item 25), the importance of transparently reporting exploratory and confirmatory analyses (item 24), and recognizing instances of HARKing (item 8). Improvement on these items, collectively, could indicate that students recognized the value of replication, the distinction between confirmatory and exploratory analyses, and the importance of being transparent about any departures from analysis plans. Such improvements make sense given that these items map onto the central themes of the three video lectures: with the first lecture highlighting that the robustness of findings can be established through repeated replications, the second lecture highlighting problematic departures from initial plans (including p-hacking and HARKing), and the third lecture delineating pathways to more transparent research.

The shift from Pre-test to Post-test in the distribution of correct versus competitor options that students chose in many of the OSCI items also demonstrates these learning gains. For example, for the item about HARKing, at pre-test the most prevalent response (16/37 responses; 43%) was that it was acceptable for the researcher to reframe his original hypothesis (that liberal attitudes predict depression) after conducting an exploratory analysis with a subset of respondents (from non-coastal states), because “he reported clearly that his findings only concern non-coastal states”. At Post-test, only 19% of students (7/37) chose that response. The most prevalent response at post-test was the correct one, indicating recognition of HARKing (“I do not agree with how Dr. Smith presented the results because excluding coastal states was not part of his original analysis plan or hypothesis.” 14/37 students; 38%).

We also considered performance on those items that addressed practices associated with p-hacking, such as outcome switching (item 17), optional stopping (item 4), and selectively reporting experiments (item 21), conditions (item 20), and outcome measures (item 19). Students exhibited relatively high accuracy at Pre-test (ranging from 72-80%) and numerically similar performance at Post-test (ranging from 72-84%). Since students in this course had taken statistics and another research methods class as prerequisites, this prior background may have allowed them to reason their way through these scenarios. To achieve greater improvement on this set of concepts around p-hacking, it might be necessary to integrate the curriculum more closely with other elements of the course, including in-class discussions and labs on data analysis.

Trust, interest, and self-efficacy. Finally, we examined any changes in trust, interest, and self-efficacy from the pre-test to post-test. Average responses to all items on the attitudes questionnaire are shown in Appendix D. In accordance with our preregistration, we calculated mean scores for items related to trust in the robustness of psychology (items 6–10), interest in pursuing careers in research (items 11–14), and self-efficacy in engaging in research (items 15–24). Each outcome was modeled with mixed effects linear regression with timepoint (Pre/Post), semester (Fall 2019, Spring 2020), and instructor as fixed effects, with random intercepts for participants. Because the preceding analysis suggested low-effort participation among some students, here we report the results after excluding participants from the lowest task duration quintile. However, the results are not meaningfully different when these analyses are performed without any exclusions, in line with our preregistration (see Supplementary Material for results with the full sample).

There was a significant effect of timepoint on self-efficacy (⁠$β$ = .19, SE = .06, 95% CI [.07, .31], t(24) = 3.06, p = .005), with self-efficacy higher at the Post-test (M = 4.18, SD = .45) than the Pre-test (M = 3.80, SD = .57). Cronbach’s alpha for self-efficacy was $α$ = .86. Self-efficacy did not differ across semesters (⁠$β$ = .02, SE = .09, 95% CI [–.15, .19], t(22) = .19, p = .85) or instructors (⁠$β$ = –.06, SE = .09, 95% CI [–.23, .11], t(22) = –.70, p = .49). We did not find any significant differences in trust (M = 3.87, SD = .48; Cronbach’s $α$ = .64) or interest (M = 4.04, SD = .79; Cronbach’s $α$ = .73) across timepoints, semesters, or instructors (all p > .31; see Appendix E).

We predicted that the introduction of open science topics would improve students’ conceptual understanding of robust and reproducible research methods, as measured by the OSCI. While there were no overall differences in the full sample, we found evidence for improvement from Pre- to Post-test in an exploratory analysis that excluded students who appeared to exhibit low-effort participation based on their short task durations. The initial non-significant finding could be an artifact of low effort responders, who in fact performed better at the Pre-test than the Post-test. This apparent low-effort participation, observed in a considerable portion of the students (32% of the sample), should be contextualized in part by the COVID-19 pandemic: the majority of the excluded participants (8 out of 12) completed the study during Spring 2020, a period of major upheavals due to COVID-19, including a mid-semester move to online instruction. Although the transition to online instruction did not directly interfere with the study procedures since the lectures and surveys were delivered online, it may have contributed to disengagement among students.

Learning gains on individual items indicated broad improvements in performance on items that were related to the key themes of the lectures, including the importance of replication and transparency, as well as potential sources of bias that emerge at different points in the research process. These results add to previous findings that brief instructional interventions can improve understanding of the epistemic foundations of scientific evidence and systemic biases that can undermine the robustness of published research (e.g., Blincoe & Buchert, 2020; Chopik et al., 2018; Sarafoglou et al., 2020). In contrast, there was less evidence of learning gains for items related to p-hacking, particularly when the target concept involved analytic flexibility without obvious signs that the researcher was withholding information (e.g., optional stopping, covariate inclusion, data exclusion). This suggests the need for more in-depth discussion or hands-on activities to help students to recognize the effects of p-hacking and sources of flexibility in analysis procedures.

In addition to our predictions about improved performance on the OSCI, we had also predicted that students would have improved attitudes toward psychology as a field and increased self-efficacy about conducting research, as measured by the questionnaire on attitudes toward research. We found an improvement in self-efficacy but no changes in trust or interest in research. As shown in Appendix D, however, students’ responses on several of the items measuring trust (e.g., items 6-8) and interest (e.g., items 11-12) were already high at the Pre-test, suggesting that there wasn’t much room for growth on these measures among this sample of psychology majors. Nevertheless, this result echoes previous findings that knowledge gains in psychology research methods are not always accompanied by improved attitudes toward conducting research or perceptions of the utility of research skills (Ciarocco et al., 2013; Holmes & Beins, 2009; Sizemore & Lewandowski, 2009). Overall, students concluded the course with improved understanding of open and transparent research practices and higher self-efficacy about engaging in research.

We set out to develop and evaluate the Open Science Concept Inventory (OSCI), an instrument designed to assess conceptual understanding of open and transparent research practices. We first created a set of scenarios depicting research dilemmas and, in Study 1, elicited open-ended responses about how researchers should respond to each situation. These responses were distilled into multiple choice options that included students’ common misconceptions and a “best practice” tied to each concept. In Study 2, we evaluated our initial set of items in a sample of undergraduate students, using an IRT analysis to select 33 items that discriminated between respondents of different ability levels. In that study, the OSCI was sensitive to students’ past experience in relevant coursework, such that those who had taken both research methods and statistics outperformed the rest. This finding provided some initial evidence that reasoning about the open science scenarios on the OSCI relies on research- and statistics-related knowledge, though it is unclear to what extent those prior courses touched on issues specific to open science and the replication crisis. Study 3 was a preliminary proof of concept in which we used the OSCI to assess the effectiveness of a targeted curriculum on open science in an undergraduate research methods course, using a pre- / post-test design. Even though we did not observe the predicted changes in performance on the OSCI in our planned analysis with the full sample, there was evidence of learning gains after excluding participants who seemed to complete the study with low effort. Among the remaining students, the largest improvements were related to the central themes of the video lectures, suggesting that students were able to apply the concepts introduced in the materials to the scenarios on the OSCI. In addition to improved performance on the OSCI, these students also reported increased self-efficacy in research, while their interest and trust in psychological research did not change. Although the results of Study 3 should be treated with some caution given their exploratory nature, they provide tentative evidence that the OSCI can measure learning gains arising from a targeted curriculum.

Our findings contribute to pedagogical research in psychology by showing that instruction on open science practices can bolster students’ conceptual foundations for research methods, including the importance of replicability in evaluating evidence, the risks of questionable research practices, and the broader systemic biases that distort the scientific literature. We see this work as complementing recent efforts to align research methods instruction with the field’s best practices, such as reorienting student projects toward replication (Frank & Saxe, 2012; Wagge et al., 2019). These approaches have clear benefits in providing hands-on, practical training in open and transparent methods, as well as providing students opportunities to contribute to the scientific literature through replication and extension. Although these educational efforts are increasingly common, attempts to systematically assess students’ conceptual understanding of open and transparent research practices have been lacking (Pownall et al., 2022). Our results indicate that the OSCI can be a useful pedagogical tool for measuring students’ ability to reason about the nature of scientific evidence. Understanding how decisions by different parties can introduce biases at each step along the scientific workflow, and how these biases can be mitigated through open and transparent research practices, is a crucial skill for both future scientists and science consumers.

Integrating the lessons of the past decade into undergraduate education does not come without significant costs in terms of time and effort. We aim to facilitate this by making our materials and assessments openly available on OSF so that they may be easily adapted in other research methods courses. The video lectures and OSCI can stand alone or be integrated into existing curricula for research methods in psychology to support student learning objectives centered on critically evaluating scientific findings, identifying robust research methods, and conducting research using the field’s best practices.

We sought to include in the OSCI a broad range of concepts that we view as being central to the replication crisis and open science. Unlike more established components of the undergraduate psychology curriculum like frequentist statistics, however, shifts in research practices are ongoing and likely will continue to evolve, as seen in ongoing debates over the distinction between confirmatory and exploratory research (e.g., Devezer et al., 2019; Muthukrishna & Henrich, 2019; Oberauer & Lewandowsky, 2019) and the importance of preregistration (e.g., Szollosi et al., 2020). We recognize that any instrument assessing research practices should also evolve and be guided by consensus among researchers about best practices. Future development of the OSCI would benefit from a more systematic, community-driven approach, for example by leveraging a collaborative knowledge base like the FORRT glossary (Parsons et al., 2022) to identify concepts that warrant representation.

Moreover, while we aimed to create a collection of scenarios that were not specific to any one discipline, some of the concepts we selected may be more or less relevant to researchers in specific sub-fields or from different methodological traditions. Most of the focus of research on QRPs has been on quantitative methodologies. Some concepts may also be applicable to qualitative analyses (e.g., lack of transparency in reporting changes to an analysis approach), while other concerns (e.g., p-hacking, statistical power) are less relevant to qualitative work that is oriented toward hypothesis generation or detailed understanding of the specific participants under study (Reischer & Cowan, 2020). The OSCI might serve as a starting point or model for specialized concept inventories that are better-suited to specific communities of practice that lean to different degrees on quantitative, qualitative, and mixed methods. Our exclusion of some of the original OSCI items following IRT analyses may highlight the tension between reconciling practices across the qualitative and quantitative traditions. For example, responders had a hard time with an item that concerned whether it was appropriate to give corrective feedback to a research assistant during the process of establishing reliability for the qualitative coding of a behavior (identifying pointing gestures in video recordings). The poor fit of this item (item 12) may be due to the lack of consensus (among responders and the scientific community at large) about what the best practices are for training and establishing reliability among coders making qualitative judgments. To move towards a more pluralistic and inclusive open science, researchers must recognize that concepts, such as “reliability,” need to be grounded to the different methodological traditions and must develop assessment instruments that reflect the epistemic roots of these traditions.

Related to this point, although for many of the concepts in the OSCI it was straightforward to identify a “best practice,” some research dilemmas may have required a more nuanced understanding of the research process, or else more context than what was provided in the vignette. One example of this was “salami slicing” or dividing a project into “minimal publishable units.” On the one hand, salami slicing could contribute to a lack of transparency (e.g., if researchers fail to report measures or analyses related to a project that are published in a different paper), and, to a cynical reader, be a product of perverse incentives to maximize the yield of publications. On the other hand, researchers who work with large, complex datasets might pursue multiple research questions that are best addressed in different papers and where it may be impractical to provide exhaustive documentation. Journal editors have acknowledged the difficulty of distinguishing between the legitimate division of large datasets and inappropriate salami slicing (e.g., Werner, 2021). Other questionable research practices may be ethically ambiguous, in that their defensibility depends on the researcher’s intention (Sacco & Brown, 2019) or positionality in academia’s hierarchical structure. Some of the items that we excluded from the OSCI based on the IRT analysis concerned such arguably ambiguous cases, such as whether to share a developmental dataset that required a lot of effort and resources to collect (item 27), whether to pay an open access publishing fee to make a paper accessible to readers (item 26), and the described example of giving corrective feedback to establish inter-rater reliability (item 12). These excluded items and the low-performing items on the OSCI (e.g., on “salami-slicing”) underscore that, for some research dilemmas, readers might require more context to determine the “best” course of action.

A related limitation of the scope of this work is that we focused on undergraduates when developing the OSCI and our instructional materials, and as a result, focused on a core set of concepts that did not depend on advanced statistical training or research experience. Although the OSCI was developed using undergraduate participants, it may nonetheless be a useful pedagogical tool for psychology graduate students, both for assessing misconceptions and for prompting discussions about best practices. As we noted in the introduction, more nuance would be appropriate at more advanced levels of research methods training to convey the different perspectives about methodological reform, theory development, and scientific inference. A further question for future work is how statistical literacy contributes to performance on the OSCI. In our studies we used past coursework as a proxy for statistics knowledge (Study 2) and delivered the survey after students had reviewed key statistics concepts in their research methods course (Study 3). However, we did not independently assess their understanding of statistics concepts such as significance testing, type I and type II errors, and power, which might have figured in their reasoning about specific items on the OSCI. While our initial evaluation of the items in Study 2 assumed that participants’ ability varied on a single latent dimension, further work could use multidimensional IRT to explore whether items differ in their reliance on domain-specific knowledge about statistics or more general abilities related to reasoning or reading comprehension.

The implementation of the OSCI in our research methods courses in Study 3 had some additional limitations. First, because data collection was simultaneous in Studies 2 and 3, we were not able to test a more refined version of the instrument (based on the results of IRT in Study 2) in a larger confirmatory sample. Instead, students in Study 3 completed all 40 items but their performance was assessed using the retained set of 33 items of the final OSCI. Second, in our exploratory analysis of performance in Study 3, we excluded a sizable proportion of participants who appeared to give low effort responses, which reduced our small sample size further. The low effort of some of the students may be unsurprising in the context of the unfolding COVID pandemic during Spring 2020. A related reason for the low effort could be fatigue, whether due to the pandemic or the length of the instrument. Participants may have experienced mental fatigue seeing that the vignettes involved detailed descriptions and that it wasn’t always clear what the “best response” was. A reassuring note regarding future work is that, as we noted, the final inventory is shorter, involving 33 items.

The information we provide in this paper can be used to guide the process of shortening the inventory further for future use. The IRT results from Study 2 can guide the selection of items with high discriminability, while taking their difficulty into account. Additionally, the topic description of items (Appendix A) can be used to sample a range of concepts, seeing that in the current instrument some concepts (e.g., publication bias, preregistration, p-hacking) are addressed by two or more items.

Another direction in which the OSCI could be revised for future use concerns its cultural sensitivity. Some of the terms used in the vignettes (e.g., “zip code” in item 23 or “small liberal arts college” in item 27) are specific to the U.S. American cultural context, whose meaning and associations may not be known to readers outside the U.S. Given that institutional structures of higher education and linguistic terms vary across cultural contexts (even among English users), we recognize that the OSCI could be improved to be more culturally sensitive and have broader applicability. In our OSF repository, we include a version of the instrument in which we substitute some of these terms with more neutral or more broadly applicable variants (e.g., “postal code” for “zip code”; see: https://osf.io/8g62m). We encourage future users of the OSCI to review the vignettes for their applicability to the cultural context they’re being deployed and tailor them to that context, as necessary.

A methodological consideration of the current work is the lack of a control group in Study 3, which leaves some doubt over whether learning gains among the remaining students were specifically tied to the open science instructional materials. It’s possible that other aspects of the course or other changes among students during the course of the semester could have accounted for the observed results. It is unknown how this group would have fared compared to a control group of students that were enrolled in the same course but didn’t complete the open science module. Similarly, it’s unclear to what extent the new materials contributed to the improvements in self-efficacy beyond the more standard research methods coursework. Our findings serve as a proof of concept for using the OSCI as an assessment, but further work is necessary to establish the pedagogical benefits of our materials, preferably with the involvement of additional instructors, a larger number of students, and additional procedures to better identify low-effort participation such as attention checks. In future work, it would also be beneficial to incorporate positive controls of the instruction materials during the intervention, such as immediate tests on concepts to verify that students attended to the curriculum. Such positive controls would help disentangle the sensitivity of the OSCI from the effectiveness of the instruction material, and would allow for a more meaningful interpretation for any effects (or lack thereof) between pre-test and post-test detected by the OSCI.

Finally, we might also see greater learning gains with a closer integration between the new materials and other class activities. In Study 3, the new materials were delivered independently of other elements of the course. Video lectures were completed through online modules and were not discussed in the context of other class activities in order to create a consistent treatment across sections. This separation also extended to the hands-on lab activities: although the statistics labs were framed around the goal of analytic reproducibility (with students using open datasets to reproduce the results of published papers), there was little explicit connection to the goals or motivation of open science as covered in the video lectures. Standardizing the experience of students from different sections as much as possible, by not linking different elements of the course, served its purpose in our study. However, we expect that closer integration of the new materials with other elements of the course would have the greatest pedagogical benefits.

The field of psychology has seen a rapid transformation in our understanding of the best practices for conducting research over just a few years. Conceptual understanding of these open and transparent research practices – and the problems they are trying to address – will help psychology students reason about scientific evidence, whether as consumers or future producers of research. Toward that end, integrating open science practices into the curriculum and assessing the impact of these efforts is essential. With the OSCI, educators can evaluate conceptual understanding of topics surrounding open science. The OSCI holds promissory potential as it can be used in future research, along with the targeted curriculum we designed, to assess learning gains in this domain with larger samples of students, interventions that involve a more seamless integration between the new curriculum and other elements of the course, and in more advanced courses in which the field’s evolving best practices are presented with appropriate nuance. We hope that the open materials we share will contribute to scalable efforts to improve and evaluate students’ conceptual foundations for conducting robust and transparent research.

Contributed to conception and design: DBM, AG

Contributed to acquisition of data: DBM, AG

Contributed to analysis and interpretation of data: DBM, AG

Drafted and/or revised the article: DBM, AG

Approved the submitted version for publication: DBM, AG

We are thankful to Mitra Mostafavi for assistance with the development of vignettes, and to Marviene Fulton and Shaina Glass for assistance with data collection. We are also thankful to Drs. Elise Demeter and Karen Singer-Freeman from the Office of Assessment and Accreditation at UNC Charlotte who provided further input on the vignettes and the survey design. Finally, we thank Drs. Sarahanne Field, Katie Corker, Jordan Wagge, and another anonymous reviewer for suggestions that significantly improved this article.

This work was supported in part by funds provided by the University of North Carolina at Charlotte through a Scholarship of Teaching and Learning Program (SoTL) grant and a Scholarship of Assessment grant.

The authors have no conflicts of interest to declare.

All data, materials, and analysis files can be found at the project OSF page: https://osf.io/fejcn. We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the studies in this paper.

Our preregistration can be found at https://osf.io/qbtyg. Study 1 was not preregistered due to its qualitative, exploratory nature. Studies 2 and 3 were preregistered before examining any data. All departures from the preregistration are described the main text.

Table A.1 summarizes the topics associated with the 41 items used in Study 1 to elicit open-ended responses from participants. Items were matched according to topic, to the extent possible, across two lists (A and B). In Study 1, participants were randomly assigned to one of the two lists (A or B). On the basis of coding from Study 1, item 30 was excluded from Study 2, due to difficulties in generating contrasting correct and incorrect options.

For Study 2, the remaining 40 items were adapted to multiple choice questions. Participants responded to all 40 items. On the basis of IRT analysis, items 11, 12, 26, 27, 32, and 34 were excluded from the final version of the instrument; item 13 was also excluded due to being paired with the scenario of item 12. The retained 33 items were used for analyses in Study 3.

Video lecture 1 (approximately 21 mins). In the first video, we introduce the credibility and replicability crisis in psychology, contextualizing the cultural norms and incentives in science that fuel problematic practices, which distort the scientific record. We describe the scientific pipeline – from the identification of research questions, specification of hypothesis, designing the study, data collection, data analysis, writing, and publication – to illustrate the stages at which problematic practices can occur. We focus specifically on problematic issues at the writing and publication stage (publication bias and the file drawer effect), and on outright unethical cases of research misconduct (e.g., data fabrication) at the data collection and analysis stages. Finally, we introduce retraction as a process of correcting the scientific record.

Video lecture 2 (approximately 22 mins). In the second lecture, we start with the point that while cases of research misconduct get a lot of attention, they’re relatively rare. We introduce Questionable research practices (QRP), as conventional behaviors that undermine the robustness of research, linked to the credibility crisis. We describe how biases toward novel findings and significant results influences decisions made at the stages of specifying hypotheses, designing studies, collecting data, analyzing data, drawing conclusions, and reporting research. In the context of an example, we include a refresher of hypothesis testing, and the concepts of false positives and false negatives. We cover practices associated with p-hacking (including optional stopping, dropping observations, outcome switching, covariate inclusion), selective reporting, and hypothesizing after the results are known (HARKing).

Video lecture 3 (approximately 27 mins). In the final lecture, we present some solutions to QRPs and more egregious research practices. We introduce open science practices aimed to correct the systematic failings of scientific practices that promote accessibility, transparency, reproducibility, and robustness in research. We cover the practices of sharing datasets, analysis code, study materials, reporting methods and results transparently and completely, preregistering hypotheses and methods, and planning of sample sizes for research studies. We provide an example of preregistration of each step of the research pipeline: how researchers must specify prior to conducting the study their hypotheses, operational definitions, sampling plan, study design, and analysis plan. We address the distinction between confirmatory and exploratory analyses, and address misconceptions about preregistration. We present resources for implementing these practices (e.g., where to complete preregistrations, how to share data, code, and material). We also address how open science practices facilitate large-scale collaborations. Finally, we cover how these practices are reinforced by other stakeholders in science (including journals, funding agencies, and scientific societies) and how the incentives, requirements, and rewards for open and transparent research have changed (e.g., badges, registered reports, open access publications and preprints).