Statistics Education in Undergraduate Psychology: A Survey of UK Curricula

Widely-used statistical practices may undermine the quality of scientific research (Falk & Greenbaum, 1995; Gigerenzer, 2004; Ioannidis, 2005; Szucs & Ioannidis, 2017). These include an over-reliance on null hypothesis significance testing (Falk & Greenbaum, 1995; Gigerenzer, 2004; Hubbard, 2019; Page & Satake, 2017; Sterne, 2001; Szucs & Ioannidis, 2017), incorrect interpretations of p-values (Haller & Krauss, 2002; Lecoutre et al., 2003; Lyu et al., 2020), publication bias in favour of positive findings (Franco et al., 2014; Scheel et al., 2020; Simmons et al., 2011), and inappropriate model or test selection (Thiese et al., 2015; Young & Karr, 2011). These practices can increase the prevalence of biased or misleading results and inaccurate conclusions, and lead to a distorted and uncertain evidence base (Ioannidis, 2005; Simmons et al., 2011).

Given the statistical shortcomings apparent in some of the published literature, several proposals have been made for how to improve the use of statistics in scientific research (e.g., Benjamin et al., 2018; Button et al., 2013; Chambers, 2013; Gigerenzer, 2004; Simmons et al., 2011). While some of these proposals have been well known for years, statistical issues in research persist (Hubbard, 2019). The incentive structure in academia and the criteria on which publishers accept manuscripts may be partially responsible (Ioannidis, 2014; Munafò et al., 2017). Some of the proposals to improve the proper use of statistics include: strengthening publishing guidelines (Friedrich et al., 2018; Thiese et al., 2015), in-principle acceptance of manuscripts before analyses are performed (Chambers, 2013), specialized statistical peer review (Hardwicke et al., 2019), and open and transparent research practices (Munafò et al., 2017; Simmons et al., 2011). A complementary avenue would be to improve the quality of statistical training that developing researchers receive (e.g., undergraduate students). Undergraduate training has a critical influence on the beliefs and practices of those who may continue into a career in research (Freeman et al., 2008). Moreover, university graduates from scientific disciplines are likely to use data skills throughout their career, regardless of whether they continue into research. Good statistical education may therefore lead to the effective use of statistics in published research and beyond.

Research describing the statistical methods taught in undergraduate science programmes remains sparse. The few relevant articles we identified focus on psychology programmes in the United States (Friedrich et al., 2000, 2018). These researchers surveyed psychology department chairs and found few changes in the content of their undergraduate statistics modules over the past 20 years. Another US survey suggests that most undergraduate psychology modules do not teach effect size or statistical power (Anglin & Edlund, 2020). The amount of time dedicated to statistics also appears disproportionately small in relation to the importance of statistics as stated in the learning outcomes of curricula (Homa et al., 2013). The emphasis that undergraduate psychology programmes place on null hypothesis significance testing appears out of step with modern statistical thought in the psychological sciences (e.g., estimation, uncertainty, and open science: Calin-jageman & Cumming, 2019; Friedrich et al., 2018).

We were unable to identify comparable research on statistical training of psychology undergraduates in the United Kingdom and aimed to fill this gap. We collected information from curricula rather than instructor surveys. Whereas instructor surveys provide detailed information, curricula describe what instructors intend to or are required to teach. Curricula also provide a target for improvements in statistics education by clearly outlining the planned teaching material and holding students and instructors accountable to the content. The data from these two methods can be triangulated to arrive at a more detailed understanding of statistics education in psychology. We therefore aim to provide a descriptive overview of the statistical concepts and techniques currently outlined in undergraduate psychology curricula in the UK.

In this paper, we use the term programme to denote the entirety of the education needed to earn an undergraduate degree in a particular field of study (often called a ‘course’ in the UK). A programme consists of multiple modules (often called ‘courses’ or ‘classes’ in the US). A module typically lasts one term and focuses on a specific subject. We use the term curricula to denote the sum of all module syllabi within a single psychology programme, not to denote a document separate from the module syllabi. For example, the sample in our study included 79 research methods syllabi, which made up 27 curricula. In other words, there were on average about 3 research methods syllabi (modules) per curricula (programme).

The preregistered protocol for this study is available at https://osf.io/q7xsu. We made minor amendments to the protocol which are all listed in Appendix A of the Supplementary Material.

We sampled from the 170 universities that the UK government considers ‘Recognised Bodies’ (UK government, 2020). Of these universities, 118 offer British Psychological Society (BPS) accredited undergraduate programmes in psychology (The British Psychological Society, 2020). We included full-time, in-person psychology programmes that lead to an undergraduate degree. We excluded part-time programmes, online programmes, and specialist programmes (e.g., sports psychology). Many universities offered multiple psychology programmes with different specializations and overlapping modules. We selected one programme per university, generally the one labelled “BSc Psychology”.

We separated our sample into research-intensive universities and other universities. We did so because research-intensive universities may systematically differ from other universities in the content they teach (e.g., they may provide more training in research methods and statistics to prepare students for a career in research). We defined research-intensive universities as those who are members of the Russell Group (i.e., a self-selected association of 24 universities that receive three-quarters of university research grant and contract income in the UK (The Russell Group of Universities, 2017)). We aimed to sample all 24 Russell Group universities and randomly sample an equivalent number of non-Russell Group universities.

For each programme we sampled, one team member (KD) recorded: (1) the number of modules that listed quantitative analysis as a learning outcome, (2) the number of research methods modules that did not include quantitative analysis as a learning outcome, and (3) the number of programming modules. Two team members (KD and RTT) screened all the quantitative module syllabi to check if they had sufficient detail to code for this study—which we defined as explicitly naming at least one topic from the “Statistical tools / techniques” section in Table 2 ¹ (inter-rater agreement was Cohen’s κ = 0.86). If the module syllabus was not available or contained insufficient detail, we requested a sufficiently detailed syllabus (see Appendix B of the protocol for the email template). If the module instructor was identifiable on the university website, we contacted them. If they were not identifiable, we contacted the programme organizer or administrator. If they were not identifiable, we contacted the director of undergraduate teaching and finally the faculty administration.

We included programmes only if all their quantitative module syllabi were of sufficient detail. To achieve our desired sample size, we contacted all Russell Group universities for whom at least one module syllabus was insufficiently detailed or not publicly available. We also contacted non-Russell Group universities with the same criteria until we reached an equivalent sample size between the Russell and non-Russell samples. However, the number of non-Russell Group universities in our sample was reduced to 12 after a second coder assessed that some of the module syllabi contained insufficient detail². In total, we contacted 34 programmes. Of these programmes, 16 had no publicly available module syllabi, 6 had publicly available module syllabi that were all of insufficient detail, and 12 had publicly available module syllabi where some, but not all, were of insufficient detail. We archived the module syllabi and indicated whether we coded them as sufficiently detailed at https://data.bris.ac.uk/datasets/1dfnj8ah83uru2hupk2jqu4jvx/StatsSyllabi_Data2ModuleSample_220710.csv. Our final sample included 15 Russell Group programmes, comprising 45 compulsory and 2 optional modules, and 15 non-Russell Group programmes comprising 31 compulsory and 1 optional module (Figure 1).

We coded whether each module syllabus mentioned each of 30 topics. We chose the topics based on their perceived relevance after reading similar studies (Anglin & Edlund, 2020; Friedrich et al., 2018), examining the literature on improving statistical practice in psychology, and discussing with instructors who teach statistics in psychology. We broadly organized these topics into either concepts (e.g., probability, Bayesian statistics), tools (e.g., correlation, ANOVA), or other skills (e.g., computer programming). Some of the topics may be taught in modules that do not include quantitative analysis as a learning outcome, and thus were not included in our sample (e.g., critical evaluation, replication, causality, reporting skills, psychometrics, and protocol writing). These findings are best interpreted in terms of whether the topic is specifically mentioned in a quantitative module syllabus, rather than on any syllabus within that programme.

All topics were coded independently by two of three coders (KD, RT, RC) and discrepancies were resolved through discussion until consensus was reached. Mean inter-rater agreement was Cohen’s κ = 0.70. The operationalization of one coding topic—applied statistics—was misinterpreted by one coder throughout, and four topics were very rare (Bayesian statistics, multiple comparisons, frequentist statistics identified, practical significance). If we remove these five topics, mean inter-rater agreement was Cohen’s κ = 0.77. The κ values and number of discrepancies for each coding topic are available in Supplementary Table 1. We also recorded the first-listed textbook, software used, reporting guidelines taught, percentage of grade given for exams, and whether an alternative teaching style was mentioned (e.g., flipped classroom). An open-ended textbox was included for additional remarks.

After coding each module individually, we combined the data across all modules within a programme. If at least one module syllabus mentioned a topic, then we coded the programme as mentioning that topic. We report the percentage of programmes that mention each topic and include 95% confidence intervals to estimate the prevalence of these topics in the finite population of programmes within the Russell Group (N=23) and non-Russell Group (N=95). Our sample mainly includes programme syllabi that were publicly available and of sufficient detail. This may lead to sampling bias in the confidence intervals because they attempt to extrapolate our results to the population of programmes that lacked publicly available syllabi of sufficient detail—a population that may differ systematically from the programmes we surveyed. This sampling bias may impact our estimates for Russell Group universities and other universities differently because more Russell Group University modules met our inclusion criteria.

In this manuscript, we remain largely agnostic about what the ideal research methods curriculum should include. However, we do hold the position that a basic understanding of data and uncertainty are necessary to effectively use inferential statistics. One could argue that undergraduate programmes are too short to include all the topics we assessed, and that many of them should be covered in graduate programmes. We maintain a descriptive—rather than prescriptive—stance, and discuss the results in relation to standards set by organizations such as the BPS.

Few module syllabi were publicly available and of sufficient detail in the non-Russell Group population. This led us to search all 95 non-Russell Group universities to match the sample size from the Russell Group. Twelve Russell Group universities and 10 non-Russell Group universities had publicly available curricula of sufficient detail for our coding procedure (Figure 1). We contacted 11 Russell Group universities and 23 non-Russell Group universities to ask for module syllabi. Three Russell Group Universities and two non-Russell Group universities which we emailed provided module syllabi. Individual university level data are available in https://data.bris.ac.uk/datasets/1dfnj8ah83uru2hupk2jqu4jvx/StatsSyllabi_Data1ProgrammeSample_220710.csv.

Russell Group universities had a mean of 3.13 (range: 2-5) quantitative modules per programme (Table 1). Of these 47 modules, 22 are taught in the first year of the programme, 20 in the second year, and 5 in the third year. Non-Russell Group universities had a mean of 2.67 (range: 2-5) quantitative modules per programme (Table 1). Of these 32 modules, 16 are taught in the first year of the programme, 15 in the second year, and 1 in the third year. The American Psychological Association (APA) guidelines were mentioned in 41% of curricula. All curricula except for one mentioned SPSS analysis software. Excel was the second most mentioned software. If we exclude optional modules, R was mentioned in only one curricula and Stata in none.

The prevalence of the topics we coded ranged from appearing in none of the curricula to appearing in all but one (Table 2). Most curricula mentioned specific tests (ANOVAs, regression, correlation, t-tests, frequency tests, and rank tests), about half mentioned effect size and statistical power, and few mentioned concepts such as confidence intervals, multiple comparisons, meta-analysis, replication, Bayesian statistics, frequentists statistics, and practical significance. When we excluded optional modules, a few topics that are already rarely mentioned—including causality, Bayesian statistics, meta-analysis, and programming—were mentioned even less often (Supplementary Table 2). Descriptively, Russell Group curricula mention several topics more frequently than non-Russell Group curricula (e.g., factor analysis, causality).

The most common first-listed textbook was Discovering Statistics using IBM SPSS Statistics (Field, 2013). It was the first-listed textbook in 7 of 47 Russell Group module syllabi and 10 of 32 non-Russell Group module syllabi. The next most common first-listed textbook was Introduction to Research Methods in Psychology (n=4/79) (Howitt & Cramer, 2006). Fourteen other textbooks were first-listed in one to three module syllabi (Supplementary Table 3).

Nine modules assigned grades based entirely on examination performance, 31 used a mix of exams and other grading schemes, 25 mention grading schemes but did not use the word exam, and 13 do not mention grading schemes (Supplementary Figure 1). We did not identify any module syllabus that highlighted teaching methods such as a flipped classroom or resequenced content. However, in the open-ended textbox of our coding form, coders noted that in some modules at least half of the hours were allocated to workshops, labs, or practicals.

Few UK undergraduate psychology programmes (19%) had publicly available syllabi that contained at least minimal detail (i.e., mentioning at least one statistical test or technique) for all their research methods modules. This finding was unexpected and arose only because we needed these documents to answer our main question regarding the statistical contents in these syllabi. This finding further suggests a lack of institutional support for sharing syllabi publicly, a lack of openness, or a lack of class planning. We also contacted 34 module instructors or programme administrators and only 5 provided additional information. Without detailed and public syllabi, it remains difficult to assess what instructors are teaching and where improvements can be made. It also leaves prospective students with a dearth of information about the programmes they are applying to and current students with little structure for self-study.

Of the curricula we could access, almost all 27 mentioned specific statistical tests. Many also mentioned critically evaluating published psychology research (81%). Specific statistical concepts that facilitate critical evaluation, however, were mentioned in fewer curricula: probability and randomness (63%), statistical power (52%), effect size (44%), treatment of outliers and missing data (30%), confidence intervals (26%), multiple comparisons (4%), and practical significance (4%). While programmes may teach these topics, their absence on curricula suggests they may not receive substantial coverage. We do not expect undergraduate programmes to teach all the topics we coded. However, almost all curricula mentioned inferential statistics and point-and-click software—such as the proprietary software package SPSS, even when free and open-source alternatives exist (e.g., JASP, R). Having the ability to perform inferential tests without an appreciation of the importance of statistical power, effect sizes, multiple comparisons, confidence intervals, and practical significance can drive a false sense of understanding. It can encourage the inappropriate use and interpretation of statistics (cf. Makin & Jean-Jacques, 2019).

Our results are broadly similar to studies that surveyed psychology instructors in the US about the topics they teach, including coverage of probability and randomness, confidence intervals, effect size, statistical power (Aiken et al., 2008; Anglin & Edlund, 2020; Friedrich et al., 2018; Sestir et al., 2021), and reproducible research practices (Anglin & Edlund, 2020). These US questionnaire studies may suffer from sampling bias (e.g., a 24%/33% response rate for the two samples in Friedrich et al., 2018). Meanwhile, our UK syllabus-based study obtains less detailed information. Nonetheless, the results are similar.

Comparing our results to the recommendations that psychological societies propose for undergraduate education could help establish whether a standard is being met; however, most of these societies offer few recommendations in terms of statistical education. The BPS serves as the exception and provides a 24-page guidance document for teaching research methods (The British Psychological Society, 2017). The document includes one paragraph that identifies specific quantitative concepts and tools that should be taught, including t-tests, ANOVAs, frequency tests, correlation, regression, and non-parametric tests (which most curricula mentioned), as well as confidence intervals and sample size selection / statistical power (which one third to half of curricula mentioned). The Quality Assurance Agency for Higher Education (QAA) in the UK—which determines subject benchmarks that universities are required to meet to receive public funds—has only a very general statistical requirement for undergraduate psychology programmes: to “demonstrate a systematic knowledge of a range of research paradigms, research methods and measurement techniques, including statistics and probability, and be aware of their limitations” (QAA, 2019). It is difficult to compare our results to broad-stroke guidelines such as these.

The APA guidelines for undergraduate psychology programmes hardly discuss research methods or statistics, but do mention effect size (American Psychological Association, 2013). A division of the APA—The Society for the Teaching of Psychology—assembled a Statistical Literacy Taskforce that established five learning goals for undergraduate psychology programmes (Addison et al., 2014). One goal was to distinguish statistical significance from practical significance, another goal was to evaluate the presentation of statistics, and the other three goals were broad (e.g., apply appropriate statistics) and thus difficult to compare with results from the present study. The Australian Psychology Accreditation Council (APAC) only mentions the term “research methods and statistics” in their accreditation standards document but contains no further details (Australian Psychology Accreditation Council, 2019). The Canadian Psychological Association does not provide guidance or accreditation for undergraduate psychology programmes. The sparsity of recommendations on research methods in the APA and APAC guidance appears disproportionate to the length of the document. Meanwhile, researchers and instructors continue to make calls to redesign statistical education in psychology programmes (e.g., Calin-jageman & Cumming, 2019; Kline, 2020; Morling & Calin-Jageman, 2020; Sestir et al., 2021). More detailed and clear-cut accreditation standards from psychological societies—developed in collaboration with instructors—could serve as a top-down method to improve statistical education.

While the present study generally achieved what it set out to do—assess the statistical content of UK undergraduate curricula—a few limitations are worth considering. First, our sample was biased towards programmes with available and sufficiently detailed syllabi. If this group systematically differs from the group without publicly available syllabi (e.g., they may be more likely to teach topics related to transparency), our inferences to the larger population of psychology programmes in the UK may be skewed. Second, we focused largely on what was taught as opposed to how it was taught. Simply teaching a topic does not guarantee that students will learn it; an effective method of instruction would be necessary. Gaining insights on how statistics is taught would require more detailed syllabi or interviews with instructors. Third, the UK syllabi we surveyed did not contain lesson-by-lesson breakdowns as some US syllabi do. This absence of detail may lead to an underestimation of topics that could receive only some attention, such as multiple comparisons or practical significance. Students may also learn such content in their final year research project, although the exact topics and level of instruction would likely vary widely among supervisors. Finally, correlation was mentioned in 96% of curricula and t-tests in 81%. If we assume that nearly every psychology programme teaches correlation and t-tests, these data suggest a margin of error in extending our findings from what is mentioned in syllabi to what is taught in class.

We aimed to assess the statistical content of undergraduate psychology curricula in the UK and found that few research methods modules had publicly available syllabi. Of the syllabi that were available, many did not mention key statistical tools and concepts that researchers and statisticians increasingly encourage the uptake of. Publicly available and detailed module syllabi would allow a clearer understanding of what is being taught in undergraduate psychology programmes and could serve as a conduit toward effective statistical education. Prospectively planned modules that include topics such as statistical power, effect size, confidence intervals, and reproducibility present one avenue that could improve how psychology undergraduates go on to consume and produce quantitative information.

There are no relevant competing interests.

Kyle Dack is supported by a PhD studentship from the MRC Integrative Epidemiology Unit at the University of Bristol (faculty matched place for MRC and Peter and Jean James Scholarship). Robert Thibault was supported by a postdoctoral fellowship from the Fonds de la recherche en santé du Québec and is now supported by a general support grant awarded to METRICS from the Laura and John Arnold Foundation and postdoctoral fellowship from the Canadian Institutes of Health Research (CIHR). Robbie Clark’s position is funded by the Wellcome Trust (214528/Z/18/Z). Jacqueline Thompson’s position was funded by Jisc (4956). Marcus Munafò and all other contributors are part of the MRC Integrative Epidemiology Unit (MC_UU_00011/7). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

An earlier draft of this manuscript was uploaded to PsyArXiv on 1 July 2020: psyarxiv.com/jv8x3/.

The preregistered protocol is available at https://osf.io/q7xsu.

Data, data dictionaries, codebooks, analysis script and materials related to this study are publicly available at the University of Bristol data repository, data.bris, at https://doi.org/10.5523/bris.1dfnj8ah83uru2hupk2jqu4jvx. To facilitate reproducibility, parts of this manuscript were written by interleaving regular prose and analysis code using R Markdown. The relevant files are available at the University of Bristol data repository, data.bris, at https://doi.org/10.5523/bris.1dfnj8ah83uru2hupk2jqu4jvx and in a Code Ocean container (https://doi.org/10.24433/CO.3003399.v1) which recreates the software environment in which the original analyses were performed. This container allows parts of this manuscript to be reproduced from the data and code with a single button press.

We thank Peter Allen, Katie Drax, Daniel Lakens, and Peder Isager for useful conversation regarding this project.