How Do Researchers in Psychology Perceive the Field? A Qualitative Exploration of Critiques and Defenses

The research reported in this paper was conducted under the IRB of the Office for Research Compliance at the University of Alabama (Protocol: 16-OR-273-ME-R2). The data for this study is coming from a larger research project funded by the National Science Foundation, Award #: 1728332 (See link).

Participants were practicing psychological researchers (i.e., current graduate students or individuals holding a graduate degree in psychology) who were recruited at psychology conferences and via email listservs as part of a larger project in which the goal was to investigate academic psychologists’ belief updating in light of new evidence (McDiarmid et al., 2021). The conferences were in the United States and Canada and the listservs have a predominantly North American membership (see Supplementary Materials for specific recruitment information). This larger study included data from two time-points: Phase I (November 2017 - March 2018) and Phase II (February 2019 - April 2019). Participants were compensated $10 for completing Phase I and $20 for completing Phase II. Here, we focus our analysis on three items from Phase II that assessed participants’ attitudes regarding the state of the field of psychology. Our sub-sample consisted of the 548 participants who responded to all three items at Phase II. See Table 1 for demographic information. The analyses conducted were exploratory in nature and thus do not support any confirmatory claims.

Two questions addressed participants’ perceptions of the state of the field. The first question asked participants to choose the most accurate statement regarding the rate of false positive findings: “The rate of false positive findings in the published psychology literature is acceptably low”(Choice 1),”The rate of false positive findings in the published psychology literature is somewhat higher than it should be, and we should try to lower it” (Choice 2), or “The rate of false positive findings in the published psychology literature is much higher than it should be and we should take major steps to lower it” (Choice 3).

The second question asked participants to choose the most accurate statement regarding research practices in psychology: “Our research practices were fine before the crisis, and we should continue employing more or less the same practices” (Choice 1), “Our research practices have been shown to be somewhat worse than we thought, and we should make some incremental changes” (Choice 2),”Our research practices have been shown to be extremely problematic, and we should make profound changes.” (Choice 3).

The third question asked participants to elaborate on their responses to the previous two questions in an open-ended fashion.

Our approach to coding aligns with recommendations for thematic coding (Braun & Clarke, 2006), and for content analysis (Hsieh & Shannon, 2005). We took a blended approach between the two to capitalize on the advantages of both approaches. Braun and Clarke (2006) write that the biggest advantage of thematic coding is its flexibility. Simultaneously, they note that this flexibility can make it difficult to develop guidelines for researchers, resulting in inconsistencies. Hsieh and Shannon (2005) write that the biggest advantage of content analysis is that it allows for rules based on the words used by participants. Meanwhile, this makes it slightly more rigid. In this blended approach, we believe we have been able to capitalize on the advantages of the approaches while mitigating their disadvantages.

Braun and Clarke (2006) lay out six phases of thematic analysis: 1) become familiar with the data; 2) generate initial codes; 3) search for themes among the codes; 4) review themes; 5) define and name themes; and 6) produce a report. These steps guided our approach, as follows. For phase 1, we sampled 100 initial responses from our data set and had three research assistants independently read (and re-read) the open-ended responses following recommendations (e.g., Hsieh & Shannon, 2005). These qualitative responses were presented in isolation from any other information, including the quantitative responses to the perceived state of the field questions.

Our research group then met with the three coders and began phase 2. In line with recommendations by Hsieh and Shannon (2005), coders shared their first impressions, and our research group used those impressions to form initial codes after several conversations. For phase 3, we followed the recommendation of using a tree diagram to arrange lower order codes and higher-order themes (e.g., Hsieh & Shannon, 2005). Two higher-order themes emerged: 1) criticisms of the field and 2) defenses of the field. The higher-order theme of criticisms included four sub-types: 1.1) statistics and/or research methods are misused and/or misunderstood, 1.2) problematic incentives in the field, 1.3) need for more transparency (e.g., open data, practices, pre-registration) and 1.4) other criticisms. The higher-order theme of defenses included five sub-types: 2.1) criticisms of replications (e.g., contextual differences), 2.2) criticisms that people are overreacting, 2.3) emphasizing psychology’s safeguards that protect the field (e.g., self-correcting nature of science), 2.4) criticisms of open science advocates and/or replicators, and 2.5) other defenses.

For phases 4 and 5, our research group discussed a sub-set of reponses in detail to clarify theme definitions and the coding process. A list of open science terminology was created for coders and frame of reference training was utilized to improve inter-rater reliability (Jackson et al., 2005). If a sub-theme was coded as present, the higher-order theme was automatically coded as present. The categories were not considered mutually exclusive, and some statements included both defenses and criticisms. As an example, one response said: “There are definitely issues in research practices used in the past that we should correct…definitely significant improvements can be made…”, and “However, I don’t believe these issues represent a ‘crisis’… most researchers are already doing many things well.” Some responses received no codes as they were not germane and/or were too ambiguous to be interpretable. For example, one response simply stated, “Replication studies are increasingly important.” In this case, it was not clear if the participant was criticizing or defending the field.

At this point, two researchers coded the entire sample of responses. Inter-rater reliability (IRR) was calculated in two ways. The first, more conservative approach, was to use the proportion of responses for which both coders said the theme was present over the total number of instances where at least one coder said the theme was present. This value was 79.3% for criticisms, and 47.4% for defenses. A more liberal approach was to use the proportion of responses for which both coders said the theme was present and/or absent over the total number of responses based off McAlister et al. (2017). This value was 81.8% for criticisms, and 85.4% for defenses. Disagreements in coding were resolved by a third coder. The prevalence of criticisms and defenses are reported in Table 2 and Table 3, respectively. See Table 4 for liberal and conservative IRRs for all sub-themes. The sixth and final phase of thematic analysis is producing the report which manifests within this article.

Twenty-three participants (4.2%) selected “the rate of false positive findings in the psychology literature is acceptably low,” 332 participants (60.6%) selected “the false positive rate is somewhat higher than it should be and there should be some attempt to lower it,” and 193 participants (35.2%) selected “the false positive rate is much higher than it should be and major steps should be taken.” See Table 5 for a comparison of the prevalence of the coded themes between the three possible responses to this question.

Eleven participants (2%) selected “research practices were fine before the crisis and the field should continue past practices as they have traditionally been implemented,” 329 participants (60%) selected ”past research practices are somewhat worse than people once thought, and that incremental changes are needed,” and 208 participants (38%) selected”past practices are extremely problematic and there is a need for profound change.” See Table 6 for a comparison of the prevalence of the coded themes between the three possible responses to this question. Those who give more critical responses to the quantitative items include more criticisms in their qualitative responses, providing some additional evidence of the construct validity of our coding method.

Of the 548 responses, 462 (84.3%) included a criticism of the field. The most common criticism was that statistics and/or research methods are misused and/or misunderstood. This theme was present in 218 (39.8%) responses and includes claims that researchers: inappropriately use statistical techniques, lack understanding of validity and/or reliability of measures, make improper claims from their data, and use questionable research practices (e.g., p-hacking, hypothesizing after the results are known (HARKing), etc.

The second most common criticism was problematic incentives within the field. This theme was present in 214 (39.1%) responses and includes claims that there are: issues within the hiring process (prioritizing high numbers of publications), biases for novelty and counterintuitive results in publication, a file-drawer problem that is obscuring a large number of null results, and a lack of incentives for conducting replication research.

The third most common criticism was a need for more transparency. This theme was present in 100 (18.2%) responses and includes recommendations that: data, syntax, and code should be open, methodology should be clear, statistical decisions should be disclosed and justified, and pre-registration and registered reports should be more common.

Lastly, there were responses that communicated some critique of the field that did not fall into the previous categories. These were coded as ‘other’ criticisms and were present in 80 (14.6%) of responses.

Of the 548 responses, 124 (22.6%) included a defense of the field. Tied as the most common sub-themes of defenses were criticisms of replications, and claims that people are overreacting in the field. These defenses were both present in 40 (3%) responses. The former includes claims that replications disregard context and fail to measure hidden moderators. The latter includes claims that replication failures are exaggerated and throwing away the credibility of the whole field is not appropriate.

The third most common defense of the field was that psychology has safeguards to protect the field. This theme was present in 13 (2.4%) responses and include claims that science

is naturally self-correcting process, statistical techniques (e.g., meta-analyses) aid in self-correction, and the detection of false-positives in our field is evidence of self-correction in action.

The fourth most common sub-theme of defenses were those that included criticisms of open science advocates and/or replicators. This theme was present in 10 (1.8%) responses and includes claims that replication advocates are spot-lighting replication failures thus hurting trust in the field, that there may be some political agenda to go after the original authors (e.g., ‘predatory nature’ of current replication efforts), and that replicators may be less competent scientists for not performing original work.

Lastly, there were responses that communicated a defense of the field that did not fall into the previous categories. These were coded as ‘other’ defenses and were present in (6.9%) responses.

With this project, we aimed to document academic psychologists’ perceptions on the state of the field, in addition to their justifications for those perceptions. The prevalence of criticisms, present in roughly 85% of responses, outweighed the prevalence of defenses, which were present in roughly 22% of responses. This suggests that our sample generally held critical views of the field and that the source of this criticism largely came from concerns that people (mis)understand/(mis)apply statistics and methodology and face problematic incentives. Among the responses that defended the field, no single defense seems to stand out, with none reaching prevalence levels over 10%.

These results should not be interpreted as the final word regarding the prevalence of psychological researchers’ criticisms and/or defenses of the field. Our sample provides the benefit of focusing on practicing psychological researchers from a range of sub-disciplines but also possesses real limitations. For people to take our survey, they would have had to attend a conference or have been included on a psychological society’s email listserv - both of which may be subject to selection effects (e.g., involvement in the field, financial resources, etc.). We were able to recruit participants from social/personality, developmental, cognitive, and clinical psychology, however, we lack representation among other subfields, such as industrial/organizational psychology. Thus, our results are likely not representative of the entire field of psychology.

It is worth noting that early career researchers are more represented than other groups (e.g., tenured faculty), which may impact the data presented here. Specifically, of the 548 participants, 53.2% identified as graduate students and 23.8% identified as professors. To get a sense of the representativeness of our sampled, we compared our sample to 2017 demographic statistics from the American Psychological Association (APA). In APA’s 2017 member profile report, 25.4% of members identified as early career researchers. This suggests that our sample overrepresented researchers earlier in their career while simultaneously underrepresenting more established academics.

A related concern is that early career scholars may be more critical and more likely to endorse systemic changes compared to established researchers, and thus our data may convey more negative perceptions than if a representative sample had been collected. To address this possibility, we examined the prevalence of qualitative and quantitative responses as a function of career stage (see Table 7 and Table 8). These data are not consistent with a trend where criticism is stronger at earlier career stages. Assistant professors had the highest rate of criticisms in their responses (87.0%), lecturers had the lowest, (75.0%), and graduate students fell in between 85.6%). Lecturers expressed the highest rate of support for “profound changes” (58.3%), full professors expressed the lowest (27.8%), and graduate students fell in between (38.0%).

Like our study, Baker’s (2016) survey addressed beliefs about the replicability of published scientific findings. Their sample differed from ours in two major ways. First, their sample was predominantly composed of researchers from biology, chemistry, earth sciences, medicine, physics, and engineering disciplines, while ours was composed exclusively of psychological researchers. Second, their survey was conducted in October through November of 2015, whereas ours was conducted in February through April of 2019. Nevertheless, considering both studies in tandem could give a more holistic picture of perceptions of reproducibility in the sciences.

In our study, 4.2% of participants indicated that “The rate of false positive findings in the published psychology literature is acceptably low” and 2% indicated that “Our research practices were fine before the crisis, and we should continue employing more or less the same practices.” By comparison, in Baker’s survey they found that 11.5% of participants disagreed with the statement “I think that the failure to reproduce scientific studies is a major problem for all fields” and 3% indicated that “there is no significant crisis of reproducibility.” Thus, in both studies, the proportion of surveyed scientists who don’t perceive a problem with replicability in science was quite low.

At the other end of the spectrum, 20.2% of Baker’s participants chose the most extreme position (strongly agree) with respect to the statement “I think that the failure to reproduce scientific studies is a major problem for all fields,” and 35.2% of our participants choose the most extreme position “The rate of false positive findings in the published literature is much higher than it should be, and we should take major steps to lower it.” Overall, there is a sizable group of participants in both studies who see serious problems with replicability in the sciences.

One challenge for interpreting our quantitative results is that some of our items are ‘double-barreled.’ For instance, participants may agree that “the rate of false positive findings in the published psychology literature is somewhat higher than it should be,” yet disagree with the conjunction “…and we should try to lower it”. The structure of the question could confuse participants and made it difficult for them to choose a numerical response that accurately captured their perceptions.

Relatedly, the first question asked about the ‘published psychology literature’ as a whole, whereas the second question asked about ‘our practices’. The former may have focused survey takers on the broad category of the field and the latter is ambiguous to the extent that it allows for personalized interpretation in a narrow sense (e.g., the participant’s subfield).

While recognizing the limitations of our descriptive work, we believe it has allowed us to learn more about how academic psychologists, especially early career psychologists, perceive the field. Recurring critiques are statistics and methodology are misused/misunderstood, the current incentive structure is problematic, and there is a greater need for transparency. Broadly, the first insight our data provides is that criticisms seem to be heavily present compared to defenses (i.e., 84.3% criticisms vs 22.6% defenses). This suggests that many academic psychologists believe that shifting away from the status quo is, at least to some degree, desirable. Fortunately, these critiques have not gone unnoticed and there are notable efforts addressing these issues at multiple levels.

We observed the greatest amount of concern regarding statistical awareness, suggesting that efforts to improve the field should place a particular emphasis on statistical training and literacy. With respect to existing resources, recent statistics textbooks, at both the introductory and graduate level, have begun adopting an open science perspective (Cummings, 2012; Cummings & Calin-Jageman, 2016) allowing individuals to teach themselves and others. The Framework for Open and Reproducible Research Training organization provides a consolidated database of teaching resources (Framework for Open and Reproducible Research Training, 2022). Creators of the FORRT database have scoured the literature in multiple fields, collecting and summarizing over 100 articles to help teachers, mentors, and students have more accessible avenues for training and self-education. In addition, Daniel Lakens offers a free Massive Open-Online Classroom (MOOC) ‘Improving your statistical inferences’ and has enrolled over 61,000 students for free (Improving Your Statistical Inferences, n.d.). The MOOC covers fundamentals of statistics, from correcting misconceptions about p-values to introducing Bayesian statistics. We believe our work suggests that these projects should continue to expand and help improve statistical literacy within the field of psychology.

A second prominent theme from our results is criticism of incentive structures. At an individual level we see academics having discussions about open science and incentives with a large audience. For example, several podcasts (e.g., Everything Hertz, Two Psychologists Four Beers, The Black Goat), blogs (e.g., The 20% Statistician, Data Colada, Replication Index), and journal clubs (e.g., ReproducibiliTea) frequently discuss the publication process, funding, null-results, and related topics with an open-science lens. There are also communities on social media platforms such as Twitter (e.g., @OSFramework, #openscience, #opendata), Reddit (e.g., r/Open_Science), and Discord (e.g., Git Gud Science), that maintain ongoing conversations about these topics. The adoption of Registered Reports at over 300 journals is another example of shifts in incentive structures; this publication format allows research projects to be evaluated on their methods rather than their results, reducing the incentives to p-hack (Center for Open Science, n.d.). These types of changes may slowly chip away at a shaky incentive structure.

The third prominent critique observed within our data was a call for greater transparency. Not only are the previously mentioned communities having conversations about systemic issues, but they are also having conversations about open data, open materials, and pre-registration. Fortunately, organizations like the Open Science Framework provide many tools to aid researchers by increasing accessibility and reducing the time burden (Open Science Framework, n.d.). OSF provides a single location in which a user can submit pre-registrations, their data, funding checklists and can be used for free. Another website, AsPredicted, speeds up pre-registration, can work within the OSF, and is suitable for getting co-authors to approve the document before making it official (AsPredicted, n.d.). Moreover, journals like Advances in Methods and Practices in Psychological Science publish articles that focus on improving research practices (Advances and Methods in Psychological Science, n.d.). Christensen and his co-authors (2019) have also recently dedicated a text on how to do transparent research, covering topics such as reporting standards, data sharing, and creating reproducible workflows.

Our findings show that psychologists are willing to identify and acknowledge weaknesses and shortcomings within the field; optimistically, this practice may be a necessary step towards improvement.”

• Substantial contributions to conception and design: J. F. Miranda, C. M. Whitt, A. McDiarmid, J. E. Stephens, & A. M. Tullett

• Acquisition of data: J. F. Miranda, C. M. Whitt, A. McDiarmid, D. Purdue, & A. M. Tullett

• Analysis and interpretation of data: J. F. Miranda, C. M. Whitt, A. McDiarmid, & A. M. Tullett

• Drafted and/or Revision of the article: J. F. Miranda, C. M. Whitt, A. McDiarmid, D. Purdue, C. Hall, & A. M. Tullett

• Approved the submitted version for publication: J. F. Miranda, C. M. Whitt, A. McDiarmid, J. E. Stephens, D. Purdue, C. Hall, & A. M. Tullett

The authors declare that there were no conflicts of interest with respect to the authorship or the publication of this article. Alexa Tullett is an author on this manuscript and is an editor for Collabra: Psychology. She was not involved in the review process of this article.

The data for this study is coming from a larger research project funded by the National Science Foundation, Award #:1728332 (See link).

Access to all presented data (both raw and clean), SPSS code, and other relevant materials can be found here.