A Perspective on the Relevance and Public Reception of Psychological Science

As scientists, it can be easy to lock ourselves inside the “science bubble” and ignore how the general public perceives our work. While some researchers may see the importance of communicating findings to the public, others may argue that “research is for researchers” or even that psychological research is too complicated to communicate to the public. Despite the variety of opinions on the matter, the decision is ultimately up to members of the public: pay attention to research if one finds it interesting and valuable or ignore the research if not. The question is, then, how does the public receive psychological research? Indeed, past researchers have asked similar questions: is psychological research relevant (Giner-sorolla et al., 2018; Lilienfeld, 2012; Silverman, 1971)? Are psychologists asking the kinds of questions that are important and interesting to non-psychologists (Sherif, 1970)?

One way to examine the level of interest in psychological research is to examine how people engage with and share research on social media websites. Reddit is a free social media website with a large worldwide audience (500+ million subscribers). Reddit is the second-most used site for news and the 18^th most-trafficked website (Alexa, n.d.) and is considered to be one of the most influential websites on the internet (Shaer, 2012). With such a large audience, this website provides an excellent window into online trends and the interests of the public.

One subsection of this website (called a “subreddit”) is called r/Science, which allows users to engage with the most recent scientific research and media coverage. All users are subscribed by default to this section. Anyone can post content, comment, “upvote” content they like, and “downvote” content they do not like. Further, posts are tagged by the poster as belonging to various scientific disciplines. However, the title of the post is often not the title of the published paper nor is the journal name required to be advertised. Further, posts to r/Science, must be peer-reviewed research (no summaries, secondhand reviews, pop articles, etc), the research must be less than 6 months old, and the subreddit is heavily moderated to enforce these rules. Thus, such a metric is a straightforward method of assessing whether psychological research is interesting to the average user and, qualitatively, what kinds of research topics are of most interest.

For the purposes of this investigation, data were downloaded from Reddit which include posts to r/Science from all of 2017 (data were downloaded in mid 2018, so 2017 was the most recent complete year). The graph below (Figure 1) depicts the mean upvotes per discipline. Posts tagged with “Psychology” have the highest mean upvotes (M = 2,759, SE = 302.95) followed closely by those tagged with “Social Science” (M = 2,602, SE = 447.34) and “Health” (M = 2,525, SE = 253.35). Clearly, Redditors are interested in research which deals with humans and psychology is a clear example of this.

Breaking down the data even further, I had four research assistants categorize each psychology post into a broad sub-discipline (social, developmental, clinical, or neuroscience) based on the topic of the research as indicated by title of the post. The coders disagreed on 28% of the titles; for these 252 titles, an independent coder reviewed the disagreements and served as the tie-breaker. Figure 2 shows the distributions of posts according to the four categories. Those categorized as social psychology were the most in number followed by clinical, developmental, and finally neuroscience, which received fewer than 1,000 upvotes.

The next goal was to examine the “Social Psychology” posts more closely. However, the post itself does not share the same name as the article, nor does it always link directly to the article (it may link to a press release). By following the link from the post on r/Science and searching the press releases, the journal in which each article was published was identified. This allowed for the journal-level metrics to be recorded, including H-index, SJR score, impact factor, and SciMago rankings (SCImago, n.d.). In addition, the number of citations from Google Scholar and the Altmetric score for each article were also recorded at the request of reviewers.

The first thing to notice about these data are that the posts with the highest number of upvotes came from a variety of journals (see Table 1), few of which are considered the highest-ranking journals in social psychology. Looking at the top 10 most upvoted posts offers a glimpse into the kinds of research Redditors find most interesting. One thing to notice about these data is that the top 10 most upvoted posts received between 37,000 and 74,000 upvotes: psychological research is hardly going unnoticed.

While some of the most upvoted posts may belong to “top tier” journals, other higher-ranking journals frequently had their articles posted on Reddit even if these were not the most upvoted posts. Looking at the journals which had the most posts (see Figure 3) we see that journal status does not necessarily indicate public interest. The Journal of Experimental Social Psychology (IF = 2.87) had some of the most upvoted articles while Psychological Science (IF = 6.13) and Journal of Personality and Social Psychology (IF = 5.73), for example, had more posts with lower mean upvotes within a tighter range.

Examining this data at the journal level changes the information we can glean. Not only are we asking what kinds of psychological research Reddit users find interesting, but we can now see whether the questions of interest and relevance to this audience are in any way related to metrics used by social psychologists. That is, are metrics like journal ranking and impact factor a good indicator of what research receives upvotes? It appears they are not.

For all posts categorized as belonging to social psychology, correlations between Reddit upvotes and the journal-level and article-level metrics were examined. Given the skew of upvotes, I also examined the natural-log of upvotes. In either case, Reddit upvotes were simply not correlated with impact factor or journal rankings (see Table 2). Please note that the n varies across correlations for each metric because not all metrics are available for all journals. For example, not all journals in the dataset have a ranking in the SciMago “social psychology” category and not all journals have published impact factors (SCImago, n.d.).

Metrics such as impact factor and journal rankings are often factored into decisions for hiring, tenure promotions, and grants (but, see Abambres, Ribeira, Sousa, & Lantsoght, 2018, for an argument as to why impact factors are poor metrics of researcher performance). Thus, these results further suggest that the kind of research that makes it into top journals is not necessarily the same research that is upvoted on Reddit.

Importantly, this data contains information about many articles from the same journal, so it’s also important to check for interdependence. Thus, I attempted to construct a multi-level model to identify what proportion of variance in upvotes can be accounted for by the journal. However, there was almost no variance at the journal-level (ICC = .005), which means that the reliability of measurements within each journal are highly inconsistent. This suggests that a mixed model accounting for both article and journal level variance is not needed to understand the patterns in these data.

What do these results mean? First, these data show that Redditors care deeply about research which deals with humans—namely psychological research, and especially social psychological research: many posts were upvoted tens of thousands of times. For comparison, the other most upvoted r/Science categories were Social Science, Health, Engineering, and Environment. The first, tenth, and twentieth number of upvotes are displayed in Table 3. As you can see, social psychology is not too far ahead nor too far behind these other categories. Thus, one take-home message is that Redditors care about psychological research.

However, it’s also worth noting that posts in r/Science are still far from being the most upvoted posts on the entire Reddit website. For example, the most upvoted post of all time on Reddit received 349,000 upvotes (Reddit, n.d.) and Reddit posts from other subreddits frequently exceed 100,000 upvotes (Pereira, 2018).

The reasons why some articles are shared on Reddit and others are not is not a question that can be answered definitively with the present data. However, examining the ten most upvoted articles in Social Psychology (Table 1) suggests that users of r/Science appreciate descriptive research and research that offers up practical findings which describe something interesting about ourselves—for example, politics, religion, dating, personality, identity, etc.

In contrast, one can also speculate about the kinds of research which people find less interesting. For example, past research suggests that over time psychology has become more complicated, esoteric, and long-winded (Witte & Brandt, 2011), as well as overly theoretical and less descriptive (Rozin, 2001), and increasingly complex (Quiñones-Vidal, José López-García, Peñaranda-Ortega, & Tortosa-Gil, 2004; Sherman, Buddie, Dragan, End, & Finney, 1999). Compared to the descriptive and practical findings noted in Table 1, this combination of characteristics seems less likely to attract the average reader even though it may gain praise from scientists (Reis & Stiller, 1992).

Second, these data point to a disconnect between the metrics of evaluation used by scientists and general audiences. These data show that the research people find interesting on Reddit is not necessarily associated with scientific publishing metrics. Other research has reported that the public attention an article receives (e.g. Altmetric score) is not a good predictor of its future citation count (Thelwall & Nevill, 2018) which further suggests a disconnect between the metrics used by science and readers. There are a few possible explanations for this disconnect which can be explored in future research (but are not testable with the present data). For example, one could speculate that scientists may be concerned with theory and lay people may be concerned with application. Perhaps scientists want something to build their new findings on and lay people want findings that apply in the present moment. Indeed, the articles shown in Table 1 are more practical rather than theoretical. Another possibility is that maybe scientists see a different goal and a different audience for their science. Again, these possibilities are speculative but could be examined in further experimental studies. Whatever the reason, I think these data suggest that people find at least some psychological research interesting and perhaps “relevant” despite concerns otherwise.

Of course, the data presented here are not meant to be prescriptive; that is, I would not recommend that scientists go on Reddit to identify the areas that lay people find interesting and start conducting their research accordingly. Perhaps this would be a great way to decide on the topic for a popular science book, instead. However, the pursuit of flashy and surprising findings is one factor leading to large-scale non-replicability (Collaboration, 2015; Camerer et al., 2018), possibly because the qualities of risky and surprising findings differ from the kinds of research yielding expected (or “boring”) results (Fiedler, 2017).

Instead of planning out research based on what will be “popular,” perhaps we can reflect on who should be the beneficiaries of our science and whether our science is having the desired effect. As psychology is the field aiming to explain human behaviors (Rad, Martingano, & Ginges, 2018), should we be producing findings that can apply to people’s life in the present? From this data, it appears that psychology as a whole does fulfill that role (to some extent) whether scientists like it or not. Perhaps scientists should be trying to identify the areas of research which are most impactful and beneficial in everyday life. Research identifying these areas of study would be most welcome.

Therefore, these findings have some important implications for issues around the replicability and integrity of our research. People will pay attention to our findings, they will share them online, they will talk about them, and they will use them in their lives. We (scientists) then have a responsibility to make sure that our published findings are accurate and hold up to scrutiny. We have a responsibility to make sure that we share data, we are transparent in our analyses, and that we minimize the reporting of false-positives. We have a responsibility to make sure that we are communicating our results accurately because whether we like it or not, our research does not just go to researchers.

The take-home message is that there both is and isn’t a “crisis of relevance” (Giner-Sorolla, 2019). In one sense, there may be a crisis because the kinds of research that lay people find interesting is not necessarily the kind of research that gets cited or published in high-impact journals; this disconnect may be concerning. However, people (at least users of Reddit) clearly find some psychological research interesting enough if it is tied for the highest mean upvotes on r/Science. I hope that we, psychologists, can continue to garner interest and that we can continue to ask the right questions. At the same time, I hope the field can continue to improve the methods we use and the way we report our findings because people are watching eagerly.