https://orcid.org/0000-0003-2426-8362
That behavioral sciences are overrepresented by some countries, in terms of samples and authors, is a well-documented finding. Considering the immediate policy implications, the present study explored whether this bias also exists for research on the coronavirus pandemic. Preprints posted on PsyArXiv between two time periods in 2020 (March-April and May-December) with keywords related to “COVID-19” were sourced and their participant and author composition were assessed. Western and rich democracies were overrepresented in terms of authors and participants; preprints posted by authors from western and democratic countries were cited more and were published in journals with a higher impact factor. Implications, especially regarding a reductionist bifurcation of research as “WEIRD” or “non-WEIRD,” are discussed.
Behavioral sciences are increasingly being used to inform policy across countries. Many practices from social and behavioral sciences have been extrapolated to understand how to change behaviors, including in the context of preventable infectious diseases, such as COVID-19. However, most published and widely shared research in the behavioral sciences has been concentrated on a very small, homogeneous section of the population (e.g., Rad et al., 2018). As the COVID-19 pandemic spread across the globe, many countries relied on such research to communicate and use social norms (instead of codified laws) to nudge citizens to engage in hygiene practices. Many researchers have also tried to assess how to communicate about the pandemic efficiently (e.g., Betsch, 2020; Lunn et al., 2020; Van Bavel et al., 2020).
In the case of a pandemic, where time and resources are scarce and there is little empirical evidence for what definitely works and what does not, behavioral interventions might prove difficult. When time and resources are scarce, interpretations of policy could be biased (Mullainathan & Shafir, 2013). This issue is also exacerbated by the fact that poor methods and statistics continue to plague science (Smaldino & McElreath, 2016). No one is doubting the intentions of social and behavioral scientists. However, even before the pandemic, science has often been conducted, interpreted, and disseminated by a homogenous group, from resource-heavy countries. Once exciting results are published and disseminated, the validity of such findings is rarely checked (but see Simonsohn et al., 2021). This is an enormous problem, and could possibly have disastrous consequences, especially during a global crisis. Instead of cross-validating their studies, many social scientists have collected data within short periods of time and have written and published their results in some form or the other within a very short time frame (Ruggeri et al., 2020), when ideally, the process takes months, if not years. How can science be considered objective if it is interpreted by a handful of people (Meadon & Spurrett, 2010) over a short period of time and overlooks a majority of the population?
That research output about COVID-19 is heavily skewed has been previously discussed by a number of commentators. For instance, in economics, it is mostly senior scientists identifying as men who have been publishing pandemic-related working papers (Amano-Patiño et al., 2020). To check if the same holds true for psychology, I assessed papers published on PsyArXiv, a preprint repository, to determine who is conducting and publishing results about COVID-19 in psychology. This is examined using two waves of data. Wave 1 includes data from March and April, 2020; Wave 2 includes data between May and December, 2020. Specifically, the following questions are asked:
RQ1A. Who is conducting studies and publishing preprints about COVID-19? B. Which of these preprints are being published, and in what journals? RQ2A. Which samples are being used to study COVID-19 in psychology? B. Which of these samples are being published, and in what journals?
Sample Diversity in the Psychological Sciences
The acronym “WEIRD” stands for Western, Educated, Industrialized, Rich, and Democratic; in other words, WEIRD provides a reasonably useful heuristic to understand the oft-discussed issue of sample diversity in psychological science. Although psychology is supposed to be a study of human behavior, it is very limited in accurately capturing the human population. For instance, Arnett (2008) has shown that, in the top six journals published by the American Psychological Association, 68% of the studies relied on US American samples, and 96% relied on WEIRD samples. Across the 2015-16 issues of Evolution & Human Behaviour and Evolutionary Psychology combined, 81% of the samples were Western (Europe/North America/Australia), and 12% were from Japan (Pollet & Saxton, 2019). In 2012, 65% of the samples from the articles published in Evolution & Human Behaviour were WEIRD (Kurzban, 2013). In the top five cross-cultural psychology journals, 64.6% of samples (and 96.7% of participants) were from WEIRD countries in 2015-2016; 33.03% of these samples were based in the US (Veillard, 2017). Between 2006 and 2010, among articles published in three top (by impact factor) experimental developmental psychology journals (i.e., Child Development, Developmental Psychology, and Developmental Science), 57.65% participants originated from the US, and 90.52% from WEIRD countries (Nielsen et al., 2017). In 2012, 96% of the papers published in Journal of Personality and Social Psychology were based on WEIRD samples (Kurzban, 2013).
Researcher Diversity in Psychological Sciences
While sample diversity has often been discussed in psychology (e.g., Arnett, 2008); researcher diversity is often overlooked. For instance, in developmental psychology journals, 61% were first-authored by those with institutional affiliations in the US, 20% in English-speaking countries, 15% in non-English speaking European countries, 4% from Asia and Israel, and 2 articles had the first author located in Central or South America, but none in Middle East or Africa (Nielsen et al., 2017). Researcher non-diversity, especially as it relates to international representation, is especially important because broader conceptualizations of scientific constructs may be possible by including more diverse researchers (see for example, Medin et al., 2017). For instance, a number of social psychological constructs that are important to understand intergroup behaviors in India, such as caste and religion, are understudied.
The Present Study
The present study aims to understand who is producing preprints about the coronavirus in the psychological sciences, and using which samples and populations. To do so, I map the sample and author diversity in terms of international representation by investigating the sample and participant characteristics of preprints on PsyArXiv as well as who authored these preprints. Data was collected in two waves: March-April, 2020 (i.e., within two months of the outbreak globally), and May-December, 2020.
It should be noted that bifurcating countries into WEIRD or non-WEIRD is reductionistic. For instance, how similar are US Americans and Cubans? Similarly, how similar are the Japanese and Indians (Muthukrishna et al., 2020)? Technically, both groups would be categorized together as “western” or “eastern.” Further, how similar are rural US Americans, Asian Americans, and economically disadvantaged US Americans? Though Singapore and Japan are often considered non-WEIRD countries, are they comparable to smaller-scale societies? Thus, a WEIRD-non-WEIRD classification may be inaccurate and oversimplified (also see Ghai, 2021).
To compensate for this limitation, I analyze the data using two samples: the first was more restrictive and used only five countries and therefore contains a narrow definition of what constitutes “WEIRD” countries. The second list included more countries, and therefore contains a broader definition of “WEIRDness.” Thus, a longer list of “WEIRD” countries was formed (Supplementary Table S1) based on certain geographical, political, and economic aspects. Keeping in mind the limitations of this approach, disaggregated results at the country-level are provided in the Supplementary Materials as ancillary analyses. For instance, I present the number of samples, participants, and authors from each country in each wave of analysis.
Method
Data and Code Availability
Data and code for this paper are available on OSF at https://osf.io/t98ud/
WEIRD Categorizations
Considering the ambiguous definition and false dichotomization of what constitutes WEIRD countries in the literature, two lists were created.
The Narrow List of WEIRD Countries. The first conceptualization of “WEIRD countries” is more restrictive and includes only five countries that are typically used while describing “WEIRD” countries: Australia, Canada, New Zealand, the UK, and the US. Countries that are not in this list are categorized as non-WEIRD.
The Broad List of WEIRD Countries. The second conceptualization of “WEIRD countries” is broader, and includes Western European countries, such as Germany, that are rich and democratic. Specifically, the coders were asked to check whether the countries are categorized as having higher rates of education and are high-income countries based on World Bank data, and whether they are democratic based on their Polity scores (Polity > 6; Marshall et al., 2017). The polity scores capture whether the state is under an authoritative regime for the purposes of quantitative or comparative analyses.
Wave 1 (March-April 2020)
In an attempt to understand the sampling and researcher diversity in preprints about coronavirus, two independent coders (one coder had a postgraduate degree in psychology; another had an undergraduate degree in Psychology) sourced papers published from up to April 2020 on PsyArXiv, a preprint server for the psychological sciences in the first wave of data collection.
A total of 42 preprints were sourced, of which, 39 were used for the analysis after deleting those that did not meet the inclusion criteria. Specifically, for the preprint to be included in the analysis, it had to meet the following inclusion criteria: 1) Not be a commentary piece, 2) Be in English, 3) Have verifiable information about the authors’ institutional affiliation, and 4) Not have data collected from before 2020 (e.g., longitudinal studies).
At the time of data collection, these papers may not have been peer-reviewed and published in academic journals, as that process might take months (others may post a version after publication). Regardless, preprints ensure rapid dissemination and visibility of research. Preprints tagged “coronavirus” or “COVID-19” or “COVID” were sourced. These papers often represent sub-discipline diversity. Considering that, other than China, most countries detected their first cases between February and March 2020, an April deadline implies up to two months between conceptualization of the study and publication of the preprint.
The raters coded each manuscript for authors and the samples. If there were multiple studies in a single paper, the sample characteristics from each study were noted separately. That is, if a paper comprised three studies, all recruiting US Americans, each study was coded separately. If the author or the sample was affiliated to more than one country, each country was coded separately. This is in line with previous studies (e.g., Arnett, 2008). These countries were then marked as WEIRD or non-WEIRD based on both, the narrow and the broad definitions of the term as described previously.
Commentaries were excluded from analyses. Similarly, papers with no verifiable data about affiliation of the authors were excluded from the analysis. A total of sixty-nine samples involving human participants were identified across the papers. In other words, the analytical sample was 69 samples across 39 preprints.
Next, between March and May 2021, five coders traced each paper from Wave 1 to assess which of them have been published and in which journals. Though imperfect, I used impact factors to assess the impact of journals, as done in previous studies (e.g., Arnett, 2008).
See Figure 1 for a visual depiction of the process.
Wave 2 (May-December, 2020)
Next, two coders (both had postgraduate degrees in psychology) sourced 614 preprints published on PsyArXiv between May and December, 2020 using the same protocol used in Wave 1. Then, three additional coders (all three had post-graduate degrees in psychology) coded the other variables based on the initial list of preprints. After deleting redacted, deleted, or duplicate preprints, a total of 577 unique preprints were included as the final sample of preprints. Commentaries, literature reviews, and other such papers without primary data about samples or participants were then excluded from analysis of the samples. To be more specific, if the preprint was a meta-analysis or a scoping review, it was included in the analysis of authors, if authors’ information was available. Here, I operationalize primary data as survey or experimental data collected for the purposes of the study; that is, secondary data, including public data from tweets or from other websites, or public data at a country/region level were excluded from the analysis. Finally, the samples within the remaining 485 preprints were analyzed.
This final sample included 3002 authors across 577 preprints and 614 samples across 485 preprints. Note that a single preprint could have multiple studies which could sample from multiple populations.
Coding Instructions
The coders were provided with a spreadsheet containing variables that need to be coded. They were then explained what needs to be coded over Google Meet. The notes from the coding instructions are included the supplementary materials (Appendix A and B). An example spreadsheet is included in the online supplementary materials: https://osf.io/n8pgw
Intercoder Agreement
The coders in the first wave coded the preprints separately; the author then verified the information provided by both coders to check whether they were in agreement. In case of disagreements, the author and the coders discussed the specifics until an agreement was reached (r=1.00 for authors, gender, countries of institutions, and authorship).
In the second wave, due to the number of preprints, the author instructed all the coders to code separate preprints. At this time, if there were inconsistencies, they were then discussed with other coders and the authors until an agreement was reached. Considering the size of data, intercoder reliability was not assessed formally for Wave 2. However, two of the coders randomly cross-verified the information for a select few variables for a subsample of preprints (such as for authors’ country of institution which are static, but not for number of citations, which is more likely to change from one point of time to another). Here, some variables such as sample sizes had changed, perhaps because of a newer version of the preprint being posted; however, the coders did not note changes or inconsistencies in authors, countries of institutions and samples, and authorship.
Variables Coded
Researchers
Authorship Order. Authorship was coded as “primary authors” for first and corresponding authors and any other authors noted as having contributed equally to the first and corresponding author. This is based on the assumption that first or corresponding authors contributed more to that particular preprint than other authors (though this may be debatable). All other authors were coded as “secondary authors.”
WEIRD/Non-WEIRDness. Based on the lists described earlier, whether the country of the author’s institution was WEIRD was noted; those that did not fall under the WEIRD label in either the narrow or the broad list were categorized as non-WEIRD.
Country name. The country where the researcher’s institution was situated was coded. If an author had multiple affiliations, each was included separately. This information is presented in the Supplementary Materials.
I also attempted to code gender and career stage of all authors; however, ultimately neither were used for analysis as they were not coded accurately (see Supplementary Materials).
Samples
Country name. Coders were instructed to look at the methods section of each preprint to code the countries from which the authors had sampled; if country names were not specified, coders were asked to code it as is. Concretely, if the manuscript states “international samples from X countries,” it was coded as “X countries.”)
Number of participants. Coders were instructed to look at the methods section of each preprint to code the number of participants in each study separately. That is, if a preprint included two studies, the number of participants were coded separately.
Preprints
Publication Status. Whether or not the paper was published (by March-May 2021) was coded for both waves.
Journal Name and Impact Factor. If a preprint has been published in a journal, the name of the journal and its impact factor was noted.
Citations. Though imperfect, citations were based on the number presented on Google Scholar; this data was coded between March and May 2021.
Link to the preprint and name of publication (Not analysed). The title of the preprint and its URL were included in the spreadsheet for ease of access to cross-verify. This information is masked in the open dataset.
Month (Not analysed). The month of publication was also coded.
Sub-field (Not analysed). The category in which a preprint was tagged (e.g., Social and Behavioural Sciences) was also coded.
Results
Wave 1
Researcher Diversity (RQ1A)
A total of 181 researchers from 23 countries were represented (in addition to one case of no information; see Supplementary Table S2A). A total of 59 (32.60%) authors were affiliated to the narrower list of WEIRD countries, and 122 (67.40%) were affiliated to the non-WEIRD countries in this list. On the other hand, 139 (76.80%) were affiliated to the broader list of WEIRD countries, and 42 (23.20%) to the non-WEIRD countries in this list.
With respect to authorship, out of the 181 authors, 50 were identified as the primary author (including sole authors). Of these, 20 (40%) belonged to countries in the narrow definition of WEIRD, and 30 (60%) to non-WEIRD countries in the narrow definition; 40 (80%) were affiliated to the broader definition of WEIRD countries and 10 (20%) to non-WEIRD.
Sample diversity (RQ2A)
In the 39 preprints, 20 countries were represented, and 14 studies included multinational samples or participants who were not disaggregated by country (Table S3). A total of 222,995 participants (Mean=5309; Range=80 (from the Netherlands) to 201,282 (from the US) ; Median=540; see Supplementary Table S2B) were included across preprints. Twenty preprints included samples from a non-WEIRD country in the narrow list, compared to nineteen in from a WEIRD country. From the broad list, 10 preprints included samples from a non-WEIRD country, whereas 29 included samples from a WEIRD country.
Publication status (RQ1B, RQ2B)
By March 2021, 20 preprints (51.28%) were published and 19 preprints (48.71%) were not. Out of the published preprints, 9 (40.91%) primary authors belonged to countries in the narrow definition of WEIRD, compared to 13 (59.09%) authors from the non-WEIRD countries; this is not a significant difference in proportions (X2=.81, p=.36, 95%CIproportions=[-0.52, .15]1). Next, 12 (54.55%) primary authors were affiliated to institutions in countries in the broader definition of WEIRD, compared to 10 (45.45%) from the non-WEIRD countries; this was also not a significant difference (X2=.09, p=.76, 95%CIproportions=[-.25, .43]).
Out of the preprints that were not published, 11 primary authors (39.29%) belonged to institutions from countries in the narrow definition of WEIRD countries, compared to 17 (60.71%) from non-WEIRD countries (X2(1)=1.79, p=.18;95%CIproportions=[-.51, .08]) and all 28 (100%) were from institutions in the broad definition of WEIRD countries.
Citations and impact factors
| Sample Characteristics | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| N | Mean | SD | Median | t(df) | p (t) | d | 95%CI (d) | W | p (W) | ||
| Citations | |||||||||||
| Narrow | Does not include non-WEIRD samples | 19 | 28.32 | 56.62 | 12 | -0.19 (28.47) | 0.85 | 0.06 | [-.71, .59] | 147 | 0.23 |
| Includes non-WEIRD samples | 20 | 31.15 | 32.63 | 23.5 | |||||||
| Broad | Does not include non-WEIRD samples | 29 | 29.55 | 51.81 | 12 | -0.08 (36.97) | 0.94 | 0.02 | [-0.7, .74] | 91.5 | 0.09 |
| Includes non-WEIRD samples | 10 | 30.4 | 17.85 | 25 | |||||||
| Impact Factors | |||||||||||
| Narrow | Does not include non-WEIRD samples | 7a | 4.46 | 4.36 | 3.6 | 0.94 (6.36) | 0.38 | 0.58 | [-4.40, 1.61] | 46.5 | 0.74 |
| Includes non-WEIRD samples | 12 | 2.88 | 0.98 | 2.68 | |||||||
| Broad | Does not include non-WEIRD samples | 10 | 4.2 | 3.6 | 3.45 2.5 | 1.31 (10.46) | 0.22 | .57 | [-.41, 1.56] | 59 | 0.27 |
| Includes non-WEIRD samples | 9 | 2.64 | 0.98 | ||||||||
| Sample Characteristics | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| N | Mean | SD | Median | t(df) | p (t) | d | 95%CI (d) | W | p (W) | ||
| Citations | |||||||||||
| Narrow | Does not include non-WEIRD samples | 19 | 28.32 | 56.62 | 12 | -0.19 (28.47) | 0.85 | 0.06 | [-.71, .59] | 147 | 0.23 |
| Includes non-WEIRD samples | 20 | 31.15 | 32.63 | 23.5 | |||||||
| Broad | Does not include non-WEIRD samples | 29 | 29.55 | 51.81 | 12 | -0.08 (36.97) | 0.94 | 0.02 | [-0.7, .74] | 91.5 | 0.09 |
| Includes non-WEIRD samples | 10 | 30.4 | 17.85 | 25 | |||||||
| Impact Factors | |||||||||||
| Narrow | Does not include non-WEIRD samples | 7a | 4.46 | 4.36 | 3.6 | 0.94 (6.36) | 0.38 | 0.58 | [-4.40, 1.61] | 46.5 | 0.74 |
| Includes non-WEIRD samples | 12 | 2.88 | 0.98 | 2.68 | |||||||
| Broad | Does not include non-WEIRD samples | 10 | 4.2 | 3.6 | 3.45 2.5 | 1.31 (10.46) | 0.22 | .57 | [-.41, 1.56] | 59 | 0.27 |
| Includes non-WEIRD samples | 9 | 2.64 | 0.98 | ||||||||
a Impact factors can only be calculated if the preprint has been published (n=20; the impact factors of only 19 preprints were coded, 1 was coded as NA); thus, the sample size is smaller. Note: Australia, Canada, New Zealand, UK, USA are included in the narrow definition of WEIRD countries; The broader definition includes those countries in the narrow definition as well as Western European countries that are rich and democratic; N=Total preprints considered, t (df)=Welsch’s t-test (degrees of freedom), p (t)= p-value associated with the t-test; d=Cohen’s d, 95%CI=95% confidence interval for Cohen’s d, W=Wilcoxon test to assess differences in the medians; p(W)= p-values associated with the Wilcoxon test.
| Author Characteristics | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| N | Mean | SD | Median | t (df) | p (t) | d | 95%CI (d) | W | p (W) | ||
| Citations | |||||||||||
| Narrow | Not WEIRD | 20 | 31.15 | 32.63 | 23.5 | 0.19 (28.47) | 0.85 | 0.06 | [-0.59, 0.71] | 233 | 0.23 |
| WEIRD | 19 | 28.32 | 56.62 | 12 | |||||||
| Broad | Not WEIRD | 9 | 31.56 | 18.53 | 26 | .21(35.31) | 0.84 | 0.05 | [-0.72, 82] | 187.5 | 0.08 |
| WEIRD | 30 | 29.23 | 50.94 | 13.5 | |||||||
| Impact Factors | |||||||||||
| Narrow | Not WEIRD | 12 | 2.88 | 0.98 | 2.68 | -0.94 (6.36) | 0.38 | 0.58 | [-1.61, .44] | 37.5 | 0.74 |
| WEIRD | 7 | 4.46 | 4.36 | 3.6 | |||||||
| Broad | Not WEIRD | 9 | 2.64 | 0.98 | 2.5 | -1.31(10.46) | 0.22 | 0.57 | [-1.56, .41] | 31 | 0.27 |
| WEIRD | 10 | 4.2 | 3.6 | 3.45 | |||||||
| Author Characteristics | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| N | Mean | SD | Median | t (df) | p (t) | d | 95%CI (d) | W | p (W) | ||
| Citations | |||||||||||
| Narrow | Not WEIRD | 20 | 31.15 | 32.63 | 23.5 | 0.19 (28.47) | 0.85 | 0.06 | [-0.59, 0.71] | 233 | 0.23 |
| WEIRD | 19 | 28.32 | 56.62 | 12 | |||||||
| Broad | Not WEIRD | 9 | 31.56 | 18.53 | 26 | .21(35.31) | 0.84 | 0.05 | [-0.72, 82] | 187.5 | 0.08 |
| WEIRD | 30 | 29.23 | 50.94 | 13.5 | |||||||
| Impact Factors | |||||||||||
| Narrow | Not WEIRD | 12 | 2.88 | 0.98 | 2.68 | -0.94 (6.36) | 0.38 | 0.58 | [-1.61, .44] | 37.5 | 0.74 |
| WEIRD | 7 | 4.46 | 4.36 | 3.6 | |||||||
| Broad | Not WEIRD | 9 | 2.64 | 0.98 | 2.5 | -1.31(10.46) | 0.22 | 0.57 | [-1.56, .41] | 31 | 0.27 |
| WEIRD | 10 | 4.2 | 3.6 | 3.45 | |||||||
Note: N=Total preprints considered, t (df)=Welsch’s t-test (degrees of freedom), p (t)= p-value associated with the t-test; d=Cohen’s d, 95%CI=95% confidence interval for Cohen’s d, W=Wilcoxon test to assess differences in the medians; p(W)= p-values associated with the Wilcoxon test.
Wave 2
A total of 614 preprints were sourced based on preprints posted between May and December 2020. Of these, after deleting preprints that did were redacted or deleted, 577 unique preprints were analyzed.
Researcher Diversity (RQ1A)
A total of 3002 authors (2341 unique authors) were coded across the 577 preprints posted between May and December, 2020. Of these, 17 did not identify a particular country, 2918 (97.20%) authors were affiliated to institutions in a single country, and the other 84 authors were affiliated to multiple countries (see Supplementary Tables S3A for a disaggregated list).
Further, 1592 authors (53%) belonged to the narrow list of WEIRD countries, and 1410 (47%) to non-WEIRD countries; the difference in these proportions was significant (X2(1) = 10.91, p<.001, 95%CIproportions2 = [0.452, 0.488]); also see Supplementary Table S4). When the broader list of countries is considered, 2593 (86.38%) authors belonged to WEIRD countries, and 409 (13.62%) to non-WEIRD countries (X2(1) = 1587.4, p<.001, 95%CIproportions = [0.124, 0.149]).
A total of 637 of all authors were identified as primary authors and 2365 were identified as secondary authors on the preprints. Of the primary authors, 330 (51.81%) were from the narrower list of WEIRD countries and 307 (48.19%) from non-WEIRD countries (X2(1) = 0.76, p=.383, 95%CIproportions=[0.443, 0.522]). Further, 553 (86.81%) from the broader list of WEIRD countries, whereas only 84 (13.19%) were from non-WEIRD countries, a significant difference (X2(1) = 343.84, p<.001, 95%CIproportions =[0.107, 0.161]).
Sample diversity (RQ2A)
After filtering the preprints (n=485) for studies (n=614) that had primary data, 358 (58.31%) had participants from the narrow list of WEIRD countries and 256 (41.69%) had participants from non-WEIRD countries. This difference in proportions was significant (X2(1) = 17.702, p<.001, 95%CIproportions=[0.378, 0.457]). A total of 509 (82.90%) preprints had participants from WEIRD and 105 (17.10%) had participants from non-WEIRD from the broad definition of WEIRD countries. The difference between these proportions was also significant (X2(1) = 264.51, p<.001, 95%CIproportions=[0. 143, 0. 203]).
Sample data collected from before 2020 (e.g., longitudinal studies) were deleted from the analysis of samples, along with studies that do not include participants directly (e.g., meta-analyses); a total of 36,909,784 participants were employed (range=1 to 32,400,000 participants; Mean=64079, Median=690.5).
The median number of participants from narrow list of WEIRD countries was 750, compared to 570 from non-WEIRD countries; however, this was not a significant or a large difference (X2(1) =2.33, p=.127, Cramer’s V=.04). Next, the median number of participants from the broad list of WEIRD countries was 793, compared to 412 from the non-WEIRD countries in this list; this difference was significant (X2(1) =13.36, p<.001, Cramer’s V=.11).
Publication status (RQ1B, RQ2B)
By May 2021, 200 preprints (34.67%) were published and 377 (65.34%) were not. Out of the published preprints, 655 (57.05%) authors belonged to the narrow list of WEIRD countries and 493 (42.94%) to the non-WEIRD countries from that list. On the other hand, when the broader list is considered, 1009 (87.89%) authors were affiliated to institutions from the WEIRD countries whereas 139 (12.10%) were affiliated to non-WEIRD countries. Specifically, 134 (55.83%) of the primary authors were from the narrow list of WEIRD countries, whereas 106 (44.17%) from non-WEIRD countries and 217 (90.42%) were from the broader list of WEIRD countries and 23 (9.58%) from non-WEIRD countries based on the broader definition.
Citations and Impact Factors (RQ1B, RQ2B)
The mean and median citations are higher for preprints where the primary author is from an institution belonging to a WEIRD country versus a non-WEIRD country in the narrow and broad definitions of WEIRD (Table 3).
| Author Characteristics | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| N | Mean | SD | Median | t (df) | p (t) | d | 95%CI (d) | W | p (W) | ||
| Citations | |||||||||||
| Narrow | Not WEIRD | 296 | 11.91 | 34.99 | 1 | -0.15 (522.28) | 0.88 | 0.01 | [-0.15, 0.17] | 46580 | 0.62 |
| WEIRD | 322 | 12.48 | 60.42 | 1 | |||||||
| Broad | Not WEIRD | 76 | 6.24 | 17.67 | 0 | -2.24* (312.3) | 0.03 | 0.14 | [-0.1, 0.38] | 16870** | 0.01 |
| WEIRD | 542 | 13.04 | 52.77 | 1 | |||||||
| Impact Factors | |||||||||||
| Narrow | Not WEIRD | 87 | 3.02 | 1.87 | 2.47 | -2.25* (187.49) | 0.03 | 0.31 | [.02, .6] | 3906 | 0.07 |
| WEIRD | 107 | 3.76 | 2.7 | 2.74 | |||||||
| Broad | Not WEIRD | 21 | 2.35 | 1.77 | 2.21 | -2.82**(29.91) | 0.008 | 0.51 | [0.03, 0.99] | 1073.5** | 0.002 |
| WEIRD | 173 | 3.56 | 2.42 | 2.74 | |||||||
| Author Characteristics | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| N | Mean | SD | Median | t (df) | p (t) | d | 95%CI (d) | W | p (W) | ||
| Citations | |||||||||||
| Narrow | Not WEIRD | 296 | 11.91 | 34.99 | 1 | -0.15 (522.28) | 0.88 | 0.01 | [-0.15, 0.17] | 46580 | 0.62 |
| WEIRD | 322 | 12.48 | 60.42 | 1 | |||||||
| Broad | Not WEIRD | 76 | 6.24 | 17.67 | 0 | -2.24* (312.3) | 0.03 | 0.14 | [-0.1, 0.38] | 16870** | 0.01 |
| WEIRD | 542 | 13.04 | 52.77 | 1 | |||||||
| Impact Factors | |||||||||||
| Narrow | Not WEIRD | 87 | 3.02 | 1.87 | 2.47 | -2.25* (187.49) | 0.03 | 0.31 | [.02, .6] | 3906 | 0.07 |
| WEIRD | 107 | 3.76 | 2.7 | 2.74 | |||||||
| Broad | Not WEIRD | 21 | 2.35 | 1.77 | 2.21 | -2.82**(29.91) | 0.008 | 0.51 | [0.03, 0.99] | 1073.5** | 0.002 |
| WEIRD | 173 | 3.56 | 2.42 | 2.74 | |||||||
N=Total preprints considered, t (df)=Welsch’s t-test (degrees of freedom), p (t)= p-value associated with the t-test; d=Cohen’s d, 95%CI=95% confidence interval for Cohen’s d, W=Wilcoxon test to assess differences in the medians; p(W)= p-values associated with the Wilcoxon test; *p<.05, **p<.01, ***p<.001
For the preprints that were published, the mean (but not median) impact factors of the journals are higher when the primary authors are from an institution that is from a WEIRD country in the broad definition. When the narrow definition is considered, the mean and median impact factors of the journals are higher when the primary authors are from an institution that is from a WEIRD country than a non-WEIRD country (Table 3).
Discussion
This study aimed to understand who is producing preprints about COVID-19 in psychology, as posted on PsyArXiv, and using what samples. To do so, the current study assessed the researcher and sample diversity in preprints about COVID-19 in psychology, hosted on PsyArXiv. In the first two months since COVID-19 was declared a pandemic (March-April 2020), a total of 65 studies were conducted and published as 39 preprints, mostly with authors and samples from western and developed countries. Further, preprints with primary authors from western and developed countries also had higher median citations and were published in journals with higher impact factors.
The preliminary analysis of papers published in the earlier stages of the pandemic is very much in line with previous studies that report an overreliance on samples from the US in published papers across the subdisciplines of psychology (e.g., Arnett, 2008; Veillard, 2017). It could be contended that there could have been opportunity costs for many researchers who could have spent their resources in other ways. However, that is the root of my argument - this is true of researchers from the overrepresented countries as well. It is a known fact that often preliminary, novel findings are publicized and enter our collective imagination; any refutations or alternative findings, especially from other countries are thought to be endogenous to these “cultures” and are often not published in top journals (see Arnett, 2008; Rad et al., 2018). This exercise could be construed as a critique of fast science, in that, I wanted to explore to what extent preliminary findings are published fast, and who is publishing them. The speed of publishing in science is a reflection of its existing power structures (see Martell, 2014), wherein underrepresented groups, including gender and racial minorities are often penalized in direct and indirect ways (Leite & Diele-Viegas, 2021). This is also in line with what I found with respect to citations and journal impact factors of preprints that were published.
In the next stages of the pandemic (May-December 2020), a similar trend was observed with respect to citations as well as impact factors, in that, authors from western, developed countries had published in “top” journals, and were cited more. Further, there was also a higher number of samples as well as median sample sizes from western, developed countries.
In addition to showing an overreliance of samples from WEIRD countries, this was also in line with anecdotal evidence about authors of certain countries or studies employing certain samples being encouraged to send their manuscripts to specialty journals (often with lower impact factors). Similarly, author status, including country of the institution, affects peer review. This includes those not from the Global North being considered peripheral and therefore, not worthy of publication citing lack of quality or contribution as the rationale. For instance, social psychologists in the Global South and South and Eastern Europe reported that a minority of their colleagues published in “international” journals (i.e., often US-based ones), compared to those in the Global North, possibly due to systemic constraints (Bou Zeineddine et al., 2021).
Preprints, on the other hand, can be posted by authors across countries; this might specifically benefit authors from small and developing countries who employ samples from their home countries, where systems of science are not as robust as the developed countries. That is, authors from smaller countries employing samples from their countries may possibly benefit by posting preprints publicly.
This is also in line with the general finding that psychology is dominated by authors from the US. It has been argued before that when applying such science to policies, caution needs to be exercised. Particularly, this also affects generalizability of theories in the psychological sciences (IJzerman et al., 2020, 2021). Perhaps more importantly, theories in psychology may have low explanatory power and/or be invalid because authors have a narrow perspective about societies that they do not have lived experiences in.
It is also important to note that researchers in developing countries often face issues relevant to the nascency of the scientific process. For instance, research infrastructure is often underdeveloped and research funds are limited (see Onie, 2020). These exacerbate the problems with understanding human behavior more broadly. For instance, if specific measures and theories are not equivalent across countries, researchers from countries where psychological science has not yet developed may find it difficult to study specific behaviors in their country. A related problem is also in studying constructs in a culture-blind manner (Berry et al., 2002), wherein even though constructs may not be culture-blind, they are assumed to be so based on previous work focusing on specific countries such as the US. Similarly, constructs not relevant to the “mainstream” frameworks of psychology, such as caste in India, are not studied (Gorur & Forscher, 2023). Further, underrepresented or Eastern countries are usually studied as absolutes, which is also reductionistic. In other words, a narrow lens, for example, based on researchers in the US, is usually employed to study other countries.
Many less developed countries did handle the pandemic well. Thus, lack of studies based on samples from non-WEIRD countries also means that important lessons and expertise about the pandemic from those countries are generally lost. This problem is exacerbated when such studies are not published in “top” journals or in journals with a wider audience.
In this light, it is also important to note the limitations of this study. First, though authors, samples, and participants were assessed, I was unable to assess the constructs, and whether there was diversity in the constructs studied. For example, were authors from countries with indigenous populations able to assess constructs specific to such populations? Relatedly, I was unable to assess whether there was within-country representation across countries. That is, though researcher diversity was assessed in terms of countries, within-country diversity in terms of class, ethnicity, race, or language, was not assessed. Further, many preprints did not indicate specific countries that were sampled, reducing accuracy, indicating that better standards for reporting of samples in psychological sciences are required. It is also important to note the limitation of the inferential tests that I conducted. For wave 1, I have presented inferential tests which are likely underpowered. Though all possible preprints that were identified within the defined period were coded, caution must be exercised while interpreting the results, considering how, for example, p-values are affected by small deviations. The data for Wave 2 somewhat solves the problem of power by including more preprints. It might be assumed that the structural and institutional advantage that those in WEIRD countries have might be diluted over time; however, observationally the problem only worsens across the time period that I have looked at. Next, here, I assumed that across a number of my variables of interests, there is an even split with respect to both WEIRD and non-WEIRD countries. A more meaningful test might have taken into account base rates of number of researchers as well as population indices to make such claims. However, data on such important covariates for many countries included here are lacking. Future studies could attempt to develop more reliable indicators so that these variables can be taken into account. I do compare authors and publication status of preprints in both waves based on the lenient criterion of an even split between countries and between a more stringent split based on the medians of the total population of these countries based on the 2019 population data provided in the World Development Index. There is much data missing in this dataset and therefore, this is presented as supplementary tables (see Tables S4A).
Next, I only assessed preprints published on PsyArXiv. It is possible that only certain researchers from some countries have started putting up their working papers onto a repository such as ArXiv, bioRxiv, or region-specific repositories such as AfricArXiv. Correspondingly, only COVID-19-related manuscripts in the specific time period between March and December 2020 were assessed. Many countries saw a second peak in 2021, and in general, the pandemic is an ongoing and evolving situation. It is likely that more studies would be conducted and published outside this time period. Similarly, it is possible that authors from other countries did not focus on research related to the coronavirus in 2020, but did so after that. This is in line with other issues such as a lack of research infrastructure and funding in some countries, which could have made it more difficult for these researchers to conduct research related to coronavirus in such a short span. Further, only preprints in English were assessed, and I could not consider those in other languages; this is especially a problem since English being the lingua franca of scientific communication has been recognized as an additional systemic barrier (Draguns, 2001). Future research should attempt to assess preprints and papers in languages other than English.
Future work may consider papers that are peer-reviewed and published, considering that not every paper is published as a working paper due to various institutional and grant-related constraints. I assessed journal and preprint impact using citations and impact factors, both of which are imperfect metrics (e.g., Allen & Iliescu, 2021; Brito & Rodríguez-Navarro, 2019). Future research could assess which preprints or published papers are used to create policies at local, national, and global levels. Similarly, the operationalization of researcher diversity could include within-country factors, such as the author’s immigrant status, their socioeconomic class, and gender. Future work can also evaluate the sampling methods used in the preprints – for example, even if the authors use participants from a non-WEIRD country, what are the educational levels and the socio-economic class of the participants? A comparatively rich and educated person from the city of Mumbai, for example, is likely to be culturally similar to an educated and rich European person; on the other hand, a person from lower socioeconomic class from rural parts of India may be similar to their counterparts in the US (see Muthukrishna et al., 2020). Though such differences and similarities should also be noted and quantified, the present study was unable to do so. Future research can also increase the time period and assess preprints produced in 2021. An interesting observation from this dataset is that the problem worsened in terms of both sample and researcher diversity. This was not empirically assessed here due to unequal volume of data in the two waves; future studies could compare this dataset with other datasets (e.g., based on studies produced after 2020). Similarly, though publication status, impact factors, and citations are assessed for Wave 2, the time period between posting of the preprints and data collection for this study was short. Future research could reevaluate these preprints over time to check whether citation counts change across time for those that employ WEIRD samples or authors.
Where, then, does this leave us? If psychological science is needed to inform policy, it is imperative that the science itself is as resolute or robust as possible. This is especially true in the case of immediate danger or major crises, as seems to be the case with the pandemic. If most researchers concentrate on collecting data from a few countries, and derive insights based on restricted sampling, scientific objectivity is questionable. It is also vital that researchers collect data about the countries and ethnicities of their participants, so that constraints on generalizability are clearer.
Though the present study only looks at preprints that have often not been peer-reviewed (there are exceptions here of course), these preprints are often cited in published works. Similarly, this study also highlights the leaky nature of the publishing pipeline, especially as it notes who is publishing preprints and how many of them are published in a peer-reviewed journal.
Conclusion
The present study assessed preprints published on PsyArXiv in two waves: immediately after the onset of the pandemic across the world, and between two and ten months after its onset. I found that in the first wave, there was an overrepresentation of authors and samples from WEIRD countries. In the second wave, though more geographical diversity was found, authors and samples from developed and European countries were overrepresented. Through the study, I hoped to highlight who conducts research in psychology, on what samples, and which of them get published, and get cited.
Contributions
AP contributed to the conception, design, analysis and interpretation of data, as well as drafting of the manuscript.
Acknowledgements
I thank the coders who assisted in collecting data - Susan Ajith, Supriya Bisht, Urvi Mange, Nikita Mehta, Yarshna Sharma, Trishla Surana, and Arunima Ticku.
I also acknowledge Hansika Kapoor, Nikita Mehta, and Tage Rai for their helpful comments.
Funding information
The author(s) received no particular financial support for the research, authorship, and/or publication of this article.
Competing Interests
No particular competing interests. However, the author has resided and worked in both a “WEIRD” and a “non-WEIRD” country.
Data Accessibility Statement
Data and code linked to this manuscript are publicly accessible in this OSF project: https://osf.io/t98ud/, which includes the data and code: https://osf.io/4rfue/
Footnotes
The 95%CI here reflects the confidence in the estimation of proportions.
The confidence intervals represent the proportions of author institutions from non-WEIRD countries.
University of California Press