Theoretical Maturation of the “Bias Blind Spot”: A Preregistered Replication Study of Pronin, Lin, and Ross (2002) in a Brazilian Sample

The outcome of our choices stems from a decision-making process guided by judgment (Rachlin, 1989). Achieving optimal judgments generally demands following the “normative rules”: relying on mathematical and statistical operations to estimate odds and probabilities. Considering the rate at which we make judgments in our everyday lives, the effort of these normative rules, and the uncertainty of the events they are embedded in, errors in judgment are likely to occur (Kahneman et al., 1982; Rachlin, 1989). A growing body of literature has shown our proneness to judgment errors (Kahneman et al., 1982) as well as their detrimental effect in several domains, such as medical diagnosis (Chen et al., 2023; Saposnik et al., 2016), law (Smith & Greene, 2005), economics (Louie et al., 2007), politics (Schuett & Wagner, 2011), and research publication (Nuijten et al., 2016; Schimmack & Bartoš, 2023). Judgment errors may occur due to heuristics - rules of thumb readily available that reduce the effort of complex predictions of judgment - or biases (also known as cognitive biases) - when people transpose their expectations and contexts to solve judgment problems (Kahneman et al., 1982; Rachlin, 1989). Although people can identify cognitive biases as well as their consequences, they also share a “bias blind spot” (BBS): a tendency to perceive their peers as more susceptible to these biases than themselves (Pronin, Lin and Ross, 2002). This phenomenon has been observed across several known cognitive biases (West et al., 2012) and tends to persist even after biased judgments are acknowledged (Hansen et al., 2014), increasing the strain of decision-making. For example, while biases affect hiring decision-making (Korte, 2003), human resources employees perceive their peers as more susceptible to biases than themselves (Thomas & Reimann, 2023). This “asymmetry” between self-other perceptions of bias susceptibility may have serious implications, such as increasing or generating conflicts, disagreements, and misunderstandings (Pronin, Puccio and Ross, 2002; Pronin & Kugler, 2007). Moreover, attempts to correct BBS have achieved little success, mostly because people fail to recognize their bias (Hansen et al., 2014; Pronin, Puccio and Ross, 2002).

Evidence suggests that BBS stems from three main sources (Pronin & Kugler, 2007): (1) Self-enhancement biases: overestimation of personal traits while disregarding contrary evidence to see themselves more positively; (2) introspection illusion: a tendency to overvalue introspective information for self-assessments while relying mostly on observable behavior and external sources for assessing others; and (3) naïve realism: when people assume that their perspective of the world is an ‘objective reality’, therefore perceiving others with different perspectives as being more biased than themselves.

The first direct experiment of BBS was provided in Study 1 of three empirical studies described in Pronin, Lin and Ross (2002). This study investigated how people would rate their susceptibility and that of others for a set of biases and short-series personal shortcomings. The former was assumed to produce a “self-other asymmetry”, whereas the latter would not, creating a difference between asymmetries. This difference was theorized due to cognitive availability: a tendency to make judgments based on readily available information (Tversky & Kahneman, 1973). According to the authors, previous evidence has shown that people are aware of specific biases that affect their peers’ responses. Additionally, people tend to rate themselves as below average for some domains in which they are aware of their skills. Therefore, shortcomings and biases that people are likely to be aware of via cognitive availability would not produce an asymmetry. On the other hand, perceptions of biases would be asymmetric due to “naive realism” as people, assuming their worldview as the genuine “objective reality”, would attribute differing perspectives of this reality as being biased, but overlooking their own biases influencing their worldview, thus leading to perceiving themselves as less susceptible to biases. The results confirmed that (1) people tend to rate themselves as less susceptible to biases than others (asymmetry of bias); (2) this tendency does not occur for personal shortcomings (asymmetry of personal shortcomings), and (3) the difference in self-other ratings for biases is larger compared with personal shortcomings (difference between asymmetries).

To date, only one direct replication has been published for this study. Chandrashekar et al. (2021) proposed to conduct a pre-registered, well-powered replication to test the replicability and generalization of the results from Pronin, Lin and Ross (2002) Study 1 in a cross-cultural setting. Table 1 summarizes the Hypotheses supported by Pronin, Lin and Ross (2002) Study 1 and the results from the replication study of Chandrashekar et al. (2021). They successfully replicated Hypothesis 1 and 3, whereas Hypothesis 2 showed an unexpected effect: instead of the expected ratings (equal or higher than others), people rated themselves in a similar way to the biases’ ratings (that is, as less susceptible than their peers), though with a smaller effect when compared to the biases ratings.

Recently, evidence of low rates for the replicability of previous findings in psychology as well as their reproducibility (Hardwicke et al., 2022; Klein et al., 2018; Maassen et al., 2020; Open Science Collaboration, 2015; Wicherts et al., 2011) elucidated the importance of replication studies and reproducible practices. Replication studies are essential for research credibility, reducing uncertainties, and promoting theoretical maturation and generalization of scientific knowledge (Nosek et al., 2022; Nosek & Errington, 2020; Schmidt, 2009). Likewise, reproducible practices allow for evaluating studies’ integrity and data reuse (Hardwicke et al., 2021; Nosek et al., 2022; Nuijten et al., 2018). Therefore, we proposed a replication of Study 1 from the seminal BBS work (Pronin, Lin and Ross, 2002) to investigate the consistency of its initial hypothesis and provide further information on their generalizability. We seek to understand if Brazilian participants will also show BBS, rating their susceptibility for bias lower than their peers. For this, we will investigate the asymmetries proposed in the original study: (1) asymmetry of bias, accounting for the sum/average of eight biases; (2) asymmetry of shortcomings, accounting for the sum/average of three personal shortcomings and (3) the difference between those asymmetries. According to the findings from the original study, we hypothesize that participants will present asymmetries for biases, but not for shortcomings. Also, we expect to find a difference between asymmetries, with a higher difference for biases. Our results will also be compared with the results from the most recent replication of the original study to provide further insight into the phenomena.

A campaign was created on social media platforms and smartphone message application groups inviting people to voluntarily participate in the data collection for this study. Participants could access the survey through a QR code or a link displayed in the description. Inclusion criteria were (a) age greater than or equal to 18 years and (b) authorization granted through the Free and Informed Consent Form (ICF). This research followed the Helsinki Declaration and received approval from the University of São Paulo local ethics board (CAAE 44195121.7.0000.5482, approval number 5.092.791). Data collection took place online between February 2022 and May 2023.

This is a cross-sectional pre-registered, open data and open material replication of the Study 1 from Pronin, Lin and Ross (2002). The open-access material and codes from the replication of Chandrashekar et al. (2021) were used as a template for this study. Brazilian Portuguese versions of survey 1 and survey 2 used in Study 1 were created and administered as a single survey in the Qualtrics XM platform. The final version comprised an online survey presenting descriptions of eight biases and three personal shortcomings in the same order described in the original study, as follows. Biases: 1) self-serving attributions for success or failures; 2) self-interest; 3) reactive devaluation of the proposal from one’s negotiation counterparts; 4) the fundamental attribution error; 5) perceptions of hostile media bias toward one’s group or cause; 6) positive halo effect; 7) biased assimilation of new information and 8) cognitive dissonance reduction after free choice. Personal shortcomings: 1) procrastination; 2) fear of public speaking, and 3) the planning fallacy. Participants were first asked to indicate their own susceptibility for each description presented, then asked to indicate the susceptibility for “most of the people” for the same descriptions, in the same order. Responses were presented on a nine-point Likert scale, ranging from 1 to 9: points 1, 5, and 9 were labeled as “not at all”, “somewhat” and “strongly”, respectively. These procedures are similar to those used in Pronin, Lin and Ross (2002), except for the exchange of “average American/student” for “most of the people”, due to cultural adaptation and translation considerations and also not counterbalancing the order of the self/other susceptibility evaluation, once Pronin, Lin and Ross (2002) stated that no effects from the counterbalancing procedure were found to influence the results. A sociodemographic questionnaire to collect information on age, sex, education, family income, and area of knowledge was included before the questions described earlier.

Power analysis was conducted on the G*Power software (Faul et al., 2007) based on the procedures described in the replication study of Chandrashekar et al. (2021), using Cohen’s D value extracted from the original study . The values extracted by Chandrashekar et al. (2021) for each hypothesis were: (1) Bias asymmetry (d = 0.86); (2) Personal Shortcomings asymmetry (d = 0.28) and (3) Difference between asymmetries (d = 0.61). We adopted the same strategy as the authors and used the smallest sample size as a basis for the required sample (in this case, hypothesis 2 [d = 0.28]). Adopting the parameters alpha = 0.05 and 1 - beta = 0.95 for a two-tailed paired t-test, the initial sample size was n = 168. Considering that our study did not rely on monetary incentives for participation, we proposed a conservative approach to account for the dropout risks of voluntary participation, and determined the final sample size at n = 250. To avoid biases, the proposed stopping rule was to count only the values from the “finished” column, which indicated the participants who completed the questionnaire, until the minimum determined sample size was reached.

The raw dataset was exported on a .csv file from the Qualtrics XM platform and uploaded to the project’s OSF page. All variables were selected, with the exception of ‘email’ and ‘IP address’. Statistical analysis was performed using R (R Core Team, 2023), via R Studio software (R Core Team, 2023). A code was developed based on the work of (Woodhead et al., 2019) to download the dataset available on the OSF page to the R Studio environment, using the packages “osfr”, “utils”, “magrittr” and “tidyverse” (Bache & Wickham, 2022; R Core Team, 2023; Wickham et al., 2019; Wolen et al., 2020). Sociodemographic variables were re-coded from Portuguese back to English using the “dplyr” package (Wickham et al., 2023) and descriptive analysis through the “report” (Makowski et al., 2023) and “jmv” packages (Selker et al., 2022). Codebook was created using “codebookr” (Cannell, 2023). The overall mean scores of “self” and “other” for biases and personal shortcomings were created. The asymmetry (self-other discrepancy) for biases and personal shortcomings was obtained by subtracting the overall mean of “others” from the “self” evaluation (“self” minus “others”). Asymmetry discrepancy was obtained by subtracting personal shortcomings asymmetry from bias asymmetry. The main hypothesis and the difference between “self” and “others” of each bias and personal shortcoming were investigated using the two-tailed paired sample t-tests, through the package “stats” (R Core Team, 2023). Effect sizes were estimated using Cohen’s D, obtained using the “effsize” package (Torchiano, 2020). Plots were created for each analysis using the packages “reshape2”, “Rmisc”, ggpubr”, “ggstatsplot”, “ggplot2”, “patchwork” and “car” (Fox & Weisberg, 2019; Hope, 2022; Kassambara, 2022; Patil, 2021; Pedersen, 2022; Wickham, 2007, 2016).

The results obtained in our study will be compared with the two previous studies (Pronin, Lin and Ross, 2002; Chandrashekar et al., 2021) using LeBel’s criteria (LeBel, 2015; LeBel et al., 2019). This criteria establishes categories for the results of a replication study based on their effect size. The results can be categorized as “signal consistent” if the replication’s effect size confidence interval (95%) includes the original effect size point estimate and excludes zero. If the point estimate is not included, but zero remains excluded, then it is categorized as “signal inconsistent”, and it should be specified whether the effect size obtained is larger or smaller when compared to the original. There is also a chance for the replication’s effect size to display an opposite direction when compared to the original effect size. In this case, the replication results should be categorized as “opposite”.

The stopping rule was reached in April of 2023. A total of 436 participants responded to the questionnaire, of which 253 were complete answers. After filtering the data set for survey tests, bots, duplicates, complete consent forms, and minimum age, our final sample size comprised n = 203 participants. Of those, the mean age was 33 years (range [18 - 79], most were female (64.53%) and had not completed undergraduate studies (37.93%). Sixty-six (32.51%) had a monthly family income of more than 15 minimum wages. Of those currently or previously enrolled in tertiary education, 107 (52.71%) were from the area of Human Sciences. A full descriptive analysis of demographic characteristics is available in Supplementary Table S1.

Results for the first hypothesis (asymmetry of biases) are displayed in Table 2. A statistically significant difference with a large effect size was found in the average ratings of the eight biases when compared between self and others (t(202) = -23.74, p <.001). For the second hypothesis (asymmetry of personal shortcomings), a statistically significant difference with a small effect size was also found between self and others in the average ratings of all personal shortcomings (t(202) = -3.76, p <.001). As for the third hypothesis, the difference in the self-other asymmetry between biases and personal shortcomings was statistically significant, with a large effect size (t(202) = -10.66, p <.001). Violin plots for each hypothesis are provided in Figure 1.

All eight biases presented statistically significant differences between self and others, with effect sizes ranging from medium to large, except for cognitive dissonance, which had a small effect size (Table S2, Figure 2). On the other hand, among the personal shortcomings, only Planning Fallacy presented a statistically significant difference, with a medium effect size (Figure 3). Figures S1 and S2 in the Supplementary materials provide Violin plots for the self-other comparison between each bias and personal shortcoming.

Our results partially replicated the hypotheses of the original study. For hypothesis one, we found a significant difference between the susceptibilities for bias, suggesting that participants view others as more susceptible to bias than themselves, thus providing evidence for replicating the “bias asymmetry” hypothesis. For hypothesis two, we found a significant difference in the self-other susceptibilities for personal shortcomings, not replicating the same findings of Pronin, Lin and Ross (2002), in which the participant’s ratings of their own susceptibilities were higher than their ratings for others. For hypothesis three, our results showed a significant difference between asymmetries, in which the bias asymmetry was greater than the personal shortcomings asymmetry, also replicating the original findings (the comparison is summarized in Table 3).

We also compared our results with those obtained by Chandrashekar et al. (2021) replication. Table 4 summarizes the comparison using the data provided by their mini meta-analysis. A comprehensive comparison table of each study is presented in Tables S3 and S4. The bias asymmetry of our replication has obtained a larger effect size, although still supporting the original null hypothesis and following the same direction as the previous studies. The remaining asymmetries showed consistent replication. Noteworthy, the same failure to replicate hypothesis 2 (described in Pronin, Lin and Ross (2002) as the higher ratings for own susceptibility to personal shortcomings) was found across both replications, thus providing evidence for the consistency of this phenomenon. It is important to note, however, that both replications found a low effect size for this phenomenon.

The results of our replication study show that (1) participants rate themselves as less susceptible to bias in comparison to other people; (2) they rate themselves as less susceptible to personal shortcomings when compared to others, but with a smaller effect size when compared to bias ratings, and (3) the self-other asymmetry between bias and personal shortcomings are different. Next, each hypothesis is discussed separately.

Our results successfully corroborated the original hypothesis proposed and empirically demonstrated by Pronin, Lin and Ross (2002), gaining further empirical validity in the series of Chandrashekar et al. (2021) pre-registered replication studies. Since its original study, the bias blind spot (BBS) phenomenon has been consistently demonstrated, with large effect sizes, showing that people tend to rate themselves as less susceptible to bias than other people. Moreover, this tendency has been shown with undergraduate students from Stanford University comparing themselves with an “average American” as well as with an “average classmate”, and with travelers at San Francisco Airport comparing themselves with their fellow travelers (Pronin, Lin and Ross, 2002), gained further cross-cultural validity when it was replicated with undergraduate students from University of Hong Kong, and MTurk participants using a simple “you/others” prompt (Chandrashekar et al., 2021). Now, it has been further expanded, in a Brazilian sample comparing themselves with “most people”. Our analysis revealed a larger effect size compared with the original study and with Chandrashekar et al. (2021) replication. It is possible that this disparity may have been caused by the homogeneous profile of our sample. Another possible interpretation of this result rests on cultural differences of the compared samples, which requires future replication studies in Brazilian samples for further investigation

The hypothesis of the null effect in personal shortcomings asymmetry was partially corroborated. Pronin, Lin and Ross (2002) predicted that participants would be aware of their personal shortcomings during the survey due to cognitive availability, thus not producing any self-other asymmetry. The null effect was found in their results (p = .15), where the authors argued it provided “preliminary support” for their concept. A pre-registered direct replication of the original study was conducted, arguing that an underpowered sample may have jeopardized some of the original study analyses, such as the null findings on personal shortcomings (Chandrashekar et al., 2021). With an adequate sample size, their results showed that personal shortcomings display the same asymmetry found in bias ratings, but with a smaller effect size (d = –0.34, 95%CI [–0.46, –0.23] for personal shortcomings, and d = –1.00, 95%CI [–1.33, –0.67] for biases).

Our results revealed an overall asymmetry with a small effect size, remarkably close to the previous replication study (d = -0.33, 95%CI [-0.40, -0.12]), thus not supporting the predictions made for Hypothesis 2 and reproducing the results of the previous replication. However, when investigating the values for each personal shortcoming assessed (Figure 3, with values displayed in Table S2), only the “planning fallacy” shows significance for the self-other asymmetry, while “procrastination” and “fear of public speaking” do not. Thus, while the null effect predicted by Pronin, Lin and Ross (2002) is confirmed for two out of three shortcomings, it is absent when using the mean of this construct for comparing self-other asymmetry. Considering both situations, we conclude that Hypothesis 2 was partially replicated.

Our findings highlight the need for refinement in the methods used to assess the predictions of Hypothesis 2. Using three personal shortcomings may not adequately balance the comparison with the eight biases assessed. Future studies may benefit from including additional personal shortcomings to balance this comparison. Additionally, given the high replicability of the BBS, some biases may also be dropped out of the survey to give space for personal shortcomings assessments without jeopardizing Hypothesis 1 while maintaining the survey length.

For the third hypothesis, we found a significant difference between asymmetries with medium effect size, thus replicating the results from the original study. Contrary to the original claims, such difference seems not to be due to a lack of or the opposite effect of personal shortcomings. Instead, evidence suggests that this disparity results from a weaker effect size for personal shortcomings when compared to bias.

According to Nosek and Errington (2020), the process of theoretical maturation can be defined as the progressive path of commitments to the replicability of a certain theory, shaping its generalizability and expectations. Successful replications help clarify boundaries, generalizability space, and expectations improving maturation. Failures to replicate reduce generalizability and replicability spaces. Considering the results of this replication study and that of Chandrashekar et al. (2021), we argue that such a process has occurred, improving the confidence, generalizability, and replicability of the “Bias Blind Spot” phenomenon (Hypothesis 1), and its significant difference when compared to “Personal Shortcomings” (Hypothesis 3). For Hypothesis 2, our results corroborated with previous replication, shrinking its generalizability and expectations. Overall, we argue that the theoretical maturation process reasserts the confidence in the “Bias Blind Spot” (Pronin, Lin and Ross, 2002) at the same time guides future studies for uncovering theoretical mechanisms underlying Personal Shortcomings.

Our results must be interpreted considering the limitations of this study. First, we obtained a homogeneous sample of mostly female, university students, and with higher socio-economic level participants. Therefore, our results cannot be generalized either for the Brazilian population or other populations outside this range of characteristics. Second, the seminal article from Pronin, Lin, and Ross (2002) comprises three studies, each with a significant role in the development of the BBS theory. Due to methodological limitations and time restraints, we were able to provide evidence of the replicability and generalizability of one of those three studies, therefore, limiting our interpretations inside this scope.

Finally, we conclude that our results provide evidence of the theoretical maturation of the BBS effect (Pronin, Lin and Ross, 2002), as shown by further evidence of its replicability and generalizability. Furthermore, we found the same significance and small effect size for the asymmetry of personal shortcomings ratings previously found in the replication study of Chandrashekar et al. (2021), thus suggesting a phenomenon similar to BBS also operating for personal shortcomings. Future studies should be performed to further comprehend the underlying mechanisms behind this disparity of effect sizes. Also, replication studies of BBS in Brazil should focus on generalizing this phenomenon in heterogeneous samples, especially low-socioeconomic participants.

Contributed to conception and design: ITCM, TLRCL, MCB

Contributed to acquisition of data: LS, ITCM, TLRCL, MCB, DF

Contributed to analysis and interpretation of data: LS, MCB, DF

Drafted and/or revised the article: LS, MCB, DF

Approved the submitted version for publication: LS, ITCM, TLRCL, MCB, DF

The authors would like to thank Prof. Gilad Feldman for his support, comments, and suggestions for improvements in the manuscript. We also like to thank the CORE Team for their transparency and openness, facilitating our replication study. Finally, we’d like to thank all the members of the Replication Lab of Studies in Psychology (LAREPsi) in Sao Paulo, Brazil, for their hard work and collaboration.

This project was supported by FAPESP process number 2022/00342-8 (scholarship to Leonardo Seda)

The authors report no conflict of interest.

All material, data, and statistical analysis used are publicly available on the OSF page of our study (https://osf.io/c9d2a).

RMarkdown file: https://osf.io/namhs

Raw Dataset: https://osf.io/xha7d

Instrument: https://osf.io/9cbu4

Registration (Portuguese): https://doi.org/10.17605/OSF.IO/5BT9F

Registration, analysis plan, rationale, and deviations from protocol (English): https://osf.io/6hcyf