Shame and guilt are unpleasant self-conscious emotions associated with negative evaluations of oneself or one’s behavior. Smith et al. (2002) demonstrated that shame and guilt are distinct and are impacted differently by public exposure, that is, the (potential) exposure to disapproving appraisals of one’s misdeeds by others. The impact of public exposure (compared to no exposure) was greater for feelings of shame than for feelings of guilt. We conducted a direct replication (N = 1272) of Smith et al. (2002)’s Study 1 and found that exposure increased both feelings of shame (ηp2 = .14, 95%, CI [.11, .17]) and guilt (ηp2 = .13, 95% CI [.10, .16]) compared with the private condition. Moreover, people who were in the high moral conditions reported both higher shame (ηp2 = .33, 95% CI [.29, .37]) and guilt (ηp2 = .36, 95% CI [.32, .39]). Shame and guilt both had moderate-to-high correlations with the shame-related and guilt-related reactions and both exposure and moral belief manipulations had effects on shame-related and guilt-related reactions. Our results suggest a failed replication: public exposure and moral belief influence both shame and guilt, so we cannot conclude that shame and guilt can be distinguished from each other solely based on public exposure, which diverges from the target article’s main theory and findings. All materials, data, and code are available at https://osf.io/j3ue4/
Emotions are said to be the interface between an organism and its environment, mediating the ever-changing contexts and the behavioral responses of the organism (Scherer, 1984). In humans, emotions play a vital role in coordinating social interactions (Gilbert, 2004; Parrott, 2019), both signaling to ourselves how we feel about the experiences (Lazarus & Lazarus, 1994) and signaling our inner world to others (Van Kleef, 2009). The social emotions of shame and guilt have been linked to morality (Dempsey, 2017; Gilbert, 2003; Parrott, 2019; Teroni & Bruun, 2011) yet the theoretical explanations for distinguishing between the two emotional states are still in debate (Dempsey, 2017; Miceli & Castelfranchi, 2018).
Question . | Hypothesis . | Sampling plan . | Analysis Plan . | Rationale for deciding the sensitivity of the test for confirming or disconfirming the hypothesis . | Interpretation given different outcomes . | Theory that could be shown wrong by the outcomes . | Observed outcome (Added in Stage 2) . |
---|---|---|---|---|---|---|---|
Does public exposure affect the magnitude of shame and guilt over one’s misconduct? | Public exposure (implicit and explicit) increases experienced shame more than guilt over one’s misconduct compared with in private settings. Explicit public exposure has a stronger effect on shame than implicit exposure. | We followed the “small-telescope” appr oach (Simonsohn, 2015) to decide the sample size. Given N = 168 in Study 1 of Smith et al. (2002), the target sample was 420 (168× 2.5). However, considering there are 3 between-subject scenarios, we decided to recruit 420 participants for each scenario, which makes the total sample size being 1260. Accounting for possible exclusions, and allowing for the potential of additional analyses, we aimed for a larger total sample of 1350 participants. (75 per condition). | First, 3 (Publicity: private vs. implicit public vs. explicit public) × 2 (Moral belief: high vs. low) × 3 (Scenario: cheating vs. steal vs. disobey) between-subject ANOVAs to examine whether Scenario interacts with other experimental manipulations. If Scenarios interacted with other experimental manipulations, we would report the three-way ANOVAs and examine the effect of public exposure and moral belief for each scenario. If not, we will run 3 (Publicity: private vs. implicit public vs. explicit public) × 2 (Moral belief: high vs. low) two-way ANOVAs and report the possible main effects of public exposure and moral belief manipulation and possible interactions. To examine whether the effects of manipulations differ between the two emotions, we will also run mixed 3 (Public Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: Low vs. High) × 2 (Emotion: shame vs. guilt) ANOVA with Emotion being the within-subject variable. Planned contrasts will be conducted using the emmeans with “fdr” corrections. | We calculated the effect sizes for the primary results in the original article: the effect of public exposure on shame is f = .39 and the effect of moral belief on guilt is f = .27. Given the very large effects, it is likely that these are overestimations of the true effect size (Button et al., 2013; Halsey et al., 2015; Lakens, 2022), and – if true - we expected to find weaker effects for the primary analyses (around Cohen’s f = .20). The planned sample size can reliably detect at least am effect of f = .12 > .20. See Sample Size Planning for details. | Based on the criteria used by Lebel et al. (2019), we examine the replicability of findings from Smith et al. (2002). | Both hypotheses are not linked to a specific theory. However, the hypotheses are from the target article are argued as consistent with the functional approach of studying emotion from an evolutionary perspective. If the hypotheses were not supported, then this should prompt adjusting our priors regarding the original’s findings and the hypotheses they argue to support. | Exposure increased both feelings of shame (ηp2 = .14, 95%, CI [.11, .17]) and guilt (ηp2 = .13, 95% CI [.10, .16]). We conclude no support for differences between guilt and shame. |
Does stronger moral belief increase guilt and shame over one’s misconduct? | Moral belief impacts guilt over one’s misconduct more than shame. Higher moral beliefs result in higher ratings for shame. | People who were in the high moral conditions reported both higher shame (ηp2 = .33, 95% CI [.29, .37]) and guilt (ηp2 = .36, 95% CI [.32, .39]). We conclude no support for differences between guilt and shame. |
Question . | Hypothesis . | Sampling plan . | Analysis Plan . | Rationale for deciding the sensitivity of the test for confirming or disconfirming the hypothesis . | Interpretation given different outcomes . | Theory that could be shown wrong by the outcomes . | Observed outcome (Added in Stage 2) . |
---|---|---|---|---|---|---|---|
Does public exposure affect the magnitude of shame and guilt over one’s misconduct? | Public exposure (implicit and explicit) increases experienced shame more than guilt over one’s misconduct compared with in private settings. Explicit public exposure has a stronger effect on shame than implicit exposure. | We followed the “small-telescope” appr oach (Simonsohn, 2015) to decide the sample size. Given N = 168 in Study 1 of Smith et al. (2002), the target sample was 420 (168× 2.5). However, considering there are 3 between-subject scenarios, we decided to recruit 420 participants for each scenario, which makes the total sample size being 1260. Accounting for possible exclusions, and allowing for the potential of additional analyses, we aimed for a larger total sample of 1350 participants. (75 per condition). | First, 3 (Publicity: private vs. implicit public vs. explicit public) × 2 (Moral belief: high vs. low) × 3 (Scenario: cheating vs. steal vs. disobey) between-subject ANOVAs to examine whether Scenario interacts with other experimental manipulations. If Scenarios interacted with other experimental manipulations, we would report the three-way ANOVAs and examine the effect of public exposure and moral belief for each scenario. If not, we will run 3 (Publicity: private vs. implicit public vs. explicit public) × 2 (Moral belief: high vs. low) two-way ANOVAs and report the possible main effects of public exposure and moral belief manipulation and possible interactions. To examine whether the effects of manipulations differ between the two emotions, we will also run mixed 3 (Public Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: Low vs. High) × 2 (Emotion: shame vs. guilt) ANOVA with Emotion being the within-subject variable. Planned contrasts will be conducted using the emmeans with “fdr” corrections. | We calculated the effect sizes for the primary results in the original article: the effect of public exposure on shame is f = .39 and the effect of moral belief on guilt is f = .27. Given the very large effects, it is likely that these are overestimations of the true effect size (Button et al., 2013; Halsey et al., 2015; Lakens, 2022), and – if true - we expected to find weaker effects for the primary analyses (around Cohen’s f = .20). The planned sample size can reliably detect at least am effect of f = .12 > .20. See Sample Size Planning for details. | Based on the criteria used by Lebel et al. (2019), we examine the replicability of findings from Smith et al. (2002). | Both hypotheses are not linked to a specific theory. However, the hypotheses are from the target article are argued as consistent with the functional approach of studying emotion from an evolutionary perspective. If the hypotheses were not supported, then this should prompt adjusting our priors regarding the original’s findings and the hypotheses they argue to support. | Exposure increased both feelings of shame (ηp2 = .14, 95%, CI [.11, .17]) and guilt (ηp2 = .13, 95% CI [.10, .16]). We conclude no support for differences between guilt and shame. |
Does stronger moral belief increase guilt and shame over one’s misconduct? | Moral belief impacts guilt over one’s misconduct more than shame. Higher moral beliefs result in higher ratings for shame. | People who were in the high moral conditions reported both higher shame (ηp2 = .33, 95% CI [.29, .37]) and guilt (ηp2 = .36, 95% CI [.32, .39]). We conclude no support for differences between guilt and shame. |
The Public-private Distinction
One of the many propositions is the distinction made between private and public emotions. In the eyes of Plato and Aristotle, individuals tend to feel shame when faced with the possibility of a worse reputation, discredit, or disgrace, whereas individuals tend to feel guilt when they have committed a wrongful act (Aristotle, 1941; Plato, 1997). Thus, public exposure is more strongly associated with shame than with guilt. This distinction was also noticed by Darwin, who referred to the association between the shame felt over norm violations and judgments by fellow man (Darwin, 2008).
In this line of reasoning, shame is regarded as a moral emotion experienced when one’s faults are made public under others’ scrutiny and judgement. On the other hand, guilt is considered an emotion rooted primarily in self-directed negative evaluation and criticism (Combs et al., 2010). Unlike shame, which involves the discrepancy between the self and the social ideal self, guilt is primarily focused on the self’s moral conduct (Higgins, 1987). That is, in the case of shame, individuals perceive the self from others’ perspectives and expectations, whereas the guilty individual refers to their own standpoint or agency. It should be noted that this presence of the “other” could be actual, presumed, or imagined (Ausubel, 1955). Thus, shame is not solely dependent on an actual “other” explicitly exposing a transgression. Rather, it depends on whether or not an individual senses that there is a discrepancy between the present, and the social ideal self—the latter of which is formed based on others’ expectations of us (Cooley, 1902/1964). Put simply, it is the recognition of another’s dissatisfaction—presumed or otherwise—that causes the experience of shame (Ausubel, 1955; Higgins, 1987).
Humans evolved in social groups with status structures, where the fitness of the individual depends on their position in the group and the relations with their conspecifics. Shame and guilt can thus be distinguished by their function in solving the re-occurring adaptive problems in social lives.
Shame and guilt address two different kinds of challenges and originate from two distinct social motives. Shame is associated with being held in low esteem by other group members and motivates one to win their place within the group or to disengage to minimize the cost of conflicts. Whereas guilt originates from a care-providing system, which motivates one to avoid harming others and make reparations when there is perceived harm (Beall & Tracy, 2020; Gilbert, 2004). de Hooge (2014) proposed shame as a general sociometer monitoring the threat of being excluded from the group, which motivates affiliative behaviors in many circumstances. Therefore, public exposure of wrongdoing or unethical behavior can induce a greater sense of shame as it increases one’s risk of exclusion by the group but not guilt because the perceived harm of one’s act is not conditioned on public exposure.
The Self-behavior Distinction
The public vs. private distinction is not the only mainstream thought regarding the differentiation between shame and guilt. Another influential explanation posits that the object of negative evaluation (self vs. behavior) can differentiate shame and guilt (Tangney et al., 2007). While shame focuses on the negative evaluation of the global self (e.g., being a bad person), guilt is associated with the negative appraisal of the specific behavior (e.g., having done a bad thing). Previous research has found that when describing shame-inducing situations, participants expressed more concerns about negative evaluations of the self, compared with guilt-inducing situations. While the opposite holds for concerns about the effect on others (Tangney, 1994).
However, as expressed by Tangney et al. (2007), the two schools of thought need not be mutually exclusive. The salience of public exposure could shift the individual’s attention to the self and thus induce a greater feeling of shame. While in private, the individual may pay more attention to the effects of their behaviors on others. Other research has also pointed out that both accounts receive empirical support, and a new scale measuring shame and guilt-proneness (Guilt and Shame Proneness scale, GASP) has been developed taking into consideration arguments from both sides (Cohen et al., 2011). GASP measures guilt-proneness using negative behavior evaluations towards private transgressions and shame proneness using negative self-evaluations towards public transgressions.
In view of the debate over the two schools of thought, it is thus more important to ensure the empirical foundations of the theorizing are reliable and replicable, which we hope to contribute by replicating one of the classic findings: Smith et al. (2002).
Choice of Replication: Smith et al. (2002)
Smith et al. (2002) explored whether the level of public exposure could differently affect the levels of shame and guilt experienced over one’s transgression and found support for the public/private shame-guilt distinction. They found that public exposure (either implicit or explicit) was more strongly associated with shame than with guilt, compared to private situations.
Their findings had vast implications from theoretical developments of moral psychology to practical applications in pedagogy or the justice system. At the time of writing (March 2022), the target article has been cited 621 times (according to Google Scholar), with many impactful follow-ups, such as the development of GASP (Cohen et al., 2011) which measures shame and guilt by referring to reactions towards public and private transgressions or failures. The public-private distinction has also contributed to the theorizing of the relationship between morality and reputation (Sperber & Baumard, 2012) and considerations of the justice system reformations (Tangney et al., 2011).
We conducted a simple scientometric analysis of research articles on shame, guilt, and moral behavior for the last two decades (2001 -2022) in Scopus with the String: shame AND guilt AND (*moral* OR norm* OR *ethical) and the results of a total of 580 publications were visualized using VOSviewer (Waltman et al., 2010). As shown in Figure 1, Smith et al. (2002) is connected to many other highly influential papers in this research area, supporting the centrality of this work. However, to the best of our knowledge, despite its impact, there seem to be no published direct replications of Smith et al. (2002).
Note. The scale of the circles reflects the weights of each paper, which are determined by their connections with other publications in the network. The color gradient represents the number of citations (from 0 to 100).
Note. The scale of the circles reflects the weights of each paper, which are determined by their connections with other publications in the network. The color gradient represents the number of citations (from 0 to 100).
Hypotheses and findings in the target article
. | Hypotheses . |
---|---|
H1 | Exposure (private, implicit, and explicit) of moral (transgressions) and nonmoral (incompetence) experiences is more strongly associated with shame than with guilt. Higher public exposure of moral (transgressions) and nonmoral (incompetence) experiences results in stronger feelings of shame. (Page 141 Line 1 in the target article) |
H2 | Moral belief (low and high) is more strongly associated with guilt than with shame. Stronger moral belief is associated with stronger feelings of guilt. (Page 141 Line 3-4 in the target article) |
H3 | Exposure and moral beliefs [do not] interact on guilt. (Page 141 Line 15-18 in the target article). [Reframed from a null hypothesis of no interaction in the target] |
. | Hypotheses . |
---|---|
H1 | Exposure (private, implicit, and explicit) of moral (transgressions) and nonmoral (incompetence) experiences is more strongly associated with shame than with guilt. Higher public exposure of moral (transgressions) and nonmoral (incompetence) experiences results in stronger feelings of shame. (Page 141 Line 1 in the target article) |
H2 | Moral belief (low and high) is more strongly associated with guilt than with shame. Stronger moral belief is associated with stronger feelings of guilt. (Page 141 Line 3-4 in the target article) |
H3 | Exposure and moral beliefs [do not] interact on guilt. (Page 141 Line 15-18 in the target article). [Reframed from a null hypothesis of no interaction in the target] |
Note. These hypotheses are interpreted and summarized in our own wordings as the hypotheses are not explicitly stated in the original article. The target article did not specify a hypothesis regarding an interaction between exposure and moral beliefs for shame. Summary in the target article: “We expected moral beliefs to enhance guilt regardless of the public exposure of the transgression. In contrast, we expected public exposure to have relatively little effect on guilt, regardless of moral belief.”
Dependent Variables . | . | Independent Variables . | F statistics . | Effect(η2) . | CIL . | CIH . |
---|---|---|---|---|---|---|
Manipulation Check | “Judged by others” | Exposure | F (2, 156) = 9.64 | 0.11 | 0.03 | 0.21 |
“Violate a personal value” | Moral Beliefs | F (1, 156) = 16.79 | 0.10 | 0.03 | 0.20 | |
Exposure × Moral Beliefs | F (2, 156) = 2.97 | 0.04 | 0 | 0.11 | ||
Explicit Emotion | Shame | Exposure | F (2, 154) = 12.47 | 0.14 | 0.05 | 0.24 |
Moral Beliefs | F (1, 154) = 11.03 | 0.07 | 0.01 | 0.16 | ||
Exposure × Moral Beliefs | F (2, 154) = 3.15 | 0.04 | 0 | 0.11 | ||
Guilt | Moral Beliefs | F (1, 154) = 14.14 | 0.08 | 0.02 | 0.18 | |
Shame-related measures | Bodily Change | Exposure | F (2, 156) = 14.47 | 0.16 | 0.06 | 0.26 |
Moral Beliefs | F (1, 156) = 4.26 | 0.03 | 0 | 0.10 | ||
Exposure × Moral Beliefs | F (2, 156) = 5.49 | 0.07 | 0.01 | 0.15 | ||
Desire to Escape | Exposure | F (2, 155) = 14.35 | 0.16 | 0.06 | 0.26 | |
Exposure × Moral Beliefs | F (2, 155) = 3.63 | 0.04 | 0 | 0.12 | ||
Embarrassment | Exposure | F (2, 155) = 4.71 | 0.06 | 0.003 | 0.14 | |
Humiliated | Exposure | F (2, 155) = 4.71 | 0.06 | 0.003 | 0.14 | |
Self-directed anger | Exposure | F (2, 155) = 6.89 | 0.08 | 0.01 | 0.17 | |
Moral Beliefs | F (1, 155) = 11.31 | 0.07 | 0.01 | 0.16 | ||
Exposure × Moral Beliefs | F (2, 155) = 4.42 | 0.05 | 0.002 | 0.13 | ||
Guilt-related measures | Inwardly guilty feelings | Moral Beliefs | F (1, 156) = 8.20 | 0.05 | 0.005 | 0.13 |
Hurt others | Exposure | F (2, 155) = 21.46 | 0.22 | 0.11 | 0.33 | |
Moral Beliefs | F (1, 155) = 4.20 | 0.03 | 0 | 0.10 | ||
Emotion Intensity | Emotion (Shame vs. Guilt) | F (1, 154) = 26.51 | 0.15 | 0.06 | 0.26 | |
Emotion × Exposure | F (1, 154) = 10.17 | 0.06 | 0.01 | 0.15 | ||
Emotion × Exposure× Moral Beliefs | F (1, 154) = 3.31 | 0.02 | 0 | 0.09 |
Dependent Variables . | . | Independent Variables . | F statistics . | Effect(η2) . | CIL . | CIH . |
---|---|---|---|---|---|---|
Manipulation Check | “Judged by others” | Exposure | F (2, 156) = 9.64 | 0.11 | 0.03 | 0.21 |
“Violate a personal value” | Moral Beliefs | F (1, 156) = 16.79 | 0.10 | 0.03 | 0.20 | |
Exposure × Moral Beliefs | F (2, 156) = 2.97 | 0.04 | 0 | 0.11 | ||
Explicit Emotion | Shame | Exposure | F (2, 154) = 12.47 | 0.14 | 0.05 | 0.24 |
Moral Beliefs | F (1, 154) = 11.03 | 0.07 | 0.01 | 0.16 | ||
Exposure × Moral Beliefs | F (2, 154) = 3.15 | 0.04 | 0 | 0.11 | ||
Guilt | Moral Beliefs | F (1, 154) = 14.14 | 0.08 | 0.02 | 0.18 | |
Shame-related measures | Bodily Change | Exposure | F (2, 156) = 14.47 | 0.16 | 0.06 | 0.26 |
Moral Beliefs | F (1, 156) = 4.26 | 0.03 | 0 | 0.10 | ||
Exposure × Moral Beliefs | F (2, 156) = 5.49 | 0.07 | 0.01 | 0.15 | ||
Desire to Escape | Exposure | F (2, 155) = 14.35 | 0.16 | 0.06 | 0.26 | |
Exposure × Moral Beliefs | F (2, 155) = 3.63 | 0.04 | 0 | 0.12 | ||
Embarrassment | Exposure | F (2, 155) = 4.71 | 0.06 | 0.003 | 0.14 | |
Humiliated | Exposure | F (2, 155) = 4.71 | 0.06 | 0.003 | 0.14 | |
Self-directed anger | Exposure | F (2, 155) = 6.89 | 0.08 | 0.01 | 0.17 | |
Moral Beliefs | F (1, 155) = 11.31 | 0.07 | 0.01 | 0.16 | ||
Exposure × Moral Beliefs | F (2, 155) = 4.42 | 0.05 | 0.002 | 0.13 | ||
Guilt-related measures | Inwardly guilty feelings | Moral Beliefs | F (1, 156) = 8.20 | 0.05 | 0.005 | 0.13 |
Hurt others | Exposure | F (2, 155) = 21.46 | 0.22 | 0.11 | 0.33 | |
Moral Beliefs | F (1, 155) = 4.20 | 0.03 | 0 | 0.10 | ||
Emotion Intensity | Emotion (Shame vs. Guilt) | F (1, 154) = 26.51 | 0.15 | 0.06 | 0.26 | |
Emotion × Exposure | F (1, 154) = 10.17 | 0.06 | 0.01 | 0.15 | ||
Emotion × Exposure× Moral Beliefs | F (1, 154) = 3.31 | 0.02 | 0 | 0.09 |
Note. Only supported findings are included. The effect sizes were calculated using the MOTE package (Buchanan et al., 2017) in R. Scripts are provided on the OSF.
Replication overview
In view of its impact and the absence of direct replications, we embarked on well-powered close replication of Study 1 in Smith et al. (2002). We chose Study 1 as the most comprehensive well-controlled baseline experimental demonstration of the effect, using simplified clear vignettes that are well-suited for online administration and our target sample. The experimental design of the target’s Study 1 was a 3 (Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: high vs. low) × 3 (Scenario: cheating vs. steal vs. disobey) × 2 (Gender: male vs. female) between-subject design. The factor Scenario was dropped from the analyses in Smith et al. (2002) as it produced no systematic effects on any dependent variables while Gender was included in the ANOVAs. The original article reported no interaction effect between Gender and Exposure or Moral belief, and gender did not seem to have any theoretical importance. Therefore, in the current replication, we collapsed the two genders (male and female) and did not include the factor Gender in the analyses.
Open Science Declaration
We provided all materials, data, and code on: https://osf.io/j3ue4/. This project received Peer Community in Registered Report Stage 1 in-principle acceptance (https://osf.io/j7kt2/; https://rr.peercommunityin.org/articles/rec?id=180) after which we created a frozen pre-registration version of the entire Stage 1 packet (https://osf.io/js5db/) and proceeded to data collection. All measures, manipulations, and exclusions conducted for this investigation are reported, and data collection was completed before analyses.
Method
Sample size plan
We calculated the target article’s η2 effect sizes (Table 2). Our calculations indicated that the effect of exposure on explicit shame was η2 = 0.14 and the effect of moral belief on explicit shame was η2 = 0.08. To make the effect sizes comparable with sensitivity analyses results (see below), we also computed the effect sizes in terms of Cohen’s f using Webpower (https://webpower.psychstat.org/models/means03/effectsize.php). The effect of exposure on shame is f = .39 and the effect of moral belief on guilt is f = .27 (for protocols see the “Additional analyses and results” section in supplementary materials). Given the very large effects, it is likely that these are overestimations of the true effect size (Button et al., 2013; Halsey et al., 2015), and – if true - we expected to find weaker effects for the primary analyses (around Cohen’s f = .20).
The “small-telescope” approach for replications (Simonsohn, 2015) proposed aiming for enough power to detect effects much weaker than those reported by the original study (d33%) with a general rule of thumb that a simplified design replication should employ a sample size 2.5 times of the original. We followed this generalized approach, even if meant for other designs, and given N = 168 in Study 1 of Smith et al. (2002) the target sample was 420 (168× 2.5), per each of the 3 between-subject scenarios, overall, 1260. Accounting for possible exclusions, and allowing for the potential of additional analyses, we aimed for a larger total sample of 1350 participants (75 per condition).
A sensitivity analysis with GPower (Faul et al., 2007) indicated that a sample of 1260 would allow the detection of f = 0.12 (groups = 18, df = 4; 3 exposure × 2 moral belief × 3 scenarios, between-subject ANOVA) and d = 0.23/ 0.19 for any contrasts between two conditions in collapsed main effects (independent samples, n = 420/630) (both: 95% power, alpha = 5%, one-tail), effects much weaker than any of the supported effects reported in the target article and considered weak in social psychology literature. We also ran sensitivity analysis for the two-way interactions (Exposure × Emotion and Moral belief × Emotion) in the mixed 3 (Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: Low vs. High) × 2 (Emotion: shame vs. guilt) ANOVA using Morepower (Campbell & Thompson, 2012). Results showed that a sample of 1260 would allow the detection of f = 0.10 with a power of 95% and alpha of 5%. The protocols are available in the “Additional analyses and results” section in supplementary materials. Overall, our target sample size was 8 times that of the original.
Participants
We recruited 1309 American participants from Amazon Mechanical Turk via CloudResearch (Litman et al., 2017). The final sample of those who completed the survey (N = 1272) consisted of 599 male participants (47.1%) and 673 female participants (52.9%) after exclusion. The average age of the sample was 44.9 (SD = 13.1), ranging from 20 to 84.
We limited participation using the following criteria: (1) over 18 years old, (2) born, raised, and residing in the US, and (3) a native English speaker. Based on previous experience of running similar replications on MTurk, to ensure high-quality data collection, we employed the following CloudResearch options: Duplicate IP Block. Duplicate Geocode Block, Suspicious Geocode Block, Verify Worker Country Location, Enhanced Privacy, CloudResearch Approved Participants, Block Low-Quality Participants, etc. We also employed the Qualtrics fraud and spam prevention measures: reCAPTCHA, prevent multiple submissions, prevent ballot stuffing, bot detection, security scan monitor, and relevantID.
Assignment pay was based on the federal wage of 7.25 USD/hour. We first pretested survey duration with 30 participants to make sure our time run estimate was accurate and then adjust pay as needed, the data of the 30 participants were not analyzed separately from the rest of the sample other than to assess survey completion duration and needed pay adjustments. For those pretest participants, if survey durations were longer than expected, they would be paid a bonus as pay adjustment. The average compensation for participation in this study was $0.7.
Procedure and materials
The overall procedure of Study 1 was as follows: Participants first read the consent form, which includes the following information: the purpose of the study, procedures, risks and benefits, compensation, confidentiality, and voluntary participation. Then they had to confirm that they were over 18 years old and gave informed consent. Then, they read a short introduction of the study and instructions for imagining the scenarios, followed by several questions to ensure that they were eligible and willing to participate in the research. Next, participants indicated their gender (Options: male, female, other, and rather not disclose), and based on their gender participants who chose either male or female were randomly assigned to one of the eighteen conditions matching their gender. The three hypothetical transgression scenarios were: (1) cheating on a lab report (plagiarize), 2) stealing (steal), and 3) disobeying parents (disobey). We informed participants that participation is limited to those who self-identify as male or female, given the gendered replication study materials, and those who did not indicate male or female were asked to return the HIT.
We used the original materials described in the target article for the stealing and cheating scenarios (summarized in the supplementary materials Table 2) yet had to reconstruct the materials for the disobeying account ourselves, as these were not provided in the original study (summarized in the supplementary materials Table 5).
Following Smith et al. (2002), moral belief manipulation was achieved by including information about the hypothetical individuals’ self-view of their characters before the wrongful act. For example, in the cheating scenario, high moral belief was manipulated by presenting that Julia [Jason] knows it would be very wrong to take the lab report, but, because of her [his] desperation, she [he] takes it. In the low moral belief condition, it was presented that she [he] resents that many of the students have an unfair advantage over her [him] and thinks that, given the high stakes, it would not harm anyone for her [him] to copy the report. For exposure manipulation, after reading about the transgression, participants read either that the individual in the account either came across or was reminded of someone who would not approve of his or her action (the implicit condition) or that the individual’s transgression is actually discovered by someone who would not approve of his or her action (the explicit condition). In the private condition, no such information was provided (for details, see Table 2 and Table 5 in supplementary materials).
Different from Smith et al. (2002), after reading the scenario, participants in all conditions first answered two comprehension questions (e.g., “Did Julia consider it wrong to take the report?” and “Did Juila see anyone nearby right after taking the report?”). Only after having chosen the correct answer for both comprehension questions, participants were reminded of the scenario and presented with a set of items depicting feelings or thoughts and indicated the degree to which they thought the item was characteristic of the individual’s experience over the transgression act (0 = Not at all characteristic; 9 = Extremely characteristic).
The main dependent measures were two manipulation checks for moral belief (“violated a personal value”) and exposure (“judged by others”) respectively as well as two explicit measures of emotions of focus (“shame” and “guilt”). Following Smith et al. (2002), we also included the measures for shame-related reactions and guilt-related measures (see Table 3 for the categorization and specific items).
IV1: Exposure (3 between) IV2: Moral belief (2 between) IV3: Scenarios (1) cheating on a lab report, (2) stealing, and (3) disobeying parents | IV1: Private condition E.g., Jody took some candy without anyone noticing | IV1: Implicit public exposure condition E.g., Jody has hidden the M&Ms and is sure that nobody has seen her take the candy, she sees her boss from a distance | IV1: Explicit public exposure condition Jody realizes that her boss had been watching her as she took the candy |
IV2: Low moral belief
(incompetence) E.g., Jody sees herself as a fairly honest person, but she does not see anything wrong with taking a little candy now and then | Manipulation checks:
Dependent variables: 0 (Not at all characteristic) to 9 (Extremely characteristic) ShameExplicit measure of shameShame-related implicit measures:
GuiltExplicit measure of guilt:Guilt-related implicit measures:
| ||
IV2: High moral belief
(transgression) E.g., Jody sees herself as a very honest person and believes that stealing is wrong but succumbs to the temptation to take some candy without paying for it |
IV1: Exposure (3 between) IV2: Moral belief (2 between) IV3: Scenarios (1) cheating on a lab report, (2) stealing, and (3) disobeying parents | IV1: Private condition E.g., Jody took some candy without anyone noticing | IV1: Implicit public exposure condition E.g., Jody has hidden the M&Ms and is sure that nobody has seen her take the candy, she sees her boss from a distance | IV1: Explicit public exposure condition Jody realizes that her boss had been watching her as she took the candy |
IV2: Low moral belief
(incompetence) E.g., Jody sees herself as a fairly honest person, but she does not see anything wrong with taking a little candy now and then | Manipulation checks:
Dependent variables: 0 (Not at all characteristic) to 9 (Extremely characteristic) ShameExplicit measure of shameShame-related implicit measures:
GuiltExplicit measure of guilt:Guilt-related implicit measures:
| ||
IV2: High moral belief
(transgression) E.g., Jody sees herself as a very honest person and believes that stealing is wrong but succumbs to the temptation to take some candy without paying for it |
Note. The table only details the two IVs that are central to the conclusions drawn from the original study.
Upon completion, participants answered a number of funneling questions about the purpose of the study as well as whether they had participated in similar studies before. Then, they answered demographic questions about age, country of origin, country of residence, social class, and English proficiency regarding the experiment materials (see Table 8 in supplementary materials). Finally, participants were debriefed about the detailed purpose of the study and compensated for their participation.
Deviations
We provided a list of deviations between the original study and the present replication in Table 9 in the supplementary materials.
Replication closeness evaluation
Design facet | Replication | Details of deviation |
Effect/hypothesis | Same | |
IV construct | Same | |
DV construct | Same | |
IV operationalization | Same | |
DV operationalization | Same | |
IV stimuli | Similar | Only 2 out of 3 scenarios were presented in the original, we reconstructed scenario 3 “Disobeying Parents”, with close consideration of the tone and wording of the original article. |
DV stimuli | Same | |
Procedural details | Similar | We specified several exclusion criteria to ensure data quality and make the sample more comparable to the original study. We added comprehension check questions before the scales measuring emotions. Originally, participants were compensated with course credit, replication participants received monetary rewards. |
Physical settings | Different | Replication was conducted online individually; the original experiment was carried out in person in groups of 30. |
Contextual variables | Different | The original study was conducted in 2002; replication was done in 2022 during the COVID-19 pandemic. The considerable time gap and pandemic may cause contextual variations. |
Population (e.g., age) | Different | The original experiment focused on university students (Age 18-22), Age range for replication was 18 and above. |
Replication classification | Close |
Design facet | Replication | Details of deviation |
Effect/hypothesis | Same | |
IV construct | Same | |
DV construct | Same | |
IV operationalization | Same | |
DV operationalization | Same | |
IV stimuli | Similar | Only 2 out of 3 scenarios were presented in the original, we reconstructed scenario 3 “Disobeying Parents”, with close consideration of the tone and wording of the original article. |
DV stimuli | Same | |
Procedural details | Similar | We specified several exclusion criteria to ensure data quality and make the sample more comparable to the original study. We added comprehension check questions before the scales measuring emotions. Originally, participants were compensated with course credit, replication participants received monetary rewards. |
Physical settings | Different | Replication was conducted online individually; the original experiment was carried out in person in groups of 30. |
Contextual variables | Different | The original study was conducted in 2002; replication was done in 2022 during the COVID-19 pandemic. The considerable time gap and pandemic may cause contextual variations. |
Population (e.g., age) | Different | The original experiment focused on university students (Age 18-22), Age range for replication was 18 and above. |
Replication classification | Close |
Data Analysis Plan
Overview
We conducted all data analyses in R (version 4. 1.2, R core team, 2021). We employed Null Hypothesis Significance Testing (NHST) to examine the hypotheses and exploratory relationships using the afex package (Singmann et al., 2015). We calculated the 95% CIs of ηp2 with the effectsize package (Ben-Shachar et al., 2020) and 95% CIs of Cohen’s d in planned contrasts and post-hocs with emmeans package (Lenth, 2022).
Outliers and exclusions
Our generalized exclusion criteria are detailed in the “exclusion criteria” subsection of supplementary materials. Note that the original article did not report any criteria or operations for exclusions. Therefore, there is a possibility that the exclusion process will constitute a deviation from the original study. However, most of the exclusion criteria we employed were to make the sample more comparable to the original sample and ensure data quality, which makes it unlikely to be a consequential deviation.
Confirmatory analyses
In the original study, the authors did not report the correlations between explicit emotion measures and the corresponding emotion-related reactions in Study 1. Although similar analyses were done in Study 4 of the original article, the measures were different. Therefore, after performing internal consistency analyses (i.e., Cronbach’s Alpha) following the original study, we ran and reported the Pearson correlations between explicit shame and shame-related reactions as well as the correlations between explicit guilt and guilt-related reactions to establish the validities of the measures.
In addition, considering that in this replication we created a scenario on our own, to ensure it is justified to collapse the three scenarios in the following reporting, we first ran 3 (Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: high vs. low) × 3 (Scenario: cheating vs. steal vs. disobey) between-subject ANOVAs for all the dependent variables to rule out that there were interaction effects between Scenario and Exposure/ Moral belief. We found that Scenario did interact with experimental manipulations, indicating that the effects of experimental manipulations differed across scenarios. However, across scenarios, the effects of exposure and moral beliefs on explicit emotions and/or emotion-related reactions were similar and differed only in magnitude, indicating that collapsing three scenarios would not result in misinterpretation of the data. To compare the replication with the target article efficiently, we decided to first report the results of two-way ANOVAs without including Scenario as a factor. Then we briefly summarized the effects in different scenarios (in subsections titled “Exploratory: Scenario interactions”) and provided the details in supplementary materials.
Main analysis: Explicit guilt and shame
To test the focal hypotheses that public exposure is associated with elevated levels of shame more than guilt experienced over one’s transgression and that high moral belief is associated more with an elevated level of guilt than shame, we ran two types of analyses. First, we ran 3 (Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: Low vs. High) between-subject two-way ANOVAs to examine the effects of exposure and moral beliefs on the level of reported explicit shame and guilt, respectively in these three situations. Second, we also transformed the data into long format and did a mixed 3 (Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: Low vs. High) × 2 (Emotion: shame vs. guilt) ANOVA following Smith et al. (2002). This was done to examine whether the effects of manipulations differ between the two emotions (i.e., the two-way interaction between exposure and emotion and the two-way interaction between moral belief and emotion).
Secondary analysis: Guilt and shame reactions
Next, following the original study, we examined the effect of exposure and moral belief on the level of reported shame-related reactions as well as guilt-related reactions with a series of 3 (Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: Low vs. High) between-subject two-way ANOVAs.
Evaluation criteria for replication findings
We aimed to compare the replication effects with the original effects using the criteria set by LeBel et al. (2019), where we provided a simplified replication taxonomy based on comparing the CI of our replication effects with that of the original article’s effects.
Results
Manipulation checks
Manipulations were successful. We conducted a two-way ANOVA and found that for the exposure check, there was a main effect of exposure, F (2, 1266) = 295.94, p <.001, ηp2 = .32, 95% CI [.28, .36] and a main effect of moral belief, F (1, 1266) = 49.94, p <.001, ηp2 = .04, 95% CI [.02, .06]. We did not find support for the interaction effect between moral belief and exposure manipulation, F (2, 1266) = 2.78, p =.062, ηp2 = .004, 95% CI [.00, .01]. As expected, our Tukey post-hoc pairwise comparisons showed that the average score of exposure check was the highest in the explicit public condition (M = 6.84, SD = 2.08), followed by the implicit public condition (M = 3.75, SD = 2.78), p <.001, which in turn was higher than the private condition (M = 2.86, SD = 2.71), p <.001. As for the manipulation of moral belief, participants in the high moral belief condition (M = 4.95, SD = 2.91) reported higher scores than those in the low moral belief condition (M = 3.99, SD = 3.13), p <.001. This suggested an association between exposure and moral beliefs.
For the moral belief manipulation check, we found support for a main effect of moral belief, F (1, 1266) = 580.59, p <.001, ηp2 = .31, 95% CI [.28, .35], a main effect of exposure, F (2, 1266) = 26.12, p <.001, ηp2 = .04, 95% CI [.02, .06], and an interaction effect between moral belief and exposure manipulation, F (2, 1266) = 9.24, p <.001, ηp2 = .01, 95% CI [.00, .03]. As expected, participants in the high moral belief condition (M =6.81, SD = 2.28) reported higher scores than those in the low moral belief condition (M = 3.34, SD = 2.93), p <.001. Our Tukey’s HSD tests showed that participants in the explicit public condition reported higher scores (M = 5.75, SD = 2.86) than participants in the implicit public condition (M = 4.94, SD = 3.19), p <.001, which in turn was nominally higher than the private condition (M = 4.51, SD = 3.26), p = .108. The effect of the moral belief manipulation was stronger in the private condition (ηp2 = .41, 95% CI [.34, .47]), followed by the implicit public condition (ηp2 = .32, 95% CI [.25, .38]), which in turn was stronger than the public exposure condition (ηp2 = .22, 95% CI [.15, .28]).
Exploratory: Scenario interactions
We supplemented these analyses with three-way ANOVAs that included Scenario as a factor and found support for a two-way interaction between Scenario and exposure manipulation on the exposure manipulation check, a two-way interaction between Scenario and exposure manipulation on moral belief check, and a two-way interaction between Scenario and moral belief manipulation on moral belief check. We provided estimated marginal means for each condition as well as the estimated marginal means collapsing the three scenarios in Table 5.
. | Conditions . | Plagiarize scenario . | Steal scenario . | Disobey scenario . | Overall . | |||||
---|---|---|---|---|---|---|---|---|---|---|
Check | Exposure | Moral Belief | Emmeans | 95%CI | Emmeans | 95%CI | Emmeans | 95%CI | Emmeans | 95%CI |
Exposure | Private | High | 3.93 | [3.36, 4.50] | 2.80 | [2.22, 3.38] | 3.93 | [3.35, 4.51] | 3.55 | [3.22, 3.89] |
Implicit | High | 3.93 | [3.37, 4.50] | 4.49 | [3.91, 5.07] | 4.37 | [3.82, 4.93] | 4.26 | [3.93, 4.59] | |
Explicit | High | 7.05 | [6.44, 7.65] | 7.76 | [7.20, 8.32] | 6.48 | [5.88, 7.08] | 7.13 | [6.78, 7.47] | |
Private | Low | 2.89 | [2.31, 3.46] | 1.34 | [0.78, 1.91] | 2.38 | [1.82, 2.94] | 2.19 | [1.86, 2.53] | |
Implicit | Low | 3.19 | [2.61, 3.76] | 3.42 | [2.85, 4.00] | 3.03 | [2.44, 3.62] | 3.22 | [2.88, 3.55] | |
Explicit | Low | 6.48 | [5.91, 7.05] | 7.17 | [6.60, 7.74] | 6.06 | [5.48, 6.63] | 6.57 | [6.24, 6.90] | |
Moral belief | Private | High | 7.07 | [6.49, 7.65] | 7.21 | [6.63, 7.80] | 5.54 | [4.95, 6.13] | 6.62 | [6.27, 6.96] |
Implicit | High | 7.03 | [6.45, 7.60] | 7.42 | [6.83, 8.01] | 5.73 | [5.17, 6.30] | 6.71 | [6.36, 7.05] | |
Explicit | High | 7.39 | [6.78, 8.00] | 7.99 | [7.42, 8.55] | 5.82 | [5.21, 6.42] | 7.11 | [6.76, 7.46] | |
Private | Low | 3.19 | [2.60, 3.77] | 2.07 | [1.50, 2.64] | 2.18 | [1.61, 2.74] | 2.47 | [2.12, 2.81] | |
Implicit | Low | 3.73 | [3.14, 4.31] | 2.82 | [2.24, 3.40] | 2.78 | [2.19, 3.37] | 3.11 | [2.76, 3.46] | |
Explicit | Low | 4.24 | [3.66, 4.82] | 4.82 | [4.24, 5.40] | 4.31 | [3.73, 4.89] | 4.46 | [4.11, 4.80] |
. | Conditions . | Plagiarize scenario . | Steal scenario . | Disobey scenario . | Overall . | |||||
---|---|---|---|---|---|---|---|---|---|---|
Check | Exposure | Moral Belief | Emmeans | 95%CI | Emmeans | 95%CI | Emmeans | 95%CI | Emmeans | 95%CI |
Exposure | Private | High | 3.93 | [3.36, 4.50] | 2.80 | [2.22, 3.38] | 3.93 | [3.35, 4.51] | 3.55 | [3.22, 3.89] |
Implicit | High | 3.93 | [3.37, 4.50] | 4.49 | [3.91, 5.07] | 4.37 | [3.82, 4.93] | 4.26 | [3.93, 4.59] | |
Explicit | High | 7.05 | [6.44, 7.65] | 7.76 | [7.20, 8.32] | 6.48 | [5.88, 7.08] | 7.13 | [6.78, 7.47] | |
Private | Low | 2.89 | [2.31, 3.46] | 1.34 | [0.78, 1.91] | 2.38 | [1.82, 2.94] | 2.19 | [1.86, 2.53] | |
Implicit | Low | 3.19 | [2.61, 3.76] | 3.42 | [2.85, 4.00] | 3.03 | [2.44, 3.62] | 3.22 | [2.88, 3.55] | |
Explicit | Low | 6.48 | [5.91, 7.05] | 7.17 | [6.60, 7.74] | 6.06 | [5.48, 6.63] | 6.57 | [6.24, 6.90] | |
Moral belief | Private | High | 7.07 | [6.49, 7.65] | 7.21 | [6.63, 7.80] | 5.54 | [4.95, 6.13] | 6.62 | [6.27, 6.96] |
Implicit | High | 7.03 | [6.45, 7.60] | 7.42 | [6.83, 8.01] | 5.73 | [5.17, 6.30] | 6.71 | [6.36, 7.05] | |
Explicit | High | 7.39 | [6.78, 8.00] | 7.99 | [7.42, 8.55] | 5.82 | [5.21, 6.42] | 7.11 | [6.76, 7.46] | |
Private | Low | 3.19 | [2.60, 3.77] | 2.07 | [1.50, 2.64] | 2.18 | [1.61, 2.74] | 2.47 | [2.12, 2.81] | |
Implicit | Low | 3.73 | [3.14, 4.31] | 2.82 | [2.24, 3.40] | 2.78 | [2.19, 3.37] | 3.11 | [2.76, 3.46] | |
Explicit | Low | 4.24 | [3.66, 4.82] | 4.82 | [4.24, 5.40] | 4.31 | [3.73, 4.89] | 4.46 | [4.11, 4.80] |
This is suggestive of differences between the various scenarios, though we caution against over-interpreting these exploratory findings. The over-arching pattern was that 1) compared with private and implicit public conditions, participants in the explicit public conditions agreed more with “being judged by others”; 2) compared with low moral belief conditions, participants in the high moral belief conditions agreed more with the statement “violating a personal value”; and 3) in the high moral belief conditions, participants in the private, implicit public, and explicit public conditions, unanimously showed high agreement with the statement “violating a personal value”.
Explicit shame and guilt: Impact of exposure and moral beliefs
To examine whether exposure and moral belief manipulations had an impact on explicit shame and guilt, we performed a 3 (Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: High vs. low) two-way ANOVAs for the two explicit measures of emotion (see Figure 2).
For explicit shame, we found support for a main effect of exposure, F (2, 1266) = 90.69, p < .001, ηp2 = .13, 95% CI [.09, .16], a main effect of moral belief, F (1, 1266) = 549.17, p < .001, ηp2 = .30, 95% CI [.26, .34], and an interaction effect between exposure and moral belief, F (2, 1266) = 15.90, p < .001, ηp2 = .02, 95% CI [.01, .04]. We conducted Tukey’s HSD tests and found that explicit shame in the explicit public condition (M = 6.11, SD = 2.73) was higher than that in the implicit public condition (M = 4.51, SD = 3.10), p <.001, which was higher than the explicit shame in the private condition (M = 3.89, SD = 3.15), p = .007.
As for the effect of moral belief manipulation on explicit shame, participants in the high moral belief conditions (M = 6.47, SD = 2.34) reported higher explicit shame than participants in the low moral belief conditions (M = 3.20, SD = 2.99), p <.001. The main effect of exposure on explicit shame was present in both high moral belief conditions, F (2, 629) = 17.91, p < .001, ηp2 = .05, 95% CI [.02, .09], and low moral belief conditions, F (2, 637) = 80.31, p < .001, ηp2 = .20, 95% CI [.15, .25], with the effect of exposure being larger in the low moral belief conditions.
We found highly similar results for our analyses for explicit guilt. We found support for a main effect of exposure, F(2, 1266) = 88.59, p < .001, ηp2 = .12, 95% CI [.09, .16], a main effect of moral belief, F(1, 1266) = 639.33, p < .001, ηp2 = .34, 95% CI [.30, .37], and an interaction between exposure and moral belief, F (2, 1266) = 19.73, p < .001, ηp2 = .03, 95% CI [.01, .05]. Explicit guilt in the explicit public condition (M = 6.26, SD = 2.75) was higher than that in the implicit public condition (M = 4.77, SD = 3.12), p<.001, which was higher than the explicit guilt in the private condition (M = 4.08, SD = 3.18), p = .003. Explicit guilt in the high moral belief conditions (M = 6.76, SD = 2.18) was higher than the explicit guilt in the low moral belief conditions (M = 3.31, SD = 3.03), p <.001. The main effect of exposure on explicit guilt was present in both high moral belief conditions, F (2, 629) = 15.94, p < .001, ηp2 = .05, 95% CI [.02, .08], and low moral belief conditions, F (2, 637) = 79.14, p < .001, ηp2 = .20, 95% CI [.15, .25], with the effect of exposure being larger in the low moral belief conditions.
Deviating from our expectations and the target’s findings, the analyses on explicit shame and guilt had very similar results not just in the presence of but also the magnitudes of the effects. We summarized the estimated marginal means for each condition in Table 6.
. | Exposure . | Moral Belief . | Plagiarize scenario . | Steal scenario . | Disobey scenario . | Overall . | ||||
---|---|---|---|---|---|---|---|---|---|---|
DV | Emmeans | 95%CI | Emmeans | 95%CI | Emmeans | 95%CI | Emmeans | 95%CI | ||
Shame | Private | High | 7.11 | [6.57, 7.65] | 6.29 | [5.74, 6.84] | 4.35 | [3.79, 4.90] | 5.93 | [5.60, 6.27] |
Implicit | High | 6.79 | [6.26, 7.33] | 7.16 | [6.61, 7.71] | 4.93 | [4.40, 5.46] | 6.27 | [5.94, 6.60] | |
Explicit | High | 7.28 | [6.71, 7.86] | 8.24 | [7.71, 8.77] | 6.00 | [5.43, 6.57] | 7.23 | [6.88, 7.57] | |
Private | Low | 2.74 | [2.20, 3.29] | 1.58 | [1.04, 2.11] | 1.42 | [0.89, 1.95] | 1.90 | [1.57, 2.23] | |
Implicit | Low | 2.89 | [2.34, 3.44] | 2.83 | [2.29, 3.38] | 2.31 | [1.75, 2.87] | 2.68 | [2.34, 3.02] | |
Explicit | Low | 3.96 | [3.41, 4.50] | 6.25 | [5.71, 6.79] | 4.90 | [4.36, 5.45] | 5.04 | [4.71, 5.38] | |
Guilt | Private | High | 7.25 | [6.71, 7.79] | 6.47 | [5.92, 7.02] | 5.06 | [4.51, 5.61] | 6.27 | [5.95, 6.60] |
Implicit | High | 7.01 | [6.48, 7.55] | 7.33 | [6.78, 7.89] | 5.57 | [5.04, 6.10] | 6.62 | [6.29, 6.94] | |
Explicit | High | 7.28 | [6.71, 7.85] | 8.32 | [7.79, 8.85] | 6.54 | [5.97, 7.11] | 7.43 | [7.09, 7.76] | |
Private | Low | 2.41 | [1.87, 2.96] | 1.55 | [1.01, 2.09] | 1.89 | [1.36, 2.43] | 1.94 | [1.62, 2.27] | |
Implicit | Low | 2.94 | [2.39, 3.49] | 2.96 | [2.41, 3.50] | 2.62 | [2.06, 3.17] | 2.84 | [2.51, 3.17] | |
Explicit | Low | 3.79 | [3.24, 4.33] | 6.39 | [5.85, 6.93] | 5.23 | [4.68, 5.77] | 5.14 | [4.81, 5.47] |
. | Exposure . | Moral Belief . | Plagiarize scenario . | Steal scenario . | Disobey scenario . | Overall . | ||||
---|---|---|---|---|---|---|---|---|---|---|
DV | Emmeans | 95%CI | Emmeans | 95%CI | Emmeans | 95%CI | Emmeans | 95%CI | ||
Shame | Private | High | 7.11 | [6.57, 7.65] | 6.29 | [5.74, 6.84] | 4.35 | [3.79, 4.90] | 5.93 | [5.60, 6.27] |
Implicit | High | 6.79 | [6.26, 7.33] | 7.16 | [6.61, 7.71] | 4.93 | [4.40, 5.46] | 6.27 | [5.94, 6.60] | |
Explicit | High | 7.28 | [6.71, 7.86] | 8.24 | [7.71, 8.77] | 6.00 | [5.43, 6.57] | 7.23 | [6.88, 7.57] | |
Private | Low | 2.74 | [2.20, 3.29] | 1.58 | [1.04, 2.11] | 1.42 | [0.89, 1.95] | 1.90 | [1.57, 2.23] | |
Implicit | Low | 2.89 | [2.34, 3.44] | 2.83 | [2.29, 3.38] | 2.31 | [1.75, 2.87] | 2.68 | [2.34, 3.02] | |
Explicit | Low | 3.96 | [3.41, 4.50] | 6.25 | [5.71, 6.79] | 4.90 | [4.36, 5.45] | 5.04 | [4.71, 5.38] | |
Guilt | Private | High | 7.25 | [6.71, 7.79] | 6.47 | [5.92, 7.02] | 5.06 | [4.51, 5.61] | 6.27 | [5.95, 6.60] |
Implicit | High | 7.01 | [6.48, 7.55] | 7.33 | [6.78, 7.89] | 5.57 | [5.04, 6.10] | 6.62 | [6.29, 6.94] | |
Explicit | High | 7.28 | [6.71, 7.85] | 8.32 | [7.79, 8.85] | 6.54 | [5.97, 7.11] | 7.43 | [7.09, 7.76] | |
Private | Low | 2.41 | [1.87, 2.96] | 1.55 | [1.01, 2.09] | 1.89 | [1.36, 2.43] | 1.94 | [1.62, 2.27] | |
Implicit | Low | 2.94 | [2.39, 3.49] | 2.96 | [2.41, 3.50] | 2.62 | [2.06, 3.17] | 2.84 | [2.51, 3.17] | |
Explicit | Low | 3.79 | [3.24, 4.33] | 6.39 | [5.85, 6.93] | 5.23 | [4.68, 5.77] | 5.14 | [4.81, 5.47] |
Exploratory: Scenario interactions
To examine whether scenario affected exposure, moral belief, or the interaction, we followed the pre-registered plan and supplemented the replication analyses with three-way ANOVAs that included scenario as a factor.
We found support for the exposure × scenario interaction effect and the moral belief × scenario interaction effect on both explicit shame and guilt. We then examined the effects of exposure and moral beliefs in each scenario using two-way ANOVAs. We found that in each scenario, there was support for a main effect of exposure on explicit shame, a main effect of moral belief on explicit shame, a main effect of exposure on explicit guilt, and a main effect of moral belief on explicit guilt (for details, see supplementary subsection “Details of the three-way ANOVAs on manipulation checks, explicit emotions, and emotion-related reactions”). Therefore, the interactions seem to suggest the effects of experimental manipulations differ in magnitude across scenarios, yet the patterns remain consistent (see Table 6), and we, therefore, caution against overinterpreting these exploratory findings.
Directly comparing shame and guilt
We ran a 3 (Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: Low vs. High) × 2 (Emotion: shame vs. guilt) mixed ANOVA and found that in addition to the main effect of exposure, F(2, 1266) = 93.97, p < .001, ηp2 = .13, 95% CI [.10, .16], and the main effect of moral belief, F(1, 1266) = 621.82, p < .001, ηp2 = .33, 95% CI [.29, .37], there was also a main effect of emotion, F(1, 1266) = 45.02, p < .001, ηp2 = .03, 95% CI [.02, .06]. Paired t-test showed that on average participants reported higher scores for guilt (M = 5.02, SD = 3.16) than for shame (M = 4.82, SD = 3.14), t(1271) = 6.68, p < .001.
We also found support for a two-way interaction between exposure and moral belief, F (2, 1266) = 18.61, p < .001, ηp2 =.03, 95% CI [.01, .05], and a two-way interaction between moral belief and emotion, F (2, 1266) = 10.73, p = .001, ηp2 = .008, 95% CI [.00, .02]. The two-way interaction between moral belief and emotion indicated that although in both low and high moral belief conditions, the score of explicit guilt was higher than the score of explicit shame, the differences were larger in the high moral belief condition (Mdiff = 0.30, t (631) = 6.96, p <.001) compared with the low moral belief condition (Mdiff = 0.10, t (639) = 2.46, p = .014). Crucially, we found no support for the two-way interaction between exposure and emotion, F (2, 1266) = 1.08, p = .34, ηp2 = .002, 95% CI [.00, .02], failing to indicate exposure having differential effects on shame and guilt. This result is again inconsistent with the original article’s finding that exposure had a greater effect on shame than guilt. Therefore, we conclude that both key tests failed to find support for the target article’s hypothesis.
Correlations between manipulation checks, explicit emotions, and emotion reactions
All multi-item measures of emotional reactions had acceptable reliability. We summarized correlations and reliabilities in Table 7.
Variable | M | SD | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 |
1. Explicit shame | 4.82 | 3.14 | (-) | ||||||||||||
2. Explicit guilt | 5.02 | 3.16 | .94 | (-) | |||||||||||
[.94, .95] | |||||||||||||||
3. Body change | 3.65 | 2.79 | .73 | .71 | (.96) | ||||||||||
[.71, .76] | [.68, .73] | ||||||||||||||
4. Desire hide | 3.90 | 2.94 | .69 | .67 | .88 | (.93) | |||||||||
[.66, .72] | [.63, .69] | [.87, .90] | |||||||||||||
5. Embarrassed | 4.42 | 3.30 | .76 | .75 | .84 | .80 | (-) | ||||||||
[.74, .79] | [.73, .77] | [.82, .85] | [.78, .82] | ||||||||||||
6. Humiliated | 3.51 | 3.23 | .69 | .66 | .85 | .80 | .85 | (-) | |||||||
[.66, .72] | [.63, .69] | [.84, .87] | [.78, .82] | [.83, .86] | |||||||||||
7. Inferior self | 3.66 | 2.57 | .68 | .64 | .84 | .81 | .75 | .79 | (.92) | ||||||
[.65, .71] | [.60, .67] | [.82, .86] | [.79, .83] | [.72, .77] | [.77, .81] | ||||||||||
8. Anger other | 2.12 | 2.31 | .03 | -.02 | .25 | .23 | .15 | .22 | .36 | (.92) | |||||
[-.03, .08] | [-.07, .04] | [.20, .30] | [.18, .28] | [.10, .21] | [.16, .27] | [.31, .41] | |||||||||
9. Anger self | 3.14 | 2.33 | .67 | .65 | .75 | .71 | .71 | .71 | .77 | .43 | (.74) | ||||
[.64, .70] | [.62, .68] | [.73, .77] | [.68, .73] | [.68, .74] | [.69, .74] | [.75, .79] | [.38, .47] | ||||||||
10. Guilty conscience | 4.73 | 2.92 | .83 | .85 | .74 | .69 | .76 | .69 | .70 | .04 | .69 | (.93) | |||
[.81, .85] | [.84, .87] | [.72, .77] | [.67, .72] | [.73, .78] | [.66, .71] | [.67, .73] | [-.02, .09] | [.67, .72] | |||||||
11. Real self | 3.97 | 2.86 | .57 | .57 | .53 | .51 | .54 | .49 | .48 | .02 | .50 | .63 | (-) | ||
[.53, .60] | [.53, .61] | [.49, .57] | [.46, .55] | [.50, .58] | [.45, .53] | [.43, .52] | [-.03, .08] | [.45, .54] | [.59, .66] | ||||||
12. Hurt other | 2.83 | 2.49 | .58 | .60 | .57 | .51 | .56 | .54 | .50 | .09 | .53 | .67 | .51 | (.71) | |
[.54, .62] | [.56, .63] | [.53, .60] | [.47, .55] | [.52, .59] | [.50, .58] | [.46, .54] | [.03, .14] | [.49, .57] | [.64, .70] | [.46, .55] | |||||
13. Undo wrong | 3.94 | 2.89 | .72 | .73 | .70 | .65 | .74 | .68 | .57 | .02 | .63 | .80 | .63 | .74 | (.97) |
[.70, .75] | [.71, .76] | [.67, .73] | [.62, .69] | [.71, .76] | [.65, .71] | [.53, .61] | [-.04, .07] | [.59, .66] | [.78, .82] | [.60, .66] | [.72, .77] |
Variable | M | SD | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 |
1. Explicit shame | 4.82 | 3.14 | (-) | ||||||||||||
2. Explicit guilt | 5.02 | 3.16 | .94 | (-) | |||||||||||
[.94, .95] | |||||||||||||||
3. Body change | 3.65 | 2.79 | .73 | .71 | (.96) | ||||||||||
[.71, .76] | [.68, .73] | ||||||||||||||
4. Desire hide | 3.90 | 2.94 | .69 | .67 | .88 | (.93) | |||||||||
[.66, .72] | [.63, .69] | [.87, .90] | |||||||||||||
5. Embarrassed | 4.42 | 3.30 | .76 | .75 | .84 | .80 | (-) | ||||||||
[.74, .79] | [.73, .77] | [.82, .85] | [.78, .82] | ||||||||||||
6. Humiliated | 3.51 | 3.23 | .69 | .66 | .85 | .80 | .85 | (-) | |||||||
[.66, .72] | [.63, .69] | [.84, .87] | [.78, .82] | [.83, .86] | |||||||||||
7. Inferior self | 3.66 | 2.57 | .68 | .64 | .84 | .81 | .75 | .79 | (.92) | ||||||
[.65, .71] | [.60, .67] | [.82, .86] | [.79, .83] | [.72, .77] | [.77, .81] | ||||||||||
8. Anger other | 2.12 | 2.31 | .03 | -.02 | .25 | .23 | .15 | .22 | .36 | (.92) | |||||
[-.03, .08] | [-.07, .04] | [.20, .30] | [.18, .28] | [.10, .21] | [.16, .27] | [.31, .41] | |||||||||
9. Anger self | 3.14 | 2.33 | .67 | .65 | .75 | .71 | .71 | .71 | .77 | .43 | (.74) | ||||
[.64, .70] | [.62, .68] | [.73, .77] | [.68, .73] | [.68, .74] | [.69, .74] | [.75, .79] | [.38, .47] | ||||||||
10. Guilty conscience | 4.73 | 2.92 | .83 | .85 | .74 | .69 | .76 | .69 | .70 | .04 | .69 | (.93) | |||
[.81, .85] | [.84, .87] | [.72, .77] | [.67, .72] | [.73, .78] | [.66, .71] | [.67, .73] | [-.02, .09] | [.67, .72] | |||||||
11. Real self | 3.97 | 2.86 | .57 | .57 | .53 | .51 | .54 | .49 | .48 | .02 | .50 | .63 | (-) | ||
[.53, .60] | [.53, .61] | [.49, .57] | [.46, .55] | [.50, .58] | [.45, .53] | [.43, .52] | [-.03, .08] | [.45, .54] | [.59, .66] | ||||||
12. Hurt other | 2.83 | 2.49 | .58 | .60 | .57 | .51 | .56 | .54 | .50 | .09 | .53 | .67 | .51 | (.71) | |
[.54, .62] | [.56, .63] | [.53, .60] | [.47, .55] | [.52, .59] | [.50, .58] | [.46, .54] | [.03, .14] | [.49, .57] | [.64, .70] | [.46, .55] | |||||
13. Undo wrong | 3.94 | 2.89 | .72 | .73 | .70 | .65 | .74 | .68 | .57 | .02 | .63 | .80 | .63 | .74 | (.97) |
[.70, .75] | [.71, .76] | [.67, .73] | [.62, .69] | [.71, .76] | [.65, .71] | [.53, .61] | [-.04, .07] | [.59, .66] | [.78, .82] | [.60, .66] | [.72, .77] |
Note. Reliability is provided on the diagonal, we calculated Cronbach’s α for scales with more than two items, and Pearson’s correlation for the two-item scale. M and SD are used to represent mean and standard deviation, respectively. Values in square brackets indicate the 95% confidence interval for each correlation. For effect sizes > |r| = .04, p<.01; for effect sizes > |r| = .09, p <.001.
A higher score on the exposure manipulation check (“judged by others”) was associated with higher scores of explicit shame, r(1272) = .54, 95% CI [.50, .58], p < .001. A higher score on the manipulation check for moral belief (“violates a personal value”) was associated with higher scores of explicit guilt, r(1272) =.76, 95% CI [.74, .78], p < .001. However, the manipulation check on exposure had a similar association with explicit guilt, r(1272) = .52, 95% CI [.48, .56], p < .001, to that of explicit shame, and the manipulation check on moral belief had a similar association with explicit shame, r(1272) =.77, 95% CI [.74, .79], p < .001, to that of explicit guilt. Both explicit shame and explicit guilt were related to shame-related and guilt-related measures in an extremely similar way.
These results suggest that the supposedly distinctive measures of shame and guilt are not as meaningfully different as expected.
Exploratory analyses: Explicit measures and emotional reactions
To further examine whether shame-related reactions were more closely related to explicit shame and guilt-related reactions were more closely related to explicit guilt, we performed regression analyses with explicit shame and guilt entered into the models. We summarized the results in Tables 8 and 9. These analyses were not part of our original analysis plan and were not pre-registered.
. | Body change . | Desire hide . | Embarrassed . | Humiliated . | Inferior Self . | Anger Other . | Anger Self . | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Est | SE | Est | SE | Est | SE | Est | SE | Est | SE | Est | SE | Est | SE | |
Intercept | 3.65*** | 0.05 | 3.90*** | 0.06 | 4.42*** | 0.06 | 3.51*** | 0.07 | 3.66*** | 0.05 | 2.12*** | 0.06 | 3.14*** | 0.05 |
[3.55,3.75] | [3.79,4.02] | [4.30,4.53] | [3.38,3.64] | [3.55,3.76] | [2.00,2.25] | [3.04,3.23] | ||||||||
Explicit Shame | 1.69*** | 0.16 | 1.77*** | 0.18 | 1.69*** | 0.18 | 2.13*** | 0.20 | 1.86*** | 1.59 | 0.91*** | 0.19 | 1.20*** | 0.15 |
[1.38,2.00] | [1.42,2.12] | [1.34,2.04] | [1.75,2.51] | [1.55,2.18] | [0.53,1.29] | [0.91,1.48] | ||||||||
Explicit Guilt | 0.38* | 0.16 | 0.28 | 0.18 | 0.88*** | 0.18 | 0.12 | 0.20 | -0.12 | 1.59 | -0.89*** | 0.19 | 0.39** | 0.15 |
[0.07,0.69] | [-0.07,0.63] | [0.54,1.23] | [-0.26,0.51] | [-0.43,0.19] | [-1.27,-0.52] | [0.10,0.67] |
. | Body change . | Desire hide . | Embarrassed . | Humiliated . | Inferior Self . | Anger Other . | Anger Self . | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Est | SE | Est | SE | Est | SE | Est | SE | Est | SE | Est | SE | Est | SE | |
Intercept | 3.65*** | 0.05 | 3.90*** | 0.06 | 4.42*** | 0.06 | 3.51*** | 0.07 | 3.66*** | 0.05 | 2.12*** | 0.06 | 3.14*** | 0.05 |
[3.55,3.75] | [3.79,4.02] | [4.30,4.53] | [3.38,3.64] | [3.55,3.76] | [2.00,2.25] | [3.04,3.23] | ||||||||
Explicit Shame | 1.69*** | 0.16 | 1.77*** | 0.18 | 1.69*** | 0.18 | 2.13*** | 0.20 | 1.86*** | 1.59 | 0.91*** | 0.19 | 1.20*** | 0.15 |
[1.38,2.00] | [1.42,2.12] | [1.34,2.04] | [1.75,2.51] | [1.55,2.18] | [0.53,1.29] | [0.91,1.48] | ||||||||
Explicit Guilt | 0.38* | 0.16 | 0.28 | 0.18 | 0.88*** | 0.18 | 0.12 | 0.20 | -0.12 | 1.59 | -0.89*** | 0.19 | 0.39** | 0.15 |
[0.07,0.69] | [-0.07,0.63] | [0.54,1.23] | [-0.26,0.51] | [-0.43,0.19] | [-1.27,-0.52] | [0.10,0.67] |
Note. Shame and Guilt were standardized when being entered into the models. Values in square brackets indicate the 95% confidence interval for each regression coefficient. ***p < .001. **p < .01. *p < .05.
. | Guilty Conscience . | Real Self . | Hurt Others . | Undo Wrong . | ||||
---|---|---|---|---|---|---|---|---|
Est | SE | Est | SE | Est | SE | Est | SE | |
Intercept | 4.73*** | 0.04 | 3.97*** | 0.07 | 2.83*** | 0.06 | 3.94*** | 0.05 |
[4.65, 4.81] | [3.84, 4.10] | [2.72, 2.94] | [3.83, 4.04] | |||||
Explicit Shame | 0.67*** | 0.13 | 0.69*** | 0.20 | 0.44** | 0.17 | 0.83*** | 0.16 |
[0.42, 0.92] | [0.31, 1.08] | [0.11, 0.77] | [0.50, 1.15] | |||||
Explicit Guilt | 1.86*** | 0.13 | 0.98*** | 0.20 | 1.07*** | 0.17 | 1.34*** | 0.16 |
[1.62, 2.11] | [0.60, 1.37] | [0.74, 1.40] | [1.02, 1.66] |
. | Guilty Conscience . | Real Self . | Hurt Others . | Undo Wrong . | ||||
---|---|---|---|---|---|---|---|---|
Est | SE | Est | SE | Est | SE | Est | SE | |
Intercept | 4.73*** | 0.04 | 3.97*** | 0.07 | 2.83*** | 0.06 | 3.94*** | 0.05 |
[4.65, 4.81] | [3.84, 4.10] | [2.72, 2.94] | [3.83, 4.04] | |||||
Explicit Shame | 0.67*** | 0.13 | 0.69*** | 0.20 | 0.44** | 0.17 | 0.83*** | 0.16 |
[0.42, 0.92] | [0.31, 1.08] | [0.11, 0.77] | [0.50, 1.15] | |||||
Explicit Guilt | 1.86*** | 0.13 | 0.98*** | 0.20 | 1.07*** | 0.17 | 1.34*** | 0.16 |
[1.62, 2.11] | [0.60, 1.37] | [0.74, 1.40] | [1.02, 1.66] |
Note. Shame and Guilt were standardized when being entered into the models. Values in square brackets indicate the 95% confidence interval for each regression coefficient. ***p < .001. **p < .01.
The Variance Inflation Factors (VIF) were 8.98, consistent with the moderate-to-high correlation between explicit shame and guilt. However, taking into consideration that the sample size of the current study is large and both explicit shame and guilt had ample variance (Gordon, 2015), we consider the results of regression analyses reliable. As shown in Tables 8 and 9, explicit shame was a more robust predictor of shame-related reactions than guilt. Interestingly, when entered simultaneously, explicit shame had a positive, and explicit guilt had a negative relationship with anger toward others. For guilt-related reactions, both explicit shame and guilt predicted all four dependent variables with larger coefficients for guilt than shame. The exploratory analyses provided limited evidence that the supposedly distinctive reactions of shame and guilt had stronger associations with explicit shame or guilt respectively but also reconfirmed that they cannot differentiate the two emotions very well.
Shame-related and Guilt-related Reactions
We conducted a series of two-way ANOVAs to examine the effect of exposure and moral belief manipulation on shame-related and guilt-related reactions.
For shame-related reactions, we found support for the main effect of exposure and the main effect of moral belief on all measures, yet mixed results regarding the interaction between exposure and moral belief (see Table 10). For guilt-related reactions, the pattern was again very similar, with support for the main effect of exposure and the main effect of moral belief on all measures, yet mixed findings regarding the two-way interactions. We summarized the estimated marginal means and standard errors of each condition in Table 11. These results again were not in support of the target’s hypothesis of a distinction between shame and guilt dependent on exposure.
. | Exposure . | Moral belief . | Exposure * Moral belief . |
---|---|---|---|
Body change | F = 149.06, p <.001 | F = 187.87, p <.001 | F = 3.30, p =.037 |
ηp2 = .19, 95%CI [.15, .23] | ηp2 = .13, 95%CI [.10, .16] | ηp2 = .005, 95%CI [.00, .01] | |
Desire to escape | F = 126.03, p <.001 | F = 144.29, p <.001 | F = 2.97, p =.052 |
ηp2 = .17, 95%CI [.13, .20] | ηp2 = .10, 95%CI [.07, .13] | ηp2 = .005, 95%CI [.00, .01] | |
Embarrassed | F = 212.03, p <.001 | F = 256.54, p <.001 | F = 11.09, p <.001 |
ηp2 = .25, 95%CI [.21, .29] | ηp2 = .17, 95%CI [.13, .21] | ηp2 = .02, 95%CI [.01, .03] | |
Humiliated | F = 194.48, p <.001 | F = 158.69, p <.001 | F = 1.73, p =.178 |
ηp2 = .24, 95%CI [.20, .27] | ηp2 = .11, 95%CI [.08, .14] | ηp2 = .003, 95%CI [.00, .01] | |
Inferior Self | F = 68.78, p <.001 | F = 166.37, p <.001 | F = 2.09, p =.124 |
ηp2 = .10, 95%CI [.07, .13] | ηp2 = .12, 95%CI [.09, .15] | ηp2 = .003, 95%CI [.00, .01] | |
Anger at other | F = 7.53, p <.001 | F = 68.95, p <.001 | F = 2.47, p =.085 |
ηp2 = .01, 95%CI [.00, .03] | ηp2 = .05, 95%CI [.03, .08] | ηp2 = .004, 95%CI [.00, .01] | |
Anger at self | F = 68.58, p <.001 | F = 188.19, p <.001 | F = 9.39, p <.001 |
ηp2 = .10, 95%CI [.07, .13] | ηp2 = .13, 95%CI [.10, .16] | ηp2 = .01, 95%CI [.00, .03] | |
Guilty Conscience | F = 81.58, p <.001 | F = 601.99, p <.001 | F = 12.25, p <.001 |
ηp2 = .11, 95%CI [.08, .15] | ηp2 = .32, 95%CI [.28, .36] | ηp2 = .02, 95%CI [.01, .04] | |
Not real self | F = 19.91, p <.001 | F = 250.37, p <.001 | F = 3.89, p = .021 |
ηp2 = .03, 95%CI [.01, .05] | ηp2 = .17, 95%CI [.13, .20] | ηp2 = .006, 95%CI [.00, .02] | |
Hurt others | F = 82.78, p <.001 | F = 284.75, p <.001 | F = 0.01, p = .990 |
ηp2 = .12, 95%CI [.08, .15] | ηp2 = .18, 95%CI [.15, .22] | ηp2 <.001, 95%CI [.00, .00] | |
Undo Wrong | F = 190.64, p <.001 | F = 323.11, p <.001 | F = 2.74, p = .065 |
ηp2 = .23, 95%CI [.19, .27] | ηp2 = .20, 95%CI [.17, .24] | ηp2 =.004, 95%CI [.00, .01] |
. | Exposure . | Moral belief . | Exposure * Moral belief . |
---|---|---|---|
Body change | F = 149.06, p <.001 | F = 187.87, p <.001 | F = 3.30, p =.037 |
ηp2 = .19, 95%CI [.15, .23] | ηp2 = .13, 95%CI [.10, .16] | ηp2 = .005, 95%CI [.00, .01] | |
Desire to escape | F = 126.03, p <.001 | F = 144.29, p <.001 | F = 2.97, p =.052 |
ηp2 = .17, 95%CI [.13, .20] | ηp2 = .10, 95%CI [.07, .13] | ηp2 = .005, 95%CI [.00, .01] | |
Embarrassed | F = 212.03, p <.001 | F = 256.54, p <.001 | F = 11.09, p <.001 |
ηp2 = .25, 95%CI [.21, .29] | ηp2 = .17, 95%CI [.13, .21] | ηp2 = .02, 95%CI [.01, .03] | |
Humiliated | F = 194.48, p <.001 | F = 158.69, p <.001 | F = 1.73, p =.178 |
ηp2 = .24, 95%CI [.20, .27] | ηp2 = .11, 95%CI [.08, .14] | ηp2 = .003, 95%CI [.00, .01] | |
Inferior Self | F = 68.78, p <.001 | F = 166.37, p <.001 | F = 2.09, p =.124 |
ηp2 = .10, 95%CI [.07, .13] | ηp2 = .12, 95%CI [.09, .15] | ηp2 = .003, 95%CI [.00, .01] | |
Anger at other | F = 7.53, p <.001 | F = 68.95, p <.001 | F = 2.47, p =.085 |
ηp2 = .01, 95%CI [.00, .03] | ηp2 = .05, 95%CI [.03, .08] | ηp2 = .004, 95%CI [.00, .01] | |
Anger at self | F = 68.58, p <.001 | F = 188.19, p <.001 | F = 9.39, p <.001 |
ηp2 = .10, 95%CI [.07, .13] | ηp2 = .13, 95%CI [.10, .16] | ηp2 = .01, 95%CI [.00, .03] | |
Guilty Conscience | F = 81.58, p <.001 | F = 601.99, p <.001 | F = 12.25, p <.001 |
ηp2 = .11, 95%CI [.08, .15] | ηp2 = .32, 95%CI [.28, .36] | ηp2 = .02, 95%CI [.01, .04] | |
Not real self | F = 19.91, p <.001 | F = 250.37, p <.001 | F = 3.89, p = .021 |
ηp2 = .03, 95%CI [.01, .05] | ηp2 = .17, 95%CI [.13, .20] | ηp2 = .006, 95%CI [.00, .02] | |
Hurt others | F = 82.78, p <.001 | F = 284.75, p <.001 | F = 0.01, p = .990 |
ηp2 = .12, 95%CI [.08, .15] | ηp2 = .18, 95%CI [.15, .22] | ηp2 <.001, 95%CI [.00, .00] | |
Undo Wrong | F = 190.64, p <.001 | F = 323.11, p <.001 | F = 2.74, p = .065 |
ηp2 = .23, 95%CI [.19, .27] | ηp2 = .20, 95%CI [.17, .24] | ηp2 =.004, 95%CI [.00, .01] |
. | Private . | Public Implicit . | Public Explicit . | |||
---|---|---|---|---|---|---|
High moral | Low moral | High moral | Low moral | High moral | Low moral | |
Body change | 3.60 (0.16) | 1.37 (0.16) | 4.21 (0.16) | 2.35 (0.16) | 5.93 (0.17) | 4.54 (0.16) |
Desire to escape | 3.75 (0.18) | 1.63 (0.17) | 4.52 (0.17) | 2.73 (0.18) | 6.08 (0.18) | 4.81 (0.18) |
Embarrassed | 4.52 (0.18) | 1.45 (0.18) | 5.09 (0.18) | 2.43 (0.18) | 7.27 (0.19) | 5.86 (0.18) |
Humiliated | 3.34 (0.19) | 1.08 (0.18) | 3.71 (0.18) | 1.80 (0.19) | 6.42 (0.19) | 4.85 (0.19) |
Inferior self | 3.96 (0.16) | 1.94 (0.16) | 4.16 (0.16) | 2.52 (0.16) | 5.41 (0.16) | 4.04 (0.16) |
Anger at other | 1.51 (0.15) | 2.24 (0.15) | 1.54 (0.15) | 2.54 (0.15) | 1.75 (0.16) | 3.16 (0.15) |
Anger at self | 3.52 (0.14) | 1.48 (0.14) | 3.77 (0.14) | 1.92 (0.14) | 4.53 (0.14) | 3.64 (0.14) |
Guilty Conscience | 5.79 (0.16) | 1.97 (0.16) | 6.18 (0.16) | 2.81 (0.16) | 7.00 (0.16) | 4.71 (0.16) |
Not real self | 4.92 (0.18) | 2.09 (0.18) | 4.92 (0.18) | 2.77 (0.18) | 5.53 (0.18) | 3.66 (0.18) |
Hurt others | 3.04 (0.15) | 0.99 (0.15) | 3.64 (0.15) | 3.63 (0.15) | 4.90 (0.16) | 2.87 (0.15) |
Undo wrong | 4.06 (0.16) | 1.41 (0.16) | 4.64 (0.16) | 2.19 (0.16) | 6.68 (0.16) | 4.75 (0.16) |
. | Private . | Public Implicit . | Public Explicit . | |||
---|---|---|---|---|---|---|
High moral | Low moral | High moral | Low moral | High moral | Low moral | |
Body change | 3.60 (0.16) | 1.37 (0.16) | 4.21 (0.16) | 2.35 (0.16) | 5.93 (0.17) | 4.54 (0.16) |
Desire to escape | 3.75 (0.18) | 1.63 (0.17) | 4.52 (0.17) | 2.73 (0.18) | 6.08 (0.18) | 4.81 (0.18) |
Embarrassed | 4.52 (0.18) | 1.45 (0.18) | 5.09 (0.18) | 2.43 (0.18) | 7.27 (0.19) | 5.86 (0.18) |
Humiliated | 3.34 (0.19) | 1.08 (0.18) | 3.71 (0.18) | 1.80 (0.19) | 6.42 (0.19) | 4.85 (0.19) |
Inferior self | 3.96 (0.16) | 1.94 (0.16) | 4.16 (0.16) | 2.52 (0.16) | 5.41 (0.16) | 4.04 (0.16) |
Anger at other | 1.51 (0.15) | 2.24 (0.15) | 1.54 (0.15) | 2.54 (0.15) | 1.75 (0.16) | 3.16 (0.15) |
Anger at self | 3.52 (0.14) | 1.48 (0.14) | 3.77 (0.14) | 1.92 (0.14) | 4.53 (0.14) | 3.64 (0.14) |
Guilty Conscience | 5.79 (0.16) | 1.97 (0.16) | 6.18 (0.16) | 2.81 (0.16) | 7.00 (0.16) | 4.71 (0.16) |
Not real self | 4.92 (0.18) | 2.09 (0.18) | 4.92 (0.18) | 2.77 (0.18) | 5.53 (0.18) | 3.66 (0.18) |
Hurt others | 3.04 (0.15) | 0.99 (0.15) | 3.64 (0.15) | 3.63 (0.15) | 4.90 (0.16) | 2.87 (0.15) |
Undo wrong | 4.06 (0.16) | 1.41 (0.16) | 4.64 (0.16) | 2.19 (0.16) | 6.68 (0.16) | 4.75 (0.16) |
Note. The numbers indicate estimated marginal means, with the numbers in the parentheses indicating standard errors.
Notably, the effect of moral belief on anger toward others was in the opposite direction to the effect of moral belief on other emotional actions. Participants in the high moral belief conditions reported higher anger toward others compared with those who were in the low moral belief conditions.
Exploratory: Scenario interactions
We conducted a series of three-way ANOVAs with scenario as the third factor, to examine the effect of exposure and moral belief manipulation on shame-related and guilt-related reactions.
For all the shame-related reactions except anger at others, we found support for the two-way interaction between exposure and scenario as well as the two-way interaction between moral belief and scenario. We did not find support for three-way interactions.
For guilt-related reactions, we found support for the two-way interaction between exposure and moral belief (for guilty conscience, not real self, undo wrong but not for hurt others), the two-way interaction between exposure and scenario (for guilty conscience, hurt others, and undo wrong), and the two-way interaction between moral belief and scenario (for guilty conscience, not real self, undo wrong but not for hurt others). Finally, we found support for a three-way interaction between scenario, exposure, and moral beliefs for undo wrong.
We then ran 3 (Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: high vs. low) two-way ANOVAs for each shame and guilt reactions in each scenario (n = 11 reactions * 3 scenarios). The overall pattern is similar to that of explicit shame and guilt: both moral belief and exposure manipulation had effects on the emotional reactions. Compared with low moral belief conditions, participants in high moral belief conditions tended to report higher emotion reactions except for anger at others. Compared with private and implicit exposure conditions, participants in the explicit conditions tended to report higher emotion reactions. For some of the shame-related reactions in the plagiarize and steal scenarios and some guilt reactions in the steal and disobey scenarios, there was support for a 3 (Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: high vs. low) interaction, indicating that the effect of exposure being greater in the low moral belief conditions compared with the high moral belief conditions.
Comparing Replication to Original Findings
We evaluated whether the replication successfully replicated the original findings based on the criteria by LeBel et al. (2019) by examining signal detection, effects overlap (whether original effect size overlaps with the replication’s confidence intervals), and effect directionality. We summarized those in Tables 12A and 12B. Note that because the target article did not report the statistical tests of some findings, we could not compute the effect sizes and confidence intervals for these. Therefore, for these effects, we assume that they were not supported and only made judgments regarding the presence/absence of signal and the direction of the effect without comparing the overlapping of confidence intervals.
Dependent Variables | Independent Variables | Original | Replication | Categorization | ||||
Effect(η2) | CIL | CIH | Effect(η2) | CIL | CIH | |||
Exposure Check | Exposure | 0.11 | 0.03 | 0.21 | 0.32 | 0.28 | 0.36 | Signal-inconsistent, larger |
Moral Beliefs | - | - | - | 0.04 | 0.02 | 0.06 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.004 | 0.00 | 0.01 | No Signal, consistent | |
Moral Belief Check | Exposure | - | - | - | 0.04 | 0.02 | 0.06 | Signal-inconsistent, positive |
Moral Beliefs | 0.10 | 0.03 | 0.20 | 0.31 | 0.28 | 0.35 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | 0.04 | 0 | 0.11 | 0.01 | 0.00 | 0.03 | Signal-inconsistent, smaller | |
Explicit Shame | Exposure | 0.14 | 0.05 | 0.24 | 0.13 | 0.09 | 0.16 | Signal-consistent |
Moral Beliefs | 0.07 | 0.01 | 0.16 | 0.30 | 0.26 | 0.34 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | 0.04 | 0 | 0.11 | 0.02 | 0.01 | 0.04 | Signal-consistent | |
Explicit Guilt | Exposure | - | - | - | 0.12 | 0.09 | 0.16 | Signal-inconsistent, positive |
Moral Beliefs | 0.08 | 0.02 | 0.18 | 0.34 | 0.30 | 0.37 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | - | - | - | 0.01 | 0.03 | 0.05 | Signal-inconsistent | |
Bodily Change | Exposure | 0.16 | 0.06 | 0.26 | 0.19 | 0.15 | 0.23 | Signal-consistent |
Moral Beliefs | 0.03 | 0 | 0.10 | 0.13 | 0.10 | 0.16 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | 0.07 | 0.01 | 0.15 | 0.005 | 0.00 | 0.01 | Signal-inconsistent, smaller | |
Desire to Escape | Exposure | 0.16 | 0.06 | 0.26 | 0.17 | 0.13 | 0.20 | Signal-consistent |
Moral Beliefs | - | - | - | 0.10 | 0.07 | 0.13 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | 0.04 | 0 | 0.12 | 0.005 | 0.00 | 0.01 | No signal-inconsistent | |
Embarrassed | Exposure | 0.06 | 0.003 | 0.14 | 0.25 | 0.21 | 0.29 | Signal-inconsistent, larger |
Moral Beliefs | - | - | - | 0.17 | 0.13 | 0.21 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.02 | 0.01 | 0.03 | Signal-inconsistent | |
Humiliated | Exposure | 0.06 | 0.003 | 0.14 | 0.24 | 0.20 | 0.27 | Signal-inconsistent, larger |
Moral Beliefs | - | - | - | 0.11 | 0.08 | 0.14 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.003 | 0.00 | 0.01 | No signal- consistent | |
Inferior Self | Exposure | - | - | - | 0.10 | 0.07 | 0.13 | Signal-inconsistent, positive |
Moral Beliefs | - | - | - | 0.12 | 0.09 | 0.15 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.003 | 0.00 | 0.01 | No signal- consistent | |
Anger to Self | Exposure | 0.08 | 0.01 | 0.17 | 0.10 | 0.07 | 0.13 | Signal-consistent |
Moral Beliefs | 0.07 | 0.01 | 0.16 | 0.13 | 0.10 | 0.16 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | 0.05 | 0.002 | 0.13 | 0.01 | 0.00 | 0.03 | Signal-inconsistent, smaller | |
Anger to Other | Exposure | - | - | - | 0.01 | 0.00 | 0.03 | Signal-inconsistent, positive |
Moral Beliefs | - | - | - | 0.05 | 0.03 | 0.08 | Signal-inconsistent, negative | |
Exposure × Moral Beliefs | - | - | - | 0.004 | 0.00 | 0.01 | No signal-consistent | |
Guilty Conscience | Exposure | - | - | - | 0.11 | 0.08 | 0.15 | Signal-inconsistent, positive |
Moral Beliefs | 0.05 | 0.005 | 0.13 | 0.32 | 0.28 | 0.36 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | - | - | - | 0.02 | 0.01 | 0.04 | Signal-inconsistent | |
Real Self | Exposure | - | - | - | 0.03 | 0.01 | 0.05 | Signal-inconsistent, positive |
Moral Beliefs | - | - | - | 0.17 | 0.13 | 0.20 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.006 | 0.00 | 0.02 | Signal-inconsistent | |
Hurt Others | Exposure | 0.22 | 0.11 | 0.33 | 0.12 | 0.08 | 0.15 | Signal-inconsistent, smaller |
Moral Beliefs | 0.03 | 0 | 0.10 | 0.18 | 0.15 | 0.22 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | - | - | - | <.001 | 0.00 | 0.00 | No signal-consistent | |
Undo Wrong | Exposure | - | - | - | 0.23 | 0.19 | 0.27 | Signal-inconsistent, positive |
Moral Beliefs | - | - | - | 0.20 | 0.17 | 0.24 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.004 | 0.00 | 0.01 | No signal-consistent |
Dependent Variables | Independent Variables | Original | Replication | Categorization | ||||
Effect(η2) | CIL | CIH | Effect(η2) | CIL | CIH | |||
Exposure Check | Exposure | 0.11 | 0.03 | 0.21 | 0.32 | 0.28 | 0.36 | Signal-inconsistent, larger |
Moral Beliefs | - | - | - | 0.04 | 0.02 | 0.06 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.004 | 0.00 | 0.01 | No Signal, consistent | |
Moral Belief Check | Exposure | - | - | - | 0.04 | 0.02 | 0.06 | Signal-inconsistent, positive |
Moral Beliefs | 0.10 | 0.03 | 0.20 | 0.31 | 0.28 | 0.35 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | 0.04 | 0 | 0.11 | 0.01 | 0.00 | 0.03 | Signal-inconsistent, smaller | |
Explicit Shame | Exposure | 0.14 | 0.05 | 0.24 | 0.13 | 0.09 | 0.16 | Signal-consistent |
Moral Beliefs | 0.07 | 0.01 | 0.16 | 0.30 | 0.26 | 0.34 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | 0.04 | 0 | 0.11 | 0.02 | 0.01 | 0.04 | Signal-consistent | |
Explicit Guilt | Exposure | - | - | - | 0.12 | 0.09 | 0.16 | Signal-inconsistent, positive |
Moral Beliefs | 0.08 | 0.02 | 0.18 | 0.34 | 0.30 | 0.37 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | - | - | - | 0.01 | 0.03 | 0.05 | Signal-inconsistent | |
Bodily Change | Exposure | 0.16 | 0.06 | 0.26 | 0.19 | 0.15 | 0.23 | Signal-consistent |
Moral Beliefs | 0.03 | 0 | 0.10 | 0.13 | 0.10 | 0.16 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | 0.07 | 0.01 | 0.15 | 0.005 | 0.00 | 0.01 | Signal-inconsistent, smaller | |
Desire to Escape | Exposure | 0.16 | 0.06 | 0.26 | 0.17 | 0.13 | 0.20 | Signal-consistent |
Moral Beliefs | - | - | - | 0.10 | 0.07 | 0.13 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | 0.04 | 0 | 0.12 | 0.005 | 0.00 | 0.01 | No signal-inconsistent | |
Embarrassed | Exposure | 0.06 | 0.003 | 0.14 | 0.25 | 0.21 | 0.29 | Signal-inconsistent, larger |
Moral Beliefs | - | - | - | 0.17 | 0.13 | 0.21 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.02 | 0.01 | 0.03 | Signal-inconsistent | |
Humiliated | Exposure | 0.06 | 0.003 | 0.14 | 0.24 | 0.20 | 0.27 | Signal-inconsistent, larger |
Moral Beliefs | - | - | - | 0.11 | 0.08 | 0.14 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.003 | 0.00 | 0.01 | No signal- consistent | |
Inferior Self | Exposure | - | - | - | 0.10 | 0.07 | 0.13 | Signal-inconsistent, positive |
Moral Beliefs | - | - | - | 0.12 | 0.09 | 0.15 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.003 | 0.00 | 0.01 | No signal- consistent | |
Anger to Self | Exposure | 0.08 | 0.01 | 0.17 | 0.10 | 0.07 | 0.13 | Signal-consistent |
Moral Beliefs | 0.07 | 0.01 | 0.16 | 0.13 | 0.10 | 0.16 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | 0.05 | 0.002 | 0.13 | 0.01 | 0.00 | 0.03 | Signal-inconsistent, smaller | |
Anger to Other | Exposure | - | - | - | 0.01 | 0.00 | 0.03 | Signal-inconsistent, positive |
Moral Beliefs | - | - | - | 0.05 | 0.03 | 0.08 | Signal-inconsistent, negative | |
Exposure × Moral Beliefs | - | - | - | 0.004 | 0.00 | 0.01 | No signal-consistent | |
Guilty Conscience | Exposure | - | - | - | 0.11 | 0.08 | 0.15 | Signal-inconsistent, positive |
Moral Beliefs | 0.05 | 0.005 | 0.13 | 0.32 | 0.28 | 0.36 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | - | - | - | 0.02 | 0.01 | 0.04 | Signal-inconsistent | |
Real Self | Exposure | - | - | - | 0.03 | 0.01 | 0.05 | Signal-inconsistent, positive |
Moral Beliefs | - | - | - | 0.17 | 0.13 | 0.20 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.006 | 0.00 | 0.02 | Signal-inconsistent | |
Hurt Others | Exposure | 0.22 | 0.11 | 0.33 | 0.12 | 0.08 | 0.15 | Signal-inconsistent, smaller |
Moral Beliefs | 0.03 | 0 | 0.10 | 0.18 | 0.15 | 0.22 | Signal-inconsistent, larger | |
Exposure × Moral Beliefs | - | - | - | <.001 | 0.00 | 0.00 | No signal-consistent | |
Undo Wrong | Exposure | - | - | - | 0.23 | 0.19 | 0.27 | Signal-inconsistent, positive |
Moral Beliefs | - | - | - | 0.20 | 0.17 | 0.24 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.004 | 0.00 | 0.01 | No signal-consistent |
Note. The target article only reported supported findings. Therefore, for effects that were not reported in the target article, we lacked the information to calculate the effect sizes and their confidence intervals and assumed that they were not supported. To make a direct comparison, scenarios were not included as a factor following the original article.
Dependent Variable | Independent Variables | Original | Replication | Categorization | ||||
Effect(η2) | CIL | CIH | Effect(η2) | CIL | CIH | |||
Emotion intensity | Exposure | - | - | - | 0.13 | 0.10 | 0.16 | Signal-inconsistent, positive |
Moral Beliefs | - | - | - | 0.33 | 0.29 | 0.37 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.03 | 0.01 | 0.05 | Signal-inconsistent | |
Emotion | 0.15 | 0.06 | 0.26 | 0.03 | 0.02 | 0.06 | Signal-inconsistent, smaller | |
Exposure × Emotion | 0.06 | 0.01 | 0.15 | 0.002 | 0.00 | 0.01 | No signal-inconsistent | |
Moral Beliefs × Emotion | - | - | - | 0.008 | 0.00 | 0.02 | Signal-inconsistent | |
Exposure × Moral Beliefs × Emotion | 0.02 | 0 | 0.09 | 0.001 | 0.00 | 0.01 | No signal-inconsistent |
Dependent Variable | Independent Variables | Original | Replication | Categorization | ||||
Effect(η2) | CIL | CIH | Effect(η2) | CIL | CIH | |||
Emotion intensity | Exposure | - | - | - | 0.13 | 0.10 | 0.16 | Signal-inconsistent, positive |
Moral Beliefs | - | - | - | 0.33 | 0.29 | 0.37 | Signal-inconsistent, positive | |
Exposure × Moral Beliefs | - | - | - | 0.03 | 0.01 | 0.05 | Signal-inconsistent | |
Emotion | 0.15 | 0.06 | 0.26 | 0.03 | 0.02 | 0.06 | Signal-inconsistent, smaller | |
Exposure × Emotion | 0.06 | 0.01 | 0.15 | 0.002 | 0.00 | 0.01 | No signal-inconsistent | |
Moral Beliefs × Emotion | - | - | - | 0.008 | 0.00 | 0.02 | Signal-inconsistent | |
Exposure × Moral Beliefs × Emotion | 0.02 | 0 | 0.09 | 0.001 | 0.00 | 0.01 | No signal-inconsistent |
Note. The target article only reported supported findings. Therefore, for effects that were not reported in the target article, we lacked the information to calculate the effect sizes and their confidence intervals and assumed that they were not supported.
For 10 out of the 20 effects that were supported in the original study, our replication detected a larger signal. Five effects were consistent with the original article and four effects were smaller compared with the original article. Only the interaction effect between moral belief and exposure on desire to escape/hide was not supported in the current replication but supported in the original. Moreover, 19 out of the 25 assumed not-supported effects in the original study received support from our replication effect. Crucially, the original study did not detect a main effect of exposure manipulation on explicit guilt. However, in the replication, we found a main effect of exposure and the interaction effect between exposure and moral belief on explicit guilt.
For the 3 (Exposure: private vs. implicit public vs. explicit public) × 2 (Moral belief: Low vs. High) × 2 (Emotion: shame vs. guilt) mixed ANOVA, we found that two supported findings in the original article did not receive support from our replication and one supported finding was present but smaller (see Table 12B). We did not detect the crucial exposure × Emotion two-way interaction, as reported in the original article. Taken together, we conclude that the current study failed to replicate the core findings of Smith et al. (2002).
Discussion
We attempted a close replication of Study 1 in Smith et al. (2002). Deviating from the findings in the target article, we found that both exposure and moral belief manipulation affected the perceived shame and guilt responses and the effects were similar for both guilt and shame. Moreover, explicit shame and explicit guilt had similar associations with shame-related and guilt-related emotional reactions.
Despite our best efforts to be consistent with the original article and to be rigorous in methodology, we found effects where none were expected (guilt) and very similar across the two emotions, which deviates from the core hypothesis of the target article claiming that the two emotions differ. It is therefore not the case that we did not find effects reported in the target, but rather a rather unique case in which we found unexpected associations which were not in line with the target’s theory or findings. This held for both planned and exploratory analyses.
In the following, we discuss the results, our deviations, and implications.
The distinction between shame and guilt
We found that the effects of exposure and moral beliefs manipulations had very similar impact on both guilt and shame. Across several other analyses we found no indication of differences between guilt and shame, suggesting that the distinction made between shame and guilt in the target article was not successfully replicated.
Moreover, our findings cast doubt on the distinction between shame-related and guilt-related emotional reactions, as both shame-related and guilt-related reactions were as closely related to explicit shame as to explicit guilt. Our exploratory regression analyses showed that explicit shame predicted shame-related reactions more than guilt and vice versa, yet the two emotions were both uniquely related to the guilt-related reactions. In addition, the experimental manipulations had similar effects on shame-related and guilt-related reactions.
This did not seem to be an issue with the manipulations - Our analyses showed that the manipulations were successful. Both exposure and moral beliefs manipulations affected the exposure and moral beliefs manipulation checks. There was a stronger effect for exposure manipulation on the exposure check and a stronger effect for moral belief manipulation on the moral belief check.
Although deviating from the original article, our replication results are consistent with other research on the shame-guilt distinction. For example, Schmader and Lickel (2006) also showed that for self-caused wrongdoings, shame and guilt were highly correlated, thus difficult to distinguish from each other. However, in the case of other-caused wrongdoings, shame and guilt not only were moderately correlated with each other but also uniquely predicted avoidance and approach motivations respectively. We see the need to revisit findings in this literature and to try and aggregate findings to try and determine whether shame and guilt are indeed distinct, and if so in what way and under what circumstances.
Comparison with the target article and theoretical implications
In the target article, Smith et al. (2002) reported that exposure had a stronger impact on shame than guilt and that guilt was only impacted by moral belief. We found that moral belief and exposure both impacted shame and guilt and the effects were highly similar between the two emotions. In addition, we found that exposure and moral belief interacted to predict the emotions.
The evaluation of replication suggests that the majority of the effects reported in the original study were also detected in the present study, with half of the effects being larger than the original. However, we found no indication for exposure having a different impact on eliciting shame than eliciting guilt.
Our replication findings seem to suggest that the public-private distinction (alone) is not enough to distinguish shame from guilt. As briefly summarized by Miceli and Castelfranchi (2018), shame and guilt share many similarities: 1) unpleasant; 2) implying a negative self-evaluation; 3) can be elicited by the same type of wrongdoings; 4) can be experienced either publicly or privately; 5) may trigger either self-defensive or reparative action tendencies; and 6) can be either adaptive or maladaptive. It is therefore not entirely surprising that we found public exposure had a similar impact on shame and guilt. However, this also means that there is much work to be done before being able to achieve a complete understanding of shame and guilt. To be able to reach such an understanding, we believe it is important to have a solid empirical foundation, which requires more replication studies of findings in this literature and the field.
Limitations and future directions
We note several limitations that may have influenced the results and might be improved in future investigations.
First, our comprehension checks were a deviation from the original study, which may have increased participants’ attentiveness to the exposure and moral belief information. This was intentional, as we wanted to ensure manipulations worked as intended, yet we could not rule out the possibility that these may have led to their impact being stronger than in the original in a way that elicited similar impact on shame and guilt.
Second, thanks to feedback from our participants, we realized several oversights in our materials. Our comprehension check to test exposure was not ideal for certain scenarios. In total, 40 participants (3.14%) indicated that they felt that the check was either wrong/inaccurate or vague. Among these 40 cases, 17 were from the plagiarize scenario explicit public condition. In hindsight, we realized that the question was not well suited for this condition as the question asked whether the main character saw anyone nearby after copying the classmate’s report and the correct answer was set to be yes, however, the story only tells that she/he received a note from another person that the other person saw her. However, we believe this flaw did not invalidate the study as the analyses done for different scenarios revealed similar patterns, and it was not the issue that we did not find effects but rather that we found effects even when effects were not expected to be found. Other minor errors in the survey also included one piped text error where the person’s name was missing in one of the sentences and one error concerning pronouns. Our pretests did not reveal these issues, yet it highlights the importance of eliciting feedback from participants for the real sample in the funneling section, to help improve future studies.
Another obvious deviation was that in the current study we collected data online in a private setting whereas the original study collected data offline in groups. The difference in settings may have impacted the emotional intensity in some way. The mean shame ratings for the private, implicit public, and explicit public conditions were 3.89, 4.51, 6.11 in our study and 5.71, 6.96, 8.11 in Smith et al. (2002). For explicit guilt, the mean guilt ratings in the low and high moral belief conditions were 3.31 and 6.76 in our study yet 7.14 and 8.26 in the original study (Smith et al., 2002). This may explain smaller effects in the target article than in our sample, though we are not certain how that may explain bigger differences between shame in guilt in their study.
We note a shared limitation of both the original study and our replication was the manipulation of the exposure. The manipulation of exposure did not simply increase perceived exposure in the implicit and the explicit conditions compared to the private condition. More precisely, the manipulation increased the perceived exposure of misdeeds to colleagues or family, with whom the characters have a personal relationship. Baumeister et al. (1994) suggested that guilt serves various relationship-enhancing functions by increasing affiliative motivations and the exposure manipulation could also have an impact on guilt by making the objects of affiliation (i.e., colleagues or family members) more salient. That is, the exposure manipulation could increase levels of both shame and guilt in the implicit public and explicit public conditions compared with the private condition via different mechanisms, which is consistent with what we observed in the current data. Future studies thus should design and validate manipulation procedures that separate exposure from exposure to affiliative targets.
Finally, as shown by the current study, the supposed shame or guilt-related actions were not uniquely related to these two emotions. The validation of these measures deserves a full-on investigation on its own and there have already been other follow-up studies that tackle this direction (e.g., the development of the GASP scale by Cohen et al., 2011). We consider the current dataset having the potential to offer more insights into this research direction and encourage anyone who would like to follow up to utilize the dataset.
Conclusion
In this close replication of Smith et al. (2002) with a larger more diverse sample, we found that exposure and moral belief manipulations impacted both shame and guilt similarly. More specifically, we failed to find support that exposure had a greater effect on shame than guilt. We, therefore, conclude this as a failed replication, not in support of the distinction made between shame and guilt in the target article (i.e., exposure). In addition to the key confirmatory analyses above, we also found that the supposed shame or guilt-related measures were not uniquely associated with shame or guilt, respectively. We note several limitations that may have impacted our replication, yet not in a way that would explain these differences. Finally, we raised issues regarding the exposure manipulation employed in the target article. We suggest caution in future studies assuming that the distinction between guilt and shame lies in exposure and using the exposure manipulation.
Competing Interests
The authors declared no potential conflicts of interest with respect to the authorship and/or publication of this article.
Funding
The project was supported by the University of Hong Kong Teaching Development Grant awarded to Gilad Feldman.
Authorship Declaration
Fung Chit Cheung, Hei Tung Wong, Lok Yee Yuen, and Hui Ching Sin designed the study, developed the experimental materials for each study respectively, and wrote an initial draft of the Registered Report Stage 1. Yikang Zhang revised the designs and experimental materials, wrote the analysis scripts, conducted the data analyses, and drafted the manuscript for submission. Hiu Tung Kristy Chow provided feedback and guidance in the initial stages. Gilad Feldman guided the replication efforts, supervised each step in the project, ran data collection and conducted the pre-registration, and edited the manuscript for submission.
Important Links and Information
Citation of the target research article:
Smith, R. H., Webster, J. M., Parrott, W. G., & Eyre, H. L. (2002). The role of public exposure in moral and nonmoral shame and guilt. Journal of Personality and Social Psychology, 83(1), 138-159. doi:10.1037/0022-3514.83.1.138
Data Accessibility Statement
We provided all materials, data, and code on: https://osf.io/j3ue4/.
Contributor Roles Taxonomy
In the table below, we employed CRediT (Contributor Roles Taxonomy) to identify the contribution and roles played by the contributors in the current replication effort.
Yikang Zhang | Fung Chit Cheung, Hei Tung Wong, Lok Yee Yuen, Hui Ching Sin | Hiu Tung Kristy Chow | Gilad Feldman | |
Conceptualization | X | X | ||
Data curation | X | X | ||
Formal analysis | X | |||
Funding acquisition | X | |||
Investigation | X | X | X | |
Methodology | X | X | ||
Project administration | X | |||
Resources | X | X | X | |
Software | X | X | ||
Supervision | X | X | ||
Validation | X | |||
Visualization | X | X | ||
Writing-original draft | X | X | ||
Writing-review and editing | X | X |
Yikang Zhang | Fung Chit Cheung, Hei Tung Wong, Lok Yee Yuen, Hui Ching Sin | Hiu Tung Kristy Chow | Gilad Feldman | |
Conceptualization | X | X | ||
Data curation | X | X | ||
Formal analysis | X | |||
Funding acquisition | X | |||
Investigation | X | X | X | |
Methodology | X | X | ||
Project administration | X | |||
Resources | X | X | X | |
Software | X | X | ||
Supervision | X | X | ||
Validation | X | |||
Visualization | X | X | ||
Writing-original draft | X | X | ||
Writing-review and editing | X | X |