The present paper reports an independent and better powered (N = 847 vs. N = 85) replication of Experiment 3 in Glikson et al. (2018). The authors of the original study reported support for their proposition that due to perceptions of (in)appropriateness, the use of smileys may backfire and produce less favorable perceptions of competence in a formal work-related setting, yet more favorable perceptions of warmth in an informal work-related setting. Our results, in contrast, indicated that smileys produce a negative effect on perceptions of competence and a positive effect on perceptions of warmth, regardless of the level of formality. Moreover, our results did not support the reported moderated mediation model involving perceptions of appropriateness. Potential explanations for the discrepancies in results are discussed. We provide data, code, and materials on https://osf.io/n7yc4/.
Introduction
Communication is increasingly taking place on text-based, computer-mediated platforms with limited opportunities to express emotions. However, digital representations of emotions, such as the prevalent “smiley” emoticon (typographic) and “smiley emoji” (pictographic), are frequently used in instant messaging and social media, and also seem to be increasingly used in professional settings, including professional emails. Skovholt et al. (2014) found that emoticons in workplace emails are typically used to signal a positive attitude, to mark humor, jokes, or irony, and to strengthen or soften the content communicated. Yet, despite their prevalent use, the interpretation of emoticons and emojis and their intended meaning can be challenging (Rodrigues et al., 2018) and highly reliant on the context. Indeed, some scholars argue that emoticons and emojis may have a negative effect in formal or work-related contexts due to impressions of, for example, insufficient verbal skills, lack of professionalism, or inappropriateness (e.g., Munter et al., 2003). This line of reasoning is supported by the findings of Kaye et al. (2016), which indicate that emoticon use on email platforms (as opposed to direct messaging and social networks) is perceived as inappropriate by many users.
So far, few studies have explicitly investigated the role of perceived appropriateness of the use of emoticons and emojis in professional digitally based communication. Exceptions include Glikson et al. (2018, Experiment 3), who reported the results from an experiment aimed to test the formality of the context as a moderator of the effects of smileys on first impressions of the sender’s warmth and competence, as well as the mediating role of perceptions of appropriateness. The authors conclude that the effects of smileys on first impressions of warmth and competence are contingent on the formality of the setting. In a formal setting, smileys were found to have a negative effect on perceptions of competence and no effect on perceptions of warmth. In an informal setting, in contrast, smileys were found to have no effect on perceptions of competence and a positive effect on perceptions of warmth. Perceptions of (in)appropriateness were found to partially mediate these effects.
Warmth and competence have previously been found to represent two fundamental dimensions of person perception (Fiske et al., 2002, 2007; Judd et al., 2005) that affect a series of important outcomes in social and work-related settings. The results presented by Glikson et al. (2018) may therefore have apparent practical implications. Provided that the use of smileys produces negative effects in a formal work setting, that is, deflates first impressions of the sender’s competence, senders are best advised to avoid smileys when initiating (email) contact in these settings.
However, to our knowledge, there are no direct replications of this study, despite several apparent methodological limitations associated with the presented results, including low statistical power due to very small sample size and inconsistencies in the reported results.
Choice of Replication
We chose to replicate Experiment 3 in Glikson et al. (2018) for several reasons. First, this is the first replication of Glikson et al.’s (2018) experimental study (Experiment 3). The original study is situated in organizational behavioral and managerial research, fields that significantly lag behind others in terms of replication studies (Tenney et al., 2021).
Second, although the authors of the original study refer to power analysis recommendations, their study was underpowered to detect typical effect sizes in psychological research (Schäfer & Schwarz, 2019). Their final sample comprises 85 participants in total — around 21 participants in each of the four experiment groups. The number of participants in each group is not specified in the original article.
Third, Glikson et al. (2018) refer to several “marginally significant findings” (i.e., p-values that fall between .05 and .10), despite recommendations against such interpretations of p-values due to their low evidential value (e.g., Olsson-Collentine et al., 2019). Finally, statcheck (Rife et al., 2016) indicated inconsistency of calculated p-values in 10 out of the 15 tests reported (see “Statcheck” folder on the OSF page for details).1 These limitations challenge the validity of the conclusions presented and call for better-powered replications.
Overview of Current Replication
. | Glikson et al. (2018) . | Replication . |
---|---|---|
Sample size | 85 | 847 |
Condition | ||
Text only, Formal | Unknown | 207 |
Text only, Informal | Unknown | 214 |
Smileys, Formal | Unknown | 212 |
Smileys, Informal | Unknown | 214 |
Geographic origin | U.S. | U.S. |
Gender | 47% female | 49.7% female |
Age (average) | 36.16 | 38.24 |
Medium | Computer (online, MTurk) | Computer (online, Prolific) |
Compensation | Yes | Yes |
Year | 2018 or earlier | 2023 |
. | Glikson et al. (2018) . | Replication . |
---|---|---|
Sample size | 85 | 847 |
Condition | ||
Text only, Formal | Unknown | 207 |
Text only, Informal | Unknown | 214 |
Smileys, Formal | Unknown | 212 |
Smileys, Informal | Unknown | 214 |
Geographic origin | U.S. | U.S. |
Gender | 47% female | 49.7% female |
Age (average) | 36.16 | 38.24 |
Medium | Computer (online, MTurk) | Computer (online, Prolific) |
Compensation | Yes | Yes |
Year | 2018 or earlier | 2023 |
Classification of Replication
According to LeBel et al.’s (2017) taxonomy, the present study meets the criteria for an exact replication (see Table 2). The replication was identical to the original study in terms of stimulus materials and operationalization of measures. It differed from the original only in that data collection took place in 2023 (in 2018 or before in the original) and that participants were recruited from Prolific (MTurk in the original).
Design facet . | Replication . |
---|---|
Independent variable operationalization | Same |
Dependent variable operationalization | Same |
Independent variable stimuli | Same |
Dependent variable stimuli | Same |
Procedural details | Same |
Physical settings | Same |
Contextual variables | Different |
Replication classification | Exact replication |
Design facet . | Replication . |
---|---|
Independent variable operationalization | Same |
Dependent variable operationalization | Same |
Independent variable stimuli | Same |
Dependent variable stimuli | Same |
Procedural details | Same |
Physical settings | Same |
Contextual variables | Different |
Replication classification | Exact replication |
Method
Data, code, and materials associated with this paper are available at https://osf.io/n7yc4/. We report how we determined the sample size, all data exclusions, all manipulations, and all measures collected in this study (Simmons et al., 2012). The study was approved by NSD (Norwegian Centre for Research Data)/SIKT (ref. no. 249710) and the ethical review board at BI Norwegian Business School (ref. no. BI-ERB P025). The experiment is preregistered on AsPredicted (https://aspredicted.org/y63ph.pdf).
Participants and Design
Participants in the U.S. were recruited through Prolific, a platform that is similar to MTurk, which was used in the original study. Participants were paid £ 0.90 for a six-minute-long experiment. We aimed for a final sample size of 800 participants and thus preregistered to recruit 1000 participants to account for potential exclusions. Data collection yielded 1002 complete responses.
In order to detect inattentive participants and enhance data quality, we preregistered to exclude participants who either (a) completed the experiment in less than one minute (n = 129) or (b) provided non-differentiated responses to measures of the three dependent variables (n = 34). We also preregistered to exclude participants who failed an attention check that was introduced after measures of the dependent variables, in the form of a manipulation check (cf. Abbey & Meloy, 2017): (“What type of invitation did the sender of the email respond to?: (1) a staff meeting, (2) a job interview, or (3) a social gathering”). However, as it may be argued that exclusions based on a manipulation check can interfere with random assignment and induce an asymmetry across conditions (e.g., Aronow et al., 2019; Varaine, 2023), we performed analyses without exclusions based on the manipulation check.2
The final sample (without exclusions based on the manipulation check) comprises 847 participants. (Mage = 38.24, SDage = 12.51; 402 males (47.5%), 421 females (49.7%), 22 non-binary/third gender (2.6%), and two prefer not to say). Most participants held a bachelor’s degree (42.3%) or a high-school diploma (28.6%).
Table 1 shows a comparison of the target article sample and the replication sample (without exclusions). We conducted a sensitivity analysis using G*Power 3.1 (Faul et al., 2007) to examine the smallest effect size detectable with our sample. The key test in the original study was the interaction between smiley and formality conditions predicting warmth, competence, and appropriateness, while controlling for gender. We, therefore, conducted a sensitivity analysis for an ANCOVA. Assuming error probability of .05, power of .80, numerator df = 2 (degrees of freedom for the interaction), number of groups = 4 (two between-subject factors with two levels each), with two control variables, our experiment could detect an effect of Cohen’s f of .11 or larger.
Procedure and Measures
We used the experimental materials as presented in Table 6 in the original article. (See the Appendix for details.) Participants were instructed to read a short email (45 words) written by a new employee in response to an invitation by an unfamiliar administrative assistant. The formality of the setting was manipulated by a minor change in the wording of the invitation: the invitation was referred to either as a “staff meeting” (formal condition) or “social gathering” (informal condition). The sender of the email was denoted by a gender-neutral name (Alex Bledow). In the email, the sender greets the administrative assistant by “Dear Sarah”, introduces himself/herself as a new employee and thanks the administrative assistant for the invitation (to either the staff meeting or the social gathering). He/she then politely asks where it will be taking place. The email either included no smileys (control condition) or two smileys (smiley condition). In the smiley condition, one smiley was positioned after the initial introduction and one after the question about the location. The smileys used were in a standardized, default format (😊) and were exhibited in yellow in the experimental materials. Participants were asked to provide their first impression of the sender by rating the sender’s warmth and competence, and the appropriateness of the message. Perceived warmth was measured with four items: (a) warm, (b) friendly, (c) positive, and (d) nice. Perceived competence was measured with three items: (a) competent, (b) intelligent, and (c) hardworking. Perceived appropriateness of the message was measured with the following three items: (a) written appropriately, (b) well-articulated, and (c) norm-violating (reversed). Similar to the original study, all items were measured by using a Likert-type scale from 1 (strongly disagree) to 7 (strongly agree).
Following the original study, we preregistered to control for age, gender, and English proficiency if they correlated with any of the DVs. In addition, we included educational level as a control for exploratory purposes. Educational level was measured with the following scale: 1 (no diploma), 2 (high school diploma or equivalent), 3 (associate’s degree), 4 (bachelor’s degree), 5 (master’s degree and above).
We also included an item drawn from one of the other experiments (Experiment 2) in the original article to explore the possible role of the perceived gender of the sender. Although Glikson et al. (2018) did not find that perceived gender moderated the effects of smiley use on perceptions of warmth or competence, it seems plausible that gender stereotypes regarding emotions may lead to inferences about a person’s gender based on their use of emoticons (cf. Lea & Spears, 1992). Hence, we included this variable as a control for exploratory purposes.
For the three DVs (warmth, competence, and appropriateness), discriminant component analysis (direct Oblimin rotation) replicated the structure reported in the original study. Factor loadings ranged from .87 to .93 for perceived warmth (α = .92), from .75 to .88 for perceived competence (α = .84), and from .68 to .91 for perceived appropriateness (α = .81). 78.62% of the variance was explained by the formed factors. Kaiser-Meyer-Olkin measure of sampling adequacy (KMO = .853) and Bartlett’s test of sphericity (p < .001) indicated suitability of data for factor analysis.
Analysis and Results
Correlations and means are presented in Table 3 and Figure 1, respectively. Participants’ gender correlated with perceived warmth and competence, and similar to the original study, we controlled for participants’ gender in subsequent analyses. (The original study controlled for participants’ gender due to a significant correlation with perceived competence and appropriateness). To control for gender as a covariate, participants who did not report to be either male or female (n = 24) were excluded from the analyses, resulting in a subsample of 823 participants. However, to provide a check for robustness, the results from analyses without controlling for gender are also presented.
. | . | I . | M (SD) . | 1 . | 2 . | 3 . | 4 . | 5 . | 6 . | 7 . | 8 . | 9 . |
---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Smiley a | - | ||||||||||
2 | Formality b | .01 | ||||||||||
3 | Warmth | 4 | 6.06 (.71) | .23*** | .06 | [.93] | ||||||
4 | Competence | 3 | 5.38 (.91) | -.10** | .07 | .52*** | [.84] | |||||
5 | Appropriateness | 3 | 5.50 (1.23) | -.40*** | -.07 | .18*** | .51*** | [.81] | ||||
6 | Gender c | .51 (.50) | .02 | .03 | .08* | .08* | -.01 | |||||
7 | Age | 38.24 (12.50) | -.00 | .00 | -.05 | -.07 | .06 | .14*** | ||||
8 | English profic.d | 6.97 (.21) | .05 | .09** | .01 | -.01 | -.03 | .02 | .02 | |||
9 | Perc. Gender c | .30 (.46) | .27*** | -.00 | .08* | -.04 | -.18*** | .23*** | -.00 | .02 | ||
10 | Education e | 3.44 (1.10) | .06 | -.01 | -.02 | -.09** | -.05 | .05 | .10* | .01 | .13*** |
. | . | I . | M (SD) . | 1 . | 2 . | 3 . | 4 . | 5 . | 6 . | 7 . | 8 . | 9 . |
---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Smiley a | - | ||||||||||
2 | Formality b | .01 | ||||||||||
3 | Warmth | 4 | 6.06 (.71) | .23*** | .06 | [.93] | ||||||
4 | Competence | 3 | 5.38 (.91) | -.10** | .07 | .52*** | [.84] | |||||
5 | Appropriateness | 3 | 5.50 (1.23) | -.40*** | -.07 | .18*** | .51*** | [.81] | ||||
6 | Gender c | .51 (.50) | .02 | .03 | .08* | .08* | -.01 | |||||
7 | Age | 38.24 (12.50) | -.00 | .00 | -.05 | -.07 | .06 | .14*** | ||||
8 | English profic.d | 6.97 (.21) | .05 | .09** | .01 | -.01 | -.03 | .02 | .02 | |||
9 | Perc. Gender c | .30 (.46) | .27*** | -.00 | .08* | -.04 | -.18*** | .23*** | -.00 | .02 | ||
10 | Education e | 3.44 (1.10) | .06 | -.01 | -.02 | -.09** | -.05 | .05 | .10* | .01 | .13*** |
Note.*p < .05, **p < .01, ***p < .001 (two-tailed). N = 847. I = No. of items. M = Mean. SD = Standard deviation. a0 = Control, 1 = Smiley. b0 = Informal, 1 = Formal. cGender (n = 823) and Perceived gender of the sender (n = 654): 0 = Male, 1 = Female. dEnglish proficiency was measured on a scale from 1 (beginner/poor) to 7 (advanced/fluent). eEducational level was measured on a 5-point scale from 1 (no diploma) to 5 (master’s degree and above).
Neither age nor English proficiency correlated with any of the dependent variables in the present study and were therefore not included in further analyses. Educational level correlated with perceptions of competence and was tested as a covariate in exploratory analyses. Following the original paper, we converted partial eta-squared (ηp2) to Cohen’s d. First, we used the effectsize package (Ben-Shachar et al., 2020) in R to convert each partial eta-squared to Cohen’s f, which we then converted to Cohen’s d using an online conversion tool developed by Lenhard and Lenhard (2016). Alternatively, the following equation can be used to directly convert from partial eta-squared to Cohen’s d (Brysbaert, 2019):
Second, we used Hayes’ PROCESS macro to test the moderated mediation models.
Levene’s test indicated that the assumption of equal variances was not met. We nevertheless decided to proceed with our analysis given that ANOVA is generally robust to this violation, especially when the sample size is large and when samples are approximately equal across experimental groups (Jaccard & Becker, 2002).
Comparison of mean scores of the dependent variables shows that in the replication, overall ratings were higher with respect to perceptions of warmth (Moriginal = 5.77 vs Mreplication = 6.06) and competence (Moriginal = 5.16 vs Mreplication = 5.38) and slightly lower with respect to perceived appropriateness (Moriginal = 5.63 vs Mreplication = 5.50).
Perceived Warmth
The original study identified no significant main effects of smiley condition or formality condition on perceived warmth after controlling for gender. However, our analyses revealed a significant positive main effect of smiley (F(1, 818) = 44.64, p < .001, ηp2 = 0.052, 95% CI = 0.026, 0.084, d = 0.47), suggesting that smileys increased perceptions of warmth. Similar to the original study, the main effect of the formality condition was not significant (F(1, 818) = 2.27, p = .132, ηp2 = 0.003, 95% CI = 0.000, 0.014, d = 0.11), and the interaction between smiley condition and formality condition on perceptions of warmth was not significant (F(1, 818) = 2.58, p = .108, ηp2 = 0.003, 95% CI = 0.000, 0.015, d = 0.11). The original article referred to the interaction between smiley condition and formality condition (p = .058) as “marginally significant”.
The same pattern of findings held when not controlling for gender. Results revealed a significant positive main effect of smiley (F(1, 843) = 47.77, p < .001, ηp2 = 0.054, 95% CI = 0.028, 0.086, d = 0.48). The main effect of the formality condition was not significant (F(1, 843) = 3.03, p = .082, ηp2 = 0.004, 95% CI = 0.000, 0.016, d = 0.12), and the interaction between smiley condition and formality condition on perceptions of warmth was not significant (F(1, 843) = 1.89, p = .169, ηp2 = 0.002, 95% CI = 0.000, 0.013, d = 0.09).
Perceived Competence
Similar to the original study, after controlling for gender, our results indicate a significant negative main effect of smiley condition (F(1, 818) = 9.67, p = .002, ηp2 = 0.012, 95% CI = 0.002, 0.031, d = 0.22), whereas the main effect of formality condition was not significant (F(1, 818) = 2.82, p = .094, ηp2 = 0.003, 95% CI = 0.000, 0.016, d = 0.12). The original article presented the main effect of formality (p = .06) as “marginally significant”. The original study also found a significant interaction between smiley condition and formality condition (p < .001), whereas we do not find a significant interaction (F(1, 818) = 2.88, p = .090, ηp2 = 0.004, 95% CI = 0.000, 0.016, d = 0.12).
The results remained similar without controlling for gender. The main effect of smiley condition was significant and negative (F(1, 843) = 8.81, p = .003, ηp2 = 0.010, 95% CI = 0.001, 0.028, d = 0.20), the main effect of formality condition was not significant (F(1, 843) = 3.76, p = .053, ηp2 = 0.004, 95% CI = 0.000, 0.018, d = 0.13), and the interaction between smiley condition and formality condition was not significant (F(1, 843) = 1.67, p = .197, ηp2 = 0.002, 95% CI = 0.000, 0.012, d = 0.09).3
Perceived Appropriateness
Consistent with the original study, after controlling for gender, smileys produced a significant negative effect on perceptions of appropriateness (F(1, 818) = 159.36, p < .001, ηp2 = 0.163, 95% CI = 0.120, 0.208, d = 0.88), indicating that the use of smileys decreased perceptions of appropriateness. The formality condition produced a significant positive effect (F(1, 818) = 4.17, p = .041, ηp2 = 0.005, 95% CI = 0.000, 0.019, d = 0.14), indicating that perceived appropriateness was higher in the informal as compared to the formal condition. The original article, in contrast, referred to the main effect of formality condition (p = .059) as “marginally significant. Consistent with the original study, the interaction between smiley condition and formality condition was significant (F(1, 818) = 4.06, p = .044, ηp2 = 0.005, 95% CI = 0.000, 0.019, d = 0.14). Planned contrasts indicated that smileys were seen as less appropriate in both the informal condition (t(818) = 7.56, p < .0001, d = 0.52, 95% CI = 0.38, 0.66) and the formal condition, (t(818) = 10.28, p < .0001, d = 0.71, 95% CI = 0.57, 0.85), but the effect was stronger in the formal condition.
When not controlling for gender, the main effect of smileys was significant and negative (F(1, 843) = 166.48, p < .001, ηp2 = 0.165, 95% CI = 0.122, 0.210, d = 0.89). The main effect of formality condition was significant and was positive (F(1, 843) = 3.91, p = .048, ηp2 = 0.005, 95% CI = 0.000, 0.018, d = 0.14). However, the interaction between smiley condition and formality condition was no longer significant (F(1, 843) = 3.37, p = .067, ηp2 = 0.004, 95% CI = 0.000, 0.017, d = 0.13).4
Moderated Mediation
Next, we tested the same two moderated mediation models (Model 7) reported in the original study, using Hayes’ PROCESS macro (with 1,000 bootstrap resamples). We did not preregister any control variables in the moderated mediation analyses since the original study did not specify their inclusion. Nevertheless, results remained the same when controlling for gender.
The first mediation model that included perceived warmth as the dependent variable was not significant (index of moderated mediation: -0.05, 95% CI = -0.12, 0.005). The second mediation model that included perceived competence as the dependent variable was also not significant (index of moderated mediation: -0.12, 95% CI = -0.25, 0.01). The results from the moderated mediation analyses are summarized in Figure 2.5
Summary of Replication Results
. | Original article . | Replication . | |||
---|---|---|---|---|---|
Effect | p | Cohen’s d [95% CI] | p | Cohen’s d [95% CI] | Summary based on Lebel et al. (2019) |
Smileys → Warmth | .11 | 0.34 [–0.10, 0.78] | < .001 | 0.47 [0.33, 0.61] | Signal – Consistent |
Smileys → Competence | .001 | 0.72 [0.25, 1.18] | .002 | 0.22 [0.08, 0.36] | Signal – inconsistent, smaller |
Smileys → Appropriateness | .01 | 0.49 [0.04, 0.93] | < .001 | 0.88 [0.74, 1.02] | Signal – inconsistent, larger |
Formality → Warmth | .32 | 0.06 [–0.38, 0.49] | .132 | 0.11 [-0.02, 0.24] | No signal – Consistent |
Formality → Competence | .06 | 0.26 [–0.18, 0.70] | .094 | 0.12 [-0.01, 0.25] | No signal – Consistent |
Formality → Appropriateness | .059 | 0.31 [–0.13, 0.75] | .041 | 0.14 [0.01, 0.27] | Signal Inconsistent, smaller |
Smileys*Formality → Warmth | .058 | 0.49 [0.04, 0.93] | .108 | 0.11 [-0.02, 0.24] | No signal – Inconsistent, smaller |
Smileys*Formality → Competence | .001 | 0.52 [0.07, 0.97] | .090 | 0.12 [0.01, 0.25] | Signal – Inconsistent, smaller |
Smileys*Formality → Appropriateness | .003 | 0.45 [0.00, 0.89] | .044 | 0.14 [0.005, 0.27] | Signal – inconsistent, smaller |
Smileys*Formality → Warmth via Appropriateness | Not reported | –0.05 [-0.12, 0.005] | No signal – NA | ||
Smileys*Formality → Competence via Appropriateness | Not reported | –0.12 [-0.25, 0.01] | No signal – NA |
. | Original article . | Replication . | |||
---|---|---|---|---|---|
Effect | p | Cohen’s d [95% CI] | p | Cohen’s d [95% CI] | Summary based on Lebel et al. (2019) |
Smileys → Warmth | .11 | 0.34 [–0.10, 0.78] | < .001 | 0.47 [0.33, 0.61] | Signal – Consistent |
Smileys → Competence | .001 | 0.72 [0.25, 1.18] | .002 | 0.22 [0.08, 0.36] | Signal – inconsistent, smaller |
Smileys → Appropriateness | .01 | 0.49 [0.04, 0.93] | < .001 | 0.88 [0.74, 1.02] | Signal – inconsistent, larger |
Formality → Warmth | .32 | 0.06 [–0.38, 0.49] | .132 | 0.11 [-0.02, 0.24] | No signal – Consistent |
Formality → Competence | .06 | 0.26 [–0.18, 0.70] | .094 | 0.12 [-0.01, 0.25] | No signal – Consistent |
Formality → Appropriateness | .059 | 0.31 [–0.13, 0.75] | .041 | 0.14 [0.01, 0.27] | Signal Inconsistent, smaller |
Smileys*Formality → Warmth | .058 | 0.49 [0.04, 0.93] | .108 | 0.11 [-0.02, 0.24] | No signal – Inconsistent, smaller |
Smileys*Formality → Competence | .001 | 0.52 [0.07, 0.97] | .090 | 0.12 [0.01, 0.25] | Signal – Inconsistent, smaller |
Smileys*Formality → Appropriateness | .003 | 0.45 [0.00, 0.89] | .044 | 0.14 [0.005, 0.27] | Signal – inconsistent, smaller |
Smileys*Formality → Warmth via Appropriateness | Not reported | –0.05 [-0.12, 0.005] | No signal – NA | ||
Smileys*Formality → Competence via Appropriateness | Not reported | –0.12 [-0.25, 0.01] | No signal – NA |
Note. Replication summary based on LeBel et al.’s (2019) taxonomy: (1) whether a signal was detected (i.e., whether the 95% confidence interval, or CI, excludes 0); (2) the consistency of the replication effect size estimate with that observed in the original study (i.e., whether the replication’s CI includes the original effect size point estimate); and (3) the magnitude of the replication’s effect size estimate in the same direction compared to original effect size. Cohen’s ds were converted from partial eta-squared values in ANOVA.
Our results detected a signal in three tests out of all the tests reported in the original. These involved the main effects of smiley usage on the three DVs. The effect of smiley on warmth was reported as nonsignificant in the original, and the effect of smiley on appropriateness was larger in the replication.
Finally, we assessed replication success using the small telescopes approach (Simonsohn, 2015) by calculating the power that the original study would have to detect the replication’s effect size for the key effects, namely, the interaction between smiley and formality in predicting the three DVs. This approach considers a result to be a failed replication if the original study lacked sufficient power—less than 33% power—to detect the replication result’s detected effect sizes. We used the pwr package in R (Champely et al., 2018).
The sample size of the original study was 85, which would give it 12% power to detect the replication’s effect size for the interaction effect predicting perceived warmth (upper bound = 45% power), 13% power to detect the interactive effect on perceived competence (upper bound = 48% power), and 17% power to detect the interactive effect on appropriateness (upper bound = 56%). These results suggest that the original study was severely underpowered to detect any of the three key effects reported in the current replication and that our study failed to replicate the original findings.
Exploratory Analyses
Perceived Gender
We performed a Chi-square test to examine differences in participants’ perceived gender of the email sender. Across conditions, 457 (54.0%) participants perceived the sender as male, 197 (23.3%) perceived the sender as female, whereas 193 (22.8%) reported being uncertain or that they did not think about a particular gender. In the control condition (no smiley), the sender was significantly more often identified as a male (82.3%) than a female (χ2 (1, 654) = 48.37, p < .001).
Next, we followed the original study and performed a series of ANCOVAs testing the effect of the perceived gender of the sender on perceptions of warmth and competence, controlling for English proficiency.
Consistent with the results from the original study, our results reveal no direct association between perceived gender and perceptions of competence (F(1, 651) = 0.57, p = .449, ηp2 = 0.001, 95% CI = 0.000, 0.011, d = 0.06, BF10 = 0.78). Perceived gender was, however, positively associated with perceptions of warmth (F(1, 651) = 4.33, p = .038, ηp2 = 0.007, 95% CI = 0.000, 0.024, d = 0.16, BF10 = 0.17) and negatively associated with perceptions of appropriateness (F(1, 651) = 21.75, p < .001, ηp2 = 0.032, 95% CI = 0.011, 0.063, d = 0.37, BF10 = 3454.90), indicating that ratings of warmth were higher whereas ratings of appropriateness were lower when the sender was perceived as female (as opposed to male).
Perceived gender did not moderate the effects of smiley condition on perceptions of warmth (F(1, 649) = 0.05 p = .828, ηp2 = 0.000, 95% CI = 0.000, 0.006, d = 0.02, BF10 = 435284.9) or competence (F(1, 649) = 2.64, p = .105, ηp2 = 0.004, 95% CI = 0.000, 0.019, d = 0.13, BF10 = 0.09), but moderated the effect of smileys on perceptions of appropriateness (F(1, 649) = 5.14, p = .024, ηp2 = 0.008, 95% CI = 0.000, 0.027, d = 0.18, BF10 = 1.16). Follow-up contrasts indicated that smileys decreased perceptions of appropriateness when the sender was perceived as being male (t(649) = 7.25, p < .001, d = 0.57, 95% CI = 0.45, 0.79) and female (t(649) = 7.05, p < .001, d = 0.55, 95% CI = 0.43, 0.77), but the effect was stronger when the sender was perceived as a female.
Controlling for Educational Level in ANOVA Predicting Perceived Competence
Due to the correlation between participants’ educational level and perceptions of competence, we performed exploratory analyses using educational level as a control variable (excluding gender as a control). The results remained similar. The main effect of smileys on perceived competence was significant and negative (F(1, 842) = 8.03, p = .005, ηp2 = 0.009, 95% CI = 0.001, 0.027, d = 0.20), the main effect of formality condition was not significant (F(1, 842) = 3.67, p = .056, ηp2 = 0.004, 95% CI = 0.000, 0.018, d = 0.13), and the interaction between smiley condition and formality was not significant (F(1, 842) = 1.70, p = .192, ηp2 = 0.002, 95% CI = 0.000, 0.013, d = 0.09).
General Discussion
We performed a high-powered replication of Experiment 3 in Glikson et al. (2018). The original results indicated that smileys have a positive effect on perceived warmth in an informal setting and a negative effect on perceived competence in a formal setting, and that these effects could partly be attributed to perceptions of appropriateness.
We found that smileys have a positive, medium effect on perceptions of warmth and a negative, small effect on perceptions of competence regardless of the level of formality, and no support for a partly mediating role of perceived appropriateness.
One potential explanation for the discrepancies in results between the two studies pertains to statistical power. Based on Simonsohn’s (2015) small-telescope approach, the original study seems to have been underpowered to detect the key effects reported in the original article and severely underpowered to detect the key effects observed in the current replication study.
Another potential explanation for the discrepancies in results is that, as proposed by Kaye et al. (2016), the increasing use of computer-mediated text-based communication has produced a shift in the perceived appropriateness of emoticons and emojis in work-related email, that is, an increased general acceptance of their usage. If so, one may expect that the direct effects of smileys as well as the (partly) mediating role of perceived appropriateness on perceived warmth and competence, as suggested by Glikson et al. (2018), will be deflated over time. The potentially deflating effect of using smileys may also be seen in the context of the general trend towards more informal writing in many domains, including business correspondence, administrative documents, journalism, and scientific writing (cf. Hyland & Jiang, 2017).
Although the time span from the collection of the original data (2018 or prior) to 2023 is relatively short, the Covid-19 pandemic accelerated digitally based communication and the volume of email, instant messaging, and social network use, due to the increase in remote working and other measures to reduce physical interaction. Hence, the rub-off effect of emoji use in instant messaging and social media may have accelerated during the pandemic. Accordingly, although smileys were found to decrease the perceived appropriateness of the message, the possibility that the use of emoticons or emojis in work-related emails has become more generally acceptable and hence of less significance when forming first impressions of the sender cannot be dismissed.
Yet, it seems reasonable to assume that acceptance and hence perceived appropriateness of smileys and other emojis will vary dependent on the level of formality in the organization as well as the degree of formality of the communication, which in turn may produce different and potentially enhanced effects on perceptions of the sender’s warmth and competence. The experimental materials used in the original study and the replication reported here did not include experimental conditions that provide the opportunity to compare organizations characterized by a high versus low level of formality in communication.
Hence, it is possible that the formality manipulations were too weak with respect to activating a realistic sense of email communication in a formal versus informal work-related setting. The formality of the setting was manipulated by only a subtle change in wording, that is, “staff meeting” versus “social gathering”. Although the manipulation check in the replication study revealed that the majority of participants (93.7%) recalled the correct formality treatment, it remains unclear whether participants experienced a sufficient sense of realism and immersion based on the respective treatments (cf. Aguinis & Bradley, 2014; Alekseev et al., 2017).
In the vignettes, the first name of the recipient only (“Dear Sarah”) was used as the initial greeting in both settings. While the latter may be customary in many organizations, an even more formal tone tends to be used in some organizations, for example, by including the last name of the recipient and potentially the job title or educational degree, particularly when making initial contact via professional email. Moreover, the experimental materials did not reveal the style and level of formality of the initial email to which the sender replies. Future research is therefore needed to investigate the degree to which the level of formality in the organization and of the specific setting may have bearing on the effects of using smileys and other emojis on first impressions of the sender’s warmth and competence.
Future research could also further investigate the potential influence of the gender of the sender, the receiver, and the observer, respectively. The experimental materials used in the original study and the present replication depict an email that is sent by a person with a gender-neutral name (“Alex”) to a female receiver (“Sarah”). Although, consistent with the original study, our exploratory analyses did not reveal an interaction between the perceived gender of the sender and smileys on participants perceptions of the sender’s warmth or competence, our findings do indicate that smileys are perceived as less appropriate when used by a perceived female sender. Yet, several studies indicate that women use emojis such as smileys more frequently than men, and that emoji preferences vary across genders (e.g., Koch et al., 2022; Oleszkiewicz et al., 2017). The pattern of findings could therefore be retested by explicitly specifying the gender of the sender through the use of two distinct gender-specific names, each representing either a female or a male sender. Similarly, since the experimental materials depict a female receiver only, future research is needed to assess the generalizability of the pattern of findings to a male recipient. Future research could also explore the potential role of the observer’s gender when forming first impressions based on emails addressed to unknown individuals, whether female or male, in comparison to emails where the observer is presented as the recipient.
Last, but not least, future research should investigate the generalizability of the findings derived from the U.S. to other countries. The U.S. is generally considered as a society characterized by high levels of masculinity and individualism, as well as a relatively low level of power distance (cf. Hofstede, 1983). Countries with other cultural profiles may place different cultural emphasis on informal communication and leveling with others, which in turn may have bearing on the effects of smiley usage on perceptions of the sender’s warmth and competence, and the appropriateness of the message in work-related email correspondence.
Conclusion
The results from the high-powered replication of Experiment 3 in Glikson et al. (2018) reported here indicate that smileys not only have a dark side by deflating first impressions of competence, as reported by Glikson et al. (2018), but also a bright side by enhancing first impressions of warmth. In contrast to Glikson et al. (2018), however, our results indicate that these effects are not sensitive to the degree of formality of the setting and are not mediated by the perceived appropriateness of the message. Future research is needed to better investigate the potential moderating role of formality as well as the gender of the sender and recipient, respectively. Hopefully, the present replication inspires further inquiries into the use of emojis and emoticons and their potential effects in work settings.
Contributions
Contributed to conception and design: LL
Contributed to acquisition of data: LL, LM
Contributed to analysis and interpretation of data: LL, LM
Drafted and/or revised the article: LL, LM
Approved the submitted version for publication: LL, LM
Acknowledgements
The authors would like to thank Senior Editor Elizabeth R. Tenney and the two anonymous reviewers for their valuable comments and suggestions for improvements in a previous version of this manuscript.
Competing Interests
The authors report no conflict of interest.
Data Accessibility Statement
Data, code, and materials associated with this paper are available at https://osf.io/n7yc4/.
APPENDIX
Experimental materials: The email presented to participants in each of the four experimental conditions. Control condition / Formal condition
Dear Sarah,
My name is Alex and I’ve started working here this week.
Thank you for sending me the invitation to join the staff meeting on Friday.
Could you please let me know where it will be taking place?
Thank you very much, Alex Bledow
Control condition / Informal condition
Dear Sarah,
My name is Alex and I’ve started working here this week.
Thank you for sending me the invitation to join the social gathering on Friday.
Could you please let me know where it will be taking place?
Thank you very much, Alex Bledow
Smiley condition / Formal condition
Dear Sarah,
My name is Alex and I’ve started working here this week. 😊
Thank you for sending me the invitation to join the staff meeting on Friday.
Could you please let me know where it will be taking place? 😊
Thank you very much, Alex Bledow
Smiley condition / Informal condition
Dear Sarah,
My name is Alex and I’ve started working here this week. 😊
Thank you for sending me the invitation to join the social gathering on Friday.
Could you please let me know where it will be taking place? 😊
Thank you very much, Alex Bledow
____________________________
Note: Drawn from Table 6 in Glikson et al. (2018).
Footnotes
Statcheck is a free and open-source tool that extracts reported statistical results from academic papers and recalculates the p-values. Inconsistency of p-values denotes that recalculated p-values, based on the reported statistical test and degrees of freedom, do not match the p-values reported by the authors (Nuijten & Polanin, 2020).
63 participants failed the manipulation check yet did not meet any of the other exclusion criteria. The proportion of male participants was higher (61.9% versus 46.3%) among participants who failed the manipulation check, and the mean age was lower (32.22 versus 38.72) compared to participants who passed the manipulation check. Discrepancies in results between analyses without and with exclusions based on the manipulation check are reported in footnotes.
When excluding participants based on the manipulation check and controlling for gender, the main effect of formality condition became statistically significant (F(1, 757) = 3.94, p = .047, ηp2 = 0.005, 95% CI = 0.000, 0.020, d = 0.14), indicating that the sender was perceived as more competent in the formal as opposed to informal condition. Our analyses produced the same pattern of results when not controlling for gender.
Analyses with exclusions based on the manipulation check, and after controlling for gender, indicated that the main effect of formality (F(1, 757) = 3.79, p = .052, ηp2 = 0.005, 95% CI = 0.000, 0.020, d = 0.14) and the interaction between smiley condition and formality condition (F(1, 757) = 3.02, p = .083, ηp2 = 0.004, 95% CI = 0.000, 0.018, d = 0.13) were not significant. The same pattern of results held when not controlling for gender.
The moderated mediation indices remained insignificant when excluding participants based on the manipulation check.