The present paper reports an independent and better powered (N = 847 vs. N = 85) replication of Experiment 3 in Glikson et al. (2018). The authors of the original study reported support for their proposition that due to perceptions of (in)appropriateness, the use of smileys may backfire and produce less favorable perceptions of competence in a formal work-related setting, yet more favorable perceptions of warmth in an informal work-related setting. Our results, in contrast, indicated that smileys produce a negative effect on perceptions of competence and a positive effect on perceptions of warmth, regardless of the level of formality. Moreover, our results did not support the reported moderated mediation model involving perceptions of appropriateness. Potential explanations for the discrepancies in results are discussed. We provide data, code, and materials on https://osf.io/n7yc4/.

Communication is increasingly taking place on text-based, computer-mediated platforms with limited opportunities to express emotions. However, digital representations of emotions, such as the prevalent “smiley” emoticon (typographic) and “smiley emoji” (pictographic), are frequently used in instant messaging and social media, and also seem to be increasingly used in professional settings, including professional emails. Skovholt et al. (2014) found that emoticons in workplace emails are typically used to signal a positive attitude, to mark humor, jokes, or irony, and to strengthen or soften the content communicated. Yet, despite their prevalent use, the interpretation of emoticons and emojis and their intended meaning can be challenging (Rodrigues et al., 2018) and highly reliant on the context. Indeed, some scholars argue that emoticons and emojis may have a negative effect in formal or work-related contexts due to impressions of, for example, insufficient verbal skills, lack of professionalism, or inappropriateness (e.g., Munter et al., 2003). This line of reasoning is supported by the findings of Kaye et al. (2016), which indicate that emoticon use on email platforms (as opposed to direct messaging and social networks) is perceived as inappropriate by many users.

So far, few studies have explicitly investigated the role of perceived appropriateness of the use of emoticons and emojis in professional digitally based communication. Exceptions include Glikson et al. (2018, Experiment 3), who reported the results from an experiment aimed to test the formality of the context as a moderator of the effects of smileys on first impressions of the sender’s warmth and competence, as well as the mediating role of perceptions of appropriateness. The authors conclude that the effects of smileys on first impressions of warmth and competence are contingent on the formality of the setting. In a formal setting, smileys were found to have a negative effect on perceptions of competence and no effect on perceptions of warmth. In an informal setting, in contrast, smileys were found to have no effect on perceptions of competence and a positive effect on perceptions of warmth. Perceptions of (in)appropriateness were found to partially mediate these effects.

Warmth and competence have previously been found to represent two fundamental dimensions of person perception (Fiske et al., 2002, 2007; Judd et al., 2005) that affect a series of important outcomes in social and work-related settings. The results presented by Glikson et al. (2018) may therefore have apparent practical implications. Provided that the use of smileys produces negative effects in a formal work setting, that is, deflates first impressions of the sender’s competence, senders are best advised to avoid smileys when initiating (email) contact in these settings.

However, to our knowledge, there are no direct replications of this study, despite several apparent methodological limitations associated with the presented results, including low statistical power due to very small sample size and inconsistencies in the reported results.

Choice of Replication

We chose to replicate Experiment 3 in Glikson et al. (2018) for several reasons. First, this is the first replication of Glikson et al.’s (2018) experimental study (Experiment 3). The original study is situated in organizational behavioral and managerial research, fields that significantly lag behind others in terms of replication studies (Tenney et al., 2021).

Second, although the authors of the original study refer to power analysis recommendations, their study was underpowered to detect typical effect sizes in psychological research (Schäfer & Schwarz, 2019). Their final sample comprises 85 participants in total — around 21 participants in each of the four experiment groups. The number of participants in each group is not specified in the original article.

Third, Glikson et al. (2018) refer to several “marginally significant findings” (i.e., p-values that fall between .05 and .10), despite recommendations against such interpretations of p-values due to their low evidential value (e.g., Olsson-Collentine et al., 2019). Finally, statcheck (Rife et al., 2016) indicated inconsistency of calculated p-values in 10 out of the 15 tests reported (see “Statcheck” folder on the OSF page for details).1 These limitations challenge the validity of the conclusions presented and call for better-powered replications.

We conducted a preregistered experiment that aimed to replicate Study 3 in Glikson et al. (2018). We used the same experimental material and a similar sample (U.S. citizens). Table 1 shows a comparison of the target article sample and the replication sample.

Table 1.
Differences and similarities between samples in original study and replication experiments
Glikson et al. (2018) Replication
Sample size 85 847 
Condition   
Text only, Formal Unknown 207 
Text only, Informal Unknown 214 
Smileys, Formal Unknown 212 
Smileys, Informal Unknown 214 
Geographic origin U.S. U.S. 
Gender 47% female 49.7% female 
Age (average) 36.16 38.24 
Medium Computer
(online, MTurk) 
Computer
(online, Prolific) 
Compensation Yes Yes 
Year 2018 or earlier 2023 
Glikson et al. (2018) Replication
Sample size 85 847 
Condition   
Text only, Formal Unknown 207 
Text only, Informal Unknown 214 
Smileys, Formal Unknown 212 
Smileys, Informal Unknown 214 
Geographic origin U.S. U.S. 
Gender 47% female 49.7% female 
Age (average) 36.16 38.24 
Medium Computer
(online, MTurk) 
Computer
(online, Prolific) 
Compensation Yes Yes 
Year 2018 or earlier 2023 

Classification of Replication

According to LeBel et al.’s (2017) taxonomy, the present study meets the criteria for an exact replication (see Table 2). The replication was identical to the original study in terms of stimulus materials and operationalization of measures. It differed from the original only in that data collection took place in 2023 (in 2018 or before in the original) and that participants were recruited from Prolific (MTurk in the original).

Table 2.
Classification of the replication based on LeBel et al.’s (2017) taxonomy
Design facetReplication
Independent variable operationalization Same 
Dependent variable operationalization Same 
Independent variable stimuli Same 
Dependent variable stimuli Same 
Procedural details Same 
Physical settings Same 
Contextual variables Different 
Replication classification Exact replication 
Design facetReplication
Independent variable operationalization Same 
Dependent variable operationalization Same 
Independent variable stimuli Same 
Dependent variable stimuli Same 
Procedural details Same 
Physical settings Same 
Contextual variables Different 
Replication classification Exact replication 

Data, code, and materials associated with this paper are available at https://osf.io/n7yc4/. We report how we determined the sample size, all data exclusions, all manipulations, and all measures collected in this study (Simmons et al., 2012). The study was approved by NSD (Norwegian Centre for Research Data)/SIKT (ref. no. 249710) and the ethical review board at BI Norwegian Business School (ref. no. BI-ERB P025). The experiment is preregistered on AsPredicted (https://aspredicted.org/y63ph.pdf).

Participants and Design

Participants in the U.S. were recruited through Prolific, a platform that is similar to MTurk, which was used in the original study. Participants were paid £ 0.90 for a six-minute-long experiment. We aimed for a final sample size of 800 participants and thus preregistered to recruit 1000 participants to account for potential exclusions. Data collection yielded 1002 complete responses.

In order to detect inattentive participants and enhance data quality, we preregistered to exclude participants who either (a) completed the experiment in less than one minute (n = 129) or (b) provided non-differentiated responses to measures of the three dependent variables (n = 34). We also preregistered to exclude participants who failed an attention check that was introduced after measures of the dependent variables, in the form of a manipulation check (cf. Abbey & Meloy, 2017): (“What type of invitation did the sender of the email respond to?: (1) a staff meeting, (2) a job interview, or (3) a social gathering”). However, as it may be argued that exclusions based on a manipulation check can interfere with random assignment and induce an asymmetry across conditions (e.g., Aronow et al., 2019; Varaine, 2023), we performed analyses without exclusions based on the manipulation check.2

The final sample (without exclusions based on the manipulation check) comprises 847 participants. (Mage = 38.24, SDage = 12.51; 402 males (47.5%), 421 females (49.7%), 22 non-binary/third gender (2.6%), and two prefer not to say). Most participants held a bachelor’s degree (42.3%) or a high-school diploma (28.6%).

Table 1 shows a comparison of the target article sample and the replication sample (without exclusions). We conducted a sensitivity analysis using G*Power 3.1 (Faul et al., 2007) to examine the smallest effect size detectable with our sample. The key test in the original study was the interaction between smiley and formality conditions predicting warmth, competence, and appropriateness, while controlling for gender. We, therefore, conducted a sensitivity analysis for an ANCOVA. Assuming error probability of .05, power of .80, numerator df = 2 (degrees of freedom for the interaction), number of groups = 4 (two between-subject factors with two levels each), with two control variables, our experiment could detect an effect of Cohen’s f of .11 or larger.

Procedure and Measures

We used the experimental materials as presented in Table 6 in the original article. (See the Appendix for details.) Participants were instructed to read a short email (45 words) written by a new employee in response to an invitation by an unfamiliar administrative assistant. The formality of the setting was manipulated by a minor change in the wording of the invitation: the invitation was referred to either as a “staff meeting” (formal condition) or “social gathering” (informal condition). The sender of the email was denoted by a gender-neutral name (Alex Bledow). In the email, the sender greets the administrative assistant by “Dear Sarah”, introduces himself/herself as a new employee and thanks the administrative assistant for the invitation (to either the staff meeting or the social gathering). He/she then politely asks where it will be taking place. The email either included no smileys (control condition) or two smileys (smiley condition). In the smiley condition, one smiley was positioned after the initial introduction and one after the question about the location. The smileys used were in a standardized, default format (😊) and were exhibited in yellow in the experimental materials. Participants were asked to provide their first impression of the sender by rating the sender’s warmth and competence, and the appropriateness of the message. Perceived warmth was measured with four items: (a) warm, (b) friendly, (c) positive, and (d) nice. Perceived competence was measured with three items: (a) competent, (b) intelligent, and (c) hardworking. Perceived appropriateness of the message was measured with the following three items: (a) written appropriately, (b) well-articulated, and (c) norm-violating (reversed). Similar to the original study, all items were measured by using a Likert-type scale from 1 (strongly disagree) to 7 (strongly agree).

Following the original study, we preregistered to control for age, gender, and English proficiency if they correlated with any of the DVs. In addition, we included educational level as a control for exploratory purposes. Educational level was measured with the following scale: 1 (no diploma), 2 (high school diploma or equivalent), 3 (associate’s degree), 4 (bachelor’s degree), 5 (master’s degree and above).

We also included an item drawn from one of the other experiments (Experiment 2) in the original article to explore the possible role of the perceived gender of the sender. Although Glikson et al. (2018) did not find that perceived gender moderated the effects of smiley use on perceptions of warmth or competence, it seems plausible that gender stereotypes regarding emotions may lead to inferences about a person’s gender based on their use of emoticons (cf. Lea & Spears, 1992). Hence, we included this variable as a control for exploratory purposes.

For the three DVs (warmth, competence, and appropriateness), discriminant component analysis (direct Oblimin rotation) replicated the structure reported in the original study. Factor loadings ranged from .87 to .93 for perceived warmth (α = .92), from .75 to .88 for perceived competence (α = .84), and from .68 to .91 for perceived appropriateness (α = .81). 78.62% of the variance was explained by the formed factors. Kaiser-Meyer-Olkin measure of sampling adequacy (KMO = .853) and Bartlett’s test of sphericity (p < .001) indicated suitability of data for factor analysis.

Analysis and Results

Correlations and means are presented in Table 3 and Figure 1, respectively. Participants’ gender correlated with perceived warmth and competence, and similar to the original study, we controlled for participants’ gender in subsequent analyses. (The original study controlled for participants’ gender due to a significant correlation with perceived competence and appropriateness). To control for gender as a covariate, participants who did not report to be either male or female (n = 24) were excluded from the analyses, resulting in a subsample of 823 participants. However, to provide a check for robustness, the results from analyses without controlling for gender are also presented.

Table 3.
Correlations and Descriptive Statistics
IM (SD)123456789
Smiley a           
Formality b   .01         
Warmth 6.06 (.71) .23*** .06 [.93]       
Competence 5.38 (.91) -.10** .07 .52*** [.84]      
Appropriateness 5.50 (1.23) -.40*** -.07 .18*** .51*** [.81]     
Gender c  .51 (.50) .02 .03 .08* .08* -.01     
Age  38.24 (12.50) -.00 .00 -.05 -.07 .06 .14***    
English profic.d  6.97 (.21) .05 .09** .01 -.01 -.03 .02 .02   
Perc. Gender c  .30 (.46) .27*** -.00 .08* -.04 -.18*** .23*** -.00 .02  
10 Education e  3.44 (1.10) .06 -.01 -.02 -.09** -.05 .05 .10* .01 .13*** 
IM (SD)123456789
Smiley a           
Formality b   .01         
Warmth 6.06 (.71) .23*** .06 [.93]       
Competence 5.38 (.91) -.10** .07 .52*** [.84]      
Appropriateness 5.50 (1.23) -.40*** -.07 .18*** .51*** [.81]     
Gender c  .51 (.50) .02 .03 .08* .08* -.01     
Age  38.24 (12.50) -.00 .00 -.05 -.07 .06 .14***    
English profic.d  6.97 (.21) .05 .09** .01 -.01 -.03 .02 .02   
Perc. Gender c  .30 (.46) .27*** -.00 .08* -.04 -.18*** .23*** -.00 .02  
10 Education e  3.44 (1.10) .06 -.01 -.02 -.09** -.05 .05 .10* .01 .13*** 

Note.*p < .05, **p < .01, ***p < .001 (two-tailed). N = 847. I = No. of items. M = Mean. SD = Standard deviation. a0 = Control, 1 = Smiley. b0 = Informal, 1 = Formal. cGender (n = 823) and Perceived gender of the sender (n = 654): 0 = Male, 1 = Female. dEnglish proficiency was measured on a scale from 1 (beginner/poor) to 7 (advanced/fluent). eEducational level was measured on a 5-point scale from 1 (no diploma) to 5 (master’s degree and above).

Figure 1.
Perceptions of warmth and competence as a function of smiley use in formal and informal contexts, including standard errors
Figure 1.
Perceptions of warmth and competence as a function of smiley use in formal and informal contexts, including standard errors
Close modal

Neither age nor English proficiency correlated with any of the dependent variables in the present study and were therefore not included in further analyses. Educational level correlated with perceptions of competence and was tested as a covariate in exploratory analyses. Following the original paper, we converted partial eta-squared (ηp2) to Cohen’s d. First, we used the effectsize package (Ben-Shachar et al., 2020) in R to convert each partial eta-squared to Cohen’s f, which we then converted to Cohen’s d using an online conversion tool developed by Lenhard and Lenhard (2016). Alternatively, the following equation can be used to directly convert from partial eta-squared to Cohen’s d (Brysbaert, 2019):

d=2×η2p1η2p

Second, we used Hayes’ PROCESS macro to test the moderated mediation models.

Levene’s test indicated that the assumption of equal variances was not met. We nevertheless decided to proceed with our analysis given that ANOVA is generally robust to this violation, especially when the sample size is large and when samples are approximately equal across experimental groups (Jaccard & Becker, 2002).

Comparison of mean scores of the dependent variables shows that in the replication, overall ratings were higher with respect to perceptions of warmth (Moriginal = 5.77 vs Mreplication = 6.06) and competence (Moriginal = 5.16 vs Mreplication = 5.38) and slightly lower with respect to perceived appropriateness (Moriginal = 5.63 vs Mreplication = 5.50).

Perceived Warmth

The original study identified no significant main effects of smiley condition or formality condition on perceived warmth after controlling for gender. However, our analyses revealed a significant positive main effect of smiley (F(1, 818) = 44.64, p < .001, ηp2 = 0.052, 95% CI = 0.026, 0.084, d = 0.47), suggesting that smileys increased perceptions of warmth. Similar to the original study, the main effect of the formality condition was not significant (F(1, 818) = 2.27, p = .132, ηp2 = 0.003, 95% CI = 0.000, 0.014, d = 0.11), and the interaction between smiley condition and formality condition on perceptions of warmth was not significant (F(1, 818) = 2.58, p = .108, ηp2 = 0.003, 95% CI = 0.000, 0.015, d = 0.11). The original article referred to the interaction between smiley condition and formality condition (p = .058) as “marginally significant”.

The same pattern of findings held when not controlling for gender. Results revealed a significant positive main effect of smiley (F(1, 843) = 47.77, p < .001, ηp2 = 0.054, 95% CI = 0.028, 0.086, d = 0.48). The main effect of the formality condition was not significant (F(1, 843) = 3.03, p = .082, ηp2 = 0.004, 95% CI = 0.000, 0.016, d = 0.12), and the interaction between smiley condition and formality condition on perceptions of warmth was not significant (F(1, 843) = 1.89, p = .169, ηp2 = 0.002, 95% CI = 0.000, 0.013, d = 0.09).

Perceived Competence

Similar to the original study, after controlling for gender, our results indicate a significant negative main effect of smiley condition (F(1, 818) = 9.67, p = .002, ηp2 = 0.012, 95% CI = 0.002, 0.031, d = 0.22), whereas the main effect of formality condition was not significant (F(1, 818) = 2.82, p = .094, ηp2 = 0.003, 95% CI = 0.000, 0.016, d = 0.12). The original article presented the main effect of formality (p = .06) as “marginally significant”. The original study also found a significant interaction between smiley condition and formality condition (p < .001), whereas we do not find a significant interaction (F(1, 818) = 2.88, p = .090, ηp2 = 0.004, 95% CI = 0.000, 0.016, d = 0.12).

The results remained similar without controlling for gender. The main effect of smiley condition was significant and negative (F(1, 843) = 8.81, p = .003, ηp2 = 0.010, 95% CI = 0.001, 0.028, d = 0.20), the main effect of formality condition was not significant (F(1, 843) = 3.76, p = .053, ηp2 = 0.004, 95% CI = 0.000, 0.018, d = 0.13), and the interaction between smiley condition and formality condition was not significant (F(1, 843) = 1.67, p = .197, ηp2 = 0.002, 95% CI = 0.000, 0.012, d = 0.09).3

Perceived Appropriateness

Consistent with the original study, after controlling for gender, smileys produced a significant negative effect on perceptions of appropriateness (F(1, 818) = 159.36, p < .001, ηp2 = 0.163, 95% CI = 0.120, 0.208, d = 0.88), indicating that the use of smileys decreased perceptions of appropriateness. The formality condition produced a significant positive effect (F(1, 818) = 4.17, p = .041, ηp2 = 0.005, 95% CI = 0.000, 0.019, d = 0.14), indicating that perceived appropriateness was higher in the informal as compared to the formal condition. The original article, in contrast, referred to the main effect of formality condition (p = .059) as “marginally significant. Consistent with the original study, the interaction between smiley condition and formality condition was significant (F(1, 818) = 4.06, p = .044, ηp2 = 0.005, 95% CI = 0.000, 0.019, d = 0.14). Planned contrasts indicated that smileys were seen as less appropriate in both the informal condition (t(818) = 7.56, p < .0001, d = 0.52, 95% CI = 0.38, 0.66) and the formal condition, (t(818) = 10.28, p < .0001, d = 0.71, 95% CI = 0.57, 0.85), but the effect was stronger in the formal condition.

When not controlling for gender, the main effect of smileys was significant and negative (F(1, 843) = 166.48, p < .001, ηp2 = 0.165, 95% CI = 0.122, 0.210, d = 0.89). The main effect of formality condition was significant and was positive (F(1, 843) = 3.91, p = .048, ηp2 = 0.005, 95% CI = 0.000, 0.018, d = 0.14). However, the interaction between smiley condition and formality condition was no longer significant (F(1, 843) = 3.37, p = .067, ηp2 = 0.004, 95% CI = 0.000, 0.017, d = 0.13).4

Moderated Mediation

Next, we tested the same two moderated mediation models (Model 7) reported in the original study, using Hayes’ PROCESS macro (with 1,000 bootstrap resamples). We did not preregister any control variables in the moderated mediation analyses since the original study did not specify their inclusion. Nevertheless, results remained the same when controlling for gender.

The first mediation model that included perceived warmth as the dependent variable was not significant (index of moderated mediation: -0.05, 95% CI = -0.12, 0.005). The second mediation model that included perceived competence as the dependent variable was also not significant (index of moderated mediation: -0.12, 95% CI = -0.25, 0.01). The results from the moderated mediation analyses are summarized in Figure 2.5

Figure 2.
Moderated mediation models
Figure 2.
Moderated mediation models
Close modal

Summary of Replication Results

Table 4 provides a summary of the replication results based on LeBel et al.’s (2019) taxonomy.

Table 4.
Summary of replication findings
Original articleReplication
Effect p Cohen’s d
[95% CI] 
p Cohen’s d
[95% CI] 
Summary
based on Lebel et al. (2019)  
Smileys → Warmth .11 0.34 [–0.10, 0.78] < .001 0.47 [0.33, 0.61] Signal –
Consistent 
Smileys → Competence .001 0.72 [0.25, 1.18] .002 0.22 [0.08, 0.36] Signal –
inconsistent, smaller 
Smileys → Appropriateness .01 0.49 [0.04, 0.93] < .001 0.88 [0.74, 1.02] Signal –
inconsistent, larger 
Formality → Warmth .32 0.06 [–0.38, 0.49] .132 0.11 [-0.02, 0.24] No signal –
Consistent 
Formality → Competence .06 0.26 [–0.18, 0.70] .094 0.12 [-0.01, 0.25] No signal –
Consistent 
Formality → Appropriateness .059 0.31 [–⁠0.13, 0.75] .041 0.14 [0.01, 0.27] Signal
Inconsistent, smaller 
Smileys*Formality → Warmth .058 0.49 [0.04, 0.93] .108 0.11 [-0.02, 0.24] No signal –
Inconsistent, smaller 
Smileys*Formality → Competence .001 0.52 [0.07, 0.97] .090 0.12 [0.01, 0.25] Signal –
Inconsistent, smaller 
Smileys*Formality → Appropriateness .003 0.45 [0.00, 0.89] .044 0.14 [0.005, 0.27] Signal –
inconsistent, smaller 
Smileys*Formality → Warmth
via Appropriateness 
 Not reported  –⁠0.05 [-0.12, 0.005] No signal –
NA 
Smileys*Formality → Competence
via Appropriateness 
 Not reported  –0.12 [-0.25, 0.01] No signal –
NA 
Original articleReplication
Effect p Cohen’s d
[95% CI] 
p Cohen’s d
[95% CI] 
Summary
based on Lebel et al. (2019)  
Smileys → Warmth .11 0.34 [–0.10, 0.78] < .001 0.47 [0.33, 0.61] Signal –
Consistent 
Smileys → Competence .001 0.72 [0.25, 1.18] .002 0.22 [0.08, 0.36] Signal –
inconsistent, smaller 
Smileys → Appropriateness .01 0.49 [0.04, 0.93] < .001 0.88 [0.74, 1.02] Signal –
inconsistent, larger 
Formality → Warmth .32 0.06 [–0.38, 0.49] .132 0.11 [-0.02, 0.24] No signal –
Consistent 
Formality → Competence .06 0.26 [–0.18, 0.70] .094 0.12 [-0.01, 0.25] No signal –
Consistent 
Formality → Appropriateness .059 0.31 [–⁠0.13, 0.75] .041 0.14 [0.01, 0.27] Signal
Inconsistent, smaller 
Smileys*Formality → Warmth .058 0.49 [0.04, 0.93] .108 0.11 [-0.02, 0.24] No signal –
Inconsistent, smaller 
Smileys*Formality → Competence .001 0.52 [0.07, 0.97] .090 0.12 [0.01, 0.25] Signal –
Inconsistent, smaller 
Smileys*Formality → Appropriateness .003 0.45 [0.00, 0.89] .044 0.14 [0.005, 0.27] Signal –
inconsistent, smaller 
Smileys*Formality → Warmth
via Appropriateness 
 Not reported  –⁠0.05 [-0.12, 0.005] No signal –
NA 
Smileys*Formality → Competence
via Appropriateness 
 Not reported  –0.12 [-0.25, 0.01] No signal –
NA 

Note. Replication summary based on LeBel et al.’s (2019) taxonomy: (1) whether a signal was detected (i.e., whether the 95% confidence interval, or CI, excludes 0); (2) the consistency of the replication effect size estimate with that observed in the original study (i.e., whether the replication’s CI includes the original effect size point estimate); and (3) the magnitude of the replication’s effect size estimate in the same direction compared to original effect size. Cohen’s ds were converted from partial eta-squared values in ANOVA.

Our results detected a signal in three tests out of all the tests reported in the original. These involved the main effects of smiley usage on the three DVs. The effect of smiley on warmth was reported as nonsignificant in the original, and the effect of smiley on appropriateness was larger in the replication.

Finally, we assessed replication success using the small telescopes approach (Simonsohn, 2015) by calculating the power that the original study would have to detect the replication’s effect size for the key effects, namely, the interaction between smiley and formality in predicting the three DVs. This approach considers a result to be a failed replication if the original study lacked sufficient power—less than 33% power—to detect the replication result’s detected effect sizes. We used the pwr package in R (Champely et al., 2018).

The sample size of the original study was 85, which would give it 12% power to detect the replication’s effect size for the interaction effect predicting perceived warmth (upper bound = 45% power), 13% power to detect the interactive effect on perceived competence (upper bound = 48% power), and 17% power to detect the interactive effect on appropriateness (upper bound = 56%). These results suggest that the original study was severely underpowered to detect any of the three key effects reported in the current replication and that our study failed to replicate the original findings.

Exploratory Analyses

Perceived Gender

We performed a Chi-square test to examine differences in participants’ perceived gender of the email sender. Across conditions, 457 (54.0%) participants perceived the sender as male, 197 (23.3%) perceived the sender as female, whereas 193 (22.8%) reported being uncertain or that they did not think about a particular gender. In the control condition (no smiley), the sender was significantly more often identified as a male (82.3%) than a female (χ2 (1, 654) = 48.37, p < .001).

Next, we followed the original study and performed a series of ANCOVAs testing the effect of the perceived gender of the sender on perceptions of warmth and competence, controlling for English proficiency.

Consistent with the results from the original study, our results reveal no direct association between perceived gender and perceptions of competence (F(1, 651) = 0.57, p = .449, ηp2 = 0.001, 95% CI = 0.000, 0.011, d = 0.06, BF10 = 0.78). Perceived gender was, however, positively associated with perceptions of warmth (F(1, 651) = 4.33, p = .038, ηp2 = 0.007, 95% CI = 0.000, 0.024, d = 0.16, BF10 = 0.17) and negatively associated with perceptions of appropriateness (F(1, 651) = 21.75, p < .001, ηp2 = 0.032, 95% CI = 0.011, 0.063, d = 0.37, BF10 = 3454.90), indicating that ratings of warmth were higher whereas ratings of appropriateness were lower when the sender was perceived as female (as opposed to male).

Perceived gender did not moderate the effects of smiley condition on perceptions of warmth (F(1, 649) = 0.05 p = .828, ηp2 = 0.000, 95% CI = 0.000, 0.006, d = 0.02, BF10 = 435284.9) or competence (F(1, 649) = 2.64, p = .105, ηp2 = 0.004, 95% CI = 0.000, 0.019, d = 0.13, BF10 = 0.09), but moderated the effect of smileys on perceptions of appropriateness (F(1, 649) = 5.14, p = .024, ηp2 = 0.008, 95% CI = 0.000, 0.027, d = 0.18, BF10 = 1.16). Follow-up contrasts indicated that smileys decreased perceptions of appropriateness when the sender was perceived as being male (t(649) = 7.25, p < .001, d = 0.57, 95% CI = 0.45, 0.79) and female (t(649) = 7.05, p < .001, d = 0.55, 95% CI = 0.43, 0.77), but the effect was stronger when the sender was perceived as a female.

Controlling for Educational Level in ANOVA Predicting Perceived Competence

Due to the correlation between participants’ educational level and perceptions of competence, we performed exploratory analyses using educational level as a control variable (excluding gender as a control). The results remained similar. The main effect of smileys on perceived competence was significant and negative (F(1, 842) = 8.03, p = .005, ηp2 = 0.009, 95% CI = 0.001, 0.027, d = 0.20), the main effect of formality condition was not significant (F(1, 842) = 3.67, p = .056, ηp2 = 0.004, 95% CI = 0.000, 0.018, d = 0.13), and the interaction between smiley condition and formality was not significant (F(1, 842) = 1.70, p = .192, ηp2 = 0.002, 95% CI = 0.000, 0.013, d = 0.09).

We performed a high-powered replication of Experiment 3 in Glikson et al. (2018). The original results indicated that smileys have a positive effect on perceived warmth in an informal setting and a negative effect on perceived competence in a formal setting, and that these effects could partly be attributed to perceptions of appropriateness.

We found that smileys have a positive, medium effect on perceptions of warmth and a negative, small effect on perceptions of competence regardless of the level of formality, and no support for a partly mediating role of perceived appropriateness.

One potential explanation for the discrepancies in results between the two studies pertains to statistical power. Based on Simonsohn’s (2015) small-telescope approach, the original study seems to have been underpowered to detect the key effects reported in the original article and severely underpowered to detect the key effects observed in the current replication study.

Another potential explanation for the discrepancies in results is that, as proposed by Kaye et al. (2016), the increasing use of computer-mediated text-based communication has produced a shift in the perceived appropriateness of emoticons and emojis in work-related email, that is, an increased general acceptance of their usage. If so, one may expect that the direct effects of smileys as well as the (partly) mediating role of perceived appropriateness on perceived warmth and competence, as suggested by Glikson et al. (2018), will be deflated over time. The potentially deflating effect of using smileys may also be seen in the context of the general trend towards more informal writing in many domains, including business correspondence, administrative documents, journalism, and scientific writing (cf. Hyland & Jiang, 2017).

Although the time span from the collection of the original data (2018 or prior) to 2023 is relatively short, the Covid-19 pandemic accelerated digitally based communication and the volume of email, instant messaging, and social network use, due to the increase in remote working and other measures to reduce physical interaction. Hence, the rub-off effect of emoji use in instant messaging and social media may have accelerated during the pandemic. Accordingly, although smileys were found to decrease the perceived appropriateness of the message, the possibility that the use of emoticons or emojis in work-related emails has become more generally acceptable and hence of less significance when forming first impressions of the sender cannot be dismissed.

Yet, it seems reasonable to assume that acceptance and hence perceived appropriateness of smileys and other emojis will vary dependent on the level of formality in the organization as well as the degree of formality of the communication, which in turn may produce different and potentially enhanced effects on perceptions of the sender’s warmth and competence. The experimental materials used in the original study and the replication reported here did not include experimental conditions that provide the opportunity to compare organizations characterized by a high versus low level of formality in communication.

Hence, it is possible that the formality manipulations were too weak with respect to activating a realistic sense of email communication in a formal versus informal work-related setting. The formality of the setting was manipulated by only a subtle change in wording, that is, “staff meeting” versus “social gathering”. Although the manipulation check in the replication study revealed that the majority of participants (93.7%) recalled the correct formality treatment, it remains unclear whether participants experienced a sufficient sense of realism and immersion based on the respective treatments (cf. Aguinis & Bradley, 2014; Alekseev et al., 2017).

In the vignettes, the first name of the recipient only (“Dear Sarah”) was used as the initial greeting in both settings. While the latter may be customary in many organizations, an even more formal tone tends to be used in some organizations, for example, by including the last name of the recipient and potentially the job title or educational degree, particularly when making initial contact via professional email. Moreover, the experimental materials did not reveal the style and level of formality of the initial email to which the sender replies. Future research is therefore needed to investigate the degree to which the level of formality in the organization and of the specific setting may have bearing on the effects of using smileys and other emojis on first impressions of the sender’s warmth and competence.

Future research could also further investigate the potential influence of the gender of the sender, the receiver, and the observer, respectively. The experimental materials used in the original study and the present replication depict an email that is sent by a person with a gender-neutral name (“Alex”) to a female receiver (“Sarah”). Although, consistent with the original study, our exploratory analyses did not reveal an interaction between the perceived gender of the sender and smileys on participants perceptions of the sender’s warmth or competence, our findings do indicate that smileys are perceived as less appropriate when used by a perceived female sender. Yet, several studies indicate that women use emojis such as smileys more frequently than men, and that emoji preferences vary across genders (e.g., Koch et al., 2022; Oleszkiewicz et al., 2017). The pattern of findings could therefore be retested by explicitly specifying the gender of the sender through the use of two distinct gender-specific names, each representing either a female or a male sender. Similarly, since the experimental materials depict a female receiver only, future research is needed to assess the generalizability of the pattern of findings to a male recipient. Future research could also explore the potential role of the observer’s gender when forming first impressions based on emails addressed to unknown individuals, whether female or male, in comparison to emails where the observer is presented as the recipient.

Last, but not least, future research should investigate the generalizability of the findings derived from the U.S. to other countries. The U.S. is generally considered as a society characterized by high levels of masculinity and individualism, as well as a relatively low level of power distance (cf. Hofstede, 1983). Countries with other cultural profiles may place different cultural emphasis on informal communication and leveling with others, which in turn may have bearing on the effects of smiley usage on perceptions of the sender’s warmth and competence, and the appropriateness of the message in work-related email correspondence.

The results from the high-powered replication of Experiment 3 in Glikson et al. (2018) reported here indicate that smileys not only have a dark side by deflating first impressions of competence, as reported by Glikson et al. (2018), but also a bright side by enhancing first impressions of warmth. In contrast to Glikson et al. (2018), however, our results indicate that these effects are not sensitive to the degree of formality of the setting and are not mediated by the perceived appropriateness of the message. Future research is needed to better investigate the potential moderating role of formality as well as the gender of the sender and recipient, respectively. Hopefully, the present replication inspires further inquiries into the use of emojis and emoticons and their potential effects in work settings.

Contributed to conception and design: LL

Contributed to acquisition of data: LL, LM

Contributed to analysis and interpretation of data: LL, LM

Drafted and/or revised the article: LL, LM

Approved the submitted version for publication: LL, LM

The authors would like to thank Senior Editor Elizabeth R. Tenney and the two anonymous reviewers for their valuable comments and suggestions for improvements in a previous version of this manuscript.

The authors report no conflict of interest.

Data, code, and materials associated with this paper are available at https://osf.io/n7yc4/.

Experimental materials: The email presented to participants in each of the four experimental conditions. Control condition / Formal condition

Dear Sarah,

My name is Alex and I’ve started working here this week.

Thank you for sending me the invitation to join the staff meeting on Friday.

Could you please let me know where it will be taking place?

Thank you very much, Alex Bledow

Control condition / Informal condition

Dear Sarah,

My name is Alex and I’ve started working here this week.

Thank you for sending me the invitation to join the social gathering on Friday.

Could you please let me know where it will be taking place?

Thank you very much, Alex Bledow

Smiley condition / Formal condition

Dear Sarah,

My name is Alex and I’ve started working here this week. 😊

Thank you for sending me the invitation to join the staff meeting on Friday.

Could you please let me know where it will be taking place? 😊

Thank you very much, Alex Bledow

Smiley condition / Informal condition

Dear Sarah,

My name is Alex and I’ve started working here this week. 😊

Thank you for sending me the invitation to join the social gathering on Friday.

Could you please let me know where it will be taking place? 😊

Thank you very much, Alex Bledow

____________________________

Note: Drawn from Table 6 in Glikson et al. (2018).

1.

Statcheck is a free and open-source tool that extracts reported statistical results from academic papers and recalculates the p-values. Inconsistency of p-values denotes that recalculated p-values, based on the reported statistical test and degrees of freedom, do not match the p-values reported by the authors (Nuijten & Polanin, 2020).

2.

63 participants failed the manipulation check yet did not meet any of the other exclusion criteria. The proportion of male participants was higher (61.9% versus 46.3%) among participants who failed the manipulation check, and the mean age was lower (32.22 versus 38.72) compared to participants who passed the manipulation check. Discrepancies in results between analyses without and with exclusions based on the manipulation check are reported in footnotes.

3.

When excluding participants based on the manipulation check and controlling for gender, the main effect of formality condition became statistically significant (F(1, 757) = 3.94, p = .047, ηp2 = 0.005, 95% CI = 0.000, 0.020, d = 0.14), indicating that the sender was perceived as more competent in the formal as opposed to informal condition. Our analyses produced the same pattern of results when not controlling for gender.

4.

Analyses with exclusions based on the manipulation check, and after controlling for gender, indicated that the main effect of formality (F(1, 757) = 3.79, p = .052, ηp2 = 0.005, 95% CI = 0.000, 0.020, d = 0.14) and the interaction between smiley condition and formality condition (F(1, 757) = 3.02, p = .083, ηp2 = 0.004, 95% CI = 0.000, 0.018, d = 0.13) were not significant. The same pattern of results held when not controlling for gender.

5.

The moderated mediation indices remained insignificant when excluding participants based on the manipulation check.

Abbey, J. D., & Meloy, M. G. (2017). Attention by design: Using attention checks to detect inattentive respondents and improve data quality. Journal of Operations Management, 53–56(1), 63–70. https://doi.org/10.1016/j.jom.2017.06.001
Aguinis, H., & Bradley, K. J. (2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods, 17(4), 351–371. https://doi.org/10.1177/1094428114547952
Alekseev, A., Charness, G., Gneezy, U. (2017). Experimental methods: When and why contextual instructions are important. Journal of Economic Behavior Organization, 134, 48–59. https://doi.org/10.1016/j.jebo.2016.12.005
Aronow, P. M., Baron, J., Pinson, L. (2019). A Note on Dropping Experimental Subjects Who Fail a Manipulation Check. Political Analysis, 27(4), 572–589. https://doi.org/10.1017/pan.2019.5
Ben-Shachar, M. S., Lüdecke, D., Makowski, D. (2020). Effectsize: Estimation of effect size indices and standardized parameters. Journal of Open Source Software, 5(56), 2815. https://doi.org/10.21105/joss.02815
Brysbaert, M. (2019). How many participants do we have to include in properly powered experiments? A tutorial of power analysis with reference tables. Journal of Cognition, 2(1), 16. https://doi.org/10.5334/joc.72
Champely, S., Ekstrom, C., Dalgaard, P., Gill, J., Weibelzahl, S., Anandkumar, A., .., De Rosario, M. H. (2018). Package ‘pwr’: basic functions for power analysis. https://cran.r-project.org/web/packages/pwr/pwr.pdf
Faul, F., Erdfelder, E., Lang, A.-G., Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/bf03193146
Fiske, S. T., Cuddy, A. J., Glick, P. (2007). Universal dimensions of social cognition: Warmth and competence. Trends in Cognitive Sciences, 11(2), 77–83. https://doi.org/10.1016/j.tics.2006.11.005
Fiske, S. T., Cuddy, A. J., Glick, P., Xu, J. (2002). A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition. Journal of Personality and Social Psychology, 82(6), 878–902. https://doi.org/10.1037/0022-3514.82.6.878
Glikson, E., Cheshin, A., van Kleef, G. A. (2018). The dark side of a smiley: effects of smiling emoticons on virtual first impressions. Social Psychological and Personality Science, 9(5), 614–625. https://doi.org/10.1177/1948550617720269
Hofstede, G. (1983). National cultures in four dimensions: A research-based theory of cultural differences among nations. International Studies of Management Organization, 13(1–2), 46–74. https://doi.org/10.1080/00208825.1983.11656358
Hyland, K., Jiang, F. (2017). Is academic writing becoming more informal? English for Specific Purposes, 45, 40–51. https://doi.org/10.1016/j.esp.2016.09.001
Jaccard, J., Becker, M. A. (2002). Statistics for the Behavioral Sciences (4th ed.). Wadsworth/Thomson Learning.
Judd, C. M., James-Hawkins, L., Yzerbyt, V., Kashima, Y. (2005). Fundamental dimensions of social judgment: Understanding the relations between judgments of competence and warmth. Journal of Personality and Social Psychology, 89(6), 899–913. https://doi.org/10.1037/0022-3514.89.6.899
Kaye, L. K., Wall, H. J., Malone, S. A. (2016). “Turn that frown upside-down”: A contextual account of emoticon usage on different virtual platforms. Computers in Human Behavior, 60, 463–467. https://doi.org/10.1016/j.chb.2016.02.088
Koch, T. K., Romero, P., Stachl, C. (2022). Age and gender in language, emoji, and emoticon usage in instant messages. Computers in Human Behavior, 126(Jan 2022), 106990. https://doi.org/10.1016/j.chb.2021.106990
Lea, M., Spears, R. (1992). Paralanguage and social perception in computer-mediated communication. Journal of Organizational Computing and Electronic Commerce, 2(3–4), 321–341. https://doi.org/10.1080/10919399209540190
LeBel, E. P., Berger, D., Campbell, L., Loving, T. J. (2017). Falsifiability is not optional. Journal of Personality and Social Psychology, 113(2), 254–261. https://doi.org/10.1037/pspi0000106
LeBel, E. P., Vanpaemel, W., Cheung, I., Campbell, L. (2019). A brief guide to evaluate replications. Meta-Psychology, 3. https://doi.org/10.15626/mp.2018.843
Lenhard, W., Lenhard, A. (2016). Computation of effect sizes. Psychometrica. https://www.psychometrica.de/effect_size.html
Munter, M., Rogers, P. S., Rymer, J. (2003). Business e-mail: Guidelines for users. Business Communication Quarterly, 66(1), 26–40. https://doi.org/10.1177/108056990306600104
Nuijten, M. B., Polanin, J. R. (2020). “statcheck”: Automatically detect statistical reporting inconsistencies to increase reproducibility of meta‐analyses. Research Synthesis Methods, 11(5), 574–579. https://doi.org/10.1002/jrsm.1408
Oleszkiewicz, A., Karwowski, M., Pisanski, K., Sorokowski, P., Sobrado, B., Sorokowska, A. (2017). Who uses emoticons? Data from 86 702 Facebook users. Personality and Individual Differences, 119, 289–295. https://doi.org/10.1016/j.paid.2017.07.034
Olsson-Collentine, A., van Assen, M. A., Hartgerink, C. H. (2019). The prevalence of marginally significant results in psychology over time. Psychological Science, 30(4), 576–586. https://doi.org/10.1177/0956797619830326
Rife, S. C., Nuijten, M. B., Epskamp, S. (2016). statcheck: Extract statistics from articles and recompute p-values [Web application]. http://statcheck.io
Rodrigues, D., Prada, M., Gaspar, R., Garrido, M. V., Lopes, D. (2018). Lisbon Emoji and Emoticon Database (LEED): Norms for emoji and emoticons in seven evaluative dimensions. Behavior Research Methods, 50(1), 392–405. https://doi.org/10.3758/s13428-017-0878-6
Schäfer, T., Schwarz, M. A. (2019). The meaningfulness of effect sizes in psychological research: Differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology, 10, 813. https://doi.org/10.3389/fpsyg.2019.00813
Simmons, J. P., Nelson, L. D., Simonsohn, U. (2012). A 21 Word Solution. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2160588
Simonsohn, U. (2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26(5), 559–569. https://doi.org/10.1177/0956797614567341
Skovholt, K., Grønning, A., Kankaanranta, A. (2014). The Communicative Functions of Emoticons in Workplace E-Mails. Journal of Computer-Mediated Communication, 19(4), 780–797. https://doi.org/10.1111/jcc4.12063
Tenney, E. R., Costa, E., Allard, A., Vazire, S. (2021). Open science and reform practices in organizational behavior research over time (2011 to 2019). Organizational Behavior and Human Decision Processes, 162, 218–223. https://doi.org/10.1016/j.obhdp.2020.10.015
Varaine, S. (2023). How Dropping Subjects Who Failed Manipulation Checks Can Bias Your Results: An Illustrative Case. Journal of Experimental Political Science, 10(2), 299–305. https://doi.org/10.1017/xps.2022.28
This is an open access article distributed under the terms of the Creative Commons Attribution License (4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.