Beyond the Lab: Validating Personality Impressions and Behavioral Observations in Online Video Calls

In everyday life, people routinely try to infer people’s personality from their behavior. For example, we can’t help, but spontaneously judge whether someone is nice, trustworthy, ruthless, or malicious. Such personality impressions form very quickly and are highly consequential for decisions in the social world, such as whom to cooperate or befriend with, whom to fear, or whom to avoid altogether (Todorov, 2017). A large body of evidence indicates that even at minimal acquaintance, people are surprisingly good at estimating others’ personality (e.g., Back & Nestler, 2016; Borkenau et al., 2004; Connelly & Ones, 2010; Connolly et al., 2007; Letzring & Spain, 2021; Vazire & Carlson, 2010; Wu et al., 2023).

In accuracy research, observer-ratings are compared to some kind of accuracy criterion. When the accuracy of personality impressions is studied, the most often used criterion with which observer-ratings are compared to are target persons’ personality self-reports (“self-other agreement”; Funder, 2012) and this was also the case in the current research. On the side of the observers, two approaches can be distinguished: aggregate-observer accuracy (also called “group” or “pooled” accuracy) and single-observer accuracy (e.g., Back et al., 2010; Bernieri, 2001; Carney et al., 2007). Aggregate-observer accuracy represents the level of accuracy gained by a group of raters (in which the idiosyncrasies of single raters have been averaged out; Block, 1961), whereas single-observer accuracy reflects how accurate a single, “typical” observer would be (Naumann et al., 2009). An advantage of the aggregate-observer approach is that due to the aggregation principle (Epstein, 1983) the reliability of the ratings is higher, which is why we used this approach in the current research.

How can personality be accurately inferred even by strangers? This process is described by the Realistic Accuracy Model (RAM; Funder, 1995), which builds upon Brunswik’s (1956) lens model. Both models suggest that personality aspects have to manifest in observable behaviors (i.e., cues), which must then be detected and utilized by observers to derive personality judgments. The RAM describes further conditions that must be met in or order for accurate personality judgements to occur. That is, the situational context must have some relevance with regard to the trait that is being judged, the trait-related behavioral cues must be available to the observers and the observers must detect and correctly interpret the behavioral cues. If all conditions are met, the total degree of accuracy is thought to depend on how easy to judge a target person is (“good target”), how visible an attribute is (“good trait”), how much and what kind of information is provided (“good information”) and how good observers are at rating others (“good judge”; Funder, 2012).

Laboratory studies have identified several traits that can be accurately detected by strangers (e.g., Carney et al., 2007; Letzring et al., 2021; Letzring & Spain, 2021; Vazire & Carlson, 2010), including the Big Five (i.e., extraversion, conscientiousness, openness, emotional stability and agreeableness; Connelly & Ones, 2010; Connolly et al., 2007), self-esteem (Hirschmüller et al., 2018; Naumann et al., 2009), and motive dispositions (i.e., the achievement, affiliation and power motives; Bassler et al., 2023; Dufner & Krause, 2023). However, researchers suggest that some traits are easier to detect than others (for an overview, see Connelly & Ones, 2010). Extraversion is considered a “good trait” due to its visibility. Highly visible traits involve tendencies that are externally expressed (mainly social behavior), making them easier to perceive (Zillig et al., 2002). On the contrary, traits low in visibility, such as emotional stability or openness to experience, involve tendencies that are more internal (e.g., thoughts or affective states), requiring additional information from the target to be detectable by others. Therefore, emotional stability is often challenging to accurately judge by unacquainted observers (Hirschmüller et al., 2015). Furthermore, highly evaluative traits, such as agreeableness (John & Robins, 1993), are often hard to detect, because targets may employ various self-presentation strategies to create a positive impression of themselves.

Other research has exclusively focused on the association between personality and social behavior. Two traits whose behavioral correlates have been well-studied are intelligence and narcissism. Studies have linked intelligence to paralinguistic cues such as fluent speaking, a pleasant speech style, and a faster speech rate (Borkenau & Liebler, 1993, 1995; Breil et al., 2021; Murphy et al., 2003; Reynolds & Gifford, 2001). Results on the behavioral correlates of grandiose narcissism indicate that admiration, which represents the assertive self-enhancement aspect of narcissism, goes along with agentic behavior (e.g., self-assuredness, activity level), whereas rivalry, which represents the antagonistic self-protection aspects of grandiose narcissism, goes along with attenuated communal behavior (e.g., less warmth and friendliness; Back et al., 2013).

In order for personality impressions to be gathered or for behavior to be coded, participants typically have to attend laboratory sessions, where they perform tasks or interact socially (Borkenau & Liebler, 1995). Typically, participants’ behavior is videotaped, and the recordings are then judged by unacquainted observers (Bakeman & Quera, 2012; Borkenau & Liebler, 1995).

A major disadvantage of this approach, however, is that it is rather inefficient, time-consuming, and costly (Furr & Funder, 2007). Whereas in many research domains, online studies are widespread (Gosling & Mason, 2015), studies on personality impressions and on the relations between personality and behavior still require a physical laboratory that can be used for at least several weeks and that is equipped with video cameras. Furthermore, visiting a laboratory is burdensome for the research participants, which makes it hard to gather sufficiently large samples and potentially leads to biased samples. After all, laboratory assessments are limited in reach and inclusion (Perry et al., 2021), as visiting a laboratory can, for example, be troublesome for persons with impaired mobility.

Over the last few years, digitalization − the use of digital technology for data collection, generation, and analysis to create value and enable innovation (Cappa et al., 2021; Goduscheit & Faullant, 2018) − has progressed rapidly. At least since the COVID-19 pandemic, online formats for chatting via text messages, talking, or meeting up with other people are finding their way into private life as well as the professional world (Nguyen et al., 2020). Digitalization also affects almost every scientific discipline including psychological research. Digital solutions can be found for data collection (e.g., online surveys, real-time experience sampling, assessment of behavioral parameters in virtual interaction games), algorithms for data analysis, and more interactive applications with human-machine communication (Ostermann et al., 2021). Digitalization adds to the toolbox for psychological research and opens up novel avenues for testing theories and concepts (Kende, 2014), including those referring to personality judgments and behavior observations (Letzring et al., 2021). Moreover, it offers great potential for interdisciplinary collaboration and points the way for a more inclusive, reliable and generalizable psychological science (Moshontz et al., 2018). Given the greater acceptance and easy use of online formats, a potential solution for the problems of conventional laboratory studies could be to study personality impressions and behavior based on online interactions.

Some initial research has already started to explore the chances of online behavioral observations. Wang and Chanel (2021) provided first evidence indicating that it might be possible to infer the target persons’ warmth and competence with some degree of accuracy after having seen them interact in a video call. In a relationship study, Perry et al. (2021) observed couples’ behavior during a video call and found that the online observed behavioral reactions were very similar to those observed in prior studies conducted in the laboratory. Diana et al. (2023) found that mimicry of multiple behaviors and perceptions of trustworthiness did not differ systematically when they were assessed in face-to-face, video call, and pre-recorded video interactions. Finally, in a study by Tissera et al. (2023) participants interacted via video calls versus face-to-face and then, meta-perceptions (i.e., estimates about how they were seen by the interaction partners) were gathered. The results indicated that the accuracy of these meta-perceptions was similar across the two formats.

Taken together, these studies indicate that when behavior is observed during video call interactions, results might be similar to cases when behavior is observed during face-to-face interactions with regard to personality impressions and to links between personality and behavior. However, no research has thus far tested systematically and comprehensively whether classic results from the literature on the accuracy of personality impressions and the links between personality and behavior would also emerge when assessments are made based on online interactions. The current research was aimed at filling this gap. If the pattern of results should be comparable to offline investigations, the use of online interactions might become standard practice in the field of personality impression research.

Personality judgments and behavior observations are important to study for both theoretical and practical reasons (Funder & West, 1993). With regard to theory, none of the discussed hypotheses concerning the processes linking personality and social evaluations and the role of moderating factors can be tested without assessments of behavior and observer evaluations. This also includes theoretical claims concerning the interplay between the person and the situation in generation of social behavior (Furr & Funder, 2018). With regard to practice, personality judgements that are based on behavior observations play an important role in settings such as personnel selection or relationship initiation − which also take place increasingly online.

Research involving personality assessments on the one side and personality impressions or behavior observations on the other is flourishing. Yet, a major impairment is the necessity for laboratory designs. An online approach that allows for an economic and efficient assessment of personality impressions and behavior observations would be highly beneficial. The aim of the current research was to test whether such an approach is viable. To be clear, the aim was neither to shed novel light on theoretical questions dealing with accuracy or the link between personality and behavior nor to introduce a novel method for the assessment of personality. Instead, we aimed to examine whether valid behavior observations can be made in the context of online video calls.

Precisely, the study tested whether results from prominent offline studies on the accuracy of personality impressions and on the link between personality and behavior can be replicated in an online video call format. Video recordings of online calls are highly similar to the ones that stem from typical interactions in the laboratory. In laboratory research, participants are often filmed from the front and at least the face and upper body are visible, so that facial expressions and gestures can be coded well. Often, the experimenter is seen from the back or not at all. The angle is therefore very similar to the view on online video calls. Furthermore, both laboratory and online interactions typically occur in real-time and synchronously, enabling the largely automatic, moment-to-moment tuning of emotional signals (Diana et al., 2023). It therefore seemed likely that results based on observations during online calls would be similar to the ones reported for laboratory studies.

As to-be-replicated findings, we selected relatively robust results on either the accuracy of personality impressions or the relation of a given trait with a specific behavioral outcome. With regard to the accuracy of personality judgments, we focused on the following traits, which can all be inferred with above chance-level accuracy in newly met others in laboratory settings: the Big Five (extraversion, neuroticism, conscientiousness, openness, agreeableness; Connelly & Ones, 2010; Connelly et al., 2007), self-esteem (Hirschmüller et al., 2018; Naumann et al., 2009), motive dispositions (Bassler et al., 2023), intelligence (Murphy et al., 2003; Reynolds & Gifford, 2001) and narcissism (Giacomin & Jordan, 2019; Lukowitsky & Pincus, 2013). With regard to the links between personality and observed behavior, we examined cues that have been linked to the respective trait in previous studies that had a very similar design like ours, but were conducted offline. We hypothesized that intelligence (general and verbal) would be related to acoustic cues such as way of speaking (i.e., clearer, more fluent, less hectic; Borkenau & Liebler, 1995), that narcissistic admiration would be positively related to agentic behavior (Back et al., 2013), that narcissistic rivalry would be negatively related to communal behavior (Back et al., 2013) and that vulnerable narcissism, which is characterized by feeling of self-importance and entitlement paired with feelings of insecurity and inferiority (Miller et al., 2017), would be positively linked with nervous behavior.

A second aim of the current study was to develop a paradigm that could be used in future research on personality perception. The consistent usage of a specific interactive paradigm is a way to ensure greater comparability of research findings, which is crucial for the cumulative sciences (Allen et al., 2017). In some research fields, such standard paradigms exist, such as the Trier Social Stress Test (TSST; Kirschbaum et al., 1993) in the field of stress research, the Cyberball paradigm (Williams & Jarvis, 2006) in research on social exclusion or the noise blaster paradigm (McCullagh et al., 1994) in aggression research. In order for a trait to manifest itself in a specific behavior, the situation needs to be trait-relevant (Funder, 1995; Hirschmüller et al., 2015; Tett & Guterman, 2000). For example, intelligence will only be linked to specific cues if the context is somewhat cognitively demanding. Accordingly, we implemented an interactive paradigm that included different questions and tasks that we considered relevant for intelligence and different aspects of personality. The aim was to add so much variety to the situation that as many traits as possible (extraversion, self-esteem, motive dispositions, etc.) would be made visible in social behavior. We reasoned that if the results should turn out as expected, our procedure could serve as a standard paradigm in future studies on personality impressions and on the relation between personality and behavior.

This study was not preregistered. Materials, participant data, and analysis scripts are available at the Open Science Framework: https://osf.io/8utzk/

The study took place between December 2022 and January 2023. The ethics commission of Witten/Herdecke University approved the study. For recruitment, an online platform hosted by the university and online social networks as well as an external panel provider were employed. Participants were required to be at least 18 years old. Participants from the panel provider received a monetary compensation (17,70€). Among the remaining participants, five 10€ Amazon vouchers were raffled off, and psychology students had the opportunity to receive course credit.

A total of 189 persons participated (99 female, 90 male) with a mean age¹ of 38.34 years (SD = 15.08, Range: 18-74 years). 24.3% of the participants had a master’s degree, 18% a bachelor’s degree, 42.3% a high school degree, and 15.3% had a lower school diploma. One hundred and eighteen participants had been recruited via the panel provider. One hundred and twenty-eight participants used a laptop, 32 used a PC, 15 used a tablet and 12 participants used a smartphone to take part in the study.

The study included an online survey in which participants completed intelligence tests and personality self-report questionnaires. (The exact order of the online survey can be found in Table S1 of the Supplemental Online Material). The survey was programmed using the software formr (Arslan et al., 2020) and took approximately 30 minutes. After participants had completed the online survey, they booked an appointment for an online interview. (If people did not book an appointment or failed to attend the scheduled appointment, their online survey data was not included. Apart from this criterion, no data have been excluded.) Five undergraduate students who were not acquainted with the participants conducted these online interviews using the online platform Zoom. The interviews lasted 5-6 minutes on average (M = 338 seconds, SD = 123 seconds) and each undergraduate student interviewed 38 participants on average.

General intelligence was assessed with the 16-item short form of the International Cognitive Ability Resource (ICAR16; Condon & Revelle, 2014), a brief measure of cognitive abilities comprising letter and number series, matrix reasoning, verbal reasoning, and three-dimensional rotation items. Each item offers seven answer options, including the possibility that none of the displayed options is correct. For each item, we coded whether it was correctly solved.

As an additional measure that taps specifically into verbal aspects of intelligence, we administered the German version of the Multiple-Choice Vocabulary Intelligence Test (MWT-B; Lehrl et al., 1995). In this test, participants are presented five letter strings, one of which represents a real word. Their task is to find out, which one the word is. A total of 37 items were presented in ascending difficulty. We again coded for each item whether it was correctly solved.

Narcissism. The German version of the Narcissistic Admiration and Rivalry Questionnaire (NARQ; Back et al., 2013) was employed to measure the two narcissism sub dimensions of grandiose narcissism, admiration and rivalry. The questionnaire consists of 18 items (1 = not agree at all to 6 = agree completely) of which 9 items refer to each dimension. Vulnerable narcissism was assessed by aggregating across the three dimensions Contingent Self-esteem, Hiding the Self and Devaluing of the German version (Morf et al., 2017) of the Pathological Narcissism Inventory (PNI; Pincus et al., 2009). The 26 items were rated from 1 = not at all like me to 5 = very much like me.

Big Five. To measure the Big Five personality traits, the German version (Rammstedt et al., 2018) of the short version of the Big Five Inventory-2 (BFI-2-S; Soto & John, 2017) was used. Thirty items were rated with a 5-point scale (1 = disagree strongly to 5 = agree strongly) with six items for each trait.

Self-Esteem. Self-esteem was assessed with the German version (von Collani & Herzberg, 2003) of the Rosenberg Self-Esteem Scale (RSES; Rosenberg, 1965), which is a well-established measure of global self-worth, and additionally with the multidimensional self-esteem scale (MSES; Schütz & Sellin, 2006), which measures different dimensions regarding self-worth. The MSES is an adapted German version of Fleming and Courtney’s (1984) multidimensional scale. For the current purposes, the general self-esteem score that is based on all items of the MSES was used. The RSES consists of 10 items (1 = strongly disagree to 4 = strongly agree), and the MSES consists of 32 items (1 = never to 7 = always).

Motive Dispositions. The short version of the Unified Motive Scale (USM-6; Schönbrodt & Gerstenberg, 2012) was employed to measure the affiliation, achievement and power motives. The scale consists of 30 items (0 = strongly disagree/not important to me to 5 = strongly agree/extremely important to me).

The online interviews were conducted via the video communication Zoom (Version 5.13.3 (11494); Zoom Video Communications, Inc.). The gallery view was used so that both the participant and interviewer appeared in equal size on the screen, had the same perspective and remained in the same position on the screen for the entire interview. Participants were instructed to position themselves directly in front of their camera, ensuring that only their face and shoulders were visible. Prior to the start of the interview, the interviewer made sure that all participants were seated in identical positions and that both the camera and microphone were functioning correctly.

The online interview consisted of nine questions and two short tasks. The online interview was standardized so that each interviewer asked the questions in a fixed order. At first, the interviewer gave a short introduction with some details about the interview:

“Now here is a short interview. I will ask you some questions. Please answer honestly, there are no right or wrong answers. During the interview, I will record the picture and sound. You don’t need to pay attention to this. When you are ready, we will start.”

Then, the interviewer asked nine personal questions (details see below). We adopted questions 1 to 8 from a study on self-esteem and self-confident behavior by Krause et al. (2016), because the set of questions performed very well in offering a trait-relevant situation for detecting self-esteem and they also seemed trait-relevant for other personality aspects we focused on, such as the Big Five, motive dispositions, or narcissism. The questions referred to different aspects of participants’ lives and were aimed at eliciting a wide range of different emotions and behaviors that might therefore be informative for a multitude of personality aspects. We added the question “How would you describe your hobbies?”, which might be relevant for traits such as openness to experience (Borkenau et al., 2004).

After the personal questions, participants engaged in the two tasks, which were primarily aimed at triggering intelligence-related behavior. The two tasks were (a) to explain the word “symmetry” and (b) to read out aloud a short weather forecast. These latter two tasks aimed at making behavioral cues visible that are linked to individual differences in intelligence. In a study by Borkenau et al. (2004), both tasks were diagnostic for measured intelligence rated by unacquainted observers. In order to explore participants affective experience, the experimenter asked each participant before and after the experiment to indicate their current mood on a scale from 1 = very bad to 5 = very good. Due to time constraints, the fact that mood effects were not central to our research question and previous findings indicating that the valence dimension of mood can be validly assessed with a single item (Weidman et al., 2017), we considered the single item measurement appropriate in our case. Data for one person was missing because the question about mood was forgotten to be asked.

Questions:

1. “Please consider what was a major success in your life?”

2. “When you look back on your life, do successes or failures predominate?”

3. “Where do your talents lie?”

4. “What would you like to change about yourself?”

5. “What do you believe makes you unique?”

6. “How would you rank yourself in comparison to your colleagues/fellow students/friends regarding your achievements and competencies?”

7. “How do you deal with negative emotions?”

8. “What makes a person particularly likable?” (pause) “Would you describe yourself as a likable person?”

9. “How would you describe your hobbies?”

10. “Explain the term ‘symmetry’.”

11. “Please read this short text aloud”: On Friday, in the eastern half, friendly periods will develop after the dissipation of fog fields. Otherwise, it is often cloudy, but initially, it remains mostly dry. Rain moves in from the west from noon. 13 to 19 degrees. Overnight into Saturday, temperatures drop to 15 degrees in the Upper Rhine region and down to 9 degrees in the Sauerland and the Erzgebirge. On Saturday, temperatures rise to 14 degrees on the North Frisian coast and up to 20 degrees in the Upper Rhine region. (For the original German version, see Supplemental Online Material)

To obtain ratings of participants’ behavior and of personality impressions, nine observers (four male and five female graduate students who were not acquainted with the target persons) watched the video recordings of the online interviews. The observers were instructed to watch each online interview while focusing solely on the participant (and ignoring the experimenter). Due to the relatively large number of ratings to be made, two independent groups of observers were formed. Group 1 (two male and two female observers) rated personality impressions (impressions of the Big Five, self-esteem, and motives), whereas Group 2 (two male and three female observers) rated the presumed behavioral correlates of intelligence and narcissism. To ensure that the behavioral assessments were reliable, each observer underwent a two-hour online observer training session along with the other observers of the specific group. At the beginning of the training session, all observers received general information about the rating procedure. Then, the two different groups were formed and each group received specific information about the to-be-rated constructs. The aim of these sessions was that the observers should develop a common understanding of their constructs. Subsequently, they rated two target persons and discussed their ratings afterwards (yet, no ratings were changed post-hoc). After the training, observers had one month to code all online interviews and were required to make their ratings independently of the other observers.

Personality impressions were assessed by using the same or slightly adapted versions of items that have been employed in offline zero-acquaintance studies on the accuracy of personality impressions. A complete list of all items can be found in Table 1. That is, impressions of the Big Five were assessed with the same items as in Borkenau et al. (2004), and impressions of motive dispositions were assessed with the items by Bassler et al., 2023 (also used by Dufner & Krause, 2023). Impressions of self-esteem were assessed with one item used by Krause et al. (2016), as molar-level assessments are suitable to capture behavioral self-confidence on a psychologically meaningful level (see Funder & Colvin, 1991). As potential behavioral correlates of intelligence, we picked paralinguistic cues such as easiness of understanding or fluent speaking that have been linked to intelligence in a laboratory study with a very similar design like ours (i.e., interview situation; Borkenau & Liebler, 1995). As indicators of agentic and communal behavior, which should be linked to narcissism, we used items from the Interpersonal Adjectives List (IAL; Jacobs & Scholl, 2005), as pilot laboratory research from our own lab indicated that behavior codings using the IAL are reliable and indeed show the expected links to the two narcissism dimensions (Schmaliy, 2020). All ratings were made on a 6-point Likert scale ranging from 1 = not at all to 6 = very much.

Using the software G*Power (version 3.1.9.4; Faul et al., 2007), we determined the required sample size to attain sufficient statistical power for our main statistical tests (i.e., Pearson correlations between the self-report measures and the observer-ratings based on the recordings of the online interviews). We looked up correlation coefficients that have been reported in offline laboratory studies with a very similar design like ours on the accuracy of personality impressions as well as on the relation between personality and behavior (the single coefficients can be found in Table S2 and Table S3). We averaged the coefficients across samples using Fisher’s r-to-z transformation and then transformed the average back into a Pearson correlation. The average correlation across all relevant effect sizes was r = .28 (please note that the identical value would have been indicated by Vazire & Carlson’s 2010, meta-analytic effect size for associations between personality and behavior more generally). Based on this value, 95 participants would have been necessary to detect the effect with a likelihood of 80 percent (alpha = .05, two-tailed), which has been proposed as a sufficient benchmark for behavioral science (Cohen, 1992). However, we also considered the possibility that this effect size estimate might have been too optimistic, as some previous findings might have been inflated due to publication bias. We therefore aimed for a larger sample sizes. Our resources allowed us to test 189 participants, and with this sample size, the power to detect an effect of r = .28 was .98 and the power to detect an effect of r = .20 (which is considered “medium” in social-personality psychology; Funder & Ozer, 2019) was .80. We will adhere to the recommendations by Funder and Ozer (2019) and interpret effect sizes of r = .10 as small, of r = .20 as medium, of r = .30 as large and of r = .40 as very large.

Concerning the number of observers, an analysis of the interrater agreement in the past relevant studies shows an average ICC value of .36 (for details, see Tables S2 and S3). Given the available resources, we had the opportunity to employ groups of four and five observers, respectively (see above). Using the Spearman-Brown formula (de Vet et al., 2017), we estimated that four observers would lead to a mean ICC of .69 and that five observers would lead to a mean ICC of .74, which can be considered moderate to good (Koo & Li, 2016).

We only present the main results here, more detailed results (on observer-agreement, accuracy, and experimenter effects and the effects of the device used during the online conversations) can be found in the Supplemental Online Material (SOM).

Descriptive statistics and reliabilities for the intelligence tasks and the self-report measures as well as their intercorrelations can be found in Table 2. As can be seen, the reliabilities of all measures were higher than .70 and thus satisfactory to good. Furthermore, different measures of the same overarching construct (i.e., the two measures of intelligence and the two self-esteem scores) were significantly positively correlated.

Descriptive statistics and observer agreements for the personality ratings can be found in Table 3 and for the behavior judgments in Table 4 (their intercorrelations are shown in Tables S4 and S5 of the SOM). To estimate observer agreement for each rating item, we calculated intraclass correlation coefficients (ICCs [2, k]). For ratings with more than one item (e.g., agency), we first computed a mean score of the rating for each observer (e.g., the mean across all agency items) and then used these mean scores to calculate the intraclass correlation across observers. (The ICCs for every single item can also be found in Table S6). With few exceptions for some intelligence-specific cues, the observer agreement was satisfactory to good for all items. It should be noted, however, that communion was ultimately computed with two instead of four items. The reversed items “hostile” and “ruthless” showed poor values (ICC [2, k] = .07 and .17), which can most likely be explained by floor effects (hostile: M = 1.52, SD = 0.30; ruthless: M = 1.55, SD = 0.33). We therefore excluded these two items from the communion composite.

Table 3 shows the results for personality impressions. Accuracy was determined by correlating the aggregated observer ratings with the self-ratings. (Accuracy for single observer are presented in Table S7.) Each of the Big Five traits was positively correlated with the respective observer judgment and the strongest of these accuracy correlations was found for extraversion, which is also the most visible of the Big Five. In each case, the observer-rating from the respective domain was the strongest correlate of the self-rated personality trait (i.e., self-rated extraversion correlated most strongly with observer-rated extraversion, etc.). Regarding self-esteem, both measures were significantly positively correlated with observer-judgments of self-esteem. Again, observer-rated self-esteem was the strongest correlate of both self-esteem measures. Looking at motive dispositions, we found the same pattern for the power and achievement motives. For each domain, a positive correlation between self-reports and observer-ratings was present, and the rating in the respective domain was the strongest correlate of the self-rating. Surprisingly, affiliation self-reports did not correlate significantly with affiliation observer-ratings.

Table 4 shows the correlations of intelligence and narcissism with behavioral cues. (Correlations for single observer are presented in Tables S8 and S9.) In line with our assumptions, intelligence as assessed with the ICAR and the MWT-B, was significantly correlated with almost every predicted cue. For narcissism, the correlational pattern was also as expected. That is, admiration was significantly positively correlated with ratings of agentic behavior, rivalry was significantly negatively correlated with ratings of communal behavior, and vulnerable narcissism was significantly positively correlated with nervous behavior.

We first examined participants’ mood. Compared to before the online interview (M = 3.95, SD = 0.76) affect scores were significantly more positive after the interview (M = 4.14, SD = 0.68), t(188) = -4.43, p \< .001, d = 0.32, indicating that participants’ mood was actually better after the online interaction.

Then, we checked our experimental paradigm for possible experimenter effects. That is, using Analysis of Variance we tested whether for each personality impression and behavior coding whether it varied systematically across experimenters. With one exception (easy to understand; p = .014), we found no significant differences, indicating that no substantial experimenter effects existed.

We next explored potential effects of the device used during the online conversations. That is, using independent samples t-tests we compared participants using a PC or laptop computer (n = 160) with participants using a smartphone or tablet (n = 27) and tested with regard to their mean personality impression and behavior codings (detailed results can be found in the SOM). Group differences for most variables were non-significant, yet there was significantly more higher-language use among participants using a PC/laptop whereas reading time and reading mistakes were higher among participants using a smartphone/tablet. A potential explanation for the latter two findings might be that the text that participants were supposed to read was smaller and therefore harder to read for smartphone/tablet users. We also explored whether the expected accuracy/personality-behavior correlations differed significantly for the two groups using Steiger’s Z test. For the behavioral correlates of intelligence, there were several cases where the correlations with intelligence were significantly higher for the smartphone/tablet group than for the PC/laptop group, but only when intelligence was measured with the MWTB.

Investigating the formation of personality impressions and testing the association between personality and behavior play a central role in personality and social-psychological research. Yet to date, virtually all of the relevant research takes place in offline laboratories, which is costly, time-consuming, and non-inclusive. In the present study, we investigated the ecological validity of personality judgment findings outside the laboratory. The results indicate that collecting observational behavior data in an online setting seems a viable alternative.

The levels of observer agreement were good for both personality impressions and behavioral judgments and similar in magnitude to agreement scores found in previous laboratory studies (Borkenau et al., 2004; Borkenau & Liebler, 1995; Krause et al., 2016). Thus, observer ratings based on online situations seem to be just as coherent as when based on laboratory situations. There was one exception: the two reversed items “hostile” and “ruthless” of communion showed substantially poorer level of observer agreement than in a previous offline study (Schmaliy, 2020). However, it is likely that this difference was not due to the digital format itself, but to the specific interactive paradigm we used. For details see below.

Most importantly, almost without exception the correlational pattern between personality impressions on the one side and personality and behavior observations on the other that has been reported in offline studies could be replicated using the online format. As in previous online studies, we found significant levels of self-other agreement for the Big Five (Borkenau et al., 2004; Borkenau & Liebler, 1993; Hirschmüller et al., 2015), self-esteem (Hirschmüller et al., 2018; Naumann et al., 2009) and the achievement and power motives (Bassler et al., 2023; Dufner & Krause, 2023). Regarding the Big Five, our findings align with the self-other correlation patterns observed in meta-analyses by Connelly & Ones (2010) and Connolly et al. (2007). Specifically, self-other correlations were highest for extraversion, followed by conscientiousness and openness, and finally neuroticism and agreeableness. Extraversion, being a highly visible trait, offers numerous easily detectable cues. Conversely, neuroticism poses greater difficulty for detection as it encompasses tendencies in internal affective states with low visibility (Zillig et al., 2002). The low accuracy correlation for agreeableness stems from its high trait evaluativeness. Since agreeableness is a valued trait for many, people often engage in self-presentation in order to appear agreeable, making it challenging to accurately observe their genuine character (Connelly & Ones, 2010; John & Robins, 1993). A similar explanation may apply to the affiliation motive, which is also socially valued in terms of establishing and maintaining relationships with others.

With regard to the behavioral correlates of intelligence, we could replicate previous findings showing that visual as well as acoustic behavioral cues correlated significantly with measures of intelligence (Borkenau & Liebler, 1995). Moreover, as hypothesized, both intelligence measures were associated with coded behavior. This shows that also when it comes to verbal cues, not only verbal intelligence tests, such as the MWT-B, but also general intelligence tests, such as the ICAR, possess predictive validity. One might wonder about the low (but still significant) correlation between the ICAR and the MWT-B. Past studies have, in fact, reported comparatively low relations between the ICAR and verbal aspects of intelligence such as vocabulary (e,g., Young & Keith, 2020), which can be explained by the fact that the ICAR aims to measure intelligence broadly, and not just its verbal aspects.

For narcissism, we could replicate earlier results indicating that the admiration dimension predicts agentic behaviors, whereas the rivalry dimension predicts a lack of communal behaviors (Back et al., 2013; Schmaliy, 2020). However, one should keep in mind that we deployed adjectives as macro-level, holistic judgments of agentic and communal behavior, rather than specific behavioral cues such as “self-assured voice” or “authentic smile” (Back et al., 2013). Future research should additionally assess micro-level behaviors in an online format. With regard to vulnerable narcissism, researchers have speculated that it goes along with shy and neurotic behavior (e.g., Van der Linden et al., 2010), even though we are not aware of any research including direct behavior observations. The present results indicate that vulnerable narcissism is indeed positively associated with observer ratings of nervousness.

Our results indicated that for both personality impressions and the links between personality and behavior, results that are based on video observations of online calls closely resemble those based on similar studies that used offline laboratory designs. This was true not only with regard to the statistical significance of the correlations, but also for their magnitudes. In almost all cases, the effect sizes reported in the literature (for an overview, see Tables S2 and S3) were within the confidence intervals of the current correlations and similar in size. Specifically, for accuracy of personality impressions, the average effect size in comparable offline laboratory studies is r = .24, while we found an average effect size of r = .29. Regarding the relationship between personality and behavior, the reported effect sizes in offline laboratory studies average at r = .26, compared to our average effect size of r = .22. In each case, the effect sizes can be considered medium-to-large (Funder & Ozer, 2019). Concerning consensus (i.e., the degree to which observers agreed in their evaluation of a given target person; Funder, 1995), the results also resembled past laboratory research. The most relevant past laboratory studies show an ICC value of .36 for a single observer. In our study the average value was highly similar (ICC = .39 for a single observer). For the scores that were aggregated across the four to five observers, the ICCs were considerably higher and mostly in a sufficient range, which indicates that even with relatively few observers sufficient agreement can be attained in the current design.

In summary, previous research has indicated that accurate personality judgements can be made in various virtual settings (see Hinds & Joinson, 2019), such as text-based interactions (Tskhay & Rule, 2014), social network sites (Azucar et al., 2018; Liu & Campbell, 2017; Settanni et al., 2018), or personal websites (Vazire & Gosling, 2004). The current results further reveal the benefits of digital assessment by showing that results on personality expression based on online video calls are highly similar to the ones based on laboratory studies in terms of accuracy. Regarding the moderators of the degree of accurate personality judgments (Funder, 2012), the current findings indicate that online video calls can offer “good information” as basis for accurate personality judgments.

The current research provides an interactive paradigm for the study of personality impressions and the link between personality and behavior. All that is needed on the side of the experimenter and the participant is an Internet-capable device that includes a camera and a microphone. With a duration of five to six minutes, the procedure is highly economic, and it is generally experienced as pleasant, as participants’ increase in positive affect from pre- to post-assessments indicates. The fact that we found the predicted results for so many different personality constructs indicates that our goal to include questions and tasks that possess trait relevance for different personality aspects was accomplished (even though it might be interesting for future research to systematically examine whether some questions possess more diagnostic values than others). An exception might be researchers interested in studying antisocial behavior, as hostile and ruthless behavior hardly ever occurred (see above). If such behaviors are in the focus, more competitive interpersonal situations might be better suited.

In many other cases, researchers who aim to study the effects of any specific personality trait on personality impressions or social behavior and who have an interest in feasibility, reach, inclusion, and comparability might consider using the paradigm (or an adapted version of it).

One should keep in mind, however, that by creating an interview situation, in which participants are also asked personal things about themselves, verbal information is likely to play a key role. To test whether this is indeed the case, future research could vary the modality of the information provided to the judges who rate the targets’ personality or behavior (e.g., with some judges receiving only visual information and other only receiving auditory information; Borkenau et al., 2004). If one wants to focus exclusively on how personality is linked to nonverbal micro-level behavior, such as the frequency of frowning or the body posture, perhaps other tasks that are likely to trigger the specific behaviors of interest (e.g., stress tests) might be more suitable.

Despite the promising results of the current research, online studies including behavior observations are facing several challenges. In the following, we will outline these challenges and give guidance on how to deal with them (for a related list of recommendations, see Perry et al., 2021).

Online approaches necessarily rely on technology capabilities (e.g., Internet connection, audio and video quality, reliability of the software used for the video calls). Whereas researchers can do their best to optimize this point on their side, they have little control over the technical equipment used by the participants. Our recommendation is to at least conduct a quick check of audio and video quality before the actual recording is started. Which kind of device should participants use in future studies with the online setting? In our exploratory analyses, we found no differences between users of different kinds of devices. There was one exception: When behavioral correlates of intelligences were investigated, some results seemed to differ, depending on whether participants used a laptop/PC or a smartphone/tablet. Yet, this conclusion must be tentative given the exploratory nature of the analyses and the modest cell size for the smartphone/tablet group. Such potential sources of noise can easily be prevented by requesting all participants of a given study to use the same kind of device.

Additionally, technical disturbances such as a slow Internet connection or a freezing camera can occur at any time, which significantly reduces the quality of the observation. We recommend researchers to develop clear guidelines on how to deal with such disturbances so that they can be dealt with maximum efficiency and consistency across experimenters. Moreover, participants may have concerns about data storage and privacy when transmitting sensible information about themselves over the Internet, even if anonymity and data protection have been assured by the experimenters. Researchers should only employ online communication software and data storage practices that comply with current privacy policies and explicitly inform participants about this practice. In addition, researchers should rely on server-focused technologies (i.e., no special software or hardware should need to be installed; Perry et al., 2021) and mobile-friendly platforms for participation using a tablet or smartphone.

In general, the online practice might scare off persons which relatively little digital experience, such as, for example, parts of the elderly, or persons who have no constant access to an Internet-capable device. This could potentially result to non-representative samples and disadvantage some groups. On the other hand, however, internet and smartphone usage continues to increase (Silver et al., 2019) so that these issues are likely to become less virulent in the future. Additionally, challenges arise when couples or families participate. Online video calls for behavioral observation do not eliminate the inherent logistical effort required to align partners’ or family members’ schedules to complete study procedures (Perry et al., 2021).

Furthermore, even though conducting studies via online conversations may be more inclusive for physically handicapped persons, one should keep in mind that such persons must have the appropriate technical equipment such as big screens, large character keyboards or screen reader for voice output.

Another challenge for researchers conducting behavior observations online is that participants should be in a comfortable and private space while taking part in the study. Hence, researchers should ensure before the start of the study that participants are home alone or in a private location where they are not disturbed and can act freely, which can be challenging for people living with their family or in flat-sharing community.

As usual in studies on the accuracy of personality impressions, observer ratings were limited to one relatively short interview situation. This practice is not ideal if one is interested in a target person’s typical behavior (Funder & Colvin, 1991). Longer observations, observations in different situational contexts or observations that are made repeatedly over time in an online setting would most likely enable even higher levels of accuracy. Through such an approach that is based on behavior aggregation (Epstein, 1983), it might even be able to use behavior observations not only as an outcome variable that is predicted by traditional assessments of personality, but as an independent personality assessment itself (Tackett et al., 2016).

Another interesting challenge for future research would be to automatize the procedure further. For example, rather than actually interacting with the experimenter, participants could receive instructions and tasks via e-mail with the request to videotape their behavior. Such an approach would, of course, be still far more economic than the current paradigm. Also on the side of personality ratings and behavior codings, the level of automatization could be increased. In emotion research, the usage of emotion recognition software that detects emotions from facial expressions or from speech signals is already widespread (for an overview, see Wani et al., 2021) and some artificial intelligence models are already being trained to infer target persons’ personality (Cannata et al., 2022). However, personality detection frequently encounters ethical dilemmas concerning the appropriate interpretation and utilization of assessment techniques (Mukherjee & Kumar, 2016). The growing integration of artificial intelligence across various domains of psychological research in recent years necessitates the development and adherence to specific legal and ethical guidelines.

The online setting appears promising for studying personality impressions and the relation between personality and behavior. However, the current study only demonstrated that the results pattern and effects sizes are similar to what has been reported based on offline settings, without precisely delineating how or whether online settings might differ from classical laboratory observations. To address this question, future research might test the same participants both settings and to directly compare the results (for a similar approach, see Kaurin et al., 2018). In such a study, one could also address the possibility that in online assessments, which typically take place in people’s private rooms, impression management concerns might be attenuated in comparison to laboratory settings, which typically feel unfamiliar and might even maybe slightly threatening.

Even though the contribution of the present research is a methodological one, the findings also raise substantive research questions. Whereas most previous studies focused on either a single personality trait or small number of traits on the predictor side and a single judgment or a small number of judgments on the outcome side, we simultaneously assessed a relatively large number of personality constructs and observer judgments. What this approach showed is that observer judgments were often not only linked to the hypothesized personality predictor, but to other traits as well. For example, observer-rated neuroticism was almost as strongly correlated to extraversion as it was to neuroticism. Such a pattern shows that even though observer judgments of personality have convergent validity, their discriminant validity can be low in many cases. Likely, there are behavioral cues that are systematically interpreted in an incorrect fashion (e.g., a behavioral correlate of extraversion is interpreted as an indicator of low neuroticism). Future research should examine this issue more systematically using lens model analyses (Brunswik, 1956) and it should do so by considering both shared and unique effects.

In doing so, researchers should also give deeper thought on the presumed ontological status of personality constructs. Typically in research on personality and behavior, a personality trait is (either explicitly or implicitly) considered a causal entity that resides inside a person and makes them likely to display a specific behavior (e.g., extraversion causing outgoing behavior). However, it is also possible to view personality and inter-related behaviors from a formative perspective (targets who show behaviors A and B may be called “C”) or from a dynamic network perspective (descriptions of A and B might correlate − and thus form a factor − because A is the cause of B; for an overview, see Leising & Borgstede, 2020). Future researchers should specify their theoretical perspective and ideally test it by using a formalized modelling approach.

Some limitations of the present research should be noted. First, our study was not pre-registered. In hindsight, this was unfortunate, given that pre-registration enhances transparency, reduces researcher bias, and fosters credibility within the scientific community (Simmons et al., 2021). We acknowledge the benefits of adopting this practice in future research endeavors.

Second, and somewhat relatedly, due to the numerous statistical tests, it cannot be ruled out that false-positive results may have occurred. Those who are cautious should interpret only effects with p ≤ .01 or even more cautiously, if with p ≤ .005 (Benjamin et al., 2018), which makes false positives highly unlikely. However, even using these conservative thresholds, most effects persist, which speaks in favor of the online paradigm.

Third, we used undergraduate psychology students as observers, a choice that was primarily guided by logistical reasons (and that has also been made in many earlier studies). An upside of using students as observers is their familiarity with the to-be-judged constructs, requiring minimal explanation or training. A downside of student observers, however, is their demographic homogeneity (i.e., predominantly young and female), potentially leading to agreement biases stemming from shared stereotypes (Kenny, 2004; Letzring et al., 2021). Furthermore, it is conceivable that due to the students’ interests and training in psychology, accuracy correlations might be higher than in the general population. Especially if the goal is to draw conclusions about personality impressions in real life, using a more representative set of observers would be advisable.

Fourth, it cannot be ruled out that the accuracy correlations could be influenced by stereotypes unrelated to actual behavior − even if more heterogeneous samples of observers are tested (see e.g., Smartt et al., 2022). To address this possibility, future studies should code behavioral cues for all traits and test if they mediate the accuracy effects.

Finally, because mood was only assessed with a single, the assessment is rather unspecific and its reliability is unknown. A nuanced, multi-item measure would have avoided these issues.

Behavioral observations offer a valuable and irreplaceable source of information in the study of personality. However, laboratory studies are cost intensive in terms of the investment of time, money, and effort and also non-inclusive. The present research demonstrates that observations of video calls yield highly similar results to those made in the laboratory when it comes to linking personality to personality impressions or behavior judgments. Our research also provides a specific paradigm for online behavioral research that can be easily implemented, is evaluated positively by participants, and provides valid results. Video calls therefore not only connect us, but also open a new lens for exploring the nuances of personality.

Contributed to conception and design: MS, MD

Contributed to acquisition of data: MS

Contributed to analysis and interpretation of data: MS

Drafted and/or revised the article: MS, MD

Approved the submitted version for publication: MS, MD

We thank Alina Klute, Annelie Schnabel, Can Aygün, Zeza Arckel, and Lennart Kraume for their help in study implementation and data collection.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

The authors declare no potential conflicts of interest concerning the research, authorship, or publication of this article.

Supplemental material can be found online at the Open Science Framework: https://osf.io/8utzk/

This study was not preregistered. Materials, participant data, and analysis scripts are available at the Open Science Framework: https://osf.io/8utzk/