Social Smartphone Apps Do Not Capture Attention Despite Their Perceived High Reward Value

Applying a value-driven attentional mechanism to a smartphone scenario, it is plausible that certain smartphone features (e.g., app icons) have been associated with social rewards through repeated use. Consequently, these features gain attentional priority and therefore attract attention and eventually harm visual search performance. As the major part of social interaction on smartphones happens via apps, we assume that app icons carry social reward to the user. For instance, social apps (e.g., Facebook, WhatsApp), particularly with a notification sign, should be associated with high social reward, as notifications usually convey social validation, such as friends liking a picture or friend requests (Reich, Schneider, & Heling, 2018). Conversely, nonsocial apps (e.g., Weather, Calculator) should not carry social rewards as they are not used for social purposes. So, analogous to the value-driven attention account, we expect that social app icons should similarly attract attention and therefore slow down visual search. Therefore, we predict that low social reward distractors (social app icons) and high social reward distractors (social app icons with a notification) result in slower reaction times compared to no reward distractors (neutral app icons; H_1a–b), and that high social reward distractors result in slower reaction times than low social reward distractors (H_1c).

In addition, it is well-established that deprivation of rewarding experiences strengthens the motivation to obtain these experiences (Seibt, Häfner, & Deutsch, 2007). For example, depriving participants of food led to a higher reinforcing value of the food compared to not hungry participants (Epstein, Truesdale, Wojcik, Paluch, & Raynor, 2003). Similarly, it is common practice to assess the true value participants assign to food after a fasting period (e.g., Chen, Veling, Dijksterhuis, & Holland, 2016). In the case of smartphones, if social apps truly are rewarding, the appeal of social apps, similar to food, should be stronger for those who have been deprived of using these apps. Evidence for such a position comes from studies showing that phone separation is associated with strong emotional reactions (Hoffner & Lee, 2015), leads to anxiety (Cheever, Rosen, Carrier, & Chavez, 2014), impairs cognitive control (Hartanto & Yang, 2016), and results in physiological stress reactions (Clayton, Leshner, & Almond, 2015). Consequently, the reward value associated with app icons should be particularly high, and hence distracting, when participants are motivated to use these apps. We thus hypothesize that all main effects of distractor are stronger for users who have previously been deprived of their phones compared to a control group (H₂).

This set-up enables us to exclude alternative explanations: If we indeed find the expected pattern for distractor, (a) low and high social reward distractors might capture attention merely because the social apps are more familiar to participants, given that they are used more; (b) high social reward distractors might capture attention more than low social reward distractors because of the red color of the notification sign. Therefore, only if the effect is amplified in the deprivation condition can we conclude that apps indeed carry reward for users, above and beyond the possible effects of familiarity and color.

To test our hypotheses, we adapted the visual search task introduced in Anderson et al. (2011b). We chose this paradigm for two reasons. First, it is a well-established method to assess the effect of reward-associated distractors on attention (for reviews see Anderson, 2016; Failing & Theeuwes, 2017; Le Pelley, Mitchell, Beesley, George, & Wills, 2016). Second, the visual search task represents a good approximation of smartphone distractions in real life scenarios. For instance, consider a student who has to write a paper, but the Facebook notification sign repeatedly captures their attention.

We deviated from the original paradigm in two major aspects. First, we omitted the reward learning phase from the current study because we assumed that people learned to associate social rewards with smartphone app icons through repeated exposure in everyday life. Therefore, we only used the testing phase of the original paradigm. Second, in order to increase ecological validity, we used smartphone app icons as distractors. By using real-life icons, we followed recent studies which show that more complex visual information, such as pictures of people or scenery, can also be associated with rewards (Failing & Theeuwes, 2015; Hickey, Kaiser, & Peelen, 2015).

Thus, in the current study, participants were instructed to find the target while they were distracted by app icons that were associated with high social rewards, low social rewards, or no social rewards. In the original paradigm, the rewarding nature of stimuli is reflected in impaired visual search. Consequently, the visual search task paradigm provides us with a test of the proposed smartphone-reward association: If social smartphone cues are indeed more rewarding than neutral smartphone cues, they, like other rewarding stimuli, should impair visual search. In other words, impaired visual search performance serves as an indicator of the reward associated with smartphone cues.

We preregistered hypotheses, sample size, inclusion and exclusion criteria, and statistical analyses. Our preregistration, experimental materials, data, and analysis are available on the Open Science Framework (https://osf.io/g8kbu/).

As power calculations are not entirely straight-forward for linear mixed-effects models (Scherbaum & Ferreter, 2008), we preregistered a rather conservative sample size. Therefore, we recruited 120 students from a Dutch university. We had four inclusion criteria: First, participants needed to have normal or corrected to normal vision. Second, students needed to own an iPhone. This ensured the icons we used as distractors would be identical to those that participants use on their iPhones every day. Icons are standardized across iOS compared to Android, where icons often differ between devices due to the open source nature of the Android OS. Third, as people under 25 report the highest smartphone use (CBS, 2018; Pew Research Center, 2017), our participants had to be younger than 25 years. Fourth, participants had to have the five distractor apps Facebook, Facebook Messenger, Instagram, Snapchat, and WhatsApp installed on their iPhone and they had to be frequent users of these apps for at least two years. These criteria were meant to ensure that reward learning had taken place, that is, stimulus features had been paired with the delivery of (social) rewards (Le Pelley et al., 2016): Using these five social apps frequently plausibly has led to an established association of social rewards with visual features of these apps.

Following our preregistered exclusion criteria, we excluded three participants as they did not reach 70% accuracy on the task. Thus, the final sample consisted of 117 students (59 in the control and 58 in the deprivation condition; 106 females, M_age = 20.85, SD_age = 1.88). Participants were compensated with monetary rewards in the form of a gift voucher (€5 or €10) or course credits. The study had IRB approval and all participants gave informed consent.

We employed a mixed design with deprivation as a between-subject independent variable (2 levels: deprivation group vs. control group), app distractor icon as a within-subject independent variable (3 levels: high social reward vs. low social reward vs. no social reward) and response time (RT) as dependent variable.

We randomly assigned participants to either the deprivation or the control condition. In the deprivation condition, we asked participants to come to the lab one hour before the experiment to hand in their iPhone, which we locked away in a drawer. Then, we told participants that they were free to go about their day within the next hour, but asked them not to engage in any social media activity until the experiment started. After one hour, they came back and performed the task (see below). After the task they received their phone. In the control condition, participants came to the lab at their assigned time slot and directly performed the task.

Before starting the task, participants reported demographics (age and gender). In order to assess whether our deprivation manipulation indeed led to an increased motivation to use their smartphones, participants then answered a short manipulation check on a 1 (not at all) to 100 (extremely) visual analogue scale (“Right now, to what extent do you feel an urge to check your phone?”). Then, they performed the visual search task. Finally, after finishing the task, participants reported a second manipulation check, namely whether they had seen 20 apps (ten of which were used in the experiment) during the course of the visual search task. With this question, we tested whether participants actually processed the distractor app icons throughout the visual search task.

We designed a visual search task based on Anderson et al. (2011b). Participants were seated about 50 cm from a monitor with a resolution of 1920 × 1080 pixels. On each trial, participants first saw a fixation cross with a visual angle of 0.5°, then six shapes organized in an imaginary circle with a visual angle of 10°; each shape had a visual angle of 3.45°; last, participants were presented with a performance feedback display (see Figure 1).

Among these six shapes, there was always one unique shape, which was defined as the target (i.e., a circle among diamonds or a diamond among circles). Each nontarget shape contained a black line tilted by 45°. The target shape always contained either a horizontal or vertical black line. On all trials, there was a distractor app icon embedded (1.73° visual angle) in one of nontarget shapes, on top of the tilted lines.

These distractor app icons represented three levels of social rewards (high, low, and no social rewards, see Figure 2). On the high social reward distractor trials (see Figure 2A), there was a social app icon with a notification sign (Facebook, Facebook Messenger, Instagram, WhatsApp, and Snapchat) within one of the nontarget shapes. We chose these apps because they are the most commonly used social apps. The red notification was identical to the one used on iOS. On the low social reward distractor trials (Figure 2B), there was a social app icon (i.e., same icons without the notification sign) within one of the nontarget shapes. As stated above, these apps are mainly used for social purposes – so we assumed they represent social reward to people, but less than these same apps with the certainty of a notification sign. Finally, on the no social reward distractor trials (Figure 2C), there was a neutral app icon (Weather, Settings, Notes, Clock, and Calculator) within the nontarget shapes. We chose these specific icons as they are pre-installed on every iPhone, so iPhone users most likely encounter them often enough; yet, they are never used for social purposes, so we assumed that participants could not have possibly associated social rewards with any of the neutral app icons. The target shape never included any distractors (i.e., icons). Target and distractor location were randomly determined; distractor app icon and the unique shape were counterbalanced.

Participants were instructed to search for the target, which was always defined as the unique shape in the search display, and report whether the line within the target shape was horizontal or vertical, by pressing the “z” and “m” keys (counterbalanced). The experiment consisted of 480 trials: 120 trials (25%) contained a high social reward distractor, 120 trials (25%) contained a low social reward distractor, and 240 trials (50%) contained a no social reward distractor. Before the task, participants did 24 practice trials. After each 96 experimental trials, participants could take a short break. The task took ~35 minutes to finish.

We conducted all of our analyses in R (version 3.5.0, R Core Team, 2018). In line with our preregistration, we tested our hypotheses using a linear mixed-effects modeling approach using the lmer function (lme4 package; version 1.1.17; Bates, Maechler, Bolker, & Walker, 2015). We aimed for a ‘maximal’ random effects structure as advocated by Barr, Levy, Scheepers, and Tily (2013) to avoid inflated Type-1 errors. Accordingly, our model predicting response time included two random intercepts; a per-participant random intercept to account for the repeated-measures nature of the data and a per-app icon random intercept to account for any additional variance in response time caused by the specific app icons included in our study. We modeled the within-subject predictor distractor as fixed effect and as random slope varying across participants. We modeled the between-subject predictor condition as fixed effect and as random slope varying across app icons.

To determine p-values, we preregistered to compute Type III bootstrapped Likelihood Ratio Tests using the mixed function (afex package; version 0.20-2.; Singmann, Bolker, Westfall, & Aust, 2018). However, this analysis led to several convergence warnings that persevered after the recommended troubleshooting steps. Thus, we followed recent recommendations by Luke (2017). Based on simulations, he compared several approaches to evaluating significance in mixed-effects models, and concluded that F-tests with Satterthwaite approximation for degrees of freedom are the most appropriate to control Type-1 error rates. Thus, we opted for this approach instead (also using the mixed function), which resulted in no convergence warnings.

Directly before starting the visual search task, participants in the deprivation condition reported a higher urge to check their smartphone (M = 51.81, SD = 21.20) than participants in the control condition (M = 32.28, SD = 27.15), t(111.44) = –4.39, p < .001, d = .80. At the end of the experiment, participants correctly classified whether or not they had seen 20 different app icons (ten of which we used as distractors) with an accuracy of 84%, indicating that they did process the distractors during the search task.

In line with our preregistration, we excluded any trial on which (a) the RT was below 300 ms (<0.01%) and (b) the RT was ±3 SDs from the participant’s mean (0.01%). For the analysis, we also excluded all inaccurate trials. Participants were accurate on 92% of the experimental trials. Across all remaining experimental trials from all participants (N = 51083) mean response time was 676.46 ms (SD = 81.17).

The main effect of distractor was not significant, F(2, 13.49) = 0.90, p = .428. Participants’ response time did not significantly differ between high social reward distractors (M = 678.82, SD = 83.11), low social reward distractors (M = 676.52, SD = 82.06), or no social reward distractors (M = 675.26, SD = 81.28). To our surprise, the main effect of condition was significant, F(1, 114.99) = 4.00, p = .048. Overall, participants in the deprivation condition (M = 661.61, SD = 76.17) responded faster than participants in the control condition (M = 691.06, SD = 83.88), irrespective of the type of distractor presented on any given trial. Last, the interaction effect of distractor and condition was not significant, F(2, 348.63) = 2.59, p = .076. To investigate whether there was indeed no interaction effect and to better understand our data, we tested the main effect of distractor in both conditions separately. The main effect of distractor was neither significant in the control condition, F(2, 16.54) = 1.18, p = .33, nor in the deprivation condition, F(2, 16.18) = 2.56, p = .11. Taken together, the effect of distractor did not significantly differ between the deprivation condition and the control condition. A visualization of the raw data associated with our analysis can be found in Figure 3.

A major limitation of our frequentist model is that it cannot quantify evidence for the null hypothesis. Therefore, to investigate to what extent our data support the lack of an effect, we conducted a Bayesian repeated-measures ANOVA with the anovaBF command (BayesFactor package; version 0.9.12-2; Morey & Rouder, 2015). The model employed the default Cauchy distribution for the prior. The Bayes Factors associated with our predictors can be found in Table 1. Comparing a model with the main effect of condition to the null model yielded inconclusive evidence, as the data were 1.63 times more likely under the null model without the effect of condition (BF₁₀ = 0.61). On the one hand, the Bayesian ANOVA does not allow an analysis as fine-grained as the frequentist mixed model, as it does not include a per-icon random intercept and random slope of condition. On the other hand, p-values close to the cut-off of α = .05 often do not represent much evidential value (Benjamin et al., 2018), which is further illustrated by the Bayes Factor we obtained. The Bayesian analysis of the main effect thus shows that we should interpret the significant main effect of deprivation with caution.

In addition, supporting the nonsignificant effect of distractor, there was strong evidence that the data were much more likely under a null model compared to a model with the effect of distractor (BF₀₁ = 755). The same holds for the interaction effect, which was not supported compared to a model with the two main effects (BF₀₁ = 155).

In order to follow up on the unexpected main effect of condition, we investigated whether there was a speed-accuracy tradeoff. A maximal generalized mixed-model with accuracy as the dependent variable did not show a significant effect of condition (χ²(1) = 0.0, p = .99). Supporting the lack of an effect, a Bayesian contingency table showed strong support for the lack of a difference between the conditions (BF₀₁ = 167). We conclude that there was no speed-accuracy tradeoff, and that participants in the deprivation condition indeed performed better (faster while equally accurate).

Contrary to our expectations, high social reward apps did not slow down visual search compared to low or no social reward apps, neither in the smartphone deprived, nor in the control condition. Based on prior work we assumed that different apps would have different levels of reward associated with them (e.g., Bayer et al., 2015; van Koningsbruggen et al., 2017). However, one possible explanation for this null effect is that social apps were not perceived as more rewarding than neutral apps. In fact, unlike in the original study series on value-driven attention, we did not directly manipulate stimulus-reward associations. In the original task, participants go through an extensive reward training, in which arbitrary stimuli, such as color, become associated with the delivery of monetary rewards. Consequently, these reward-associated stimuli slow down visual search; that is, impairment of visual search is an indicator of attentional capture by the reward of the stimuli. However, in our application of this paradigm we did not manipulate reward, but assumed the reward value of apps had been established in real life, through repeated use prior to the experiment. The lack of an effect on visual search speed might then either reflect that the stimuli are not rewarding, or that they are rewarding, but not rewarding enough to cause differences in attentional capture. Due to the design of Study 1, we cannot be certain that participants indeed perceived social apps as more rewarding than nonsocial apps. Therefore, we need to establish whether our reward manipulation was effective after all to rule out the alternative explanation that the app categories were not different in their associated reward.

To address this possible alternative explanation for the null effect of reward-associated distractors, we conducted a survey where participants rated all 15 apps we used during the experiment on how rewarding they found them. We expected that, if the three levels were truly to manipulate social reward, we should at least be able to detect a difference on how people themselves perceive these different apps. Accordingly, we hypothesized that high social reward apps would be rated higher than both low social reward apps and no social reward apps. In addition, we expected low social reward apps to receive higher ratings than no social reward apps.

We preregistered hypotheses, sample size, inclusion and exclusion criteria, and statistical analyses. The preregistration, experimental materials, data, and analysis are available on the Open Science Framework project of this article (https://osf.io/g8kbu/).

Because we expected an experimental manipulation to induce at least a medium-sized effect (η_p² = .05) on a manipulation check, we aimed to obtain 95% power to detect an effect of at least that size at α = .05 for the main effect in a repeated-measures ANOVA. Thus, we preregistered to recruit 160 (150 needed for 95% power plus ten to account for exclusions) valid responses on the online platform Prolific. We counted those submissions as valid that passed an attention check (see below), as Prolific lets researchers resample participants if a participant fails an attention check.

We aimed to obtain a sample as similar as possible to our sample in Study 1. Overall, 252 participants from the UK between the ages of 18 and 25 opened the survey. All participants were screened and had to currently own an iPhone and have used an iPhone for at least the past two years. Furthermore, participants had to have the five social apps from Study 1 installed and had to have used them for at least the past two years. In addition to these inclusion criteria, we preregistered several exclusion criteria. First, 52 participants were excluded because they did not finish the survey. Second, of the remaining 200, 40 did not pass an attention check (see Procedure). Third, we excluded two participants who indicated that they were older than prescreened by Prolific. No participant fulfilled our fourth exclusion criterion of having a variance of zero across all rated apps, or our fifth exclusion criterion of spending less than 30 total seconds on the 15 apps to rate (M_seconds = 72, SD_seconds = 31). Thus, the final sample consisted of 158 participants (M_age = 21.56, SD_age = 2.40) of which 110 were female (70%).

Participants were informed that the aim of the study was to find out how people experience different apps. In particular, participants were informed that they were to rate different apps on how rewarding they find them. To make clear what we meant with rewarding, we provided several clarifications (e.g., feeling happy when using the app, feeling a strong need to use it, liking the app). To avoid participants overthinking their responses, we instructed them to respond promptly, based on their immediate thoughts about each app. To avoid confusion about the difference between a high social reward app (i.e., a social app with a notification sign) and a low social reward app (i.e., the same social app without a notification sign), we instructed participants that the apps would sometimes have a notification sign and that they should treat the app as if they saw it in that form on their own phone. Because understanding the task instructions was crucial to accurately rate the apps, we implemented two measures to ensure participants properly read the instructions. First, going to the next page was only possible after 20 seconds. Second, at the end of the task description, we instructed participants to select “No” to proceed to the task as an attention check.

Participants then proceeded to rate all 15 stimuli used in Study 1 on the question “How rewarding do you find this app?” on a visual analogue scale ranging from –100 (not at all) to 100 (very much). Presentation order of the apps was randomized. The entire survey, on average, took about three minutes (M_seconds = 185, SD_seconds = 71) and participants received £0.50. The study had IRB approval and all participants gave informed consent.

We conducted a repeated-measures ANOVA with app category (within: high social reward vs. low social reward vs. no social reward) as predictor and ratings of how rewarding participants found those apps as outcome. As the assumption of sphericity was violated (W = .30, p < .001), we report the F-statistic with Greenhouse-Geisser correction. The main effect of category was significant and large, F(1.18, 184.75) = 150.77, p < .001, η_G² = .32. The strength of evidence for an effect of category was further supported by a Bayesian repeated-measures ANOVA with the standard Cauchy prior (BF₁₀ = 1.63e + 107). To test our predicted contrasts we conducted three post-hoc two-tailed paired t-tests without correction for multiple testing, as correction for multiple testing is not necessary for designs with only one factor with three levels. We present paired Bayesian t-tests alongside the frequentist results.

In line with our predictions, high social reward apps (M = 36.99, SD = 33.18) received significantly higher ratings than low social rewards apps (M = 25.46, SD = 34.31), t(157) = 7.61, p < .001, BF₁₀ = 3.06e + 09, d_z = 0.61, and significantly higher ratings than no social reward apps (M = –22.00, SD = 43.48), t(157) = 13.20, p < .001, BF₁₀ = 1.15e + 24, d_z = 1.05. In addition, low social reward apps received significantly higher ratings than no social reward apps, t(157) = 11.64, p < .001, BF₁₀ = 7.36e + 19, d_z = 0.93. The residuals within each condition were roughly normally distributed and the results were robust to removal of outliers. A visualization of the raw data associated with our analysis can be found in Figure 4.

The goal of the current study was to test whether smartphone distractions stem from the high social rewards associated with smartphone apps. Participants engaged in a visual search task while they were distracted by smartphone app icons. Although we show that participants perceive social apps as more rewarding than neutral apps, that perceived reward did not impair performance in a visual search task. Also, depriving participants of their smartphone did not amplify such an effect. However, surprisingly, participants who were deprived of their smartphones performed better. In short, these results suggest that even if people perceive social apps as more rewarding than nonsocial apps, being exposed to these apps as distractors does not influence performance on a visual search task. However, there are several alternative explanations, both theoretical and methodological, for our findings.

One possible alternative explanation for the lack of an effect of the three app groups is that participants did not perceive social apps as more rewarding than neutral apps. Instead of manipulating reward, as is common with the visual search paradigm, we assumed that users repeatedly obtain social validation and gratification from social apps (Karapanos et al., 2016; Reich et al., 2018), such that they would learn to associate social reward value with these apps. Therefore, we expected that social app icons, particularly those with a notification sign, gained their reward value in everyday life and should be as powerful as a controlled reward training phase in the lab. To provide evidence for this line of reasoning, Study 2 showed that people themselves report social apps to be more rewarding than neutral apps, especially if social apps have a notification sign. Importantly, the effect we obtained was large. As a consequence, we can be more confident that the lack of an effect is not due to a failed manipulation of reward.

That being said, there are several caveats to this objection which do not allow to draw a clear conclusion from our behavioral data regarding the reward value of apps. First, Study 1 and Study 2 were run on different samples, albeit matched on demographics. Technically, insights from the ratings in Study 2 might not apply to participants in Study 1. In addition, although we show a difference in how the different app sets (i.e., high, low, and no social reward) are perceived, these ratings are relative to each other. We cannot be certain a social app with a notification sign does truly feel rewarding – or just more rewarding compared to a neutral app, which might not feel rewarding at all. As such, the relative difference in perceived reward value might not manifest itself on a behavioral level because, in absolute terms, the reward associated with apps is not large enough to attract attention (Potter, 2011).

Furthermore, we did not have a no-app control condition. Such a control condition would be informative by testing whether all apps, regardless of their perceived value, slow down visual search. Similarly, implementing a control condition with an arbitrary symbol (e.g., a symbol similar in shape to app icons) as distractor could provide a test whether app icons attract attention above and beyond any other distractor. This view aligns with the lack of an interaction between app icons and the deprivation manipulation. We predicted that social apps would be particularly distracting if users had been deprived to access them (Epstein et al., 2003; Seibt et al., 2007). Yet our data show that deprivation did not affect whether participants were more or less distracted by different apps. The lack of an interaction provides additional evidence for the explanation that perceived reward did not manifest itself on a behavioral level. Future research could consider using a no-reward control condition, an arbitrary symbol control condition, or even contrast apps with the low and high monetary reward condition used in the original paradigm (Anderson et al., 2011b) to test such a proposition.

Our findings echo work demonstrating that people’s perception about smartphones do not necessarily translate to behavior. For instance, Johannes et al. (in press) found that receiving a notification during a cognitive control task did not impair performance despite participants reporting to find the notification highly distracting. In a similar vein, other studies found that people are not good estimators of their own smartphone or internet use (Ellis, Davidson, Shaw, & Geyer, 2018; Scharkow, 2016). Taken together, these findings suggest that there might be a gap between what people themselves report about the distractions of their smartphones and the actual behavioral impairment these devices exert on them.

Our null findings have another potential explanation. In our experiment, we presented app icons in complete isolation from their usual context, which may have reduced their reward value altogether. Drawing from the theory of grounded cognition (Barsalou, 2009), experiences are stored in people’s mind within a complex structure of sensory input, cognitions, affective states, and situational cues. It is possible that drawing a specific cue from such a rich, situated experience reduces the value of that cue. It is likely that it is only in their real-life context that smartphone apps represent social reward, because context is part of the reward value-association. Consequently, participants might explicitly evaluate app icons as rewarding if there is no time pressure and they can imagine the icons within the context of their own phones, as in Study 2. In contrast, when these app icons get isolated from their real context in the lab, they may not affect attention (Best & Papies, 2017). Thus, the lab situation may not be appropriate for behaviorally measuring reward associations with smartphone apps. Supporting this reasoning, it has been shown that other types of rewards, such as food, are often stored in memory in terms of situations, for instance, where people eat them (e.g., popcorn is associated with cinema) and whom people eat those foods with (e.g., family events; Papies, 2013). Future research could address this issue by measuring smartphone distractions in their natural context, for example, on people’s own smartphones.

In sum, our results suggest that social app icons do not impair visual search. Given that social apps are rated as more rewarding outside of the lab, the lack of an effect of these apps on attention could be due to a loss of associated reward when taken out of context or insufficient absolute reward levels.

To our surprise, we found that participants who had locked away their phone an hour prior to the experiment were overall faster on the visual search task than participants who had not locked their phone away. However, this effect should be interpreted with caution for several reasons. First, we did not predict nor preregister a main effect of the deprivation condition. As such, the effect needs to be regarded as exploratory until independently replicated. Second, the p-value for the main effect of deprivation was extremely close to the alpha-level and many scholars argue that p-values of that size have limited evidential value (Benjamin et al., 2018). Our Bayesian analysis supports a need for caution regarding the effect, as the Bayes Factor in favor of the effect was inconclusive. Third, given the complexity to calculate power for our analysis, we cannot be certain we had sufficient power to detect a main effect. As a consequence, it is possible that our design was underpowered for the main effect, which is problematic for several reasons. Most important, underpowered studies that yield a significant estimate will necessarily overestimate the true effect size (Vasishth, Mertzen, Jäger, & Gelman, 2018). Thus, we can only reiterate that the surprising main effect of deprivation requires high-powered, preregistered replication.

Last, we cannot exclusively attribute the effect to phone deprivation, as there was more than one difference between the non-deprived and the deprived group. Whereas participants in the non-deprived group came to the lab and immediately did the visual search task, participants in the deprived group came to the lab one hour earlier to lock away their phone and were free to do as they pleased during the one hour of deprivation, except for checking social media. Consequently, those deprived participants had one more contact point with the researchers and there was no control over what they did during the hour of deprivation. This difference might present an alternative explanation for the main effect: For instance, all participants were informed in the recruitment text for Study 1 that some of them might have to lock away their phone. Thus, at the point of locking away their phones participants in the deprived group could easily deduct that they were in the experimental condition. This knowledge might have induced reactance in the form of motivation to show that they could still perform well without their phones. Future research employing such a deprivation manipulation should consider having all participants come to the lab an hour prior to the task.

That being said, if we assume that the effect reflects a true difference between the conditions, regardless of possible confounding factors, it is plausible that locking participants’ smartphones away increased their motivation, which resulted in better performance. The idea that participants could get their smartphone back and access its social rewards when they were done with the task may have motivated them to perform faster. In other words, being able to check their social media and their messages after 1.5 hours may have acted as an incentive that they could receive at the end of the experiment (Aarts, Custers, & Veltkamp, 2008). In line with such a view, participants in the deprivation condition indeed reported a higher urge to check their phones. Interestingly, this improvement in speed did not come at the cost of performance, as accuracy was almost identical across the two conditions. Such an interpretation corroborates the well-established idea that incentives boost cognitive performance (Botvinick & Braver, 2015). Another account suggests that removing the smartphone as a distractor may have resulted in better performance, as smartphones have been shown to impede attention (Stothart et al., 2015; Thornton et al., 2014; Ward et al., 2017).

Regardless of whether our data support a positive effect of deprivation or no effect, at the very least our findings stand in contrast to previous experiments, where participants who were separated from their phones performed worse on cognitive tasks (e.g., Hartanto & Yang, 2016). These studies argue that smartphone separation increases anxiety (Cheever et al., 2014), which in turn leads to worse task performance. In light of these findings, our results are quite surprising and, when replicated, may have important implications. On the one hand, this apparent discrepancy might result from the difference in manipulations between studies. Whereas previous work deprived students of their phones strictly during the tasks, participants in our experiment were deprived both before and during the task. On the other hand, previous work also showed that anxiety due to phone separation rose with time (Cheever et al., 2014); if anything, increasing the deprivation duration should have had an even stronger negative effect on performance. Even taking into account our methodological concerns surrounding the effect of deprivation, there is a clear need for more research addressing this inconsistency. Doing so is of importance, as many policy makers, for example in schools, base their policies on findings demonstrating a detrimental effect of smartphones.

Overall, according to our findings, app icons are perceived as rewarding, but they do not capture attention and therefore do not distract participants from their task. Moreover, being deprived of access to these apps might not always be detrimental. However, these conclusions are constrained to our specific study design, which is subject to several possible alternative explanations. Our findings highlight that investigating media effects is complex and requires thorough designs that take the context of smartphone stimuli into account. As such, we believe that the current inconsistencies in the literature warrant more highly powered, preregistered research before making recommendations to policy makers.

All experimental materials, data, and analysis code are available on the Open Science Framework (https://osf.io/g8kbu/).

The authors would like to thank Francesco Mazzeo for his help with data collection, as well as Gijs Bijlstra, Harm Veling, and Erik Bijleveld for their feedback on earlier drafts of this manuscript.

This research was partially supported by the Emerging Scholars Research Grant of the Mobile Communication Interest Group of the International Communication Association (ICA).

The authors have no competing interests to declare.

All authors contributed equally to all parts of the article. Author order was determined randomly with the following command in R:

The author(s) of this paper chose the Open Review option, and the peer review comments are available at: http://doi.org/10.1525/collabra.207.pr