Do Reward-Related Distractors Impair Cognitive Performance? Perhaps Not

To test our hypothesis, we used our original paradigm (Rusz et al., 2018) – without the motivation manipulation in the test phase – which was originally designed based on research on value-driven attention (Anderson et al., 2011) and on research on distraction during math performance (Beilock et al., 2004; Beilock & Carr, 2005). First, in a training phase, participants learned to associate earning high vs. no monetary rewards with two different colors. Later, in a test phase, participants performed a mental arithmetic task (they had to add up four numbers), while the previously rewarded colors reappeared as distractors. So, participants had to prioritize the arithmetic task, while ignoring distractors, even though these were sometimes associated with reward. As simple mental additions can take up to 800–900 ms (Ashcraft, 1992; Ashcraft & Battaglia, 1978), we expected that the rather fast presentation of the numbers (700 ms per number) would make it especially difficult to perform these mental additions while trying to ignore distractors. Therefore, we expected that distractors associated with reward (vs. no reward) will disrupt cognitive performance, which we operationalized as the percentage of accurate responses.

To explore the circumstances under which reward-based disruption – if we can detect it at all – is strongest, we designed three variations of our task described above. In these variations, we tested whether distractors associated with reward impede cognitive performance more when they were previously always associated with rewards (Experiment 1A) vs. when they were randomly associated with rewards on 80% of trials (Experiment 1B), or when they were located further away from task-relevant cues (Experiment 1C).

We preregistered our study on AsPredicted (https://aspredicted.org/3j7gw.pdf). Unless otherwise noted, inclusion criteria and statistical analyses were pre-registered. The experimental task, data, and analysis scripts can be found via https://osf.io/dwa89/. This research was approved by the Ethics Committee of the Social Science Faculty (ECSW2017-0805-50).

Participants were 90 students from Radboud University. We recruited fluent Dutch speakers, who were 25 years old or younger, slept at least 6 hours the night before the experiment, and were not colorblind. After data collection, 2 participants nevertheless reported to have slept less than 6 hours, so they were excluded from the final analysis. Moreover, we excluded 7 participants who performed below 60% accuracy. The final sample consisted of 81 students (27 participants per task variation; 59 females and 22 males; M_age = 22.35 years, SD_age = 1.96). Participants received compensation in cash based on their performance (maximum 6 €).

Participants were seated in a cubicle in front of a computer. First, participants gave their written informed consent to participate in the study. Next, they reported their demographics (age, sex), hours of sleep the night before, and their need for money on a 1 (not at all) to 7 (very much) scale (“To what extent are you in need for money at the moment?”). Afterwards, they carried out the task (see below). Then, they reported how motivated they were, and how demanding and difficult they felt the task was on a 1 (not at all) to 7 (very much) scale. Finally, they filled out the Dutch version of the Barratt Impulsiveness Scale (BIS-11; Patton, Stanford, & Barratt, 1995; see Table 1 for descriptive statistics of all subjective measures). We collected these subjective measures in order to explore whether current states and traits affect reward-based distraction (based on results of Anderson et al., 2011). The experiment took around 35–40 minutes to finish. At the end, participants were thanked, paid, and debriefed.

The task consisted of two parts, a training and a testing phase. In the training phase (see Figure 1), participants first saw a fixation cross, then four times a number + letter combination presented one after each other (e.g., 8W, X5, 9Y, and Z7; see Figure 1). Their task was to report whether the letters were in the correct alphabetical order (e.g., W, X, Y, Z). They responded by pressing “Q” for correct and “P” for incorrect trials. On half of the trials, one of the letters had a different color. If this letter was blue (or red, counterbalanced across all participants, N = 90), participants could earn a monetary reward (8 cents). If it was red (or blue, counterbalanced), participants could earn no monetary reward. On no-reward trials (e.g., red), responses were followed by visual feedback indicating “Good” or “False”. High-reward trials (e.g., blue) were additionally followed by reward feedback (+ 8 cents) and the total amount that has been earned during the task so far. Participants were not informed about the reward contingency beforehand. In total, participants completed 4 practice trials and 150 training trials.

In the test phase (see Figure 2), participants saw a fixation cross and four times a number + letter combination presented one after each other (e.g., 8W, X5, 9Y, and Z7), and lastly, a two-digit number that was +/–1 to the sum of these numbers (e.g., 28). Their task was to add up the four numbers and report whether the sum of the presented numbers (e.g., 8 + 5 + 9 + 7 = 29) was higher or lower than the number presented in the last display (e.g., 28). They responded by pressing “Q” for smaller and “P” for larger sums (29 is bigger than 28, so the correct response would be “P”). Identical to the training phase, on half of the trials, one letter was always red and the other half of the trials one letter was always blue. In this case, the letters served as distractors, previously associated with monetary (vs. no monetary) rewards. Importantly, because the spatial location of the distractor (left vs. right) was randomized, participants could not predict the location of the distractor, which made distractors difficult to ignore. In total, participants completed 10 practice and 64 test trials.

As mentioned above, we designed three variations of the experiment (between-subject factor), to which we randomly assigned participants (N = 30/variation). All three variations were in line with the general task description above, with only slight differences. In Experiment 1A, reward contingency in the training phase was 100%. That is, high-reward distractors were previously associated with earning money on all trials. To test whether a variable ratio schedule (Thorndike, 1898) makes the hypothesized disruption effect stronger, Experiment 1B used an 80%–20% reward contingency. That is, high-reward distractors were previously associated with earning money on 80% of the trials (vs. all trials). Finally, Experiment 1C was identical to Experiment 1A with one difference: both in the training and test phases, target (e.g., 8) and distractor (e.g., W) were located spatially further away from each other (8.5 mm vs. 34.2 mm difference between target and distractor/1680 × 1050 display). In this version, the goal was to test whether reward-based disruption is stronger when it takes longer to direct eye movements from distractor (e.g., red colored W) to target (e.g., 8). If indeed eye movements are rapidly captured by reward-related distractors, it should take longer to direct attention back to task relevant cues (Henderson & Macquistan, 1993; Laberge & Brown, 1986; Shulman, Wilson, & Sheehy, 1985), which may result in even worse performance on the math task (for an alternative perspective, see Gaspar & McDonald, 2014; Hickey & Theeuwes, 2011).

We should note that we did not preregister these analyses; thus, these results should be interpreted with caution. As we had no a priori hypotheses about the learning phase, we mainly rely on effect sizes when we interpret the results (Forstmeier, Wagenmakers, & Parker, 2017; Greenland et al., 2016; Nosek, Ebersole, DeHaven, & Mellor, 2018). Additionally, the p-values that are related to exploratory analyses in this paper are not corrected for multiple comparisons.

In order to examine reward learning, we ran a GLM analysis with reward (high vs. no) and block (first vs. second) as independent variables, separately for accuracy and response time (RT) as dependent variables. Based on prior research (e.g., Anderson, 2016b; Sha & Jiang, 2016), the first goal of this analysis was to explore whether participants are more accurate and/or faster when there is a reward (vs. no reward)-predictive color in the sequence. In line with analyses from previous studies (Anderson, Laurent, & Yantis, 2014; Bourgeois, Neveu, Bayle, & Vuilleumier, 2015; Failing & Theeuwes, 2015), the second goal was to test whether this difference would change over time. More specifically, if there is a difference in responses to reward predictive vs. non-predictive trials, this difference should be especially pronounced by the end of the training phase (e.g., in the second block) – when participants had enough time to pick up on stimulus-reward associations. We ran our GLM, first, on the whole sample and second, separately for each experiment. Results are summarized in Table 2 and Figure 3a–d.

Inspection of Table 2 shows that high (vs. no) reward predictive colors did not influence accuracy or RT when considering the pooled data from all experiments. When examining the experiments separately, however, we found that reward learning seemed more pronounced in Experiment 1A and Experiment 1B (80–20% random reward) compared to Experiment 1C (increased distance). In Experiment 1A, participants seemed to be more accurate on high-reward trials, though this was not accompanied by faster response times. In Experiment 1B, participants became faster in the second half of the experiment, particularly on high-reward trials, but they did not become more accurate. So, while there were indications of reward learning in both Experiments 1A and 1B, these indications were not entirely consistent.

We further explored whether task-related subjective measures (motivation, difficulty, demands), current states (need for money, fatigue), and trait impulsivity were related to performance in the training phase. We found that trait impulsivity was negatively related to accuracy scores on reward predictive trials (r = –.34, p = .002). It could be that for participants who score high on this trait, reward predictive colors triggered some impulsive responses that were less likely to be accurate. Furthermore, self-reported motivation was positively related to accuracy scores in both no-reward (r = .26, p = .017), and high-reward conditions (r = .37, p < .001). All other correlations were not significant (ps > .05).

Responses that were three standard deviations faster or slower than the participant’s mean and responses faster than 300 ms (which were considered guesses) were deleted, which resulted in the exclusion of 1.4% of trials. To test our hypothesis, we performed a GLM analysis with the reward value of the distractor (high vs. no) as a within-subject independent variable, and accuracy scores (percentage of correct responses) as the dependent variable. There was no significant difference in accuracy scores between high vs. no distractor reward trials, F(1,80) = 0.26, p = .608, η_p² = .00. Thus, contrary to our hypothesis, we found no evidence that reward-related distractors impede cognitive performance.

To test whether different task variations had any effect on performance in high (vs. no) reward distractor trials (see Table 3 for descriptive statistics), we ran the same GLM as before, now adding task variation as a between subject factor. Crucially, the distractor reward × task variation interaction was also not significant, F(2,78) = 0.15, p = .864, η_p² = .00. To conclude, we did not find evidence that reward-related distractors harm more when reward contingency is random (no difference between Experiment 1A and Experiment 1B) and when the distractor is located further away from the target (Experiment 1C).

To further explore these null results, visually inspecting individual accuracy scores in high vs. no reward distractor conditions might be helpful. Figure 4 shows each participants’ difference score (i.e., high-reward accuracy minus no-reward accuracy scores) across task variations. A difference score below 0 would mean that our manipulation worked, that is, participants had lower accuracy (i.e., performed worse) in the high-reward distractor condition. When inspecting Figure 4, it is clear that participants did not clearly score below 0.

As a secondary analysis, we tested whether high (vs. no) reward-related distractors had an impact on people’s speed. To this end, we ran another GLM with distractor reward (high vs. no) as within subject independent variable and RT as dependent variable. We found no significant difference in RTs between high vs. no distractor reward trials, F(1,80) = 0.90, p = .346, η_p² = .01. The distractor value × task variation interaction was not significant either, F(2,78) = 0.57, p = .563, η_p² = .01. In sum, we found no evidence that reward-related distractors slow down people’s responses on a math task.

To provide more conclusive evidence for this null effect, we tested our predictions with a Bayesian approach (Dienes, 2014) in JASP (JASP Team, 2018). A Bayesian paired-samples t-test with distractor reward (high vs. no) as independent and accuracy scores as dependent variable yielded a BF₀₁ = 6.68, which is considered moderate evidence for the null hypothesis (see Figure 5A). Another Bayesian paired-samples t-test with distractor reward (high vs. no) as independent variable and RTs as dependent variable also showed moderate evidence for the null hypothesis, i.e., BF₀₁ = 5.26 (see Figure 5B). In sum, these results strengthen findings from our preregistered analyses: they directly support the idea that reward-related distractors do not harm cognitive performance.

To test whether different task variations had an effect on performance in high (vs. no) reward distractor trials, we ran a Bayesian repeated-measures ANOVA with distractor reward and task variation as independent variables, and separately for accuracy and RTs as dependent variables. The distractor reward × task variation interaction on accuracy as dependent variable yielded a BF₀₁ = 91.29 – and on RTs as dependent variable yielded a BF₀₁ = 52.68. These results show strong evidence against the idea that different task variations moderated performance in high vs. no distractor reward trials.

We explored whether task-related measures (motivation, difficulty, demands), current states (need for money, fatigue), and trait impulsivity were related to the amount of reward-related capture (calculated as RT when a high-reward distractor was present minus RT when no-reward distractor was present; based on Anderson et al., 2011). Difficulty was positively related to the amount of reward-related capture (r = .25, p = .028). This implies that the more difficult one felt like the task was, the more they got distracted (i.e., in this case, became slower) from high reward distractors. Other correlations were not significant (ps > .05).

Finally, we further explored whether performance in the training phase was related to the distractor effects in the test phase. For this purpose we computed a reward learning score for each participant (i.e., the difference between RTs on no vs. high reward trials on the first part of the training phase minus the difference between no vs. high reward trials on the second part of the training phase). The higher the score is, the bigger the RT difference in between no vs. high reward predictive trials in the second vs. the first phase of the learning phase and, thus, the more successful reward learning. This reward learning score was not related to the amount of reward-related capture (r = .06, p = .58); thus, we found no evidence for a relationship between reward learning and reward-based distraction.

Note: in our previous paper, in Experiment 2, we found that reward-related distractors impaired performance only when they appeared early vs. late during the trial. To be consistent, we performed that same analysis (GLM with distractor reward and timing as within subject independent measures and accuracy as dependent measure) on the current data too. This analysis yielded no meaningful effect (F < 1). This means that unlike in our previous study (Rusz et al., 2018), distractor timing did not moderate reward-based distraction.

The goal of the present research was to test whether reward-related distractors harm cognitive performance. Participants first learned to associate reward (vs. no reward) to different colors in the training phase via a deterministic (Experiment 1A and 1C) or a random reward schedule (Experiment 1B). After, they performed a math task, which involved difficult operations (e.g., the active maintenance and updating of task-relevant information), while they were exposed to reward-related, but task-irrelevant cues. Task-irrelevant and task-relevant stimuli appeared spatially close together (Experiments 1A and 1B) or far apart (Experiment 1C).

Our main confirmatory test suggested that being exposed to reward-related distractors do not necessarily thwart cognitive performance on a mental arithmetic task. This finding is unexpected, given that a large body of prior work showed that reward-related distractors do capture visual attention (e.g., Anderson et al., 2011; Bucker & Theeuwes, 2017; Le Pelley et al., 2015). Furthermore, this finding is not fully in line with our own previous findings on math performance. In particular, in our previous study (Rusz et al., 2018), we found only weak evidence for reward-driven distraction; in the current study, we found evidence against reward-based distraction. Although the current study’s method was somewhat different from the studies in Rusz et al. (2018), the studies collectively suggest that reward-related distractors do not harm math performance. Now, we will discuss three possible explanations for finding evidence against our hypothesized effect.

First, it is plausible that reward-related distractors simply have no influence on task performance that relies on cognitive control processes. As discussed in the introduction, the rapid visual capture effect of reward-related cues may deteriorate in later processing stages. Supporting this possibility, Theeuwes and Belopolsky (2012) showed that distractors associated with reward rapidly captured visual attention (i.e., saccades), but did not affect eye movements after this capture. Along these lines, recent research showed that reward-related distractors more likely influence early (vs. late) stages of visual processing (Failing et al., 2015; Lee & Shomstein, 2014; Maclean & Giesbrecht, 2015). Corroborating this finding, Le Pelley, Seabrooke, Kennedy, Pearson, & Most (2017) found that the negative effect of reward-related distractors was rather short-lived on a temporal attention task (but see Failing & Theeuwes, 2015). Based on these studies, one could speculate that reward-related distractors impede early cognitive processes, leading to misguided saccades. However, such early “disruptions” may have only limited effect on later processing stages, because people may be able to quickly correct for their impact.

Indeed, results from Experiment 1C seem consistent with the possibility that reward-related distractors rapidly capture visual attention, but do not affect later cognitive operations. In this version of our task, we increased the distance between target and distractor, assuming that this would increase the chance for visual attentional capture by the distractor associated with reward – thus maximizing the chance for reward-driven disruption. Surprisingly, this manipulation had no effect on how accurate participants were in high vs. no distractor reward conditions. Nevertheless, participants were somewhat slower on trials where a high (2210 ms) vs. no (2113 ms) reward-related distractor appeared. This effect was small (d = .29) and non-significant, so it should be interpreted with extreme caution. Still, based on this finding, one might speculate that although people showed misguided saccades to reward-related cues, they might have been able to overcome this initial shift (within 700 ms) as they were still able to respond accurately on the math task.

As we did not measure people’s gaze during our study, we cannot conform nor disconfirm the latter possibility, but we do feel this would be a promising future direction. Such a design—i.e., a design that combines eye tracking measures with math performance—would help to better understand whether optimal performance on the math task is still possible, even if attention is initially rapidly captured by reward-related distractors. Specifically, it would be informative to see what aspects of gaze behavior (e.g., whether the first fixation is on the distractor vs. the target; how quickly people can disengage from the distractor; whether people can avoid fixating on the distractor at all) is most likely to impair performance (Failing et al., 2015; Le Pelley et al., 2015).

Second, it is possible that reward-related distractors do impair cognitive processes beyond visual capture, but that our task was not sensitive to detecting such an effect. After inspecting the results, one may argue that our task was too easy; overall, participants were quite accurate (i.e., above 80%). Thus, even if participants were initially disrupted by reward-related distractors, they may have managed to protect goals from these interfering stimuli: they could quickly (i.e., within 700 ms) correct for this and apparently were still able to process the targets (e.g., 8) and to perform mental operations on these targets (e.g., 9 + 8). In line with this explanation, exploratory findings revealed that perceived task difficulty was related to the susceptibility for reward-based distraction. Specifically, participants who thought that the task was more difficult, were slowed down more by high reward-related distractors. This explanation is in line with previous research that shows that under high working memory load, people are more susceptible to distractions (for a review, see Lavie, 2010). Speculatively, future research should make the task more difficult (e.g., by increasing the pace), or take into account individual differences, such as working memory capacity (e.g., Anderson et al., 2011).

The potential lack of difficulty might also explain why our results are not in line with Krebs et al. (2011, 2010), who reported that irrelevant-reward associations interfere with conflict processing. For two reasons, the Stroop task used in Krebs’ studies is different from the current task. First, in the Stroop task, distractor colors are always associated with a competing response, so in those studies, distractor effects may reflect response interference, not just attentional capture. In our task, such response competition was absent, as participants never had to respond to the colors themselves. Second, in the Stroop task, the concept color is always task-relevant, which makes ink colors difficult to strategically ignore. In our task, it may have been easier for people to categorically ignore the color dimension during the test phase, thus preventing distraction before it began. The latter issue could be addressed in future research, e.g., by introducing mini-blocks of learning and testing phases (e.g., Lee & Shomstein, 2014). Such a design could prevent people from strategically only focusing on numbers, not colors, thus making distractors even harder to ignore.

Third, although we found some indications that participants learned reward associations in the training phase (especially in Experiment 1B), these stimulus-reward associations may have been too subtle to disrupt performance during the test phase. Indeed, our learning phase was different from most previous studies in this area in three ways. First, in previous studies, the learning phase often employed instrumental learning—where action towards a certain cue (e.g., finding a red circle) was directly associated with earning money. In our study, however, reward learning was arguably less instrumental in nature. Specifically, while stimuli (e.g., a blue X) predicted the availability of reward contingent on people’s action (indicating the correct order of the letters), people were never required to approach this cue, or actively search for it. Second, our training task required more focused attention, and perhaps greater working memory capacity, than learning tasks in prior studies (which often used visual search procedures). This strong attentional focus may have caused people to readily suppress task-irrelevant dimensions (in this case, color), impairing acquisition of stimulus-reward associations. Third, our training phase was shorter than training phases in previous studies (e.g., 240 in Anderson et al., 2011), which may have given simply less time for participants to learn reward associations. We should note, though, there are existing studies that used training phases with less instrumental reward learning (Bucker & Theeuwes, 2016; Le Pelley et al., 2015; Pool et al., 2014), more focused attention learning tasks (Anderson, 2016b; Mine & Saiki, 2015) and shorter training sessions (e.g,. 144 trials in Sali et al., 2014); these prior studies have shown reward-based attentional capture in a test phase, suggesting that our learning phase, in principle, should be conducive to reward learning. We recommend that future studies use a more established reward learning procedure, to facilitate interpretation of results.

We further examined how different variations of our task affected performance in high vs. no distractor reward conditions. First, we introduced a random reward schedule in Experiment 1B (Skinner & Ferster, 2015; Thorndike, 1898), but confirmatory results from Experiment 1B (80% random reward) were similar to those of Experiment 1A (100% reward). Second, we located the distractor further away from the target in Experiment 1C. In this version of the task, misguided saccades towards the distractor should have had the strongest detrimental effects for performance. Nevertheless, by contrast to mounting evidence from vision research (Anderson, 2016a; Failing & Theeuwes, 2017), we did not find evidence for such an effect. It could be that even if participants’ eye movements were initially captured by reward-related distractors, they could quickly disengage from these cues and still perform the math task well. To conclude, task variations did not influence the magnitude of disruption of performance by reward-related distractors.

In the current research, we found no evidence that reward-related distractors harm performance during a mental arithmetic task. In our view, this finding is interesting as it contradicts findings from visual search tasks (but see Sha & Jiang, 2016), from conflict processing tasks (Krebs et al., 2011, 2010), and from our previous research on mental arithmetic tasks (Rusz et al., 2018). All in all, our research suggests that reward-related distractors may not harm performance on all types of tasks that require cognitive control.

The preregistration, all the stimuli, presentation materials, participant data, and analysis scripts can be found on this paper’s project page on the Open Science Framework (https://osf.io/dwa89/).

The authors have no competing interests to declare.

• Contributed to conception and design: DR, EB, MK

• Contributed to acquisition of data: DR, EB

• Contributed to analysis and interpretation of data: DR, EB

• Drafted and/or revised the article: DR, EB, MK

• Approved the submitted version for publication: DR, EB, MK

The author(s) of this paper chose the Open Review option, and the peer review comments are available at: http://doi.org/10.1525/collabra.169.pr