Why Does the IAT Reveal a Preference for Stimuli Said to Be Paired With an Unpleasant Sound? Stalking the Unexpected

Experiment 1 was based on the same method and design adopted in De Houwer et al. (2019). We changed one specific aspect of the procedure, that is, the order of administration of the IAT and the CSs rating phase. In our previous study, CSs ratings were taken right after the onset of each US and before the potential occurrence of the unpleasant sound. The IAT was administrated only after the CS ratings were completed. Crucially however, none of the CSs ratings were followed by the occurrence of the sound. Because the CS rating phase provided a disconfirmation of the instructions (i.e., unlike to what participants had been told, CSs were never followed by USs), the effect of the instructions on the IAT might have been reduced (e.g., if participants no longer believed the instructions) or even reversed (e.g., if participants distrusted the instructions and suspected that the reverse might be true) simply because the IAT was always administrated after the CS rating phase. To test this idea, in Experiment 1, the IAT was presented immediately after the instruction phase. By implementing this change, we expected the impact of instructions on the IAT to be more similar to the effect on self-reported liking (i.e., a preference for CS_noUS over CS_neg).

Experiment 2 introduced two important new elements. First, IAT instructions made clear that during the IAT participants would not hear any sound (but the sounds would occur in the later phase). This should prevent participants from expecting the occurrence of the unpleasant noise while performing the IAT (which, if participants would find this unexpected absence positive, could drive the unexpected IAT effect). Second, we used a personalized version of the IAT (pIAT, Fazio et al., 1986). In a typical IAT (e.g., with positive, negative, and stimulus category labels), many influences other than liking-related influences can be used to complete the task (see Meissner et al., 2019). Most prominently, participants might draw on specific propositional information that is unrelated to participants’ beliefs about the valence of the stimuli ( De Houwer, 2014). For instance, in De Houwer et al. (2019), some participants might draw on the propositional information that there is a positive relation between the CS_neg and a sound (e.g. “CS_neg and the unpleasant noise will co-occur”) and a negative relation between the CS_noUS and a sound (e.g., “CS_noUS and the unpleasant noise will not co-occur”) when completing the IAT. This might lead to better performance in IAT blocks where participants respond to CS_neg with the key assigned to positive words. Circumventing this problem, the pIAT uses “I like” and “I dislike” to capture specific evaluative beliefs about the target stimuli. This should therefore encourage participants to perform the task based on beliefs that relate to their actual liking or disliking the target stimuli.

Experiment 3 further explored the idea that beliefs unrelated to liking might affect IAT performance. One such belief relates to the “figure-ground” effect (see Rothermund & Wentura, 2001). When instructions inform participants that an aversive sound will be sometimes present and sometimes absent after seeing one CS but always absent after seeing another CS, then the presence of the sound becomes more salient (the figure) than the absence (ground). As a result, a CS predicting the US (CS_neg) might become more salient. Assuming that the IAT category “I like” is also more salient than the IAT category “I do not like”, the salience account would predict faster IAT performance in blocks in which the CS_neg and “I like” are assigned to the same key, which is what we observed. To test this idea, we changed two aspects of the procedure. First, we manipulated the instructions referring to a contrast (non-negative) CS (CS_contrast: CS_noUS vs. CS_pos) such that for half of the participants they indicated that the CS would never be followed by the unpleasant sound (CS_noUS: as before) and for the other participants, we included a pleasant sound and instructions indicated that the CS would always be followed by this pleasant sound (CS_pos). In the latter condition, both CS_neg and CS_pos signaled the presence of a valenced sound. This should reduce differences in CS salience and ultimately prevent a figure-ground effect in the IAT. Second, half of the participants completed a standard IAT, whereas the other participants completed a recoding-free IAT in which the assignment of two categories to the same response switches around from trial to trial (rather than in separate IAT blocks). It has been argued that the recoding-free IAT is less susceptible to the effects of figure-ground asymmetries (Meissner et al., 2019). Hence, if figure-ground asymmetries are responsible for the unexpected effects in the IAT, then the recoding-free IAT should reveal a preference for the CS_contrast rather than for the CS_neg.

For all the experiments, we pre-registered the materials, sampling plan, exclusion rules, analysis plans, and hypotheses on the Open Science Framework. We report all data exclusions, manipulations, measures, and how we determined our sample sizes. The materials, data and analysis scripts of all experiments are available at osf.io/f6u2n/.

In-line with De Houwer et al. (2019), Experiments 1 and 2 adopted a 2 (CS type: CS_neg vs. CS_noUS) x 2 (CS-US Statistical Contingency: high vs. low) x 2 (CS-US Co-occurrence: high vs low) mixed design, with the first factor measured within and the other two factors measured between subjects. We also counterbalanced the identity of CS_neg, the assignment of each CS to either positive or negative stimuli in the first block of the IAT, and the order in which the two CSs were evaluated (the last two factors were nested).

In Experiment 3, we opted for a 2 (CS Type: CS_neg vs. CS_contrast) x 2 (Statistical Contingency between CS_neg and US: high vs. low) x 2 (CS_contrast Instructions: absence of negative sound (CS_noUS) vs. presence of a positive sound (CS_pos)) x 2 (IAT type: standard IAT vs. recoding free IAT) mixed design, with the first factor measured within and the other three factors measured between subjects (see Figure 1 for a schematic overview of the procedure adopted in the original and the present studies). All the experiments received approval from the Ethical Committee of the Faculty of Psychology and Educational Sciences at Ghent University.

Participants. Participants were recruited via the Prolific Academic website (https://prolific.ac/). The target sample size was set via a-priori power analyses conducted on G-Power 3.1. For Experiments 1 and 2, we determined that 200 participants needed to complete our experiment to have high power (> 0.90) to observe a moderate sized effect (d = 0.50) in a planned t-test on IAT scores (i.e., a significant preference for the CS_noUS over the CS_neg), at alpha = 0.05. In Experiment 3, we planned a sample size of 280 participants to have at least 255 participants in the final analyses which provides sufficient power (0.90) to observe a difference of small to moderate effect size (η²_p = 0.04) in an ANOVA testing the interaction between the manipulated factors of instructions and IAT Type on IAT scores, at alpha = 0.05. In all experiments, we opted for slight oversampling to control for possible exclusions and recruited 228, 230, and 301 participants, respectively (Expriment1: 125 women, Mage = 33.71, SD = 11.08; Experiment 2: 138 women, Mage = 34.30, SD = 12.07; Experiment 3: 156 women, Mage = 37.12, SD = 12.26).

In line with our data analysis plan, participants who (1) failed to complete the whole session or had incomplete data, or (2) did not accurately describe the sound or indicated that they could not hear it, (3) had error rates above 30% across the IAT or above 40% for any one of the four critical IAT blocks, or (4) completed more than 10% of IAT trials faster than 400ms, were excluded from the analyses. Applying these screening criteria led to 198 (Experiment 1), 190 (Experiment 2), and 262 (Experiment 3) participants included in the final analyses.

Stimuli. The same stimuli were used as in De Houwer et al. (2019). The negative US was the sound of a woman’s scream. Two nonsense words (MORAG and STRUAN) were used as CSs. It was counterbalanced across participants which word served as CS_neg or CS_noUS (or CS_pos). In Experiment 3, the sound of baby laughter was used as the positive US.

Procedure. Experiment 1. After providing informed consent and demographic information, participants were instructed to put on their headphones and set the sound volume to 30%. Participants then listened to the unpleasant auditory stimulus and were asked to briefly describe it in an open response format.

Participants received verbal instructions about the number of times in which two different visual stimuli (CS_neg and CS_noUS) were meant to be followed by the aversive sound they just listened to (US) in a following task. Four different between-subjects instruction conditions resulted from crossing instructed CS-US contingency (high: 80% vs. low: 50%) and instructed CS-US co-occurrence (high: 8 trials vs. low: 4 trials). This instruction phase was followed by a retention test that served to assess participants’ recollection of the correct number of events for both words. Otherwise, participants saw the instructions screen again. Both the instructions and the retention test referred to the US stimulus as “the sound”, without qualifying it based on its pleasantness.

Participants then completed an IAT. The IAT consisted of three practice blocks and two experimental blocks. It began with 16 practice trials in which the target CSs were sorted under their own categories, followed by 16 practice trials sorting the attribute words. Next, participants completed 64 trials in which CS_noUS and positive words shared one response key and CS_neg and negative words shared another response key. Participants then practiced sorting the CSs on 16 trials with the reversed response key assignment. Finally, they completed a second set of 64 trials in which the CS_noUS shared a response key with negative words and CS_neg shared a response key with positive words. On each trial, a valence word or CS was presented in the center of the screen until the participant pressed one of the two valid keys (i.e., ‘E’ or ‘I’). In case of correct response, the stimulus disappeared and was replaced by the next stimulus, presented 400ms later. If the response was incorrect, a red ‘X’ replaced the stimulus for 200ms and participants were to correct their mistake to proceed to the next trial.

After the IAT, participants completed a CSs ratings phase that consisted in the individual presentation of the two CSs on screen. Participants were informed that between the offset of the CSs and the potential onset of the aversive sound they would be asked to rate the CSs on two distinct dimensions. The two CSs were presented individually and in counterbalanced order in the center of the screen for 4 seconds, followed by both a valence and a US-expectancy rating scale, administered in counterbalanced order between participants. No sound was displayed after the scales and participants were informed that the condition they have been assigned to involved a single trial for each stimulus. Participants rated the extent to which they liked the two CSs on a 21-category scale (-10 = very negative; 0 = neutral; +10 = very positive). US expectancy ratings were collected on each trial using a 9-point Likert scale presented below the CSs with 3 anchor points: 1 = “not at all”, 5 = “uncertain”, 9 = “certainly”. Finally, participants answered two questions about clarity and credibility of the experimental instructions. In this questionnaire, participants indicated the clarity and believability of the instructions on a 10-point scale and could additionally provide general remarks about the experiment (Raes et al., 2014). A second memory test, identical to that administered after instructions, was repeated at the very end of the session.

Experiment 2. Both the design and the experimental procedure mirrored those used in Experiment 1, except for two main modifications. First, we administered a pIAT. There were two differences between the standard version of the IAT used in Experiment 1 and the pIAT: First, the attribute category labels were the Dutch words for “I like and”I don’t like" instead of “Positive” and “Negative”. Second, as is typically the case in pIATs, there was no error feedback for the attribute stimuli (because we do not know what stimuli participants like or dislike). Second, instructions preceding the task specified that participants should not expect the occurrence of any sound after presentation of each CS on screen.

Experiment 3. The procedure of the experiment was identical to that of Experiment 1, except for variations in (1) instructions about stimulus pairings (and the presence of an additional auditory US) and (2) the nature of the implicit measure. Specifically, participants assigned to a first instruction condition (with two USs) were exposed to both a pleasant and an unpleasant sound at the very beginning of the session. The pleasant US reproduced the laugh of a baby. The two USs were matched in duration. After hearing the USs, participants received verbal instructions about the number of times the two CSs would be followed by the USs. One CS (CS_neg) was said to be followed by the aversive US (either 8/10 times or 8/16 times, depending on Contingency), while the other CS (CS_contrast) was said to be either never followed by the unpleasant US (CS_noUS) or always followed by an alternative US selected for being pleasant (CS_pos). The retention test for participants assigned to this condition varied according to the manipulation and so did the options of answer. Participants assigned to a second instruction condition (with one US) had the same instructions as in Experiments 1 and 2. Note that, like in Experiment 1-2, there was no explicit mention to the pleasantness of the USs. The instructions and the retention test referred to the US stimuli as “the first sound” and “the second sound”.

Participants assigned to a first measurement condition completed a recoding free IAT (IAT-RF, Rothermund et al., 2006). The IAT-RF consisted of a first block of two blocks with 16 trials in which participants sorted CS_neg vs. CS_contrast stimuli and positive vs. negative stimuli, respectively, a second block with 32 combined practice trials, and a final block with 128 test trials. In contrast to the typical IAT, compatible and incompatible assignments of categories to responses varied randomly between trials within a single experimental block. Each trial began with response assignments indicated as category labels in the respective corners of the screen. Due to the random switching between compatible and incompatible response, participants were forced to categorize all presented stimuli on the basis of their task-relevant category membership. Participants assigned to the second measurement condition completed the typical IAT. Finally, along with the exploratory questions, participants provided their evaluation of the sound(s) on an 11-point scale ranging from -5 to +5 (extremely unpleasant/extremely pleasant).

We present the results for the effect of CS type on both self-reported liking and IAT scores by reporting the results of a paired-sample t-test (comparing self-report CS ratings) and a one-sample t-test (comparing the IAT score to zero), respectively. For Experiments 1-2, we also report results of an ANOVA examining the impact of both Contingency and Co-occurrence on both self-report ratings and IAT score. For Experiment 3, we report results of an ANOVA examining the impact of Contingency (there was no co-occurrence manipulation), IAT type and US instructions.³ Because not of crucial importance for the purpose of the present project, we do not report results for US expectancy ratings in the main text (results on US expectancy for all the experiments are available at osf.io/f6u2n/).

Experiment 1. Self-report ratings. A paired-sample t-test indicated a typical EC instruction effect, t(197) = -3.79, p < .001, d = .27. Participants indicated less liking for the CS_neg (M = –0.76, SD = 4.41) than for the CS_noUS (M = .73, SD = 4.56). A mixed ANOVA with CS Type as within-subjects factors and Contingency and Co-occurrence as between-subjects factors showed only the main effect of CS Type, F(1,194) = 13.47, p < .001, but no significant interaction between CS Type and Contingency, CS Type and Co-occurrence, or the three considered together in a three-way interaction, ps > .27.

IAT scores. IAT scores were calculated using the D2-algorithm (Greenwald et al., 2003) so that a positive score indicates a preference for the CS_neg over the CS_noUS. Split-half reliability was r(196) = .69. Contrary to our hypotheses, we observed an overall preference for CS_neg (M = .09, SD = .44), t(197) = 2.97, p = .003, d = .21.⁴ The factorial ANOVA included Contingency and Co-occurrence as between-subjects factors (but not CS Type because the IAT score is a relative score so the main effect of CS Type is reflected in the Intercept). This ANOVA revealed neither main effects of Contingency or Co-occurrence, nor an interaction between the two, ps > .14. We also did not observe a significant correlation between the IAT score and self-reported liking difference scores, r = .12, p = .09, BF₀₁ = 1.55.

Experiment 2. Self-report ratings. A paired-sample t-test indicated an EC effect, t(189) = -3.78, p < .001, d = .27. Participants indicated less liking for the CS_neg (M = –1.02, SD = 4.69) than for the CS_noUS (M = .54, SD = 4.76). A mixed ANOVA showed an effect of CS Type, F = 14.22, p < .001, but no significant interactions with either Contingency or Co-occurrence, or both, ps > .31.

IAT scores. Split-half reliability was r(188) = .62. An overall preference for the CS_neg emerged, M = .08, SD = .41, t(189) = 2.64, p = .009, d = .19. Neither main effects of Contingency or Co-occurrence nor an interaction between the two was found, ps > .30. Again, we observed no significant correlation with self-report liking difference scores, r = .03, p = .72, BF₀₁ = 5.58.

Experiment 3. Self-report ratings. The mixed ANOVA with CS Type as within-subjects factor and Contingency, IAT Type and CS_contrast instructions revealed an effect of CS Type, t(261) = -4.83, p < .001, d = .30, indicating less liking for the CS_neg (M = –.73, SD = 4.82) than for the CS_contrast (M = 1.05, SD = 4.76). No other main effects or interaction effects were observed, ps > .12.

IAT scores. Split-half reliability for both the standard and the IAT-RF were r(136) = .81 and r(122) = .42, respectively. We found an overall preference for the CS_neg, M = .06, SD = .34, t(261) = 2.60, p = .010, d = .16. The 2 (Contingency: High vs. Low) × 2 (IAT Type: standard IAT vs IAT-RF) x 2 (CS_contrast Instructions: absence of negative sound vs. presence of a positive sound) ANOVA did not reveal any main effects or interactions, ps > .06. There was a significant correlation between IAT scores and self-report liking, r = .16, p = .012, BF₁₀ = 3.22.

US evaluation. Participants who were exposed only to the unpleasant sound rated it as negative (M = -3.17), t(127) = -13.32, p < .001. Participants exposed to both the USs evaluated the pleasant US more positively (M = 1.96) than the unpleasant (M = -2.57), t(133) = -12.65, p < .001. For participants exposed to a single sound, self-report liking towards the CS_neg was not correlated with the evaluation of the US, r = .08, p = .37. Instead, for participants exposed to both USs, self-reported CS_neg liking significantly correlated with evaluation of the unpleasant US, r = .23, p = .009, and CS_pos liking correlated with that of the pleasant US, r = .29, p = .001. No significant correlation between IAT scores and US evaluations was found.

Data from the three experiments were combined (N = 650) to investigate the overall effect of the instruction-based conditioning procedure on both self-report ratings and IAT score. Because we were not interested in any other effect at this stage, none of the additional variables manipulated across the three studies (i.e., contingency, co-occurrence, IAT Type and CS_contrast instructions) were entered in this analysis. Bayes factors were reported for each analysis to quantify the evidence for the alternative over the null hypothesis (BF₁₀).

Self-report ratings. A paired-sample t-test revealed a preference for the CS told to be never paired with the unpleasant sound, t(649) = -7.13, p < .001, d = .28, BF₁₀ > 1000.

IAT scores. A one-sample t-test revealed an overall preference for CS_neg, t(649) = 4.75, p < .001, d = .19, BF₁₀ = 2738.16. The correlation between the IAT score and self-report ratings was significant, although weak, r = .10, p = .008, BF₁₀ = 2.99.

Across three experiments, we consistently replicated the results of De Houwer et al. (2019). Most importantly, we observed that on a self-report measure of liking, a CS said to be paired with an aversive auditory sound was evaluated less positively than a CS said to be never followed by the US (Experiment 1-2) or always followed by a pleasant auditory sound (Experiment 3). In contrast, this evaluative pattern never emerged when looking at the effect on the IAT. Experiments 1-3 explored several explanations of this unexpected effect in terms of measurement order, the type of IAT measure, and whether instructions referred to either the consistent absence of the negative US or to the presence of a positive US after the contrast CS. Surprisingly, the difference in results on self-reported liking versus IAT scores was observed in all of the conditions. Although the three studies add robust evidence for the unexpected findings of De Houwer et al. (2019), it remains unclear what might explain the inconsistency with previous results on instruction-based EC (De Houwer, 2006; Gast & De Houwer, 2013).

In a final effort to try and explain these findings, we used an approach in which we wanted to first establish the typical instruction-based EC effect with the current stimuli and then move more closely to the current procedure until we could identify the procedural aspect that eliminates the typical effect. To this end, we conducted a number of pilot studies (materials, data and analyses code for the pilot studies are available at osf.io/f6u2n/). In a first pilot study, we simply instructed participants that one CS nonword would always be followed by an unpleasant sound, while the other CS nonword would be followed by a pleasant sound. Importantly, in this study, instructions made the valence of the USs explicit (i.e., a sound was labelled as pleasant or unpleasant), and participants were never exposed to the USs. We adopted this procedure because it mirrored what has been done in previous studies testing the impact of EC-instructions on IAT scores (e.g., De Houwer, 2006). As expected, a sample of 39 Prolific Academic participants (26 women, Mage = 39.03, SD = 13.21) exhibited a preference for the CS_pos over CS_neg on the IAT, M = -.17, SD = .43, t(38) = -2.50, p = .017, d = .40.

The first pilot study showed the expected effect of instructed CS-US pairings on the IAT score. However, there were multiple procedural differences between this pilot study and the previous experiments that might have accounted for the discrepancy in the results. Among such differences, one refers to the contingency between the CSs and the USs. In fact, whereas in the first pilot study instructions indicated perfect contingencies (i.e., CSs always followed by either a negative or a positive sound), in the previous experiment involving both a positive and a negative US (Experiment 3) the contingency between the CS_neg and the US was either high (8/10 trials) or low (8/16 trials), but never perfect. Therefore, in a second pilot study, 20 Prolific Academic participants (10 women, Mage = 32.80, SD = 14.17) underwent the exact same procedure but this time both CS_neg and CS_pos were instructed to be either followed by the unpleasant (CS_neg) or by the pleasant sound (CS_pos) in only 8 out of 10 trials. This contingency manipulation did not appear to be the critical factor producing the dissociation as participants again revealed a similar sized preference for the CS_pos over the CS_neg on the IAT (M = -.21, SD = .43), t(19) = -1.96, p = .065, d = .44. Data from the two pilot studies were combined (N = 59) to estimate evidence for the overall impact of the instruction-based conditioning procedure on self-report and IAT scores, also with Bayes factors. A paired-sample t-test revealed a preference for CS_pos on self-report ratings, t(58) = -3.54, p < .001, d = .46, BF₁₀ = 32.73. A one-sample t-test revealed a preference for CS_pos on IAT scores, consistent with self-report liking, but contrasting with the results of Experiments 1-3, t(58) = -3.20, p = .002, d = .42, BF₁₀ = 13.34.

Having ruled-out an explanation in terms of CS-US contingency, we turned our attention to another key difference between the pilot studies and prior experiments, that is, the way the USs were described in the instructions. In Experiments 1-3, participants were first exposed to one sound (or two sounds, Experiment 3) and then received instructions that referred to the presence or absence of one sound after the presence of either CS. However, those instructions did not refer to the valence of the sound(s), which was supposed to have been learned in the initial exposure of the sound(s). Instead, there was no exposure to the sounds in the pilot studies and participants were informed about the valence of the US via instructions. A third pilot study changed this aspect of the procedure. Thirty-three participants (21 women, Mage = 35.73, SD = 12.54) were exposed to the same USs used in Experiment 3 at the very beginning of the study and then saw the contingency instructions. Importantly, and in line with the procedure used in Experiment 3, these instructions now referred to the USs as either a “first” or a “second” sound rather than “positive” or “negative” sounds. Crucially, this time the IAT score did not indicate a preference for the CS_pos (M = -.01, SD = .54), t(32) = -.06, p = .95, d = .01.

Taken together, the results of these pilot studies support a new explanation for the unexpected effects of EC instructions on IAT scores that were observed in Experiments 1-3. In those studies, instructions about CS-US relations did not specify the valence of the US, whereas instructions in studies that did find typical instruction effects on IAT scores always informed participants about the valence of the stimuli that would be paired with the US (e.g., De Houwer, 2006; Gregg et al., 2006; Kurdi & Banaji, 2017). For instance, De Houwer (2006) instructed participants that either a “positive” or a “negative” stimulus would follow the onset of the target non-words and Kurdi & Banaji (2017) told participants that members of two alternative social groups were about to be paired with “positive” or “negative” things.⁵ In studies that use stimuli for which valence is very salient (e.g., images of death people), stimulus presentation may evoke labeling in valenced terms and presenting valenced labels in the instructions might not be crucial. However, as suggested by previous inconsistencies in finding effects on the IAT following conditioning with fearful stimuli (e.g., Boschen et al., 2007), reference to the valence of the US stimuli might be very important to qualify the nature of this type of stimuli. Without such reference, fearful stimuli might not be categorized by participants in negative terms (e.g., the “unpleasant” stimulus) but in neutral terms or even positive terms (e.g., the “crucial”, “exciting”, or “important” stimulus). If we assume that such labelling of the USs also influences the way in which the CSs are labelled during the IAT (e.g., “nonword that was paired with the unpleasant sound” vs. “nonword that was paired with the crucial sound”) this could produce an impact of US labelling on IAT performance such that the presence or absence of a valenced label determines whether the IAT score indicates more positive or more negative evaluation of the US.

For instance, if participants in our main experiments labelled CS_neg as “the nonword that was paired with the unpleasant sound”, this might have facilitated responding to that CS in the same way as responding to unpleasant attribute stimuli, leading to IAT scores indicative of a negative evaluation of the CS_neg. In contrast, if participants labelled CS_neg in a different way, such as “the nonword that was paired with the crucial sound”, this might have facilitated responding to the CS_neg with the same key as responding to pleasant attribute stimuli, leading to IAT scores indicative of a positive evaluation of the CS_neg. Whereas labelling CS_neg as “the nonword that was paired with the crucial sound” seems likely when the unpleasant sound is the only US told to be paired with either CS (Experiment 1-2), it might be less likely when both a pleasant and an unpleasant US are used (Experiment 3). However, a fearful sound is not only more negative, but also more arousing than pleasant sound. Past research has shown that the conditioning of affective responses can occur simultaneously along one evaluative dimension (i.e., valence) and along the dimension of arousal (Gawronski & Mitchell, 2014). Therefore, it is still possible that CS_neg was conceived as “the nonword that was paired with the arousing sound”. Although these considerations are somewhat speculative at this stage, they are in line with the fact that IAT performance depends heavily on the nature of the category labels that are used in the IAT, with identical stimuli producing different effects depending on the labels that are used to categorize them. For instance, in Mitchell et al. (2003) the automatic evaluations of the very same black vs. white social stimuli changed based on the reference category used to sort stimuli throughout the IAT (e.g., the name “Michael Jordan” to be categorized as either a Black person or an athlete). Whereas differences in automatic evaluations observed in that study emerged from a direct manipulation of the category labels, here we propose that even in the absence of any direct manipulation, participants might spontaneously use specific labels when categorizing target stimuli during the IAT, and that such a labelling might ultimately affect their automatic evaluations.

To test the idea that the unexpected IAT effect depends on whether instructions inform participants about the valence of the US, we conducted a well-powered fourth experiment. In this study, we first exposed all participants to both the sounds, and then we directly compared a condition in which instructions overtly referred to the valence of US sounds with effects of an alternative procedure in which instructions did not refer to their valence. This experiment was tightly matched to the procedure of Experiment 3 in which participants were exposed to two sounds, were given instructions that one CS would be followed by the first sound 8/10 times and another CS would be always followed by the second sound, and received a typical IAT. Importantly, however, for one group of participants the instructions referred to ‘an unpleasant sound’ and ‘a pleasant sound’. Therefore, participants in both the experimental and the control condition were pre-exposed to the pleasant and the unpleasant sound, but only for the former the valence of such sound was explicit in the contingency instructions.

Experiment 4 was run as a registered report. The design and analysis and sampling plan were preregistered at OSF and approved at Collabra:Psychology (https://osf.io/5necg). There were no deviations to report. All data and analysis code are available in the OSF repository at osf.io/8kguz/.

Experiment 4 had a 2 (CS type: CS_neg vs. CS_pos) x 2 (Instructions type: explicit reference vs. no reference to US valence) mixed design, with the first factor manipulated within and the second between participants. We also counterbalanced across participants (i) the identity of the CS to be related with either US, and (ii) the order of presentation of the two CSs in the evaluative ratings and the order in which the two critical blocks of the IAT were administered (the two factors will be nested into a single one).

As in the previous experiments, participants were recruited via Prolific Academic. In line with our sampling plan, we stopped data collection when 200 participants completed the entire session (84 females, 79 males, 37 missing information, M_age = 33.13, SD_age = 12.76). Considering our inclusion criteria, we expected a screened-out rate of approximately 10% of the entire sample so this sample size would allow us to have a good power (0.95) to observe a moderate difference between the two conditions (d = 0.50) in a one-tailed t-test at alpha = 0.05 (required N=176)⁶.

We adopted the same exclusion criteria as in the previous and excluded the data of participants who (1) failed to complete the whole session or showed incomplete data (4 participants), (2) did not accurately describe the sound or indicated that they could not hear it (1 participant), (3) had error rates above 30% across the IAT or above 40% for any one of the four critical IAT blocks (4 participants), or (4) completed more than 10% of IAT trials faster than 400ms (2 participants). Analyses were performed on the data of the remaining 190 participants.

The stimuli, measures, and instructions were identical to Experiment 3 with few exceptions. After participants provided informed consent and answer demographic questions, they listened to the two auditory stimuli (US+ and US-) and briefly described them.

Next, participants were assigned to two instructions conditions. In the explicit instructions condition, participants were informed that in 8 out of 10 trials one CS was followed by the unpleasant sound they listened to, and that the other CS was always followed by the pleasant sound they listened to. For participants in the control condition, instructions referred to “the first[second] sound”, without any explicit reference to their pleasantness. Participants then underwent the same retention test administered in Experiment 3. In this test, they were asked to indicate the correct number of times in which, based on instructions, the relevant US would occur upon presentation of each CS. Importantly, neither the questions nor the response options presented in the control condition made explicit reference to the pleasantness of the USs, instead they were simply defined as the “first” and the “second” sound. In case of incorrect responding, participants were exposed to the initial instructions again. Next, participants completed the IAT, followed by the same CSs rating phase used in the previous studies, in which both self-reported liking towards the CSs (i.e., “How much do you like this stimulus?”; response scale: -10 = very negative; 0 = neutral; +10 = very positive) and US expectancy (i.e., No reference to US valence: “To what extent do you expect that the first [second] sound you listened to will occur?”; Explicit reference to US valence: “To what extent do you expect that that the pleasant [unpleasant]sound you listened to will occur?”; response scale: 1 = not at all, 5 = uncertain, 9 = certainly) were measured. Participants answered exploratory questions, including the same questions about clarity and believability as in our previous experiments. We also measured self-report evaluation of both the sounds (i.e., “How pleasant was the first(second) sound you listened to at the beginning of the study?”) on a 11-point scale (-5 = very unpleasant; +5 = very pleasant). The same memory question administered in the retention test was repeated at the end of the session.

We expected to find a difference in the EC effect on IAT scores as a function of the type of instructions. Namely, the IAT score should reveal a preference for CS_pos over CS_neg for participants in the explicit valence instructions condition. Participants assigned to the control condition should show a preference for the CS_neg, as was the case in Experiment 3. We also expected to replicate prior findings on self-reported liking: in both the instruction conditions, participants should exhibit a preference for CS_pos over CS_neg.

Self-report ratings. A paired-sample t-test showed that the CS_neg was evaluated more negatively (M = 0.20, SD = 3.95), than the CS_pos (M = 1.30, SD = 3.54), t(189) = -2.93, p = .002, d = .21. The mixed-ANOVA, with Instructions Type (explicit reference vs. no reference to US valence) as between-subjects factor and CS Type (CS_neg vs. CS_pos) as within-subjects factor, showed a main effect of CS Type, F(1,188) = 8.35, p = .004, $ηp2$ = .04, and a significant interaction, F(1,188) = 5.86, p = .02, $ηp2$ = .03. In accordance with our hypotheses, a preference for CS_pos (M = 1.51, SD = 3.81) over CS_neg (M = -0.45, SD = 3.97) was evident when instructions were explicit about the valence of the sounds, t(97) = -3.25, p = .002, d = 0.33, BF₁₀ = 38.96. However, in contrast with our hypotheses and with the results of Experiments 1-3, we did not observe this preference for CS_pos (M = 1.08, SD = 3.24) over CS_neg (M = 0.90, SD = 3.82) when instructions did not refer to the valence of the US, t(91) = -0.42, p = .67, d = 0.04, BF₁₀ = 0.48.

IAT score. Split-half reliability was r(188) = .74. The overall IAT score revealed no preference for either CS (M = 0.01, SD = 0.50), t(189) = 0.15, p = .88, d = 0.01. The crucial, expected, difference between the two IAT scores was significant in the planned one-tailed t-test, t(188) = 1.72, p = .043, d = 0.25, BF₁₀ = 1.49. In contrast to our hypothesis and the results of Experiment 3, we did not observe a significantly positive IAT score (i.e., preference for CS_neg over CS_pos) in the control condition (i.e., when instructions did not mention the valence of the USs, M = 0.07, SD = .46), t(91) = 1.44, p = .076, d = 0.15, BF₁₀ = 1.34. We also did not observe a significantly negative IAT score (i.e., preference for CS_pos over CS_neg) in the explicit instructions condition (i.e., when instruction did mention the valence of the USs, M = -0.05, SD = .53), t(97) = -1.02, p = .16, d = 0.10, BF₁₀ = 0.81. The IAT score and the self-report liking ratings were positively correlated, r(188) = .19, p = .008.

US expectancy. Expectancy of the unpleasant US was higher for CS_neg (M = 6.23, SD = 3.32), as compared to CS_pos (M = 3.07, SD = 2.59), t(189) = 11.78, p < .001, d = .85. Similarly, expectancy of the pleasant US was higher for CS_pos (M = 6.61, SD = 2.72), as compared to CS_neg (M = 2.82, SD = 2.30), t(189) = -13.06, p < .001, d = .95. Two mixed ANOVAS that included the factors CS Type and Instruction Type showed no significant interaction effect on either US negative expectancy, F(1,188) = .06, p = .80, or US positive expectancy, F(1,188) = .19, p = .66.

Comparison between self-report ratings and IAT score. A mixed ANOVA with Measurement type (IAT vs. self-report) as within-subjects factor and IAT order, CS type and Instructions type as fixed factors showed a significant effect of Measurement Type, F(1,182) = 6.23, p = .01, $ηp2$ = .03, indicating stronger effect when preferences were measured via self-report. We also observed a main effect of Instructions type, F(1,182) = 8.21, p = .005, $ηp2$ = .01, but no interaction effect of Instructions Type and Measurement, F(1,182) = .71, p = .40, $ηp2$ = .004.

US evaluation. The evaluation of the unpleasant US was more negative (M = -3.72, SD = 1.86) than that of the pleasant US (M = 2.19, SD = 2.24), t(189) = -25.55, p < .001, d = 1.85. When US evaluations were submitted to a mixed ANOVA with Instructions type as fixed factor, we did not find a significant interaction, F(1,188) = 1.72, p = .19. Instruction Type also did not show any significant impact on evaluation ratings of either US when considered in isolation (unpleasant US: F(1,188) = .23, p = .63; pleasant US: F(1,188) = 2.14, p = .15).

Correlation analyses. For both the self-report and the IAT scores, we inspected the correlations with the other relevant scores (i.e., expectancy of the unpleasant US, expectancy of the pleasant US, US evaluation). A differential liking score (CS_pos-CS_neg) correlated negatively with the differential expectancy of the unpleasant US (CS_pos-CS_neg), r(188) = -.30, p < .001, and correlated positively with the differential expectancy of the pleasant US (CS_pos-CS_neg), r(188) = .30, p < .001, and with differential US evaluation ratings (US_pos-US_neg), r(188) = .23, p = .001.

We did not observe a significant correlation between IAT scores and ratings of the expectancy of the unpleasant US, r(188) = .11, p = .12, the expectancy of the pleasant US, r(188) = -.08, p = .243, and US evaluation, r(188) = -.02, p = .793.

Meta-analysis. Findings from Experiment 4 diverged from previous experiments with respect to the impact of EC instructions (in the condition that did not refer to the valence of the USs) on self-reported ratings. Therefore, we performed a meta-analysis of the results of the four experiments that examined the impact of EC instructions when instructions did not refer to the valence of the sound(s). Data from the four experiments were combined, with the exception of participants that in Experiment 4 were assigned to the condition in which instructions referred to the pleasantness of the USs. This led to a final sample of 742 participants. Self-report ratings showed a significant preference for the CS_pos over the CS_neg,t(741) = -4.84, p < .001, d = 0.18, BF₁₀ > 1000. The IAT score revealed the opposite pattern, with a significant preference for the CS_neg, t(741) = 4.93, p < .001, d = 0.18, BF₁₀ > 1000. Neither Self-report ratings nor IAT scores were qualified by type of experiment, ps > .39.

The Implicit Association Test (IAT) is typically used as a measure of automatic evaluations (i.e., evaluations that occur under automaticity conditions). Research has shown that evaluative conditioning procedures (i.e., the pairing of a neutral stimulus, US, with a valenced stimulus, CS) typically lead to changes in IAT scores reflecting a preference for USs paired with positive CSs over USs paired with negative CSs (Hofmann et al., 2010). Similar effects have been observed when participants are merely instructed about upcoming CS-US pairings (e.g., De Houwer, 2006; Gast & De Houwer, 2013; Kurdi & Banaji, 2017).

In a recent study, De Houwer et al. (2019) found an unexpected IAT effect of EC instructions that referred to upcoming pairings of an unpleasant sound (US) with non-words CSs. Specifically, IAT scores reflected a preference for a CS told to be followed by the unpleasant US (CS_neg) over a CS told not to be followed by the unpleasant US (CS_noUS). In contrast, self-report ratings showed the expected effect of instructions, with more negative evaluations of the CS_neg than the CS_noUS.

The present research set out to explore this unexpected pattern of results. Experiment 1 replicated the unexpected effect of instructions on IAT scores when the IAT immediately followed the EC instructions. This ruled-out an alternative explanation of the IAT effect observed in De Houwer et al. (2019) in terms of actual pairings that occurred after the EC instructions and before the IAT. Experiments 2 and 3 further replicated the IAT effect and did not reveal differential effects on different types of IAT measures such as the personalized IAT (Experiment 2), and the recoding-free IAT (Experiment 3). Experiment 3 also did not show an effect of whether instructions referred to a contrast CS that was never followed by the aversive sound (CS_noUS) or that was always followed by a pleasant sound (CS_pos). Thus, the three experiments suggested that the unexpected effect on the IAT was robust and did not depend on manipulations of the IAT structure or instructions about the contrast CS.

Building on the results of three pilot studies, we hypothesized that the IAT effect might reflect the labelling of the CSs, such that participants readily refer to the CS_neg in positive terms (e.g., as the stimulus that goes together with the “crucial” or “arousing” sound), which facilitates categorization of the CS_neg in positive terms in the IAT. From this perspective, instructions that explicitly refer to the valence of the US might facilitate labelling the CS_neg in negative terms (e.g., as the CS that goes together with the negative sound), which might eliminate the unexpected IAT effect. Although we did not observe a significant typical IAT effect in the explicit instructions condition of Experiment 4, the unexpected effect found in prior studies was absent. Moreover, the type of instructions had a significant effect on the IAT effect, although this effect was weak and significant only in a (planned) one-sample t-test. In sum, our studies shed new light on the conditions under which the unexpected IAT effect does and does not arise.

The fact that instructions about the US were crucial in obtaining the unexpected IAT effect fits well with prior research on the malleability of IAT scores. These prior studies showed that labelling of IAT categories is an important determinant of IAT scores. For instance, whether Michael Jordan is evaluated as positive or negative in an IAT depends on whether that person is labelled as being a Black person or as being an athlete (Mitchell et al., 2003). Likewise, in our studies, IAT scores might have depended on whether participants labelled the CS_neg in negative terms (e.g., as the stimulus that co-occurs with the negative sound) or in positive terms (e.g., as the stimulus that co-occurs with the “crucial” or “arousing” sound). This labelling of the CS_neg might in its turn have depended on whether and how the US was labelled by the experimenter (e.g., as a “negative” sound). Note that this would be a very subtle malleability effect. Whereas previous studies manipulated the way in which an IAT category was labelled (e.g., label Michael Jordan as a Black person or as an athlete), we only varied the way in which stimuli related to an IAT category were labelled (i.e., whether the sound that would go together with the CS_neg was described as “negative”).

Although our results make sense from the perspective of the malleability of IAT scores, it is important to realize that they provide only weak evidence for such malleability. The only direct evidence comes from the significant effect of instruction type on the IAT effects in Experiment 4. However, this effect was small and Bayes Factors were inconclusive. Instruction type might have had only a small effect in Experiment 4 because instructions were presented only after participants had listened to the two sounds. The direct experience with the USs at the beginning of the session might have affected CSs labelling and could have counteracted the effect of instructions. Future studies might thus replicate Experiment 4 while presenting the US instructions before the USs are presented or might eliminate the presentation of the USs altogether. Also note that whereas CS_neg was told to be followed by the unpleasant US “in eight trials out of ten trials”, CS_pos was told to be “always followed the pleasant sound”. This inconsistency might have somehow encouraged participants to ascribe greater salience to the aversive US (e.g., “The US that is more exceptional, or more difficult to predict given the relevant CS”) irrespective of the explicit instructions about the USs. Thus, a direct follow-up study could compare IAT effects in a condition in which both USs are matched in terms of statistical contingency.

It would also be interesting to examine the role of US labelling in paradigms that produce strong typical IAT effects of EC instructions (i.e., liking of CS_pos over CS_neg). In the present studies, we started from a procedure that produces unexpected IAT effects of EC instructions and tried to eliminate this unexpected effect. In a similar vein, future studies could examine whether typical IAT effects of EC instructions can be reduced or even eliminated when USs are not labelled as positive or negative stimuli. It is indeed striking to see that most if not all studies that documented effects of EC instructions on IAT scores used instructions in which USs were labelled explicitly as positive or negative (but see Footnote 5).

If the explicit labelling of the USs is found to systematically influence (typical and unexpected) effects of EC instructions on IAT scores, the next step would be to examine whether similar US labelling effects can be found also on other indices of automatic evaluations, such as evaluative priming effects (e.g., Fazio et al., 1986). If this is the case, it would strengthen the conclusion that US labelling influences automatic evaluation of the CSs. If the effects are specific to the IAT task, it might point at non-evaluative influences on IAT performance, more specifically, the impact of US labels on the strategies that participants use to optimize IAT performance (e.g., Meissner et al., 2019; Rothermund & Wentura, 2004). Because it is currently not clear whether the IAT effects that we observed reflect automatic evaluation or non-evaluative influences, we refrain from drawing conclusions about the mental processes involved in EC and automatic stimulus evaluation.

In sum, across four experiments we found that instructions about upcoming pairings of neutral stimuli with an unpleasant (fearful sound) US can lead to an unexpected (reversed) effect on the IAT, confirming previous findings by De Houwer et al. (2019). Experiments 1-3 ruled-out several potential explanations for this effect (e.g., related to the moment of IAT administration, IAT type, or the instructions about the control stimulus). In Experiment 4, we did succeed in eliminating the unexpected IAT effect by explicitly labelling the aversive US as a negative stimulus. Although this finding is in line with previous studies showing the malleability of IAT effects, future studies are required to verify whether US labelling significantly moderates unexpected and standard IAT effects of EC instructions and whether these US labelling effects also arise for other measures of automatic evaluation.

SM, PVD and JDH developed the research design, SM prepared the experimental materials, SM and PVD performed the analysis, SM and PVD drafted the initial manuscript, SM, PVD and JDH revised the manuscript.

The authors declare that they have no conflict of interest.

For all the experiments, we pre-registered the materials, sampling plan, exclusion rules, analysis plans, and hypotheses on the Open Science Framework. We report all data exclusions, manipulations, measures, and how we determined our sample sizes. The materials, data and analysis scripts of all experiments are available at osf.io/f6u2n/.