A Comparative Investigation of Indirect Measures of Sexual Attraction

We first conducted two preliminary studies (reported in the Supplemental Online Materials [SOM]) to select suitable indirect measures of sexual attraction – some have been used before and some are novel. The indirect measures that we tested included the gender-comparative IAT and several single-gender measures – the Single-Target IAT (ST-IAT; Karpinski & Steinman, 2006; Wigboldus et al., 2004), and two versions of single-gender IATs, each with a different stimuli set (see details below). As part of our preliminary studies, we also compared two sets of attribute labels for the IATs: Sexually Attractive vs. Not Sexually Attractive, and their personalized version (as recommended by Olson & Fazio, 2004): I Am Sexually Attracted vs. I Am Not Sexually Attracted.

These studies resulted in three notable findings that impacted our choice of stimuli and measures for the main study: First, because the ST-IAT showed poor validity we decided not to test it further. Second, we decided to use the attribute labels I Am Sexually Attracted and I Am Not Sexually Attracted because they provided better validity than the pair Sexually Attractive vs. Not Sexually Attractive. Third, we decided to test two single-gender versions of the IAT that showed reasonable validity in the preliminary studies – a version similar to a single-gender IAT reported in previous research (entitled the Men/Not-Men IAT and the Woman/Not-Woman IAT in our study; Snowden et al., 2020, 2024; Snowden & Gray, 2013) and a novel version we created (the Men/People IAT and the Women/People IAT). In the main study, we added to these two single-gender versions of the IAT a few additional measures. These included the gender-comparative IAT, which was tested in one of the preliminary studies, and a few measures that were not included in the preliminary studies: the gender-comparative Sequential Priming Task (SPT), the single-gender Questionnaire-Based IAT (qIAT), and the Single Association Test (SAT). Thus, the main study included two gender-comparative and four single-gender indirect measures.

The first gender-comparative measure was the IAT described above, in which participants categorized stimuli to Men and Women target categories, and to I Am Sexually Attracted and I Am Not Sexually Attracted attribute categories. The score of this IAT reflects how much participants associate images of women compared to images of men with words that are related to being sexually attracted, and so a higher score indicates greater sexual attraction to women compared to men. The second measure was based on a Sequential Priming Task (SPT) in which participants categorized stimuli to I Am Sexually Attracted and I Am Not Sexually Attracted attribute categories, primed with Men or Women images. Faster reaction times when sexual attraction words are primed with images of Women, compared to reaction times when they are primed with images of Men, is interpreted as a greater association between women and sexual attraction, compared to the association between men and sexual attraction. These measures were chosen as part of the current investigation because they are two of the most widely used indirect measures in social cognition research, and because they have been previously used to assess sexual attraction (e.g., Snowden et al., 2008, 2020; Snowden & Gray, 2013).

As for single-gender measures, we included two types of IAT to explore the possibility that the IAT could be reliably used as a single-gender measure, and to determine which version may be superior to the other. The first IAT was based on an existing IAT (Snowden et al., 2020, 2024; Snowden & Gray, 2013), with the target categories Men and Not-Men or Women and Not-Women. As mentioned above, this version was created by Snowden and Gray (2013) and contrasts images from one gender group (e.g., Men) with neutral images of images of scenery and household items (e.g., Not-Men, in our example), rather than with images of another gender group (e.g., Women), making it possible to compute a score of sexual attraction toward a specific gender. However, previous research suggested that using this IAT, the association between sexual attraction and men/women may be more common than associating sexual attraction with nature scenery, regardless of one’s self-reported sexual orientation (Snowden et al., 2020, 2024; Snowden & Gray, 2013). Due to this limitation, in the second single-gender IAT we contrasted a gender category with a non-gendered category consisting of images of people (men and women). Thus, the target categories in this IAT were Men and People or Women and People, and the items of the contrast category were images of several people, always including at least one man and one woman. Although this approach may still suffer from some confounds (e.g., sexual attraction is typically associated with a specific gender rather than with no particular gender), it may still outperform the previous version. In all the IATs, we used the attribute categories I Am Sexually Attracted and I Am Not Sexually Attracted. The items for these categories were words selected by each participant for each attribute category, from a list of candidate words.

The third single-gender measure was a novel measure – a Single Association Test (SAT) – that we developed based on the BIAT (Bar-Anan & Nosek, 2014; Sriram & Greenwald, 2009). The SAT was developed to measure the association between one gender group (e.g., Men) and one attribute (e.g., I Am Sexually Attracted) without a clear comparison category (e.g., People) or a comparison attribute (e.g., I Am Not Sexually Attracted). In the SAT, we compared blocks in which the two concepts (e.g., Men and I Am Sexually Attracted) are the focal categories to blocks in which only one of the categories is the single focal category (e.g., I Am Sexually Attracted). Although it is clear that people would be faster in blocks with a single focal category than in blocks with two focal categories, we expected that advantage to be smaller, the stronger the two categories are linked in one’s memory.

The fourth measure was an adaptation of the Questionnaire-Based IAT (qIAT; Yovel & Friedman, 2013), in which the task’s stimuli are statements taken from self-reported measures. The two attribute categories include True statements (e.g., I am looking at a computer screen) and False statements (e.g., I am climbing a steep mountain). We adapted the qIAT to measure sexual attraction to men and to women, separately, by transforming questions from our sexuality questionnaire into statements. In each qIAT, one target category was for statements describing sexual attraction to one gender (i.e., to either men or women; e.g., I often feel attraction towards men), and the other target category was for statements describing lack of attraction to that gender (e.g., I rarely have sexual fantasies about men). The score of each qIAT reflects how much participants associate statements that describe them as attracted to one gender with the concept True, compared to statements that describe them as unattracted to that gender. The qIAT has shown good validity as an indirect measure of several personality constructs, including self-esteem (Yovel et al., 2022), extraversion (Friedman et al., 2022; Yovel & Friedman, 2013) and conscientiousness (Friedman et al., 2022).

In the present study, we tested measures that assessed sexual attraction to the only two normative categories of gender – women and men – out of the many gender categories that individuals currently use to self-identify (e.g., American Psychological Association, 2015; Matsuno & Budge, 2017; Richards et al., 2016). In addition, the indirect measures that used pictures of men and women as target stimuli (i.e., all indirect measures except the qIAT), included mainly images of White, young and thin individuals, with appearances that adhere to Western gender presentation and beauty standards. This methodological choice, which does not capture the full spectrum of human sexuality and gender diversity, inherently limits the generalizability of our findings.

Test-Retest Reliability. We computed the correlation between two administrations of the same measure in different times, for returning participants who happened to complete the same measure twice. Because most retest data were collected within the same day, this criterion relates to the reliability of the indirect measures irrespective of the stability of the sexual attraction to women and men over longer periods of time.

Internal Consistency. We computed internal consistency from different parts of each task (Bar-Anan & Nosek, 2014).

Convergent Validity. An important quality of a good measure is how strongly it is associated with measures of closely related constructs. Here we used several ways to assess the convergent validity of the different indirect measures.

Known Groups Differences. A valid indirect measure should reveal differences between groups of individuals who are expected to differ in the construct of interest – in the present context, groups of individuals with different self-reported sexual orientation, or of men and women with the same self-reported sexual orientation. Larger differences would suggest better validity.

Correlations with Other Measures of the Same Construct. We computed the correlation between the measure’s score and other indirect and direct measures of sexual attraction that assessed sexual attraction in a similar relative way. We also computed single-gender attraction scores for each gender-comparative measure from a subset of trials that included only stimuli of that gender. A stronger correlation of a measure with other measures of the same construct would suggest better convergent validity.

Correlations between indirect and direct measures of the same construct are common for evaluating the validity of implicit measures (Bar-Anan & Nosek, 2014; Bluemke & Friese, 2008; Greenwald et al., 2003), under the assumption that even if indirect measures are more sensitive to unintentional processes than self-report measures, the constructs tapped by direct and indirect measures are not completely unrelated. Indeed, previous research has found positive correlations between indirect and direct measures of different attitudes (e.g., Bar-Anan & Nosek, 2014; Nosek, 2005), including sexual attraction (e.g., Bartels et al., 2017; Imhoff et al., 2011; Snowden et al., 2020).

One problem with using correlations between the indirect and direct sexuality measures to validate the former is that such correlations may reflect an unwanted sensitivity of the indirect measures to deliberate processes. Therefore, we conducted exploratory analyses that assessed whether the indirect measures capture unintentional processes that bypass self-presentation efforts. To do so, we used participants’ political identity and compared the relations between indirect and direct measures of same-gender sexual attraction in Liberal versus Conservative individuals, who self-identified as exclusively straight. The reasoning underlying this analysis is based on the finding that negative stances toward homosexuality are more prevalent among Conservatives compared to Liberals (e.g., Herek, 2000; Monto & Supinski, 2014; Prusaczyk & Hodson, 2020; Wood & Bartkowski, 2004; Yang, 1998) and that identifying as anything but exclusively straight means acknowledging some degree of non-heterosexual feelings (e.g., Savin-Williams & Vrangalova, 2013; Vrangalova & Savin-Williams, 2012). Therefore, while same-gender sexual attraction has been reported by self-identified exclusively straight men and women (e.g., Savin-Williams & Vrangalova, 2013; Vrangalova & Savin-Williams, 2010), such feelings may conflict with their self-definition, and this conflict may be greater for individuals who identify as Conservative compared to Liberals, leading to higher self-presentational concerns regarding same-gender sexual attraction in Conservatives compared with Liberals. Therefore, if the indirect measures are less controlled than the direct measures, we expect to see stronger relations between the direct and indirect measures of same-gender attraction among Liberals than among Conservatives. We also conducted similar analyses for the relations between indirectly- and directly-measured “other”-gender attraction. Although social sanctions may exist against individuals who report low levels of sexual attraction (e.g., MacInnis & Hodson, 2012; Thorpe & Arbeau, 2020), we were not aware of studies reporting different attitudes of Conservatives and Liberals towards such individuals, and therefore had no reason to expect that Conservatives and Liberals will differ in the relations between direct and indirect measures of “other”-gender sexual attraction.

Discriminant Validity. Building on evidence from direct measures that sexual attraction is comprised of two separate (but related) constructs (sexual attraction to women and to men), we expected negative correlations between the scores of single-gender indirect measures (and of single-gender attraction scores computed from the gender-comparative measures) and directly-measured sexual attraction to the “other” gender group. Yet, because attraction to women and attraction to men are supposed to be related but separate constructs, larger differences between the absolute value of the (positive) correlation of each single-gender score with directly-measured attraction toward the same-gender, and the absolute value of the (negative) correlation of that score with the directly-measured attraction toward the “other” gender would reflect better discriminant validity.

The study was a part of a larger study in the Project Implicit participant pool (implicit.harvard.edu; Nosek, 2005), assessing sexuality, gender identity and their relations, both directly and indirectly. In each session, participants completed one to four indirect measures of sexual attraction or gender identity, followed by corresponding direct measures of the relevant constructs. We omitted study sessions that did not include sexual attraction measures, and we did not analyze the gender identity measures for the present research.

Participants were adult (at least 18 years old) volunteers from the Project Implicit participant pool who were randomly assigned to this study from the study pool. We planned a sample size that would provide 95% power to detect a correlation of r = .20 between self-reported attraction and each indirect measure’s score. However, because we collected data for a long period of time as part of a larger study for purposes unrelated to the current report, we obtained a much larger sample than needed. Importantly, the stopping criteria for data collection did not depend on the results and their significance.

We excluded participants with missing information regarding their age, current gender, or sexual orientation. Table 1 displays descriptive statistics by current gender and sexual orientation groups. The number of participants who identified as transmen, transwomen, transgenders, genderqueers, or ‘other’ was too small to allow meaningful statistical analyses. Therefore, the analyses reported in the present study included only cisgender men (individuals who reported being registered male at birth, raised as a boy, and currently identify as a man), or cisgender women (individuals who reported being registered female at birth, raised as a girl, and currently identify as a woman). The number of participants who identified their sexual orientation as ‘asexual’ or ‘other’ was also too small to be included in the present study. The number of participants who identified as bisexual or pansexual was larger, but still too small to analyze separately, so these participants were grouped into a single sexuality group titled ‘bisexual/pansexual’. Thus, the study included a sample of 50,535 cisgender men and women from five sexuality groups: exclusively straight, mostly straight, bisexual/pansexual, mostly gay/lesbian, and exclusively gay/lesbian. We allowed being randomly assigned to this study more than once, resulting with multiple study sessions for 10,177 of the participants (all together, the study included 68,646 sessions). Of the participants in the final sample, 73% stated they were US citizens. The main reported racial categories were White (62.3%), Hispanic (9.8%), Asian (9.7%), Black or African American (9.6%), and multiracial (5.4%).

We excluded data for indirect measures in which the participant’s performance was poor, based on each measure’s exclusion norms (see SOM for each measure). Table 2 displays how many participants 1) started the study, 2) completed the study, and 3) were included after we implemented each exclusion criterion.

At the beginning of the study, we informed participants that we would ask them questions about their sexuality, and that they would see images of individuals in minimal clothing (when such images were included). Following this information, participants consented or left. At the beginning of the study, in conditions that included word stimuli in the attribute categories, participants chose from a list of specific words, the words that they considered most relevant to each attribute category (De Houwer et al., 2004). For the category I Am Sexually Attracted, participants chose four words from the following list: Attraction, Excitement, Desire, Lust, Aroused, Stimulated, Passionate. For the category I Am Not Sexually Attracted, the list consisted of the words: Apathy, Indifferent, Neutral, Uninterested, Boredom, Trivial, Aloof. In all studies, participants completed the indirect measure(s) first, followed by the corresponding direct measures, a demographics questionnaire, and finally, we thanked and debriefed them.

The type and number of indirect measures participants completed varied between the randomly assigned conditions. In some conditions, participants completed one indirect measure that assessed sexual attraction to men, to women, or to women compared to men. In most conditions, participants completed two single-gender indirect measures – one that assessed sexual attraction to men, and the other toward women. In these instances, they completed the measures consecutively and in random order. Before the sexual attraction indirect measure(s), some participants completed indirect measure(s) of gender identity. After the indirect measure(s), participants completed direct measures of sexuality (those who also completed indirect measures of gender identity, completed also direct measures of gender identity). The ethics committee of Tel-Aviv University approved the study.

All procedures, stimuli and instructions appear in the SOM, and all materials, analysis codes and data are accessible at https://osf.io/juezp/.

We presented each indirect measure in the Introduction. Table 3 displays the outline of trials and blocks for each indirect measure. The full procedure of each indirect measure is detailed in in the SOM.

All self-report items appear in the SOM.

Self-Reported Sexuality. Similar to the sexual orientation questionnaire used by Jacobson & Joel (2018), participants reported the level of sexual attraction they experience towards each gender group, using a 6-point Likert scale that ranged from none to very high; how often they had erotic fantasies about each gender, using a 6-point Likert scale ranging from never to very often; and how often they had sexual encounters with each gender and romantic relationships with each gender, using a 5-point Likert scale, ranging from none to all. Additionally, we created a gender-comparative self-report item with a 5-point Likert scale that ranged from only attracted to men, not to women to only attracted to women, not to men, with equally attracted to men and women in the middle, and a sixth response option – not attracted to either, that was recoded to 3 (mid-scale).

Self-Reported Attractiveness Associations to Gender Categories. In two separate items, participants reported to what extent they associate men (or women) with sexually attractive as compared to not sexually attractive. These questions might have higher structural fit with the indirect measures because they were developed to capture mental associations. In two additional items, participants reported to what extent each gender group is associated with the concept sexually attractive or with not sexually attractive in their culture. All items used a 7-point Likert scale that ranged from with ‘sexually attractive’ extremely more than with ‘not sexually attractive’ to with ‘not sexually attractive’ extremely more than with ‘sexually attractive’. These items helped test the sensitivity of the indirect measures to people’s knowledge about the typical association in their culture between sexual attractiveness and each gender (Olson & Fazio, 2004).

Demographic Information. As part of their registration to Project Implicit’s website, participants were invited to provide general demographic information, including birth month and year, ethnicity, race, occupation, place of residence, religious affiliation, and level of religiosity, as well as their political identity using a 7-point Likert scale ranging from strongly conservative to strongly liberal. In our study, we asked participants to indicate their sex at birth (male / female / other), the gender they were raised as (boy / girl / other), their current gender (man / woman / transman / transwoman / transgender / genderqueer / other), and sexual orientation (exclusively straight / mostly straight / bisexual / mostly gay/lesbian / exclusively gay/lesbian / pansexual / asexual / other). Whenever an other response was chosen, participants were given space to specify. Following Rotondi et al. (2011), participants who reported identifying as trans or queer (transman / transwoman / transgender / genderqueer), or whose current gender was different than the gender they were raised in, were also asked to indicate whether they live in their felt gender (full time / part-time / no) and how many years they have done so.

For returning participants who completed the same direct or indirect measure more than once, our analyses focused on their first score for each sexuality measure.

We processed and scored each indirect measure using the most common approach in the literature for that measure or, in the case of novel measures, for the most similar measure (see the SOM for details). The category titles and stimuli used in each indirect measure appear in the SOM. For the gender-comparative measures, a greater score was interpreted as stronger sexual attraction to women compared to men. For the single-gender measures, a greater score was interpreted as stronger sexual attraction to the assessed gender group (women or men).

Because the study includes many comparisons (between many measures in many criteria), each specific comparison does not provide strong statistical inference. Therefore, inference about the superiority or inferiority of one measure in comparison to others relied on consistent results throughout multiple different criteria, especially consistency in performance of each pair of measures (one for attraction toward women and one for attraction toward men) because these provide independent tests of each measurement method.

Test-Retest Reliability. For returning participants, we computed Pearson’s correlations between the indirect measure’s scores of the first and second sessions. Within each type of measure (e.g., single-gender measures toward women), we compared all test-retest correlations using Fisher’s z-test, to determine which measures were superior to others.

Internal Consistency. We computed a Cronbach’s alpha (Cronbach, 1951) for each indirect measure from three parcels. The 1^st, 2^nd, and 3^rd trial of each three consecutive trials were divided to the 1^st, 2^nd, and 3^rd parcel, respectively (Bar-Anan & Nosek, 2014). Due to the large sample size, any difference in internal consistency was significant (using the Feldt test; Feldt, 1969).

Known Group Differences. For each indirect measure, we calculated several effect sizes to determine to what extent the measure reflects known group differences. For all comparisons, participants who reported being exclusively or mostly straight, and participants who reported being exclusively or mostly gay/lesbian, were grouped to two separate groups – Straight individuals and Gay/lesbian individuals, respectively. Then, we computed effect sizes for the indirect measures’ scores for the following comparisons: 1. straight men versus straight women, 2. gay men versus lesbian women, 3. straight men versus gay men, and 4. straight women versus lesbian women. For all comparisons, we calculated Hedge’s g (Hedges & Olkin, 1985) and its confidence intervals, using the R package ‘effectsize’ (Ben-Shachar et al., 2020). We chose Hedge’s g because it corrects for the potential bias of small samples (such as those we used when comparing gay men and lesbian women), it is equivalent to Cohen’s d (Cohen, 2013) in sample sizes > 20, and is also suitable when comparing groups with unequal variances (Marfo & Okyere, 2019). To estimate the overall validity of each indirect measure, we conducted a meta-analysis of the effect sizes of all four comparisons for each indirect measure’s score. Using the R package ‘meta’ (Schwarzer, 2007), we used a random-effects model that assessed the weighted mean effect size for each indirect measure, based on the four effect sizes we computed. We interpreted non-overlapping confidence intervals of the known-groups effect sizes as significant differences between indirect measures.

Correlations with Indirectly-Measured Sexuality. We calculated Pearson’s correlations between the single-gender score of each indirect measure and the corresponding single-gender score of each one of all the other indirect measures. Because the correlations of each indirect measure with other indirect measures could only be computed for participants with two or more indirect measures’ scores, leading to a high non-random occurrence of missing data, we could not reliably aggregate all indirect-indirect correlations for each measure, nor compare them across measures of the same type. So, for each measure we display a range of the correlations with other indirect measures that assess the same construct.

Correlations with Directly-Measured Sexuality. We created three composite self-report scores, one of sexuality toward women, one of sexuality toward men, and one of sexuality toward women compared to men. The first two composite scores averaged the standardized responses to four directly-measured single-gender sexuality facets: sexual attraction, frequency of erotic fantasies, frequency of sexual encounters, and frequency of romantic relationships. The gender-comparative composite score was the difference between the single-gender composite scores. We calculated Pearson’s correlations between the indirect measure’s score and the corresponding composite score of directly-measured sexuality (i.e., the comparable single-gender composite score for single-gender measures, and the gender-comparative composite score for the gender-comparative measures). Each correlation was calculated only for participants who completed all relevant measures – the indirect measure’s score and all direct measures included in the composite score. For statistical inference, we used Fisher’s z-test to compare each pair of mean correlations for measures that target the same construct (e.g., Men-qIAT vs. Men-SAT). In the SOM, we report the results of simulation analyses that computed the average correlations of the indirect measures with directly-measured sexual attraction when the sample was balanced with an equal number of straight and gay/lesbian individuals.

We tested whether the positive correlation of each indirect measure (e.g., the qIAT-Women) with directly-measured sexual attraction toward the same gender group (i.e., attraction to women) was larger (in absolute terms) than the negative correlation with directly-measured sexual attraction toward the “other” gender group (i.e., attraction to men). We computed the difference between the absolute values of these correlations and interpreted a zero or negative difference as an indication of poor discriminant validity (Bar-Anan & Nosek, 2014). We used the Williams test to test the differences between the absolute values of the two correlations, while accounting for their dependency (Steiger, 1980), and considered significant differences as support for the measure’s discriminant validity. We also computed similar scores of discriminant validity for direct measures of sexuality, to provide comparative context to the discriminant validity values found for the indirect measures. For this purpose, we used four single-gender sexuality facets (frequency of erotic fantasies, frequency of sexual encounters, frequency of romantic relationships, and personal sexual associations), and tested to what extent the positive correlation of each sexuality facet with same-gender sexual attraction (i.e., attraction to women) was larger (in absolute terms) than the negative correlation with directly-measured sexual attraction toward the “other” gender group (i.e., attraction to men).

In the exploratory analyses, we included only individuals who self-identified as exclusively straight and had scores in all the relevant variables – self-reported sexual attraction toward the same gender and toward the “other” gender, the corresponding scores on at least one single-gender indirect measure, and political identity. For same-gender attraction, for each indirect measure we conducted a multiple regression analysis that predicted indirectly-measured same-gender sexual attraction from self-reported attraction to the same gender, self-reported political identity, and their interaction, while controlling for current gender. Similar regression models were conducted for “other”-gender attraction – for each indirect measure, the model predicted indirectly-measured attraction to the “other” gender from self-reported sexual attraction to the “other” gender, self-reported political identity, and their interaction, while controlling for current gender. All continuous variables were standardized prior to the analysis. A moderation effect for the prediction of same-gender attraction was expected. Notably, due to a violation of the assumption of homoscedasticity in all moderation analyses, we repeated the same analyses with permutation tests for linear models. Because their results were almost identical to the results of the original parametric models, we report here the parametric multiple regression models.

The scores of the indirect measures among women and men in each of the self-reported sexual orientation groups appear in Figures 1-3. Overall, these figures reveal large within group variability and the expected group-level differences on all measures, albeit to different degrees. Table 4 displays a summary of the results of all evaluation criteria for all indirect measures. The SOM provide descriptive statistics for the indirect measures, which were combined with those of the preliminary studies that appear in the SOM.

Test-retest correlations (Table 4) were significant (p < .001) for all measures. The Men/Women IAT had the largest test-retest reliability altogether, r(560) = .61, and the SAT had the lowest test-retest reliabilities – Men-SAT: r(131) = .37, Women-SAT: r (146) = .40. All other single-gender measures had similar test-retest correlations to those previously reported for similar indirect measures (Bar-Anan & Nosek, 2014).

Table 4 presents the internal consistencies of all measures. Of the two gender-comparative measures, the Men/Women IAT (α = .91) was more internally consistent than the Men/Women SPT (α = .65). The qIAT surpassed all other single-gender measures (Men-qIAT: α = .91, Women-qIAT: α = .92), and was equivalent to the Men/Women IAT. The worst single-gender measure was the SAT (Men-SAT: α = .71, Women-SAT: α = .69).

Known Group Differences. The raw effect sizes and confidence intervals for each between-groups comparison are shown in Table 5. The mean (meta-analyzed) effect size for each measure/score appears in Table 4. As these tables show, the Men/Women IAT had greater sensitivity to differences between groups than the Men/Women SPT. Across the single-gender measures and scores, the single scores computed from the Men/Women IAT were the best, followed by the qIAT. A possibly disappointing finding was that in most single-gender indirect measures, regardless of self-reported sexual orientation, the mean indirect measure’s score was not significantly below zero (Figures 2 and 3). Similarly, the mean score of the Men/Women IAT for straight women suggested that as a group, they did not display any preference for men over women (Figure 1). We will discuss possible reasons for these findings in the Discussion.

Correlations with Directly-Measured Sexuality.Table 6 presents the correlations between all indirect measures and the main facets of directly-measured sexuality. All indirect measures were positively associated with the corresponding direct measure of attraction. As detailed in Table 4, among the gender-comparative measures, the IAT performed better than the SPT, and the qIAT was the best-performing single-gender measure.

Correlations with Indirectly-Measured Sexuality.Table 7 presents all correlations among the indirect measures. All the single-gender measures were positively correlated with other indirect measures of attraction toward the same gender. As shown in Table 4, the Gender/People IATs had the strongest mean correlation with indirectly-measured sexuality (r = .28 for the Men/People IAT, r = .32 for the Women/People IAT), which were not much different from the qIAT (r = .28, for the Men-qIAT, r = .25 for Women-qIAT).

Disappointingly, although all the direct measures of all sexuality facets showed the expected negative relation between attraction-to-men and attraction-to-women scores with a moderate-to-strong correlation (r range = -.51 to -.81, Table J1 in the SOM), this was hardly the case with most indirect measures (Table 7). The correlation between the Gender/Not-Gender IATs was positive, r(4,976) = .15, and the correlation between the Gender/People IATs was near zero, r(5,197) = .07. The Men-qIAT and the Women-qIAT were only weakly negatively correlated, r(3,350) = -.13.

Table 4 presents the discriminant validity of all single-gender measures and single-scores of the gender-comparative measures. The qIATs and the Gender/People IATs were the only measures that displayed a significant difference between the (absolute) mean correlations with the same-gender versus “other”-gender direct measures. Both qIATs showed the best discriminant validity, with a .12 difference for the Women-qIAT, and a .07 difference for the Men-qIAT. The single-scores computed from the Men/Women SPT and IAT had rather weak discriminant validity with values close to zero, and the SATs displayed negative values, which indicate no discriminant validity. For perspective, the discriminant validity of the eight direct measures of sexuality ranged from -.07 to .25 (see Table J2 in the SOM).

Political identity in the study sample ranged from -3 (strongly conservative) to 3 (strongly liberal). When comparing the study sample to the subgroup of exclusively straight individuals, the results indicated that the latter was slightly less liberal (M = 0.45, SD = 1.64) compared to the total sample (M = 0.76, SD = 1.64), t(76,898) = 26.62, p < .001, and slightly older (M = 38.15, SD = 15.19) compared to the total sample (M = 36.32, SD = 14.87), t(78,310) = 17.72, p < .001.

For most single-gender measures (all except the SAT) and for the IAT (but not the SPT), there was a significant interaction between the indirect measure’s score and political identity reflecting a weaker direct-indirect correlation of same-gender sexual attraction for more conservative (vs. more liberal) participants (Figure 4 and Table 8). Notably, these effects had rather small beta coefficients, but there were no similar moderation effects for any of the measures in the models that predicted “other”-gender sexual attraction (Figure 5 and Table 9). The SOM includes further analyses that explore the robustness of the moderation effects to alternative explanations.

In the present research, we tested and compared the validity and reliability of seven indirect measures of sexual attraction to men and women: Men/Women IAT, Men/Women SPT, Gender/People IAT, Gender/Not-Gender IAT, SAT, qIAT, and Single-Target IAT. We found that most indirect measures displayed acceptable internal consistency and mostly adequate test-retest reliability. Moreover, all indirect measures discriminated between groups that are expected to display different levels of sexual attraction, based on their gender and sexual orientation. Further, all indirect measures toward the same gender group were positively related to direct measures of sexuality toward that gender group and to one another, and negatively related to direct measures of sexuality toward the “other” gender group. We found some evidence for discriminant validity of the qIAT and Gender/People IAT. Notably, these measures, which were used here for the first time to study sexual attraction, showed better performance than the other single-gender indirect measures. Our results provide evidence for the validity of the indirect measures, highlight their potential weaknesses, and may guide selection of indirect measures in future research.

For most indirect measures, the relations between directly- and indirectly-measured sexual attraction of straight individuals were stronger for Liberal individuals in the case of same-gender attraction, but not for “other”-gender attraction. Even though the moderation effects were small, this finding is important because capturing unintentional aspects of a construct is the main appeal of indirect measures over direct measures of sexuality. Notably, no previous research has shown any evidence that indirect measures of sexual attraction capture sexual attraction among people who may be less willing to report that attraction.

As part of our research, we compared two gender-comparative indirect measures of sexual attraction: the IAT and SPT. Our findings suggest the IAT clearly surpassed the SPT on all qualities, consistent with conclusions of other comparisons of the reliability and validity of these two tasks (Bar-Anan & Nosek, 2014). Importantly, the IAT had closer relations with directly-measured sexuality, in concordance with previous reports (e.g., Snowden et al., 2008).

For the gender-comparative measures we also computed single-gender scores. Because these scores relied on a subset of the trials used to compute the gender-comparative scores, it is not surprising that they showed reliability and convergent validity that were inferior to those found with the gender-comparative scores. However, these scores did not produce convincing evidence for discriminant validity, reaffirming previous conclusions that computing single-construct scores from a subset of trials in the comparative measures is not useful for capturing non-relative constructs (Bar-Anan & Nosek, 2014), such as attraction to one gender.

In line with contemporary understanding of sexuality, a main part of our research focused on constructing single-gender indirect measures of sexual attraction, assessing sexual attraction to men and to women separately and independently. As part of this endeavor, we compared the validity and reliability of two single-gender versions of the IAT. The first version was based on the approach of previous studies (Snowden et al., 2020, 2024; Snowden & Gray, 2013), contrasting images of men or women with images of scenery or household items. In the other version, developed in the present research, the contrast category was labelled People and included images depicting groups of both men and women. We found that the latter version was equal or superior to the former version in all psychometric criteria, suggesting that single-gender IATs of sexual attraction produce more valid results with a neutral People category compared to a neutral category of inanimate objects. We found similar results in a preliminary study reported in the SOM using a smaller sample (Study S2).

Although the results suggest that the neutral category of the Gender/People IAT enhanced the validity of the single-gender IAT compared to the previous version of this task (Snowden et al., 2020, 2024; Snowden & Gray, 2013), they also suggest that both approaches are not ideal. In both IATs, the attempt to measure single-target constructs using the IAT’s inherently comparative design may have introduced unwanted noise into these measures. In the Gender/Not-Gender IATs, some people might associate sexual attraction less strongly with nature scenery than with the concept Men (or Women) simply because they find very little relation between nature scenery and sexual attraction, and not because they are sexually attracted to men (or to women). Similarly, in the Gender/People IATs, some people might associate sexual attraction with the concept People less than with the concept Men (or Women) because sexual attraction is typically associated with specific gender groups (e.g., Women) rather than with the concept People. In other words, some individual differences in the associations that influenced those IATs might have been unrelated to the participants’ sexual attraction. This could explain why most sexual orientation groups showed mean scores that reflected sexual attraction to both genders (see more about this finding below).

Of the single-gender indirect measures we used, the qIAT was the best performing measure, with the strongest discriminant validity, high internal consistency, and the highest correlations with directly-measured sexuality. Indeed, past studies have established high reliability and validity for the qIAT in the personality domain (e.g., Friedman et al., 2021; Yovel & Friedman, 2013). Another possible advantage of the qIAT is that its stimuli are only statements, with no photos of men or women, reducing possible bias by the photos’ compatibility with people’s specific sexual preferences (within gender), or by culture-based sexual associations. Nevertheless, the reliance of the qIAT on statements taken from self-report questionnaires may also be a disadvantage. Specifically, the high correlations of the qIAT with self-report measures in the present and in previous research may raise the suspicion that it is more sensitive to intentional processes than other indirect measures. However, the moderation effect we found for this measure, in which Liberals showed stronger qIAT/self-report relations compared to Conservatives, was stronger than for the other measures, providing some evidence for the unintentional nature of the processes that influence the qIAT. Based on all performance criteria, we conclude that the qIAT is the best single-gender indirect measure of sexual attraction.

The worst performing measure in our research was the SAT, a novel measure that we developed for this research, but found only modest evidence for its validity. The SOM includes a study that found poor performance for the ST-IAT (Karpinski & Steinman, 2006; Wigboldus et al., 2004), a more established and popular indirect measure of social cognition. Our evidence suggests that these two measures may not be adequate options for measuring sexual attraction.

Several findings in the present research raise some concerns about the validity of single-gender indirect measures. Most single-gender measures did not show even a moderate negative correlation between attraction to men and attraction to women, as is consistently found in directly-measured sexuality (e.g., Jacobson & Joel, 2019). The exception was the qIAT, but even the qIAT showed only a slight negative correlation between attraction to men and attraction to women. A related concern, consistent with similar results in previous studies (Snowden et al., 2020; Snowden & Gray, 2013), is that the mean scores of most single-gender measures often reflected sexual attraction even toward the gender that the participant reported not being attracted to. This might reveal that all people are attracted, at least to some extent, to both genders. However, that theoretical interpretation of our results is inconsistent with present knowledge about sexuality (e.g., Vrangalova & Savin-Williams, 2012). Therefore, a more likely account for these results is a methodological weakness in indirect single-gender measures of sexual attraction. Specifically, single-gender measures may be sensitive to a factor that does not reflect sexual attraction. For example, perhaps people associate each gender with sexuality to some extent because, even if they are not attracted to that gender, they are still more attracted to individuals from one gender group than to anything that does not belong to one gender group. The qIAT seems the least vulnerable to that validity threat, perhaps because it is the only measure that is based on comparing attraction to a specific gender with no attraction to that gender.

The results pertaining to the discriminant validity of the indirect measures raise another concern regarding the validity of single-gender indirect measures. Specifically, most of these measures could not clearly differentiate between sexual attraction to men and to women. While the weak discriminant validity could be a limitation of the indirect measures, it is also possible that our approach to evaluating the measures’ discriminant validity was limited by relying on direct measures of sexual attraction to men and to women, which were highly (negatively) correlated (r = -.77).

Finally, the results of the moderation analyses suggest that some of the indirect measures may be sensitive to unintentional processes, thus providing a major contribution to knowledge on the automatic aspects of sexual attraction. Currently there is no consensus on the extent to which indirect measures, such as the IAT and its variations, capture automatic processes (Vianello & Bar-Anan, 2021). While previous research suggests that the IAT is sometimes sensitive to nonautomatic or nonevaluative factors (e.g., Olson & Fazio, 2004; Rothermund et al., 2009; Teige-Mocigemba & Klauer, 2013; Uhlmann et al., 2006), there is also evidence that indirect measures (the IAT, SPT, and the qIAT) are sometimes sensitive to quick unintentional mental processes that may require relatively little cognitive resources (De Houwer, 2006; Friedman et al., 2022; Hofmann et al., 2008; Schmitz et al., 2013; Spruyt et al., 2007). The present finding that exclusively straight Liberals showed a stronger relation between indirectly and directly measured same-gender attraction compared to exclusively straight Conservatives, provides evidence for the sensitivity of these indirect measures to unintentional processes.

The current research is limited by the non-representative sample, with an over-representation of participants who are young, White, Liberal, North American, educated, and identify as women. Moreover, random sampling may have been limited by the warning that the study includes sexual materials and the resulting self-selection. However, as detailed in the SOM, there were no meaningful differences in the age and political identity of participants who saw this warning and participants who did not.

Another notable limitation is our use of a binary construction of gender and sexuality. Although a major aim of our research was to allow for independent measurement of sexual attraction, all our measures still rely on a two-gender system thinking framework, by assessing attraction vis-à-vis the distinct gender groups Men and Women. Furthermore, the indirect measures that used pictures as stimuli (i.e., all indirect measures except the qIAT), used images of White, thin, cisgender-presenting young people, with appearances that adhere to common beauty standards. As detailed in the Introduction, due to these choices, our results do not capture the full spectrum of human sexual and gender diversity. Therefore, our knowledge on people’s sexuality could benefit from future developments of additional, more inclusive measures of sexual attraction, in which sexuality is construed in ways that transcend the binary gender framework, such as by assessing sexual attraction to additional gender groups or to particular people, rather than categories.

The current research provides evidence that can guide sexual attraction researchers in using indirect measures and inspire further investigation into the nature of these measures. Based on our findings, researchers who wish to use indirect measurement of individual differences in sexual attraction to a specific gender should consider using the qIAT and the Gender/People IAT. However, the Men/Woman IAT might be the best choice for researchers who are not particularly interested in the measurement of sexual attraction to a single gender. Another contribution of the present research pertains to the finding that political identity moderates the relation between direct and indirect measures of same-gender sexual attraction. Although this finding was exploratory, it was consistent across multiple measures and could inspire further research to examine whether it reflects an unintentional reflection of one’s sexual attraction.

Should researchers use the indirect measures of sexual attraction that we tested in the present research? Currently, we do not have a definitive answer to that question. These measures are certainly sensitive to sexual attraction, and it is quite possible that using them may capture variance in sexual attraction that is not always captured by self-report measures. This may be particularly beneficial in cultures that are less accepting of same-gender sexuality than Western cultures, as there may be larger discrepancies between direct and indirect measures of sexual attraction in these cultures. On the other hand, any finding that would be unique to the indirect measures will require further investigation, before one can conclude that it reflects sexual attraction rather than a different, currently unknown, construct. That might limit the usefulness of these measures until further research clarifies the open questions about which factors other than sexual attraction may influence their between-individuals variability.

Although some of our results cast doubts on the validity of the indirect measures, they also raise questions that could inspire further research. What are the factors that produced unexpected mean scores in all the indirect measures, and poor discriminant validity in the indirect single-gender measures? Identifying these factors may help improve the indirect measures by implementing modifications that would reduce sensitivity to these factors, or by controlling for these effects with separate measures of the biasing factors.

Further research could also explore whether the results that we considered validity weaknesses are in fact discoveries about the nature of sexual attraction. Namely, we cannot rule out the possibility that the cause for the bias of the indirect single-gender measures toward scores that presumably suggest sexual attractions to both genders is that most people do harbor some form of sexuality-related processes to both genders. Similarly, perhaps the factor that diminished the negative correlation between attraction-to-men and attraction-to-women scores on the single-gender measures is in fact a general sexual attraction factor that self-report measures have not yet uncovered.

In the present research, we compared the psychometric qualities of several indirect measures developed to tap sexual attraction toward men and women, separately or comparatively. The ‘classic’ Men/Women sexual attraction IAT had the best psychometric qualities but no ability to differentiate between attraction to men and to women. Of the single-gender measures, the best was a novel adaptation of the qIAT. Based on our results, we recommend the qIAT for studies requiring independent measurement of sexual attraction to women and to men. However, given the shorter length of the Men/Women IAT and its superior psychometric qualities, we recommend it for studies in which a comparative measurement of sexual attraction suffices. Our findings further suggest that indirect measures may provide relevant information on top of that provided by self-report mainly in societies or cultures with less acceptance of same-gender sexuality. All in all, our results may be used to guide future selection of indirect measures of sexual attraction, while also providing directions for further research into their validity and improvement.

All authors contributed to the following endeavors: conception and design, acquisition of data, analysis and interpretation of data and revision of the article. Finally, all authors approved the submitted version for publication.

The study’s design and analyses were not pre-registered.

This work was co-funded by the Israel Science Foundation (ISF; grant No. 1684/21) awarded to Yoav Bar-Anan, and by a grant by the European Union (ERC, BeyondGenderBinary, Project No.101054741) awarded to Daphna Joel. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.

The authors have no competing interests to declare.

All materials, analysis codes and data are accessible at: https://osf.io/juezp/.