Timbre Semantic Associations Vary Both Between and Within Instruments: An Empirical Study Incorporating Register and Pitch Height

Most of the seminal timbre psychoacoustics literature uses stimuli of the same note to control for the perceptual influence of fundamental frequency (Grey 1977; Kendall & Carterette, 1993; McAdams et al., 1995; Zacharakis et al., 2015). Several notes have been used in previous work, most commonly E♭4 (e.g., Eerola, Alluri, & Ferrer, 2012; Saitis & Siedenburg 2020; Wallmark, 2019b). E♭4 corresponds to a fundamental frequency of approximately 311 Hz (in an equal tempered scale tuned at 440 Hz), which represents the “average” pitch in many music corpora (Huron, 2001). Although keeping the note played and other parameters such as duration and loudness constant reduces confounding variables, this paradigm suffers in ecological validity (Siddiq et al., 2018). Furthermore, in studies of instrument timbre, keeping the note played constant across stimuli may necessarily exclude certain instruments or involve the inclusion of extreme low/high notes, which can lead to non-idiomatic tone production.

Contrastingly, relatively few experiments have investigated interactions between pitch and timbre, despite evidence that distinctions are relevant to timbre processing. Differences over an octave may increase the difficulty of identifying common sound sources (Handel & Erickson, 2001; Marozeau et al., 2003), though Steele and Williams (2006) demonstrated that musicians appear to be more successful than nonmusicians at this task. Speeded classification tasks have observed interactions of pitch and timbre (Krumhansl & Iverson, 1992). An experiment reported by Allen and Oxenham (2014), moreover, observed interference effects of fundamental frequency and timbre, with results suggesting that fundamental frequency and auditory brightness are tightly related (see also Cousineau et al., 2014; Marozeau & de Cheveigné, 2007; Melara & Marks, 1990; Schubert & Wolfe, 2006; Siedenburg, 2018).

Timbral differences across instrumental registers have also been considered in experiments assessing the semantic and affective connotations of timbre. In an interlanguage semantic ratings study, Zacharakis et al. (2014) developed stimuli consisting of acoustic instrument signals of a few pitch classes spread across three octaves. They reported that fundamental frequency was negatively correlated with mass-related terms among English speakers, and positively correlated with luminance-related terms among Greek speakers. McAdams et al. (2017) investigated effects of music training, pitch register, and instrument family on affect ratings. Using isolated orchestral instrument samples of E♭ across seven octaves (E♭1 to E♭7), the researchers reported that pitch register was a significant predictor of valence, tension arousal, and energy arousal and was also involved in significant interactions in these models. Finally, Reymore (in press) used the 20-dimensional semantic model presented in Reymore and Huron (2020) to collect ratings of recordings of two instruments, oboe and French horn, playing at three dynamics across four relative register levels. These results revealed that within-instrument timbral variability may be quite complex. While some trends between register and mean semantic ratings were linear, others followed different patterns; for example, soft/singing was less prevalent in the highest and lowest registers of the oboe but was especially prominent in the middle registers. Similarities between the oboe and French horn were evident within some semantic scales, but others showed different activity between the two instruments, and interactions were observed between register and dynamics for some scales. Several terms showed significant relationships to register for both instruments, including shrill/noisy, rumbling/low, and sparkling/brilliant.

Such semantic interactions between timbre and pitch or register have also been reported in studies of cross-sensory associations between music and color, offering converging evidence from another domain of the importance of timbre/pitch relationships. Using piano and string tones, Ward et al. (2006) found the increase of brightness with ascending pitch as a common underlying mechanism for timbre-color mappings in both sound-color synesthetes (individuals who involuntarily experience a color when hearing a certain note/sound) and non-synesthetes. A correlation of higher fundamental frequencies and brighter grayscales was also reported for non-synesthetes by Adeli et al. (2014), who examined mappings between musical instrument sounds and colored shapes (though note that pitched stimuli were filtered to share the same normalized temporal envelope, which might have compromised their timbral integrity). Reuter et al. (2018) investigated timbre-color associations in non-synesthetes using 60 orchestral sounds (ten instruments, three notes, two playing styles). Similar to our study, each instrument was represented by a low, a medium, and a high note relative to its range, but only one pitch class (E) was used in sampling notes across registers (we used two pitch classes, see “Method”). Although some instruments were significantly matched with certain colors (e.g., violin with yellow, bassoon/cello/tuba with brown), grouping the most frequent color selections for each instrument by register revealed that timbre-color mappings were dominated by the concurrence of ascending pitch and increasing brightness. Generally, higher stimuli were more often matched with lighter colors (consistent with Ward et al. 2006), as well as more saturated colors, while stimuli from the lower registers tended to be paired with darker colors. A systematic shift of color hue towards more yellow tones at higher notes was also observed.

The main objective of this study is to investigate how timbre semantic associations vary with instrument register and pitch height across instruments. Although several timbre studies have incorporated pitch differences into their design, registration remains undertheorized in the timbre semantics literature. Further, sample sizes of previous studies have been relatively small, diminishing the generalizability of the findings. The present experiment aims to explore the contribution of instrument and register in the generation of semantic associations involving timbre. To do so, we recruited a large global online sample of English-fluent participants to rate eight orchestral instrument sounds spanning two pitch classes (C and G) across five octaves (C2 to C7) on 20 semantic scales. Our approach is built from the understanding that timbre is a perceptual phenomenon, and that timbre is both a quality and a contributor to source identity (Siedenburg & McAdams, 2017).

This project was also motivated by the authors’ involvement with the ongoing interinstitutional Composer-performer Orchestration Research Ensembles, or CORE Project, which is realized in the context of the international ACTOR Partnership (Analysis, Creation, and Teaching of Orchestration; actorproject.org/working-groups/core). CORE doubles as an educational experience for performance and composition students and as an opportunity to document and research creative processes with specific focus on timbre and orchestration. In choosing instruments as stimuli for the current study, we included instruments from the CORE ensemble: violin, bass clarinet, trombone, and vibraphone. The CORE Project continues to produce dozens of new compositions by emerging composers for an instrumentation which is common to all participating institutions; thus, the results of the current experiment, by addressing this instrumentation, provide valuable information for analyzing perceptual effects created in these pieces. By studying the variation of semantic associations with register among CORE Project instruments, we aim to: 1) contribute to the ongoing work of analyzing the resultant pieces, particularly in understanding semantic and perceptual effects; and 2) provide a resource for composers in future rounds of this project, who will be composing using the same instrumentation.

For this online perceptual study, a general global sample of 590 adults were recruited using the Prolific platform (prolific.com). Thirty-six participants were removed due to incomplete responses (time-outs). We next assessed the data for suspected “bots” (computer program) and/or “farmers” (phony participants in server farms), which have grown increasingly prevalent in online subject pools in recent years (e.g., Chmielewski & Kucker, 2020). Multiple data quality checks were performed in this assessment, and data were flagged if answers appeared to be repeated or random; interrater correlation was low (r < .10); headphone check (following Woods et al., 2017) failed two consecutive times; time spent on the instructions or total experiment was impossibly low; and/or answers given in the free response attention check questions were missing or nonsensical (see “Procedure”). Based on a holistic consideration of these flagged cases, 13 participants were cut. We also cut one duplicate participant.

The remaining 540 participants (242 females, 293 males, 3 other, 2 not disclosed) were fluent in English (self-reported via Prolific) and came from a range of musical backgrounds. Using the single-question item from the Ollen Musical Sophistication Index (2006), 81% of the participants self-identified as non-musicians (“non-musician” and “music-loving non-musician”); 19% identified as musicians (“amateur,” “serious amateur,” “semiprofessional,” and “professional”; see Supplementary Material at mp.ucpress.edu for full breakdown). The average participant age was 26.9 years (range 18–71, SD = 8.7). Participants were paid $4.15 USD for taking part in the study. The experiment took M = 36 minutes to complete (SD = 12 m). The study was approved by the University of Oregon Institutional Review Board.

In this repeated-measures design, participants listened to brief recordings of instruments playing sustained notes (approximately two seconds) and rated these sounds on 20 semantic scales. We manipulated two independent variables: register and instrument. Specifically, the stimuli included eight instruments (flute, oboe, bass clarinet, trumpet, trombone, violin, cello, vibraphone); each instrument was represented by notes from three registers (low, middle, high), for a total of 24 stimuli. Participants were shown word prompts consisting of 20 orthogonal semantic dimensions for timbre (see “Semantic Scales” below for further detail).

Stimuli came from the Vienna Symphonic Library Cube (2011), a sample library of recorded sounds played by professional musicians. Stimuli included the natural attack and were edited to 1.5s sustains plus the natural decay of the sound, which varied slightly in length among the stimuli, dependent upon the envelope. All sounds were adjusted to a matching ANSI-loudness level (American National Standards Institute) using the Genesis loudness toolbox in MATLAB. In this procedure, the loudness of each sound was first computed using the Moore model. Next, the median of all loudness values was computed across all stimuli, and finally, each sound was adjusted to this median value. Although this process helps to equalize loudness, additional variability is likely present due to differences in individual perception and participant headphones. In choosing the instruments to be rated, we began by including the four instruments that have been used in the CORE Project (see “Study Aims”): violin, bass clarinet, trombone, and vibraphone. Vibraphone sounds were bowed, rather than struck, to maintain consistency of excitation type across the instruments of the stimuli set; all other stimuli were produced with standard playing techniques. We then added flute, oboe, trumpet, and cello in order to balance the range of the stimuli and to maximize the variability of orchestral timbres tested. We note that the vibraphone emerges as something of an outlier among our chosen instruments: the only percussion instrument, the only idiophone, the only instrument played with a technique other than its default, and the only instrument that is not a standard member of the traditional symphony orchestra. As our goals included understanding how instrument type might affect the relationship between a semantic category and register, we determined that the variability offered by the vibraphone was advantageous.

Each of the eight instruments was represented by a low, a medium, and a high note relative to its range. We sought to minimize the number of pitch classes used to avoid potential effects of intervals between stimuli on semantic judgments. However, use of a single pitch class precluded thorough representation of all three registers in all instruments; accordingly, two pitch classes, C and G, were used in sampling notes across registers, which allowed access to all three registers on all instruments. The interval of a perfect fifth, moreover, was deemed preferable to intervals that could confound our results with affective connotations via order effects (e.g., a minor third). The selection technique prioritized this low-medium-high distribution across individual instrument ranges. The resulting set of notes is distributed reasonably well across the orchestral ambitus, though not all notes are represented the same number of times in the stimulus set (see Figure 1).

The semantic scales used in the rating task were derived from the 20-dimensional model presented in Reymore and Huron (2020), which was built from the results of open-ended interviews and rating tasks. In that study, principal component analysis (PCA) was used to generate the model’s dimensions; these 20 dimensions have proven useful in capturing timbre semantic variation (e.g., Reymore, 2021; Reymore, in press). The dimensions of the original model include varying numbers of descriptors, up to seven individual terms per dimension. To reduce cognitive load on participants and decrease the length of the current experiment, we included a maximum of three terms per scale. Table 1 contains the 20 semantic scales used in the experiment.

Given the range of sounds of the stimuli set, which included eight instruments varying in register, we first considered the possibility that some of the dimensions might display little variance and could thus be trimmed from the experiment to increase its tractability. Accordingly, we ran a pilot study with 20 participants to finalize the stimuli set and to potentially reduce the number of semantic scales. However, after considering analyses of pilot study results (correlation matrix among ratings, PCA results, and hierarchical clustering), we determined that, consistent with previous work, the original 20 scales were indeed optimal for this study.

Participants were routed from the Prolific recruitment site to GlistenIQ (Bailly, 2020), a custom platform hosted on Google Cloud Platform and created using Node.js and standard web technologies. After consenting, participants answered demographic questions and questions about their musical background. Participants were asked to wear headphones for the duration of the experiment, and a headphone check following Woods et al. (2017) was implemented: participants judged the relative loudness of pairs of tones, where one of the tones was presented 180° out of phase across stereo channels, a task that is easy with headphones but difficult over loudspeakers due to phase-cancellation. If a participant failed the check, they were given one chance to redo the check. Next, for the purpose of familiarization with the range of stimuli, participants listened to a sound file containing 400 ms clips of all the stimuli (in a randomized sequence). Two trial ratings of sounds not included in the main study were made so that participants could adjust to using the interface. Instruction texts are included in the Supplementary Materials at mp.ucpress.edu.

During the main task, participants were provided with a semantic descriptor and then listened to a set of 24 recordings of musical instruments playing sustained sounds, rating each sound according to the given descriptor. Specifically, they were prompted to rate how well each scale described the sound being played. This process was repeated for each of the 20 semantic scales; that is, all 24 sounds were rated on a single scale before moving on to the next scale. Instructions specified that when scales consisted of more than one word, participants should base their rating on the word in each set that they felt was most applicable. Ratings were made using the number keys on the keyboard on a scale from 1 (“does not describe at all/does not apply”) to 5 (“describes extremely well”). The midpoint of the scale (3) was designated as “describes moderately well.” Sounds were automatically loaded and played at the beginning of each new prompt, and participants were permitted to replay a sound once. A progress bar was displayed at the top of the screen to inform participants of total study duration.

The order of presentation of the scales was randomized for each participant, as was the order of audio stimuli within each scale block. Two attention check questions were employed at random over the course of the experiment requesting that participants identify, to the best of their ability, the last instrument they heard.

All analyses were carried out in R (R Core Team, 2020; version 4.0.5). Our final dataset consisted of 540 participants’ ratings of the 24 instrument stimuli on 20 semantic scales. Table 2 below reports the mean and standard deviation of each stimulus that received the highest mean rating for each semantic scale. Ratings were generally in accord with expectations based on casual observation. For example, the highest rated stimulus on the airy/breathy scale was the flute in the middle register, while the sound rated highest on woody was deemed to be the cello in the low register, and the most sparkling/brilliant/bright sound was the oboe in the high register. Conversely, we can consider the top descriptors for each stimulus. For example, the top descriptor for oboe in the low register was sustained/even, for the middle register was pure/clear/clean, and for the high register was shrill/harsh/noisy.

Figure 2 illustrates mean ratings by instrument type and register. Although some terms show consistent patterns across registers for all instruments (e.g., deep/thick/heavy, sparkling/brilliant/bright), it is evident from the figure that other terms show varied types of relationships between semantic association and register, dependent on instrument type (e.g., brassy/metallic, resonant/vibrant). For illustrations of mean ratings on register (averaged across instruments) and mean ratings on instrument (averaged across register), see the Supplementary Materials at mp.ucpress.edu.

One intriguing observation was that mean ratings of brassy/metallic among the 24 stimuli show the least amount of variance (range = 2.21–2.96), despite the inclusion of two brass instruments, suggesting that perceived brassiness may be a product of interactions between instruments and other parameters such as dynamics and or playing technique. Additionally, as all instruments played for approximately the same duration, it is unsurprising that there is very little variance among ratings of sustained/even (range = 2.86–3.74). The exception to this was the vibraphone, whose bowing technique, along with the extended ring at the end of the sound, scored lowest overall on sustained/even.

Finally, we can illustrate the results using radar plots (Figure 3), following the approach of Reymore (2021). In examining these plots, the vibraphone stands out as demonstrating relatively little semantic variation across registers as compared to the other instruments. By way of contrast, for example, flute, violin, and cello were more likely to be considered shrill/harsh/noisy in the upper register, and bass clarinet, trombone, and cello were rated as more deep/thick/heavy in the lower register compared to middle and high.

First, the correlation matrix of ratings by dimension was examined to assess the independence of the 20 categories, following Reymore (2021). That is, if the 20 semantic categories are perceptually distinct for the stimuli in this study, we would expect that no two categories should demonstrate strong correlations. Indeed, no strong correlations were observed between pairs of dimensions. The matrix included 12 moderate correlations ranging in absolute value from r = .30 to .43; the other correlations were each less than .30. The strongest positive correlation was between deep/thick/heavy and raspy/grainy/gravelly, r(12742) = .43, 95% CI [.42, .45], p < .001, and strongest negative correlation was between deep/thick/heavy and sparkling/brilliant/bright, r(12762) = –.42, 95% CI [–.44, –.41], p < .001. These results are similar to those of Reymore (2021), in which participants used the same 20 semantic categories to rate imagined typical sounds of 34 instruments: in that study, only 26 of the unique correlations had absolute values ≥ .30.

Hierarchical clustering was performed on the rating data to illustrate relationships among stimuli within the 20 semantic categories (Figure 4). This analysis was carried out using the hclust() function in the stats package, which uses Ward linkage with Euclidean distance.

A number of explanations for the semantic groupings represented in this dendrogram seem plausible. Most obviously, we might expect sounds produced by the same instrument to generate similar semantic profiles. Intriguingly, however, nearest-neighbor pairs suggested by this model are not primarily grouped by instrument (apart from the vibraphone, which stands alone in that its low, medium, and high notes are clustered together). We might also expect to observe grouping by instrument family, with strings, woodwinds, brass, and percussion producing internally consistent and distinct semantic profiles. This also does not appear to be a strong factor, at least not according to the conventional orchestral families, as all the clusters contain representatives of at least two different families (with the exception of the vibraphone-only cluster). We also considered the groupings in relation to Hornbostel-Sachs system as used in organology. However, although all the instruments in the first cluster are aerophones, three of the five groupings also cross boundaries of the basic Hornbostel-Sachs categories, including both chordophones and aerophones. Excitation method also seems not to be a strong factor, as the bowed vibraphone is more closely clustered with the blown aerophones than with the bowed chordophones. In sum, these results are not easily explained on the basis of instrumental differences alone.

Considering pitch height, however, provides a much more satisfying interpretation of the clusters. The clusters themselves group neatly by overall orchestral register, containing between one and four notes that are always consecutive according to our overall schema (again, excepting the vibraphone-only cluster). We may consider these to be a high cluster (G5–C7), a medium-high cluster (C5–G5), a medium-low cluster (G3–G4), and a low cluster (C2). This model is not perfect, as there is a small amount of overlap: both the high cluster and the medium-high cluster contain the note G5, indicating that there is still a role for instrumental differences. Nevertheless, the trend for semantic groupings to correspond with pitch height is clear. Relative register also appears to be a predictor, although it is difficult to disentangle the relative roles of pitch height and register. For example, the low cluster contains only the lowest notes from each of the three instruments represented. All other clusters contain mixtures of relative registers: for example, the medium-high cluster contains the high note of the bass clarinet and the middle notes of the oboe, flute, and trumpet.

To first determine the optimal modeling approach for our data, we computed exploratory models that examined relationships between ratings of each semantic dimension and instrument, register, and musical identity (“musician” vs. “nonmusician”) of the participant. As noted above, register was categorical, relative to each instrument (3 levels: “high,” “middle,” or “low”). Musician identity was derived from responses to the Ollen (2006) single-measure item as a binary category, either “musician” or “nonmusician.” Specifically, we considered instrument, register, and musical background as predictors of mean semantic ratings.

The model building process included an initial assessment of the 20 models’ random structures and consideration of the inclusion of the musical background variable via log likelihood ratio tests. Specifically, log likelihood ratio tests were first used to compare models with random intercepts only (for participant and stimulus) to models with a maximal random effects structure (based on the methods of McAdams et al., 2017). The maximal effects structure included random intercepts for participant with random slopes for instrument and register and random intercepts for stimulus with random slopes for musical background. We observed that the addition of random slopes significantly increased the goodness-of-fit of all 20 models, and so the maximal random structure was maintained moving forward. Next, log likelihood tests were used again, this time to determine whether musical background significantly contributed to each model. Results were mixed: while most models did not see a significant improvement in goodness-of-fit with the musical background variable, five models did (percussive, raspy/grainy/gravelly, airy/breathy, woody, and muted/veiled). Consequently, the musical background variable and the random slope for musical background were included only in these five models moving forward.

Multiple linear mixed effects models for each semantic dimension were then assessed using the dredge() function from the MuMIn package (Barton, 2009), a model selection approach that compares fit among multiple model configurations. Both instrument and register were fixed for inclusion in the final model. This analysis consistently suggested the inclusion of only instrument and register as fixed effects (no interactions), based on consideration of model rankings by AIC, BIC, and AICc. This conclusion applied to all 20 models, including those five that had retained the musical background slope and variable in the previous step. Although the musical background variable was removed from the final models for these five dimensions, the musical background slope was still included.

Statistical significance for these final linear mixed effects models was determined through Type III Wald chi-squared tests. Results are shown in Table 3. For most models, main effects of both register and instrument were significant. Brassy/metallic was exceptional in that neither register nor instrument were significant predictors of scale performance. Several models indicated significant main effects of one variable but not the other: for example, while register was significant for open, instrument was not; conversely, while instrument was significant for focused/compact, nasal/buzzy/pinched, and sustained/even, register was not. Conditional and marginal R² values for the final models were calculated using the r.squaredGLMM() function from the MuMIn package (Barton, 2009). The semantic scales in Table 3 are listed in order of marginal R² values (that is, R² for fixed effects only) from highest to lowest. Terms that showed the strongest positive associations with register include shrill/harsh/noisy, sparkling/brilliant/bright, ringing/long decay, and percussive; terms with the strongest negative associations with register include deep/thick/heavy, raspy/grainy/gravelly, hollow, and woody (see Figure 5).

In addition to asking the question of how register, a variable relative to each instrument, affects semantic associations with timbre, we can also consider how pitch height affects the same semantic associations. Recall that the term “register” refers only to the position of a given note within the overall available range on a given instrument. Because instruments vary greatly in range, the same note may be in the low register of one instrument but in the high register of another. Our principal interest in register came in part from the possibility that certain timbral qualities may be associated with register; that is, the highest and lowest notes an instrument is capable of playing may share particular semantic associations. The results of the exploratory modeling described above suggest that this is the case.

However, as we saw earlier in the hierarchical cluster analysis, another question remains as to what extent explanations of semantic variance from register overlap with explanations from pitch height and whether one of these factors explains more variance than the other. Thus, we focus in our post hoc analysis on critically comparing pitch height and register. Two models were considered for each semantic scale, one using only pitch height as a fixed effect and one using only register as a fixed effect. Both types of models included the random effects structures determined through the initial analysis; however, the pitch height models included pitch height as a random slope, whereas the register models included register as a random slope. As with register, pitch height was modeled as a categorical variable. Recall that we designed the experiment to principally consider register; across stimuli, registers were represented equally (that is, each instrument was represented once at each of three relative levels). However, notes were not distributed evenly across stimuli due to the instruments used; although this distribution is not ideal for modeling, all models nonetheless converged successfully.

Log likelihood tests showed that all pitch height-only models demonstrated significantly better goodness-of-fit than register-only models (all p values < .001). We also compared marginal R² values between the two types of models. Table 4 reports marginal R² (fixed effects only) for models that demonstrated a significant effect for either pitch height, register, or both. The marginal R² for pitch height models was consistently higher than that of the register models, suggesting that although both register and pitch height are related to timbre semantic associations, pitch height explains more variance in semantic ratings than does register. It should be noted that the pitch height variable included nine categories, whereas the register variable included three; the additional degrees of freedom may contribute to the increase in R².

The present experiment examined the effects of instrument and register on timbre semantic associations. Analyzing semantic ratings data from a large global sample, we found, for the first time, that most of the 20 ubiquitous semantic categories for timbre vary with register and instrument. Exceptions include brassy/metallic, which did not significantly vary with either register or instrument; sustained/even, nasal/buzzy/pinched, and focused/compact, which varied with instrument but not with register; and open, which varied with register but not with instrument. However, considering marginal R² as a measure of effect size, it is apparent that magnitude varies greatly among semantic categories. This measure is particularly important in the interpretation of the results. For example, it is apparent that for those cases described above in which instrument, register, or both were not significant, the marginal R² values are low (all under .10). In other cases, as with airy/breathy and resonant/vibrant, both register and instrument were significant, yet R² is again less than .10. The secondary analyses comparing pitch height-only and register-only models also provide insight into the effect size with respect to pitch height and register specifically (in comparison to the instrument-register models reported in the primary analysis, where R² values may include overlapping variance). In other words, although our primary analysis demonstrated significant relationships between register and semantic category for 16 of the 20 models, the effect size for many of these categories is relatively small. Indeed, the secondary analysis reveals that register alone explains 5% or more of the variance for only 8 of the 20 categories. Among these, four categories stand out as most relevant for both register and pitch height: deep/thick/heavy, sparkling/brilliant/bright, shrill/harsh/noisy, and raspy/grainy/gravelly.

In interpreting our results, one should also keep in mind that our a priori model selection method resulted in models with no interaction terms, as interactions did not significantly improve model fit. However, examination of the mean ratings by both instrument and register (Figure 1) demonstrates that there may often be at least potential mild interaction between instrument and register, in that not all instruments always change on a given category with respect to register in the same manner. Thus, the results of the exploratory modeling reported here suggest which semantic categories demonstrate global effects of register on semantic rating, but this does not necessarily mean that each individual instrument always follows the same pattern. Furthermore, it is also possible that terms that did not reach significance or that explain little variance globally (such as nasal/buzzy/pinched or open) might actually demonstrate variation by register for certain individual instruments but not for others. For example, register explained very little overall variance for ratings of open. However, observation of rating means (Figure 1) suggests that open hardly varied at all with register for oboe, violin, and vibraphone but did show notable variation for other instruments, such as the bass clarinet and trombone.

To explore in more detail how the semantic-register relationship can vary among instruments, consider Figure 2. We can see that some terms showed directional effects of register that were clearly consistent across all eight instruments, including deep/thick/heavy, sparkling/brilliant/bright, woody, hollow, and ringing/long decay. However, even among terms that demonstrated overall significance, we can observe that not all instruments behaved in similar ways. For example, for smooth/singing/sweet, the bass clarinet and trombone showed increased mean ratings with higher register (positive relationship). However, flute, oboe, trumpet, violin, and cello were all deemed to be most smooth/singing/sweet in their middle registers but less so in both the low and high register (convex relationship). The bowed vibraphone was exceptional in that it showed almost no difference with respect to this semantic category in relation to register. As a second example, although most instruments decreased on ratings of raspy/grainy/gravelly as register increased (negative relationship), the flute and the oboe were least raspy/grainy/gravelly in their middle registers (concave relationship).

Thus, it may be useful to consider the categories that did not demonstrate significant relationships with register in order to refine our understanding of how these relationships may or may not vary by instrument. For example, all instruments showed little variation on brassy/metallic with respect to register; however, flute, oboe, and cello demonstrated concave relationships, with the middle register rated as least brassy/metallic. The trumpet decreased on brassy/metallic with register (negative relationship), while the violin increased (positive relationship). Although these effects are small, they nonetheless may be compositionally useful in some circumstances. Previously, we noted that mean brassy/metallic ratings demonstrated the least amount of variation among the stimuli and suggested that this semantic category may be more closely related to parameters other than pitch, such as dynamics or playing technique. Another possible contributing factor to this lack of variation might be differences in the understanding between musicians and nonmusicians of the terms “brassy” and “metallic” as applied to timbre. Note that the semantic category was derived from studies involving professional musicians, whereas our current participants were largely nonmusicians, and our musician participants were nearly all amateurs. It is unclear, for example, to what extent knowledge of an instrument’s material influences ratings of relevant terms, including brassy/metallic and woody.

In general, more research is needed to understand potential differences in how musicians and nonmusicians apply semantic terms to timbre. Our consideration of this question in the current experiment is to some extent limited by the imbalance between musicians and nonmusicians in our sample: note that as our research question did not involve differences between these groups, we did not aim to recruit equal numbers. For this reason, musician identity was included in the exploratory modeling procedure as a potential fixed effect. Although the dredge() step of the procedure ultimately did not recommend including musician identity in the final models, earlier steps showed that the addition of musician identity as a predictor significantly improved model fit for 5 of the 20 semantic categories. This suggests that music training may play a role in timbre semantics; however, this preliminary result requires direct testing in future work.

It should be noted that because of potential instrument-register differences, our global results from the models may be in part a product of the particular set of eight instruments that were chosen for the experiment and the extent to which they demonstrated similar trends to the other instruments in the group. That is, they do not necessarily represent global trends across all musical instruments. However, the eight instruments used here are diverse in instrument family and range, providing a solid representation of typical Western orchestral instruments. In future research, it will be necessary to expand the palette of instrumental sounds to determine whether our results generalize across a broader range of musical timbres.

Collecting data online necessarily introduces variability that may affect results, including variation in the quality of listening devices used by participants. To mitigate this, we aimed to recruit a large sample size and incorporated a headphone check at the beginning of the experiment. Although this does not provide control over headphone quality, our results show significant, systematic differences in ratings among stimuli for most of the semantic scales, suggesting that consistent responses can be obtained with variation in headphone quality. Furthermore, as discussed below, a portion of our data replicates data collected in Reymore (in press), a study that was carried out in a sound booth with quality headphones and controlled volume. Both studies collected ratings on the same 20 semantic categories for the oboe across registers, and results are remarkably consistent despite the differences in platform (laboratory vs. online) and participant pool (music majors vs. mixed, primarily nonmusicians).

Reymore (in press) collected ratings across four registers and three dynamic markings for the oboe and French horn. The three dimensions in the current study with the highest marginal R² values (deep/thick/heavy, sparkling/brilliant/bright, and shrill/harsh/noisy) also appeared to be among the most important dimensions for both the oboe and the horn in the previous study. Several other dimensions overlap as important for both instruments in that study and for the assessment across eight instruments in the current study, including raspy/grainy/gravelly, smooth/singing/sweet, open, and resonant/vibrant. Reymore (in press) also shows converging evidence that registral effects on semantic associations can be instrument-specific; for example, the equivalent dimensions to projecting/commanding/powerful and watery/fluid were shown to be significantly affected by register for the horn, but not for the oboe, while woody and muted/veiled were significantly associated with register for the oboe, but not for the horn. Interestingly, the category including the term “nasal” did not demonstrate significance across the eight instruments in the current study but did so for the in press study, for both oboe and horn. More specifically, however, ratings for the oboe are similar across both studies, with the lowest ratings for nasal/buzzy/pinched in the middle registers, suggesting reliability across studies and further suggesting that, as described earlier, it may be the case that we did not observe a global significant effect (across instruments) in the current study for nasal/buzzy/pinched in part because different instruments vary on these terms in different ways with respect to register.

Hayes et al. (2022) further demonstrate that effects of register on timbre-semantic associations can be context-specific. Presented with a frequency modulation (FM) synthesizer, sound designers were asked to create a sound that has more or less of a given semantic attribute (e.g., is rougher or less rough) than a played reference. Participants were then asked to rate the created sound against the reference on additional semantic attributes (e.g., how much thicker or less thick it is). Reference sounds were presented at three registers, but no significant relationship was found between stimulus register and the semantic ratings. It is possible that the use of comparative versus absolute (Reymore, in press; Zacharakis et al., 2014; this study) ratings effectively rendered any registral effects null. On the other hand, their lack might be the result of the specific characteristics of FM synthesis. The introduction of sidebands both above and below the fundamental frequency of the carrier operator during the synthesis task might have falsely implied a lower or higher pitch/register, likely due to interference between pitch height and auditory brightness. In other words, in certain contexts perceptual interactions between pitch and timbre may obscure semantic interactions between the two attributes.

How do our findings compare to the written discourse of orchestral timbre? A number of best-fit adjectives in our study correspond to patterns gleaned from a corpus of orchestration treatises (Wallmark 2019a). For example, in the current study, the mid-register flute was rated highest for smooth/singing/sweet and airy/breathy, while “sweet” and “breathy” are among the most common descriptors for the flute across the orchestration corpus. Low trombone was the most nasal/buzzy/pinched (“nasal” is a top descriptor of the trombone in the corpus), and low cello the most deep/thick/heavy (“deep” is a common cello adjective). Besides these few convergences, the present findings differ considerably from the corpus: in fact, none of the top 10 corpus descriptors for the oboe, trumpet, and violin were replicated in the current study. Perhaps this lack of convergence is the result of the specific way in which register was defined in the current study (Wallmark, 2019a, only examined descriptions at the level of individual instruments); further, treatises were the product of professional musicians, while the present study drew upon ratings from a general global sample.

A more detailed examination that would account for semantics in relation to different groupings of instruments (for example, by orchestral family or Hornbostel-Sachs category) remains elusive at this point and will require more data in future studies. A related question pertains to the extent to which instrument recognition may affect semantic judgments, particularly regarding semantic categories that are nominally associated with an instrument’s material (i.e., woody and brassy/metallic). Although we did not directly measure participant instrument recognition ability, data collected from the attention check questions suggest that between one half and three quarters of our participants were able to recognize instruments to some extent. However, the results do not show evidence that knowledge of an instrument’s material were driving ratings. For example, we did not find instrument to be a significant predictor of ratings of brassy/metallic. From Figure 2, it is apparent that in some cases, we observed the opposite of what would be expected if ratings were based on knowledge of the instrument material. For example, all three registers of the trombone and the high register of the violin received about equal ratings on brassy/metallic, and the high register of the trombone was judged to be woodier than the high register of the violin. As another example, the low register of the flute was considered to be woodier than the low register of the oboe. However, it remains plausible that prior knowledge may affect ratings, possibly more so with respect to the more typical middle register sounds of an instrument, which are likely more easily identifiable. The extent to which instrument knowledge affects semantic judgments may also vary by semantic category; such questions provide an intriguing topic for future research.

The respective contributions of relative register and pitch height also remain to be clarified, though our comparative modeling demonstrates that models using pitch height rather than instrument register to predict semantic ratings explain more of the variance and offer a significantly improved model fit. In general, the pitch height-only models tended to explain around twice as much variance as did the register-only models. This is an intriguing finding that challenges the intuitive assumption that corresponding registers between instruments would be more semantically similar than corresponding notes. This finding contributes to developing understanding of the concept of metatimbres (Soden, 2020), collections of timbres related by shared attributes—in this case, pitch—that may lead them to be grouped perceptually and semantically. Metatimbres may also be organized around other shared attributes such as attack quality, spectral composition, dynamic level/contour, and so forth. Specifically, results from the comparative modeling suggest that it is possible that pitch height may be a greater influence on the perceptual organization of metatimbres than register or instrument. As noted above, however, this post hoc comparative approach has limitations, in that the study was designed using stimuli equally distributed by register but not pitch height, and that the additional degrees of freedom in the pitch height-only models have the potential to increase R². Thus, we understand our finding of the relative strength of pitch height to provide a hypothesis for future experiments, rather than a complete account. Future studies should explore the many possibly relevant parameters for metatimbre by directly testing the relative influences of pitch height, register, and instrument.

Finally, acoustical modeling of listeners’ semantic patterns will help further delineate the nature of semantic interactions between timbre and pitch height/register. For instance, in analyzing spectral envelopes from 50 sustained orchestral instruments sampled across their entire range, Siedenburg et al. (2021) found complex statistical dependencies between both pitch and register and two key correlates of timbre perception, namely the position of the envelope, as modeled by the spectral centroid (Caetano et al., 2019) and spectral shape.

Overall, our results suggest that both instrument type and fundamental frequency have crucial roles in determining a wide range of timbre-semantic associations. While this may seem intuitive and in keeping with the descriptions found in orchestration textbooks, it represents an advancement in the field of timbre semantics research, which has tended to exclude register from the discussion. Further, our study is the first of its kind with the statistical power to detect small effects. Across our group of eight varied instruments, most semantic categories demonstrated a significant relationship with both register and instrument. For some terms, register explained more of the total variance in semantic ratings, especially deep/thick/heavy, sparkling/brilliant/bright, shrill/harsh/noisy, and raspy/grainy/gravelly. Other categories, such as nasal/buzzy/pinched and sustained/even, seemed to be driven more by instrument (and likely other factors) than register. Results also demonstrate that there can be differences in the types of relationships that instruments have with register; for example, the relationship between register and raspy/grainy/gravelly is concave for the flute, with middle register rated lower than the high and low registers, but linear for the bass clarinet, where ratings decrease as register increases.

In addition to the specific findings reported above, we take the present study to be a proof of concept that sound sources, in particular musical instruments, are not necessarily the optimal “units” of timbre semantics in research and creative practice. Common descriptions such as “the oboe is nasal” and “the trumpet is brilliant” are partial accounts, representing semantic nuclei associated with prototypical subsets of the timbres that such instruments are capable of producing. Since each instrument represents “a constrained universe of timbres” (McAdams & Goodchild, 2017, p. 129), each also represents a constrained universe of potential semantic associations, with vast potential for meaningful variation within a single instrument and even vaster potential for meaningful correspondences and interactions with other instruments. Composers and musicians have long tapped into these potentials: the challenge before us as researchers is to come to an explicit understanding of mechanisms that have been implicitly applied for centuries. In this study, we hope to have provided a model of analysis that may be productively applied to this end, which may help to mitigate the significant challenges posed by the complexity of the many-to-many mappings underlying semantic associations, the advanced cognitive processes involved in relating attributes of sound to extramusical domains, and the large degree of intersubjective variation bound to be encountered in any area of musical interpretation. We anticipate that the results, especially the timbral profiles provided by mean ratings of semantic descriptors, will be useful in composition and orchestration as well as for music analysis, both in practice and in pedagogy. They may also be informative for future research in timbre perception—for example, in informing stimuli selection and providing a foundation for conceptualizing relationships between timbre-semantic associations and register across instruments. Results also offer a unique opportunity to begin analyzing the relative contributions of pitch height and register in timbre semantics.

The authors would like to express their thanks to the Analysis, Creation, and Teaching of Orchestration (ACTOR) Partnership for supporting data collection with ACTOR Strategic Project Funding.

Zachary Wallmark is co-founder of Glisten Labs, LLC, a venture that hopes to make commercially available the GlistenIQ experiment design platform, a developmental version of which was used free of charge in this research.