Reflections on Sound Associations and Sonic Digital Environments

This research resulted from the process of making an interactive sound exhibition.² It made use of the HTC Vive as a user interface for the audience and it distributed multichannel sound in a loudspeaker array. The HTC Vive controllers were used as the main interface for interaction with the “digital sound environment,” and the HTC Vive headset was kept as a complementary option to visualize the data of that environment (because the system would not run without the headset anyway). In order to discuss ideas for the installation and develop the sounds that would compose it, we explored the process through which one sound suggests another, or is associated with another (sonic associations). This was exercised in different ways among us: We tried to create a stream of consciousness by sharing sounds among ourselves without contextualization (and answering back through other sounds without nomination). We tried to explore our own references and memories, listening to long sessions of white noise with slight frequency filtering, etc. We also went to the gallery where our composition would be installed and recorded the space individually, to then share one’s perspective with the others. During this process, we encountered a well-known riddle in our discussion: “If a tree falls in a forest and there is no one there to listen…does it make a sound?”

With this in mind, it became clear that the concept of interactive sound installation often relies on action-reaction feedback systems.³ How would one sound lead to the other in an interactive way but still follow the logic of the associations? It seemed to us that there would be a limited set of possible states and events defined by the author’s intentions and expressed through utilized technologies. Then these would be unfolded based on the audience’s interaction. Clearly, this is not exclusive to sound-art, as participation in art has been encouraged since the 1960s;⁴ and it falls in line with the Kockelman-Peircean idea that “any interaction is a semiotic process,”⁵ that is, a process based on action-reaction-action, or on signs versus interpretants. This is also Kohn’s argument, that the whole world is semiotics, which should include entities beyond the human—an idea also long explored by Haraway ever since her multispecies manifestos, which recently culminated in the proposal of Chthulucene: that is, an engagement to stay “with the trouble of living and dying together on a damaged earth.”⁶

However, this concept of interactive installation also seems to lie on either a false feeling of flexibility (the illusion that the composition is open) or on a completely random experience (the fact that the composition is not fixed). In other words, “a Gramscian hegemony: having no choice about the field of options within which one may freely choose.”⁷ In this sense, how can the installation follow a pattern of sound associations in such an open environment? How would one sound lead to the other in this associative way, but still be dependent on the visitor’s input? This raises questions of agency in general, which in fact are also part of the riddle itself; and of residential agency in particular. Accordingly, “residential agency describes the degree to which one may control the expression of a sign, compose the relation between a sign and an object, and commit to the interpretant of this sign-object relation.”⁸

Furthermore, the riddle acquires a different formulation if it is considered in the context of the digital tools used for sonic production: How can the tree fall in a virtual environment (even if just a simple DAW—a digital audio workstation), does it make sound, and who is the listener? Because in this environment, the tree only falls if someone makes it do so, or only sounds if someone designs it so. Is the programmer/sound designer the witness? Does the experience become less physical? Less anthropocentric? Did it actually depend on physicality and/or on human perception? By the same token, the sound associations would only follow a certain pattern (in the installation) if we made them so—in spite of the participant’s actions. This would be the same as forcing poiesis to become esthesis; that is, assuming that the creative processes that generate the work will be the same processes that the receivers perceive as the work.

With this new context in mind, the research followed the different elements in the riddle in order to arrive at its formulation in a digital environment. This process would inform the terms in which these sonic associations could occur in such a digital context—because, in the end, that would be the presentation’s format: a virtual sonic exhibition based on sound associations within which the listener could navigate and interact. Hence, the tree in the digital environment is analogous to the sonic associations, because both premises presuppose witnessing a path that corroborates the experience. Working on an interactive path of sonic associations contended possible answers to the riddle, while the riddle itself enlightened the possible choices for the installation.

On this account, the riddle is hereby approached from four different angles: the falling tree, which discusses the basis of perception; the thought of the tree, which addresses sonic associations and objects of thought; the sound of the fall, which puts forward the topic of sound object and reduced listening; and finally the digital tree, which comprehends the digital domain as inseparable from the actual experience. In this way, this thought experiment will move toward an exploration of sound associations in a digital environment.

Does a falling tree in the forest make a sound when there is no one to hear it? This “thought experiment” addresses fundamental questions about sound, perception, and elementary experience of the world. The term thought experiment originates in the German Gedankenexperiment and was first used by Hans Christian Ørsted.⁹ It is a process of speculation based on a conceptual hypothesis. This one in particular encompasses multiple interpretations, from different fields such as physics, philosophy, and psychology. Although from a certain perspective sentience is not a requirement for events to happen, this question suggests that events are relational in their nature and that the sentience (or sensorial interfaces of any kind) has a crucial role in the event’s existence—or at least, in the understanding of that existence. Is sound not a sound, if not perceived? Is the tree’s occurrence dependent on human acquaintance? How does this work?

From Berkeley’s perspective, there are no trees in the park nor books on a shelf unless there is a mind to at least contemplate them.¹⁰ In other words, thinking of things without a mind to think of them is simply impossible due to the lack of what Kockelman calls “mediating relationality.”¹¹ For instance, sound events might be understood from the perspectives of their various elements like pitch, timbre, loudness, or duration and their effects on the neural networks, but these individual aspects will not explain the miscellaneous hearing experience, including spatial awareness.¹² The environment seems not to be a composition of individual constituents, but most likely it expresses itself as a network of numerous interconnected and mutually influencing actants of which the experiencing sentience is part. If the falling tree is experienced, its sonic expression cannot be accessed solely, ripped out of its context or surrounding environment, as a figure isolated from its background. It is a ground combination of symbols, indices, and icons (as the Peircean semiotic process mentioned above, where ground refers to the kind of sign relationship between sign and object)—which therefore should include a variety of interpretants: affective, energetic, and representational, each of those including their ultimate versions.¹³ In this case, the sound of the falling tree would be an index (contiguity) of its fall. Besides, in the context of a virtual environment, that would be the residential agency as mentioned above.¹⁴

Additionally, according to James, there are two processes of thought, empirical thinking and reasoning. Empirical thinking is reproductive only, while reasoning is productive. Associative processes are placed (or drifting) in between the two and contain elements of both. They depend on a vast number of factors such as the subject’s past experience, attention, and expectation. Additionally, “some one brain-process is always prepotent above its concomitants in arousing action elsewhere.”¹⁵ In the meantime, Carlson uses different classifications for mental processes, where associations are a part of a higher-level or a complex mental process while very close to the process of learning and reasoning.¹⁶ These associations are unpredictable: Each input creates a certain neural path and connection in the brain, for as much as certain structures of neurons and nodes are connected to each other by synapses. Accordingly, “by the synapse-state theory, mental life is embodied in excitations in a neural structure composed of synapses, neurons, and nodes.…The excitations originate in the sense cells […,] are transmitted along the neurons, through the synapses, and are summed and distributed in the nodes.”¹⁷ In this sense, synapses have a variable sensitivity threshold, which becomes higher each time it is being used.

In this way, these connections learn to be more reactive than the ones not being used. That is, “when two elementary brain-processes have been active together or in immediate succession, one of them, on reoccurring, tends to propagate its excitement into the other.”¹⁸ An association is then an overlap of these neural paths and their synapses with the highest threshold. The neural paths used more often are biased to reconnect again—they are more conductive. Each sensory input causes a series of excitations of the brain.¹⁹ These excitations always occur in groups, as a sum of elements that are inseparable, and conceive one object of thought. In other words, “however complex the object may be, the thought of it is one undivided state of consciousness.”²⁰

Consequently, the question of attention is at stake. Attention is always selective, and it only works in relation to perception, not sensation: In every sensation, there are many sensory inputs received, but not perceived. Attention can be both voluntary and involuntary: Voluntary attention is connected to the distinction and focus on the perceived object (for example, face recognition); while involuntary attention reduces the number of objects perceived. In other words, “voluntary attention enhances the perceptual representation whereas involuntary attention affects the tendency to respond to stimuli in one location or another.”²¹ According to neurophysiology, there are at least two stages of filtering sensations in the brain on their path to perception.²² The moment of involuntary attention occurs in each sensory input. Without this filtering, it would be impossible for the human mind to deal with the number of sensations. In spite of the level of subjective idealism, James clarified:

In line with that, the unconscious inference is the immediate connection between the partial sensory input and what the mind offers as its cause.²⁴ That is, “our ‘perceptions,’ or recognitions of what objects are before us, are inferences of this kind. We feel a patch of color, and we say ‘a distant house,’ a whiff of odor crosses us, and we say ‘a skunk,’ a faint sound is heard, and we call it ‘a railroad train.’”²⁵ Unconscious inference stands for empirical thinking and analysis stands for reasoning, but they may use the same mechanisms as both look for reason. In more complex cases, conscious approaches come first, such as analysis and abstraction. The received object is being split so that some of its qualities become abstracted. Through that, essential characteristics can be recognized.

However, because an unconscious inference is one of the few moments in which there is no time to divide the perceived objects into their elements, the object is examined as a whole. And so the unconscious inference provides a hint about the brain’s tendency to acquire the whole object out of the individual elements. For Bradley, identity and contiguity are the base for associations. So there is a process of reintegration in which “any part of a single state of mind tends, if reproduced, to reinstate the remainder; or any element tends to reproduce those elements with which it has formed one state of mind.”²⁶

In that sense, neither sensation nor perception can be understood as an individual entity; it is a rather complex entirety of that thing, its environment, and all the connections it contains in itself. As “the elementary experience generated in perception is not a question of a recognition of the core of the thing perceived, but one of the experiences of the thing as embedded in the fringe of its relations.”²⁷ Within those determinations, it is possible to construct hypotheses attempting to describe such particular occurrences, but their stances on experiencing reality will always be reductive. The more the theories try to unfold the basics of events, the more they become estranged from intuitive understanding.²⁸ Besides, if the focus is on the sensation of hearing and the feeling of the sound vibration, how can the rest of the senses be left out of this experience and consequently the mental process behind it and associated with it? As Don Ihde recalls, “Xenophanes voiced the note that experience in its deepest form is global: ‘It is the whole that sees, the whole that thinks, the whole that hears.’”²⁹ Perception is multimodal.

Considering the objects of associations, James’s approach is tied in with subjective idealism, but it is also grounded in the neurophysiology of perception. Instead of approaching the sensory input, it treats the object of thought that is being associated from/to another one of its kind—in spite of being invoked by the sensory input. For example, let’s say that the sound of rain is associated with the smell of wet dirt. In this case, it is not the actual sensory input of the heard sound that is being associated, but an object of thought that this sound becomes in one’s mind. And this object of thought (the sound of rain) is then associated with another object of thought (the smell of wet dirt). As he also claimed, “in fact, the ‘objects’ of our perception, as trees, men, houses, microscopes, of which the real world seems composed, are nothing but clusters of qualities which through simultaneous stimulation have so coalesced that the moment one is excited actually it serves as a sign or cue for the idea of the others to arise.”³⁰ This also corroborates Kockelman’s elaboration of Peircean semiotics, that “any semiotic process has three components: sign, object, interpretant. There are three kinds of sign-object relations, or grounds: iconicity (quality), indexicality (contiguity), and symbolism (convention).”³¹

Furthermore, even though the objects of thought are usually not classified, they can be also distinguished between directional and bidirectional sequences. In bidirectional sequences, object A associates with object B and object B associates with object A. In directional sequences, object A associates with object B, but object B does not associate with object A (rather, object C).³² The object of thought is possibly modifiable by will, due to a step of attention between the sensory input and the source object of thought. So far, it is not acknowledged that this possibility could be generalized to all objects of thought. But if the source is to be modified, then its effect and result are also modified indirectly. As it is possible to make the sound of the rain the main attribute of the source object of the thought, then it should also be possible to make an associated object of thought to be based on sound.

In the example mentioned above, the sound of rain would rarely happen as a sole sensory input because it is a whole set of sensory inputs throughout all the senses in typical circumstances. They may vary in intensity, but all senses are present. In fact, the sound of rain might not be objectively the strongest aspect of the sensory input, but through the means of attention, it can be perceived as the crucial one. Thus, the object of thought is established in one’s mind, which then triggers synapses with the highest sensitivity threshold. Such invokes an association to the object of thought, which will then be called the smell of wet dirt. Here the emphasis is on what will be called the smell of wet dirt. First of all, because it is not a scent, it is an object of thought. Second, because calling this object of thought the smell of wet dirt does not explain what it really is.

Therefore, an object of thought is a complex sum of elements, but strictly abstract. Describing it through words is problematic and inaccurate because it reduces it to an idea of its essential characteristics, which does not necessarily define the experience thereof. However, it is still necessary to verbalize and describe what is perceived, in order to analyze the sensations from a psychological or social point of view (instead of neurophysiological). These empirical perspectives can clarify the processes of perception themselves but not much about their content. For the latter to be understood, it would be necessary to address it with dedicated vocabulary. Besides, languages have a certain degree of flexibility, and one word might denote a rich set of quite heterogeneous processes and attitudes to understand them.³³ For this reason, this discussion should be centered on sound phenomenology rather than language semiotics.

From the intuitive mechanistic perspective, a sound is an event independent of the perceiver; but, as Sterne outlined, it seems that language itself renders this independence a fallacy. For example, the prefixes infra and ultra are used to describe phenomena of the physical vibration that resides out of the aural range of the human body.³⁴ Yet there are many possibilities to construct an infrasound experience within human audible capacities, such as an acoustic beat caused by the small difference in frequency of two oscillators playing simultaneously. In addition, the sensory boundaries are considerably individual and nonpersistent in time. Everyone hears differently and with age, the sensitivity of the hearing apparatus significantly decreases. In fact, the average range of the human senses still delimits the language of scientific categories and how listening is addressed.

The term sound describes the change of pressure moving through a physical substance. The sensorial mechanism translates the physical pressure into an electrochemical signal, which will be further processed by sentience. Without these, there would be no sensual experience of sound. In other words, sound is an expression for a somewhat conventionalized and circumscribed process, an embodied experience of a certain kind. As Sterne puts it, “We can say either that sound is a class of vibration that might be heard or that it is a class of vibration that is heard, but, in either case, the hearing of the sound is what makes it.”³⁵ And yet, paraphrasing phenomenological studies, vision is the objective sense.³⁶

In fact, the concept of acousmatic listening addresses precisely the disposal of vision. In sonic practices, the term acousmatic refers to a sound of which one does not see the source. When Pierre Schaffer adopted the term into the foundation Musique Concrète, it was believed to have its origins in Pythagorean Greek mythology.³⁷ For Schaeffer, the act of listening should focus on the sound itself (the sonic traits) instead of its source or meaning. This core approach of concrete music is based on the idea of a sonorous object, in which “an ‘intentional unit’ [is] constituted by our own mental activity.”³⁸ The objet sonore is “an organized totality that one can assimilate into a ‘gestalt.’”³⁹ In other words, the idea of “intentional unit” turns every fragmentable piece of sound into a “sound-object,” which interweaves with James’s definition of “object of thought” as an act of consciousness. Hence, a sound object is an object of thought.

Moreover, many approaches to listening also deal with voluntary and involuntary attention. For example, deep listening is an attempt to extend attention (what is perceived) to the auditory input as much as possible.⁴⁰ It bases perception, the object of thought, as much as possible only on the sonic environment. Along the same lines, reduced listening is a process of directing attention to the auditory input only, dismissing causal reasoning.⁴¹ Both concepts share the principles of acousmatic listening for trying to manipulate unconscious inference. In a way, reduced listening is an attempt at creating its own object of thought, or at least at controlling the chain of processes that leads to it—for the sound should not be perceived in the complexity and multitude of an object of thought as unconscious inference would do, but should be consciously poured down to sonic input only. It is a specific perspective over a general experience, given that, for reduction to occur, “there must be a division of the experience itself.”⁴²

But, as Demers points out, “reduced listening and object-hood both call on the observer to accept materials uncritically, to reflect only on their role within the artwork rather than within the world at large.”⁴³ In this sense, analysis would allow for joining new labels and connections to a perceived object (although not as far as reasoning would). Drawing from Kane, she asserts that “this uncritical acceptance is inherent to the phenomenological method itself, which secures ‘an a priori ontological foundation, but the supposed benefits of such a foundation are attained at the expense of historically sedimented ‘residual signification.’”⁴⁴ By the same token, the process of acousmatic listening itself enlightens much of this object of thought insofar as both are based on conceptual principles that cannot be truly materialized: For listening to a sound within this “thought experiment” implies perceiving it as “the sound of a tree falling,” while the principles of concrete music propose to liberate the sounds from nominative identification, particularly due to the limitations already mentioned above.

In line with this, the thought experiment of the tree falling could be enlightened if de-contextualized into other environments. In the same way that Schaffer de-contextualized sound by recording it and taking it into a studio, what if we take the idea of the tree and/or the forest into a digital environment?

In the contemporary digital domain, does the falling tree make a sound in a virtual environment when there is no one interacting with it? The context of computer games, for example, frames this question within the software’s development and its usage within a network of mediating interfaces and computing devices.

The first stance of the falling tree in a virtual environment presupposes that the game is fully developed but not being played yet. All the content is compressed into many hidden options—possibilities that temporize in the future and may or may not become actual experiences. It is their virtual state: The game world dwells on the computer’s hard drive, often in an obfuscated binary state. The content is there, but the relations of individual bits are scrambled and only a certain path or sequence of actions will render them meaningful. The data is literally inaccessible to the player by any other means than entering the executive mode of the game. Playing the game unfolds its preconstructed possibilities, transforming the data into an actual relatable experience of the falling tree. That applies also to the games that implement the concept of a “persistent world” that is again only mere data when not shown to the user. In line with that, to perceive this data it is necessary to translate the information encoded in electrical charges into a tangible domain of flesh and bone. Mediating interfaces such as loudspeakers and displays suit this purpose.

When the player encounters the falling tree, it will happen in the first stage as a computation. Depending on the player’s interaction, the computer will execute commands that decide what to translate and how and where to deliver the resulting signals. From the user’s perspective, the signals and their meanings are only understandable in the form of bodily experience; for example, as a physical vibration emanating from the loudspeaker’s membranes. The user’s physical experience then closes the feedback loop of human computer interaction: It translates virtual into actual. Without these interrelated interfaces—computing devices, output peripherals, matter, time, and embodied sentience—the data will never become sound as a lived experience, it will never become a part of “lived space.”⁴⁵ In fact, “Massumi reminds us that there is actually no such thing as digital sound, whether generated on or off a computer; if it is audible, it must be analog. Digital code is audible only after it is transduced into sound waves.”⁴⁶ In other words, the user’s experience of digital environments is inextricably tied to a broad network of physical actants that make it possible. As such, it can be seen as an extension of our physical surroundings—an effort of mankind to adapt one’s affordance of the environment in order to suit it.⁴⁷

The second stance concerns the process of fabrication of the game that aims to exhibit the hypothetical falling tree. There also, the question of an ongoing reality independent of the sentience is of no importance. Digital means are a complex creation of human thought. Every possible content expressed through them is a fabricated product of the mind. The development process is a reversed version of the previous mode of software operation. In this phase, the creator/programmer folds individual media files into a constructed game environment with the intention to induce some specific experience. During this procedure the creator has to constantly switch their perspective into the one of the player, to verify that the construct fulfills its purpose, that it induces an acceptable spectrum of reactions.

To simplify, in the context of computer games, the creator and the user have diverse possibilities to access the content of the incipient digital world. The fabricated data can be assembled to form relational structures that will interact with input data provided by the future user. The creator can only assume that the functional structure will express itself as a falling tree, which, in its turn, will make a sound as an experience of the user.

From the perspective of contemporary science, uncertainty and contingency seem to be fundamental features of nature itself. In fact, “first phenomenology is a philosophy of presence.”⁴⁸ In other words, there is no solution for this “thought experiment” besides questioning its possibilities. It is rather about understanding what the question implies, more than seeking a closed answer. On the one hand, there is no universal rule to unfold the fundamental truth behind this question, because there is nothing like it, except the uncertainty of affirmations.⁴⁹ On the other hand, the falling tree experiment provides an applicable basis for examining the vibrating world of sound. It reveals an anthropocentric relation with something that should not be reduced to what is heard or who heard it but should be extended to what might be heard and who else is there too. The question implies a sense of witnessing, which itself can have multiple interpretations, particularly if not reduced to the human auditory range.

For example, any form of notation is a symbolic representation of a communication system (code, sheet music, etc.). If one understands that system, one can perceive its meaning. This perception itself is an experience, perhaps not of the thing per se but nevertheless related to it. And, in fact, there are multiple ways of experiencing a tree if it fell in the forest, as much as there are many trees and forests, and multiple beings that can experience it in spite of humans acknowledging it. The issue with this subjective idealism is, of course, a dependency on the self—the listener—as experiencing the tree falling in the forest is subjective. And in the end, if one considers multiple realities, dimensions, or environments, the question is ultimately if there is a tree, if it fell, if there is a forest, if there is sound. In this specific case, the existence of the tree itself should not need validation. The tree should subsist, it is the tree that falls and makes sound—or not. As Kohn puts it, it is its own response.⁵⁰ Perhaps no one will perceive it, perhaps someone will, but it is up to the tree to make its own sound.

***

This study originated at the Academy of Performing Arts in Prague as part of the Digital Unrealities project, supported by special-purpose funds intended for the specific tertiary education research provided by the Ministry of Education in 2018.