Echolocation is a rare ability of some bats, dolphins, and humans with reduced sightedness or visual impairment. Often visualized as a type of auditory sight, echolocation has no true analog for sighted humans without the ability, resulting in a wide range of interpretations when game designers attempt to capture this subjective experience. Video games have depicted echolocation with varying degrees of fidelity and realism, from musical scales and maps to fully realized three-dimensional worlds. This variety may be attributed in part to the inaccessible experience of the echolocating subjects. Designers must rely on their own subjective experiences to create a mental image of this ability. Synthesizing aspects of acoustic and biological sciences, philosophy, and disability studies, this article examines depictions of echolocation in video games, demonstrating that games require players to incorporate their own experiences in order to bridge the “explanatory gap” between the subjective experiences of visually impaired characters and knowledge of the objective processes of echolocation that are accessible to sighted players. With examples from Ecco the Dolphin (1992) and Perception (2017), this article will show that designers support their echolocation mechanics with narrative and supplementary information rather than actualizing the experience with gameplay.
Introduction
In an influential 1974 article titled “What Is It Like to Be a Bat?,” Thomas Nagel examines the problem of human consciousness by asking readers to consider the perspective of a microbat (Microchiroptera) as it creates a spatial representation of its surroundings through echolocation, the use of echoed sound waves to sense elements of an environment. His purpose in posing this question is to emphasize the subjective experience of the bat as it sends out vocalizations and then reacts to the echoed sound waves as they return. While humans are able to understand the basic physical processes of bat echolocation, the only way for us to consider what it is actually like to be a bat is through our imaginations, and even then our understanding of the true bat experience is only speculative. Nagel emphasizes this idea:
[Bat echolocation] is not similar in its operation to any sense that we possess, and there is no reason to suppose that it is subjectively like anything we can experience or imagine. This appears to create difficulties for the notion of what it is like to be a bat. We must consider whether any method will permit us to extrapolate to the inner life of the bat from our own case, and if not, what alternative methods there may be for understanding the notion.1
As humans we are bound to our own subjective experiences, and our imaginations (active though they may be) still cannot produce an objective knowledge of a bat’s point of view. We are required to bring our own experiences to the table (along with knowledge of the physical processes) to create a mental image that imitates a bat’s means of perception.
While most humans are not able to experience echolocation directly, game designers have still attempted to represent this ability through game mechanics, visual effects, and narrative elements. These efforts result in depictions of echolocation with varying degrees of fidelity and realism; echolocation mechanics in games have provided spatial information through musical scales, maps, and even fully realized three-dimensional worlds. This diverse set of representations may be attributed in part to the inaccessible experience of the echolocating subjects. When a game designer sets out to recreate the concept of echolocation, they are often approaching the design problem from the perspective of a sighted, non-echolocating human being, one who is attempting to understand the subjective experience of the echolocator but who has no firsthand knowledge of this ability.2 Without reference to that experience, the outcomes can vary quite dramatically.
As an example of how far afield representations of echolocation can be, consider Matthew Hubbard’s Dolphin, published by Activision for the Atari 2600 in 1983. The cartridge box for the game sets the scene:
They appeared on Earth 49 million years before we did.
Their brains are large and complex. Their radar system—more powerful than any known to man.
They are dolphins! And now, they need your help. This unique mammal is calling upon you and all her powers to outwit one of the deep’s deadliest predators.
The longer the chase, the more frantic it becomes. Sonar is the Dolphin’s code. A lone seagull, her only hope. And you. Are you listening? Each sound is your cue to act, to guide the dolphin in its treacherous trek, through seahorses and undercurrents, away from danger.
There is much to learn and not a moment to lose. The dolphins await, and so does—the giant squid!3
In this game the player controls a dolphin that is being chased by a giant squid. To safely navigate away from this enemy, the dolphin must pass through small holes in a towering wall of seahorses, shown in Figure 1. These walls appear rapidly from offscreen, giving little time for the player to react, but attentive listeners will still be able to navigate this game world effectively, as the manual for Dolphin emphasizes: “Only by learning the dolphin’s sonic language can you guide her through schools of seahorses, battling a monstrous squid, to gain magic powers from an elusive seagull. Hurry! Lend an ear…”4 This suggestive ellipsis (original to the manual) leads into a description of the game’s primary mechanics. Just before each seahorse wall appears, an audio cue plays that indicates the location of an opening. A high-frequency sound indicates a higher opening in the wall while a low-frequency sound indicates a lower opening, as demonstrated in the gameplay of Video 1.5 Since the highest and lowest seahorses never disappear, there are five openings that can occur in the wall, corresponding to five pitches that serve as cues to the player, shown in Figure 2.6
A screenshot from Dolphin showing the wall of seahorses that the player character must navigate in order to avoid the giant squid.
A screenshot from Dolphin showing the wall of seahorses that the player character must navigate in order to avoid the giant squid.
The five pitches that indicate the location of openings in the seahorse walls from Dolphin.
The five pitches that indicate the location of openings in the seahorse walls from Dolphin.
Gameplay from Dolphin exhibiting the five pitches that serve as cues.
While the choice of these specific pitches may warrant further investigation, the aspect most relevant to our current discussion is the domain in which these signals are being transmitted.7 Using an auditory signal as a cue for an unseen object relates this game mechanic directly to animal echolocation (or biosonar), but the mechanic strays slightly from our common understanding of echolocation since dolphins don’t use differences in pitch to determine the location of objects in the their surroundings, especially not to determine the height of a particular object from the seafloor. This mechanic is not meant to be a faithful reproduction of dolphin echolocation; rather, it is game designer Matthew Hubbard’s artistic representation of an idea that is foreign to his own experience.
When game designers ask “What is it like to be a dolphin?” and choose to rely on their own experiences to answer this question, the mental images they create (and the gameplay outcomes that result from them) carry a vestige of their own perceptual and phenomenal understanding of the world. The subjective nature of experience promotes the diversity we see in representations of echolocation. In this article I will examine these differing depictions of echolocation in video games, specifically focusing on how game designers use echolocation mechanics to consider the subjective experiences of player characters with reduced sightedness, visual impairment, or blindness.8 Video games commonly use visuals to represent the perspective of a subjective agent—for example, first-person games allow players to see through a character’s eyes—but echolocation mechanics intermix sonic and visual effects, pushing the concept of sound toward the forefront when players attempt to engage and connect with the experience of a player character.9 To examine these mechanics, I will begin by discussing the philosophical underpinnings of subjective experience, specifically with regard to game design. I will then consider the mechanical aspects of dolphin echolocation before looking at representations of this concept in video games, with a focus on the Sega Genesis title Ecco the Dolphin (1992). Finally, I will examine depictions of human echolocation and how representations of disability interact with ideas of subjective experience. This article treats the representation of human echolocation as an extension of media depicting human blindness, helping underscore the discussion of subjective experience with its cultural implications.
Depictions of both human and nonhuman echolocation attempt to bridge the gap between the hypothetical experiences of characters with reduced sightedness and the apperceptions of sighted players, but as we will see in the examples of this article, game designers often support their echolocation mechanics with narrative and supplementary information rather than actualizing the experience with realistic gameplay. By sidestepping the responsibility of fully recreating these experiences, game designers require players to actualize the experience themselves, filling in this gap with their imaginations. The idea of “imagination” in this sense is not something unreal, but the imagined statuses and actions of gameplay do not substitute for experiences; rather, they act as simulations of those experiences.10 As Espen Aarseth notes, a simulated experience in a game “is created by the player’s actions and strategies” rather than by a “writer.”11 Relying on aspects of acoustic and biological sciences, philosophy, and disability studies, I will demonstrate that representations of echolocation require players to incorporate their own experiences, ultimately informing them less about the represented characters and more about their individual perceptions.
Subjective Experience and the Player Character
Nagel’s basic proposition in “What Is It Like to Be a Bat?” is that our current models of the physical world are inadequate for describing subjective experiences. He compares the current state of this area of inquiry to the hypothetical status of Einstein’s ideas “if uttered by a pre-Socratic philosopher,” writing: “We do not have the beginnings of a conception of how it might be true.”12 The essential problem in this scenario is what Joseph Levine termed the “explanatory gap” between objective physical processes (such as the neurobiological reactions that induce our sense of pain) and the subjective experiences that they cause in a perceiver (such as the experience of actually feeling pain).13 We can accurately describe actions and operations with our language, but we lack the ability to describe perspectives and experiences with specific terminology. In this section I will examine how video games express and enact this explanatory gap by first detailing the philosophical foundations of the idea before considering how it affects the relationship between players and their characters.
As an example of this explanatory gap, Levine considers the human perception of color, one of the more common case studies in the examination of subjectivity. He writes that the physical nature of color alone does not capture the entire meaning of color in terms of our experience:
Let’s call the physical story for seeing red “R” and the physical story for seeing green “G.” My claim is this. When we consider the qualitative character of our visual experiences when looking at ripe McIntosh apples, as opposed to looking at ripe cucumbers, the difference is not explained by appeal to G and R. For R doesn’t really explain why I have the one kind of experience—the kind I have when looking at McIntosh apples—and not the other.14
In this sense, the physical explanation of the color red does not describe the quale of red, the unique subjective experience of red recognized by the perceiver. To extend this example a little further, consider those who experience red-green color blindness; whereas people with common trichromatic color vision would perceive red and green objects as having separate colors, those with red-green color blindness would experience some reds and some greens as being the same.15 The physical stories for seeing red and green (R and G) do not change outside of the perceivers’ visual mechanisms—the wavelengths of red and green light are still the same—but since the visual mechanisms of those with red-green color blindness are different from those with trichromatic color vision, the same visual information results in different understandings of the colors in question. The phenomenal and linguistic associations that people with red-green color blindness make with particular colors must also differ from those with trichromatic color vision; if some reds and some greens appear to be the same color, then the words red and green and the ideas associated with them take on different meanings. While the physical and biological differences between these two kinds of vision are well understood, the phenomenological experiences of the color-perceiving subjects, their distinct internal senses of those red and green objects, have resisted precise description.
Many philosophers have proposed ideas to bridge this explanatory gap, but they often involve incomplete solutions.16 In a related philosophical endeavor, David Chalmers examines the problem of consciousness and suggests dividing the problem into separate parts by thinking about the mind in terms of different properties that are either physical, psychological, or phenomenal. The connections between these properties can then serve to outline and bridge the explanatory gap in sections. This division does not make the problems of consciousness or subjective experience any easier for Chalmers to resolve, but it does highlight two distinct problem sets in this area of research:
Many books and articles on consciousness have appeared in the past few years, and one might think that we are making progress about consciousness untouched. Often, such work addresses what might be called the “easy” problem of consciousness: How does the brain process environmental stimulation? How does it integrate information? How do we produce reports on internal states? These are important questions, but to answer them is not to solve the hard problem: Why is all this processing accompanied by an experienced inner life? Sometimes this question is ignored entirely; sometimes it is put off until another day; and sometimes it is simply declared answered. But in each case, one is left with the feeling that the central problem remains as puzzling as ever.17
Nagel himself proposes the construction of “an objective phenomenology not dependent on empathy or the imagination” to tackle this problem. He suggests that detailing each of the processes that our minds perform in objective terms can at least shorten the explanatory gap and help define the true difficulties of understanding unique consciousness and subjective experience, but he also admits the shortcomings of such a practice: “One would reach a blank wall eventually, but it should be possible to devise a method of expressing in objective terms much more than we can at present, and with much greater precision.”18 While this process might assist in defining the extent of Chalmers’s hard problem, an objective phenomenology would be fraught with issues from the outset due to the inherently subjective nature of phenomena; it would not be able to solve the problem outright.
Notably, Nagel’s suggestion to exclude empathy and imagination from his method seems at odds with the motives of game designers who attempt to represent subjective experiences through game mechanics and narratives, raising the question: How do game designers’ efforts fit in with these philosophical attempts to resolve subjectivity? While game designers are typically not responding to the philosophical question of consciousness, they are often attempting to answer the question “What is it like to be _____?” That blank can be filled in with almost any personal descriptor (e.g., “a treasure hunter,” as in the games of the Uncharted franchise) or the name of any animal (e.g., the numerous animals that a player can control in the game Tokyo Jungle). Players have little problem assuming control of most humanoid or animal characters and empathizing with their concocted experiences because they can conceptualize that there is (in Nagel’s terms) “something that it is like to be” one of those “beings.”19 While players do not directly perceive these experiences or interact with them firsthand—players generally use an intermediary device to initiate action (e.g., a controller or keyboard) and must rely on an intermediary device to receive stimuli (e.g., a computer monitor or television screen)—they can imagine “what it is like to be” the subject when granted a sense (or even an illusion) of control over the perceptual, cognitive, and responsive capabilities of the characters in question.20
When filling in the blank of this prompting question (What is it like to be _____?), game designers have even stretched beyond the realm of conscious beings, placing objects without recognizable subjective experiences in the role of player character (e.g., a piece of bread in I Am Bread). Taking control of these objects requires not only a sense of imagination but an ability (and willingness) to suspend disbelief.21 In these cases, as with those of human or animal subjects, game designers can use narrative objectives (e.g., becoming an edible piece of toast) to make a player empathize with their character through shared struggles. This is all the more important when the player character has no correlate for the experience beyond the constructed narrative. This empathetic connection and the sense of player authority over game decisions enhance one another, even in situations where the subjective experience is irrational.
Empathy and imagination are integral to the designing and playing of video games, especially those games in which a player controls a dissimilar character.22 Nagel’s suggestion to avoid empathetic and imaginative faculties seems to outline a philosophical focus that is different from our current examination of subjectivity in games. Games can take part in creating a phenomenological model for experiences, but the process of playing a game includes the player’s subjective interactions with the phenomena in the form of empathetic and imaginative responses, responses that are drafted by game designers but enacted and actualized by players. Julianne Grasso describes this interaction as a “co-creative” process, “in which player subjectivity collides with a prefabricated virtual reality.”23 Games can give us an outline of a character’s actions, personality, temperament, and mentality, but engaging a character’s experiences requires players and designers to incorporate their own experiences.
Alternatively, Nagel wants to remove this step; he wants to take part in the subject’s experiences firsthand. In this sense Nagel is looking for a way to explain not just “what it is like to be a bat” but “what it is like for a bat to be a bat”24—that is, not simply for us to imagine its experience or to recreate it in a virtual world but for us to know what it is like for the subject itself to actually experience it. While video games can traverse the explanatory gap to a certain degree, they are not currently equipped to remove our own subjective experiences and replace them with those of another subject.25 Empathy and imagination are required insomuch as they are unavoidable for representing a subjective experience in games, but any attempt that includes these faculties should not be written off as a completely faulty endeavor in the examination of subjectivity. While we would have to make significant breakthroughs in neuroscience (and potentially many other areas) to be able to fully perceive another being’s experience, attempting this feat with our current abilities can offer valuable information as to the extent of the explanatory gap if we recognize and acknowledge that those moments of empathy and imagination are not actual subjective experiences, that they require us to take an additional step. By considering Nagel’s prompting question, we can clarify the role that empathy and imagination play in our interactions with video games and how depictions of echolocation exemplify the explanatory gap.
Dolphin Echolocation and Its Representation in Video Games
Invoking echolocation allows Nagel to illustrate the problem of subjective experience by emphasizing the difference between the ways that bats and sighted humans collect information about their surroundings, but general discussions of echolocation often focus on the similarity between this ability and visual perception, primarily for the purpose of empathetic reference (to allow non-echolocating humans a sense of “what it is like to be” an echolocator). Echolocation is often conceptualized (and represented) as a type of auditory sight, an ability of certain beings to create a map of their surroundings with projected and echoed sounds.26 In some sense, comparisons of echolocation to sight are apt; both echolocators and non-echolocators have to perform complex operations in order to activate the visual cortex of the brain and process its information. Through their infancy, sighted humans have to learn how to comprehend and respond to visual information (through recognition, depth perception, spatialization, etc.), a process that can take years to master. Echolocation similarly engages the visual cortex but uses an elaborate auditory system (detailed in the following paragraphs) to achieve this effect. While both systems involve significant information processing, echolocation requires an auditory acuity that is foreign to the experience of most humans.
Employing echolocation as a game mechanic can offer an additional layer of audiovisual interaction for the player, but for those who have no direct knowledge of the echolocation experience, any contrived gameplay situation involving this ability will also require empathy and imagination by default. Because of this separation between real-world echolocation and its depiction in games, designers make sincere efforts to justify its usage as a game mechanic; they often incorporate background information or elements of lore to situate the player in the gameworld, as is the case for the 1992 Sega Genesis title Ecco the Dolphin and its sequels. By considering the technical aspects of dolphin echolocation, the representation of this ability in the game Ecco the Dolphin, and the formative ideas that led to this gameplay mechanic, this section will demonstrate that game designers attempt to bridge the explanatory gap of dolphin experience through supplementation rather than actualization; designers reinforce the fabricated experience of echolocation in their games with narrative support and supplementary information (often through marketing and manuals), not by recreating an experience precisely. Much like Nagel’s proposed objective phenomenology, games representing dolphin echolocation can only offer partial solutions to the problem of realizing that subjective experience.
Examining the mechanical processes of sight and echolocation can highlight their unique properties. Unlike biosonar, which uses echoed sounds that originate from the subject, sight does not require an animal to actively produce “visual rays” that emanate from and bounce back to the subject’s eyes.27 Rather, visual information is generated from a constant stream of reflected electromagnetic waves that are collected and decoded by the visual system.28 While biosonar requires active involvement on the part of the subject, sight is a relatively passive experience. As you will see in many of the examples in this article, games often require player input for their echolocation mechanics, distinguishing this mode of play from the ability of sight by virtue of an additional interaction—there is rarely a button that you have to push to engage sight in a video game.
While depictions of sight (through various first-person or third-person camera perspectives) are relatively commonplace and even standardized in video games and other media, echolocation has seen a multitude of representations, and rightfully so; echolocation is not a singular ability but a range of abilities used by various species with different biological mechanisms. Concordantly, scientific definitions of echolocation open the concept up to different interpretations and methods of sonic interaction beyond those typically associated with bats and dolphins. For instance, a definition from the bioacoustics researcher Whitlow Au generalizes the idea, stating that echolocation is “the process by which an organism projects acoustic signals in order to obtain a sense of its surroundings from the echoes it receives.”29 Under this definition there are a number of routine human actions that qualify as echolocation. Consider the process of hanging a heavy picture on drywall. You can rap the wall with a knuckle to determine the location of studs; hollower or more resonant sounds (those that echo more) indicate spaces where a nail will not be supported by additional wood framing. While this action is related to the ability of echolocation under Au’s definition, it differs from our traditional understanding of biosonar in both the transmission and judgment of the acoustic signals. The projected signals in this instance are not completely biological in origin; they require a surface that is not part of the human subject’s body to generate the initial sound. Additionally, judging the distance and location of objects in this stud-finding example requires attention to timbre, different from the element of sound seemingly used by echolocating bats and dolphins, time delay.
For a direct comparison, consider the operations involved with dolphin echolocation. Dolphins determine the range and direction of objects by producing clicks that are sent out into the water and echo back to them. As Au writes: “Dolphins produce sounds within their nasal system, and the signals are projected out through the melon.”30 The “bulbous melon” contains “translucent lipid very rich in oil,” sometimes referred to as “acoustic fat,” which transmits the auditory signal.31 The time delay between sound production and echo reception determines the range of an object; the longer the delay, the farther an object is from the original sound source.32 Dolphins sense the returning echoes through bilateral sound receptors in their lower jaws.33 These receptors contain the same lipid used to transmit the signal from the melon.34 Similar to human hearing, dolphins detect differences in intensity between the signals from these sound receptors to determine the direction of an object, but unlike most humans, dolphins are able to perform this task with echoed sound waves that they produce internally.
Conceptualizing the dolphin echolocation experience is an intricate task for humans without firsthand knowledge of biosonar. Rather than using an external sound source (as in the human stud-finding process), dolphins both produce and receive auditory signals with their own biological mechanisms. The acute ability to detect differences in time between transmission and reception of auditory signals may seem humbling (if not baffling) to humans untrained in this form of echolocation.35 To comprehend the echolocation experience of dolphins, a non-echolocating human would have to imagine not only the ability to project sounds from a bulbous, fat-filled organ in their head but also the ability to perceive the echoes of those projections within milliseconds of their creation.
The ability of humans to echolocate has seen a dramatic increase over the past two decades, largely due to the efforts of organizations like World Access for the Blind, but human echolocators can only empathize with the dolphin or bat processes to a certain extent.36 The extensive training required for humans to echolocate does not correspond to the seemingly innate (or at least deep-rooted) abilities of some bats and dolphins.37 Humans can understand the mechanical processes involved with dolphin echolocation precisely (as many of the authors cited in this section do), and they may be able to perform echolocation acts that replicate some of the biomechanical operations of dolphins, but the precise experience of dolphin echolocation will still be unavailable to them, a version of the explanatory gap in action. If humans want to consider the perspective of an echolocating dolphin, this disconnect has to be filled by imagination, a task that many game designers have cultivated and promoted.
The finely tuned abilities of dolphins allow them to generate three-dimensional maps of their surroundings through auditory signals and even differentiate between dangerous objects (such as harsh terrain or predators) and desirable objects (such as food sources or other members of their species).38 This relation of echolocation to mapmaking is evident in a number of video games, most prominently in Ecco the Dolphin. The instruction manual for the game, written by Carol Ann Hanshaw, outlines the different abilities that the player character, Ecco, can use to navigate the gameworld, focusing on the ability to produce what the game designers call “dolphin song.” Hanshaw writes:
Songs are powerful. They can:
• Call to other singers, who will respond with songs of their own. You may get clues, pleas for help or important directions.…
• Ward off the deadly Hungry Ones and other enemies.
• Get new songs, special powers and information from Glyphs scattered throughout the mazes of the sea.39
The manual even goes into detail about “Mapping with Songs,” outlining the function of echolocation in the game:
Songs that echo back to you bring long-range information. This is called “echolocation.” When you hold down Button A, your song reverberates through the currents and caverns, returning to you with a map of your surroundings.40
As the player character echolocates, the map provides information about enemies, health locations, and passages through difficult levels. Hanshaw poetically implores the player to use the echolocation ability when difficulties arise: “Look beyond your eyes with your song.”41 Calling this your song (not Ecco’s song) emphasizes the connection that the designers are making between the player’s experience and the character’s experience.42
The gameplay of Video 2 shows this mechanic in action; Ecco uses his “song,” which propagates outward from his head in the two-dimensional gameworld (Figure 3), to communicate with other dolphins and to map the surroundings for areas of interest. The combination of these two abilities into the singular “dolphin song” results in a simple interface for players, one button to interact with the world through sound, but it does not fully replicate the real-world dolphin experience; dolphins’ echolocating clicks and their whistling communications have different means of production, different sonic effects, and, in general, different purposes.43 Ecco’s song mimics the whistles commonly used in dolphin communication, but due to the electronic medium of the Sega Genesis, the resulting sound is quite similar to that of sonar used for submarine navigation. In this sense, the electronic sound of “dolphin song” combines the concepts of navigation and communication from different domains, that of naval vessels for the former and that of aquatic mammals for the latter.
Gameplay from Ecco the Dolphin demonstrating the dolphin song mechanic.
The combination of these abilities into a single mechanic can be traced back to the main inspiration for the game’s narrative, the 1982 novel Sounding, written by Hank Searls. Ed Annunziata, the lead game designer for Ecco the Dolphin, spoke of this inspiration in an interview with Sef Highwind:
After I had the idea for the play mechanic, I started to study about the dolphins. It was easy to learn the basics from the library and books I bought (no Google back then) but the real inspiration [for] making the creatures sentient was a book I read: Sounding by Hank Searls. This story was from the point of view of a whale. It really inspired me to think of Ecco as [a] thinking, feeling creature. Also, this story inspired me to think more and more about how the dolphin’s echolocation works—so that was added to the main controls.44
Annunziata’s focus on the sentience and perspective of aquatic mammals resonates with Nagel’s examination of the subjective experience of a bat, considering the character not just as an agent of the biomechanical process of echolocation but also as a “thinking, feeling creature.” Similarly, the novel Sounding examines not only the actions of a sperm whale as it navigates the Atlantic Ocean but also its internal thoughts and decisions—that is, not just the “processing” but the “experienced inner life” mentioned by Chalmers.45
Searls details multiple instances of whale navigation and communication in Sounding that map to corresponding mechanics in Ecco the Dolphin. For example, the opening sequence in the book details the process of using natural landmarks to navigate:
Tonight he was blatting, from spring-taut internal lips in his head-case, a loud and repeated peal: “Blang…blang…blang… ” The sound, like a mallet slamming an empty steel drum, would have deafened a man in the water. The sperm whale was echo-scanning peaks of the Mid-Atlantic Ridge below him, listening for reverberations from the walls of an undersea canyon that scored the ocean bottom and was the ninth mark he must fix in his mind before turning southwest.46
The book also describes long-range communications between different groups of aquatic mammals:
He heard the finbacks again, much closer now. He queried them, and this time they heard him.…There were a dozen whales in the pod, and they spoke simultaneously, leaving it for him to sort the input. But they spoke in a code that all whales, and dolphins too, knew well. First they located themselves, spoke of potential dangers, and only then gave him gossip of the sea. When finally it came, he listened indulgently.47
These passages serve as precursors to Ed Annunziata’s “dolphin song” mechanic; they try to capture the thoughts of a sperm whale in those critical moments when sound production and sound reception are the primary means of interacting with the world.
While this novel and the game it inspired both portray the experiences of echolocating aquatic mammals, they do not present those experiences objectively; their renderings of the aquatic mammalian inner life do not satisfy Nagel’s proposition of an objective phenomenology. Rather, they require the reader or player to interpret the described circumstances through their own subjective experiences in order to connect with the characters’ experiences. These representations of echolocation and communication in whales and dolphins require empathy and imagination on the part of the reader or the player, precisely what Nagel is trying to avoid in his examination of “what it is like to be” another being. Sounding and Ecco the Dolphin can only relay their characters’ potential experiences, necessitating an extra step from players.
Game designers are aware of the additional measures that are required for players to have a sincere connection with their protagonists. Beyond relaying information about game mechanics, the instruction manual for Ecco the Dolphin includes references to literature about aquatic mammals and their habitats, even going so far as to incorporate a page-long quote from Jacques-Yves Cousteau’s The Living Sea.48 These elements prompt readers to learn more about dolphins from nonfiction sources. Similarly, the instruction manual for Activision’s 1983 Dolphin includes a section that details the traits of the common bottlenose dolphin (Tursiops truncatus), specifically their speech, hearing, respiration, sonar, maximum velocity, and information about their brains. The manual then encourages readers to seek out more resources on dolphins: “Find out more at your library!”49 Beyond simply supplementing the game with ancillary information, these manuals foster a culture of player exploration outside the gameworld. Game designers offer players a sense of agency by inviting them to assist in this supplementary engagement.
In the cases of Dolphin and Ecco the Dolphin, the game designers seem equally invested in explaining the game mechanics of echolocation and selling players on the experience of actually being a dolphin. By presenting factual information about dolphins alongside desired player actions (both inside and outside the gameworld), game designers develop a specific perspective, a clear motivation, and a means for players to imagine and enact a response. This process of supplementing the gameplay experience may even resolve a portion of Chalmers’s “easy” problem, the connection between the physical and the psychological properties of consciousness by helping players better understand the psychophysiological natures of the characters they are controlling. A solution to the “hard” problem, connecting to the phenomenal properties of consciousness (i.e., “what it is like for a dolphin to be a dolphin”), lies beyond the current scope of the video game medium; supplementary information only serves as a substitute for this solution, still requiring players to bridge the gap with empathy and imagination, but these games at least provide a genuine attempt at understanding and reinforcing the unique perspective of echolocating dolphins, though not actualizing it.
Human Echolocation and Representations of Disability
Advancements in audio and graphics in the 1990s encouraged the development of more realistic environments and promoted new opportunities for player engagement through the reappraisal of traditional game mechanics.50 The mechanics introduced in Ecco the Dolphin, for example, have been adapted for use in more modern games that represent echolocation in three-dimensional virtual worlds. The Sega Dreamcast title Ecco the Dolphin: Defender of the Future (2000) shifts the series from a primarily two-dimensional environment with a side-scrolling perspective to a three-dimensional environment with a third-person trailing perspective. This change in the Ecco the Dolphin series mimics a larger trend in games toward using the camera perspective to produce what Karen Collins calls a “natural subjective viewpoint.”51 Players can still use the same echolocation mechanic to produce the in-game map of the character’s surroundings, but the map projection has also shifted from a side-on view of the ocean to a top-down view that is overlaid on the screen as the main character traverses the gameworld (Figure 4). Whereas the in-game map from the original Ecco the Dolphin provided only a wider view of the side-scrolling gameplay perspective, the overlay map for Ecco the Dolphin: Defender of the Future provides a perspective that differs from that of regular gameplay, one that is not from the viewpoint of the player character.52 Though the map is still a two-dimensional projection, its combination with the third-person trailing perspective situates the player in a different frame of reference, encouraging new gameplay interactions.
Comparison of the in-game maps from Ecco the Dolphin (top) and Ecco the Dolphin: Defender of the Future (bottom).
Comparison of the in-game maps from Ecco the Dolphin (top) and Ecco the Dolphin: Defender of the Future (bottom).
With the ability to generate a natural subjective viewpoint that replicates human vision, three-dimensional gameworlds heavily promote the use of human characters as analogs for players. Consider, for instance, the slew of games released in 1996 that feature human player characters in three-dimensional gameworlds using either a first-person or third-person perspective: Duke Nukem 3D, Resident Evil, Quake, Super Mario 64, and Tomb Raider, among others. Aside from the fixed camera angles of Resident Evil, these games all offer some degree of player control over the camera, allowing the player to enact the thought processes of the character by scanning the terrain for points of interest or information necessary to the continuation of the narrative. This effect is intensified with character animation in some third-person games; for example, the character model for Tomb Raider’s Lara Croft follows the player’s camera perspective in “Look” mode, shifting Lara’s gaze in a corresponding direction, reinforcing the connection between player and character.
This common synthesis of elements in three-dimensional gameworlds (player-controlled camera perspectives, replicated natural subjective viewpoints, and more realistic environments and character models) allows for representations of human echolocation that further connect players to the hypothetical experiences of their characters. Differing from the two-dimensional map representation of dolphin echolocation in Ecco the Dolphin: Defender of the Future, human echolocation is most often depicted with an altered version of the three-dimensional gameworld that replaces the standard color palette and texturing of game objects with a new (often simplified) appearance. Narratively, this effect can appear as a form of technological enhancement when game designers either reduce visibility of the player character’s surroundings or require interaction with an invisible game object, forcing them to rely on some novel intervention, an item or ability. One example of this type of intervention is the Echo Visor in Metroid Prime 2: Echoes (2004), which has two related but distinct abilities. First, the item allows the player character, Samus Aran, to detect sounds within inaudible frequency ranges and locate their sources (invisible sonic emitters) to unlock new areas of the game. When obtained, the Echo Visor allows Samus to both hear and visualize previously imperceptible sonic markers. Second, the Echo Visor boasts a form of three-dimensional sonar, which allows Samus to sense her surroundings using propagated sound waves (Figure 5). While the game can be completed without using this sonar ability—only the sonic detection of the Echo Visor is necessary—this early representation of a first-person echolocation experience (with a monochromatic color palette and simplified object textures) heavily influences later depictions of human echolocation in video games.53
Since the fictional technology in Metroid Prime 2: Echoes requires suspension of disbelief by default, the issue of realizing the character’s inaccessible subjective experience, what it is like to be Samus Aran, may seem of little import; players must already imagine a world in which this echolocation technology exists, so it seems fitting that they should engage their imaginations (and incorporate their own lived experiences) when encountering this representation, thus negating their ability to connect with the experience (though fictional) on its own terms. Some recent games, however, have attempted realistic portrayals of human echolocation in three-dimensional worlds, using player characters who are blind or have some visual impairment. Perception (2017), a horror game from The Deep End Games, and Blind (2018), a virtual reality game from Tiny Bull Studios, represent echolocation using a first-person perspective that is visually and functionally comparable to Metroid Prime 2: Echoes but with player characters who are required to use this ability to visualize and navigate the gameworld.54 The narratives of these games do not rely on electromechanical forms of sonar to enact echolocation; both Perception and Blind incorporate a mechanic that requires player characters to tap a cane against surfaces and objects in the gameworld to produce auditory signals and thereby illuminate the character’s surroundings, a visual representation of echoing sound waves.
Since many blind people use canes for the purpose of mobility, the designers may have considered this form of gameworld interaction to be fitting for a blind player character, but the association of cane navigation with echolocation mischaracterizes both of these practices to an extent. Mobility canes, often called white canes, generally allow users to detect their surroundings through the sense of physical touch, not through sound; the hand transduces vibrations and pressure created by cane interactions with objects and surfaces, allowing the user to process the information and react accordingly. Alternatively, human echolocation often follows the basic practices of bats and dolphins; human echolocators are primarily trained to project clicking sounds from their mouths and attend to the echoes that return to them.55 The combination of these two actions into a singular gameplay mechanism, tapping a cane to echolocate, is similar to Ecco the Dolphin’s combination of communication and echolocation abilities, simplifying player actions into a single press of a button, but in this case the simplification results in a misrepresentation of the experience of most blind people.
This imprecise representation is especially vexing in the case of Perception since its creators tend to focus on authenticity in their discussions of the game’s design philosophy. In a series of interviews with Ryan McCaffrey of IGN, the game’s creative director, Bill Gardner, noted the efforts to understand and represent the experience of being blind. The quotes that follow represent a peculiar mix of understanding and misunderstanding, a quality that has become representative of this game’s production and marketing:
I spent my final semester of my Master’s degree studying blindness and the different tools available.…I spent days with my phone set to accessibility mode, where it turns everything to text-to-speech. I also did things like doing many daily tasks with my eyes closed.…I interviewed many blind people, spoke to a number of accessibility experts, and tried to learn as much as I could. It’s an ongoing process, but it’s been fascinating. For every answer, there are ten more questions. You don’t know how little you know until you start to try to understand something.56
This method of learning has been roundly criticized by Erin Hawley, who notes that the “blindfold for a day” approach is “an unsuccessful way to gain any insight into blindness and navigating the world with low vision.”57 This measure of success may seem to hold true, considering Perception’s flawed representation of the ability with their echolocation mechanic.
Bill Gardner’s stated intent with Perception also indicates a certain level of misrepresentation inherent in the game’s design:
My goal has always been to try to create the feeling of being a blind person who uses echolocation.…Now, it’s a game, and of every media, games are perhaps driven the most by visuals. As such, I take liberties for the sake of the game. Early on in development, your echolocation revealed a lot less detail in the world. As we put it in front of players, they rejected it. Many found it too frustrating. So we scaled it back and began to show more detail. It will continue to be an iterative process throughout development, but Perception was never meant to be a simulation. We have to make something that people really want to play. In that sense, I think we have a very strong balance that both captures some of the feel of what it might be like to use echolocation while still remaining “fun.”58
Gardner seems caught between competing desires to recreate the “feeling of being a blind person” and to appeal to his intended audience of sighted players by not fully simulating the blind experience, an attempted balance of representation and entertainment that is difficult to accomplish and potentially damaging to the represented group.59
Some writers praise Perception’s representation of blindness, including Mieke Dix, who describes crying in eager excitement when she first came across the game. She writes: “It wasn’t until I became partially-sighted that I realised how little positive representation there is of disability in multimedia.”60 Others focus on the inherent problems of using blind characters for the purposes of entertainment; Joseph Bein from Out of Sight Games frames this depiction of blindness as a form of commodification rather than as a representation:
Any time a blind protagonist features in a game for sighted players, which blind gamers can’t play, you are essentially commodifying the experience of blind people. You are taking a population that is already underserved, and instead of making a game they can enjoy, which is a very rare thing for them, you’re instead using their experience to make a game that sighted gamers (who have plenty of games made just for them) can enjoy.61
Perception can seemingly be considered from two opposing perspectives, one that deems the representation to be incredibly important for the blind community and another that regards it as exploitative.62 To paraphrase a leading question from the epilogue of Dana Plank’s dissertation, do games that feature blindness so prominently “succeed in fostering empathy” for the represented group, or do they “merely exploit disability and reduce it to an interesting commodity that can set a game apart in the market and increase profit for the developer?”63 And further, does the answer to this question depend on the game’s representation of the experience, or does it depend on the player’s own experiences as they encounter the game? The question of subjective experience and its representation seems inherent to Perception’s design. Considering this, I would like to conclude by examining the context of this game’s echolocation mechanic, its representation of blindness, and its framing of disability in the genre of horror.
Horror, Blindness, and Echolocation in Perception
Perception participates in a long history of artistic works in the horror genre that use disability to accompany (or provoke) situations of fear or terror. Despite the game’s stated motive of representation, depictions of disability in horror media largely contribute to an atmosphere of othering. Neil Lerner notes that “the entire horror genre repeats narrative patterns that see the continued repression and oppression of anything defined as nonnormative.”64 As indicated by Angela Smith, representations of disability in horror films have the “potential for social disruption in the visibly deformed body of a singular monster,” that is, the potential to cause viewers to confront their own preconceptions about disability, but they are most often seen to “play out a titillating but temporary sexual threat to female bodies, and to conclude safely with the monster’s expulsion and a return to social norms,” the most classic example being Frankenstein (1931), which Smith details in her opening chapter.65 Horror games have similarly used disability to invoke sensations of thrill, most commonly in the survival horror genre where zombies (or some comparable stand-in) generate waves of excitement as they appear onscreen in larger and larger numbers through a game’s progression. Diane Carr notes that these sensations require a sense of disturbed boundaries between the player character’s body and the “abject bodies” being represented.66
While Perception includes some elements of this brand of horror in the form of the primary enemy in the game, known as the Presence, whose appearance mixes monstrous qualities with a looming human stature, the game’s representation of disability generally stands in contrast to these more sensational examples. Perception attempts to deny any othering of the disabled body by placing the disability on the player character (Cassie Thornton) herself, but Cassie’s depiction is inherently entangled in the complex history of representing blindness in media. As mentioned by Lerner in his analysis of Pride of the Marines (1945), films often fetishize blindness by using recurring tropes such as “face-feeling” and the “wooden stare,” both mentioned by Lauri Klobas in her catalog of representations of blindness in film and television media.67 Klobas notes that while “blindness is the most frequently seen physical limitation onscreen,” the elements to represent it are “often misused and misunderstood,” as with Perception’s cane-tapping echolocation mechanic.68
At its most basic level, Perception engages the genre of horror not through shock or depictions of the grotesque but through a game design that enhances a common childhood anxiety, fear of the dark. Bill Gardner emphasizes the connection between the suppression of visual information, the echolocation mechanic, and the horror genre:
I’m a huge horror fan and am always thinking of ways to bring a fresh angle to the genre. When you think about it, information is the enemy of horror. And so, the notion of finding a way to strip away information and make that into an actual mechanic seemed like a natural fit. It’s rare that you can find narrative, aesthetic and gameplay unite around one single idea.69
Amanda Gardner, Perception’s lead writer, also highlights “fear of the dark” and “fear of the unknown” as drivers for the game’s design and narrative.70 Historical psychoanalytical literature, which heavily influenced and occasionally utilized the horror genre, links fear of the dark, sometimes called nyctophobia or scotophobia, with the fear of losing one’s eyesight.71 Sigmund Freud describes this link in his discussion of Ernst Jentsch’s concept of “the uncanny” and E. T. A. Hoffman’s “Der Sandmann,” writing that “the fear of damaging or losing one’s eyes is a terrible one in children” and that “many adults retain their apprehensiveness in this respect, and no physical injury is so much dreaded by them as an injury to the eye.”72 Freud associates fears of “going blind” with fears of blind people themselves via the uncanniness of their ability to navigate the world despite being unable to visualize it.73
This connection between blindness and horror (through fear of the dark) is played out in a number of modern video games that use forms of echolocation, including Dark Echo (2015), which depicts sounds as slowly moving linear emanations that bounce around a darkened two-dimensional game world to generate a map of each level, and Lurking (2014) and Stifled (2018), which use microphone input to illuminate three-dimensional environments. All of these games make use of enemies that can detect sounds produced by player actions, a situation that discourages players from overusing the echolocation mechanic and in Perception’s case hinders the attempted representation of blindness.74 Perception’s desire to instill a sense of authenticity in its design, however imperfect its depiction of echolocation may be, and its integration of entertainment through the genre of horror result in a unique gameplay experience not found in its peers.
Perception emphasizes its commitment to authenticity from the outset of the game; the opening title card even claims that the story is “inspired by true events.”75 This realist approach is enhanced by supplementing the primary echolocation gameplay with secondary mechanics that support the representation of the blind player character. These mechanics involve the use of assistive technologies and mobile applications for people with visual impairment. Cassie Thornton gathers information using the Delphi Text-to-Speech app and connects with sighted users on the Friendly Eyes app to help her assess situations in which echolocation is not effective. Edwin Evans-Thirlwell notes the importance of this type of representation in comparison to more extravagant options: “This ludic engagement with everyday practicalities contrasts the sensational role played by disability aids in blockbusters such as Deus Ex: Human Revolution or Call Of Duty: Black Ops 3, where protagonists are endowed with bionic limbs and enhancements.”76 These appeals to realism resemble the efforts of the designers of Dolphin and Ecco the Dolphin, justifying the use of an echolocating character by supplementing the mechanic with context about the experience of being the character, but the designers of Perception choose to augment gameplay rather than simply provide ancillary information.
Perception’s promotion of a realist aesthetic and its emphasis on visual information necessitate a depiction of echolocation (through a combination of sight and sound) that captures both the sensory effects of this ability and the sensational affects caused in the subject. Cassie can tap her cane to both illuminate surrounding objects and produce a resonant sound associated with the object being struck. This process is demonstrated in the tutorial for the game shown in Video 3, a flashback to Cassie’s first experience with echolocation. Any sound source casts a wave of monochromatic light onto the gameworld, representing the perceived sound waves echoing off of nearby objects. The illuminated objects resonate momentarily before eventually shifting back into darkness, fading from Cassie’s memory after a few seconds. This mechanic serves a significant ludic purpose, increasing player uncertainty by withholding information about the character’s surroundings in moments of complete silence.
Gameplay tutorial from Perception introducing Cassie’s echolocation ability.
In order to create a convincing gameworld using this mechanic, audio director Jim Bonney produced hundreds of cane tapping sounds that vary depending on the targeted surface, some of which are demonstrated in Video 4. Bonney notes that these different sounds provide “a sense of the way Cassie would be experiencing” the location, indicating a desire to connect not just to the visual representation of Cassie’s ability but also to its sonic qualities.77 Notably, this design replicates the “stud-finding” process described above, rapping a knuckle on a wall to hear differences in timbre, quite different from the uniform clicking sounds typically used by human echolocators. This action and its sonic results emphasize the game’s purpose as a form of entertainment for sighted individuals, not as a simulation of human echolocation as it is more commonly practiced.
Demonstration of cane tapping sounds on different surfaces in Perception.
Bonney’s attention to this timbral experience extends to environmental sounds, which are primarily used to generate unsettling audiovisual effects as Cassie explores Echo Bluff, the mysterious house that serves as the primary setting for the game. An example of this effect is given in Video 5; a nursery mobile slowly chimes out the melody to Brahms’s “Wiegenlied,” revealing the environment in waves of sound and light that are broken up by complete darkness. Since sounds can stimulate an echolocator’s visual cortex, the conglomeration of sonic and visual effects supports the game designers’ attempt at replicating the human echolocation experience, but environmental sounds and cane taps are unlikely to produce information about surroundings with the same fidelity as self-produced clicks.78
A nursery mobile playing Brahms’s “Wiegenlied” in Perception.
Perceptions’s desire for realistic representation in its audiovisual effects carries over into the modeling of objects and characters, but some elements of the gameworld are so detailed that the echolocation mechanic seems to extend beyond realism into hyperrealistic territory. For example, the screenshots of Figure 6 display textures that are unlikely to be as defined for real-world echolocators as they are in the gameworld. Firsthand descriptions do not indicate that human echolocators would be able to discern the details in the pattern of wallpaper, the grain of a piece of wood, or the individual fronds of a fern.79 These overly detailed objects and surfaces attempt to project realism without realizing—that is, making real—the perceptual faculties of echolocators.
Perception’s depiction of echolocation is an artistic interpretation, filtered through the experiences and imaginations of a sighted development team, not an actualization of the experience of being an echolocator. “Perception was never meant to be a simulation,” but it may also fail “to create the feeling of being a blind person who uses echolocation” as Bill Gardner intended. Despite the three-dimensional gameworld, superior graphics, and the sense of player perspective and control, the game can only approach the explanatory gap but never fully cross it. This is not to say that Perception (or any other game) does not offer a compelling answer to Nagel’s unanswerable question about subjectivity, only that the answer still requires players to account for this gap, filling it in with their own experiences and imaginations. Games can be part of an extended cognition in which objects external to our own bodies, whether three-dimensional virtual worlds or controllers or computer screens, affect our cognitive actions, becoming part of what Andy Clark and David Chalmers call our “extended mind.”80 As Grasso writes, “Being in the world, adopting new environmental paradigms with particular interactive affordances, involves a necessary extension of the bodily self into a world on screen.”81 We may consider the visual, sonic, and tactile properties of a game as part of our cognitive process, but they will not be able to fully substitute for the experiences they represent.
Conclusion
What is it like to be _____? As the examples from this article demonstrate, video games have a limited ability to answer this question. The representation of echolocation in Perception is imperfect not just because the game mischaracterizes some of the actions and sensations of echolocators but because the medium itself is incapable of reproducing the inner life of any being or individual. Recreating the subjective experience of dolphins is similarly fraught with issues; dolphin echolocation has a complex evolutionary history that is only more complicated in the modern world, and no amount of supplementary information or narrative structuring will remove a player’s preconceptions about this ability to experience it firsthand.82 Games are not simulations, nor are they magical corrective lenses that allow us to experience other perspectives without prejudice; rather, they are mirrors that elucidate our prejudices.
Instead of simply responding to Nagel’s question, we should consider what it means for us to ask it. The wide range of interpretations that game designers produce when they ask themselves this question should cause us to consider subjectivity more closely, not in reference to the represented character but to ourselves as creators and consumers of these representations. What does it mean that these interpretations vary so widely not just from each other but from real-world experiences? When game designers attempt to replicate a subjective experience, they give players a glimpse into their presuppositions, and they potentially provide players a sense of their own. Why do we, as game designers, have to replace palatal clicks with cane taps? Why do we, as players, need to see the precise details of objects and surfaces in a gameworld? Why do we reduce complex abilities and experiences to simpler, digestible forms? Answering these questions can lead to a form of empathy that video games cannot fully attain with fabricated perspectives and experiences alone; considering our own experiences and how they affect our unique perspectives can cause us to (at least) recognize the experiences and perspectives of others.83
Echolocation is one of many concepts that can be used to examine subjective experience in games, but no matter the mechanism that is being interpreted, there will still be an explanatory gap that players must fill with their own experiences. Considering this, we don’t have to stop asking the question “What is it like to be _____?” We just have to be honest about how effectively video games can answer it.
Notes
Thomas Nagel, “What Is It Like to Be a Bat?,” Philosophical Review 83, no. 4 (1974): 438.
Several tools have emerged to change this trend and grant people with visual impairment (and potentially human echolocators) the ability to design video games. See Viktor Stadler and Helmut Hlavacs, “Blind Adventure: A Game Engine for Blind Game Designers” (presentation, Chi Play 2018, Melbourne, Australia, October 29, 2018). Also see Chella Ramanan, “Sable: A Tool Helping Blind Gamers Make Their Own Games,” Game Industry News, February 22, 2019, accessed January 12, 2021, https://www.gameindustry.com/editorials/eurofiles/sable-the-tool-helping-blind-gamers-to-make-their-own-games/.
Cartridge box for Dolphin, 1983, Activision, English, Atari 2600. Note that the mention of “radar” is not technically accurate when discussing echolocation since radar uses radio waves instead of sound waves.
Instruction manual for Dolphin, 2.
The game also includes other sonic cues, including a “Danger Claxxon” for when the dolphin’s “energy charge is dwindling” and a “Thud” to indicate that the dolphin has “bumped into seahorses,” among others. The manual catalogs these cues in a visual audio chart that provides sound wave representations for each sound. See the instruction manual for Dolphin, 4.
The notes of Figure 2 indicate the best approximations of the pitches for the sonic cues, but the actual frequencies vary slightly from standard twelve-tone equal-tempered tuning. See Karen Collins, Game Sound: An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design (Cambridge, MA: MIT Press, 2008), 21–22. The “sonic echoes” (as described in the game’s manual) use the sound and resulting pitches associated with distortion values 12 and 13 in the TIA (Television Interface Adaptor) code for the Atari 2600. I am grateful to the pioneering (unpublished) work in this area by Glenn Saunders and Eckhard Stolberg. Collins also cites Stolberg’s research in her discussion of the Atari 2600’s tuning capabilities.
As a side note to this discussion, consider these interesting facts about the collection of pitches in Figure 2: (1) the pitches all fall into only two pitch-class categories, A and E; (2) all of the four intervals between consecutive pitches are so-called perfect intervals, specifically the perfect fourth, fifth, octave, and eleventh; (3) all of these intervals are different from one another and increase as the scale progresses upward. The combination of these facts results in a modified subharmonic series (with an added perfect eleventh) that aligns well with the game’s underwater setting.
The term “player character” refers to a character directly controlled and (to a degree) inhabited by a player. See the discussion of “inhabitable protagonists” in Katherine Isbister, How Games Move Us: Emotion by Design (Cambridge, MA: MIT Press, 2016), 11. While there are examples of non-player characters who use echolocation, e.g., the “clicker” enemies in The Last of Us (2013), this article focuses on echolocating player characters in order to highlight the subjective relationship between human participants and fictional agents. This study also considers the intent of game designers who promote a connection between players and their inhabited characters, resulting in what Michelle Grosser calls the “avatar/player,” where a “player identifies with a character in the game and constructs a relationship in which they virtually coexperience the game with the avatar.” See Michelle Grosser, “Avatar/Player Subjectivity: An Agential Analysis of Crypt of the NecroDancer,” Journal of Sound and Music in Games 1, no. 3 (2020): 2, accessed June 15, 2021, https://doi.org/10.1525/jsmg.2020.1.3.1.
Dolphin’s use of an audio cue without visuals for its echolocation mechanic stands out among representations of this ability. As demonstrated in the examples of this article, many games focus on visual effects to recreate the experience of echolocation. While there may be some potential of reproducing the echolocation experience in audio-based games, members of this genre, such as Papa Sangre (2010) and The Nightjar (2011), often use atmospheric sounds as cues for gameplay but not to represent echolocation.
Johan Huizinga writes of “actualization by representation,” which allows a player’s imagination to become “reality” even if the imagined statuses and actions do not exist outside the boundaries of the game. See Johan Huizinga, Homo Ludens: A Study of the Play-Element in Culture, trans. Richard Francis Carrington Hull (London: Routledge & Kegan Paul, 1949), 14.
Espen Aarseth, “Genre Trouble: Narrativism and the Art of Simulation,” in First Person: New Media as Story, Performance, and Game, ed. Noah Wardrip-Fruin and Pat Harrigan (Cambridge, MA: MIT Press, 2006), 52.
Nagel, “What Is It Like,” 447.
Joseph Levine, “Materialism and Qualia: The Explanatory Gap,” Pacific Philosophical Quarterly 64 (1983): 354.
Levine, “Materialism and Qualia,” 357.
Following Nagel’s examination of consciousness and subjectivity, a number of philosophers have used color blindness to evaluate the premise of subjective experience. For an example of this, see Peter Smith and Gregory McCulloch, “Subjectivity and Colour Vision,” Proceedings of the Aristotelian Society, Supplementary Volumes 61 (1987): 245–81.
Notably, there are some philosophers who do not foresee a resolution to this problem. See Colin McGinn, “Can We Solve the Mind-Body Problem?” Mind 98, no. 391 (1989): 349–66.
David Chalmers, The Conscious Mind (New York: Oxford University Press, 1996), xi–xii.
Nagel, “What Is It Like,” 449.
Nagel writes that “an organism has conscious mental states if and only if there is something that it is like to be that organism—something it is like for the organism.” See Nagel, “What Is It Like,” 436.
Katherine Isbister aligns this sense of control with Mihaly Csikszentmihalyi’s concept of flow: “Well-designed games, with the control they offer users over actions in a novel world, readily engage players in a flow state.” Isbister notes how this sense of control affects flow state and player engagement in a number of games, including Journey (2012). See Isbister, How Games Move Us, 4 and 120.
Johan Huizinga’s concept of a temporary play-world (or a “magic circle” that surrounds the ideas and materials of gameplay) allows for this suspension of disbelief to occur. See Huizinga, Homo Ludens, 10.
Isbister echoes this sentiment, noting that games even “play a powerful role in creating empathy.” Isbister, How Games Move Us, xvii.
Grasso notes: “This process of interactivity and understanding in games reflects a dual agency between players and designers.” See Julianne Grasso, “Video Game Music, Meaning, and the Possibilities of Play” (PhD diss., University of Chicago, 2020), 60.
Nagel, “What Is It Like,” 439.
This ability is currently confined to the realm of science fiction. One popular example can be found in the Rick and Morty episode “Mortynight Run,” which features a virtual reality game, Roy: A Life Well Lived. The fictional game replaces the memories of the player with those of the game’s title character, Roy, and allows the player to experience and determine the events of Roy’s life, continuing until the character’s death (the player’s game over). The player loses a sense of self while playing the game and takes on the subjective experiences of the character.
Many writers refer to “acoustic images” produced by echolocation. For an example of this type of description, see Gareth Jones, “Echolocation,” Current Biology 15, no. 3 (2005): 487.
The concept of “visual rays” projecting from the eyes aligns with descriptions of vision leading back to Euclid’s Optics. This theory, called extramission, was furthered in later philosophical and scientific writings, including works by Augustine of Hippo and Roger Bacon. See David C. Lindberg and Katherine H. Tachau, “The Science of Light and Color, Seeing and Knowing,” in The Cambridge History of Science, Volume 2: Medieval Science, ed. David C. Lindberg and Michael H. Shank (Cambridge, UK: Cambridge University Press, 2013), 485–511. Notably, the idea of visual rays is similar to the process of ray casting in computer graphics, but the application of this idea to real-world vision is not furthered in modern scientific writings.
See Richard Haglund, “The Properties of Light,” in Springer Handbook of Lasers and Optics, ed. Frank Träger (New York: Springer, 2007), 3–32.
Whitlow W. L. Au, “Echolocation in Dolphins,” in Hearing by Whales and Dolphins, ed. Whitlow W. L. Au, Richard R. Fay, and Arthur N. Popper (New York: Springer-Verlag, 2000), 364.
Au, “Echolocation in Dolphins,” 365.
Whitlow W. L. Au and Roderick A. Suthers, “Production of Biosonar Signals: Structure and Form,” in Biosonar, ed. Annemarie Surlykke, Paul E. Nachtigall, Richard R. Fay, and Arthur N. Popper (New York: Springer-Verlag, 2014), 66.
Brock M. B. Fenton, Frants H. Jensen, Elisabeth K. V. Kalko, and Peter L. Tyack, “Sonar Signals of Bats and Toothed Whales,” in Biosonar, ed. Annemarie Surlykke, Paul E. Nachtigall, Richard R. Fay, and Arthur N. Popper (New York: Springer-Verlag, 2014), 13.
Whitlow W. L. Au, “Echolocation in Dolphins with a Dolphin-Bat Comparison,” Bioacoustics 8, no. 1–2 (1997): 138.
Au and Suthers, “Production of Biosonar Signals,” 66.
The discriminating abilities that are required for bats and dolphins to make functional use of echolocation have caused some scientists to refuse that the process is actually possible. See Jones, “Echolocation,” 488.
For information about World Access for the Blind and human echolocation training, see Daniel Kish and Jo Hook, Echolocation and FlashSonar (Louisville, KY: American Printing House for the Blind, 2016).
While humans must be trained in order to meaningfully excite the visual cortex with sonic stimuli, echolocating bats instinctively supplement their vision with echoed sounds. For an example of this innate sense, see Manfred Kössl, Cornelia Voss, Emanuel C. Mora, Silvio Macias, Elisabeth Foeller, and Marianne Vater, “Auditory Cortex of Newborn Bats Is Prewired for Echolocation,” Nature Communications 3, no. 773 (2012). The innateness of echolocation in toothed whales is still an unsettled matter, but their biological mechanisms are uniquely predisposed to this ability.
Au, “Echolocation in Dolphins,” 137–38.
Carol Ann Hanshaw, Instruction manual for Ecco the Dolphin, 1992, Novotrade/Sega, English, Sega Genesis, 11.
Hanshaw, Instruction manual for Ecco the Dolphin, 12.
Hanshaw, Instruction manual for Ecco the Dolphin, 12.
This connection resonates with Michelle Grosser’s “avatar/player” and the “coexperiencing relationship” that the player has with their player character through “an experience of virtual subjectivity.” See Grosser, “Avatar/Player Subjectivity,” 13.
While the separation of dolphin clicks and whistles into different functional categories is commonly accepted, echolocation clicks have been shown to serve communicative functions in some circumstances. For references to this effect, see Fenton, Jensen, Kalko, and Tyack, “Sonar Signals of Bats,” 42–43. Regardless, clicks and whistles seem to (at least) serve different communicative functions. The combination of all sonic interactions into one game mechanic in Ecco the Dolphin effectively simplifies the player action.
Ed Annunziata quoted in Sef Highwind, “Sef’s Interview with Ed Annunziata, Game Designer of Ecco the Dolphin,” Gaming Furever, April 3, 2013, accessed January 12, 2021, https://web.archive.org/web/20151105145927/http://www.gamingfurever.com/editorials/727-sef-s-interview-with-ed-annunziata-game-designer-of-ecco-the-dolphin.
Intriguingly, the novel reinforces the connection between human experience and that of a thinking, feeling sperm whale by alternating chapters between the perspective of a human character using an electromechanical form of sonar in a submarine and the perspective of the whale character using biosonar.
Hank Searls, Sounding (New York: Random House, 1982), 3.
Searls, Sounding, 15–16.
Marketing for Ecco the Dolphin also used Cousteau’s likeness, most notably in a television commercial with an actor portraying Cousteau and claiming that the game allows players to experience the ocean’s “danger, mystery, and beauty all through the eyes of a dolphin.” This emphasis on recreating the dolphin experience is continued in advertisements for the direct sequel, Ecco: The Tides of Time (1994), in which a goldfish dreams about being a dolphin.
Instruction manual for Dolphin, 6.
See Dominic Arsenault, Pierre-Marc Côté, Audrey Larochelle, and Sacha Lebel, “Graphical Technologies, Innovation and Aesthetics in the Video Game Industry: A Case Study of the Shift from 2D to 3D Graphics in the 1990s,” GAME 2, no. 1 (2013): 79–89.
This viewpoint typically involves the use of a first-person perspective, but as Collins notes, the third-person trailing perspective “can lead to greater extension into the game space” by virtue of its ability to display the character’s immediate surroundings. See Karen Collins, Playing with Sound: A Theory of Interacting with Sound and Music in Video Games (Cambridge, MA: MIT Press, 2013), 52. Collins cites research by Laurie N. Taylor to support this position. See Laurie N. Taylor, “Video Games: Perspective, Point-of-View, and Immersion” (master’s thesis, University of Florida, 2002).
The echolocation map in Ecco the Dolphin: Defender of the Future allows players to see beyond the hazy depiction of the environment (shown in Figure 4), which replicates the effects of water turbidity, a property that limits vision over long distances and makes echolocation all the more useful for real-world dolphins. See Jones, “Echolocation,” 484.
Notably, the simplified object textures recall the vector graphics of some early video games, but the alignment of simplified textures with echolocation is not prevalent until after Metroid Prime 2: Echoes.
The first-person perspective is enhanced in the case of Blind by virtue of the title being exclusive to virtual reality consoles.
The use of clicking sounds is the primary method of echolocation promoted by Daniel Kish of World Access for the Blind. This is not to say that mobility canes have not been used for a similar purpose, only that the use of clicking sounds is the predominant practice in the community and that it replicates the familiar echolocation practices of bats and dolphins. Merry-Noel Chamberlain has written about her experience of using a cane for echolocation purposes, but the practice is quite different from the use of internally produced clicking sounds. See Merry-Noel Chamberlain, “Echolocation with the Long White Cane,” Future Reflections 32, no. 1 (2013), accessed January 12, 2021, https://nfb.org//sites/default/files/images/nfb/publications/fr/fr32/1/fr3201tc.htm.
Bill Gardner, quoted in Ryan McCaffrey, “Ex-BioShock Devs’ Perception: Designing Horror around a Handicap,” IGN, January 21, 2016, accessed January 12, 2021, https://www.ign.com/articles/2016/01/21/ex-bioshock-devs-perception-designing-horror-around-a-handicap-ign-first.
Erin Hawley, “On Perception and Authentic Disability Representation,” The Geeky Gimp, July 2, 2017, accessed January 12, 2021, https://geekygimp.com/on-perception-and-authentic-disability-representation/.
Bill Gardner, quoted in McCaffrey, “Ex-BioShock Devs’ Perception.”
Jed Whitaker’s review of Perception expresses this sentiment, stating that the game is neither “fun” nor “insightful.” Jed Whitaker, “Review: Perception,” Destructoid, May 25, 2017, accessed January 12, 2021, https://www.destructoid.com/review-perception-436946.phtml.
Mieke Dix, “Why Perception’s Representation of Disability in Video Games Is Important,” Cultured Vultures, October 29, 2016, accessed January 12, 2021, https://culturedvultures.com/perception-disability-in-video-games/.
Bein became aware of Perception at the Game Developers Conference when his wife pointed out that the game’s booth used braille characters on its signage, a stylization of the title that is replicated in most promotional materials for the game and another example of elements of the blind community being used for the entertainment of sighted people, not for the accessibility of blind people. Joseph Bein, quoted in Hawley, “On Perception and Authentic Disability Representation.”
This aligns with Ann Fox’s conclusion that “the gains disability representation makes happen over a long time, and within the context of much back-and-forth, much stress and strain” and that “performing Disability Studies means at once understanding that representations…should be seen in context, and understood to be inherently messy, contradictory, sometimes immensely dissatisfying, yet never completely without worth as cultural images.” See Ann M. Fox, “Scene in a New Light: Monstrous Mothers, Disabled Daughters, and the Performance of Feminism and Disability in The Light in the Piazza (2005) and Next to Normal (2008),” in The Oxford Handbook of Music and Disability Studies, ed. Blake Howe, Stephanie Jensen-Moulton, Neil Lerner, and Joseph Straus (New York: Oxford University Press, 2016), 793.
Dana Plank, “Bodies in Play: Representations of Disability in 8- and 16-Bit Video Game Soundscapes” (PhD diss., The Ohio State University, 2018), 307.
Neil Lerner, “The Horrors of One-Handed Pianism: Music and Disability in The Beast with Five Fingers,” in Sounding Off: Theorizing Disability in Music, ed. Neil Lerner and Joseph N. Straus (New York: Routledge, 2006), 86. Lerner’s focus on horror films allows him to examine “broader cultural anxieties about disability” as they appear in mass media.
Angela M. Smith, Hideous Progeny: Disability, Eugenics, and Classic Horror Cinema (New York: Columbia University Press, 2011), 24. In order to uncover some latent historical perceptions of disability in horror film, Smith frames these representations as an attempt by the genre to “grappl[e] with the politics and aesthetics of disability representation,” not as essentially othering. See Smith, Hideous Progeny, 29.
In her discussion of “abject bodies,” Carr considers the Necromorphs in Dead Space (2008), whose “bodies mix grotesque swellings, or rapier-like appendages, with bare, human feet,” and notes that “by combining hostile alien sentience with human body parts, they ignore any border between the human and the alien, and they override the boundaries between individual human subjects.” Diane Carr, “Ability, Disability and Dead Space,” Game Studies 14, no. 2 (2014), accessed January 12, 2021, http://gamestudies.org/1402/articles/carr.
Neil Lerner, “Hearing a Site of Masculinity in Franz Waxman’s Score for Pride of the Marines (1945),” in The Oxford Handbook of Music and Disability Studies, ed. Blake Howe, Stephanie Jensen-Moulton, Neil Lerner, and Joseph Straus (New York: Oxford University Press, 2016), 861.
Lauri E. Klobas, Disability Drama in Television and Film (Jefferson, NC: McFarland, 1988), 2–3. Lerner cites this passage and Klobas’s specific discussion of Pride of the Marines.
Bill Gardner quoted in Victor Ren, “Deep End Games Founder Bill Gardner Talks Perception, a New Horror Project,” Game Skinny, May 31, 2015, accessed January 12, 2021, https://www.gameskinny.com/dxjme/deep-end-games-founder-bill-gardner-talks-perception-a-new-horror-project.
Amanda Gardner quoted in McCaffrey, “Ex-BioShock Devs’ Perception.”
Note that many of the findings in this historical literature are opposed outright by modern psychologists, but these early writings still have an effect on contemporary depictions of horror.
Sigmund Freud, “The ‘Uncanny,’” in An Infantile Neurosis and Other Works, vol. 17 of The Standard Edition of the Complete Psychological Works of Sigmund Freud, ed. James Strachey, trans. Alix Strachey (London: Hogarth Press, 1953), 231.
In true Freudian fashion, he extends this association from fear of the dark, fear of the unknown, and fear of blindness to fear of “being castrated.” See Freud, “The ‘Uncanny,’” 231.
By denying a blind player character her ability to navigate and requiring her to hide from an enemy in situations where she takes such liberties, Perception presents a narrative of overcoming blindness not only as a disability but as a lifestyle. The game even frames itself with the overcoming narrative in its final moments, displaying a quote attributed to Helen Keller just before the credits roll: “Although the world is full of suffering, it is also full of overcoming it.” Neil Lerner and Joseph Straus provide a basic outline of overcoming narratives: “The person with a disability…overcomes the impairment, as perceived by people without disabilities, to relieve them of their own anxieties about experiencing similar fates.” See Neil Lerner and Joseph N. Straus, “Introduction: Theorizing Disability in Music,” in Sounding Off: Theorizing Disability in Music, ed. Neil Lerner and Joseph N. Straus (New York: Routledge, 2006), 2. Julia Miele Rodas notes that “the pervasive image of the disabled overcomer ultimately harms people with disabilities by fortifying unrealistic expectations of what they can and must do and by denying the real limitations of bodies and environments in which they live.” See Julia Miele Rodas, “Overcome and Overcoming,” in Encyclopedia of American Disability History, ed. Susan Burch (New York: Facts on File, 2009), 691.
Statements like these are common in horror media whether the claim is genuine (as in the case of Jay Anson’s 1977 novel The Amityville Horror) or not (as in the case of the 1999 film The Blair Witch Project and its related marketing materials). Only at the end of Perception (in another intertitle series) do players learn about the origin of this claim: “Perception was inspired by Susannah Martin, an ancestor of Director Bill Gardner. Susannah was accused of witchcraft twice before being found guilty and hanged in 1692. We dedicate this game to anyone who has been misunderstood or misjudged. To anyone who has been underestimated. And to anyone who has been told they can’t do something.”
Edwin Evans-Thirlwell, “‘It’s Ingrained in Humanity to Be Scared’—How Games Are Challenging the Taboos of Disability and Mental Health,” GamesRadar, August 11, 2018, accessed January 12, 2021, https://www.gamesradar.com/its-ingrained-in-humanity-to-be-scared-how-games-are-challenging-the-taboos-of-disability-and-mental-health/.
Jim Bonney and Bill Gardner, “Ex-BioShock Devs’ Perception: Using Audio as Gameplay,” IGN, January 27, 2016, accessed January 12, 2021, https://www.ign.com/videos/2016/01/27/ex-bioshock-devs-perception-using-audio-as-gameplay-ign-first.
Kish and others have noted the benefit of palatal clicks for echolocation due to their “intensity and uniformity.” See Juan Antonio Martínez Rojas, Jesús Alpuente Hermosilla, Pablo Luis López Espí, and Rocío Sánchez Montero, “Physical Analysis of Several Organic Signals for Human Echolocation: Oral Vacuum Pulses,” Acta Acustica united with Acustica 95, no. 2 (2009): 328.
The images in Figure 6 are reminiscent of the “radar” sense appearing in media from the Daredevil franchise, including panels from the comic book series and scenes from the 2003 film and the 2015 television series. Daredevil’s self-described abilities (in the comics) combine elements of sonar and radar, incorporating ambiguous “sensations” that allow for more precise mobility. First-person representations of the character’s experience as a blind crimefighter also deviate from descriptions of real-world echolocation. Depictions of superhero characters with disabilities often emphasize heightened senses and can misrepresent the lived experiences of people with disabilities. Notably, the Daredevil character has been used to subvert narratives of overcoming disability and is cited by José Alaniz as being (at one time) “the most progressive image of disability in superhero comics.” See the discussion of the 1985 Daredevil stand-alone story “The Price” in José Alaniz, Death, Disability, and the Superhero: The Silver Age and Beyond (Jackson: University Press of Mississippi, 2014), 49–51.
Clark and Chalmers cite David Kirsh and Paul Maglio’s examination of player movements in Tetris, which take two distinct forms: pragmatic actions, for which players rotate or translate a tetromino piece (or tetrimino) in order to drop it into a particular position, and epistemic actions, for which players rotate or translate a piece to help visualize its potential position. Clark and Chalmers propose that epistemic actions imply an “active externalism” that links the “human organism” with an “external entity,” which “jointly govern behavior.” They indicate that the external entity is critical to the functioning of the system: “If we remove the external component, the system’s behavioural competence will drop, just as it would if we removed part of its brain.” See Andy Clark and David Chalmers, “The Extended Mind,” Analysis 58, no. 1 (1998): 7–9. For Kirsh and Maglio’s original work on Tetris, see David Kirsh and Paul Maglio, “On Distinguishing Epistemic from Pragmatic Action,” Cognitive Science 18 (1994): 513–49. Julianne Grasso also cites Clark and Chalmers in her discussion of “Play as Cognitive Process.” See Grasso, “Video Game Music,” 40–42.
Grasso, “Video Game Music,” 41.
The effects of anthropogenic activities on dolphin habitats and behaviors have been well documented in recent research. Increased vessel noise has even resulted in modified dolphin vocalizations. See Leila Fouda, Jessica E. Wingfield, Amber D. Fandel, Aran Garrod, Kristin B. Hodge, Aaron N. Rice, and Helen Bailey, “Dolphins Simplify Their Vocal Calls in Response to Increased Ambient Noise,” Biology Letters 14, no. 10 (2018), accessed January 12, 2021, https://doi.org/10.1098/rsbl.2018.0484.
As Nagel writes: “The subjective character of the experience of a person deaf and blind from birth is not accessible to me.…This does not prevent us each from believing that the other’s experience has such a subjective character.” See Nagel, “What Is It Like,” 440.
University of California Press