Ludomusicology: Normalizing the Study of Video Game Music

The creation and development of a research area that would permit the examination of video games in their specificity was the primary objective of the first theorists that dedicated themselves to the study of video game products. Academic interest in digital games began in the last decades of the twentieth century, first within the fields of anthropology and cultural studies and much later as part of work on multimediality and hypertextuality. At the beginning of the twenty-first century, a new autonomous discipline began to be visible; with its broader focus, ludology, or game studies, is gradually becoming more established, thanks to the appearance of publications, research groups, and academic events focused on analyzing specific problems in studying video games and other ludic products.² The first works investigating video game music can also be framed within this context of searching for academic independence; from the start, they declared it necessary to develop specific tools for the study of a new form of sonic expression, one that cannot be explained solely through the analysis of written musical notation or its phonographic reproduction.

Trying to break into a new field of research, such as the study of video game music, meant meeting the challenge that the recognition and consideration of the academic community entailed. When in 1999 Matthew Belinkie wrote his seminal historical overview on the music of video game products, he saw it necessary to begin by claiming the seriousness of an object of investigation that, until then, had been considered a simple complement to an ordinary juvenile entertainment product; in fact, the very title of his article alluded to the value of video games as an object of study (“Video Game Music: Not Just Kid Stuff”).³ Karen Collins, one of the pioneers of ludomusicology, remembers that in the inaugural academic articles dedicated to the study of video game music, published in the first years of the twenty-first century, it seemed necessary to always include a preamble to justify, with facts and figures, the relevance of the video game industry in terms of economic value, demographics, and cultural impact.⁴ Added to this were the difficulties that any researcher who wanted to work on digital products had to face in a radically different context from the one that we are in today. In a recent article, Collins reflected on the obstacles that she herself had to overcome in the early years of the twenty-first century when she began her research for Game Sound, considered to be the first academic monograph dedicated to the theory, history, and practice of music and sound design in video games:

Both the professional and academic worlds at that time seemed very unreceptive to the study of video game music. Collins relates the difficulties she had to deal with in the early years of her work:

Two decades later, the researchers who work on video game music no longer need to dedicate a considerable amount of their time to justifying the value of their field. In the United States as well as a good part of Europe, ludomusicology has begun to be normalized in academia, the result of substantial earlier work to disseminate and edit monographs, edited collections, and journal articles; these have contributed to laying the theoretical bases and defining lines of research beginning to consolidate.⁷ Also, holding conferences and other academic events has facilitated the creation of contact networks between researchers in different countries. In 2012 the Ludomusicology Research Group, originally formed by the researchers Michiel Kamp, Tim Summers, and Mark Sweeney in the United Kingdom, organized at the University of Oxford the first academic conference dedicated specifically to video game music. This event was the root of a conference that has been held in different parts of Europe under the generic title of Ludo. Two years later, at Youngstown State University (in the United States), the North American Conference on Video Game Music (NACVGM) took place. Both initiatives have been repeated annually since then. Thanks to the momentum of these academic events, at the end of 2016 the first international society dedicated to the study of music and sound in video games, the Society for the Study of Sound and Music in Games (SSSMG), was founded. The founders of this organization were members of the Ludomusicology Research Group and the organizers of NACVGM and the Audio Mostly conference, an international event that, since its first meeting in 2006, has also focused part of its interests on the study of sound in video games. The growth of this society, which currently has more than two hundred members from all over the world, has been cemented by the publication in January 2020 of the first issue of the Journal of Sound and Music in Games (University of California Press).⁸

Despite the international productivity that ludomusicology has had over the last decades, the Hispanophone academic community has to date demonstrated a certain resistance to accepting video game music as a field of research comparable to the studies that have been done on music in other accepted and legitimized audiovisual media such as film, advertising, or television. However, the labor of several researchers from fields such as audiovisual communication, sociology of music, or musicology, who have addressed this topic in a timely fashion and from diverse perspectives, should be recognized.⁹ Likewise, over the last decade, video game music has begun to have a presence at conferences and academic events on music and audiovisual culture held in Spain and in Latin America. It is not easy to understand the causes of this situation. In addition to organizational and budgetary problems, always present in policies that govern centers of research and training, researchers and university professors have difficulties funding new areas of research and proposing approaches that permit the creation of paradigms and epistemological models that, on many occasions, do not agree with the needs and interests of digital society.¹⁰

When a player in the twenty-first century, their hands occupied by a video game controller, is observed strolling calmly through a reproduction of the Odeon of Pericles and stopping in front of a group of women in order to listen to a piece of music played on instruments that bring to mind the shapes and sounds of classical Greece—as in one of the episodes of the Assassin’s Creed: Odyssey Discovery Tour (Ubisoft, 2019)—it is immediately apparent that, apart from the ludic-only aspects, video game music belongs to a complex digital ecosystem that generates a multiplicity of coded messages (see Figure 1).¹¹ Producing meaning from the elements and references that comprise a video game is not a simple task, and it implies the development of a series of intellectual, social, and ethical proficiencies that allow critical interaction with information and its recreation. The complex process through which meaning is generated in a video game is part of ludoliteracy. This concept, which comprises the ability to generate and understand meanings in and through video games, has crystallized from the theories of José P. Zagal. In addition to considering functional abilities, such as knowing how to play or being able to create and develop games, ludoliteracy includes a more analytical and reflexive dimension that aims to improve competence in explaining, discussing, describing, framing, situating, interpreting, and positioning games, and placing them in four distinct contexts: (1) as cultural artifacts, (2) in the context of other games, comparing games and genres, (3) in the context of the technological platform that is used, and (4) deconstructing them and understanding the relationship of their components, how they interact, and how they facilitate certain experiences for the players.¹²

Based on the approach proposed by Zagal, Isabella van Elferen has suggested the model of ludomusical literacy (game music literacy) for video game music, which combines the concept of ludoliteracy with that of musical media literacy. The latter involves the capacity to interpret the tropes of film, television, or advertising music based on frequent exposure to them.¹³ Van Elferen suggests that, through intertextual references from other audiovisual media, video game players can develop the ability to recognize musical traits and compositional styles that allow them to become more actively involved in the dynamics of the ludofictional worlds. Studying in depth the model of ludoliteracy can, therefore, be a useful tool for the advancement of research on the process of ludomusical literacy through which video game music constructs its meaning. Starting from this premise, a personal proposal for the application of Zagal’s model to video game music is outlined in the following points.¹⁴

Placing video games in their cultural context is the first step in the approach proposed by José P. Zagal. To understand the elements of a video game, it is necessary to begin by understanding the dependencies and correlations that are generated between video games and other cultural productions such as printed products (books, comics, board games) or audiovisual media (movies or television series). The situation is especially clear when it comes to video games that belong to the same franchise, but it can also be analyzed in those that are a direct adaptation of other media, and even in those that share the aesthetic or thematic qualities of a particular genre. The relationships that are forged between video games and other cultural products contribute to the creation of a complex process of unified and coordinated convergence. The player-consumer, being familiar with a particular product, can easily identify and interpret the tropes and elements that appear in a different medium. An example of this can be seen in the relationships generated around the successful video game series The Witcher (various developers, 2007–2015), which takes its plot from the series of fantasy novels by author Andrzej Sapkowski, published from 1990 on. Before moving to the video game medium, Sapkowski’s novels had been adapted into comic format in six volumes published between 1993 and 1995 that also served as a reference for some visual aspects of the first video game. The books segued to the big screen in the film Wiedźmin (2001), directed by Marake Brodszki, and in the television series The Hexer, released in 2002 with a soundtrack by Grzegorz Ciechowski; one of his musical themes was directly used in the video game The Witcher 3: Wild Hunt (CD Projekt, 2015).¹⁵ Furthermore, in 2001 the role-playing game Wiedźmin, also inspired by the Sapkowski novels, was published, and over the past decade several card games, comic series, and even a coloring book for adults were released.¹⁶ The subject has recently returned with the series The Witcher, released in late 2019 on Netflix; although it takes its plot from Sapkowski’s novels, it borrows a number of visual elements from the video game series.

The multiplicity of relationships and references that are generated in the processes of convergence is also one of the core aspects of ludomusical literacy. In many video games, musical meaning is constructed not only through things taken directly from a preceding audiovisual or literary product, as in the aforementioned case of the video game soundtrack for The Witcher 3: Wild Hunt, but also from imitations, recreations, or similarities that give rise to a constant flow of content that the player recognizes and interprets, thanks to associative and implicit learning. To illustrate this process, what follows will consider how the musical environment was developed in the three games of the Red Dead series, set in Mexico and the American West, in which a self-referential language is created from sonic topics from film.

The first video game in the series, Red Dead Revolver (Rockstar, 2004), takes place around 1880 in the American West. Its soundscape was constructed from a collection of musical themes taken from spaghetti western films, a genre that reached its peak in the 1960s and 1970s. The musical selections in the game include songs by important composers such as Ennio Morricone, Nino Rota, Bruno Nicolai, Francesco De Masi, Roberto Pregadio, and Luis Bacalov, author of the main theme of the film Lo Chiamavano King (1971), which is used to open the game.¹⁷

Due to its direct relationship with the cinematographic medium, the soundtrack of Red Dead Revolver is limited to using a wide array of the main sonic topics of the spaghetti western that the player knows and can associate with various situations in the game, such as trumpet solos for important moments, whistles as popularized through Ennio Morricone’s soundtracks, and other sonic cues such as bells, the use of the whip, the anvil for percussion, or the mouth harp. Despite its lack of originality, this acoustic environment was what was used as a reference when Bill Elm and Woody Jackson were commissioned for the music for Red Dead Redemption (Rockstar, 2009). The action of this video game, the plot of which has no relationship with the previous one, shifts to the year 1911. The protagonist is John Marston, an outlaw who is pressured by two law enforcement agents to capture one of his former fellow adventurers. In their musical design, Elm and Jackson retain the sonic topics of the genre but introduce certain innovations in instrumentation that primarily intend to characterize the three fictional spaces of Mexico and the United States where the game takes place: New Austin, Nuevo Paraíso, and West Elizabeth. However, unlike its ludic and cinematographic precedents, the music of Red Dead Redemption behaves in a dynamic manner in a good part of the game and adapts to the main situations that the player must face, such that, in this case, the process of ludomusical literacy appears to be shaped by technological constraints. For example, to facilitate their combination within the video game, all of the pieces that comprise the soundtrack are written in the key of A minor and arranged at 130 beats per minute.

Throughout the game, the sonic topics of the spaghetti western are intended to react dynamically in situations similar to those that the player would experience if they were the main character in a film in this genre. If, during the game, the player shoots another character without a clear reason, they will have to run away immediately, pursued by a group of agents. During the chase, the video game engine will have the player hear a musical theme in which topics such as bells, mouth harps, and whistles appear, accompanied by other diegetic sounds. When the agents begin to approach the character controlled by the player, the musical intensity increases in order to generate tension; if the player manages to get away from them, it progressively decreases.

A similar situation occurs in duels, in which John Marston has to confront an antagonist. In this case, the player hears a whistle, accompanied by a drum that taps out a bolero rhythm with an ostinato that starts softly and progressively increases in intensity until the moment when the shots are heard (see Video 1).

The player’s previous exposure to other audiovisual products, through which their ludomusical literacy was acquired, allows them to recognize the sonic code of the whistle accompanied by percussion as a symbol of the lone cowboy alert to risky situations; this musical topic was popularized by composer Ennio Morricone in his famous Dollars Trilogy, a series of three spaghetti western films directed by Sergio Leone in the 1960s.

The intertextual references to film music and to previous games in the series were also taken into account when the latest installment in the series, Red Dead Redemption 2 (2018), was designed. On this occasion, however, an attempt was made to overcome the literal reproduction or mere imitation of sonic tropes to emphasize that, from the perspective of plot, the new video game should be considered a prequel to the previous one. The developer Rockstar Games proposed creating a new sound design that, without losing its essence, would serve to contextualize a game centered around outlaws from the Wild West set in 1899.

As is quite common in the world of video games, in order to begin to work on the sound design of Red Dead Redemption 2, a series of sonic references was taken as a starting point, such as the album Teatro (1998) by the American singer-songwriter Willie Nelson and the soundtrack by Bruce Langhorne for the 1971 film The Hired Hand, directed by Peter Fonda. The latter was included because it evoked a soundscape that moved away from the stereotypes of the film genre without losing its roots in North American folk music.¹⁸ Additionally, an attempt was made to imitate the sound of the Wrecking Crew, a versatile group of studio musicians who worked in Los Angeles in the 1960s and 1970s with artists such as Frank Sinatra, Cher, and Art Garfunkel, among many others.¹⁹ With these references, Woody Jackson set out to compose a soundtrack that would ignore stale cinematic sonic tropes—without losing sight of the fact that players should continue to recognize the result as part of the soundscape of the Wild West. For this, Jackson focused specifically on adjusting timbral elements. For example, he replaced the bells typical in a spaghetti western with a mandolin, and he introduced other instruments such as the violin, mouth harp, jug, and harmonica; all of these were played in jug bands, popular musical groups in America in the early twentieth century.²⁰ Likewise, original melodies appropriate to the game’s historical context were included; these are interpreted in a realistic manner by nonplayable characters (NPCs), and the players can interact with them or simply listen to their versions.

As seen in the preceding examples, originality or complexity are not significant characteristics of video game music, but there is a solid reason for this. It should be easy to learn and identify the sonic tropes that ludomusical literacy requires. When music plays an important role in interaction, the player must be able to identify and interpret, with the greatest precision possible, the trope that will be encountered throughout the game. Recognizing the significance of the sound of a whistled melody before a duel can be crucial to being prepared to appropriately resolve the situation that the scene presents. In this way, when sonic intertextuality based on places commonly known to the players is relied on, the process of ludomusical literacy becomes one of the defining aspects for the player to achieve immersion in the videoludic space.²¹

Ludoliteracy also includes, according to Zagal, the contextual relationship that is established between the video game being analyzed and other games, whether digital or not. This aspect attempts to understand the conventions and design decisions that are determined by factors such as mechanics derived from another similar video game or from other analog games that served as inspiration or reference for the design. An example would be experience points, which a character obtains by doing concrete actions such as collecting objects, completing missions, or overcoming certain challenges, and which enable the player to improve the characteristics or abilities of their character.

With regard to music, this type of relationship supposes a recognition, through the process of ludomusical literacy, of those elements that are integrated into the mechanics of video games and that have particular meaning for the player. In the vast majority of video games, sonic elements are present in the most critical moments and indicate, for example, if certain points have been achieved through an action, if the things that have been collected are the right ones, or if the character being controlled has many enemies lying in wait. In this fashion, an ascending or descending glissando or slide usually signifies the increase in or loss of energy of the character throughout the game, and the high-pitched sound of a small bell or glockenspiel is commonly associated with the reward achieved for collecting an object.

The association of musical sounds or gestures with the actions or situations in a game are not arbitrary; such an association responds to a process of sonic codification that was generated throughout the twentieth century, and it originated in the first pinball machines and other mechanical and electromechanical predecessors that incorporated sound as part of the configuration of their aesthetic parameters. Although it is evident that today, thanks to technology, much more information can be communicated through sound—for example, spatialization, more precise frequencies, changes in timbre, or digital signal processing effects—the function of providing basic information to the player through certain sound features, as Karen Collins suggests, continues to be an important factor in today’s video games.²²

To adequately understand the nature and values of the video game, it is also necessary to consider the technical means by which it is reproduced and the restrictions or technical possibilities of each of the devices in which it is implemented. This aspect is especially significant in the field of music and sound; a researcher must be aware of recent developments in order to precisely refer to the acoustic environment of each genre and to locate, from both the synchronic and diachronic points of view, the possible sonic tropes that can be found in a video game. For example, if one wished to delve into the musical aesthetics of classic video game series such as The Legend of Zelda or Super Mario Bros.—still a force to be reckoned with through the latest versions of the games and their permanence in orchestral repertoire or arrangements disseminated through YouTube—it becomes essential to understand the technical characteristics of the Nintendo Entertainment System (NES), one of the highest-selling video game consoles in the 1980s and ’90s, for which the first video games in these series were created.

The NES, released in 1985, had a sound chip (the Ricoh RP2A03) with five channels. Two of these were square waves that allowed for several options to configure timbre. One of these channels could vary frequency to create glissando or slide effects, appropriate for the sounds of laser guns firing or flying saucers. The third channel was a triangle wave, an octave lower than the pulse waves, and was rather limited in pitch. The fourth channel produced white noise and was commonly used to emulate percussion instruments and other effects. The last channel allowed for samples of real references with delta (analog-digital) modulation, making available to the composer a rudimentary version of 1-bit sounds.²³

The technological limitations of the NES chip contributed to the development of new compositional strategies that were incorporated into the sonic aesthetic of Nintendo products. For example, given that only three of the five channels could be heard simultaneously, it was common to use arpeggios at high speed in one channel, creating the impression of polyphony and generating greater harmonic richness. The two remaining channels were thus freed up to arrange one or two melodies that could work in contrapuntal fashion, or to enrich the texture with doublings and different timbral effects. In the short loop entitled “Poison Mind” that Kinuyo Yamashita composed for one of the boss fights in Castlevania (Konami, 1987), the use of this instrumentation technique is evident. With a reinforced two-voice texture, a fast sixteenth-note arpeggio appears in one of the two square-wave channels. The triangle wave, which doesn’t permit volume control, is used as a harmonic bass that develops over the primary chord progression, while the noise channel simulates percussion effects; see Video 2.

The process of ludomusical literacy, which occurs through the perception of the sonic result combined with the previous theoretical knowledge of the technical development of video game consoles, would help the researcher who listens to the previous Castlevania loop to identify its aesthetic features and place it in the chronological setting of third-generation video game consoles that were manufactured in the 1980s.

The last of the aspects that, following the model proposed by Zagal, can contribute to the process of ludomusical literacy is an understanding of the functions of the different components that comprise the workings of a video game, how they interact, and how they facilitate specific experiences for the players.

The recurring activities and missions that have to be accomplished in Japanese role-playing games, or JRPGs, helps demonstrate that, in the field of video games, listening to the same music with few variations can paradoxically be an effective way to generate meaning. The musical structure that shapes a JRPG experience usually has eight loops or short pieces related to situations or places in the game. It is a reiterative scheme that replicates the general model established by composer Koichi Sugiyama in the score made for the first game in the Dragon Quest series (Enix, 1986).²⁴ Of the eight musical pieces that usually appear in this game genre, four represent a standardized sonic location of the environments or places through which the player has to travel: castle, town, field (overworld), and dungeon. The remaining four are used for different moments within the play structure itself. The first musical piece, in looped form, is the introduction or game presentation, which also occasionally includes a short cutscene contextualizing details of the characters or narrative. The second usually sonically represents combat, while the third piece illustrates the final boss fight. The last music heard in the game is related to the final victory and the music from the opening credits.²⁵

The use of the same eight-piece scheme, codified over more than three decades, has become firmly entrenched in the identity of JRPGs. Thanks to the learning involved in exposure to other titles of the same genre, a player is able to identify and assign meaning, from the first moment and even without previously knowing the game, to each one of the loops that they hear while traveling through the different game areas.

The process of ludomusical literacy that involves the codification of the eight musical pieces used in JRPGs has also transcended the world of video games to be incorporated into the pedagogical field. A curious circumstance is thus produced, where a musical model has an educational character in both the virtual and real worlds. Tim Summers has observed the use of a piece, taken from a JRPG, in a volume of teaching materials published as part of the official curriculum for music in elementary schools in Japan.²⁶ Given that it is a book whose contents have been approved by the authorities, the presence in the pedagogical field of such music also appears to reveal a conscious practice of ludomusical literacy, promoted by official institutions in order to educate new generations in the topics and aesthetics of Japanese video game music.²⁷

As part of a complex and multidimensional medium, video game music requires an approach that uses sources, materials, and methodologies different from those used in other kinds of musical analysis. The audio in a video game is not always linear, and in many cases it has to adapt to the player’s reaction. Due to its immediacy, interactivity, and playability, video game music is essentially perceived as a real-time event, which a traditional score could only blurrily represent. For this reason, when analyzing musical compositions made for video games, the researcher cannot limit themselves to studying structure, harmonic or contrapuntal contents, or instrumentation. They have to address specific aspects of the medium, like the roles that music plays in immersion, in the game narrative, or in the relationship between the music and the player’s movements—which is dealt with in kinesthetic design—or the affordances and constraints of available technology, among others. Analyzing the music of a video game therefore means developing new methodological approaches and working with sources that are unusual for the typical activity of a musicologist.

In order to explain how sonic content relates to gameplay—meaning to identify places where dynamic changes, musical transitions or interruptions, or those moments when the game engine reacts musically in an unexpected manner for the player occur—the researcher must take the controller or keyboard in their hands and dedicate part of their time to playing with an analytical and critical eye.²⁸ Playing conventionally, following the routes or instructions indicated, can be a good way to get to know the general outline of the musical activity as it relates to game mechanics. Sometimes it is useful to have a “reactionary” game experience, meaning to act unpredictably in order to learn how the music reacts when expectations are subverted—for example, taking a character to a place different than the one indicated by the game, or making them walk slowly when they need to run away immediately. Likewise, an experience of “recidivist gaming”—playing the same level again but observing how musical conditions are retained or altered—can help detect, for example, possible problems caused by technological mechanisms in transitions, fade speeds between the two tracks, or repetitions of looping sequences. This methodology, based in analytical play, is highly recommended, especially when one intends to analyze a game that contains generative or procedural music, a technique that allows the creation of music by algorithms in real time; this happens, for example, in No Man’s Sky (Hello Games, 2016), an open-world space exploration adventure with minimalist music composed by the British electronic experimental rock group 65daysofstatic (see Video 3).

In this type of game, it can also be interesting to study the programming code that affects music or the middleware systems with which sound objects are implemented—such as FMOD, Wwise, or Microsoft Direct Music—although it is not always possible to obtain this information, as it is usually protected by copyright.²⁹

Accessing the musical contents of a game is not usually an easy task either. In some products, it is possible to extract the files in MIDI format, for example, which can enable the musical transcription that the researcher must usually do in order to make up for the lack of reliable scores accurately reproducing the music heard in the game. Except on some occasions, when the composers make scores available to the public—as for example with the pieces for piano solo Garry Schyman shared of the music he composed for the BioShock series—the materials that appear online are usually fan-made transcriptions, which do not always faithfully reflect the sonic content of the game.³⁰ Arranged anthologies published by commercial publishers can also be useful tools for analysis, especially if the composer has participated in them.³¹

To supplement or complement the data obtained in investigating the game elements themselves, it is usually also useful to consult other secondary sources, as noted by Tim Summers.³² Although it is not easy to access this kind of information, the development companies sometimes release production documents and reports or game design documents (GDDs) that occasionally include detailed descriptions of how they planned to relate the music to the game’s mechanics and narratives. On the other hand, commercial recordings of the soundtrack released on CD or in digital format should be used with caution. Generally, these are remixed and rearranged partial versions that offer a static photograph of the musical content, decontextualized from the narrative, lacking dynamic change and the contributions made by the player’s actions. To deepen the relationship that the players establish with the music of a specific video game, it might also be interesting, as Summers suggests, to analyze the reviews published in specialized communities on the internet and the comments in forums and YouTube videos, a thought-provoking methodological approach that links up with reception studies.³³

Isabella van Elferen, professor at Kingston University (London), has developed a comprehensive model for studying the relationship established between immersion and musical elements in video games.³⁴ The model, called ALI (affect, literacy, interaction), proposes three superimposed working concepts: musical affect, media literacy, and interaction, which work together in the process of meaning and identification of video game music. As a psychological component, affect is related to memory, emotion, and identification. This model calls attention to the important emotional connections that video game players create through music. It is an affective immersion that generates subjective and unpredictable meanings, but it can be guided by the model’s second element, media literacy, which is understood as previous knowledge of certain tropes and musical conventions that the player has assimilated more or less subconsciously through other audiovisual media. For example, hearing the sound of a cello with dissonant motifs in the lower part of its range can be interpreted as a foreshadowing of death, because the player has often heard this sonic gesture in a similar situation in a horror movie.

The third concept in van Elferen’s ALI model is musical interaction in video games, which occurs both with performative situations, in which the gameplay is based on the player’s control of musical elements—for example, in rhythm games that emulate interpretation of a musical theme—and with those in which the player can control the diegetic music in the game, such as turning off the radio that the character is listening to in their car. Musical interaction also occurs in situations in which extradiegetic music helps guide the player through the space. The analysis of the way in which the three elements—musical affect, media literacy, and interaction—converge in the sound design of a video game permits precise knowledge of the mechanics of musical immersion, and furthermore provides an interesting connection between empirically obtained data and theoretical reflection.

The study of the aforementioned performative aspects of video game music forms another of the productive lines of research that have begun to develop in recent years, in which ludomusicology converges with other fields such as music pedagogy or psychology. Understanding music as an experience or as a set of complex actions that produce an active and participatory experience, as synthesized by the expression musicking created by Christopher Small, allows the expansion of the concept of musical interpretation to include new ways of producing music mediated by ludic interfaces.³⁵

So-called music video games, in which music itself is the center of gameplay, offer specific opportunities for fun and learning and can become a valuable complement to a music education program. This type of game involves active participation and a new form of musical interpretation that, as has been shown in empirical studies, contributes to improving memory, creativity, rhythm, and cognition in a wide array of people that cannot invest the time or money required to learn a musical instrument.³⁶ For example, in a now-classic video game such as Rock Band (Harmonix, 2007), the players can choose to perform different musical works with drums, bass, guitar, or voice, either as a soloist or as a band. In this game, drummers play a special drum kit consisting of four electronic pads, cymbals, and a kick pedal. Guitarists and bassists use controllers that resemble their respective instruments, but the strings and pick are replaced by five colored buttons on the neck and a toggle switch for effects. A microphone allows the game to analyze the pitch and rhythm of the vocalists. All of the players perform from an unconventional “score” or graphic representation of the music that scrolls vertically. These graphics are synchronized with an audio recording. The score a player achieves is based on the rhythmic precision of the percussion players and also the melodic precision, in the case of the guitar players; for vocalists, pitch is also evaluated.

The possibilities of expression and control through the interfaces of musical video games, such as in Rock Band and other similar games like Guitar Hero (Harmonix, 2005) or Rocksmith (Ubisoft, 2011), have created the need to reconsider the definition of a musical instrument. The similarities between video game controllers and acoustic or electric instruments invite the extension of the conventional definition of a musical instrument to include any device capable of producing sounds that is controlled by a series of physical gestures and that reacts to user input in order to generate sound, as in many music video games in which the players physically interact with electronic interfaces—whether or not they are in the shape of a musical instrument.³⁷ In this fashion, musical performance is expanded to the complex phenomenon of “kinesonic synchresis,” exclusive to interactive products in which multisensory experiences can be generated through the synchronization between action, sound, and image.³⁸

The search for adequate conceptual tools for analyzing the relationships between visual and sonic elements of video game products is another research area that ludomusicology has cultivated from the start.

The similarities that exist between the sonic elements that make up the structure of video games and those in other audiovisual media, and the fact that a large number of video games today use soundtracks similar to epic Hollywood films, have led to the adoption of a diegetic/extradiegetic binary. In literary and audiovisual theory, it is common to use it to classify the difference between sonic events that originate within the ludofictional world—such as sounds that come from a music player or from a character that plays a musical instrument at some point in the game—and those others that, on the other hand, lack direct connections with a visible, or at least suggested, sound source within the ludic space. Unlike what happens with other audiovisual products, however, the boundaries between diegetic and extradiegetic sonic elements in video games are not always clearly defined, and on occasion are consciously broken for a functional purpose. In a film screening, the sound components are immutable, and as a result they are always heard in the same way every time the film is repeated. But the interactive element in video games means that all sounds, including musical ones, can vary depending directly on the decisions and actions adopted by the player, who thus becomes an often-involuntary cocreator of the game’s soundscape. The complexity of this approach increases in the case of multiplayer games, where the sound sources branch out in proportion to the number of players that are participating in the game session. This generates the creation of multiple sounds in real time. In many video games, moreover, the player can decide to adapt the acoustic experience to their personal tastes: turning off the music, changing the volume of the sound effects or dialogue, and in some games personalizing the music. For example, in Grand Theft Auto V (Rockstar North, 2013), as in previous games in the series, cars have several different radio stations that broadcast different musical genres. The player can select the station they want to listen to, or personalize their listening with a playlist that includes their favorite musical works, and therefore they intervene directly in the sonic diegesis.

As the preceding examples show, the theoretical concepts that have been used to analyze other audiovisual products are insufficient to explain the complexity of the relationships established between sonic and visual elements in video games, which has created the necessity to suggest new theoretical developments to modify the conceptual framework of reference. Kristine Jørgensen proposes the term “transdiegetic” to define sounds, including those of a musical nature, that do not come from a source in the game, but that maintain the ability to inform about events within it.³⁹ Jørgensen’s idea emphasizes the functional importance of music and sounds in relation to the player’s actions, and it draws attention to the new relationships that are established between sound and space in the ludofictional world.

Transdiegesis suggests two subcategories. The first, called external transdiegetic sounds, includes those sonic events that do not have a direct reference to a sound source in the game but might serve to communicate information on diegetic aspects, for example, by generating feedback for the player on their own actions. Adaptive ambient music could also be included in such external transdiegetic sounds—a soundtrack that accompanies the player dynamically, generating input from the game, and allowing the player’s decisions to influence certain musical parameters and how the pieces develop over time. The location in the game space of the character being controlled, the presence of enemies, or the state of health are vital factors for adapting the musical contents to the game experience and to the emotions it generates. The second subcategory of Jørgensen’s proposal is that of internal transdiegetic sounds, those that emanate from diegetic sources but, in principle, seem to have no relationship with the gameworld. Their main function is to communicate directly with the player located in the real world and to create a sense of separation from a diegetic sound. An example of this could be one of the characters referring to the player’s actions through a burst of music or through direct dialogue.

Jørgensen’s concept of transdiegesisis is especially useful when analyzing the new relationships with sound in virtual reality products, in which external sounds such as the player’s breathing or heartbeat can be linked with the game through biometric sensors or microphones and be translated into information of a rhythmic nature. This information can be incorporated, for example, into a generative soundtrack in order to adapt musical events to the biological parameters of the player, which enhances their immersion in the game by taking part in real time in the cocreation of the music and in the sonic diegesis itself.

In some situations, the music of video games can also create its own virtuality, its own flow, and thus maintain independence from other game components with which it interacts in order to complement or intensify certain aspects. Unlike what happens in other audiovisual products, in video games music can extend an autonomous visual space at those points where it is desirable to increase immersive and diegetic qualities. For example, if music and sound were persistently absent in a suspense or horror film sequence, even ignoring the effect of silence for moments of maximum tension, the essential elements of the message would continue to be transmitted to the viewer, even though some of the effectiveness that the sonic tropes communicate to us would be lost. But in certain survival horror video games, a lack of sonic feedback would make it difficult for the player to have sufficient references to advance in the game.

Conscious of the terminological limitations to define situations such as the one just described, Isabella van Elferen proposes an expansion of the diegetic and extradiegetic categories and introduces the concept of “semidiegetic” sounds to define those scenarios where the diegetic sounds blend with nondiegetic and interface sounds.⁴⁰ The complexity of the contexts in which these situations occur in some video games, where reality and virtuality overlap, can be appreciated via an analysis of an example from Resident Evil 4 (Capcom, 2005), to which van Elferen alludes. In this horror game, the protagonist Leon S. Kennedy has to rescue the daughter of the president of the United States, who has been kidnapped by an organization in Spain that is part of a horde of violent zombies that look like villagers. The sequence analyzed illustrates the moment when the protagonist bursts into the space inhabited by the furious zombies. When Leon enters the field of vision, the sound of a chainsaw begins to be heard in the foreground, which coincides with the perfectly identifiable diegetic cry of one of the zombies, who warns the rest of his companions of the character’s presence: “¡Un Forastero!” To enhance the effect of immersion, the shouted voice is accompanied by a short, static, dissonant, and extradiegetic musical burst (see Video 4).

Although the characters cannot hear the music external to the action, the sonic amalgam helps model for the player an ambivalent situation of semidiegesis, in which the music belongs to both the reality of the game’s sonic ambience—with regard to the cry of the zombie—as well as the virtuality that presumes that the player, and not the character they control, is the only one capable of hearing all of the various sonic elements.

Analyzing video game music from an academic perspective is not just an end in itself. The different aspects studied in this article have tried to highlight some contributions of ludomusicology to the restructuring of categories, concepts, methods, and analytical techniques needed to approach the understanding of music’s role in new forms of artistic expression derived from digital culture. Understanding, for example, how the music of a video game reaches complex levels of meaning through the process of ludomusical literacy, or incorporating concepts such as immersion or performativity into common research tools, can effectively contribute to a serious overhaul of musicology.⁴¹

Given that the audible phenomena occurring in video games create and shape a sonic space that includes the individual perspectives and expectations of the player, ludomusicology also suggests the possibility of reexamining music comprehension from the perspectives of listening and phenomenology, by allowing the critical evaluation of the sutures and fissures that can be generated between reality and virtuality in ambivalent sonic situations generated in ludomusical environments. This occurs in the rhythmic action games cited in this article that combine diegetic and nondiegetic sonic functions, in which the player not only focuses on musical performance but must also use their memory to respond adequately to visual and sonic stimuli. There is, therefore, an intensification of the immediacy that characterizes conventional musical performance. This type of approach can also be useful when studying the presence of music in new products developed for augmented reality or virtual reality devices, where an immersive three-dimensional and multisensory experience is created, or in games based on musical algorithms, in which the player does not have a specific objective and their gaming experience is captured in experimenting with sound, music, and abstract images.

In “liquid modernity,” in which the generational dynamics of music consumption seem to point to a certain disjunction in the interests of traditional musicology, the analysis of contingency and interactivity that characterize ludic video products also provokes a reflection on the current objectives that guide teaching and research in the humanities in general, and musicology in particular.⁴² Studying new ways of producing and consuming music, as proposed by ludomusicology, also invites the deepening of research approaches such as musical experience, meaning, and performativity, which allow for the enrichment of the field of musical comprehension in the context of digital society.

This article was originally published in Anuario Musical 75 (2020): 181–99 and has been translated for the Journal of Sound and Music in Games. https://doi.org/10.3989/anuariomusical.2020.75.09