A Composers Perspective on Game Audio – Adaptive Audio Concepts and Work Flow

October 3rd, 2011 | By Lance Hayes

Note to Readers

This article is from the collection of blogs I wrote for Keyboard Magazine’s website in 2010 and 2011. For more information about Keyboard Magazine please go to their website.

Cheers!

A Composers Perspective on Game Audio – Adaptive Audio Concepts and Work Flow

I’m often asked, “What exactly is adaptive music?” With that in mind, this entry is dedicated to a core game-music production concept. I’ll do my best to offer up a basic outline and explain, with some examples, what adaptive audio is and a little about how it is created.

Non-Linear Music

One of the first things that game composers must tackle conceptually is use and application of non-linear music production. We all know that linear music is music that plays from one end to the other without interruption. By contrast, adaptive music is expected to twist and bend, lead and define on demand as the player moves through the game while maintaining structural integrity. That is to say it has to sound like music, communicate with the player and engage the listener while being custom-built by the action in the game. The very best does that so transparently you don’t realize you’re listening to a custom-created soundtrack.

On a very high level, adaptive music is best described as music that is created to augment or inform player actions or progress through a game. It is similar to the concepts of sound design in games where you create a set of sounds that play at a precisely orchestrated moment to enhance an action in the game (for example, the sound of a door opening when you open a door in the game). Music can function in much the same way to enhance the player experience.

What’s in a Name?

Adaptive music is also often referred to as “interactive” or “dynamic.” There is some debate and splitting of hairs about these terms, but the important thing to keep in mind is that they all mean a system of music that can be broken down and used in various capacities within a game. Properly implemented adaptive music by any other name can heighten the player experience and give rise to a musical score for the gameplay that can be at once unique under repeat play and allow for leitmotif development to occur naturally to varying degrees of refinement.

I prefer the title “adaptive” to the other options because this word is the most closely aligned to the actual process that takes place when the music is adjusting to gameplay. Adaptive music can result in a nearly endless series of possible types of experiences in a game. The sky’s the limit.

Before we move on, it may also help to know that “implementation” is the term used in the game industry to describe the process of placing, mixing, augmenting and managing the music and sound-design assets in the game audio engine.

Applications

If it’s well-conceived and implemented, adaptive music develops into a subtle audio experience that leaves the player unaware that the musical playback is following their progress or directing their actions. The idea is that the music is triggered in a way that will allow them to feel as if the score for the game is happening not because they are now in an area or experiencing an event that triggers a new set of music, but because the game is scoring their progress seamlessly in an effort to give context or clues. With that in mind, you can imagine how adaptive music could be employed to achieve almost any mood or result at any juncture during gameplay.

Approaches

Adaptive music is designed and produced in a number of ways. The music itself can be created using virtually any approach, from one person in a studio or session musicians working up parts to full orchestral sessions with choir. Or all of the above blended together, as is the case with many soundtracks now. The approach is similar to other media at that point.

Depending on the nature of the project, the composer often has to keep the adaptive process in mind as he or she composes. You have to consider that the music will be played back in a non-linear fashion, and so that affects, among other things, use of modes, key changes, and tempo as you work up a track. You can still do all of those things, but the timing has to be pre-established so they don’t clash with each other and layers can play back-to-back or layered as stems.

After the loops and stems are established, they can be used in numerous ways. A common method is back-to-back triggered playback of loops or sections of music activated by player action or location in the game space. In that instance, the music is played back as complete blocks of audio, much like triggering block-type MIDI or sample files in your DAW.

Another approach is to use the stems as musical or percussive layers that can be flown in over a core composition stem to add enhanced tension, happiness, stress, excitement, and so on. Real-time DSP effects are sometimes used on the music to enhance the final sound and give it additional context. A popular example of that is a grenade going off in the game near the player, causing all of the sound and music to be ducked and run through a lowpass filter to simulate the short-term effect of temporary deafness.

Examples

Thanks to fellow composer and violinist extraordinaire Jeff Ball for suggesting this video capture from popular audio middleware FMOD that succinctly shows what is happening to the layers of music as situations develop in the game. You can’t see the gameplay from this perspective, but this is what’s happening behind the curtain if you look at the process from the viewpoint of the middleware. The music flows seamlessly from simple to complex and percussive, and from mellow to aggressive.

A great video (NSFW) of adaptive music being highlighted during actual gameplay was created by the team at WallOfSound as an example of their work on Mass Effect 2. They created a massive hybrid score that incorporated electronics and full orchestra seamlessly into the game. The video features work by Jack Wall, Sam Hulick, David Kates, and Jimmy Hinson, with audio implementation by Brian DiDomenico. If you want a good overview of the adaptive process, this video gives you an idea of what’s happening as the action ramps up. Keep an eye on the titles at the bottom of the screen to see how many events shape the music you are hearing as the soundtrack progresses.

From a different perspective, there is great example of using layers to create an adaptive score in this video interview (also NSFW) with Jason Graves. The whole interview is worth a look, but starting at 5:44 Jason shares some great information about his approach to layered adaptive score creation and how he developed those layers for his soundtrack for Dead Space 2. He also discusses working with the team at the studio.

Another fantastic illustration of an adaptive score is 2009’s “Flower.” Using a combination of synth- and orchestra-style cues layered together, composer Vincent Diamante created an entire interactive world of music that matches the beautiful images and simple gameplay from this elegant masterpiece.

Wrap

The information here is really only intended to give you an indication of what adaptive audio is. Adaptive audio is an amazing, continuously growing frontier in games and increasingly in outside media. If you are interested in game music, this is one of the most fundamental concepts in the medium.

Cheers!

Pro Tip

The Sony PSN hacking event helped illustrate to many people just how deeply games and the game industry impact our everyday lives. When the 77 million user accounts suddenly became unavailable due to an intrusion into the Sony network, there was hue and cry and then some real-time issues when it became clear that credit card accounts had been violated. I was reminded—along with millions of others—that it’s a long dark night without Call of Duty: Black Ops, and for some there was no solution other than a trade in.

It will be interesting to see how this all shakes out for Sony, but they are back online now and hopefully this will all be a memory as once again players enjoy competing across one of the biggest game networks on earth.

Gear Tip

Michael Patti and Michael Barry at Cinesamples are hard at work on an exciting new brass library. Recorded by legendary veteran Dennis Sands at the Sony Pictures Scoring stage in Los Angeles, the CineBrass collection looks like it will be exceptional. The library features all the individual brass sections sampled with true legato transitions and a remarkable new articulations engine. Check out the CineBrass patches teaser video.

Comments are closed.

Lance Hayes