The stereotype goes that when directors are figuring out how to frame a shot, they extend their arms, create a rectangle using their thumbs and index fingers, and peer through that box.
The stereotype goes that when directors are figuring out how to frame a shot, they extend their arms, create a rectangle using their thumbs and index fingers, and peer through that box. And because movies and TV shows have traditionally been viewed on one kind of box or another, the process provides a rough gauge of what the scene will look like.
But when shooting 360° video, what does framing even mean? How does a director draw the viewer’s attention to a specific point when there’s so much to look at, with continuous imagery at every direction?
Filmmakers dipping a toe into this kind of video are learning that virtual reality is a completely new beast, one that forces them to rethink the traditional storytelling techniquesthey’ve come to rely on.
In the process, they are discovering new things about the power of audio.
George Lucas is widely creditedwith having said that sound is 50% of the moviegoing experience. While audio indeed has played an important role in cinema since the 1920s, with the rise of talkies, it’s proving an especially vital tool in virtual reality. With composition and framing essentially boiling down to making sure the horizon is straight and keeping objects and people that don’t belong in a shot out of it, subtle sound cues like footsteps from behind or a knock at the door can help guide viewers to particular points in a scene.
The unexpected VR company
For someone who’s never tried virtual reality before, donning a pair of goggles for the first time can be an exercise in sensory overload. People are often so stunned by the visuals that they hardly notice the audio—but in fact it’s playing a crucial role.
“If you put earplugs in and try to go about your day-to-day life, you could say half the experience is gone,” says Joel Susal, Dolby’s director of virtual and augmented reality.
Audio has always been Dolby’s bread and butter. The division Susal runs only came about in March—largely because Jaunt, a VR studio whose backers include Disney, was searching for audio technology to create immersive “3D” sound. One of Jaunt’s board members, Peter Gotcher, happened to be the chairman of Dolby, and helped make the connection.
Dolby turned out to be a good fit because it already had a technology that could be adapted for virtual reality: Dolby Atmos. If you’ve ever been presented with the option to pay an extra dollar for “enhanced audio” at a movie theater, chances are it was for a film that was mixed in Atmos.
Atmos builds on the idea of surround sound, which literally surrounds viewers with speakers placed around them: in the center (behind the screen), left-front, right-front, left-side, right-side, left-back, and right-back.
Atmos goes a step further by placing an array of speakers on the ceiling. This setup allows filmmakers to position and move so-called “audio objects” anywhere in a three-dimensional space. So if you see a scene with a helicopter flying overhead, for example, you’d be able to hear very precise movement as the sound travels from speaker A to B to C to D.
This 3D audio technology provided a solid foundation for an extension into VR audio, but as Susal notes, it was “certainly not a layup.”
From Atmos to VR
For starters, Atmos was designed for theaters, including home setups. Virtual reality, however, is a solitary experience, experienced with a pair of goggles and a set of headphones.
To translate Atmos to headphones, Dolby updated its algorithm to render audio in such a way that tricks the brain into believing sound is coming from multiple sources when there are only two speakers embedded in the headphone cups.
“It’s not really about the number of speakers,” explains Brett Crockett, Dolby’s vice president of research and development. “It’s about recreating spatial resolution.” To achieve this, Atmos directs the audio at specific angles so the sound waves bounce off parts of the ears before reaching the ear canal. (Dolby has also partnered with speaker manufacturers to create Atmos-enabled computer speakers and TV sound bars. In those products, the sound waves bounce off the ceiling to create 3D audio.)
Virtual reality also needs to take into account the fact that people’s left and right ears perceive sound differently, including time delay and volume. To create a general audio profile that’ll translate to a wide audience, many VR filmmakers are creating binaural recordings, a century-old recording techniquethat’s been revived largely because of virtual reality.
To get a binaural recording, microphones are placed in ear-shaped cavities on the sides of a dummy or a stand. This was how the LA Philharmonic recorded the audiofor its VR performance of Beethoven’s Fifth, which viewers could experience in a special truck that was outfitted with carpet and seats from the Walt Disney Concert Hall—along with headphones. (The truck, dubbed the VAN Beethoven, was part of the orchestra’s outreach efforts to provide a vivid concert hall experience for Angelenos who might otherwise never attend a live orchestra performance.)
Because VR viewers are constantly moving their heads to see different parts of a scene, audio needs to remain consistent, so that a knock from the door doesn’t appear to come from the wall just because a viewer changed positions.
“It turns out to be quite difficult to do sound correctly,” says Arthur van Hoff, Jaunt’s cofounder and chief technical officer.
A sense of presence
Filmmakers also have to reconcile moving audio objects and stationary sound—an orchestral soundtrack, for example, that doesn’t come from any particular direction.
“This is a VR-specific feature that never existed before,” explains Susal. Adam Somers, a senior sound engineer at Jaunt, says the VR studio had the capability to mix both fixed and moving audio, but “I think our collaboration with Dolby helped us understand all the nuances of 3D audio.”
Because of the immersive nature of virtual reality, any inconsistencies—including audio—can become glaringly obvious. But done right, virtual reality can create a sense of presence, which in essence fools the viewer’s eyes and brain into believing what’s virtually around them is in fact real.
To Jaunt’s van Hoff, Black Mass , a short film the studio released in 2014 around Halloween, is one of the best examples of how audio can transport viewers to another place. The horror flick opens in an abandoned warehouse, where the viewer soon realizes that she’s been drugged and held captive. “I think you’re bleeding,” a small girl says. In between flashes of darkness, the scenery reveals chain saws, dismembered babies, skulls, dead animals, a hysterical woman dragged by who knows what under a garage door. One CNET writer said the experience “ scared the bejeesus out of me.”
Black Mass was recorded with a type of surround sound called ambisonic audio. “It was scary, but it wasn’t as scary as we were hoping it’d be,” says van Hoff. But after mixing with Atmos in post-production, the short film “felt a lot scarier because you felt people creeping around you. You hear them in the right direction, and it really enhances the experience.”
Fine-tuning Atmos for VR is still an ongoing process. But Jaunt and Dolby have made substantial progress to overcome a number of hurdles.
Read this next: Sorry, YouTube: The virtual-reality revolution will not come via shaky footage shot by amateurs