London designer Arthur Carabott has worked with composer Anna Meredith to create an augmented reality app that immerses listeners in a piece of music.
Called Moonmoons AR, the application allows users to place six virtual speakers anywhere in their surrounding environment, whether close or far away, in front of or behind them.
Meredith's song, also named Moonmoons, is then played by the app with each instrument – from the bass to the cello – played through a different one of these speakers.
By physically moving between the different sound sources, users can craft their own personal "aural experience", changing the direction from which they hear the instrument and its volume – the closer they get, the louder it will be.
This effect is called spatial audio, which Carabott explains is "a bit like virtual reality for the ears".
"It means the listener has a sense of the direction and distance of each sound source, much like being in a room with a group of musicians: if the drummer is to your left, and the cellist to your right, you can tell where they are just by listening," he told Dezeen.
"A spatial audio engine allows you to position sounds in a virtual space, for example to the left of the listener, and if they are wearing headphones it will sound as if that instrument is coming from that location."
The app was designed specifically for Moonmoons, a song which lends itself to spatial audio both thanks to its easily isolated instruments, as well as more conceptually through its underlying theme.
“Moonmoons are the moons of moons," said Meredith. "And this idea of intergalactic worlds within worlds and of doubling and playing with scale, of zooming in and out is explored right through the piece."
Carabott built the app using the development platform Unity and used Google's Resonance software development kit to process the spatial audio. For the virtual speakers, they decided to digitally recreate the ceramic objects which Meredith's sister Eleanor had originally created for the cover of her album Fibs.
"The prototype used very generic objects as speakers: cubes, spheres, and pyramids," said Carabott. "Once we decided to use Eleanor's ceramics, I set about learning about photogrammetry – 3D scanning using a regular camera – which I did using Agisoft's Metashape software."
The designer believes that spatial audio is the next logical step in the symbiotic relationship between technology and music.
"New pieces of technology have always lead to new musical ideas, and musicians have had new ideas that have demanded new pieces of technology," he explained.
"It's easy to forget that recording is a relatively new invention, where before the only way to hear music was to be in a room with musicians. Nowadays most people's 'listening venue' is a pair of headphones, which opens up new opportunities for things like spatial audio."
But crucially, that doesn't mean that in the future, all audio will be spatial.
"I think the key is to make it optional," said Meredith. "Of course, you can just listen to this as a straight track and hear the mix that I wanted for the final thing. But some of the people who like my stuff really like to dig around under the hood of the music, and unpick the layers. This offers a playful and personal way to do that."
Carabott is an interaction designer, who has previously created a superhydrophobic water fountain and an audio-visual device that helps musicians improve their posture and technique.
Other recent experiments hoping to make music more immersive include water-filled, bone-conducting headphones by designer Rocco Giovannoni and an AR campaign for the release of The 1975's song People.