Imogen Heap launches funding drive for gloves that turn gestures into music

News: musician Imogen Heap is to put an experimental electronic glove into production, creating a tool that will allow anyone to interact with their computer remotely via hand gestures. Update: this interview is featured in Dezeen Book of Interviews, which is on sale now for £12 (+ interview + movie).

Heap has launched a Kickstarter campaign to raise £200,000 to develop and produce a limited production run of open-source Mi.Mu gloves, with a wider production planned for the future.

"Funding this campaign will enable us to make a really important developmental leap to finalise the gloves' design so they're ready to go into production," Heap said in a video accompanying her Kickstarter campaign.

Each gesture-control glove contains a range of sensors that track the position, direction and velocity of the wearer's hand, the degree of bend in their fingers and the distance between their fingers. It can also understand "postures" such as an open palm, a finger-point or a closed fist.

The resulting data is sent wirelessly to a computer, keyboard and other electronic music equipment, allowing musicians to create music by moving their hands rather than by playing a keyboard or pressing buttons.

"Fifty percent of a performance is racing around between various instruments and bits of technology on stage," Heap told Dezeen in an exclusive interview ahead of the Kickstarter launch. "I wanted to create something where I could manipulate my computer on the move wirelessly so that music becomes more like a dance rather than a robotic act like pressing a button or moving a fader."

The latest version of the gloves was developed as part of Heap's ongoing The Gloves Project, which began four years ago. In 2012 she performed with an early version of the gloves at the Wired 2012 conference.

While developed for musicians, Heap said the gloves could be "hacked" for other uses.

"I'm not claiming they're going to be the answer to every interaction with the computer but there's a lot of applications where it just feels wrong to use a mouse and a keyboard," she said. "You might want to be able to make something in some architecture software where you could stretch a building or draw little windows and quickly move them around like play-dough and maybe we'll get to the point where people will start to develop software like that."

She added: "It's essentially a remote control and anything that you could potentially do with your hands, you could do with your gloves."

The key piece of technology in each glove is an x-IMU board developed by x-IO Technologies, which is mounted on the back of the hand and contains an accelerometer, magnetometer, gyroscope and wifi transmitter. The latest version of the gloves features e-textile technology, where movement sensors are integrated into the fabric.

Below is an edited transcript of the interview with Heap:

Marcus Fairs: Tell us about the gesture-controlled gloves you’ve been working on.

Imogen Heap: I’m a musician but more recently I've been developing some gloves with an amazing team of people to help me make music on the move gesturally, enabling me to interact more naturally with my music software, to more freely create music on the move, in the flow of things.

Marcus Fairs: So they’re to allow you to make music without having to be tied to keyboards or other physical instruments?

Imogen Heap: Fifty percent of a performance is racing around between various instruments and bits of technology on stage. For instance, pressing a record button doesn't look or feel very expressive but actually that moment of recording something is a real creative act; it's a musical act.

But these actions have always been hidden from the audience and they disengage me in my performance, so I wanted to find a way to do that and integrate it into the performance. It's the unseen that I'm interested in bringing out of hiding.

There are so many types of sounds or effects that don't have a physical existence. They are software, they are hidden inside the computer. A bass-line might sound sculpted; it might have this blobby, stretchy sound. For me doesn't feel natural to play a sound like that on a keyboard because a keyboard is very restrictive and very linear and you only have two hands. I can play a melody but if I wanted to manipulate any kind of parameter of that sound, my other hand is completely used up. It’s quite restrictive.

I wanted to find a way to be really expressive in using these software instruments and effects that feel like how I feel they should be played and how I feel that represents the sound that's coming out of the speakers.

So in order to free myself up on the stage from my various bits of technology and to bridge the gap between what’s going on on stage and the audience, I wanted to create something where I could manipulate my computer on the move wirelessly so that music becomes more like a dance rather than a robotic act like pressing a button or moving a fader.

Marcus Fairs: How do the gloves work?

Imogen Heap:  They have bend sensors in the fingers, they have lights for feedback, they have buzzers integrated in the side so I can sense where I am if I want to get a haptic feedback. They also contain a microprocessor unit that has an accelerometer, a magnetometer and a gyroscope in it.

We've been developing them for about four years and they've come a long way. We started with fibre-optic bend sensors in the fingers but we quickly realised that we needed positional data, accelerometer data, gyroscope data so that we could really be inside the music. Because actually just having the bend sensors in the beginning was almost like just pressing buttons. It felt very unnatural.

It began with little lapel microphones, which are made by Sennheiser. Seven years ago I began to stick them onto my wrists so that I could make sound with wine glasses or I could play my mbira on stage. I would be able to avoid putting microphones on stage for festivals or touring so it would cut down on the weight and the transport costs, which is also a reason for the gloves.

In the early versions of the gloves it all connects to a hub that I wear on my upper body. It's quite complicated but it basically communicates with the computer wirelessly so I can use it to manipulate music software, enabling me to unchain myself from the computer, to humanise the missing bits of how I interact with technology in music.

I use these gloves with a Kinect so that I can have an extra dimension on top of local gestural movements; I can use the stage as a playground like different zones for whole different presets. I could map the centre of the stage for a certain key, and if I go over to the right and I combine it with a gesture so that I don't accidentally go in there, then I can have a whole different key or a whole different set of sounds to play with. I could unmute and mute different instruments that are inside the music software.

There's really nothing out there on the market like this, that enables me to be this expressive with music on the move in the studio and on the stage. It's very exciting. When you see me play, not maybe every time because maybe it goes wrong or I go wrong, but when it works, when it's effortless and when your movement is part of the music, it's almost like a dance. It's so natural that the tech disappears.

Marcus Fairs: Tell us about the latest version of the gloves.

Imogen Heap: It’s very exciting because it's so much simpler and it needs less gear, less setting up. As you can see it's compact and doesn't need so many extra wires and the main reason for that is this: it has an x-IMU board by Seb Magdwick of x-IO Technologies containing an accelerometer, a gyroscope, a magnetometer but the main difference is it now has wifi built into the glove. So it doesn’t need an extra unit to send information to the computer.

That is incredible because it's sending Open Sound Control data instead of MIDI serial data. There are two bend sensors in the wrists and we've still got the bend sensors in the fingers and "forchettes" between them telling us how closed or open my hands are and equally how much my hands are bending. We're finding that the bend sensors so far are the simplest solution but really we want to get to the point where it's all e-tech style. So that we can separate the hard tech from the soft tech.

Marcus Fairs:What’s e-tech?

Imogen Heap:  E-tech is electronic textiles. So information is passing through fabric by using conductive threads or materials. This is where we are and it's beautiful.

But at the moment it's really simple, it just sees this exoskeleton as a device and then it comes up on your computer as a wifi device and you're ready to go. It's super simple and it's great.

Marcus Fairs: Could the gloves be used for other creative uses besides music?

Imogen Heap: A lot of people have been in touch. For instance a guy suggested that he could take all the international sign language which you only need one hand for and translate that sonically, so that each posture for a word or gesture for a word could be mapped and generate a word. You could hack a little speaker onto the system so it could actually speak for you as well. So that's one idea.

And in the video for Me, The Machine, which is a song that I wrote with the gloves and for the gloves, you see me manipulating visuals with them. Just drawing lines onto a screen that's in front of me so you can see me drawing in real time.

It’s great fun to do. I can draw little arrows and houses and people. It's not like using a pencil; it's incredible to be able to create these grand shapes, to be able to shift everything, painting out of nothing and spinning it around and stopping it and moving it over here. So I imagine a few people might start to use them with visuals.

Marcus Fairs: What about non-creative uses? Could these gloves be used by surgeons for example, or pilots or bus drivers?

Imogen Heap: I think there's a lot of applications; it doesn't have to be like you're painting or making music with them. For our Kickstarter campaign, we've been starting to think about funny things we could pretend to do with them. So I suppose as long as you can access your computer inside your car, there's no reason why you couldn't just sit in the back of your car and indicate right or left. It's a remote control. It feels like an expressive musical instrument sometimes but it's essentially a remote control and anything that you could potentially do with your hands, you could do with your gloves.

Marcus Fairs: Do you plan to manufacture and sell them as a product?

Imogen Heap: We would love the gloves to be as affordable as something like a MIDI keyboard in time. Imagine if this was something that people would just go to use as one of those expressive things that they feel can't be done with certain types of more rigid technology, because what is exciting about them is that they're totally customisable.

You can even hack them, so you might want a screen or maybe you'll want a push button thing, but something that gives off a smell when you move your hand. It's really exciting to see what people might do with hacking them. The software is going to be open source and so is the hardware. We can't wait to see what people do with them. It's early stages.

Marcus Fairs: There’s a lot of talk about how wearable technology could remove the need to interact with computers. How do your gloves fit into that trend?

Imogen Heap: I'm not claiming they're going to be the answer to every interaction with the computer but there's a lot of applications where it just feels wrong to use a mouse and a keyboard. You might want to make something in some architecture software where you could stretch a building or draw little windows and quickly move them around like play-dough and maybe we'll get to the point where people will start to develop software like that. That would be amazing.