In a new research study, AI-driven facial animation company Speech Graphics, teamed up with scientists from UC San Francisco and UC Berkeley to help a paralysed woman in the US speak using a digital avatar that’s controlled through a brain-computer interface (BCI).
The study managed to turn the signals from the woman’s brain into three forms of communication such as text, synthetic voice and facial animation on a digital avatar with corresponding lip sync and emotions.
Dr. Edward Chang, chair of neurological surgery at UCSF with over a decade of experience working on brain-computer interfaces led the group of researchers. The team implanted a paper-thin rectangle with 253 electrodes on the woman’s brain, specifically in areas they found are vital for speaking.
The electrodes picked up the brain signals that would have been sent to the muscles in her tongue, jaw, larynx and face if she had not had a stroke. A cable connected the electrodes to a bank of computers through a port fixed to her head. Over several weeks, AI algorithms were trained to recognise the brain activity associated with a vocabulary of more than 1,000 words.
With the help of the AI, the woman could communicate verbally by ‘writing’ text or ‘speaking’ using a synthesised voice that was created from recordings of her real voice before she was paralyzed.
“Our goal is to restore a full, embodied way of communicating, which is really the most natural way for us to talk with others,” said Professor Chang in a blog post. “These advancements bring us much closer to making this a real solution for patients.”
The company claims that this is the first time facial animation has been synthesised from brain signals and has released a paper that details its research breakthrough in the August edition of the science journal Nature.
Using video game technology
In one approach, the researchers collaborated with Michael Berger, CTO and co-founder of Speech Graphics to develop a system that could decode brain activity into facial movements.
The company’s AI-based facial animation technology is commonly used to create realistic facial animation in popular games like The Last of Us II, Hogwarts Legacy and Halo Infinite can simulate muscle contractions over time, including speech articulations and nonverbal movements.
Then the researchers used the woman’s synthesised voice as a proxy for her actual voice to drive the muscles. Speech Graphics’ real-time software then converted the muscle movements into 3D animation in a video game engine. This resulted in a realistic avatar of the woman that accurately pronounced words in sync with the synthesised voice.
AI-driven faces beyond video games
In a second approach, the woman’s brain signals were directly connected to the simulated muscles, which acted as a substitute for her non-functioning muscles. She could also use the simulated muscles to express specific emotions and move individual muscles.
“Creating a digital avatar that can speak, emote and articulate in real-time, connected directly to the subject’s brain, shows the potential for AI-driven faces well beyond video games,” said Berger. “When we speak, it’s a complex combination of audio and visual cues that helps us express how we feel and what we have to say.
“Restoring voice alone is impressive, but facial communication is so intrinsic to being human, and it restores a sense of embodiment and control to the patient who has lost that. I hope that the work we’ve done in conjunction with Professor Chang can go on to help many more people,” added the CTO.
The team is optimistic that their research will lead to the development of an FDA-approved system that can enable people with paralysis to speak again using their brain signals.
Isa Muhammad is a writer and video game journalist covering many aspects of entertainment media including the film industry. He's steadily writing his way to the sharp end of journalism and enjoys staying informed. If he's not reading, playing video games or catching up on his favourite TV series, then he's probably writing about them.
