AI converts brain waves into speech for paralysed patients in real-time

California-based researchers have developed a groundbreaking AI-powered system that enables real-time speech generation for individuals with paralysis, using their own voices.

This cutting-edge technology, created by scientists at the University of California, Berkeley, and the University of California, San Francisco, represents a significant advancement in brain-computer interface (BCI) research.

The system utilizes neural interfaces to measure brain activity and AI algorithms that reconstruct speech patterns.

It marks a major leap forward from previous efforts, allowing for near-instantaneous voice synthesis—a capability previously thought to be years away.

"Our streaming approach brings the same rapid speech decoding capacity of devices like Alexa and Siri to neuroprostheses," said Gopala Anumanchipalli, assistant professor of electrical engineering and computer sciences at UC Berkeley and co-principal investigator of the study, which was published this week in Nature Neuroscience.

"Using a similar type of algorithm, we found that we could decode neural data and, for the first time, enable near-synchronous voice streaming. The result is more naturalistic, fluent speech synthesis."

The technology can work with various brain-sensing interfaces, including high-density electrode arrays placed directly on the brain’s surface, microelectrodes that penetrate brain tissue, and non-invasive Surface Electromyography (sEMG) sensors that measure muscle activity on the face.

The neuroprosthetic device samples neural data from the motor cortex—the brain region responsible for speech production.

AI then decodes this data into audible speech. Study co-author Cheol Jun Cho explained, "What we’re decoding is after a thought has happened—after we’ve decided what to say, after we’ve chosen the words and planned our vocal tract movements."

To train the AI, researchers collected data from patients silently attempting to speak words displayed on a screen.

This enabled the system to map neural activity to specific speech patterns. Additionally, a text-to-speech model was developed using recordings of the patient’s voice from before their paralysis, ensuring a more natural sound.

The system can begin decoding brain signals and producing speech within a second of a patient attempting to speak—an improvement from the eight-second delay recorded in a 2023 study.

While the generated speech is not yet perfectly fluid, it is significantly more natural and intelligible compared to previous BCI-based speech synthesis technologies.

This innovation could dramatically enhance the quality of life for individuals with conditions like ALS or severe paralysis by enabling more expressive and natural communication with caregivers, loved ones, and the broader world.

Researchers plan to further refine the AI model to speed up processing times and enhance the expressiveness of synthesized speech.

As advancements continue, this breakthrough could pave the way for broader accessibility and improved communication tools for those with severe speech impairments.