Brain Implants Can Now Decode Our Gestures, Too

New technology that simultaneously transfers the attempted speech and movements of people with paralysis to a digital avatar is prompting us to reconsider the limits of human communication.

Imagine telling someone an exciting story with your hands behind your back. The words remain, but something may be missing from your expression and emphasis. A hand movement showing surprise or the distance you indicate when saying “it was this big” disappears. Does communication consist only of words, or are bodily movements also part of how we express our thoughts?

For people who have lost their ability to speak and move due to paralysis, this question is not merely theoretical but directly affects daily life. Although expressing thoughts in writing is an important achievement, communication often goes beyond words. Waving, indicating an object's size, or supporting speech with gestures adds different layers of meaning.

A study published in Nature Neuroscience on September 14, 2026, marked an important step toward restoring this multilayered communication through brain-computer interfaces. Researchers at the University of California, San Francisco (UCSF), developed a system that simultaneously decodes attempted speech and gestures using signals from a single brain implant.

Are Gestures Merely Decorative Elements of Speech?

We generally consider hand movements behaviors that accompany speech. However, psycholinguistic research shows that gestures do more than decorate communication.

For example, a 1998 study by Jana Iverson and Susan Goldin-Meadow demonstrated that people who are blind from birth spontaneously gesture while speaking. This behavior continues even when their conversation partner is also blind. Thus, gesturing does not require learning by observing others or having a listener who can see the movements.

This finding suggests that gestures do not exist solely to convey visual information. Hand movements may also be related to organizing and expressing thoughts. Therefore, technologies aimed at restoring natural communication may exclude an important aspect of human expression if they focus only on words.

Previous Advances in Neurotechnology

Brain-computer interfaces (BCIs) record electrical brain activity and convert these signals into commands a computer can process. Technologies developed for people with severe paralysis who have lost their ability to speak have advanced in recent years.

In a 2023 study published in Nature, Edward Chang's team at UCSF developed a system that converted brain signals from attempted speech into text and synthetic speech while animating a digital facial avatar. The participant silently attempted speech movements, while the system analyzed neural activity patterns to predict the intended expression.

However, human communication involves more than movements of the speech organs; we use our entire bodies. This raises a question: Can brain signals related to speech and gestures be decoded simultaneously?

The New Study: One Implant, Two Communication Channels

The 2026 study by Samantha Brosler and colleagues used high-density electrocorticography (ECoG). In this method, electrodes are surgically placed inside the skull, on the brain's surface, to record cortical electrical activity. The implant collected signals from a broad sensorimotor cortex region associated with speech production and bodily movements.

Initially, researchers examined neural activity associated with speech and different body movements in three participants with varying degrees of paralysis. They then conducted experiments with two participants to decode speech and gestures together and control personalized digital avatars.

Because participants differed in movement abilities and strategies, researchers selected different sets of expressions and gestures. Participant Bravo-6 used ten speech expressions, ten gestures, and one hundred speech–gesture combinations. Participant Bravo-1r used five speech expressions, four gestures, and twelve combinations.

Participants attempted specific expressions and gestures. Bravo-1r attempted silent speech and gestures through limited movements, while Bravo-6 made overt speech attempts and mentally imagined gestures.

Two artificial intelligence models operated in parallel, one classifying neural patterns associated with speech and the other with gestures. Their outputs were transferred to personalized, full-body digital avatars.

The avatar performed predicted gestures while speech appeared as text. Participants could therefore communicate digitally using predefined expressions and gestures, separately or together.

How Successful Were the Results?

When Bravo-6 attempted speech and gestures simultaneously, the system classified speech expressions with 70% accuracy and gestures with 66% accuracy.

Both models selected among ten target classes and one resting class, giving an approximate chance level of 9%. These results indicated that the system extracted meaningful information from neural activity patterns.

Researchers also evaluated the system when participants attempted to answer conversational questions rather than repeat predefined expressions. For Bravo-6, average accuracy reached 75% for speech and 85% for gestures.

For Bravo-1r, who used a smaller set of expressions and gestures, both models achieved a median accuracy of 100% across three conversational blocks.

Another important finding was that the system could decode speech–gesture combinations it had not encountered during training. Since teaching every possible combination separately would be impractical, this finding is important for developing more flexible communication systems.

Are Speaking and Gesturing Independent Processes in the Brain?

One of the study's most interesting findings for psychology and cognitive neuroscience concerned difficulties in decoding both movements simultaneously.

The researchers found that models trained on speech and gestures separately performed less accurately when both movements were attempted together.

One reason was that neural activity related to speech and gestures partially overlapped in the sensorimotor cortex. Attempting a movement alone or alongside another movement could also affect recorded activity patterns.

This reduced classification accuracy and caused false activations between models. For example, models trained using only genuine resting examples sometimes predicted one movement when participants attempted the other. For Bravo-1r, the median false-positive rate reached 76% when the alternative movement type was attempted.

To address this, researchers included examples of the opposite movement during training. The speech model learned to recognize both genuine rest and attempted gestures as examples without speech. The same approach was used for the gesture model.

This adjustment substantially reduced false activations when the other movement was attempted.

What does this tell us about human cognition?

Speech and gestures may not be entirely independent at the neural level. However, the study does not establish that both behaviors rely on a single shared mental mechanism. Instead, it shows that neural patterns associated with speech and gestures can be influenced by behavioral context, which must be considered when training artificial intelligence models.

Does the System Really Read Our Body Language?

In everyday language, body language generally includes facial expressions, posture, emotion-related movements, and unconscious behaviors.

However, this system does not decode hidden emotions or unconscious bodily movements. Instead, it classifies brain signals associated with attempts to produce specific speech expressions or gestures.

Therefore, this is not mind-reading technology but a system that converts neural activity related to movement production into previously learned categories.

The technology remains limited. Participants cannot freely construct sentences or move as they wish. Communication relies on expressions and gestures defined during training.

A practical interface will require larger vocabularies, more flexible movement control, and faster, more reliable decoding methods.

Questions Psychology Has Yet to Answer

Self-Perception and Body Ownership

To what extent might people experience a digital body controlled by their brain activity as an extension of themselves over time?

The rubber hand illusion, introduced by Botvinick and Cohen in 1998, demonstrated that combining visual and tactile information under certain conditions could make people perceive a hand that was not theirs as belonging to their body.

However, how a person with paralysis would experience a digital avatar controlled through brain signals remains a separate research question. Controlling an avatar does not automatically produce a sense of body ownership.

The Perception of Others

Communication is a two-way process. Therefore, evaluating this technology should not focus solely on how accurately users produce expressions.

How natural would another person find a digital avatar's gestures? Could presenting words and movements simultaneously make messages seem more understandable or personal?

Autonomy and Freedom of Expression

Perhaps the most important issue is how much control individuals have over their communication.

If someone can use only predefined gestures and expressions, to what extent can they communicate in a way they have chosen?

As technology develops, researchers will need to evaluate not only accuracy but also how well systems reflect users' preferences, how errors can be corrected, and who controls communication.

The current study does not answer these questions. However, advances in neurotechnology make these fundamental psychological questions about human experience increasingly important.

Conclusion: What Does It Mean to Restore Communication?

The idea that human communication involves more than words is not new to psychology. However, reconstructing this multilayered structure using brain signals represents an important advance in neurotechnology.

Brosler and colleagues demonstrated that attempted speech and gestures can be decoded simultaneously from a single cortical implant and expressed through a digital avatar.

Of course, the system does not yet capture the complexity of natural communication. The small number of participants, predefined expressions, and controlled experimental conditions require caution when generalizing findings to everyday life.

Nevertheless, the study encourages us to reconsider what restoring communication actually means.

Communication is not simply delivering a message. Sometimes, accompanying a greeting with a hand movement, gesturing while telling a story, or expressing thoughts in a personally chosen way is equally important.

Perhaps the goal of future brain-computer interfaces will be not only to restore speech but also to develop tools that allow people to express themselves as naturally, personally, and freely as possible.

Kaynakça

• Brosler, S. C., Liu, J. R., Silva, A. B., et al. (2026). Simultaneous speech and gesture decoding for multimodal communication in paralysis. Nature Neuroscience, 29, 2445–2455. https://doi.org/10.1038/s41593-026-02446-2
• Metzger, S. L., et al. (2023). A high-performance neuroprosthesis for speech decoding and avatar control. Nature, 620, 1037–1046. https://doi.org/10.1038/s41586-023-06443-4 Neuroscience: Brain implant brings speech and gestures together (Nature Neuroscience). (z.d.). Nature Portfolio. https://www.natureasia.com/en/info/press-releases/detail/9440
• Iverson, J. M., & Goldin-Meadow, S. (1998). Why people gesture when they speak. Nature, 396, 228. https://doi.org/10.1038/24300
• Botvinick, M., & Cohen, J. (1998). Rubber hands ‘feel’ touch that eyes see. Nature, 391, 756. https://doi.org/10.1038/35784
• Yee, N., & Bailenson, J. (2007). The Proteus effect: The effect of transformed self-representation on behavior. Human Communication Research, 33(3), 271–290. https://doi.org/10.1111/j.1468-2958.2007.00299.x