Semantic Decoder: An AI System that Converts Brain Activity into Text

Researchers at The University of Texas at Austin have developed a new AI system, called a semantic decoder, which is capable of translating a person's brain activity into a stream of text while listening to or imagining a story. The system could potentially aid individuals who are mentally conscious but physically unable to speak, such as stroke victims, to communicate effectively again. The researchers published their work in the journal Nature Neuroscience, relying on a transformer model similar to those used in Open AI's ChatGPT and Google's Bard. Brain activity is measured using a functional MRI scanner following extensive training of the decoder, during which the individual listens to hours of podcasts in the scanner.

Artificial Intelligence Aiding in Early Detection of Breast Cancer in Certain US Hospitals:

Participants open to having their thoughts decoded later listened to a new story or imagined telling a story, allowing the machine to generate corresponding text from brain activity alone.

While the result is not a word-for-word transcript, it captures the gist of what is being said or thought.

About half the time, when the decoder has been trained to monitor a participant’s brain activity, the machine produces text that closely – and sometimes precisely – matches the intended meanings of the original words.

A participant who listened to a speaker say that they do not have their driver's license yet had their thoughts translated as, "She has not even started to learn to drive yet."

The researchers said they were getting the model to decode continuous language for extended periods of time with complicated ideas.

AI Chatbot's "Bedside Manner" prefferred over conventional Doctors by Shocking margin, according to blind study

The researchers conducted a study where the participants were required to watch four short, silent videos while inside a scanner. The semantic decoder was able to accurately describe certain events from the videos using their brain activity. In addition to listening or thinking about stories, this approach was used to test the system's performance. However, it was observed that when the system was tested on people who had not been trained, the results were unintelligible.

The newly developed semantic decoder system, which has the ability to translate a person's brain activity into a continuous stream of text, is currently not suitable for use outside the laboratory due to its dependency on the time needed for an fMRI machine. Nevertheless, researchers have suggested that this technology could be adapted for use with other, more portable brain-imaging systems in the future. Despite concerns that such technology could be misused, study leader Jerry Tang, a doctoral student in computer science, emphasized that the team has taken steps to prevent this from happening and to ensure that the technology is only used when necessary and for beneficial purposes.