A nonspeaking person is the star of this opera. AI made that possible

The unlikely opera star: How AI helped a nonspeaking performer find his voice

What if artificial intelligence is used to amplify human creativity rather than replace it?

Rachel Feltman: For Scientific American ’s S cience Quickly , I’m Rachel Feltman. Today we’ll be hearing from a few folks, one of whom is a young man named Jakob Jordan.

Feltman: Jakob has autism and apraxia, which is a disorder of the brain and nervous system that keeps people from carrying out physical actions at their own command. In Jakob’s case, it can be difficult or even impossible to speak his thoughts aloud.

If you're enjoying this article, consider supporting our award-winning journalism by subscribing . By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.

So it might surprise you to learn that Jakob just starred in an opera called Sensorium Ex .

Feltman: In a moment, you’ll hear Jakob using a custom-built AI speech synthesis device that made this opera sing—one that was specifically designed to sound like him and, crucially, to give him expressive control over his own voice.

Jordan: Hearing my voice produce my desired words feels so exhilarating. My mouth is constantly spouting off repetitive loops that I get stuck in. It is not my intended speech, so it can be frustrating when I can’t express my intentions. Sensorium Ex gave me the gift of hearing my trapped thoughts come to life. It’s like my voice was released from prison after a wrongful conviction. I’m so grateful to the Sensorium Ex team for making dreams that felt impossible come true.

Feltman: That sound bite was processed with the AI tech that Jakob was referring to, but later on we’ll hear how it contrasts with his more traditional text-to-speech device.

Along with Jakob, today we’ll be chatting with New York University professor Luke DuBois, who developed Jakob’s AI speech synthesis device, and composer Paola Prestini, who wrote the opera this tech was designed for. Their work together raises a radical question about AI: What if we built a tool not to replace human creativity but to amplify it?

Paola Prestini: It started with a commission from an opera company that then didn’t end up participating in the final run. But they wanted me to do something in relation, actually, to blindness, and at that point I was very, you know, determined to only do something that I had lived experience in or collaborate with someone who had. And I was already working with this amazing poet Brenda Shaughnessy, and Brenda has lived experience as a mother to a nonverbal and nonambulatory child.