From Audrey to Eleven v4: How Voice Agents Grew Up

Today a voice agent can pick up a customer call, understand an accent it has never heard, check a booking system and reply in a natural voice before the caller notices a pause. It can switch from English to Arabic mid-sentence. That feels like it arrived overnight, but it is the result of seventy years of work on two separate problems: getting machines to understand speech, and getting them to speak back.

This post walks through the milestones that matter for anyone building voice products today, and ends with the model ElevenLabs released on September 28, 2026.

1950s to 1990s: machines that recognized a handful of words

In 1952, Bell Labs built Audrey, a machine that could recognize the spoken digits zero to nine, but only from a single speaker whose voice it had been tuned to. A decade later, IBM showed the Shoebox at the 1962 Seattle World’s Fair. It understood 16 words: the ten digits and a few arithmetic commands.

The 1970s brought government money and bigger vocabularies. Carnegie Mellon’s Harpy system, built under a DARPA speech research program, could handle about 1,000 words by 1976. Through the 1980s, statistical methods based on hidden Markov models replaced hand-written rules and became the foundation of speech recognition for the next thirty years.

By the late 1990s the technology reached consumers. Dragon NaturallySpeaking (1997) let people dictate continuous speech on a home PC, and interactive voice response systems put “say or press one” menus on almost every customer service line. They worked, but only inside narrow scripts, and anything unexpected sent the caller back to the main menu.

2011 to 2016: an assistant in every pocket

Apple’s Siri arrived with the iPhone 4S in 2011, followed by Google Now, Microsoft’s Cortana, Amazon’s Alexa on the Echo speaker in 2014, and Google Assistant in 2016. Speech recognition improved sharply as deep neural networks replaced older acoustic models, and suddenly hundreds of millions of people talked to their devices every day.

These assistants were still command systems underneath. Each request had to map to a predefined intent, such as setting a timer, playing a song or checking the weather. Anything outside the list got a polite apology. They could recognize speech well; they could not really hold a conversation.

2016 to 2022: neural voices and near-human transcription

The speaking side changed next. DeepMind’s WaveNet (2016) generated audio sample by sample with a neural network, and Google’s Tacotron 2 (2017) produced speech that listeners struggled to tell apart from a human recording. In 2018, the Google Duplex demo showed an AI calling a hair salon to book an appointment, complete with “um”s and natural pauses.

On the listening side, OpenAI released Whisper in September 2022: an open-source model trained on 680,000 hours of audio that transcribed dozens of languages robustly, including noisy recordings. Accurate multilingual transcription stopped being a competitive advantage and became something any developer could download.

2023 to 2024: the LLM cascade and the first real voice agents

Two months after Whisper, ChatGPT arrived, and developers quickly built the architecture that powered the first generation of real voice agents: a cascade of speech-to-text, then a large language model, then text-to-speech. For the first time, the “brain” in the middle could handle open-ended questions instead of a fixed list of intents.

ElevenLabs became the default voice layer for many of these systems, with expressive multilingual text-to-speech and voice cloning, and platforms such as Vapi, Retell, LiveKit Agents and Pipecat made cascades easier to wire into phone lines. The weak point was latency: three models in a row often meant a second or more of silence before each reply, which callers notice immediately.

In May 2024, OpenAI demonstrated GPT-4o speaking and listening natively, and in October 2024 it opened a Realtime API for speech-to-speech models that skip the text step entirely. That set up the debate that still shapes voice architecture today: a cascade you can inspect and customize at each step, or a single native model that responds faster and sounds more natural.

2025 to 2026: reasoning voices and the race to 100 milliseconds

The last two years turned voice agents from demos into production systems. OpenAI moved its realtime model to general availability in 2025, then released GPT-Realtime-2 in May 2026, bringing GPT-5-class reasoning, a 128,000-token context window and adjustable reasoning effort to voice agents. Google answered on September 15, 2026 with Gemini 3.8 Live, native speech-to-speech models that detect and switch between 97 languages mid-conversation.

At the same time, text-to-speech kept getting faster and more expressive, which matters because many production agents still use a cascade for control, compliance and tool use. The competition is now measured in milliseconds of time to first audio, how well a voice handles interruptions, and whether it still sounds like the same person twenty minutes into a call.

Eleven v4: the latest step, released yesterday

On September 28, 2026, ElevenLabs released Eleven v4 and Eleven v4 Turbo. Eleven v4 is built on a new architecture that reads tone, pacing, emotion and context from the text, so a line can come out urgent, tender or conversational while the voice keeps its identity over long passages. According to TechCrunch, it supports more than 90 languages, up from 70 in v3, lets you stack several expression tags and have them followed in order, and can clone a voice from 10 seconds of audio.

Eleven v4 Turbo is the version built for voice agents. ElevenLabs reports a median inference latency of about 100 milliseconds and around 150 milliseconds to first speech, and it streams in both directions, so audio starts playing while the LLM behind it is still writing the sentence. The release notes also call out better handling of the hard parts of real calls: confrontations, escalations and putting a caller on hold. Both models are available in ElevenAgents, ElevenCreative and the API, including the free tier, and early coverage reports Eleven v4 ranked first on Artificial Analysis, with roughly 75% of listeners preferring it in blind comparisons.

For builders, the practical effect is that the slowest link in a cascade just got much shorter. With speech output starting around 150 milliseconds, a well-built cascade can now feel close to a native speech-to-speech model while keeping every step observable and replaceable.

What this means if you are building a voice agent

Seventy years after Audrey, the hard problems have moved. Recognizing speech and producing a natural voice are largely solved. The work that decides whether a voice agent succeeds is now:

• The latency budget across the whole loop, including your own tools and databases, not just the model.
• Choosing the architecture per use case: a cascade when you need control, audit trails and custom models; native speech-to-speech when conversational feel matters most.
• Language and dialect coverage for your actual callers. For the UAE and the wider Gulf, that means Arabic and English in the same conversation.
• Guardrails and handoff, so the agent knows when to escalate to a person.

At BrilliMinds we build AI-powered products for companies in the UAE and abroad, including conversational features that have to work in both Arabic and English. If you are planning a voice agent, talk to us about the architecture before you pick a model.