Baseten Launches Voice Agent Stack at Roughly One-Second Latency
Baseten opens its Voice Agent Stack in private preview with roughly one-second median turn latency, per-component scaling and $0.012 per audio minute at 32 sessions.
By Daniel Okafor
4 min read
What's News
- Baseten announced the Voice Agent Stack and Realtime Voice in private preview on November 16, 2026.
- Median time to first audio rose from 0.94 seconds at one concurrent call to 1.14 seconds at 56.
- Baseten estimated $0.012 per audio minute at 32 concurrent sessions and $0.008 per call-minute at 56.
- Coval ranked Baseten-hosted Qwen3 ASR 1.7b second with 33 ms median Time to Final Segment and a 2.1% word error rate.
- OpenAI's Python SDK cannot connect directly because Baseten's endpoint does not serve the /realtime path.
By Daniel Okafor
Baseten reported roughly one-second median turn latency across up to 56 concurrent sessions when it opened its Voice Agent Stack and Baseten Realtime Voice to private preview on November 16, 2026. The stack co-locates speech models with orchestration, lets customers pick each model independently, and scales every pipeline stage on its own. Access runs through a waitlist.
What does co-location actually fix?
The Voice Agent Stack runs on Baseten Dedicated Inference. The orchestrator and the models sit in one cloud region, connected over an in-cluster network. Baseten said a remote orchestrator can add hundreds of milliseconds of network round trips per turn — even when speech-to-text, the language model and text-to-speech all run on Baseten already. The stack adds inference-runtime optimization, inter-model networking, traffic autoscaling, cross-cloud failover and model post-training support.
Realtime Voice serves as the reference implementation. It runs a cascade of Baseten-optimized open-weight models for speech-to-text, language processing and text-to-speech, driven by a custom Pipecat-based orchestrator. An optional delegation split does the interesting work: a fast, co-located talker model carries the conversation while a stronger external thinker model asynchronously handles reasoning, web search, decisions and tool calls. Server-side tools execute on the co-located orchestrator, which can also call remote Model Context Protocol servers and avoid extra client round trips. Client-side tools are supported too.
How much control do operators get?
Customers choose the speech-to-text, language and text-to-speech models independently, mix dedicated components with third-party APIs, and move individual pieces onto dedicated infrastructure over time. Post-training options cover adapting transcription to industry jargon, tuning language-model behavior for calls, and supervised fine-tuning of speech with a customer's own voice data. A live session can switch its language model, voice or transcription route without redeployment.
The API matches the core OpenAI Realtime event protocol. It supports audio-in/audio-out and text-in/audio-out modes, plus transcription keyword boosting, configurable acknowledgments and voice cloning. Compatibility has limits. Clients must accept a full WebSocket URL. OpenAI's Python SDK cannot connect directly because it appends /realtime to its base URL, a path Baseten's endpoint does not serve. Browser clients need a backend relay to hold the API key and proxy the socket, since browsers cannot set the required Authorization header.
What do the latency and cost numbers show?
Median time to first audio rose from 0.94 seconds at one concurrent call to 1.14 seconds at 56. The tests used contexts up to 16,000 tokens. Longer contexts can push turn latency up, while configurable summarization holds each session to a target context length.
The test configuration ran Qwen3-ASR on one RTX-6000, Gemma-4-31B on one B200, Qwen3-TTS on another RTX-6000 and GLM-5.3 Fast through Baseten Model APIs for delegation. Each component scales independently against its own concurrency target, so operators add capacity where a stage saturates instead of duplicating the whole pipeline. Baseten said the design preserves strict p90 and p95 latency tails while keeping GPU utilization high.
Cost: Baseten estimated $0.012 per audio minute at 32 concurrent sessions, excluding per-token reasoning-model delegation, and $0.008 per call-minute at 56 sessions including Model API delegation, under its stated utilization assumptions. Dedicated pricing amortizes hardware across sessions, which makes sustained utilization central to the economics.
What does the benchmark leave out?
An independent Coval leaderboard, with data timestamped 17:30 UTC on November 16, 2026, ranked Baseten-hosted Qwen3 ASR 1.7b second by median Time to Final Segment at 33 milliseconds, with a 2.1% word error rate over 456 samples. The metric starts at a shared end-of-speech point and stops when the final transcript arrives. It excludes the time a live system spends detecting that a turn ended, so it does not measure the full agent response.
What comes next?
Baseten listed future work: more complex orchestration flows, easier migration of existing voice pipelines, native speech-to-speech full-duplex systems, omni models with dedicated speech-generation heads and new scheduling primitives across the cascade. The launch makes orchestration the control point for voice-agent economics — model choice, tool execution and component-level capacity now sit in one place, giving operators several ways to tune latency and GPU use without rebuilding the pipeline.
Source: unite.ai
More from Daniel Okafor
Show full bio
Correspondent covering money, funding and market shifts around entrepreneurship, startups and venture funding at Business Bearings.
2,189 articles