Integrate
Realtime protocol
The WebSocket that carries audio and conversation events, and how the avatar video arrives.
A live conversation is two connections: a WebSocket for microphone audio and conversation events, and a WebRTC stream for the avatar's video and voice.
Connecting
Open a WebSocket to the voice_relay_url returned by session creation. The
session is already authorised by that URL; there is no separate handshake.
The connection can drop — mobile networks, laptop sleep, flaky Wi-Fi. Reconnect to the same URL with backoff; the session survives a brief disconnect, and the conversation resumes rather than restarting.
Messages you send
| Type | Payload | Purpose |
|---|---|---|
audio_data |
{ audioData, metadata: { sampleRate: 16000, channels: 1, format: 'wav' } } |
Microphone audio, base64-encoded, streamed continuously |
text |
{ text } |
Inject a user turn as text — for typed input or testing |
vad_status |
{ speaking: true | false } |
Client-side voice activity hint |
control |
{ action: 'interrupt' } |
Stop the avatar talking, immediately |
bot_audio |
{ active: true | false } |
Tells the server whether avatar audio is currently playing out of the speaker, so it can suppress echo |
Send 16 kHz mono. Resampling elsewhere costs accuracy, and the recognition model is tuned for it.
bot_audio matters more than it looks: without it, a kiosk with speakers
hears its own avatar and transcribes it as user speech.
Messages you receive
Connection lifecycle
| Type | Meaning |
|---|---|
connected |
Relay accepted the socket |
backend_connected / backend_disconnected |
The GPU worker attached or dropped |
ready |
The pipeline is warm — from here, speech is processed |
state |
Coarse pipeline state changes |
What the user said
| Type | Meaning |
|---|---|
partial_transcript |
Interim recognition, changes as the user keeps talking |
transcription_start / transcription_chunk / transcription_end |
Streaming final transcript |
transcription |
Complete final transcript for one user turn |
What the avatar says
| Type | Meaning |
|---|---|
response_start |
Generation began |
response_chunk |
Incremental answer text — use it for subtitles |
response_end |
The answer is complete |
chat_message / chat_history |
Individual turns, and backfill on reconnect |
processing_status |
Progress hints for long operations |
Conversation-level
| Type | Meaning |
|---|---|
flow_event |
Progress through a structured flow — step entered, field captured, tool called |
conversation_end |
The conversation is finished. Tear down the UI and DELETE the session |
Treat unknown message types as ignorable. New ones get added; a client that throws on an unrecognised type will break on an upgrade that harms nobody else.
Video
2D backends return a whep_url. WHEP is a small, standard HTTP handshake
for WebRTC playback: POST your SDP offer, get an SDP answer, attach the
resulting media stream to a <video> element. Expect roughly 1 Mbps of H.264
video plus AAC audio.
UE5 backends return a stream_url instead and use the Unreal pixel-streaming
transport.
Play the video element muted until the user interacts, then unmute — browsers block autoplay with sound, and this is the difference between a working avatar and a silent one.
Turn-taking
The server owns turn-taking: it runs voice activity detection, decides when a
user turn has ended, and cuts the avatar off when the user interrupts. Your
client does not need to arbitrate. What it should do is send bot_audio
honestly, stream microphone audio continuously rather than in gated bursts, and
stop local playback the moment the user starts speaking.