IMMERSA Voice Chat API

Message flow

Session states

The server tracks a state machine per session. State is included in the start_session ack.

CLIENT → SERVER start_session

Initialise the conversation. Send once after connection_established.

JSON payload

{
  "type": "start_session",
  "character_id": "s1",
  "sample_rate": 16000,
  "audio_format": "wav_base64_chunks"
}

Fields

CLIENT → SERVER audio_chunk

One chunk of the user's audio, base64-encoded. Send in order, starting at chunk_index: 0.

JSON payload

{
  "type": "audio_chunk",
  "chunk_index": 0,
  "audio": "<base64-encoded WAV data>"
}

Fields

CLIENT → SERVER end_of_utterance

Signal that the user has finished speaking. This triggers the full pipeline: final STT → LLM → TTS streaming.

JSON payload

{
  "type": "end_of_utterance"
}

CLIENT → SERVER close_session

Gracefully end the session. The server sends a final ack then closes the WebSocket.

JSON payload

{
  "type": "close_session"
}

SERVER → CLIENT connection_established

Sent immediately after the WebSocket handshake. Contains the session ID you should log for debugging.

JSON payload

{
  "type": "connection_established",
  "session_id": "3f4a1b2c-8d9e-4f5a-b6c7-d8e9f0a1b2c3",
  "message": "WebSocket connected successfully"
}

Fields

SERVER → CLIENT ack

Generic acknowledgement for every client message. The event field tells you which message is being acked. Extra fields depend on the event.

ack for start_session

{
  "type": "ack",
  "event": "start_session",
  "message": "Session started successfully",
  "session_id": "3f4a1b2c-...",
  "character_id": "s1",
  "sample_rate": 16000,
  "audio_format": "wav_base64_chunks",
  "state": "LISTENING"
}

ack for audio_chunk

{
  "type": "ack",
  "event": "audio_chunk",
  "message": "Audio chunk received",
  "chunk_index": 0,
  "total_chunks": 1
}

ack for end_of_utterance

{
  "type": "ack",
  "event": "end_of_utterance",
  "message": "Processing started"
}

ack for close_session

{
  "type": "ack",
  "event": "close_session",
  "message": "Session closed successfully"
}

SERVER → CLIENT partial_transcript

Intermediate STT result emitted every 5 audio chunks while the user is still speaking. Use for live captions.

JSON payload

{
  "type": "partial_transcript",
  "text": "What is hydraulic pressure",
  "chunk_index": 4,
  "window_size": 5
}

Fields

SERVER → CLIENT final_transcript

Complete transcription of the user's full utterance, produced after end_of_utterance.

JSON payload

{
  "type": "final_transcript",
  "text": "What is hydraulic pressure and how is it measured?"
}

Fields

SERVER → CLIENT reply_text_done

The character's complete text reply from GPT. Emitted before TTS audio starts — show a text bubble immediately without waiting for audio.

JSON payload

{
  "type": "reply_text_done",
  "text": "Well, hydraulic pressure is the force exerted by a confined fluid...",
  "length": 147
}

Fields

SERVER → CLIENT tts_audio_chunk

One chunk of the character's voice audio, base64-encoded MP3. Buffer and play sequentially.

JSON payload

{
  "type": "tts_audio_chunk",
  "chunk_index": 0,
  "audio": "<base64-encoded MP3 data>"
}

Fields

SERVER → CLIENT tts_done

All TTS audio has been sent. The session returns to LISTENING state — you can start the next turn.

JSON payload

{
  "type": "tts_done"
}

SERVER → CLIENT error

Sent when something goes wrong. The session may still be alive after a non-fatal error — check the message and decide whether to retry or reconnect.

JSON payload

{
  "type": "error",
  "message": "Failed to decode base64 audio",
  "chunk_index": 3
}

Fields

Characters

Pass character_id in start_session to select which character you are speaking with.

Audio format reference

Encoding audio in the browser

// Record with MediaRecorder, then send each chunk:
const reader = new FileReader();
reader.onload = () => {
  const base64 = reader.result.split(',')[1];
  ws.send(JSON.stringify({
    type: 'audio_chunk',
    chunk_index: index++,
    audio: base64
  }));
};
reader.readAsDataURL(audioBlob);

Decoding TTS audio in the browser

// Collect chunks, then play:
const chunks = [];
ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === 'tts_audio_chunk') {
    const binary = atob(msg.audio);
    const bytes  = Uint8Array.from(binary, c => c.charCodeAt(0));
    chunks.push(bytes.buffer);
  }
  if (msg.type === 'tts_done') {
    const blob = new Blob(chunks, { type: 'audio/mpeg' });
    const url  = URL.createObjectURL(blob);
    new Audio(url).play();
    chunks.length = 0;
  }
};