Voice agent session reference

Voice agent sessions speak JSON text frames in both directions; binary WebSocket frames are ignored. Every frame has a type field, and audio travels base64-encoded inside the JSON.

1. The client connects and sends session.update as the first frame.
2. The server emits session.ready with the session_id.
3. The client streams input.audio frames continuously.
4. While the caller speaks, the server emits input.speech.started, transcript.user.delta partials, then input.speech.stopped and a final transcript.user.
5. The server replies: reply.started, a stream of reply.audio chunks, transcript.agent, and reply.done.
6. When tools are configured the server may emit tool.call mid-reply; the client answers with tool.result and the reply continues.
7. The client sends session.end; the server emits session.ended and closes the connection.

session.update

Defines the agent. Send it as the first frame after connecting; the server confirms mid-session updates with session.updated.

{
  "type": "session.update",
  "session": {
    "system_prompt": "You are a friendly support agent for Acme Internet. Keep answers short and confirm the account before making changes.",
    "greeting": "Hi, thanks for calling Acme. How can I help today?",
    "input": { "format": { "encoding": "audio/pcm" } },
    "output": { "format": { "encoding": "audio/pcm" } },
    "tools": [
      {
        "type": "function",
        "name": "get_account_status",
        "description": "Look up the status of a customer account",
        "parameters": {
          "type": "object",
          "properties": { "account_id": { "type": "string" } },
          "required": ["account_id"]
        }
      }
    ]
  }
}

input.audio

{
  "type": "input.audio",
  "audio": "<base64 encoded PCM16 audio>"
}

audio is a base64-encoded chunk of 16-bit PCM. Send frames continuously after session.ready; the server handles speech detection and turn taking.

tool.result

{
  "type": "tool.result",
  "call_id": "call_5d1a2b",
  "result": "{\"status\": \"active\", \"plan\": \"fiber_500\"}"
}

reply.create

{
  "type": "reply.create",
  "instructions": "Let the caller know the account check is still running."
}

Triggers a spoken reply without waiting for caller speech, guided by instructions. Useful for filling silences, for example while a slow tool runs.

session.end

{ "type": "session.end" }

Finalizes the session immediately. The server responds with session.ended and closes the socket. No fields.

session.ready

{
  "type": "session.ready",
  "session_id": "sess_9f2c81"
}

Sent once the agent has accepted your configuration. Do not send input.audio before it arrives.

transcript.user.delta and transcript.user

{
  "type": "transcript.user.delta",
  "text": "is my internet plan"
}

{
  "type": "transcript.user",
  "text": "What is my internet plan called?",
  "item_id": "item_2b8c4d"
}

Deltas stream while the caller talks; the final transcript.user carries the full turn and an item_id you can use to correlate with the agent's reply.

reply.audio

{
  "type": "reply.audio",
  "data": "<base64 encoded PCM16 audio>"
}

data is a base64-encoded chunk of 16-bit PCM agent speech in your configured output format. Decode and play chunks in arrival order.

transcript.agent

{
  "type": "transcript.agent",
  "text": "You are on the Fiber 500 plan.",
  "reply_id": "reply_7e3f90",
  "item_id": "item_2b8c4d",
  "interrupted": false
}

The text of the agent's reply. interrupted is true when the caller talked over the agent and the reply was cut short.

reply.done

{ "type": "reply.done", "status": "interrupted" }

Marks the end of a reply. status is present when the reply did not complete normally, for example interrupted when the caller barged in; stop playback of any buffered reply.audio when that happens.

tool.call

{
  "type": "tool.call",
  "call_id": "call_5d1a2b",
  "name": "get_account_status",
  "arguments": { "account_id": "acct_1042" }
}

session.ended

{
  "type": "session.ended",
  "session_duration_seconds": 312.4,
  "audio_duration_seconds": 296.8,
  "timestamp": 1784721600.512
}

The final event of every session, with wall-clock and audio duration accounting. The socket closes right after it.

In all three cases billing stops when the connection ends. Prefer an explicit session.end so you receive session.ended and its duration accounting.

Failures during a session arrive as session.error events. When a session cannot start at all (for example the service is temporarily unavailable), you receive a session.error with only a message and the socket closes.

{
  "type": "session.error",
  "code": "invalid_format",
  "message": "Invalid message format"
}