Build a voice agent

A voice agent session is one WebSocket that runs the whole conversational loop: it listens to the caller's audio, transcribes it, reasons about what to say (including calling tools you define), and speaks the reply back as audio. You configure the agent's behavior at the start of the session, then stream audio in and play audio out.

The protocol is JSON text frames in both directions. Audio travels inside the JSON as base64-encoded PCM16, so a single session carries user speech, agent speech, transcripts, and tool calls.

1. Your server calls POST /v2/voice-agent/token with your rtvk_ API key and receives a token, the WebSocket URL, and the effective session cap.
2. The client connects to wss://api.realtimevoicekit.com/v2/voice-agent/ws with the token as a query parameter.
3. The client sends session.update as the first frame: system prompt, greeting, tools, and input/output configuration.
4. The server answers with session.ready once the agent is live.
5. The client streams input.audio frames with base64 PCM16 microphone audio.
6. The server emits events: user transcripts, agent audio replies, and tool calls (which the client answers with tool.result).
7. The client sends session.end; the server finalizes, emits session.ended, and the socket closes.

Sessions are metered per second of connection time, so a 5 minute call costs about $0.45. Call GET /v2/models for current rates.

From your server, exchange your API key for a short-lived session token. Voice agent tokens are deliberately shorter-lived than realtime transcription tokens: redeem the token on the WebSocket within expires_in seconds (5 minutes by default).

curl -X POST https://api.realtimevoicekit.com/v2/voice-agent/token \
  -H "Authorization: Bearer rtvk_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "whisper-voicekit-agent", "max_session_duration_sec": 1800}'

{
  "token": "eyJhbGciOiJIUzI1NiIs...",
  "websocket_url": "wss://api.realtimevoicekit.com/v2/voice-agent/ws",
  "expires_at": "2026-07-22T12:05:00Z",
  "model": "whisper-voicekit-agent",
  "max_session_duration_sec": 1800
}

Request fields

Response fields

If your free API credit is used up and you have no active API subscription, this endpoint returns HTTP 402 with a message asking you to add billing.

Open the WebSocket with the token in the query string, then send session.update as the first frame. It defines everything about the agent: its instructions, greeting, tools, and audio formats.

import asyncio
import base64
import json
import websockets

async def run(session, session_update):
    url = session["websocket_url"] + "?token=" + session["token"]
    async with websockets.connect(url) as ws:
        await ws.send(json.dumps(session_update))  # always the first frame
        await asyncio.gather(stream_audio(ws), handle_events(ws))

{
  "type": "session.update",
  "session": {
    "system_prompt": "You are a friendly support agent for Acme Internet. Keep answers short and confirm the account before making changes.",
    "greeting": "Hi, thanks for calling Acme. How can I help today?",
    "input": { "format": { "encoding": "audio/pcm" } },
    "output": { "format": { "encoding": "audio/pcm" } },
    "tools": [
      {
        "type": "function",
        "name": "get_account_status",
        "description": "Look up the status of a customer account",
        "parameters": {
          "type": "object",
          "properties": { "account_id": { "type": "string" } },
          "required": ["account_id"]
        }
      }
    ]
  }
}

Wait for session.ready before streaming audio. If the token is invalid or expired the socket closes immediately with code 4401; an unknown model closes with 4400. See the session reference for every session.update field.

Send caller audio as input.audio frames: 16-bit PCM chunks, base64-encoded inside the JSON. Binary WebSocket frames are ignored on this endpoint; audio always travels inside JSON text frames.

{
  "type": "input.audio",
  "audio": "<base64 encoded PCM16 audio>"
}

function sendAudioChunk(ws, pcm16ArrayBuffer) {
  const bytes = new Uint8Array(pcm16ArrayBuffer);
  let binary = "";
  for (let i = 0; i < bytes.length; i++) {
    binary += String.fromCharCode(bytes[i]);
  }
  ws.send(JSON.stringify({ type: "input.audio", audio: btoa(binary) }));
}

Stream continuously; there is no push-to-talk framing. The agent detects when the caller starts and stops speaking and emits input.speech.started and input.speech.stopped events as it happens.

Everything the agent does arrives as JSON events on the same socket. The four you must handle to have a working conversation:

• transcript.user: the final transcript of what the caller just said (with transcript.user.delta partials while they speak).
• reply.audio: a chunk of the agent's spoken reply as base64 PCM16. Decode and play in arrival order.
• transcript.agent: the text of what the agent said, for logs and UI.
• tool.call: the agent wants one of your tools executed. Run it and send back tool.result with the same call_id.

async def handle_events(ws):
    async for raw in ws:
        event = json.loads(raw)
        etype = event.get("type")
        if etype == "session.ready":
            print("session live:", event["session_id"])
        elif etype == "transcript.user":
            print("caller:", event["text"])
        elif etype == "reply.audio":
            play_audio(base64.b64decode(event["data"]))
        elif etype == "transcript.agent":
            print("agent:", event["text"])
        elif etype == "tool.call":
            result = run_tool(event["name"], event["arguments"])
            await ws.send(json.dumps({
                "type": "tool.result",
                "call_id": event["call_id"],
                "result": json.dumps(result),
            }))
        elif etype == "session.ended":
            break

When a tool call arrives, the agent pauses that reply until your tool.result comes back, then continues speaking with the result woven in. Keep tool handlers fast; the caller is waiting on the line.

Send session.end when the conversation is over. The server finalizes the session, emits session.ended with duration accounting, and closes the socket.

client -> server   {"type": "session.end"}
server -> client   {"type": "session.ended", ...}
                   socket closes

{
  "type": "session.ended",
  "session_duration_seconds": 312.4,
  "audio_duration_seconds": 296.8,
  "timestamp": 1784721600.512
}

If your connection drops or you close the socket without sending session.end, the service finalizes the session for you. Sessions never outlive the WebSocket, they are not resumable after a disconnect, and billing stops when the connection ends.

• Each session is capped by max_session_duration_sec: 3300 seconds (55 minutes) by default, and requests are clamped to the service ceiling. The token response returns the effective cap.
• When the cap is reached the service finalizes the session and closes the socket; start a new session to continue.
• Billing is metered per second of connection time at $5.40 per session hour, from the moment the session is accepted until the socket closes.
• Redeem tokens promptly: the WebSocket must be opened before expires_at (60 to 600 seconds after minting).