Realtime API (OpenAI)

The Realtime API is OpenAI’s bidirectional audio + text streaming endpoint — the only path for sub-second voice conversations with GPT models. Architecturally it’s an event-driven WebSocket (or WebRTC) connection rather than a request-response REST call; clients send a stream of session.update, input_audio_buffer.append, conversation.item.create, response.create events and receive a stream of response.audio.delta, response.text.delta, response.function_call_arguments.delta, response.audio_transcript.delta, input_audio_buffer.speech_started, and ~40 other event types. The current models are gpt-realtime-2 (frontier, 2025-released) and gpt-4o-realtime-preview (older). Audio is PCM16 at 24 kHz, with G.711 (μ-law / a-law) supported for telephony interop. Function calling works mid-stream. Voice activity detection (VAD) is server-managed; interruption (“barge-in”) is first-class.

1. WebSocket vs WebRTC

Transport	Best for	Tradeoffs
WebSocket	Server-side bridges (your backend talking to OpenAI on behalf of a user)	Need to handle audio I/O yourself; simpler protocol; works in Node / Python
WebRTC	Browser / mobile clients connecting directly	Browser handles audio I/O natively; lower latency on client; SDP signaling overhead

WebSocket connection

URL: wss://api.openai.com/v1/realtime?model=gpt-realtime-2
Headers:
  Authorization: Bearer $OPENAI_API_KEY
  OpenAI-Beta: realtime=v1

WebRTC connection

Sign an ephemeral session token server-side first:

session = client.beta.realtime.sessions.create(
    model="gpt-realtime-2",
    modalities=["text", "audio"],
    voice="alloy",
    instructions="You are a friendly voice assistant.",
)
ephemeral_key = session.client_secret.value
# Send ephemeral_key to browser; browser uses it for WebRTC SDP offer

Browser then opens a peer connection with https://api.openai.com/v1/realtime using the ephemeral key.

2. Session configuration

# Server-side (Python SDK over WebSocket would use a lib like openai-realtime-api-beta)
session_config = {
    "model": "gpt-realtime-2",
    "modalities": ["text", "audio"],
    "instructions": "You are a friendly voice assistant.",
    "voice": "alloy",
    "input_audio_format": "pcm16",          # or "g711_ulaw", "g711_alaw"
    "output_audio_format": "pcm16",
    "input_audio_transcription": {"model": "whisper-1"},  # transcribe user input
    "turn_detection": {
        "type": "server_vad",                # or "none"
        "threshold": 0.5,
        "prefix_padding_ms": 300,
        "silence_duration_ms": 500,
    },
    "tools": [...],                          # function declarations
    "tool_choice": "auto",
    "temperature": 0.8,
    "max_response_output_tokens": 4096,
}

Send this as a session.update event right after connection.

3. Audio formats

Format	Sample rate	Use case
`pcm16`	24 kHz	Default; raw little-endian 16-bit PCM
`g711_ulaw`	8 kHz	Telephony (USA / Japan)
`g711_alaw`	8 kHz	Telephony (rest of world)

PCM16 is the highest quality. G.711 formats are for SIP / phone-system integration where 8 kHz is required.

Audio is sent as base64-encoded chunks in input_audio_buffer.append events:

import base64
 
# Pseudocode for chunking + sending
chunk = audio_bytes[i:i+8192]
event = {
    "type": "input_audio_buffer.append",
    "audio": base64.b64encode(chunk).decode(),
}
ws.send(json.dumps(event))

Commit a buffer to finalize input:

ws.send(json.dumps({"type": "input_audio_buffer.commit"}))

Or rely on server-side VAD (see below) to auto-commit on detected silence.

4. Voice activity detection (VAD)

Mode	Behavior
`server_vad` (default)	Server detects start + end of speech automatically
`semantic_vad`	Server uses content cues for turn detection
`none`	Client manually commits buffers

"turn_detection": {
  "type": "server_vad",
  "threshold": 0.5,             // 0-1, sensitivity
  "prefix_padding_ms": 300,     // ms of audio before speech start to include
  "silence_duration_ms": 500    // ms of silence to declare turn end
}

When VAD detects the user finished speaking, the server emits input_audio_buffer.speech_stopped and automatically generates a response — no need for the client to call response.create.

For push-to-talk UX, use type: "none" and manually commit buffers + create responses.

5. Interruption (barge-in)

User speaks while the model is generating audio:

Server detects new speech (input_audio_buffer.speech_started)
Server automatically truncates the current response (sends response.audio.done with truncated: true)
Client should stop playing the in-flight audio immediately
Server generates a new response based on the interruption

Client must handle this:

on("input_audio_buffer.speech_started", () => {
  audioElement.pause();
  audioElement.currentTime = 0;
  audioBuffer.clear();
});

The model’s awareness of what it actually said vs what it intended to say (which may differ if interrupted mid-sentence) requires you to send a conversation.item.truncate event with the actually-played duration:

{
  "type": "conversation.item.truncate",
  "item_id": "msg_abc123",
  "content_index": 0,
  "audio_end_ms": 2400
}

This tells the server “the user only heard 2.4 seconds of that response” so the next turn’s context is accurate.

6. Function calling mid-stream

session_config = {
    ...,
    "tools": [
        {
            "type": "function",
            "name": "get_weather",
            "description": "Get current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    ],
    "tool_choice": "auto",
}

When the model calls a function:

Server emits response.function_call_arguments.delta (incrementally)
Then response.function_call_arguments.done with full args
Client executes function locally
Client sends back:

{
  "type": "conversation.item.create",
  "item": {
    "type": "function_call_output",
    "call_id": "call_abc",
    "output": "{\"temp\": 22, \"condition\": \"sunny\"}"
  }
}
{"type": "response.create"}    // trigger model to continue

Then response.audio.delta events arrive with the model’s spoken answer.

7. Key events — outbound (client to server)

Event type	Purpose
`session.update`	Update session config (instructions, voice, tools, VAD)
`input_audio_buffer.append`	Send audio chunk
`input_audio_buffer.commit`	Finalize current audio buffer
`input_audio_buffer.clear`	Discard pending audio
`conversation.item.create`	Inject text message / function output
`conversation.item.truncate`	Mark how much of an audio response was played
`conversation.item.delete`	Remove an item from conversation
`response.create`	Generate a response
`response.cancel`	Cancel in-progress response

8. Key events — inbound (server to client)

Event type	Purpose
`error`	Error in session
`session.created` / `session.updated`	Confirms session state
`conversation.created`	New conversation started
`conversation.item.created`	New item (message / function_call / function_call_output)
`conversation.item.input_audio_transcription.completed`	User speech transcribed (if Whisper enabled)
`input_audio_buffer.speech_started`	VAD detected speech onset
`input_audio_buffer.speech_stopped`	VAD detected speech end
`input_audio_buffer.committed`	Server committed input buffer
`response.created` / `response.done`	Response lifecycle
`response.output_item.added` / `.done`	New output item in response
`response.content_part.added` / `.done`	New content part
`response.text.delta` / `.done`	Text generation increments
`response.audio.delta` / `.done`	Audio output increments (base64 PCM16 chunks)
`response.audio_transcript.delta` / `.done`	Transcript of model’s audio response
`response.function_call_arguments.delta` / `.done`	Function call arg streaming
`rate_limits.updated`	Rate-limit state changes

40+ events total. The official openai-realtime-api-beta library wraps these into higher-level callbacks.

9. Reference client libraries

Official (Node)

npm install openai/openai-realtime-api-beta

import { RealtimeClient } from "@openai/realtime-api-beta";
 
const client = new RealtimeClient({ apiKey: process.env.OPENAI_API_KEY });
await client.connect();
client.updateSession({
  instructions: "You are a friendly voice assistant.",
  voice: "alloy",
  turn_detection: { type: "server_vad" },
});
 
client.on("conversation.updated", ({ item, delta }) => {
  if (delta?.audio) {
    playAudio(delta.audio);   // your audio output
  }
});
 
// Send audio
client.appendInputAudio(audioChunk);

Agents SDK (Python)

The openai-agents-python package wraps Realtime in a RealtimeAgent class — gives you handoffs, tools, sessions across voice conversations. See agents-sdk-openai.

Community libraries

Many open-source Realtime wrappers exist (pipecat, livekit-agents, etc.) that handle audio I/O, VAD bridging, and platform integration.

10. Pricing

Per platform.openai.com/docs/pricing:

Model	Text input	Text output	Audio input	Audio output
`gpt-realtime-2`	(varies — premium)	(varies)	premium per-MTok	premium per-MTok
`gpt-4o-realtime-preview`	$5 / MTok	$20 / MTok	$100 / MTok	$200 / MTok
`gpt-4o-mini-realtime-preview`	cheaper	cheaper	cheaper	cheaper

Audio tokens are roughly 20× more expensive than text tokens. A 1-minute voice conversation (input + output) typically costs ~$0.05 - $0.20 depending on model.

Cached input also applies on Realtime — long system prompts benefit from automatic caching same as the REST APIs.

11. Browser security pattern

Don’t expose your OPENAI_API_KEY to browsers. Use ephemeral session tokens:

# Server endpoint that mints tokens
@app.post("/realtime-session")
def create_session():
    session = client.beta.realtime.sessions.create(
        model="gpt-realtime-2",
        modalities=["text", "audio"],
        voice="alloy",
        instructions="...",
    )
    return {"client_secret": session.client_secret.value}

// Browser
const { client_secret } = await fetch("/realtime-session", { method: "POST" }).then(r => r.json());
const pc = new RTCPeerConnection();
// ... use client_secret in WebRTC offer SDP ...

The ephemeral token is scoped to that session and expires quickly. Safer than long-lived API keys in browser-resident code.

12. Common pitfalls

Forgetting OpenAI-Beta: realtime=v1 header — WebSocket connection fails or returns no model events. Required for WebSocket transport.
Audio sample rate mismatch — sending 16 kHz when format is pcm16 (24 kHz) produces high-pitched output. Resample client-side.
Not handling interruption — old audio keeps playing while user is talking → confusing UX. Always handle speech_started event.
Forgetting conversation.item.truncate after interruption — model’s context drifts (thinks it said the whole response). Always truncate.
server_vad too sensitive — cuts off mid-sentence on a pause. Increase silence_duration_ms to 700-1000 for natural pacing.
server_vad too insensitive — long pauses leave user hanging. Decrease threshold.
WebSocket reconnection — Realtime sessions don’t survive disconnections. On reconnect, recreate session from scratch. Persist conversation state client-side or via the new conversation API.
Browser audio APIs — AudioContext and MediaStreamTrack have quirks; use a well-tested library for production browser apps.
Function calls don’t auto-resume responses — after submitting function_call_output, you must send response.create to continue.
Cost surprise — audio output at $200/MTok can hit $1+ per minute on gpt-4o-realtime-preview. Budget carefully; monitor; use gpt-4o-mini-realtime for cost-sensitive workloads.

13. Use cases

Voice agents — customer support, scheduling, transactional voice UX
Real-time interpreters / translators — listen-translate-speak in near-real-time
Language tutoring — pronunciation feedback, conversational practice
Phone IVR — replace touch-tone with natural conversation (via G.711 + Twilio bridge)
Accessibility — voice interfaces for users who can’t type quickly

14. Comparison to other vendors

Aspect	OpenAI Realtime	Gemini Live API	Anthropic
Status	GA (`gpt-realtime-2`)	GA (Gemini Live)	No direct equivalent
Transport	WebSocket / WebRTC	WebSocket	n/a
Audio format	PCM16 24kHz, G.711	PCM16 16kHz, MP3	n/a
Latency	sub-second TTFT	sub-second	n/a
Function calling mid-stream	Yes	Yes	n/a
Voice options	6-10 voices, modulatable	Several voices	n/a
Cost (per minute)	$0.05-$0.20	similar range	n/a

15. Further reading

Realtime API guide — canonical reference
openai-realtime-api-beta — Node reference client
WebRTC quickstart — browser path
Agents SDK Realtime — RealtimeAgent class
Events reference — full inbound/outbound event list

Compendium

Explorer

Realtime API (OpenAI)

Realtime API (OpenAI)

See also

1. WebSocket vs WebRTC

WebSocket connection

WebRTC connection

2. Session configuration

3. Audio formats

4. Voice activity detection (VAD)

5. Interruption (barge-in)

6. Function calling mid-stream

7. Key events — outbound (client to server)

8. Key events — inbound (server to client)

9. Reference client libraries

Official (Node)

Agents SDK (Python)

Community libraries

10. Pricing

11. Browser security pattern

12. Common pitfalls

13. Use cases

14. Comparison to other vendors

15. Further reading

Graph View

Table of Contents