Prompt caching (OpenAI)

OpenAI’s prompt caching is automatic, zero-configuration, and free — no cache_control blocks, no caches.create API, no breakpoints. Any prompt ≥ 1,024 tokens is automatically eligible for caching; subsequent requests within the cache window (typically 5-10 minutes, sometimes longer during off-peak hours) that share an identical prefix get a 50 % discount on the cached portion of input tokens. The mechanism is server-side KV-cache reuse on OpenAI’s inference fleet, exposed to you only as a usage.prompt_tokens_details.cached_tokens field in the response. There’s nothing in the SDK to “turn on.” Compared to Anthropic Claude (90 % discount with manual breakpoints) and Google Gemini (75 % discount with an explicit cache resource), OpenAI’s design trades savings for simplicity — you don’t think about caching, it just happens.

See also

• openai-models-and-capabilities — per-model availability
• openai-api-and-sdks — prompt_cache_key parameter for routing
• prompt-caching — sister doc; manual breakpoints + 90 % discount
• context-caching-gemini — sister doc; explicit resource + storage fee
• inference-optimization — KV-cache reuse at the serving-layer level

1. How it works

Per platform.openai.com/docs/guides/prompt-caching:

1. You send a request with a prompt ≥ 1,024 tokens
2. The serving infra hashes the prefix and stores the KV-cache state alongside the hash
3. On subsequent requests within the cache window with an identical prefix, the KV state is reloaded — the model doesn’t recompute the cached portion
4. You’re billed at 50 % of the standard input rate for the cached tokens
5. The cache window is 5-10 minutes typical, sometimes longer (up to ~1 hour during off-peak)

No opt-in required. Works on:

• Chat Completions API
• Responses API
• Assistants API (legacy)

2. Eligibility threshold

Threshold	Behavior
Prompt < 1,024 tokens	Not eligible — no cache
Prompt ≥ 1,024 tokens	Eligible; cached on first request
Cached portion granularity	128-token chunks: a cache hit reports tokens cached in multiples of 128

If your prompt is just under 1,024 tokens, the cache silently doesn’t kick in. Verify by checking usage.prompt_tokens_details.cached_tokens after a second request.

3. What’s cacheable

Per the docs:

• System / instructions messages — the most common cache target
• User messages — text content, including long prefixes shared across users
• Tool definitions (tools[]) — function schemas count toward cacheable prefix
• Image input — images can be cached (their tokenized representation)
• Audio input — yes, for audio-capable models

What’s NOT separately controllable:

• You can’t mark just one section “cache please”; the whole prefix is treated atomically
• The prefix is determined by the boundary where requests start to diverge

4. Cache key composition

The cache key is the full byte-exact prefix up to the point where requests diverge. Any byte difference in the prefix breaks the cache.

Order of evaluation:

1. model (different models = different caches)
2. tools[] (any change invalidates downstream)
3. instructions / system message
4. Prior conversation turns (oldest first)
5. Latest user message

A single different character anywhere in the prefix → no hit.

prompt_cache_key parameter

To route requests to specific cache shards for higher hit rates in multi-tenant scenarios:

client.responses.create(
    model="gpt-5",
    input="...",
    prompt_cache_key="user-12345",      # route to this user's cache shard
)

Without prompt_cache_key, OpenAI’s load balancer picks a shard. Multiple requests for the same user may land on different shards, missing the cache. Setting prompt_cache_key to a stable per-user / per-session value pins requests to one shard, raising hit rate at the cost of slightly worse load balancing.

When to set prompt_cache_key:

• High-traffic API where each user has a long system prompt + history
• Long-running agent sessions (set to session ID)
• Multi-tenant SaaS where you want per-tenant cache isolation

When not to:

• Low-traffic single-user dev work — the load balancer handles it
• Highly variable prefixes (no benefit from sharding)

5. Pricing

Per model, on the cached portion:

Model	Standard input	Cached input	Discount
GPT-5	~$2.50 / MTok (varies)	~$1.25 / MTok	50 %
GPT-4o	$2.50 / MTok	$1.25 / MTok	50 %
GPT-4o-mini	$0.15 / MTok	$0.075 / MTok	50 %
o-series	varies; cache discount applies	50 % off	50 %

Output tokens are unaffected by caching — generated tokens always bill at full output rate.

Worked example: GPT-4o (50k-token system prompt, 100 requests)

Without caching:

• 100 × 50k × $2.50 / MTok = $12.50 input

With caching (cache hits after first request):

• First request: 50k × $2.50 / MTok = $0.125 input (full price; writing to cache)
• Next 99 requests: 50k × $1.25 / MTok = $0.0625 each = $6.19 total
• Plus per-request user message + output (small, unaffected)
• Total saved: ~$6.18, roughly half

Compared to other vendors

Vendor	Discount on cache hit	Manual control	Storage fee
OpenAI	50 %	None (automatic)	None
Claude	90 %	Manual breakpoints	None
Gemini (explicit)	~75 %	Create cache resource	Yes (hourly)
Gemini (implicit)	(best effort)	None	None

OpenAI’s design optimizes for “fewer footguns, less savings.” For long-running agents with massive system prompts, Claude’s 90 % beats OpenAI’s 50 %. For most workloads, the automatic behavior is hard to beat in convenience.

6. Cache window / TTL

Per the docs: 5-10 minutes typical, up to 1 hour during off-peak hours. The window is server-side and not directly controllable.

Implications:

• Sustained high-traffic prompts stay cached as long as traffic continues
• A 30-minute gap in traffic likely evicts the cache
• TTL extends on cache hit — a continually-hit prefix can stay cached indefinitely
• Cold prefixes get evicted under capacity pressure (heavy load = shorter TTL)

For long-gap workloads (e.g. hourly cron jobs against the same agent), expect the cache to be cold at each invocation. Either accept the cost or use the cron job to “warm” the cache periodically (a no-op request with max_output_tokens=1 works).

7. Tracking cache performance

The usage field reports:

Chat Completions

resp.usage.prompt_tokens                          # 5170 (total input)
resp.usage.prompt_tokens_details.cached_tokens    # 5120 (cached portion)
resp.usage.completion_tokens                      # 200

Responses API

resp.usage.input_tokens                           # 5170
resp.usage.input_tokens_details.cached_tokens     # 5120
resp.usage.output_tokens                          # 200

Hit rate:

def cache_hit_rate(resp):
    if hasattr(resp.usage, "input_tokens_details"):
        cached = resp.usage.input_tokens_details.cached_tokens or 0
        total = resp.usage.input_tokens
    else:
        cached = resp.usage.prompt_tokens_details.cached_tokens or 0
        total = resp.usage.prompt_tokens
    return cached / total if total else 0

In well-tuned applications:

• Long-running agents: 70-95 % hit rate
• Multi-user with prompt_cache_key: 50-80 %
• One-shot prompts: typically 0 (no second request to hit cache)

8. Best practices

8.1 Put static content first

The cache key is the prefix. Structure prompts with static content at the start:

input = [
    {"role": "system", "content": large_static_system_prompt},   # cacheable
    {"role": "user", "content": user_query_changes_every_call},  # not cacheable
]

Wrong:

input = [
    {"role": "user", "content": "Query: ..."},                    # changes per call
    {"role": "system", "content": large_static_system_prompt},   # would-be cacheable but order kills cache
]

Conversation history grows at the end — that’s fine, the static prefix stays cached.

8.2 Hit the 1,024-token threshold

A 500-token prompt is never cached. If you have a short system prompt but anticipate many requests, pad it to ≥ 1,024 tokens with useful content (examples, edge-case handling, persona detail). The added tokens cost ~zero after the first cached request.

8.3 Use prompt_cache_key for tenant isolation

In a multi-user SaaS:

client.responses.create(
    model="gpt-5",
    input=[...],
    prompt_cache_key=f"tenant_{tenant_id}",
)

Same prompt_cache_key → same cache shard → maximizes hit rate within a tenant.

8.4 Keep tool definitions stable

tools array changes invalidate the cache. If you have 10 tools and add/remove one mid-session, all subsequent caches are cold. Either keep tools stable across a session, or accept periodic cache misses on tool churn.

8.5 Cache verification gate

When debugging, log hit rate:

hit_rate = cache_hit_rate(resp)
log.info(f"cache hit: {hit_rate:.1%}, cached_tokens={...}")

A consistent 0 % despite long stable prefixes means the prefix isn’t actually stable (byte-level diff) or you’re below the 1,024 threshold.

9. Pre-warming

You can warm a cache with a lightweight call:

client.responses.create(
    model="gpt-5",
    input=[
        {"role": "system", "content": large_system_prompt},
        {"role": "user", "content": "ok"},
    ],
    max_output_tokens=1,           # generates one token (~negligible cost)
)
# Cache is now warm; subsequent calls with the same system prompt hit

For real cron / scheduled workloads, run a no-op warm call ~1-2 minutes before the main batch.

10. Cache vs Batch API

Property	Cache	Batch
Discount	50 % on cached input	50 % on all input + output
Latency	Real-time (no impact)	Up to 24 hours
Setup	None (automatic)	None (just submit to Batch API)
Applies to	Repeated prefixes only	Independent requests

For massive offline workloads (1M+ requests), Batch + Cache stack: batch gets you 50 % off everything; cache (if prefixes repeat) gets you another 50 % off the input portion. They’re not redundant.

11. Cache vs prompt routing (load balancing)

Without prompt_cache_key, OpenAI routes requests using its load balancer. Hit rates can be variable. Setting prompt_cache_key pins requests to a cache shard — better cache, slightly worse load balancing. For most workloads the trade-off favors pinning.

If your traffic spikes hard, the load balancer may bypass the pinned shard for capacity reasons — the cache miss in that case isn’t a bug.

12. Cross-organization isolation

Cache is organization-scoped. Two different orgs with byte-identical prompts don’t share a cache. Two projects within the same org may share a cache (varies by configuration).

For SaaS that exposes the API to end users, this means caches naturally isolate by your tenant orgs.

13. Common pitfalls

1. Below 1,024 tokens — cache silently doesn’t apply. Check cached_tokens > 0.
2. Byte-level diffs in prefix — a single comma added in the middle of a system prompt invalidates. Pin your prompts.
3. Tool changes invalidate — adding / removing tools mid-session wipes cache. Plan tool stability.
4. Reasoning tokens not cached — o-series reasoning happens fresh each request. Caching saves on the input portion only.
5. prompt_cache_key collision — same key across very different prefixes still doesn’t share cache (byte-identity required) but does force same-shard routing, possibly hurting load balance. Use semantically-coherent keys.
6. Cache hit + streaming = correct usage in last chunk only — when streaming on Chat Completions, set stream_options.include_usage=True to get cache stats in the final chunk; Responses streaming emits a response.completed event with usage.
7. Migrating from Chat Completions to Responses changes the prefix — the message-shape differences mean caches don’t survive the migration. Re-warm.
8. Multi-region / Azure deployments — caches don’t share across Azure deployments or across OpenAI / Azure boundaries. Each deployment has its own cache.

14. Diagnostics

If you expect cache hits but cached_tokens is consistently 0:

1. ✅ Is the prompt ≥ 1,024 tokens? Check count_tokens or tiktoken
2. ✅ Is the prefix byte-identical across requests? Diff two consecutive requests
3. ✅ Same model + same org + within ~5-10 minute window?
4. ✅ Are tools / instructions stable?
5. ✅ Try setting prompt_cache_key="debug-{your-id}" — forces consistent shard
6. ✅ Try lightweight pre-warm call

If still no hit after all of the above — file a support ticket; some accounts have had caching unavailability in specific edge cases.

15. Further reading

• Prompt caching — canonical reference
• Pricing — cached input rates by model
• tiktoken — for pre-flight token counts
• Anthropic Claude caching — different design for comparison
• Gemini context caching — different design for comparison