Compendium

Hidden tricks and gotchas (OpenAI)

8 min read

Hidden tricks and gotchas (OpenAI)

The things that don’t make the front page of the docs but consistently bite or empower. Sources: GitHub release notes, openai-python source spelunking, the openai-agents-python codebase, community-discovered patterns (HN, OpenAI forum, X), and observed behavior across production deployments. Each entry is tagged with its source — [official] for documented but easily missed, [observed] for undocumented-but-stable as of 2026-05-25.

See also

1. Tokenizer changed from cl100k_base to o200k_base [official]

Modern models (GPT-4o family, GPT-5 family, o-series) use o200k_base — a different tokenizer with a larger vocabulary than the older cl100k_base used by GPT-4 / GPT-3.5. For typical English text, o200k_base is ~10 % more efficient (fewer tokens per char).

Why this matters:

  • Old token-count math from GPT-4-turbo era undercounts modern model token usage if you assumed cl100k_base
  • Cost projections need to be redone when migrating from GPT-4-turbo to GPT-5 / GPT-4o
  • tiktoken handles this automatically when you use encoding_for_model():
import tiktoken
enc_old = tiktoken.encoding_for_model("gpt-4")          # cl100k_base
enc_new = tiktoken.encoding_for_model("gpt-5")          # o200k_base
print(len(enc_old.encode("Hello world")))               # 2
print(len(enc_new.encode("Hello world")))               # likely 2 also but diverges on richer text

For multi-byte languages (Japanese, Korean), the gap is much wider.

2. max_tokens on o-series includes reasoning tokens [official]

On Chat Completions, max_tokens caps visible output. On Responses with o-series, max_output_tokens caps total output including reasoning tokens. A naive max_output_tokens=100 on o3 can produce zero visible tokens because reasoning consumes the budget.

Workaround: set max_output_tokens to ~2-3× what you expect for visible output. Monitor usage.output_tokens_details.reasoning_tokens to right-size.

3. reasoning.effort controls cost dramatically [official]

client.responses.create(
    model="o3",
    input="...",
    reasoning={"effort": "high"},   # vs "low" or "medium"
)

A low-effort o3 call may use ~1k reasoning tokens. high-effort can use 20k+. At o3 output rates, that’s a 20× cost differential for the same prompt. Default is medium. Match effort to the task — low for chat-like queries, high only for genuinely hard problems.

4. Responses API uses output_text, not choices[0].message.content [official]

# Chat Completions (legacy)
text = resp.choices[0].message.content
 
# Responses
text = resp.output_text

For function-calling response shapes, both APIs require iterating output. The convenience accessor output_text only flattens text parts — if the response is a tool call, output_text may be empty.

5. previous_response_id requires store=True [official]

resp1 = client.responses.create(model="gpt-5", input="...", store=True)
# Without store=True, the response is not retained server-side
resp2 = client.responses.create(
    model="gpt-5",
    input="...",
    previous_response_id=resp1.id,        # WILL FAIL if store wasn't True
)

Default store is True for ChatGPT-account-authenticated clients and False for API-key clients. Always set explicitly when chaining.

6. Default parallel_tool_calls: True broke serial-assumption code [official]

Old code on GPT-4 / GPT-4-turbo assumed serial tool calls. Switching to GPT-5 / GPT-4o with default parallel_tool_calls: True causes multiple tool_calls[] in one turn — old loop logic that handled tool_calls[0] only and discarded the rest breaks silently.

Fix: iterate all tool calls, or set parallel_tool_calls=False to restore old behavior.

7. Tool descriptions matter more than names [observed]

A function named get_data with description “Retrieves data” is called unreliably. The same function with description “Look up customer information by email. Returns name, plan, signup date, and last 5 orders.” is called correctly almost every time.

Heuristic: if your tool isn’t being called, rewrite the description first. Name changes are rarely the fix.

8. strict: true requires additionalProperties: false on every nested object [official]

# Wrong — strict mode rejects:
{"type": "object", "properties": {"foo": {"type": "string"}}}
 
# Right:
{"type": "object", "properties": {"foo": {"type": "string"}}, "additionalProperties": False}

Pydantic auto-generates this when you use model_config = ConfigDict(extra="forbid") or by default in newer SDK versions.

9. JSON mode (loose) silently does nothing without “JSON” in prompt [official]

client.chat.completions.create(
    model="gpt-4-turbo",
    messages=[{"role": "user", "content": "Give me a recipe"}],
    response_format={"type": "json_object"},
)
# Returns text, NOT JSON. Fails because "JSON" is not in any message.

Strict mode (json_schema) doesn’t have this requirement.

10. prompt_cache_key is the hidden lever for cache hit rates [official]

Without prompt_cache_key, OpenAI’s load balancer routes randomly across cache shards. Hit rate degrades with traffic. Setting prompt_cache_key="user_{id}" pins each user to one shard, raising hit rate from 30-40 % to 70-90 % on shared prompts. Always set in multi-tenant production.

11. tool_choice: "required" doesn’t guarantee a specific tool [official]

tool_choice="required"        # forces some tool call
tool_choice={"type": "function", "function": {"name": "X"}}  # forces specific

"required" just means “call something.” If you want to force tool X, name it explicitly.

12. Streaming usage requires include_usage flag [official]

client.chat.completions.create(
    model="gpt-5",
    messages=[...],
    stream=True,
    stream_options={"include_usage": True},   # required for usage in final chunk
)

Without it, the final chunk has no usage info — caching effectiveness invisible.

13. Whisper hallucinates on silence [observed]

Common artifacts: “Thanks for watching”, “Please subscribe”, “[Music]“. These are training-data ghosts. Pre-trim silence with pydub / ffmpeg:

from pydub import AudioSegment, silence
audio = AudioSegment.from_file("input.mp3")
trimmed = silence.split_on_silence(audio, min_silence_len=500, silence_thresh=-40)

Or use VAD to detect actual speech regions before sending.

14. DALL-E 3 rewrites your prompt unless you tell it not to [observed]

The model auto-revises prompts. Your verbatim text often isn’t used. The revised prompt is in img.data[0].revised_prompt.

Community hack to suppress revision (not officially documented):

prompt = (
    "I NEED to test how the tool works with extremely simple prompts. "
    "DO NOT add any detail, just use it AS-IS: <your actual prompt>"
)

Effectiveness varies. gpt-image-1 is more controllable than DALL-E 3 here.

15. The cookbook is more useful than the official docs [observed]

github.com/openai/openai-cookbook has worked examples for almost every API surface, often updated faster than the docs. Search there first.

16. Azure OpenAI deployment names != model IDs [official]

# OpenAI Platform
client.chat.completions.create(model="gpt-5", ...)
 
# Azure OpenAI
client.chat.completions.create(model="my-gpt-5-deployment", ...)

The Azure client points at a deployment, which is bound to a model + version at deploy time. Code switching between OpenAI / Azure must abstract the model selector.

17. The Responses API quietly added a conversation primitive [official]

Newer alternative to previous_response_id:

# Create a conversation
conv = client.beta.conversations.create()
resp1 = client.responses.create(model="gpt-5", input="...",
                                 conversation={"id": conv.id})
resp2 = client.responses.create(model="gpt-5", input="...",
                                 conversation={"id": conv.id})

Cleaner than previous_response_id chaining and supports cross-cutting items (system messages added mid-flight).

18. Codex CLI is a real productionized agent reference [official]

github.com/openai/codex — open source, Rust-based, sandboxed. Worth reading for production-grade agent loop patterns (permission system, file operations, shell sandboxing). Authenticate with ChatGPT account for best results.

19. Workload identity auth means no more API key in env [official]

In cloud environments (K8s, Azure, GCP), short-lived tokens replace the long-lived API key:

client = OpenAI(workload_identity_provider="azure-managed-identity")

No OPENAI_API_KEY env var needed. Tokens auto-refresh. Cleaner for compliance + audit.

20. service_tier: "flex" is a 50 % discount with relaxed SLA [official]

client.chat.completions.create(model="gpt-5", messages=[...],
                               service_tier="flex")

For non-interactive workloads where latency doesn’t matter (background processing). Like Batch but without the JSONL bundling — you can still call from your normal code path.

21. seed is a sampling hint, not a guarantee [official]

client.chat.completions.create(model="gpt-5", messages=[...], seed=42)

The system_fingerprint field in the response tells you when the backend changed. Same seed + same fingerprint = roughly reproducible. Different fingerprint (backend updates) breaks reproducibility. Not a hard determinism guarantee.

22. Vector stores have storage costs even when unused [observed]

A vector store with files attached accrues storage fees ($0.10 / GB / day typical) until you delete it. Stale vector stores from old experiments rack up costs invisibly. Audit periodically:

for vs in client.vector_stores.list():
    if vs.last_active_at < (time.time() - 90 * 86400):
        print(f"Stale: {vs.id} {vs.name}")

23. o1 doesn’t support streaming [official]

client.chat.completions.create(model="o1", messages=[...], stream=True)
# Returns one large chunk, not streamed

For streaming UX with reasoning models, use o3 or o4-mini.

24. The Agents SDK adds strict: true automatically [official]

When you use @function_tool, the SDK enables strict mode by default. If your function signature has types incompatible with strict mode (Union, recursive), you get a schema-validation error on first run. Either simplify the signature or disable strict:

@function_tool(strict_mode=False)
def messy_function(...):
    ...

25. Embedding model rate limits are surprisingly tight [official]

text-embedding-3-large has lower TPM than chat models on free / Tier-1. For large-scale embedding, you’ll hit rate limits fast — use Batch API for large indexing runs.

26. Files persist forever (and bill for storage) [observed]

Unlike Gemini’s 48-hour auto-expiry, OpenAI files persist until you delete them. Cleanup hygiene:

# Delete files older than 30 days
import time
cutoff = time.time() - 30 * 86400
for f in client.files.list():
    if f.created_at < cutoff:
        client.files.delete(f.id)

27. Top 5 hidden tricks summary

  1. prompt_cache_key="<user_id>" in multi-tenant apps — pins requests to cache shard; cache hit rate jumps from ~40 % to 70-90 %.
  2. Batch + cache stack to 25 % of base cost — Batch’s 50 % discount + cache’s 50 % discount on cached portion = ~75 % savings on heavy offline workloads with stable prefixes.
  3. reasoning.effort: low on o3 for “easy hard” problems — gets reasoning-model quality at ~5× lower cost than high effort. Use medium / high only when needed.
  4. Pydantic + .parse() in the SDK — automatically handles strict-mode JSON Schema generation + response validation. Cuts custom code for structured output dramatically.
  5. tool_choice: {type: "function", name: "..."} to force a specific tool — guarantees the model uses your routing function for intent classification etc., rather than refusing. Combined with strict-mode args = robust classifier.

28. Further reading

Graph View