Batch API and fine-tuning (OpenAI)

OpenAI’s Batch API offers a 50 % discount on input + output for non-interactive workloads with a 24-hour SLA — submit a JSONL file of requests via the Files API, get a JSONL file of responses back, no rate-limit pressure on your sync endpoints. It supports /v1/chat/completions, /v1/responses, /v1/embeddings, and a few other endpoints; same model surface, just delayed and cheaper. The fine-tuning path supports supervised fine-tuning (SFT) on gpt-4o-mini-2024-07-18, gpt-3.5-turbo, and a growing list of newer models, direct preference optimization (DPO) on preference pairs, and reinforcement fine-tuning (RFT) in preview — the latter being OpenAI’s newer technique for o-series reasoning models where you supply a grader function and the model is trained to maximize the grade.

See also

• openai-models-and-capabilities — which models support fine-tuning
• openai-api-and-sdks — client.batches.* and client.fine_tuning.* surfaces
• prompt-caching-openai — pairs with Batch for compounding discounts
• batch-api-and-files-api — sister doc for Claude
• gemini-api-and-sdks — Gemini batch section

1. Batch API basics

Per platform.openai.com/docs/guides/batch:

• 50 % discount on input + output tokens
• 24-hour completion target (most batches finish in 1-4 hours)
• Higher rate limits than sync endpoints (per-batch limits, not RPM)
• Supports: /v1/chat/completions, /v1/responses, /v1/embeddings, /v1/moderations
• Same models as sync (no special “batch model” — same gpt-5, gpt-4o, etc.)
• Per-request body identical to sync endpoint shape

2. Workflow

1. Prepare JSONL input file (one request per line)
2. Upload via Files API (purpose="batch")
3. Submit batch job referencing the file
4. Poll status (or use webhooks)
5. Download output JSONL when complete
6. Parse + use

3. JSONL input format

One request per line. Each line:

{
  "custom_id": "req-12345",
  "method": "POST",
  "url": "/v1/chat/completions",
  "body": {
    "model": "gpt-4o-mini",
    "messages": [
      {"role": "system", "content": "Translate to French."},
      {"role": "user", "content": "Hello world"}
    ],
    "max_tokens": 100
  }
}

custom_id is your tracking ID — any string, must be unique within the batch. The output JSONL preserves custom_id so you can correlate requests + responses.

body is the standard endpoint payload (whatever you’d send to a sync chat.completions.create).

File limits

• Max 50,000 requests per batch
• Max 200 MB per JSONL file
• For larger workloads, split into multiple batches

4. Full Python workflow

import json
from openai import OpenAI
 
client = OpenAI()
 
# 1. Build JSONL
requests = []
for i, prompt in enumerate(my_prompts):
    requests.append({
        "custom_id": f"req-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4o-mini",
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 500,
        },
    })
 
with open("batch_input.jsonl", "w") as f:
    for r in requests:
        f.write(json.dumps(r) + "\n")
 
# 2. Upload
batch_file = client.files.create(
    file=open("batch_input.jsonl", "rb"),
    purpose="batch",
)
 
# 3. Submit
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"description": "Translation batch"},
)
print(f"Batch ID: {batch.id}, status: {batch.status}")
 
# 4. Poll
import time
while True:
    batch = client.batches.retrieve(batch.id)
    print(f"Status: {batch.status}, "
          f"completed: {batch.request_counts.completed}/{batch.request_counts.total}")
    if batch.status in {"completed", "failed", "expired", "cancelled"}:
        break
    time.sleep(60)
 
# 5. Download output
if batch.status == "completed":
    output_content = client.files.content(batch.output_file_id).read()
    for line in output_content.decode().splitlines():
        result = json.loads(line)
        custom_id = result["custom_id"]
        if result.get("response"):
            answer = result["response"]["body"]["choices"][0]["message"]["content"]
            print(f"{custom_id}: {answer}")
        else:
            print(f"{custom_id}: error - {result.get('error')}")
 
# Optional: download errors file
if batch.error_file_id:
    errors = client.files.content(batch.error_file_id).read()
    print(errors.decode())

5. Batch states

State	Meaning
validating	File being checked
failed	Validation failed (malformed JSONL, schema errors)
in_progress	Running
finalizing	Wrapping up
completed	Done — output_file_id available
expired	Hit 24-hour deadline; partial results in output
cancelling	Cancellation requested
cancelled	Cancelled by user

client.batches.cancel(batch.id)        # request cancellation
client.batches.list(limit=20)          # list recent batches

6. Webhook notifications

Instead of polling, configure webhooks at the org level. OpenAI POSTs to your endpoint on batch state changes:

{
  "id": "evt_abc",
  "type": "batch.completed",
  "data": {"id": "batch_xyz", "status": "completed"}
}

Verify webhook signatures using SDK helpers:

from openai.webhooks import construct_event
 
event = construct_event(payload, signature, webhook_secret)

7. Batch + cache stacking

Prompt caching applies to batch requests. If many of your batch entries share a system prompt:

• Cache discount: 50 % on cached input tokens
• Batch discount: 50 % on input + output

These stack — a cached input token in a batch is $0.25 of base rate (25 % of original), since cache makes it 50 % and batch makes that 50 %.

For massive offline workloads with stable prefixes, this can drive effective cost to a quarter of standard pricing.

8. Batch on embeddings + Responses

# Embeddings batch
batch = client.batches.create(
    input_file_id=...,
    endpoint="/v1/embeddings",
    completion_window="24h",
)
 
# Responses batch
batch = client.batches.create(
    input_file_id=...,
    endpoint="/v1/responses",
    completion_window="24h",
)

Same shape, different per-line url field in the JSONL:

{"custom_id": "emb-1", "method": "POST", "url": "/v1/embeddings",
 "body": {"model": "text-embedding-3-large", "input": "..."}}

9. Use cases for Batch

• Bulk transformation: translate / summarize / classify 100k documents
• Embedding indexing: generate embeddings for a 1M-record corpus
• Synthetic data: generate training data via model self-play
• Periodic batch evaluations: run eval suites overnight
• Background enrichment: enhance records with LLM-derived fields
• Cost-sensitive workloads where 24h latency is acceptable

NOT for:

• User-facing real-time requests
• Interactive workflows
• Anything needing sub-minute response

10. Fine-tuning — basics

Per platform.openai.com/docs/guides/fine-tuning:

Method	What it does	When to use
SFT (supervised fine-tuning)	Trains on (input, output) pairs	Adapt style, format, domain knowledge
DPO (direct preference optimization)	Trains on (input, chosen, rejected)	Improve helpfulness, safety, preference alignment
RFT (reinforcement fine-tuning)	Trains o-series with a grader function	Optimize for measurable outcomes on complex tasks

11. Supervised fine-tuning (SFT)

Training data format

JSONL, one example per line:

{
  "messages": [
    {"role": "system", "content": "You are a helpful customer support agent for Acme Corp."},
    {"role": "user", "content": "How do I reset my password?"},
    {"role": "assistant", "content": "To reset your password, go to acme.com/reset..."}
  ]
}

Minimum: 10 examples to start (recommend 50-100 for meaningful results).

Submitting

# Upload training data
train_file = client.files.create(file=open("train.jsonl", "rb"), purpose="fine-tune")
val_file = client.files.create(file=open("val.jsonl", "rb"), purpose="fine-tune")
 
# Create job
job = client.fine_tuning.jobs.create(
    training_file=train_file.id,
    validation_file=val_file.id,
    model="gpt-4o-mini-2024-07-18",
    hyperparameters={
        "n_epochs": 3,
        "learning_rate_multiplier": 1.0,
        "batch_size": 4,
    },
    suffix="acme-support",          # appears in fine-tuned model ID
)
print(f"Job: {job.id}, status: {job.status}")
 
# Monitor
while True:
    job = client.fine_tuning.jobs.retrieve(job.id)
    print(f"Status: {job.status}")
    if job.status in {"succeeded", "failed", "cancelled"}:
        break
    time.sleep(60)
 
if job.status == "succeeded":
    print(f"Fine-tuned model: {job.fine_tuned_model}")
    # Use it:
    resp = client.chat.completions.create(
        model=job.fine_tuned_model,
        messages=[{"role": "user", "content": "..."}],
    )

Checkpoints

for cp in client.fine_tuning.jobs.checkpoints.list(job.id):
    print(cp.fine_tuned_model_checkpoint, cp.metrics)

Each epoch produces a checkpoint; pick the best by validation metrics, not necessarily the final one.

Models that support SFT

Model	Status
gpt-4o-mini-2024-07-18	✅
gpt-4o-2024-08-06	✅
gpt-3.5-turbo-0125 and others	✅
gpt-5-mini	Rolling availability
o3-mini	Preview support
GPT-5 frontier	Not yet (as of 2026-05)
o1 / o1-pro	Not supported

Pricing

Operation	Cost
Training (gpt-4o-mini)	~$3 / 1M training tokens
Training (gpt-3.5-turbo)	~$8 / 1M training tokens
Inference (fine-tuned gpt-4o-mini)	premium over base — ~3-5× depending on model
Hosting / storage	included while model is in use

For a 1M-token training run on gpt-4o-mini: ~$3 for training; per-call inference at fine-tuned rates.

12. Direct preference optimization (DPO)

Training data shape:

{
  "input": {
    "messages": [{"role": "user", "content": "..."}]
  },
  "preferred_output": [{"role": "assistant", "content": "..."}],
  "non_preferred_output": [{"role": "assistant", "content": "..."}]
}

job = client.fine_tuning.jobs.create(
    training_file=...,
    model="gpt-4o-mini-2024-07-18",
    method={
        "type": "dpo",
        "dpo": {"hyperparameters": {"beta": 0.1, "n_epochs": 3}},
    },
)

DPO is well-suited for tuning the model away from undesirable outputs (verbose, unsafe, off-style) without requiring positive examples to dominate.

13. Reinforcement fine-tuning (RFT) — preview

The newer technique for o-series reasoning models. Instead of (input, output) pairs, you provide:

1. A set of training examples (input + ground truth answer)
2. A grader — a function or model that evaluates how good a response is

The model is then trained via reinforcement learning to maximize grader scores. Best for tasks with measurable correctness (math, code generation, structured tasks with verification).

job = client.fine_tuning.jobs.create(
    training_file=...,
    validation_file=...,
    model="o4-mini-2025-XX-XX",
    method={
        "type": "rft",
        "rft": {
            "grader": {
                "type": "model",                       # or "code", "score_model"
                "model": "o4-mini-grader",
                "instructions": "Score 0-10 ...",
            },
            "hyperparameters": {"n_epochs": 1, "reasoning_effort": "low"},
        },
    },
)

Preview-only as of 2026-05. Limited model + grader availability. Higher cost than SFT (training requires many more rollouts).

Graders

client.fine_tuning.alpha.graders.* API provides standalone grader management:

grader = client.fine_tuning.alpha.graders.create(
    name="Math correctness",
    type="model",
    config={...},
)

14. Continued fine-tuning

You can fine-tune a previously fine-tuned model:

job = client.fine_tuning.jobs.create(
    training_file=...,
    model="ft:gpt-4o-mini-2024-07-18:acme:abc123",        # prior FT result
)

Useful for incremental updates as your data grows.

15. Fine-tuning + Batch API

Fine-tuned models work with the Batch API at the same 50 % discount. Combination of caching + batch + fine-tuned model gives a powerful cost / quality / speed tradeoff:

fine-tuned gpt-4o-mini  →  better quality on your task
+ batch API             →  50 % cost reduction
+ prompt caching         →  another 50 % on cached portion

Effective cost for a heavy offline workload can be a fraction of using gpt-4o sync.

16. Evaluation

OpenAI provides client.evals.* API for systematic evaluation:

eval_obj = client.evals.create(
    name="customer-support-eval",
    data_source_config={"type": "stored_completions"},
    testing_criteria=[
        {"type": "label_model", "model": "o4-mini", "instructions": "Score helpfulness..."},
    ],
)
run = client.evals.runs.create(eval_id=eval_obj.id, data=...)

Run before / after fine-tuning to measure impact.

17. Common pitfalls — Batch

1. JSONL formatting errors — one bad line fails validation of the whole batch. Validate locally first.
2. custom_id duplicates — silently merged or fails. Ensure unique.
3. Mixed endpoints in one batch — endpoint is set at batch level, not per-request. All requests in a JSONL must target the same endpoint.
4. Expired batches — running past 24h leaves partial results. Submit smaller batches.
5. No streaming in batch — output is delivered atomic per request when batch completes.
6. No retries — failed individual requests in a batch don’t auto-retry. Inspect error_file_id, re-submit failures in a new batch.
7. Image / audio data inflates JSONL — base64 of large images explodes file size. Use URL inputs where possible.

18. Common pitfalls — Fine-tuning

1. Too little data — <50 examples typically doesn’t move the needle. Bias overfits to small data.
2. Train-eval split — without a held-out validation set, you can’t catch overfitting. Always supply validation_file.
3. learning_rate_multiplier too high — model “forgets” base capabilities. Default is usually fine; tune carefully.
4. System prompt in training but not at inference — if you trained with a specific system prompt, use the same one at inference. Drift breaks behavior.
5. Fine-tuned model has different rate limits — typically lower than base. Plan capacity.
6. DPO needs balanced preferences — wildly imbalanced or noisy preferences harm more than help.
7. Re-fine-tuning on the same data — incremental gains diminish. Add diverse new data instead.
8. Forgetting to A/B test — fine-tuned model “feels” different but may not be measurably better. Always eval.

19. Comparison to other vendors

Aspect	OpenAI	Claude	Gemini
Batch discount	50 % (input + output)	50 % (input + output)	50 % (input + output)
Batch SLA	24h	24h	24h
Batch input format	JSONL	JSONL	JSONL
Fine-tuning available	SFT, DPO, RFT	No (not offered)	SFT, preference (Vertex)
Models for fine-tuning	gpt-4o-mini, gpt-3.5, others	n/a	gemini-2.5-flash, others
Custom grader / RFT	Yes (preview)	n/a	Limited

20. Further reading

• Batch API guide — canonical reference
• Fine-tuning guide — SFT + DPO + RFT
• Reinforcement fine-tuning announcement — RFT background
• Evaluations API — eval workflows
• openai-python examples — batches — worked examples