Vision and long context (Gemini)

Gemini is natively multimodal end-to-end — every chat-tier model (3.x, 2.5, 2.0, 1.5) accepts text + image + audio + video + PDF in any combination, decoded by the same model rather than routed to separate sub-models. Combined with the 1M-token context window (2M on Gemini 1.5 Pro), this is the line’s signature capability: you can drop a 60-minute video, a 1000-page PDF, or 9 hours of audio into a single request and reason across the whole thing. The cost discipline that makes this economical is context caching — without caching, processing a 500k-token document repeatedly is expensive; with explicit caching, the cost drops by ~75 %.

See also

• gemini-models-and-capabilities — context windows per model
• context-caching-gemini — caching long content for cost control
• gemini-api-and-sdks — Files API surface
• vision-and-multimodal — sister doc for Claude
• vision-and-multimodal-openai — sister doc for OpenAI

1. Modality matrix

Per ai.google.dev/gemini-api/docs/vision and the modality-specific pages.

Modality	Token cost	Per-request limits
Text	1 token ≈ 4 chars	Up to 1M (or 2M on 1.5 Pro)
Image (≤ 384 × 384 px)	258 tokens flat	Up to 3,600 images per request
Image (larger)	258 tokens × tile_count (split into 768×768 tiles)	Same
PDF	~258 tokens / page	Up to ~3,600 pages (token cap is the real limit)
Audio	32 tokens / second (~115k tokens / hour)	Up to ~9.5 hours per 1M context
Video	1 fps sampled, ~263 tokens / frame (image cost) + 32 / sec audio	Up to ~1 hour per 1M context

2. Image input

Two input methods

Inline (Base64, under 20 MB total request size):

import pathlib
from google.genai import types
 
resp = client.models.generate_content(
    model="gemini-3.5-flash",
    contents=[
        "Describe this image.",
        types.Part.from_bytes(
            data=pathlib.Path("photo.jpg").read_bytes(),
            mime_type="image/jpeg",
        ),
    ],
)

Files API (for larger files, repeated reuse, or files >20 MB):

img = client.files.upload(file="big-image.jpg")
resp = client.models.generate_content(
    model="gemini-3.5-flash",
    contents=["Describe this image.", img],
)

URL inputs are not supported directly (unlike OpenAI’s image_url) — fetch externally and pass bytes, or upload to Files API first.

Supported formats

image/png, image/jpeg, image/webp, image/heic, image/heif. No image/gif (which Claude supports).

Sizing rules

• Per ai.google.dev/gemini-api/docs/vision: “Images ≤ 384 pixels on both dimensions cost 258 tokens. Larger images are divided into 768 × 768 tiles, each costing 258 tokens.”
• A 1024×768 image = 4 tiles = 1,032 tokens (rounded up; 2×2 tile grid covers 1536×1536 worst case).
• A 4K image (3840×2160) ≈ 6 × 3 = 18 tiles = ~4,644 tokens.

media_resolution parameter

config=types.GenerateContentConfig(
    media_resolution=types.MediaResolution.MEDIA_RESOLUTION_LOW,
)

Three levels: LOW, MEDIUM, HIGH. Trades detail-fidelity for tokens. LOW reduces the image to a single tile (258 tokens) regardless of size — useful for classification where the model just needs to know what the image is, not read fine print. Default is MEDIUM.

Best practices

Per the docs:

• Position the prompt text after image parts in the contents list
• Use sharp, well-lit, unblurred images
• Ensure correct rotation (EXIF auto-rotation may not be applied)
• For object detection / bounding boxes: outputs are normalized to [0, 1000] scale relative to image dimensions

Object detection

The model can return bounding boxes directly when asked:

resp = client.models.generate_content(
    model="gemini-3.5-flash",
    contents=[
        "Detect all cars in this image. Return bounding boxes as JSON: "
        "[{label, box_2d: [ymin, xmin, ymax, xmax]}].",
        image,
    ],
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
    ),
)

Coordinates are normalized to [0, 1000]. Convert to pixels:

ymin_px = int(ymin / 1000 * image_height)
xmin_px = int(xmin / 1000 * image_width)

3. PDF input

Native PDF support — no preprocessing, no extracted-text-only path. The model sees both layout + text.

pdf = client.files.upload(file="annual-report.pdf")
resp = client.models.generate_content(
    model="gemini-3.5-flash",
    contents=["Summarize this report. What was Q3 revenue?", pdf],
)

Per ai.google.dev/gemini-api/docs/vision:

• Max ~3,600 pages per request (constrained by 1M context: ~258 tokens / page × 3,600 ≈ 928k tokens, leaving headroom for the prompt)
• Each page renders as an image (table layout, charts, formulas, signatures preserved)
• Inline PDFs limited to 20 MB; use Files API for larger

PDFs are the killer Gemini use case for long-context work — financial reports, legal contracts, research papers all parse natively without an OCR step.

4. Video input

Upload

video = client.files.upload(file="meeting-recording.mp4")
# Wait for processing
import time
while video.state.name == "PROCESSING":
    time.sleep(2)
    video = client.files.get(name=video.name)
if video.state.name == "FAILED":
    raise RuntimeError("Video processing failed")

Videos require a processing step before they’re queryable. Larger videos take longer — a 1-hour video may take 1-2 minutes to process.

Limits

Per ai.google.dev/gemini-api/docs/video and observed limits:

Per-file limit	Value
Max duration (any single video)	1 hour (on 1M-context models); proportionally longer on 2M (1.5 Pro)
Max file size	2 GB
Supported formats	MP4, MOV, MPEG, AVI, FLV, MPG, WEBM, WMV, 3GPP

Frame sampling

The Files API samples at 1 fps by default. A 60-minute video = 3,600 frames × ~263 tokens / frame ≈ 947k tokens, plus 60 min × 60 sec × 32 tokens/sec = 115k audio tokens. That’s over 1M — the practical limit on a 1M-context model is ~30-45 minutes if you want headroom for the prompt + response.

Custom sampling

You can pass video_metadata to limit segment processing:

contents=[
    types.Part(
        file_data=types.FileData(file_uri=video.uri, mime_type="video/mp4"),
        video_metadata=types.VideoMetadata(
            start_offset="120s",
            end_offset="180s",
            fps=2,           # request 2 fps sampling
        ),
    ),
    "Describe what happens between 2:00 and 3:00.",
]

Custom sampling beyond 1 fps + start/end offsets is a more recent addition; check the docs for current support.

YouTube URL support

Pass YouTube URLs directly without uploading:

contents=[
    "Summarize this video.",
    types.Part(file_data=types.FileData(file_uri="https://www.youtube.com/watch?v=...")),
]

Restricted to public videos. Subject to YouTube usage limits.

5. Audio input

Upload

audio = client.files.upload(file="podcast.mp3")
resp = client.models.generate_content(
    model="gemini-3.5-flash",
    contents=["Summarize this podcast. List the key points.", audio],
)

Limits

Per-file limit	Value
Max duration	~9.5 hours per 1M context (32 tokens/sec × 9.5 × 3600 = 1.09M, slightly over)
Practical max	8 hours (leaves headroom)
Supported formats	MP3, WAV, FLAC, AAC, OPUS, AIFF, etc.

Capabilities

• Transcription (per the docs, accuracy “competitive with dedicated ASR systems”)
• Summarization across multiple speakers
• Sentiment / topic detection per segment
• Q&A with timestamp references (Gemini will cite “around 1:23:45”)
• Audio + visual together (in video): “what does the speaker mean at 0:42?“

6. Long context: when it works, when it doesn’t

Per ai.google.dev/gemini-api/docs/long-context:

Strong cases:

• Summarize a 500k-token document
• Q&A against a single retrieved document with full context
• Many-shot in-context learning (hundreds of examples in the prompt — can match fine-tuned model performance)
• “Where in this video does X happen” (cross-modal retrieval)
• Single-needle retrieval: ~99 % accuracy on finding one specific fact in long context

Weak cases:

• Multi-needle retrieval (“find all 5 mentions of X across this 800k-token corpus”) — accuracy varies widely
• Tasks that require synthesis across many distant chunks (“compare claims in chapter 1 vs chapter 15”) — sometimes degrades vs RAG
• Time-sensitive performance — long-context calls have higher latency (proportional to input length)

Best practice from the docs

“Place your query at the end of the prompt for optimal performance with lengthy context.”

Long-doc → query, not query → long-doc.

Context cost tradeoff

A 500k-token call on Gemini 3.5 Flash:

• Input: 500k × $1.50 / MTok = $0.75 per call
• Output: ~5k × $9.00 / MTok = $0.045
• Total: ~$0.80 per call

Across 50 questions on the same document = $40. With explicit caching: ~$12 (see context-caching-gemini). Always cache for any long-context workflow with reuse.

7. Files API specifics

Per ai.google.dev/gemini-api/docs/files-api:

Property	Limit / behavior
Per-file size	2 GB
Total project quota	20 GB
Auto-expiry	48 hours after upload
Visibility	Project-scoped (no cross-project sharing)
Files in cache	Cached files require source File to exist; deleting File invalidates cache

# Lifecycle
f = client.files.upload(file="path/to/file")
print(f.name, f.state, f.expiration_time)
 
# Wait for processing (videos especially)
while f.state.name == "PROCESSING":
    time.sleep(2)
    f = client.files.get(name=f.name)
 
# Use
client.models.generate_content(model="...", contents=["prompt", f])
 
# Delete (optional — auto-expires at 48h)
client.files.delete(name=f.name)
 
# List all files
for f in client.files.list():
    print(f.name, f.display_name, f.size_bytes, f.expiration_time)

For workloads where files are reused beyond 48 hours, re-upload periodically. For permanent storage, host externally and pass bytes per-request (subject to 20MB inline limit) or upload to Vertex AI which has different lifetime semantics.

8. Combining modalities in one request

contents = [
    "Compare this document to what's said in this audio recording.",
    pdf_file,
    audio_file,
    types.Part.from_bytes(data=chart_image, mime_type="image/png"),
    "Focus on financial claims and flag discrepancies.",
]
resp = client.models.generate_content(
    model="gemini-3.5-flash",
    contents=contents,
)

The order matters: per the docs, place the prompt text at the end when working with media. Token budget accumulates across all parts.

9. Sizing pre-flight check

Before sending a large multimodal request, count tokens:

count = client.models.count_tokens(
    model="gemini-3.5-flash",
    contents=[prompt, pdf_file, audio_file],
)
print(f"Will cost: {count.total_tokens} tokens of input")

count_tokens is free and gives accurate per-modality breakdown. Use to:

• Decide whether to chunk
• Compare cache-create cost vs per-request cost
• Pre-flight rate-limit math

10. Common pitfalls

1. Processing state ignored — client.files.upload returns immediately for images / PDFs but video / large audio needs PROCESSING → ACTIVE transition. Always poll for video.
2. 48-hour file expiry — long-running workflows lose file references. Re-upload or stash bytes.
3. image/gif not supported — convert to PNG / WebP first.
4. URL image inputs not supported on chat models — must upload to Files API or send bytes.
5. Tile-count math — uploading a 16-megapixel photo costs ~18 tiles ≈ 4,644 tokens unless you set media_resolution: LOW. Surprise cost.
6. Video > 30 min on 1M context — token budget exhausted; either chunk, downsample fps, or use media_resolution: LOW (which propagates to video frames).
7. File deletion mid-request — deleting a referenced file while a request is in flight returns 400. Sequence the lifecycle.
8. Long context latency — a 500k-token call may take 30-60 seconds even on Flash. Don’t block UI; stream or background.
9. YouTube videos that are unlisted / private — fail silently. Only public videos work.
10. Audio without language hint — if the audio is in a non-English language, including a hint in the prompt (“This is in Japanese”) improves quality.

11. Migrating long-context patterns

From RAG → long context

If your retrieval returns the same top-3 chunks repeatedly, just put the whole doc in context (cached):

# Before: RAG
top_chunks = retrieve(query, k=3)
context = "\n\n".join(top_chunks)
resp = llm.complete(prompt + context)
 
# After: long context + explicit cache
cache = client.caches.create(model="...", config=...(contents=[whole_document]))
# ... reuse cache.name across many queries ...

Tradeoff: RAG retrieval is fast and precise; long context is slower (higher latency, larger TPMs) but avoids retrieval bugs (wrong chunk, missed connection across chunks). For docs under ~500k tokens with frequent reuse, caching usually wins.

From transcription pipeline → direct audio Q&A

Old:

audio -> Whisper -> transcript -> LLM

New:

audio -> Gemini (direct Q&A)

Cuts one model out of the path; preserves prosody / non-text cues; halves latency.

12. Further reading

• Vision — image + PDF input
• Long context — 1M / 2M tradeoffs
• Audio — audio input + capability
• Video — video sampling + limits
• Files API — upload lifecycle
• Pricing — token costs by modality — exact rates