Compendium

Workers AI and Cloudflare Agents

10 min read

Workers AI and Cloudflare Agents

Cloudflare’s AI stack has four products that fit together: Workers AI (managed inference for 50+ open-weight models — Llama, Phi, Mistral, DeepSeek, Stable Diffusion, Whisper, etc. — billed per request, no infra to manage), Vectorize (managed vector database for embeddings + semantic search), AI Gateway (analytics + caching + rate limiting + fallback chains across both Workers AI and external providers like OpenAI / Anthropic / Google), and the Cloudflare Agents SDK (durable AI agents backed by Durable Objects — state, tool use, scheduled execution, WebSocket-streaming UI, MCP server hosting). Together these let you build a complete AI application on Cloudflare without leaving the edge.

See also

1. Workers AI — catalog and API

Per developers.cloudflare.com/workers-ai/. Open-weight models hosted on Cloudflare’s GPU fleet. Pay-per-request, no instance management.

Model categories

CategoryNotable modelsUse
Text generation (LLM)@cf/meta/llama-3.3-70b-instruct, @cf/meta/llama-3.1-8b-instruct, @cf/microsoft/phi-3-medium-4k-instruct, @cf/mistral/mistral-small-3.1-24b-instruct, @cf/deepseek-ai/deepseek-r1-distill-qwen-32bChat, completion, instruction-following
Embeddings@cf/baai/bge-base-en-v1.5 (768d), @cf/baai/bge-large-en-v1.5 (1024d), @cf/baai/bge-m3 (1024d, multilingual)Vector search, semantic similarity
Reranking@cf/baai/bge-reranker-baseRAG post-retrieval reranking
Image generation@cf/black-forest-labs/flux-1-schnell, @cf/stabilityai/stable-diffusion-xl-base-1.0, @cf/lykon/dreamshaper-8-lcmText-to-image
Image understanding@cf/llava-hf/llava-1.5-7b-hf, @cf/meta/llama-3.2-11b-vision-instructImage captioning, VQA
Speech-to-text@cf/openai/whisper, @cf/openai/whisper-large-v3-turboAudio transcription
Text-to-speech@cf/myshell-ai/melottsAudio synthesis
Translation@cf/meta/m2m100-1.2b100-language translation
Object detection@cf/facebook/detr-resnet-50Bounding-box detection
Text classification@cf/huggingface/distilbert-sst-2-int8Sentiment, intent

Full catalog (50+ models): developers.cloudflare.com/workers-ai/models/.

Binding

[ai]
binding = "AI"

Inference

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    // Text generation
    const chat = await env.AI.run("@cf/meta/llama-3.3-70b-instruct", {
      messages: [
        { role: "system", content: "You are a coding tutor." },
        { role: "user", content: "Explain async/await." },
      ],
      max_tokens: 512,
      temperature: 0.3,
      stream: false,
    });
    // chat.response is the generated text
    
    // Embedding
    const embed = await env.AI.run("@cf/baai/bge-base-en-v1.5", {
      text: ["Hello world", "Goodbye world"],
    });
    // embed.data[0] is the first embedding (768-dim array)
    
    // Image generation
    const img = await env.AI.run("@cf/black-forest-labs/flux-1-schnell", {
      prompt: "A cat in a wizard hat",
      steps: 4,   // FLUX schnell uses few steps
    });
    return new Response(img, { headers: { "Content-Type": "image/jpeg" } });
  },
};

Streaming

const stream = await env.AI.run("@cf/meta/llama-3.3-70b-instruct", {
  messages: [...],
  stream: true,
});
 
return new Response(stream, {
  headers: { "Content-Type": "text/event-stream" },
});

The SSE stream contains data: { ... } JSON events with incremental tokens.

Tool use

const result = await env.AI.run("@cf/meta/llama-3.3-70b-instruct", {
  messages: [
    { role: "user", content: "What's the weather in Paris?" },
  ],
  tools: [
    {
      type: "function",
      function: {
        name: "get_weather",
        description: "Get current weather for a city",
        parameters: {
          type: "object",
          properties: {
            city: { type: "string" },
          },
          required: ["city"],
        },
      },
    },
  ],
});
 
if (result.tool_calls) {
  for (const tc of result.tool_calls) {
    const args = JSON.parse(tc.function.arguments);
    const weather = await getWeather(args.city);
    // Continue conversation with tool result...
  }
}

Pricing

Per developers.cloudflare.com/workers-ai/platform/pricing/. Different unit per model category:

CategoryUnitExample
Text generationPer-million tokensLlama 3.3 70B: $0.59/MTok in, $0.79/MTok out
EmbeddingsPer-million tokensbge-base-en-v1.5: $0.012/MTok
Image generationPer-megapixel-stepFlux schnell: $0.0001 / megapixel-step
WhisperPer-audio-second$0.000012 / sec

Free plan: 10,000 “neurons” per day (Cloudflare’s normalized AI unit) — enough for hundreds of LLM calls or thousands of embeddings.

2. Vectorize — managed vector database

Per developers.cloudflare.com/vectorize/. Vector index with metadata filtering and namespacing. GA in early 2024.

Setup

wrangler vectorize create my-index --dimensions=768 --metric=cosine
[[vectorize]]
binding = "VECTORS"
index_name = "my-index"

Insert / upsert

await env.VECTORS.insert([
  {
    id: "doc-1",
    values: new Array(768).fill(0).map(() => Math.random()),   // 768d vector
    metadata: {
      source: "blog",
      url: "https://example.com/post-1",
      published_at: 1716508800,
    },
  },
]);
 
// Upsert (overwrites if ID exists)
await env.VECTORS.upsert([{ id: "doc-1", values: [...], metadata: {...} }]);

Query

const results = await env.VECTORS.query(queryEmbedding, {
  topK: 5,
  returnValues: false,
  returnMetadata: true,
  filter: {
    source: { "$eq": "blog" },
    published_at: { "$gt": 1700000000 },
  },
  namespace: "tenant-123",   // optional logical separation within an index
});
 
for (const match of results.matches) {
  console.log(match.id, match.score, match.metadata);
}

Metadata indexes (re:Invent 2024)

Add specific metadata fields to a searchable index for faster filtered queries:

wrangler vectorize create-metadata-index my-index --property-name source --type string
wrangler vectorize create-metadata-index my-index --property-name published_at --type number

Limits

  • Max dimensions: 1536
  • Max vectors per index: 5M (Paid)
  • Max indexes per account: 100 (Paid)
  • Max metadata per vector: 10 KB
  • Distance metrics: cosine, euclidean, dot-product

Pricing

  • Stored vector dimensions: $0.05 / 100M dimensions / month
  • Queried vector dimensions: $0.01 / 1M dimensions

For a 1M-vector index at 768 dimensions:

  • Storage: 1M × 768 = 768M dimensions = $0.38/mo
  • 1M queries against it: 1M × 768 = 768M = $7.68

Significantly cheaper than Pinecone for similar workloads.

3. AI Gateway — observability and resilience across providers

Per developers.cloudflare.com/ai-gateway/. A proxy that sits between your Worker (or any client) and your AI providers (Workers AI, OpenAI, Anthropic, Google, Mistral, Replicate, Bedrock, Azure, Cohere, Groq, Together, HuggingFace, Perplexity, etc.).

Setup

Create a Gateway in the dashboard → AI → Gateway. You get a base URL like https://gateway.ai.cloudflare.com/v1/<account>/<gateway>/<provider>.

Use via direct HTTP

const resp = await fetch(
  "https://gateway.ai.cloudflare.com/v1/<account>/<gateway>/anthropic/v1/messages",
  {
    method: "POST",
    headers: {
      "x-api-key": env.ANTHROPIC_API_KEY,
      "anthropic-version": "2023-06-01",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-6",
      max_tokens: 1024,
      messages: [{ role: "user", content: "Hello!" }],
    }),
  }
);

Use via the AI binding (Workers AI Gateway)

const response = await env.AI.run(
  "@cf/meta/llama-3.3-70b-instruct",
  { messages: [...] },
  {
    gateway: {
      id: "my-gateway",
      cacheTtl: 3600,
      skipCache: false,
      collectLog: true,
    },
  }
);

Features

  • Logging — every request + response is logged with token counts, latency, cost. Searchable via dashboard.
  • Caching — exact-match request cache. TTL configurable. Especially valuable for embedding requests (deterministic queries).
  • Rate limiting — per-IP, per-API-key. Block runaway calls before they hit upstream.
  • Fallbacks — if Anthropic 429s, automatically retry on OpenAI.
  • Cost tracking — aggregated spend by provider, model, token usage.
  • Real-time analytics — request volumes, error rates, token usage curves.

Pricing

Free for first 100k requests/day; Paid usage-based. The savings on cached requests typically pay for the Gateway many times over.

4. Cloudflare Agents SDK

Per developers.cloudflare.com/agents/ and github.com/cloudflare/agents. A TypeScript framework for building stateful AI agents on Cloudflare. Built on:

  • Durable Objects — for per-agent state (each agent instance is a DO)
  • WebSocket Hibernation — for cost-effective real-time UI
  • SQLite-in-DO — for built-in storage
  • Workers AI / AI Gateway — for LLM inference
  • Cron triggers + alarms — for scheduled / proactive agent execution

Install

npx create-cloudflare@latest --template cloudflare/agents-starter

This scaffolds an agent project with React frontend + Worker backend + DO-backed agent class.

Basic agent

import { Agent, callable } from "agents";
 
interface State {
  count: number;
  messages: { role: string; content: string }[];
}
 
export class CounterAgent extends Agent<Env, State> {
  initialState: State = { count: 0, messages: [] };
  
  @callable()
  async increment(): Promise<number> {
    this.setState({ ...this.state, count: this.state.count + 1 });
    return this.state.count;
  }
  
  @callable()
  async chat(message: string): Promise<string> {
    const messages = [
      { role: "system", content: "You are a helpful assistant." },
      ...this.state.messages,
      { role: "user", content: message },
    ];
    
    const result = await this.env.AI.run("@cf/meta/llama-3.3-70b-instruct", {
      messages,
    });
    
    this.setState({
      ...this.state,
      messages: [
        ...messages.slice(1),
        { role: "assistant", content: result.response },
      ],
    });
    
    return result.response;
  }
}

AIChatAgent — multi-turn chat with persistence built in

import { AIChatAgent } from "agents/ai-chat-agent";
import { streamText } from "ai";
import { workersai } from "workers-ai-provider";
 
export class ChatAgent extends AIChatAgent<Env> {
  async onChatMessage(onFinish: any): Promise<Response> {
    const model = workersai(this.env.AI)("@cf/meta/llama-3.3-70b-instruct");
    
    const result = streamText({
      model,
      messages: this.messages,        // pre-populated history
      tools: {                         // SDK-provided tools
        getWeather: { ... },
      },
      onFinish,
    });
    
    return result.toDataStreamResponse();
  }
}

The AIChatAgent class automatically:

  • Persists conversation history in the DO’s SQL storage.
  • Syncs state to connected WebSocket clients in real time.
  • Plays well with useAgent React hook for client-side chat UIs.

React client hook

"use client";
import { useAgent } from "agents/react";
 
export function ChatUI() {
  const { messages, sendMessage, state } = useAgent({
    agent: "chat-agent",
    name: "user-7281-session",   // DO addressing — same name = same agent
  });
  
  return (
    <div>
      {messages.map((m, i) => <div key={i}>{m.role}: {m.content}</div>)}
      <form onSubmit={(e) => { e.preventDefault(); sendMessage(input); }}>
        <input ... />
      </form>
    </div>
  );
}

Tools — server-side

import { Agent, tool } from "agents";
import { z } from "zod";
 
export class TripPlannerAgent extends Agent<Env> {
  async runTrip(destination: string) {
    const result = await this.askAI({
      messages: [{ role: "user", content: `Plan a trip to ${destination}` }],
      tools: {
        searchFlights: tool({
          description: "Search for flights",
          parameters: z.object({
            from: z.string(),
            to: z.string(),
            date: z.string(),
          }),
          execute: async ({ from, to, date }) => {
            // Hit your flight API
            return { flights: [...] };
          },
        }),
      },
    });
    return result;
  }
}

Scheduled execution

Agents can self-schedule via alarms (inherited from DO):

async checkSubscription() {
  const expires = this.state.subscriptionExpires;
  if (expires - Date.now() < 86400000) {   // < 1 day
    await this.sendReminderEmail();
  }
  
  // Schedule next check in 1 day
  await this.ctx.storage.setAlarm(Date.now() + 86400000);
}
 
async alarm() {
  await this.checkSubscription();
}

Human-in-the-loop

Pattern: agent emits a requiresApproval event, the UI shows it to the user, user approves → agent continues. Built-in helpers:

async approveAction(action: Action): Promise<boolean> {
  this.setState({
    ...this.state,
    pendingApproval: action,
  });
  
  // Wait for approval (resolves via DO alarms or external trigger)
  return new Promise((resolve) => {
    this.approvalResolvers.set(action.id, resolve);
  });
}

MCP server hosting

Agents can expose themselves as MCP (Model Context Protocol) servers — other AI agents (Claude Code, Cursor, etc.) can call them as tools. Configure in the agent class:

import { McpAgent } from "agents/mcp";
 
export class MyMcpAgent extends McpAgent<Env> {
  async initializeMcp(server: McpServer) {
    server.tool("search-docs", { description: "Search internal docs" }, async ({ query }) => {
      // ... search logic
      return { content: [{ type: "text", text: results }] };
    });
  }
}

Then any MCP-aware client (Claude Code, Cursor, MCP Inspector) can connect to your agent via the MCP transport (Streamable HTTP).

5. The AI SDK Workers AI provider

Per npm.cloudflare.com/workers-ai-provider. Use Vercel’s AI SDK syntax with Workers AI:

import { createWorkersAI } from "workers-ai-provider";
import { generateText } from "ai";
 
const workersai = createWorkersAI({ binding: env.AI });
 
const { text } = await generateText({
  model: workersai("@cf/meta/llama-3.3-70b-instruct"),
  prompt: "Hello!",
});

Lets you write Vercel-AI-SDK-compatible code that runs on Cloudflare with Workers AI instead of an external provider.

6. RAG pattern — end-to-end on Cloudflare

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const { query } = await request.json();
    
    // 1. Embed query
    const embed = await env.AI.run("@cf/baai/bge-base-en-v1.5", { text: [query] });
    const queryVec = embed.data[0];
    
    // 2. Search Vectorize
    const matches = await env.VECTORS.query(queryVec, {
      topK: 5,
      returnMetadata: true,
    });
    
    // 3. Rerank (optional)
    const rerankInput = matches.matches.map(m => m.metadata.text as string);
    const reranked = await env.AI.run("@cf/baai/bge-reranker-base", {
      query,
      contexts: rerankInput.map(text => ({ text })),
    });
    
    // 4. Build context
    const context = reranked.response
      .sort((a, b) => b.score - a.score)
      .slice(0, 3)
      .map(r => rerankInput[r.id])
      .join("\n\n---\n\n");
    
    // 5. Generate answer
    const answer = await env.AI.run("@cf/meta/llama-3.3-70b-instruct", {
      messages: [
        { role: "system", content: `Answer based on:\n${context}` },
        { role: "user", content: query },
      ],
    });
    
    return Response.json({ answer: answer.response });
  },
};

All four stages (embed, search, rerank, generate) run on Cloudflare. No external API keys, no egress fees, single-vendor billing.

7. Costs at scale — Workers AI vs OpenAI/Anthropic

Comparing Llama 3.3 70B on Workers AI vs Claude Sonnet 4.6 on Anthropic for a typical workload:

ProviderInputOutputNote
Workers AI Llama 3.3 70B$0.59/MTok$0.79/MTok~6× cheaper
Anthropic Claude Sonnet 4.6$3/MTok$15/MTok~10× more capable on many tasks

Workers AI wins on price; Claude wins on quality. For agent-style workflows where Llama 3.3 is “good enough” (classification, extraction, basic tool-calling), Workers AI’s price advantage compounds rapidly. For nuanced reasoning, stick with Claude (via AI Gateway).

8. Limits

  • Workers AI requests per minute: 720 default per account on Paid; higher quotas available.
  • Max tokens per generation: model-dependent (typically 4-32k).
  • Max image size for vision: 4 MB.
  • Vectorize ingestion rate: 100 vectors/sec/index sustained; bursts higher.

Further reading

Graph View