Put AI Gateway in front of every provider call: logs, cache, retries

The answer. Every LLM integration eventually grows the same plumbing: request logs with token counts, a cache for repeated prompts, per-key rate limits, retries with backoff, a fallback model. AI Gateway is that plumbing as a proxy. The cheapest adoption is a base-URL swap: point the OpenAI or Anthropic SDK at https://gateway.ai.cloudflare.com/v1/<account>/<gateway>/<provider> and keep your provider key. Since 2026 there is also a unified REST API on api.cloudflare.com (/ai/v1/chat/completions, /ai/v1/messages, /ai/run) that accepts provider/model ids and can bill third-party models through your Cloudflare account. Per-request behavior is set with cf-aig-* headers: cache TTL and key, skip-cache, request timeout, retry attempts and backoff, and metadata that lands in the log. A gateway named default is created on first use.

The pattern.

import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
  apiKey: env.ANTHROPIC_API_KEY,
  baseURL: `https://gateway.ai.cloudflare.com/v1/${env.CF_ACCOUNT_ID}/content-pipeline/anthropic`,
  defaultHeaders: {
    "cf-aig-authorization": `Bearer ${env.CF_AIG_TOKEN}`,   // only if the gateway requires auth
    "cf-aig-cache-ttl": "3600",                             // identical prompts hit the cache for an hour
    "cf-aig-max-attempts": "3", "cf-aig-backoff": "exponential", "cf-aig-request-timeout": "60000",
    "cf-aig-metadata": JSON.stringify({ run: runId, stage: "draft" }),
  },
});

Watch out.

Caching is keyed on the whole request. A timestamp or a random id inside the prompt defeats it; use cf-aig-cache-key to group equivalent requests deliberately.
The gateway sees your prompts and completions in its logs. Decide what is allowed to be logged (cf-aig-collect-log: false per request) before routing customer data through it.
Timeouts count to first byte, so a streamed response that starts fast never times out mid-stream.