Check provider balances on a cron; a card lapse killed runs for 24 h

Alert on failed runs, not just bad output, and poll each paid provider's balance from the cron: a daily digest plus hourly under-floor alarms, KV-deduped.

Observability

· Chapter

22

·

3

min read

The answer. A pipeline paying several third-party APIs has a failure mode no test catches: a prepaid card lapses, one provider refuses, and every run fails looking like a content-quality problem. Two fixes, both in the cron you already have. First, make failed runs a first-class alert — if alerts only bump on "output rejected", a run that never produced output bumps nothing. Second, a balance check-in: call every provider's balance endpoint, send a morning digest, and raise an hourly under-floor alarm when one drops below PROVIDER_BALANCE_LOW_USD (default $10). Dedupe alarms in KV so a low balance emails once per window, not every tick.

The pattern.

// lib/providerBalances.ts — read-only probes; providers without an API get a console link
const PROBES = [
  { name: "llm-a",  url: "https://api.example-llm.com/user/balance",            read: (r: any) => Number(r.balance_infos?.[0]?.total_balance) },
  { name: "llm-b",  console: "https://console.example-llm-b.com/billing" },                                       // no balance API exists
];
export async function providerBalanceCheckIn(env: Env, opts: { digest: boolean }) {
  const floor = Number(env.PROVIDER_BALANCE_LOW_USD ?? "10");
  const rows = await Promise.all(PROBES.map(async (p) => ({ ...p, usd: await probe(env, p) })));
  if (opts.digest) await sendEmail(env, "Provider balances", renderTable(rows));          // 05:00 UTC tick
  for (const r of rows.filter((r) => r.usd != null && r.usd < floor)) {
    const key = `balance-alarm:${r.name}:${new Date().toISOString().slice(0, 13)}`;      // hourly dedupe
    if (await env.KV.get(key)) continue;
    await env.KV.put(key, "1", { expirationTtl: 3600 });
    await sendEmail(env, `LOW balance: ${r.name} $${r.usd!.toFixed(2)}`, `floor is $${floor}`);
  }
}
// scheduled(): ctx.waitUntil(providerBalanceCheckIn(env, { digest: new Date().getUTCHours() === 5 }));

The receipt. In our API Worker, 2026-08-02: article runs had failed since 2026-08-01 with "too many empty sections". The owner's card had been cancelled after theft, the LLM provider behind the section writers couldn't charge, and five empty sections failed the run downstream. No alert fired: the reliability delta system knew only article_hard_rejected and autopilot_slot_canceled, so failed runs bumped nothing for about 24 hours. Same evening a run_failed alert type went live, proven with a backfilled count of 3 → one email at the 19:00 UTC tick; the balance check-in shipped alongside, with the 05:00 UTC digest and hourly alarms. Five balance endpoints were live-verified; a provider with no balance API at all (404 on every path probed) shows as a console link.

Watch out.

  • The provider that lapsed has no balance API — the digest shows a console link; only the run_failed alert would have fired. Ship both.
  • A send-only email key can't tell you whether an alert arrived. Verify delivery once with a full-access key, then rotate it.
  • Never auto-top-up from the Worker; a runaway run plus auto-top-up is how a spend guard gets bypassed.

Related: cron-heartbeat-dead-mans-switch · cron-one-tick-many-isolated-legs · kill-switch-var-fail-closed-ceilings-fail-open