The answer. Give the product one deliberate, instant lever: a plain Worker var (RUNS_KILL_SWITCH) checked at the single choke point every dispatch passes through. A var edited in the dashboard takes effect without a deploy, and the check runs before any database work, so it holds even when everything behind it is broken — fail-closed by design. Beside it, add automatic runaway detection: a per-tenant and a global ceiling on work started in the trailing 24 hours. Those must fail open: if the counter query errors, log and allow. A counting outage must never become a self-inflicted product outage.
The pattern.
// lib/spendGuard.ts — evaluated at the one dispatch choke point
export function parseCap(raw: string | undefined, fallback: number) {
const n = Number.parseInt(raw ?? "", 10);
return Number.isFinite(n) && n > 0 ? n : fallback; // "0" or a typo must not block every run
}
export async function checkSpendGuard(env: Env, orgId: string) {
if (env.RUNS_KILL_SWITCH === "true") // fail CLOSED, no I/O first
return { ok: false, code: "runs_disabled" };
const since = new Date(Date.now() - 86_400_000).toISOString();
try {
const [global, org] = await Promise.all([countRuns(env, since), countRuns(env, since, orgId)]);
if (org >= parseCap(env.MAX_RUNS_PER_ORG_PER_DAY, 60)) return { ok: false, code: "org_run_limit" };
if (global >= parseCap(env.MAX_RUNS_PER_DAY, 500)) return { ok: false, code: "global_run_limit" };
return { ok: true };
} catch (err) {
console.warn("[spendGuard] ceiling check failed open:", err);
return { ok: true }; // fail OPEN
}
}
[env.prod]
vars = { RUNS_KILL_SWITCH = "false", MAX_RUNS_PER_DAY = "500", MAX_RUNS_PER_ORG_PER_DAY = "60" }
The receipt. In our API Worker, added 2026-06-20 as launch prep: every run (on-demand, autopilot, make-up) triggers paid third-party calls, and the monthly article quota caps published articles, not attempts. A dispatch bug, an n8n retry storm, a burst of trial sign-ups or a compromised account could still run spend away. The defaults are safety ceilings, not throttles: 500 runs per 24 h globally and 60 per org, against a projected ~84 articles a day. The kill switch sits among ~40 prod vars, which is exactly why our deploy script never passes --var. Whether either ceiling has ever tripped in production is not recorded — unmeasured.
Watch out.
- Parse caps defensively: an unset or
"0"var must fall back to the default, not block every run. - Check the org cap before the global cap so the more specific reason reaches the user.
- The other fail-closed gate in the same Worker idled crons silently for 11 days. Fail-closed levers must be visible on
/health.
Related: cron-gate-outage-kill-switch-returned-early · deploy-with-rollback-target-health-probe