Kill switch as a Worker var: fail closed; spend ceilings: fail open

RUNS_KILL_SWITCH lives in dashboard-editable vars (no deploy) as the only fail-closed guard; rolling-24h run ceilings beside it fail open on a counting error.

Security

· Chapter

12

·

3

min read

The answer. Give the product one deliberate, instant lever: a plain Worker var (RUNS_KILL_SWITCH) checked at the single choke point every dispatch passes through. A var edited in the dashboard takes effect without a deploy, and the check runs before any database work, so it holds even when everything behind it is broken — fail-closed by design. Beside it, add automatic runaway detection: a per-tenant and a global ceiling on work started in the trailing 24 hours. Those must fail open: if the counter query errors, log and allow. A counting outage must never become a self-inflicted product outage.

The pattern.

// lib/spendGuard.ts — evaluated at the one dispatch choke point
export function parseCap(raw: string | undefined, fallback: number) {
  const n = Number.parseInt(raw ?? "", 10);
  return Number.isFinite(n) && n > 0 ? n : fallback;   // "0" or a typo must not block every run
}
export async function checkSpendGuard(env: Env, orgId: string) {
  if (env.RUNS_KILL_SWITCH === "true")                    // fail CLOSED, no I/O first
    return { ok: false, code: "runs_disabled" };
  const since = new Date(Date.now() - 86_400_000).toISOString();
  try {
    const [global, org] = await Promise.all([countRuns(env, since), countRuns(env, since, orgId)]);
    if (org >= parseCap(env.MAX_RUNS_PER_ORG_PER_DAY, 60)) return { ok: false, code: "org_run_limit" };
    if (global >= parseCap(env.MAX_RUNS_PER_DAY, 500))     return { ok: false, code: "global_run_limit" };
    return { ok: true };
  } catch (err) {
    console.warn("[spendGuard] ceiling check failed open:", err);
    return { ok: true };                                  // fail OPEN
  }
}
[env.prod]
vars = { RUNS_KILL_SWITCH = "false", MAX_RUNS_PER_DAY = "500", MAX_RUNS_PER_ORG_PER_DAY = "60" }

The receipt. In our API Worker, added 2026-06-20 as launch prep: every run (on-demand, autopilot, make-up) triggers paid third-party calls, and the monthly article quota caps published articles, not attempts. A dispatch bug, an n8n retry storm, a burst of trial sign-ups or a compromised account could still run spend away. The defaults are safety ceilings, not throttles: 500 runs per 24 h globally and 60 per org, against a projected ~84 articles a day. The kill switch sits among ~40 prod vars, which is exactly why our deploy script never passes --var. Whether either ceiling has ever tripped in production is not recorded — unmeasured.

Watch out.

  • Parse caps defensively: an unset or "0" var must fall back to the default, not block every run.
  • Check the org cap before the global cap so the more specific reason reaches the user.
  • The other fail-closed gate in the same Worker idled crons silently for 11 days. Fail-closed levers must be visible on /health.

Related: cron-gate-outage-kill-switch-returned-early · deploy-with-rollback-target-health-probe