Our fail-closed cron gate was the outage: 11 silent days

A kill-switch pinned to true made scheduled() return early; nothing alerted for 11 days. Expose gate state on /health and check it after every deploy.

Queues & Crons

· Chapter

5

·

3

min read

The answer. A fail-closed gate at the top of scheduled() is the right design: a freshly provisioned environment must not run customer-mutating crons until someone has proven its secrets and its deployment. But "fail closed" means "fail silently" unless you do three more things. Expose the gate's decision and its reason on /health, so the state is one curl away. Pin the vars that open the gate in wrangler.toml for the prod env, not in a deploy script that a direct wrangler deploy bypasses. And check it after every deploy: a Worker var holding the wrong string produces no error, no log line, no failed request. Just an idle cron.

The pattern.

// lib/scheduledMutationGate.ts — fail-closed; reasons are data, not prose
export function scheduledMutationGate(env: Env) {
  if (env.SCHEDULED_MUTATIONS_KILL_SWITCH === "true")
    return { enabled: false, reason: "kill_switch" };
  if (env.DEPLOYMENT_GATE_STATUS !== "verified")
    return { enabled: false, reason: "deployment_gate_unverified" };
  return { enabled: true, reason: null };
}
// index.ts
scheduled(_c, env, ctx) {
  const gate = scheduledMutationGate(env);
  if (!gate.enabled) { console.warn("cron gated", gate.reason); return; }
  /* legs... */
}
// GET /health -> { runtime_gates: { scheduled_mutations_enabled, reason } }
[env.prod]
vars = { SCHEDULED_MUTATIONS_KILL_SWITCH = "false", DEPLOYMENT_GATE_STATUS = "verified" }

The receipt. HarperFlow's autopublish was dead from about 2026-07-10 (a database cutover) to 2026-07-21. SCHEDULED_MUTATIONS_KILL_SWITCH="true" sat in the prod env block, so scheduled() returned on line one and no sweep ran at all. At diagnosis the hosted database had zero published articles since the cutover. Two days after the fix it broke again: the guarded release pipeline set DEPLOYMENT_GATE_STATUS=verified as its final attestation step, and every direct wrangler deploy --env prod left it unset. Pinning both vars in wrangler.toml on 2026-07-23 ended the class; that evening the first unassisted publish landed on the 20:30 UTC tick, six hours overdue. A second, stacked bug in the due-publisher was bridged the same week.

Watch out.

  • The instant run kill switch (a dashboard-editable var) is fail-closed by design; the spend ceilings beside it fail open on a counting error. Know which is which.
  • Any gate that only a pipeline step opens will close on the first out-of-band deploy.
  • Add a watchdog on durable state (scheduled rows more than 6 hours overdue); it catches what no test layer can.

Related: cron-heartbeat-dead-mans-switch · cron-one-tick-many-isolated-legs