launchd Calendar Fires Die Across Reboot: The Recovery Net Pattern

I armed a one-shot LaunchAgent for 08:00 UTC on a Saturday. The machine rebooted at 19:24 the night before. When morning came, the agent had not run. There was no error in the runner log. There was no launchd complaint. The plist was still loaded. The fire was gone.

That is the class of bug that feels unfair the first time you hit it, because half of what you know about launchd is true and the other half is a trap. Sleep coalesces missed StartCalendarInterval fires. Reboot discards them. Same API. Opposite reliability. If your automation must happen, a bare calendar one-shot is a bet that the laptop never restarts between arm and fire — and laptops restart. Solo builders who run a Mac as the control plane learn this the expensive way: the job you trusted to fire after a power cycle never shows up.

What people think launchd does

The folklore on a Mac control plane is reasonable:

• Prefer launchd over cron.
• Use StartCalendarInterval when you care about wall-clock time.
• Trust sleep recovery: if the machine was asleep at 07:00, launchd runs the job when it wakes.

That last sentence is true for sleep. It is not true for reboot. I treated them as the same recovery class for years because the docs and the tribal knowledge both talk about "missed intervals" without drawing a hard line between power states. Sleep is deferred. Reboot is deleted. No banner. No exit code. No ==== fired line in your runner log. Silent drop.

For optional hygiene jobs, silence is annoying. For irreversible or time-bound work — decommission a rollback host, verify a production promote, close a decision gate — silence is a reliability failure dressed up as an empty morning.

The live miss: the Fly rollback decommission

During the thesolostack.dev Ghost→Pyre cutover window I scheduled a one-shot to decommission the idle Fly.io Ghost rollback once the soak window closed. The runner was deliberate and fail-closed:

• Label: a dedicated one-shot, one label per job
• Fire: 08:00 UTC, the morning the soak window closed
• Payload: dispatch a pre-authored prompt; never reason about Fly inside the runner
• On verified dispatch: self-disarm
• On failure: stay armed and escalate (fail-armed-and-loud)

The control Mac power-cycled at 19:24 the night before the fire. After reboot, every edge-triggered piece of that campaign failed the same way: the calendar fire never appeared, a bare gate-server process was dead, heartbeats went stale. Everything level-triggered or static survived. That contrast is the lesson. The calendar agent was not "slow." It was not "queued." It was discarded.

I recovered the decommission work inline in the next session. That is the human recovery path: notice the silence, kickstart or run the payload, write the scar down. Human recovery does not scale. The next one-shot in the same campaign was hours later and could not depend on me staring at logs.

The recovery net pattern

The fix is not a cleverer plist key. There is no RetryAcrossReboot flag. The fix is a second agent that is level-triggered:

1. One-shot calendar agent — still the preferred way to name "fire at this wall-clock time." Keep the narrow runner. Keep fail-armed-and-loud on dispatch failure.
2. Recovery watcher — RunAtLoad + StartInterval (I used 30 minutes). On every tick it asks a durable question: is the scheduled time past the grace window and is the success artifact still missing?
3. On lost fire — kickstart the original label (or run the same runner). Do not invent a second payload path.
4. On verified success — the one-shot self-disarms (plist leaves ~/Library/LaunchAgents). The recovery net sees the target plist gone and disarms itself.
5. Hard deadline — if the job never fires by T+24h, escalate loud and stop spinning forever.

That is the shape I shipped for the next critical fire the following day (a promote-verify one-shot with a paired -recovery watcher label). The recovery script's comment block is the whole doctrine in one paragraph: launchd drops missed calendar fires across reboot; sleep only defers them; this watcher closes the class.

The condition the watcher checks is intentionally boring. It greps the runner log for a known start marker (==== fired). If the fire time plus grace has passed and that marker is absent, the fire was lost. Kickstart. If the marker is present but the agent is still armed, the runner already owns escalation — recovery stays quiet. If the target plist is gone, success already happened — recovery disarms. No LLM. No database write on the happy path. Just time, a file, and launchctl kickstart.

Sleep vs reboot

If you only test by sleeping the Mac overnight, you will ship a false confidence. Reboot is the acceptance test for calendar one-shots. Force a reboot before the fire time in a staging label. Watch the runner log. If you see nothing at T+grace, your recovery net is the product, not the calendar agent.

When not to use bare calendar agents

Do not put irreversible work behind a bare one-shot calendar fire:

• Destroying a rollback host
• Rotating production credentials
• Promoting a publish that has no independent public check
• Closing a human decision gate that only listens while a local server is up

For those, either:

• Pair calendar + recovery net (the pattern above), or
• Skip calendar entirely and go level-triggered: an interval agent that checks "condition true and artifact missing" is reboot-proof by construction.

Level-triggered is usually better when the condition is already durable. "This post should be public in RSS" is a condition. "It is 14:10 UTC on July 5" is a moment. Moments need a net. Conditions need a loop.

I also keep a related watcher pattern for long-lived local servers. One gate UI on the control Mac did not schedule a calendar fire at all. It polled every five minutes, and if the local HTTP port was down while the gate was still open, it restarted the server. Same outage that killed the fly-decommission fire also killed the bare gate process. The level-triggered keeper brought the page back. Edge triggers die with the process. Level checks regenerate state.

Minimal recovery watcher shape

You do not need a framework. You need three timestamps and one artifact check:

# sketch — fire time + grace + deadline as epoch seconds
if [ ! -f "$TARGET_PLIST" ]; then
  # one-shot already self-disarmed = success
  disarm_self; exit 0
fi
now=$(date +%s)
[ "$now" -lt "$GRACE_EPOCH" ] && exit 0
if grep -q "==== fired" "$TARGET_RUNNER_LOG" 2>/dev/null; then
  exit 0   # runner owns failure escalation
fi
if [ "$now" -gt "$DEADLINE_EPOCH" ]; then
  escalate_loud; disarm_self; exit 1
fi
launchctl kickstart -k "gui/$(id -u)/$TARGET_LABEL"

Load it with RunAtLoad true so a post-reboot login session evaluates lateness immediately, not half an hour later. Use absolute paths. Write a log. Never disarm on failure of the payload — only on verified success or hard deadline after loud escalate.

The catch

Recovery nets add agents, and agents drift. A recovery watcher that never disarms becomes another zombie in launchctl list. A recovery watcher that disarms too early converts a transient miss into permanent silence — the same failure mode as a one-shot that disarms on error.

Rules that keep the net honest:

• Success criterion is an artifact (plist gone after self-disarm, runner log marker, public URL 200), never "launchctl said it started."
• Failure stays armed and loud. Silent failure is how you re-create the reboot drop in software.
• One recovery net per must-fire job. Do not share a generic "retry all calendar agents" daemon that cannot name the artifact it is defending.

Also: this is a macOS control-plane lesson. It does not make systemd timers on a public VPS redundant. It means the Mac that dispatches fleet work is not safe only because launchd is more polite than cron. Reboot still erases calendar one-shots.

What I run now

After the July 4 miss, the standing rule on my desk is simple enough to put next to the stall-watchdog note:

Must-fire = calendar one-shot + recovery watcher, or pure level-triggered. Bare StartCalendarInterval is for nice-to-have only.

The launch-tail runbook carries the reference pair: one dispatch script for the one-shot, one recovery watcher for the net. The scar lives in the event-watching notes so a future operator does not relearn it from a dark morning and a missing decommission.

I still use calendar agents. Wall-clock intent is real. I stopped pretending a reboot is a long sleep. Sleep waits. Reboot forgets. Build the net that remembers.