Evergreen Watchdog — Crash-Safe Supervisor

Wraps any job or subprocess across crashes, sleep, no-net, reboots. Gates, backoff, singleton lock, bounded logs.

Problem

Long-running scheduled jobs die on crashes, sleep/wake, dead internet, and reboots — and naive retry loops spam empty cycles instead of recovering.

Approach

A Watchdog type wrapping a Rust closure OR a subprocess command. Pre-cycle gates (internet probe with fallback, RAM-pressure pause, custom flags); exponential backoff capped at max, then a 24h operator pause after N failures; singleton lock via racefree-filelock surviving double-fired schedulers; self log rotation. Ships a CLI binary wrapping any command without writing Rust.

Code

From src/watchdog.rs — the job is data, so the supervisor owns it:

/// What the supervisor runs each cycle.
pub enum Target {
    /// A Rust callable that returns an exit code. Boxed so the supervisor
    /// owns it. `&mut self` (via FnMut) lets the closure keep state
    /// across cycles.
    Callable(Box<dyn FnMut() -> i32 + Send>),
    /// A subprocess command. Non-zero exit code counts as failure.
    Command(Vec<String>),
}

Preview

Rust library + CLI (v0.1.0 source-only, install from git — crates.io publish planned). The honest scope table in the README says what it’s not: not systemd, not supervisord, not Kubernetes. Verify via the repo’s gates/backoff/lock-takeover tests.

Results

Exactly one supervisor even when the OS fires it twice; blocked cycles skip and retry; failures back off instead of spamming; disk stays bounded.