Update Architecture

End-to-End Flow

Control Plane Subjects

Source of truth: pkg/bus/subjects.go.

Watchdog status flows through update-status KV heartbeats, not a dedicated event subject.

Rollout Controller Behavior

Source of truth: pkg/update/rollout.go, wiring in cmd/zester-master/main.go.

• Rollouts are persisted in update-rollouts KV with CAS revision control.
• Node list is resolved from update-status keys.
• Targets currently support:
  • standard targeting expressions (glob, E@ PCRE, L@ list, G@ fact, compound)
  • evaluated against the set of nodes currently reporting in update-status for the component
• Pre-flight eligibility checks run before batching (skipped when the status KV is unavailable; they also apply to dry runs):
  • Degraded exclusion — nodes whose NodeStatus.Degraded is set are excluded, with an Info log listing them (excluding degraded nodes from rollout). If exclusion empties a non-empty target set, the rollout errors with no eligible nodes (N degraded excluded).
  • Min-protocol gating — when the target manifest declares MinProtocol > 0, every remaining node's NodeStatus.Protocol is checked; any node below it refuses the whole rollout with an error naming each incompatible node as node (protocol X < Y). Nodes reporting Protocol 0 (the field was never set — treated as compatible everywhere else) and nodes with no status record fail the check only when MinProtocol > 0; a manifest with MinProtocol 0 never refuses anyone. Watchdogs stamp NodeStatus.Protocol with proto.ProtocolVersion (see pkg/proto/doc.go for the additive-only wire policy that MinProtocol gates).
• Batch execution runs commands in parallel per batch:
  • prepare on batch
  • apply on batch
  • wait soak_time (each node's watchdog independently polls the child's readiness during this window and auto-rolls back on sustained failure — see Watchdog Runtime)
  • confirm on batch — must arrive before the watchdog's confirm-deadline (3× soak time, floored at 5 minutes), or the node auto-rolls back
  • wait batch_pause between batches
• max_failed is global per rollout; once reached, rollout transitions to aborted.
• Abort resets the in-flight batch: every node still staged, applying or soaking this rollout's version receives a rollback (staged nodes just discard the staging slot; soaking nodes restart the previous binary), recorded per node as rolled_back. Already-confirmed nodes from earlier batches are not touched. Abort works from any master: if the rollout is not locally active, the abort is CAS-written to the rollout record and the aborting master performs the node reset itself; the owning driver (if alive) also stops and resets on its next save or abort-poll — a second rollback to an idle node is refused harmlessly. Aborting an already-aborted rollout is idempotent.

Rollout Resume (Driver Failover)

A rollout survives the death of the master driving it:

• The rollout record carries DriverID and DriverHeartbeat (msgpack driver_id / driver_heartbeat, additive). The driving controller refreshes the heartbeat on every batch step and at least every 10s while waiting or soaking.
• Every master runs the resume loop (RolloutController.RunResumeLoop, scan period rollout_resume_interval, default 60s). It scans update-rollouts for non-terminal, non-dry-run rollouts whose heartbeat is older than rollout_stale_after (default 60s; floored at twice the 10s driver heartbeat so a live driver that missed one tick is never adopted) and CAS-adopts each — writing its own driver ID and a fresh heartbeat; losing the CAS race just skips. A zero heartbeat (never written) reads as maximally stale and is adopted immediately.
• Every batch starts with reconciliation: the controller sends a status probe to each node and derives what it still needs — idle/confirmed and already running the target version → skipped (no restart); idle → full prepare → apply → soak → confirm; staged with the target version → resume at apply; soaking with the target version → confirm (immediately when the watchdog reports soak_passed, else after one soak window); a pending version that is NOT the target (a leftover from another attempt) → rollback first, then prepare; transient states (preparing/applying/rolling_back) are re-probed for up to 60s. Only unreachable or stuck nodes count toward max_failed. A resumed (adopted) rollout therefore continues where the dead driver left off instead of re-sending prepare to nodes that would refuse it — the single-master self-update case, where the driver always dies with the old master process, now completes with a confirm from the new process. Adopted aborting records just finish the abort (including the node reset).
• A master shutting down (context cancelled) stops driving without aborting, leaving the record for another master's resume loop. A mid-run persistence failure, by contrast, loudly aborts the rollout.
• Brief double-drive during adoption is safe by design: the watchdog handler rejects duplicate prepare/apply commands, and a superseded driver stops on its next CAS save.

Rollout Persistence Model

Each rollout state tracks:

• config (version, component, target and timing controls)
• deterministic batches
• per-node result map (prepared, applied, confirmed, failed)
• current batch index
• failed count
• start/finish timestamps
• state (created, rolling, completed, aborting, aborted, etc.)
• driver identity and liveness (driver_id, driver_heartbeat) for resume adoption

Scope Notes

• This architecture is for watchdog-mediated binary swaps, not state.apply.
• Rollout targeting only sees nodes currently reporting to update-status — an offline watchdog is invisible to a rollout.