Upgrading Zester
Two upgrade channels — the self-update plane for running binaries, packages/config through apt or the pkg module — and the rule that a node must never restart its own service from a job.
Zester has two independent upgrade channels, and keeping them straight is what makes "zester manages zester" safe.
| Channel | Upgrades | How | When |
|---|---|---|---|
| Self-update plane | The running peel/master binary | zester update rollout (soak + auto-rollback), coordinated by the watchdog | The normal way to move the fleet to a new version |
| Packages & config | The dpkg-recorded version, config files, on-disk /usr/local/bin binary | apt / the .deb, or zester's own pkg module (pkg.latest, pkg.installed) | Aligning package metadata and config; provisioning |
The doctrine
1. Upgrade the running binary on the self-update plane — not with apt. A rollout stages the binary, soaks it against readiness, and auto-rolls-back on failure. The watchdog owns the running binary via its slot manager; that is the supported path to change what's actually executing.
2. Align packages and config through the pkg module (or apt) — a package upgrade must never restart the service. The .deb prerm only stops/disables the unit on a real removal (prerm remove), never on an upgrade (prerm upgrade); postinst only ever starts (a no-op if already running), never restarts. So apt upgrade zester-peel — or pkg.latest zester-peel run through zester — updates the dpkg record and drops the new on-disk binary without touching the running process.
3. A node must never synchronously restart its own service from inside a job. If a job stops or restarts the very service the node runs under, it kills its own cgroup mid-job: the watchdog, the agent, and the apt/dpkg it spawned all die together, the job return is lost, and dpkg is left half-configured. If you must restart a node's own service, do it detached (e.g. systemd-run --on-active=2s systemctl restart zester-peel) or let the self-update plane perform the swap-and-restart for you.
Learned the hard way
Running pkg.latest zester-peel through zester used to make the peel kill itself: the old package's prerm stopped the service unconditionally, and because apt was a child of the peel's own cgroup, stopping the unit took down watchdog + peel + apt + dpkg mid-unpack. Fixed by gating prerm on remove — but the rule stands: don't synchronously restart your own service from a job.
Release promotion and auto-rollout
Promotion turns a published version into the release, in one command:
zester update publish zester-peel --component peel --version 0.5.0 # TTL 30d by default
zester update promote --component peel --version 0.5.0A promoted version never expires, and every master with auto-rollout
enabled (the default) automatically starts a rollout to the latest promoted
version whenever live nodes lag behind it — using the exact same machinery as
a manual update rollout: batches, per-node soak with auto-rollback, the
failure budget, degraded-node exclusion. Auto-rollout never downgrades (nodes
at or ahead of the promoted version are untouched), runs at most one rollout
per component at a time, and never retries a rollout an operator aborted.
A completed run does not freeze convergence: nodes that were offline during
it (or enrolled later) are picked up by a follow-up generation targeting
just the laggards.
The fleet-wide switch lives in NATS — flip it at runtime, no master restart:
zester update auto status # ON by default (nothing rolls until you promote)
zester update auto off # freeze auto-rollouts fleet-wide
zester update auto onMasters additionally honor a local update_auto_rollout config knob (plus
update_auto_components, update_auto_batch_size, update_auto_soak_time,
update_auto_max_failed, update_auto_interval for tuning).
Version lifetimes
Expiry is per-version, carried in the manifest (the object-store bucket has no TTL anymore); a master-side GC reaps expired versions hourly, always skipping promoted versions and anything an in-flight rollout still needs:
zester update publish ... --ttl 720h # explicit lifetime (0 = never)
zester update set-ttl --component peel --version 0.4.9 --ttl 168h
zester update demote --component peel --version 0.4.9 --ttl 24h # promotion off, expiry resumes
zester update versions --component peel # PROMOTED + EXPIRES columns
zester update unpublish --component peel --version 0.4.8 # immediate removal
zester update rollouts # rollout history with idsunpublish is refused while a live rollout references the version and needs
--force for promoted versions. Node-side rollback never re-downloads —
nodes revert from their local previous-binary slot — so removing old
versions cannot break rollbacks.
Managing zester with the pkg module
Once the prerm fix is in place, aligning packages fleet-wide is safe from the master:
# Refresh indexes, then bring packages to the latest published version.
zester '*' cmd.run 'apt-get update'
zester 'role:peel' pkg.latest zester-watchdog
zester 'role:peel' pkg.latest zester-peel # does NOT restart the running peel
zester '*' cmd.run 'dpkg -l zester-peel zester-watchdog zester | grep ^ii'The apt provider runs fully non-interactively — DEBIAN_FRONTEND=noninteractive plus --force-confdef --force-confold — so a modified conffile (an operator-edited /etc/zester/peel.yaml, say) resolves to your edited version without prompting. Without this, a conffile prompt would hang the peel's serialized exec worker and block every mutating job behind it.
Channel mixing: keep versions aligned
The running binary (managed by the self-update slots) and the dpkg-recorded version are independent. After a rollout, the running binary can be newer than what dpkg records; a later apt upgrade zester-peel then overwrites the slot-managed on-disk binary with the package's. This is fine as long as you know it — prefer one channel per concern: rollouts to change what runs, packages to align metadata/config, and keep them at the same version so a package upgrade is a no-op for the running process.
Deployment
This guide covers installing Zester binaries, configuring master and peel services, running in containers, and performing upgrades.
Monitoring
Both daemons serve three local HTTP endpoints — /healthz (liveness), /readyz (readiness), and /metrics (Prometheus) — and log structured JSON via Go's log/slog package.