Monitoring

Health Check Endpoints

Both zester-peel and zester-master serve a plain-HTTP listener on a loopback address, started before anything else during daemon startup:

The defaults are distinct so a peel and a master can colocate on the same host. Each listener serves three endpoints: /healthz, /readyz, and /metrics.

/healthz — Liveness

Always returns 200 while the process is up and serving HTTP, regardless of NATS or JetStream state:

$ curl -s http://127.0.0.1:9090/healthz
{"status":"ok","component":"peel","version":"v0.5.0"}

$ curl -s http://127.0.0.1:9091/healthz
{"status":"ok","component":"master","version":"v0.5.0"}

The version field is parsed by the watchdog for fleet status reporting — keep it if you front the endpoint with a proxy.

/readyz — Readiness

Runs named per-subsystem checks (in parallel, 2s timeout per check) and reports one of three statuses per check:

The endpoint returns 503 only when at least one check is down — degraded stays 200 so load balancers don't drop the node and the watchdog's update soak doesn't roll back healthy binaries over non-fatal conditions. The JSON body always carries per-check status, message, and latency_ms for operators and monitoring.

Peel checks:

Master checks:

$ curl -s http://127.0.0.1:9090/readyz | jq .
{
  "status": "ok",
  "checks": {
    "nats": {"status": "ok", "latency_ms": 1},
    "kv": {"status": "ok", "latency_ms": 3}
  },
  "version": "v0.5.0",
  "uptime": "2h3m0s"
}

Watchdog Health Monitoring

zester-watchdog uses both endpoints, for different decisions:

• --health-url (default http://127.0.0.1:9090/healthz) — liveness. Drives restart monitoring and the post-apply WaitForHealthy gate. Must match the child's --health-addr — the packaged systemd units pair them accordingly (:9090 for zester-peel.service, :9091 for zester-master.service).
• --ready-url (default: derived from --health-url by replacing the path with /readyz) — readiness. Polled only during the update soak phase, so an alive-but-functionally-dead child fails soak and auto-rolls back.

Related watchdog flags: --health-timeout (default 5s), --health-interval (default 10s), and --health-retries (default 3 — consecutive failures before a post-update rollback). See Watchdog Runtime for the full flag list and soak policy.

Peel Presence (Heartbeats)

The health endpoints answer "is this process alive?" — fleet-wide peel presence comes from the peel heartbeat bucket. Every connected peel rewrites its own key in the peel-heartbeat KV bucket on a fixed cadence:

Key presence is the liveness signal: a peel is considered offline after ~3 missed beats, when the TTL expires its key. Two consumers read the bucket:

• zester peel list derives its ONLINE and LAST-SEEN columns from the heartbeat keys (fleets without the bucket show - in both columns).
• The master counts the bucket's keys every 15 seconds to feed the zester_connected_peels gauge.

Heartbeat writes are non-fatal on the peel (Debug-logged, retried on the next tick). A peel whose credentials lack the heartbeat KV grant cannot write the bucket — it runs fine but looks offline in presence views.

Prometheus Metrics

Both daemons serve Prometheus metrics (OpenMetrics format) at /metrics on the same health_addr listener as the health endpoints.

Master metrics

Peel metrics

Both registries also expose the standard Go runtime, process, and build-info collectors (go_*, process_*, go_build_info).

Scrape Configuration

The metrics listener binds to health_addr, which defaults to loopback. To scrape from a central Prometheus, either run a node-local agent (Prometheus agent mode, Grafana Alloy, etc.) or set health_addr to a routable interface — if you do the latter, firewall the port, as the endpoints are unauthenticated.

scrape_configs:
  - job_name: 'zester-master'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['master-host:9091']

  - job_name: 'zester-peel'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['web-01:9090', 'web-02:9090']

NATS Server Monitoring

Complement Zester-native metrics with the NATS server's built-in monitoring endpoints for broker-side visibility:

# Server status
curl -s http://nats-host:8222/varz | jq .

# Connected clients (peels + master)
curl -s http://nats-host:8222/connz | jq '.num_connections'

# JetStream status (storage, streams, consumers)
curl -s http://nats-host:8222/jsz | jq .

# Cluster routes
curl -s http://nats-host:8222/routez | jq .

Add the NATS monitoring endpoint to your prometheus.yml:

scrape_configs:
  - job_name: 'nats'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['nats-host:8222']

Structured Logging

Zester uses Go's log/slog package for structured logging. All daemons log JSON by default; level and format are configurable:

The flags exist on zester-master, zester-peel, and zester-watchdog (the watchdog has no YAML config — flags only). Invalid values fail startup with an error listing the valid options.

Every log line carries base attributes: component (master, peel, or watchdog) and version (build version). The master additionally attaches master_id (its KSUID) and the peel attaches peel_id once known — so a watchdog and its wrapped child can share one stdout stream and still be told apart.

Log Format

{
  "time": "2026-07-01T14:30:00.123Z",
  "level": "INFO",
  "msg": "received exec request",
  "component": "peel",
  "version": "v0.5.0",
  "peel_id": "web-01",
  "jid": "2oHfKnCPMQnLEYQeBQsNtUiJp3r",
  "module": "state.apply"
}

Log Levels

Multi-Master Log Lines

In multi-master deployments, KV publishing (settings files, state files, GitFS sync) and per-peel secrets encryption are gated by advisory leader leases in the leases bucket (publisher and facts-secrets). Standby masters are healthy while logging only the candidate lines — do not mistake them for a stuck startup:

Log Aggregation

Ship logs to your log aggregation platform via journald:

# View logs
journalctl -u zester-master -f --output json

# Export to a collector
journalctl -u zester-master -f --output json | your-log-shipper

Use tools like Filebeat, Fluentd, Promtail, or Vector to ship log files to Elasticsearch, Loki, Splunk, or your platform of choice.

Useful Log Queries

When using a log aggregation system, these queries help with operations:

Alerting Recommendations

Base alerts on Zester-native metrics where available, plus NATS server metrics (at http://<nats-host>:8222/metrics) for broker health:

Critical Alerts

Warning Alerts

Dashboard Recommendations

Dashboard 1: Zester Fleet

• Connected peel count (zester_connected_peels on the master, fed from peel heartbeats)
• Peel connectivity (zester_peel_connected across the fleet)
• Job throughput and final status breakdown (zester_jobs_total by status)
• Job duration percentiles (zester_job_duration_seconds by function)
• Module execution rate and error ratio (zester_peel_state_apply_total by result)
• Job reclaims (zester_job_reclaims_total)

Dashboard 2: NATS Infrastructure

• Connection counts (total clients = master + peels)
• Messages per second (published + received)
• Bytes per second throughput
• JetStream storage utilization
• Slow consumer events
• Reconnection events (zester_nats_reconnects_total / zester_nats_disconnects_total)