Peel Configuration

A peel is a managed node that connects to the NATS server as a client. Configuration is provided via a YAML config file and/or command-line flags. The underlying ClientConfig struct controls connection and resilience behavior.

Configuration File

The peel loads configuration from a YAML file. The default search path is /etc/zester/peel.yaml. Use --config to specify a custom path.

Precedence (lowest to highest):

1. Built-in defaults
2. Config file values
3. Command-line flags

Example Config File

id: web-01
# Only tls:// URLs are accepted — plaintext nats:// is rejected at startup.
nats_url: tls://nats:4222
nats_ca: /data/auth/nats-ca.crt
master_url: https://master:8443
enroll_ca: /data/auth/enroll-ca.crt
states_cache: /data/states-cache
health_addr: "127.0.0.1:9090"

The packaged config installed by the .deb/.rpm (packaging/config/peel.yaml) uses nats_url: "tls://localhost:4222" with nats_ca under /var/lib/zester/auth/.

Config File Reference

Command-Line Flags

Flags explicitly passed on the command line override config file values; default flag values do not. Flags and YAML fields are generated from the same tagged config struct (PeelConfig in internal/config/peel.go), so every flag has a matching YAML field with a shared default; the precedence stays flag > config file > built-in default.

The peel credentials file is loaded from /data/auth/<peel-id>.creds. If no credentials file exists and a master URL is configured (master_urls or master_url), the peel runs the enrollment flow automatically, trying each URL in order.

On startup, the peel syncs state files from the state-files KV bucket to --states-cache and watches for updates. If the cache is empty (e.g., first boot), the peel falls back to baked-in states at /data/states. The effective states directory is re-evaluated before every execution, so a peel that boots before state files reach KV switches to the cache once it fills. See State File Distribution for details.

Offline-First Startup

The peel starts in two phases and never exits just because NATS is unreachable:

1. Local phase — health endpoints, enrollment, local fact collection, provider detection, the states engine (cache or baked fallback), settings warm-start from the on-disk snapshot, the scheduler, and the exec/cancel subscriptions (registered while the client is still connecting; NATS replays them on connect).
2. Connected phase — everything JetStream-dependent: facts publishing, the settings resolver and watchers, state-file cache sync, the basket publisher, and the peel heartbeat. It starts once the NATS connection reports healthy; failures here are retried or degrade gracefully — they never terminate the daemon.

Only genuinely local problems abort startup: a bad config file, no credentials and no master URL to enroll with, an invalid TLS configuration, or unreadable credentials. NATS-dependent failures (facts bucket not yet initialized, resolver errors, state-file sync) are retried in the background.

The zester-peel ready log line appears after the local phase. When NATS is reachable at boot, the peel first waits (up to 60s) for the connected phase to finish so the observable startup sequence matches earlier releases. When NATS is down, the line is preceded by the warning NATS unreachable, starting in offline mode; scheduler and cached state remain active, and the peel enforces from local state — scheduler entries, cached state files, snapshot settings — attaching to the control plane whenever the connection arrives.

On-Disk Runtime State

Offline-first operation relies on two small peel-managed files, both written atomically with mode 0600:

Peel Heartbeat

While connected, the peel writes a liveness record to the peel-heartbeat KV bucket (key <peel-id>, bucket TTL 30s) every 10 seconds, so a peel reads as offline after roughly three missed beats. The heartbeat feeds the ONLINE column of zester peel list and the master's zester_master_connected_peels gauge. Failed heartbeat writes are logged at debug level and retried on the next tick — the peel just looks offline in presence views, nothing breaks. The write is authorized by the peel credential's $KV.peel-heartbeat.<peel-id> grant, issued as part of enrollment.

Execution Queue and Read-Only Fast Path

Mutating executions (state applies, cmd.run, package operations, ...) go through a bounded queue (depth 64) drained by a single worker. When the queue is full, the peel rejects the request immediately: request/reply callers receive the error peel busy: execution queue full, and job dispatches produce a failed return with the same message. Jobs that are queued but not yet running can be cancelled with zester job kill.

Read-only modules bypass the queue and execute concurrently, so liveness probes and data queries keep answering during a long state.highstate. The read-only set is exactly: facts.* (except facts.set, which mutates the custom-facts file and stays on the serialized path), settings.* and their pillar.* aliases, test.ping, grains.*, and sys.list_functions.

Logging

The peel emits structured slog logs with a configurable level and format:

• log_level / --log-level — debug, info, warn, or error (default info)
• log_format / --log-format — json or text (default json)

Invalid values abort startup with an error listing the valid options. Every log line carries the base attributes component=peel and version=<build version>, plus peel_id=<id>.

Health and Metrics Endpoints

The local listener on health_addr (default 127.0.0.1:9090) serves three endpoints:

Readiness checks:

/readyz reports 503 from process start until the NATS connection is established, so it is suitable for systemd/Kubernetes readiness gating and is what the watchdog polls during the update soak phase (--ready-url).

NATS TLS and CA Resolution

The peel refuses to start with a plaintext NATS URL: bus.ValidateTLSNATSURLs rejects anything that is not tls:// before connecting (the master and watchdog enforce the same rule).

The CA certificate used to verify the NATS server is resolved in this order:

1. Explicit --nats-ca flag / nats_ca YAML field
2. NATS_CA_FILE environment variable
3. /data/auth/nats-ca.crt, if the file exists
4. The host's system trust store

If your NATS server certificate is signed by a public or OS-trusted CA, no configuration is needed. For a private CA, either drop the CA cert at /data/auth/nats-ca.crt or point nats_ca at it.

Schedule

The schedule map configures the peel-side scheduler: named entries that run modules at fixed intervals or cron times. Entries can also come from the settings pipeline (hot-reloaded); peel.yaml entries are static.

schedule:
  refresh-facts:
    module: "facts.items"
    interval: "10m"
    splay: "1m"
    return_job: true
  nightly-highstate:
    module: "state.highstate"
    cron: "0 3 * * *"
    maxrunning: 1
    return_job: true

ClientConfig Reference

The ClientConfig struct in pkg/bus/client.go controls the NATS connection:

Full Example

id: web-server-01
nats_url: tls://nats-1.example.com:4222
nats_ca: /data/auth/nats-ca.crt
master_urls:
  - https://master-1.example.com:8443
  - https://master-2.example.com:8443
enroll_ca: /data/auth/enroll-ca.crt
states_cache: /data/states-cache
health_addr: "127.0.0.1:9090"
log_level: info
log_format: json

schedule:
  nightly-highstate:
    module: "state.highstate"
    cron: "0 3 * * *"
    maxrunning: 1
    return_job: true

Authentication

A peel authenticates using a .creds file located at /data/auth/<peel-id>.creds. This path is derived from the peel's id and is not a configurable YAML field.

The credentials file is a standard NATS decorated format containing a JWT (with publish/subscribe permissions) and an nkey seed. It is generated by the enrollment system or provisioned manually.

TLS Configuration

When the master requires mutual TLS (verify_client: true), the peel must provide a client certificate and key.

Connection Resilience

Zester peels are designed to maintain persistent connections to the master. The client handles network interruptions automatically.

Reconnection Behavior

• Unlimited reconnects (max_reconnects: -1): The peel never gives up trying to reconnect. This is the default and recommended setting.
• Jitter: NATS automatically adds random jitter (500ms-5s) to reconnect_wait to prevent thundering herd when many peels reconnect simultaneously.
• Reconnect buffer: Messages published while disconnected are buffered up to reconnect_buf_size (default 8 MB). If the buffer fills, publish calls will return an error.

Health Monitoring

The NATS client sends periodic ping messages to verify the connection is alive:

• Every ping_interval (default 20s), a ping is sent.
• If max_pings_out (default 3) pings go unanswered, the connection is considered dead and reconnection begins.
• The IsHealthy() method reports the current connection state.

Disconnect/Reconnect Notifications

The client exposes channels for monitoring connection state changes:

client, _ := bus.NewClient(cfg)

// Monitor in a goroutine
go func() {
    for {
        select {
        case <-client.DisconnectNotify():
            log.Warn("lost connection to master")
        case <-client.ReconnectNotify():
            log.Info("reconnected to master")
        }
    }
}()

Multiple Server URLs

When multiple URLs are provided, the client connects to the first URL and uses the rest as failover targets:

peel:
  urls:
    - tls://master-01.example.com:4222
    - tls://master-02.example.com:4222
    - tls://master-03.example.com:4222

Shutdown Behavior

Graceful shutdown follows this sequence:

1. Drain() is called on the NATS connection, which:
   • Unsubscribes from all subscriptions.
   • Waits for pending messages to be processed.
   • Flushes any buffered outgoing messages.
2. If drain does not complete within drain_timeout, the connection is force-closed.
3. All internal channels are closed.

ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
client.Shutdown(ctx)

CLI Configuration

The Zester CLI uses a separate configuration structure focused on connecting to the master for administrative operations:

CLI Config File Locations

The CLI searches for configuration in this order:

1. Path specified by --config flag
2. /etc/zester/master.yaml
3. ~/.zester/config.yaml

If no configuration file is found, defaults are used (connect to tls://localhost:4222 without authentication).

CLI Config Example

master:
  urls:
    - tls://master.example.com:4222
  creds_file: /home/admin/.zester/admin.creds
  tls_ca: /etc/zester/tls/ca.crt