Embedded CA & Zero-Config Bootstrap

This guide covers Zester's embedded certificate authority and the zero-config peel bootstrap it enables: initializing the CA, issuing the NATS server certificate, starting the master in embedded mode, distributing the SPKI pin, the peel trust ladder (including TOFU), NATS endpoint discovery, verifying trust at the approval gate, and the CA rotation runbook.

For the general enrollment workflow (approve, reject, revoke, state machine), see the Enrollment Operations Guide.

Why an Embedded CA

Zester requires TLS everywhere: the enrollment HTTPS API refuses to start without a certificate, and only tls:// NATS URLs are accepted fleet-wide. Without an embedded CA, that means bringing your own PKI (OpenSSL ceremony, Vault, cert-manager) before the first peel can enroll — a significant barrier for a fresh cluster.

The embedded CA removes that requirement. A single offline command generates a root + signing-intermediate hierarchy (stdlib crypto/x509, ECDSA P-256 — no external dependencies, no NATS connection needed):

zester ca init --dir /var/lib/zester/auth/ca

This produces:

/var/lib/zester/auth/ca/
  root.crt           # Root CA certificate (long-lived trust anchor)
  root.key           # Root private key
  intermediate.crt   # Signing intermediate
  intermediate.key   # Intermediate private key

and prints the root fingerprint and its SPKI pin (sha256:<hex>) — the one value you distribute to peels. You can reprint it at any time:

zester ca fingerprint     # prints the root SPKI pin (sha256:<hex>)
zester ca print           # certificate details + PEM bundle

With the CA in place, the master self-issues its enrollment HTTPS certificate, peels verify the master against a two-line config (or nothing at all, via TOFU), and NATS endpoints are discovered rather than configured. Everything the enrollment guide describes still applies — the embedded CA only changes how trust is established, not the enrollment state machine.

Bootstrap Sequence

Bringing up a new cluster from zero:

1. Initialize the CA (on the master host)

zester ca init --dir /var/lib/zester/auth/ca

Record the printed SPKI pin — you will put it in every peel.yaml. The default CA directory is /var/lib/zester/auth/ca; the master's ca.dir config must point at the same location if you choose a different one.

1b. Initialize the NATS auth hierarchy (on the master host)

zester ca handles the TLS plane; zester nats-auth init handles the NATS auth plane — the operator/account/user JWTs your daemons authenticate with. It is also offline (no NATS needed):

zester nats-auth init \
  --dir /var/lib/zester/auth \
  --nats-conf /etc/nats/nats-server.conf

This generates the operator/account/system JWTs, account.seed (the fleet trust root — replicate it to every master), master.creds, admin.creds, and a nats-server.conf (operator mode, MEMORY resolver, JetStream) whose tls{} block references the NATS server cert you issue next. Zester does not ship or run nats-server — install NATS yourself and start it with -c /etc/nats/nats-server.conf.

# nats-server.conf
max_control_line: 16384

2. Issue the NATS server certificate

zester ca issue nats-server \
  --dns nats-1.example.com,nats-2.example.com \
  --ip 10.0.1.10 \
  --out /etc/nats/tls

This writes the server certificate as a leaf + intermediate chain plus its key. Install them on the NATS host and configure the NATS server's TLS block to use them. Peels validate the chain against the root they trust, so the served chain must include the intermediate.

An enrollment certificate can be issued explicitly with zester ca issue enroll, but in embedded mode you normally do not need to — the master self-issues it (next step).

3. Start the master in embedded mode

nats_url: tls://nats-1.example.com:4222
nats_advertise_urls:
  - tls://nats-1.example.com:4222
  - tls://nats-2.example.com:4222

ca:
  mode: embedded          # or leave at the default "auto"
  dir: /var/lib/zester/auth/ca

In embedded mode the master self-issues its enrollment HTTPS certificate from the CA and hot-renews it in-process (via GetCertificate) — enroll.tls_cert / enroll.tls_key are unused and no cron or reload is needed for renewal. A ca readiness check on /readyz reports OK, degraded when the certificate is near expiry, or down.

If you already run your own PKI, set ca.mode: external and keep configuring enroll.tls_cert / enroll.tls_key as described in the enrollment guide — the embedded CA is entirely optional.

4. Distribute the SPKI pin to peels

Put the pin from step 1 into each peel's config (see the next section). This is the only secret-free trust artifact that needs to travel — one string, safe to bake into provisioning templates, cloud-init, or AMIs.

5. Enroll peels as usual

Peels start, discover the master, enroll, get approved, and then discover the NATS endpoints automatically. The enrollment approval workflow is unchanged, with one addition: the TRUST column (see Verifying Trust at the Approval Gate).

Zero-Config Peel Configuration

With the embedded CA, a production peel config is two lines:

master_urls: ["https://master-1.example.com:8443"]
enroll_ca_pin:
  - "sha256:9f2a4c...e81b"

Everything else is discovered: the peel verifies the master's enrollment TLS against the pinned root, enrolls, fetches the bootstrap document, learns the fleet's NATS URLs from nats_advertise_urls, caches them locally, and connects. No nats_url, no nats_ca, no certificate files on the peel.

The packaged zester-peel.service runs the peel under zester-watchdog with --bootstrap-cache, so the watchdog follows the peel's discovered NATS endpoints too — self-updates keep working against discovered infrastructure.

If you rely on the convention master (zester resolving in DNS), the config shrinks to one line — just the pin.

The Peel Trust Ladder

When the peel opens a TLS connection to the enrollment API, it establishes trust by walking a strict ladder — the first configured rung wins, and higher rungs are never consulted:

enroll_trust controls whether the last rung exists: tofu (the default) allows first-contact trust; strict removes it, so an unverifiable master is always fatal.

Two properties of the ladder are worth internalizing:

• A configured pin or anchor that mismatches is FATAL — never a downgrade. The peel does not fall through to system trust or TOFU when a strict rung fails. There is no configuration in which an attacker's certificate is "tried against the next rung".
• Pins gate the handshake anchor itself. The pin selects which root is allowed to anchor the TLS chain — the peel never blindly trusts a whole CA bundle served by the master. A poisoned bundle (legitimate CA plus an attacker CA appended) buys the attacker nothing: only the pinned root verifies.

TOFU and the Persisted Anchor

With no enroll_ca, no pin, and no system-trusted certificate, a brand-new peel accepts the master's CA on first contact — the same model as Salt's auto-accepted minion_master.pub or Puppet's first-run cert exchange. On acceptance the peel:

1. Logs the trusted CA fingerprint (sha256:<hex>) at INFO — capture this in provisioning logs if you want an after-the-fact audit trail.
2. Persists the CA certificate to <auth_dir>/enroll-ca.crt as the anchor.

From then on the anchor is rung 3 of the ladder: every future contact must present a chain rooted in that exact CA. TOFU fires only on genuine first contact — a peel that already holds credentials and encounters an unknown authority fails hard, it never re-TOFUs. This is the equivalent of Salt's "The master key has changed" refusal: an infrastructure that silently re-trusted a new CA would hand a MITM every re-connecting node.

Break-glass — when the master's CA legitimately changed outside the rotation procedure (e.g. rebuilt from scratch without restoring /var/lib/zester/auth/ca):

# On the affected peel — deliberate operator action, not automation:
rm /data/auth/enroll-ca.crt
systemctl restart zester-peel

Note that an already-enrolled peel also holds NATS credentials signed by the old deployment; a full master rebuild typically means re-enrollment (see the enrollment guide's disaster-recovery section), not just anchor removal.

The Convention Master: https://zester:8443

When neither master_urls nor master_url is configured, the peel tries https://zester:8443 — the same convention Salt (salt) and Puppet (puppet) use. Point a DNS record (or search-domain entry) named zester at your master and unconfigured peels find it automatically.

This applies to the enrollment path only: a peel that already has credentials never contacts the convention host and never re-enrolls through it.

NATS Endpoint Discovery

What the master advertises

nats_advertise_urls is the list of fleet-facing NATS URLs the master hands to peels. It is validated at startup: every entry must be tls://, and loopback (localhost, 127.0.0.0/8, numeric-loopback forms), unspecified (0.0.0.0), and link-local addresses are rejected. This mandate exists because the master's own NATS view is frequently tls://localhost:4222 (co-located NATS) — a view that must never leak to peels, which would then dial their own loopback and fail confusingly. If a URL is not reachable from a peel's network position, it does not belong in nats_advertise_urls.

The bootstrap document

The master serves the advertised URLs plus the CA bundle in one place:

GET /api/v1/enroll/ca      (unauthenticated, cacheable, relaxed rate limit)

{
  "v": 1,
  "ca_bundle_pem": "-----BEGIN CERTIFICATE-----\n...",
  "fingerprint": "sha256:9f2a4c...e81b",
  "nats_urls": ["tls://nats-1.example.com:4222", "tls://nats-2.example.com:4222"],
  "issued_at": "2026-07-08T10:00:00Z"
}

The same document is republished to the secrets KV bucket under the key _cluster_info; peel JWTs carry a read grant for $KV.secrets._cluster_info, so already-connected peels receive endpoint updates live over their NATS connection, without touching the HTTP API.

How the peel picks its NATS servers

At boot the peel resolves its NATS server list by precedence:

1. Explicit nats_url in config (non-empty) — always wins; discovery is bypassed.
2. Bootstrap cache — <data_dir>/nats-bootstrap.msgpack from a previous discovery. The cache is keyed by an identity hash of master_urls + pins, so repointing the peel at a different cluster invalidates it rather than replaying stale endpoints.
3. Enrollment discovery — fetch the bootstrap document over anchor/pin-verified TLS.
4. Builtin tail — tls://nats:4222, the historical convention default.

Because rung 2 works without any master reachable, discovery is offline-first: a peel rebooting during a full master outage connects to NATS from its cache and enforces from local state, consistent with the rest of the peel's offline-first design.

Two mechanisms keep the endpoint list current on a running peel:

• KV watch: updates to _cluster_info are applied to the live connection as they arrive.
• Recovery loop: if NATS stays unhealthy for more than 5 minutes, the peel re-fetches the bootstrap document over anchor/pin-verified TLS and repoints the live connection via SetServerPool + ForceReconnect (nats.go v1.52.0) — no process restart. This is the path that saves a fleet whose NATS cluster moved to new addresses while peels were disconnected.

Every candidate URL list — freshly fetched, KV-delivered, or cached — passes the same tls:// + no-loopback validator before use. A poisoned or corrupt list cannot point a peel at a plaintext or loopback endpoint.

Verifying Trust at the Approval Gate

TOFU's first-contact window has a second line of defense at approval time. During enrollment the peel signs the SPKI of the CA it decided to trust into its submission (trusted_ca_spki + trust_signature, Ed25519 — domain-separated, challenge-bound, and capability-bound, so a relay MITM cannot strip or replay the fields). The master compares that fingerprint against its own root and flags a TrustMismatch on the record.

zester enroll list shows the result in the TRUST column:

ID                              PEEL ID   HOSTNAME               STATE     TRUST      CREATED
enr-2JFK0003ABCD1234567890      web-03    web-03.prod.internal   pending   ok         2026-07-08 10:00:12
enr-2JFK0006EFGH9876543210      app-01    app-01.prod.internal   pending   MISMATCH!  2026-07-08 10:02:33
enr-2JFK0007IJKL5555555555      db-02     db-02.prod.internal    pending   -          2026-07-08 10:03:01

A MISMATCH! means the peel's first TLS contact terminated at a CA that is not yours — the signature proves it end-to-end, so a relay MITM that forwarded the enrollment to your real master is caught here, at the human approval gate. zester enroll show <id> prints both fingerprints (the peel's trusted SPKI and the master's root) for comparison.

Enforcement:

• zester enroll approve refuses a mismatched record. --force overrides — use it only when you have positively explained the mismatch (e.g. the peel enrolled through a TLS-terminating proxy you own, and you have verified the proxy's CA fingerprint is the one on the record).
• Reactor auto-approval rules refuse mismatched records unconditionally — there is no force path for automation. A mismatch always requires a human.

The All-In-One Box Exception

A single machine running NATS + master + peel cannot be zero-config. The master's nats_advertise_urls must contain fleet-reachable URLs — it may legitimately advertise nothing loopback — so a co-located peel relying on discovery would receive URLs meant for the outside network (or none at all). On an all-in-one box, keep the explicit setting:

master_urls: ["https://localhost:8443"]
enroll_ca_pin:
  - "sha256:9f2a4c...e81b"
nats_url: tls://localhost:4222     # explicit — bypasses discovery (rung 1 of the precedence)

Explicit nats_url is precedence rung 1, so discovery, the cache, and the recovery loop are all bypassed for this peel. Fleet peels on the same cluster remain zero-config.

Multi-Master

The CA replicates to every master as files, exactly like account.seed — never through NATS KV. The CA is upstream of the bus (it signs the NATS server certificate; distributing its keys through NATS would be circular), and the first master needs it before NATS even has a certificate to serve.

• What each master needs in ca.dir: root.crt, intermediate.crt, intermediate.key. The root key may live on one machine or offline — masters load fine without it (leaves are signed by the intermediate); only intermediate rotation needs it.
• Each master self-issues its own enrollment leaf from the shared intermediate (own hostname/ca.enroll_sans SANs) and hot-renews it independently. Leaves differ per master; peels don't care — they anchor the root, so every master verifies against the same pin and persisted anchor, and master_urls failover is transparent.
• Drift is detected, not repaired: every master publishes its CA bundle + fingerprint to _cluster_info; a master seeing a different fingerprint already published warns loudly ("split-brain CA? replicate ONE ca directory to every master"). A peel hitting a diverged master fails TLS verification (closed, never a downgrade) and rotates to the next master URL. Keeping ca.dir in sync is a configuration-management job, like account.seed.

CA Rotation Runbook

Intermediate rotation (routine)

Peels trust the root; the intermediate only signs leaves. Rotating it is invisible to the fleet:

1. Issue a new intermediate from the root, on the machine that holds root.key (replace intermediate.crt/intermediate.key in ca.dir).
2. Multi-master: replicate the two new intermediate files to every master's ca.dir.
3. Re-issue the NATS server certificate (zester ca issue nats-server ...) and drop the new chain on the NATS host.
4. Restart or reload each master, one at a time — each self-issues a fresh enrollment certificate chained through the new intermediate. During the roll, masters on the old and new intermediate coexist: both chains verify against the same root.

No peel-side action. No pin change (the pin is the root SPKI). Anchors remain valid.

Root rollover (rare, two-phase)

The root is the fleet's trust anchor, so rollover uses an overlap bundle — at no point does any peel see a chain it cannot verify:

Phase 1 — introduce the new root alongside the old:

1. Generate the new root hierarchy (new zester ca init into a staging dir).
2. Distribute the overlap bundle (old root + new root concatenated) as the trust source: update peels' nats_ca-equivalent trust material and, where pins are used, stage the new pin into provisioning for new peels. During overlap, chains from either root verify.
3. Re-issue the NATS server certificate from the new CA and drop the new chain files on the NATS hosts. No NATS or peel process restarts are needed: the NATS client's RootCAs callback re-reads the CA file on every reconnect, so the rollover is literally a file drop — peels pick up the new trust on their next reconnect.
4. Switch each master's ca.dir to the new hierarchy, one master at a time — during the overlap window peels verify chains from either root, so masters on the old and new hierarchy coexist (expect the _cluster_info divergence warning while they do; it clears when the last master switches). Each self-issues a new enrollment certificate. Peels' persisted anchors are refreshed through the bootstrap document / _cluster_info path.

Phase 2 — retire the old root:

5. Once the whole fleet has reconnected under the new root (verify via zester peel list — everyone ONLINE — and the ca readiness check), drop the old root from the bundle.
6. Update any remaining enroll_ca_pin values in configuration management to the new pin (zester ca fingerprint on the new CA dir).

Threat Model Summary

The default (TOFU) is deliberate Salt/Puppet parity: zero-friction bootstrap with an auditable, sticky anchor. Shipping the SPKI pin — one string in a two-line config — upgrades every peel to strict verification and closes the first-contact window entirely. There is no configuration in which a verification failure silently degrades to a weaker rung.