Embedded CA Architecture & Security

This document defines the architecture and security model of Zester's embedded certificate authority: the root + signing-intermediate hierarchy generated by zester ca init, the SPKI-pin trust anchoring peels use for verified first contact, the hardened trust-on-first-use (TOFU) fallback, the signed trust binding that surfaces first-contact MITM at the approval gate, and the rotation model. Requirements are stated normatively (CA-*-N), in the same style as the Enrollment Security Specification.

The embedded CA exists to make peel bootstrap zero-config: a peel configured with nothing but master_urls and (ideally) an enroll_ca_pin discovers the fleet CA bundle and NATS endpoints from the master, instead of an operator copying certificates to every node out of band. The CA is the root of the fleet's TLS trust — the enrollment HTTPS listener and the NATS server certificates both chain to it — so its design errs on the side of failing closed everywhere a downgrade would otherwise be silent.

Implementation lives in pkg/ca (hierarchy, pins), pkg/enroll/trust.go (the peel trust ladder), pkg/enroll/verify.go (the signed trust binding), internal/masterd/ca.go (master-side CA manager and self-issued enrollment certificate), and internal/masterd/discovery.go (the bootstrap document).

Hierarchy: Root + Signing Intermediate

zester ca init --dir <dir> generates a two-tier hierarchy into a single directory (default /var/lib/zester/auth/ca; the master resolves ca.dir, defaulting to <auth_dir>/ca):

CA-HIER-1: Peels MUST anchor trust on the root only. Authority.Bundle() returns the root certificate alone; issued leaves carry the full served chain (leaf + intermediate), so a verifier holding only root.crt can build the path. This is the entire reason the hierarchy has two tiers: rotating the intermediate never touches a peel trust anchor. A compromised or expiring intermediate is replaced by signing a new one with the root key — no peel re-configuration, no bundle redistribution, no pin change.

CA-HIER-2: The intermediate MUST be constrained to MaxPathLenZero (a signing-only CA that cannot mint further CAs). A leaked intermediate key can issue server leaves until rotated, but can never extend the hierarchy.

CA-HIER-3: The root key MAY be taken offline after initialization. ca.Load tolerates a missing root.key (root-offline mode): issuance still works — leaves are signed by the intermediate — and only intermediate rotation fails, with a clear error. zester ca print reports Root key: offline in this state.

CA-HIER-4: Certificate serials MUST be random (128-bit, from crypto/rand). Random serials avoid Puppet-style serial-file state and the locking it requires, and keep multi-host issuance collision-free without coordination.

CA-HIER-5: NotBefore on every generated certificate is backdated 24 hours. Freshly provisioned nodes frequently have not completed NTP sync; a certificate issued "now" on the master can be "in the future" on the peel. (Puppet backdates a full day for the same reason.)

Default validities follow the Puppet/Caddy convention — long root, shorter signing tier: root 10 years, intermediate 5 years (DefaultRootValidity / DefaultIntermediateValidity in pkg/ca/ca.go). ca init refuses to overwrite existing CA material (ErrExists) — an existing root is never silently replaced.

Why stdlib crypto/x509

The CA is implemented entirely on Go's standard library (crypto/x509, crypto/ecdsa, ECDSA P-256) — deliberately not on smallstep/certificates or another CA framework:

• The needs are small and fixed. Two CA certs, two server-leaf profiles (nats-server, enroll), SPKI pins. That is a few hundred lines of stdlib code (pkg/ca is ~560 lines including pins); a CA framework brings an order of magnitude more dependency surface for features Zester does not use (ACME, OIDC provisioners, remote signing APIs).
• The trust root of the fleet should have a minimal audit surface. Every dependency in the certificate-issuance path is code an attacker can target; stdlib crypto/x509 is the most heavily reviewed X.509 implementation in the Go ecosystem.
• Offline by construction. The zester ca verbs are pure local file I/O — no NATS, no master, no daemon. CA creation must work before any bus exists (the external NATS server needs its certificate first), which rules out anything that assumes a running service.

Private keys are held behind crypto.Signer, so a future HSM/KMS-backed implementation is an additive change, not a rewrite.

Trust Distribution: the Bootstrap Document

The master serves a JSON bootstrap document at GET /api/v1/enroll/ca — unauthenticated, cacheable (Cache-Control: public, max-age=60), on the relaxed rate-limit budget of the enrollment listener:

{
  "v": 1,
  "ca_bundle_pem": "-----BEGIN CERTIFICATE-----\n...",
  "fingerprint": "sha256:9f2a…",
  "nats_urls": ["tls://nats-1.example.com:4222", "tls://nats-2.example.com:4222"],
  "issued_at": "2026-07-08T09:00:00Z"
}

fingerprint is the root SPKI pin; nats_urls is the validated nats_advertise_urls list from the master config. The same document is republished to the secrets KV bucket under the _cluster_info key (peel JWTs carry a read grant for exactly that key), giving enrolled peels a live refresh channel without touching HTTP.

CA-DISC-1: The document is deliberately served on an idempotent, re-fetchable route — never bundled into the single-use /creds response — so a peel that lost its cache or rotated trust can always re-fetch it. The envelope is additive-only (BootstrapDoc in pkg/enroll/bootstrap.go), matching the fleet-wide wire-evolution policy.

CA-DISC-2: Advertised NATS URLs MUST pass bus.ValidateAdvertisableNATSURLs: tls:// scheme only, and loopback / unspecified / link-local / localhost / 0.0.0.0 / numeric-loopback hosts are rejected with a per-URL warning. A master's own tls://localhost:4222 view of the bus must never leak to the fleet — a peel that cached it would dial itself. Peels apply the same validator to every candidate list they handle (fetched over HTTPS, delivered via the _cluster_info watch, or read back from the on-disk bootstrap cache), so a poisoned or corrupted list can never brick a boot.

CA-DISC-3: The KV republish is idempotent and compare-before-put: content-equal documents (ignoring issued_at) are not re-written, so steady-state multi-master deployments produce no KV churn — and a byte-level mismatch between masters is logged as a divergence warning (see publishClusterInfo in internal/masterd/discovery.go).

Note the layering: the bootstrap document is discovery data, not trust. Nothing in it is believed until the trust ladder below has authenticated the server that produced it (or, for the pin path, until the pinned root inside it has been independently verified).

SPKI-Pin Anchoring

Peels pin the root's SubjectPublicKeyInfo hash, in the RFC 7469 style:

enroll_ca_pin:
  - "sha256:<64 hex chars>"

zester ca init prints the pin at generation time; zester ca fingerprint prints it any time after (one line, for provisioning templates). It is verifiable out of band with standard tooling:

openssl x509 -in root.crt -pubkey -noout \
  | openssl pkey -pubin -outform der \
  | openssl dgst -sha256

CA-PIN-1: The pin MUST cover the SPKI, not the certificate. An SPKI pin survives re-issuing the root certificate with the same key (extended validity, corrected subject), so routine certificate maintenance never invalidates fleet provisioning data. zester ca print additionally shows the certificate-DER fingerprint as a human cross-check value — that one does change on any re-issue, which is exactly what makes it useful for "is this the same file?" comparisons and useless as a pin.

CA-PIN-2 (bundle-poisoning resistance): A pin gates the handshake anchor, never a whole served bundle. When a peel resolves trust via pin, ca.FindPinnedRoot scans the served PEM bundle for a CA certificate whose SPKI matches the pin and puts exactly that certificate — and nothing else — into the TLS root pool. This is the only way trust may be established from an unauthenticated bundle fetch: a bundle containing the real root plus an attacker CA yields only the real root. Trusting the served bundle wholesale would let anyone who can respond on the wire append their own CA to an otherwise-honest bundle and ride along inside the peel's trust store forever.

CA-PIN-3 (no downgrade): A configured pin (or persisted anchor) that fails to match is fatal — ErrTrustPinMismatch — never a fallback to TOFU or system trust. The error class exists precisely so that no code path can catch a pin failure and "helpfully" continue. This holds in every combination: an enroll_ca file whose root contradicts the pin, a persisted anchor that contradicts a later-added pin (the error tells the operator to delete the anchor and re-fetch under the pin), or a served bundle containing no pinned root.

CA-PIN-4: Peel YAML parsing is strict (KnownFields): a typo'd key — enroll_ca_pins, enrol_ca_pin — fails startup loudly instead of being silently ignored and landing the peel in TOFU. A misspelled security knob must never quietly weaken the trust posture it was meant to establish.

The Peel Trust Ladder

enroll.ResolveTrust (pkg/enroll/trust.go) resolves the TLS trust for the enrollment endpoint by walking a strict precedence ladder:

The candidate-bundle fetch used by rungs 2 and 5 is the only InsecureSkipVerify code path in the peel, scoped to exactly GET /api/v1/enroll/ca, and its result is trusted only after pin verification (rung 2) or logged first use (rung 5). Every other request the peel ever makes — enrollment submission, credential download, discovery refresh, the recovery loop — runs over verified TLS (FetchBootstrap takes the resolved, verifying TLS config).

Hardened TOFU

TOFU is the default because it is the Salt/Puppet operating model — salt-key -A accepts a minion key the master has never verified; Zester's TOFU accepts a master CA the peel has never verified, then holds it forever. But the implementation is deliberately narrower than naive TOFU:

CA-TOFU-1 (first contact only): TOFU fires only on genuine first contact — the peel has no credentials yet (TrustConfig.FirstContact). An already-enrolled peel that finds itself without a trust anchor and facing an unknown authority does not re-TOFU; it fails closed with ErrTrustPinMismatch. The reasoning: after enrollment the peel provably knew the real fleet CA once, so "unknown authority" now means either the CA rolled over without following the rotation procedure, or someone is impersonating the master. Both are incidents. Silently re-trusting would hand an attacker a second first-contact window on every re-provisioned node.

CA-TOFU-2 (persistence): A successful TOFU (or pin resolution) persists the trusted root to <auth_dir>/enroll-ca.crt (0600). All subsequent boots verify strictly against that anchor (rung 3) — the peel behaves exactly like a pin-configured peel from then on. This is the direct equivalent of Salt's minion_master.pub: later contact with a different authority produces a hard "the master's CA changed" refusal, never a re-prompt. The break-glass path for a legitimate authority change is explicit and manual: rm <auth_dir>/enroll-ca.crt and restart.

CA-TOFU-3 (loud, actionable logging): The TOFU event is logged at WARN with both the certificate fingerprint and the SPKI pin, plus the instruction to verify against zester ca fingerprint on the master and to set enroll_ca_pin to close the window:

WARN TOFU: trusting enrollment CA on first contact — verify against 'zester ca
     fingerprint' on the master; set enroll_ca_pin to prevent this window
     fingerprint=sha256:41ac… spki_pin=sha256:9f2a…

CA-TOFU-4: enroll_trust: strict disables rung 5 entirely: with no enroll_ca, no pin, and no anchor, the peel relies on the system trust store and fails closed on an unknown authority. Fleets that provision the pin through their config-management pipeline should run strict — TOFU then never executes at all.

The Signed Trust Binding

TOFU leaves one structural gap: on first contact the peel cannot know whether the CA it just trusted is the fleet's. The trust binding moves that judgment to the place that can make it — the master's approval gate — by making the peel sign what it trusted into the enrollment submission.

Alongside the primary enrollment signature (over challenge || curve_public_key, see ENROLL-SIG-2), a current peel submits two additional fields: trusted_ca_spki (the SPKI pin of the CA it anchored on) and trust_signature, an Ed25519 signature over:

challenge(32 bytes) || "zester-enroll-trust-v1\x00" || "bind" || 0x00 || trusted_ca_spki

(see trustSignatureMessage / SignTrustBinding / VerifyTrustBinding in pkg/enroll/verify.go). Each component is load-bearing:

CA-BIND-1 (domain separation): The fixed prefix zester-enroll-trust-v1\x00 ensures the trust signature can never be confused with, or replayed as, the primary enrollment signature (which is challenge || curve_key with no separator). The two proofs are made by the same key over the same challenge; without domain separation, a cross-protocol substitution would be possible in principle.

CA-BIND-2 (challenge binding): The 32-byte challenge is included, so the binding inherits the full anti-replay properties of the enrollment challenge (single-use, 5-minute TTL, bound to the (peel_id, public_key) tuple — ENROLL-REPLAY-1..4). A trust signature captured from one enrollment is worthless in any other.

CA-BIND-3 (capability binding): The fixed marker bind sits inside the signed message. Its presence under the peel's Ed25519 signature makes the peel's claim explicit: "I implement trust binding and this is the CA I trusted." A relay MITM cannot rewrite trusted_ca_spki to the real master's pin (it does not hold the peel's key — any tampering with a present binding fails verification and the submission is rejected with 401, a hard integrity failure, not an advisory flag). Its only remaining move is to strip both fields entirely and pass the submission off as a pre-feature client — which is exactly the residual CA-BIND-5 addresses.

CA-BIND-4 (the approval gate): The master verifies a present binding, then compares the reported SPKI against its own root pin. A mismatch sets TrustMismatch on the enrollment record — deliberately without blocking submission, because the record must exist for an operator to see the flag. The gate is downstream:

• zester enroll list shows a TRUST column: ok (binding matches this master's root), MISMATCH!, or - (no binding — pre-feature peel).
• zester enroll show prints both fingerprints on a mismatch.
• zester enroll approve refuses a mismatched record unless --force is given, with an error explaining the possible first-contact MITM (Store.ApproveForce in pkg/enroll/store.go).
• Reactor auto-approval (enroll.approve reaction actions) always passes force=false — a flagged record can never be auto-approved, no matter what the rule says.

CA-BIND-5 (residual: the full strip): Because the fields are additive (mixed-version fleets must keep working), a relay MITM that strips both fields produces a record indistinguishable from a legacy peel — TRUST: -. This is the honest residual of the design: the strip converts an explicit MISMATCH! into a conspicuous absence, it cannot manufacture an ok. Operationally: on a fleet running current peels, treat - on a new enrollment with the same suspicion as MISMATCH! — a current peel always sends the binding, so its absence means either an old binary or an active relay.

Threat Model

The summary position: TOFU default = Salt/Puppet parity; pin = strictly better. Zester's TOFU is additionally hardened over the naive form by first-contact-only scoping (CA-TOFU-1), anchor persistence (CA-TOFU-2), and the approval-gate binding (CA-BIND-4) — protections Salt's minion_master.pub model does not have an equivalent for.

Multi-Master Operation

CA-HA-1: The CA directory is replicated to every master as files, exactly like account.seed — never through NATS KV. This is deliberate, not an omission:

• The CA is upstream of the bus. It signs the NATS server certificate and the enrollment certificates; distributing its private keys through the system whose trust it establishes would be circular (a NATS compromise could then mint the certificates that secure NATS).
• Private keys do not belong in KV. The secrets bucket carries per-peel-encrypted material; CA keys have no per-recipient encryption story and no reason to transit the bus at all.
• Bootstrap ordering. The first master needs the CA before NATS has a certificate to serve — there is no bus to replicate through yet.

zester ca init prints this as its step 3: "Multi-master: replicate this CA directory to every master, exactly like account.seed." At minimum every master needs root.crt, intermediate.crt, and intermediate.key (each master self-issues its own enrollment leaf); root.key may live on one machine or offline.

CA-HA-2: Each master independently self-issues and hot-renews its own enrollment HTTPS leaf from the shared intermediate (caManager in internal/masterd/ca.go): served via tls.Config.GetCertificate, re-issued at ~2/3 of its validity (ca.enroll_cert_validity, default 90 days), swapped atomically with no listener restart. The enroll.tls_cert / enroll.tls_key files are unused in embedded mode. The ca readiness check reports Degraded when the served leaf enters its renewal window and Down if no leaf is issued — an approaching expiry surfaces on /readyz before it becomes an outage.

CA-HA-3: CA mode is ca.mode: auto | embedded | external (default auto = embedded iff <auth_dir>/ca/root.crt exists). embedded with absent CA material is a fatal startup error (the message names the missing directory and the zester ca init command) — a master that was promised a CA and cannot find one must not fall back to serving operator files that may not exist either. Divergent CA material across masters is surfaced by the cluster-info divergence warning (CA-DISC-3): masters publishing byte-different bundles to _cluster_info warn on every reconciliation attempt.

Rotation

CA-ROT-1 (intermediate: invisible to peels): Peels trust the root (CA-HIER-1). Rotating the intermediate — new key, new certificate signed by the root — changes only what servers present. Roll the NATS server certificate and let each master re-issue its enrollment leaf; no peel configuration, anchor, or pin changes. This is the routine rotation and it is a non-event by design.

CA-ROT-2 (root: two-phase overlap): Root rollover cannot be atomic across a fleet, so it runs as an overlap:

1. Widen trust: distribute a bundle containing both roots (old + new) to the peels' NATS CA path (/data/auth/nats-ca.crt) — a file.managed state does this fleet-wide.
2. Roll servers: re-issue the NATS server certificate (and masters' enrollment material) under the new hierarchy; nats-server --signal reload is hitless.
3. Narrow trust: once no server presents old-root chains, drop the old root from the bundle. Update enroll_ca_pin / persisted anchors as part of the same config rollout.

CA-ROT-3 (rotation is a file drop): The peel's NATS client deliberately does not use nats.RootCAs() (which reads the CA file once, eagerly, at option-build time). It sets RootCAsCB directly (pkg/bus/client.go), so the CA file is re-read on every individual (re)connect attempt. Dropping a new nats-ca.crt on disk takes effect on the next reconnect — no zester-peel restart, no watchdog involvement. The same property makes a not-yet-materialized CA non-fatal: a missing file fails only that connect attempt, and the standing retry loop heals the moment the file appears.

CA-ROT-4 (recovery beats rotation mistakes): If a peel is left behind by a botched rotation (its cached endpoints or trust no longer work), the discovery recovery loop re-fetches the bootstrap document over anchor/pin-verified TLS after the NATS connection has been unhealthy for 5 minutes, revalidates the endpoint list, and repoints the live connection (SetServerPool + ForceReconnect) without a process restart (internal/peeld/bootstrap.go). Note this loop runs over verified TLS — a rotation that invalidated the peel's enrollment anchor is not self-healing (by design, per CA-TOFU-1) and requires the documented break-glass anchor removal.

Requirements Cross-Reference

For operating procedures (initial provisioning, pin distribution, rotation runbooks) see the operations documentation; for the enrollment protocol the CA secures, see the Enrollment Security Specification.