NATS JetStream Deep Dive

NATS JetStream is the unified messaging, persistence, and KV layer that underpins every Zester operation. This document covers the rationale for choosing NATS, the subject hierarchy, KV bucket design, JetStream streams, and deployment topologies from single-node to multi-region superclusters.

Why NATS

Zester evaluated several messaging technologies before selecting NATS JetStream as its backbone. The decision was driven by the need for a single technology that could serve as both the real-time message bus and the persistent data store, eliminating the operational complexity of running separate messaging and database systems.

Alternatives Considered

Why Not ZeroMQ

ZeroMQ (used by SaltStack) is a powerful messaging library but lacks persistence, clustering, and authentication. Salt compensates by layering its own PKI, job cache (returners), and failover logic on top. Zester eliminates this complexity by using NATS, which provides all of these features natively.

Why Not Redis

Redis Streams provide persistence and pub/sub, but running a separate Redis cluster adds operational overhead and introduces a second failure domain. Redis ACLs operate at the command level, not the subject/topic level, making fine-grained per-peel authorization harder to implement.

Why NATS Wins

1. Single dependency -- NATS serves as message bus, KV store, and event stream. No external database needed.
2. Standalone server -- NATS runs as its own process with dedicated resources, simplifying operations and enabling independent scaling.
3. Native Go -- Written in Go, with first-class Go client libraries. No CGo, no FFI.
4. Subject-level authorization -- JWT permissions control exactly which subjects each peel can publish to or subscribe from.
5. Built-in clustering -- RAFT-based consensus for high availability without external coordination (no ZooKeeper, no etcd).
6. Leaf nodes -- Edge deployment pattern where remote sites connect through a leaf node, enabling WAN-optimized topologies.

Subject Hierarchy

Zester organizes all NATS communication under a structured subject hierarchy rooted at zester.. Every subject follows a consistent pattern that maps to the operation type and target identity.

Subject Prefix Tree

Subject Reference Table

Queue Groups

Zester uses NATS queue groups to distribute job dispatch across multiple master instances in an HA deployment. When masters subscribe to zester.dispatch with queue group zester.masters, NATS delivers each incoming request to exactly one member of the group using round-robin delivery.

nc.QueueSubscribe("zester.dispatch", "zester.masters", handler)

Queue groups provide automatic load balancing and failover with no external coordination. When a master disconnects, NATS removes it from the group and redistributes to surviving members. When a new master starts, it automatically joins by subscribing with the same group name.

Wildcard Patterns

NATS supports two wildcard tokens that Zester uses extensively:

Helper Functions

The pkg/bus/subjects.go file provides type-safe helper functions for constructing subjects, preventing string formatting errors:

Subject Permissions (JWT)

Peel user JWTs (auth.PeelUserJWTOptions in pkg/auth/jwt.go) are least-privilege: a peel can only publish on its own peel-scoped subjects (zester.event.<peel-id>.>, zester.fact.<peel-id>, zester.job.*.ack.<peel-id>, zester.job.*.return.<peel-id>, zester.job.*.schedule.<peel-id>) plus the JetStream API subjects for the KV buckets it needs. Three grants support the peel-side services:

The admin subjects (zester.admin.enroll.*) are served by the masters with full-permission master credentials; operator NATS credentials must be allowed to publish on them for zester enroll approve|reject|revoke to work. Peel JWTs deliberately have no grant for zester.admin.enroll.*, and no $KV.jobs / $KV.job-returns grants — a compromised peel cannot administer enrollments or tamper with other peels' job records.

KV Buckets

Zester uses NATS JetStream KV buckets as its primary persistence layer. All buckets are created during master startup via InitializeStorageOpts() in pkg/bus/kv.go (the InitializeStorage() convenience wrapper delegates to it).

Bucket Reference

Replica Tiering

Replica counts scale automatically with the NATS cluster: bus.EffectiveReplicas(explicit, clusterSize) returns an explicit operator-set count when > 0, otherwise min(3, max(1, clusterSize)) — capped at 3, the JetStream RAFT sweet spot (R5 buys little for KV workloads). InitializeStorageOpts(ctx, js, StorageOptions{Replicas, ClusterSize}) applies this to every bucket, stream, and Object Store.

Every asset — critical and ephemeral alike — gets the effective replica count; heartbeat and lease buckets must survive a single node loss for liveness detection to keep working. The DurabilityTier (TierCritical / TierEphemeral on BucketConfig.Tier) controls only warning behavior:

• A critical asset whose effective replica count is below min(3, ClusterSize) (an operator explicitly forced a low count in a cluster) logs a loud warning per asset — a single-node disk failure then permanently loses that data.
• A failed replica upgrade on an existing asset (older servers, insufficient cluster resources) never aborts startup: the asset is re-created at 1 replica with a warning that includes the manual migration hint (nats stream edit KV_<bucket> --replicas=N).

Bucket Lifecycle

KV Operations

All KV operations use MessagePack serialization transparently:

The bus.KV Interface Boundary

Domain packages never touch nats.go KV types directly. The bus.JetStreamAPI interface's KV methods return bus.KV — a bus-owned narrow interface (Get/Put/Create/Update/Delete/Keys/ListKeys/ListKeysFiltered/Watch/WatchAll/WatchFiltered) with bus-owned companion types: KVEntry (Key/Value/Revision/Operation), KeyWatcher (nil-sentinel end-of-replay; the channel closes on consumer loss), KeyLister, and the op constants KVOpPut/KVOpDelete/KVOpPurge.

At the production boundary, Client.JetStream() returns *bus.JS, an adapter over the raw jetstream.JetStream that implements JetStreamAPI, ConsumerAPI, and ObjectStoreAPI (Unwrap() exposes the raw context; bus.NewJS wraps a self-built one). The raw jetstream.KeyValue is adapted exactly once, in pkg/bus/kv_jetstream.go. The error sentinels bus.ErrKeyNotFound, bus.ErrKeyExists, and bus.ErrNoKeysFound are aliases of the jetstream sentinels, so errors.Is works with either form. Tests use bustest.NewFakeJS() / bustest.NewFakeKV() against the same interfaces.

Object Store Buckets

Zester currently defines one default Object Store bucket:

JetStream Streams

In addition to KV buckets (which are themselves built on streams), Zester defines explicit JetStream streams for event logging.

job-events Stream

The job-events stream captures messages published under zester.job.> (for example dispatch, ack, return, status, cancel, and scheduler schedule results). This provides:

• Audit trail -- Complete history of who ran what, when, and what happened.
• Replay capability -- Consumers can replay the stream from any point to reconstruct job state. The master relies on this in two places: the shared durable schedule-results consumer persists scheduler return_job results, and job reclaim replays zester.job.<jid>.return.> via an ephemeral pull consumer so returns published while no watcher was running are recovered.
• Debugging -- Inspect the exact sequence of events for any job.

events Stream

The events stream captures every publish under zester.event.> — peel event.send events, beacons, operator events, and master-synthesized events — durably. The masters' shared durable consumer reactor (DeliverNew, explicit ack, AckWait 60s, MaxDeliver 5) delivers each event to exactly one master fleet-wide and replays events published while all masters were down. The size caps bound flood damage from a compromised peel; the duplicate window collapses redelivered reactor-derived events published with deterministic message IDs. See Reactor architecture.

Deployment Topologies

Zester supports multiple deployment topologies, scaling from development to multi-region production.

Single Node (Development / Small Scale)

The simplest deployment: one NATS server with the master connecting as a client.

Characteristics:

• NATS runs as a separate process (or container) alongside the master
• No high availability -- single NATS failure = total outage
• Peels continue running last applied state during outage
• Suitable for development, testing, and small deployments (< 100 peels)
• JetStream data persisted to local disk (survives process restarts)

3-Node NATS Cluster (Production)

For production high availability, deploy a 3-node NATS cluster with RAFT consensus.

Characteristics:

• Tolerates 1 node failure (quorum = 2 of 3)
• Committed writes are never lost (RAFT guarantee)
• KV bucket and stream replicas = 3
• Election on leader failure is sub-second
• Master connects to cluster as an external client
• Peels receive connection URLs for all 3 nodes; NATS client handles failover

5-Node NATS Cluster (High Availability)

For environments requiring tolerance of 2 simultaneous failures:

• Quorum = 3 of 5; tolerates 2 failures
• Higher write latency (must replicate to 3 nodes)
• Recommended for critical infrastructure with SLA requirements

Superclusters (Multi-Region)

For multi-region deployments, NATS superclusters use gateways to connect independent clusters:

Characteristics:

• Each region has an independent RAFT cluster
• Gateways route interest-based messages between regions
• JetStream domains isolate per-region data (JetStreamDomain in ServerConfig)
• Cross-region peels connect through their local cluster
• Provides geographic redundancy and data locality

Leaf Nodes (Edge / WAN)

For edge deployments or sites behind restrictive firewalls, NATS leaf nodes provide a WAN-optimized connection:

Characteristics:

• Leaf nodes multiplex all local peel connections into a single WAN connection to the hub cluster
• Reduces WAN bandwidth and connection count
• Peels connect to the local leaf node over LAN (low latency)
• Leaf nodes can operate in a degraded mode during WAN outages (local messaging continues)
• Subject mapping can remap subjects at the leaf boundary for namespace isolation

Network Partition Behavior

Understanding NATS RAFT behavior during network partitions is critical for production operations.

Partition Scenarios (3-Node Cluster)

One follower dies: Leader continues. Quorum maintained (2/3). Writes succeed. Dead node catches up on rejoin via log replay.

Leader dies: Remaining 2 nodes detect missing heartbeats. New leader elected (sub-second). Writes pause during election. No committed data lost.

Network split (1 vs 2): Minority partition (old leader) cannot commit writes. Majority partition elects new leader. Old leader steps down when it discovers higher term. Uncommitted entries on old leader are discarded.

Peel Behavior During Partitions