Scaling

Resource Sizing

Master Resources

CPU burst occurs during mass state.apply or full fact sync. Steady-state load comes from heartbeats and incremental fact updates.

Peel Resources

Peels are lightweight agents with minimal resource requirements:

Network Bandwidth Estimates

NATS Cluster Topologies

Single Master (Development / Small Deployments)

┌─────────────┐     ┌─────────────┐
│   Master    │────│    NATS     │ ◄── all peels connect here
│  (client)   │     │  (server)   │
└─────────────┘     └─────────────┘

• Suitable for up to ~1,000 peels
• No high availability — master failure means no new jobs
• Peels survive master outage with cached state
• Simplest to operate

3-Node Cluster (Production Standard)

┌──────────┐     ┌──────────┐     ┌──────────┐
│ Master-1 │────│ Master-2 │────│ Master-3 │
│  (RAFT)  │     │  (RAFT)  │     │  (RAFT)  │
└──────────┘     └──────────┘     └──────────┘
     ▲                ▲                ▲
     └── peels distribute connections ─┘

• Tolerates 1 node failure
• RAFT consensus for JetStream data
• Automatic leader election (~2 seconds)
• Recommended for 1,000--10,000 peels

5-Node Cluster (Large Scale)

• Tolerates 2 simultaneous node failures
• Better read distribution across nodes
• Required for 10,000--50,000 peels
• Same topology as 3-node, with two additional nodes

Cluster + Leaf Nodes (Edge / DMZ)

┌────────── Core Cluster ──────────┐
│ ┌────────┐ ┌────────┐ ┌────────┐│
│ │Master-1│ │Master-2│ │Master-3││
│ └────────┘ └────────┘ └────────┘│
└──────────────────────────────────┘
         ▲            ▲
    ┌────┴────┐  ┌────┴────┐
    │  Leaf   │  │  Leaf   │
    │ (DMZ)   │  │ (Edge)  │
    └─────────┘  └─────────┘
      ▲   ▲        ▲   ▲
    peels         peels

Leaf nodes act as local proxies, similar to Salt's syndic but built into NATS:

• Reduce latency for remote/edge peels by providing a local NATS endpoint
• Cross firewall boundaries with a single outbound connection from leaf to core
• Isolate traffic — local peel-to-peel communication stays on the leaf
• Scale horizontally — each leaf handles its own connection pool

Configure a leaf node:

# On the leaf node server
nats:
  listen: "0.0.0.0:4222"
  leaf_node:
    remotes:
      - url: "tls://core-master-01:7422"
        credentials: "/etc/zester/leaf.creds"

Supercluster (Multi-Region / Global)

┌─── Region: US-East ────┐     ┌─── Region: EU-West ────┐
│  ┌──────┐  ┌──────┐    │     │    ┌──────┐  ┌──────┐  │
│  │ M-1  │──│ M-2  │    │     │    │ M-1  │──│ M-2  │  │
│  └──────┘  └──────┘    │     │    └──────┘  └──────┘  │
│       ▲                 │     │         ▲               │
│     peels               │     │       peels             │
└─────────┬───────────────┘     └─────────┬───────────────┘
          │       NATS Gateway            │
          └───────────────────────────────┘

NATS superclusters connect multiple independent clusters via gateways:

• Each region operates independently if gateways fail
• Cross-region job dispatch and fact sharing via gateway routing
• JetStream data replication across regions (configurable)
• Replaces Salt's complex syndic chains

Performance Tuning

Server Configuration Parameters

These parameters control the NATS server behavior (configured on the external NATS server):

Client Configuration Parameters

These parameters in ClientConfig control peel-to-master connections:

NATS Tuning for Scale

For deployments above 10,000 peels, tune the NATS server configuration:

nats:
  max_connections: 20000
  max_payload: "8MB"
  write_deadline: "10s"
  jetstream:
    max_file_store: "500GB"
    max_mem_store: "4GB"
    limits:
      max_ack_pending: 50000
      duplicate_window: "600s"

KV Bucket Tuning

Default bucket configurations from the codebase:

To update bucket replicas for a clustered deployment:

# Each KV bucket is a stream prefixed with KV_
nats stream edit KV_facts --replicas 3
nats stream edit KV_jobs --replicas 3
nats stream edit KV_job-returns --replicas 3
nats stream edit job-events --replicas 3

Operating System Tuning

For masters handling 10,000+ connections:

# /etc/sysctl.d/99-zester.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.core.netdev_max_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1

# File descriptor limits (also set in systemd unit)
fs.file-max = 2097152

# Apply without reboot
sysctl --system

Scaling Checklist

When scaling to a new tier, work through this checklist:

Moving to 1,000+ Peels

• Deploy 3-node NATS cluster for high availability
• Set KV bucket replicas to 3
• Configure Prometheus monitoring and alerting
• Set up automated JetStream backups
• Increase LimitNOFILE in systemd unit to 65536

Moving to 10,000+ Peels

• Deploy 5-node NATS cluster
• Switch to NVMe storage for JetStream
• Tune max_connections, max_payload, and write_deadline
• Apply OS-level tuning (sysctl, file descriptors)
• Use Prometheus service discovery for peel scraping
• Consider leaf nodes for remote sites

Moving to 50,000+ Peels

• Deploy leaf nodes for edge and DMZ networks
• Increase ReconnectWait on peels to 5s to reduce thundering herd
• Increase JetStream memory allocation to 16+ GB
• Monitor and tune max_ack_pending for job dispatch throughput
• Consider target slicing/run partitioning to limit concurrent fan-out

Moving to 100,000+ Peels

• Deploy NATS supercluster for multi-region
• Configure JetStream domain isolation per region
• Increase ReconnectBufSize on peels for longer outage tolerance
• Implement graduated rollout for mass state applies (partitioned target sets)
• Dedicated monitoring infrastructure for Zester metrics