NATS and JetStream · Stellar Control

NATS is the only infrastructure Stellar Control needs: its messaging carries the control plane and the data plane, and JetStream holds every state that is not in Git. The services are stateless; back up and replicate JetStream, and you back up and replicate the MCS.

Requirements#

NATS server 2.12 (the development setup and the Nomad deployment use 2.12.15) with JetStream enabled, on file storage.
Decentralized JWT authentication in production: an operator, one account per cell, and a resolver. The reconciler holds a signing key of the account and issues the JWTs of the drivers, transports, gateways and connectors (see NATS Accounts and Credentials).
TLS on client ports, leaf nodes and routes (see TLS, Encryption and Secrets).
Replication: 1 in development, 3 in production (a NATS cluster of three servers).

Everything the MCS keeps lives under fixed names: nothing is prefixed per tenant, since cells are isolated by NATS accounts.

Subjects#

The control plane is addressed by instance, the data plane by logical role and target. Every variable part is a NATS token: ASCII letters, digits, - and _.

text

stellar.ctl.register                                   registrations (request-reply)
stellar.ctl.hb.<kind>.<instance>                       heartbeats
stellar.ctl.rpc.<kind>.<instance>.<verb>               control verbs (request-reply)
stellar.tc.submit.<target>                             direct telecommands
stellar.tc.encode.<codec>.<target>                     telecommands to encode
stellar.tc.wrap.<transport>.<target>                   units to frame by a transport
stellar.tc.uplink.<gateway>.<target>                   frames to uplink
stellar.tc.evt.<target>.<tc_id>                        telecommand events
stellar.tm.raw.<gateway>.<target>                      raw frames received
stellar.tm.unit.<transport>.<target>                   units unframed by a transport
stellar.tm.decoded.<target>                            raw values decoded by a driver
stellar.param.<target>.<component>.<instance>.<measure>  measure samples
stellar.alarm.evt.<target>.<component>.<instance>.<alarm>  alarm transitions
stellar.alarm.control.<target>                         operator commands on alarms
stellar.file.chunk.<target>.<file_id>                  file chunks
stellar.file.decode.<target>                           files to decode
stellar.stream.<target>.<stream_id>                    continuous stream segments
stellar.sim.evt.<target>                               fault changes of simulations
stellar.metrics.<gateway>.<target>                     throughput reports
stellar.metrics.connector.<name>                       lag of output connectors
stellar.run.submit                                     runs submitted
stellar.run.evt.<run_id>                               run logs
stellar.run.approval.<run_id>                          answers to questions and decisions
stellar.run.control.<run_id>                           suspend, resume, abort

The NATS Contract Reference details the messages.

Streams#

Durable subjects are captured by the streams below (ADR 0001). stellar.ctl.> and stellar.metrics.> stay on core NATS, without a stream. Every stream uses file storage and the old discard policy: the oldest messages go first.

Stream	Subjects	Max age	Dedup window	Created by	Content
`TC_COMMANDS`	`stellar.tc.submit.>`, `stellar.tc.encode.>`, `stellar.tc.wrap.>`, `stellar.tc.uplink.>`	7 d	10 min	executor, API, transfer manager	Telecommands submitted, to encode, to frame and to uplink; `Nats-Msg-Id` = the telecommand ULID (with `-encode`, `-uplink`), so a retry after a failure is deduplicated
`TC_EVENTS`	`stellar.tc.evt.>`	30 d	2 min	executor, API, compute, alarms	Lifecycle of every telecommand, replayable
`TM_RAW`	`stellar.tm.raw.>`	30 d	none	reconciler	Opaque frames as received, re-decodable
`TM_DECODED`	`stellar.tm.decoded.>`	1 d	none	compute	Raw values decoded by the drivers, before calibration
`PARAMS`	`stellar.param.>`	30 d	none	compute, executor, alarms, transfers	Measure samples, real-time and deferred; source of the current values and of the temporal derived measures after a restart
`RUNS`	`stellar.run.submit`, `stellar.run.evt.>`, `stellar.run.approval.>`	365 d	2 min	executor, API, scheduler	Event sourcing of the runs and their answers; source of the reports
`ALARMS`	`stellar.alarm.evt.>`	90 d	2 min	alarms, API	Alarm transitions
`FILE_CHUNKS`	`stellar.file.chunk.>`	7 d	10 min	transfer manager	Chunks received, until the transfer manager stores them
`STREAMS`	`stellar.stream.>`	`archive.streams.max_age`	none	reconciler	Continuous stream segments, never decoded; also bounded by `archive.streams.max_bytes`

Archive of the continuous streams#

The reconciler creates STREAMS with archive.streams.max_age and max_bytes when it starts. JetStream reserves max_bytes on the disk of the server. When the server cannot reserve it (more than its JetStream storage), the reconciler creates the archive bounded by max_age only and logs a warning:

text

archive.streams.max_bytes exceeds the JetStream storage of the server: the archive of the
streams is bounded by archive.streams.max_age only

KV buckets#

Observed and operational state that is read by key lives in KV buckets. Keys are made of tokens joined by .; a single-instance component uses _ as instance. Values are JSON, so that components in any language can read them.

Bucket	Key	Value	Writer
`stellar_config`	`current`	Hash of the current snapshot, publication time and publisher	`stellar compile --publish`
`stellar_instances`	`<kind>.<instance>`	Registration and last heartbeat; TTL of three heartbeat periods	Reconciler
`stellar_leader`	`reconciler`, `compute`, `alarms`, `transfers`	Leader instance and its epoch; TTL `reconciler.leader_ttl`	The candidates, by conditional writes
`stellar_bindings`	`<target>.<link>`	Driver, transport and gateway bound to a link, subjects, revision, epoch	Reconciler leader
`stellar_readiness`	`<target>`	Ready flag, reason per link, configuration revision, epoch	Reconciler leader
`stellar_runs`	`<run_id>`	Resolved run, launcher, executor holding it, last renewal, end	Executor holding the run
`stellar_leases`	`<target>`	Runs holding the target, shared flag, last renewal	Executor
`stellar_values`	`<target>.<component>.<instance>.<measure>`	Latest sample of each measure	Compute stage
`stellar_passes`	`<target>.<pass_id>`	Passes, merged from their sources	API (feeders)
`stellar_pass_gateways`	`<target>.<pass_id>`	Gateway serving the station of a pass, or why none	Reconciler leader
`stellar_schedule`	`<schedule_id>`	Schedules and their state	API and scheduler
`stellar_schedule_rules`	`<rule_id>`	Recurring rules	API
`stellar_alarms`	`<target>.<component>.<instance>.<alarm>`	Alarm state, acknowledgement, shelving	Alarm service
`stellar_transfers`	`<target>.<file_id>.<generation>`, `….up` for an upload	Transfer state and ranges	Transfer manager, executor and API
`stellar_cop1`	`<target>.<link>`	COP-1 FOP state and sequence counters	Transports
`stellar_modes`	`<target>`	Current mode of the target: mode, since, by, run (history 5)	API (operators) and executor (`configure`); read by the reconciler and the executor

Writers that compete for a key (leaders, leases, runs) always write with an expected revision.

Object stores#

Object store	Object name	Content	Writer
`stellar_ir`	hex SHA-256	Compiled snapshots and resolved runs (postcard)	`stellar compile --publish`, API, scheduler (recurring rules)
`stellar_files`	`<target>/<file_id>/<generation>[/<offset>]`	Chunks received and files assembled	Transfer manager; read by drivers and connectors
`stellar_uploads`	hex SHA-256	Contents to upload	API (`POST /v1/uploads`); read by the transfer manager and CFDP entities
`stellar_reports`	`<run_id>`	HTML test reports	Executor

Development server#

The repository of Stellar Control runs a development server with Docker or Podman, with JetStream and decentralized JWT authentication:

Shell

dev/nats/setup.sh               # once: generates identities in dev/nats/secrets/ (git-ignored)
docker compose up -d nats       # or: podman compose up -d nats
dev/nats/smoke-test.sh          # checks JetStream and permissions (needs the nats CLI)

dev/nats/setup.sh runs nsc in the nats-box image and generates:

File	Content
`secrets/creds/dev.creds`	User `dev` of account `stellar`: full access, for developers and tests
`secrets/creds/bootstrap.creds`	User `bootstrap`: may only publish `stellar.ctl.register`, `stellar.ctl.hb.>` and `_INBOX.>`, and subscribe to `_INBOX.>` and `stellar.ctl.rpc.>`
`secrets/stellar-account-signing.nk`	Seed of the signing key of account `stellar`, for the reconciler
`secrets/resolver.conf`	Operator, system account and memory resolver, included by `dev/nats/nats-server.conf`
`secrets/tls/`	A development CA, server and client certificates

setup.sh --force regenerates everything; restart the server afterwards so that it loads the new operator. The server listens on 4222 (clients) and 8222 (monitoring), with JetStream in the nats-data volume. With NATS_CONF=nats-server-tls.conf docker compose up -d nats, it requires TLS: give the clients tls://localhost:4222 and STELLAR_NATS_CA=dev/nats/secrets/tls/ca.pem.

Point a component at it with STELLAR__NATS__CREDENTIALS=dev/nats/secrets/creds/dev.creds.

dev/nats/smoke-test.sh [server-url] checks that the dev user can create a stream, publish and read back, and that the bootstrap user can only register and send heartbeats.

compose.yaml also starts timescaledb, the database of the reference output connector stellar-timescale (user, password and database stellar, port 5432).