Nomad

deployment/nomad in the repository of Stellar Control runs the whole stack on Nomad: NATS with JetStream, the services of the core, the web editor, the simulated targets of the example configuration (sim-1 and psu-sim-1) and, optionally, a Traefik reverse proxy. Every component is a plain process (raw_exec driver): no container runtime is needed.

As shipped, it targets a single development host. The same jobs serve a shared platform, one cell per tenant, provisioned by stellar-admin; a production platform runs its cells in containers instead, behind Traefik.

Files#

Deploy#

deployment/nomad/deploy.sh            # cargo build --release, agent, nats, stellar
deployment/nomad/deploy.sh --no-build # binaries already built

Then:

nomad job status stellar                     # NOMAD_ADDR=http://127.0.0.1:4646, UI at /ui
nomad alloc logs -job stellar -task executor # logs of a component
stellar watch sim-1 --api http://127.0.0.1:8080
stellar send sim-1 'tcu[TCU1].ping' --api http://127.0.0.1:8080
stellar run examples/config/runs/hot-standby.yaml --api http://127.0.0.1:8080

The web console is at http://127.0.0.1:8080 when core/frontend is built.

Stop with deployment/nomad/stop.sh; add --purge to start again from an empty JetStream.

Variables of deploy.sh#

NATS_PORT=14222 NATS_MONITORING_PORT=18222 API_PORT=18080 deployment/nomad/deploy.sh

How it works#

• Configuration. Before the reconciler starts, the prestart task publish-config runs stellar compile examples/config --publish <nats>: every restart of the reconciler group publishes the repository as it is. After a change, restart it (nomad alloc restart <reconciler alloc>) or run the job again.
• Order. Nomad does not order jobs: deploy.sh waits for NATS before running stellar. A component that cannot reach NATS retries its connection; a failed task restarts every 5 s.
• Web application. The API serves core/frontend/dist (api.web_dir) when it is built.
• Editor. stellar-editor listens on 127.0.0.1:8090; the API relays /v1/editor to it (api.editor_url). Its remote is by default a bare mirror of the repository, $STELLAR_NOMAD_DATA/config.git, whose main follows the repository at each deployment and where the draft branches are pushed. Its clone and working copies live in $STELLAR_NOMAD_DATA/editor. The forge comes from editor.forge and STELLAR_FORGE_TOKEN.
• Metrics. Each component serves /metrics on its own dynamic port (STELLAR__OBSERVABILITY__METRICS_ADDR), shown by nomad alloc status.
• Security. As shipped, development only: NATS without authentication, no signing key (no JWT issued), declared identities accepted, everything on the loopback interface.

Variables of the jobs#

nats

stellar

proxy

Scaling#

Every group scales from 0 to its maximum with nomad job scale stellar <group> <n>; the initial counts come from the counts variable.

At 0, a component stops: api at 0 closes the web application and the CLI. See Running the Services.

A small cell (all_in_one=true) runs stellar-mcs in the group mcs beside its simulators: about 55 MiB at rest instead of 210 MiB for the seven separate services (debug builds).

TLS#

NATS_TLS=1 deployment/nomad/deploy.sh      # clients: tls://127.0.0.1:4222 and STELLAR_NATS_CA
NATS_TLS=mtls deployment/nomad/deploy.sh   # and client certificates

The development certificates come from dev/nats/tls.sh, generated once in the data directory. A real deployment uses the certificates of its own CA: tls_dir and tls_verify for the nats job, nats_ca, nats_cert and nats_key for the stellar job, and leafnode_port, websocket_port for the benches.

Reverse proxy#

With PROXY=1, Traefik serves the API, the web application and their WebSockets over HTTPS. The CLI checks its certificate with STELLAR_API_CA:

PROXY=1 deployment/nomad/deploy.sh
STELLAR_API_CA=deployment/nomad/.data/tls/ca.pem \
  stellar send sim-1 'tcu[TCU1].ping' --api https://localhost:8443

The proxy routes each host to the API of its cell from the Nomad variables nomad/jobs/proxy/cells/<cell> (host, backend): adding, changing or deleting one updates the routes within seconds, without redeploying the proxy. In production it listens on 80 and 443 and gets its certificates from Let's Encrypt (acme_email); with Nomad ACLs, the task needs a policy reading nomad/jobs/proxy/cells/*.

Secrets and encryption at rest#

No secret is written in a job or in the repository: each lives in a Nomad variable (encrypted at rest by Nomad, readable by the tasks of its job only) or with the operator of the platform.

With NATS_ENCRYPT=1, deploy.sh generates the JetStream key once (.data/jetstream.key) and puts it in nomad/jobs/nats; NATS then encrypts its streams, KV buckets and object stores (ChaCha20-Poly1305). See TLS, Encryption and Secrets.

A shared platform#

A shared platform hosts one cell per tenant: the nats job includes the resolver configuration of the operator (resolver_conf), and each cell runs its own job cell-<name>, in the namespace of the cells, with secrets=true. Cells describes how stellar-admin provisions them.

Containers#

On a production platform the cells run in containers (Docker driver, no raw_exec) of the image stellar-mcs (Container images), behind a Traefik that finds them with its Nomad provider: stellar-admin runs jobs/containers/cell.nomad.hcl for each cell (runtime: containers in platform.yaml). Its groups and their scaling are those of the stellar job; what changes:

• Routes from tags. The service cell-<name>-api carries the router of Traefik: its host, entry point, TLS and certificate resolver. Traefik watches the namespace of the cells (providers.nomad.namespaces=[cells], exposedByDefault=false); a new cell is routed within its refresh interval, a deleted one disappears with its services. The proxy of the platform (jobs/proxy.nomad.hcl) stays for the cells as processes, which it reaches on the loopback interface.
• Ports of their own. Each API instance listens on 8080 in its container, published on a port chosen by Nomad; Traefik spreads the requests among them. The editor is found by the API through its service cell-<name>-editor.
• Hardened containers. Read-only root file system, no capabilities, no-new-privileges, the user of the image (10001); the secrets are files of that user, mode 0400. The only host path is the configuration repository of the cell, mounted read only (read and write for the editor): the Docker plugin needs volumes { enabled = true }.
• NATS off the loopback interface. The cells reach NATS at nats_url: with jobs/containers/nats.nomad.hcl, its ports are published on the host network host_network of the clients, a private one on a host with a public address (agent-containers.hcl).

Clients of cells in containers keep the images of the cells longer than a cell sleeps (gc { image_delay = "168h" } of the Docker plugin, agent-containers.hcl): a cell woken up starts at once rather than pulling its image again.

cells-admin (jobs/containers/admin.nomad.hcl), the life cycle of the cells:

It reaches Nomad through the API socket of its task with the token of its workload identity; with ACLs, bind the job to policies/cells-admin.hcl (nomad acl policy apply -namespace default -job cells-admin cells-admin policies/cells-admin.hcl).