Cells

A shared deployment of Stellar Control (demonstrations, integration tests, development environments) hosts one cell per customer: a NATS account and a dedicated set of core services connected to it. The same cell is what an on-premises installation deploys. Cells share the NATS cluster and the Nomad cluster; nothing else.

flowchart LR
  subgraph NATS["NATS cluster (operator, full resolver)"]
    A["account demo"]
    B["account lab"]
  end
  subgraph Nomad["Nomad cluster, namespace cells"]
    subgraph NA["job cell-demo"]
      SA["api, reconciler, executor, compute, alarms, scheduler, transfers, editor, simulators"]
    end
    subgraph NB["job cell-lab"]
      SB["stellar-mcs (small cell)"]
    end
    P["Traefik"]
  end
  SA --- A
  SB --- B
  P -->|demo.mcs.example.org| SA
  P -->|lab.mcs.example.org| SB
  Bench["bench of lab: gateways, drivers"] -->|leaf node, account lab| B

Why accounts#

An account is the unit of isolation that NATS itself enforces: the subjects, streams, KV buckets and object stores of an account are invisible to the others. So:

• Names do not change. No tenant prefix anywhere: the NATS contract, the drivers and their permissions are the same in a cell, on premises and in development. A missed prefix cannot leak data between tenants.
• Quotas come for free. The JetStream limits of the account (storage, streams, consumers, connections) bound what a cell can use.
• Each reconciler signs for its own account. The reconciler of a cell holds a signing key of the account of its cell, and the JWTs it issues are valid in that account only.

Services per cell#

Each cell has its own services: api, reconciler, executor, compute, alarms, scheduler, transfers, editor and simulators. Sharing services between tenants would put the tenant in every snapshot, election, lease and partition: it is not planned.

• A small cell runs the services of the core in one process, stellar-mcs (see Running the Services).
• An idle cell sleeps: every service at zero instances. It wakes up with its state intact, since the state is in its account.
• Benches join the cell of their customer: their gateways, transports and drivers connect through a NATS leaf node bound to the account of the cell, over TLS or WebSocket on 443.

Security of a cell#

Encryption at rest protects disks and backups; between tenants, the accounts isolate. See TLS, Encryption and Secrets.

stellar-admin#

stellar-admin provisions the cells. It is a Python tool of the operator of the platform, distinct from the stellar CLI and the SDKs and never published: it works only with the secrets of the operator (operator key, system account credentials, Nomad token), which no user holds.

Setup#

export STELLAR_ADMIN_DIR=/etc/stellar-admin   # secrets of the operator and platform.yaml
export NOMAD_ADDR=https://nomad.example.org:4646 NOMAD_TOKEN=…

stellar-admin operator init --name stellar-platform --resolver-dir /var/lib/nats/accounts

operator init writes, in STELLAR_ADMIN_DIR (mode 700, secrets 600): the seed of the operator (operator.nk), the system account and a user of it (system.nk, system.creds), and resolver.conf, to include in the configuration of the NATS servers (resolver_conf of the nats job): the operator, the system account and a full resolver, so that accounts come and go at runtime without a restart.

The platform itself is described by platform.yaml, in the same directory:

nats_url: tls://nats.example.org:4222   # of the tool and of the cells
nats_ca: /etc/stellar/tls/ca.pem        # optional
bin_dir: /opt/stellar/bin               # release binaries on the Nomad clients
config_file: /opt/stellar/stellar.yaml  # global configuration of the cells
repository: /opt/stellar/config         # configuration repository of the cells
jobs_dir: /opt/stellar/deployment/nomad/jobs
domain: mcs.example.org                 # a cell is <name>.<domain>
api_port_base: 18100                    # APIs of the cells from there, two ports each
limits: {disk_storage: 2000000000, mem_storage: 268435456, streams: 64}

Those cells run as processes, behind the proxy of the platform. In production they run in containers, behind Traefik (Containers):

runtime: containers
ingress: traefik                        # the only one of the containers
nats_url: nats://172.17.0.1:4222        # reached from the containers, and from the tool
repository: /usr/share/stellar/examples/config   # the template of the repositories of the cells
cells_dir: /srv/stellar/cells           # a repository per cell: <cells_dir>/<cell>/config
cells_owner: "10001:10001"              # the user of the services, owner of the repositories
jobs_dir: /usr/share/stellar/jobs       # in the image stellar-admin
domain: cells.example.org
image: registry.example.org/stellar-mcs
tag: 0123456789ab                       # immutable
traefik_certresolver: letsencrypt
traefik_wildcard: "*.cells.example.org" # one certificate for every cell
limits: {disk_storage: 2000000000, mem_storage: 268435456, streams: 64}
variables:                              # more variables of the job of the cells
  memory: '{default = 128, mcs = 384, editor = 256, simulators = 64}'

Commands#

stellar-admin cell create demo --ttl 7d       # https://demo.<domain>
stellar-admin cell create lab --count editor=1
stellar-admin cell create mini --small
stellar-admin cell credentials demo > bench.creds   # STELLAR_NATS_CREDENTIALS of its benches
stellar-admin cell secret lab forge_token < token
stellar-admin cell sleep demo
stellar-admin cell wake demo
stellar-admin cell delete demo

What cell create provisions#

1. An account, signed by the operator and pushed to the resolver of the NATS servers ($SYS.REQ.CLAIMS.UPDATE), with the JetStream limits of platform.yaml and, with --ttl, its expiry: past it, the servers refuse its connections.
2. A signing key of the account for the reconciler of the cell, and two users signed with it: services (the core services) and bootstrap (the instances of the benches: registration, heartbeats and control verbs, until the reconciler issues their own JWT).
3. With cells_dir, its configuration repository <cells_dir>/<name>/config: the files of its template (repository of platform.yaml by default), committed on main as its user, given to cells_owner. Its services compile and publish it when they start, and its web editor publishes the drafts proposed there at once (direct forge, see Web Editor).
4. Its Nomad job cell-<name>, in the namespace of the cells (cells), with the secrets of the cell in the variable nomad/jobs/cell-<name>: Nomad gives them to that job only. Its services run as processes or in containers (the runtime of platform.yaml, see Nomad).
5. Its record in stellar/cells/<name> and its route: with the proxy of the platform, the variable nomad/jobs/proxy/cells/<name>, read without redeploying it; with Traefik, the tags of the service cell-<name>-api, read by its Nomad provider.

cell delete removes all of it: the services; the streams, KV buckets and object stores of the account (renewed first when it expired, so that its data can be reached); the account ($SYS.REQ.CLAIMS.DELETE); the secrets, the service registrations of its job, its repository, the route and the record. Nothing of the cell is left, on disk either.

Each step is noted in the record once done: a creation interrupted (the cell still provisioning) goes on from there when asked again, and cell tick deletes one left more than an hour.

Life of a trial cell#

An online trial gives each user a cell of their own, ready to use: provisioning.yaml, next to platform.yaml, says what a cell gets and how long.

offers:
  trial:
    ttl: 7d                 # life of a cell
    max_ttl: 30d            # longest life, extensions included
    idle_sleep: 2h          # asleep after two hours without an access to its API
    small: true             # the core in one container
    counts: {editor: 1}     # with its web editor
    limits: {disk_storage: 536870912, mem_storage: 67108864, streams: 32}
    variables: {memory: '{default = 128, mcs = 384, editor = 256, simulators = 64}'}
templates_dir: /usr/share/stellar/templates    # the templates of the image stellar-admin
templates:                  # more, or overriding one of the directory
  my-bench:
    path: /srv/stellar/templates/my-bench          # a configuration repository
default_offer: trial
quotas: {per_owner: 1, total: 40}
warn_before: 1d             # the user warned a day before the expiry
hook: {url: https://app.example.org/hooks/cells, token_env: STELLAR_ADMIN_HOOK_TOKEN}

• Ready to use. The cell starts from its template: its repository committed as its user, the configuration published before its services start, its simulators running. It is ready once every service is up and healthy.
• Asleep when idle. Its API notes its last access (bucket stellar_activity, key api, at most once a minute; its health checks excluded). Past idle_sleep without one, cell tick puts the cell to sleep; waking it up is the first thing its next visit does (the onboarding application, through wake).
• Expiry. warn_before its expiry, the hook receives expiring (once); past it, the cell is deleted and the hook receives deleted. extend pushes it back, within max_ttl: its account is signed again with the new expiry.
• Events. The hook receives a JSON POST per event (expiring, deleted, asleep): the cell, its URL, its user (id, name, email) and its expiry, with the bearer token of token_env. An event not received is sent again at the next round.
• Periodically. deployment/nomad/jobs/containers/admin.nomad.hcl runs cell tick every five minutes in the image stellar-admin, with the directory of the operator and the directory of the cells; with Nomad ACLs, bind it to deployment/nomad/policies/cells-admin.hcl.

Templates#

A template is a configuration repository that runs as it is, every target of its demonstration simulated, with a template.yaml: its title, description and use case, and the procedure and run request of its demonstration, proposed to the user at their first visit. The images stellar-admin and stellar-onboarding carry them in /usr/share/stellar/templates, one directory each, named after the template.

Where the templates come from#

deployment/docker/build.sh runs deployment/docker/templates.sh before building, which fills deployment/docker/.templates for the Dockerfile:

• From the catalog of Stellar Systems (STELLAR_CATALOG, the catalog.yaml of stellar-catalog). Each use case of Stellar Control is a repository of its own, github.com/<company.github>/<repo.name>, cloned at the tag repo.ref pinned in the catalog; a branch (main) is refused, so that an image never changes with a repository. The template is named after the use case (use-cases/<id>), the name the onboarding offers for #/new?usecase=<id>. The catalog goes into the image (/usr/share/stellar/catalog/catalog.yaml, ONBOARDING_CATALOG of stellar-onboarding), and /usr/share/stellar/templates.MANIFEST records the repository, tag and commit of each template.
• From this repository without a catalog, as before: examples/use-cases, and examples/config as space-missions; no catalog in the image, the onboarding then shows the templates as they describe themselves.

STELLAR_CATALOG=../stellar-catalog/catalog.yaml TEMPLATES_TOKEN=$(gh auth token) \
    STELLAR=target/release/stellar deployment/docker/build.sh

The workflow Publish images does the same when the repository variable STELLAR_CATALOG_REPO (owner/name of stellar-catalog, at STELLAR_CATALOG_REF, main by default) and the secret TEMPLATES_TOKEN are set.

A new version of a template is a tag on its repository (v0.1.1), then the same tag in repo.ref of the catalog, then a build of the images: the cells created from then on start from it; the cells already created keep their own repository.

The templates of the catalog (use cases of Stellar Control):

The examples of this repository: test-benches, remote-assets, flight-test, research-facilities, critical-operations, space-missions.

The simulators of a cell are read from the topology of its template: each target whose default link has the driver of a simulation of simulations/ runs that simulation, on the gateway of the link.

The use cases the onboarding presents, and in what order, come from the catalog of Stellar Systems when it is given (ONBOARDING_CATALOG, catalog.yaml of stellar-catalog): a template is matched to the use case of its use_case. A link to #/new?usecase=<id> opens the wizard on it, after the sign-in when needed; each environment then lists its first steps: the CLI from get.stellar-systems.eu, the public repository of its template, its console, its demonstration run from the command line.

The onboarding application uses the same steps as a Python library:

from stellar_admin.cells import Owner
from stellar_admin.provisioning import Provisioner, QuotaError

provisioner = Provisioner.open(Path("/etc/stellar-admin"))
status = await provisioner.create(Owner("u-42", "Ada Lovelace", "ada@example.org"), template="test-bench")
status = provisioner.status(status.cell)      # provisioning → starting → ready
provisioner.cells_of("u-42")                  # the cells of a user
await provisioner.extend(status.cell)         # a week more
provisioner.wake(status.cell)                 # on the first visit to a sleeping cell
await provisioner.delete(status.cell)

create is idempotent: asked again for the same user, it returns their cell, or goes on with the one being created; past the quotas, QuotaError.

Benches of a cell#

A bench runs its gateways, transports and drivers on site. They reach the cell through a leaf node of the NATS server of the bench, bound to the account of the cell, and start with the bootstrap credentials of stellar-admin cell credentials <cell> in STELLAR_NATS_CREDENTIALS. Once bound, the reconciler of the cell gives each of them its own short-lived JWT (see NATS Accounts and Credentials).