Production

The production platform runs the cells of the online trial on one Nomad node, in Docker containers, behind the Traefik of the machine: the images come from the registry of the organization (ghcr.io/stellar-factory), and deployment/ in the repository holds the two scripts that install and update it.

Their settings, without any secret, are in deployment/production.env: the registry, the directories of the operator and of the cells on the node, SELinux, the domain of the cells, the witness cell, the entry point and certificate resolver of Traefik (le), the host name of the API of Nomad.

flowchart LR
  Dev["release.sh"] -->|push| GHCR["ghcr.io"]
  Dev -->|NOMAD_TOKEN, HTTPS| T["Traefik"]
  T -->|nomad.stellar-systems.eu| N["Nomad (docker0)"]
  T -->|"*.control.stellar-systems.eu"| C["cells"]
  T -->|control.stellar-systems.eu| O["onboarding"]
  O --> N
  N --> C
  N --> NATS["NATS (docker0)"]
  N --> A["cells-admin, cells-upgrade"]
  GHCR -->|pull| N

First installation#

On the node, as root, beforehand: Docker running, and docker login ghcr.io with a token that reads the packages of the organization. Then, from a clone of the repository:

STELLAR_REGISTRY=ghcr.io/stellar-factory deployment/docker/build.sh --push   # prints the tag
deployment/bootstrap.sh root@prod.stellar-systems.eu 0123456789ab

bootstrap.sh copies deployment/bootstrap to the node and runs it there; each step is kept when already done, so it can run again after a failure:

1. Nomad, installed from the packages of HashiCorp, one server and client: the Docker driver with volumes relabeled for SELinux, the credentials of the registry, the images kept a week (a sleeping cell wakes up without pulling), no raw_exec, ACLs on. Its ports and those of the tasks are on the bridge of Docker (docker0) only: nothing of Nomad or of the cells on the public address.
2. ACLs: the management token, the namespace cells, the policies of the jobs of stellar-admin, of Traefik (the services of the cells) and of the releases, and a token for each.
3. The operator of NATS (stellar-admin operator init), platform.yaml and provisioning.yaml in /etc/stellar-admin (from deployment/bootstrap), the directory of the cells /srv/stellar/cells.
4. NATS, in a container, published on docker0.
5. The first release, the jobs cells-admin (the life cycle of the cells, every five minutes) and cells-upgrade, and the witness cell canary (small, never expiring), checked after each release; the release is confirmed once it is ready.
6. What is left by hand, printed at the end: the Nomad provider of Traefik (its endpoint and token), the route of the API of Nomad (a file of the file provider of Traefik), the DNS records (*.cells, nomad, app), and where the tokens are.

The onboarding#

The Try now of the site opens the onboarding on control.stellar-systems.eu (job onboarding, image stellar-onboarding): sign-up and sign-in through Logto, a wizard that picks a use case (the templates of Cells), the environment of the user made in the background and followed live, and My environments (open, extend, sleep, wake up, delete). Each environment is a cell on <cell>.control.stellar-systems.eu.

• A sleeping cell has no route of its own: Traefik sends its host to the onboarding, at the lowest priority, which wakes it up and opens it once ready.
• Its emails: the environment ready, about to expire, expired (from the hook of stellar-admin cell tick).
• Its data: a database on the Postgres of the infrastructure (users, environments, tasks, journal of every action); its measures on /metrics (with a token).
• Its settings and secrets: the Nomad variable nomad/jobs/onboarding. The bootstrap generates the session key and the tokens; the database, the application of Logto (a confidential client, redirect URI https://control.stellar-systems.eu/auth/callback) and the mail server are added by hand (see deployment/README.md).

Releases#

export NOMAD_ADDR=https://nomad.stellar-systems.eu NOMAD_TOKEN=…   # the token of the releases
deployment/release.sh                  # the commit checked out, built and pushed
deployment/release.sh 0123456789ab     # images already in the registry (workflow "Publish images")
deployment/release.sh --rollback       # the previous confirmed release

1. The images of the commit are built and pushed (deployment/docker/build.sh --push), unless a tag is given: the workflow Publish images of the repository builds, checks and pushes them.
2. The release is recorded in the Nomad variable stellar/release (stellar-admin release set): the tag, who, when. From then on, a cell created or woken up runs it.
3. The job cells-admin takes the image of the release, and the job cells-upgrade runs every cell again with it (stellar-admin cell upgrade --all): one at a time, each running cell ready again before the next, a sleeping cell left asleep. A cell only restarts.
4. Smoke test: the witness cell answers through Traefik (its topology and its web console).
5. The release is confirmed. On a failure of 3 or 4, the previous confirmed release goes through the same steps: release.sh ends in error, the platform back on its last good version. A rollback never goes back to a release that did not come up.

stellar-admin release show prints the release, the previous confirmed one and the history (the last ten: tag, date, author, confirmed).

Upgrading NATS#

The releases leave NATS alone: its version is that of jobs/containers/nats.nomad.hcl (nats_version), and running its job again restarts it under every cell. Change it on purpose, at a quiet time: nomad job run of that job with the variables of the bootstrap.