Observability

Every component of Stellar Control writes structured logs and serves Prometheus metrics and a health endpoint. The services of the core configure them in the observability section of the global configuration; components built with the Rust SDK take STELLAR_METRICS_ADDR.

Logs#

• Format. One JSON object per line on standard output (observability.log_format: json, the default), flat, with the fields of the event and of the current span; or human-readable text (text) for development.
• Filter. observability.log_level, in the syntax of tracing directives: info, debug, stellar_reconciler=debug,info. RUST_LOG, when set, takes precedence.
• Context. Every line of a service carries the component and instance. The handling of a NATS message runs in a message span with its operation, subject, correlation (the telecommand or run identifier) and config (the configuration revision), so every log line about a telecommand can be found by its identifier.
• At start, a service logs its effective configuration (effective configuration), the address of its metrics endpoint, and component started.

The language server logs to standard error, at warn unless RUST_LOG says otherwise.

Endpoints#

A service serves them on observability.metrics_addr (0.0.0.0:9100 by default); give each instance on a host its own port with STELLAR__OBSERVABILITY__METRICS_ADDR. The all-in-one binary serves one endpoint for all its services. A driver, transport, gateway or connector of the Rust SDK serves them on STELLAR_METRICS_ADDR, and none when it is unset.

Every metric carries the labels component (the kind: reconciler, driver…) and instance. Durations are histograms with buckets from 0.5 ms to 5 s.

Metrics#

Every component#

Services of the core#

Drivers, transports and gateways (Rust SDK)#

Gateways also publish their throughput on NATS, stellar.metrics.<gateway>.<target>, every metrics_period (5 s): the transfer manager uses it, and the web console shows it (see Writing a Gateway).

Connector lag#

Every 15 s, the reconciler leader measures each consumer of each output connector: messages not delivered, messages not acknowledged, age of the oldest of them. It publishes them on stellar.metrics.connector.<name> and as the stellar_connector_* gauges; GET /v1/connectors gives the same on demand.

A consumer is at risk when the retention of its stream is about to remove messages it has not read: its oldest unread message has reached 80 % of the max_age of the stream, or the stream, bound by max_bytes, is 90 % full while its oldest message is still unread. The reconciler then logs a warning. Alert on it:

groups:
  - name: stellar-connectors
    rules:
      - alert: ConnectorAboutToLoseData
        expr: stellar_connector_at_risk == 1
        labels: {severity: critical}
        annotations:
          summary: "Connector {{ $labels.connector }} ({{ $labels.kind }}) is about to lose data"

Suggested alerts#

Readiness in NATS#

Health of the instances is also visible without Prometheus:

• every instance publishes a heartbeat on stellar.ctl.hb.<kind>.<instance> (healthy, or degraded with a reason; a gateway adds whether its link is available). Its key in stellar_instances expires after three heartbeat periods: an expired instance is unbound and its targets are no longer ready;
• the readiness of each target, with a reason per link, is in stellar_readiness;
• GET /v1/topology, GET /v1/instances and GET /v1/instances/{kind}/{instance} show them, as do the Topology and Instances views of the web console.

NATS itself serves its monitoring on its HTTP port (8222 in the development setup): /healthz, /varz, /jsz.