File Transfers

A file transfer is a first-class object, like a telecommand or a run: it has an identity, a state and a history, and it can span several passes. Only the missing ranges are asked again at each pass, and a transfer left unfinished resumes by itself at the next one. The transfer manager (stellar-transfers) drives the transfers, stores the chunks, assembles and verifies the files, and offers the verified files to the drivers for decoding.

The catalogue declares the on-board directory, the file types and the transfer protocol of a platform in its standard files component: see Files and Streams.

The on-board directory#

The files component exposes the on-board directory as measures, one instance per on-board file, named by its identifier in decimal:

The generation is part of the identity of a file (a counter or a creation date). A file rewritten on board is never confused with its previous version, and the chunks of two generations never mix.

The listing telecommand of the catalogue (transfer.list) brings the directory down. The driver decodes it into samples of files, which follow the path of every measure: compute stage, PARAMS, current value table. A file deleted on board keeps its last values, dated by the last listing where it appeared.

stellar files sat1-fm              # the directory, from the last listing
stellar files sat1-fm --refresh    # send the listing telecommand, then wait for the new listing

files[12]  lttm  65536 B  generation 3  checksum 9a3f0c21  transfer VERIFIED
files[13]  log  4096 B  generation 1  checksum 0c77e1b5  transfer -

--refresh sends the listing telecommand as a direct telecommand and polls the directory for up to 10 seconds until a newer listing arrives. The API gives the same: GET /v1/targets/{target}/files (each file with the date it was last listed, listed_at) and POST /v1/targets/{target}/files/refresh. A platform without a listing telecommand answers 422 files::no-listing; a platform without a files component 404 files::no-directory.

States of a transfer#

stateDiagram-v2
    [*] --> REQUESTED
    REQUESTED --> PARTIAL: first range
    PARTIAL --> COMPLETE: every byte received / written
    COMPLETE --> VERIFIED: checksum matches
    COMPLETE --> CORRUPTED: checksum differs
    VERIFIED --> PROCESSED: decoded by a driver
    REQUESTED --> SUPERSEDED: newer generation
    PARTIAL --> SUPERSEDED: newer generation
    PROCESSED --> [*]
    CORRUPTED --> [*]
    SUPERSEDED --> [*]

PROCESSED, CORRUPTED and SUPERSEDED are final. VERIFIED is final too for a file that no driver decodes. Each change of state is published as the measure files[f].transfer, so a procedure checks it like any measure:

expect sat.files[lttm].transfer is PROCESSED within 8 min

A transfer is stored in the stellar_transfers bucket under <target>.<file>.<generation> (an upload under <target>.<file>.<generation>.up): size, checksum and type from the last listing, state, ranges received [start, end), priority, requester, reason.

Downloading a file#

stellar download sat1-fm 12                     # the current generation of file 12
stellar download sat1-fm 12 --priority 10       # before the others of the target
stellar download sat1-fm 12 --wait              # follow it; exit 0 once VERIFIED or PROCESSED
stellar transfers --target sat1-fm              # every transfer of the target

sat1-fm  file 12 (3)  download  lttm  65536 B  37 %  PARTIAL

A download targets the generation of the last listing. A file never listed is refused (422 transfer::not-listed: refresh the directory first). Asking again for a generation already transferred returns the existing transfer: a corrupted file is downloaded again at its next generation.

POST /v1/transfers creates a download ({target, file_id, priority}); GET /v1/transfers (?target=) lists the transfers by priority then age; GET /v1/transfers/{target}/{file_id}/{generation} (?direction=upload for an upload) gives one.

From a procedure#

download is a statement of a step. It creates the transfer, with run <id> as requester, and goes on to the next statement without waiting: the step waits for the state with expect.

step "Download LTTM"
  uses sat: platform-v3
  input lttm: file_id of lttm
  download sat.files[lttm]
  expect sat.files[lttm].transfer is PROCESSED within 8 min

The file is an input file_id of <type>, whose type is declared by the platform; a literal identifier is refused. The existence of the file is checked at run time: a file absent from the directory fails the step.

How chunks are asked (chunked protocol)#

With protocol: chunked, the transfer manager drives the transfer with the telecommands of the catalogue: read asks a range, the driver extracts the chunks from the telemetry.

• Every 250 ms, for each target whose default link is available, it sends the read telecommand for missing ranges. Each range is at most max_chunk bytes; transfers are served by priority, then by age.
• Pipeline. The bytes asked and not received yet stay under two seconds of rate, so as not to flood the link.
• In flight. A request follows the events of its telecommand. Failed (not sent, rejected, timed out), its range is asked again. Sent, its chunk is awaited 15 s, then the range is asked again. A telecommand without a final event is forgotten after 60 s. A chunk lost at the end of a pass is thus asked again at the next pass, and a cut of the gateway is caught up the same way.
• Normal telecommands. The requests go through the path of direct telecommands, with the identity transfers: requires, evidence and permissions apply.

The chunks come down on stellar.file.chunk.<target>.<file> (stream FILE_CHUNKS) with their generation and offset in headers. The transfer manager stores each in the stellar_files object store under <target>/<file>/<generation>/<offset> and ignores a range already received. Once the ranges cover the file, it assembles <target>/<file>/<generation> and the transfer is COMPLETE.

Rates#

The rate that sizes the requests is, by order of preference:

1. the downlink throughput measured by the gateway (stellar.metrics, less than 15 s old), unless it is under a quarter of the next value — that of an idle link, not of a slow one;
2. the rate declared by the pass in progress (rates.downlink);
3. the theoretical rate the gateway announced at its registration;
4. 64 kbit/s.

The Prometheus counter stellar_transfers_bytes_total (labels target, outcome: asked, received) shows the bytes asked and received.

Verification#

At COMPLETE, the transfer manager computes the checksum of the protocol (transfer.checksum: crc16, crc32, md5 or sha256) on the assembled file and compares it with the checksum of the last listing:

Supersession#

A listing, or a chunk, of a newer generation of the same file makes the active transfers of the older generations SUPERSEDED, with the new generation as reason. Their requests in flight are forgotten.

Decoding LTTM files#

Once verified, a file of long-term telemetry (LTTM) is decoded by the driver into samples dated by on-board time and marked deferred. The transfer manager offers each VERIFIED file to the drivers of its target, by a request on stellar.file.decode.<target> (queue group of the drivers): {file_id, generation, file_type, object}. The driver reads the file in stellar_files, decodes it and publishes its samples on stellar.tm.decoded.<target> with Stellar-Delivery: deferred, one message per on-board instant. It answers:

A decoding lasts at most 5 minutes. Deferred samples are archived and feed pure derived measures, but never raise alarms nor feed temporal functions, and never replace a more recent current value. See Time and Freshness.

CFDP#

With protocol: cfdp, files move under CFDP class 2 (CCSDS 727.0, acknowledged mode). The ground CFDP entity lives in the driver of the default link; the transfer manager drives it.

• Implementation. A fork of cfdp-rs 0.3.0 (sat-rs project, Apache-2.0) in vendor/cfdp-rs, which fixes defects of the acknowledged mode — among them those that made a metadata-only transaction, hence the Proxy Put Request, impossible. The SDK runs one entity per target, with the settings the driver gives (Driver::cfdp: entity identifiers, PDU size, timers and limits of the ACKs and NAKs, inactivity).
• Requests. Every 5 s, while the transfer is active and the link available, the transfer manager asks the driver of the default link through its cfdp verb (stellar.ctl.rpc.driver.<instance>.cfdp): {target, file_id, generation, direction, file_type, size, source}. The request is idempotent; the reply is running (with the progress), completed or failed. A failed transfer is started again at the next request.
• Download. The entity asks the file from the spacecraft by a Proxy Put Request. The spacecraft sends it in its own transaction; the entity sends the NAKs and checks the checksum of the EOF. The data received are published as chunks on stellar.file.chunk.…: storage, assembly, verification and states are those of the chunked protocol.
• Upload. The entity reads the content in stellar_uploads and sends it. Delivered whole (Finished without error), the transfer is COMPLETE, then verified at the next listing, or VERIFIED without a listing telecommand.
• PDUs travel on the uplink subject of the link, without telecommand events; CFDP asks again what is lost. The checks (available link, priorities) apply to the requests of the transfer manager, not to each PDU.
• State. The state of the transactions is not persistent. After a restart of the driver, the next request starts the transaction again; in a download, the chunks already stored remain.

With CFDP, the catalogue declares neither read nor write. The listing telecommand stays optional: without it nothing can be downloaded, and an upload is verified by the on-board CFDP entity only.

Uploading a file#

An upload is the same object in the other direction. It is created only by a run, so that its request is validated like any run: environment policies, package status.

1. Store the content: stellar put patch.bin (or POST /v1/uploads with the raw body, 256 MiB at most) stores it in the stellar_uploads object store under its SHA-256 and prints the hash.
2. Give the hash as the value of a file input of the run.
3. The procedure uploads it with upload <input> to sat.files[x].

stellar put patch.bin
# 3b8e…c41a  18432 B

procedure "Patch software"
  uses sat: platform-v3
  input patch: file
  input slot: file_id of sw_patch

  step "Upload patch"
    upload patch to sat.files[slot]
    expect sat.files[slot].transfer is VERIFIED within 30 min

  step "Activate patch"
    ask operator "Activate the patch?"
    send activate_file to sat.files[slot]

run: Patch software
environment: AIT
targets: {sat: flatsat-1}
inputs: {patch: 3b8e…c41a, slot: 21}

• Transfer. The statement creates an upload for the generation that follows the last listing (1 for a file never listed). The same content returns the existing transfer; another content is refused while an upload of that file is in progress.
• Writing. The transfer manager sends the write telecommand of the catalogue (identifier, offset, data in hexadecimal) for the ranges not acknowledged yet, at most max_chunk each, within two seconds of uplink rate, while the link is available. The rate is the one declared by the pass, else the theoretical uplink rate of the gateway; the measured throughput is used only above it, since it reflects only what the transfers send.
• Acknowledgement. A range counts once its telecommand reaches VERIFIED (on-board echo, verify: [echo] on the write telecommand) or COMPLETE; a telecommand failed or refused is sent again. A cut or the end of a pass is caught up at the next pass.
• On-board check. Every range acknowledged, the transfer is COMPLETE and the transfer manager asks a listing every 10 s until it gets a newer one. Generation, size and on-board checksum as expected: VERIFIED; otherwise CORRUPTED, with the differences. Without write nor list telecommand, or when the content is no longer stored, the upload is CORRUPTED with the reason.

Transfers and leases#

The telecommands of a transfer started by a run carry the header Stellar-For-Run: they pass the lease that run holds on the target. Other direct telecommands, those of transfers started from the CLI or the API included, stay refused while a run holds the target.

Deleting files on board#

Deletion on board is never automatic. It goes through an explicit telecommand in a procedure, conditioned on the state of the transfer:

step "Free LTTM"
  uses sat: platform-v3
  input lttm: file_id of lttm
  check sat.files[lttm].transfer is PROCESSED
  send delete_file to sat.files[lttm]

A telecommand of files with an argument named file_id receives it from the instance: it is not written in with.

Simulated files#

The simulator generates on-board files (files: {lttm: {every: 10 min, size: 64 KiB}}), serves the list, read and write telecommands, runs a CFDP entity on board for a CFDP platform, and can rewrite or corrupt a file to exercise supersession and verification.

See also#

• Files and Streams: declaring the files component.
• Continuous Streams.
• API reference: files.