Backups

Every Joinery site can back itself up: on a schedule, encrypted, uploaded to an S3-compatible bucket, with retention enforced. No agent, no SSH, no management node. server_manager is a fleet layer on top of this, not a prerequisite for it — a site running server_manager backs itself up through exactly the same path as a standalone install, because no site's recovery may depend on another machine being alive.

Configured at Admin → System → Backups (/admin/admin_backups).

Two parties, two profiles

A site can be backed up by more than one party. It backs itself up, and a management node managing it may take its own copies. Those are not two ways of doing one thing — they are two parties' backups, on two schedules, onto two shelves, answerable to two people. They open with the same key, because that key belongs to the machine and its administrator, not to whoever asked for the run.

A profile (includes/BackupProfile.php) is the unit that keeps them apart:

Where a manager run writes: the broker

A Managed node holds no key to the bucket its management node keeps for it. Its backup job carries, where a credential would go, a broker run: the management node's broker address, a run opened for this job when the node's agent claimed it, and that run's token. The run's engine hands that to S3Signer in place of a credential (BackupBroker, an S3LinkSource): every upload request becomes one call to the broker for a signed link, then the request itself on that link, so a stream, a multipart upload and a retry work exactly as they do against a site's own target.

• begin says where the run writes: the storage space, its base key ({path_prefix}/{slug}/manager/), the bucket and the target's name. It is asked before the run decides which chain it extends, because a chain stays in one space (bkh_remote_space_id); a run the broker takes in another space starts a new chain there.
• sign answers one link for one write inside the base key: a put, or a multipart create, a batch of ten part links, a complete. Nothing outside the base key, no read and no delete is ever signed for a node. *Nothing is written twice: a key a finished run completed is answered exists, with the size and sha256 recorded for it, instead of a link. In a space whose target locks, a put's and a multipart create's link signs the object lock headers and the answer names them; the node sends them as given (§ Object lock).
• finish names every object the run wrote, with the bytes and sha256 the node sent, and the chain. The broker records them (svo_sha256), cancels whatever else the run signed, and writes the run's ledger file (below). Only then is the run committed on the node; a run the broker did not record is a failed run, undone like any other, and its retry signs the same names again. A finish whose reply was lost is asked again and answered with what was recorded. A node that never heard back (it died after the broker recorded the run) finds its next run number already taken: that run fails saying so, and marks the chain closed, and the run after it starts a new chain (name_taken). A name already holding the same bytes counts as stored.
• abort says the run failed: what it signed is cancelled.

The asymmetry in the last row is the safety argument: the site profile depends on nothing. A management node that is down, retired or hostile costs a site nothing it was relying on.

Everything a run touches that could collide with another run is derived from the profile: the working directory (backups/ for the site, backups/manager/ for a management node's), and therefore the lock, the tar snapshot, the chain manifest, the envelope scratch and the local sweep. Sharing a snapshot alone would corrupt both chains — each run advances it, so each profile would treat the other's work as already archived.

Two locks are held. The per-profile lock is correctness: two runs of one profile share a snapshot and a manifest. The machine-wide lock is courtesy and I/O: a run that finds the other profile working reports itself skipped and waits for its next tick.

What a backup is

Two shapes, chosen by How backups are taken.

Incremental (default). A chain: one full, then runs that carry only what changed. Measured on a real site, the first run's file archive was 193 MB and the next was 37 kB. The site is archived as two parts, each with its own incremental history: data, the site directory less public_html/ (uploads, config, static files), and code, public_html/.

{path_prefix}/{slug}/{profile}/chain-{YYYYMMDD_HHMMSS}/
    manifest-0000.json      the restore contract — order, levels, hashes, sealed keys
    data-0000.tar.gz.enc    the site less public_html: the full
    code-0000.tar.gz.enc    public_html, rooted at public_html: the full
    db-0000.sql.gz.enc
    meta-0000.tar.gz.enc    shape.json + virtualhost + a note of the run
    data-0001.tar.gz.enc    an incremental
    code-0001.tar.gz.enc    a full again when an upgrade swapped the code
    db-0001.sql.gz.enc
    manifest-0001.json      run 1's manifest: every run up to and including it
    ...

The manifest (version 3) records a level on every artifact of a kind that increments — data, code, and a physical database backup, pgdata — so one kind can start over inside a chain while the others carry on; a run's own level is 0 exactly when every kind in it is. A restore takes each kind from its newest level 0 at or before the run, forward to the run (BackupChain::restore_plan(); restore_chain.sh applies the same rule).

Every run writes its own manifest, manifest-{seq}.json, naming every run of the chain up to and including it, so nothing in a chain is ever written twice. Readers take the highest number (BackupChain::newest_manifest_name()); a prune deletes every manifest of a chain with the chain. On the machine the chain's working copy is manifest.json, and staging lands the chosen manifest under that name, so everything that reads a local chain reads one file.

A version 2 manifest is the same layout with one manifest.json that every run rewrote; a version 1 manifest is a chain written before the data/code split: one files-NNNN.tar.gz.enc a run, the whole site directory, and one level per run that is the files archive's. Every reader reads versions 1 to 3, and refuses a version it does not know, by name, rather than restoring part of a chain. A runner never extends a chain of another version: the next run starts a new chain (layout_split).

Full every time. One self-contained archive per run:

{path_prefix}/{slug}/{profile}/{project}-{YYYYMMDD_HHMMSS}.tar.gz.enc            the archive
{path_prefix}/{slug}/{profile}/{project}-{YYYYMMDD_HHMMSS}.tar.gz.enc.keys.json  its envelope

A database-only backup is the same shape with {database}-{YYYYMMDD_HHMMSS}.sql.gz.enc as the archive. All three families carry the same stamp, which is what a management node's retention sorts backup storage by.

Nothing an engine produces lands on this disk. The data and code archives, the standalone archive and the database dump are each one pipeline — tar | openssl or pg_dump | gzip | openssl — whose stdout the runner hands to S3Signer::put_stream(), which uploads it as it flows. What a run holds on disk is the metadata artifact (kilobytes), the chain manifest, the snapshot, and, for a standalone archive, the compressed database dump for the length of the tar (it is a member of that archive, staged beside the output and removed with the staging directory). A standalone archive is built from the live tree — a second -C into the site root with the members renamed under project_files/ — so no copy of the site is ever made. On the smallest nodes this is the difference between a site that can be backed up and one that cannot: peak disk during a run is the site itself.

The engine's verdict arrives after its bytes. tar reports a file that changed while it was being read as exit 1 (accepted — the normal case on a live tree) and a real failure as 2 or more; pg_dump reports its own. So each engine in stream mode (--archive -) writes its statuses to a report file (--report FILE) once the stream has been fully produced, and the runner completes the upload only when the process exited 0, the report says tar 0 or 1 (or pg_dump 0), openssl 0, and at least 64 bytes went up — an openssl envelope around an empty stream is 32 bytes, and a backup of nothing is never recorded as a backup. Anything else aborts the upload: nothing partial or empty is ever in backup storage, and a chain run that fails this way is discarded under the ordinary rule (snapshots and manifest put back as they stood, the run's other objects deleted where the credential can delete).

Everything is AES-256-CBC (PBKDF2, random salt). slug defaults to the project directory name — the same value a management node would use for this site — so a standalone site that later joins a fleet keeps one location instead of starting a second pile beside the first. profile separates the parties, so a listing can always say whose backup an object is and each party's retention addresses only its own backup storage.

What a backup carries besides files and database

A backup has to be able to rebuild a site on hardware that is not the hardware it came from, so it records the facts about that hardware which a restore has to settle. shape.json, written by reconcile_site.sh --print-shape into the archive (full mode) or the meta artifact (chain mode):

{prefix}/{slug}/{profile}/
    chain-{id}/
        objects-0003.json.gz          the run's index — plain gzipped JSON, like the manifest
    {slug}-{stamp}.objects.json.gz    a standalone full's index, same stamp as its archive
    objects/
        {epoch}/
            envelope.json             the epoch's sealed data key
            envelope-{fpr}.json       the same key re-sealed to a later recovery key
            {fbb_stored_name}.enc     one object per offloaded blob
            mailbox/{id}.eml.enc      one object per offloaded message

Epochs are the key model. One data key with one envelope at objects/{epoch}/envelope.json — the ordinary envelope, sealed to the same two recipients as every chain — encrypts every object stored while the epoch is current, in the same aes-256-cbc-pbkdf2 form as an archive, so the Opening a backup with no Joinery anywhere procedure opens an object unchanged. A new epoch starts when there is none, when the recovery recipient changed, when the site key cannot open the current envelope (degrade, never fail every run), or when backup storage is proven to have lost an object stored under the current epoch: a fresh encryption is never the ciphertext the backup sets naming the lost key recorded, so it is never written there. On recovery-key rotation the older epochs are re-sealed, not re-encrypted: each envelope the site key opens is rebuilt for the current recipients and uploaded beside it as envelope-{fpr}.json, named by the first 16 hex of the new recovery key's fingerprint — nothing in backup storage is written twice. The new key opens everything through those, and the old key still opens what it always did through envelope.json. An epoch whose envelope re-sealed to the current key is already there is left alone. An epoch the site key cannot open stays sealed to the retired key alone; the run writes those down (objects/retired-epochs.json), names them in its message (so a management node's job result and this site's history say so), and Recovery Readiness says so on the recovery-key card — "N objects (X GB) open only with a retired recovery key" — with no automatic re-copy. Keep that key.

Enabled profiles, and the hold. A profile stores offloaded files when it is enabled (BackupProfile::enabled()): the site profile when a target is enabled, the recovery key proven and the backup type includes files; the manager profile when this machine has joined a management node and a manager run carrying the object store has been here (the objects/enabled marker). An offloaded file's local bytes stay on this server until every enabled profile holds it — the site profile by the store the offload tick makes on the way out (docs/cloud_storage.md § An offloaded file is in backup storage before its local copy goes), the manager profile by its held.json, which the management node's runs write. Only then are the original and its variants released. With no profile enabled nothing is held, and the file store alone serves the file.

The run. Before the archive: the epoch; what backup storage holds (the site profile lists its objects/ prefix; the manager profile reads the newest manager index through the presigned objects_index_url in its request, union held.json). held.json names the destination it describes — the target for the site profile, the management node's space for the manager profile — and a run reads it only on that destination: a profile that moved storage, or a file that names no destination, starts from the new destination's own index, and the run stores every object that index lacks.

Objects backup storage lost. An object a backup set records as stored and backup storage does not hold is not held, and the run stores it again from the site's copy — its local original, else the file bucket — under the current epoch, a new one when the lost object was under the current one. The site profile proves it itself: an object its previous index names that its own listing lacks. The manager profile is told: the management node's backup storage check lists every location an index it read marks stored that the listing lacks, and the next run's request carries them as objects_missing ({space, locations: [epoch/name]}), taken only from a listing that completed for that space; the node applies it only when its run writes to the same space. The job carries it as two parameters the agent declares, objects_missing_space and the list objects_missing, and only to an agent at 1.83.0 or newer (an older agent refuses a run that names them, so it is not told and the check keeps naming the objects until it is updated). One run names as many as fit in a job, at most 2,000; the rest are still missing at the next check and go with a later run. The run's index names the object where it now is, so the newest restore point is whole again. Earlier restore points still name the lost key and cannot bring the object back; the check keeps naming them until retention removes their backup set. An object the site no longer has a copy of cannot be stored again: the run says so, its index keeps naming the lost location, and the check keeps reporting it.

Then the exclude list; then the store — every cloud blob not held, one at a time under the offload engine's per-row lock, local originals first, then catch-up: a blob with no local bytes is fetched from the file bucket, encrypted and uploaded, both temporaries deleted. One budget covers the whole step, 2 GB or 20 minutes (BackupRunner::OBJECT_STORE_BUDGET_*, constants); what it leaves is indexed stored: false and taken next run, and the Backups page says "N files (X GB) still to copy from the file store" until it reaches zero. Then the engines with the exclude file, the index, the commit, and — after the run is offsite — held.json and the release. A run whose request did not ask for objects (a management node not running the object store, a database-only backup) stores nothing and holds nothing.

The manager run request carries these fields from a management node running the object store: objects: true, objects_index_url (a presigned GET for the newest manager index, or absent when there is none), epoch_envelope_urls ({epoch id: presigned GET} for each epoch's newest envelope in the listing, what the re-seal reads) and, when its backup storage check found any, objects_missing (above). A node whose management node sends none of them behaves as if the object store did not exist.

Node disk. Every step is bounded:

What each backup storage protects against. The object store protects against a deleted bucket, a revoked key and an accidental rm in the file store. It does not protect against losing the account backup storage is on: when the site's backup target and the file store share an access key, the Backups page and the cloud-storage page both say "Backup storage and the file store share one account; losing it loses both. A management node's copy would survive it." The same-key test is the whole rule.

On the pages. The Backups page's Offloaded files box: how many files live in the file store; per enabled backup, "Offloaded files in backup storage (this site's backup): N objects, X GB; last indexed at the run of …"; what is still to copy from the file store; "M files (X GB) waiting for the management node's backup before their local copy is released" (every cloud row whose original is still on disk — one stat per row on page load, no column, no cache; BackupObjectsStatus); the same-account line; and the daily file-store check with Bring them back (§ The file store is checked too). The cloud-storage page's Status block has the waiting count and size. The node's Backups tab in the reads the backup storage listing: "Offloaded files in backup storage: N objects, X GB by this management node; last indexed at the run of …", with Bring them back. And because a count on a page is read by nobody, BackupObjectsNotice stands on every admin page when the waiting bytes pass 2 GB or anything waits for a backup that has not succeeded in 7 days: "N files (X GB) are waiting for the management node's backup, which last succeeded D days ago. They stay on this server until it does." It clears itself when nothing waits.

What a restore reconciles

Both restore paths — the archive and the chain — end in reconcile_site.sh, which makes the restored site agree with the machine it landed on. It reports each value it changed and refuses rather than papering over a mismatch it cannot fix.

The identity, in both places it lives:

A host whose restrictions refuse unsigned code (unsigned_packages=refuse, written by install.sh docker --multi-tenant; see Installation) carries sites for other people. A copy or restore there brings the source's whole code tree and its config/, so both restore scripts check the archive before anything is written or the database is loaded, and refuse it whole if any check fails:

• Only a site's own folders. The archive may hold nothing at the site root but public_html, maintenance_scripts, config, uploads, static_files, storage and the release manifest. Anything else (a vendor/, say) holds code no signature covers, and is named in the refusal. An upgrade's own trees are never restored on any host (below), so an older archive carrying them is not refused for it.
• Code exactly as shipped. Core, every plugin and every theme are checked against their own signed manifests (utils/verify_package.php --tree, over PackageSignature::verifyTree()). The verifier and the keys are the machine's: the tree the restore script runs from, never anything inside the archive. An unsigned style theme passes; nothing in it runs. Each package that fails is named.
• The machine's trust files. The archive's copies are never used (above).

php maintenance_scripts/sysadmin_tools/backup_envelope.php open \
    --sidecar manifest-0007.json --manifest-sha256 {recorded hash} \
    --private ~/recovery.key --key-out /tmp/k
bash maintenance_scripts/sysadmin_tools/restore_chain.sh {project} \
    --artifacts {downloaded chain dir} --key-file /tmp/k \
    --manifest-sha256 {recorded hash} [--seq N] [--domain d]

Download the chain folder; the newest manifest-NNNN.json names every run, and restore_chain.sh reads it (a version-2 chain's manifest.json likewise). The manifest names every archive's size and hash, so the archives are only as good as it is: backup_envelope.php open opens a chain manifest only against the sha256 recorded when it was written — the management node shows it beside each run on the node's Backups tab, and the run's ledger file in the bucket holds it (Ledger files, below) — and restore_chain.sh checks it again and prints the hash it used. A manifest whose hash differs is refused before anything is read. Only for a bucket with neither record does --trust-bucket-manifest open one, saying that nothing vouches for it.

Every artifact is checked against its recorded size and hash before anything is written, so a truncated download fails while the live site is still intact. --seq N restores as at run N; the default is the newest. --dry-run reports the plan and needs no key. --domain names the domain the restored site is to answer to; without it the site keeps the domain this machine's config already names. --skip-ssl is passed to the reconcile, which then arms no certificate retry.

Extraction replays the chain the way it was taken: anything in the target that the archive does not list is removed, at any depth, including a directory the files engine never archives (vendor/, say) when the source had none of that name. A directory the source had but the archive left out (logs/) is kept.

A database dump does not open without its site's key. Every secret setting (mail, payment and API credentials) and every other sealed value is sealed with the secret_box_key of the site that wrote it, which lives in config/Globalvars_site.php, not in the database. A dump loaded on a site with another key restores those values unreadable: the site reads them as unset. restore_database.sh checks the key canary after the load and says so (RESTORE_SECRET_KEY_MISMATCH, beside RESTORE_OK); the remedy is the key the dump was sealed with, which --adopt-secret-key below brings across.

--adopt-secret-key is for a chain from another machine. This machine keeps its own config/Globalvars_site.php and takes the chain's secret_box_key into it: read from the chain's copy of the file by pattern (never run), required to be 32 base64-encoded bytes, written with var_export, and linted before it replaces the file, which keeps its owner and mode. Every value the source sealed then opens here, and the database password, the paths and the rest stay this machine's. It is refused before anything is written when this machine has no config of its own, and after the files when the chain's config carries no usable key (this machine's key is then left in place and the database is not loaded).

Offloaded files. The archives carry no file the site offloaded to its file bucket; those are in backup storage under objects/{epoch}/, named by the run's index (objects-NNNN.json.gz beside the run's archives). Download that tree with the site's own credential and hand it over:

bash maintenance_scripts/sysadmin_tools/restore_chain.sh {project} \
    --artifacts {downloaded chain dir} --key-file /tmp/k --objects {downloaded objects dir} \
    [--objects-mode missing|all] [--epoch-key epoch-20260901_000000=/tmp/e]

The step runs after the database is loaded, because the blob row is what says where each file belongs and how big it is. It runs the restored tree's own utils/restore_objects.php: each object is checked against the index's size and hash before it is decrypted, decrypted into its placement, checked against its row (size, and fbb_sha256 where the row records one), put in place, and only then is the row set to local. missing (the default) brings home only what the file bucket cannot serve — the bucket is HEADed per file — and all brings every offloaded file home, for a site leaving its bucket. Nothing on disk is overwritten (a file already at the placement is adopted when it matches its row and refused by name when it does not) and nothing in any bucket is deleted; running it again finishes what an interrupted run left. Variants are regenerated on demand. Each epoch's envelope opens with the machine's own backup_site_key; for an epoch it does not open, recover the key with backup_envelope.php open --sidecar {epoch}/envelope.json --private … and pass it as --epoch-key. The script can be run on its own, and --dry-run says how many files would come home and from which epochs:

php public_html/utils/restore_objects.php --index {chain dir}/objects-0003.json.gz \
    --objects {objects dir} [--mode missing|all] [--dry-run]

From the dashboard: the node's Backups tab lists every run in the node's backup storage, newest first, with the last backup, the last full backup and the oldest backup held stated above the list — and, when a run in backup storage carries an offloaded-files index, an Offloaded files row with Bring them back, which runs the restore_objects loop below in missing mode against the newest such run without restoring anything else; each row's Restore button runs the restore_chain job for that run. Prepare (stage_chain) recovers the chain key on the node from the node's own backup_site_key, so no recovery private key travels in a job record. A chain another server made (the server this site was on before a switch-over) does not open with this machine's key: Prepare stages it without one (STAGE_KEY=recovery), and the restore's approval opens it with the recovery key the operator pastes to approve, in their browser (Approve it on the node, below). Once the restore completes, the run's offloaded files follow in missing mode as the paged restore_objects jobs described under Restoring a managed node below. A chain taken by a machine that no longer exists goes onto a new one as a copy from backups (Handing a chain to a copy, below), or from a shell with the recovery key, as above.

Checking a restore: the census

A restore onto another machine is checked by counting what each side holds and comparing the counts. maintenance_scripts/sysadmin_tools/site_census.php (no arguments, run as root so it can read every directory) prints one CENSUS= line with a JSON object:

• tables: the rows in every table (a table outside public is named schema.table);
• files: for each top-level directory of the site, the files, directories and bytes, and a digest over every path with its size, so a renamed file or two sizes that changed by amounts that cancel still show; loose files at the site root are counted under .;
• secrets: whether the key canary opens (ok, dead, absent, or nokey when the site has no secret_box_key), and how many registered sealed secrets are stored and how many of them are dead, by where they live.

• copy_export, on the source. The source's own operator approves it on the source's Backups page with the recovery key (the export approval scope), against a statement naming the copy by its agent key fingerprint. The source then requires the chain's local manifest to match its ledger, opens the chain data key with its own config/backup_site_key, and seals the key, the manifest's hash (the vouch) and the host's certificate and DKIM keys to the copy's agent key, signed with its own. backup_site_key itself never leaves. When both agents carry it (JobCommandBuilder::COPY_HISTORY_MIN_AGENT_VERSION), the export also lists the site's backup history: the newest manifest of each chain the source's ledger holds, by hash, size and upload time.
• copy_import, on the copy. It checks the signature by the source key its dormant install recorded, the target, the expiry and that the bundle is newer than the last one taken, then writes chain.key into restore_<chain_id>, the vouch to /etc/joinery/sites/<site>/vouched, the history to history beside it, and the certificates.
• copy_stage, on the copy. It downloads the chain from the source's backup storage, keeps the manifest only when its hash is vouched for, and checks every artifact against it (CopyStaging). The ordinary staging, which trusts only this machine's own ledger and key, is not used. It also records in the copy's own upload ledger, marked handed over, the vouched manifest, every artifact it names, and the history's manifests (The upload ledger, below), so once the copy is the site, its backups from before the switch restore there through the ordinary staging; their key opens with the recovery key at the restore's approval. A copy that cannot write its ledger refuses to stage, by name.

php -r '
$env = json_decode(file_get_contents($argv[1]), true);
$sk  = base64_decode(trim(file_get_contents($argv[2])));
$kp  = sodium_crypto_box_keypair_from_secretkey_and_publickey(
           $sk, sodium_crypto_box_publickey_from_secretkey($sk));
foreach ($env["recipients"] as $r) {
    $key = sodium_crypto_box_seal_open(base64_decode($r["sealed"]), $kp);
    if ($key !== false) { file_put_contents("data_key.txt", $key); exit; }
}
fwrite(STDERR, "no recipient in this envelope opens with that key\n"); exit(1);
' {archive}.keys.json recovery_private.b64

openssl enc -d -aes-256-cbc -pbkdf2 -pass file:data_key.txt \
    -in {archive}.tar.gz.enc -out {archive}.tar.gz

php maintenance_scripts/sysadmin_tools/escrow_keypair.php generate --private-out ~/recovery.key
php maintenance_scripts/sysadmin_tools/escrow_keypair.php unseal   --private ~/recovery.key

It exists for one adversary: the party that chooses where a restore's bytes come from. When a management node runs this machine's backups it also owns the bucket and signs the download, and the person approving a restore approves a name and a date — they never see the bytes. The ledger is the only thing on the machine able to say whether those bytes are the ones it uploaded under that name. Two attacks fail against it, and the second is the one that carries it:

• Forgery — an artifact whose content is simply made up. The hash does not match.
• Replay — this machine's own genuine month-old archive, served under a fresh-looking name. Every signature verifies and every envelope opens, because it really is this machine's backup; sealing does not touch this at all. The name it is offered under has no record, so it is refused.

• Handed over (record_handed_over()). A site copied onto this machine keeps the backups its previous server made. That server lists the newest manifest of each chain from its own ledger, in the export it signs for the copy, and the copy's final manifest in the vouch it signs at the switch (Handing a chain to a copy, below). The copy records them with the other server's upload times, so after the switch the site's backups from before it restore here under their own names. A manifest that server did not sign for is still one this machine never heard of.
• From a manifest (record_manifest_artifacts()). A manifest fixes the name, size and hash of every artifact its runs made. Once a manifest's bytes match the ledger, staging records the artifacts it names that the ledger holds no line for, before fetching any of them. A name the ledger already holds keeps its hash.

{path_prefix}/{slug}/ledger/{run id}.json

bash maintenance_scripts/sysadmin_tools/restore_project.sh {project} /backups/{archive}

php maintenance_scripts/sysadmin_tools/backup_envelope.php open \
    --sidecar {archive}.keys.json --private ~/recovery.key --key-out /tmp/k
bash maintenance_scripts/sysadmin_tools/restore_project.sh {project} {archive} --key-file /tmp/k

Bring the backup back. Every node deletes its local archive once it is safely uploaded, so the normal state of a machine is that its backups are all offsite — and a restore takes the name of a file it expects to find in its own backup directory. Bring back to node on the node's Backups tab (or Prepare, for a chain) fetches it. No bucket credential is sent: the management node signs one object key, for no longer than the job's own claim budget, and the node receives the signature. A node holds no bucket credential at all — it writes through the broker, which signs it writes and nothing else — because a node that could read backup storage is a node whose compromise reaches every other node's backups. Everything fetched is checked against the node's own upload ledger before it lands, and lands 0600 — created that way, not chmod'd afterwards, because on a container node the backup directory is inside the site tree and a descriptor opened during a multi-gigabyte transfer stays open. The transfer is capped at the size the ledger recorded, so a response that declines to say how big it is cannot run the node's disk to zero; and a failed download reports its HTTP status without its body, because the plane picks the URL and reads the transcript.

Ask for the restore. The management node dispatches it as a primitive. It sends a name and a profile — no path, no key, no bucket, no domain. A chain restore also carries a project, and the node treats it as a claim to check rather than an instruction: restore_chain.sh spends that value twice, on the tree it replaces and on the database name it loads over, so the node uses its own project and refuses a job naming any other.

Approve it on the node. The node's agent claims the job and runs nothing. It composes its own statement of what it would do — which project, which database, which archive, the archive's real age, size and fingerprint — from its own records, seals a one-time challenge to the backup recovery public key it already holds, binds it to that job and that statement, and stages it for the node's own site. The node's Backups page shows the pending approval; the operator opens the challenge there with their recovery key, in their browser, and answers. The agent verifies the answer against what it sealed and only then restores.

A chain staged without a key (another server made it) carries its key box in the statement: the recovery recipient's ephemeral public key and the recovery key's fingerprint, bound into the challenge with the rest. The statement says the backup opens only with the recovery key, and names the server before the switch-over as its maker when the ledger line was handed over. The page works out, from the same pasted key, the X25519 of that key with the box's ephemeral key (recovery-readiness.js openKeyBox), the one value that opens that box, and posts it with the answer; a backup sealed to an earlier recovery key asks for that key in a second box, by its fingerprint. The agent finishes opening the box as copy_take_key does, writes the chain key where Prepare would have, and only then restores. A right answer without the value, or a value that does not open the box, refuses the restore.

Nothing else writes while it loads. restore_database.sh renames the database {name}__restoring for the load, so a page or a scheduled task that asks for it by name finds nothing and writes nothing: a row written mid-load collides with the dump's own and stops the load after the schema is dropped. It is renamed back once loaded, or as it is when the load fails. A work database left by a restore that was killed is the one replaced when nothing has the site's name; with both present the restore refuses (RESTORE_DB_BUSY) and touches nothing.

A chain restore finishes with its own release's tools. Its extraction puts back the maintenance_scripts/ the backup carried, which for an older run are older tools. restore_chain.sh copies this release's sysadmin_tools/ and the install_tools/ scripts aside before extracting, and runs the database restore and reconcile_site.sh from that copy.

What a database restore leaves behind, and what it does not roll back. A database restore replaces the database and nothing else, so a dump taken before an upgrade lands under files that are still on the newer release: the site's recorded system_version and its settings go back to the older value while the code on disk stays put. This is self-correcting — the next upgrade re-runs the migrations, which are idempotent — but it means the version a freshly restored node reports from its database can trail its files until then. A restore also leaves the archive it was given in the backup directory; the local sweep expires it on the ordinary keep_local_days schedule.

A run records itself as running before it dumps the database, so every dump holds the run that took it as running. After the load, restore_database.sh settles every such row, since none is running on this machine: the run a chain restore names (--restored-run, which restore_chain.sh reads from the manifest) is recorded as finished, with its chain, run, size and time, because its archive is the one just loaded; it is the newest row running at or before that time. Any other running row is marked never finished.

Bring the offloaded files home. The archives carry no file the site offloaded to its file bucket, so once the chain restore completes the plane brings those home too, in missing mode — only what the file bucket cannot serve. The node never lists its backup storage and holds no read credential, so every object arrives by a link the plane signs, and a job is bounded, so the work is paged and the plane drives the loop (FleetObjectRestore): a survey job (the restore_objects primitive with the run's index link and no object links) has the node read the index — ledger-checked like every artifact — HEAD its file bucket per offloaded file, and answer with the names it would bring home (RESTORE_OBJECTS_WANT, at most a thousand, RESTORE_OBJECTS_MORE when there are more). The plane signs a page of links from that answer — object links plus the envelope of each epoch those objects are sealed under, filled to the job's byte ceiling and never more than 150 — and each page's result issues the next, so no link waits in a queue past its expiry. When the pages are done and the survey said there was more, a fresh survey names the next thousand; a survey that names nothing ends the loop. The node opens each envelope with its own key, fetches objects one at a time, checks each against the index before decrypting it and against its row after, and sets the row to local only once the file is in place. restore_objects is an operate primitive: it overwrites nothing and deletes nothing in any bucket, so no page needs an approval. A failed page ends the loop with its reason on its own job; a restore whose result is read long after it finished starts none. The job's record names the survey it pages and its slice, and the survey's result holds the names, so a store of ten thousand files is stored once, not once per page.

The management node is not in that path at all — not as a gate, and not as a relay. The challenge and the answer live entirely between the node's own site and its own agent, and the restore vocabulary declares no parameter through which an approval answer could travel, so relaying one is impossible by wire format rather than by care. A management node can dispatch a restore and can do nothing whatsoever to get it approved.

The costs are deliberate and worth stating. Restoring in place requires the node's site to be up. A node whose site will not boot goes onto a new machine as a copy from backups (Handing a chain to a copy, above), where the owner opens the backup with the recovery key on the new machine's own page, or by hand: a fresh site installed on a new machine, then the backup restored onto it from a shell with the recovery key (Rebuilding a site on new hardware). No approval is asked on either path, because holding the recovery key is the proof. An unanswered challenge expires and the job is refused, so a restore nobody is watching fails rather than pinning the node. And a support-driven restore requires the customer to be reachable: there is no unattended destructive path, including for us.

The archive's age is on the approval screen as an age, above the key box, because it is the one fact no automatic check can substitute for — a replayed archive is genuine, signed and openable, and only its date is wrong.

Verifying backups

A backup that has never been opened is a hope. Backup storage listing proves a backup is present; restore_chain.sh --dry-run proves it is intact against its manifest; Prepare proves the key this machine holds opens its envelope. Verification is the step that proves a backup is recoverable, without restoring it, and records the proof where people look: the site's own Backups page, the node's Backups tab on its management node, and the node's card on the management dashboard. The word every page uses is verified restorable, with a date, and the level it was proven at:

The node's history row is the authority. A verify stamps the run it opened (bkh_verify_time, bkh_verify_level 2 or 3, bkh_verify_outcome pass or fail, bkh_verify_message in plain words) whichever profile took the run. A verify that was skipped — not enough disk, a throwaway database that could not be created, a machine busy with a backup for too long — stamps the message only, because nothing was proven either way: the last real outcome stands and the skip's reason rides beside it. A request refused before anything was read is noted the same way. The management node's copy (mgn_backup_verify_*) is refreshed from the job's result lines and from the node's status report, so a verify the site ran itself is seen from the management node too.

A failed verify is surfaced exactly like a failed backup, in the node's own words, and never triggers anything automatic — not a re-run, not a deletion, not a fresh full. A failing verify is a person's problem to look at.

The engine

utils/verify_backup.php is the script a node runs (as the verify_backup agent primitive, or from the site's own scheduled task and Backups page). It takes JSON on stdin and nothing on argv: the chain, the profile, the level, a signed link to the manifest and a signed link to every object under the chain, keyed by bare name, optionally the run, and — when the run carries offloaded files — a signed link per epoch envelope the run's index names (epoch_envelope_urls, keyed by epoch id) and, for a rehearsal, the sample (object_urls, keyed by the object's name in the index, at most 20). Whoever signs the links reads the run's index from backup storage to pick them: the site's own launcher (BackupVerifyLauncher::object_links()) and the management node's builder alike. Everything about what may be fetched is BackupStaging's, shared with Prepare, so a verify can never fetch something a Prepare would refuse: the manifest is read on the machine, the artifact list comes from BackupChain::restore_plan(), every fetch is checked against the upload ledger, and the chain key is recovered from the machine's own config/backup_site_key. Offloaded files are not ledgered — their index is — so BackupStaging::fetch_objects() checks each sampled object against the index's recorded size and hash instead, and lands it under objects/{epoch}/ in the working directory. BackupVerifier is the level 2 and 3 engine over an already-staged directory (read_all, rehearse, disk_needed, sample_objects). A management node sends the two link maps with every verify of a run whose index names offloaded files; a run that carries offloaded files but whose request links no envelope for an epoch they need fails by name, because the objects under it cannot be opened here.

The script works in {site}/backups/{profile}/verify-{pid}/, under the same two locks as a backup run (so it never reads a chain a run is writing), checks free disk before downloading anything (the set for level 2; the set plus twice each tree archive's full plus three times the dump for level 3), and removes its working directory, scratch tree and throwaway database on every exit path, including a fatal. The local sweep removes any verify-* directory older than a day that a killed process left behind.

It answers with one key per line:

VERIFY_RESULT=pass|fail|skipped
VERIFY_LEVEL=2|3
VERIFY_RUN=<chain_id>/<seq>
VERIFY_RUN_TIME=<manifest run time, UTC>
VERIFY_ARTIFACTS=<n read>
VERIFY_BYTES=<bytes read>
VERIFY_FILES=<entries listed (2) or files restored (3)>
VERIFY_OBJECTS=<offloaded files proven recoverable: stored, epoch envelope opened>
VERIFY_OBJECT_BYTES=<their bytes in backup storage>
VERIFY_OBJECTS_SAMPLED=<n opened and compared>     (level 3 only)
VERIFY_TABLES=<n>                            (level 3 only)
VERIFY_ROWS=usr_users:<n>,<table>:<n>,…      (level 3 only)
VERIFY_DURATION=<seconds>
VERIFY_REASON=<one line>                     (fail or skipped only)
VERIFY_NEEDS_BYTES=<n>                       (skipped for disk only)
VERIFY_FREE_BYTES=<n>                        (skipped for disk only)

Exit 0 on pass or skipped, 1 on fail, 2 on a request it could not understand. A skip's reason is disk (with both numbers, so the card can say "needs N free, has M"), createdb, or busy. An object retention deleted from under a verify in flight fails with reason gone, naming the object — an archive, an epoch envelope or a sampled offloaded file alike; the next pass verifies the newer backup, and the retention pass never holds a deletion for a verify in flight. What keeps a backup is a verify that passed (What retention will not delete).

The site's own backups

A site verifies its own backups the same way: the Backup verification scheduled task (tasks/BackupVerify.php, switched on together with Backup by BackupNightly) opens and reads the site's newest own backup every backup_verify_every_days (7 by default; 0 never), signing the links from the site's own target through S3Signer::presign_get — no credential reaches the script. It is due when the interval has passed since the last verify (pass or fail) and a newer backup exists, or when nothing has ever been verified. A site that takes no backups of its own has nothing to verify and the task says so, once, as a skip. The Backups page's Verify the newest backup button runs the same thing on demand, and Rehearse a restore runs level 3; both run in the background like Run a backup now (the request crosses on the detached process's stdin, handed over on an explicit descriptor — BackupVerifyLauncher::detach), and the result lands on the run's row under Recent backups and in the Status box's Last verified restorable line. A verify that proves nothing either way — skipped for disk or a busy machine, or refused before it read anything — leaves its reason as the run's message with no outcome (BackupVerifier::stamp_history / note_history); the row says not verified with the reason, or since then: beside a proof that still stands, and the Status box says Last attempt: when it is about a backup no older than the last proof.

BackupVerifyLauncher is the site side (newest_run, due, request, run, start).

The file store is checked too

Verification proves backup storage holds what the index says. The other side — does the file bucket still hold every offloaded file it is serving? — is asked daily by the offload tick (CloudStoreInventory, docs/cloud_storage.md § The file store is checked daily): every cloud blob is HEADed in its bucket, a slice per tick, and a file the bucket lacks or holds at the wrong size is written down by name. The Backups page's Offloaded files box and the cloud-storage page's Status block both say when it last looked and, when anything is missing, "N offloaded files are missing from the file store; the backup holds M of them" — M counted against the enabled profiles' held sets — with Bring them back beside it:

• A site with a backup target of its own runs it here, in the background (BackupObjectRestoreLauncher::start_newest() → utils/bring_back_objects.php, the request on stdin as a verify's is). The launcher reads the newest run's offloaded-files index off the site's own backup storage, surveys it in missing mode (the bucket is HEADed per file, so a file the bucket has recovered by then is left alone), and brings the rest home a page at a time — each page's object and envelope links signed here with the site's own credential (S3Signer::presign_get) and opened with its own key; the engine is the same BackupObjectRestore the node path runs. One at a time, by a lock in the profile's objects/ directory. What it did lands on the inventory record, both pages read it, and the names it brought home leave the missing list at once.
• A site backed up only by its management node holds no backup storage credential, so the box names the management node and its URL: from that node's Backups tab, Bring them back (the restore_objects backup action → FleetObjectRestore::start()) runs the paged survey-and-page loop described under Restoring a managed node, in missing mode, against the newest run in backup storage that carries an offloaded-files index. The management node does not see the node's missing list; the survey job asks the node.
• A site with neither is told nothing holds them.

php public_html/utils/bring_back_objects.php --chain chain-20260912_044520 --seq 3 [--mode missing|all]

maintenance_scripts/sysadmin_tools/verify_backup.sh drives the same engine over a chain directory an operator downloaded by hand, with a key the operator recovered. This is the one verification path that exercises the recovery private key, and the runbook for proving that the key in a password manager still opens a real backup:

# 1. Download the chain's whole directory from the bucket to DIR
#    (its manifests and every artifact they name).
# 2. Recover the chain key with the recovery private key, against the newest
#    manifest's recorded sha256 (Restoring a chain, above).
php backup_envelope.php open --sidecar DIR/manifest-NNNN.json --manifest-sha256 HASH \
    --private /path/to/recovery.key --key-out /tmp/chain.key
# 3. Open and read (level 2), or rehearse a restore (level 3).
./verify_backup.sh --artifacts DIR --key-file /tmp/chain.key --level 2
./verify_backup.sh --artifacts DIR --key-file /tmp/chain.key --level 3 [--seq N] [--project NAME]
# 4. Shred the key file.
shred -u /tmp/chain.key

It runs from a site's maintenance_scripts/sysadmin_tools (the engine is PHP in the public_html beside it, and a rehearsal uses restore_chain.sh and restore_database.sh from the same directory), prints the same VERIFY_* report, leaves the chain directory as it was (a rehearsal's scratch/ is removed), and exits 0 on pass or skipped, 1 on fail, 2 on a request it could not understand. --project names the directory the archive carries, for a rehearsal; left out it is read from the archive, and a wrong name is refused. This path holds a chain key, not the site key, so it proves the archives and reports the run's offloaded files unproven (VERIFY_OBJECTS=0); those are proven by the fetching path, which opens their epoch envelopes with the site key.

Disk and egress

Opened-and-read downloads the whole set each time; on Backblaze that is egress. At the monthly default on a 1 GB set it is inside the free allowance for a fleet of nodes; weekly across nine nodes would still be under 40 GB a month. A rehearsal needs scratch disk of roughly twice the site plus the database, and a PostgreSQL role that can create a database. Both are checked before anything is downloaded, and the machine says the numbers when it declines.

A backup needs no scratch disk beyond the site itself. Every archive and every dump streams from its engine into the bucket; a chain run's peak disk is the live tree plus the metadata artifact, a database-only run's is nothing, and a standalone whole-site run's is the live tree plus its compressed dump for the length of the tar. Memory is one upload part, 100 MiB. The double-space need is a restore's and a verify's, not a backup's.

The node tool

maintenance_scripts/sysadmin_tools/backup_envelope.php reads a backup's key back. open, relabel and site-key are deliberately standalone — no platform bootstrap — so they work during disaster recovery when the site will not boot, which is the moment that matters. mint runs only on a live site and does read its settings, because it must know which recovery key that site holds: nobody hands it one, and --recovery-pub is refused.

The Backup scheduled task (tasks/BackupRun.php) runs this site's own backups — it is pinned to the site profile, so a management node's copies can never be started by editing a row in this site's task table. It is not active on install: a site with no target configured runs nothing and warns about nothing. It switches itself on once a scheduled target and a proven recovery key both exist (see Recovery key setup); Scheduled Tasks sets its frequency and time, and switches it off or back on. It supports a dry run, which reports exactly what a real run would do without producing or deleting anything. The Backup verification task (tasks/BackupVerify.php) is switched on with it and proves the newest backup restorable on its own interval — see Verifying backups.

A run is recorded in bkh_backup_history before it starts and updated when it finishes — including when it fails. Every row carries bkh_profile (whose backup it was) and bkh_recovery_fpr (which private key opens it), so a restore never has to infer from today's settings what was true when the archive was made. A site whose backups have been failing for a month looks identical to a healthy one if only successes are written down.

A failed site run is an admin notice. From the first failure, every admin page (superadmin) carries the notice (includes/SiteBackupNotice.php, a core AdminNotices renderer): when the run failed, which target, and the engine's last line (bkh_message, colour codes stripped, the last |-joined segment). It reads the newest site-profile row; a run still recorded as running is not a failure, and a manager-profile row never speaks for the site's own backup. The next site-profile success clears it. A site that has never configured a target has no row and hears nothing.

A row still marked running whose start is more than six hours old (SiteBackupNotice::STALE_RUN_HOURS) is a run that died without writing its end — the kernel killed it, or it lost its database and could not get it back. The notice names it as started at … and never finished*. The runner makes the second case rare: when recording a failure throws, BackupRunner::fail() asks DbConnector::reconnect() for a fresh connection once and saves again, because the failure that ended the run (a full disk, a PostgreSQL restart) is often the one that killed the connection. reconnect() rebuilds the current mode's link from the same settings; it is for a long-running CLI process, and nothing that runs inside a web request calls it.

Because manager-profile rows land in the site's own database, the site can answer "does someone back me up?" locally: BackupHistory::manager_coverage() returns the newest manager-profile success that reached its bucket within MANAGER_COVERAGE_DAYS (7), or null. The setup wizard's Backups step reads it as a green condition, so a fleet-backed node is not asked to configure a bucket it is already archived to; coverage goes stale on its own if the management node's runs stop.

Saving a target

One form adds and edits a target: BackupTargetForm, drawn by the site's Backups page, the setup wizard (a short form: no name, folder or Enabled box) and, on a management node, the Targets page. One save path sits behind all three (BackupTargetForm::save()). The provider select shows only the region and endpoint fields the provider asks for, and only those are read from the post; the lock days field shows only for a provider that takes object lock (§ Object lock, below).

A site's first enabled target becomes where its backups go (backup_target_id) as it is saved. Nothing ever picks "the one enabled target" later.

The file store is a target row too, of purpose files (bkt_purpose), set up on the Cloud Storage page through the same form in its files mode (docs/cloud_storage.md). It is never a place backups go: every list of targets is the backup targets unless it asks for file stores (MultiBackupTarget, purpose), and this form's save refuses one. Its location is fixed once an offloaded file records it.

A target's location is fixed once it is used. Provider, endpoint, region, bucket, folder and lock days are drawn read-only, and refused on save, once anything was stored in the target (a history row points at it) or, on a management node, a storage space on it is still live (BackupTarget::location_refusal()). The name and the key stay editable. To use another bucket, add a target. The folder is stored in one form — no leading or trailing slash, no empty segment (BackupTarget::normalise_prefix()) — and no two targets share a name, since a name says which target a backup went to.

Switching off and deleting. A switched-off target takes no new backups; listings, restores and pruning carry on. It is not switched off while it is where new backups go or, on a management node, an owner's new backups go to it (an active storage space; disable_refusal()). It is not deleted while either holds, while it still holds backups (runs not yet pruned), or while it keeps an owner's older backups (a draining space; delete_refusal()). Each refusal is a sentence naming what still uses the target.

A target is proven before it is saved: the save runs TargetTester::test() on an enabled target and refuses when it fails, printing why with the values kept; the separate Test button runs the same check on a saved target. A disabled target is saved untested, and enabling it is a save. The steps, in order, stopping where the rest would be meaningless:

1. Its own bucket. Not the file store's bucket (BucketCheck::collision_step, by name and endpoint host). Refused outright, before the network: one bucket for files and backups means one deletion or one public-read setting takes both.
2. Reach. The main key lists the bucket (one ListObjectsV2, one key).
3. Write. The main key stores a probe object under the target's prefix.
4. Private. An anonymous read of that probe is refused. Backups are encrypted, but a bucket anyone can read is refused all the same.
5. Prune. The main key deletes the probe; retention needs that. A bucket that locks every new object by default (object lock with a default retention) fails here, saying so: retention could not prune on its own schedule. Lock days on the target are how backups are locked.
6. Backblaze key. The key states what it may do (b2_authorize_account's allowed, kept by B2Client::authorize()): a key pinned to another bucket is refused; one that opens every bucket on the account passes with a warning naming the file store buckets it also opens; a missing capability (listFiles, readFiles, writeFiles, deleteFiles, plus readBucketRetentions, readFileRetentions, writeFileRetentions on a target that locks) is refused naming it.
7. Lock, on a target that locks. The bucket's object lock setting is on (GetObjectLockConfiguration), and a probe written locked for one day is refused a delete. A bucket without lock is refused, saying lock can only be turned on when a bucket is created. The probe stays until its date, under _joinery_lock_probe/ beside the target's folders; each test first deletes the probes whose date has passed.

Object lock

A backup target can lock what is written to it: under object lock in compliance mode, nobody can delete or replace an object before its retain-until date — not this site, not the management node, not the key's owner at the provider's console. A break-in on the machine holding the key cannot erase the backups.

• Where. A backup target on a provider that takes object lock (StorageProvider::object_lock(): Backblaze B2, Amazon S3, Linode). The bucket must have object lock on, which a provider allows only when the bucket is created; the connection test refuses a target set to lock whose bucket has none. A file store never locks: deleting a member's file has to delete it.
• How long. bkt_lock_days, 0 for off, part of the location (fixed once anything is stored). The form suggests the site's retention plus its full-backup interval (BackupTarget::suggested_lock_days()), which covers each backup for as long as its set can be restored from.
• Every write is locked. A site writes through the target's write credential (BackupTarget::write_credentials(), S3Signer::with_lock()): every whole-object PUT and multipart create carries x-amz-object-lock-mode: COMPLIANCE and a retain-until date that many days out, and a locked PUT carries its body's Content-MD5, which Backblaze requires. Through the broker, the lock headers are signed into each link (ShelfBroker::lockHeaders(), S3Signer::presign()'s signed headers) and named in its answer, so the node sends them and cannot leave them out: the provider refuses a request missing a signed header. The run's ledger file is locked the same way. A node on a release older than JobCommandBuilder::LOCK_MIN_CORE_VERSION is refused a backup or a re-upload to a space that locks, naming the release.
• A delete waits for the date. S3Signer::delete() answers LOCKED (423, with locked_until) for a version the lock still holds, read from the version's retain-until date. Retention deletes a point whole or not at all: while the lock holds any of it — judged from when it was written plus the lock days, with fifteen minutes for clocks (BackupTarget::held_until(), BackupHistory::held_until(), and for a chain the newest LastModified in its folder, which a failed run may have written after the last success; the listing's LastModified on the management node) — a surplus point is left as it is, still recorded surplus, and the first pass after its date deletes it. Should the clocks disagree by more than that and a delete inside a point be refused, the rest of the point waits the same way, without an error. The offloaded-files prune remembers each delete it could not make, and every candidate of a pass that could not read a retained run's index (unpruned.json beside the site's objects, per target), and offers it again, through the same check of the retained runs' indexes, once its date has passed; an epoch whose re-sealed envelope is still locked is tidied again the same way. A customer's lapse prune is finished — its ledger rows marked, the customer told — only once nothing the lock holds is left; until then each pass tries again. Stored Backups and Empty backup storage say the date in their refusal.
• What a refusal is read as. A delete the provider refuses is read as locked when the version's retain-until date is still ahead. So on a locking target, a key that has lost its delete permission looks held until that date, and then fails with the real reason; the connection test's Prune step, whose probe is not locked, shows a missing permission at once.
• Cost. Storage for a locked object is paid until its date even when retention would delete it sooner, and nothing can shorten that; a mistaken large upload costs its lock period. The form says so beside the field.

A target an installer creates

A first-boot installer that was handed a bucket and its key pair (the Linode deploy form's optional backup fields) creates the first target itself with utils/install_backup_target.php: the target is saved, a Backblaze credential's region and endpoint are filled from Backblaze's authorize answer (BackupTarget::complete_credentials, the same call the Backups page makes), the target is proven (§ Saving a target), and a first target becomes the scheduled one. A target that fails is removed again so the setup wizard asks for one. The recovery key is never created here — it is shown once to a human — so nightly runs still wait on the wizard's key ceremony, exactly as they do for a target saved on the Backups page. Inputs are environment variables (JOINERY_BACKUP_BUCKET, JOINERY_BACKUP_KEY_ID, JOINERY_BACKUP_KEY, optional JOINERY_BACKUP_PROVIDER, any provider in StorageProvider, default b2, with JOINERY_BACKUP_REGION or JOINERY_BACKUP_ENDPOINT where the provider asks for one); the first output line is INSTALL_BACKUP_TARGET=ok or =error.

Artifact naming

includes/BackupNaming.php owns which files are backups, what each one is, and what restoring it would do. Every surface consults it — the management API's local listing, the node Backups tab, the job builder's globs.

Recognized suffixes are matched longest-first: .sql.gz is a suffix of .sql.gz.enc, so a shortest-first match would classify every encrypted dump as plaintext and hand the restore engine a file it will not decrypt.

Settings

The account backups run as

The scheduled task runs as the web user, which is what fix_permissions.sh makes the owner of the site tree. A backup that cannot read part of the tree fails rather than shipping a partial archive, and says which file it could not read — so a permissions problem surfaces as a failed run with a specific cause, not as a backup that turns out to be incomplete on the day it is needed.