Encrypted data root

A host's data root (/srv/joinery, see Installation § The data root) can sit on a LUKS2 volume, so an offline copy of the machine's disk (a provider backup, a snapshot, a disk taken out of a data centre) holds its data only as ciphertext. This page covers what is encrypted, how the volume is opened at boot, who holds its keys, and the commands and files involved. The design and its reasons are in specs/hosted_data_promise_disk_encryption.md.

Nothing offers or creates an encrypted data root unless the core setting data_root_encryption_offered is on (off by default). Everything below acts only on a machine whose data root is encrypted; a plain data root behaves as Installation describes.

What is encrypted, and what is not

The volume holds the data root, and on an encrypted machine these move onto it, each as a bind mount recorded in /etc/joinery/data_binds:

Also on an encrypted machine:

• PHP's temporary files go to /var/lib/php/tmp (upload_tmp_dir, sys_temp_dir in each PHP-FPM version's conf.d/99-joinery-data-root.ini).
• /tmp is a tmpfs from the next boot (/etc/systemd/system/tmp.mount, size=50%, enabled and not started).
• Core dumps are off: DefaultLimitCORE=0 (system.conf.d), Storage=none (coredump.conf.d), kernel.core_pattern=|/bin/false (sysctl.d/60-joinery-data-root.conf).
• A swap device that is neither dm-crypt nor compressed memory is turned off by the tick. The host's own swap is compressed memory (zram, host_swap.sh), which never reaches a disk.
• The agent's kept job results go to /srv/joinery/agent-outbox while the data root is mounted, and to /etc/joinery-agent/outbox while it is not.
• A container's /tmp is in its writable layer, under Docker's data root (/srv/joinery/docker), so it is covered already.

/var/log and early boot

Two early-boot units need /var/log before any unlock can happen: systemd-update-utmp (the boot record in wtmp) and systemd-journal-flush. On an encrypted machine each runs from a copy of its package unit file in /etc/systemd/system/, regenerated each tick, without its /var/log mount requirement, which would otherwise be an ordering cycle with an unlock that waits for the network. The boot's flush carries ConditionPathIsMountPoint=/var/log, so the journal stays in memory (RuntimeMaxUse=32M) until joinery-journal-flush.service moves it onto the data root once /var/log is mounted. The root disk's /var/log underneath the bind stays writable for what early boot writes there; once each boot, after the unlock, the tick appends those files to the data root's logs, deletes them, and trims. rsyslog, fail2ban and sysstat wait for the data root, as the database and web stack do.

On a machine encrypted where it stands, /var/log, PHP's sessions and temporary files and the DKIM keys move onto the data root only once encrypt has finished, never into a volume still part plain. The move restarts what holds a log it moved: PostgreSQL, whose log is its standard error, among them. So encrypt stops the services twice for a few seconds: at its start, and when it finishes. The Backups page and the node page say so before they ask.

The host converger's path trigger (joinery-host-converger.path) is not ordered before paths.target, since its folder (cache/root_requests) is on the data root.

Making one

• Born encrypted. install.sh server and install.sh docker take --encrypted-data-root --recovery-public-key=KEY (the base64 public key the Backups page shows), and --no-tpm2 for a machine whose TPM2 chip must never open it. Where no agent is installed yet, the installer first places the release's own agent binary (agent_dist/, checked against its manifest) at /usr/local/bin/joinery-agent, with no unit and no identity: it seals the break-glass copy and opens the volume at every boot, including one before the site install. The agent's service comes with the site install, or at the end of install.sh docker. Provisioning in one of our cloud accounts passes both flags and creates the instance with the provider's disk encryption off (cvp_data_root_encrypted, see the server_manager overview, "One layer of encryption, never two").
• In place. joinery_data_root.sh encrypt [--keep-provider-layer] stops the data root's services for seconds, starts LUKS2's online encryption (--reduce-device-size 32M, the filesystem's 512-byte sectors), and brings them back; joinery-data-encrypt.service then encrypts at idle I/O priority for hours. The start goes ahead only once every mount of the data root's filesystem is gone: a service that keeps a file on one of its binds (rsyslog keeps its socket in Postfix's chroot) is stopped for those seconds, and anything else holding one refuses the start with nothing changed. Before the volume is touched, everything a recovery needs (the sealed copies, the record, the units) is synced to disk. A plain data root on a device keeps its last 32 MiB outside its filesystem for the header; one made before that needs its volume grown at the provider first. From the first stop to the last start, however the start ends (a refusal, a failure, a signal), it starts the data root and its services again on the volume as it then is, and leaves no keys or sealed copies behind; cryptsetup opens the volume as its start ends and a signal waits for that, so a start cut there closes the volume before it lets go of the image's loop device and puts the header back; it never stops the service it runs in (a site's Backups page asks through the host converger, and the request lets go of its transcript, which is on the data root, for the start). A cut during the start, before any data is encrypted, is undone by the next tick (the data moved for the header goes back, and the data root mounts plain), and the tick starts a run that is recorded and not running. A tick that finds the data root not mounted with no run of the script holding its lock starts it again, closing first a volume a cut-off start left open over a plain one, and then starts what stopped with it; on a plain one it removes keys and sealed copies a cut-off start left. On a site whose config/ is on the data root, the host converger runs that tick before it waits for the data root, so a plain data root left unmounted comes back by itself. The tick also brings /usr/local/sbin/joinery-data-root, the copy a run's unit starts, up to the release's once no run is recorded. A management node offers it on a node page (Encrypt the data root) for a machine in one of our accounts, once the node's recovery key is proven and a verified backup is under a day old; a self-hosted site offers it on its own Backups page, sealed to the site's recovery key.
• What its management node sees. The node page, the enrolment and the address swap's check read the machine's last host report. A data root job that ends (an encryption or its undo once under way, a re-key, an enrolment, a new recipient) queues one at once, and while the report says an encryption, its undo or a re-key is running, the uptime pass asks for one every five minutes (RunNodeUptimeChecks::DATA_ROOT_RUN_REFRESH_SECONDS), so the run's end, and the enrolment it calls for, is seen within minutes.
• Undo. joinery_data_root.sh decrypt decrypts online from wherever it stands (Decrypt (undo) on the node page). Once the node reports its data root mounted and plain, its management node drops what it kept as escrow holder for it (DataRootEscrowSync::from_plain_report(), a retired row excepted), so its page offers Encrypt the data root again. An encryption made again starts with no escrow holder or slot recorded on the machine (encrypt removes /etc/joinery/data-root/holder, written for the old header), and enrols afresh under the same unlock identity.
• A site copy. A dormant copy on a server born encrypted is enrolled with its management node as any machine is (agent 1.75.5 or newer takes data_root_enrol and data_root_recipient while the site is quiet), so the restart an address swap makes opens it by escrow; its escrow row moves with its machine at the switch-over and on the way back (SiteCopySwap). An address swap is refused while the copy's data root has no escrow holder: none named in its last host report, and no enrolled key held here for the identity that report names. A machine the copy lets go (a discarded copy, or the old server of a kept switch-over) has the key held for it retired (DataRootEscrowSync::let_go()), so its holder answers it no more. So does a node removed permanently (Remove Permanently); one hidden from the dashboard keeps its key, as it keeps every record, until Restore to Dashboard.

Keys

• The volume key is random, 512 bits, and never written to a disk. Every open uses --disable-keyring, so it sits in the device-mapper table and root can read it from the open mapping (dmsetup table --showkeys).
• Key slots are PBKDF2-SHA256 at 1,000,000 iterations, about a second on a 1 GB machine (the 512-bit key takes two PBKDF2 blocks); check treats a PBKDF2-SHA256 slot as one the platform made only at a count some release has made its slots at, 1,000,000 or 2,000,000 (LUKS_ITERATIONS_MADE), so a slot added by hand at any other count is named as not ours. A machine with a TPM2 chip has a slot sealed to PCR 7; an escrow machine has a slot whose secret its holder keeps. Every enrolment prunes tokens whose slots are gone (joinery-agent data-root prune-tokens).
• The break-glass copy is the volume key sealed to a recovery public key (the backup envelope's format), beside the image (/srv/joinery.img.recovery). The key it is sealed to is in /etc/joinery/data-root/recovery_recipient, with who set it (set_by=operator|site|install). When the recipient changes, the tick seals the copy again from the open mapping and replaces it; a copy that has gone missing is made again the same way.
• Whose key. A machine in one of our cloud accounts is sealed to the management node's own recovery key, and the management node sends the current one to every such machine after a rotation (DataRootEscrowSync; a new key at once, the same key again at most hourly); its Backups page counts how many are sealed to it. A self-hosted machine is sealed to its site's own recovery key. A machine never takes back a key it was moved off: recipient records each key it replaces in /etc/joinery/data-root/retired_recipients and refuses those, so a management node rolled back to before a rotation cannot re-seal the fleet to the old key. A restore keeps the site's proven recovery key in any case (restore_database.sh).
• The TPM2 chip. On a machine with one, the chip gets a slot (sealed to PCR 7) and opens the volume at boot with no one asked. That keeps a disk taken out of the machine locked, but not the machine taken whole: whoever boots it as it is gets its data. A machine that must stay locked when it is taken asks for its key at every boot instead: create --encrypted --no-tpm2 and encrypt --no-tpm2 never enrol the chip, and joinery_data_root.sh tpm2 off takes its slot away later (through a slot made for the moment from the open volume's key). The choice is recorded as tpm2=off in /etc/joinery/data_root, and every enrolment follows it: create, an encryption's and a rekey's finish, and the re-seal after a boot the chip did not open. tpm2 on clears it and enrols the chip again; during a run the choice is recorded and the run's finish follows it. A decrypt clears it. A site's Backups page offers only the stricter way, encrypting with the key at every boot or stopping the chip opening an encrypted one (RootRequest kinds data_root_encrypt with tpm2: off, and data_root_tpm2_off), because the request queue is the web user's to write; letting the chip back in is root's own command. status and the status file the site reads (tpm2_chip, tpm2) say whether there is a chip and what was chosen.
• A key that may have been seen. joinery_data_root.sh rekey re-encrypts the volume under a new volume key, online, driven by a temporary slot sealed to the recovery key. When it finishes, the new break-glass copy replaces the old, the TPM2 chip is enrolled again, and the escrow slot is made again with a new secret: the machine proves the change to its holder with the old volume key's report key, kept meanwhile at /srv/joinery/.joinery-rekey/. A management node runs one at a time (Re-encrypt under a new key… on the node page); a self-hosted site starts it from its Backups page.

Boot

joinery-data-unlock.service (the agent's unlock mode) opens the volume; srv-joinery.mount follows it, then joinery-data.target and every service that requires it. The agent daemon starts beside the unlock: a site whose config/ is on an unmounted data root starts in site posture, waiting, and restarts to read its config within 15 seconds of the unlock. The host converger, fix_permissions.sh and the outbound limits wait for the data root too.

The unlock tries, in order:

1. The TPM2 chip, up to five times two seconds apart (a boot process extending a PCR during the unseal makes it refuse once, TPM_RC_PCR_CHANGED). A boot it did not open is recorded in /run/joinery/data-root.tpm2-failed, and once something else opens it the tick seals the chip's slot again to that boot's measurements.
2. The escrow holder, every ten seconds.
3. The recovery key: at the console, and on the machine's own unlock page on ports 80 and 443 (503 with X-Joinery-Locked), at once where there is no holder and after a minute where there is. The holder's node page can also unlock it: the recovery key stays in the browser, which works out the one value that opens the break-glass copy.

Locking it by hand. joinery_data_root.sh stop stops the data root and every service on it, closes the volume and lets its loop device go, so the machine is locked as at a boot. The sockets that would start a consumer (syslog.socket, docker.socket) stop first: a log line arriving through one would start rsyslog, which pulls the data root up again and cancels the stop. What it stopped is recorded in /run/joinery/data-root.stopped, the tick leaves a stopped data root alone, and status says STOPPED. The status file (/run/joinery/data-root.status, stopped_by_hand=) and the host report (data_root_stopped) say when, so the management node shows the machine as stopped by hand, a warning, and raises no site-down or locked alarm for it or the site containers on it. A site that stops answering after the machine's last report has the machine asked for a report within two minutes, and those alarms wait for it, ten minutes at most. joinery_data_root.sh start asks for the key as a boot does and starts what the stop stopped once the volume opens; a reboot starts it as ever. Closing the volume tells the escrow holder of the shutdown, so the holder answers the next ask. A filesystem that something will not let go of is started again, and stop names where it is still mounted. A plain data root stops and starts the same way, without the key.

The escrow holder

Usually the machine's management node; the two management nodes hold each other's. Requests go to /api/v1/agent/escrow/{enrol,confirm,unlock,report,preflight,rekey}, signed by the machine's unlock identity (Ed25519, /etc/joinery/data-root/unlock_identity, domain joinery-unlock-v1). Confirm, report, preflight and rekey also carry X-Joinery-Unlock-Report-Mac, an HMAC under a key derived from the volume key, so a copy of the root disk, which has the identity but not the volume, cannot change what the holder records.

• Enrolment is approved at once when the machine's node key vouches for it (its own management node), and by a person comparing fingerprints on the holder's dashboard otherwise; the dashboard names the machine by its site's domain and node (joinery-agent data-root enrol --name overrides). A holder takes new enrolments only where data_root_encryption_offered is on (DataRootEscrow::takes_new_enrolments()): none is made anywhere until the release signing keys are rotated. Enrolments made already keep answering. A machine is not enrolled while an encryption, its undo or a change of key is under way (the mapping may hold either key): the agent refuses data_root_enrol then, and the management node asks at the first report after the run ends.
• The holder's own backup. Rebuilt from a backup, a holder knows only the keys it held then. Within the hour of a change (an enrolment confirmed, a key changed or given up) it backs itself up (EscrowHolderBackup, the Back Up After an Escrow Change task, after ten minutes of quiet), and the node page of a machine enrolled since its newest offsite backup says so.
• Unlock is answered only from the machine's stated or seen addresses (IPv4 and IPv6, compared as addresses), once per boot.
• Tier 2. For a machine in one of our accounts whose key holds events:read_only, the holder reads the provider's activity log before it answers (EscrowTier2) and refuses a boot after a rescue boot, a root-password reset, a rebuild, a clone or image of its disks, a backup restored onto it, or an address taken away, or one asked from an address that is not its instance's. It reads the log twice, a few seconds apart, and refuses while the log cannot be read. A refused machine shows its event on the node page, with two ways on: I did this myself (recorded with the user's name, excusing every event up to then), or a copy from its backups onto a new server. A machine in its owner's own account stays at Tier 1, and its node page says why.
• A key that stops working keeps it at Tier 2. Once a machine is in one of our accounts held by a lasting key with events:read_only, it is Tier 2 whether or not that key works today: a key the provider refuses, one past the expiry the provider reported, or one this node cannot unseal means its boots are refused (it opens by its recovery key meanwhile), never answered unchecked, since revoking this node's key is the first thing a stolen hosting login can do. A 401 while reading the log marks the account refused for the Hosting tab. The daily transfer watch first asks each API key about itself (OperatorCloudAccount::check(), which records the key's own expiry where the provider lists it: Linode's profile/tokens, readable with any scope), the Hosting tab shows the expiry and Expires soon within a week, and plane:cloud_account_key raises an incident on each machine Tier 2 covers: critical when the key does not work, a warning within a week of its expiry. A key whose scopes were never read (a check that has not reached the provider yet) is treated the same way, refused until a check reads them, and raises the same critical incident. Tier 1 by design: a machine in its owner's own account, one in an account held by a two-hour Linode sign-in, one whose checked key lacks events:read_only, and one whose account was removed from this management node. Tier 2 guards against a stolen hosting login; removing an account is an act of this management node's own operator, which it trusts by definition, and the Hosting tab's Remove names the encrypted servers it drops to Tier 1 before it does.

On the site

The data root tick writes /run/joinery/data-root.status (world-readable, nothing secret) for the site to read (DataRootStatus). The Backups page shows whether the data root is encrypted, what opened it, and whose key its break-glass copy is sealed to, with advice when the copy is missing or sealed to a key the site has retired; the recovery key's card on Recovery Readiness says it opens the data root too; VaultHealth adds a row for the break-glass copy. The page asks root through RootRequest (data_root_encrypt, data_root_reseal, data_root_rekey), and root checks every condition again itself.

Commands and files

maintenance_scripts/install_tools/joinery_data_root.sh, run from the site's tree by the host converger; an encrypt, decrypt or rekey run installs a copy at /usr/local/sbin/joinery-data-root for its unit:

The agent's words: data_root_enrol, data_root_encrypt, data_root_decrypt, data_root_rekey and data_root_recipient (operate; script words of joinery_data_root.sh except enrolment). joinery-agent data-root is the script's helper: seal, unseal, reseal, slots, add-slot, prune-tokens, identity, enrol, report, preflight, report-key, escrow-rekey.

Tests: tests/integration/joinery_data_root_gate.sh, plugins/server_manager/tests/data_root_escrow_test.php, escrow_tier2_test.php, provision_one_layer_test.php, tests/vault/vault_health_test.php, and the agent's go test.