Deploy and Upgrade Systems

Every byte in an archive is then derivable from those commits. What ships is a file the commit holds or one publish builds itself (ReleaseCommit::joineryFileShips()): the agent bundle under public_html/agent_dist/, the relay sealer binaries under the mailbox plugin's provisioning/bin/, the mail gateway binaries under the mail_apps plugin's provisioning/bin/, the license files copied into public_html/, and each release statement. A file git ignores never ships, wherever it sits: the manifests list only shipping files, the component tree hashes that decide version bumps read only them, and every archive is cut from its own manifest's listing (TreeManifestPublisher::archiveMembers()) rather than from a directory. The rest is generated deterministically:

• Install SQL — create_install_sql.php writes no timestamp, passes a fixed --restrict-key to pg_dump, seeds the admin row with the locked marker ! (no plaintext ever existed; _site_init.sh sets the real password), and compresses with gzip -n. Two runs against one schema are one byte string, and the file is committed.
• Go binaries (agent, relay sealer, mail gateway, parser-jail launcher) — each go.mod pins toolchain goX.Y.Z, and the build uses the official toolchain of that version and nothing else (GoBinaryPublisher::officialGo). The box's own Go only fetches it: asked with GOTOOLCHAIN=goX.Y.Z, Go downloads the golang.org/toolchain module, checks it against Go's public checksum database and unpacks it in the build's module cache; a Go that is itself the pinned version, such as a distribution's patched build, is refused, because it compiles different bytes. The build then runs that toolchain with GOTOOLCHAIN=local. With -trimpath, CGO_ENABLED=0 and -buildvcs=false a build is bit-identical on any machine. agent_dist/manifest.json records the toolchain.
• Support bundle — `tar --sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner, normalised modes, gzip -n`: one staging tree, one tarball.

Tests: tests/unit/release_commit_test.php (the clean-tree rule, what ships and the archive cut from its manifest, the toolchain pin, the key lists) and tests/core/install_sql_deterministic_test.php (two runs of the generator, byte-identical).

The release statement and the public log

Every release the publishing site authors is written to Sigstore's public Rekor log before it ships, as one signed release statement: the version, both commits, the Go toolchain and compressor versions, the hash of every artifact's manifest, every agent binary's raw sha256, the support bundle, the relay sealer and mail gateway hashes, and every key the release installs.

Watching for Sigstore's log changeover. Sigstore moves to a new log about once a year and lists it months ahead. Every node must receive the new log's key from a release logged before the changeover. The Watch Release Log task (hourly; it asks Sigstore once a day, and every hour while anything is wrong) runs the same discovery and stores its conclusion in the server_manager_release_log_watch setting. Reconcile Incidents raises, on this node only:

• plane:release_log — a future log whose key nodes do not hold yet: not published by Sigstore, not pinned here, or pinned but not yet shipped by a logged release (warning; critical inside 30 days of the changeover), or any other reason a publish would be refused (critical);
• plane:release_log_blind — the watch cannot conclude: Sigstore unreachable or answering in a format the client cannot read, no conclusion for three days, or the task missing or turned off (warning; critical after two weeks). A run that cannot conclude never clears plane:release_log.

• plane:release_log_entry — an entry under our key that the ledger does not hold, or holds with other bytes; it names each entry's log, index and entry bundle (critical);
• plane:release_log_tail_blind — a log nodes trust not read up to a settled checkpoint for a day, no completed check against Sigstore's trusted root for three days, or the task missing or turned off (warning; critical after a week).

The reader checks each checkpoint's signature but does not hash the entries it reads back to the checkpoint's root: a log showing it different entries than it proves to nodes is a split view, which only witnesses catch.

The public releases page. The site that logs releases serves /server_manager/releases, with no login: every logged statement, newest first (version, date, both commits linked to GitHub, its log entry, when the reader saw it, whether it shipped, and the keys it installs on nodes, each log key marked with whether Sigstore's trusted root lists it), how far each log nodes trust has been read and when it last caught up, any unaccounted entry or unlisted key at the top, and the verify_release command for the newest release with this site as its source. Any other site answers that it does not publish releases. Its logic (releases_logic) is also an API action that needs no session. Test: plugins/server_manager/tests/releases_page_test.php.

Tests: tests/unit/release_log_client_test.php (discovery, the shard rules, the recorded log answer), tests/unit/release_statement_test.php (the listed statement, the payload, the key chain and its walk, the statement key, logging end to end against a log built in the test, the watch and its incidents, the publisher's order) and plugins/server_manager/tests/release_log_entries_test.php (the record of logged statements, spent versions, the unfinished-row guard) and plugins/server_manager/tests/release_log_tail_test.php (reading a log built in the test: where it starts, the settled checkpoint, our entry matched, a stranger and a changed entry unaccounted for, the budget, blind).

Preparing a commit, and publishing without an agent session

Sessions share one working tree and one git index, so a commit is prepared in a private index per session (.git/work-commits/NAME/) and landed by the owner with work NAME --commit. work is the owner's session launcher on the dev box; its source is a local-only repository at ~/work-tool and ships nowhere. It never types into a session, so preparing a commit does not depend on which assistant, if any, is at the other end.

Preparing. Any assistant, or a person at a terminal, prepares a commit in four steps:

1. Find the files for the task: git status --porcelain and work --owners. Leave out files another running session owns.
2. A finished spec moves with its work: set its **Status:** line to IMPLEMENTED <date>, mv it from public_html/specs/ to public_html/specs/implemented/, and list both paths. A file already in implemented/ is not edited.
3. Write the message to a file. The subject is imperative, has no trailing period and is 72 characters or fewer; then a blank line; then plain - bullets. No double quote or backtick anywhere (both break the shell here-documents a release commits with), and no trailer such as Co-Authored-By. New files and version bumps are named.
4. Stage it: work --prepare-commit --name NAME --message-file FILE -- PATH... stages the paths into NAME's private index, checks that they differ from HEAD, and writes nothing to the shared index. It refuses a message that breaks the rules above and drops an agent's attribution lines from the end.

Landing. work NAME --status shows what is prepared, laid out like git status; work NAME --diff [PATH...] shows the changes; `work NAME --commit` shows the commit, asks, commits, and offers to push. With nothing prepared, --commit exits at once and says how to prepare.

What blocks a release. work --held lists the uncommitted files a publish would refuse (the held paths above), grouped by the session behind each, with the prepared commit that already holds one; work NAME --held lists one session's. It asks ReleaseCommit::ownerBlockers() through maintenance_scripts/dev_tools/held_blockers.php, which prints JSON, so the rule is the publisher's own. It exits 1 while anything is held, so a script can gate on it.

Emergency deploy. A fix that must ship while no agent session can prepare it follows the same path with a person at the terminal:

1. work --held. A held file that is part of the fix goes in the fix's commit. One that is another session's finished work is prepared and committed under that session's name (work SESSION --prepare, then work SESSION --commit). One that is an unfinished draft from an idle session is moved out of the tree or reverted before the publish; it is not committed.
2. Prepare and land the fix: work NAME --prepare ..., then work NAME --commit, and confirm the push.
3. php tests/run.php db --changed, and work --held exits 0.
4. Publish: `php plugins/server_manager/includes/publish_upgrade.php "release notes", or Publish Upgrade at /admin/server_manager`.
5. Roll out from the node's Updates tab at /admin/server_manager. A node that does not come back healthy is diagnosed from its detail page and the management agent first, and over SSH only after those.

The site container engine

Container hosts run our own build of Podman with netavark and aardvark-dns (specs/site_container_engine.md), not Ubuntu's package. maintenance_scripts/install_tools/site_engine/versions.env pins each part by upstream tag and commit, and the official Go toolchain Podman is compiled with. When versions.env or build.sh changed since the engine last published, publish builds it (SiteEnginePublisher, about twelve minutes) in an Ubuntu 26.04 build root at /var/lib/joinery-site-engine-build/resolute. The owner makes that root once, as root, on the publishing box:

sudo bash maintenance_scripts/install_tools/site_engine/build_root.sh create

build.sh refuses its own output unless the binary carries the pinned commit, the pinned Go and the seccomp and apparmor build tags. The archive, joinery-site-engine-<version>-<arch>.tar.xz, is published under static_files/site_engine/; publish writes its name and SHA-256 into site_engine/ENGINE and commits that file with the release, so the signed manifest covers it. A build owed and not done (no build root, a failed build) refuses the release.

A site that republishes what it received does not build: it fetches the archive its received ENGINE names from its upgrade_source, keeps it only when the hash matches, and serves it from its own static_files/site_engine/; one it cannot serve refuses the republish. A host installs the engine with install_site_engine.sh converge, which reads ENGINE from its own signed tree and fetches and installs the archive only when its current engine is not that one.

Tests: tests/integration/site_engine_gate.sh (the host installer and converge, run against a fixture root) and tests/unit/site_engine_publisher_test.php (build when owed, publish, the republishing site's fetch).

Distribution Architecture

Updates are distributed as separate archives:

• Core archive (joinery-core-X.XX.upg.zip) - Main application without themes/plugins
• Theme archives (theme-THEMENAME-X.XX.upg.zip) - Individual themes
• Plugin archives (plugin-PLUGINNAME-X.XX.upg.zip) - Individual plugins

• Independent versioning of themes and plugins
• Selective updates (update core without touching themes)
• Smaller download sizes for incremental updates
• Third-party theme/plugin distribution

Quick Reference

install.sh

Location: /var/www/html/joinerytest/maintenance_scripts/install_tools/install.sh

Universal installer for Docker and bare-metal deployments. Supports --themes flag to download published themes/plugins from the upgrade server after site creation (extensions whose manifests have included_in_publish: true).

Full documentation: Installation Guide

build_dev_from_source.sh

Location: /var/www/html/joinerytest/maintenance_scripts/install_tools/build_dev_from_source.sh

Note: This script is functional but not recommended for production. Use upgrade.php for production deployments. build_dev_from_source.sh is suitable for development environments where git-based deployment is convenient.

# Basic deployment
./build_dev_from_source.sh joinerytest

# Verbose mode (recommended)
./build_dev_from_source.sh joinerytest --verbose

# Disable auto-rollback for debugging
./build_dev_from_source.sh joinerytest --norollback

# Manual rollback
./build_dev_from_source.sh joinerytest --rollback

Features:

• Git-based deployment from repository
• Pre-deployment validation (PHP syntax, bootstrap tests)
• Automatic rollback on failure (trap-based)
• Preserves extensions marked receives_upgrades: false
• Composer integration and database migrations

upgrade.php

Location: /utils/upgrade.php

Web Usage: an upgrade source answers ?serve-upgrade=1 (what it offers, as JSON) and ?serve-verify-key=1 (its release verification keys). Any other browser request is sent to the Updates page (/admin/admin_updates, below), where an update is asked for and watched.

CLI Usage:

# Basic upgrade
php /var/www/html/joinerytest/public_html/utils/upgrade.php

# Verbose mode
php /var/www/html/joinerytest/public_html/utils/upgrade.php --verbose

Features:

• Downloads packages from upgrade server (configured via upgrade_source setting)
• Downloads core, themes, and plugins as separate archives
• Pre-deployment validation via DeploymentHelper
• Preserves extensions marked receives_upgrades: false
• Enhanced rollback (preserves failed deployments with timestamps)
• Database migrations and composer integration
• Install history — every command-line run that takes the upgrade lock writes a ReleaseInstall row (rin_release_installs) as it exits: the version it started from, the version it was installing, and the outcome — installed (with the core and agent commits and the log entry the release's statement names), refused (a verification refusal: every upgrade_abort title that starts Upgrade refused), rolled_back (with the step), or stopped (anything else, with the reason). The Updates page lists them. Recording never changes a run's outcome; a failure to record is one line in the transcript.
• Declared-dependency install — after the file swap, installs any PHP extension the new code declares (root composer.json ext-* + plugin requires.extensions, resolved by utils/list_dependencies.php --apt) and reloads web PHP. Then runs the core host installers and every active plugin's declared host_installer via _plugin_installers_start.sh so new host requirements land with the deploy. Both need root, which every run has: the agent's apply_update, a docker exec, an operator's shell, and the Updates page's request, which the host converger (below) carries out as root.
• The host converger (specs/implemented/host_converger.md) — a root timer every site install leaves behind (install_host_converger.sh, a core host installer: a systemd timer where PID 1 is systemd, a cron.d entry otherwise) that runs _plugin_installers_start.sh --when-changed every minute. Nothing runs until the deployed VERSION, the runner, its installers or the set of active plugins changes, then everything converges; once a day regardless. A tick while the site's upgrade holds uploads/.upgrade.lock does nothing: the upgrade runs the installers itself once its tree is finished, and a tick in between would converge a half-installed tree. Every other tick first holds the agent running where its switch is on: under systemd, an agent that is stopped, or whose unit is not enabled, while /etc/joinery-agent/enabled is absent or 1 is started and enabled within the minute, on a site and on a machine with no site alike (agent_running_hold). Every run records cache/host_converger.last; a converger silent for a day is named in the admin header (HostConvergerNotice) and in VaultHealth with the reinstall command. On a managed node the same job any apply_update or run_plugin_installers runs reinstalls it; an unmanaged box that will not open a shell can pair to a management node for that one job.
• One runner at a time — whatever invoked it (the timer, a cron tick, install.sh, upgrade.php, an agent job, an operator's sudo), the runner holds a kernel flock from before its first change to the host until it exits: as root on /run/joinery/host-installers.<site>.lock, a file only root can create, so nothing the web user can do blocks the run that takes the tree back from the web user. After taking it the holder writes its pid and start time (UTC unix) into the file. A run that finds the lock held waits for it, ten minutes at most (flock -w; long enough for a full converge or an upgrade's installer run, far under the service timeout below), then takes it and does its own work, its record replacing the holder's; the timer's tick holds the lock for about a second, so a run that lands on a tick waits a moment and runs. Only when the wait runs out does it print one line — another run holds the lock (pid N since T) - waited 600s, or holder unknown when the record is unreadable — record nothing, and exit 0. The lock cannot go stale (a kernel flock dies with its holders, and every child the runner starts inherits it), but a hung installer holds it for as long as it lives; the timer's oneshot service carries TimeoutStartSec=1h, and systemd's default KillMode=control-group kills the installer and its children together, releasing the lock.
• One installer — _plugin_installers_start.sh --only=<name> runs that one core installer (a name from CORE_INSTALLERS; anything else is refused with exit 2 before the lock is taken) under the same lock and after the ownership assertion, and nothing else: no converge stamp, no host_converger.last, no PHP extensions, no plugin installers, no root requests. It prints the same transcript lines as the full run, and an installer that fails still exits 0, so a caller reads the transcript and checks the host, never the exit code. This is the path an agent's host_converge word takes on a site.
• One plugin's installer — _plugin_installers_start.sh --only-plugin=<plugin> runs one active plugin's declared host_installer, and nothing else, through the same checks a full run applies (path inside the plugin, root put it there, a package we built). A name that is not a plugin identifier, or --only-plugin beside --machine, --when-changed or --only, is refused with exit 2; a plugin that is not active, or declares no installer, is refused in the transcript. This is the path an agent's run_installer plugin:NAME takes.
• The site's own files — site_housekeeping.sh, a core installer, writes the site's /etc/logrotate.d/joinery-{site} and, on bare metal, its /etc/cron.d/joinery-{site} entry, each only when absent; _site_init.sh calls it at install. host_housekeeping.sh likewise writes mpm_event.conf, the journal cap and the PHP-FPM php.ini (rebuilt from php.ini-production) when absent, from _host_files.sh, the definition install.sh uses too. Absent-only, so an owner's tuning survives every converge; the agent's reclaim_managed_file moves one aside (keeping a dated copy) and runs its owner to put it back.
• The quiet state — a site is quiet while root's /etc/joinery/sites/<site>/state exists, holding quiet copy (a dormant copy; only install.sh site --dormant --copy-of=ID writes it) or quiet switchover (a source frozen for a switch-over; only sysadmin_tools/site_quiet.sh on, the agent's site_quiet word, writes it). The runner reads it first, right after taking the lock, whatever mode it was given: while it exists, the runner asserts the quiet measures, records quiet-copy or quiet-switchover in host_converger.last, keeps no converge stamp, and runs no installer, core or plugin, and no root request. The measures are the machine's, in install_tools/_site_state.sh, and no line of the site's own code asks about them: (1) the nftables table inet joinery_site_state rejects every outgoing connection from the web user and from Postfix's own client except over loopback, loaded from /etc/joinery/sites/<site>/quiet.nft by joinery-site-state.service at boot (before Apache, cron and Postfix) and again on every run, so a table deleted by hand is back within the minute — root is not blocked, so the agent, a backup run and a restore still reach storage, and the site census (site_census, run as root) asks the file bucket about a sample of offloaded files; (2) conf-enabled/joinery-quiet-<site>.conf answers every request on every host with a 503 — under copy, a secret cookie (set by /.joinery-look/<secret>, the secret in look_secret, root-only) lets the owner look, and under switchover everyone gets the maintenance page; (3) the site's /etc/cron.d file, and certbot's if there is one, move into held/ and nothing writes them back; (4) certbot's timer is off; (5) every Postfix smtpd service answers 4xx (smtpd_client_restrictions=defer on its master.cf entry), so senders keep their mail and retry. held/ records what quiet took away — the cron files as they were, whether certbot's timer was on, each smtpd service's own setting — and clearing puts back exactly that. A source frozen for a switch-over whose address moves by DNS forwards while resolvers let go of it (site_quiet.sh forward ADDRESS, the agent's site_quiet word with forward): /etc/joinery/sites/<site>/forward names the copy's address, the drop-in proxies every request there over https with the copy's certificate checked against the site's own name (the ACME challenge path over http, so the copy can be issued one), and the table lets the web user reach that one address on 80 and 443; in a container the host's table lets it (site_quiet_host.sh forward SITE ADDRESS). site_quiet.sh on stops forwarding, and clearing removes it. site_quiet.sh off clears quiet switchover, and clears quiet copy only with --copy-promoted, which the agent passes only once the machine holds the node id recorded in copy_of. Once the state file is gone, a run that finds the measures still in place clears them, and with no stamp the first run after the state clears converges in full: the cron file is written if none came back, and every installer runs. On bare metal one site is on a machine, so a quiet site is a quiet machine. In a site container (either engine's marker; specs/site_container_engine.md D9) the machine's measures are its host's: the host keeps its own record, /etc/joinery/sites/<site>/quiet on the host (copy, written only by the dormant install; switchover, by the site_quiet_host word), and sysadmin_tools/site_quiet_host.sh loads the same table into the site's own network namespace, refusing every account but the container's root (meta skuid != { 0, BASE }), where the site, holding no CAP_NET_ADMIN, cannot change it. The site container engine's OCI hook (/etc/containers/oci/hooks.d/joinery-site-quiet.json, at createRuntime) loads it before a recorded site's first process and stops the start when it cannot; host housekeeping installs the hook and asserts every recorded site's table on each run. The container keeps measures 2, 3 and 5 itself, with no boot unit, nftables or certbot timer of its own; its start command asserts them before cron and Apache start, and a cron file it could not hold stops the start. The host lifts a copy's table (site_quiet_host.sh off) only once the copy's own state file is gone; a switch-over's is lifted first on the way back, while the site still shows its 503. A copy's host record is written once, by its install, through the host's installed copy of the script, and a site its host records quiet is never rebuilt.
• A machine with no site — _plugin_installers_start.sh --machine is the runner on a host that has no site tree: a Docker host, a relay. Its root is the agent's verified support bundle (/opt/joinery-agent/tree, whose layout is a site root's; --site-root= overrides it for a fixture), its name is host (so root's lock is /run/joinery/host-installers.host.lock), and its set is HOST_INSTALLERS — host_housekeeping.sh and install_host_converger.sh, each called as <installer> --machine ROOT. Nothing a site has and a bundle does not runs: no ownership assertion, no permissions pass, no secrets, no release keys, no PHP extensions, no plugin installers, no certificate summary, no root requests. --when-changed hashes the bundle's stamp beside the tree instead of VERSION, and root keeps the stamp and last-run record under /var/lib/joinery/host, which a bundle refresh does not replace. --only= under --machine names a host installer only; a site name is refused with exit 2. The mode is an argument, never derived from where the copy lives. The machine's timer (install_host_converger.sh --machine ROOT) runs the entry point with --when-changed --machine --site-root=ROOT, logs to the journal on a host with a data root (/var/log/joinery/host_converger.log on one without), has no path unit (no web user queues a root request there), and leaves a unit a site already owns alone. This is the path an agent's host_converge word takes on a siteless machine, and what the agent runs once on its own when a new support bundle lands, whether its management node sent it or install.sh docker installed the release's own (joinery-agent bundle install).
• Graceful handling of missing archives — if a theme or plugin archive returns 404, the upgrade warns and skips it instead of aborting. The core upgrade and all other themes/plugins proceed normally. A summary of skipped items is shown at the end.
• Post-deploy smoke test — after the new code is in place and migrations have run, the deploy test tier runs against it. A failure restores the previous release as one unit — public_html from public_html_last, and the maintenance_scripts and RELEASE_MANIFEST(.sig) the deploy replaced from public_html_last/.release_last, where the deploy keeps them before it touches either — and preserves the broken tree for diagnosis. The site root never holds one release's manifest over another's code, so the agent, which verifies every script it runs against that manifest, can still run upgrade.php and take the node forward. This is the first thing on the node that reads a line of the code being installed. The tier takes a couple of seconds, is entirely reads, and asks only whether the code runs on this machine — every deployable PHP file compiles, the core classes load, every active plugin's bootstrap loads, the database answers, the declarative manifests parse, and the site returns a page over HTTP. See The deploy tier below for why it is not safe. The rollback returns the code but not the schema, because migrations ran first; schema changes are additive so the previous code normally runs against them, but the output says plainly that this is a recovery rather than a clean undo, and the node should be upgraded forward rather than left there.
• Site-root maintenance_scripts/ — the core archive carries install_tools/ and sysadmin_tools/ alongside public_html/, and the upgrade syncs them into the site root after the deploy swap and before fix_permissions.sh runs, so a release applies as one piece rather than with its own tooling a version behind. The previous copy, with the previous manifest pair, is kept under public_html_last/.release_last first (DeploymentHelper::keepPreviousRelease), so a rollback restores them with the code; it shares the backup's lifecycle — cleared before a deploy, removed after a completed one, never archived. The sync compares by content (rsync --checksum), because the staged files carry the publishing box's timestamps while the node keeps its own and a same-size edit can otherwise look unchanged. It does not delete wholesale: a node can legitimately hold scripts the archive does not ship. What it removes is exactly the files a release stopped shipping — each maintenance_scripts/ path the previous release's signed RELEASE_MANIFEST (still at the site root when the sync runs) lists and the new one does not. Two scripts retired before signed manifests existed, install_tools/deploy.sh and install_tools/_reconcile_stock_assets.sh, are removed by name for now. A file no release listed is local and stays. The removal runs after the release has passed its deploy tier, so an upgrade that rolls back removes nothing. *.sh are made executable afterwards. A failed sync is a warning, not an abort — public_html is already live — and it says explicitly that the node's backup, restore and permission scripts are still the previous version.

A site other sites upgrade from takes everything its source publishes. A release-serving site such as getjoinery republishes what is on its own disk — the publish walks plugins/ and theme/, not the plugin registry. So when DeploymentHelper::servesReleases() is true (an upgrade server that has published at least one release, the same condition under which its ?serve-upgrade answers), its upgrade downloads every plugin and theme the source publishes, installed there or not, and the transcript names the ones it did not have. They land on disk only: the sync registers them, nothing is activated, and no tables are created. Without this, a plugin missing from that site's disk is missing from every release it serves, and each downstream site running it keeps its old copy (marked stale) against the new core.

What root verifies before a deploy

Every archive the publisher ships carries a signed listing of its files: RELEASE_MANIFEST (the sha256 of every shipped file, paths relative to the site root) and RELEASE_MANIFEST.sig, an Ed25519 signature over it. A node holds the public key(s) in config/release_verify_keys — root's, 0644, one base64 key per line. Until the node has recorded its trusted set (below), the host converger writes it on every tick from the agent bundle's manifest.json in the tree (signing_public_key and release_keys), and install.sh from <upgrade source>/utils/upgrade?serve-verify-key=1 on a box whose tree ships no bundle; the upgrade writes it from the same bundle when it finds none. serve-verify-key answers, on the publishing box, with the release keys its repository lists (release_keys/release/), and on any other site with the set it holds.

PackageSignature::verify() (includes/PackageSignature.php) reads the pair back and answers with one verdict. signed means the signature verifies against a key in the file, every file present is listed with a matching hash, and every listed file is present. Everything else is its own verdict with a sentence: unsigned (no manifest or signature), unknown_key (signed by a key this node does not hold), no_keys (no usable key on this node), tampered (a file's bytes differ from its signed hash), extra_file (a file, or a symlink, beside the signed set), missing_file, unreadable (not the format the publisher writes, a manifest describing a different directory, or a signature file holding anything but one signature), unauthorized_keys (a logged release that changes the keys this node trusts without being signed by the keys allowed to change them). Paths a manifest never lists — .git, cache, logs, uploads, backups, specs, .claude, node_modules anywhere; config and vendor at the site root only; .gitignore and the manifest pair themselves — are neither required nor counted against a package in a live tree. A plugin's own vendor/ and config/ are listed and checked.

A fresh archive (staging during an upgrade, an upload, a marketplace download) is checked with fresh: a file on one of those paths is extra_file, since an archive has no reason to carry one. The one allowance is the core archive's config/default_Globalvars_site.php, which must be the listed maintenance_scripts/install_tools/default_Globalvars_site.php byte for byte. The publisher tars every archive without those paths.

The release log. A node that holds config/release_log_required installs only releases written to Sigstore's public log: signed also needs the package's RELEASE_STATEMENT to verify offline against keys the node already holds — config/release_statement_keys (P-256, one base64 key per line) and config/transparency_log_keys (<origin> <base64 key> per line). The statement's envelope, its log entry, the log's checkpoint and the inclusion proof are all checked, the statement must record this manifest (PackageSignature::statementSubject()), and every RELEASE_STATEMENT the manifest lists must be that statement. Anything less is unlogged. The statement's key_chain is walked in memory, from the keys held, before the manifest's signature is checked, so a node that missed the release carrying a new release, statement or log key still verifies.

The trusted set and its authority. The release and statement keys a node trusts are one set, the one the last release that changed it installed; each statement names, beside the keys it installs (keys_installed), its authority: the release and statement keys allowed to sign the next change (release_keys/authority in the repository; without that file, every listed key). A statement from before the field has the whole set as its authority. Walking the chain, a link that does not verify under the statement keys held adds nothing; one that installs the set already held adds the log keys it installs; one that installs a different set replaces the set only when its statement is signed by a statement key of the authority held, and otherwise adds nothing. The manifest's signature is then checked against the release keys the walk ends with. A release whose own set differs from that one must be signed, statement and manifest, by keys of that authority, or it is refused whole as unauthorized_keys. So a key is retired by a release, signed by the authority, that leaves it out, and once the authority has moved to new keys the old ones cannot change the set back. A manifest signed by a key the node has retired is unknown_key, and says the key is retired. The set a verdict leaves comes back on it (trust, and keys_proven['trust'] for the node's own key files) with any log keys the walk proved; the root callers — the upgrade, install_extension.php, verify_package.php, the extension manager — write them (PackageSignature::persistProvenKeys()): log keys appended, the set replacing config/release_verify_keys and config/release_statement_keys, and recorded in config/release_trust (the set, its authority, the release that set it, and the keys it dropped, kept only to name them). An installer writes the log keys at the verdict, before it installs, and the set only once the code is in place — the upgrade after its deploy tier passes, install_extension.php and the extension manager once the package is placed — so an upgrade that rolls back keeps the set its code is signed under; a set an installer did not write, root's next verify of the live tree writes: verify_package.php --tree, the code check each housekeeping tick on a host that refuses unsigned code, writes the live core's set, and the plugin installers' run, at least daily, writes each plugin's. The verifier itself never writes. Log keys are Sigstore's, not ours, and only ever grow. The release-log checks live in includes/TransparencyProof.php, which is not in the self-update set (the re-run after a self-update is one-shot); a node whose live tree lacks it loads the staged copy, as root, only after its sha256 matches the core archive's signed listing. An upgrade that keeps a plugin with a host installer and no release statement (a local fork from before the log) says in its transcript that the installer will not run on a node that requires the log. The agent's self-update makes the same checks on its own next binary, against keys compiled into it and kept under /etc/joinery-agent/ (Server Manager overview, agent self-update).

The host converger writes config/transparency_log_keys from the bundle's log_keys, and config/release_statement_keys from its statement_keys until the node holds config/release_trust: from then the release and statement keys are that record's, changed only by a verdict, never by a bundle manifest any installed release could have put in the tree. It creates config/release_log_required once it holds a key of each kind — never before, since a node required to check the log with nothing to check it by would refuse every release. It never removes the file. A fork owner publishing their own releases opts out by creating config/release_log_optional by hand; nothing writes it, and the verifier ignores one that anyone but root or the tree owner could have written. These files and config/release_trust are pinned root:root 0644 by fix_permissions.sh, and a trust record anyone else could have written is not read.

The upgrade verifies at every point where bytes would otherwise move from the archive into the live tree, and before anything reads them:

• the core archive, as a whole, right after extraction into staging and before the self-update copies any of its files into live;
• the staged core again on the re-run after a self-update, because staging lives under uploads/ and could have changed in between;
• each theme and plugin archive, after its extraction into staging, with the listing required to describe exactly public_html/theme/<name> or public_html/plugins/<name>.

Deployment self-update (read before editing upgrade.php)

A release can change the deployment tooling itself, so before deploying anything the upgrade compares five files between the staged archive and the live site:

utils/upgrade.php
utils/update_database.php
includes/DatabaseUpdater.php
includes/DeploymentHelper.php
includes/PackageSignature.php

Any that differ are copied to live immediately and the pipeline re-executes from the start, so a release is applied by its own tooling rather than by the previous version's. The comparison runs from the fully staged tree, after the whole archive has verified; nothing is copied out of an archive before that.

All five move as one set, never individually. They call each other, so refreshing a subset produces a new file running against an old API — which is the same breakage the self-update exists to prevent, arrived at from the other direction.

The rule this imposes: during that window the new upgrade.php is running against the old core — every file outside the set of five is still the previous release. Anything those five call must therefore already exist on the oldest site expected to upgrade, or travel inside that same set. The verifier is the one member that can be absent on the re-run (a node whose previous release predates it): upgrade.php then loads includes/PackageSignature.php from the staged release for that run and says so, which is the same trust the previous upgrade.php placed in the archive when it copied the deployment files out of it.

publish_upgrade.php

Location: plugins/server_manager/includes/publish_upgrade.php Access: Requires the Server Manager plugin to be active. Superadmin only (permission level 10).

Preferred usage: Use the Publish Upgrade form on the Server Manager dashboard (/admin/server_manager). Enter release notes and submit — the plugin dispatches a publish_upgrade job to the management node's own agent, which builds all archives as root. The management node is a node of itself for this: on its own Management Node page (/admin/admin_management_node) it connects to its own URL, and the request is approved at the top of its own Server Manager dashboard — no shell involved (sudo /usr/local/bin/joinery-agent join --management-node=<own URL> files the same ask from a terminal). The form says so when that has not happened yet.

Auto-notes. With the Joinery AI plugin active, an Auto-notes button under the release notes box asks the model Joinery AI selects for this site (its selection policy, under its monthly token ceiling) for a few short lines for a general audience, and puts them in the box to edit before publishing. A site that mints its own version summarizes the commits since the release it last published — core commits after that release's recorded core commit, agent commits after its agent commit, without the Release x.y.z commits. A site that republishes what it received summarizes the release notes of the upstream releases it carries: it asks its upgrade source (utils/upgrade?serve-release-notes=1&after=<the newest version it published before>&through=<the version it runs>) for the notes of each release in that range. Nothing is saved until the form is submitted (ReleaseNotesDrafter, action server_manager/release_notes_draft).

A publish ends by queuing a run_plugin_installers job on the management node's own paired agent, so the host installers a release ships (the parser jail launcher, the agent artifact, a plugin's services) converge on this machine the way an upgrade converges them on every other node; the management node's tree is the source and is never upgraded, so nothing else would run them here. A site with no agent of its own is told the one command to run instead.

The agent runs only scripts that match the signed release manifest, and a publish is what signs that manifest, so the one file the button cannot carry is the publisher itself. After an edit to publish_upgrade.php, the first publish runs from a shell under sudo (below); it signs the changed publisher into the new release and the button works again. The form detects this state and shows the command.

CLI usage (root, because the signing key at config/agent_signing_key is 600 root:root):

sudo /usr/bin/php plugins/server_manager/includes/publish_upgrade.php "release notes here"
# Auto-detects the next version number

A publish that runs as root gives every file it creates the owner and group of the file's parent directory before it exits, so the tree is left as it was found apart from the new release.

Note: The legacy location utils/publish_upgrade.php still exists for backward compatibility during the Phase 1 transition. It will be removed in a future release once all remote nodes have been upgraded.

Features:

• Creates separate archives for core, themes, and plugins
• Core archive excludes theme/ and plugins/ directories
• Each theme/plugin with included_in_publish: true gets its own versioned archive
• Prevents overwriting existing versions
• Refuses a release whose tree fails its tests (see Publish refusals below)
• Automatic cleanup on failure
• Registers upgrade in stg_upgrades table

static_files/
├── joinery-core-3.26.upg.zip        # Core application
├── theme-falcon-3.26.upg.zip        # Falcon theme
├── theme-default-3.26.upg.zip       # Default theme
├── plugin-bookings-3.26.upg.zip     # Bookings plugin
├── plugin-controld-3.26.upg.zip     # ControlD plugin
└── ...

Component version integrity:

Each publish records a per-release snapshot of every published component's (version, tree_hash) in upg_upgrades.upg_component_state (JSON, keyed by themes/plugins). The snapshot of the most recent release row that carries one is the baseline for change detection; rows without a parseable snapshot are skipped, so an aborted publish never poisons the baseline. When no prior snapshot exists, the run re-baselines — records everything, bumps nothing.

The tree_hash is a deterministic, git-independent SHA-256 of the component's working tree (.git/ and .gitignore excluded). The manifest (theme.json / plugin.json) is hashed with its version member removed, so the hash measures content-minus-version.

Per component, before archiving, the publisher applies a four-way decision against the baseline entry:

1. No baseline entry — first publish of this component: record as-is, no bump.
2. Manifest version higher — author bumped deliberately: respect and record.
3. Manifest version lower — record and archive as-is, with a warning line in the publish summary naming the component and both versions. Not aborted: a backward component version has no destructive effect at publish time, and any resulting depends violation is caught fail-closed at activation.
4. Manifest version equal — compare hashes. Equal → unchanged, carry the entry forward. Differ → auto patch-bump the manifest's version (targeted string edit, no reformatting) and archive under the new version.

Publish log:

Every publish writes the full text of its run to {site root}/logs/publish/publish-{version}-{YmdHis}.log, outside public_html so it is not web-served. The log is written by a shutdown handler rather than at the end of a successful run, so a publish that exits early — a refusal, a die on a missing prerequisite, a fatal error — is captured too. Those are the runs worth reading: a stage that reports a problem through the publish summary and continues leaves no other trace once the page or terminal is gone. The header records the version, whether the run was CLI or web, and which account it ran as; a fatal error is appended to the end. The newest 20 logs are kept and older ones removed on each publish. A successful run names its log path as the last line of the summary.

Publish refusals:

These conditions stop a publish before anything is written — no VERSION change, no archives, no release row:

1. The requested version is lower than the one already in the VERSION file.
2. LICENSE.md is missing or empty (every archive carries the license text).
3. A default install bundle names a plugin this release cannot carry.
4. The release fails its own tests. The publisher runs the deploy tier (every deployable file compiles, the core classes load, the database answers, the manifests parse, the site answers over HTTP — seconds, all reads) — see The deploy tier. Where the publishing site has a release staging site, the tier runs there, on the commit being released: publish copies the commit's public_html/, maintenance_scripts/install_tools/, maintenance_scripts/sysadmin_tools/ and root license files into it and runs that copy's own test runner, so the pages are served by Apache from the code that ships, and a file another session has left half-edited in the working tree neither refuses the release nor hides a file the commit lacks. Without one, the tier runs on the working tree. The staging site is <site root>_test, served by the rendered vhost's test-site block as test.<domain>. maintenance_scripts/dev_tools/setup_release_staging.sh, run once as root, makes it: its config requires the site's own config (the same database) and overrides only the paths, webDir (http://test.<domain>) and protocol_mode (http), and it answers 127.0.0.1 and ::1 only, since the vhost is plain HTTP on every address. Publish uses the directory only when its config/release_staging marker is present, so a real companion test site there is never written to. The development tiers are not a publish precondition: they are the developer's working loop and pre-checkin gate (Testing), and the publisher runs as root inside the plane's own publish_upgrade primitive, where they cannot run. Per-suite PASS lines are left out of the publish output; failures, skips and the summary are kept. There is no flag to skip the gate: the fix is the failing test or the code it caught.
5. This box holds agent source newer than the bundled agent artifact and the rebuild fails (see Server Manager).
6. A Go program a plugin ships prebuilt cannot be built: the mailbox plugin's relay sealer or the mail_apps plugin's mail gateway (GoBinaryPublisher::PLUGIN_PUBLISHERS).

publish_theme.php

Location: plugins/server_manager/includes/publish_theme.php, served at /admin/server_manager/publish_theme (and the legacy utils/publish_theme.php). No session is asked for: the catalog and the downloads are public.

# Catalog, read by the marketplace and upgrade.php (&site=DOMAIN scopes audience-limited entries)
https://yoursite.com/admin/server_manager/publish_theme?list=themes
https://yoursite.com/admin/server_manager/publish_theme?list=plugins

# The published archive of a theme or plugin, at the version its live manifest names
https://yoursite.com/admin/server_manager/publish_theme?download=falcon
https://yoursite.com/admin/server_manager/publish_theme?download=bookings&type=plugin

# Redirect to the newest core archive
https://yoursite.com/admin/server_manager/publish_theme?core

A download serves the archive publish_upgrade.php wrote to static_files/themes/ or static_files/plugins/, byte for byte. Only publish builds an archive; a version with no published archive is a 404, and the endpoint never cuts one from the live directory.

How It Works

Deployment Flow

1. Download/extract to staging directory
2. PackageSignature verifies every archive against config/release_verify_keys
3. DeploymentHelper validates:
   - PHP syntax on all files
   - Plugin class loading
   - Bootstrap/core components
4. DeploymentHelper preserves extensions marked `receives_upgrades: false`
5. Backup current installation to public_html_last/
6. Deploy staged files to public_html/
7. Keep the previous maintenance_scripts/ and RELEASE_MANIFEST(.sig) under public_html_last/.release_last, then sync the staged maintenance_scripts/ into the site root (files the release stopped shipping are removed once it passes its deploy tier)
8. Fix permissions (`fix_permissions.sh --production`: the code to root at 755/644, the data to www-data)
9. Run composer_install_if_needed.php (composer under umask 022), then fix permissions again, so what composer installed is owned and moded like the tree
10. Run database migrations (update_database.php)
11. Run the deploy test tier against the new code

If ANY step fails → Automatic rollback of the whole previous release (public_html, maintenance_scripts, the manifest pair)

Directory Structure

/var/www/html/{site}/
├── public_html/              # Current live installation
├── public_html_last/         # The previous code, during an upgrade (for rollback)
├── public_html_failed_*/     # The most recent failed deployment (timestamped)
├── upgrades/                 # Staging: the downloaded release, extracted
├── upgrade_archives/         # The downloaded archives, emptied by a successful upgrade
└── static_files/             # Published upgrade packages
    ├── joinery-core-X.X.X.tar.gz
    ├── themes/THEMENAME-X.X.X.tar.gz
    └── plugins/PLUGINNAME-X.X.X.tar.gz

The code is swapped by moving, not copying: the live tree's contents move into public_html_last/ and the staged tree's contents move into public_html/. On one filesystem each move is a rename. A successful upgrade removes public_html_last/ and every public_html_failed_*; a rollback keeps only the newest failed tree, removing older ones. User data (uploads, static_files, backups, cache, logs, config, vendor) lives beside public_html/, never in it, so an upgrade never copies it. Staging and the archives stay beside the code and out of uploads/: user data may be on another filesystem (the data root, /srv/joinery), and a staged tree there would be copied into place file by file instead of renamed.

Before downloading, an upgrade checks for free space in the deploy root measured from the size of the live code tree: three times that (the archives, the staged copy, and a failed copy if it rolls back), and never less than 500 MB.

A container site has a deploy volume, mounted at /var/www/html/{site}/deploy/. Where it exists, an upgrade keeps its staging (deploy/upgrades/), its archives (deploy/upgrade_archives/), the previous code (deploy/public_html_last/) and a failed deployment's code (deploy/public_html_failed_*) there instead, and checks for free space there. The volume is outside the site's disk allowance on a shared host, so a site at its allowance can still be upgraded, and none of it lands on the container's capped writable layer (DeploymentHelper::deployRoot(); utils/upgrade.php applies the same rule itself, since it updates ahead of that class).

Archive Naming Convention:

• joinery-core-X.XX.upg.zip - Core application (no themes/plugins)
• theme-{name}-X.XX.upg.zip - Individual theme archive
• plugin-{name}-X.XX.upg.zip - Individual plugin archive

Extension Distribution Flags

Themes and plugins carry two independent boolean flags in their manifests (theme.json / plugin.json) that control distribution. Both default to true when missing, and they govern different sides of the pipeline:

Example manifest (theme.json or plugin.json):

{
    "name": "controld",
    "version": "2.1.0",
    "description": "ControlD DNS management plugin",
    "receives_upgrades": true,
    "included_in_publish": true
}

update_database Behavior

Advisory Lock

update_database.php uses a PostgreSQL advisory lock (pg_try_advisory_lock(99999)) to prevent concurrent runs. If a second process tries to run while one is already in progress, it exits immediately with "already running." The lock is released automatically when the database connection closes.

Halt on Migration Failure

Migrations stop on the first failure — subsequent migrations are skipped. Fix the failing migration and re-run update_database.php to continue.

Migration test Semantics

Each migration has an optional test SQL query that returns a row with a count column. The runner interprets it as:

• count > 0 → migration is skipped (already applied)
• count = 0 → migration runs

// Insert a row — skip if it already exists
$migration['test'] = "SELECT count(1) as count FROM emt_email_templates WHERE emt_name = 'my_template'";
$migration['migration_sql'] = "INSERT INTO emt_email_templates (emt_name, emt_body) VALUES ('my_template', '...')";

Note: do not use migrations to seed stg_settings rows. Setting names and defaults are declarative — see "Declarative Settings (no migration)" below.

The same principle applies to core admin/profile menu rows (declared in public_html/admin_menus.json) and plugin menu rows (declared in plugin.json under adminMenu / profileMenu).

Column Defaults

Agent File Regeneration

After migrations and plugin sync, update_database regenerates DB-managed agent files (CLAUDE.md, GEMINI.md, etc.) from the agf_agent_files table — the table is the source of truth, the on-disk files are generated output. Only rows previously written to disk (agf_last_written_time IS NOT NULL) are regenerated, so a never-written customer baseline row stays dormant until the customer opts in.

A drift guard protects out-of-band edits: if a target file on disk was changed since it was last written (its sha256 no longer matches agf_last_written_hash), the row is skipped with a warning rather than overwritten. Resolve a skipped row from /admin/admin_agent_files — writing from there prompts for confirmation and backs the on-disk copy up as <filename>.old before overwriting. See specs/implemented/agent_files_management.md for the full design.

Agent File Upgrades (Customer Baseline)

The default_agents_template.md file ships inside the upgrade tarball. When update_database runs it compares the template's normalized SHA-256 against each Customer baseline row's agf_template_baseline_hash:

An updated agent template is available. [Compare] [Switch to new version]

• Compare opens a read-only side-by-side view of the current content vs the candidate.
• Switch to new version moves target filenames from the active row to the candidate, archives the previously-active row (name prefixed Archived — , target filenames cleared), and writes the new content to disk.

DeploymentHelper API

Validation:

DeploymentHelper::validatePHPSyntax($directory, $verbose)
DeploymentHelper::testBootstrap($stage_dir, $verbose)

Theme/Plugin Preservation:

DeploymentHelper::preserveExtensionsAcrossDeploy($stage_dir, $backup_dir, $verbose)

Rollback:

DeploymentHelper::performRollback($target_site, $preserve_failed, $verbose)

All methods return structured arrays with success status, errors, and detailed results.

Who owns the code, and how the browser upgrades

The code a site runs belongs to root, and the web server cannot write it. Every file the PHP pool executes or includes — public_html, maintenance_scripts, vendor, the release manifest, the site directory itself, config/*.php — is owned by the tree owner at 755/644, and the pool has read and nothing else. Everything the pool writes — uploads, static_files, cache, logs, storage, backups — is www-data:www-data 0770, and none of it is ever executed.

The point is what a break-in gets. A bug that makes the web server write one file can write data, and data does not run. Before, the same bug left PHP in the tree, and the tree is what the next request executes.

The tree owner is root on a node and the developer's account on a development box. fix_permissions.sh records which in {site}/config/tree_owner, root-owned so nothing else can name it, and the root actor reads it before it re-owns anything. The record is the answer once it exists: --dev and --production decide only what is written the first time, so a tool that passes a mode without knowing the box — upgrade.php passes --production everywhere — cannot change whose a tree is.

The Updates page

/admin/admin_updates (System > Updates, superadmin) is a site's one page about updates. It shows:

• What this site runs, from the release's own RELEASE_STATEMENT at the top of the code tree (ReleaseProvenance): the version, the public core and agent commits (linked to getjoinery/joinery and getjoinery/joinery-agent), the public log entry, and the utils/verify_release.php command that checks them from another computer (below). It says whether the site installs only logged releases (PackageSignature::nodeLog()). The page is served by the machine it describes, so it points rather than proves.
• Update: the running version, the upgrade source and the version it offers (asked live), and Update now, which queues an upgrade root request and shows its transcript. A site with a management node (ManagementNodeStatus::is_managed()) is told its updates come from there and gets no button; the origin publishes releases and gets none either.
• Install history: the last 25 ReleaseInstall rows.
• Installed with Install anyway: every plugin and theme whose trust is unsigned or unlogged, with who approved it and when, from the unsigned_package_installed event log row the installer writes. Said plainly as not logged.
• Kept out of updates: every local copy (receives_upgrades: false) and the version the source ships. Allow upgrade stays on the Plugins and Themes pages.

Verifying a release

utils/verify_release.php checks, from any computer, that a release is built from the public code and is in the public log. It runs from a clone of the public repository with no site, settings or database (ReleaseVerifier, which resolves core classes with ClassAutoloader::restrictToCore() and loads the four plugin classes it reuses by path):

php utils/verify_release.php 0.8.467 --source=https://getjoinery.com

It downloads the core archive from the upgrade source (/static_files/joinery-core-<version>.tar.gz, served without a credential) and reads the release statement in it. The core and agent commits the statement names must each be on the main branch of getjoinery/joinery and getjoinery/joinery-agent on GitHub: the branch's history is fetched (commits only) and the commit must be an ancestor of its tip, the test publish applies (ReleaseCommit::onRemote()). A commit GitHub serves by hash is not enough, since it serves any commit in a repository's fork network that way. Each commit's files are then read from GitHub, or from a local clone when one is named. Then:

• The statement: signed by a statement key the core commit lists, its log entry proven under a log key the commit lists, its key chain walked from genesis, and the keys it installs exactly the commit's release_keys/ (ReleaseStatementPublisher::verifyDocument()).
• Sigstore: every log key the commit lists is in Sigstore's trusted root (ReleaseLogClient::trustedRoot()); the log serves the statement's entry bytes at its index today, and the checkpoint it serves today is signed by its key (LogTileReader, which reads Rekor v2's checkpoint and entry bundles). The served bytes are compared, not proven under today's checkpoint; the entry's place in the log is proven by the statement's inclusion proof under its signed checkpoint.
• Plugins and themes: each manifest is rebuilt from the core commit and must be the one the statement records. The prebuilt Go binaries a plugin ships (the relay sealer in mailbox, the mail gateway in mail_apps) are taken from the statement and their source stamp from the commit; a release that records none of a program's binaries shipped none.
• The core archive: signed by a release key the commit lists, carrying exactly what its manifest lists, with that manifest the one the statement records; and every manifest line is either the commit's own file with the same bytes or a file publish builds that passes its check: the agent binaries (decompressed, against the statement), manifest.json (its signatures verify, its keys are the commit's, its source_commit is the statement's agent commit), the systemd unit (the agent commit's file), the support bundle (the statement's hash, its own signed manifest, every file the commit's or a plugin's prebuilt Go binary the statement records) and its index, the license copies (the commit's files), and every RELEASE_STATEMENT (the statement itself). And the other way round: every file the core commit ships into the core archive (its public_html less what publish leaves out, and maintenance_scripts/install_tools and sysadmin_tools) must be listed, so an archive that drops one does not verify.
• Rebuilds: the agent is rebuilt from the agent commit with the inputs manifest.json records, and the relay sealer and the mail gateway from the core commit, each with the official toolchain its go.mod pins (the statement's go_toolchain must be the agent's pin), and each must be byte-identical. Any Go 1.21 or later fetches that toolchain, checksum-verified, as publish does (--go=PATH names which Go); without one, or with one that is itself the pinned version, the two are reported as not run.

Root requests

Installing a plugin, upgrading, saving a doc from the admin editor — anything that writes the code — is a root request: a small JSON file the web side writes into {site}/cache/root_requests/ naming a kind from a fixed list.

Installing an uploaded package

On a host that refuses unsigned packages (unsigned_packages=refuse in /etc/joinery/host_restrictions/restrictions, read through HostRestrictions; a multi-tenant host, see Installation) there is no way past the refusal. The page shows "This host installs only signed plugins and themes" with the package's facts and no button, PackageAcknowledgement::mint() refuses before any step-up, and install_extension.php refuses an unverified plugin or page theme with or without --acknowledged, and from disk even when its row already says unsigned. A style theme still installs. Each refusal exits 3, as any unverified package does. PackageSignature::publisherBox() is false there whatever config/ holds, and the host converger never takes the publishing-box shortcut there.

The shell form still works for a self-hoster with one, and is the same code:

sudo -u <tree owner> php utils/install_extension.php plugin --staged=<dir> [--replace] [--acknowledged]

--replace is needed to overwrite an extension that is already installed; the copy it replaces is kept beside it. --acknowledged is the operator's answer to the warning the command prints for an unverified package; the restrictions apply exactly as they do from the page, with "an operator at a shell" as the approver.

What a web-tier attacker gets from this: someone already running code as the web user can write the step-up marker row and forge the acknowledgement, and root will copy their PHP into plugins/ — under the restrictions, with every superadmin told. That is the accepted residual (the read-only tree's "nothing persists" becomes "nothing persists unless they forge one row"); root never runs their code as root. A stolen admin session cannot do it at all: the marker needs the second factor at the keyboard.

A host installer runs only out of a package we built

The converger runs every active plugin's declared host_installer as root. Before it does, it verifies the plugin directory — `php utils/verify_package.php <dir>, a thin CLI over PackageSignature` — and skips the installer, with the verdict in the converger's log (the journal, journalctl -u joinery-host-converger, on a host with a data root; logs/host_converger.log on one without), on anything but signed. An unsigned plugin stays installed and active; it simply never has a script run as root out of its directory. The publishing box, which holds config/agent_signing_key, trusts its own tree: every plugin there is the source the archives are built from, and its live manifests are stale the moment a file is edited.

The host converger carries them out — a root timer running every minute, plus a .path unit that fires the moment a request is queued, so an operator watching a transcript sees it move in seconds. The queue is worked under the runner's one lock (above), so two ticks inside one upgrade cannot both take it. The page that submitted the request polls its transcript and shows it as it grows.

A machine with no converger cannot upgrade or install anything until it has one, because nothing else can write the new code in. Every page that submits a request says so before its button rather than after, and the request is written either way — a converger that comes back finds it waiting. install.sh installs one on every new box; HostConvergerNotice reports a missing or stalled one.

Common Issues

Permission Errors:

One script sets every permission on a site, and it is the only thing that should. Correcting them by hand risks handing the code back to the web user, which is the state the ownership model exists to prevent.

sudo bash /var/www/html/{site}/maintenance_scripts/install_tools/fix_permissions.sh {site} --production

Validation Failures:

• The most recent failed deployment is preserved in a public_html_failed_* directory
• Inspect for syntax errors or missing dependencies
• Fix and redeploy

• Check manifest has "receives_upgrades": false
• Restore from public_html_last/ if needed

• Check public_html_last/ exists
• Manually restore from backup
• Fix permissions after restore

The Joinery vhost bakes a RewriteCond %{HTTP:CF-Visitor} !"scheme":"https" guard into the redirect rule, so it cannot loop in any CF SSL mode (or with no CF at all). The admin Settings page also surfaces a yellow warning banner whenever the platform detects it's being served from Flexible-mode CF — so the misconfig is visible without needing to read logs.

Apache Vhost

Every site installed by install.sh has the same Apache vhost shape, regardless of whether it sits behind Cloudflare, behind another CDN, or is exposed directly to the public internet, and regardless of whether the origin has its own TLS certificate.

Shape. Defined by template files in maintenance_scripts/install_tools/ (single source of truth per deployment mode) — default_proxy_vhost.conf for Docker reverse-proxy sites, default_virtualhost.conf for bare-metal sites. install.sh write_universal_vhost substitutes the placeholders ({{DOMAIN_NAME}}, {{SITE_NAME}}, {{PORT}}) and writes the result to /etc/apache2/sites-available/${sitename}.conf.

Every host answers on any address (*:80, *:443) and is chosen by name, so a change of the machine's address (an IP swap, or a provider re-addressing it) needs nothing re-rendered. A machine carries one bare-metal site. Several sites share a machine through Docker, whose proxy vhosts are on any address too. Apache considers vhosts pinned to a connection's exact address before any * vhost, so a second, hand-made site pinned to the machine's IP would outrank the Joinery site on every request to that address.

1. The name reaches this box (name_reaches_here: a nonce is written where the site serves files, fetched through the name with redirects followed, and compared; it costs nothing at Let's Encrypt) → HTTP-01 through certbot certonly --apache, for the apex and www whenever www resolves and reaches here too. --apache answers the challenge through the running vhost; certonly keeps certbot from editing that vhost, which the renderer owns. A --deploy-hook 'systemctl reload apache2' is recorded as the lineage's renew_hook, so a renewal writes the files and reloads Apache and nothing else. Whether the name reached the box directly or through an edge is a word in the log, not a gate.
2. The name does not reach this box, or HTTP-01 failed, and a matching DNS-API credentials file exists at /etc/letsencrypt/<provider>.ini → LE DNS-01 with the matching certbot plugin. Plugin map:

Each plugin reads its credential file in its own standard format; certbot's docs cover the schemas.

1. Neither path produces a cert → the install proceeds on HTTP. The :443 vhost stays dormant via the <IfFile> guard, and the retry timer below keeps asking.

• DNS points at the box: issued during the install.
• DNS points at Cloudflare (or another edge): issued during the install, through the edge; the placeholder is what lets the edge's https hop complete. Cloudflare must be on Full, not Full (Strict), until the first certificate lands: Strict refuses the placeholder (526), so the challenge cannot get through. That is the one true limit, and the installer, the retry timer's journal line and the node's health panel all say so when they see it. The probe tells that case from the other way an https hop can fail: 525 or no answer means this box completed no handshake at all — the placeholder is missing — which is a defect on the box, named as one, with render_vhost.sh as the fix (name_reaches_here returns 2 for the first and 3 for the second).
• DNS points nowhere yet: the retry timer issues it once the name reaches here.

Issuing or re-issuing by hand. sudo /var/www/html/<site>/maintenance_scripts/sysadmin_tools/setup_ssl.sh <domain> re-enters the same decision tree. sudo .../sysadmin_tools/issue_origin_cert.sh <domain> is the operator's tool for a name that needs www added to an existing lineage, or for a container's certificate issued on its Docker host: it probes reach for both names, prints the exact certbot line, asks, passes --expand only when the lineage already exists, reloads Apache and verifies at the origin with SNI that each name is covered. Neither deletes anything. sysadmin_tools/strict_readiness.sh <name>=<origin-ip> ... is the read-only check of the same fact from outside, one line per apex and www.

A certificate deferred at install time arrives on its own. When no certificate landed, arm_ssl_retry.sh installs joinery-ssl-retry@<domain>.timer: every five minutes it asks name_reaches_here, does nothing until a request for the name lands on the box, then issues once and disables itself. A self-signed placeholder does not count as done. The reach probe is what makes an indefinite retry safe — Let's Encrypt counts failed validations, and a failed probe is one HTTP fetch that costs nothing. A restore that lands the site on a different domain re-arms the timer for the new name and disarms the old one.

Renewal. install.sh server enables certbot.timer (or asserts /etc/cron.d/certbot where there is no systemd) rather than inheriting it from the package. host_housekeeping.sh, on every converge, sets every lineage the machine renews through Apache to installer = None with a renew_hook that reloads Apache (host_files_heal_renewal_confs in _host_files.sh), so certbot never edits a vhost. That covers every certificate on the machine, whatever its name, and a Docker host's proxy vhosts through the host's own converger; a lineage issued by webroot or DNS is left alone, and a renew_hook already set is kept. In render_vhost.sh, the lines certbot's Apache installer inserts (an Include of its options file and its redirect block) are recognised as certbot's, not an operator's, so a vhost carrying them is re-rendered rather than held for a hand-apply.

The strict-ready contract

A node guarantees this to an owner who flips Cloudflare to Full (Strict), with no operator step on the node:

• The origin presents a certificate that names the apex and www under SNI, issued over HTTP-01 through the edge, with no credential file and no operator step.
• It renews on certbot's timer, and a renewal never edits the vhost.
• When renewal stops — more than a day past the two-thirds-of-life due date with the same certificate still on disk, or the timer off, or www uncovered — the admin header and the health panel (includes/CertificateNotice.php, VaultHealth::checkCertificates()) say so, with the command. They read cache/certificates.json, which the host converger writes on every run from /etc/letsencrypt; a managed node's management node watches the same origin certificate over the wire (RunNodeUptimeChecks::check_cert_expiry()).
• It answers TLS from the first minute, with the placeholder until the real certificate lands, so an edge that redirects the challenge to https can deliver it.
• The one limit: the edge must be on Full, not Strict, until the first certificate has landed. Strict refuses the placeholder, and the HTTP-01 challenge cannot get through. The retry timer keeps asking; the health panel says to set Full until then.
• On a managed node the management node also says when the origin answers with a certificate for other names — the placeholder, or a shared fallback vhost — because that is exactly what Strict would reject.
• The edge must let the challenge through. Cloudflare's Under Attack mode, or a firewall rule that challenges every request, stops Let's Encrypt's validator at the edge; renewal fails while that is on, the alerts say so, and nothing on the node can work around it. Outside that, the encryption mode does not matter once a certificate exists: Flexible, Full and Full (Strict) all pass the challenge.

Rebuilding a site on new hardware

A backup carries the whole site tree, config/ included, but a restore keeps the target machine's own config/Globalvars_site.php and config/backup_site_key over the copies inside the backup: the first holds the machine's database password and its secret_box_key, the second its identity as a recipient of its own backups, and all of that belongs to the machine. So a rebuild is install, then restore onto the installed site.

Sealed values survive a chain rebuild only with --adopt-secret-key. By default the new machine keeps its own secret_box_key, so every value sealed on the old one is dead there: stored credentials (the backup target, OAuth client secrets, IMAP passwords, cloud accounts) are entered again, and regenerable secrets are minted again, which changes the Joinery Direct signing key. restore_chain.sh --adopt-secret-key keeps the new machine's config and takes only the old machine's secret_box_key into it, so everything sealed there opens here. restore_project.sh has no such flag. See Sealed Secrets.

# 1. On the new box: server prerequisites, then the site
sudo ./install.sh -y server
sudo ./install.sh -y site --bare-metal <site> --password-file=/root/.joinery_postgres_password <domain>

# 2. Bring the backup down and restore onto it
bash maintenance_scripts/sysadmin_tools/restore_project.sh <site> <archive> --domain <domain> --force
#    or, for a chain:
bash maintenance_scripts/sysadmin_tools/restore_chain.sh <site> \
     --artifacts <chain dir> --key-file /tmp/k --domain <domain> --adopt-secret-key --force

The restore reconciles the result to the new box: the domain, the deployment shape (docker vs baremetal), the paths, a regenerated virtualhost, and an armed certificate retry. It refuses if the restored database will not open with the new machine's credentials, rather than leaving that to show up as SQLSTATE[08006] on every page. What it reconciles and why is in Backups.

Cross-shape rebuilds work in both directions with no extra step: a container backup landing on a plain server, or the reverse. Neither installs the virtualhost the backup carries — see the same section.

The distinction matters beyond the upgrade endpoint: any control that edits a file the upgrade replaces wholesale belongs only on a publishing instance, because on a consuming site the edit is discarded at the next upgrade. Version-controlled manifests like Joinery AI's recipes.json are for the same reason edited only on the publishing checkout.

PostgreSQL and PHP memory

maintenance_scripts/sysadmin_tools/tune_postgres_memory.sh sizes PostgreSQL from the RAM the machine actually owns and writes the result as conf.d/20-joinery-memory.conf: shared_buffers at 20% of RAM (floor 64MB, cap 2GB) and effective_cache_size at 50%. It is idempotent, writes nothing when the drop-in already matches, and restarts the cluster unit (postgresql@{version}-main) only when it wrote. --dry-run prints what it would write; --no-restart writes without restarting.

In a container, tune_php_fpm.sh sizes PHP-FPM's worker pool from the same budget, and PostgreSQL's max_connections follows the pool. The split lives in one sourced file, sysadmin_tools/_memory_plan.sh, which both tuners read:

"RAM the machine owns" is the whole question, and on a shared host only the container knows the answer. The budget is resolved in one order, and the script skips rather than guesses:

Where it runs. On bare metal, install.sh server runs tune_postgres_memory.sh during the install: that host is the machine. On Docker both tuners run from the container start command on every start, each before the service it sizes starts, so the values apply from the first process. It is deliberately not part of the base image — install.sh server is what docker build runs to bake that image, so a figure computed there would be the build host's RAM, frozen into the image and shipped to every container on every host — and deliberately not in _site_init.sh, which runs only on first boot while /etc/postgresql is reset by a container rebuild.

Giving a container a budget. install.sh site --memory=SIZE (any size Docker takes: 512m, 1G, 1.5g; at least 6m) sets the limit, and --memory-swap is pinned to the same figure so the cap is not doubled by swap. The size is checked before anything is stopped. The limit is recorded in the site's run spec (below), so every later rebuild keeps it; a rebuild given another --memory records that one instead, and --memory=none lifts it. docker update --memory=512m --memory-swap=512m NAME changes a running container's budget until its next rebuild, which reads the run spec again. The container picks up the sizing at its next start. Containers are unlimited by default: a limit that arrives uninvited OOM-kills a site that was fitting fine, so it is a decision for whoever knows how many sites share the host. On any host running more than one site it is worth setting — it is the only thing that tells each container what share of the host is its own.

CPU and process ceilings. install.sh site --cpus=N caps the container at N cores (1.0, 0.5; at least 0.01), and PostgreSQL's parallel workers follow it (tune_postgres_memory.sh reads cpu.max). Without a ceiling, sites on a busy host still share the CPU evenly, because Docker gives every container the same weight. --pids-limit=N caps the container's processes and threads together, so a fork bomb stops there. A new site gets 512; an idle site runs 54 to 75 processes and threads, so a value below 128 is refused because the site would not start. Both are recorded in the run spec like the memory budget: a rebuild keeps them, a rebuild given the option records the new value, and none lifts either. A bad value is refused before anything is stopped, and the message says whether it came from the option or the run spec. A CPU ceiling above the CPUs Docker counts on the host (docker info's NCPU) is refused the same way, since Docker would refuse it only after the old container was gone; the rebase and code-volume scripts check the spec for this before they stop anything too. docker update --cpus=N NAME changes a running container until its next rebuild. On a bare-metal site all three options are refused.

The run spec. A Docker host keeps one file per site, /etc/joinery/sites/{site}/run_spec, that records how its container is run: hostname, restart policy, memory, CPU and process limits, published ports and volumes, and the site's own network (network=, bridge=, subnet=, subnet6=; see the installation guide's Each site's network). It holds no secret; the environment travels as an env file. Everything that creates a site container builds its docker run arguments from it (_site_run_spec.sh's run_spec_args): install.sh site, rebase_site_container.sh (whose rollback recreates the old container from the copy it kept, checked before anything is removed) and migrate_site_to_code_volumes.sh (which adds its new volumes to the spec). Nothing rebuilds a container's arguments from docker inspect, which would drop its limits. run_spec_args also makes the network the spec names exist, exactly as the spec says, before it gives --network, so no caller can run a site onto a missing or different network. install.sh writes the web and database ports and the standard volumes; any other published port or volume in the spec is the site's own, and a rebuild keeps it.

A container made before the file existed has it read from Docker once, by whichever of those runs first, while the container still exists. Anything a spec cannot carry (a bind mount, added capabilities, privileged mode, a network other than the default or the site's own, extra hosts, devices, a CPU quota or set, restart retries, swap set apart from memory) stops that read with the thing named, and nothing is changed: recreating the container would lose it. Each line is checked when written and when read: a value that would not be exactly one argument, or a newer format than the script reads, is refused. A rebase swapped before run specs existed rolls back from the argument list it kept then. remove_account.sh removes the file, and the site's network, with the site.

The deploy tier

php tests/run.php deploy is the set upgrade.php runs after a swap. It pulls in no other tier and no other tier pulls it in.

It exists because safe, db and test-db are development gates. They run in a checkout and are entitled to assert things about one: that the full first-party plugin set is present, that the components manifest lists what the repository holds, that maintenance_scripts has the layout the installer expects. A deployed site has none of that and never should — it carries the plugins it uses and no repository around it. Eleven of those suites fail on a production node, for reasons that say nothing whatever about the release.

So deploy asks the only question that means anything on a node: does the code that just landed run on this machine.

• No assumption of a repository. Not the plugin set, not the theme set, not a git checkout, not a sibling directory.
• Reads only. It runs on production, after a swap, with a rollback hanging on the result.
• An unreachable dependency is a SKIP, not a failure. Reverting a working release because a socket would not open is worse than the thing being guarded against.

Where a site's upgrade_source comes from

The setting is bounded: an https origin with a host name and optional port and nothing after it, vault gated like every setting that decides what the server runs. Root verifies every archive it downloads against the release key before deploying it, so the setting chooses where a verified archive is fetched from and nothing more — pointing it at a server serving anything else gets a refused upgrade with the verdict, and the bound keeps a typo from becoming that refusal.

It is not a decision anyone makes twice. _site_init.sh writes it at install time from the endpoint the install actually fetched its code from — install.sh's UPGRADE_SERVER, which defaults to https://getjoinery.com and is overridden with --upgrade-server=URL. One rule covers both audiences: leave the flag off and the site tracks stable releases; pass it and the site follows wherever it was installed from.

The root node

One deployment is the origin of the estate: the code is written there, and the themes and plugins are published from there. The root_node setting names it by domain — dev.getjoinery.com — and a site is the origin when its own webDir matches that name. It is machine-written and never rendered on the settings page, because it is a fact about the estate's shape rather than a preference anyone tunes.

Naming it by domain rather than raising a flag is what makes a site copy or a restored backup safe. The copy carries the same value, which still names the origin, and the copy correctly concludes it is not the origin itself.

Five behaviours follow from it, each fixing a state that has no sensible reading otherwise:

• It serves its own catalog. MarketplaceClient::source() answers with the site itself rather than upgrade_source, so the Marketplace lists what is actually on this disk. upgrade_source records where a site was installed from, which on the origin is a site running an older copy of this very tree — reading a catalog from it offers to install yesterday's theme over today's.
• It sees every extension. An audience naming a customer's sites does not have to also name the origin; see Plugin Developer Guide.
• It refuses to upgrade itself. upgrade.php stops before its first fetch, because applying the last release published from here would replace new work with an older copy of itself. Serving upgrades to other sites is a different branch and is unaffected — the origin still publishes normally.
• It always may mint a release number. Everywhere else that is decided locally, by whether the tree has moved past what upstream delivered; on the origin it must not depend on anything — see The release channel.
• It refuses to install over itself. Everything in its catalog is already on its disk, so a marketplace install, or the upstream refresh a plugin install performs, could only overwrite the working copy with an archive of itself. The publisher caches archives per version, so an edit made without a version bump would be replaced by the code it replaced.

The release channel

getjoinery.com serves stable releases. It is a site rather than a flag on a row: no schema, no promotion state to forget, and it reuses the chaining utils/latest_release already implements.

The question is deliberately local. Asking which site is the origin would mean every deployment had to be told, and a deployment that was never told reads as free to mint — so the rule would fail exactly where nobody had thought to apply it. The one exception is the origin itself, which always may mint: its root_node is already set, and its authorship should not hang on never having received an upgrade — a restore or one accidental upgrade.php run would otherwise strand it. A site that has received something but cannot read its running version at all is refused; minting is the unrecoverable direction.

# on getjoinery, after upgrade.php brings it to 0.8.199
sudo /usr/bin/php plugins/server_manager/includes/publish_upgrade.php "release notes"
# -> Republishing 0.8.199 -- this deployment is running exactly what it received, not new work.

Prerequisites

• upgrade_source setting must be configured (URL of the upgrade server), or the site is the origin named by root_node and serves its own catalog
• The upgrade server must have the Server Manager plugin active

Overwrite Protection

• A marketplace install of a name already on disk replaces the directory with the verified archive (refreshFromUpstream() sets the old copy aside as .refresh-<name>-<pid> until the new one is fully in place, and sweeps a stale one at its next run). It refuses when the live manifest says receives_upgrades: false: that directory is a local fork, and the refusal names Allow upgrade as the way to take the shipped version instead.
• A local fork — a live manifest saying receives_upgrades: false — is carried over the incoming copy by every deploy, whatever the incoming manifest says (DeploymentHelper::preserveReason()). Every uploaded package is marked one when it is installed. Allow upgrade on the Plugins or Themes page writes true, and the next deploy replaces the directory with the shipped version; public_html_last is the undo.
• An edit made on the node leaves a package Joinery's: it is not a fork until the operator says so. The Plugins and Themes pages compare each shipped, non-fork package's files with its signed listing (PackageSignature::editedSinceInstall(), once per page load, reading only the files and the node's key) and show Edited since install, naming the first differing file, beside one that no longer matches. Keep my changes marks it a local fork through the same set_receives_upgrades root request as Disable upgrade, and the confirmation says the cost. An edited package that is not a fork is still replaced by the next deploy, and the deploy's extension table says so: "Edited here, will be replaced", with the file. An unsigned package has no listing to compare with and is never reported as edited.
• An upload that names an installed extension shows the replace panel on the Plugins or Themes page (both copies' version, author and file count; Replace or Discard). Replace queues install_package with replace: true, and the installer sets the installed copy aside as <name>.replaced.<UTC time> (the newest kept copy only) after the verdict and the acknowledgement, never before. The event log has a package_replaced row. A system extension is refused: every deploy pulls it fresh.

Catalog Endpoint Fields

The publish_theme.php catalog endpoints (?list=themes, ?list=plugins) include:

• name — display name (unchanged for backward compatibility)
• directory_name — filesystem directory name (used for matching and downloads)
• display_name, version, description, author, is_system, included_in_publish

Related Documentation

• CLAUDE.md - System architecture and development guidelines
• Plugin Developer Guide - Plugin development
• Server Manager - Server management, publishing, and backup targets
• Specifications: - /specs/implemented/upgrade_system.md - Feature parity analysis - /specs/implemented/fix_publish_upgrade_system.md - Publish upgrade fixes - /specs/implemented/theme_plugin_distribution_refactor.md - Separate archive distribution - /specs/implemented/server_manager_publish_upgrade.md - Moving publish/upgrade into server_manager plugin - /specs/implemented/upgrade_graceful_theme_download.md - Graceful handling of missing archives