File Blob Layer — Physical Storage as a First-Class Object
Every uploaded file on the platform splits into two objects:
File(fil_files) — the logical file. Identity, ownership, visibility, title, the URL a user links to. Everything a user sees.FileBlob(fbb_file_blobs) — the physical bytes underneath it. The stored name on disk / in the bucket, size, content hash, storage driver, and offload state. AFilereferences one blob viafil_fbb_file_blob_id.
Any feature that stores bytes — entity photos, inbound-email attachments, AI
chat uploads, profile pictures, Drive — gets this for free by going through the
one ingestion path below. No feature touches fbb_file_blobs directly.
Storing bytes
There is exactly one byte-ingestion path. Use it; never write a File row and a
file on disk by hand.
// From an uploaded temp file (a $_FILES entry, or any staged path):
$file = File::createFromUpload($tmp_path, $display_name, $mime, $owner_id, $restrictions);
// From bytes you already hold in memory (generated / received / fetched):
$file = File::createFromBytes($bytes, $display_name, $mime, $owner_id, $restrictions);$restrictions is an array of visibility columns set at creation, e.g.
['fil_private' => true] or ['fil_min_permission' => 5, 'fil_source' => File::SOURCE_ENTITY_PHOTO].
Origin tags — where a file came from
Every file carries fil_source, a short key naming the subsystem that created it
(NULL means a legacy row from before the tags). *Stamp one when you store a file,
and declare it. Core's own tags are File::SOURCE_* constants declared in
File::core_source_catalog(). A plugin declares its tags under fileSources in its
plugin.json and keeps the constant on one of its own classes (see the
Plugin Developer Guide).
File::source_catalog() merges the two and is the one place that says what each
tag is:
| Field | Meaning |
|---|---|
label | Human name. Listing filters and pickers show this. |
internal | A file the system made for its own use. Browse surfaces must not list it. |
default_view | Included when a listing opens with nothing picked. |
File::internal_sources() (feed to MultiFile's
sources_not), File::default_view_sources(), File::source_label(),
File::source_is_listable(). An undeclared tag counts as listable — a
subsystem that forgets to declare itself shows up in an admin listing where
someone notices, rather than having its files quietly disappear.internal is not a permission. It means "don't list this", never "this is
secret" — File::is_viewable() alone decides who may read any file's bytes.
Classification lives on the source rather than on the row because it is a property of what created the file: every sealed search index is internal, and there is no user-visible one. So the tag stamped at creation already carries the answer, with no per-row column for a writer to get wrong.
MultiFile filters on origin three ways, and they resolve to exactly one SQL
condition: sources (allowlist), sources_not (blocklist), source_not (its
single-value form). An allowlist combined with a blocklist is the allowlist minus
the blocked entries. NULL rows survive a blocklist — they are not what it names —
and can only enter a result through an explicit File::SOURCE_UNCLASSIFIED in the
allowlist, because fil_source IN (...) never matches NULL.
Both route through FileBlob::createFromPath(), which:
- Hashes the bytes (sha256) and measures the size.
- Dedup lookup — a live blob with the same
(sha256, size, is_private)? Hit → retains that blob (refcount +1) and discards the new bytes. Miss → moves the bytes into the correct visibility directory under a collision-freefbb_stored_nameand inserts a new blob. - Detects the honest MIME type from the bytes (the caller's
$mimeis only a fallback) and stores it — sofil_typeis trustworthy by construction, never by caller discipline.
fil_name (URL identity) equals the
blob's fbb_stored_name (physical identity). They diverge only under dedup and
versioning: a deduped file gets its own unique fil_name but points at the
shared blob, and the serving path resolves the physical bytes through the blob.Image variants live at <size>/<stored_name> and are shared by every file
referencing the blob. Ingestion does not make them: call $file->resize() to
build every registered size up front, which is what a path that knows its images
will be browsed immediately should do.
Nothing else has to. A variant that does not exist yet is generated the first
time it is requested, and only the one size asked for: /uploads/<size>/<name>
resolves the file, applies the normal is_viewable() gate, and then calls
$file->ensure_variant($size) before serving. So a subsystem can store an image
without thinking about sizes at all — an inbound mail attachment, a chat upload —
and get_url('avatar') still resolves, while an 80px listing never causes a
1920px copy to be written. Generation sits behind the gate on purpose: resizing
is real CPU and the requested name comes from the URL, so a caller who may not
view the file cannot spend it.
The on-demand path is for local blobs; a cloud variant means download, resize and re-upload, which does not belong inside a page request. Sealed and client-encrypted files are skipped either way — their stored bytes are a container, so resizing them would produce garbage.
Reference counting and deletion
Lifecycle is code-owned (refcounting isn't expressible in declarative deletion rules):
File::permanent_delete()callsFileBlob::release($blob_id). Release decrements the count inside a transaction guard and, at zero, deletes the physical original + every image variant (local unlink or cloud driver delete) and the blob row. A blob still referenced by another file is left untouched.FileBlob::retain($blob_id)is the increment twin (a new file pointing at an existing blob — the dedup path calls it).- A reference is taken before the row that holds it exists.
createFromPathinserts a fresh blob already counting one, andcreateFromExistingBlob/drive_upload_init's dedup branch callretainbefore inserting. Every path out from there has to give the reference back if the row is not created — the insert legitimately fails when two uploads race for one name, and the partial unique index refuses the loser. A blob left counting a reference nothing holds is never reclaimed: the bytes stay on disk, keep billing against the owner, and can be reached by nobody.
CloudOffloadEngine is mid-push.A practical consequence for tests and callers: deleting one of several files that share a blob (identical bytes) does not remove the bytes — they survive for the remaining references. That is correct dedup behavior, not a leak.
Visibility
Public vs private is a physical placement — public bytes go to the fast-serve
directory and stay on this server, private bytes to the restricted directory
and from there to the private bucket — so it is a blob property,
fbb_is_private. But it
is derived from the referencing files. The invariant: every file referencing
a blob is in the same visibility class. It is maintained by:
- Dedup scoping. A dedup match requires equal
(sha256, size, is_private). The same bytes uploaded once public and once private become two blobs — never a shared blob straddling both stores. - Visibility change (
File::move_to_correct_directory(), called aftersave/soft_delete/undelete). When a file's desired class no longer matches its blob's: - refcount 1 → flip the blob (move its bytes between dirs, or pull home from the wrong bucket). - refcount > 1 → copy-on-write split: copy the bytes to a new blob in the target class, repoint this file, decrement the old blob. The siblings keep the shared blob. - Soft-delete exception. Soft-deleting one of several references leaves a
shared blob where it is — the deleted file's own URLs are already gated by
File::is_viewable(), and the siblings keep serving. A refcount-1 soft delete flips the blob private (as before); undelete flips it back.
What lives where
| Concern | Owner |
|---|---|
Identity, ownership, title, fil_name (URL), visibility gates, signed URLs | File (data/files_class.php) |
| Stored name, size, sha256, MIME, driver, offload counters, refcount | FileBlob (data/file_blobs_class.php) |
| Paths, reads, resize, variant layout, cloud put/get/delete, pull-back, flip / split | FileBlob — File delegates (get_filesystem_path, read_bytes, remote_key_for, resize, delete_resized, storage_driver) |
| Offload eligibility (private blobs only) + per-row enumeration for the shared engine | BlobStorageProfile |
Drive — folders, quotas, sharing, versioning
The member Drive at /drive is personal file storage built on the blob layer
above. It adds folders, a per-member quota, file versioning, sharing (member
grants and public links), trash, a resumable upload protocol, and a change feed
for sync clients. It is off by default: the drive_active setting gates the page,
the menu entry, and every drive_* action.
Data model
Drive files are ordinary File rows (private, fil_source = 'drive') placed
in the tree by fil_fol_folder_id (NULL = the drive root, which is implicit —
there is no root row). The drive source tag is the boundary of the whole
feature: listings, search, trash, the trash purge, and quota accounting all
scope to it, and DriveHelper::load_file resolves only Drive files — so a
Drive verb can never list, trash, share, or destroy a file another subsystem
owns (an avatar, a mail attachment, a store image). Everything else is a small
model of its own:
| Model | Table | Role |
|---|---|---|
Folder | fol_folders | A node in a user's tree. Sibling-name uniqueness among live rows is a partial unique index on (owner, COALESCE(parent,0), name) WHERE delete_time IS NULL. Save folders through DriveHelper::save_folder_unless_name_taken() — see below. |
File | fil_files | Platform-wide, so Drive's own rows are fil_source = 'drive'. Sibling-name uniqueness among live Drive rows is a partial unique index on (owner, COALESCE(folder,0), title) WHERE delete_time IS NULL AND fil_source = 'drive'. Save through DriveHelper::save_file_unless_name_taken() — see below. |
DriveUsage | dru_drive_usage | One recomputed byte total per user (the quota gate + storage meter). |
FileVersion | fvr_file_versions | Prior content of a file — one row pins one historical blob. |
FileAccessGrant | fga_file_access_grants | A share of a file/folder to another member (viewer / editor). |
FileShareLink | fsl_file_share_links | A durable, revocable public link (/s/{token}). |
FileChange | fch_file_changes | Append-only change feed; the primary key IS the sync cursor. |
FileUpload | fup_file_uploads | Pending state for a resumable upload. |
The other places a member's files live
The Drive rail has a second list, Also in your account: one entry per
listable origin tag other than drive under which the member owns at least one
live file, with its count — Mail attachments, AI chat uploads, Photos,
Unclassified for rows older than the tags. drive_list returns them as
other_sources on every response (DriveHelper::other_sources_for_user), and
view=source&source=<tag> lists that source: the caller's own files only,
newest first, a search staying inside the source. A source view is
read-only — Drive hands the file over and nothing more; renaming, moving or
deleting it belongs to the feature that stored it. Drive's own tag and every
internal tag are refused as a source view.
A file whose source opens through a decrypt hook (File::source_opens_through_hook
— a mail attachment sealed under its message key) is exported with
requires_window: true, no thumbnail URL (the stored bytes are that consumer's
ciphertext; a variant resized from them would be noise), and syncable: false.
The page opens it the way it opens a Private file: a HEAD, an unlock on 423,
then the download.
Access and tree logic lives in includes/DriveHelper.php; the verbs are drive_*
API actions (logic/drive_*_logic.php, each with a _logic_descriptor()), which
page JavaScript (assets/js/drive.js) calls with the browser-session credential.
No two live items may share a name in one folder — folders or files — and a
partial unique index is what enforces it. Files are scoped to
fil_source = 'drive', because fil_files is platform-wide and an avatar, a
store image and a mail attachment all sit at folder NULL where duplicate
titles are ordinary.
DriveHelper::folder_name_taken() and file_name_taken() are fast paths that
give a clean answer before any work is done, but two requests can both pass one
and both reach the insert — so every create, rename, move and restore saves
through DriveHelper::save_folder_unless_name_taken() or
save_file_unless_name_taken(). On a uniqueness refusal these ask the same
question again and, if the name is now taken, return false for the caller to
answer exactly as it would have a moment earlier; anything else is rethrown.
A file arriving by upload is created rather than saved, so the two upload paths
carry the rule themselves: they ask file_name_taken() first and translate a
23505 from the insert into the same refusal. There are two of them, and both
matter — drive_upload_complete for a transferred file, and the dedup
short-circuit in drive_upload_init, which creates the file itself when the
account already holds identical bytes and never reaches completion at all. Both
put the destination folder in the creation, so an upload never occupies the
root's namespace on its way to where it was going.
A file refusal carries reason: name_taken in the error data. Sync clients
branch on that marker rather than on the message, and treat it as a state that
resolves rather than a failure: the sibling holding the name is something the
next index walk will tell them about, and landing it is what frees the name.
Without the rule, two devices that conflict on one file both upload the same
conflicted-copy name, the server takes both, and every device can then
materialize only one of them — leaving the rest unsyncable forever with nothing
the user can do about it, since they cannot see the second file to rename it.
A trashed folder is not a place
Nothing new may be placed into a folder that is in the trash. Every verb that
puts an item somewhere — drive_folder_create, drive_upload_init,
drive_upload_complete and drive_move — asks DriveHelper::folder_is_trashed()
about its destination and refuses with reason: parent_trashed.
load_folder() resolves trashed folders on purpose, since listing the trash and
restoring out of it both need them, so the check belongs at each placement site
rather than in the loader.
The refusal exists because a live item under a trashed parent is invisible and
permanent: no listing walks into a trashed folder, so no client can be told
where the item lives, no user can see it to move or delete it, and it holds
quota nobody can account for. A sync client told about one through the change
feed is worse off still — it waits for a parent that will never arrive and
never reports itself finished again. The state arises from an ordinary
two-device race — one device writes into a folder while another deletes it —
and once the write is taken there is nothing to repair it. Sync clients treat
the marker as Overtaken rather than a failure: the index walk carrying the
delete also trashes their copy of the folder, and anything inside it the server
never took is rescued out on the way.
That first check is a fast fail, not the guarantee. It is made before the
expensive part of the verb — quota, bytes, sealing, thumbnails, a
protection-level conversion — and a folder trashed during that work would be
marked deleted, sweep the children it could see, and finish before the new row
landed underneath it. DriveHelper::place_into($folder_id, $write) closes that
window: it takes a Postgres advisory lock keyed on the destination, re-reads the
folder, refuses with DriveDestinationTrashedException if it went to the trash,
and only then runs the write. Every verb that puts something into a folder goes
through it, drive_upload_init included: when the server already holds bytes
with the hash the client sent, that verb answers on the spot and creates the
file itself rather than staging an upload, which makes it a placement like any
other. soft_delete_folder_cascade() takes the same lock
around marking a folder deleted and sweeping what is directly inside it. Two
orderings remain and both are safe: the placement finishes first and the cascade
sweeps what it made, or the cascade finishes first and the placement is refused.
The cascade always takes a parent's lock before a child's, which is what keeps
two overlapping cascades from deadlocking; drive_upload_complete takes the
quota lock outside the placement lock, and nothing takes them the other way
round.
A file in the trash is not a place either. drive_upload_init and
drive_upload_complete both refuse a new version of one, with
reason: file_trashed and the file id. The row is hidden from every listing, so
a version admitted into it is a save that reports success and shows nothing —
bytes charged to quota that the owner cannot see, find, or delete. Coming back
out of the trash is drive_restore's job; a sync client whose edit beat
somebody else's delete sends the rescued bytes up as a NEW file where the old
one lived, which it can only do if this is refused. drive_upload_complete
re-asks after drive_upload_init has already asked, because the file can go to
the trash while the bytes are in flight.
Each verb turns the exception into its own refusal, because each has cleanup
only it knows about. drive_upload_complete discards the staged upload as it
refuses, rather than leaving it to expire — the destination is gone and the
transfer has nowhere left to land. drive_restore re-roots rather than refusing
when the parent it is coming back to is already trashed; it holds the same lock
over the write, so a parent trashed between the re-rooting decision and the
write leaves the item safely in the trash instead of bringing it back
invisible.
One case the rule cannot reach, stated honestly: a Fortress file's fil_title
is an opaque enc-… identifier and its real name lives inside encrypted
metadata the server cannot read, so two client-encrypted files can share a real
name and differ here. That is the price of the server not being able to see the
name, and noticing it is the client's job.
Protection levels
Every folder carries a protection level (fol_protection_level), and every file
records the level it was stored at (fil_protection_level). Drive shows three
rungs of the platform ladder (includes/ProtectionLevel.php):
| Standard | Private | Fortress | |
|---|---|---|---|
| Promise | The server manages these files for you | Encrypted at rest — opened only while you're present | Plaintext never exists on the server |
| Custody | none | server (sealed to your vault, opened in-window) | client (your browser holds the keys) |
| Bytes on disk | plaintext | SealedFileContainer | browser-made ciphertext |
| Previews, thumbnails, AI | always | while your unlock window is open | never, server-side |
| Names, sizes, types | plaintext | plaintext | encrypted (in fil_encrypted_metadata) |
| Public links | yes | no | files only, key in the URL fragment |
| Member grants | yes | no | yes (per-user wrapped keys) |
| Sync clients | yes | no — a daemon holds no unlock window | yes, under device custody |
| Survives a stolen database or backup | ✗ | ✓ | ✓ |
File::is_encrypted() means Fortress, exactly and only; File::is_sealed()
means Private. Both exports carry protection_level and a syncable flag, and
encrypted on an export means Fortress so a client that only ever knew two
modes keeps reading it.A Private file's export also carries requires_window: true — its bytes and
its thumbnail open only inside the owner's unlock window, and that window is
keyed to the browser session (see Sealed Vault). A caller
holding an API key has no session cookie to present when it follows a link, so
for that caller download_url and thumb_url are omitted rather than minted:
every one of them would answer 423, and a listing of broken tiles is a worse
answer than a stated reason. A browser-session caller gets the URLs as normal.
A protected tree is a top-level tree. A folder's level is the floor for everything inside it, and subtrees are uniform: a level is chosen when a top-level folder is created, a subfolder inherits its parent's, and the create/move/link triad refuses anything else. That uniformity is what lets a public link on a Standard folder trust its whole subtree.
A refusal reads a file's effective level — the stronger of its own and its
folder's (DriveHelper::effective_file_level()). The two disagree on purpose
while a level change converges: the folder's column is the truth about what is
promised, the file's about what its bytes are. Public links and member grants
are refused on the promise, so nothing is minted in that gap only to be revoked
when the batch arrives. Byte work — sealing, unsealing, quota — reads the file's
own level, which is the one describing what is on disk. A move that would seal a
file carrying a live link or grant is refused outright, naming what is in the
way.
Changing a level (drive_level_change) is Standard ↔ Private only — the two
the server holds a key wrapping for. It runs the platform's one level-change
sequence (Protection Levels) with the tree as the scope
(DriveFolderLevel): the owner confirms with a second factor when they have one
(the action answers requires_stepup and the page confirms with a passkey and
applies again), the folder changes at once, so everything uploaded from that
moment lands at the new level, and the files already inside are converted
afterwards by repeated drive_level_batch calls, each bounded by a byte budget
rather than a row count; whatever the dialog does not finish, the vault's deferred
work (drive_level) finishes in the owner's window, and applying the same level
again resumes it. Raising needs only the owner's vault public key and so
runs locked; lowering decrypts and needs the window. Going Private ends any
public links and member grants in the subtree — the first call reports them and
does nothing until the caller confirms, and a prerequisite that refuses the
change anyway answers before that prompt. Each change records a
level_changed row in the change feed.
Private is described in full in Drive Encryption, which covers both custody models.
Access and grant semantics
A Drive file is private, so File::is_viewable() returns true for the owner or an
admin. Sharing adds one clause: a viewer or editor grant on the file, or on
any ancestor folder also grants view. The one grant-reach implementation is
DriveHelper::grant_reaches() (a recursive-CTE ancestor walk bounded by
drive_max_folder_depth); is_viewable and every drive_* action resolve
sharing through it. DriveHelper::can_write() adds the same reach for editor
— an editor may rename, save a new version, and upload into a shared folder.
Delete, trash, and share stay owner-only; an editor grant never confers
them. "Shared with me" is FileAccessGrant::entity_ids_for_user; filename
search spans the caller's own files plus granted files and granted folder
subtrees.
Trees have a single owner. Everything under a folder belongs to the
folder's owner: an upload into a shared folder creates a file owned by (and
billed to) the folder owner, and a folder made inside one is owned by the
folder owner too, whoever performs either (the actor is recorded in the change
feed); and drive_move only accepts a destination folder owned by the moved
item's owner, or the top level when the caller is that owner: the top level is
the owner's, and an editor's grant, which reaches the item through a share,
does not cover taking it out of the share. That invariant is what makes the
folder trash and delete-forever cascades — which select by folder — safe: they
can never touch another user's rows. It is also why a grantee browsing a shared
folder sees the full listing (children are listed by the folder owner's id). A
grantee's breadcrumb is cut at the topmost folder that carries a direct grant
for them, so the owner's private ancestor names are never exposed.
Grant management is a full-set reconcile: drive_share_sync takes grants (a
JSON object mapping each grantee — a user id or an email, resolved
server-side — to a role) and FileAccessGrant::sync_for_entity inserts new
grants, updates changed roles, and hard-deletes ones no longer in the set
(revocation is a row deletion; there is no delete_time). Each member it removes
is told through a change-feed row addressed to them (see Change feed).
Newly-granted users get a drive_share notification, gated by their
NotificationPreference.
Public links (drive_link_create, owner-only, gated by the drive_share_links
tier feature) mint a FileShareLink: the raw token is returned once, only its
SHA-256 is stored. A live link (not revoked, not past fsl_expires_time, password
satisfied) grants view at /s/{token} — anonymous, no login. The page streams
files through a short-lived signed URL and renders folders as a read-only listing
scoped to the shared subtree; the link is the durable revocable grant, the signed
URL is the transport. drive_link_revoke stamps fsl_revoked_time.
Encrypted files (files inside an encrypted vault folder) add a second layer to
the share dialog: a FileAccessGrant grants access, and a FileKeyGrant grants
readability — the owner's browser wraps the file key to each recipient's Drive
vault public key, and an upload seals the new file's key to the destination's
full reader set. drive_key_grants_sync reconciles each file's key set to the
listed members (the owner's own key is always kept) and tells each member whose
key it added or removed through the change feed. An encrypted vault folder
exists only at the Drive root or inside another vault, so a plaintext subtree
never hides encrypted content.
Encrypted files rename via their re-encrypted metadata (drive_rename with
encrypted_metadata), never a plaintext name. Public links carry the file key
in the URL fragment and are single-file only (encrypted folders can't use them).
See Drive Encryption.
Upload protocol (resumable, sequential chunks)
Uploads never overwrite bytes directly — they flow through the one blob-ingestion path, so dedup and accounting are automatic. The protocol is the complete server contract for sync clients:
This protocol is not Drive-only. It is the platform's route for any file larger than a single request, and other subsystems use it by passing a
purposetodrive_upload_init— see API § Uploading something that is not a Drive file. Everything described below is thedrivepurpose specifically: quota, folders, encryption and the dedup short-circuit belong to Drive and do not run for others.
drive_upload_init— `{name, folder_id?, file_id?, size_bytes, sha256?, mime_type?}`. Gates folder/file write access, then the per-file size (drive_max_file_bytes; for an encrypted destination the gate isDriveHelper::encrypted_size_ceiling()— the cap plus the client container's fixed per-chunk overhead, since the cap means plaintext bytes and the upload arrives as ciphertext) and quota (`bytes_used + size_bytes <= drive_storage_bytes`) of the owner who will be billed — either limit at 0 is no limit, and the disk-space reserve (FileUpload::space_refusal) is then the only bound — the target file's owner, else the destination folder's owner, else the actor. Ifsha256matches a private blob the actor already possesses it short-circuits: retain the blob, create theFile(or a newFileVersionwhenfile_idis set), and return the file — no bytes transferred. Possession means the blob is already referenced by one of the actor's own files or file versions (FileBlob::find_dedup's$possessed_by_user_id): a client-claimed hash is not proof of content, so a foreign hash+size never matches — no cross-user disclosure and no existence oracle. Otherwise it creates aFileUploadand returns{upload_token, chunk_bytes}(the raw token; only its hash is stored).PUT /api/v1/drive_upload/{token}— the raw-body chunk transport (a pre-CRUD branch inapi/apiv1.phpthat reads its own body and writes its own response). Chunks are sequential only: the request carries `Content-Range: bytes <start>-<end>/<total>, and<start>must equal the server'sreceived_bytes` or the response is 409 with{received_bytes}so the client resumes from the right offset. Appends are serialized per upload with a Postgres advisory lock, and the offset is re-read under the lock, so concurrent PUTs for one token cannot interleave writes.GETon the same path returns{received_bytes, expected_bytes}. Scratch bytes accumulate in a part-file under{site_root}/storage/drive_uploads/, outside the web root. Chunk requests use their ownapi_uploadrate-limit bucket so a large upload never drains the general API budget; the transport's response helpers write the request-log row with the actual outcome (success on 2xx, failure with the status code otherwise).drive_upload_complete—{upload_token}. Verifies the byte count and (when given) the sha256, re-validates write access, and enforces the quota here, where bytes are admitted to storage: under a per-owner advisory lock it recomputes usage fresh and rejects the complete if the upload would land past a non-zerodrive_storage_bytes(the init check is only a fast-fail — N uploads opened while under quota cannot all complete past it; a rejected upload keeps its pending row so the user can free space and retry). Then it ingests throughFileBlob::createFromPath(server-side dedup applies — the server hashed the actual bytes), creates theFileowned by the folder owner — or a newFileVersionwhen the upload targeted an existing file — recomputes the owner's usage, records the change, and clears the pending row. Safe to retry, and covered by the standardIdempotency-Keymachinery. For an encrypted destination the complete also carries the opaque key/metadata payloads, and an encrypted version upload must reuse the file's key — see Drive Encryption for that contract.
drive_upload_chunk_bytes (8 MiB). Abandoned uploads (rows + part
files idle past drive_stale_upload_retention_hours, default 24) are swept by the
daily retention sweep.Versioning
Saving new content to a file demotes the current head blob to a FileVersion
and repoints the file at the freshly-ingested blob — the head blob's reference
is transferred to the version row, so refcounts stay correct without a
retain/release. The demotion holds the file's row (SELECT ... FOR UPDATE) and
reads the head from inside that hold rather than from the caller's copy, which
is what stops two uploads finishing at once from demoting the same blob twice
and leaving the head one of them installed referenced by nothing. The same hold
serialises fvr_version_number, which is MAX + 1 against a unique index. Restoring swaps the roles back. Both prune to the owner's
drive_versioning_depth (oldest versions released and deleted). Because a version
IS a reference to its blob, FileVersion::permanent_delete releases it, and the
file's fvr_fil_file_id rule is permanent_delete so deleting a file releases
every version's bytes. drive_versions lists a file's history; drive_version_restore
promotes one back to head. The web UI uploads a new version by sending file_id
to drive_upload_init.
Quota accounting
Everything a member owns counts. DriveUsage::sum_for_user($user) sums the
blob sizes of every file with the member's id on it — a Drive upload, a mail
attachment, a photo, a chat upload, a legacy untagged row — plus the prior
versions of their files, excluding only the system's own internal sources
(File::internal_sources(), files the member can neither see nor free).
Trashed files count until purged. Version bytes bill the file's owner (a
version row's fvr_usr_user_id records who saved it — audit only, since an
editor may save a version of someone else's file), and an upload into a shared
folder bills the folder owner, per the single-owner-tree rule above. Each
logical file bills its full size even when its bytes are deduped onto a shared
blob (dedup saves disk, not quota).
The sum is computed when asked, never incremented: files arrive through
every subsystem, most of which know nothing about Drive, so a cached counter
would go stale the moment an attachment landed. The meter
(DriveUsage::current_bytes) and both upload gates read the live sum;
DriveUsage::recompute($user) persists it to dru_drive_usage after Drive's own
mutations (upload / new version / restore / permanent delete), and the daily
DriveUsageReconcile task re-runs it for every file-owning user. The sum walks
fil_files by fil_usr_user_id, which is indexed for it.
A quota of 0 is no quota: the meter shows the number alone, and uploads are bounded only by the server's disk-space reserve.
The admin's site-wide meter
An admin (permission ≥ 5) sees a second widget under their own meter,
Everyone on this site: what every member's files weigh as stored and what
this server has left. drive_list returns it as site_storage (null for a
member) from DriveUsage::site_totals() — counted over blobs, not files, so
a deduped byte is counted once, and split into bytes_local / bytes_cloud
because a blob offloaded to the cloud bucket takes no room here.
bytes_available is the free space where uploads land, net of the reserve
uploads keep (DiskSpace::DEFAULT_FLOOR_BYTES), so it is what can actually
still be stored; it is null, and shown as unknown, when the host hides
disk_free_space.
Trash
Trashing soft-deletes; DriveHelper::soft_delete_folder_cascade deletes the
folder first (earliest timestamp in its cascade) then every descendant, so
restore_folder_cascade — which restores only descendants with
delete_time >= the folder's own — leaves a child trashed independently earlier
in the trash. "Delete forever" runs an impact preview
(DriveHelper::delete_impact) before permanent_delete_tree destroys the subtree
(a raw folder permanent-delete instead orphans its files to root via the
fil_fol_folder_id → null rule). The daily retention sweep permanently deletes
Drive items (fil_source = 'drive', plus folders) trashed longer than its
window (default 30 days) — a soft-deleted file belonging to another subsystem
is that subsystem's to reclaim and is never touched. Restoring a folder whose
parent is still in the trash re-roots it (with a name-collision suffix when
needed) so it never reappears inside an unreachable parent.
Change feed
Every mutation records one FileChange via `FileChange::record($kind,
$entity_type, $entity_id, $owner_id, $actor_id) — kinds created, content`,
renamed, moved, trashed, restored, deleted, grant_changed, level_changed (a
folder tree's protection level). The feed is
append-only and the primary key is the cursor. drive_changes takes a {cursor}
and returns the changes after it that the caller may see — their own entities plus
entities shared to them — with next_cursor. With scope: 'mine', a sync
client's scope, it returns only the changes to entities the caller owns: an
editor's change to them included, since that row is recorded against the owner,
and nothing shared in to the caller. A cursor that cannot be proven
contiguous with the retained window returns {reset: true}, with the feed's
head as next_cursor, so the client re-lists from scratch rather than silently
missing changes: a cold start (cursor 0) on a site that has recorded any
change, because the log keeps only its retention window and a file untouched
for longer has no row left; a cursor before the earliest retained row; a
cursor ahead of the head, which this server never issued (a site restored from
an older backup, or a copy of it); and, when the log is empty after a purge, a
cursor short of the head. The head is the change-id sequence's, which outlives
the trimmed rows, so a client reset once on a quiet site parks there and is not
reset again on every poll.
The daily retention sweep trims rows past drive_change_feed_retention_days
(default 30).
Rows addressed to one member. The feed decides what a caller may see by
where an entity is now, so a change that takes something away from a member
would never reach them. Such a change also records a row for each member it
touches, passing them as FileChange::record's sixth argument
(fch_audience_usr_user_id). drive_changes returns that row to them beside
their own and their shared entities' rows (not under scope: 'mine'), and its
owner_id still names the real owner. Anyone else who can see the entity sees
the row too; the feed returns no audience field, so a row says nothing of whom
it was for.
| Change | Row for each member it touches |
|---|---|
drive_share_sync removes a member's grant | grant_changed on the entity |
| a folder made Private ends its grants | grant_changed on each entity whose grant ended |
drive_move, or drive_restore bringing an item back at the top because its folder is still in the trash, takes the item out of a member's share | moved on the item (a folder's row speaks for its subtree) |
drive_key_grants_sync adds or removes a member's key to an encrypted file | grant_changed on the file |
DriveHelper::roots_changed_by_reparent). The row is moved whatever the
action was called: to the member it left, the item went somewhere their share
does not reach.A member who no longer reaches the item at all asks drive_stat about it and
finds it under not_yours, which carries no name and no place: the row tells
them it is gone for them, never where it went. One it moved to another of their
shares finds it there. A member given a key finds it in the file's
wrapped_file_key. A key re-wrapped for a member who already holds one is not
announced: the owner's browser re-wraps every recipient's key on every change to
a share, and a sealed box is randomized, so a different blob is the ordinary
case.
drive_move, a re-rooting drive_restore and drive_key_grants_sync write
the change and its addressed rows in one transaction, so no device reads a move
or a key the feed has not announced.
FileChange::record swallows and logs a failure only outside a transaction,
where the mutation has already committed; inside the caller's transaction it
rethrows, and the mutation fails whole.
Sync clients
A desktop client keeps a folder on a computer matching the user's Drive. Four server surfaces serve it, on top of the change feed above. The client itself — its state model, conflict policy, per-filesystem naming rules, and health model — is Drive Sync Client.
Content identity on exports
DriveHelper::file_export carries three fields a syncing client needs and a
browser ignores:
| Field | What it is |
|---|---|
content_sha256 | The head blob's hash. Plaintext bytes for a plaintext file, ciphertext bytes for an encrypted one — either way it identifies the content a client holds. |
modified_time | The mtime the uploading client declared (fil_content_modified_time), so a file copied back down keeps its original timestamp. Plaintext files only. |
head_change_id | The feed position of the change that established the current content (content, or created for a file uploaded once). Lets a client say "what I have matches position N" without hashing. 0 means no feed row exists — compare hashes instead. |
modified_time: a timestamp on
ciphertext would leak when the file was last worked on, so the real mtime lives
inside the encrypted metadata blob. drive_upload_init refuses a
modified_time parameter for a vault destination for the same reason.Listings prime these in two queries for the whole page
(DriveHelper::prime_sync_meta()); a mutation that changes head content within
a request calls DriveHelper::forget_sync_meta() so the export it returns
reflects what it just wrote.
Standing refusals
A refusal the same request would get again — against the same entity, the same
target and the same account limits — carries standing: true in its data,
beside any reason: a quota or per-file limit, an owner-only trash or move, no
write access to the destination, a protection boundary, a file in the trash, an
invalid destination, the folder depth limit, a malformed request. A sync client
holds a standing refusal rather than asking again until something it asked
about changes. A refusal that depends on bytes in flight (a size or checksum
mismatch), a cursor (changed_since), time or the server's health is not
standing, and is retried.
The completion's integrity refusals (bytes missing, the wrong size, a failed
checksum) say reason: 'upload_incomplete': a client sends the bytes again,
and never reads a name clash into a refusal that names its reason.
A new version says what it was based on. drive_upload_init and
drive_upload_complete take base_sha256, the head content the edit started
from (for an encrypted file, the ciphertext's hash, as content_sha256 gives
it), and for an encrypted file base_metadata_sha256, the hash of the metadata
blob the new one was sealed from. A head holding other content refuses with
head_moved, a metadata blob rewritten since by a rename with moved_since;
both carry the head as it stands (head_sha256, head_size,
head_change_id) and are not standing. Init asks before a byte is sent, when
the upload names its hash; the answer that counts is given where the version is
written, under the file's row lock, and a refused completion discards its
staged bytes. A head already holding the upload's content is no new version:
the upload landed before, or two devices saved the same bytes, and the call
succeeds with nothing written. A version that sends no base is not checked: it
replaces whatever the head holds, as a browser's replace does. A success does
not say whether the server checked a base.
Refusal reasons
Every refusal from the actions a sync client calls (drive_changes,
drive_index, drive_stat, drive_list, drive_upload_init,
drive_upload_complete, drive_folder_create, drive_rename, drive_move,
drive_trash, drive_public_keys) carries a machine-readable reason in its
data, and drive_changes and drive_index answer with refusals: 'reasoned'
to say so. A client that has seen the marker acts on the reason alone, never
on the message, which is prose for people.
| Reason | Meaning |
|---|---|
name_taken | another file or folder holds the name there (with the holder) |
parent_trashed | the destination folder is in the trash |
file_trashed | the file is in the trash |
changed_since | something in it changed after the client's cursor |
moved_since | the item stands in another folder, or under another name (for an encrypted file, another metadata blob), than the move, rename or new version was decided against |
head_moved | a new version's file holds other content than the head it was based on (with the head: head_sha256, head_size, head_change_id) |
upload_incomplete | the upload's bytes are missing, the wrong size or fail their checksum |
quota, file_too_large | the account's storage or per-file limit |
no_room | the server's disk is short of space |
storage_failed | the bytes could not be stored, sealed or converted |
not_found | the entity, the file to update or the upload is gone |
parent_not_found, invalid_parent | the parent or destination folder is gone or not one |
forbidden | no permission, no access, or not the owner's to do |
protection_boundary, level_change_refused | a protection level the request would cross or change |
vault_needed | a Private change needs the vault set up or unlocked |
key_unavailable | the caller does not hold the file's key |
move_into_self | a folder moved into its own subtree |
too_deep | the folder depth limit |
name_too_long | the name is over the server's limit |
invalid_request | the request itself is malformed: a field missing or out of range, an unknown scope or cursor, encrypted metadata or keys missing |
drive_disabled, signed_out | Drive is off on the site, or there is no session |
idempotency_key_reused | the key was spent on a different request |
action_failed | the action itself broke (an exception, for any action) |
name_taken refusal names what holds the name, from every action that
refuses one (DriveHelper::name_taken_data()): its type, its id, its name as
the server stores it, and for a file the hash of the bytes the server holds
(the content_sha256 drive_index reports; for an encrypted file, of the
ciphertext). A sync client that holds those same bytes knows the name is held
by its own upload, one that landed while its answer was lost, and adopts that
file instead of sending a second copy.{
"reason": "name_taken",
"holder_type": "file",
"holder_id": 1963621,
"holder_name": "notes.txt~",
"holder_sha256": "833be3070c2f9a6a1f0de7c3d2b1a4e5f60718293a4b5c6d7e8f9a0b1c2d3e4f"
}A folder holder has no holder_sha256. The holder is named only to a caller
who can read it: a user granted one file alone may rename it onto a sibling's
name, and is told only that the name is taken. A holder already gone when the
refusal is made (the loser of an insert race asks again) also leaves the bare
reason. This example, and a folder's, is
sync/jd-proto/tests/fixtures/name_taken_refusal.json; the platform's
drive_sync_contract test, the client's parser and jd-sim's server are all
tested against that file.
A move or rename can say where the client last saw the item: drive_move
takes from_parent_id (0 for the top of the Drive) and drive_rename takes
from_name. Where the item stands somewhere else now, the request is refused
with moved_since and changes nothing, so a client that decided a move before
another device's move reached the server cannot put the item back over it
unseen. Each asks only about the half it changes: a rename elsewhere does not
refuse a move, and a move elsewhere does not refuse a rename. A move to where
the item already is, a rename to the name it already has, and a rename to the
blob it already holds succeed whatever they say. drive_move asks again under the owner's write lock where the move
lands; drive_rename asks once, so two requests a few milliseconds apart can
still both land. from_name is read for plaintext names only: an encrypted
file's title is its opaque placeholder. An encrypted file's rename takes
from_metadata_sha256 instead, the SHA-256 (hex) of the metadata blob the new
one was sealed from; a stored blob that is another, rewritten by a rename or a
new version since, is refused the same way. A request that says none of these
is carried out wherever the item stands, as the Drive pages' own moves and
renames are.
A request the API's own input schema refuses, before the logic runs, comes
back as a ValidationError instead.
drive_stat — batch fetch
The change feed carries id-only rows so it stays cheap however far behind a
client is; this turns a list of ids back into entities in one round trip.
{entities: [{entity_type, entity_id}, …]}, up to 500, deduped. Entities that
are gone or no longer visible come back under missing rather than as an
error — a client must be able to tell "delete the local copy" from "retry
later". Those that still exist but are no longer the caller's to see are also
listed under not_yours, a subset of missing; with scope: 'mine' an entity
the caller does not own is answered that way too, however it is shared to them.
Signed URLs are withheld unless urls: true.
drive_index — full walk
Cold start, and whatever drive_changes returns {reset: true}. Keyset
paginated: pass the previous page's next_after_id back as after_id and stop
when done is true. The cursor is an opaque token (folder:123 / file:456)
because folders and files have separate id spaces and a bare integer cannot say
which it points into. Folders come before files, so a client materializing as it
reads always has somewhere to put the next file.
scope: 'mine' walks everything the caller owns; scope: 'shared' walks
everything they reach through grants, each item annotated with grant_root —
the granted entity it hangs off — so the client can mount it under the right
"Shared with me" root. Trashed items are included with deleted: true: a client
that could not see the trash would read a trashed file as vanished and delete
the local copy, and would never recognize a restore.
vault_client_probe — lean vault probe
{scope} (default drive; any registered client-custody scope) →
{set_up, public_key, key_generation}, reachable with a session key. Enough to seal file keys for uploads and to notice a key rotation.
Wrappings, salts, and KDF parameters are unlock material and stay on the
browser-only vault_client_status.
Device linking
A sync client cannot sign a user in well: it has no trustworthy password field, WebAuthn does not work outside a browser, and a passkey-first account has no password at all. So it does not try. It opens a ceremony, shows a code, and the user approves in the browser they are already signed into.
- The client calls
POST /api/v1/auth/device_linkwith its name, platform, and (optionally) an X25519 public key, and gets a code plus a poll token. - The user opens
/profile/devices/link?code=…, which requires a signed-in session and a recent step-up, and shows what is asking — name, platform, and the address the request came from. If they have encrypted folders they may tick a box to give this device access, and one more box for each other browser-held vault they have set up (the password vault, say): for each ticked vault the browser unlocks it and seals its secret key to the device's public key (session.sealSecretKeyTo()). Approval callsdrive_device_link_approve(Drive's key insealed_vault_key, the others insealed_vault_keys), which mints the sessionApiKey, creates theSyncDevicewith the scopes it was handed (sde_vault_scopes), and parks the sealed keys and the encrypted one-time secret on the ceremony row. - The client polls
POST /api/v1/auth/device_link/{poll_token}and collects the credential exactly once — withsealed_vault_keyandsealed_vault_keyswhen it was given vaults; the row is scrubbed immediately after.
SyncDevice (sde_sync_devices) is the device identity, paired
1:1 with the session key it authenticates with, holding the device public key,
last check-in, and last acknowledged cursor. DeviceLink (dlk_device_links)
is the ten-minute ceremony state: code and poll token stored only as hashes, the
minted secret SecretBox-encrypted at rest and deliverable once.
The daily retention sweep removes finished rows past
drive_device_link_grace_minutes (default 60).Codes are 8 characters of a Crockford-style alphabet, and lookalike characters
fold on entry (O→0, I/L→1) so a font cannot defeat someone reading a
code off another screen. Wrong codes are counted per IP address rather than per
ceremony — a guesser by definition matches no row — and 20 misses in 15 minutes
shuts that address out.
Management. drive_devices, drive_device_rename, drive_device_revoke,
surfaced on /profile/security alongside App Sessions. Revoking unlinks the
device
Liveness comes for free. When the caller of drive_changes maps to a
SyncDevice, the handler stamps sde_last_seen_time (throttled to hourly, like
apk_last_used_time) and sde_last_cursor from the request. There is no
separate heartbeat call and therefore no client that can forget to send one, and
the security page can say "last synced 4 minutes ago" — which is what makes a
stalled device visible instead of silent.
Settings and tier features
Settings (settings.json): drive_active ('0'), drive_max_folder_depth
('32'), drive_upload_chunk_bytes ('8388608'), api_upload_rate_limit_requests
('10000'), api_upload_rate_limit_window ('3600').
Tier features (includes/core_tier_features.json), read via
SubscriptionTier::getUserFeature($uid, $key, $default): drive_storage_bytes
(quota over everything the member owns; 0 is no quota), drive_max_file_bytes
(per-file cap; 0 is no cap), drive_share_links (boolean),
drive_versioning_depth (versions kept). A tierless member — the owner of a
self-hosted site — uploads without limit, bounded by the disk-space reserve.
