Status: Normative. This document is written to be published verbatim (D-007). Audience: implementers of Copy Shelter, and anyone writing an independent reader.
If it is on disk, it is described here. There are no undocumented fields, no reserved-for-internal-use regions, and no behaviour that depends on state outside the repository.
Keywords MUST, MUST NOT, SHOULD, MAY are used as in RFC 2119.
The format exists to satisfy six properties. Every decision below traces to one of them.
| P1 | An interrupted write MUST NOT produce an object that appears valid. |
| P2 | A repository MUST be fully recoverable using only its own contents plus a key. |
| P3 | Replication MUST reduce to "copy the objects the target lacks", in any order, resumably. |
| P4 | Deleting one snapshot MUST NOT invalidate any other retained snapshot. |
| P5 | The destination holder MUST NOT learn filenames, paths, sizes of individual files, or content. |
| P6 | The format MUST tolerate destinations that are slow, case-insensitive, eventually consistent, and hostile to many small files. |
<repository-root>/
copyshelter.repo plaintext, ~1 KB
keys/
<keyID>.key wrapped master key, one per unwrap path
packs/
<first-2-hex>/<pack-id>.pack immutable, 32–64 MB target
index/
<index-id>.idx immutable
snapshots/
<snapshot-id>.snap immutable
refs/
head mutable, signed
replicas/
<replica-id>.state mutable, signed
gc/
<run-id>.mark mutable, signed
holds/
<hold-id>.hold mutable, signed
locks/
<lock-id>.lock ephemeral
devices/
<device-id>.status mutable, signed
tmp/
<uuid> scratch; deleted on open
Every filename is a hexadecimal identifier or a fixed literal. No user-derived string ever appears in a path (P5).
packs/ is sharded by the first two hex characters of the pack ID, giving 256 directories, so no directory exceeds a few thousand entries on realistic repositories. This matters on SMB and File Provider (P6).
copyshelter.repoThe only plaintext file. It contains public parameters required to *attempt* decryption; it contains no secrets.
{
"magic": "copyshelter-repository",
"formatVersion": 1,
"repositoryID": "1HQ702Et94FXRv7oucFuqjKXHvYpXUYPovlrUyFZK9Y=",
"createdAt": "2026-09-01T14:22:31.000Z",
"createdBy": "CopyShelter/1.0 (26.5.0)",
"cipher": "ChaCha20-Poly1305",
"chunker": {
"algorithm": "FastCDC",
"minSize": 524288,
"avgSize": 1048576,
"maxSize": 4194304,
"normalization": 2
},
"hash": "SHA-256",
"blobID": "HMAC-SHA256",
"signature": "Ed25519",
"packTargetSize": 67108864
}
repositoryID is base64 of the 32 raw bytes, not hex. Those 32 raw bytes, decoded, are what every derivation below uses as its HKDF salt. This document said "32 hex bytes" until an independent reader was written against it and failed on the first field it needed.
Keys are sorted alphabetically and timestamps carry milliseconds, because this file is written with a canonical encoder so that two implementations produce identical bytes.
There is no kdf block. Argon2 parameters would be needed only by the passphrase unwrap path, which format version 1 does not implement: the two paths that exist are recoveryKey and keychain, and neither uses Argon2. A writer MUST NOT emit one and a reader MUST NOT require one.
A reader MUST refuse to operate on a formatVersion it does not understand. A reader MUST NOT trust any value here to be authentic: this file is unauthenticated by necessity, since it describes how to derive the keys that would authenticate it. Tampering with it can only cause key derivation to fail, never to succeed with a wrong key: every subsequent object is AEAD-authenticated.
masterKey := 32 cryptographically random bytes, generated once per repository dedupKey := HKDF-SHA256(ikm: masterKey, salt: repositoryID, info: "copyshelter.v1.dedup", L: 32) objectKeyRoot := HKDF-SHA256(ikm: masterKey, salt: repositoryID, info: "copyshelter.v1.object", L: 32) signingSeed := HKDF-SHA256(ikm: masterKey, salt: repositoryID, info: "copyshelter.v1.sign", L: 32) metadataKey := HKDF-SHA256(ikm: masterKey, salt: repositoryID, info: "copyshelter.v1.meta", L: 32) signingKey := Ed25519 private key seeded with signingSeed
masterKey MUST NOT be stored anywhere in plaintext, including logs, crash reports and diagnostics.
keys/<keyID>.keyOne file per unwrap path. keyID = first 16 bytes of SHA-256(wrappingParameters), hex.
{
"keyID": "d9150c93124da8e969a47973763cc04d",
"method": "recoveryKey" | "keychain",
"createdAt": "2026-09-01T14:22:31.000Z",
"hint": "optional, user-supplied, MAY be empty",
"nonce": "base64, 12 bytes",
"wrapped": "base64, 48 bytes: the 32-byte masterKey sealed with ChaCha20-Poly1305, tag appended"
}
The method string is the associated data. The seal is
wrapped := ChaCha20-Poly1305.seal(masterKey, key: wrappingKey, nonce: nonce, aad: utf8(method))
so a key file cannot be relabelled from recoveryKey to keychain and offered to a different unwrap path. This was missing from earlier drafts of this section, and without it every unwrap fails its authentication tag with nothing to say why.
There is no separate check field. Earlier drafts described one, over the literal ASCII copyshelter-key-check; nothing has ever written it. The AEAD tag on wrapped already answers the only question a check field would: a wrong key fails to open it. A reader MUST NOT require a check field and MUST NOT treat its absence as damage.
Wrapping key by method:
| Method | Wrapping key |
passphrase | Argon2id(passphrase, kdfSalt, params from copyshelter.repo) |
recoveryKey | HKDF-SHA256(ikm: recoveryKeyBytes, salt: repositoryID, info: "copyshelter.v1.recovery", L: 32) |
keychain | 32 random bytes stored in the macOS Keychain, referenced by keyID |
A repository MUST have at least two key files at all times, and MUST have at least one that is not keychain. Implementations MUST refuse to delete the last non-keychain key file.
recoveryKeyBytes is 25 random bytes (200 bits), not 16. This is deliberate, not a rounding error: 200 bits is exactly 40 Crockford Base32 characters (200 / 5 = 40) with nothing left over, so the encoding below is a straight bijection. There is no padding bit, no derived filler, and nothing implementation-defined. An earlier draft of this section specified 128 bits rendered as 40 characters, which is inconsistent (128 bits needs only 26 Crockford characters) and was caught during CS-031: it could not have been implemented interoperably as written. 200 bits costs nothing extra to generate or store and removes the ambiguity entirely.
Alphabet. Crockford Base32, no I, L, O, U:
0123456789ABCDEFGHJKMNPQRSTVWXYZ
Encoding (data characters). Read the 25 bytes as a single 200-bit big-endian integer. Starting from the most significant bit, take 5 bits at a time and map each 5-bit value through the alphabet above by index. This produces exactly 40 characters, grouped for display as 8 groups of 5. Decoding is the exact inverse: concatenate the bits each character represents, most significant character first, and read the result back out as 25 bytes.
Checksum (group 9). Let data be the 40 characters above, upper case, with no hyphens.
mac := HMAC-SHA256(key: literal "copyshelter.v1.rkcheck", message: data) // 32 bytes checkBits:= mac[0], mac[1], mac[2] // first 3 bytes, 24 bits
24 bits is one bit short of the 25 bits five Crockford characters hold. Append a single 0 bit at the least significant end (standard base32 tail padding) to make 25 bits, then split those 25 bits into 5 groups of 5 bits, most significant first, and map each through the alphabet. That is group 9.
Normalization on input. Decoding MUST accept lower case, and MUST map I and L (either case) to 1, and O (either case) to 0. Hyphens and any whitespace are ignored, in any position.
Validation order. A reader MUST validate the checksum before attempting Argon2/HKDF, so a mistyped key produces "that key is mistyped" rather than "wrong key".
Worked example, so an independent implementation can check itself byte-for-byte:
input (25 bytes, hex): 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 data characters (40): 000G40R40M30E209185GR38E1W8124GK2GAHC5RR checksum (group 9, 5 characters): 7X3W6 formatted (45 characters, 9 groups of 5): 000G4-0R40M-30E20-9185G-R38E1-W8124-GK2GA-HC5RR-7X3W6
A reader that decodes 000G4-0R40M-30E20-9185G-R38E1-W8124-GK2GA-HC5RR back to bytes MUST get the 25-byte input above exactly, and MUST compute 7X3W6 as the checksum over the 40 data characters.
Every timestamp in a repository is ISO 8601 with fractional seconds, in UTC, and carries millisecond precision. For example 2026-09-01T14:22:31.482Z.
Implementations MUST round a timestamp to milliseconds before writing it. This is stated rather than left implicit because a value taken straight from a system clock has finer precision than the encoding, so writing it unrounded makes a record that is not equal to itself after a round trip. Nothing fails when that happens: timestamps simply drift by microseconds between memory and disk, and comparisons quietly start lying.
A blob is the unit of deduplication. Every blob has a type and an ID.
blobID := HMAC-SHA256(key: dedupKey, message: plaintext) // 32 bytes
Keying the hash with a per-repository secret means a party holding the repository but not the key cannot test whether a known file is present (P5).
| Blob type | Contents |
0x01 data | A chunk of file content produced by the chunker |
0x02 tree | A directory object (§7) |
0x03 meta | Reserved. Format version 1 stores every xattrs value and acl inline in the tree entry (§7), with no size limit other than the field's own u32 length prefix. Nothing in format version 1 produces a 0x03 blob; a future version MAY use it to spill oversized metadata out of a tree entry, which would be a new formatVersion, not a change to existing objects. |
objectKey := HKDF-SHA256(ikm: objectKeyRoot, salt: blobID, info: "copyshelter.v1.blob", L: 32) nonce := 12 zero bytes sealed := ChaCha20-Poly1305.seal(plaintext, key: objectKey, nonce: nonce, aad: blobID || blobType)
A fixed nonce is safe here and MUST be used: objectKey is uniquely determined by blobID, which is uniquely determined by the plaintext. Identical plaintext therefore produces identical ciphertext, which is required for deduplication, and no (key, nonce) pair is ever reused with different plaintext.
Implementations MUST include blobID || blobType as associated data, so a blob cannot be relabelled or substituted for another.
A pack is an immutable concatenation of sealed blobs plus a sealed directory.
offset 0 ┌────────────────────────────────────────────────────────────────────┐ │ HEADER 8 bytes "CSPK" 0x01 0x00 0x00 0x00 │ ├────────────────────────────────────────────────────────────────────┤ │ BLOB 0 sealed bytes │ │ BLOB 1 sealed bytes │ │ ... │ ├────────────────────────────────────────────────────────────────────┤ │ DIRECTORY sealed with metadataKey; a packed array of fixed │ │ size records, 45 bytes each: │ │ blobID(32) type(u8) offset(u64 LE) length(u32 LE)│ │ preceded by a u32 LE record count │ ├────────────────────────────────────────────────────────────────────┤ │ TRAILER 12 bytes directoryLength(u32 LE) │ │ "CSPKEND" 0x00 │ └────────────────────────────────────────────────────────────────────┘ packID := hex( SHA-256( all bytes of the file ) ) filename := packs/<packID[0..2]>/<packID>.pack
A pack is self-verifying. Its name is the hash of its own contents, so a truncated or corrupted pack can never be mistaken for a valid one (P1). A reader MUST verify the pack hash before trusting its directory, except during streaming restore where it MUST verify each blob's AEAD tag instead and MUST verify the whole-file hash before reporting the pack as verified.
Packs SHOULD target 64 MB and MUST NOT exceed 256 MB. A pack MAY be smaller when a backup run ends.
The directory is sealed with metadataKey and a random 12-byte nonce stored in the first 12 bytes of the sealed directory region, because directory plaintext is not content-addressed.
The directory is a packed array of fixed size records rather than a self describing encoding such as CBOR. Every field is a fixed width integer or a fixed length identifier, so a self describing encoding would add a parser and a dependency while describing nothing a reader does not already know from this document. Tree objects (§7) use the same reasoning and the same style of encoding, with a per-entry presence bitmap standing in for the optional fields a fixed size record cannot express.
A reader MUST refuse a directory whose declared record count does not match the region length exactly, and MUST refuse any record whose offset + length falls outside the blob region. A directory pointing outside the pack is corruption and must be rejected before anything reads there.
A tree is a 0x02 blob describing one directory: a flat list of entries, one per child. It is packed binary with a per-entry field-presence bitmap, not CBOR.
Every other self-describing decision in this document (§6) exists to preserve unknown keys across a rewrite. Trees never rewrite: they are content addressed, so an entry with a changed field produces a different blob under a different ID, not the same object edited in place. There is therefore nothing for a self-describing encoding to protect here, and it would only add a parser and a dependency to describe fields already fixed by this document. A future field addition bumps formatVersion (§17) instead of relying on unknown-key preservation. Every field below is a fixed width integer or a length-prefixed byte run, exactly like the pack directory (§6) and the index file (§8).
All integers are little endian.
u32 entryCount
per entry:
u16 presenceBitmap which optional fields follow, see the bitmap table below
u8 entryType 1 file, 2 dir, 3 symlink, 4 fifo, 5 socket, 6 char, 7 block
u8 status 0 ok, 1 unstable (changed during read), 2 unreadable
u16 nameLength, then that many RAW name bytes
u32 mode POSIX mode bits
u32 uid uid as recorded
u32 gid gid as recorded
i64 mtimeSeconds, u32 mtimeNanoseconds
i64 ctimeSeconds, u32 ctimeNanoseconds
then, only where the matching presence bit is set, in this fixed order:
birthtime i64 seconds + u32 nanoseconds
size u64 logical size in bytes
bsdFlags u32 BSD st_flags (UF_HIDDEN, UF_IMMUTABLE, ...)
userName u16 length + bytes UTF-8 text; for cross-machine remapping
groupName u16 length + bytes UTF-8 text
symlinkTarget u16 length + RAW bytes
hardlinkGroup u64 hard-link group id, snapshot-local
xattrs u16 count, then per xattr:
u16 nameLength + name bytes
u32 valueLength + value bytes
acl u32 length + bytes NFSv4 ACL text form, from acl_to_text
sparseRanges u16 count, then per range: u64 offset + u64 length
contentBlobs u32 count, then that many 32 byte blob identities, in order
childTree 32 byte blob identity
Presence bitmap. Bit order and field order are the same thing on purpose: a reader never has to jump around to know what comes next. Bit 0 is the least significant bit.
| Bit | Field |
| 0 | birthtime |
| 1 | size |
| 2 | bsdFlags |
| 3 | userName |
| 4 | groupName |
| 5 | symlinkTarget |
| 6 | hardlinkGroup |
| 7 | xattrs |
| 8 | acl |
| 9 | sparseRanges |
| 10 | contentBlobs |
| 11 | childTree |
Bits 12 through 15 are unused by format version 1. A reader MUST ignore unset bits it does not recognise only insofar as format version 1 defines no such bits; in practice this means bits 12-15 MUST be zero when writing, and a reader of format version 1 has nothing else to skip, because there is no unknown-field region left to preserve the way there would be in a self-describing encoding. A version that adds a field defines a new bit and a new formatVersion, per §17.
Notes that matter:
name and symlinkTarget are raw bytes, never text. macOS stores NFD-normalised UTF-8 onHFS+/APFS; other filesystems do not, and some filenames are not valid UTF-8 at all. Copy Shelter stores exactly what the filesystem returned and restores exactly those bytes. It MUST NOT normalise, because normalising is lossy and irreversible. userName and groupName are the one text field in an entry: they are POSIX account names used only for cross-machine remapping, never filesystem paths, so UTF-8 text is the right representation for them.
hardlinkGroup groups hard links within a snapshot. On restore, the first occurrence ismaterialised and subsequent occurrences are linked to it. Hard links spanning outside the backup set cannot be preserved; they are restored as independent files and reported.
status: 1 (unstable) means the file changed while being read and Copy Shelter could not obtaina consistent copy after 3 attempts. The content stored is the last read attempt. This MUST be surfaced in the UI and MUST NOT be reported as a clean backup.
acl is the NFSv4 ACL text form produced by acl_to_text. On restore to a different machine,entries referencing unresolvable principals are dropped and reported rather than failing the file.
com.apple.decmpfs): content is stored decompressed. The compression xattr isnot preserved. Restored files are functionally identical and occupy more space.
in the xattrs list with valueLength == 0; the latter never appears in the list at all. The same distinction holds one level up: presenceBitmap bit 7 set with xattrs count 0 (the field is present, the directory has no xattrs to report) differs from bit 7 unset (the field was not captured).
before using it to read further, and MUST refuse the object rather than read out of range or return a partially decoded result. A truncated or corrupted tree is corruption, not a smaller tree.
Every blob is compressed before it is encrypted, and the result is stored as:
[method: 1 byte][original size: 8 bytes, little endian][payload]
That whole structure is what gets sealed, so the marker and the size are covered by the same authentication tag as the content and cannot be altered independently.
method is 0 for stored as-is and 1 for LZFSE. A reader MUST refuse a blob whose method it does not recognise rather than returning the payload unchanged.
Order. Compression happens before encryption because that is the only order that achieves anything: ciphertext is indistinguishable from random and does not compress. Identity is still the keyed hash of the ORIGINAL plaintext, so deduplication is unaffected and two implementations that make different compression choices still agree on what a blob is.
A blob is only stored compressed if it shrank by at least 10%. Marginal wins are not worth decompressing on every read for the rest of the backup's life, and refusing them keeps already-compressed media on the fast path. Blobs under 1 KiB are not attempted at all.
A writer may decline to attempt compression at all. This implementation compresses a 64 KiB sample, spread across the blob as 128 windows rather than taken from one place, and skips the full attempt when the sample saves less than 5%. That is a writer's choice about how to spend its own time and it changes nothing a reader has to do: the method byte says what happened, and a blob stored uncompressed is read the same way whether that was because compression did not help or because it was never tried. Two writers that make different choices here still agree on every blob's identity, because identity is taken over the original plaintext.
The original size is recorded, not inferred. compression_decode_buffer and its equivalents fill whatever buffer they are given and return a non-zero count when the output did not fit, so a reader that grows a buffer until the call "succeeds" never grows and returns a truncated file that looks whole. A reader MUST decode into a buffer of exactly the recorded size and MUST refuse the blob if the number of bytes produced differs.
What compressing before encrypting costs. The size of a stored blob reveals something about its plaintext: compressible data produces smaller blobs. Against somebody holding a stolen drive this is a weak signal, roughly "this chunk was text rather than a photo". It becomes a real attack only where an adversary can inject chosen content and observe the resulting size, which is the CRIME class of attack against TLS and does not apply to a backup: nobody is choosing what goes into somebody's Documents folder and then measuring their drive. The trade is stated here rather than left implicit.
Measured. On a mixed corpus of source code and prose, 11.2 MB of files stored in a 4.9 MB repository. On photographs it saves nothing, which is expected and is why the attempt is abandoned rather than forced.
index/<indexID>.idx indexID := random UUIDv4, hex without hyphens
Sealed with metadataKey, random nonce in the first 12 bytes. Plaintext is packed binary, in the same style as the pack directory (§6) and for the same reason: every field is fixed width, so a self-describing encoding would add a parser and a dependency while describing nothing this document does not already fix. All integers are little endian.
u32 sectionCount
per section:
packID(32)
u32 entryCount
per entry:
blobID(32) type(u8) offset(u64) length(u32) // 45 bytes, exactly as in §6
Earlier drafts of this section said CBOR. Nothing has ever written CBOR here. The 45-byte record is byte-identical to the pack directory record, which is the point: an implementation reads one record shape whether it is rebuilding from packs or loading an index.
Index files are immutable and additive. A reader loads all of them and merges. Duplicate blobIDs across index files are permitted and MUST resolve to any one of the entries: they are byte-identical by construction.
Index files are a cache derivable from the packs themselves. An implementation MUST be able to rebuild the entire index by reading every pack directory, and MUST do so automatically when the index is missing, unreadable, or inconsistent. This is what makes P2 hold even after catastrophic partial loss.
Index compaction: write a new merged index file, flush it to stable media, then delete the superseded ones. Safe in any crash ordering, because deleting an index never loses data.
Before a pack is deleted, every index entry naming it MUST be gone. In that order, and never the other way round.
The index is what a writer consults to decide a blob is already stored. An index naming a pack that no longer exists will tell the next backup that a blob is present when it is not, so that backup writes nothing, and commits a snapshot referencing bytes that are not there. It reports success and cannot be restored, and no reader can detect it until somebody tries to get the file back.
The rebuild path does not save an implementation from this. It runs when the index is missing or unreadable, and an index that is merely wrong is neither: it loads cleanly and lies. So the compaction above is not an optimisation to perform when convenient. It is the step that has to happen first.
Written down because this implementation had the bug and shipping the document without the rule would hand it to the next one. It was found by interleaving a backup after a prune, which is a sequence no test here had ever run.
snapshots/<snapshotID>.snap snapshotID := random UUIDv4, hex without hyphens
Layout:
┌──────────────────────────────────────────────────────┐ │ nonce 12 bytes │ │ sealed payload ChaCha20-Poly1305 with metadataKey │ │ signature 64 bytes, Ed25519 over (nonce||sealed)│ └──────────────────────────────────────────────────────┘
snapshotID is a UUID. It appears with hyphens inside refs/head and inside the payload, and without hyphens, lower case in the filename. A reader MUST strip hyphens when turning an identifier into a path.
Payload (JSON, keys sorted, UTF-8, no trailing newline):
{
"v": 1,
"id": snapshotID,
"counter": u64, // strictly increasing per repository
"deviceID": bytes(16),
"deviceName": text,
"parent": snapshotID or null,
"createdAt": "2026-09-01T14:22:31Z",
"hostPlatform": "macOS 26.1",
"appVersion": "1.0.0",
"sources": [ { "path": bytes, "kind": "folder"|"wholeMac"|"photos"|"mobileSync" } ],
"rootTree": blobID,
"excludeRules": [ text ],
"stats": { "files": u64, "dirs": u64, "bytesLogical": u64, "bytesNew": u64,
"bytesDeduped": u64, "unstableFiles": u64, "unreadableFiles": u64 },
"flags": [ "anomalyDetected", "partial", "inconsistentSource", ... ],
"anomaly": { "signals": [...], "detectedAt": "..." } // present iff flagged
}
Every field holding raw bytes, including rootTree and deviceID, is base64. Earlier drafts of this section said CBOR; nothing has ever written CBOR here, and an implementer following it decoded the first byte as a CBOR major type and stopped.
A snapshot is written only after every blob it transitively references has been durably committed (P1, P3). A reader encountering a snapshot referencing a missing blob MUST report the repository as damaged and MUST identify the affected files, rather than silently restoring a truncated tree.
refs/headThe only frequently-mutated file in the repository.
┌──────────────────────────────────────────────────────┐
│ nonce 12 bytes │
│ sealed payload ChaCha20-Poly1305 with metadataKey │
│ signature 64 bytes, Ed25519 │
└──────────────────────────────────────────────────────┘
payload = JSON, keys sorted, UTF-8:
{
"counter": u64, // max snapshot counter ever committed
"snapshots": [ snapshotID ], // the authoritative set, in commit order, hyphenated UUIDs
"updatedAt": "...",
"writerDeviceID": base64 of 16 bytes
}
JSON, not CBOR, for the same reason as §9: it is what is written. The signature covers nonce || sealed, and the sealed payload has no associated data.
A client MUST persist, per destination, the highest counter it has ever observed. On open:
counter > remembered → normal, update the remembered value.counter == remembered → normal.counter < remembered → rollback suspected. The destination is marked Degraded,the user is alerted, and no pruning may run against it. Backups may continue.
A client SHOULD additionally compare counters across destinations of the same repository and flag divergence.
refs/head is written with the standard write protocol (§11). A crash mid-update leaves the previous refs/head intact; the new snapshot exists on disk but is not yet referenced, and is collected as garbage on a later prune. Data is never lost by this ordering: only invisible for a moment.
Every write to a repository, without exception:
1. create tmp/<uuid> 2. write the complete contents 3. F_FULLFSYNC the file descriptor (see note below) 4. rename tmp/<uuid> → final path 5. F_FULLFSYNC the containing directory > **Why F_FULLFSYNC and not fsync.** On macOS, `fsync` hands the data to the > drive and returns. The drive may keep it in its own volatile cache and lose it > on power loss, and Apple documents `F_FULLFSYNC` as the only request for > stable media. Every ordering guarantee in this format assumes that a pack > genuinely survives before an index or a snapshot refers to it; with `fsync` > alone, a power cut can produce a repository whose contents point at bytes that > were never written. Implementations of this format must use `F_FULLFSYNC` on > macOS, or the equivalent elsewhere, and may fall back to `fsync` only where a > filesystem refuses it outright. > > It costs. Measured on an Apple silicon SSD, sequential writes went from > 949 MB/s to 201 MB/s. The first number was page cache speed and never > represented anything reaching a disk.
For content-addressed files (packs/) the final path is derived from the completed bytes, so step 4 is what makes the object exist. A crash before step 4 leaves an orphan in tmp/, which is deleted on next open. A crash between 4 and 5 may lose the rename on some filesystems, which is equivalent to a crash before 4.
On destinations where capabilities.durableFsync == false (some SMB servers, all File Provider volumes), step 3 and 5 are best-effort and the destination is marked as offering weak durability. Such a destination MUST NOT be used as Primary without an explicit user acknowledgement, and its objects MUST be re-verified by read-back after write.
locks/<lockID>.lock lockID := random UUIDv4 hex
Plaintext-authenticated CBOR (sealed with metadataKey):
{ "v":1, "kind": "shared"|"exclusive", "deviceID": bytes(16), "pid": u32,
"createdAt": "...", "refreshedAt": "...", "ttlSeconds": 300 }
refreshedAt every 60 s.now > refreshedAt + ttlSeconds. Stale locks MAY be removed by any client.exclusive requires no other lock to exist. shared requires no exclusive lock to exist.On conflict, remove your own lock and back off with jitter. This is safe without atomic compare-and-swap because the only exclusive operation (prune) is never time-critical.
| Operation | Lock |
| Backup | shared |
| Replication | shared |
| Restore | shared |
| Verification | shared |
| Prune: mark | shared |
| Prune: sweep | exclusive |
Concurrent writers from *different devices* are outside the format's guarantees. Each device MUST use its own repository (D-014).
gc/<runID>.mark sealed + signed, same envelope as snapshots
{ "v":1, "runID":..., "markedAt":"...", "candidates":[packID],
"reachableDigest": SHA-256 over the sorted reachable blobID set,
"retainedSnapshots":[snapshotID] }
Sweep MUST recompute the reachable set and MUST abort if reachableDigest differs. Sweep MUST NOT run until at least 7 days after markedAt.
holds/<hold-id>.hold sealed + signed
{ "v":1, "id":..., "reason":"anomalyDetected"|"entitlementDowngrade"|"userRequested"|"verificationFailure",
"note": text or null, "protectsSnapshotsBefore":"...", "createdAt":"...", "clearedAt":"..." or null }
A hold forbids pruning any snapshot created before protectsSnapshotsBefore, whatever the retention policy says.
Holds live in the repository, not on the machine that made them. A hold is what stops a former customer's history being deleted after their subscription ends. If it lived only in a local database, reinstalling the application would lose it and the next prune would delete exactly what the hold existed to defend.
A reader that does not recognise a reason MUST still honour the hold. Refusing to understand a hold is not a reason to ignore it and delete data.
replicas/<replicaID>.state sealed + signed
{ "v":1, "replicaID":..., "kind":"external"|"network"|"cloud"|"local",
"label": text, "lastSyncAt":"...", "holdsSnapshots":[snapshotID],
"holdsPacksDigest": SHA-256 over sorted packIDs held,
"durability":"writtenLocally"|"believedUploaded"|"verified",
"lastVerifiedAt":"..." }
Written to each destination, describing that destination, so a foreign machine can reconstruct the replication picture from any single copy (P2).
devices/<deviceID>.status sealed + signed
{ "v":1, "deviceID":..., "deviceName": text, "platform":"macOS"|"iOS"|"iPadOS",
"appVersion":"1.0.0", "lastSnapshotID":..., "lastBackupAt":"...",
"lastVerificationAt":"...", "repositoryHealth":"healthy"|"degraded"|"damaged",
"bytesPending": u64, "destinationCount": u32 }
Deliberately minimal. It MUST NOT contain filenames, paths, source lists, or any content-derived value.
| Protected from a party holding the destination | Leaked to that party |
| File contents | Total repository size |
| File and directory names | Number of snapshots and their approximate times (file mtimes) |
| Directory structure | Volume of change per backup |
| Individual file sizes | Pack count |
| Timestamps, xattrs, ACLs | Repository ID, format version, cipher and KDF parameters |
| Whether a specific known file is present (keyed blob IDs) | That Copy Shelter is in use |
| Device names, source paths |
Pack sizes are held near a uniform target specifically to limit inference from size distribution.
Not protected against: an adversary who can delete files. Authentication detects tampering and rollback; it cannot prevent destruction. See 04-THREAT-MODEL.md.
formatVersion is a single integer. Readers MUST refuse versions they do not implement.repository and replicating into it, with both readable throughout.
Tree objects (§7) are exempt from this requirement: they are packed binary, not CBOR, and they are never rewritten. A field a reader does not recognise cannot occur within one formatVersion, because §7 defines every presence bit that version writes; a field addition is a new formatVersion instead. Unknown *blob types* MUST be preserved by replication and treated as opaque.
An independent implementation needs, in order:
1. Read copyshelter.repo; check formatVersion == 1. 2. Read keys/*.key; unwrap masterKey with a passphrase or recovery key (§4). 3. Derive dedupKey, objectKeyRoot, signingSeed, metadataKey. 4. Read and verify refs/head; pick a snapshot. 5. Read the snapshot; verify signature; decrypt. 6. Load index/*.idx, or rebuild by scanning pack directories (§8). 7. Walk from rootTree, resolving blob IDs through the index to (packID, offset, length). 8. Read, verify and decrypt each blob; concatenate content blobs in order.
Everything required is inside the repository. No network, no server, no original machine.
csctl restore is the reference implementation of exactly this algorithm and is published alongside the specification.
This document is checked against a real repository on every build. tools/independent-reader.py implements the algorithm above from this document alone, in another language, sharing no code with the product, and scripts/check-format-doc.sh runs it against a repository the shipping app just wrote. csctl could never have caught the errors that check found, because csctl is built from the same source as the app and therefore only demonstrates that the code agrees with itself.
When it was first written, this document was wrong in seven ways that each stopped an independent reader dead: repositoryID documented as hex and written as base64; a check field in the key file that nothing writes; the key file's associated data undocumented; refs/head, snapshot payloads and index files all documented as CBOR and all written as JSON or packed binary; and snapshot identifiers hyphenated in one place and not the other. Every one of them is now corrected above, and the check exists so that "if we disappear, your backups still open" is something this repository verifies rather than something it asserts.