Persistence

This page explains how the server stores data, for contributors changing storage code. Operators should start with Storage and backups.

originchats-osl use <directory> changes into the database directory before loading anything. dbGetPath(name) joins a name onto the current directory, so every storage path is relative to the active database, not the source checkout.

The server keeps small, hot resources (users, roles, channels, webhooks, access settings) in memory and writes them through as JSON documents. Messages go in numbered JSONL files called shards, so adding a message never rewrites a whole channel.

Database layout

Data	Storage	Code
Configuration	config.json (overrides only)	src/config/
Users	users.json, keyed by internal user ID	src/db/users/
Roles	roles.json, keyed by role name	src/db/roles/
Channels	channels.json, an ordered array	src/db/channels/
Channel messages	channels/<channel>/0000.jsonl, 0001.jsonl, and so on, each with an optional .pending edit journal	src/db/messages/, src/db/shards/
Message indexes	.message-index/ in each channel and thread directory	src/db/message_index/
Threads	threads/<id>.json details, threads/<id>/*.jsonl messages, and threads_index.json	src/db/threads/
Unread state	users/<id>/unreads.json	src/db/unreads/
Mention history	users/<id>/mentions/0.jsonl, 1.jsonl, and so on, plus mentions-version.json	src/db/mentions/
Polls	polls/<id>.json	src/db/polls/
Reports	reports.json	src/db/reports/
Webhooks	webhooks.json	src/db/webhooks/
Audit log	modlog.jsonl	src/db/modlog/
Access settings	access.json	src/db/access/
Timeouts	rate_limits.json	src/db/rate_limits/
Welcome state	welcome.json	src/db/welcome/
Attachment details	attachments/<id>.json, attachment_ids.txt, and attachment_stats.json	src/db/attachments/
Attachment files	attachments/<id>, attachments.local_dir, or an S3 bucket	src/attachments/storage.osl
Emoji and stickers	serverEmojis.json and serverEmojis/, serverStickers.json and serverStickers/	src/db/assets/
Plugins	plugins/<name>/	src/db/plugins/
Open transaction	write-transaction.json and *.undo, only while a transaction runs	src/db/transactions.osl

Treat the directory as one unit. A partial copy can keep a message but lose its thread index, poll, unread state, or attachment details.

Atomic document writes

writeFileAtomic(path, content) writes path.tmp, renames it over the destination, and removes the temporary file on failure. Resource storage functions use it for mutable JSON documents and rewritten shards.

The rename provides a clear old-or-new result only when the temporary file and destination are on the same filesystem. Keep them together. If a deployment replaces this helper, it must also consider file sync and directory sync guarantees required by its operating system.

In-memory mutation and rollback

For users, roles, and channels, reads normally come from memory. A safe mutation keeps the previous state until save succeeds:

lock when shared mutation needs it
copy previous value
apply next value in memory
write durable representation
on failure, restore previous value
unlock
only then return success or broadcast

Never let a failed save leave memory newer than disk. On restart, that apparent success disappears.

Channel list replacement stores the previous array and restores it on save failure. Channel rename moves the message directory and updates the channel list, persisted mention locations, thread index, and thread parent metadata. If the file transaction fails, it attempts to move the directory back.

File transactions

dbCommit(changes) in src/db/transactions.osl writes several files as one unit. Each change either replaces a file or appends to it.

1. It takes the storage lock and first recovers any earlier interrupted transaction.
2. A single replacement skips the journal and uses writeFileAtomic.
3. Otherwise it records each file’s size, and copies each file it will replace to <file>.undo.
4. It writes that list to write-transaction.json, then applies the changes.
5. When every change succeeds, it deletes the journal and the .undo copies.
6. When one fails, it calls dbRecover, which truncates appended files to their old size, restores replaced files from .undo, and deletes files that didn’t exist before. If that fails too, it throws Storage rollback failed; write-transaction.json must be recovered before accepting writes.

Buffered shard changes follow the same result. shardsWrite commits its buffers only after dbCommit succeeds, so a rejected transaction leaves both the shards and their .pending journals unchanged.

Startup calls dbRecover before loading anything. If recovery fails, the server exits.

Message bodies and their recipient mention entries share a transaction. Unread counters and notification delivery happen afterward. Directory moves and other resource mutations still have their own rollback paths.

Message shards

The shared helpers in src/db/shards/ enumerate numeric .jsonl filenames, count entries, and append batches. A shard holds up to 1,000 newline-separated entries. Filenames sort numerically, so 10.jsonl follows 9.jsonl. Message files retain at least four digits, starting at 0000.jsonl; mention files start at 0.jsonl.

messages_batch retrieves up to 100 IDs from one channel or thread. Results follow request order, duplicate IDs appear once, and missing or severed messages are omitted. Batch reads resolve cached bodies first, group the remaining IDs by known shard location, and scan each required shard once. Uncached IDs use a persistent hash index. Missing IDs do not require a history scan.

The server stores immutable serialized message bodies in the cache and parses a fresh object for each result. An edit cannot mutate cached state before persistence succeeds. The existing message_get command uses the same batch storage path.

Shard reads and writes go through an in-memory buffer bounded to eight shards and 256 KiB per shard. Small rewrites append a line-range patch to a .pending journal next to the shard instead of rewriting the file, so bursts of edits cost a small append each. Every committed read replays the journal, so buffered edits are visible immediately after shardsWrite succeeds. Backups must copy .pending files alongside their shards; a shard without its journal loses staged edits.

After 64 journal patches, when a shard exceeds 256 KiB, or when a single change replaces more than 256 KiB, the buffer checkpoints: it rewrites the shard file from memory and clears the journal. Shards larger than 256 KiB bypass the buffer entirely and keep the previous rewrite-on-edit behavior, as do any files that are not numbered message shards. An evicted buffer reloads from disk plus its journal, and messageLruClear drops buffered shards so recovery paths can re-read committed state.

Message creation must persist before message_new is returned. Unread recording follows a successful append.

Edit and delete locate the shard containing the ID, compute a line-range patch against its buffered content, and commit that patch through the journal above. Malformed lines are preserved during patching rather than silently discarded. Bulk user-message deletion applies its per-file replacements through the same path, so it also sees staged but uncheckpointed edits.

Persistent message indexes

Each channel and thread stores a rebuildable .message-index/ directory beside its message shards. It contains JSON index pages and JSONL pin bodies; it does not use SQL. Shard paths inside the cache are relative, so a channel directory rename also moves its index without rewriting historical paths.

The ID index partitions SHA-256 hashes into a radix tree. Each leaf holds at most 512 distinct IDs and their shard positions. Branch directories have at most 16 child directories. A shard sidecar stores its message IDs and physical line positions. Blank lines and malformed JSON do not contribute to pagination counts; valid JSON objects do, including objects without an ID.

Four levels of count pages summarize message and pin counts, with at most 256 children per page. These counts locate a page by rank without walking the preceding shards or allocating the skipped messages. Shard numbers are mapped to separate slots for the current and legacy nested thread layouts. The supported numeric shard range is below 2,147,483,648, far above the roughly 100,000 full shards needed for 100 million messages.

messages_around locates its target through the ID index, calculates its rank from the count tree, and streams only the shards containing the requested window. messages_get uses the same window reader. A numeric start skips that many newest records; a string start is an exclusive message-ID cursor for older history. Both return messages in chronological order. The cost of an indexed page depends on the index depth and selected shards, not its distance from the newest message.

Pinned message bodies are stored in separate files per shard. Pin queries follow only count-tree branches with pins and read those files in reverse history order. Edits, pin changes, and deletions refresh the cached bodies before the next indexed read. Public serialization and permission checks still run for each request. The current pin command returns every pin, so its response memory necessarily grows with the number and size of pinned messages.

The first indexed read builds the cache. It streams each message shard, then partitions a temporary ID stream into bounded buckets. This avoids rewriting random ID buckets for every record during the initial build. Construction keeps one shard’s metadata and bounded partition buffers in memory, plus the directory listing of shard paths. Temporary partition files require additional disk space while building. A restart can reuse completed shard sidecars, but an unfinished ID partition pass restarts. The first build and repairs run under the shared messages lock and can delay other message operations on a large existing store.

Before a source transaction changes a shard, the server atomically writes a dirty marker if that scope has an index. Repeated writes coalesce to one marker per shard. The next indexed read rebuilds affected locations, pin bodies, and counts, then removes the marker. Replacements and recomputed counts make repair repeatable after an interrupted update. Failed source transactions can leave dirty markers; repair reads the rolled-back source. Bulk user-message deletion uses the shared file transaction layer and the same invalidation path.

The source JSONL remains authoritative. Stop the server and remove the affected .message-index/ directory after manually editing, importing, or replacing shard files outside server APIs. Do the same to rebuild a damaged cache. Restoring a backup must restore matching source and index files, or omit the derived index and let the server rebuild it.

Message memory limits

Cache	Maximum entries	Additional bound
Recent channel windows	8	At most 100 messages per window; windows above 262,144 serialized characters are not retained
Shared message bodies	128	Bodies above 32,768 serialized characters bypass the LRU
Message append directories	128	One path per active channel or thread
Inbox accounts	64	Latest 4,096 references per account
Append shard state	512	One current shard per storage directory
Shard line counts	8,192	One count per file; writes invalidate or update it
Buffered shards	8	One content snapshot per file, at most 262,144 serialized characters each; journaled patches cap at 64 before a checkpoint rewrite

These caches use LRU eviction and a one-hour TTL. The server no longer preloads a window for every channel at startup, and historical ID maps are not retained in RAM. Serialized character limits do not include object overhead and are not process-wide byte limits. Request bodies, selected message windows, pin responses, and other server state also use memory.

Recent windows load on demand. Batch and single-message reads check them before the shared body LRU and disk index. Edits replace cached bodies, and deletions refill affected windows. Readers parse fresh objects so callers cannot mutate a cached message before persistence succeeds. messageLruClear clears shared derived caches without discarding the bounded recent-window cache or persistent disk indexes.

pings_get caches account-specific references from mention files. It selects a page before batching message body reads. Pages beyond the cached prefix read more mention entries but retain only the latest 4,096 references. New mentions update active inbox caches after persistence succeeds. Permissions are checked again when selecting the page.

Message index reads, batch reads, inbox reads, mutations, and cache updates share the messages lock. Channel mutations acquire channels before messages; shard writes then acquire shards before the buffer’s shard-buffers lock and the transaction’s storage lock. Pin mutations hold the message lock across lookup and rewrite.

Persistent mention history

Each line records one message location for the account that received the ping:

["general","message-id"]
["support","thread-message-id","thread-id"]

The server resolves direct mentions, role mentions, and replies when saving the message. It records joined recipients who can view the channel and thread, excluding the author. Removing a recipient’s role later leaves those entries intact. Gaining a role does not add old messages to that account’s history.

An edit retains existing recipients and adds accounts reached by newly added mentions. Removing a mention does not erase its history. Deleted message references also remain on disk, but pings_get omits missing bodies. Current channel and thread permissions still control whether the recipient can read a historical message.

Entries append in notification order. Inbox reads visit them newest first. Pagination counts stored references, including entries whose bodies are now missing or inaccessible. A response can therefore contain fewer messages than its requested limit.

Stored message bodies carry an internal mention_users list to avoid duplicate entries during edits. Public message serialization removes it.

Link preview work

Link previews use four workers and a queue capped at 64 jobs. Plain messages do not enqueue work. When the queue is full, the server saves the message and skips its automatic preview. Preview downloads have a 1 MiB response limit and an eight-second timeout.

Threads and related records

Thread creation writes message storage and metadata, then adds the thread to threads_index.json. If index persistence fails, it destroys the newly created thread data.

Thread metadata carries message_count and the last-message pointer (last_message_id, last_message). Appends update the pointer from the saved message; deletions re-read the shard tail only when the removed ID matches the stored pointer. Threads without the pointer derive it from the last shard line at read time, so listings read at most one shard tail per legacy thread.

Thread deletion removes the index entry before destroying current and legacy metadata and message paths. Because a failure after the index update can leave orphaned files, backups and repair tooling should treat the index and directories as one unit.

Poll creation writes a poll record and a backing message. The message creation operation removes the new poll if later message creation fails. The file transaction helper covers operations that explicitly use it; it does not automatically include the caller’s other writes.

Attachment metadata and objects

Attachment creation writes bytes, then metadata, aggregate stats, and an append-only ID index under a lock. If a later metadata step fails, it removes the object and rolls back stats where possible.

Deleting metadata and object bytes is intentionally tolerant of a missing object. Expired records are skipped by reads and list pages. If attachment bytes live in S3 or an external local directory, database backups alone are incomplete.

Public serialisation

Never return stored records directly. Public helpers convert internal IDs to protocol usernames and remove private fields.

For messages, publicMessageFrom:

• resolves the author ID
• derives mention and reply data
• caps reply previews at 40 characters
• converts reaction user IDs
• serialises slash interactions without internal connection data
• hydrates poll state for the requesting user
• sets type: "message"

For users, userToPublic omits password hashes and internal-only fields, resolves display color, and adds local-account avatar fields. Apply equivalent boundaries to every new resource.

Parsing stored data

Loaders should validate disk data into a safe shape. A malformed optional record can be skipped or replaced with a default when the resource permits it. Missing required database files should stop startup rather than create a half-empty server over the wrong path.

Do not use unchecked stored strings as paths. Attachment, unread, channel, and thread helpers validate IDs or pass them through path-base checks before file access.

Migration

Migration code lives in src/db/migrations/ (attachments, audit log, threads, and unreads) and src/db/mentions/migration.osl. Updating lists when each runs and what it changes.

• main.osl calls mentionsMigrate and attachmentMigrate at startup, after dbSeedMemory. Either failing stops the server.
• unreadsLoad calls unreadMigrate during dbSeedMemory. Its result is ignored, so a failure leaves unreads.json in place to retry next start.
• modlogMigrate runs inside the audit log’s load and save functions, so it runs on first use.
• threadMigrateLegacy runs when a thread has no non-empty shard in the current layout.
• attachmentMigrate is also called by the attachment storage functions, but only does work once per process.

When mentions-version.json is absent, mention migration scans existing channel and thread messages once. It stages recipient references in mentions-migration/, sorts each account’s entries by message timestamp, removes duplicates, and writes numbered mention shards. It preserves accounts that already have mention shards so an interrupted migration can resume. After all accounts succeed, it writes version 1 and removes staging files.

This backfill uses current account names, roles, and permissions. The old message store cannot reconstruct role membership at the time each message was sent. Once migrated, normal inbox reads use the persisted history and never rerun this backfill.

When you write a migration:

• Make it safe to run twice.
• Keep the old data until the new form is complete.
• Fail before the server starts changing live state.
• Test it on a copy of a database in the oldest layout you support.

Backup and restore

Operator steps are in Storage and backups. For storage code, the rules that matter are:

• .pending journals are part of the data. A shard copied without its journal loses staged edits.
• .message-index/ is derived. A restore must bring back matching source and index files, or leave the index out so it’s rebuilt.
• Attachment files may live outside the directory, in attachments.local_dir or S3.

Legacy formats

These are compatibility inputs for existing databases, including data from the legacy Python originchats server. New storage code and import tools should write the current layout.

Legacy format	Current layout and behavior
attachments.json	Per-ID metadata plus ID and statistics indexes. Startup migration creates these and renames the old file to attachments.json.migrated after success. It also deletes .attachments-sidecar-v1 and .attachments-sidecar-v2.
unreads.json	users/<id>/unreads.json. Each stored message ID becomes last_viewed, the timestamp of that message, with ping_count: 0. The old file is renamed to unreads.json.migrated.
plugins/welcome/data.json	welcome.json. The old file is read when welcome.json doesn’t exist.
plugins.welcome config block	Top-level welcome. Read only when config.json has no welcome object.
modlog.json array	modlog.jsonl. Migration writes JSONL and removes the old file. If JSONL already exists, it takes precedence.
threads/{id}/thread.json	threads/{id}.json. The old metadata location remains a read fallback.
threads/{id}/messages/*.jsonl	threads/{id}/*.jsonl. Both shard locations remain readable.
threads/{id}.jsonl or threadMessages/{id}.jsonl, including .bak variants	When no populated shard layout exists, the compatibility reader copies the first available old file into threads/{id}/0000.jsonl.
No persisted mention history	Startup backfills mention references from existing messages using current permissions and roles. It cannot reconstruct historical role membership.

Keep a complete backup before migration. These readers do not imply a complete database conversion guarantee. In particular, the audit migration removes an invalid old modlog.json instead of preserving it, and thread fallback files can remain after copying. Review the migrated data before deleting the legacy backup. Do not run both servers against the same database.