Introduction
Corium is a database system in the style of Datomic. It is immutable, time-aware, and fact-oriented. Queries run in the application process against an immutable database value. A single transactor process owns all writes to a database.
This manual is for the person who runs Corium. It covers configuration, initialization, keys, authorization, schema, backup, restore, availability, and recovery. It does not teach Datalog query authoring. For the query language, read the query engine design document.
How to read this manual
The manual has six parts.
- Theory of operation explains datoms, the log, the indexes, and the process roles. Read this part first. Every operational rule in the manual comes from one of these ideas.
- Running Corium covers installation, the transactor, storage backends, the database catalog, schema, and index publication.
- Client surfaces covers the console, the dashboard, the SQL shell, the PostgreSQL wire server, and the peer server.
- Security covers authentication, authorization, encryption at rest, and attribute protection.
- Availability and data care covers high availability, backup, restore, forks, and garbage collection.
- Operations collects the metrics and the runbooks.
The Reference part holds the command list, the environment variables, the default values, and the glossary.
Conventions
Commands appear as the corium binary. A source build runs the same command
as cargo run -p corium-cli -- <command>. The manual writes corium for
both.
Angle brackets mark a value that you supply, for example <database>.
This manual marks incomplete work in an aside. Two labels are used.
Not implemented. The feature is specified in a design document, but no code implements it. Do not plan a deployment around it.
Partly implemented. Some of the feature works. The aside states what works now and what does not.
Asides that carry neither label give background information.
Terminology
This manual uses one word for one idea.
| Word | Meaning |
|---|---|
| transactor | The process that owns writes for a database. |
| peer | A library, or a process, that holds a database value and queries it locally. |
| storage service | The blob store and the root store together. |
| database value | An immutable snapshot of a database at one basis. |
basis, t | The transaction number that a database value covers. |
| datom | One fact: entity, attribute, value, transaction, and assert or retract. |
The glossary holds the full list.
Getting started
This chapter builds Corium, starts a local system, and runs a query. The whole system is in memory. Nothing is written to disk, so there is nothing to clean up.
The steps take about ten minutes. Use four terminals.
Step 1 — Build the binary
Corium needs Rust 1.85 or newer. From the repository root, run:
cargo build -p corium-cli --release
The binary is target/release/corium. This manual writes corium for that
path.
To build the optional storage backends, add their features. The installation chapter lists every feature.
Step 2 — Start a transactor
In terminal 1, run:
corium transactor --store mem --data-dir ./corium-data --listen 127.0.0.1:4334
The transactor prints the databases it serves. --store mem keeps everything
in the process. The process loses the whole database when it stops.
--data-dir is required for every store. The mem store does not write to
it.
Step 3 — Write a schema file
Create schema.toml in terminal 2:
schema-version = 1
[[entity]]
name = "person"
[entity.attributes]
name = { type = "string", unique = "identity", index = true }
age = "long"
This file declares two attributes, :person/name and :person/age. The
[[entity]] block is an authoring group. It does not create an entity type.
The schema chapter explains the full format.
Step 4 — Create the database
corium db create people --schema schema.toml
corium db list
corium db stats people
db create sends the schema to the transactor as an ordinary transaction.
db stats prints the basis, the datom count, and the transactor counters.
Step 5 — Write some data
The CLI has no transact command. Writes come from a client library, or from the PostgreSQL wire server. This step uses the wire server, because it needs no code.
In terminal 3, start the server with writes enabled:
corium postgres-server --listen 127.0.0.1:5432 --allow-writes
In terminal 4, insert two rows with psql:
psql 'host=127.0.0.1 port=5432 dbname=people' \
-c "INSERT INTO corium.person (name, age) VALUES ('Ada', 36), ('Grace', 45)"
Each statement is one transaction. The SQL chapter states which statements the write path accepts.
The Rust, Clojure, Python, and Java clients all transact directly against the transactor. Use them for real data loading. See
clients/pythonandclients/java.
Step 6 — Query the data
Open the Datalog console:
corium console people
Enter a query:
[:find ?name ?age
:where [?e :person/name ?name]
[?e :person/age ?age]]
The console also runs pull forms and time-view commands:
(pull [:person/name :person/age] 1000)
Type :basis to see the current transaction number. Type :quit to leave.
The console chapter lists every command.
Run the same question in SQL:
corium sql people -c "SELECT e, name, age FROM corium.person ORDER BY name"
Open the dashboard to watch the system live:
corium tui people
Step 7 — Change the schema
Add an attribute to schema.toml:
[entity.attributes]
name = { type = "string", unique = "identity", index = true }
age = "long"
email = { type = "string", doc = "primary contact" }
Ask for the plan:
corium schema update people --schema schema.toml
The command writes nothing. It prints the plan and the digest of that plan. Apply exactly the plan you read:
corium schema update people --schema schema.toml --apply --plan <digest>
The last line of the plan is the invocation to run. The schema chapter explains the execution classes and the acknowledgement codes.
Step 8 — Stop the system
Press Ctrl-C in each terminal. The mem store discards the database.
Next steps
- To keep the data, restart with the default
fsstore. Read storage backends. - To run a production process, read the transactor.
- To understand what the system does with your data, read how Corium works.
How Corium works
An operator who knows the data model can predict what Corium does under load, after a crash, and during recovery. This part of the manual explains the model. Every later chapter refers back to it.
Five ideas carry the whole system.
- A fact is a datom. Nothing is updated in place. A change asserts a new fact, and it retracts the old one. See datoms and the fact model.
- The log is the truth. A transaction is durable when its record is durable. Everything else is derived. See the transaction log.
- Indexes are a fold of the log. They are immutable, content-addressed blobs. They are an optimization, never a durability requirement. See indexes and storage.
- A database value is a snapshot. Time views name a basis. They do not copy facts. See time and database values.
- Writes and reads are separate roles. One transactor writes. Many peers read locally. See processes and roles.
The consequences an operator sees
The five ideas above produce the operational rules in this manual.
- Index publication can lag without risk to durability. Lag costs cold-start time and backup freshness.
- Garbage collection can strand garbage, but it cannot lose data, because it deletes only unreachable blobs after a retention window.
- Crash recovery equals startup. The transactor replays the log tail. There is no repair step.
- A peer never blocks on the transactor to get a database value.
- A deposed transactor cannot publish, because every root write is a compare-and-set on the record that holds the lease.
Where the design documents are
This manual states what an operator needs. The design documents state why.
They are in
docs/design, and the
decisions are recorded as ADRs in
docs/adr.
Datoms and the fact model
The datom
The unit of information is the datom. A datom is a five-part fact.
| Part | Name | Content |
|---|---|---|
e | entity | A 64-bit entity id. |
a | attribute | The entity id of an attribute. |
v | value | A typed value. |
tx | transaction | The entity id of the transaction that recorded the fact. |
added | assert or retract | true for an assertion, false for a retraction. |
A datom is never modified. A new value for a cardinality-one attribute is one transaction that retracts the old datom and asserts the new one. The old datom stays in the history indexes.
Entity ids and partitions
An entity id is a 64-bit number. The high 22 bits hold the partition. The low 42 bits hold a sequence number.
Entities in one partition sort together in the EAVT index. The partition is therefore the locality control. Three partitions are built in.
| Partition | Holds |
|---|---|
:db.part/db | Schema entities, such as attributes. |
:db.part/tx | Transaction entities. |
:db.part/user | Application entities, by default. |
Not implemented. User-defined partitions are described in the design documents. The engine has three partitions and no way to add a fourth. All application entities land in
:db.part/user.
Transaction numbers
A transaction id is an entity id in :db.part/tx. The basis, written t, is
the sequence part of that id. Conversion between t and tx is a bit
operation.
t increases by one for each committed transaction. Every value of t up to
the current basis names a real transaction.
Transaction time is data
Every commit asserts :db/txInstant on its own transaction entity. The commit
time is a datom, not log metadata. Three consequences matter to an operator.
- The commit time joins like any other fact. The clause
[?tx :db/txInstant ?inst]binds the time of a transaction. - The attribute is AVET-indexed, so a wall-clock time resolves to a basis with an index seek.
- Transaction entities are ordinary entities. Datom counts and entity counts include them.
The transactor stamps max(now, last + 1). This rule keeps commit times
monotone. A transaction can supply its own :db/txInstant, which is how an
import keeps original timestamps. The transactor rejects a supplied instant
that does not advance the clock.
Value types
The engine has nine value types.
| Schema type | Holds |
|---|---|
:db.type/boolean | true or false. |
:db.type/long | A signed 64-bit integer. |
:db.type/double | A double, totally ordered. |
:db.type/instant | Milliseconds since the Unix epoch, UTC. |
:db.type/uuid | A 128-bit UUID. |
:db.type/keyword | An interned keyword. |
:db.type/string | UTF-8 text. |
:db.type/bytes | A byte array. |
:db.type/ref | An entity reference. |
Keywords are interned per database. A keyword comparison is therefore an integer comparison.
One binary encoding serves the indexes, the log, and the wire. The encoding is sortable: the byte order of two encoded values equals the semantic order of the values. Index segments therefore compare without decoding.
Not implemented. Arbitrary-precision integers and decimals appear in the data-model design document, but the engine has no such value type. Store a big number as a string, or as a scaled long.
Not implemented.
:db.type/fulltextbehavior, tuple types,:db.type/uri, and:db.type/symbolare out of scope for version 1. See ADR-0009.
A stored value has one more shape than the nine above. Sealed holds a value
encrypted under a protection class key. No schema declares it. It appears when
the writing peer seals a value on a protected attribute, and it sorts after
every plaintext type. See
attribute protection.
Schema is data
An attribute is an entity in :db.part/db, described by datoms. The
schema chapter covers the attribute properties.
Because schema is data, a schema change is a transaction. corium db create
installs the first schema. corium schema update compares a file with the
installed schema, and applies the plan you reviewed.
A database also carries a schema generation. It is a counter, separate from the basis. It advances once for each committed transaction that contains a schema change. The basis says when a change happened. The generation says whether two database values use the same schema.
Excision
Not implemented. Excision, which removes historical facts, is out of scope for version 1. It is the one operation that breaks immutability. The design reserves space for a filter set in the database root, applied at read time.
The transaction log
What the log is
The log is a totally ordered sequence of transactions. Each record holds t,
the transaction instant, and the datoms of that transaction.
The log is the source of truth. The indexes are a deterministic fold of the log. Anything except the log can be rebuilt from the log.
The commit point
A transaction is durable when its log record is durable. The transactor acknowledges the caller after that point, and never before it.
Each record uses this frame:
payload-length | (1 << 63): u64 big-endian
payload: [u8; payload-length]
crc32c: u32 big-endian
The CRC32C covers the length word and the payload. Replay rejects a corrupted record even when the payload still decodes. A crash before the checksum is durable leaves a torn tail, and recovery truncates the whole incomplete frame.
The transaction pipeline
One logical thread of control serves one database. That thread is the write serialization point.
- Receive the transaction data.
- Resolve database functions. Built-in functions are native Rust. User
:db/fncode runs in a sandboxed Clojure interpreter. - Expand map forms and nested entities into list form.
- Resolve lookup references and tempids. A
:db.unique/identitycollision becomes an upsert. - Validate against the schema: types, cardinality, and uniqueness.
- Retract the prior value of each cardinality-one attribute.
- Assign the transaction entity id and
:db/txInstant. - Append to the log and flush. This is the durability point.
- Apply the datoms to the in-memory live index.
- Acknowledge the caller.
- Broadcast the transaction report to subscribed peers.
Steps 1 to 5 are pure functions of the database value and the input.
Group commit
Concurrent transactions to one database commit as a batch under one durability
boundary. Each transaction keeps its own t, its own report, and its own
acknowledgement.
A caller enqueues its work and then contends to lead a flush. The leader validates each transaction against a staging value that already includes its predecessors. Uniqueness, cardinality-one retraction, and compare-and-set therefore see the same state as a sequence of single transactions.
The batch is one atomic log object. A takeover keeps all of the batch or none of it. A rejected transaction fails alone, and the rest of the batch still commits.
Batch size is capped by count and by encoded bytes. Under no contention a batch holds one transaction, so light-load latency is unchanged.
Partly implemented. Group commit works. Three write-path items remain: optimistic-apply overlap across separate batches, more than one flush in flight, and an explicit bounded queue with fast-fail backpressure. See write-path-scaling.md.
Where the log is stored
The log layout follows the store.
| Store | Log layout |
|---|---|
mem | An in-process registry. The log dies with the process. |
fs | Versioned files under the data directory, named <db>.v<N>.log. |
postgres, turso | One row per transaction, keyed by database, lease version, and t. |
s3 | One create-only object per transaction, with the same key. |
On the native backends each commit is one create-only write. Success of that write is the durability point. An append is therefore O(1), and it does not read and rewrite a growing object.
The create-only condition is the fence of the log. A given lease version and
t are written at most once.
Partly implemented. Log sealing is future work on the native backends. The design concatenates the per-transaction tail into content-addressed chunks and reclaims the small objects. Without that step, replay cost and list cost grow with the tail since the last index publication. Frequent index publication keeps that tail short.
Lease-versioned files and takeover
Each transactor appends only to the file, or the key prefix, of its own lease version. A pre-HA log reads as version 0.
Readers merge the versions in order. A reader discards any record in an older
version whose t is at or after the first record of a later version. Those
records are exactly the never-acknowledged appends of a deposed writer.
CAUTION: Never edit or delete log files by hand. Old lease-version files are inert history that readers need to merge correctly.
Reading the log
Two surfaces read the log directly.
tx-range(from-t, to-t)streams(t, instant, datoms)to any peer. It does not touch the covering indexes.corium log --data-dir <dir> --db <name>prints committed transactions from a filesystem data directory. The transactor does not need to be running.
An encrypted log needs the key. Pass --storage-key to corium log. See
encryption at rest.
Indexes and storage
The four covering indexes
Corium keeps four covering indexes. Each one is a total sort of all datoms, or of a subset of them. Every index holds whole datoms, so an answer comes from one index without a second lookup.
| Index | Sort order | Contains | Serves |
|---|---|---|---|
| EAVT | e, a, v, tx | All current datoms | Entity access, pull |
| AEVT | a, e, v, tx | All current datoms | Column scans, clauses with a known attribute |
| AVET | a, v, e, tx | Datoms of :db/index and :db/unique attributes | Value lookups, ranges, uniqueness, lookup refs |
| VAET | v, a, e, tx | Reference-typed datoms | Reverse references, graph walks, component traversal |
Each index has a history variant. The history variant keeps retractions and
superseded assertions. Attributes marked :db/noHistory are not retained in
the history indexes.
The operational rule follows from the table. An attribute needs :db/index
or :db/unique before a query can seek it by value. Without one of them, a
value lookup is a bounded scan of AEVT.
Immutable segments
Each index is a persistent tree. A leaf segment holds a sorted run of encoded datoms, about 50 KB to 100 KB compressed. An inner segment holds separator keys and child hashes.
A segment is addressed by the BLAKE3 hash of its bytes. Segments are write-once. A new tree reuses every unchanged subtree by hash.
Content addressing has three operational effects.
- A segment is cacheable anywhere with no invalidation protocol.
- A publication uploads only the leaves that changed, not the whole database.
- A stale read is impossible. A reader either has the bytes or does not.
The database root
A database is named by its root record in the root store. The root holds the
database name and id, the basis t, and the index basis t. It also holds
the eight index roots, the log root, and the keyword table root. Last, it
holds the schema revision, the garbage collection epoch, the format version,
and the write lease.
The lease lives inside this record. Every lease acquisition, every renewal, and every index publication is a compare-and-set on the same bytes. One atomic operation therefore fences the writer, which is why no consensus protocol is needed.
The current database value equals the index trees at the index basis, merged with the log tail after that basis.
What a peer holds today
Partly implemented. The published format names content-defined leaf chunks under a manifest, so consecutive roots share every untouched chunk. The inner tree levels are future work. A reader therefore cannot seek into a published index, and it materializes the whole index in memory.
This limit has three consequences that an operator must plan for.
- A peer keeps every datom it has seen, including retractions. Its memory tracks total history, not the size of the live database.
- A cold time view costs a fold of the whole history, not of the view. First
touch of a distinct
as-of,since, orhistoryview is slow. - Facts are allocated once and shared by handle, so the four indexes cost keys and pointers rather than copies.
Size a peer against total history. The time chapter states the cost of each view.
Incremental publication
One rule decides where a sorted key stream is cut. The published format and the in-memory segment obey the same rule, so a segment leaf is exactly one published chunk.
The indexing job therefore folds the log tail into the segments of the last publication. It rebuilds only the leaves that the tail touched, and it carries every other leaf across by handle. Work per pass tracks the tail, not the size of the database.
A pass that cannot reuse the last publication rebuilds each segment from the database value. A rebuild is a full re-encode, but not a full re-upload, because content-defined boundaries reproduce the chunks of the previous process.
The pass can carry chunks over only while it can prove that the root that names them stays live. It pins the index state through the root compare-and-set. If the pin fails, nothing is installed, and the pass retries from a rebuild.
Storage traits
The storage service has two parts.
- The blob store holds immutable, content-addressed objects. Its operations are idempotent and need no ordering guarantee.
- The root store holds a small mutable map of named pointers. It is updated by compare-and-set, and it is the only strongly consistent state in the system.
A writer uploads every segment before it publishes a root that names the segment. Any root that a reader obtains is therefore fully dereferenceable.
Garbage collection
Old roots keep old segments alive until no reader needs them. Collection is epoch-based and never urgent.
- The transactor bumps the collection epoch and records the live roots.
- Mark: walk the live roots and collect every reachable hash.
- Sweep: delete unreachable segments older than the retention window.
Deletion is the only mutation, and it touches only unreachable data. A bug in collection can therefore strand garbage, but a generous window makes data loss a non-risk. See garbage collection.
Segment cache
A read-through, size-bounded cache wraps blob store reads. A peer server can
add an SSD tier with --segment-cache-dir and --segment-cache-capacity.
The cache never covers mutable roots, and it is not part of durability.
Why this is safe
- Segments are immutable, so no read sees a torn or stale segment.
- A root is published only after every referenced segment is durable.
- A root update is a compare-and-set fenced by the lease version, so a late publish by a deposed transactor fails cleanly.
Time and database values
The database value
A peer holds a database value. The value is immutable. A query runs against one value, so the answer cannot change while the query runs.
A time view wraps the same datoms with a different fold policy. A view never copies a fact.
The views
Given a connection whose latest known basis is t-now:
| View | Meaning |
|---|---|
db() | Current facts at t-now. |
as-of(t) | Facts as they stood at basis t. |
since(t) | Only facts added after t. |
history() | Every assertion and every retraction ever recorded. |
sync(t) | Completes when the basis of the peer reaches t. |
as-of and since also accept a wall-clock instant. Corium resolves the
instant to the last transaction committed at or before it. An instant older
than the database resolves to basis 0.
The as-of and history views disable the uniqueness shortcuts of the
planner. Uniqueness holds only in the current view.
Naming a view by wall clock
The t to instant correspondence is part of the database value, because every
commit asserts :db/txInstant. Resolution in both directions is O(log n).
A derived view keeps the whole correspondence. An instant therefore means the same thing whatever value it starts from.
Five surfaces accept an instant.
Db::as_of_instantandDb::since_instantin Rust.d/as-ofandd/sincein the Clojure API.as_of_instantandsince_instantinDbViewSpecon the wire.:as-of <timestamp>and:since <timestamp>in the console.\as-of <timestamp>and\since <timestamp>in the SQL shell.
The cost of a view today
Partly implemented. The design opens a view by descent through the segment tree. The implementation folds the view in memory from the recorded log. The costs below are what a peer pays now.
as-offolds the log up to its basis.historyfolds the whole log.sincefolds the whole log and then filters. It narrows before it projects, so the floor is applied while the index is built.
The result is cached in the database value, and it is shared by the clones of that value. A view that selects exactly the datoms of an already-folded view reuses that fold. A genuinely distinct view pays a full fold on first read.
The operational rule: a report that opens many distinct historical views is expensive on a peer. Reuse one view where possible.
Transaction reports
A peer receives a stream of transaction reports. Each report holds the basis before, the basis after, the datoms, and the tempid map for the submitting peer.
The peer applies the datoms to its live index and then offers the report to
registered listeners. One stream serves three needs: keeping the peer basis
current, sync, and application change feeds.
Reports arrive in t order with no gaps for a connected peer. After a
reconnect the peer declares its basis, and the transactor backfills the gap.
Reports are not durable per consumer. A consumer that needs exactly-once
delivery must track its own high-water t. tx-range replays the gap.
Reading the log directly
tx-range(from-t, to-t) streams transactions from the log tree. It is
available on any peer, and it does not touch the covering indexes. Use it for
audit, for replay, and for change-data capture.
Processes and roles
Corium separates three roles. Early versions can run them in one process, but no part of the core assumes that they share a process.
┌────────────────────┐
transact ───────►│ Transactor │──── append ────► ┌─────────────┐
│ (single writer) │──── segments ──► │ Storage │
│ tx pipeline │ │ service │
│ indexing job │ │ (blob store │
└─────────┬──────────┘ │ + roots) │
│ tx-report stream └──────┬──────┘
┌───────────────┼───────────────┐ │
▼ ▼ ▼ read segments
┌──────────┐ ┌──────────┐ ┌───────────┐ │
│ Peer │ │ Peer │ │Peer server│ ◄────────────┘
│ (in-proc │ │ │ │ (hosts db │
│ query) │ │ │ │ for thin │
└──────────┘ └──────────┘ │ clients) │
└─────┬─────┘
│ gRPC query/transact
┌─────┴─────┐
│Thin client│ (any language)
└───────────┘
Storage service
The storage service is passive. It has a blob store for immutable segments and a root store for named pointers.
Five backends exist: mem, fs, postgres, turso, and s3. See
storage backends.
Transactor
The transactor is the single writer for a database. It serializes transactions, validates them, appends to the log, acknowledges the caller, and streams reports to peers.
A background job publishes fresh index trees. Exactly one transactor holds the write lease for a database at a time.
One transactor process serves many databases. It can be active for some databases and standby for others.
Peer
A peer is a library in the application process. It keeps a live connection for transaction reports, reads segments from storage through a cache, and merges them into an immutable database value.
All query execution happens on the peer: Datalog, pull, the entity API, index scans, and time views. Getting a database value never blocks on the transactor.
Peer state is either immutable or disposable. A peer crash loses nothing.
Peer server and thin clients
The peer server is a peer hosted as a standalone process. It exposes query, pull, and transact over gRPC for languages without the peer library.
One peer server hosts one database. See peer server and thin clients.
Operator service
Not implemented. An operator peer service is specified. The design runs backup, restore, fork, garbage collection, index publication, and the encryption migrations as resumable, auditable jobs, behind an API and a web interface. The CLI runs all of those duties in-process today. See operator-service.md.
Transactor fleet
Not implemented. A fleet topology is specified. The design places many databases across many machines behind one client address, with the same lease and failover guarantees. The implemented topology is an active and standby pair. See transactor-fleet.md.
Which process needs what
| Process | Needs transactor address | Needs storage credentials | Needs storage key |
|---|---|---|---|
corium transactor | No | Yes | Yes, for encrypted databases |
corium peer-server | Yes | Only with --peer-bootstrap | Yes, for encrypted databases |
corium console, tui, sql | Yes | Only with --peer-bootstrap | Yes, for encrypted databases |
corium postgres-server | Yes | No | No |
| Thin client | No, it uses the peer server | No | No |
corium backup | Yes | Yes | Not supported yet |
corium restore, offline gc, log | No | Local data directory | Yes, for encrypted databases |
A thin client receives plaintext over TLS. It never holds a storage key.
Installation
Toolchain
Corium builds with a stable Rust toolchain, version 1.85 or newer. It uses edition 2024.
Build the CLI:
cargo build -p corium-cli --release
The binary is target/release/corium. Copy it to a directory on the path of
the operator, for example /usr/local/bin/corium.
Run the test suite before you promote a build:
cargo test --workspace
Cargo features
Optional backends and authentication methods are Cargo features of
corium-cli. A feature that is not compiled in makes its flags fail at
startup with a clear error.
| Feature | Default | Enables |
|---|---|---|
cljrs | Yes | The :db/fn Clojure transaction-function runtime. |
postgres | No | --store postgres. |
turso | No | --store turso. |
s3 | No | --store s3. |
oidc | No | OIDC bearer tokens with a JWKS file. |
oidc-discovery | No | OIDC, and JWKS fetch from the issuer. |
Build a production binary with the backends that you deploy:
cargo build -p corium-cli --release --features postgres,s3,oidc-discovery
A backend can also be loaded at run time instead of compiled in. Build the driver crate on its own, and give the transactor its library path:
cargo build -p corium-store-turso --release
Do not enable the static-link feature when you build a loadable library.
That feature is for a host that links the driver in. See
storage plugins.
Workspace build note
corium-cljrs and the MusicBrainz example are excluded from the default
workspace members. A --workspace build unifies the Clojure runtime into
no-gc mode and degrades their garbage-collection semantics.
Build and test those two crates on their own:
cargo test -p corium-cljrs
cargo test -p corium-mbrainz
What a deployment needs
A minimal deployment has one transactor process and one storage backend.
Add a peer server only when a client language has no peer library. Add a PostgreSQL wire server only when a SQL client must reach the data.
| Process | Default port |
|---|---|
corium transactor | 4334 |
corium peer-server | 4336 |
corium postgres-server | 5432 |
| Metrics endpoint | None. Set --metrics-listen. |
Directory layout of the fs store
The filesystem store keeps two directories under --data-dir.
| Path | Content |
|---|---|
<data-dir>/store | Blobs and root records. |
<data-dir>/logs | Versioned transaction log files. |
Back up the data directory as a unit, or use
corium backup. Do not edit files in either
directory by hand.
Process supervision
Run the transactor under a supervisor, such as systemd. Two rules apply.
- Give the transactor a stable
--ownervalue. A restarted member re-acquires its own unexpired lease at once. - Stop the transactor with
SIGINT, whichCtrl-Csends. The transactor releases its leases on the way out. A standby then takes over without waiting for the lease to expire.
Partly implemented. The transactor and the peer server listen for
SIGINTonly.SIGTERMkills the process, which leaves the lease held until it expires. A shutdown bySIGTERMis safe, because takeover is ordinary crash recovery, but failover then costs one full lease time-to-live.
For systemd, set the stop signal explicitly:
[Service]
ExecStart=/usr/local/bin/corium transactor --config /etc/corium/transactor.edn
KillSignal=SIGINT
Restart=on-failure
The transactor
The transactor owns writes, logs, indexing, leases, and scheduled garbage collection. One process serves every database in its catalog.
Start a transactor
corium transactor --data-dir /srv/corium --listen 0.0.0.0:4334
The process prints the databases that it serves and the databases for which it
stands by. It exits with an error when it cannot acquire a lease and --ha is
not set.
Every flag below has an equivalent key in the configuration file. A flag on the command line always wins.
Identity and network
| Flag | Default | Effect |
|---|---|---|
--data-dir <path> | None. Required. | Data directory for the filesystem store and for logs. |
--listen <addr> | 127.0.0.1:4334 | gRPC listen address. |
--owner <id> | transactor-$HOSTNAME | Stable identity in lease records. Set it. |
--advertise <url> | None | Client endpoint that peers use to find the lease holder. |
--metrics-listen <addr> | None | Prometheus endpoint at /metrics. |
Set --owner to a stable value per member, for example the host name. A
restarted member then re-acquires its own unexpired lease at once. A service
manager usually does not export HOSTNAME, so the default becomes
transactor-local on every member.
CAUTION: Keep the metrics listener on a private network. The endpoint has no bearer-token authentication.
Storage selection
--store picks the backend: mem, fs, postgres, turso, or s3. The
default is fs. Each backend has its own flags and its own Cargo feature.
--store-plugin <path> loads a storage driver at startup, and
--store <kind>:<json> then selects it. See
storage backends.
Lease and availability
| Flag | Default | Effect |
|---|---|---|
--ha | Off | Stand by when another transactor holds the lease, instead of failing at startup. |
--lease-ttl-ms <n> | 5000 | Failover detection bound. Renewals run at one third of this value. |
--lease-wait-ms <n> | 15000 | How long startup waits for a held lease before it gives up. Ignored with --ha, which waits without limit. |
--heartbeat-ms <n> | 10000 | Subscription heartbeat interval. |
A lower time-to-live gives faster takeover. It also costs more root-store traffic, and it tolerates shorter pauses on the active member. See high availability.
Index publication pacing
| Flag | Default | Effect |
|---|---|---|
--index-interval-ms <n> | 5000 | Base interval between publications. |
--index-backoff <n> | 4 | Minimum wait before the next publication, as a multiple of the duration of the last one. 0 disables it. |
--index-tail-threshold <n> | 0 | Defer publication while fewer than this many datoms are pending. 0 publishes any pending work. |
--index-tail-deadline-ms <n> | 60000 | Longest that a small tail defers publication. |
These four values can also be changed per database at runtime. See index publication.
Garbage collection
| Flag | Default | Effect |
|---|---|---|
--gc-interval <duration> | 1h | Interval of the scheduled sweep. off disables it. |
--gc-window <duration> | 72h | Retain unreachable blobs for at least this long. |
Collection is serialized with index publication. See garbage collection.
Database functions
| Flag | Default | Effect |
|---|---|---|
--db-fn-fuel <n> | 1000000 | Execution credits per :db/fn call. |
--db-fn-memory-bytes <n> | 16777216 | Managed memory per :db/fn call. |
User :db/fn code runs on the transactor in a restricted Clojure interpreter.
The interpreter has no input or output access. These two budgets bound a
runaway function. Both flags need the cljrs feature, which is on by default.
Authentication, authorization, and TLS
| Flag | Effect |
|---|---|
--serve-token <secret> | Require this exact bearer token. Strict mode. |
--require-auth | Require the shared development token. Reject anonymous callers. |
--serve-open | Accept every request as anonymous. |
--oidc-issuer <url> | Accept tokens signed by this issuer. Strict mode. |
--authz-db <name> | Authorize every request against this policy database. |
--tls-cert <pem>, --tls-key <pem> | Serve TLS. Both are required together. |
The default is permissive. The server recognizes the shared development token,
and it also admits anonymous callers. Any of --serve-token, --require-auth,
or --oidc-issuer switches the server to strict mode.
Read authentication and TLS before you expose a transactor outside a private network.
Encryption keys
--storage-key <uri> names a key that this process can resolve. The flag is
repeatable, because one transactor hosts databases under different keys.
The same keyring holds key-encryption keys and protection class keys. A transactor needs no class key: it commits sealed values without opening them.
The process resolves every named key at startup. A misconfigured process therefore fails at startup and names the key. See encryption at rest.
Logging
Tracing is human-readable by default. --log-format json writes structured
logs. RUST_LOG filters them.
RUST_LOG=corium_transactor=debug,corium_peer=info \
corium --log-format json transactor --data-dir /srv/corium
--log-format is a global flag. It comes before the subcommand.
A production example
corium transactor \
--config /etc/corium/transactor.edn \
--listen 0.0.0.0:4334 \
--advertise http://txor-a.internal:4334 \
--owner txor-a \
--ha \
--metrics-listen 127.0.0.1:9464 \
--serve-token "$CORIUM_SERVE_TOKEN" \
--authz-db corium_authz \
--tls-cert /etc/corium/tls/server.pem \
--tls-key /etc/corium/tls/server.key
The configuration file holds the storage selection and the read-only discovery credentials. The command line holds the identity of the member.
Storage backends
--store selects where the transactor keeps blobs, root records, and the
transaction log. Five backends are built in.
| Store | Cargo feature | Blobs and roots | Log | Shared between hosts |
|---|---|---|---|---|
mem | Built in | Process memory | Process memory | No |
fs (default) | Built in | <data-dir>/store | <data-dir>/logs | Only on a shared filesystem |
postgres | postgres | PostgreSQL tables | PostgreSQL rows | Yes |
turso | turso | Turso database file | Turso database file | No |
s3 | s3 | S3 objects | S3 objects | Yes |
--data-dir is required for every store. The mem store does not write to
it.
A backend is a driver that enters a process-wide registry under a backend
kind. mem and fs live in the engine. The other three are separate crates
that a Cargo feature links in. A driver can also be loaded at run time. Read
storage plugins below.
Read-only discovery credentials
A storage-aware peer, and an online backup, read storage directly. They ask
the transactor for connection details with the GetStorageInfo call.
GetStorageInfo never returns the primary write credential of a service
backend. For PostgreSQL and S3 you must provision a separate read-only
credential. Without it, storage-aware peer bootstrap and online backup fail
with an explicit error. They do not fall back to the write credential.
Local filesystem and Turso stores need no separate credential.
mem
corium transactor --store mem --data-dir ./corium-data
Everything lives in one process. The database is lost on exit. Use mem for
demonstrations and tests.
An online backup cannot open a process-local memory store. The backup command rejects it clearly.
fs
corium transactor --store fs --data-dir /srv/corium
Blobs are files under <data-dir>/store. Logs are versioned files under
<data-dir>/logs.
For a high-availability pair, both members must see the same directory over a shared filesystem. Never run two members against diverged copies of a data directory.
postgres
corium transactor --store postgres \
--postgres-url 'postgresql://corium@db.example/corium?sslmode=require' \
--postgres-read-only-url \
'postgresql://corium_reader@db.example/corium?sslmode=require' \
--data-dir /srv/corium
The backend creates corium_blobs and corium_roots in the current schema of
the connection. It stores log objects as fenced root records with log:
names. TLS uses the platform certificate store.
Provision the read-only role with SELECT on both tables. Pass its URL with
--postgres-read-only-url, or with CORIUM_POSTGRES_READ_ONLY_URL.
Readers use ordinary MVCC. They do not contend with the root compare-and-set writer.
turso
corium transactor --store turso --data-dir /srv/corium --turso-path /srv/corium/store.db
Turso is an embeddable SQLite database. --turso-path defaults to
<data-dir>/store.db.
Turso 0.7 needs its experimental multi-process write-ahead log when independent processes open one file. Corium enables that mode. Every process that touches the file must run the same mode.
s3
AWS_REGION=us-east-1 \
corium transactor --store s3 \
--s3-bucket corium-prod --s3-prefix corium/ \
--s3-region us-east-1 \
--s3-read-only-role-arn arn:aws:iam::123456789012:role/corium-reader \
--data-dir /srv/corium
Blobs go under {prefix}blobs/. Roots, including versioned log objects with
log: names, go under {prefix}roots/.
Root publication is fenced with S3 conditional writes, If-None-Match and
If-Match. The bucket, or the S3-compatible substitute, must support them.
Provision the bucket yourself. Corium does not create it, because bucket creation involves region and ownership choices that Corium must not make for you.
Primary credentials come from the standard AWS configuration chain:
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_PROFILE, and instance or
task roles. Region and endpoint come from that chain, or from --s3-region
and --s3-endpoint-url.
Read-only discovery credentials take one of two forms.
- Static keys. Pass
--s3-read-only-access-key-idand--s3-read-only-secret-access-key, and an optional session token. Restrict these keys to reads of the Corium prefix yourself. - AWS STS role. Pass
--s3-read-only-role-arn. Corium generates short-lived credentials on everyGetStorageInfocall, with a session policy limited tos3:GetObjectand prefix-scopeds3:ListBucket. The generated token cannot write, even when the role has broader permissions. The AWS identity of the transactor must be allowed to assume the role.
The two forms conflict. Use one of them.
A custom --s3-endpoint-url implies path-style addressing.
Storage plugins
A storage driver can be a dynamic library that the transactor loads at startup. An external author can therefore add a backend without a rebuild of the engine.
corium transactor \
--store-plugin /opt/corium/plugins/libacme_gcs.so \
--store 'acme-gcs:{"bucket":"corium-prod"}' \
--plugin-read-only-config '{"bucket":"corium-prod","role":"reader"}' \
--data-dir /srv/corium
| Flag | Environment | Effect |
|---|---|---|
--store-plugin <path> | CORIUM_STORE_PLUGINS | Load a plugin library. Repeatable. |
--store <kind>:<json> | Select a registered backend and pass it a JSON configuration object. | |
--plugin-store <kind>:<json> | The same form under an older name. It conflicts with --store. | |
--plugin-read-only-config <json> | CORIUM_PLUGIN_READ_ONLY_CONFIG | The read-only configuration returned by GetStorageInfo. |
CORIUM_STORE_PLUGINS takes a path-separator-delimited list of files and
directories. Corium searches a directory for platform dynamic libraries only.
It never adds the working directory.
The EDN configuration file carries the read-only
configuration under :plugin-read-only-config. It does not carry the plugin
paths or a kind:{json} store.
--plugin-read-only-config is what GetStorageInfo returns to a
storage-aware peer or an online backup. Corium never returns the primary
configuration of a plugin backend. Without the read-only configuration,
discovery against a plugin store fails.
Verifying a backend
corium store verify acme-gcs '{"bucket":"verification"}' \
--store-plugin /opt/corium/plugins/libacme_gcs.so
The command runs the blob and root conformance suite against a live backend. It creates uniquely named objects, exercises idempotence, listing, compare-and-set fencing, and deletion, and removes the objects afterward.
Use a disposable namespace. Do not run the verifier against a bucket or a database that holds live Corium data.
What a plugin costs
CAUTION: A plugin is native code that Corium runs in its own process. Install plugins only in directories that you control, and name them by explicit path.
- A plugin can receive storage credentials, because backend configuration crosses the boundary as JSON.
- A loaded library is never unloaded.
- Each library carries its own async runtime and its own process globals, so a process that loads several drivers pays for each.
- Encryption sits above the storage interface. A driver receives ciphertext and never receives key material.
Partly implemented. Only
corium transactorandcorium store verifyload plugins.corium peer-server,corium console, and the other client commands do not, so--peer-bootstrapagainst a plugin backend fails with “storage backend is not available”. Use a built-in backend where a storage-aware peer is needed.
The plugin contract is documented in storage-plugins.md.
Keeping secrets out of the process arguments
Static secret fields are also read from the environment:
CORIUM_S3_READ_ONLY_ACCESS_KEY_IDCORIUM_S3_READ_ONLY_SECRET_ACCESS_KEYCORIUM_S3_READ_ONLY_SESSION_TOKENCORIUM_POSTGRES_READ_ONLY_URLCORIUM_PLUGIN_READ_ONLY_CONFIG
Prefer the environment, or a protected configuration file, over a process argument.
Choosing a backend
| Situation | Backend |
|---|---|
| Demonstration or test | mem |
| Single host, simple operation | fs |
| High availability without a shared filesystem | postgres or s3 |
| Existing PostgreSQL operations practice | postgres |
| Large database, object storage economics | s3 |
| Single-file embedded deployment | turso |
| A service Corium does not support | A plugin |
A high-availability pair on fs needs a shared filesystem. postgres and
s3 remove that requirement, because they store the log natively.
The configuration file
corium transactor --config <path> reads storage selection and read-only
discovery credentials from one EDN file. A flag on the command line overrides
the value in the file.
The file holds one EDN map. Keys are plain keywords without a namespace. An unknown key is an error, and the message names the key.
Example
{:store :s3
:data-dir "/srv/corium"
:s3-bucket "corium-prod"
:s3-prefix "corium/"
:s3-region "us-east-1"
:s3-read-only-role-arn "arn:aws:iam::123456789012:role/corium-reader"
:s3-read-only-role-duration-seconds 900}
corium transactor --config /etc/corium/transactor.edn
Keys
| Key | Type | Equivalent flag |
|---|---|---|
:store | Keyword: :mem, :fs, :postgres, :turso, :s3 | --store |
:data-dir | String | --data-dir |
:turso-path | String | --turso-path |
:postgres-url | String | --postgres-url |
:postgres-read-only-url | String | --postgres-read-only-url |
:plugin-read-only-config | String holding a JSON object | --plugin-read-only-config |
:s3-bucket | String | --s3-bucket |
:s3-prefix | String | --s3-prefix |
:s3-region | String | --s3-region |
:s3-endpoint-url | String | --s3-endpoint-url |
:s3-read-only-access-key-id | String | --s3-read-only-access-key-id |
:s3-read-only-secret-access-key | String | --s3-read-only-secret-access-key |
:s3-read-only-session-token | String | --s3-read-only-session-token |
:s3-read-only-role-arn | String | --s3-read-only-role-arn |
:s3-read-only-role-session-name | String | --s3-read-only-role-session-name |
:s3-read-only-role-duration-seconds | Integer | --s3-read-only-role-duration-seconds |
:s3-read-only-role-external-id | String | --s3-read-only-role-external-id |
What the file does not hold
The file covers storage only. It does not hold the listen address, the owner identity, or the lease values. It does not hold the index pacing, the garbage collection schedule, the authentication flags, or the storage keys.
:store names a built-in backend only. A plugin backend needs
--store <kind>:<json> on the command line, and the file carries no plugin
paths. See storage plugins.
Put those on the command line, or in the unit file of the service manager.
Not implemented. There is no configuration file for the peer server, the PostgreSQL wire server, or the client commands. They take flags and environment variables only.
Protecting the file
The file can hold static secrets. Two rules apply.
- Set the file mode so that only the transactor user can read it. For example,
run
chmod 600 /etc/corium/transactor.edn. - Prefer the file, or the environment variables listed in environment variables, over process arguments. Process arguments are visible to every user on the host.
The database catalog
One transactor serves many databases. The corium db commands operate the
catalog. Every one of them talks to a running transactor.
Connection flags
Every client command shares the same connection flags.
| Flag | Default | Effect |
|---|---|---|
--transactor <url> | http://127.0.0.1:4334 | Transactor endpoint. A comma-separated list gives failover order. |
--token <secret> | Shared development token | Bearer token. --token "" connects anonymously. |
--ca <pem> | None | CA certificate to trust. Enables TLS. |
--tls-domain <name> | None | Domain expected on the server certificate. |
--peer-bootstrap | Off | Read the published snapshot from storage instead of replaying the log from basis 0. |
CORIUM_TOKEN sets the token for every command.
Administrative commands use the first endpoint in the list. Peer connections fail over across the whole list.
Create a database
corium db create people --schema schema.toml
The command prints {:db "people" :created true}.
A database name holds 1 to 128 characters. Only ASCII letters, digits, -,
and _ are allowed.
The schema file is EDN, or TOML when the path ends in .toml. Omit
--schema to create an empty database. See
schema management.
To encrypt every durable artifact of the database, add --storage-key:
corium db create people --schema schema.toml --storage-key file:/etc/corium/storage.key
Encryption is fixed at creation. See encryption at rest.
db createis idempotent, and it does not update an existing database. An existing name prints{:db "people" :created false}, and the schema file is ignored.
To change the schema of a database that already exists, use
corium schema update. See
schema management.
List databases
corium db list
The command prints the names that the transactor serves.
Inspect a database
corium db stats people
The command connects a peer, syncs it, and prints one EDN map:
{:basis-t 1240 :index-basis-t 1200 :datoms 91234 :entities 20114
:attributes 37 :index-lag 40 :tx-count 1240 :tx-failures 2
:tx-queue-depth 0 :gc-runs 17 :gc-swept-blobs 214}
| Field | Meaning |
|---|---|
:basis-t | Newest committed transaction that the peer has seen. |
:index-basis-t | Transaction covered by the published indexes. |
:datoms, :entities, :attributes | Counts in the current value. |
:index-lag | Transactions committed after the published index basis. |
:tx-count, :tx-failures | Transactor counters since process start. |
:tx-queue-depth | Commit queue depth now. |
:gc-runs, :gc-swept-blobs | Garbage collection counters since process start. |
db stats replays from basis 0 unless --peer-bootstrap is given. On a large
database that is slow. Add --peer-bootstrap when the client can reach the
storage backend.
Partly implemented.
db statsdoes not print the lease owner. TheMetricspanel ofcorium tuishows lease ownership and the advertised endpoint, from the sameStatuscall.
Delete a database
corium db delete people
The command prints {:db "people" :deleted true}.
CAUTION:
db deleteasks for no confirmation, and it cannot be undone. The command deletes the database root, the metadata root, the key manifest, and every log record at once. Blobs stay until garbage collection sweeps them. Take a backup first.
Fork a database
corium db fork creates a new database that duplicates an existing one at a
transaction basis. Use it for a writable sandbox against real data. See
forking a database.
Index publication
corium db request-index and corium db index-policy control when the
transactor publishes fresh index trees. See index publication.
Schema management
A schema declares attributes. An attribute has a name, a value type, a cardinality, and optional properties.
The schema is data. Corium keeps it in the same log as every other fact. The
schema of a database at basis t is therefore a question you can ask.
Two commands install attributes.
| Command | Purpose |
|---|---|
corium db create --schema <file> | Install the first schema with the database. |
corium schema update <db> --schema <file> | Compare a file with the installed schema, and apply the plan you reviewed. |
Both commands read the same file. The CLI reads TOML when the path ends in
.toml. Every other extension is read as EDN.
corium db create people --schema schema.toml
corium schema update people --schema schema.toml
The rest of this chapter describes the file format, the properties, and the update procedure.
TOML format
The TOML format is an authoring layer over the flat attribute model.
schema-version = 1
[[entity]]
name = "person"
[entity.attributes]
id = { type = "uuid", unique = "identity" }
name = { type = "string", index = true }
age = "long"
tags = { type = "keyword", many = true }
address = { type = "ref", component = true }
[[entity]]
name = "organization"
[entity.attributes]
name = "string"
employees = { type = "ref", cardinality = "many" }
The first block declares :person/id, :person/name, :person/age,
:person/tags, and :person/address. A bare string is shorthand for that
type with cardinality one.
An [[entity]] block is an authoring group only. It supplies the keyword
namespace. It does not create an entity type. It does not constrain which
attributes can appear together, and it does not constrain the target of a
reference attribute.
Each group name can appear in at most one [[entity]] block.
schema-version is the version of the file format. It is not a migration
number, and Corium does not compare it between runs.
Flat attributes
Top-level declarations express ungrouped attributes, or add attributes to a group without an entity block:
[[attribute]]
name = "created-at"
type = "instant"
index = true
[[attribute]]
group = "audit"
name = "created-by"
type = "ref"
These declare :created-at and :audit/created-by. Declaring one canonical
attribute through both syntaxes is an error.
Attribute options
Every detailed declaration requires type.
| Option | Values | Default |
|---|---|---|
type | boolean, long, double, instant, uuid, keyword, string, bytes, ref | Required |
many | Boolean cardinality shorthand | false |
cardinality | "one" or "many" | "one" |
unique | "identity" or "value" | Unset |
index | Boolean | false |
component | Boolean | false |
no-history | Boolean | false |
doc | Documentation string | Unset |
protection | A declared class, as "protect/<name>" | Unset |
Use only one of many and cardinality on a declaration. A unique attribute
receives index coverage whether or not index = true is present.
A [protect.<name>] section declares a protection class. Read
attribute protection for the class options and
for what protection costs.
Name rules
Group and attribute names are preserved exactly. They must be valid EDN keyword components, so that the resulting idents work in queries, transactions, and console input.
A name cannot start with a digit. It cannot contain whitespace, /, :, or
EDN delimiter and reader-macro punctuation.
A quoted TOML key carries an EDN-valid name that is not a valid bare TOML key. Quoting does not bypass the validation.
[entity.attributes]
"active?" = "boolean"
EDN format
The EDN format is the Datomic-style attribute map. The file holds one vector of maps, or a sequence of bare maps.
[{:db/ident :artist/gid
:db/valueType :db.type/uuid
:db/cardinality :db.cardinality/one
:db/unique :db.unique/identity}
{:db/ident :artist/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/index true
:db/doc "Credited name of the artist."}
{:db/ident :medium/tracks
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many
:db/isComponent true}]
| Key | Values | Default |
|---|---|---|
:db/ident | Keyword. Required. | None |
:db/valueType | :db.type/ plus boolean, long, double, instant, uuid, keyword, string, bytes, ref. Required. | None |
:db/cardinality | :db.cardinality/one or :db.cardinality/many | one |
:db/unique | :db.unique/identity or :db.unique/value | Unset |
:db/index | true | false |
:db/isComponent | true | false |
:db/noHistory | true | false |
:db/doc | String | Unset |
:db/protection | A declared class, for example :protect/pii | Unset |
What each property does
:db/unique. :db.unique/identity makes a tempid collision an upsert.
:db.unique/value makes a collision an error. Both give the attribute AVET
coverage, so a lookup reference [:person/email "a@b.c"] works.
:db/index. The attribute gets AVET coverage. A query can then seek it by
value or by range. Without it, a value lookup is a bounded scan of AEVT.
Index coverage costs write throughput and storage.
:db/isComponent. A reference attribute whose targets are retracted with
the parent by :db/retractEntity, and pulled recursively by default. Use it
for owned sub-entities, not for shared references.
:db/noHistory. The attribute is not retained in the history indexes. Use
it for high-churn values whose past does not matter, for example a counter.
The past of that attribute cannot be recovered afterward.
:db/doc. Free documentation text. It has no effect on queries.
:db/protection. Values on the attribute are sealed by the writing peer
under the key of the named class. Read
attribute protection.
Reference attributes are always covered by VAET. A reverse-reference query therefore needs no extra declaration.
Protection cannot be combined with :db/index, :db/unique, or
:db.type/ref. Ciphertext order is not value order, so a protected attribute
can never appear in a value-ordered index.
Enumerated values
An entity outside the :db.part/db partition can carry :db/ident as an
ordinary name. An ordinary transaction writes it, and a keyword in a reference
position resolves through it.
[{:db/ident :status/active}
{:db/ident :status/retired}]
A :db.type/ref attribute can then name :status/active as its value. Where
this shape is not needed, a plain :db.type/keyword value is simpler.
Updating an installed schema
corium schema update compares a schema file with the schema installed in a
database. It prints the changes, their cost, and their meaning.
corium schema update people --schema schema.toml
The command is read-only without --apply. It plans against one immutable
database value, so every count in the plan is measured at one basis and one
schema.
The plan
Each difference is one property-level change with an execution class.
| Class | Meaning | Examples |
|---|---|---|
additive | No existing fact is inspected or rewritten. | A new attribute. Cardinality one to many. |
validate-reindex | Existing facts stay valid, but a bounded scan, a constraint validation, or an index rebuild is needed. | Add index or unique. Change the uniqueness mode. Toggle isComponent. Retire an attribute. |
rewrite | Current facts must change first. | Collapse cardinality where an entity holds several values. |
destructive | Information or historical meaning is lost. | Change the value type of an attribute in place. |
Risk is reported beside the class, and the two are independent. An AVET
backfill is expensive and semantically harmless. A metadata-only
isComponent change is cheap and changes the meaning of every live reference.
A plan carries a digest. A later schema change, or a failed safety condition, invalidates the digest. Ordinary data writes do not.
database: people basis: 418 schema-generation: 3
desired: sha256:709f947a…
plan: sha256:ad5cb3c4…
ADDITIVE
+ :person/email string cardinality-one
+ :person/tags keyword cardinality-many
VALIDATE-REINDEX
~ :person/address component false -> true
live refs: 8109
[ack: component-enable] existing references acquire cascade retract and pull semantics
note: existing facts are not rewritten; future pull and retract-entity semantics change
UNMANAGED
:legacy/import-id use --prune to retire
3 change(s) planned. Nothing was written.
To apply: corium schema update people --schema <file> --apply --plan sha256:ad5cb3c4… --allow validate-reindex --ack component-enable
The last line is the exact invocation to run. Copy it, and add the path of the schema file.
Acknowledgement codes
A change whose meaning changes carries a stable code. Pass the code back
with --ack.
| Code | What you accept |
|---|---|
component-enable | Existing references acquire cascade retract and pull semantics. |
component-disable | Existing references lose those semantics. |
unique-mode-change | Upsert and conflict behavior changes for future writes. |
no-history-enable | History stops being recorded from this transaction onward. |
no-history-disable | History resumes. The interval already omitted cannot be reconstructed. |
retire-live-attribute | New assertions are refused while existing facts stay readable. |
protection-forward-only | Protection changes are forward-only and cannot re-seal existing facts. |
Allowing an execution class says which work can run. An acknowledgement says that you understood what the change means. The two are separate on purpose.
Partial files and retirement
A file manages the declarations that it contains. An installed attribute that
the file does not name is reported as unmanaged, and the command leaves it
alone. --prune turns every unmanaged attribute into a retirement request.
Retirement is not deletion. A retired attribute keeps its ident, its metadata, and its history. New assertions are refused. Retractions stay legal, which is what makes retirement a usable step when an application moves to a replacement attribute.
A retirement step prints the current datoms, the current entities, and the
recorded history datoms. It needs --ack retire-live-attribute only when the
attribute still holds live facts. Retracting those facts is separate work.
Idents are matched exactly. A removed ident and an added ident are two changes, never an inferred rename. An incorrect rename aliases two meanings permanently.
Engine attributes such as :db/txInstant are never managed by a file. A file
that declares one is rejected with an explicit error.
Applying a plan
Run the same command again with --apply, the digest that the plan printed,
and every allowance and acknowledgement that the plan asked for.
corium schema update people --schema schema.toml \
--apply --plan sha256:ad5cb3c4… \
--allow validate-reindex --ack component-enable
The transactor recomputes the plan under its writer queue. If the digest does not match, the transactor refuses the apply and changes nothing. Ordinary writes between the review and the apply are safe. Only a schema change or a failed condition invalidates the plan.
A successful apply prints the basis and the new schema generation:
Applied 3 change(s) to people at basis 419 (schema-generation 4).
+ :person/email
+ :person/tags
An apply that has already landed succeeds and prints No changes. Installing
a change is what invalidates the digest that described it, so the command
re-plans, finds nothing to do, and writes nothing. The command is therefore
safe in a pipeline.
Applying needs the alter-schema authority. That authority is separate from
transact, so an application writer cannot broaden its own vocabulary. Read
authorization.
Flags
| Flag | Effect |
|---|---|
--schema <path> | The desired schema file. Required. |
--prune | Retire the installed attributes that the file omits. Part of the digest. |
--json | Print the versioned machine contract instead of the human report. |
--detailed-exit-code | Exit 0 for no change and 2 for changes planned. |
--apply | Apply the plan. Requires --plan. |
--plan <digest> | The digest that the read-only plan printed. |
--allow <class> | Permit an execution class above additive. Repeatable. |
--ack <code> | Acknowledge a semantic change. Repeatable. |
A successful plan exits 0 whether or not it finds changes, so && chains keep
working. --detailed-exit-code changes that to 0 for no change and 2 for
changes planned.
A failure exits 1. With --json the failure carries a stable code:
parse-error, connect-error, plan-error, plan-mismatch,
allow-required, ack-required, blocked-change, or apply-failed.
Scripts must read --json. The human report is not a contract.
The audit trail
Every applied schema transaction records the requester, both digests, the
observed basis, the tool version, the execution classes, and the
acknowledgements. The transaction entity carries them under
:db.schemaUpdate/*.
These are ordinary queryable attributes, so the schema history of a database is a Datalog query. An ordinary transaction cannot write them, so a transaction cannot claim that it was a schema update.
What an update cannot do
Partly implemented. Every
rewritechange is reported as blocked, and--allow rewritedoes not enable it. Resolving cardinality conflicts, copying values to a replacement attribute, and sweeping current facts are jobs that do not exist yet. Do that work through ordinary transactions first, then plan again.
A blocked change stops the whole apply. Resolve it in the database, or remove it from the file, and then apply the rest.
A value-type change in place is destructive and can never run. The plan
prints a replacement-attribute recipe instead. Follow it by hand.
- Add a new attribute with the wanted type.
- Convert the current values, and assert them under the new attribute.
- Compare the counts, and record the values that no conversion accepted.
- Move the application reads and writes to the new ident.
- Retire the old attribute with
--prune.
An explicit rename is not specified yet. Excision, which erases historical
facts, is separate work with its own approval path. schema update never
hard-deletes anything.
Not implemented. A schema update points an attribute at an installed protection class. It cannot install a class. Decide the classes before you create the database.
Inspecting the installed schema
The console prints the schema:
:schema
:schema person/name
The Schema panel of corium tui shows the same table,
and it filters with /.
SQL exposes the same data as relations:
SELECT * FROM corium_sys.attributes;
Reserved idents
The engine installs its own attributes first. Attribute entity ids below 100
in the :db.part/db partition are reserved.
:db/txInstant is installed by the engine at the same id that Datomic uses.
A schema file that declares it is rejected.
An update allocates a new attribute id above the highest durable id. Database creation keeps its positional allocation, so a database built from one file is reproducible.
Index publication
The transactor republishes the covering indexes in the background. A cold peer then bootstraps from a snapshot instead of replaying the whole log.
Each index is published as content-defined leaf chunks under a small manifest. A publication uploads only the chunks that the changes landed in. Building the snapshot still costs processor time in proportion to the database, and pacing bounds that cost.
Indexing is never a durability requirement
The log append is the commit point. The transactor serves from its in-memory value whatever the index lag is.
Deferred publication has two costs, and no risk.
- Cold-peer bootstrap gets slower, because the log tail after the published basis is replayed.
- A backup is less fresh, because it reads through the published basis.
Pacing
| Flag | Default | Effect |
|---|---|---|
--index-interval-ms | 5000 | Base interval between publications. |
--index-backoff | 4 | Minimum wait before the next publication, as a multiple n of the duration of the last one. 0 disables it. |
--index-tail-threshold | 0 | Defer a due publication while fewer than this many datoms are pending. 0 publishes any pending work. |
--index-tail-deadline-ms | 60000 | Longest that a below-threshold tail defers publication. |
The backoff bounds indexing to at most 1/(1+n) of wall-clock time and of
storage bandwidth as publications get slower. With the default of 4, indexing
takes at most one fifth of the time.
The tail threshold makes small writes coalesce. Without it, a trickle of transactions rewrites the indexes on every interval.
Runtime overrides
All four values can be changed per database while the transactor runs. Omitted flags are unchanged. An override lasts until the process restarts.
corium db index-policy people --interval-ms 60000 --tail-threshold 1000000
Read the current policy back with no flags:
corium db index-policy people
The command prints one EDN map:
{:db "people" :interval-ms 60000 :backoff 4 :tail-threshold 1000000 :tail-deadline-ms 60000}
Publish now
corium db request-index people
This request bypasses pacing entirely. Use it after a bulk load, and before a backup, when the snapshot must be current.
Bulk loading
For a bulk load, follow this procedure.
- Raise the tail threshold, for example to one million datoms:
corium db index-policy <db> --tail-threshold 1000000. - Run the load. The backoff keeps the indexing duty cycle bounded as the database grows.
- Watch
:index-lagincorium db stats, or the metrics endpoint. - Run
corium db request-index <db>when the load is complete. - Restore the normal policy, or restart the transactor.
Without step 4, the final tail publishes within the tail deadline of the last transaction.
Partly implemented. On the native backends the per-transaction log objects are not yet sealed into chunks. Replay cost and list cost therefore grow with the tail since the last publication. A long deferral on
postgres,turso, ors3makes recovery and cold bootstrap slower than the same deferral onfs.
Watching the lag
Three surfaces report index lag.
corium db stats <db>prints:index-basis-tand:index-lag.- The
Metricspanel ofcorium tuiplots index lag. - The metrics endpoint exposes
corium_transactor_index_duration_seconds. See monitoring.
A lag that grows without limit means that publication cannot keep up. Lower
--index-backoff, or give the transactor faster storage.
Query console
corium console opens an interactive Datalog console. The console is a peer.
Every query runs locally in the console process.
corium console people --transactor http://127.0.0.1:4334
Input
The console accepts three kinds of input.
EDN Datalog queries. Enter the query on one line:
[:find ?name ?age :where [?e :person/name ?name] [?e :person/age ?age]]
Pull forms.
(pull [:person/name :person/age] 1000)
Console commands. Every command starts with a colon.
Partly implemented. A console query takes database inputs only. A query with
:inparameters other than$is rejected. Use a client library for a parameterized query.
The console is read-only. There is no transact command. See getting started for the write paths.
Commands
| Command | Effect |
|---|---|
:basis | Print the basis and the active view. |
:as-of <t> | Fix the view at transaction <t>. |
:as-of <timestamp> | Fix the view at a UTC timestamp. |
:since <t> | Show only facts added after <t>. |
:since <timestamp> | The same, named by timestamp. |
:history on | Show every assertion and retraction. |
:history off | Return to the current view. |
:current | Return to the current view. |
:schema | Print every attribute. |
:schema <attr> | Print one attribute, for example :schema person/name. |
:stats | Print the basis and the datom, entity, and attribute counts. |
:timing on | Report time and datoms scanned after each query. |
:timing off | Stop reporting them. |
:watch | Tail live transaction reports until Ctrl-C. |
:help | Print the command list. |
:quit, :exit | Leave the console. |
Timestamps
:as-of and :since accept a transaction number, or a UTC timestamp.
:as-of 10
:as-of 2026-07-25T09:30:00Z
:since 2026-07-25 09:30:00
A timestamp is YYYY-MM-DD, optionally with HH:MM, HH:MM:SS, or
HH:MM:SS.mmm. A timestamp selects the last transaction committed at or
before it. Resolution reads the :db/txInstant datom that every commit
asserts.
The SQL shell accepts the same two forms with \as-of and \since.
Cost of a time view
Partly implemented. A distinct time view costs a fold of the whole history on first read, not a fold of the view. A
:history onconsole session on a large database is slow and uses memory in proportion to total history. See time and database values.
Bootstrap
By default the console replays the log from basis 0. On a large database that is slow.
Add --peer-bootstrap when the console host can reach the storage backend:
corium console people --peer-bootstrap
The console then reads the published snapshot and subscribes from the index basis. The transactor supplies the storage connection details, using the read-only credential that you configured. See storage backends.
An encrypted database also needs --storage-key.
Watching transactions
:watch tails the transaction report stream. Each report prints t, the
commit instant, and the datom count. Press Ctrl-C to stop the tail and
return to the prompt.
The Transactions panel of corium tui shows the same stream with
a datom detail pane.
Terminal dashboard
corium tui opens a full-screen dashboard over one database.
corium tui people --transactor http://127.0.0.1:4334
| Flag | Default | Effect |
|---|---|---|
--refresh-ms <n> | 2000 | Metrics sample interval. The minimum is 250. |
The dashboard also accepts every connection flag.
The process owns the terminal, so it writes no tracing output.
Navigation
Press Tab to cycle the panels. Press 1 to 4 to jump to one panel, from
outside the query editor.
Quit with Ctrl-C anywhere, with q outside the query editor, or with
:quit.
Query panel
An editor for EDN Datalog queries, (pull …) forms, and every
console command.
Enter runs the form when its brackets balance. Otherwise Enter inserts a
newline. Alt-Enter always inserts a newline.
A relation result renders as a scrollable table with :find headers. Every
run reports wall-clock time, the datoms scanned, and the basis that it ran
against. ↑ and ↓ recall the query history.
Metrics panel
Data-store statistics, sampled from the transactor Status call on the
refresh interval:
- Basis, index basis, and index lag.
- Datom, entity, and attribute counts.
- Commit queue depth, transaction totals, and failure rate.
- Indexing and garbage collection counters.
- Lease ownership and the advertised endpoint.
The panel also draws sparklines for transaction frequency, status round-trip latency observed by the peer, and index lag. It reports peer-side query latency as last, average, and maximum.
This panel is the only surface that shows lease ownership without reading the root record directly.
Transactions panel
A live feed from the transaction report subscription of the peer. Each row
shows t, the commit time, and the datom count.
A detail pane shows the datoms of the selected transaction. Press f to
toggle follow-newest.
Schema panel
The attribute table: ident, value type, cardinality, uniqueness, and the index, component, and history flags.
Press / to filter the table.
When to use the dashboard
Use the dashboard for live observation during a load, a failover test, or an incident. Use the metrics endpoint for recorded monitoring. See monitoring.
SQL shell and PostgreSQL server
Corium executes SQL inside a peer, against immutable database values. SQL does not change the storage model into tables. The relations are a projection.
The SQL dialect is the DataFusion dialect. Wire compatibility with PostgreSQL
does not imply dialect compatibility or pg_catalog compatibility.
The relational projection
Attributes are grouped by keyword namespace. Given :artist/name,
:artist/country, and :artist/tags, SQL sees:
corium.artist(e BIGINT, name TEXT, country TEXT, tags LIST<TEXT>)
The rules of the projection are:
eis the Corium entity id, and the name is reserved.- A cardinality-one column is a nullable scalar.
- A cardinality-many column is a non-null list. An absent attribute is an empty list. Values are unique and ordered deterministically, but the order carries no meaning.
- One entity can occur in several namespace tables. These are projections, not entity types.
- An attribute without a namespace is grouped in
corium._global. - Names are preserved exactly. Use double quotes for a name such as
release-group.
Three system relations are available in every view.
| Relation | Content |
|---|---|
corium_sys.datoms | e, a, attr, typed value columns, tx, t, added. |
corium_sys.attributes | The schema. |
corium_sys.idents | Entity id to keyword ident. |
Partly implemented. A history session exposes
corium_sysrelations only. Wide history tables are reserved for a later validity-interval design.
The SQL shell
corium sql people
corium sql people -c "SELECT * FROM corium.artist LIMIT 10"
corium sql people -f report.sql
An interactive statement ends with a semicolon. Each statement captures a
fresh current database value, unless a time view is selected. Ctrl-C drops
the running query.
The shell is read-only.
| Command | Effect |
|---|---|
\as-of <t> | Fix later sessions at <t>, or at a UTC timestamp. |
\since <t> | Use a since view. Timestamps are accepted. |
\history on | Expose history events. |
\history off | Return to the current view. |
\current | Return to the current view. |
\basis | Print the basis and the view. |
\dt | List relations. |
\d <table> | Print the result columns of a relation. |
\timing on | Report execution time. |
\q | Quit. |
List functions come from DataFusion:
SELECT e, name FROM corium.artist WHERE array_has(tags, 'ambient');
The shell takes no key flag, so it prints <redacted> for a value on a
protected attribute.
The PostgreSQL wire server
corium postgres-server --listen 127.0.0.1:5432
One server exposes the whole database catalog of the transactor. A connection
picks its database with the standard startup database parameter. It can
switch at any time with USE <database>. SHOW DATABASES lists what is
available.
psql 'host=127.0.0.1 port=5432 dbname=people' \
-c "SELECT e, name FROM corium.person ORDER BY name LIMIT 10"
| Flag | Default | Effect |
|---|---|---|
--listen <addr> | 127.0.0.1:5432 | Listen address. |
--database <name> | All | Restrict the exposed set. Repeatable. |
--password <secret> | None | Require this cleartext password. Ignored once authentication is configured. |
--allow-writes | Off | Enable guarded DML. |
The server also takes the connection flags, the
serving flags, and --storage-key.
Databases are opened lazily and cached. One peer connection is shared by every client that uses that database.
The server supports the simple and the extended query sub-protocols, including
$1 bound inputs. Common scalar parameters accept text and binary encodings.
Results support both encodings.
Not implemented. Array inputs are not supported on the wire.
Writes through SQL
corium postgres-server is read-only by default. --allow-writes enables a
narrow DML subset.
corium postgres-server --listen 127.0.0.1:5432 --allow-writes
In autocommit each statement is one transaction. An expected-basis fence rejects a stale read-modify-write plan before it commits.
- Only existing
corium.<namespace>projections are writable.corium_sys, the time views, DDL, and schema changes are read-only. INSERTrequires an explicit column list. It supportsVALUESor a query source. Omitefor a tempid. An explicitemust not already occur in that projection. ANULLinput omits the attribute.UPDATEsupports one plain target table, predicates, expressions, andRETURNING. AssigningNULLclears a cardinality-one attribute. AssigningARRAY[...]replaces the whole cardinality-many set.DELETEsupports one plain target table, predicates, andRETURNING. It retracts every attribute in the target namespace, and it preserves attributes of other namespaces on the same entity.RETURNINGworks for all three. Delete rows come from the pre-commit snapshot. Insert and update rows come from the committed value.
Not implemented. Joined and multi-table mutations, conflict clauses and upserts, ordered or limited mutations, new keyword interning, and DDL are deferred.
Explicit transactions
An explicit BEGIN block pins the database value of its first statement. DML
is staged against a provisional value, so a later statement in the block reads
what the earlier ones wrote.
ROLLBACK discards the staged forms. COMMIT submits them as one atomic
Corium transaction. A concurrent basis change fails the commit with SQLSTATE
40001, which a client reads as a serialization failure and retries.
SET, RESET, and DISCARD are compatibility no-ops.
Object-relational mappers
The server answers the PgJDBC metadata probes for SQL keywords, current schema and catalog, and transaction isolation. Hibernate therefore selects its PostgreSQL dialect on its own.
The runnable
postgres-hibernate
example exercises Hibernate ORM 7.4 with PgJDBC 42.7. It inserts with a
generated id, reads, updates, runs an HQL query, and deletes. Every step uses
an ordinary Hibernate transaction.
Not implemented. Broader
pg_catalogintrospection, DDL-based schema management, savepoints,COPY, and sequences are absent. Declare the schema withcorium schema updaterather than with the schema tool of the mapper.
Security of the wire server
CAUTION: The PostgreSQL wire server does not terminate TLS. It rejects
--tls-certand--tls-keyrather than accept flags it cannot honor. Put a TLS-terminating proxy in front of it, or bind it to loopback.
Restrict the exposed set with --database when only some databases must be
reachable.
A SQL client is a Corium principal
Set any of --serve-token, --oidc-*, or --authz-db, and the server
authenticates each client for itself.
PostgreSQL has no bearer-token field, so the password field carries the
token of the caller. The startup user is informational.
corium postgres-server --listen 127.0.0.1:5432 \
--oidc-issuer https://issuer.example --oidc-audience corium \
--authz-db corium_authz
psql "host=127.0.0.1 port=5432 dbname=people user=alice password=$JWT"
CAUTION: The token crosses the wire in the clear. The server prints this warning at startup whenever authentication is configured.
Every statement is then authorized as that principal. SELECT needs query.
DML needs transact. SHOW DATABASES lists only what the principal can
inspect.
Reads are answered through the view of the principal and through its own
protection class keys. A column that the policy hides keeps its declared type,
reports NULL, and never takes a pushed-down predicate. A principal whose
view hides attributes cannot write. Read
authorization and
attribute protection.
--password still applies when no authentication flag is set. It is one
shared secret and it maps to no principal.
Partly implemented. A write still commits through the peer connection of the server, so the transactor additionally applies the bearer principal of that connection. Give that connection an identity that can transact every database the server exposes.
Peer server and thin clients
A peer server is a peer hosted as a standalone process. It exposes query, pull, transact, datom scans, and transaction ranges over gRPC.
Use it for a language that has no peer library. For a language with a peer library, embed the peer instead. An embedded peer queries in-process.
Start a peer server
corium peer-server --db people --listen 0.0.0.0:4336 \
--transactor http://127.0.0.1:4334
One process hosts one database. Run one process per database.
| Flag | Default | Effect |
|---|---|---|
--db <name> | None. Required. | Database to host. |
--listen <addr> | 127.0.0.1:4336 | gRPC listen address. |
--max-fuel <n> | 10000000 | Ceiling on datoms touched per query. |
--metrics-listen <addr> | None | Prometheus endpoint at /metrics. |
The peer server takes the same connection flags as the other client commands, and the same serving flags as the transactor.
Query fuel
Fuel bounds a runaway query. A client can request less fuel. fuel = 0
requests the server default. The server clamps every request to --max-fuel.
An exhausted budget returns INVALID_ARGUMENT.
Storage bootstrap
By default the peer server replays the log from basis 0 at startup. On a large database that is slow.
corium peer-server --db people --peer-bootstrap \
--storage-key file:/etc/corium/storage.key
--peer-bootstrap reads the published snapshot from storage and subscribes
from the index basis. The peer needs network reach to the storage backend, and
a build with the matching storage feature.
The transactor supplies the connection details through its GetStorageInfo
call, using the read-only credential that you configured. See
storage backends.
Segment cache
A peer server can keep a local SSD cache of segments.
| Flag | Default | Effect |
|---|---|---|
--segment-cache-dir <path> | None | Dedicated directory for the cache. |
--segment-cache-capacity <size> | None. Required with the directory. | Disk capacity, for example 256GiB. |
--segment-cache-memory <size> | 64MiB, or the capacity when smaller | Memory front tier. |
A size accepts the suffixes B, KiB, MiB, GiB, TiB, kB, MB, GB,
and TB.
The cache requires --peer-bootstrap. Without it, the process fails at
startup with a clear message.
Give the cache a dedicated directory. Corium manages the contents.
Failover
Pass every transactor endpoint, active first:
corium peer-server --db people \
--transactor http://txor-a:4334,http://txor-b:4334
The peer rotates the list on failure. A standby rejects subscriptions with a
standby status, which the peer treats as a reason to try the next endpoint.
See high availability.
The thin-client contract
The wire contract is documented in
thin-client-protocol.md.
The canonical schema is crates/corium-protocol/proto/corium.proto.
Six rules matter to an operator.
- Every transact and subscribe request sends a
protocol_version. This build speaks version 3 and accepts version 1 and later. An unsupported version isFAILED_PRECONDITION, never a silent downgrade. - Malformed input is
INVALID_ARGUMENT. An unknown database or entity isNOT_FOUND. Upstream loss isUNAVAILABLE. - Query results stream in chunks. A client concatenates them and stops at
last = true. Subscribe.from_basis_tis exclusive. The server backfills every later transaction without gaps, then continues live.- The subscription handshake advertises the heartbeat interval. A client treats silence for a few multiples of it as a dead upstream.
Transactgives read-your-writes on the serving peer before it responds.
A client is conformant when it reproduces the behavioral corpus in
tests/conformance.
Version 3 adds the sealed value tag. A client older than version 3 never
receives a sealed value. An unopened value reaches it as the tagged EDN
element #corium/redacted, which every EDN reader parses.
Serving many principals
A peer server serves every client from one process and one database value. Which facts a client sees is a policy question, not a process question.
- A view hides attributes from a principal. Read authorization.
- A key grant decides which protection classes a principal can open. Read attribute protection.
A principal whose view hides attributes cannot Transact, and it cannot
Subscribe. Transaction data is opaque bytes by the time authorization runs.
A subscription proxies the stream of the transactor. Neither call can honor a
filter, so the server refuses rather than serve unfiltered data.
Language clients
| Client | Location |
|---|---|
| Rust | corium-peer, corium-client |
| Python | clients/python |
| Java | clients/java |
| Clojure | corium-cljrs, the corium.api namespace |
The Python and Java clients each offer two peers behind one interface.
| Peer | Where it runs |
|---|---|
LocalPeer | Embeds a full peer in the process. It indexes and queries in process, and it talks to a transactor directly. |
RemotePeer | Connects to a peer server over gRPC. |
Both produce the same database values, and every time view works the same way on both. Only a remote peer can join across databases in one query.
An embedded peer can also read published segments straight from storage, and it can hold its own class keys. Use it where the keys must stay in the application process.
Authentication and TLS
Every network surface establishes a request-scoped principal. Authentication answers who is calling. Authorization answers what that caller can do.
The default is permissive
A server with no authentication flags does two things.
- It recognizes the shared development token, and it gives that caller the
identity
operator, vouched for by the providerstatic-token, with the roleadmin. - It also admits an anonymous caller.
This default makes a local database usable with no flags. It is not safe on a shared network.
CAUTION: Never expose a default-configured transactor or peer server outside a trusted network. The shared development token is a compiled-in constant, and anonymous callers are admitted.
Strict mode
Any of three flags switches a server to strict mode. In strict mode an absent or unrecognized credential is rejected.
| Flag | Effect |
|---|---|
--serve-token <secret> | Require this exact bearer token. It replaces the development token. |
--require-auth | Require the development token, or --serve-token. Reject anonymous callers. |
--oidc-issuer <url> | Accept tokens signed by this issuer, and the static token. |
--serve-open goes the other way. It disables authentication completely, and
every request arrives as anonymous. It conflicts with --serve-token,
--require-auth, --oidc-issuer, and --authz-db.
Read the secret from CORIUM_SERVE_TOKEN rather than from a process argument:
CORIUM_SERVE_TOKEN=$(cat /etc/corium/serve.token) \
corium transactor --data-dir /srv/corium --require-auth
Client tokens
A client sends its token with --token, or with CORIUM_TOKEN.
CORIUM_TOKEN=$(cat /etc/corium/serve.token) corium db list
--token "" connects anonymously. With no flag and no variable, the client
sends the shared development token.
OIDC
--oidc-issuer <url> accepts bearer tokens signed by an OIDC issuer. The
static token keeps working alongside it.
| Flag | Effect |
|---|---|
--oidc-issuer <url> | Issuer URL. |
--oidc-audience <aud> | Accepted audience. Repeatable. Set it. |
--oidc-jwks-file <path> | Read the JWKS from a file instead of fetching it. |
Two Cargo features apply. oidc verifies against a JWKS file. oidc-discovery
also fetches the JWKS from the issuer over HTTP. A binary without the feature
rejects the flags at startup and names the feature.
Set at least one audience. Without it, a token minted for another service of the same issuer is accepted.
TLS
A server terminates TLS when both certificate flags are present:
corium transactor --data-dir /srv/corium \
--tls-cert /etc/corium/tls/server.pem \
--tls-key /etc/corium/tls/server.key
A client enables TLS by naming a CA, a domain, or both:
corium db list --transactor https://txor-a:4334 \
--ca /etc/corium/tls/ca.pem --tls-domain txor-a.internal
Three surfaces have no TLS of their own.
- The metrics endpoint. Keep it on a private operations network.
- The PostgreSQL wire server. Put a proxy in front of it.
- The storage backends use their own transport security. PostgreSQL uses
sslmodein the URL. S3 uses HTTPS.
Recommended settings
| Deployment | Settings |
|---|---|
| Laptop, single user | No flags. |
| Shared development host | --require-auth, or --serve-token. |
| Production, machine clients | --serve-token, TLS, --authz-db. |
| Production, human identities | --oidc-issuer with --oidc-audience, TLS, --authz-db. |
Authentication alone permits every action. Add authorization to restrict what a principal can do.
Authorization
Servers authorize every request permit-all by default. --authz-db <name>
switches them to a relationship policy that is stored in an ordinary Corium
database.
The model is relationship-based, in the style of Google Zanzibar and OpenFGA. A decision is a bounded, cycle-safe walk over relationship tuples.
The policy database is an ordinary database. Backup, restore, fork, as-of,
and the log API all work on it.
Bootstrap
Bootstrap is two steps, in this order.
Step 1. Against a transactor started without --authz-db:
corium authz init --admin alice --provider oidc
corium authz grant 'group:eng#member' writer database:music
corium authz grant bob member group:eng
corium authz check bob transact --database music
authz init creates the database corium_authz, installs the reserved
schema, installs the default permissions, and grants the first administrator
owner on catalog:* and on database:*.
Step 2. Restart the surfaces with enforcement on:
corium transactor --data-dir /srv/corium --authz-db corium_authz
corium peer-server --db music --authz-db corium_authz
The default administrator
authz init defaults its administrator to operator, pinned to the provider
static-token. That is the identity a token client presents, so the CLI keeps
working after enforcement is on.
| Flag | Default | Effect |
|---|---|---|
--db <name> | corium_authz | Policy database name. |
--admin <id> | operator | Subject id of the first administrator. |
--provider <name> | static-token | Provider that must vouch for the administrator. any accepts every provider. |
--no-admin | Off | Install schema and permissions only. Grant nobody anything. |
Subjects, relations, and objects
A tuple says subject relation object.
corium authz grant alice writer database:music
corium authz revoke alice writer database:music
| Position | Forms |
|---|---|
| Subject | user:alice, group:eng, role:ops, or the userset group:eng#member. A bare name reads as user:<name>. |
| Relation | A name, for example owner, writer, viewer, member, parent. |
| Object | database:music, tenant:acme, catalog:*, database:*. |
Relation names are data, not built-in values. The permissions decide which relation satisfies which action.
Actions
Fifteen actions exist. Each belongs to one class.
| Class | Actions |
|---|---|
| Read | query, pull, datoms, tx-range, subscribe, inspect, list-databases |
| Write | transact |
| Admin | create-database, delete-database, fork-database, garbage-collect, manage-index, manage-keys, alter-schema |
alter-schema is deliberately separate from transact. An application writer
that can add facts must not be able to broaden the vocabulary that it writes
them under. The action is admin-class and database-scoped, so the default
permissions give it to a database owner only. Read
schema management.
The default permissions that authz init installs bind classes to relations.
| Object type | Class | Relations that satisfy it |
|---|---|---|
database | read | viewer, writer, owner |
database | write | writer, owner |
database | admin | owner |
catalog | read | viewer, owner |
catalog | write | owner |
catalog | admin | owner |
A permission entity can also name one action instead of a class, and *
matches any action or any object type.
Testing a decision
corium authz check bob transact --database music
The command runs the same evaluator that a server runs, and it prints the matched path. Use it before and after a change.
| Flag | Effect |
|---|---|
--database <name> | Target database. Omit it for catalog-wide actions. |
--provider <name> | Provider that vouched for the subject. Defaults to oidc. |
--role <name> | A role that the credentials of the principal assert. Repeatable. |
--claim <key>=<value> | A claim that the principal carries. Repeatable. |
Status
corium authz status
The command prints the compiled basis, :authz-t, and the entity counts.
Operating notes
Fail closed. A surface that cannot read or compile the policy denies every request. It does not refuse to start. It logs the remedy and recovers on its own when the database appears, so an ordering mistake is not fatal.
Changes propagate without a restart. Each server watches the policy database and recompiles off the request path. A grant takes effect in milliseconds.
Every decision is logged with its basis. The tracing target is
corium_authz::audit. Denials log at info. Grants log at debug.
--authz-fresh-writes makes write and admin actions re-read the policy
before they decide. The cost is one snapshot read per such request. Reads keep
using the pinned snapshot.
--authz-max-depth <n> bounds the relation hops of one check. The default
is 8.
--authz-break-glass-role <role> admits a role while the policy is
unreadable. It never overrides a deny.
Recovering from a lockout
A policy that denies everyone cannot be repaired through the policy.
- Stop the transactor.
- Start it again without
--authz-db. - Fix the tuples with
corium authz grant. - Stop it, and start it again with
--authz-db.
Break-glass does not help here, because the policy is readable. It denies.
Interaction with authentication
--authz-db conflicts with --serve-open, because that flag authorizes
requests whose identity was never established.
An anonymous caller is still admitted in permissive mode. With an authorizer
in place, that caller arrives as user:anonymous. Public read with
authenticated write is therefore a policy question, not a flag.
Views: attribute and key filtering
A view narrows what a successful read returns. A binding attaches a view to one relation on one object.
;; Hide every attribute outside the allowlist.
{:authz.view/name "support"
:authz.view/filter-type "attribute-allowlist"
:authz.view/attribute [":person/name" ":person/city"]}
{:authz.binding/relation "support"
:authz.binding/object "database:people"
:authz.binding/view "support"}
| Attribute | Meaning |
|---|---|
:authz.view/name | Name a binding refers to. Unique. |
:authz.view/filter-type | attribute-allowlist or attribute-denylist. Optional. |
:authz.view/attribute | Attribute idents the filter names. Repeatable. |
:authz.view/key | Protection class key ids the view permits. Repeatable. |
:authz.binding/relation | Relation the view attaches to. |
:authz.binding/object | Object the view attaches to. type:* is allowed. |
:authz.binding/view | Name of the view to apply. |
:authz.binding/unfiltered | Marks the relation as granting full attribute visibility. |
A view can name attributes, key ids, or both. :authz.view/filter-type is
needed only when the view names attributes.
Hidden means hidden
A datom on a hidden attribute is dropped inside the scan. It never binds, it never joins, and it never satisfies a predicate.
Every path that reaches an index directly is closed with it. A pull omits the
key. SQL reports NULL and refuses to push a predicate down that column.
get-else falls to its default. missing? reports missing. A lookup
reference does not resolve. Reverse-reference traversal returns nothing.
Each of those is an existence test over the attribute that the view withholds.
Combining views
When several relations succeed, their views intersect. Holding one more relation can never reveal more than holding it alone.
:authz.binding/unfiltered is the escape. Use it for a relation such as
owner that must see everything.
CAUTION:
:authz.binding/unfilteredgrants attributes and no keys. Keys are named by key id, and that binding names none. A relation that must read protected values names those key ids on a view.
Key grants
:authz.view/key names the protection class keys that a principal can use.
Read attribute protection, which states the key policy modes
and the strict default.
Surfaces that cannot filter refuse
A principal whose view hides attributes cannot write on any surface. The peer
server refuses Transact and Subscribe. The transactor refuses a write. SQL
refuses DML.
Transaction data is opaque bytes by the time authorization runs, and
Subscribe proxies the stream of the transactor, so neither can honor a
filter. All four ask the same question, so one principal under one policy gets
one answer whichever surface it reached.
A view that names every attribute takes nothing away, so it is not treated as hiding. A view that restricts only keys is refused nowhere.
Protecting the policy database
The policy database is governed by the policy that it holds. The database:*
ownership of the administrator is what keeps corium authz grant working.
Back it up like any other database. Read backup and restore.
Encryption at rest
Every durable artifact of an encrypted database is sealed. Index blobs, transaction-log record payloads, and cached segments are all covered.
The seal uses a per-database data key. That data key is itself wrapped by a key-encryption key, written KEK, that Corium never stores.
Encryption is fixed at creation
corium db create people --schema schema.toml --storage-key file:/etc/corium/storage.key
A database created without a storage key stays unencrypted forever. A database created with one stays encrypted forever.
There is no in-place migration. Migrating an unencrypted database means a backup and a restore into a new database.
Partly implemented.
corium backuprefuses an encrypted database. Backup format 1 cannot carry the key manifest, so no restore can open the resulting archive. An encrypted database therefore has no supported backup path today. Protect it with storage-level replication and snapshots until backup format 2 lands.
Key identities
A key identity is a URI.
| Scheme | Resolves | Content |
|---|---|---|
file:<path> | Yes | 32 raw bytes, or 64 hexadecimal characters. |
env:<NAME> | Yes | The same two forms, from an environment variable. |
awskms:, gcpkms:, vault: | No | Recognized and rejected as unsupported. |
Surrounding whitespace is ignored, so a key file with a trailing newline works.
Not implemented. KMS key identities are recognized, but no keyring resolves them. A process that names one fails at startup. Use
file:orenv:today.
Create a key:
head -c 32 /dev/urandom > /etc/corium/storage.key
chmod 400 /etc/corium/storage.key
CAUTION: Corium never stores the KEK. If you lose it, the database cannot be read again. Keep the key off the machine that holds the data, and keep a copy in a separate system.
Which processes need the key
Every process that reads storage directly needs the key.
corium transactor --data-dir /srv/corium --storage-key file:/etc/corium/storage.key
corium peer-server --db people --peer-bootstrap --storage-key file:/etc/corium/storage.key
--storage-key is repeatable, because one node can host databases under
different KEKs. CORIUM_STORAGE_KEY sets it from the environment.
A process resolves every named key at startup. A process without a key that its database needs therefore fails at open, and the error names the key. It does not fail later at its first read.
Thin clients and peer-server callers need no key. They receive plaintext over TLS.
corium db create --storage-key names the key for the transactor to
resolve. No key material leaves the transactor host.
Offline commands
Two offline commands read blob and log content, so they need the key.
corium log --data-dir /srv/corium --db people --storage-key file:/etc/corium/storage.key
corium gc --data-dir /srv/corium --storage-key file:/etc/corium/storage.key
Offline garbage collection refuses to run without the key. Without the key, the command cannot follow the index chunks, and a sweep deletes them.
Inspecting keys
corium keys status people
The command prints the KEK, the storage-key epochs, their states, and the share of the nonce budget that each epoch has spent.
| Field | Meaning |
|---|---|
:encrypted | Whether the database is encrypted at all. |
:kek | The key-encryption key that the manifest names. |
:rotation-due | true when the active epoch has spent half its nonce budget. |
:keys-unavailable | This node cannot load a manifest change. |
:keys-fenced | This node cannot load the epoch that the manifest opened. |
:storage-keys | One map per epoch: state, algorithm, KEK epoch, opening t, records sealed, budget used, live objects. |
Rotation
corium keys rotate people
Rotation opens a new storage-key epoch. New writes use it at once. It rewrites no stored object.
An older epoch stays readable. It drains as ordinary re-indexing rewrites its objects. An epoch retires only when no live object carries it.
Rotate when corium keys status reports :rotation-due true. That fires at
half the log-record nonce budget. A log record uses a random 96-bit nonce, so
an epoch must seal well under 2³² records. That count is the span of t that
the epoch covers.
Re-wrapping
corium keys rewrap people --kek file:/etc/corium/storage-2026.key
Re-wrapping re-encrypts the data keys under a new KEK. It reads, rewrites, and re-encrypts no stored object.
The transactor must resolve both KEKs at once. Follow this procedure.
- Start the transactor with both
--storage-keyflags. - Run
corium keys rewrap <db> --kek <new>. - Confirm the new KEK with
corium keys status <db>. - Restart the transactor with the new key only.
When a node cannot load a key change
A key change made elsewhere is picked up within one lease-renewal tick. When that load fails, the effect depends on which change it was.
| State | Cause | Effect |
|---|---|---|
:keys-unavailable true | The manifest changed, and this node cannot load it. Usually a re-wrap to a KEK it cannot resolve. | Warning only. Reads and writes continue under the keys it already holds. The corium_keys_unavailable gauge rises. |
:keys-fenced true | The manifest opened an epoch that this node cannot load. | Writes refuse with FAILED_PRECONDITION, naming both epochs. Reads, index publication, and the lease continue. |
The difference is deliberate. A re-wrap leaves the data keys unchanged. A write refusal therefore turns a key-service outage into a write outage for no confidentiality gain. A rotation is different in kind: records sealed under a closed epoch draw on a budget that has stopped counting them.
Both states clear as soon as a load succeeds. The fix is the same for both.
- Give the process a
--storage-keythat resolves the KEK that the manifest now names. - Restart the process.
Only the fenced state stops writes while you do this.
The second layer
Storage encryption protects the medium. It does not protect a fact from a reader that Corium serves.
The second layer is attribute protection. Values on a protected attribute are sealed by the writing peer under a class key. Only a process whose keyring resolves that key sees them in the clear.
The two layers are independent. A database can use either, both, or neither.
--storage-key supplies the keys of both, because one process keyring holds
key-encryption keys and class keys alike.
Attribute protection
Encryption at rest protects the medium. Attribute protection protects facts from readers.
Values on a protected attribute are sealed by the writing peer under the key of a protection class. Only a process whose keyring resolves that key sees them in the clear. The transactor does not. A peer without the key does not. An operator with storage credentials does not.
Declaring a class
A protection class names a key. It never holds key material. Declare classes in the schema file that creates the database.
[protect.pii]
key = "file:/etc/corium/pii.key"
padding = 64
on-missing-key = "redact"
[[entity]]
name = "person"
[entity.attributes]
name = "string"
ssn = { type = "string", protection = "protect/pii" }
A [protect.<name>] section declares the class :protect/<name>.
| Option | Values | Default |
|---|---|---|
key | Key identity, for example "file:/etc/corium/pii.key" | Required |
algorithm | "aes-256-gcm-siv" | "aes-256-gcm-siv" |
scope | "attribute" or "entity" | "attribute" |
padding | Bytes to round the plaintext up to. At least 16. | Unset |
on-missing-key | "redact", "hide", or "error" | "redact" |
legacy-plaintext | "redact" or "pass-through" | "redact" |
epoch | Key epoch that new values seal under | 1 |
The EDN form declares the same class as an entity with :db.protect/*
attributes, and points an attribute at it with :db/protection.
Not implemented. A schema update points an attribute at an installed class. It cannot install a class. Decide the classes before you create the database.
Partly implemented.
scope = "entity"parses and stores, and a writing peer refuses to seal under it. Use the default attribute scope today.
What each option costs
scope decides what the sealing determinism leaks. Sealing must be
deterministic, because the transactor compares values as bytes and holds no
key. Under "attribute" a reader without the key can tell that two entities
share a value on that attribute. Under "entity" the seal also binds the
entity, so it leaks only the repeated value of that one entity.
padding rounds the plaintext up to a multiple of that many bytes before
sealing. It costs storage and removes the length side channel for short,
guessable values.
on-missing-key decides what a reader who cannot open a value gets.
| Policy | Result |
|---|---|
redact | The value binds in redacted form. EDN prints #corium/redacted. SQL prints NULL. |
hide | The datom is dropped from every scan. The entity leaves the join. |
error | The read fails. |
Under all three, an unopenable value never satisfies a constant and never satisfies a predicate. It binds, it disappears, or it raises. It never matches by accident.
legacy-plaintext decides the same question for a value that was written
before the attribute became protected.
Protection is not free
A protected attribute cannot also carry :db/index, :db/unique, or
:db.type/ref. Ciphertext order is not value order, so a protected value can
never appear in a value-ordered index. The schema rejects the combination
rather than surprising a range query later.
An attribute that has ever been protected can never gain :db/index or
:db/unique, even after it is unprotected again.
Where an application needs an indexed lookup on a protected field, add a second unprotected attribute that holds a keyed hash of the value. That leak is then explicit.
Which processes need class keys
--storage-key is the keyring of the process. It resolves key-encryption keys
and protection class keys alike. Give a class key to the processes that must
read the class, and to no others.
corium peer-server --db people \
--storage-key file:/etc/corium/storage.key \
--storage-key file:/etc/corium/pii.key
A process with no class keys is a fully working peer. It commits, it indexes, it syncs, and it answers every query that touches no protected value identically. Protected values come back redacted, hidden, or refused, as the class policy says.
corium console, corium sql, and corium tui take no key flag. They print
protected values in redacted form.
Serving many principals from one process
A peer server and a PostgreSQL wire server serve many principals from one process. Which of the keys of that process a request can use is a policy question.
Name the key ids on a view, and bind the view to the relation that can read them:
{:authz.view/name "pii-reader" :authz.view/key ["file:/etc/corium/pii.key"]}
{:authz.binding/relation "hr" :authz.binding/object "database:people"
:authz.binding/view "pii-reader"}
A guarded server defaults to the strict key policy. A principal whose decision names no key id hydrates nothing.
| Mode | A decision that names no key id | Default when |
|---|---|---|
strict | Grants no class key. | Authentication is configured. |
server-wide | Grants the whole keyring of the process. | Authentication is off. |
--key-policy strict and --key-policy server-wide override the default on
peer-server and on postgres-server.
Three rules matter.
- Key grants combine across successful paths by intersection. One more relation can never widen a key set.
- Granting a key id that the process does not hold does nothing. Policy narrows the keyring of the process. It never extends it.
:authz.binding/unfilteredgrants full attribute visibility and no keys. Keys are named by key id, and that binding names none. A relation that must read protected values names them.
CAUTION: This is authorization, not cryptography. A key-holding server still holds the plaintext and is choosing not to disclose it. A compromised or misconfigured server defeats it.
For a genuinely less-trusted deployment, run the server with no class keys. Let each entitled application embed a peer with its own keyring, so the keys stay in the process that owns them.
Not implemented. Seal-through mode, in which the server forwards sealed values and the thin client opens them itself, does not exist. A key set is always resolved in the server.
Upgrading a guarded server that holds keys
Turning authorization on changes what a key-holding server discloses. Under
the derived strict default, a principal whose policy names no key id stops
seeing protected plaintext and starts seeing redactions.
That is the safe direction, and it is loud. Plan the change.
- List the relations that must read each class.
- Create one view per class, naming its key ids on
:authz.view/key. - Bind each view to its relation with
:authz.binding/*. - Restart the server with
--authz-db.
To keep the old behavior while you write those grants, pass
--key-policy server-wide deliberately.
Changing protection
Protection is not fixed at database creation.
corium schema update
can protect an attribute, unprotect it, or move it to another class.
The change is forward-only. Values written from that basis onward take the new form. Every value already stored keeps the form it had.
- Protecting an attribute does not seal the plaintext already stored.
- Unprotecting an attribute does not open the ciphertext already stored.
The plan reports both consequences and requires
--ack protection-forward-only. It also reports that lookup references and
value-ordered reads through the attribute stop working from that basis on.
Not implemented. Sweeping the current values into the new form is a rewrite job that does not exist. There is no
corium keys protect,corium keys unprotect, orcorium keys audit. Class key rotation and crypto-shredding commands are also absent.
What a protected value looks like on the wire
A sealed value needs thin-client protocol version 3. An older client never
receives one. Values reach a thin client as boundary EDN, which renders an
unopened value as #corium/redacted, a tagged element that every EDN reader
parses.
High availability
One transactor holds the write lease per database. A warm standby polls the lease and takes over when it lapses.
This is an active and standby pair. It is not a consensus protocol. The root store is the single arbiter.
Starting a pair
Start both members identically, each with its own identity and endpoint:
corium transactor --data-dir /srv/corium --ha \
--owner txor-a --advertise http://txor-a:4334 --listen 0.0.0.0:4334
corium transactor --data-dir /srv/corium --ha \
--owner txor-b --advertise http://txor-b:4334 --listen 0.0.0.0:4334
Whichever starts first becomes active. The other stands by.
The standby rescans the catalog every lease-renewal interval, so a database
created on the active is picked up. It rejects client work with a standby
FAILED_PRECONDITION that names the current lease holder.
Give each member a stable --owner. A restarted member then re-acquires its
own unexpired lease at once.
Storage requirements
Both members must see the same blob store, root store, and log.
| Store | Shared storage for a pair |
|---|---|
fs | Needs a shared filesystem for <data-dir> on both members. |
postgres, s3 | Shared by construction. No shared filesystem is needed. |
mem, turso | Not usable for a pair. |
CAUTION: Never run two members against diverged copies of a data directory. The store is the source of truth.
Peer failover
Peers list both endpoints and fail over automatically:
corium peer-server --db people \
--transactor http://txor-a:4334,http://txor-b:4334
A library peer passes the same list through ConnectConfig::with_failover. A
peer with storage credentials can also rediscover the advertised endpoint of
the current holder from the database root.
Guarantees
- Takeover is ordinary crash recovery. The standby acquires the lapsed lease, which atomically fences the deposed writer, replays the log tail, and serves.
- No acknowledged transaction is ever lost or duplicated. A post-append ownership check runs before every acknowledgement.
- A deposed transactor can never publish. Every root write is a compare-and-set on the record that holds the lease.
- Peer subscriptions reconnect and backfill without gaps.
- A deposed member returns to standby on its own. No operator action is needed after a garbage collection pause or a partition.
Unavailability window
Writes are unavailable from the crash until takeover. The bound is the sum of three terms.
- One lease time-to-live, because the last renewal of the active must expire.
- One standby poll interval, which is one third of the time-to-live.
- Reconnect backoff on the peer.
With the defaults that is about 6.7 seconds plus backoff.
Ambiguous transactions
A transact call that fails before it reaches the commit point is retried
transparently within failover_timeout. Standby rejection and connection
refusal are both in this class.
A call whose connection died mid-request is ambiguous. The transaction is committed, or it is absent. The call surfaces an error, exactly like a transactor crash between durability and reply.
On an ambiguous error, run
syncand read before you resubmit. A blind retry can write the data twice.
Tuning
| Knob | Default | Effect |
|---|---|---|
--lease-ttl-ms | 5000 | Failover detection bound. Renewals run at one third of it. |
--heartbeat-ms | 10000 | Subscription heartbeats. A peer presumes the transactor dead after 3 missed intervals. |
Peer reconnect_min / reconnect_max | 100 ms / 5 s | Reconnect backoff while endpoints rotate. |
Peer failover_timeout | 30 s | How long a safe-to-retry transact failure rides out a takeover. |
A lower lease time-to-live gives faster takeover. It also costs more root-store traffic, and it tolerates shorter garbage collection or input and output pauses on the active.
Keep the heartbeat interval at or below the lease time-to-live. The heartbeat is what detects a partition that TCP has not noticed.
Clock skew between members shifts detection latency only. It never affects safety, because the root store is the single arbiter.
Runbooks
The runbooks chapter holds the procedures for planned failover, a crashed active, split-brain suspicion, and both members down.
Fleet topology
Not implemented. A fleet topology is specified. The design distributes databases across overlapping candidate sets and gives clients one load-balanced address, with the same lease and failover guarantees. It does not change the commands in this chapter. See transactor-fleet.md.
Backup and restore
Backup is online. Restore is offline.
How backup works
The backup contacts the running transactor once. That call fixes the current transaction basis, and it returns the connection details of the underlying storage.
The backup then reads the storage log independently, through that basis only. Transactions committed while the backup runs are left for the next incremental run.
corium backup --transactor http://127.0.0.1:4334 people /backups/people.corium
| Flag | Default | Effect |
|---|---|---|
--transactor <url> | http://127.0.0.1:4334 | Transactor used for storage discovery. |
--token <secret> | Development token | Bearer token. --token "" connects anonymously. |
--ca <pem>, --tls-domain <name> | None | TLS for the transactor connection. |
The positional arguments are the database name and the destination file.
Incremental backup
Run the same command with the same file for an incremental refresh.
The backup reads only the transaction records after its existing checkpoint,
and it appends one new checkpoint frame. The report prints
:replayed-transactions.
The first run embeds the immutable snapshot blobs and retains that index snapshot as a replay base. Later runs do not repeat that work.
The report is one EDN map:
{:db "people" :backup-format 1 :writer-version "…" :basis-t 1240
:index-basis-t 1200 :replayed-transactions 40 :copied-blobs 12 :reused-blobs 480}
The archive
A backup has exactly one representation: a binary .corium archive.
The header carries an independent backup-file format version, and the Corium version that created it. Every incremental checkpoint records the version that appended it.
An unsupported future format fails before restore, and the error names its writer.
--log-format human|json controls diagnostic logging only. It never changes
the artifact.
Not implemented. There is no
dumpcommand. Human, JSON, and EDN export belong in one, not in backup or restore.
Where a backup can run
| Store | Requirement |
|---|---|
fs, turso | Run where the absolute local storage path of the transactor is reachable. |
postgres, s3 | Connect to the same native storage that the transactor advertises. S3 credentials come from the standard AWS environment. |
mem | Rejected. A separate process cannot open process-local memory storage. |
The advertised PostgreSQL connection is read and write in this version. A future release can substitute read-only credentials without a protocol change.
Partly implemented.
corium backuprefuses an encrypted database. Backup format 1 cannot carry the key manifest. See encryption at rest.
Restore
Restore is offline, and it refuses to overwrite a database. The target transactor must be stopped.
corium restore /backups/people.corium --data-dir /srv/corium-restored --as-db people
Restoring under a new name creates a clone:
corium restore /backups/people.corium --data-dir /srv/corium --as-db people-staging
| Flag | Effect |
|---|---|
--data-dir <path> | Target transactor data directory. Required. |
--as-db <name> | Target database name. It can differ from the source name. Required. |
Restore writes a filesystem data directory. It does not write to a postgres,
turso, or s3 store directly.
Backup-container and database-storage versions are checked separately before publication.
After a restore
- Start the target transactor on the restored data directory.
- Wait until
:index-lagincorium db statsreaches zero. - Compare the basis with
:basis-tin the backup report. - Compare datom, entity, and attribute counts.
- Run a known query and compare the result.
- Redirect peers only after those checks pass.
Backup policy
Three rules make a backup useful.
- Run
corium db request-index <db>before a backup when the snapshot must be current. A backup reads through the published basis. - Keep the first full archive and its incremental chain together. An incremental run needs the checkpoint in the same file.
- Test a restore on a schedule. An untested backup is not a backup.
For a database that corium backup refuses, back up the underlying storage
instead. See the runbooks.
Forking a database
A fork creates a new database that duplicates an existing one at a transaction basis. The result is a sandbox wound back to a point in time.
Use a fork to debug against real data, or to try an alternative approach, without touching the original.
Unlike backup and restore, forking is online. It runs against the live transactor through the catalog service.
Commands
corium db fork people people-debug --as-of 1234
corium db fork people people-scratch
Without --as-of, the fork is taken at the current basis.
The command prints one EDN map:
{:db "people-debug" :forked-from "people" :basis-t 1234 :created true}
What a fork copies
The fork copies only the transaction-log prefix through the requested basis.
Every t up to the basis of the source names a transaction, so any value in
range is exact.
Schema metadata is shared. Index segments deduplicate by content address in the blob store, so a fork is cheap in storage.
The new database replays that prefix, publishes its own indexes, and then transacts completely independently of its source.
Rules
- A basis ahead of the source is refused.
- An existing target is never overwritten. The command prints
:created false, and nothing is changed. - The fork is a full database. It accepts writes, it needs its own backups, and garbage collection covers it.
When not to fork
A read-only view of a point in time does not need a fork. A peer gets one
locally with as-of, at no storage cost.
Fork only when the sandbox must accept writes.
Cleaning up
Delete a finished fork like any other database:
corium db delete people-debug
Blobs shared with the source stay reachable from the source root. Garbage collection sweeps only what no live root reaches. See garbage collection.
Garbage collection
Old index roots keep old segments alive until no reader needs them. Garbage collection deletes the segments that no live root reaches.
Collection is epoch-based, and it is never urgent.
The retention rule
A sweep deletes an unreachable blob only when the blob is older than the retention window. The window must cover any reader that can still hold a stale root.
The default window is 72 hours.
Scheduled collection
The transactor runs collection on a schedule.
| Flag | Default | Effect |
|---|---|---|
--gc-interval <duration> | 1h | Interval between sweeps. off disables the duty. |
--gc-window <duration> | 72h | Retention window. |
Collection is serialized with index publication. The two never run at the same time.
Manual collection
Online collection asks a running transactor to sweep:
corium gc --transactor http://127.0.0.1:4334 --window 72h
Offline collection reads a data directory directly. The transactor must be stopped:
corium gc --data-dir /srv/corium --window 72h
Both use the same retention rule. --data-dir and --transactor conflict.
An encrypted database needs the key for offline collection:
corium gc --data-dir /srv/corium --window 72h --storage-key file:/etc/corium/storage.key
Offline collection refuses to run on an encrypted database without the key. Without the key, a sweep deletes the index chunks that it cannot follow.
Zero window
CAUTION: Use
--window 0only when no stale root and no in-flight reader can exist. A zero window deletes a blob as soon as it is unreachable. A reader that still holds an older root then fails.
A safe use is a stopped system with no peers running.
Why collection is safe
Deletion is the only mutation, and it touches only unreachable data. A bug in the mark phase can therefore strand garbage. A generous window makes data loss a non-risk.
Deleting a database deletes its root, and the sweep reclaims the blobs afterward.
Monitoring
Three counters report collection.
| Source | Fields |
|---|---|
corium db stats <db> | :gc-runs, :gc-swept-blobs |
| Metrics endpoint | corium_transactor_gc_runs_total, corium_transactor_gc_swept_blobs_total, corium_transactor_gc_retained_blobs_total |
corium tui | The Metrics panel |
A retained count that grows steadily means that the window is longer than it needs to be, or that a root is pinning old segments.
Tuning
| Situation | Setting |
|---|---|
| Peers hold long-lived database values | Raise --gc-window above the longest reader lifetime. |
| Storage cost matters more than reader tolerance | Lower --gc-window, and watch for reader errors. |
| Bulk load in progress | Set --gc-interval off, and run one manual sweep afterward. |
| A pause on the active transactor causes failover | Raise --lease-ttl-ms, or lengthen --gc-interval. |
Collection competes with index publication for processor time and storage bandwidth. On a busy system, run it less often rather than with a shorter window.
Monitoring
Three surfaces report the state of a Corium system: the metrics endpoint,
corium db stats, and the tracing log.
The metrics endpoint
Pass --metrics-listen to a transactor or a peer server:
corium transactor --data-dir /srv/corium --metrics-listen 127.0.0.1:9464
The endpoint serves Prometheus text at /metrics.
CAUTION: Keep the metrics listener on a private operations network. The endpoint has no bearer-token authentication.
Transactor metrics
| Metric | Type | Meaning |
|---|---|---|
corium_transactor_transactions_total | Counter | Committed transactions. |
corium_transactor_transaction_failures_total | Counter | Rejected transactions. |
corium_transactor_transaction_latency_seconds | Histogram | Commit latency. |
corium_transactor_queue_depth | Gauge | Commit queue depth. |
corium_transactor_index_duration_seconds | Histogram | Index publication duration. |
corium_transactor_gc_runs_total | Counter | Collection runs. |
corium_transactor_gc_swept_blobs_total | Counter | Blobs deleted. |
corium_transactor_gc_retained_blobs_total | Counter | Unreachable blobs kept by the window. |
corium_keys_unavailable | Gauge | Nodes that cannot load a key manifest change. |
Peer server metrics
| Metric | Type | Meaning |
|---|---|---|
corium_peer_queries_total | Counter | Queries served. |
corium_peer_query_latency_seconds | Histogram | Query latency. |
corium_peer_query_fuel_spent_total | Counter | Datoms touched. |
Segment cache metrics
A peer server with --segment-cache-dir adds these.
| Metric | Labels | Meaning |
|---|---|---|
corium_peer_segment_cache_requests_total | result, tier | Hits and misses per tier. |
corium_peer_segment_cache_native_fetches_total | result | Fetches that went to storage. |
corium_peer_segment_cache_bytes_read_total | source | Bytes read per source. |
corium_peer_segment_cache_admissions_total | result | Admissions and rejections. |
corium_peer_segment_cache_used_bytes | tier | Bytes in use. |
corium_peer_segment_cache_capacity_bytes | tier | Configured capacity. |
On-demand statistics
corium db stats people
The command prints the basis, the index basis, the index lag, the counts, and the transactor counters. See the database catalog.
The transactor Status call carries the same data. The Metrics panel of
corium tui samples it live, and it is the only surface
that shows lease ownership.
Auditing schema changes
Every applied schema update records its requester, its digests, its observed basis, and its acknowledgements on the transaction entity. Those are ordinary attributes, so the schema history of a database is a query.
[:find ?when ?who ?tool
:where [?tx :db.schemaUpdate/requester ?who]
[?tx :db.schemaUpdate/tool ?tool]
[?tx :db/txInstant ?when]]
[[#inst 1785899778642 "static-token:operator" "corium-cli/0.1.0"]]
The requester is the authenticated principal. A caller never supplies it.
An ordinary transaction cannot write :db.schemaUpdate/*, so no transaction
can claim to have been a schema update. See
schema management.
Logging
Tracing is human-readable by default. --log-format json writes structured
logs, and RUST_LOG filters them.
RUST_LOG=corium_transactor=info,corium_peer=warn \
corium --log-format json transactor --data-dir /srv/corium
--log-format is a global flag, so it comes before the subcommand.
Useful targets:
| Target | Content |
|---|---|
corium_transactor | Commit pipeline, indexing, leases, garbage collection. |
corium_peer | Connection, subscription, failover. |
corium_authz::audit | Every authorization decision, with its basis. Denials at info, grants at debug. |
Log lines worth an alert:
standby took over write lease— a failover happened.deposed— this member lost the lease. It stands down on its own.standing by; lease held elsewhere— normal on a standby at startup.
What to alert on
| Signal | Condition | Meaning |
|---|---|---|
| Index lag | Grows without limit | Publication cannot keep up. |
corium_transactor_queue_depth | Stays high | The write path is saturated. |
corium_transactor_transaction_failures_total | Rises sharply | Validation errors, or a fenced writer. |
corium_keys_unavailable | Above zero | A node cannot load a key manifest change. |
corium_transactor_gc_retained_blobs_total | Grows steadily | Storage is not being reclaimed. |
| Lease owner | Changes unexpectedly | An unplanned failover. |
Peer memory is not exported as a metric. Watch the resident memory of the process, because a peer holds the whole history. See indexes and storage.
Runbooks
Each runbook is a procedure. Read the whole procedure before you start it.
Planned failover
Use this before maintenance on the active member.
- Stop the active member with
Ctrl-C, or withSIGINT. It releases its leases on the way out. - Watch the log of the standby for
standby took over write lease. Takeover happens within one third of the lease time-to-live. - Do the maintenance.
- Start the member again with the same
--ownerand--ha. It rejoins as standby.
If the supervisor sends SIGTERM, the process dies without releasing the
lease. Takeover then costs one full lease time-to-live.
Crashed active
Nothing is required for service. The standby takes over within the time-to-live plus one third.
- Confirm the takeover. Watch the basis advance with
corium db stats, and read the lease owner in theMetricspanel ofcorium tui. - Start the crashed member again under its supervisor with
--ha. It rejoins as standby. - Investigate the crash afterward, not before.
Split-brain suspicion
Both members print ownership messages in their logs.
This is not possible for durable state. The root record is owned by exactly one lease version, and every publish and acknowledgement is fenced by it.
- A member that logs
deposedis the loser. It stands down on its own. - Trust the root record, not the process logs.
corium tuireads the lease owner from theStatuscall. - Take no other action.
Both members down
- Start either member. Prefer the one with the newest data-directory modification times if storage is not shared.
- The member waits out any unexpired lease. Without
--hait waits up to--lease-wait-ms. With--hait waits without limit. - It recovers by log replay, and it serves.
- Start the second member. It becomes standby.
Recovery from a backup
- Stop the affected transactor, and preserve its data directory. Do not delete it.
- Restore the newest backup into an empty directory, or under a new name:
corium restore <file> --data-dir <empty-dir> --as-db <name>. - Start a transactor on the restored directory.
- Wait until
:index-lagincorium db statsreaches zero. - Compare the basis, the datom count, the entity count, and the attribute count with the backup report.
- Run a known query, and compare the result.
- Redirect peers only after those checks pass.
Transactor will not start
Read the error first. Four causes are common.
| Error names | Cause | Fix |
|---|---|---|
| A lease holder | Another transactor holds the lease. | Add --ha to stand by, or stop the other member. |
| A storage key | The process cannot resolve a named KEK. | Give it a --storage-key that resolves. |
| A Cargo feature | The binary lacks the storage or OIDC feature. | Rebuild with the feature. |
| A missing data directory | --data-dir is absent. | Pass it, or set :data-dir in the configuration file. |
The process fails at startup for a key it cannot resolve. That is by design. It does not fail later at the first read.
Writes refuse with FAILED_PRECONDITION
Two causes give this status.
A standby. The message names the current lease holder. Point the client at that endpoint, or pass the whole endpoint list.
A fenced key. Run corium keys status <db>. If :keys-fenced is true,
the manifest opened an epoch that this node cannot load.
- Give the transactor a
--storage-keythat resolves the KEK that the manifest names. - Restart the transactor.
- Confirm that
:keys-fencedisfalse.
See encryption at rest.
An ambiguous transaction
The result of a transact call whose connection died mid-request is unknown.
The transaction is committed, or it is absent.
- Do not resubmit yet.
- Run
syncon the connection. - Read the data back and decide from the result.
- Resubmit only if the write is absent.
Changing the schema of a live database
Writes continue throughout. Only a blocked change stops the procedure.
- Edit the schema file. Keep every attribute that must survive, because a file that omits one reports it as unmanaged.
- Plan it:
corium schema update <db> --schema <file>. Nothing is written. - Read every execution class, every count, and every acknowledgement code.
- Run the invocation that the last line of the plan prints, and add the path of the schema file.
- Confirm the new schema generation in the output of the apply.
- Compare
:attributesincorium db statswith the expected count.
A plan is invalidated by a schema change or a failed condition, never by ordinary data writes. Re-plan if the digest is refused.
See schema management.
A schema plan is blocked or refused
Read the reason that the plan prints under the change.
| Reason | Meaning | Fix |
|---|---|---|
value-type-mutation | A value type cannot change in place. | Follow the replacement-attribute recipe that the plan prints. |
unique-duplicates | Duplicate values exist. | Retract the duplicates, then plan again. |
cardinality-conflicts | An entity holds several values where the file asks for one. | Choose a winner per entity and retract the rest. |
ever-protected | The attribute has been protected at some time. | It can never gain index or unique. Drop them from the file. |
protection-conflict | The declaration combines protection with index, unique, or ref. | Drop the conflicting property from the declaration. |
An apply can also fail after a clean plan.
| Error code | Meaning | Fix |
|---|---|---|
plan-mismatch | The schema changed between the plan and the apply. | Plan again and read the new plan. |
allow-required | A change needs --allow <class>. | Add the exact allowance the plan names. |
ack-required | A change needs --ack <code>. | Add the exact code the plan names. |
Every request is denied
The policy denies, or the policy is unreadable.
- Run
corium authz status. A missing basis means the policy is unreadable. - If it is unreadable, a break-glass role admits an operator. See authorization.
- If the policy denies, stop the transactor.
- Start it again without
--authz-db. - Fix the tuples with
corium authz grant. Test each one withcorium authz check. - Restart with
--authz-db.
Index lag grows without limit
- Read
:index-lagincorium db stats, and the publication duration in the metrics endpoint. - Lower
--index-backoff, so publication takes a larger share of wall-clock time. - Lower
--index-tail-thresholdif a large threshold defers the work. - If neither helps, the storage backend is the limit. Give it faster storage, or reduce the write rate.
Index lag never risks durability. It lengthens cold-peer bootstrap and it makes a backup less fresh.
A peer uses too much memory
A peer holds every datom that it has seen, including retractions.
- Confirm the cause. Memory tracks total history, not the size of the live database.
- Restart the peer with
--peer-bootstrap, so it starts from the published snapshot rather than replaying the log from basis 0. - Avoid opening many distinct time views in one process. Each distinct view costs a fold of the whole history.
- Split the workload across more peer processes.
See indexes and storage.
Backing up an encrypted database
corium backup refuses an encrypted database.
- Stop the transactor, or accept a crash-consistent copy.
- Copy the underlying storage with its own tool. Use a filesystem snapshot, a PostgreSQL dump, or S3 replication.
- Copy the KEK separately, and keep it in a different system.
- Test the restore path on a separate host before you rely on it.
Storage is full
- Run a manual sweep:
corium gc --transactor <url> --window 72h. - If that reclaims little, read
corium_transactor_gc_retained_blobs_total. A large retained count means that the window is holding the blobs. - Lower
--gc-windowonly when no reader holds a root older than the new window. - Delete finished forks and staging clones with
corium db delete.
Emergency: recreate a database from the log
The log is the source of truth. A data directory with an intact log recovers by replay.
- Stop every transactor that touches the directory.
- Preserve a copy of the whole directory.
- Start one transactor on the directory. Startup replays the log tail after the last published index basis.
- Compare
corium db statswith the last known values.
Never edit or delete files under <data-dir>/logs by hand. Old lease-version
files are inert history that readers must merge.
Command reference
This chapter lists every corium command. Run corium <command> --help for
the authoritative flag list of a build.
Global
corium [--log-format human|json] <command>
--log-format comes before the subcommand. corium tui writes no tracing
output, because it owns the terminal.
Shared flag groups
Connection flags apply to peer-server, postgres-server, every db
subcommand, schema update, every authz subcommand, every keys
subcommand, console, tui, and sql.
| Flag | Environment | Default |
|---|---|---|
--transactor <url> | http://127.0.0.1:4334 | |
--token <secret> | CORIUM_TOKEN | Shared development token |
--ca <pem> | None | |
--tls-domain <name> | None | |
--peer-bootstrap | Off |
Serving flags apply to transactor, peer-server, and
postgres-server.
| Flag | Environment |
|---|---|
--serve-token <secret> | CORIUM_SERVE_TOKEN |
--require-auth | |
--serve-open | |
--oidc-issuer <url> | |
--oidc-audience <aud> | |
--oidc-jwks-file <path> | |
--authz-db <name> | CORIUM_AUTHZ_DB |
--authz-fresh-writes | |
--authz-break-glass-role <role> | |
--authz-max-depth <n> | |
--key-policy strict|server-wide | |
--tls-cert <pem> | |
--tls-key <pem> |
--key-policy decides which protection class keys a caller can use. Only
peer-server and postgres-server act on it. postgres-server rejects
--tls-cert and --tls-key, because it does not terminate TLS.
Key flags apply to transactor, peer-server, postgres-server, gc,
and log.
| Flag | Environment |
|---|---|
--storage-key <uri> | CORIUM_STORAGE_KEY |
One keyring serves both purposes. It resolves key-encryption keys and protection class keys. The flag is repeatable, and the environment variable takes a comma-separated list.
Servers
corium transactor
Runs a transactor over a data directory. See the transactor.
Storage flags: --config, --store, --store-plugin, --plugin-store,
--plugin-read-only-config, --data-dir, --turso-path,
--postgres-url, --postgres-read-only-url, --s3-bucket, --s3-prefix,
--s3-region, --s3-endpoint-url, --s3-read-only-access-key-id,
--s3-read-only-secret-access-key, --s3-read-only-session-token,
--s3-read-only-role-arn, --s3-read-only-role-session-name,
--s3-read-only-role-duration-seconds, --s3-read-only-role-external-id.
Process flags: --listen, --owner, --advertise, --metrics-listen.
Lease flags: --ha, --lease-ttl-ms, --lease-wait-ms, --heartbeat-ms.
Index flags: --index-interval-ms, --index-backoff,
--index-tail-threshold, --index-tail-deadline-ms.
Collection flags: --gc-interval, --gc-window.
Function flags: --db-fn-fuel, --db-fn-memory-bytes.
corium peer-server
Hosts one database for thin clients. See peer server and thin clients.
Flags: --db, --listen, --max-fuel, --metrics-listen,
--segment-cache-dir, --segment-cache-capacity, --segment-cache-memory.
corium postgres-server
Serves the catalog over the PostgreSQL wire protocol. See SQL shell and PostgreSQL server.
Flags: --database (repeatable), --listen, --password, --allow-writes.
Storage administration
corium store verify <kind> <config>
Runs the blob and root conformance suite against a live backend. <config> is
the JSON configuration object of the backend. Flag: --store-plugin
(repeatable). See storage backends.
Use a disposable namespace. The command writes and deletes objects.
Catalog
corium db create <name>
Creates a database. Flags: --schema <path>, --storage-key <uri>.
corium db delete <name>
Deletes a database. It asks for no confirmation.
corium db list
Lists the databases that the transactor serves.
corium db stats <name>
Connects a peer and prints statistics.
corium db fork <name> <target>
Duplicates a database at a basis. Flag: --as-of <t>. See
forking a database.
corium db request-index <name>
Publishes the indexes now, bypassing pacing.
corium db index-policy <name>
Reads or overrides the pacing of one database. Flags: --interval-ms,
--backoff, --tail-threshold, --tail-deadline-ms. With no flags it prints
the current policy.
Schema
corium schema update <db> --schema <path>
Compares a schema file with the schema installed in a database and prints the
plan. It writes nothing without --apply. See
schema management.
Flags: --schema, --prune, --json, --detailed-exit-code, --apply,
--plan, --allow (repeatable), --ack (repeatable).
Authorization
See authorization.
| Command | Effect |
|---|---|
corium authz init | Creates the policy database with schema, permissions, and a first administrator. Flags: --db, --admin, --provider, --no-admin. |
corium authz grant <subject> <relation> <object> | Asserts a relationship tuple. Flag: --db. |
corium authz revoke <subject> <relation> <object> | Retracts a relationship tuple. Flag: --db. |
corium authz check <subject> <action> | Prints what the policy decides. Flags: --database, --provider, --role, --claim, --db. |
corium authz status | Prints the compiled basis and entity counts. Flag: --db. |
Keys
See encryption at rest.
| Command | Effect |
|---|---|
corium keys status <db> | Prints the KEK, the epochs, and the nonce budget. |
corium keys rotate <db> | Opens a new storage-key epoch. |
corium keys rewrap <db> --kek <uri> | Re-wraps the data keys under another KEK. |
Data care
corium gc
Sweeps unreachable blobs. Flags: --data-dir or --transactor (they
conflict), --token, --ca, --tls-domain, --window, --storage-key. See
garbage collection.
corium backup <db> <destination>
Creates or refreshes a backup from a live transactor. Flags: --transactor,
--token, --ca, --tls-domain. See
backup and restore.
corium restore <source>
Restores a backup offline. Flags: --data-dir, --as-db. Both are required.
Interactive surfaces
| Command | Effect |
|---|---|
corium console <db> | Datalog console. See query console. |
corium tui <db> | Dashboard. Flag: --refresh-ms. See terminal dashboard. |
corium sql <db> | SQL shell. Flags: -c/--command, -f/--file. See SQL. |
Offline inspection
corium log
Prints committed transactions from a filesystem data directory. The transactor does not need to be running.
Flags: --data-dir, --db, --from, --to, --storage-key.
--from is inclusive. --to is exclusive, and 0 means open-ended.
Commands that do not exist
Not implemented. There is no
corium transactcommand. Writes come from a client library, or fromcorium postgres-server --allow-writes.
Not implemented. There is no
corium dumpcommand. Human, JSON, and EDN export are deferred.
Not implemented.
corium schemahas only theupdatesubcommand.status,history, and job inspection are planned and absent.
Not implemented. There is no
corium keys protect,corium keys unprotect, orcorium keys audit. Protection changes go throughcorium schema update. See attribute protection.
Environment variables
Every variable below has an equivalent flag. A flag on the command line overrides the variable.
Corium variables
| Variable | Equivalent flag | Used by |
|---|---|---|
CORIUM_TOKEN | --token | Every client command. |
CORIUM_SERVE_TOKEN | --serve-token | transactor, peer-server, postgres-server. |
CORIUM_AUTHZ_DB | --authz-db | transactor, peer-server, postgres-server. |
CORIUM_STORAGE_KEY | --storage-key | transactor, peer-server, postgres-server, gc, log. |
CORIUM_STORE_PLUGINS | --store-plugin | transactor, store verify. |
CORIUM_PLUGIN_READ_ONLY_CONFIG | --plugin-read-only-config | transactor. |
CORIUM_POSTGRES_READ_ONLY_URL | --postgres-read-only-url | transactor. |
CORIUM_S3_READ_ONLY_ACCESS_KEY_ID | --s3-read-only-access-key-id | transactor. |
CORIUM_S3_READ_ONLY_SECRET_ACCESS_KEY | --s3-read-only-secret-access-key | transactor. |
CORIUM_S3_READ_ONLY_SESSION_TOKEN | --s3-read-only-session-token | transactor. |
CORIUM_S3_READ_ONLY_ROLE_ARN | --s3-read-only-role-arn | transactor. |
CORIUM_S3_READ_ONLY_ROLE_EXTERNAL_ID | --s3-read-only-role-external-id | transactor. |
CORIUM_STORAGE_KEY accepts a comma-separated list, because one process can
hold several keys. The same keyring resolves key-encryption keys and
protection class keys.
CORIUM_STORE_PLUGINS accepts a path-separator-delimited list of files and
directories. Corium searches a directory for platform dynamic libraries only,
and it never adds the working directory.
Rust and AWS variables
| Variable | Effect |
|---|---|
RUST_LOG | Tracing filter, for example corium_transactor=debug,corium_peer=info. |
HOSTNAME | Supplies the default --owner value, transactor-$HOSTNAME. |
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY | Primary S3 credentials. |
AWS_PROFILE, AWS_REGION, AWS_ENDPOINT_URL | Standard AWS configuration. |
The transactor takes its primary S3 credentials from the standard AWS chain, which also covers instance and task roles.
Handling secrets
Process arguments are visible to every user on the host. Three rules follow.
- Put a token, a password, or a connection URL in an environment variable, or in the configuration file.
- Set the mode of a key file and of a configuration file so that only the service user can read them.
- Prefer a
file:storage key over anenv:storage key. A file has a mode. An environment variable is inherited by child processes.
Default values
Network
| Setting | Default |
|---|---|
| Transactor listen address | 127.0.0.1:4334 |
| Peer server listen address | 127.0.0.1:4336 |
| PostgreSQL server listen address | 127.0.0.1:5432 |
| Metrics listen address | Disabled |
| Client transactor endpoint | http://127.0.0.1:4334 |
Storage
| Setting | Default |
|---|---|
--store | fs |
--data-dir | None. Required. |
--turso-path | <data-dir>/store.db |
--s3-prefix | Bucket root |
| S3 read-only role duration | 900 seconds |
Lease and availability
| Setting | Default |
|---|---|
--lease-ttl-ms | 5000 |
--lease-wait-ms | 15000 |
--heartbeat-ms | 10000 |
--ha | Off |
--owner | transactor-$HOSTNAME |
| Peer reconnect backoff | 100 ms to 5 s |
| Peer failover timeout | 30 s |
Index publication
| Setting | Default |
|---|---|
--index-interval-ms | 5000 |
--index-backoff | 4 |
--index-tail-threshold | 0 |
--index-tail-deadline-ms | 60000 |
Garbage collection
| Setting | Default |
|---|---|
--gc-interval | 1h |
--gc-window | 72h |
corium gc --window | 72h |
Query and function budgets
| Setting | Default |
|---|---|
--max-fuel (peer server) | 10000000 |
--db-fn-fuel | 1000000 |
--db-fn-memory-bytes | 16777216 (16 MiB) |
--authz-max-depth | 8 |
Segment cache
| Setting | Default |
|---|---|
--segment-cache-dir | Disabled |
--segment-cache-capacity | None. Required with the directory. |
--segment-cache-memory | 64 MiB, or the capacity when smaller |
Security
| Setting | Default |
|---|---|
| Authentication | Permissive: development token accepted, anonymous admitted |
| Authorization | Permit-all |
| TLS | Off |
| Encryption at rest | Off |
| Authorization database name | corium_authz |
authz init --admin | operator |
authz init --provider | static-token |
| Attribute protection | Off |
--key-policy | strict when authentication is configured, server-wide when it is not |
on-missing-key (protection class) | redact |
legacy-plaintext (protection class) | redact |
scope (protection class) | attribute |
Schema
| Setting | Default |
|---|---|
corium schema update mode | Read-only. --apply is needed to write. |
| Permitted execution class | additive only |
--prune | Off. Absent attributes are reported as unmanaged. |
| Exit code with changes planned | 0, or 2 with --detailed-exit-code |
Interactive surfaces
| Setting | Default |
|---|---|
corium tui --refresh-ms | 2000, minimum 250 |
corium log --from | 0 |
corium log --to | 0, meaning open-ended |
corium postgres-server write mode | Read-only |
Duration and size formats
A duration accepts one of the suffixes ms, s, m, h, and d, for
example 1h, 30m, 72h. A bare number is seconds. off disables the
scheduled collection duty.
A byte size accepts B, KiB, MiB, GiB, TiB, kB, MB, GB, and
TB. A bare number is bytes.
Glossary
Attribute. An entity in :db.part/db that declares a name, a value type,
a cardinality, and optional properties. See
schema management.
AEVT. The covering index sorted by attribute, entity, value, transaction. It holds all current datoms.
AVET. The covering index sorted by attribute, value, entity, transaction. It holds the datoms of indexed and unique attributes only.
Acknowledgement code. A stable, kebab-case name for a schema change whose
meaning changes, passed back with --ack.
Basis. The transaction number, written t, that a database value covers.
BLAKE3. The hash function that addresses blobs.
Blob store. The immutable, content-addressed half of the storage service.
Cardinality. Whether an attribute holds one value or many per entity.
Chunk. A content-defined run of a sorted key stream. A published leaf is exactly one chunk.
Class key. The key that seals values on the attributes of one protection class. A process resolves it through its own keyring.
Covering index. An index that holds whole datoms, so an answer needs no second lookup.
Database root. The record in the root store that names a database: its basis, its index roots, its log root, and its write lease.
Database value. An immutable snapshot of a database at one basis.
Datom. One fact: entity, attribute, value, transaction, and assert or retract.
EAVT. The covering index sorted by entity, attribute, value, transaction. It holds all current datoms.
Entity id. A 64-bit number: a 22-bit partition and a 42-bit sequence.
Epoch (storage key). One generation of the per-database data key. New writes seal under the newest open epoch.
Execution class. How much work a schema change needs: additive,
validate-reindex, rewrite, or destructive.
Fence. The mechanism that stops a deposed writer. Every root write is a compare-and-set on the record that holds the lease.
Fork. A new database that duplicates an existing one at a basis, and then diverges. See forking a database.
Fuel. A budget on the datoms a query touches, or on the work a database function does.
Group commit. Committing concurrent transactions as one batch under one
durability boundary, while each keeps its own t and its own acknowledgement.
Ident. A keyword that names an entity, above all an attribute.
Index basis. The transaction number that the published index trees cover.
Index lag. The difference between the basis and the index basis.
KEK. Key-encryption key. It wraps the per-database data key. Corium never stores it.
Key policy. Whether a serving process hydrates a caller with the class
keys that policy names (strict) or with its whole keyring (server-wide).
Lease. The right to write one database. It lives in the database root and is renewed by compare-and-set.
Lease version. The generation of a lease. It names the log file, or the log key prefix, that its holder appends to.
Lookup reference. A [attribute value] pair on a unique attribute, usable
anywhere an entity id is expected.
Partition. The high bits of an entity id. Entities of one partition sort together in EAVT.
Peer. A library, or a process, that holds a database value and queries it locally.
Peer server. A peer hosted as a standalone process for thin clients.
Plan digest. The hash of a schema plan. --apply refuses a digest that no
longer describes the change.
Principal. The identity of a request, produced by an identity provider.
Protection class. A named key identity and sealing policy. An attribute that names one has its values sealed by the writing peer. See attribute protection.
ReBAC. Relationship-based access control, the authorization model that
--authz-db enables.
Retirement. The schema change that refuses new assertions on an attribute while keeping its ident, its metadata, and its history readable.
Root store. The small, mutable, strongly consistent half of the storage service. It is updated only by compare-and-set.
Schema generation. A per-database counter that advances once for each committed transaction containing a schema change.
Segment. An immutable, content-addressed node of an index tree.
Standby. A transactor that polls a lease held elsewhere and takes over when it lapses.
Storage plugin. A dynamic library that registers a storage backend at run time. See storage backends.
Tempid. A transaction-local placeholder for an entity id, resolved at
commit. A collision on a :db.unique/identity attribute becomes an upsert.
Transaction report. The record that a commit broadcasts to peers: the basis before, the basis after, the datoms, and the tempid map.
Transactor. The single writer for a database.
Unmanaged attribute. An installed attribute that the desired schema file
does not name. corium schema update leaves it alone unless --prune is
given.
Upsert. Unifying a tempid with an existing entity through a
:db.unique/identity attribute.
VAET. The covering index sorted by value, attribute, entity, transaction. It holds reference-typed datoms.
View. A policy object that hides attributes from a principal, names the class keys the principal can use, or both.
t. The sequence part of a transaction id, and the name of a basis.
:db/txInstant. The commit time of a transaction, asserted as a datom on
the transaction entity.