High availability
One transactor holds the write lease per database. A warm standby polls the lease and takes over when it lapses.
This is an active and standby pair. It is not a consensus protocol. The root store is the single arbiter.
Starting a pair
Start both members identically, each with its own identity and endpoint:
corium transactor --data-dir /srv/corium --ha \
--owner txor-a --advertise http://txor-a:4334 --listen 0.0.0.0:4334
corium transactor --data-dir /srv/corium --ha \
--owner txor-b --advertise http://txor-b:4334 --listen 0.0.0.0:4334
Whichever starts first becomes active. The other stands by.
The standby rescans the catalog every lease-renewal interval, so a database
created on the active is picked up. It rejects client work with a standby
FAILED_PRECONDITION that names the current lease holder.
Give each member a stable --owner. A restarted member then re-acquires its
own unexpired lease at once.
Storage requirements
Both members must see the same blob store, root store, and log.
| Store | Shared storage for a pair |
|---|---|
fs | Needs a shared filesystem for <data-dir> on both members. |
postgres, s3 | Shared by construction. No shared filesystem is needed. |
mem, turso | Not usable for a pair. |
CAUTION: Never run two members against diverged copies of a data directory. The store is the source of truth.
Peer failover
Peers list both endpoints and fail over automatically:
corium peer-server --db people \
--transactor http://txor-a:4334,http://txor-b:4334
A library peer passes the same list through ConnectConfig::with_failover. A
peer with storage credentials can also rediscover the advertised endpoint of
the current holder from the database root.
Guarantees
- Takeover is ordinary crash recovery. The standby acquires the lapsed lease, which atomically fences the deposed writer, replays the log tail, and serves.
- No acknowledged transaction is ever lost or duplicated. A post-append ownership check runs before every acknowledgement.
- A deposed transactor can never publish. Every root write is a compare-and-set on the record that holds the lease.
- Peer subscriptions reconnect and backfill without gaps.
- A deposed member returns to standby on its own. No operator action is needed after a garbage collection pause or a partition.
Unavailability window
Writes are unavailable from the crash until takeover. The bound is the sum of three terms.
- One lease time-to-live, because the last renewal of the active must expire.
- One standby poll interval, which is one third of the time-to-live.
- Reconnect backoff on the peer.
With the defaults that is about 6.7 seconds plus backoff.
Ambiguous transactions
A transact call that fails before it reaches the commit point is retried
transparently within failover_timeout. Standby rejection and connection
refusal are both in this class.
A call whose connection died mid-request is ambiguous. The transaction is committed, or it is absent. The call surfaces an error, exactly like a transactor crash between durability and reply.
On an ambiguous error, run
syncand read before you resubmit. A blind retry can write the data twice.
Tuning
| Knob | Default | Effect |
|---|---|---|
--lease-ttl-ms | 5000 | Failover detection bound. Renewals run at one third of it. |
--heartbeat-ms | 10000 | Subscription heartbeats. A peer presumes the transactor dead after 3 missed intervals. |
Peer reconnect_min / reconnect_max | 100 ms / 5 s | Reconnect backoff while endpoints rotate. |
Peer failover_timeout | 30 s | How long a safe-to-retry transact failure rides out a takeover. |
A lower lease time-to-live gives faster takeover. It also costs more root-store traffic, and it tolerates shorter garbage collection or input and output pauses on the active.
Keep the heartbeat interval at or below the lease time-to-live. The heartbeat is what detects a partition that TCP has not noticed.
Clock skew between members shifts detection latency only. It never affects safety, because the root store is the single arbiter.
Runbooks
The runbooks chapter holds the procedures for planned failover, a crashed active, split-brain suspicion, and both members down.
Fleet topology
Not implemented. A fleet topology is specified. The design distributes databases across overlapping candidate sets and gives clients one load-balanced address, with the same lease and failover guarantees. It does not change the commands in this chapter. See transactor-fleet.md.