Scaling and high availability

A Tricryption key server can run as a cell: several key-server members, each on its own host, sharing one key database, reached by every client through one address. Losing a member or a whole zone leaves the cell serving; the key database is the one component the cell cannot lose, so it is the one you make highly available by other means.

The shape, and why one address

• Members. Each member is a key-server instance on its own host, with its own instance id. All members run the same encrypted configuration and read and write the same key database. Place them in at least two zones of one region.
• The key database. One PostgreSQL database, highly available in its own right (for example a managed instance with a standby in a second zone), reached by every member over verify-full.
• One address. The Console, the REST Gateway and the C and C++ SDKs each connect to exactly one key server address. Put a TCP load balancer in front of the members and give clients its address; the member list is yours, never a client's. The remote engine is the exception: it takes a list of key servers of its own.
• One anchor. Each member presents a self-signed root. A member that joined another carries that member's configuration, so the members of one cell present the same root; build the anchor a client trusts (KS_SERVER_ANCHOR for the Console and the Gateway) from every member all the same, in one PEM file, so a member installed separately is covered too. A member answers with the same name whichever address it is dialled by, so the load balancer's address needs no certificate of its own.

Choose the member count before the first install

The number of instance ids a key database serves is fixed when the database is installed. Every persistent object a member creates is counted per instance id, those counters are written only by the install, and a member started with an id that has no counter rows refuses to start. So decide the count first:

• --instance-total N at the first install provides instance ids 0 to 2^k - 1, where k is the number of bits in N - 1. --instance-total 3 and --instance-total 4 both provide ids 0, 1, 2 and 3.
• Install for one more member than you plan to run. The spare id is what you add a member with later, or join a replacement with while the old one is still being taken apart.

Install the first member (instance id 0) against the key database. Every member listens on 0.0.0.0 and on the same port, because a passthrough load balancer delivers each connection to the members unchanged, addressed to the load balancer's own address:

sudo tricryption-ks-provision --instance main --instance-total 3 --instance-id 0 --xauth-policy \
  --bind-host 0.0.0.0 --port 8893 --db tecloud --db-role teks_db \
  --db-host 192.0.2.10 --db-password-file /opt/te/managed-db/db-password \
  --db-sslmode verify-full --db-sslrootcert /opt/te/managed-db/server-ca.pem \
  --db-servername db.example.com

Add a member

A new member joins a running one. It copies the source member's protection material and encrypted configuration, relocates its own listener, backs that configuration up into the database under its own id, and starts. It creates nothing in the database and changes no existing row.

The join material is the source's encrypted configuration, its master, administrator and database passwords, its system id and its instance record: ks-config _MASTERPASSWORD_ _SYSTEM_ _ADMINPASSWORD_ _DBPASSWORD_ ks-instance.txt in /var/lib/pisoft/ks/main. Move it from host to host without writing it anywhere in between: stream it from the source over one SSH session into the next, into a directory only root can read, and remove it after the join:

ssh ks-a.example.com 'sudo tar -C /var/lib/pisoft/ks/main -cf - ks-config _MASTERPASSWORD_ _SYSTEM_ _ADMINPASSWORD_ _DBPASSWORD_ ks-instance.txt' \
  | ssh ks-b.example.com 'sudo sh -c "umask 077; rm -rf /root/join-src; mkdir -p /root/join-src && tar -C /root/join-src -xf -"'

Compare the configuration's digest on both sides, then join on the new host. Install the tricryption-ks package there first, and place the key database's CA file at the same path the source uses:

ssh ks-a.example.com 'sudo sha256sum /var/lib/pisoft/ks/main/ks-config'
ssh ks-b.example.com 'sudo sha256sum /root/join-src/ks-config'
ssh ks-b.example.com 'sudo tricryption-ks-provision --check-join --join-from /root/join-src --instance-id 1 --bind-host 0.0.0.0'
ssh ks-b.example.com 'sudo tricryption-ks-provision --join-from /root/join-src --instance-id 1 --instance main --bind-host 0.0.0.0 --port 8893; sudo rm -rf /root/join-src'

--check-join reads the source's record and this member's command line exactly as the join does, prints the member it would make, and writes nothing. The provisioner refuses an id the database has no rows for, the source's own id, and, for a source instance on the same host, that source's own port and address. The new member takes the source's database settings from its record; do not pass --db-* options with --join-from.

Add the new member to the load balancer once it listens.

Remove a member

1. Take it out of the load balancer first, and wait until the load balancer no longer lists it as a backend. Connections already open on it finish; new ones go elsewhere.
2. Stop its unit: sudo systemctl stop tricryption-ks@main.
3. Move its state aside, outside /var/lib/pisoft/ks and /etc/pisoft/ks. Purging the package deletes both trees, and a removed member's state is evidence you may want later.
4. Purge the package.

The database is not touched: the other members keep serving every key the removed one made. Measured with a client making a fresh request about once a second: taking the member out of the load balancer first (the backend list no longer named it 14 seconds later) and only then stopping it failed none of 95 requests.

Replace a member

A member whose host is lost is replaced by joining, not restored:

1. Make sure the old host is stopped or deleted, and that it is out of the load balancer.
2. Bring up a new host in the same zone and join it from a live member, with the lost member's instance id (a join refuses a source whose id is the new member's own).
3. Add the new host to the load balancer.

The replacement keeps the lost member's instance id and the cell's system id. Measured with a member's host deleted outright and a client making a fresh request about once a second: none of 489 requests failed while the load balancer took the deleted host out, the longest time between two successful requests was 8.3 seconds, and the replacement was in rotation 13 seconds after its join.

This procedure was qualified on a cell of three members installed with --instance-total 3 (ids 0 to 3), adding a fourth member on the spare id and removing it again.

The load balancer's requirements

• TCP passthrough. It must not terminate TLS: each member's TLS and certificate authentication run end to end between the client and the member.
• Connection-level balancing. One connection stays on one member for its whole life; a client's next connection may land on another. Nothing a client does depends on reaching the same member twice.
• A TCP health check on the members' port. A member that stops listening is taken out of rotation; one that comes back is put back.
• Health-check sources admitted by the members' firewall, as well as the clients' networks.

On Google Cloud this is an internal passthrough Network Load Balancer: one unmanaged instance group per zone, a regional TCP health check, a regional internal TCP backend service, a reserved internal address and a forwarding rule on the members' port. See Installing on Google Cloud.

Provision the Console and the Gateway against the load balancer's address, with an anchor from every member and administration against one member directly:

sudo tricryption-tier-provision --ks-host 192.0.2.50 --ks-port 8893 \
  --anchor-endpoint 192.0.2.21:8893 --anchor-endpoint 192.0.2.22:8893 --anchor-endpoint 192.0.2.23:8893 \
  --admin-endpoint 192.0.2.21:8893 --admin-password-file /root/ks-admin \
  --keydb-host 192.0.2.10 --keydb-name tecloud --keydb-user teks_db \
  --keydb-password-file /opt/te/managed-db/db-password --keydb-sslmode verify-full \
  --keydb-sslrootcert /opt/te/managed-db/server-ca.pem --keydb-servername db.example.com \
  --system-id 20260912 --xauth-policy

--system-id is the key database's system id (in any member's ks-instance.txt) and --xauth-policy says the members were provisioned with it. The tool writes the delegation policy into the database and prints RESTART-REQUIRED:: each member reads that policy when it starts, so restart every member in turn, waiting for the load balancer to report it healthy before you restart the next.

Upgrading a cell

Upgrade one member at a time, in the order of Upgrading:

1. Take the member out of the load balancer and wait until the load balancer no longer lists it.
2. Install the new tricryption-ks package over the old one. The package restarts the member's unit; wait for it to listen.
3. Put it back and wait until the load balancer reports it healthy. Go on to the next member.

Upgrade the tier last. At code-signing level 2 the Gateway and the Console start only with a registered release, so register the new release before you install it, from the old tier, with the new Console bundle's signed BUNDLE-MANIFEST.txt, .sig and .cert.pem in a directory on the tier host:

sudo tricryption-tier-provision --register-release --release-dir ./next-release \
  --ks-host 192.0.2.50 --ks-port 8893 \
  --anchor-endpoint 192.0.2.21:8893 --anchor-endpoint 192.0.2.22:8893 --anchor-endpoint 192.0.2.23:8893 \
  --admin-endpoint 192.0.2.21:8893 --admin-password-file /root/ks-admin \
  --keydb-host 192.0.2.10 --keydb-name tecloud --keydb-user teks_db \
  --keydb-password-file /opt/te/managed-db/db-password --keydb-sslmode verify-full \
  --keydb-sslrootcert /opt/te/managed-db/server-ca.pem --keydb-servername db.example.com \
  --signer-cert ./signer.pem --signer-ca ./cacert.pem --signer-crl ./ca.crl.der

--release-dir is available when the installed tier's tricryption-tier-provision --help lists it. With a tier that does not have it yet, install the new tier packages first and then run --register-release without --release-dir: the Gateway answers 503 until the registration completes.

A release signed by a different signer. --signer-cert, --signer-ca and --signer-crl name the signer of the release you register, which need not be the signer of the release you run: the registration adds that signer's CA, its CRL and the publisher beside the ones already trusted, and both releases are admitted afterwards. Until it is registered, a client that attests the new release is refused with the signer certificate does not chain to a trusted CA.

So the order, at code-signing level 2:

1. Register the new release from the old tier, with the new release's signer files and --release-dir.
2. Roll the members one at a time, as above.
3. Install the new tier packages last.

Measured on a three-member cell at code-signing level 2, upgrading from an earlier release signed by another signer, with a client making a fresh request about once a second: no request failed while the three members were rolled in turn (0 of 117, 102 and 124 at the three steps; the longest time between two successful requests was 3.3 seconds), and none failed while the tier was upgraded (0 of 32). Each member was out of the load balancer for about two minutes. A key encrypted before the upgrade decrypted after every step, whichever member served it. Registered ahead, the upgraded tier was admitted at level 2 as it started; the Gateway answered 503 for 1.4 seconds while it restarted. A second roll, to the next build signed by the same signer, had the same shape: no request failed at any step, and the longest gap was 2.2 seconds.

Measured again on 2026-10-27, rolling a three-member cell at code-signing level 2 to 8.1.0+3578 (which refuses an expired key, below), signed by another signer: no request failed while the three members were rolled in turn (0 of 101, 106 and 101 at the three steps; the longest time between two successful requests was 3.4 seconds), and none failed while the tier was upgraded (0 of 33). Each member was out of the load balancer for about two minutes. A key encrypted before the upgrade decrypted after every step. The upgraded tier was admitted at level 2 as it started, and the Gateway answered 503 to one readiness probe while it restarted. Upgrade the members before the tier, as above: a member that has the new release serves an older tier unchanged.

Measured again on 2026-10-29, rolling a three-member cell at code-signing level 2 from 8.1.0+3578 to 8.1.0+3647 (which ships Node.js 24.22.2 and answers a ciphertext that does not decrypt with 422 DECRYPT_FAILED, below), signed by another signer: no request failed while the three members were rolled in turn (0 of 96, 103 and 96 at the three steps; the longest time between two successful requests was 4.3 seconds), and none failed while the tier was upgraded (0 of 38). A key encrypted before the upgrade decrypted after every step. The upgraded tier was admitted at level 2 as it started.

Reading the audit across members

Every member writes its audit rows to the one key database, so the Console's audit view, and its GET /api/audit route, list the rows of every member together. Each member numbers its rows in a range of its own, so the view does not page by one highest row number. The first page of a selection answers a ceiling: the highest row of each member at that moment. Every later page of that selection is read under it, so rows the members write while you page do not push rows from one page onto the next or into it. A client of GET /api/audit pages the same way: send the first page's ceiling back, unchanged, with every later offset.

Measured on a three-member cell, with every member writing between two pages: the view's first page carried one ceiling per member, and the pages after it listed every row the selection held at the first page, from all three members, each once (69 rows over three pages), and none of the 75 rows written while it paged. The same walk without the ceiling listed rows twice, missed others and never reached its last page.

When something fails

A member stops. The load balancer's health check takes it out of rotation; the next connection is served by another member. A request in flight on the member as it died either completed or is answered with a retryable error; it never hangs. Revocation crosses members through the key database: a key's access withdrawn on one member is refused on every member at the next request, and so is a session whose certificate a CRL imported through one member revokes, whichever member serves its next request. (Releases before 8.1.0+3538 could serve that session's next request once more on a member it had just used; upgrade a cell that relies on CRL revocation.) Measured on a three-member cell, with a client opening a fresh connection to the load balancer's address about once a second: with the health check probing every 5 seconds and taking a member out after 2 failed probes, the killed member was out of rotation 14 seconds after the kill; 5 of 92 requests in that window failed to connect, each answered at once with a connection error, and the longest time between two successful requests was 5.4 seconds. The member, started again, was back in rotation 14 seconds later. Retry a request that fails to connect: the next connection is balanced again.

A key is expired or removed. An expiry or a removal through any member is refused on every member at the key's next use, whichever member serves it, even a member that has just used the key: an expired key answers 410 KEY_EXPIRED and its record stays readable, a removed key answers 404 KEY_NOT_FOUND. Measured on a three-member cell running 8.1.0+3578, each member expiring a key once and using a key another member expired once: the next decrypt on a member that had just decrypted the key was refused at its first attempt, three times out of three, and a removal the same way. (Releases before 8.1.0+3578 recorded an expiry but went on serving the key, on every member.) Measured again on 8.1.0+3647, three times out of three.

A ciphertext does not decrypt. A ciphertext encrypted under another key, damaged or cut short answers 422 DECRYPT_FAILED on every member, and the session that sent it goes on working. Measured on a three-member cell running 8.1.0+3647: through each member, a ciphertext cut by one byte and one with its last byte changed were each answered 422 DECRYPT_FAILED, the member's own audit row recorded the failure as Decrypt failed, and the same session then decrypted the intact ciphertext. (Releases before 8.1.0+3647 answered 403 ACCESS_DENIED.)

A zone is lost. The members in the other zone keep serving through the same address. Measured with the zone holding one of three members stopped: 3 of 250 requests failed to connect while the load balancer took that member out, the longest time between two successful requests was 11.5 seconds, and the member was back in rotation 14 seconds after its host started.

The key database fails over to its standby. The members detect the dropped connections and reconnect on their own; no unit restarts, and the database's address does not change. On a managed regional instance the provider reported the standby primary 92 seconds after the failover was issued; the cell served again about 70 seconds after the failover was issued (requests made in between waited, and 2 of 303 failed), and the longest time between two successful requests was 32 seconds. A member in the primary's original zone then reached the database in the other zone and served. Fail the database back to its primary zone afterwards: the failback was the same shape (2 of 137 failed, 31.9 seconds), and work measured with the database in another zone from the members is not comparable.

The key database pauses or restarts. Members recover without a restart once the database answers again: a member gives back a connection whose transaction could not begin, and a key generator that fails keeps retrying. If a member stays slow after the database is back, check its journal for database errors and restart that member.

A member stops doing work while the database is idle. A member waiting on its own database connection pool, with every backend idle in the database, is a pool held by work that needs a second connection. Current releases authorize before they open a transaction, so this no longer wedges a member; if you see it, the member's pool is smaller than the number of threads that can each hold one connection, and raising the pool at install time is the fix.

Audit records appear in the journal. When an audit insert fails, the audit writer retries the batch, then the rows one by one, and journals only a row that still fails, with a line beginning AuditLogJournal: and a summary AuditLog: <n> record(s) written to the journal (stdout) instead of the table. The next batch tries the table again. Watch for those lines: they mean audit rows are reaching the journal rather than the database.

Recovering the whole cell

A member is replaced by joining. The cell is recovered from the key database: it is the single point every member depends on. Recovering a key server is the procedure; on a cell with a managed key database it runs as follows.

The recovery set. Take it while the cell serves: the dump's start is the cut, and the dump is one consistent snapshot of the database. From a member, as the database's role, over verify-full:

sudo sh -c 'umask 077; PGSSLMODE=verify-full PGSSLROOTCERT=/opt/te/managed-db/server-ca.pem PGHOSTADDR=192.0.2.10 pg_dump -Fc --no-owner --no-privileges -h db.example.com -U teks_db -d tecloud > /root/keydb.dump'

Then the first member's state root, without its master password, and the recovery password kept apart from both:

sudo tar --numeric-owner -C / --exclude=var/lib/pisoft/ks/main/_MASTERPASSWORD_ -cf /root/ks-main-state.tar var/lib/pisoft/ks/main etc/pisoft/ks/main.env
sudo cat /root/.tricryption/ks-main.recovery

When the database and every member are lost:

1. Create a new database instance, highly available like the old one. A managed service may not give a deleted instance's name back for days, so it has a new name, a new private address and a new certificate name. Restore the dump into its empty database as the role, with pg_restore --no-owner --no-privileges --exit-on-error.

2. On a new host for the first member, install the same release and lay the state root back. In the plaintext ks-config.xml, put the new instance's address and certificate name in both connections, and recover the master password with the recovery password (in a mode-600 secrets file with RECOVERY=, the new role password as DBPASSWORD=, and a placeholder MASTERPASSWORD=):

   sudo ./ks-setup -RECOVERY -CONFIGFILE=ks-config.recovery -SECRETS=ks-recovery.secrets -PROTECTOR=PasswordProtector -DBUSER=teks_db -IC=3 -ID=0 -MASTERPASSWORDFILE=_MASTERPASSWORD_

   Then repair the recovered configuration to the new instance (the secrets file now carries MASTERPASSWORD=, REPAIRDBPASSWORD= and REPAIRDBPASSWORD2=), and start the member:

   sudo ./ks-setup -CONFIGFILE=ks-config -SECRETS=ks-repair.secrets -PROTECTOR=PasswordProtector -ID=0 -REPAIR -REPAIRDBSERVER=192.0.2.11 -REPAIRDBSSLSERVERNAME=db2.example.com -REPAIRDBSERVER2=192.0.2.11 -REPAIRDBSSLSERVERNAME2=db2.example.com
   sudo systemctl restart tricryption-ks@main

3. Keep client access closed: stop and mask the Gateway and the Console. Join every other member from the first one again.

4. Reconcile. Everything after the cut is not in the set. A revocation made after the cut is served again by the recovered cell until it is replayed: replay every change that took access away, directly to the key server, and read each back.

5. Unmask the Gateway and the Console and provision the tier against the new database.

Measured on a three-member cell whose instance and every member were deleted 11.6 seconds after the cut, with one revocation made inside that window: the recovered cell presented the same root and system id, decrypted every ciphertext made before the cut byte for byte, served the revocation made after the cut until it was replayed and refused it after, and refused the one made before the cut throughout. From the first restore command to the first verified decrypt took 49 minutes; without the two interruptions of that run, its parts took 11 minutes to create the instance, 9 minutes to restore the first member, 9 minutes to join the other two, 5 minutes to reconcile and 7 minutes to reopen client access.

What was qualified

On 2026-10-20 every procedure on this page ran on Google Cloud, with a client making a fresh request through the load balancer's address about once a second throughout: three members on Compute Engine e2-standard-4 (Ubuntu 24.04) in two zones of one region, a Cloud SQL for PostgreSQL 16 db-custom-2-8192 regional instance (primary in the first zone, standby in the second) reached over verify-full on a private address only, one internal passthrough Network Load Balancer, and the Console and the Gateway on a host of their own. Release 8.1.0 was installed and upgraded member by member to the next build at code-signing level 2.

On 2026-10-26 the same shape (image ubuntu-2404-noble-amd64-v20261024, Cloud SQL 16.15) was installed from the release qualified on 2026-10-20 (8.1.0+3328) and rolled without being taken down to 8.1.0+3507, signed by another signer, and on to 8.1.0+3538, at code-signing level 2 with the same client running: the upgrade, the audit view across members, the encrypt size limit and the refused request's audit row, certificate revocation across members, and a member's loss were measured on it, and every check passed on 8.1.0+3538. A zone's loss, the database failover, adding, removing and replacing a member, and recovering the whole cell are the 2026-10-20 measurements.

On 2026-10-27 the same shape was installed from 8.1.0+3507 and rolled without being taken down to 8.1.0+3578, signed by another signer, at code-signing level 2 with the same client running. Before the roll an expired key still decrypted on another member; after it, the upgrade, a key's expiry and its removal across members, certificate revocation across members, the Console's first change of the code-signing level after a restart, and a member's loss were measured, and every check passed.

On 2026-10-29 the same shape was installed from 8.1.0+3578 and rolled without being taken down to 8.1.0+3647, which ships Node.js 24.22.2, signed by another signer, at code-signing level 2 with the same client running. Before the roll a ciphertext that does not decrypt was answered 403; after it, the upgrade, a key's expiry across members, a failed decrypt through every member, certificate revocation across members, the Console's first change of the code-signing level after a restart, and a member's loss were measured, and every check passed.

Qualified at engine master afc556ab71cf.