Recovering a key server

A key server, its Console and Gateway, and its local PostgreSQL can be lost along with their host. You can bring them back on a replacement host as the same logical system. The system id and the instance id stay the same. Every ciphertext encrypted before the backup decrypts byte for byte. Every revocation made before the backup is still refused. Client access stays closed until every revocation made after the backup has been applied again.

This procedure was qualified on the Ubuntu packages, with the paths on Linux (Ubuntu). It uses only the product's own mechanisms: pg_dump and pg_restore of the one key database, the packages' state roots, and the key server's own recovery path. It is not a backup service, and nothing in it runs on its own schedule.

What to back up

A recovery point is these items, all taken at one cut:

Item	Where	How
the key database	PostgreSQL	`pg_dump -Fc` of the one database
its database role, with its password verifier	PostgreSQL	that one role's lines from `pg_dumpall --roles-only`
the key-server instance	`/var/lib/pisoft/ks/<instance>/` (configuration, `_MASTERPASSWORD_`, `_SYSTEM_`, `_ADMINPASSWORD_`, `_DBPASSWORD_`, `_DBUSER_`, `ks-instance.txt`, `ks-config.xml`) and `/etc/pisoft/ks/<instance>.env`	`tar`, keeping numeric owners and modes
the Console and Gateway tier's identity	`/etc/pisoft/console-gateway/` (`console.env`, `gateway.env`) and `/var/lib/pisoft/console-gateway/` (`ks-anchor.pem`, `secrets/`, `pki/`, `registered-release/`)	`tar`, keeping numeric owners and modes

Take the point in this order:

Stop the Gateway and the Console, then the key server.
Dump the database and the role. The cut is the moment the dump starts. Record it in UTC.
Archive the state roots.
Start the key server, then the tier, and check that both readiness routes answer 200.

Leave out three things. The packages under /opt/pisoft/tricryption are reproduced by installing the same release. The logs and the PostgreSQL cluster files are not needed, because the dump is the logical copy. The recovery password is held separately (next section).

The point holds every password the instance has. Encrypt it, keep its key apart from it, and check after sealing that every item decrypts and hashes to what you stored.

The recovery password, held apart

When you provision the key server you can give it a recovery password (tricryption-ks-provision --recovery-password-file <file>). On the host that file lives at /root/.tricryption/ks-<instance>.recovery (root, mode 600). The key server seals a recovery envelope of the master password under it, inside the key database.

Keep a copy of the recovery password in your own custody. Never keep it in the recovery point or beside the instance's configuration.
After you change the master password, re-seal the envelope. You change the master password with ks-setup -UPDATE=secret. Only an update whose secrets file also carries RECOVERY= with the recovery password seals a new envelope under the new master. Without it, ks-setup prints WARNING: the master password changed and the recovery envelope was NOT re-sealed. A later recovery would then open the old master password and fail.
With a re-sealed envelope you may leave the master password out of the recovery point entirely, and recover it on the replacement instead.

Restoring onto a replacement host

The replacement host carries the same release as the one that was lost. That means the same four .deb packages and PostgreSQL at the same version. It is not provisioned: a replacement host receives state and never initializes it. Do not run tricryption-ks-provision on it. Do not create a new system id or a new key database, and do not weaken TLS for the new address. The tier's trust anchor is the key server's own root, and it is restored with it.

Install the release's packages. See Install the packages.
Restore the role and the database. Create the role from its saved lines, create the empty database owned by it, and pg_restore --exit-on-error the dump into it.
Lay back the state roots, keeping numeric owners and modes. If the tricryption service user has a different uid or gid on this host, change the restored files' owners to it.
Keep client access closed. Before anything starts, mask the Gateway and the Console:
```
sudo systemctl mask tricryption-gateway.service tricryption-console.service
```
If the master password is lost too, recover it from the envelope with the recovery password from custody. Run this in the restored instance directory (/var/lib/pisoft/ks/<instance>/), which holds ks-setup as a link into the installed payload — the link must resolve, or the laid-back state does not match the installed release:
```
sudo ./ks-setup -RECOVERY -CONFIGFILE=<copy of ks-config.xml> -SECRETS=<secrets file> -PROTECTOR=PasswordProtector -DBUSER=<role> -IC=<instance_total> -ID=<instance_id> -MASTERPASSWORDFILE=_MASTERPASSWORD_
```
The secrets file is mode 600 and carries RECOVERY= (the recovery password) and DBPASSWORD= (from _DBPASSWORD_). instance_total and instance_id are in ks-instance.txt. ks-setup -RECOVERY opens the newest envelope in the key database and writes the master password to _MASTERPASSWORD_ (mode 600). It also re-encrypts the configuration into the copy you named. Move that copy over ks-config, give both files to the tricryption user with mode 600, and delete the secrets file. It mints a new Product uuid. The system id and the instance id are the identity, and they do not change.
Start the key server (sudo systemctl enable --now tricryption-ks@<instance>.service). Check that the installed ks-server is what listens on its port.

The reconciliation gate

The recovery point does not hold anything that happened after the cut. A grant revoked after the cut is served again by the restored key server. This was measured, not assumed. Keep the Gateway and the Console masked until every change made after the cut that took access away has been applied again:

Read the lost host's audit since the cut if you still have it. Otherwise use your own record of changes. List every successful change that reduced access: ACL entries removed or updated, principals removed or changed, roles removed or changed, keys expired, CRLs imported, trusted CAs or servers removed, code-signing entries removed.
Apply each one to the replacement's key server directly, on its loopback listener. Confirm each with a fresh read.

Only when every listed change is confirmed, unmask and start the tier:

sudo systemctl unmask tricryption-gateway.service tricryption-console.service
sudo systemctl start tricryption-gateway.service tricryption-console.service

Proving the recovery

Use fresh Gateway sessions on the replacement:

every ciphertext you hold from before the cut decrypts byte for byte (compare its hash with the plaintext's);
each recipient is served the keys it holds a grant on, and is refused the others, including every grant revoked before the cut and every grant you revoked again at the gate;
GET /api/v1/server/system-id answers the lost host's system id;
the key server presents the lost host's own root certificate.

Rehearsing it

The procedure on this page was qualified with a rehearsal kit: a driver for every step above, an archive sealer and verifier, a versioned recovery manifest, and the measured recovery point and recovery time objectives. Pi Soft support provides the kit and its runbook on request. Rehearse on hosts you can lose before you need this page for real.

Qualified at engine master ac4545d27c95.

What to back up​

The recovery password, held apart​

Restoring onto a replacement host​

The reconciliation gate​

Proving the recovery​

Rehearsing it​