Backup and Recovery

A CoderFlow server keeps its state in three locations, plus a few files around them. A usable backup holds all of them, copied together while the server is stopped, with their owners and permissions.

Location	Where	What it holds
Server data directory	SERVER_DATA_PATH, by default ~/.coder/data in the home directory of the account that runs CoderFlow	Users, roles, settings, integration and AI provider credentials, API keys, the server's encryption keys and their markers, and, unless moved, task storage
Setup directory	The directory set with coder-server config set coder_setup_path	Environments, templates and the setup repository. Each environment's .secrets.json and .connection-secrets/ folder are kept out of Git by the repository's .gitignore, so the repository's remote does not have them.
Task storage	tasks in the data directory, unless TASK_STORAGE_PATH_OVERRIDE or task_storage in setup.json puts it elsewhere	Task records, output, and each task's encrypted masking history (runtime/held-secret-values.enc)

Also keep:

• the server account's ~/.coder/config.json (the settings made with coder-server config set);
• any file on the host that a File on host environment secret, or client_secret_file in oidc.json, points to outside these locations.

Docker images and containers are not part of a backup; images are built again from the setup directory.

Encrypted credentials and their keys

Stored credentials are encrypted with keys kept in the data directory. Next to each key is a marker file, named after the key with .encrypted added, that records which key the stored data belongs to. A key is only useful together with its marker and the files it protects, so always back them up and restore them together.

Key (with its .encrypted marker)	Protects
.user-secrets-key	Environment secret values (each environment's .secrets.json); personal secrets (user-secrets.json), personal IBM i passwords (personal-ibmi-credentials.json) and Portal launch trust (portal-launch-trust.json); MFA authenticator secrets in users.json; the assistant telemetry endpoint and signing secret (assistant-telemetry.json); Git provider secrets (.git-provider-secrets/); AI provider accounts (provider-accounts.json, user-provider-accounts.json); the Slack, Microsoft Teams, Jira and SMTP credentials (slack.json, microsoft-teams.json, jira.json, smtp-config.json); the SSO client secret in oidc.json; and every task's masking history
.git-oauth-key	Users' Git connections, including their refresh tokens (user-git-tokens.json)
.external-connections-key	Connection passwords and SSH private keys (each environment's .connection-secrets/)

Only the secret values are encrypted. The rest of each file stays readable JSON, so settings such as allow_local_auth in oidc.json can still be edited by hand. A secret typed into one of these files by hand is accepted, and is encrypted the next time it is saved or when the server next starts. Settings pages show whether a secret is saved, never its value.

API keys (api-keys.json) and password setup and reset links (password-tokens.json) are stored as one-way hashes instead. No key can turn a hash back into the original key or link.

CoderFlow never replaces a key while data encrypted with it exists. When a key is missing, damaged or not the one the data was encrypted with, CoderFlow refuses to read or save the credentials that depend on it and changes nothing on disk. See Lost server key.

Back up before upgrading

An upgrade that changes how credentials are stored converts them the first time the new version starts, and earlier versions cannot read the converted files. The backup taken just before the upgrade is the way back.

1. Stop the server: sudo systemctl stop coderflow.service under systemd, or coder-server stop.
2. Copy the data directory, the setup directory and every task storage location, including hidden files: the key files and their .encrypted markers, .secret-quarantine/ if present, and each environment's .secrets.json and .connection-secrets/. Add ~/.coder/config.json and any host files that secrets point to. See Backup commands for a recipe that also handles folders that are symbolic links.
3. Keep the backup private. It contains every stored credential with the keys that open them, and a backup taken before the upgrade also holds the credentials earlier versions stored unencrypted. Protect it like the passwords themselves.
4. Install the new version and start the server.

Backup commands

Any of these locations, or a folder inside them such as a month or day of task storage, can be a symbolic link to another disk, and an archive of a link holds only the link. The commands below archive the real folders, plus every folder reached through links inside them, including links inside those folders, each once. The backup then restores the same layout. Run them as root, with the server stopped, after setting the first five lines for your installation:

HOME_DIR=/home/coderflow                          # home of the account that runs CoderFlow
DATA=$(readlink -f "$HOME_DIR/.coder/data")       # or your SERVER_DATA_PATH
SETUP=$(readlink -f "$HOME_DIR/mycompany-coder-setup")
TASKS=$(readlink -f "$DATA/tasks")                # or your TASK_STORAGE_PATH_OVERRIDE or task_storage location
BACKUP=/secure/backups/coderflow-$(date +%F).tar.gz

ROOTS="$DATA $SETUP"
case "$TASKS/" in ("$DATA"/*) ;; (*) ROOTS="$ROOTS $TASKS" ;; esac
# Folders reached through links, followed again inside them; each is kept once.
FOLDERS=$ROOTS SCAN=$ROOTS
while [ -n "$SCAN" ]; do
  SCAN=$(find $SCAN -type l -xtype d -exec readlink -f {} \; | sort -u | while read -r dir; do
    for kept in $FOLDERS; do case "$dir/" in ("$kept"/*) continue 2 ;; esac; done
    echo "$dir"
  done)
  FOLDERS="$FOLDERS $SCAN"
done
# Leave out folders that are inside another archived folder.
FOLDERS=$(for dir in $FOLDERS; do
  for other in $FOLDERS; do [ "$other" = "$dir" ] || case "$dir/" in ("$other"/*) continue 2 ;; esac; done
  echo "$dir"
done | sort -u)
tar --create --gzip --preserve-permissions --file "$BACKUP" "$HOME_DIR/.coder/config.json" $FOLDERS

Add any host files that secrets point to. The commands assume paths without spaces.

Restore

1. Stop the server.

2. Move the current data directory, setup directory and task storage aside, keeping them until the restored server works, and put all of them back from one backup with their owners and permissions. With an archive made by the backup commands, run as root the same lines up to the tar command, with BACKUP set to the archive, then:

   for dir in $FOLDERS; do mv "$dir" "$dir.before-restore"; done
   tar --extract --gzip --preserve-permissions --same-owner --file "$BACKUP" --directory /

   The archive holds the real folders, so links that point to them, such as ~/.coder/data, work again unchanged.

3. Start the server.

Never combine a key from one backup with data from another, or a backup's data with the current key. Values encrypted with another key cannot be read, and CoderFlow does not replace the key. Everything changed after the backup was taken is lost.

Rolling back an upgrade

There is no downgrade command. To go back to the version you upgraded from:

1. Stop the server.

2. Install the earlier version without starting it, for example:

   npm install -g --@profoundlogic:registry=https://npm.profoundlogic.com/ @profoundlogic/coderflow-server@<earlier-version>

3. Restore the backup taken before the upgrade. Do not put back that backup's assistant-actions directory: handle Ask's record as described in Rolling Back to an Earlier Version.

4. Start the server.

Do not start the newer version on the restored files: it would convert them again. If you install the earlier version from Server Administration instead (see Server Updates), choose Later when asked to restart, then stop the server and restore the backup before starting it.

Everything changed after the backup is lost: credentials entered or changed since are back to their earlier values, API keys created since must be issued again, and password setup or reset links sent since must be sent again.

Moving a server

1. On the old host, stop the server and take a backup as described above.
2. On the new host, install the same CoderFlow version as described in Installation, with the same service account, and stop it.
3. Restore the backup with the service account as owner. An archive made by the backup commands extracts at / to the same real paths as on the old host; recreate any symbolic links you used there, such as ~/.coder/data. Or point the new server at the new paths: SERVER_DATA_PATH, coder-server config set coder_setup_path, and TASK_STORAGE_PATH_OVERRIDE or task_storage in setup.json.
4. Put the host files that File on host secrets or client_secret_file use at the same paths.
5. Start the server, then build the base image and the environment images on the new host (see Installation). If the server's address changed, update site_url and the callback URLs your integrations use.

Never run the old and the new server on the same data at the same time. Keep the old host stopped until the new one works.

An environment export is not a way to move secrets: exports never contain them.

Lost server key

When a key file is missing, unreadable, truncated, or is not the key the data was encrypted with:

• The server keeps running. Everything that does not need that key works as usual.
• Reading or saving a credential protected by that key is refused with Credential encryption authority unavailable. Restore the matching server key and store snapshot from backup. The server log names the affected store (credential_encryption_authority_unavailable). Nothing on disk is changed.
• CoderFlow does not create a replacement key while data encrypted with the lost one exists. Do not create a key file or marker yourself, and do not delete the marker: a different key leaves every value it should protect unreadable.

While .user-secrets-key is unavailable, SSO sign-in also fails, because the client secret in oidc.json cannot be opened (a secret kept in client_secret_file is not affected). Sign in with a password meanwhile; on a server that allows only SSO, follow Recovering Access.

To recover:

1. Stop the server.
2. Put back the key file and its .encrypted marker from the backup the stored data came from, normally the most recent one. If the protected files have changed since that backup, restore everything from the same backup instead (see Restore).
3. Start the server.

Entering the credentials again does not help while the key is missing or wrong: those saves are refused too. Re-entering helps only when one stored value is damaged; see A damaged credential.

A damaged credential

When the key is fine but one stored value no longer opens (for example after disk damage or a hand edit), only that value is affected:

• Slack, Teams, Jira, SMTP and SSO settings. Slack and Teams run as not configured, Jira import is unavailable, email is not sent, and SSO sign-in is unavailable. CoderFlow does not switch to the JIRA_* or SMTP_* environment variables instead. The server log names the store (secret_entry_unavailable). Enter the secret again on its settings page and save. A save that leaves the secret blank, which normally keeps the saved value, is refused while that value cannot be opened, so the damaged value is never carried over.
• AI provider accounts. Reads of the whole account store (provider-accounts.json for server accounts, user-provider-accounts.json for personal ones) are refused, because task output masking relies on every account. The message is A stored AI provider account credential cannot be read, and the server log names the provider and account ID (provider_credential_unavailable), never the value. Restore that file from your most recent backup, or stop the server, copy the file to a private folder, remove that one account's entry from the provider's list in the file, start the server and connect the account again.
• Personal secrets, Git connections and connection credentials. Enter the value again where it was saved.

If no copy of the key exists

The values that key protected cannot be recovered. Starting over means taking every file encrypted with the lost key out of service, so that CoderFlow can create a new key. Moving only the key or the marker aside does not work: CoderFlow still finds the encrypted data and keeps refusing. Move files into a private folder rather than deleting them: if the key turns up, putting them back with it makes the values readable again.

Each key's quarantine archive, .secret-quarantine/<key>.json, also counts as data encrypted with that key: move it aside with the other files. Keep it with them, since it holds the only copy of values quarantined earlier.

Git connections (.git-oauth-key). Stop the server, move user-git-tokens.json, .git-oauth-key, .git-oauth-key.encrypted and, if present, .secret-quarantine/.git-oauth-key.json aside, and start the server. Each user connects their Git account again. Nothing else uses this key.

Connection credentials (.external-connections-key). Stop the server. Move .external-connections-key, .external-connections-key.encrypted, .secret-quarantine/.external-connections-key.json if present, and the *.json credential files in every environment's .connection-secrets/ folder aside; leave its secret-files/ folder in place. Start the server and enter each connection's password or SSH private key again in the connection editor. The connections' other settings are kept.

Everything else (.user-secrets-key).

1. Stop the server.
2. In the data directory, move aside .user-secrets-key, .user-secrets-key.encrypted, .secret-quarantine/.user-secrets-key.json if present, user-secrets.json, personal-ibmi-credentials.json, portal-launch-trust.json, assistant-telemetry.json, .git-provider-secrets/, provider-accounts.json, user-provider-accounts.json, slack.json, microsoft-teams.json, jira.json, smtp-config.json and oidc.json. Without oidc.json, SSO is off and password sign-in works until you set SSO up again.
3. Move aside every environment's .secrets.json in the setup directory.
4. In task storage, move aside every task's runtime/held-secret-values.enc and any held-secret-values.enc.unavailable next to it. Those tasks keep their output, but their view-only shared pages stay withheld, because their masking history is gone.
5. Start the server and sign in as a Server Admin. If your own account uses MFA, or MFA is required, sign in with a one-time login link. Reset TOTP for every user enrolled in MFA, as described in Recovering a locked-out user: their authenticator secrets in users.json were encrypted with the lost key, and CoderFlow creates the new key, at the next credential save, only once they are reset.
6. Enter the credentials again: Git provider secrets, AI provider accounts, Slack, Teams, Jira, SMTP, SSO, environment secrets and the assistant telemetry settings, and set up Portal launch trust again. Users enter their personal secrets and IBM i passwords again and enroll in MFA again.

The moved files keep their other settings as readable JSON, such as Slack channel mappings and each environment secret's name and variable, which helps to set them up again.

Quarantined credentials

When CoderFlow first meets a key that has no marker yet, for example on the first start after the upgrade that added the markers, it checks the key against the encrypted values already stored. If the key opens at least one of them, it is kept. Values it cannot open, left over from an earlier key, are moved to a private recovery archive so that everything else keeps working:

• The archive is .secret-quarantine/ in the data directory: one file per key, readable only by the server account. Each entry records the store, the entry and its original encrypted value; no key is stored in it.
• The server log records one warning per affected store with the number of entries (secret_entries_quarantined). Values and encrypted data are never logged, returned by any API or included in exports.
• A quarantined credential reads as not saved. Enter it again: personal secrets, Git connections, connection passwords and SSH keys, AI provider accounts, and the Slack, Teams and SSO settings.
• A quarantined Jira API token or SMTP password is replaced by an "unavailable" placeholder instead. Jira import stays unavailable while the integration is turned on, and email is not sent; CoderFlow does not switch to the JIRA_* or SMTP_* environment variables in their place. The settings page shows the secret as not saved, and a save that leaves it blank is refused. Entering the token or password again restores the integration. To use the environment variables instead, disconnect Jira in Settings → Jira, or remove smtp-config.json from the data directory.
• MFA authenticator secrets, personal IBM i passwords, Portal launch trust, the assistant telemetry settings and Git provider secret files are only copied to the archive and stay where they are, so their own rules keep applying: an enrolled user still needs an authenticator code until an administrator resets their TOTP, a custom telemetry endpoint stays unset rather than becoming the default, and Git provider and Portal trust credentials keep refusing until they are entered again.
• A task whose masking history cannot be opened keeps the history file, gets a held-secret-values.enc.unavailable marker next to it, and its view-only shared pages stay withheld. Other tasks are not affected. Removing the history file does not lift this.
• If the key opens none of the stored values, nothing is moved and every read and save is refused: see Lost server key.

Keep the archive. Back it up with the data directory, never publish it, and never copy it into the setup directory or a task's output. CoderFlow never deletes it, and it has no download or restore command. If the earlier key turns up, the archived values can be recovered offline from a private copy with that key; contact Profound Logic support. Do not replace the server's current key with the earlier one.