#Runbook — backup and restore

An untested backup is not a backup. The restore procedure in this file was run end to end on 14 August 2026: the database was dropped completely and brought back from a backup. The result is in section 4.

Operator-facing text is English, the interface is bilingual. Logs, CLI output and start-up errors are fixed English (NSHIP-118) so that someone searching an error message finds results. The product interface follows each user's own language setting.

#1. How backups are taken

Every night at 02:00 UTC the application runs pg_dump --format=custom and writes the output under /data/backups (notusship-YYYYMMDD-HHMMSS.dump). The default retention is 14 days; older files are deleted automatically.

The job is scheduled with Hangfire. If the process restarts during a backup the job is not lost.

To take one by hand:

docker compose exec app dotnet NotusShip.Api.dll backup-now

This runs the same code as the scheduled job. Always run it before an upgrade.

#Getting backups off the server

V0 writes backups to a local volume. If the whole server is lost, the backup goes with it — which is why syncing the notusship-backups volume outward is the operator's job:

# example: nightly off-box copy (cron)
docker run --rm -v notusship_notusship-backups:/backups:ro -v /mnt/backup:/target \
  alpine sh -c 'cp -a /backups/. /target/'

Direct upload to S3-compatible storage arrives in V1 — S3 storage is scoped to V1, and pulling in an SDK for a single backup feature would have weighed V0 down.

#Attachments are backed up separately

pg_dump only captures the database. Issue attachments live in the notusship-files volume and must be copied as well. Restoring the database without restoring the files turns every attachment into a broken link.

#2. Restore procedure

Takes about 2 minutes. The application is down for that time.

#Step 0 — choose which backup to use

docker compose exec db ls -lh /backups

#Step 1 — stop the application

docker compose stop app

This step cannot be skipped. If you restore while the application is running, the migrations that run at start-up create the tables first, and pg_restore swallows hundreds of conflict errors and leaves you with a half-filled database. This is exactly what happened while this procedure was being written.

#Step 2 — empty the database

docker compose exec db psql -U notusship -d postgres -c 'DROP DATABASE notusship;'
docker compose exec db psql -U notusship -d postgres -c 'CREATE DATABASE notusship OWNER notusship;'

#Step 3 — restore

docker compose exec db pg_restore -U notusship -d notusship \
  --no-owner --no-privileges /backups/notusship-20260814-182013.dump

The command runs from the db container; the backup volume is mounted there read-only. The application container is not needed — it is already stopped.

The output should be empty. If you see an error line, do not continue; see section 3.

#Step 4 — start the application

docker compose start app
docker compose logs app --tail 20

You should see Database is up to date, no migrations to apply. in the log. If it says N migration(s) to apply instead, the backup is older than your version; that is normal, and the migrations are applied on top.

#Step 5 — verify

curl -s localhost:8080/health/ready

Then sign in through the interface and open an issue. If reading works but writing does not, the issue counter may not have come back — see section 3.

#2b. Restore — if you use an external PostgreSQL

There is no db container, so pg_restore is run by the application container; the backup directory is already mounted there.

The order is a little different: you cannot stop the application, because it is the container that runs the command. Instead --clean --if-exists is used — the restore drops the objects the start-up migrations created and puts the ones from the backup in their place.

# 1) Choose the backup
docker compose exec app ls -lh /data/backups

# 2) Restore (the application is UP, --clean is required)
docker compose exec app sh -c 'PGPASSWORD=<password> pg_restore \
  --host <db-host> --port 5432 --username <user> --dbname <database> \
  --no-owner --no-privileges --clean --if-exists /data/backups/<file>.dump'

# 3) Restart the application — the connection pool is looking at stale tables
docker compose restart app

You should see Database is up to date, no migrations to apply. in the log.

Without --clean --if-exists you get hundreds of "already exists" errors and end up with a half-filled database. This is not needed in the packaged installation, because there we can stop the application and restore into an empty database (§2).

Tested: 16 August 2026, against a real external PostgreSQL. The database was DROPped, restored from the application container, the user signed in again, and the migration history was in place.

#3. Common problems

#4. The drill (14 August 2026)

Setup: a clean docker compose up -d, first-user registration, one project (GERI), one issue (GERI-1).

1. backup-now                    → notusship-20260814-182013.dump (77 KB, 0.07 s)
2. docker compose stop app       → stopped
3. DROP DATABASE notusship       → 0 tables left in the public schema
4. pg_restore …                  → exit code 0, no errors
5. docker compose start app      → "Database is up to date, no migrations to apply."

Verification:

The counter coming back mattered: had projects.issue_counter not been restored, new issues would have collided with existing keys, and that would only have been noticed days later.

#5. Upgrading

docker compose exec app dotnet NotusShip.Api.dll backup-now   # back up first
docker compose pull
docker compose up -d
docker compose logs app --tail 30                              # watch the migrations

If a user sees a white screen and "Something went wrong." after an upgrade, their browser holds an index.html fetched before this release. One hard refresh (Cmd/Ctrl + Shift + R) fixes it, and it does not recur.

That banner follows the language cookie even though it lives in the page shell and is shown before the application starts — the shell reads the cookie itself (NSHIP-118).

Migrations are applied automatically at start-up and written to the log. Because dropping and renaming a column happens in two stages, going back one version does not lose data — but take a backup first anyway.