Skip to main content

Upgrade & Rollback

Each release of octopize-deploy-tool bundles the templates and the image versions of one Avatar release. Upgrading Avatar therefore means upgrading the tool, then letting it regenerate and restart the stack.

Deployment set up by hand, or before Authentik?

If your deployment directory has .env and .secrets/ at its top level and no .deployment_states/, run the one-time Legacy Migration first.

Before you upgrade​

  • Read the release notes of the target version.
  • Plan a maintenance window: the stack restarts, which usually takes 5 to 15 minutes, plus the time for database migrations.
  • Check that you have about 6 GB of free disk space for the new images.
  • Back up the databases and the deployment directory.

Backup​

Back up two things: both databases (Avatar's and Authentik's) and the whole deployment directory, which holds the secrets, including the storage encryption key. The backup files are written next to the deployment directory:

cd ./app/avatar/generated_files
STAMP=$(date +%Y%m%d-%H%M%S)
DB_ADMIN=$(cat .secrets/db_admin_user)

# Both databases, as SQL that drops and recreates their objects on restore
for DB in $(cat .secrets/db_name) $(cat .secrets/authentik_database_name); do
  docker compose exec -T db pg_dump -U "$DB_ADMIN" --clean --if-exists -d "$DB" \
    > "../../avatar-backup-$STAMP-$DB.sql"
done

# The deployment directory: configuration, every rendered version, and the secrets
tar czf "../../avatar-backup-$STAMP-app.tar.gz" -C ../.. avatar

Store these files somewhere safe: they contain every credential of the deployment.

Datasets in SeaweedFS are temporary and expire on their own, so they are not part of the backup. Users download their results.

For production, schedule the database backup, keep at least 30 days of backups, and test a restore regularly.

Upgrade​

1. Upgrade the deploy tool​

pip install --upgrade octopize-deploy-tool

If you run the tool from Docker, pull quay.io/octopize/deploy-tool:latest instead.

2. Regenerate the configuration​

octopize-deploy-tool update --app-root ./app/avatar

Your values are carried forward. The tool renders the new templates into a new version slot, shows the diff, starting with the service versions that change, and applies it. It does not restart anything yet.

The deploy tool releases after 3.5.1 need the new secret seaweedfs_admin_password. update generates it. To give your own value, see SeaweedFS admin password.

update ends by listing which Python client versions work with the new API. Tell your users before they next run a job if their client needs upgrading.

3. Restart the stack​

octopize-deploy-tool start --app-root ./app/avatar

This pulls the new images, recreates the containers and re-applies the Authentik blueprint. The API runs the database migrations as it starts.

4. Verify​

octopize-deploy-tool verify --app-root ./app/avatar --test-email you@company.com

Then log in through the web client and run a small avatarization. See Verification & Troubleshooting if anything fails.

Authentik moves one release line per upgrade​

Authentik upgrades its database one release line at a time (2026.2, then 2026.5, then 2026.8), and from 2026.8 it refuses to start on a database that skipped one. From deploy tool 3.6.0, update moves Authentik at most one release line from the version that your Authentik database has reached. A new installation gets the latest Authentik directly.

A deployment on an Authentik older than 2026.5 therefore needs one upgrade for each release line. For example, from 2026.2:

  1. The first update gives Authentik 2026.5.7. Run start and verify as above. They record that Authentik runs 2026.5.7.
  2. Run the upgrade again. This update gives Authentik 2026.8.

The diff that update shows lists the Authentik version that it chose. verify also checks that Authentik runs the version of the deployment, and fails if it does not.

Do not skip a release line

If you set AVATAR_AUTHENTIK_VERSION in your configuration, keep it at most one release line ahead of the version that Authentik runs. Otherwise Authentik stops at start with Major version skips are not allowed. update warns you and names the version to use. Authentik cannot downgrade its database: back it up before each upgrade.

Rollback​

Roll back if the API does not become healthy, a migration fails, or login, avatarization or storage is broken and cannot be fixed within the maintenance window. First collect diagnostics, so the issue can be investigated afterwards.

Database migrations are not reversible: a database migrated forward does not work with the previous API version. A rollback therefore restores the databases from the backup.

# 1. Stop the stack
octopize-deploy-tool stop --app-root ./app/avatar

# 2. Put the previous rendered version back in place (list the slots with: ls ./app/avatar/.deployment_states)
octopize-deploy-tool apply --app-root ./app/avatar --version v1

# 3. Restore both databases
cd ./app/avatar/generated_files
docker compose up -d db
DB_ADMIN=$(cat .secrets/db_admin_user)
for DB in $(cat .secrets/db_name) $(cat .secrets/authentik_database_name); do
  docker compose exec -T db psql -U "$DB_ADMIN" -d "$DB" < "../../avatar-backup-<STAMP>-$DB.sql"
done
cd -

# 4. Start the previous version
octopize-deploy-tool start --app-root ./app/avatar

Replace v1 with the slot that was deployed before the upgrade, and <STAMP> with the timestamp of your backup. Then verify the deployment.

If the deployment directory itself is damaged, restore it from the tar.gz archive first.