Verification & Troubleshooting

Use these checks after an installation or an upgrade. docker compose commands run from the generated_files/ directory:

cd ./app/avatar/generated_files

Quick verification

octopize-deploy-tool tasks provision-admin --app-root ./app/avatar
octopize-deploy-tool verify --app-root ./app/avatar --test-email you@company.com

provision-admin creates an admin API key for the akadmin account, in generated_files/.secrets/avatar_api_key. It only needs to run once. verify then checks the API with that key and sends a test email through Authentik.

Manual checks

Containers

docker compose ps

Every service should be running or healthy, none restarting. The services are api, web, nginx, db, dask-scheduler, dask-worker, master, volume, filer, s3, authentik_server and authentik_worker. init-db, init-storage and authentik_init_db exit once they have finished.

API health

curl -k https://<your-domain>/api/health

This returns {"message":"ok"} and needs no authentication. For a full check of the configuration, call /health/config-full-check with the admin API key:

curl -s -k https://<your-domain>/api/health/config-full-check \
  -H "Authorization: api-key-v1 $(cat .secrets/avatar_api_key)" | python3 -m json.tool

It checks the environment and secrets, the database and pending migrations, storage access, email, the web login page, the workers and telemetry. It returns 200 when everything passes, 207 with warnings and 500 when a check fails. The body details each check.

Email

Emails are sent by Authentik:

docker compose exec authentik_worker ak test_email you@company.com

Storage encryption

Check that SeaweedFS can read and write the encrypted buckets:

docker compose logs -t s3 --tail=100

SSE-S3 errors mean that .secrets/storage_encryption_key does not match the key the filer holds. Compare them:

docker compose exec master weed shell
> fs.cat /etc/s3/sse_kek

If they differ, put the filer's value back in your configuration and restart the stack. Never keep a new key: every stored object would become unreadable. See the SeaweedFS documentation.

Login

1. Open https://<your-domain>/.
2. Log in with an account created by tasks create-users, or as akadmin with the AUTHENTIK_BOOTSTRAP_EMAIL and AUTHENTIK_BOOTSTRAP_PASSWORD from .env.
3. Check that you land back in the Avatar interface and that it shows the expected version.

The server has no browser

Forward the HTTPS port over SSH from your workstation:

ssh -L 8443:localhost:443 user@server

Map the domain to your machine by adding 127.0.0.1 <your-domain> to your local /etc/hosts, then open https://<your-domain>:8443/.

Sample avatarization

Upload a small dataset, run an avatarization, and download the result. This is the only check that exercises the workers and the storage end to end.

If a check fails

Symptom	Fix
verify reports "API key not found"	Run tasks provision-admin first.
Containers start but cannot reach each other	Docker uses an HTTP proxy. Add the noProxy setting, then recreate the containers.
install stops: "Volume 'avatar_postgres_data' already exists"	The server holds data from an earlier deployment. Use update on that deployment's directory to keep it, or uninstall it to discard it.
install or update says the directory is "pre-versioned"	Run the Legacy Migration.
update stops: "Apply is blocked until conflicts are resolved"	Someone edited generated_files/. Move those edits into a configuration file, then accept the new render.
Pulling images fails (unauthorized, denied)	Run the docker login quay.io command Octopize sent you again.
The API never becomes healthy after an upgrade	Read docker compose logs api: a failed database migration is reported there. See Rollback.
Writes to storage fail, SSE-S3 errors in the s3 logs	The storage encryption key changed. See Storage encryption.
No email arrives	Check the SMTP settings and that the server can reach your SMTP server, then rerun ak test_email.
SSO changes do not take effect after update	Run octopize-deploy-tool tasks reconcile-blueprint --app-root ./app/avatar.
You are locked out of Authentik	See Recovering admin access.
The disk is full	Remove unused images with docker image prune -a, or move Docker's storage.

Add --verbosity 2 to any deploy tool command for detailed output.

Starting over

Use octopize-deploy-tool uninstall, which asks for confirmation and removes the containers, the volumes and the deployment files together. It deletes every dataset and user account. Do not run docker compose down --volumes by hand: it deletes the database but leaves the deployment files behind.

Collecting diagnostic information

Before contacting Octopize support, collect:

docker compose ps > diagnostics-ps.txt
docker compose logs --tail=20000 > diagnostics-logs.txt 2>&1
cp .env diagnostics-env.txt

warning

Remove passwords, tokens and secrets from these files before sharing them.

Send them to support@octopize.io with a description of the issue.