Upgrades
The online helpers switch the portal and report-runner images together. The
verification record lists the focused checks and isolated
product upgrade/rollback drill. They
require Bash, standard Linux utilities (including timeout), Docker Compose v2, the bundled
Postgres service, and the default single-host combined-worker topology:
scripts/upgrade.sh v1.2.3
# Noninteractive, after reviewing release notes:
scripts/upgrade.sh v1.2.3 --yes
Host Python is optional. If Python 3.11+ is unavailable, the wrapper builds a
small interpreter/CLI image from scripts/deployment.Dockerfile before any
application changes. This online setup needs access to the Docker and Python
base images. Image setup has a five-minute deadline. The adapter supports a local Unix Docker socket only, verifies
that the daemon sees the deployment directory at its exact host path, and
mounts that directory, backup directory and legacy dump directory at the same
paths. Keep referenced Compose files inside the deployment directory. Remote
Docker contexts and Docker Desktop filesystem translation require an operator
procedure. Registry authentication using external host credential-helper
executables also requires an operator procedure when using the adapter.
Published defaults are ghcr.io/trellumhq/trellum:<version> and
ghcr.io/trellumhq/trellum-runner:<version>. For a mirror, set
TRELLUM_REGISTRY and TRELLUM_REPO (the runner repository defaults to the
portal repository plus -runner). To select arbitrary repositories or digests,
set both TRELLUM_TARGET_IMAGE and TRELLUM_TARGET_RUNNER_IMAGE. Each image
must report the requested product version. The helper inspects and smoke-runs
both images before downtime, then pins their immutable local image IDs in
.env. This is a version/import check, not signature verification; verify
release signatures and attestations using Verify what you received.
TRELLUM_COMPOSE selects the Compose command, including project/file options.
Run from this deployment's .env and services. Alternate --env-file,
--project-directory, daemon flags and COMPOSE_ENV_FILES are rejected; use
docker compose -p <project> -f <file> options and the calling shell's Docker
context instead. Do not export
TRELLUM_IMAGE or TRELLUM_RUNNER_IMAGE in the calling shell: they can override
.env. A preflight configuration check rejects image overrides that prevent the
paired pins from taking effect. The release-only overlay selects images through
TRELLUM_VERSION and is excluded; migrate it to the base image-pin Compose
configuration first on a disposable copy: record both image references from
docker compose config, put those exact references in .env as TRELLUM_IMAGE
and TRELLUM_RUNNER_IMAGE, then remove only docker-compose.release.yml from
COMPOSE_FILE. Keep any supported storage override. Compare the rendered
configuration before/after: both images and all database/data mounts must stay
the same. This configuration conversion does not require an image upgrade or
container restart. Retain the original .env for recovery. External databases
and split/multi-host setups remain outside the helper procedure.
External Postgres, offline upgrades, split workers, profiles, and multi-host installations are excluded. These require a database/operator-specific backup, drain, migration and restore procedure across all hosts. The helpers reject external database URLs and active split services rather than backing up an unused bundled database. Custom overlays must preserve the base service roles, database and shared-data mounts; arbitrary overlay semantics are not validated. An existing local image does not bypass the online target pull.
Before you start
- Know which version you are on.
curl -s https://bi.example.com/api/version, or the operator/systempage. - Let
/systemtell you there is a newer one. SetTRELLUM_UPDATE_CHECK=trueand the page lists the latest releases and marks the one this instance runs. It is off by default; on, it is one outbound request a day. - Read the release notes for every version between yours and the target. Anything needing a manual step says so under Upgrade notes.
- Upgrade one version at a time unless the notes say a jump is safe. Migrations are written to be applied in order.
- Be healthy first.
docker compose exec web python manage.py doctorshould pass. The script checks this too and stops otherwise — upgrading an instance that is already unwell turns one problem into two, and the upgrade gets the blame. - Upgrading from v0.7.0 or earlier: run cleanup once first. Those releases
never ran the scheduled retention cleanup, so
doctorfails withcleanup has never run. Preview withdocker compose exec web python manage.py cleanup --dry-run, then run it without--dry-run. That first run deletes everything past its retention window since installation.
What the script does, and why in that order
| Step | Why |
|---|---|
doctor preflight |
Refuses to upgrade an unhealthy instance |
| Pull, inspect and import/version-check both images | Fail before downtime; select immutable IDs |
| Stop web, worker and backup | Drain writers before the database snapshot; stop failures abort the operation |
pg_dump, then pg_restore --list |
Save a readable custom-format dump plus an adjacent manifest containing both prior image IDs, original configured pins, version and dump hash |
migrate |
New schema, with no old code running against it |
| Start | The new version comes up on a schema it agrees with |
| Verify | /healthz, then assert /api/version reports the expected version and schema: ok |
The helper operates on the selected Compose project. It starts only the
combined-worker services, without rebuilding, pulling or implicitly rerunning
migrations. Verification checks the serving version/schema, actual web and
worker image IDs, worker runner configuration and doctor. It does not submit
a report build automatically. Build a representative report and inspect its
sandbox image before accepting the upgrade.
Expect a short outage: seconds, plus however long your migrations take, plus up to ten minutes if a report build is mid-flight when the worker drains.
Why not just docker compose up -d
Compose starts the one-shot migrate service while the old web and
worker containers are still serving. For the length of the migration the
running code and the schema disagree, and nobody chose that. It also takes
no backup, and reports success when the containers start — not when the new
version is actually serving a schema it agrees with.
Air-gapped sites
No offline installation or upgrade bundle is provided; see
Verify what you received.
An offline upgrade must transfer and verify both images and the operational
files, and test the full upgrade and rollback procedure without network
access. The upgrade helper still calls docker pull, so an already-loaded
portal image alone does not make that script an offline upgrade path.
How you know it worked
The script tells you, but to check by hand at any time:
curl -s http://127.0.0.1:8050/api/version
{"version": "1.2.3", "sha": "a1b2c3d…", "build_time": "…",
"branch": "", "source_url": "https://github.com/trellumhq/trellum/tree/v1.2.3",
"framework": {"tag": "v1.2.3", "meta_schema": 1}, "schema": "ok"}
The values to check are:
versionis the release actually serving. If it still shows the old one, the old container is still up.framework.tagmirrors the same product version for compatibility with existing clients. The framework and platform are released together.framework.meta_schemais the generated-report compatibility number. It can remain unchanged across product releases.source_urlidentifies the exact source for the running release.schemais whether the database matches that code:ok— in step.pending— the code is newer than the database. The migration did not run or did not finish. The instance may work until something touches a column that is not there. Rundocker compose run --rm migrate.error— the database could not be read at all.
manage.py doctor and /system report the same thing, and name the missing
migrations.
Note
Report output is derived data. After an upgrade, existing reports keep serving their last build; they refresh on their normal schedule or when you trigger a run.
Rolling back
Migrations are forward-only
Putting the old image back on a database whose schema has moved gives you old code reading new columns. The database has to go back too, and the only way back is the dump.
scripts/rollback.sh ./upgrade-backups/pre-<stamp>-<pid>.dump
This discards every database change made since that dump was taken — report runs, audit rows, settings, invitations. That is not caution, it is arithmetic: the dump is a snapshot, and going back to it means going back to it.
So:
- Prefer rolling forward. If the new version is merely misbehaving, fix the fault and upgrade again. Reach for rollback when the new version cannot serve.
- Keep the dump and its
.images.jsonmanifest until you are satisfied. The script leaves them inupgrade-backups/(override withTRELLUM_BACKUP_DIR) and prints the recovery command. Keep the recorded local images too; do not prune them during the recovery window. Rollback uses their immutable IDs and requires both to remain available locally. - Report output and uploaded data-source files are untouched by either script.
For an older custom-format dump without a manifest, retain the previous portal image/version positional argument and supply the matching runner explicitly:
scripts/rollback.sh ./backups/old.dump v0.9.1 \
--runner-image ghcr.io/trellumhq/trellum-runner:v0.9.1 --confirm-legacy-pair
# For a custom portal image, also supply --previous-version 0.9.1.
--confirm-legacy-pair confirms that the backup and both selected images belong
to the same previous release. Establish this from your deployment records;
there is no safe way to infer the image pair from a database dump. Both images
are pulled (unless immutable local IDs), imported and checked against the
expected version; the dump is hashed and validated before downtime. The usual
ROLLBACK confirmation still applies (or explicit --yes for automation).
Missing metadata or a corrupt existing manifest cannot silently fall back to
legacy recovery. New backups continue to use manifests as the preferred path.
Keep the bundled db service running for dump validation and restore. If it
is stopped, start only it with docker compose up -d db before recovery.
Rollback checks the dump hash/readability and both prior image versions before
stopping writers. It first saves a verified before-rollback database dump,
then restores into a fresh scratch database with
--exit-on-error --single-transaction. Only after that succeeds does a separate
transaction rename the current database to a retained recovery name and the
scratch database to the configured name. This requires bundled-Postgres
superuser privileges and space for both databases plus their dumps. New tables
from the abandoned release cannot survive in the restored database. The
retained database name is printed and recorded in the recovery marker during
the operation, and in a sidecar beside the safety dump; remove it only after
accepting the rollback. A restore or
database-switch error is fatal. It restores both image pins, checks serving
version/schema and worker configuration, and retains all backups. It never restores or deletes uploaded
files, encryption keys, report output, or other shared-volume data; protect
those separately with Backups & restore.
If an upgrade fails midway
Before stopping services, pull, image-version, configuration and preflight
failures leave the deployment running and its pins unchanged. After the stop
begins, failures leave application services stopped and retain
.trellum-deployment-incomplete.json. A dump that failed verification is not a
usable recovery backup. If migration has begun, use the printed dump and paired
manifest to roll back; never just restart the old code on a possibly new schema.
A failed migration is not retried; its named one-shot container is terminated on error/timeout. If termination cannot be confirmed, investigate that container before restoring. A failed restore or timeout may have an unknown outcome: inspect Postgres and the retained recovery marker before considering a manual retry. The helper refuses a second rollback after an unconfirmed rollback. It never automatically restores, retries a write, or restarts after failure.
The directory lock .trellum-deployment-lock prevents concurrent helper runs.
After a killed host process, check that no helper or migration/restore is still
running before manually clearing a stale lock. Clear an incomplete marker only
after reconciling database, both pins and service state. Do not delete these
files simply to get past a failed check. The default per-command timeout is
900 seconds (TRELLUM_COMMAND_TIMEOUT); startup verification has a 180-second
polling budget (TRELLUM_HEALTH_TIMEOUT) with individually bounded commands.
A slow command can extend that polling budget by one command timeout.
Bundled Postgres commands also use its container-side timeout utility so
terminating a Docker client cannot leave a database write running indefinitely.
See the isolated verification record for the tested path and its limits.
Versioning
Trellum follows semver. The Python framework and self-hosted platform have one product version and are released together.
- MAJOR — a breaking configuration change, an upgrade that needs a manual step, or a migration that isn't rollback-safe. Always called out in the release notes.
- MINOR — new backwards-compatible features or report-building behaviour.
- PATCH — fixes only; no schema or configuration change.
/api/version keeps a framework object for compatibility, but its tag
matches the product version. Its separate meta_schema number decides
whether report output built by an older release can still be read.
If you're upgrading from before report sandboxing
Report builds now run in disposable per-build containers instead of inside the worker process. Crossing that change needs two things on the host, the first time only:
- Docker Engine 26+ for named-volume storage, or 25+ with
host-folder storage.
Before changing the stack, run
bash scripts/check-docker.sh(add--bind-dataonly for the host-folder setup). The worker also checks its actual storage type and API version at startup. DOCKER_GID— add it to.env(see Configuration); without it the worker cannot reach the socket it needs to start build containers.
Then run the firewall rule once, if you haven't already.
If your data volume predates this change, it may still be owned by root from an older image that ran as root — the current, unprivileged process can't write to it until you fix ownership once, with the stack stopped:
docker compose down
docker compose run --rm --user root web chown -R 10001:10001 /data
docker compose up -d
manage.py doctor names this exact command if it detects the problem.