trellum
Docs / Operations / Sizing

Sizing

Applies to: latest

Report builds usually drive CPU and memory needs. Most viewers read pre-built static output; allow for serving that traffic too. Reports with opt-in live queries also need web and data-source capacity for interactive requests.

Starting points

Situation Rough shape
Evaluation, a handful of reports 2 CPU, 4 GB RAM
A team, tens of reports 4 CPU, 8 GB RAM
Several teams, heavy dataframes 8 CPU, 16 GB+ RAM

The variable that matters is the largest amount of data a single report pulls into memory at once, not how many reports exist.

What consumes what

  • CPU — concurrent builds; one worker process can run several builds at once (WORKER_MAX_CONCURRENT defaults to 3). Additional runner processes each have their own concurrency limit
  • RAM — the biggest single build. A report that loads a very large frame sets the floor for the whole instance
  • Disk — built output, the database, and backups. Output size depends on the data included: reports can contain row-level datasets as well as aggregates. Output can be moved to object storage; checkouts and uploaded files stay on the volume either way

Explicit TRELLUM_RUN_TIMEOUT environment settings, including 1800, retain their value on upgrade; check the worker service environment before expecting the new default. The build budget begins at process start, excluding queue time. Driver connect and statement timeouts and worker shutdown drain budgets remain independent. Shutdown still drains for 540 seconds, then terminates remaining builds.

Sizing report-build memory precisely

The starting points above get you running; once you have real traffic, size memory from evidence instead. Every build gets the same admission allocation, TRELLUM_DEFAULT_JOB_MEMORY_MB (1024 by default), and the same wall-clock limit, TRELLUM_RUN_TIMEOUT (3600 seconds / 60 minutes by default). report.yaml carries no sandbox limits, so size both for the largest report on the instance. Give the runner a budget sized to what's left once the rest of the stack has what it needs:

TRELLUM_RUNNER_MEMORY_BUDGET_MB = total RAM
                              - RAM for the database
                              - RAM for the portal
                              - headroom
VM size Suggested TRELLUM_RUNNER_MEMORY_BUDGET_MB Concurrent 1 GB builds
4 GB 1536 1
8 GB 5632 5
16 GB 13824 13

Leaving the budget at 0 (the default) keeps the simpler behaviour: only a concurrency limit (WORKER_MAX_CONCURRENT) caps how many builds run at once, with no memory accounting. A budget controls admission to that runner, not physical memory reservation. Allow headroom for builds reaching their caps, and divide the available memory between runner processes sharing a host.

Two details worth knowing:

  • Docker sandboxes enforce a container memory cap of TRELLUM_DEFAULT_JOB_MEMORY_MB x TRELLUM_JOB_MEMORY_HEADROOM (headroom defaults to 1.5, with a minimum container cap of 256 MB) and disable swap. Admission uses the base amount, so simultaneous builds can consume more than the configured budget.
  • Unsandboxed builds on POSIX instead use an address-space limit when TRELLUM_JOB_MEMORY_ENFORCE=true. Address space can be larger than resident memory for dataframe workloads. Unsandboxed Windows builds have no such memory cap; their wall-clock timeout still applies.

A Docker-confirmed memory kill is recorded as oom_killed; a Python MemoryError is recorded as error and means an allocation failed. An unexplained SIGKILL (exit 137) is recorded as error: that exit code alone cannot establish an out-of-memory kill. Runner diagnoses are included in both the completed run log and the report error summary. The log preserves the original stderr tail; the shorter summary retains the diagnosis and final exception. Memory diagnoses distinguish the admission allocation from the applied container or address-space cap and the sampled peak. A later failed connection check is secondary evidence, not a replacement for the build error.

Operations shows observed memory use. Its max observed value is the largest sampled peak among completed runs in the displayed history window, not a configured limit. Sampling can miss the final spike before termination. A database connection-closed message alone does not identify which side or resource caused the failure. Compare the runner diagnosis and original trace with the container/host and database logs before attributing it to memory. Use those measurements and failures to size limits from real workloads.

Run heavy reports one at a time

With one runner process, set WORKER_MAX_CONCURRENT=1 to leave other builds queued until the active build finishes. One worker container alone does not ensure serial execution. Each additional runner process can still admit its own build.

Serial execution does not raise the per-build cap. Adding host RAM also does not change TRELLUM_DEFAULT_JOB_MEMORY_MB or its headroom multiplier. Size the applied cap for the largest report while leaving measured room for the OS, database, portal and other services. A nonzero runner admission budget must also accommodate the configured job allocation. These settings are read when the runner process starts; existing build containers keep their original caps.

To reduce a report's peak, first identify whether it occurs while fetching, transforming or rendering. Reduce unnecessary result rows and columns, perform appropriate aggregation in the database, and avoid retaining redundant frames. Fetching in batches only bounds memory when each batch can be processed and released; collecting every batch into a final full frame still requires that frame to fit. Browser data chunking does not make the build stream its input.

When one machine is not enough

Choose the topology in Scale builds on one server when the queue needs more capacity on one host, or in Run across multiple servers when runner capacity spans hosts. This page keeps the sizing tables and runner-pool details used by both guides.

Assign studios to runner pools

Runner pools route builds to selected runners, for example a larger machine for a studio with heavy reports. An instance operator opens the organization's detail page under /operator/orgs/ and changes the studio's Pool in the Studios table. The choices are small, standard (the default), and large. The assignment applies to the whole studio; already queued runs keep the pool recorded when they were queued.

Set TRELLUM_RUNNER_POOLS separately for each runner service: large accepts only large-pool jobs, while small,standard accepts those two pools. Leaving it empty accepts all pools. For dedicated routing, configure every runner's pool list; an unrestricted runner can otherwise claim any job. Keep at least one runner serving each pool you use, or its jobs stay queued.

Pool names do not set memory or CPU limits. Configure the runner's capacity with the build settings above; shared storage is still required across pools.

Uploaded data-source files

A data source can be a file the portal stores directly instead of a git repository — the way to use a large extract without committing it. Two limits bound this, since uploads are otherwise the only tenant-controlled consumer of the data volume:

  • Per file — Max upload size (MB) on the operator /system page, 512 MB by default. 0 removes the limit.
  • Per organization — a Max uploaded files (MB) quota, set per organization in the operator console. 0 means unlimited.

An upload is also refused outright if it would leave under 1 GB free on the data volume, regardless of either setting above — a full volume takes down every tenant, not just whoever filled it. manage.py doctor reports free space.

Reverse proxy: request size

Whatever you set above, your reverse proxy sees the upload first, and most proxies cap request bodies well below it:

  • nginx defaults to client_max_body_size 1m — raise it (client_max_body_size 512m;) to match, and give proxy_read_timeout room too (300s), or every upload over 1 MB fails with a 413 before it reaches the portal.
  • Traefik has no body limit by default.
  • Caddy has no body limit by default; only request_body max_size adds one.

Watching it

/system (operators) shows live workers and recent runs across all studios, including memory figures per run. A report whose memory figure creeps upward release over release is the usual early warning.

GET /healthz is a lightweight liveness check — the web process and the database — for whatever uptime monitor or load balancer you point at the instance.

Completed output replaces the prior directory only after a successful build. Linux uses an atomic directory exchange; on other filesystems/platforms a rename-and-rollback sequence can briefly make the directory unavailable, while keeping partially generated files out of the served report.

Automatic recovery requires confirmed process-tree termination. POSIX workers launch each build in a separate process group; Docker workers confirm container exit through inspection. A missing container or surviving process group remains unconfirmed and holds the report's active-run fence. Unsandboxed Windows workers cannot currently prove descendant termination and also retain this fence; use Docker for builds requiring automatic recovery. Operator investigation is needed when a termination outcome cannot be confirmed.