Sizing
Report builds usually drive CPU and memory needs. Most viewers read pre-built static output; allow for serving that traffic too. Reports with opt-in live queries also need web and data-source capacity for interactive requests.
Starting points
| Situation | Rough shape |
|---|---|
| Evaluation, a handful of reports | 2 CPU, 4 GB RAM |
| A team, tens of reports | 4 CPU, 8 GB RAM |
| Several teams, heavy dataframes | 8 CPU, 16 GB+ RAM |
The variable that matters is the largest amount of data a single report pulls into memory at once, not how many reports exist.
What consumes what
- CPU — concurrent builds; one worker process can run several builds at
once (
WORKER_MAX_CONCURRENTdefaults to 3). Additional runner processes each have their own concurrency limit - RAM — the biggest single build. A report that loads a very large frame sets the floor for the whole instance
- Disk — built output, the database, and backups. Output size depends on the data included: reports can contain row-level datasets as well as aggregates. Output can be moved to object storage; checkouts and uploaded files stay on the volume either way
Explicit TRELLUM_RUN_TIMEOUT environment settings, including 1800, retain their value on upgrade;
check the worker service environment before expecting the new default. The build
budget begins at process start, excluding queue time. Driver connect and statement
timeouts and worker shutdown drain budgets remain independent. Shutdown still
drains for 540 seconds, then terminates remaining builds.
Sizing report-build memory precisely
The starting points above get you running; once you have real traffic, size
memory from evidence instead. Every build gets the same admission allocation,
TRELLUM_DEFAULT_JOB_MEMORY_MB (1024 by default), and the same wall-clock
limit, TRELLUM_RUN_TIMEOUT (3600 seconds / 60 minutes by default). report.yaml carries
no sandbox limits, so size both for the largest report on the instance. Give
the runner a budget sized to what's left once the rest of the stack has what
it needs:
TRELLUM_RUNNER_MEMORY_BUDGET_MB = total RAM
- RAM for the database
- RAM for the portal
- headroom
| VM size | Suggested TRELLUM_RUNNER_MEMORY_BUDGET_MB |
Concurrent 1 GB builds |
|---|---|---|
| 4 GB | 1536 | 1 |
| 8 GB | 5632 | 5 |
| 16 GB | 13824 | 13 |
Leaving the budget at 0 (the default) keeps the simpler behaviour: only a
concurrency limit (WORKER_MAX_CONCURRENT) caps how many builds run at once,
with no memory accounting. A budget controls admission to that runner, not
physical memory reservation. Allow headroom for builds reaching their caps,
and divide the available memory between runner processes sharing a host.
Two details worth knowing:
- Docker sandboxes enforce a container memory cap of
TRELLUM_DEFAULT_JOB_MEMORY_MB x TRELLUM_JOB_MEMORY_HEADROOM(headroom defaults to 1.5, with a minimum container cap of 256 MB) and disable swap. Admission uses the base amount, so simultaneous builds can consume more than the configured budget. - Unsandboxed builds on POSIX instead use an address-space limit when
TRELLUM_JOB_MEMORY_ENFORCE=true. Address space can be larger than resident memory for dataframe workloads. Unsandboxed Windows builds have no such memory cap; their wall-clock timeout still applies.
A Docker-confirmed memory kill is recorded as oom_killed; a Python
MemoryError is recorded as error and means an allocation failed. An
unexplained SIGKILL (exit 137) is recorded as error: that exit code alone
cannot establish an out-of-memory kill. Runner diagnoses are included in both
the completed run log and the report error summary. The log preserves the
original stderr tail; the shorter summary retains the diagnosis and final
exception. Memory diagnoses distinguish the admission allocation from the
applied container or address-space cap and the sampled peak. A later failed
connection check is secondary evidence, not a replacement for the build error.
Operations shows observed memory use. Its max observed value is the largest sampled peak among completed runs in the displayed history window, not a configured limit. Sampling can miss the final spike before termination. A database connection-closed message alone does not identify which side or resource caused the failure. Compare the runner diagnosis and original trace with the container/host and database logs before attributing it to memory. Use those measurements and failures to size limits from real workloads.
Run heavy reports one at a time
With one runner process, set WORKER_MAX_CONCURRENT=1 to leave other builds
queued until the active build finishes. One worker container alone does not
ensure serial execution. Each additional runner process can still admit its
own build.
Serial execution does not raise the per-build cap. Adding host RAM also does
not change TRELLUM_DEFAULT_JOB_MEMORY_MB or its headroom multiplier. Size the
applied cap for the largest report while leaving measured room for the OS,
database, portal and other services. A nonzero runner admission budget must
also accommodate the configured job allocation. These settings are read when
the runner process starts; existing build containers keep their original caps.
To reduce a report's peak, first identify whether it occurs while fetching, transforming or rendering. Reduce unnecessary result rows and columns, perform appropriate aggregation in the database, and avoid retaining redundant frames. Fetching in batches only bounds memory when each batch can be processed and released; collecting every batch into a final full frame still requires that frame to fit. Browser data chunking does not make the build stream its input.
When one machine is not enough
Choose the topology in Scale builds on one server when the queue needs more capacity on one host, or in Run across multiple servers when runner capacity spans hosts. This page keeps the sizing tables and runner-pool details used by both guides.
Assign studios to runner pools
Runner pools route builds to selected runners, for example a larger machine
for a studio with heavy reports. An instance operator opens the organization's
detail page under /operator/orgs/ and changes the studio's Pool in the
Studios table. The choices are small, standard (the default), and
large. The assignment applies to the whole studio; already queued runs keep
the pool recorded when they were queued.
Set TRELLUM_RUNNER_POOLS separately for each runner service: large accepts
only large-pool jobs, while small,standard accepts those two pools. Leaving
it empty accepts all pools. For dedicated routing, configure every runner's
pool list; an unrestricted runner can otherwise claim any job. Keep at least
one runner serving each pool you use, or its jobs stay queued.
Pool names do not set memory or CPU limits. Configure the runner's capacity with the build settings above; shared storage is still required across pools.
Uploaded data-source files
A data source can be a file the portal stores directly instead of a git repository — the way to use a large extract without committing it. Two limits bound this, since uploads are otherwise the only tenant-controlled consumer of the data volume:
- Per file — Max upload size (MB) on the operator
/systempage, 512 MB by default.0removes the limit. - Per organization — a Max uploaded files (MB) quota, set per
organization in the operator console.
0means unlimited.
An upload is also refused outright if it would leave under 1 GB free on the
data volume, regardless of either setting above — a full volume takes down
every tenant, not just whoever filled it. manage.py doctor reports free
space.
Reverse proxy: request size
Whatever you set above, your reverse proxy sees the upload first, and most proxies cap request bodies well below it:
- nginx defaults to
client_max_body_size 1m— raise it (client_max_body_size 512m;) to match, and giveproxy_read_timeoutroom too (300s), or every upload over 1 MB fails with a 413 before it reaches the portal. - Traefik has no body limit by default.
- Caddy has no body limit by default; only
request_body max_sizeadds one.
Watching it
/system (operators) shows live workers and recent runs across all studios,
including memory figures per run. A report whose memory figure creeps upward
release over release is the usual early warning.
GET /healthz is a lightweight liveness check — the web process and the
database — for whatever uptime monitor or load balancer you point at the
instance.
Completed output replaces the prior directory only after a successful build. Linux uses an atomic directory exchange; on other filesystems/platforms a rename-and-rollback sequence can briefly make the directory unavailable, while keeping partially generated files out of the served report.
Automatic recovery requires confirmed process-tree termination. POSIX workers launch each build in a separate process group; Docker workers confirm container exit through inspection. A missing container or surviving process group remains unconfirmed and holds the report's active-run fence. Unsandboxed Windows workers cannot currently prove descendant termination and also retain this fence; use Docker for builds requiring automatic recovery. Operator investigation is needed when a termination outcome cannot be confirmed.