trellum
Docs / Recommended workflow / Validate report changes in CI

Validate report changes in CI

Applies to: latest

Run a build on each pull request to catch report errors before merging. The existing --fail-on-validation option exits unsuccessfully when validation finds FAIL-level issues. WARN and INFO findings remain visible but do not fail this gate. Generator/query exceptions also fail the build.

Prepare the project

Run from the analytics project root containing reports/. Commit a requirements-ci.txt with a pinned Trellum version and every dependency your report generators need. For example, start with trellum==0.9.5 and use the version tested by your team; pin any additional packages too.

CI executes your generators and reads their data. Use synthetic fixtures or a disposable test database for pull requests. The workflow below assumes those sources are already available without production credentials. Check that the build log lists the reports you intend to validate; an empty or misplaced reports/ directory is not evidence that your reports passed.

Try the same command locally first:

python -m pip install -r requirements-ci.txt
python -m trellum.run --all --no-serve --no-cache --fail-on-validation --max-concurrent 2

--no-cache prevents reused query results from hiding source changes. The build writes local output and exits without starting a preview. Do not add --production for this gate: production mode also selects the publishing path and requires an output backend. Do not add --no-fail-on-validation or ignore the command's exit status.

GitHub Actions example

Save this as .github/workflows/validate-reports.yml in your analytics repository:

name: Validate reports

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read

concurrency:
  group: validate-reports-${{ github.ref }}
  cancel-in-progress: true

jobs:
  validate:
    name: Validate reports
    runs-on: ubuntu-latest
    timeout-minutes: 15
    env:
      FW_PROJECT_ROOT: ${{ github.workspace }}
    steps:
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
        with:
          persist-credentials: false
      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
        with:
          python-version: '3.12'
      - name: Install report dependencies
        run: python -m pip install -r requirements-ci.txt
      - name: Build and validate every report
        run: python -m trellum.run --all --no-serve --no-cache --fail-on-validation --max-concurrent 2

Replace main if your default branch differs. For a nested project, set both defaults.run.working-directory and FW_PROJECT_ROOT to that project directory; keep its requirements-ci.txt alongside reports/. Adjust the job timeout and concurrency for your test workload. Maintain the action pins with your dependency update process.

If a trusted CI job needs test-source credentials, provide the environment names referenced by data-sources/config.yaml through GitHub Actions secrets. Portal credentials are not automatically available in a checkout. Fork pull requests do not receive repository secrets: keep their validation fixture-based. Do not use pull_request_target to execute untrusted report code with secrets.

Make Validate reports a required status check in your repository's branch protection or ruleset if failures should prevent merging. The workflow itself reports failure; it does not configure that merge policy.

Read a failed build

The log names the failing report and its validation findings or exception. Fix the source and rerun the same command. When validation aborts generation, its _validation.json and _details.json are written into that report's output directory. A failure earlier in loading or querying may not produce these files. Review generated output before sharing it: reports and logs can contain data.

This gate verifies generation and FAIL-level validation for the reports it builds. It does not replace browser interaction tests, data-quality assertions, or checks of a deployed portal's permissions. See Build failures and The analytics repository.