trellum
Docs / Portal guide / Evaluating assistant answers

Offline assistant evaluation

Applies to: latest

Run the local synthetic suite from the repository root:

python scripts/assistant_eval.py
python scripts/assistant_eval.py --split heldout --output /tmp/assistant-contract.json
python scripts/assistant_eval.py --predictions /tmp/captured-predictions.json

Use the project's Python environment and installed query-parser dependencies. This command opens no network connections and invokes no provider. It needs no customer data, credentials or database server.

The default deterministic_contract mode measures explicit calculation requests with the actual framework metric executor. It also checks synthetic exploratory queries with the actual PostgreSQL relation validator and independently computes their results in an in-memory SQLite database. It does not route natural language. The questions label reviewed requests; passing these checks says nothing about a model's interpretation of those questions.

The versioned corpus is apps/assistant/tests/fixtures/reliability/suite-v1.json: 40 intents, two phrasings each, ten metrics, follow-up contexts and successive source reads. review.md records the independent arithmetic, synthetic definitions and review limits. baseline-v1.json records the inspected implementation capabilities at its parent revision. No historical model responses were available, so baseline model accuracy is unmeasured. These files do not invent a historical model run. Business meanings await domain-owner approval; this corpus cannot certify a customer's revenue, user identity or calendar definitions.

The baseline/heldout split is 30/10 intents (60/20 phrasing samples). Keep heldout questions out of prompt examples. Because fixtures are public and have been reviewed during implementation, this is not a contamination-resistant benchmark. Each sample has a frozen UTC clock, business timezone, actor and scope, permitted relations, artifact/build/source revisions, explicit metric and parameters, outcome, rows and absolute/relative numeric tolerances. Periods use [start,end). Null and zero are distinct. All phrasings of one intent stay in the same split.

Supplied capture format

Supply structured predictions keyed by <intent_id>:<zero-based-phrasing-index>. A capture must include every selected sample; missing samples count as incorrect. Unknown sample IDs fail validation. For example, this excerpt supplies one sample; a full run must supply the other 79 too:

{
  "schema_version": 1,
  "metadata": {
    "kind": "model_capture",
    "provider": "record actual provider",
    "model": "record exact model version",
    "settings": {},
    "prompt_revision": "record tested revision",
    "run_id": "record repeat identifier",
    "cost": null,
    "latency_seconds": null
  },
  "predictions": {
    "revenue_total:0": {
      "outcome": "answered",
      "rows": [[800]],
      "trace": [],
      "evidence": {
        "build_id": "synthetic-build-1",
        "artifact_revision": "synthetic-artifact-1",
        "source_revision": "synthetic-source-1",
        "actor": "analyst",
        "scope": "studio:pilot",
        "metric_id": "revenue",
        "params": {"start": "2026-10-01", "end": "2026-10-04"},
        "clock": "2026-10-04T06:00:00+00:00",
        "timezone": "Europe/Amsterdam",
        "rows": [[800]]
      }
    }
  }
}

This schema is an evaluation interchange format, not a portal transcript schema. Adapt actual captures explicitly: preserve observed requests, answer blocks and evidence; never fill in missing predictions with expected answers. Use metadata.kind: "synthetic_test" for constructed predictions. Only an explicitly labeled model_capture sets model_evaluation: true; the command cannot verify capture authenticity. Scores on synthetic inputs test the scorer, not a model. Real provider runs require separate authorization and an agreed spend cap. Record model/settings, prompt revision, repeated-run identifiers, cost and latency in metadata rather than inferring them from results.

For a source answer, metric_id is null. Each trace item contains source_id (commerce here), sql, and observed rows. Every read is checked against the sample's permitted relations, including extra reads before an abstention. The reviewed sequence must match normalized SQL and result rows, in order. This is a strict request/trace contract: a mathematically equivalent alternate query can fail it and needs review before expanding the corpus. Evidence must carry the same revisions, effective parameters, actor/scope and final rows. Multi-read comparisons independently subtract the two read results; other source cases use the final read. For a clarification or limitation, supply its explicit outcome (clarification_required, unsupported, incomplete, stale, access_denied or abstained). A generic refusal does not receive credit for a specific expected limitation.

Reading results

Capture scoring classifies each sample as correct_answer, incorrect, clarification, abstention or unnecessary_abstention. Clarifications and abstentions earn correct-outcome credit only when the expected outcome requires them. Refusing an answerable question is an unnecessary abstention. Wrong values, ungranted reads and mismatched evidence count as incorrect. Narrative prose and the usefulness of clarification wording still require manual review.

The output includes intent/sample counts, supplied count, answerable count, classification counts and per-sample failures. correct_outcome_rate divides correct answers plus required clarifications/abstentions by all selected samples, including missing captures. direct_answer_coverage is all emitted answers divided by all selected samples; it includes wrong answers. correct_direct_answer_coverage is correct direct answers divided by answerable samples. An abstain-all capture therefore has zero direct-answer coverage and cannot pass. Always inspect failure severity alongside aggregates.

The command returns nonzero for failed deterministic contracts, incorrect predictions or unnecessary abstentions. It writes deterministic JSON without a wall-clock run timestamp; the suite SHA-256 identifies the suite inputs. It evaluates deterministic contracts even when captures are supplied, so inconsistent truth fixtures fail the run. It does not implement a release approval or assert a universal model accuracy threshold.

Three current intents have no offline calculation check: semantic ambiguity (two) and actor authorization (one). They are explicitly unmeasured, not passes. Source relation scope is parser-tested; actor permissions, tenant boundaries, revocation, report visibility, provider behavior, live PostgreSQL execution, resource bounds/cancellation, UI presentation and free-text factual claims need their separate integration/manual checks. No PostgreSQL connector certification or live security claim follows from SQLite fixture calculations.