TrueVectors — Architecture

How TrueVectors is put together — the tiers a request flows through, described by role. This page tracks implemented behaviour only; roadmap items are marked separately. Implementation status does not imply public distribution.

Components

Clients

CLItruevectors / tv

VS Code / Cursor extensioneditor integration

React dashboardbrowser UI

↓

Service

Local API serverFastAPI · serves the dashboard

The CLI calls the auditor directly; the extension and dashboard go through the server.

↓

Core

Outcome auditorthe audit orchestrator

↓  fans out to

Audit engines

Deterministic verifiersstatic analysis + detectors

LLMAnthropic · temp 0 · tool-use

Falsification sandboxexecution + data-contract checks · opt-in

↓  converge on

Fusion

Adjudicatorfusion + evidence-tier stamping

↓

Persistence

Local store.truevectors/ · runs + feedback

The audit pipeline

One engine powers both the TrueVectors Auditor and the TrueVectors Hub. A request flows through a deliberately short pipeline, where every stage has a single responsibility and every finding carries an audit trail back to the stage that produced it.

Stage	Role
Orchestration	Resolves intent, enumerates the files in scope, plans the work, loads relevant session context, runs a pre-spend cost gate, calls the model per file, then adjudicates and persists.
Prompt construction	Builds the system and user messages, grounding the model with the facts the deterministic layer has already proven.
Closed taxonomy	A canonical, enumerated mode_id vocabulary with a normaliser that maps synonyms back onto the canonical id; unknown ids are rejected at parse time.
Result schema	A typed result/finding shape carrying each finding's evidence tier — proved, corroborated, or model opinion.
Large-file handling	Overlapping-window split with per-window merge and dedupe. Fault-tolerant: a window that fails every retry is skipped, the surviving windows still merge, and the unaudited range is surfaced as a coverage gap — never a silent pass.
PR integration	Proved-tier-first: the summary leads with the proved tier, inline comments are proved-tier only, and the model-only tier is collapsed; the required check is configurable and non-blocking by default.
Triage policy	One shared proved-tier-first policy — split, count, and summarize findings by evidence tier — used identically by the CLI, the API, and the PR surfaces.
Profiles	fast / standard / deep-verify presets that bundle the feature flags; resolved from a profile flag or a workspace config file, with the environment always winning.
Trends	Descriptive per-run trends from persisted run records — raw counts by evidence tier and severity, deliberately no composite score.
Published spec export	Generates the public failure-mode taxonomy spec from the engine's source of truth, with a CI drift guard.
Claims routing	Parses agent-session transcripts into typed claim records, then routes each claim to the cheapest verifier that can settle it (deterministic detector → counterexample → artifact invariant → otherwise an unverified-claim).

The deterministic layer

This is the symbolic half of the neuro-symbolic engine — the part that proves or refutes rather than opines. It runs only against the changed line numbers in a diff, so it never re-litigates code you didn't touch.

Capability	What it proves	Status
Diff-scoping core	The shared verifier model and the scoping that limits every check to changed lines, plus the soundness contract that lets an absence-of-signal refute a false claim.	Implemented
Swallowed-error & test-gaming detection	Swallowed exceptions; gamed tests (tautological asserts, skip-added, no-reference-to-subject, inline reimplementation, source-string search).	Implemented
Missing-output-contract detection	A row-reducing transform returned without an output guard.	Implemented
Conservation detection	Fan-out followed by a conserved-sum candidate, with false-positive controls.	Implemented
External-tool oracles	ruff / mypy run as isolated oracle verifiers behind an opt-in flag, with an explicit rule → mode map.	Implemented (opt-in)
JS/TS analysis	tree-sitter queries for swallowed errors and test-gaming signals in JavaScript / TypeScript.	Implemented
Adjudication	Post-model fusion: stamp model-only findings, promote corroborated ones, soundly refute disproven ones, append proven facts, and recompute the summary.	Implemented
Dataflow / taint	Semgrep / CodeQL for unvalidated-input and taint sinks.	Roadmap

The falsification engine (opt-in per repo)

For claims that reduce to a property over inputs, the auditor goes past static analysis and executes:

• A typed property specification is compiled from the model's intent, a Hypothesis sampling backend hunts for a violating case, and a CrossHair / Z3 symbolic backend either returns a concrete counterexample or proves the property across the whole declared input domain.
• A pandera + metamorphic data-contract harness checks a transform's output schema and the relations it must preserve, over the same sandbox.
• A candidate-invariant inference pass proposes likely invariants (labelled inferred / likely, never proved).
• Everything runs in a resource-capped subprocess sandbox (CPU / memory / wall-clock limits, network blocked), gated by two keys: a global enable switch and a committed .truevectors/counterexample.toml allowlist naming the exact callables allowed to run. Nothing executes by surprise.

The generic outcome & behavioural auditor

The same deterministic checker that proves source-code facts also proves a pipeline's realized output and an agent run's declared policy:

• A deterministic Tier-0 checker reads an output artifact (CSV / JSON / Parquet / DataFrame) and evaluates each declared invariant — no LLM on this path.
• An optional, gated LLM step proposes candidate invariants in a small DSL; it never decides whether they hold and fails loudly when its key is missing.
• A metamorphic conservation suite adds duplication- and order-invariance relations.
• Behavioural adapters flatten recorded agent runs onto the identical checker: simple sessions, ATIF trajectories (auto-detected), and whole agent-lens / Harbor run directories (policy lifted from the run's own config.yaml). A unified invariants front door produces receipts and a gated, receipts-only explainer.
• A calibration component fits and serves probability calibration, with a refusal guard that declines to serve on a synthetic map.
• A deterministic eval harness with a frozen baseline backs a CI regression gate.

Runtime: server + CLI

• The CLI exposes serve, outcome-audit, audit, and invariants subcommands (with --profile and --all). The invariants command takes --workspace, --session, or --agentlens-run (no LLM, no key) plus --report / --json and a gated --explain. It ships under the truevectors and tv entry points.
• The API server is a FastAPI app that serves the built dashboard and exposes the audit API (runs, findings with an evidence-tier filter, descriptive read-only trends, feedback, exports) and an invariants API that backs the dashboard's Invariants page.

Apps

• Editor extension — the TrueVectors VS Code / Cursor extension. It bundles the Outcome Audit commands and ships a partial TypeScript deterministic layer (swallowed-error plus the two test-gaming value signals for JS/TS). The full detector set, the tool-runner oracles, and the counterexample / data-contract stages are Python-only, so the Python CLI is authoritative.
• Dashboard — a React + Vite single-page app served by the backend. Findings default to the proved evidence tier (toggle to All); a descriptive trends panel charts per-run proved / unverified counts from the persisted run records. An Invariants page drives the deterministic outcome / behavioural checks from the browser.

Project-level assets

These are implemented project assets, not public distribution references. Authorized enterprise evaluations receive private, versioned CI delivery; there is no public Action, container image, or installer today.

Asset	Role
CI wrapper	An internal composite-action implementation: dry-run by default, with a profile input, that posts the proved-tier-first PR comment. It is dogfooded internally but has no public Action reference.
Published taxonomy spec	The public failure-mode taxonomy and findings JSON Schema (semver + changelog + contribution process), generated from the engine's source of truth.
Benchmark	A public, tool-agnostic Agent Failure Mode Benchmark: Python / TS / JS cases with hard negatives, a schema-based scorer, and misses-first results.
Module boundary	A documented split between the proprietary core and its surfaces, enforced by a CI guard that asserts the layering holds and the core stays encapsulated.

Persistence

Audit results and verifier output live under a per-workspace .truevectors/ directory (runs and per-run finding verification). Run identifiers containing path separators are rejected to prevent traversal.

Read more

• Overview — the platform and two products.
• TrueVectors Auditor — prove what an agent did.
• TrueVectors Hub — durable system of record.
• How it works — the per-file pipeline.
• Delivery and evaluation — authorized CI-first delivery and access.