Agents say "done." We check the receipts.
Independent, reproducible audits of what AI agents actually did — proved with math, not LLM opinions.
TrueVectors is an independent research project on Agent Behavioral Reliability — the discipline of proving what an agent actually did against what it claimed. We publish audits of public AI-agent work — benchmarks, leaderboards, autonomous runs — and every finding ships with the evidence and the exact command to re-run it yourself. "It looks right" and "the tests passed" were never proof.
Proved in the wild
The first look said fine. The math disagreed — and showed its work. Recent audits on public AI-generated outcomes — reproducible, no API key, every offender named:
AI coding agents · SWE-bench Lite leaderboard
67% of "fixes" didn't fix
We re-checked 75 agents on the public SWE-bench Lite leaderboard. Of 21,593 patches submitted as bug fixes, 14,482 never actually resolved the issue — even the best agent left 119 unfixed. Claimed ≠ verified, universally.
AI model outcomes · Large Language Monkeys
28.78M attempts graded
Several models answered test questions 10,000 times each; every attempt was auto-graded. Only ~1 in 6 was correct overall, and 997 of 2,878 problems were never solved even once — genuine blind spots on the hard benchmarks.
AI security agents · AIRTBench
954 runs over budget
Of 8,066 autonomous red-team runs with a declared $10/run cap, 954 blew it — the worst single run cost $437.49. The checker summed the real cost column and named every over-budget run.
See all 14 public audits → — full receipts, plain-English findings, reproduce commands.
Why the findings hold up
A finding you can't check is an opinion. Ours are built to be checked.
Re-run it yourself
Every published audit links a public repro bundle — a standalone script that fetches the cited artifact and recomputes the finding. Verifiable by anyone, in minutes, with no TrueVectors software and no API key.
Only proved findings
If a check didn't prove it, it isn't in the public record. Hypotheses stay hypotheses — and what can't be decided is labeled undecidable, never green-stamped.
Right of reply & errata
Anyone we audit can respond, and the response is published alongside the finding. If we get one wrong, the correction is public and permanent — the errata page exists before it's ever needed.
The referee isn't selling the players. TrueVectors is an independent research project — unaffiliated with any model vendor, agent framework, or code-review product. We have no stake in any claim we test. Independence isn't a policy; it's the product.
The trust gap is the defining risk of the agentic era
Confidence scores dress up guesses — a number on a hallucination is still a hallucination. A passing check proves something ran, not that it did the right thing: the report that balanced by dropping rows, the agent that drifted off its policy, the change that gamed its own tests. And once an agent finishes, there is rarely a durable record of what it actually did. As agents take on more of the work, "trust me" does not scale — and "looks right" is how the worst failures ship.
One category, three kinds of outcome
Agent Behavioral Reliability is holistic by design. The same proof discipline applies wherever an agent produces something that can be checked against a declared rule — not just code.
AI-generated code
Does the change do what was claimed — or does it swallow errors, game its own tests, and quietly breach the scope it was given?
Model outcomes
Does a realized output — a report, a total, a pipeline result — obey the invariants it must hold, or were records silently dropped along the way?
Agent behavior
Did a recorded session stay inside its tool allow-list and spend budget — and actually deliver what it said it delivered?
Did the agent do what it said?
The review question changed — from "does this look right?" to "did the agent actually do what it said?" We answer it with proof: the AI suggests, the math decides — and every verdict is stamped with where it came from, so a proof is never confused with a guess.
Compliance-ready evidence
Every audit is sealed into a tamper-evident, signed ledger, and high-severity unverified findings can be gated on a recorded human review. We map these to the frameworks your program already reports against — as evidence you consume, not a certification we issue.
Tamper-evident attestation
Each run is sealed into an append-only, hash-chained ledger and signed with a local Ed25519 key — verifiable offline, no server, no trust required.
Enforceable human oversight
An opt-in CI gate in the EU AI Act Art. 14 style: critical / high findings the engine could not prove must carry a recorded reviewer disposition before the check passes.
Standards crosswalks
Control mappings with explicit gap columns for EU AI Act (Arts. 12–14), ISO/IEC 42001, NIST AI RMF, and SOC 2 — written for your auditor, honest about what we do and don't cover.
EU AI ActISO/IEC 42001NIST AI RMFSOC 2
TrueVectors is not a SOC 2 report, an EU conformity assessment, an ISO/IEC 42001 certificate, or a correctness guarantee — it produces the artifacts your compliance program relies on. See the compliance evidence →
This is how AI earns trust
Agent Behavioral Reliability turns "trust me" into "here's the proof" — the missing layer that lets autonomous systems be adopted where being wrong is not an option. Follow the record, check the work, and hold us to the same standard we hold the agents to.