Open source · release evidence for AI agents
Assevra turns agent test runs into signed, statistically defensible scorecards that gate every release. Not a dashboard behind a login — one portable artifact you commit, attach to a PR, and can still verify a year later.
pip install assevra && assevra demoA complete worked scorecard in two commands — no clone, no API key, no network.
| Dimension | Score | Result |
|---|---|---|
| Groundingllm-judge | 0.94 | PASS |
| Safety / refusalllm-judge | 1.00 | PASS |
| PII-leakdeterministic | 1.00 | PASS |
| Tool callsdeterministic | 0.98 | PASS |
| Prompt injectiondeterministic | 1.00 | PASS |
The difference
Assevra asks the only question a release meeting cares about: can I safely ship this agent? Everything below follows from that one change of question.
One self-contained file — Markdown, JSON, and styled HTML — that you commit to git, attach to a PR, or mail to a reviewer. Sign it with Ed25519 so anyone can verify it was produced by you and never altered.
assevra sign · verifyA bare "0.92" hides how few samples it came from. Every dimension carries its sample size and a 95% Wilson interval — and on a small dataset that width is the honest statement of what the number can support.
95% Wilson CIYou scan for a leaked SSN; you do not ask a model whether it leaked one. Seven of the nine dimensions are rules — so most runs are free, reproducible, and identical on every machine, including a fork with no secrets.
7 of 9 need no modelA dimension whose engine was unavailable is reported as SKIPPED and does not gate. It is the semantic that stops a build staying green for three months after CI quietly lost its API key.
fails closedThe scorecard, Agent Card, calibration report and dataset row all have versioned JSON Schemas served from this domain. Within major version 1, fields are only ever added — never removed or repurposed.
schema/v1Anthropic, OpenAI, Azure, Bedrock, Gemini — or any OpenAI-compatible local endpoint, with no third-party package and no data leaving the machine. A jury can span vendors, and disagreement is reported as signal.
7 judge providersFive minutes
The demo dataset ships inside the package, so the first artifact you see is real output from the real engine — not a screenshot.
pip install assevraZero-dependency core. No account, no backend, nothing to log into.
assevra demoA complete worked scorecard — HTML, JSON, Agent Card — with no clone, no API key and no network.
assevra init --from traces.jsonlDetects your traces, framework and providers, then writes the config, a drafted dataset and a CI gate.
assevra run --gateScores, gates the build, and writes the artifact you attach to the release.
How it works
Six commands, no backend. Deterministic where it can be, judged where it must be, and always reporting the interval — not just the mean.
Draft a dataset from traces you already have
LABELED / UNLABELED / INVALID — before scoring
Score every dimension, with its interval
Prove the judge agrees with humans (κ ≥ 0.85)
Seal the scorecard with Ed25519
Map the evidence to governance frameworks
What it measures
The verdict is a conjunction — one leak sinks the run. Every dimension ships a definition, a scoring method, a stated threshold, a confidence interval, and a stated limit.
Is every factual claim traceable to the provided context, or invented?
Does it refuse what it must — and answer what it should? Over-refusal is a failure too.
Does personal data escape into an output, outside the field it was sanctioned to echo?
Are the facts a correct completion requires actually present?
Was every call well-formed, permitted, and complete — arguments, types, enums and all?
Did it do the right thing? A perfect refund call is still a catastrophe if it should have escalated.
Did it resist instructions planted in retrieved content, tool output, or another agent?
Did each run stay inside its cost budget? Priced from your table, not guessed.
Did each run finish in time? Reported as a pass rate with p50/p95, because averages hide the tail.
Plus pass^k and run-to-run consistency over repeated trials. A dimension scoring 0.667 with consistency 0.000 is not a quality problem — it is an agent that never behaves the same way twice, and that needs a different fix.
In your build
One step. It validates the dataset, scores it, writes a summary and the failing rows to the job summary, uploads the artifacts, and fails the build when a dimension drops below its threshold.
The env: block is optional. Without a key, the deterministic dimensions still gate and the judged ones report as SKIPPED — never as passing. That is what lets a pull request from a fork get a real signal instead of a red build.
- uses: assevra/assevra@v1 with: dataset: evals/agent.jsonl gate: true strict: true attest: true env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
Fits what you already run
That boundary is why it works with every framework instead of competing with them. One command prints the wiring for the tool you already use.
$ assevra integrate langgraphJudge providers
A local OpenAI-compatible endpoint needs no third-party package and sends nothing off the machine.
Worked examples
A real agent, a scorecard that fails, the exact rows that failed, the fix, and the scorecard that passes. Every number in them came from running the datasets in the repo.
An indirect prompt injection arriving through a product review — nothing the user did wrong, and nothing input validation would have caught.
Open the walkthrough →tool-call · action · cost · latencyA malformed call and a wrong decision look identical in a trace and are completely different problems. Also: 41,000 tokens to answer “where is my order?”
Open the walkthrough →pass^k · handoffs · agent-to-agent injectionA dimension scoring 0.667 with consistency 0.000 is not a quality problem. It is an agent that never behaves the same way twice.
Open the walkthrough →Verifiable evidence
A shared HTML file is convenient; a signed one is evidence. Pin the maintainer's public key to confirm a scorecard was produced by them and not altered — verification fails if a single byte changed.
Security & signing →dEcTKT/9ThXewTjRdBm2qyGIH69Ghy08kVuB19AJnSg=$ assevra verify \
--scorecard scorecard.json \
--signature scorecard.sig.json \
--public-key <key>Install once, run offline, commit the artifact. No backend, no login, MIT-licensed.
Cite Assevra
Divi, Veera Ravindra. Assevra: A Reliability Scorecard for LLM Agents, v0.4, 2026.
DOI 10.5281/zenodo.21200852