Assevra is not an eval dashboard and not an observability backend. It produces release evidence: a portable artifact — Markdown, JSON, and a self-contained HTML report — that you commit to git, attach to a pull request, hand to a security reviewer, and can still verify a year from now.
Five minutes, start to finish
pip install assevra
assevra demo # a full worked scorecard, offline
assevra scan --from traces.jsonl # score YOUR traces — nothing labeleddemo writes a complete worked scorecard — no clone, no API key, no network.
Open assevra-demo/scorecard.html and you have seen the whole product. scan
then does the same for traces you already have: five dimensions, no answer key.
See zero-label scoring, or
try it in your browser with no install at all.
Then point it at your own system:
assevra init --from traces.jsonl # detects your traces, framework and providers
assevra validate --strict # is every row actually labeled?
assevra run --gate # score, gate the build, write the artifactsWhat to read next
| If you want to… | Go to |
|---|---|
| Score real traces without labeling anything | Zero-label scoring |
| Try it with no install at all | In your browser |
| Get a scorecard for your own agent | Getting started |
| Understand what a “dimension” or “skipped” means | Concepts |
| Know exactly what each dimension checks | Dimensions |
| Stop typing flags | Configuration |
| Call it from Python instead of a shell | Python SDK |
| Wire it to LangGraph, Langfuse, Phoenix, OTel… | Integrations |
| Gate a build on it | CI & the GitHub Action |
| Parse the scorecard from another tool | Schemas |
| Prove a scorecard was not altered | Security & signing |
| Answer a security review | Governance mapping |
The three commitments
Everything in these docs follows from three rules. They are worth reading once, because they explain most of Assevra’s design decisions — including the ones that look inconvenient.
Deterministic before judge. If a property can be decided by a rule — a leaked SSN, a malformed tool call, a blown latency budget — decide it with a rule. Rules are reproducible, free, and auditable. Ask a model only for what genuinely requires judgment, and then pin and hash the rubric so the number stays comparable.
Report the interval, not just the mean. A bare 0.92 hides how few samples it
came from. Every dimension carries its sample size and a 95% Wilson confidence
interval, so nobody over-reads a small-sample move. On a small dataset the
interval is embarrassingly wide — that width is the honest statement of what
the number can support.
Skipped is not passed. A dimension whose engine was unavailable — no judge
configured, no detector installed — is reported as SKIPPED and does not gate.
It is never silently counted as a pass, because a missing measurement is not a
passing one.
What Assevra deliberately does not do
- It does not run your agent. You capture the outputs; Assevra scores them. That boundary is what makes it work with every framework instead of competing with them.
- It does not certify anything. A pass means the agent behaved on the dataset you gave it. Conformity determinations are made by auditors and authorities.
- It has no backend, account, or login. The artifact is the product.
Project
- GitHub · PyPI · DOI 10.5281/zenodo.21200852
- Roadmap — what is planned, and what is deliberately not
- Governance — the bar a methodology change has to clear
- MIT licensed. A personal open-source research project by Veera Ravindra Divi.