Agent evaluation. From FAILURE → verified FIX.
Turn captured agent runs into a clear decision: what failed, what to change, and whether the fix holds.
Try a sample scorecard in under 5 minutes.
Get your first scorecard →Explore a failure and its fix →No signup. No installation. Use the sample or drop in your own trace. Browser scans provide initial triage; full release checks require setup.
FAIL · ACTION CORRECTNESS
An unauthorized request triggered a refund. The workflow should have escalated it.
4 cases · 3 runs per case · action, tool contract, and PII checks. A local fixture, with no model calls or customer data.
Start with the failure your team cannot afford
Catch unsupported policy claims, leaked identifiers, and instructions hidden in retrieved content.
Then act: Review sources, repair the retrieval or response policy, then rerun labeled cases.
Explore the evaluation approach →Catch unauthorized refunds and incorrect final state—even when the tool call is well-formed.
Then act: Move authorization before the side effect and verify the resulting order state.
Explore the refund walkthrough →Check captured runs against cost, latency, and correctness requirements before shipping a change.
Then act: Compare the full suite against a compatible, reviewed baseline in CI.
Explore the evaluation approach →Nine dimensions. One inspectable decision.
Inspect what the agent said, what it did, and what the run cost. Choose the checks your release actually requires.
Showing all 9 dimensions
Does the answer follow from the sources?
Compare claims with captured context using a model judge.
Explore groundingDoes it refuse the right requests?
Check both required refusals and permitted responses against labels.
Explore safetyDid private information escape?
Detect supported sensitive patterns, with optional Presidio coverage.
Explore piiDid the answer include what was required?
Check explicit required strings; use outcome assertions for business state.
Explore task_completionWere the arguments valid and permitted?
Validate full tool schemas, allowed calls, and expected arguments.
Explore tool_callDid the workflow do the right thing?
Assert action order, forbidden side effects, and observed final state.
Explore action_correctnessDid untrusted content redirect the agent?
Evaluate labeled injection cases with canaries or a configured judge.
Explore injectionDid the run stay within budget?
Use recorded cost or token usage with your explicit price table.
Explore costDid the workflow finish in time?
Compare recorded duration with your budget and inspect timing summaries.
Explore latencyModel judgments require calibration against representative human labels. Detector and rule coverage depend on the configuration; a passing fixture does not establish general reliability.
A scorecard should start the next step.
Import traces, evaluate Python records, or capture a command. Declare the outcomes and dimensions that matter to your workflow.
See the assertion, case identity, trace reference, and suggested next action. Evaluator errors stay separate from agent failures.
Repair the prompt, tool boundary, retrieval, or workflow. Suggestions guide your investigation; they do not modify your agent.
Check the same inputs and policy. Required missing checks block release; compatible history enables regression comparison.
Start with a working example
Run the included refund workflow before and after its authorization fix. Open both reports, inspect the findings, then adapt the assertions to your own agent.
Python 3.10+. Installation downloads packages. This example runs locally without API keys.
macOS / Linux · Run each step in order
Use the same folder for every step. Wait for each command to finish successfully before continuing.
python3 --version
python3 -m venv .assevra-venv
.assevra-venv/bin/python -m pip install assevra==0.6.0
.assevra-venv/bin/python -m assevra.reference --out-dir reference-output
If python3 is missing or older than 3.10, install Python, reopen your terminal, and start at step 1. On Linux, if creating the environment fails, install your distribution’s Python venv package first.
Prefer no setup? Try the browser sample →
Open reference-output/after/scorecard.html
Fits around your agent pipeline
Assevra evaluates captured evidence. Start with an adapter or recipe, then supply your business assertions. Integration types are listed explicitly below.
| Your stack | Available today | How it helps |
|---|---|---|
| Python / command line | SDK + recorder | Evaluate records in memory, or capture a command you provide. |
| LangGraph | Capture recipe | Retain tool calls across the message history and label workflow outcomes. |
| OpenAI Agents SDK | Capture recipe | Export run items, tool calls, and final output to the generic adapter. |
| OpenTelemetry / Phoenix | Trace adapter | Read supported OTLP and OpenInference exports with trace and span identity. |
| Langfuse | Export recipe | Bring observations into a local evaluation dataset. |
| MCP / OpenAI / Anthropic tools | Contract import | Check recorded arguments against your tool definitions, including nested constraints. |
Recipes and format adapters are covered by local tests. Hosted customer environments require your own smoke test. These are integrations, not vendor partnerships.
Choose an integration →Evidence with boundaries
Dimension pass rates include sample sizes and 95% Wilson intervals. Required checks cannot disappear behind a green headline. A scan is TRIAGE; a mock evaluator run is SELF_TEST.
Choose thresholds and coverage for your use case. Validate model judges against human labels before relying on their judgments.
Read the methodology ↗Open source, portable by design
Keep reports in your repository or attach them to a release. JSON schemas are versioned. Optional Ed25519 signatures protect canonical scorecard JSON; trust depends on the signing key you verify.
Cloud judges receive the fields in their evaluation prompts. Use deterministic checks or a configured local judge when your data must stay local.
Read the documentation →More than a single score
See sample sizes and Wilson intervals alongside dimension pass rates. Use repeated trials to inspect consistency and pass^k.
Measurement methodology →Use Anthropic, OpenAI, Azure, Bedrock, Gemini, or a compatible local endpoint. Panel disagreement remains visible.
Judge calibration →Sign canonical JSON with Ed25519. Share the portable report and verify it against a trusted public key.
Signing and data flow →Generate an Agent Card with indicative mappings to NIST AI RMF, EU AI Act, ISO/IEC 42001, and OWASP control families.
Explore Agent Cards →Governance mappings help reviewers locate evidence; they are not a compliance determination or certification.
No. Export supported traces or use the Python SDK and recorder. Assevra adds evaluation and release evidence around the runs you already capture.
Inspect the case, trace, and acceptance criteria. Review the suggested action, change the agent, execute it again, and rerun the complete declared suite. Follow the guided example.
Deterministic checks run locally after installation. A configured local model endpoint can handle model judgments. Cloud judges receive the fields in their rubric prompts.
No. Scans are TRIAGE and mock runs are SELF_TEST. A release PASS requires completed evidence for every included and required dimension.
Yes, under the MIT license. The code, methodology, versioned schemas, and release archives are public. You can extend scorers and reporters through the Python SDK.