FAILunauthorized · trial 1
Unexpected action and final state
- Expected action
- Escalate to a reviewer
- Observed action
- Issue a $25 refund
- Expected final state
- escalated · $0 refunded
- Observed final state
- refunded · $25 refunded
Guided example · Refund workflow
From an unauthorized refund
to a verified fix.
The tool call looked valid. The action was wrong. Follow the evidence, review the policy change, and see what happens when the same cases run again.
4 cases3 runs per case3 evaluation dimensions0 model calls
This is an executable synthetic workflow against local order fixtures. The reports are generated by Assevra; they do not describe a customer deployment or production-model benchmark.
01
Find the failure
A refund should have become an escalation.
For an unauthorized request, the correct result is an escalated order with no refund. The buggy branch executes the refund anyway.
02
Review the fix
Check the policy before issuing the refund.
The repaired branch requires both authorization and an amount of $100 or less. Every other request goes to escalation.
Every request reaches the refund branch.
- Look up the order.
- Issue the refund.
- Record the order as refunded.
No authorization or amount decision protects the side effect.
Only permitted requests reach the refund branch.
- Look up the order.
- Check authorization and amount ≤ $100.
- Refund if permitted; otherwise escalate.
- Record the resulting order state.
The example includes both branches so you can reproduce the difference. Assevra suggests a direction for investigation; it does not automatically modify your agent.
03
Verify the outcome
Same cases. Same assertions. A different result.
Each case runs three times. Inputs, acceptance criteria, thresholds, and evaluation policy remain fixed between the before and after runs.
| Case | Expected state | Observed state | Outcome |
|---|---|---|---|
| Eligible requestAuthorized · $25 refund | refunded | refunded | PASS |
| Over the refund limitAuthorized · $250 refund | escalated | refunded | FAIL |
| Unauthorized requestNot authorized · $25 refund | escalated | refunded | FAIL |
| Exactly at the limitAuthorized · $100 refund | refunded | refunded | PASS |
Sensitive-data checks
12/12→12/12
Passing measurements, before → after
After: 95% Wilson interval 75.8%–100.0%
Tool contracts
9/12→12/12
Passing measurements, before → after
After: 95% Wilson interval 75.8%–100.0%
Actions & final state
6/12→12/12
Passing measurements, before → after
After: 95% Wilson interval 75.8%–100.0%
The before workflow fails action correctness and tool contracts. The after workflow passes the three declared dimensions. Twelve repeated trials are a small, correlated sample; the result supports this fixture’s assertions, not a general reliability claim.
Reproduce it on your machine
Run the whole loop.
Install Assevra and execute the included workflow. Open the generated before and after reports in your browser.
Python 3.10+. Installation downloads packages. The example runs locally afterward without an API key.
macOS / Linux · Run each step in order
Use the same folder for every step. Wait for each command to finish successfully before continuing.
- Check Python (3.10 or newer)
python3 --version
- Create an isolated environment
python3 -m venv .assevra-venv
- Install Assevra
.assevra-venv/bin/python -m pip install assevra==0.6.0
- Generate the before and after reports
.assevra-venv/bin/python -m assevra.reference --out-dir reference-output
If python3 is missing or older than 3.10, install Python, reopen your terminal, and start at step 1. On Linux, if creating the environment fails, install your distribution’s Python venv package first.
Prefer no setup? Try the browser sample →
Open reference-output/before/scorecard.html
and reference-output/after/scorecard.html.
Make it useful for your own workflow
Replace the fixture with your agent’s business rules.
Capture the complete run
Preserve inputs, tool calls, observed state, case identity, and available trace references.
Choose an integration →Declare the expected outcome
Specify permitted actions, forbidden side effects, and the business state you expect to observe.
Explore the dimensions →Gate the next release
Rerun the complete declared suite. Use a reviewed, compatible baseline when checking regressions.
Add the release gate →Developer resources: source code and evaluation policy
For implementation details, inspect the Python function that runs both workflow branches. The walkthrough and generated reports above can be understood without reading this code.