Case study
Project Aegis
Agent oversight and evaluation
Summary
Problem: Audit a shopping experience with a browser agent without trusting the agent to grade itself.
- Role
- Product designer — design, implementation, and evaluation
- Team
- Solo
- Timeline
- One month
Outcomes
- Measured
8 of 8 agreement with golden labels on local test cases. — Not yet run against production merchants.
Product walkthroughs
Two scripted walkthroughs of the product: the shopping task the agent carries out, and the report it produces. They recreate the interface; they are not live recordings. I built them in code with Claude Code.
Mystery Shopper Agent
Paused. Press play to watch.
Agent Access Scorecard
Paused. Press play to watch.
Context and stakes
A browser agent can walk through a shopping experience the way a customer would. The risk is letting that same agent decide whether the audit passed. I designed Aegis so the agent gathers evidence and a separate, rule-based verifier judges it.
I designed, built, and evaluated it alone over one month.
Where AI entered and why
The shopper agent browses the shopping experience. A separate deterministic verifier checks the audit evidence.
Three key decisions
Separate the shopper agent from a deterministic verifier.
Set hard stops before checkout, payment, and account creation.
Make the audit console read-only.
Trust and control design
The shopper agent has hard stops: it cannot proceed through checkout, payment, or account creation.
The audit console is read-only, so a reviewer can inspect a run but cannot alter it. It separates four things into their own views: what the agent was asked to do, what it did, the evidence it collected, and the verifier's independent checks. When a piece of evidence is missing, the console says so instead of hiding the gap.
Evaluation
Result. On eight local test cases, the verifier's verdict matched my reviewed answer key on all eight, and its stated reason matched on all eight. Measured.
What the cases cover. Checkout, payment, quote, a redirect to a site outside the allowed list, product match, wrong product variant, prompt injection, and account creation. Seven cases should pass and one should fail. Catching the failing case counts as agreement, so 8 of 8 means the verifier judged every case correctly, not that eight shopping journeys succeeded.
How I checked it. I re-ran the comparison offline against the saved test runs, using the project's own validator and scorer, and confirmed each saved run was the one the answer key referred to.
Limits. These are local test cases; Aegis has not been run against production merchants. The re-check compared saved records with the answer key. It was not a new browser run. When I reviewed the evidence in November 2026, five of the capture files referenced by the saved runs were no longer available, so the fidelity ratings are my original judgments and were not re-verified against the captures. I have not yet documented a failure, fix, and re-run cycle.
The source repository is private. I can walk through the test cases, answer key, and verifier rubric on request.
Outcome
Measured: 8 of 8 agreement with golden labels on local test cases. Not yet run against production merchants.



