Case study

Project Aegis

Agent oversight and evaluation

Summary

Problem: Audit a shopping experience with a browser agent without trusting the agent to grade itself.

Role
Product designer — design, implementation, and evaluation
Team
Solo
Timeline
One month

Outcomes

  • Measured

    8 of 8 agreement with golden labels on local test cases. — Not yet run against production merchants.

Product walkthroughs

Two scripted walkthroughs of the product: the shopping task the agent carries out, and the report it produces. They recreate the interface; they are not live recordings. I built them in code with Claude Code.

Mystery Shopper Agent

The shopper agent filters electricity plans on Lamplight, a demo marketplace I built for testing, opens the plan that matches its task, and stops there. It does not enroll, create an account, or enter personal data.

Paused. Press play to watch.

Agent Access Scorecard

The scorecard rates how ready a store is for AI agents across six signals. This example scores the Lamplight demo store at 45 out of 100. It is a score for the store, and is separate from the 8 of 8 evaluation result reported under Outcomes.

Paused. Press play to watch.

Context and stakes

A browser agent can walk through a shopping experience the way a customer would. The risk is letting that same agent decide whether the audit passed. I designed Aegis so the agent gathers evidence and a separate, rule-based verifier judges it.

I designed, built, and evaluated it alone over one month.

Where AI entered and why

The shopper agent browses the shopping experience. A separate deterministic verifier checks the audit evidence.

Three key decisions

Separate the shopper agent from a deterministic verifier.

Set hard stops before checkout, payment, and account creation.

Make the audit console read-only.

Trust and control design

The shopper agent has hard stops: it cannot proceed through checkout, payment, or account creation.

The audit console is read-only, so a reviewer can inspect a run but cannot alter it. It separates four things into their own views: what the agent was asked to do, what it did, the evidence it collected, and the verifier's independent checks. When a piece of evidence is missing, the console says so instead of hiding the gap.

Evaluation

Result. On eight local test cases, the verifier's verdict matched my reviewed answer key on all eight, and its stated reason matched on all eight. Measured.

What the cases cover. Checkout, payment, quote, a redirect to a site outside the allowed list, product match, wrong product variant, prompt injection, and account creation. Seven cases should pass and one should fail. Catching the failing case counts as agreement, so 8 of 8 means the verifier judged every case correctly, not that eight shopping journeys succeeded.

How I checked it. I re-ran the comparison offline against the saved test runs, using the project's own validator and scorer, and confirmed each saved run was the one the answer key referred to.

Limits. These are local test cases; Aegis has not been run against production merchants. The re-check compared saved records with the answer key. It was not a new browser run. When I reviewed the evidence in November 2026, five of the capture files referenced by the saved runs were no longer available, so the fidelity ratings are my original judgments and were not re-verified against the captures. I have not yet documented a failure, fix, and re-run cycle.

The source repository is private. I can walk through the test cases, answer key, and verifier rubric on request.

Outcome

Measured: 8 of 8 agreement with golden labels on local test cases. Not yet run against production merchants.