agent-reliability-harness
A refund-support agent built on one idea: the LLM expresses goals, deterministic guardrails enforce policy, and every real failure becomes a permanent regression test.
The separation that matters
The model is trusted to say what the customer wants, never to decide whether it is allowed. Policy lives in plain Python, and denial messages are templated from the guardrail's reason code so a customer never gets a hallucinated explanation.
Proof it is tested, not demoed
Hypothesis states each invariant (an already-refunded order is always denied) and tries 200 random inputs per property to break it. A 17-phrasing sweep checks how real customers talk.
Two real bugs, caught by the loop
A partial refund request issued the full amount because dollar-amount parsing was missing. It was annotated into a regression test before touching the fix. And the phrase "What's your return policy?" triggered a refund, because a bare keyword matched with no context.
Stack
Python · Hypothesis · OpenAI API · Pytest
Honest notes
There was no LLM API key, so a deterministic rule-based client drives every test. A real OpenAI client is implemented behind the same interface. Language understanding of a live model is not claimed as tested.