Software Engineer @ Microsoft · Agent Reliability

Measuring whether agents are right — and stopping them before they're wrong.

I build the measurement and enforcement layer for AI agents — the part that tells you whether an agent's answer is actually correct, and the part that blocks a dangerous action before it happens.

Most "agent safety" today is an LLM grading another LLM after the fact. That's a wobbly sensor bolted on after the money already moved. My work is the opposite: deterministic checks that gate, calibrated judges that only report, and evidence ranked by how hard it is to forge.

🏢 Microsoft · Redmond, WA

Two questions, two pillars. Everything below serves one of them.

One principle ties them together: rank evidence by independence, not cost. A document-ID match or a tool-argument check means the same thing every run — so it gates. An LLM judge wobbles between runs — so it only reports. Never let a number that flips a coin block a merge or a payout.

A safety gate that says "no" before an agent does something dangerous, not after. Rust core (PyO3), checked before every tool step in tens of nanoseconds, with an identical verdict every time. Drop-in adapters for LangGraph, CrewAI, AutoGen, LlamaIndex.

Enforces structural facts a judge can't reliably see: prerequisites (ExecuteRefund requires prior VerifyIdentity), runaway-loop detection, depth/token budgets, tool-argument schemas.

 gate-agent  vs  LLM-as-a-judge   —  same unsafe refund trace
------------------------------------------------------------------
 Blocks unsafe refund          100% every run        62% of runs
 Unverified refunds leaked            0.0%                 37.5%
 When it decides             BEFORE the refund       AFTER the refund
 Verdict                        deterministic        wobbles (SD~0.20)
 Latency / step                    ~9 µs                  ~700 ms
------------------------------------------------------------------

Why it matters: no evaluation run afterward can un-refund $5,000. Safety has to be a pre-execution decision.

A zero-dependency TypeScript toolkit for evaluating agent output — in tests, on production transcripts, and as a CI gate. Built on an independence-first tier pyramid:

Most agent failures (crashes, stale runs, format breaks, hallucinated paths) are caught by Tier 1+2 alone. Gate vs. grade: a check either gates (binary) or grades (0–1). A low grade is information, not a failure — never coerce one into the other.

A .NET 8 reference implementation of three-tier evaluation, and the repo where I make the case that calibration is non-negotiable — with numbers from actually running it:

• Split the RAG judge into Retrieval / Groundedness / Relevance. Never average them. One blended "quality score" hides the exact failure you need to fix.
• Calibration caught the Groundedness judge scoring outright fabrication at exactly 3.0 every time while also penalizing good answers — two errors that cancel into a healthy-looking −0.17 bias. Moving the floor 3.0 → 3.5 lifted band agreement 50% → 75%.
• The Retrieval judge returned 5, 2, 4, 5, 2 on identical input — 17% of merge decisions would flip at random, so an exact expectedChunkIds check does the gating and the judge only advises.
• The judge costs ~250× the agent — measuring quality is the entire bill, so caching and counting judge calls is the real engineering.

The thesis: a threshold picked without calibration is just taste.

Where the reliability work gets battle-tested — real agents and prompt tooling to point the evals and gates at.

_{Full portfolio — languages, distributed systems, viz — in PROJECTS.md.}

• Agent evaluation & judge calibration — why a single quality score lies, and how to measure the measurer
• Deterministic pre-execution safety — enforcing invariants on the hot path instead of grading traces after the fact
• AI agent identity, accountability & self-replication safety — governing autonomous agents
• Published in IEEE and Springer · Chair, ICGIS 2026 · 📄 Papers