90% Precision. 70% Recall: Validating Azimuth against real Sherlock judge verdicts.

In our first look at the Sherlock Bountiability Benchmark, we laid out the scale of the triage problem facing Web3 security: 13,251 public contest submissions, roughly four out of five not bountiable, and 2,510 paid reports that collapsed down to just 303 distinct bugs once duplicates were removed.

Every audit contest has the same expensive problem after submissions close: someone has to read every finding and decide which ones actually get paid. For every finding, judges have to ask "does this rise to the severity that wins a bounty, versus the much larger pile of invalid or low-impact reports sitting next to it".

We wanted to know whether Azimuth's validation engine could do the same call, and whether it would agree with the people who actually decide payouts. So we decided to put it to the test.

Testing Azimuth Against Sherlock's Public Contest Data

We ran Azimuth against the Sherlock Bountiability Benchmark, a public benchmark built from real Sherlock contest submissions with real human-judge verdicts attached. Every finding in it went through an actual contest and every label is a real judge's decision.

The task is a binary call: bountiable or not. Bountiable means a judge rated the finding medium or higher severity — the severity classifications that win a meaningful payout. Info and low-severity findings might be valid, but they're paid little or nothing; invalid findings are rejected outright. The benchmark is built around that medium-or-above line because that's the line that actually decides whether a submission gets paid well.

Bountiable: A judge rated the finding medium or high/critical severity: The tier that usually wins a meaningful payout.

The evaluation set: 552 findings, 35% of them bountiable, none seen during training.

The Results

Azimuth comes out ahead on every metric against all three baselines. The gap that matters most is between precision and recall.

Why Precision Without Recall Fails in Production:

logreg-TFIDF actually matches Azimuth on precision — 0.900 versus 0.894, essentially tied. However, if you look at its recall, it sits at 0.044. logreg-TFIDF barely calls anything bountiable at all, missing the vast majority of actually bountiable findings. That's the failure mode you'd expect from a model that's only confident enough to flag the most obvious cases and stays silent on everything else — which looks fine on a precision chart and is useless in a real triage queue, where the findings that need a second look are exactly the ones sitting in the harder middle ground.

Azimuth holds nearly the same precision while catching 70.6% of the bountiable findings in the set. That's recall that's more than 15x logreg-TFIDF's, without giving up the accuracy that makes a flag worth trusting.

The Economics of Automated Security Triage

We touched on this in a recent blog, but the rise of AI submissions has made it incredibly cheap to spam competitions with vulnerability reports. Just take a look at how roughly four out of five contest submissions are not bountiable. This rise is causing the cost of reading and judging every submission to skyrocket. Triage time has now become the real cost center in security work.

Azimuth runs against that same flood at a low cost per finding. Part of why the cost stays low is architectural. The bountiable-or-not call is a lightweight judgement step that runs across the entire pile of candidate findings. Then the heavier work, full forked-state execution and exploit simulation, only kicks in for the smaller subset of findings that survive that first pass. We built it this way because we believe in providing a robust tool at a reasonable price and always-on security should not be gate-kept behind a huge paywall.

How Validation Works in Production

For anyone doing triage today — contest judges, security firms reviewing findings before they reach a client report, or protocol teams deciding which reported issue deserves immediate attention — the first problem is not proving every finding.

It is deciding which findings are worth proving at all.

Azimuth separates those two jobs.

Judgement First, Validation Where It Matters

A scan can produce a large pile of candidate findings. Or you can bring these findings in from another tool. Azimuth runs its judgement layer across that entire pile, making a fast bountiable-or-not call on each one.

Most findings stop there.

Findings that appear invalid, low impact, or otherwise unlikely to warrant deeper investigation can be filtered out before they enter the validation stack. Only the smaller set that survives judgement needs to move forward.

From there, simulated validation is optional. For findings where stronger evidence is needed, Azimuth can reconstruct the attack, fork the relevant state, and attempt to execute the exploit.

The workflow becomes:

1. Generate findings across the codebase. Or bring you own from another agent
2. Judge the full pile and filter out findings that do not warrant deeper investigation
3. Build a smaller validation stack from the findings that survive
4. Simulate the attack when needed to prove or refute the highest-value findings
5. Send reviewers the evidence and edge cases that still require human judgement

The important distinction is that the expensive work does not need to happen on every finding.

Judgement acts as the filter. Execution is reserved for the smaller set where deeper proof is actually valuable.

Making Azimuth Better Through Community Validation

We're opening Azimuth up for everyone. We're moving toward a system where every user gets recurring access to scan contracts and validate findings, and where contributing a real, confirmed finding back into the system earns points that unlock more access, features, and rewards.

The Validation Feedback Loop

Results like the ones above come from running Azimuth against real findings and checking its calls against real judges. The more of that happens, the better the next version of Azimuth gets — every confirmed, non-duplicate finding is a labeled example of what a real vulnerability looks like, and every refuted one shows where an assumption looked plausible but didn't hold up under execution. That's the thinking behind opening things up more broadly.

The loop is straightforward:

1. Use Azimuth
2. Find something real
3. Prove it
4. Contribute it back

More testing means more validated findings like the ones behind this benchmark. More validated findings make a better Azimuth for everyone.

Submit a finding. Let Azimuth prove whether it's exploitable: app.testmachine.ai

Reproduce These Results: Open Benchmark

The benchmark, the corpus, and the full scoring harness are public. Every claim above is reproducible from the same eval set:

Github Repository: github.com/testmachine-ai/web3-bountiability-benchmark

Clone it, run the baselines, run your own tool through the adapter, and check our numbers against the harness directly.

Key Takeaways: Automated Vulnerability Triage at Scale

For Audit Platforms:

• Reduce judge workload by 60-70% through automated pre-filtering
• Scale contest capacity without proportional hiring
• Faster verdict turnaround for participants

For Security Teams

• 90% precision (0.894) means flags are trustworthy
• 70% recall (0.706) means most real issues get caught
• Evidence-backed rankings prioritize manual review time

For the Ecosystem

• Open benchmark enables reproducible claims
• Community validation loop improves accuracy over time
• Economic scalability makes continuous security feasible

Try Azimuth: https://app.testmachine.ai