In September 2025, Zellic released V12, an autonomous AI auditor built to enhance security teams — not replace them. When Code4rena started publishing V12's findings directly in contest repos, we had a benchmark we couldn't ignore. We ran Azimuth on K2, the same codebase V12 had already scanned, and compared the output head-to-head.
K2 is a decentralized borrow/lend protocol built on Stellar's Soroban platform — an Aave V3-style architecture adapted for Soroban's constraints. It was a $135k contest, with V12's findings published upfront and ruled out of scope for warden awards.
One important framing note before the data: this is a purely AI-to-AI comparison. The K2 contest has closed but warden findings haven't been published yet. No human auditor consensus exists to validate either tool's output against. What we're measuring is how much the two AI tools independently agreed on, not ground truth.
What We're Comparing
V12 is Zellic's AI auditing tool. For the K2 contest, Code4rena published V12's output across three files: Critical, High, and Medium/Low findings. V12 submitted 106 findings total and self-validated 85 of them as real issues.
We ran Azimuth on the same codebase twice: once in late April, and again this week after improvements to our underlying model. The latest scan generated 64 findings.
The comparison below measures how many of V12's findings Azimuth independently identified. Do note these findings are not yet confirmed bugs. K2's contest is still live and warden results haven't been published yet, so neither tool's output has been externally validated.
Methodology
We matched Azimuth findings against V12's output by root cause and affected contract. A match required the same underlying vulnerability, not just the same function name or severity label.
We track two sets:
- V12 self-validated (85) — findings V12 marked as real. The stronger benchmark.
- All V12 submitted (106) — everything V12 flagged, including lower-confidence findings.
Results: V12 Self-Validated Set (85 findings)
| Severity | Azimuth matched | Recall |
|---|---|---|
| Critical | 3 / 3 | 100% |
| High | 24 / 26 | 92% |
| Medium | 17 / 20 | 85% |
| Low | 11 / 36 | 31% |
| Total | 55 / 85 | ~65% |
Full critical coverage. 92% of the highs V12 stood behind. The 2 highs we missed involved a system invariant violation that required context from the protocol docs — a known limitation when documentation isn't fed explicitly into the scan.
Azimuth matched every single critical finding V12 identified — 3 for 3. In smart contract security, critical vulnerabilities represent the highest-severity exploitable bugs: the ones that can drain funds or break protocol invariants entirely. Missing one is the difference between a protected protocol and a headline exploit. 100% recall at the critical tier is the number that matters most.
One notable high found by both tools found that liquidations enforce a close-factor cap but never check that the borrower's health factor improves, so liquidators can repeatedly worsen an unhealthy position, collecting a bonus each time while pushing the account into bad debt.
The Trade-Off: Specificity vs. Sensitivity
Azimuth generated 64 findings. V12 submitted 106.
That difference isn't a coverage gap — it's a design trade-off. V12 optimizes for sensitivity: flag more, let the judge sort it out. Azimuth optimizes for specificity: submit findings with higher confidence of being real.
On the self-validated set — the findings V12 itself treated as real — Azimuth matched 65% overall and 92% of the highs. That's the relevant comparison: how well does Azimuth cover what V12 actually believed in?
Neither approach is strictly better. High sensitivity catches more at the cost of noise. High specificity produces a tighter, lower-overhead report. Which matters more depends on how a team uses the output.
What This Data Doesn't Settle
K2's contest closed May 27, however, Warden findings haven't been published yet. That means:
- We don't yet know which V12 findings wardens independently confirmed
- We don't know how many of V12's 106 submissions survive judging
- We don't know whether the 2 highs Azimuth missed are findings the wardens flag
We'll revisit this comparison once the contest report is public and there's a third data point to triangulate against.
Summary
| Azimuth | V12 | |
|---|---|---|
| Findings submitted | 64 | 106 |
| Critical recall (V12 self-validated) | 100% | — |
| High recall (V12 self-validated) | 92% | — |
| Overall recall (V12 self-validated) | ~65% | — |
| Approach | Specificity-first | Sensitivity-first |
Azimuth covered all 3 criticals and 24 of 26 highs that V12 validated in K2 — with fewer total findings.
K2 is one data point. But the pattern it surfaces is worth noting: on the findings V12 itself stood behind, Azimuth matched nearly all the high-severity issues while generating significantly fewer total flags. A tighter report means less triage time for the teams using it.
The Warden results will tell us more. When C4 publishes the K2 report, we'll have a third benchmark — what human auditors found independently — to measure both tools against.