We recently ran Azimuth on EVMBench Detect and it received a 78.6% recall, placing it top of the leaderboard. However, in citing the limitations of EVMBench Detect, we noted that it does not prove exploitability. So to put a number on Azimuth's exploitability, not just its detection, we ran it against EVMBench Exploit.
OpenAI and Paradigm’s EVMBench Exploit asks agents to execute end-to-end fund-draining attacks against contracts deployed on a sandboxed blockchain. EVMBench Detect asks the question “Can the agent find this bug”. EVMBench Exploit asks “Can the agent actually drain the funds”.
Azimuth achieved a 91% validation success rate, validating 20 out of 22 findings with automated validation scripts.
EVMBench Exploit Break Down
EVMBench Exploit asks a narrower, more demanding question than Detect: can an AI agent not just identify a vulnerability, but actually carry it out, end-to-end, against a live chain? In Exploit mode, vulnerable contracts are manually deployed to a local instance of the Ethereum blockchain. The agent receives an RPC endpoint, a funded private key, and any metadata it needs, and from there it's on its own — analyzing the chain, deploying helper contracts if required, and crafting and sending the transactions that actually drain the funds.
Grading is programmatic: custom per-vulnerability scripts re-execute the agent's transactions and check the resulting chain state via transaction replay and on-chain verification. There's no ambiguity — either the funds moved, or they didn't.
That's the real difference between the two modes. Detect asks whether a vulnerability was named in a report. Exploit asks whether it was actually realized as a working attack.
How Azimuth was tested
Standard EVMBench Exploit harness. Each task runs on a sandboxed Anvil chain. Azimuth receives the RPC endpoint, a funded private key, and the metadata bundle defined by the benchmark — nothing else.
Agent operates end-to-end. Azimuth analyzes the deployed bytecode and source, forms a hypothesis, deploys any helper contracts it needs, and submits the transactions intended to drain the target.
Programmatic grading. Custom per-vulnerability scripts replay Azimuth's transactions and verify the resulting chain state. No human reviewer — pass/fail is determined entirely on-chain.
How Azimuth Achieved 91% on EVMBench Exploit
Azimuth validated 20 of an initial batch of 22 findings — a 91% success rate. Validation here is programmatic: transaction replay and on-chain verification confirm whether the agent's transactions actually moved the funds. No judge, no report to interpret — the transactions either drained the contract or they didn't. 91% of the time, Azimuth did.
We then extended validation further, bringing total coverage to 56 findings across the dataset — same engine, same standard of proof, just broader coverage across more vulnerability classes.
The Limitations of EVMBench Exploit
Exploit mode is a harder bar than Detect, but it isn't a complete picture of real-world exploitability either. A few structural constraints are worth naming, because they shape what a result here can and can't claim.
Timing is one. Agent transactions replay sequentially in the grading container, and the harness doesn't reproduce the original timestamps exactly. Any exploit that depends on precise timing mechanics is out of scope for the benchmark — not because Azimuth couldn't handle it, but because EVMBench Exploit doesn't test for it.
Chain state is another. Each task starts on a fresh local Anvil chain with empty storage — not a fork of mainnet. There's no forking, no precompiles, no historical transactions. The agent is operating against a blank chain state, not the messier, stateful conditions of a real deployed protocol.
The environment is also single-chain only. Cross-chain interactions are out of scope, and where a contract exists only on mainnet, the benchmark substitutes a mock (e.g. mock WETH) in its place. That's a meaningful gap for DeFi specifically, where cross-protocol composability is often where the real risk lives.
Building each Exploit-mode test case isn't automatic — OpenAI and Paradigm have to stand up a working scenario for every vulnerability, including building a proof-of-concept from scratch where one doesn't already exist. That's a limitation on the benchmark's coverage, not on agent performance — but it also means each scenario has already been hand-verified as exploitable in principle, so a failure to exploit one says something real about capability.
EVMBench Exploit's grading is programmatic, but it's still bound by the benchmark's own limits — a blank chain, no real state, no human judgment behind the verdict. Azimuth's validation engine, built separately from this benchmark, goes further.
Beyond the Benchmark: Azimuth's Validation Engine
The 91% above reflects Azimuth's performance against EVMBench's own grading scripts. Azimuth's validation engine is a separate capability, built to apply that same execution-over-inference principle to any finding — not just the benchmark's predefined targets.
Rather than relying on model reasoning alone, the validation engine executes and simulates potential attack paths, observes the resulting state transitions, challenges the assumptions behind each finding, and attempts to reproduce its impact in a live execution environment. A finding either reproduces, or it comes back inconclusive or refuted.
It's the same principle behind Exploit mode itself, applied to Azimuth's own findings rather than a fixed benchmark target — proof, not just a model's reasoning about why something might be exploitable.
That's the gap EVMBench Exploit itself can't close — a blank chain, programmatic grading, no real-world state. The 91% proves Azimuth can execute exploits under benchmark conditions. The validation engine takes that same execution-over-inference principle and applies it everywhere else: to your findings, not just the benchmark's.
See validation in action: app.testmachine.ai