Which AI model finds the most vulnerabilities? 5 tested

I tested five AI models on finding vulnerabilities in code, in two rounds. The first used 44 textbook Java files from the OWASP Benchmark, and every model flagged every vulnerable file it answered on, so that round ranks nothing. The second used 30 real vulnerabilities fixed between June and September 2026 in open-source JavaScript, TypeScript and Python projects. Each model saw the code before the fix and after it, in two separate calls, and had to call the first version vulnerable and the second safe. The highest score was 15 of the 30 pairs and the lowest 9, and with 30 pairs these gaps are too small to rank the five with confidence. The cheapest model, Claude Haiku 5.5, scored 13 for about a thirtieth of the price of Claude Opus 5.5, which scored 15.

Round 1: 44 textbook files, passed by all five models

The OWASP Benchmark is a public set of small Java web programs, each labelled vulnerable or safe. I drew 22 vulnerable and 22 safe files across its 11 classes of weakness, removed the comments and names that state the answer, and asked each model through OpenRouter for a verdict in JSON, with reasoning effort set to low.

These files are easy for the five models. In 16 of the 44, one line decides, for example the choice of DES or AES. In the other 28, the user’s input either reaches the dangerous call or is replaced by a constant a few lines earlier. The benchmark has also been public since 2016, so its files may be in the training data. Three false alarms in 110 answers on safe files are too few to separate five models. The missing answer is a request that was blocked before Opus replied, with the message “This request triggered restrictions on violative cyber content and was blocked under Anthropic’s Usage Policy”; I sent it once, so I don’t know whether a second attempt would pass.

Round 2: 30 real fixes, each shown before and after

I took the cases from the GitHub Advisory Database: the 1,977 reviewed advisories for npm and PyPI packages published between 1 July and 9 October 2026. A script kept an advisory when, among other conditions, it is rated high or critical and links to exactly one fix commit, dated June 2026 or later, that changes one to three source files of at most 600 lines in total. 109 advisories passed. A written rule and a fixed random seed then drew 30 of them, one per project, spread over 15 classes of weakness, with 12 fixes that span two or three files.

Each advisory gives a pair: the changed files before the fix and after it. The model got one version at a time, with the file paths, and without the advisory, the commit message or the project’s name. A script removed the comments that name an advisory or describe an attack, which happened in 5 pairs. The settings were those of round 1, and the model also had to say in one line where the flaw is. A pair counts as told apart when the model calls the version before the fix vulnerable and the version after it safe, so a model that calls everything vulnerable scores zero.

What round 2 shows, and its limits

No model told more than half of the pairs apart, and 6 of the 30 pairs were told apart by none of the five. All five still do better than a model that ignores the code: four of them never gave the reversed answer, vulnerable version safe and fixed version vulnerable, and Gemini gave it 3 times against 10 pairs told apart.

The table is sorted by pairs told apart, but I can’t defend that order as a ranking. The widest gap is between Opus and Qwen: 7 pairs that only Opus told apart and 1 that only Qwen did, a split that two equally good models would produce about 7 times in 100. Between Opus and Haiku the split is 4 pairs to 2. I also made one call per version, and a second run could move a few answers. Qwen and Gemini lost 3 and 2 pairs before any comparison: 4 of their answers used up the 4,096 tokens I allowed while the model was still reasoning, and one Gemini answer gave a class name where the verdict should be.

GPT-6.1 Sol behaves differently from the other four. It answered “vulnerable” 44 times out of 60, against 21 to 25 for the others, so it flagged 28 of the 30 vulnerable versions and also 16 of the 30 fixed ones.

Two limits of the test affect the count of pairs. The first is that I label the fixed version safe, although it can hold another flaw. In 13 of the 36 cases where a model called both versions vulnerable, it named the advisory’s class of weakness before the fix and a different class after it. On one pair, three models pointed at a login token whose signature was never verified, and then, in the fixed version, at a request body read without a size limit. The second limit is that the code scores the verdict and ignores the reason. A pair can count as told apart when the model pointed at code the fix did not touch, and I did not grade the reasons.

The model also saw only the files the fix changed. Fixes that span two or three files were told apart less often than one-file fixes, 19 times in 60 against 40 in 90 over the five models. In 16 pairs the fixed version contains a comment that the vulnerable version lacks, which can help a model. Recent code can still be in training data, and the code before a fix may be years old.

Cost of round 2

Round 2 cost $2.80 in total. Opus cost $1.58 and Haiku $0.05 for the same 60 answers, a ratio of 32, for 15 pairs told apart against 13. These are OpenRouter prices on the day of the run, for files of 79 to 564 lines and low reasoning effort.

What a team can take from this test

All five models catch textbook weaknesses in a single file, so a set like round 1 no longer helps to choose between them.

On real vulnerabilities, with one low-effort pass over the changed files only, the highest score was half of the pairs. A “safe” answer from these models is weak evidence: Opus called 10 of the 30 vulnerable versions safe, Haiku and Qwen 12 each, and Gemini 16.

I would choose on behaviour and price before I choose on this order. GPT-6.1 Sol flags almost every real flaw and about half of the fixed code, which suits a team that can review every alert. Opus and Haiku each raised an alert on 5 of the 30 fixed versions and missed a third or more of the real flaws, with Haiku at about a thirtieth of the price of Opus.

A real audit also reads what this test leaves out: the code around the changed files, the configuration, the permissions, the dependencies and the git history. I wrote up one such audit in Security audit of an AI-generated codebase. A ranking of the models would take more pairs and several runs per pair, and I have not run that test.