How Accurate Are AI Detectors? What the RAID Benchmark Shows

How accurate are AI detectors? Many advertise 99% or more. The RAID benchmark, presented at ACL 2024, was built to test that kind of claim on a shared, difficult dataset. Its answer: detectors can reach very high accuracy, but only under narrow conditions, often only when allowed many false positives, and simple changes to how text is generated or lightly altered can push them close to failure.

Here is what RAID contains, how it measures detectors, and why its numbers rarely match the figures on vendor landing pages.

What is inside RAID

RAID is large on purpose. Earlier detection datasets tended to use one or two generators, one decoding setting and no attacks, which flatters detectors. RAID varies all of these at once.

Ten percent of the data is held back without labels as a hidden test set, and the authors run a public leaderboard so new detectors can be compared on the same footing. A separate RAID-extra set adds 2.3 million generations of Python code, Czech news and German news.

Paper abstracts are one of the eight domains, which makes RAID more relevant to academic writers than benchmarks built only on news or web text.

Why the false positive rate changes everything

A detector outputs a score. To turn it into "AI" or "human", someone picks a threshold. The RAID authors show that this choice matters more than most marketing admits.

First, they checked what happens with a naive threshold such as 0.5. Several open-source detectors, especially the metric-based ones, produced what the authors call dangerously high false positive rates. The commercial detectors were better calibrated, with none above a 1.7% false positive rate at the common thresholds tested.

Second, they plotted accuracy against false positive rate. Detectors could reach the very high accuracies cited in viral reports, but only when they were allowed a similarly high false positive rate. As the tolerated rate fell, most dropped steeply, and few could operate below 1%. Some never reached the lowest rate tested at all, levelling off at 16.9% (ZeroGPT), 0.88% (Fast-DetectGPT) and 0.62% (Originality). Binoculars was the clear exception, holding up well at low false positive rates.

• Detection rate on AI text
• False positive rate on human text

Illustrative example of the tradeoff RAID measures, not data from the paper.

For the rest of the paper, every detector is calibrated to a 5% false positive rate on RAID's human texts, and accuracy is reported at that point. That is the only fair way to compare tools, and it is also why RAID numbers look lower than vendor claims, which often do not state a false positive rate at all.

Sampling, repetition penalties and unseen models

Some of RAID's most striking results involve no attack at all, only ordinary generation settings.

Repetition penalty. Adding a repetition penalty, which discourages a model from reusing the same tokens, cut accuracy by up to 32 points across detectors, generators and domains. The authors note that no earlier study of detector resilience had reported on it, even though penalties make output sound more human.

Decoding strategy. Every detector category did substantially better on greedy decoding than on random sampling.

Unseen generators. The RoBERTa detector trained on GPT-2 output scored above 95% on GPT-2 text in five domains, but rarely above 60% on text from other models in the same domains. Detectors do best on the models and domains they were trained on, and the authors observed similar patterns in commercial tools.

The paper sums it up plainly: changing the generator, the decoding strategy or adding a repetition penalty was enough to introduce error rates of 95% or more for detectors that otherwise looked excellent.

How adversarial attacks hit different detectors

RAID's attacks are deliberately simple, black-box edits. Their effect varied a lot by detector, which is one of the paper's more useful findings.

• Originality, no attack85.0%
• Originality, homoglyph attack9.3%
• Binoculars, no attack79.6%
• Binoculars, synonym swap43.5%
• GPTZero, no attack66.5%
• GPTZero, homoglyph attack66.2%

Source: Dugan et al. 2024, Table 6.

• Swapping a small portion of words for synonyms cut Binoculars and other metric-based methods by as much as 36.1 points.
• Homoglyphs, characters that look identical but have different code points, hurt almost every detector. GPTZero lost only 0.3 points; five others dropped by an average of 40.6.
• Adversarially trained detectors such as RADAR held up much better.
• Some attacks helped. The GPT-2 RoBERTa detector improved by 16.2 points after T5 paraphrasing, likely because the paraphraser moved the text closer to the detector's own training data.

The pattern behind all of this is the same one seen in smaller studies such as the 14-tool test by Weber-Wulff and colleagues: detectors read surface statistics, and different detectors read different ones. A change that leaves meaning untouched can still move a text across a detector's line.

Why benchmark numbers differ from vendor claims

When a product page says 99%, it usually means 99% on the vendor's own test set, drawn from the models, domains and settings the detector was trained for, at a threshold the reader is not told. RAID differs on every one of those points:

• It fixes and reports the false positive rate, so accuracy cannot be bought with misflagged human text.
• It includes models and domains a detector may never have seen.
• It varies decoding settings that real users and tools change routinely.
• It applies cheap edits that real text goes through anyway, such as typos, formatting changes and rewording.

Neither number is fake. They answer different questions. A vendor figure tells you how a tool does in the conditions it was built for. RAID tells you how it behaves when conditions shift, which is closer to what happens to a real manuscript. That gap also explains much of what we cover in why AI detectors disagree on the same paragraph.

The authors draw a firm line in their ethics statement: given unsolved false positive problems, they oppose using detectors in disciplinary or punitive contexts until better evaluation standards are widely adopted.

What this means for academic writers

If you write with AI help, RAID is a reminder that any single detector score is conditional on settings you cannot see. Treat a result as one reading, check what your institution or journal policy says about AI assistance, and keep records of your drafting process. If a score is used against you, benchmark evidence like this is a fair basis for asking for human review; our post on what to do if you are falsely flagged as AI covers the practical steps. Does Turnitin detect ChatGPT, Claude and Gemini? looks at the same question for Turnitin, which RAID did not test.

Where Academic Humanizer fits

Because detectors disagree and shift, Academic Humanizer checks its results against several of them rather than one. It is validated against Turnitin, GPTZero, ZeroGPT, Copyleaks, Sapling and Originality.ai. The Humanizer rewrites only the sentences that detectors flag, keeps your terminology, numbers and citations, and its AI Detection Report shows the draft and the output side by side on the same measure, including any sentences that are still flagged, so you can decide what to revise before you submit.