Fast-DetectGPT and Binoculars Explained: Cheaper AI Text Detection

Fast-DetectGPT and Binoculars are two zero-shot AI text detectors that made the DetectGPT idea cheap enough to run at scale. Both decide whether a passage is machine-written by asking how predictable its words are to a language model, and both do it in a single pass instead of the hundred rewrites DetectGPT needed. Fast-DetectGPT appeared at ICLR 2024; Binoculars appeared at ICML 2024.

This post explains both methods for readers outside NLP, sets out the numbers the authors reported, and then looks at the question that matters most for researchers: what these detectors do with careful, formal human prose.

Where they come from: the cost problem in DetectGPT

DetectGPT showed that machine text tends to sit at a local peak of a model's probability. To measure that, it rewrote each passage about a hundred times with a separate model and rescored every copy. It worked, but it was slow. Fast-DetectGPT's authors timed it on their benchmark at 79,113 seconds on an A100 GPU. Their own method took 233 seconds.

How Fast-DetectGPT works: conditional probability curvature

The key move is to work word by word. At each position in the passage, a language model has a distribution over possible next words given the words before it. Fast-DetectGPT asks: was the word actually used one of the model's favourite choices at that position, compared with what the model would typically pick there?

1. Fix the contextKeep the real passage in place so every position has its true preceding words.
2. Sample alternativesAt each position, draw alternative tokens from a sampling model, independently of the other positions.
3. Score bothA scoring model gives the log probability of the real tokens and of the sampled ones.
4. NormalizeSubtract the average sampled score and divide by its spread. A high result means the text picks the model's favourite words unusually often.

The authors call this conditional probability curvature. Because the samples are drawn per token with the real context held fixed, the whole calculation needs only a forward pass or two rather than a hundred. In the black-box setting, where the source model is unknown, they used GPT-J as the sampling model and GPT-Neo-2.7B as the scoring model.

What Fast-DetectGPT reported

Averaged over five source models in the white-box setting, Fast-DetectGPT reached an AUROC of 0.9887 against 0.9554 for DetectGPT. On ChatGPT and GPT-4 text in the black-box setting, it reached 0.9338 against 0.7225. AUROC runs from 0.5, a coin flip, to 1.0, perfect separation.

• DetectGPT, white-box0.9554
• Fast-DetectGPT, white-box0.9887
• DetectGPT, black-box (ChatGPT, GPT-4)0.7225
• Fast-DetectGPT, black-box (ChatGPT, GPT-4)0.9338

Source: Bao et al., Fast-DetectGPT, ICLR 2024, Table 1

How Binoculars works: perplexity divided by cross-perplexity

Binoculars, from Abhimanyu Hans and colleagues, tackles a different weakness. Plain perplexity, a measure of how surprised a model is by a text, confuses two things: whether the text was machine-written and whether the topic is unusual. The authors give a memorable example they call the capybara problem. Ask a chatbot to write about a capybara who is an astrophysicist, and the result will surprise a model even though a machine wrote it, because the subject itself is odd.

Binoculars corrects for this with two closely related models, an observer and a performer (in the paper, Falcon-7B-Instruct and Falcon-7B). It computes two quantities:

• Log perplexity: how surprised the observer model is by the actual text.
• Log cross-perplexity: how surprised the observer is, on average, by what the performer model would have predicted at each position. This is a baseline for how surprising machine text on this topic would be.

The Binoculars score is the first divided by the second. A strange prompt raises both numbers, so the ratio stays low for machine text. Human writing tends to be more surprising than the machine baseline, so its ratio is higher.

• Machine-written text
• Human-written text

Illustrative example of the idea, not data from either paper

What Binoculars reported

Across a range of document types, Binoculars detected over 90% of ChatGPT-generated samples at a false positive rate of 0.01%, which is one human text in ten thousand. It had never seen ChatGPT data during development. The authors also tested essays by non-native English writers from an online essay forum, in original and grammar-corrected versions, and reported the same accuracy, 99.67%, on both. Whether commercial tools generalize the same way across chatbots is a separate question, covered in does Turnitin detect ChatGPT, Claude and Gemini?

Limits, and false positives on formal text

Both papers report strong numbers on their benchmarks. A few caveats matter when these ideas end up inside tools that judge real manuscripts.

Predictable human writing still scores as predictable

Both detectors reward surprise. Academic writing often avoids it on purpose: standard method descriptions, fixed phrases, cautious hedging and field terminology all make text easier to predict. The Fast-DetectGPT authors themselves warn that detectors of this kind may show higher false positive rates on text by non-native English speakers. Binoculars did well on its ESL essay test, which is encouraging, but one dataset of student essays is not the same as a methods section in clinical research. Our post on detector bias against non-native writers covers the study that raised this concern.

Memorized text looks like machine text

The Binoculars team tested 11 famous texts. Three fell on the machine side of their threshold, because passages a model has memorized are highly predictable to it. The authors note this is consistent with how the detector works, and that it may be acceptable for some uses but a problem for others. Quoted definitions and standard passages in academic writing raise a similar concern.

Paraphrasing and model mismatch

Neither paper claims resistance to deliberate rewriting, and the Binoculars authors note that they did not study adversarial attacks. Fast-DetectGPT also gets weaker when the scoring model differs from the source model and when passages are short. Since every zero-shot method of this type reads token probabilities, a passage that has been substantially rewritten is a different object to score. Research on paraphrasing attacks, covered in can AI-generated text be reliably detected?, shows how far scores can move.

What this means with Academic Humanizer

Because detectors like these respond to how predictable each stretch of text is, a mostly human draft can still contain a few sentences that read as machine-like. Academic Humanizer targets exactly those: it finds the sentences detectors flag, rewrites only them while keeping your argument, data, citations and terms, and leaves every other sentence untouched. The AI Detection Report then compares the original and revised versions on the same measure, so you can review each change, and you should still check your institution's or journal's AI policy before submitting.