Are AI Detectors Biased? False Positives on Non-Native English Writers

Are AI detectors biased? In the best-known test, yes: they produced far more false positives on writing by non-native English speakers than on writing by native speakers. That is the central finding of "GPT detectors are biased against non-native English writers", a 2023 study by Weixin Liang, James Zou and colleagues at Stanford, published in the journal Patterns. Seven popular detectors labelled, on average, 61% of TOEFL essays written by real students as AI-generated, while handling essays by US eighth graders almost perfectly.

For international researchers and students, this paper is the clearest evidence that a detector verdict can reflect how you write English rather than whether you used AI. Here is how the study worked, why the bias happens, and what has changed since.

How the Liang et al. study was designed

The design is simple, which is part of why it was so persuasive. The authors collected two sets of essays written entirely by humans:

• 91 TOEFL essays by non-native English speakers, gathered from a Chinese educational forum.
• 88 essays by US eighth-grade students, taken from the Hewlett Foundation's Automated Student Assessment Prize dataset.

They ran every essay through seven detectors that were widely used at the time: Originality.AI, Quill.org, Sapling, OpenAI's classifier, Crossplag, GPTZero and ZeroGPT. Since all essays were human-written, any AI verdict was a false positive.

The results: 61% of TOEFL essays flagged

The detectors were close to perfect on the US essays. On the TOEFL essays, the average false positive rate across the seven tools was 61.22%. Worse, the errors were not scattered randomly.

Eighteen of the 91 essays were labelled AI-generated by every single detector. And 89 of 91 were flagged by at least one. If a teacher or editor runs a non-native writer's essay through whichever tool happens to be at hand, these numbers suggest a high chance of at least one false alarm.

Why it happens: perplexity and constrained vocabulary

Many detectors lean on perplexity, a measure of how surprised a language model is by a text. Machine output tends to have low perplexity because models pick likely words. The trouble is that human writing can also have low perplexity for perfectly innocent reasons.

Writers working in a second language often use a smaller, safer vocabulary and more common sentence structures. That is not a flaw in their thinking; it is how most people write in a language they learned later. But to a language model, those choices are predictable, and predictable looks machine-made. The authors measured perplexity directly and found the TOEFL essays had significantly lower perplexity than the US essays.

• Machine-generated text
• Non-native human writing

Illustrative example of the mechanism, not data from the paper

The word-enrichment experiment

To test whether vocabulary really drove the bias, the authors ran the experiment in both directions using ChatGPT as an editor.

• Enriching non-native essays. They asked ChatGPT to improve the word choices in the TOEFL essays so they sounded more like a native speaker. The average false positive rate fell from 61.22% to 11.77%, and only 1 of 91 essays was still flagged by all seven detectors.
• Simplifying native essays. They asked ChatGPT to simplify the word choices in the US essays to mimic non-native writing. Misclassification rose from an average of 5.19% to 56.65%.

• TOEFL essays, original61.22%
• TOEFL essays, word choices enriched11.77%
• US 8th-grade essays, original5.19%
• US 8th-grade essays, word choices simplified56.65%

Source: Liang et al., "GPT detectors are biased against non-native English writers", Patterns 2023

The content of each essay stayed the same. Only the vocabulary changed, and the verdicts swung by roughly fifty percentage points in each direction. The authors also showed the flip side: asking ChatGPT to rewrite its own college essays and scientific abstracts with more literary language cut detection sharply. For abstracts, detection fell from up to 68% to up to 28%. Their conclusion was that detectors in their current form may penalize writers with constrained linguistic expression while being easy for determined users to sidestep.

What it means for international researchers

Scientific English already pushes everyone toward standard phrasing. Add writing in a second language, and the prose of many international researchers lands squarely in the zone detectors associate with machines. A few practical points follow, and our guide on how to avoid AI detection in academic writing turns them into revision steps.

• A flag is not evidence of misconduct. The authors explicitly cautioned against using detectors in evaluative or educational settings. If you are flagged, our guide on what to do if falsely flagged as AI covers how to respond.
• Keep your process visible. Drafts, notes and version history are the strongest answer to a disputed score. See how draft history supports academic integrity.
• Formal is not the same as machine-made. The same mechanism explains why formal prose can look AI-written even from native speakers.

Limits of the study and later work

The study has real limits. The samples are modest, 91 and 88 essays, and they are student essays rather than research papers. The detectors were tested in early 2023, and every vendor has updated its models since. OpenAI withdrew its classifier later that year. So the exact percentages describe those tools at that time, not today's versions.

Later research gives a mixed picture. The authors of Binoculars, a zero-shot detector presented at ICML 2024, tested essays by non-native writers in original and grammar-corrected form and reported 99.67% accuracy on both, suggesting some methods are less sensitive to this effect. On the other hand, the Fast-DetectGPT authors flagged higher false positive rates on non-native writing as a risk for detectors of their type. We compare these methods in Fast-DetectGPT and Binoculars explained. The underlying tension has not gone away: any detector that rewards surprising word choices will find plain, careful English harder to judge.

How Academic Humanizer approaches this

Academic Humanizer was built for researchers whose own ideas get mislabelled as machine text, including many who write in English as a second language. It checks a draft sentence by sentence, rewrites only the sentences detectors flag, and keeps your claims, numbers, citations and field terminology intact; sentences already read as human stay exactly as you wrote them. The AI Detection Report shows each change side by side so you approve the final wording, and it is good practice to confirm your journal's or institution's AI policy alongside it.