How Do AI Detectors Work? And Why Your Own Writing Gets Flagged

AI detectors work by measuring how predictable a piece of text looks to a language model, not by reading for meaning, and that is also why your own writing can be flagged as AI. Some use simple statistics such as perplexity and burstiness, some a classifier trained on labelled examples, and a few hidden watermarks placed in the text at generation time. None of them knows who wrote the text. They judge whether its pattern resembles machine output.

Each method has a different weak spot, and almost all of them share one: careful, formal, conventional prose is predictable by design. That is why a well-edited methods section can score closer to "AI" than a messy blog post. Here is how the main approaches work, what the research says about their limits, and why academic writing sits so close to the decision line.

The core signal: how surprised is the model?

A language model assigns a probability to every possible next word (more precisely, every next token). When a model generates text, it tends to pick high-probability tokens, because that is what sampling from it does. Human writers are not bound by that rule. We choose an odd verb, refer back to a detail from three pages earlier, or break a pattern for emphasis.

Perplexity turns this into a number. Run a passage through a model, ask how likely each token was given what came before, and average the result. Low perplexity means the model found the text easy to predict. High perplexity means it kept being surprised. GPTZero's own explainer defines perplexity in roughly these terms and treats higher scores as a sign of human authorship (GPTZero).

The results of this study suggest that soil moisture plays an important role in seedling survival, though only after the second frost.

Green: tokens a model expected. Neutral: topic words. Red: tokens the model found unlikely. The first half reads as template; the late qualifier is what a human reader, and a detector, notices.

This idea is older than ChatGPT. GLTR, a 2019 tool from Harvard and IBM researchers, colored each word by its rank in a model's predictions so humans could spot generated text. In their study, the visualization raised untrained readers' detection rate from 54% to 72% (Gehrmann et al., ACL 2019 demo).

Burstiness: variation across sentences

Perplexity is measured over tokens. Burstiness looks one level up: how much predictability changes from sentence to sentence. Human prose tends to alternate. A short declarative line, then a long sentence with two subordinate clauses and a parenthetical, then a fragment. Generated text often holds a steadier rhythm, with sentences of similar length and similar structure.

A detector that tracks burstiness asks whether the passage has those spikes and dips. Flat profiles push the score toward "AI". The catch is that many genres are flat on purpose. Abstracts, protocol descriptions, and legal or regulatory text all reward uniform sentences.

Perplexity and burstiness are easy to explain, which is why they dominate popular articles. They are no longer the whole story, though. GPTZero itself says it moved to a deep learning architecture and now treats these two measures as a subset of its indicators, not the primary method.

Trained classifiers

Most commercial detectors today are supervised classifiers. A team collects large sets of human-written and machine-written text, labels them, and fine-tunes a neural network to tell them apart. The network learns whatever separates the two piles: word choice, phrase frequency, punctuation habits, paragraph shape, and patterns no one has named.

Classifiers can be very accurate on text that resembles their training data. Ghostbuster, a research system that feeds features from several weaker language models into a classifier, reported 99.0 F1 across domains on its benchmark (Verma et al., NAACL 2024). The problem is drift. A classifier trained on one generation of models, one set of genres, and one decoding setting can falter when any of those change. The RAID benchmark, with over 6 million generations across 11 models, 8 domains and 11 adversarial attacks, found that detectors are easily fooled by changes in sampling strategy, unseen models and simple attacks (Dugan et al., ACL 2024).

The clearest cautionary tale came from OpenAI. Its own AI text classifier, released in January 2023, correctly flagged only 26% of AI-written text in its challenge set and mislabelled 9% of human text. OpenAI withdrew it in July 2023, citing its low rate of accuracy (OpenAI).

Zero-shot statistical methods

A second family of detectors skips training altogether. These zero-shot methods use a language model's own probabilities to test a passage.

DetectGPT, from Stanford, rests on one observation: text sampled from a model tends to sit at a local peak of that model's log probability. Nudge a few words with a second model (the authors used T5) and machine text loses probability fast, while human text changes less predictably. On fake news articles from a 20B parameter model, DetectGPT reached 0.95 AUROC against 0.81 for the strongest zero-shot baseline (Mitchell et al., ICML 2023).

• Machine-generated passage
• Human-written passage

Illustrative shape, based on the method described by Mitchell et al. (2023)

DetectGPT is expensive, since it scores many perturbed copies of each passage. Fast-DetectGPT replaced the perturbation step with a cheaper sampling estimate and reported about a 75% relative improvement and a 340x speedup (Bao et al., ICLR 2024). Binoculars compares how two closely related models score the same text and reported detecting over 90% of ChatGPT samples at a 0.01% false positive rate without training on ChatGPT data (Hans et al., ICML 2024).

Watermarks

Watermarking moves detection to the moment of generation. In the scheme proposed by Kirchenbauer and colleagues, the model splits its vocabulary into "green" and "red" lists at each step using a hash of the previous token, then nudges sampling toward green tokens. A detector that knows the hashing rule counts green tokens and runs a statistical test. Human text lands near chance; watermarked text has too many greens (Kirchenbauer et al., ICML 2023).

Watermarks only work if the model that wrote the text applied one, and only for detectors that know the key. They also weaken under heavy editing. Krishna et al. showed that a trained paraphraser cut DetectGPT's detection rate from 70.3% to 4.6% at a 1% false positive rate, and that watermark detection also degraded (Krishna et al., NeurIPS 2023). So in practice, the detectors your university or journal uses are classifiers and statistical scorers, not watermark readers.

Sentence scores versus document scores

Detectors report at different grains. Some return one probability for the whole document. Others score each sentence or short window, highlight the suspicious ones, and then roll those up into a percentage. Turnitin, for example, highlights sentences and has described a sentence-level false positive rate of around 4% in its own guidance (Turnitin).

Sentence scoring is more informative but noisier. One sentence has few tokens, so its perplexity estimate is unstable. Document scoring is steadier but hides where the problem is. We cover the trade-off in detail in sentence-level AI detection, and why different tools disagree in why AI detectors disagree.

Why formal academic prose sits near the boundary

Every method above rewards surprise and penalizes convention. Academic writing is built on convention. Fields standardize their phrasing so readers can scan papers quickly: "Participants were randomly assigned to", "These findings are consistent with", "Further research is needed to". Style guides discourage flourishes. Non-native writers often rely on a narrower, safer vocabulary. All of that lowers perplexity.

The clearest evidence comes from Liang et al. at Stanford. They ran seven detectors on TOEFL essays written by non-native English speakers and on essays by US eighth graders. The detectors were near perfect on the native essays but misclassified the TOEFL essays as AI an average of 61.3% of the time. When the researchers used ChatGPT to enrich the vocabulary of those same essays, the average false positive rate fell to 11.6% (Liang et al., Patterns 2023).

• Original essays61.3%
• After vocabulary enrichment11.6%

Source: Liang et al., "GPT detectors are biased against non-native English writers", Patterns 2023

The lesson is not that detectors are useless. It is that their scores reflect style as much as authorship. A careful researcher writing plain, disciplined English can land on the wrong side of the line without any AI involvement. We look at this effect more closely in our review of the Liang study and in why formal prose can look AI-written. For revision techniques that keep your own ideas intact, see how to avoid AI detection in academic writing.

Where Academic Humanizer fits

Academic Humanizer was built for that last problem. The Humanizer scans a draft sentence by sentence, rewrites only the sentences detectors flag, and leaves the rest exactly as you wrote them, keeping your argument, numbers, citations and terminology. The built-in AI Detection Report shows each flagged sentence next to its rewrite and compares both versions, and results are checked against detectors including Turnitin, GPTZero and Originality.ai. Before you use any tool on submitted work, check your institution's or journal's AI policy, and read the output yourself before it goes anywhere.