Does Turnitin Detect AI Watermarks? LLM Watermarking Explained

Turnitin's public description of its AI detector is a trained classifier that scores writing patterns; it does not mention reading watermarks. LLM watermarking is a different idea: a way to hide a statistical signal in AI-generated text at the moment it is written, so that software holding a secret key can later confirm where the text came from. The best-known method, "A Watermark for Large Language Models" by John Kirchenbauer and colleagues (ICML 2023), nudges the model toward a secret, randomly chosen set of "green" words. Readers cannot see it, but a simple statistical test can count green words and detect the pattern in a short passage.

Watermarks come up often in conversations about AI detection, and they are frequently misunderstood. This post explains how the green-list method works, what the paper reported, how edits weaken the signal, and what is actually deployed today.

How the green-list watermark works

A language model writes one token at a time, where a token is a word or a piece of a word. Before choosing each token, the watermarking scheme does the following:

1. Seed from the previous tokenHash the token just generated to seed a random number generator.
2. Split the vocabularyUse that seed to divide the whole vocabulary into a green list (a fraction γ of words) and a red list (the rest).
3. Tilt the oddsAdd a small bonus, δ, to the scores of green tokens before sampling, so green words become more likely.
4. Sample as usualPick the next token. Repeat for every position in the text.

The paper describes two versions. A hard watermark bans red tokens outright, which can damage text when the only sensible next word happens to be red. The soft watermark, the one most people mean, only tilts the odds. When the model is confident (low entropy, as in "Barack" followed by "Obama"), the bonus barely matters and quality is protected. When many words would work equally well, the bonus decides, and green words pile up.

The survey results suggest that rural clinics face persistent staffing gaps across both regions.

Illustrative example. Green = token on that position's green list, red = red list. Unwatermarked text would show roughly a γ share of green.

Detecting the watermark: the z-score test

Detection needs no access to the model. Anyone with the secret hashing key can rebuild each position's green list from the preceding token and count how many tokens in the passage are green. Human text knows nothing about the lists, so about a γ fraction of its tokens will be green by chance. Watermarked text will have more.

The test asks how surprising the green count is. With T tokens, of which G are green, the paper uses the one-proportion z-test:

z = (G − γT) / √(T × γ × (1 − γ))

If z is above a threshold, the text is flagged as watermarked. With a threshold of z = 4, the probability that unwatermarked text crosses it by chance is about 0.00003, or three in a hundred thousand. The arithmetic also shows why short spans can work: with γ = 0.5 and a hard watermark, 16 tokens that are all green already give z = 4.

In the paper's experiments with a multi-billion-parameter OPT model, sequences of about 200 tokens, δ = 2 and γ = 0.5 under ordinary multinomial sampling, the test caught 98.4% of watermarked texts with no false positives in that setting. Larger δ makes the watermark stronger and easier to detect, at some cost to text quality.

How well the watermark survives edits

Because the green list for each position depends on the previous token, changing one word disturbs the green-red status of a couple of positions, not the whole text. The authors tested an attack that replaced spans with a T5 model. It weakened the watermark, but the replacements also made the text noticeably worse, so there was a trade-off between removing the signal and keeping the writing usable.

Later work tested harder attacks:

• Paraphrasing. Sadasivan and colleagues reported that recursive paraphrasing cut detection of watermarked 300-token passages from 99.3% to 9.7% at a 1% false positive rate. See can AI-generated text be reliably detected?
• Spoofing. The same paper showed that an attacker could estimate the green lists from model outputs and forge watermarked-looking human text, driving the detector's AUROC from 99.8% to 1.3%.
• A defense of reliability. Kirchenbauer's team answered in "On the Reliability of Watermarks for Large Language Models" (ICLR 2024): watermarks stayed detectable after human and machine paraphrasing, because paraphrases keep fragments of the original. In one setting with strong human rewording, detection needed about 800 tokens on average at a very low false positive rate.

The fair summary: light edits leave the watermark intact, heavy rewriting weakens it, and longer texts give the detector more to work with. Our post on DIPPER paraphrasing covers the rewriting side in more detail.

Why watermarks are not what Turnitin uses

A watermark can only be found in text from a model that applied it, and only by someone who holds the key. Most text people paste into a detector comes from many different models, most of which do not share a watermark key with any detection vendor. That is why commercial detectors work differently. Turnitin's public description of its AI writing detection describes a trained transformer classifier: the submission is split into overlapping segments, each sentence is scored, and the scores are combined into a document estimate. Nothing in that description involves reading a watermark. We explain the indicator in Turnitin's AI detection score explained, and the broader detector types in how AI detectors work. What Turnitin says it does detect across chatbots is covered in does Turnitin detect ChatGPT, Claude and Gemini?

Where watermarking stands in industry

Watermarking has moved from research into at least one large production system. Google DeepMind published "Scalable watermarking for identifying large language model outputs" in Nature in October 2024, describing SynthID-Text, which changes only the sampling step and detects without running the language model. The team reported a live experiment covering nearly 20 million Gemini responses that showed no loss in user-rated quality.

OpenAI took a more cautious line. In an August 2024 update it said its own text watermarking method was highly accurate against localized tampering such as paraphrasing, but less so against global changes like translation or rewording with another model, and that it could disproportionately affect groups such as non-native English speakers. It had not released the tool at that time.

So watermarks exist and work in the narrow case they were built for: confirming that a specific provider's model produced a piece of text. They do not answer the general question of whether any AI was involved in a manuscript.

How this relates to Academic Humanizer

For most researchers, the detectors that matter day to day are classifiers that score each sentence's style, not watermark readers. That is the level Academic Humanizer works at: it finds the sentences detectors flag, rewrites only those while keeping claims, numbers, citations and terms intact, and leaves the rest of the draft as written. The AI Detection Report lays out each flagged sentence next to its rewrite for your review, and you should still follow your institution's or journal's AI disclosure rules.