Are AI Detectors Accurate? What a 14-Tool Study Found

Are AI detectors accurate? Not very, according to one of the most carefully documented public tests so far. In 2023, eight academic integrity researchers wrote original texts, generated others with ChatGPT, edited and paraphrased some of those, and submitted all 54 documents to 14 detectors. None reached 80% accuracy. The conclusion, published in the International Journal for Educational Integrity, was blunt: the tools were "neither accurate nor reliable".

This post explains how the study was built, what the numbers actually show, and why its most useful lesson for researchers is narrower and more practical than "detectors are useless". A detector score is a signal. It is not evidence.

How the study was designed

The strength of this paper is its test set. Nine researchers (the eight authors plus one collaborator) each prepared six documents, so the ground truth for every case was known. Human texts of about 10,000 characters were written fresh, at roughly undergraduate level, in fields such as civil engineering, computer science, economics, history, linguistics and literature. None of it had ever been on the internet, which rules out the chance that a model had seen it in training.

The detectors were Check For AI, Compilatio, Content at Scale, Crossplag, DetectGPT, Go Winston, GPT Zero, the GPT-2 Output Detector Demo, the OpenAI Text Classifier, PlagiarismCheck, Turnitin, Writeful GPT Detector, Writer and ZeroGPT. Twelve were free web tools; Turnitin and PlagiarismCheck were commercial systems that gave the team access. That made 756 individual tests, run between March and May 2023.

Because tools report probabilities rather than yes or no, the authors mapped every output onto a five-step scale and computed accuracy three different ways. The ranking of tools barely changed between methods, which makes the results harder to dismiss as an artefact of scoring.

The headline: no tool reached 80%

Every tool scored below 80% accuracy, and only five scored above 70%: Turnitin, Compilatio, the GPT-2 Output Detector, Crossplag and Go Winston. Turnitin ranked first under all three scoring methods. The GPT-2 Output Detector placing third surprised the authors, since it was never trained on GPT-3.5 output.

• Turnitin76%
• Compilatio74%
• GPT-2 Output Detector Demo72%
• GPT Zero54%
• PlagiarismCheck39%
• Content at Scale33%

Source: Weber-Wulff et al. 2023, Table 5 (binary approach, 54 documents per tool).

An overall accuracy figure hides most of the story, though. The interesting part is where each tool failed.

Where accuracy collapsed

Averaged across tools and scoring methods, human-written English was classified correctly 96% of the time. Unedited ChatGPT text came in at 74%. After that, the numbers fell quickly.

• Human-written (01-Hum)96%
• ChatGPT, unedited (03/04-AI)74%
• ChatGPT, hand-edited (05-ManEd)42%
• ChatGPT, machine-paraphrased (06-Para)26%

Source: Weber-Wulff et al. 2023, section 5.1.1 (mean of all accuracy approaches).

Machine translation. Human texts that were written in another language and machine translated into English were classified correctly about 20 points less often than native English texts. Translation leaves statistical traces that tools read as machine output, even when every idea came from a person.

Light human editing. Swapping synonyms and reordering clauses, the kind of quick pass the authors call patchwriting, pulled average accuracy down to 42%.

Machine paraphrase. One pass through Quillbot dropped accuracy to 26%. Most paraphrased AI texts were read as human.

Why would such shallow changes matter so much? Most detectors do not read for meaning. As we explain in how AI detectors work, they score how predictable each word is and how uniform the rhythm of the text looks. Change enough word choices and the statistical fingerprint shifts while the content stays the same. That is a fragility in the detector, not a property of the writing.

A bias toward "human", and the false accusation problem

With the notable exception of GPT Zero, every tool was more accurate on human text than on AI text. The authors read this as a probably deliberate design choice, and argue it is the safer direction in education. Still, errors ran both ways.

The team defined a false accusation ratio: the share of human-written cases where a tool's output would plausibly lead a teacher to accuse a student. For native English texts, the average across tools was 2.4%. For machine-translated texts it was 11.1%. Half the tools never produced a false accusation; GPT Zero was the outlier at 50%, which the authors called unsuitable for academic use. Turnitin had no false accusations in this set.

On the other side, about 20% of unedited AI texts went undetected on average, and around half of obfuscated ones did. The team also saw the same document get different results on different days: one Check For AI test reported low risk, then high risk five days later. For anyone facing an accusation, that inconsistency matters as much as the error rates.

The translation result deserves attention from international researchers in particular. Writing in your first language and translating your own work is a legitimate practice, yet it raised the risk of a false flag in this study. It lines up with the findings we cover in GPT detectors are biased against non-native writers.

What the authors recommend

The conclusions are aimed at educators, and they are firm:

• Detector output should never be the only basis for a misconduct report. Unlike text-matching software, AI detectors provide no source that can be checked, so a student has nothing concrete to rebut.
• A flag can prompt a conversation with the student, not a verdict.
• Prevention should come before detection: clear guidance on ethical and unethical uses of generative AI, and assessment redesigned to value the writing process over the final product.
• Institutions should think about the privacy and legal side of uploading student work to cloud-based detection services.

The paper is also open about its limits. It tested English only, a single generator, and tools as they existed in early 2023. Vendors have updated their products since, so the specific percentages should not be read as current scores. The method and the failure patterns are what carry forward, and later work such as the RAID benchmark found the same kinds of weakness at much larger scale.

What this means if you write with AI assistance

If you are a researcher or student, three practical points follow. First, check what your institution or target journal actually allows, and disclose AI use where required. Second, keep your drafts, notes and version history; the study's authors stress that detectors offer no evidence, and your own process records are the strongest evidence you can have (see how draft history supports academic integrity). Third, if you are flagged, the published error rates give you a reasonable basis to ask for a human review. Our guide on what to do if you are falsely flagged as AI covers that conversation. If an instructor raises a flag, how professors detect AI explains what else they tend to look at besides a score.

Where Academic Humanizer fits

Studies like this one are why Academic Humanizer exists: formal, carefully edited scholarly prose can sit close to the line detectors draw, and good ideas get mislabelled. The Humanizer scans a draft sentence by sentence, rewrites only the sentences detectors flag, and leaves the rest exactly as you wrote them, keeping your claims, numbers and citations intact. The built-in AI Detection Report shows each flagged sentence next to its rewrite and compares both versions on the same measure, so you can see what changed and review every line yourself before you use it.