The Issue With AI Detectors

Introduction

In mid February of 2023, my brother submitted his midterm exam for a Middle Eastern history class. About a week later, he logged in to his student portal to find the following message:

William, unfortunately it appears as though this exam is plagiarized. The answer to Q3, in particular, is drawn from ChatGPT or similar AI software, and consequently, drawn from a variety of internet sources without attribution or citation. The consequences for submitting plagiarized work in this course is a grade of 0/20, and a citation to OSSJA for the issue of academic integrity.

William was confused. Of course he was. He hadn't cheated. He only had a vague idea of what ChatGPT even was. And so, of course, he tried to plead his case. His first email was, quite frankly, pathetic. His professor ignored him.

On the advice of his best friend, William reached out to our parents. We're very fortunate to have supportive parents, and in this case, our parents did not fail him. After he proved - quite easily - that he had definitely, absolutely not cheated by using "ChatGPT or similar AI software", my father drafted an email for him to send to his professor. Our father is an attorney, and when he drafted the email he was a pissed off attorney, so naturally his email was full of legalese. That certainly got William's professor's attention, because she finally responded to William:

Hi William,

Thank you for your email, and for getting in touch. I understand that this is a deeply frustrating experience. I can walk you through the process that happens from here. Yes, you will have an opportunity to share your work and discuss with OSSJA; I have forwarded the midterm exam, your exam paper, and the reports from the AI detector to them. They are tasked with figuring this out from here, and for presenting the evidence to you as they proceed with their process. You will be shown everything, and you will be given ample opportunity to discuss.

The campus policy here is for faculty to submit the material over to OSSJA and have them manage cases of possible plagiarism. There are two reasons for this. First, this ensures that each case is dealt with fairly and equitably. Second, this process ensures that every students gets ample opportunity to understand and demonstrate their case in the way you're describing. I strongly encourage you to follow that process through. If this is your first time with OSSJA, you will find that is it not like a court proceeding. It is also very rare that a first-time citation turns into serious academic discipline.

You are always welcome to visit me during my office hours. However, that is not the time or place to "plead your case," nor will it reverse the OSSJA citation. Your email suggests that this is a deeply distressing event. I definitely empathize with your frustration. I want to put forth a reminder that you will need to maintain your professionalism as we interact. I can discuss the course, the exam, and your progress. But when it comes to the OSSJA citation, we both must follow their process at this point.

I hope this is clear. Please feel free to reach out to me if I can further clarify.

Best wishes,

Prof. Fahrenthold

We personally found the email fairly condescending. First, this was William's first "offense". How was he to know he wasn't allowed to contact his professor? He was only attempting to appeal to his professor's better nature (although as we later discovered, his professor does not have a better nature).

Preparation

Faced with the knowledge that William would not be able to simply settle the matter with his professor, and would have to suffer through an embarrassing, stressful "informal" meeting with UC Davis's Office of Student "Support" and Judicial Affairs (OSSJA), we went to work mounting his defense. My father - the attorney - started looking for a legal way to defend my brother. That was fairly simple, it turns out. The school had never sanctioned use of AI detectors, and the professor had defamed my brother in accusing him of cheating based on unsanctioned software.

Meanwhile, I dealt with the AI detection side. I work in AI, and I'm familiar with machine learning models. I was by far best equipped to understand how AI detectors worked. So I passed William's midterm responses through every AI detector I could find (for free). The only one that flagged his work was GPTZero.

GPTZero

Let me tell you something about GPTZero. Of all the AI detectors that don't work (all of them), GPTZero is by far the worst. That's not particularly surprising. GPTZero was built in about three days by a 22-year-old undergrad named Edward Tian. He trained GPTZero on about 5000 texts - half human, half AI. If that sounds insufficient, that's because it is. And while I want to believe that by the time William's professor passed his exam answers through GPTZero, the model was a little more robust, trained on a little more data. However, GPTZero's false positive rate would suggest otherwise.

Because here's the thing: I've passed nearly 300 demonstrably human-written academic texts through GPTZero, and over 30% were falsely flagged as being partially or entirely written by AI. How do I know for sure these texts were written by humans? Easy. All of these texts were written before November 30, 2022 when ChatGPT was released. Many of the flagged texts were written (by me) way before any GPTs were available to anyone. I'm pretty sure I didn't use a time machine to get ChatGPT to write any of my college admission essays.

So how did we test GPTZero for William's case? We passed through famous texts that could prove that GPTZero sucks. We passed through, among other things, among other texts, the ending to Martin Luther King, Jr.'s "I Have a Dream" speech:

We passed through Professor Stacy Fahrenthold's "About Me" from her website:

We also passed the following (not pictured) texts: Genesis Chapter 2 (RSV), the email from Professor Fahrenthold to William, the email from the OSSJA to William and the Beatitudes (RSV). All of these were partly written by AI.

(As an aside, I have a bone to pick with Edward Tian. ChatGPT and other large language models (LLMs) do not write anything. They generate texts. If you're going to build an AI detector, you should understand that difference.)

AI Detectors

But I didn't just stop at GPTZero. No. I tested five other AI detectors: OpenAI, Crossplag, Copyleaks, Content at Scale, and Kazan SEO. None had a false positive rate lower than 2% (which is the false positive rate Edward Tian claims for GPTZero).

False Positive Rates

• GPTZero: 37..22%

• OpenAI: 3.15%

• Crossplag: 6.62%

• Copyleaks: 4.1%

• Content at Scale: 23.97%

• Kazan SEO: 20.19%

The First Meeting

The first meeting actually went very well. Although our father was not allowed to speak, and thus had trouble advising my brother, William was able to clearly lay out the evidence: how bad GPTZero is, and his Google Docs edits that showed every single one of his edits over two hours of writing his exam. This was recorded for his professor to review.

That was the good part. The bad part is that we learned that 26 other cases had been referred to the OSSJA for just this reason. William was apparently the first student not to confess immediately to using ChatGPT. But without Google Docs - or an understanding of AI detectors - it's incredibly difficult to prove you haven't used ChatGPT to generate your essays. So how many of those students were "confessing" just to get things over with because they couldn't prove their innocence? I know if I had been accused of this and hadn't been able to prove my innocence, I would have confessed to get things over with.

The woman from the OSSJA also expressed surprise that William was supposedly the first person to express distress from having to go through this process. Excuse me? That's just being willfully ignorant. Of course a student who has been accused of cheating, whether or not they're guilty, will feel distress. My brother has friends who were falsely accused and are still traumatized by the experience. At least two people have reached out about their own false accusations that are causing them undue stress. My brother's advisor fully empathized with his situation because they had also been falsely accused of cheating in their undergrad days and still felt discomfort at th

e memories. Our grandfather heard about my brother's situation and immediately was worried my brother would consider suicide. Why? Because 70 years ago, our grandfather was falsely accused of cheating, considered suicide at the time, and was still traumatized by the false accusation now, in 2023.

Finally, we also learned that in addition to GPTZero, William's professor had passed his work through ChatGPT itself and a second, unnamed AI detector. Unfortunately, William was not shown the screenshots of this "evidence". So let me break down the other "evidence" for you:

• ChatGPT absolutely cannot tell you if it has generated a text in another person's account. What ChatGPT can do, however, is tell you exactly what you want to hear. So if you ask it if it generated a particular text, it will usually say 'yes'. In fact, nine times out of ten, I can get ChatGPT to say whatever I want. In fact, I convinced ChatGPT to tell me that it was a good dog. I ask you, does this sound like a reliable source?

• Like I said earlier, I passed William's exam responses through almost every single AI detector available (about 15 at the time William was accused). Only GPTZero flagged his responses. So what was this second, unnamed AI detector that supposedly proved William had cheated? We not only never got a name, but that's not even one of the screenshots Stacy Fahrenthold provided to the OSSJA. Does that feel weird to you? Because it certainly felt weird to me.

The Second Meeting

My brother requested a follow-up with the OSSJA. Really what he wanted was an apology from his professor for the stress she caused him. After all, my brother - who had gone almost four years of college without significant mental health issues - had multiple panic attacks throughout this process.

We also wanted to discuss the danger of AI detectors with the university. See, during the week where we waited for a response from the OSSJA, I had gone much deeper into the AI detection world. I had passed all those texts through, gotten a sample that started to be statistically significant.

I had also come across some interesting research. See, in my brother's case, we claimed that there were no studies published about the reliability of AI detectors. That is no longer true. Researchers from the University of Maryland found that as GPTs become more advanced, AI detectors will become less and less effective.

But none of that mattered. In fact, the second meeting with the OSSJA was very, very disappointing.

First, we learned that Stacy Fahrenthold would not be providing us with her "evidence". Why? Because William was "threatening". That's right. When pressed, we learned why Stacy Fahrenthold found William "threatening":

• He sent her an email with "suggestive" language. This turned out to be the email that our father wrote on my brother's behalf. The one in legalese strongly denying the accusations against him. Unfortunately, I do not have the text of that email, but everyone who read it - besides Stacy Fahrenthold - agreed that there was nothing "threatening" about the email.

• William "stalked" her "private" social media and website. That I can refute absolutely. Why? Because I'm the one who "stalked" her social media. What did I do? A few things. First, I accidentally clicked on her LinkedIn (her public profile). That's probably why she blocked me, although I can't imagine why because she had nothing on her LinkedIn. Second, I checked out her Twitter (before and after she locked it the first time). She has almost 6,000 followers on Twitter, so clearly she's a very private person. It was also necessary in mounting a defense for my brother because she tweeted (several times) about how bad ChatGPT is and how easy it is to fool. In fact, she tweeted those things before accusing my brother of cheating. Third, her "private" website is actually very much public.

• William showed up to class in a hoodie. Yes, you read that right. Following the cheating accusation, William showed up to class in a hoodie. In the third meeting with the OSSJA, the mediator tried to justify this by saying that with the current climate, many teachers who refer students to the OSSJA are worried about violent retribution from students. The word "school shooting" was even thrown into the conversation. From what I overheard, it sounded a lot like Stacy Fahrenthold thought my brother could be a school shooter simply because he showed up to class in a hoodie. If Stacy Fahrenthold would like to prove me wrong, she can agree to the restorative justice meeting my family has been asking for. Until then, I'm going to interpret this the way I heard it.

• In fact, until the USA Today article was published, we were determined to keep Stacy Fahrenthold's name out of the press. I no longer have such qualms after the vicious, damaging insults she sent my brother's way. Stacy Fahrenthold is not a good person, and I have no problem attaching her name to my grievances (although I do not want anyone to go after her. She does not need trolls rubbing her nose in her misbehavior; she needs to realize it for herself).

During the second meeting, we also learned that UC Davis would be adopting Turnitin's AI detector. So instead of taking William's case as a cautionary tale, they were doubling down.

Turnitin

Turnitin is a plagiarism checker. For the last two years, they've been working on an AI detector, which means their AI detector is trained primarily on output from GPT3 (as ChatGPT was only released in late 2022).

Turnitin claims their AI detector has a false positive rate of less than 1%. The problem with that? Prior to them turning on the AI detection feature, they wouldn't let anyone test the feature out. That's kind of a problem because what is happening now is that they're beta testing their feature on unsuspecting students. And according to a Washington Post article, it doesn't really work as advertised. But universities weren't really given the option to turn the feature off (universities in the UK were able to opt out, but it took a lot of work, a lot of backlash, and the opt-out movement did not reach the States).

But I decided to take Turnitin at their word. They claim their false positive rate is less than 1%? Well, okay.

So I created a hypothetical university that I called Sample University. I based it on statistics available from UC Davis, which has about 31,000 undergrad students. Assuming that the chance that any single paper gets flagged by Turnitin is less than 1%, I found between 3% and 14% of honest students would get falsely flagged over the course of a single academic year. In other words, between (roughly) 930 and 4,340 students would be referred per year. That's a lot of students. A lot of new cases for the OSSJA to juggle. And a lot of innocent students facing undue stress.

The Emotional Toll

My brother, who did not suffer from severe mental health issues through most of college, had full-blown panic attacks during this period of his college career. I watched him break down while trying to write an essay for that class. Not only that, but his emotional distress was dismissed - he was the "only" student they'd seen that had such a strong response, most students weren't so affected.

But that's blatantly and willfully ignoring college depression statistics. An estimated 30% of college students experience depression. Of that 30%, 13% had suicidal ideation, and 5% had plans to commit suicide.

When I was in college, I was part of that 5%. I had a plan to commit suicide, and there was very little keeping me from following through. I can say with absolute certainty that if I had been falsely accused of cheating, I would not have done the smart thing. I would not have gone to my parents for help. I would not have been able to face a judiciary board and prove my innocence. I would have confessed to the charges, gone home, and followed through with my plan. My parents would have brought me home in a body bag.

Now let's go back to Sample University. Assume 9% of innocent students are falsely flagged by Turnitin (this represents a false positive rate of 0.75%). Then say that only 20% of those cases are referred to the judiciary office. That translates to 558 students referred in one academic year. If college depression statistics hold, than an estimated 9 of those referred students have a plan to commit suicide. How many of those students will be so devastated by a false accusation that they'll follow through with their plan?

That's just one campus. There are about 15 million undergraduate students in the United States. 15.3 million high school students. How many vulnerable students, already prone to suicidal ideation, will see a false accusation as a reason to move forward with a suicide plan? Is that a death toll that we as a country are ready to accept? I sincerely hope not.

The Current Situation

My brother's case is now closed. Stacy Fahrenthold, through her actions and inactions, has made it clear that she still believes that William cheated. Her claims continue not to add up, and she seems incapable of realizing that she has defamed my brother multiple times. Throughout this process she has acted unprofessionally and proved that she is not only a bad teacher, but a bad person.

But the issue is larger than Stacy Fahrenthold now. The issue is about the other students who will be falsely accused. In fact, just recently my brother was contacted by a UC Davis student who read the article about him in USA Today. This student was accused of cheating because Turnitin's AI detection feature flagged an assignment they wrote. This student also has Google Docs edits, so we feel confident they'll be able to beat the charge. But it's still a lot of stress, and we can't defend them as well as we could William because we have no way of testing Turnitin's AI detector. I highly doubt Turnitin's AI detector has as low a false positive rate as it claims.

But it doesn't matter. Even if it's false positive rate is as low as Turnitin claims (again, doubtful to the point of being false advertising) any false positive rate for a single paper translates to a significant portion of innocent students being falsely flagged. At the same time, Turnitin's AI detector is easily defeated by AI paraphrasers. So dedicated cheaters get away scot-free, and innocent, honest students suffer through the stress and humiliation of a cheating accusation.

It is irresponsible of universities to continue using AI detectors. It is unethical of GPTZero and other companies to sell AI detectors that cannot have as low false positive rates as they claim. It is unethical for Turnitin to test its AI detector on unsuspecting students without giving universities a choice in the matter. It is unprofessional of teachers to refer students for disciplinary action based on AI detectors without first speaking to students. And it is unfair to students that they have to be caught in the crossfire of an AI arms race.