ChatGPT Essay Detection
ChatGPT Detector
How can we English teachers detect ChatGPT-produced essays? We can bury our heads in the sand like snooty ostriches and claim that our superior literary backgrounds will detect artificial essays submitted by our students in the new year, but having spent some time researching this artificial intelligence (AI), I will say that it’s time we English teachers need to admit that “Houston, we’ve got a problem.”
To produce an AI essay in ChatGPT, simply copy and paste a writing prompt into the textbox on this site: https://openai.com/. However, before you do so, I suggest reading the rest of the article to find answers to some of the questions you will have as you start to play with this tool.
I selected a commonly-used writing prompt used with The Outsiders and pasted the following in the ChatGPT textbox:
In S.E. Hinton’s The Outsiders, each of the Greasers could be considered a hero. Include your own definition of hero in an essay, and provide textual evidence to explain how Johnny, Ponyboy, and Dally matches your definition. Include MLA citations with page numbers.
The results? A coherent, well-written five paragraph essay. A bit formulaic and repetitive. But that’s not the point, is it? As a first draft, the essay is more than passably good. With revision, the essay could be AP A quality. I’ll share one of the body paragraphs as an example:
Dally is a hero because of his determination. Despite his tough exterior and rebellious behavior, Dally is deeply loyal to his friends and willing to do whatever it takes to help them (Hinton 48). When Johnny and Ponyboy run away, Dally helps them by giving them money and a place to stay (Hinton 66). He also helps them come up with a plan to turn themselves in and clear Johnny’s name (Hinton 76). Dally’s determination to stand by his friends and do what is right, even when it is difficult, makes him a hero.
Works Cited Hinton, S.E. The Outsiders. Viking Press, 1967.
And, yes, I generated another essay from the same prompt. Results were equally good, if not better, because the AI learns from its experience. It predicts based upon previous writing samples and so improves. Additionally, the essay content was largely different than that of the first essay. Teachers also say that the AI will vary the vocabulary and syntax to reflect that reflective of specific grade levels.
Furthermore, the ChatGPT is quite capable of revising student first drafts. I found it alarmingly, surprisingly, encouragingly good at syntax revision, for example.
In the December 9, 2022 issue of The Atlantic, author Daniel Herman (a high school English teacher in the San Francisco Bay Area) shares this AI capability:
I passed it (a student’s first draft of an essay) along to OpenAI’s bots. “Can you fix this essay up and make it better?” Turns out, it could. It kept the student’s words intact but employed them more gracefully; it removed the clutter so the ideas were able to shine through. It was like magic.”
Possible Solutions to Chat GPT-Produced Essays
1. In the December 29, 2022 Washington Post article, “Handwritten Essays Defeat CHATGPT,” a link to an opinion piece, “Here’s how teachers can foil CHATGPT,” author, Markham Heid, suggests that in-class handwritten essays can solve the plagiarism problem of artificially-produced digital writing. Moreover, the author posits that handwriting produces valuable cognitive benefits. Heid states,
For one thing, neuroscience research has revealed that, to the human brain… stimulates greater activity in a broader group of brain regions when compared with typing. However, the author admits that “much of the research comparing the differing neurological effects of handwriting and typing has focused on children or younger students.”
2. Fight fire with fire. Artificial intelligence can also be used to predict with remarkable accuracy the likelihood of “predicted probabilities” that an essay has been created via AI. I used the https://huggingface.co/openai-detector and found upwards of 99% probability that an essay had been generated by ChatGPT with numerous submissions. Of course this company does require payment after numerous submissions; however, it shows what may be possible to detect cheating that is currently beyond the scope of TurnItIn or Google. Additionally, the software builds on its expertise by experience and lists the number of “tokens” (I presume similar submissions) to provide teachers with some sense of reliability.
3. TEACHER SOLUTIONS?
I posed this question on several FB groups and received some brilliant responses. I will summarize for the sake of brevity and keep names private:
- Give an on demand essay to serve as a baseline. Easy to compare with AI-generated essays and provides good backup for teachers who may be subject to student and/or parent pushback.
- Focus on smaller segments of writing craft. Perhaps one paragraph in-class. Know your students’ writing and you won’t get fooled. For example, students who consistently make capitalization, comma, or subject-verb errors are easy to spot. However, teachers report that telling the program to include errors in spelling, grammar, etc. typical of specified grade levels and the AI integrates these into its product.
- Use screen monitoring tools, such as GoGuardian provides, or Google draft histories. I hear Draftback is an easy-to-use app to monitor writing history in Google docs. However, these won’t help with students opening separate windows and manually writing or typing from the AI-generated sources.
Making Lemonade out of Lemons
Perhaps rather than looking at AI and ChatGPT solely as opportunities for devious students to plagiarize or avoid writing essays and trying to figure out ways to outsmart this new threat, we might do better if we embrace the tool and design our instruction accordingly. A few ideas…
- The AI is quite good at lower level cognitive tasks, such as data collection. To use Jane Schaffer’s writing terminology, the ChatGPT excels at gathering concrete details and citations. Less time thumbing through a novel for quotations. Less time Googling “global warming warning signs” on dozens of sites (and ads). More time analyzing the data and composing original commentary on the concrete details. More time revising word choice and syntax to create essays will original style and voice. In other words, more time for teachers to teach the important stuff about writing.
- As a reading specialist, I’m excited to see AI used to produce relevant background knowledge. Think of it as a filter (and teach students how to do so) to refine a directed task for ChatGPT. Via trial and error I got more and more out of the AI with more specific parameters in my directions. These refinements will lead to the AI learning how to produce a quality knowledge base. Having access to this background information is key to advanced reading comprehension.
- Another idea is to use ChatGPT to provide drafts for revision. I find this solution enticing, as it’s the revision step of the writing process that improves students’ writing sophistication and provides us the opportunity to share our writing teacher set of tools that we can share with our developing writers.
***** ADVERT*****
Better Writing Feedback
One such set of revision tools that will teach students the art and science of writing revision for both student-produced and ChatGPT-produced essays is my e-Comments Chrome Extension. This easy-to-use app allows you to insert hundreds of canned comments from each of four different comment levels (including AP) into Google docs and slides with just one click from the pop-up e-Comments menu. Each instructional comment identifies, explains, and shows your writers how to revise a specific writing issue in stories, essays, and reports. These comprehensive comments don’t simply flag errors, they help your writers learn and improve. Plus, you can add your own comments! Works in all LMS and online Word 365. Add links, audio and video comments, and record the screen. Check out the free 10-day trial on the Chrome Web Store.
“An incomparable time saver for annotating student essays. Custom comments are easily added and modified, and each generation of this extension has improved significantly upon the last. Truly excellent.” K Mason Schecter
“This is an amazing extension. It not only allows you to grade quickly using ACTIONABLE feedback, but it also allows you to leave video and audio feedback all in one place! AWESOME!” Dr. Desiree Alexander
“Open an assignment in Google Classroom and it’s ready to go! Furthermore, create custom sets aside from the existing (overwhelmingly thorough and comprehensive) sets. While the pre-loaded sets deal with writing and conventions, the customizable sets could easily be utilized by teachers from other content areas who wish to expedite the grading of short/long answers and essays with comprehensive feedback. This extension isn’t just for English teachers – it’s for teachers.” Adam Huttenga