Despite widespread complaints about AI-generated content, a new study suggests that readers frequently cannot tell the difference between human and machine-written stories—and may even prefer AI text when they believe it was crafted by a person. The research, published in the journal Judgment and Decision Making, found that participants rated AI-generated stories more favorably on measures of quality and engagement when they were told a human had written them, underscoring the powerful role of perceived authenticity.
Study Design and Findings
Dr. Deena Weisberg, senior author of the study from the Department of Psychological & Brain Sciences at Villanova University, and her team asked participants to read six short stories. Three were extracts from published books by human authors, while the other three were generated by OpenAI's ChatGPT. The participants were then asked to identify which stories were AI-written and to rate various qualities of the texts.
In one of the trials involving 424 participants, only 39.39% correctly identified the human-authored story, while 40.8% correctly spotted the AI-generated one. In a third trial with 481 participants, 51.97% correctly identified the AI story when asked more pointed questions. The results varied across tests, with accuracy rates ranging from roughly 31% to 67% depending on the specific story and question format.
Perceived Quality Over Actual Origin
The study also measured how participants rated the stories on scales from -3 (strongly disagree) to +3 (strongly agree). On average, AI-generated stories scored 1.54 for being well-written and 1.42 for story absorption, compared to human-written texts which scored 1.00 and 0.97 respectively. Crucially, these higher ratings occurred when participants believed the AI stories were human-written.
“It shows that people value authenticity,” said Weisberg. “People may also prefer art that has been created only by humans. I think that this finding also reveals the biases that people have about what AI systems are capable of.”
Why AI Writing Is Hard to Detect
Weisberg explained that AI models, such as large language models, learn to write by analyzing vast amounts of text from the internet, detecting patterns and predicting the most likely next word in a sequence. This allows them to mimic human writing styles with increasing sophistication, making it harder for readers to distinguish between human and machine authorship.
“AI does have a somewhat distinct way of writing, a style that we can learn to recognize with experience,” Weisberg noted. “And it is getting harder to tell this style apart from human writing, as new models are getting better at mimicking other styles.”
Implications for the Future of Writing
The findings suggest that while AI may not be inherently “better” at writing than humans, it is becoming increasingly difficult to tell the difference. The study does not conclude that AI surpasses human creativity, but rather that reader perception plays a significant role in how writing is received.
“So when they encounter a story that they like, they assume that it was written by a human and not an AI,” Weisberg added, “even though that may not be the case.”
As AI continues to advance, the study raises important questions about transparency, authenticity, and the value we place on human-made art. While some polls show distrust in AI's creative abilities, this study suggests that our biases may lead us to prefer AI work—provided we think it came from a human hand.
