ChatGPT outscores human authors in new fiction study

Researchers Sydney Sears and Deena Skolnick Weisberg published a study in the journal Judgment and Decision Making involving over 2,500 participants across three experiments. In the first phase, 1,682 readers evaluated 1,000-word short stories. Three stories were selected from established literary magazines, while three others were generated by ChatGPT 4.0 using prompts that mirrored the style and themes of the human originals. Half of the readers were told the true author, while the other half were given false attribution.

The results showed that ChatGPT 4.0 outperformed human writers on a scale ranging from minus 3 to plus 3. For perceived quality, the AI-generated stories scored a mean of 1.54 compared to 0.97 for human texts. In terms of reader immersion, the AI stories led with a score of 1.42 against 1.00 for human-authored pieces. However, a clear bias emerged: regardless of who actually wrote the text, participants gave significantly higher ratings when they believed a human was the author. In two subsequent experiments with 905 participants, readers attempting to identify the source of the texts performed no better than random chance.

The researchers noted that AI-generated prose is often more fluid, emotionally positive, and simpler to digest, whereas high-quality human literary fiction is frequently complex and challenging. For creative practitioners and publishers, this study suggests that generative models have already mastered the structural and stylistic elements that satisfy general audiences. While professional readers can still spot generic AI outputs, previous research from Stony Brook University and Columbia Law School indicates that when models are fine-tuned on specific author styles, experts prefer AI-generated text eight times more for style imitation and twice as much for overall quality. This shifts the practitioner's focus from basic drafting to highly customized style training.