The Wrong Fight over AI and Writing
The Binary Debate Misses the Only Question That Really Matters
A writer expostulating on truth in the age of AI used AI for his research and inadvertently passed off a bunch of bogus—that is to say, untrue—information in the process. Embarrassment, thy name is Steven Rosenbaum. His new book, The Future of Truth, features not only fabricated quotes but also misinterpreted source material.
Rosenbaum, who disclosed his use of AI for research in his acknowledgements, apparently didn’t do enough of his own to recognize the problem while writing or editing his book. An example, cited by Benjamin Mullin when covering of the story for the New York Times:
“The most sophisticated A.I. language model is like a mirror,” said tech journalist Kara Swisher, supposedly. “It reflects our own morality back at us, polished and articulate, but ultimately empty behind the surface. It’s not bound by Asimov’s laws or any ethical framework—it’s bound by the patterns in its training data and the objectives set by its creators.”
But when the Times asked Swisher about the quote, she denied ever having uttered the words. Rosenbaum’s chatbot made them up—and he never verified them.
Fabricated quotes are bad enough, but it seems to me there’s a deeper problem. A writer can profitably use AI to find sources, even summarize them, depending how those summaries are used. AI pioneers in the 1940s and ’60s, men such as Vannevar Bush and J.C.R. Licklider, imagined machine help with sifting through the vast literature relevant to researchers as a tremendous boon. I agree and wrote about this benefit in The Idea Machine. But if you haven’t bothered to read the sources for yourself, how can you be sure you even understand what you’re ascribing to them? Rosenbaum went astray here too.
The book attributes some quotes to neuroscientist Lisa Feldman Barrett and her book How Emotions Are Made, which I’ve read and is quite good. But the attributed quotes? Fairy dust. Worse, the way her quotes were phrased assigns ideas to Barrett that aren’t even hers or accurate. “Ms. Barrett said in an email to The Times that the quotes ‘don’t appear in the book and they are also wrong’,” said Mullin.
I feel bad for Rosenbaum. I hate seeing a guy shamed in public. But this is basic hygiene, right? Authors can get stuff wrong on their own. They don’t need large language models to misattribute quotes or misunderstand a claim; happens every day. But a conscientious author (which is, I assume, what we expect all authors to be) checks their friggin quotes. You double back in your endnotes to make sure you got things right. If bits look squirrelly, you check again. You might still get stuff wrong, but the number of such errors in Rosenbaum’s book doesn’t speak well of his method, chatbot or not.
What Rosenbaum did subverts the hope that people like Bush and Licklider possessed for machine-aided research. They imagined a human actor—scientist, researcher, author, whatever—vetting the outputs of their machines, not passing off the results sans scrutiny or additional cognitive labor.
ChatGPT “f---ed up the book,” Rosenbaum told Atlantic writer Will Oremus when discussing the situation. But that’s wrong. To quote that sage of the inner city, Ice-T (don’t worry, I checked this), “you played yourself.”
Of course, there’s more, because we all know the most recent raft of AI writing, concern about AI writing, and thus writing about AI writing is only the first float in what will prove a long and raucous parade.
At a recent conference, the Polish author and Nobel laureate in literature Olga Tokarczuk admitted to using AI, confessing surprise at “how fantastically it expands horizons and deepens creative thinking.” She said she asks her chatbot, “how could we beautifully develop this?” and called AI “an asset of truly unbelievable proportions.”
People lost their minds, which prompted Tokarczuk to later explain herself, throttling back her apparent excitement. “None of my texts, including the novel that will appear in Polish this fall, has been written with the help of artificial intelligence—except for using it as a tool for faster preliminary research,” she clarified, adding that she takes the step Rosenbaum skipped: checking the outputs.
Other writers admit use without hedging or hesitation. A couple of years ago Japanese novelist Rie Qudan won the coveted Akutagawa Prize with a story written in part by ChatGPT—5 percent’s worth, Qudan gleefully admitted. “This is a novel written by making full use of a generative A.I.,” she said in her acceptance speech. “I plan to continue to profit from the use of A.I. in the writing of my novels while letting my creativity express itself to the fullest,” she also said. Despite her announcement, the book went on to international fame through a string of translations—English, Italian, French, and German in 2025—and was well reviewed.
But authors can see what happens to people like Tokarczuk or Shy Girl author Mia Ballard in terms of public perception and so deny any use at all. When the prize awarded by the Commonwealth Foundation and featured by literary journal Granta went to a writer whose work seems very likely to have been written with AI, he denied it—same with the runners up, some of whose works contain the sort of constructions and tics chatbots tend to use. Oremus said he spotted the same sort of tics in Rosenbaum’s book. If you read enough of it, it starts punching you in the face as you hazard your eyes across the page or screen.
One of the things obvious from the AI writing discourse is that just like writing itself, using AI isn’t one thing, nor does it serve one purpose. Rosenbaum, Tokarczuk, and Qudan have all used AI but have done so differently and to different effects. And there are countless other authors pursuing countless other strategies. When we adopt a binary position on the question, we’re missing the diversity of use and purposes at play.
Where do you draw the line—and why there and not somewhere else—and how could you possibly police it once AI has oozed into every part of the process and there's no telling what’s been touched, or how, or how much? Oremus gives one amusing example:
Consider that The New York Times, which has endured a spate of AI writing scandals, maintains two different standards. Its freelancers can use AI tools for “high-level brainstorming” and almost nothing else. Newsroom employees are encouraged to experiment with what the paper’s guidelines tout as “a powerful tool that, like many technological advances before it, may be used in service of our mission.”
At what point does high-level brainstorming shade into nitty-gritty research? Is there a ruler for that? Somebody got a metric? This strikes me as an unenforceable, CYA standard—and not one to which the paper holds its own staff. The rule exists to firewall the paper from contributors it can’t oversee, whose AI use it has no way to monitor. With its own writers it assumes a firmer grip. It’s not going to hamstring them; it just wants some measure of control, however illusory, over what ends up in its pages.
These sorts of complications will bubble up—are already bubbling up—everywhere. Barnes & Noble CEO James Daunt recently said he had “no problem” selling AI generated books so long as they’re flagged as such, a statement he also walked back a bit. But how will that work? Do Tokarczuk’s “beautifully develop[ed]” ideas count as generation? B&N currently stocks Qudan’s book with no mention of its 5 percent AI generation. Does that kick in at 6 percent?
How much machine help flips the switch? We don’t expect writers to tell us they used Google. But they didn’t find all that crap on their own; Larry Page and Sergey Brin’s marvelous machine went out and found it for them. As readers, we do expect authors to cite their sources within the conventions of their genre. That’s obvious enough for nonfiction. But novelists rarely share the sources of their research; it would get annoying if they did.
I’m uncomfortable and ambivalent about all of this. I don’t hate AI. I use it every day at work. And I use it here for research, fact checking, and proofing my work. Like Henry Oliver, I don’t have “a” view about this. Maybe we can AI-detect our way through this morass, as Kelsey Piper suggests in an insightful piece about the Commonwealth controversy for The Argument. It might work. Personally, I don’t think so; everything’s evolving too fast. Not to mention the downstream effects: If writers know that AI detectors flag x, y, or z as tells, they’ll shape their style to avoid the detectors, whether they’re using AI or not. So real live humans with moms and everything have to intentionally alter how they write—their style, whatever it is—to avoid the appearance of AI generation? That’s a distortion of human art I find more troublesome than AI generated text.
As I said, I’m ambivalent. But I’m increasingly of the mind that our focus is on the wrong thing to begin with. Where do we draw the line? I think we have to draw it the same place we drew it before Dawn of ChatGPT—the product itself. I personally don’t care how enthusiastic Tokarczuk or Qudan are about their LLMs, or even how they use them. I care about their artistic judgment displayed on the page. Is it any good?
The only metric that matters at some level is our personal appraisal of the words themselves. We don’t passively absorb text. We read it, actively scrutinizing and criticizing as we go. That’s how Rosenbaum’s flub was found out. And it would have been every bit as problematic if he didn’t use AI because the problem wasn’t AI slop; the problem was he was sloppy.
Sans any awareness of how an LLM may or may not have been used, that’s all we have anyway. If the words are up to par, we’re satisfied; if not, then we’re not. Does it have to be more complicated than that? I’d happily read whatever Tokarczuk writes next, same with Qudan. And should Rosenbaum ever write again, I’d read him too if I cared about the topic. Naturally, I’d also spot check his quotes.
Thanks for reading! If you enjoyed this post, please hit the ❤️ below, restack, and share it with your friends (especially if they’re opinionated about AI writing).
Not a subscriber? Take a moment and sign up. Free subscribers get two-week access to all new posts. Paid subscribers get access to the full archive, going back to 2022, plus an exclusive member chat and monthly member calls.
While you’re here, check out👇
Thanks, Joel! All I can say is: AI and writing: the slipperiest of slippery slopes.
"I personally don’t care how enthusiastic Tokarczuk or Qudan are about their LLMs, or even how they use them. I care about their artistic judgment displayed on the page. Is it any good?"
With respect, I think I disagree. This stance fails to grapple with the fact that writing has long had a strong social or communal character. In my own field, I see the extraordinary literary production of Greek writers in the 5th century B.C. and Roman writers in the Age of Augustus. One suspects that without Herodotus, no Thucydides; without Sophocles, no Euripides; without Plato, no Aristotle; with Vergil, no Ovid. So we have to ask: what is supportive of a vibrant community of writers who are continually influencing, inspiring, and challenging one another to higher and higher levels of achievement? Put differently, it can't be just about the product; it has to also be about the people behind the works. A writer who looks to her preferred AI—not to her friends and rivals who are nearby likewise sweating over their manuscripts—for creative sustenance is not giving anything back. In effect, her relationship to her cohort is parasitical. She wants to benefit from what that community generates, e.g., a kind of nursery for creativity and a reading public eager to buy books, without contributing to it. She thinks, "I have my AI. What do I need them for?"