AI put "synthetic quotes" in his book. But this author wants to keep using it.

I wish I knew how to quit you

Journalist and author Steven Rosenbaum has more reasons than most to distrust AI.

His new book, The Future of Truth: How AI Reshapes Reality, is all about “how Truth is being bent, blurred, and synthesized” thanks to the “pressure of fast-moving, profit-driven AI.” Yet a New York Times investigation this week found what Rosenbaum now acknowledges are “a handful of improperly attributed or synthetic quotes” linked to his use of AI tools while researching the book.

These quotes include one that tech reporter Kara Swisher told the Times she “never said” and another that Northeastern University professor Lisa Feldman Barrett said “don’t appear in [my] book, and they are also wrong.” Rosenbaum is now working with editors on what he says is a full “citation audit” that will correct future editions.

Speaking to Ars in the wake of the controversy, Rosenbaum says he “learned a lesson” and is “going to be much more suspicious” and “reticent to trust” AI outputs going forward.

But he also can’t tear himself away from the tools. Rather amazingly, Rosenbaum is not interested in going back to the AI-free research process he used to write previous books.

“The idea of taking X years off [from AI] while it sorts itself out, and going back to, like, Microsoft Word … it’s just not in my nature,” he told Ars. “[AI] is magical. Because it connects, it knits together ideas and gives you pathways to think about things that you’re not going to come up with on your own.”

It’s also magical in another way: Like J.R.R. Tolkien’s One Ring, AI convinces many of those who use it that they can control its power properly. But can they?

Slipping through the cracks

Rosenbaum used AI tools during his writing process, he told me, “to surface ideas, locate articles, summarize themes, identify people or papers I might want to look into.” He draws a hard line between this kind of research and the “actual reporting, narrative structure, interviews, arguments, and conclusions in the book,” which he says are “entirely mine… There was never a time when AI was writing the book.”

In addition to chapters based on transcribed interviews that Rosenbaum says he conducted himself, The Future of Truth also includes more research-based chapters in which Rosenbaum said, “We’re pulling facts and then knitting them together into a narrative.” Tools like OpenAI’s ChatGPT and Anthropic’s Claude were used heavily to gather information, he said, with any nuggets mined by those tools tagged with a “this came from AI” warning in his notes.

It’s strangely creative and crafty and unusual in all these ways … and then it betrays you in ways that are just really quite horrible.

Steven Rosenbaum

Those tagged AI-generated notes were then passed on to a fact-checker and two copy editors provided by the publisher, Rosenbaum said. Of the 285 outside citations in the book, six have been identified by the Times as problematic, including three so-called “synthetic quotes” that have no apparent source. (More examples could turn up as the book undergoes further review. And it’s worth noting that most writers manage to include zero made-up quotes when they write a book.)

“I think we did that [double-checking] incredibly effectively, but not a hundred percent,” Rosenbaum told Ars. “We’re doing the work, we’re doing the best we can. We look at it, it looks right. We double-check it, and then we made a mistake.”

But the significant failure here highlights how the traditional fact-checking process might be ill-equipped to handle AI-assisted research. In the past, a fact-checker could be reasonably confident that any author quoting cited written works had simply copied down those quotes directly. These quotes would need to be checked, of course, but the fact that they’re so easy to verify makes them less inherently suspicious. If AI tools are involved anywhere in the pipeline, though, that assumption goes out the window, and there needs to be an extra layer of skepticism that those quotes had been copied correctly or that they even exist at all.

The widespread adoption of AI tools among writers of all stripes also comes at a time when financially pressured newsrooms and publishers are increasingly cutting copy editors and fact-checkers from their workflows. We’ve seen how AI-generated errors like this can make it into a published book even with a fact-checking layer. The risk of using those tools only increases for the many books that never go through any fact-checking before publication.

Rosenbaum, for his part, agreed that “publishers are going to need new verification workflows designed specifically for AI-era research. That probably includes mandatory source tracing for quotations, better provenance tracking, clearer standards around AI-assisted research, and potentially (more irony here) AI tools that audit citations against primary materials.”

“I didn’t set out to fabricate anything,” Rosenbaum continued. “What happened is what increasingly happens to journalists, students, researchers, lawyers, and authors working with these systems every day: [There was] AI-generated information that looked authoritative, and some of it made its way too far downstream before being caught.”

Cursing at the machine

Instances of prominent AI-generated errors are becoming distressingly common across a number of fields. Last year, the Chicago Sun-Times printed an advertorial summer reading list full of non-existent books dreamed up by AI. The New York Times recently had to issue a significant correction after published quotes attributed to Conservative leader Pierre Poilievre turned out to be “an A.I.-generated summary of his views.” Publications including Wired and Business Insider pulled down multiple articles attributed to “Margaux Blanchard” that appeared to be AI-generated. Scholarly conferences have been beset by papers with hallucinated citations, and pre-print clearinghouse arXiv has recently implemented a zero-tolerance ban policy to try to stem the problem.

Ars itself is not immune to this problem. Earlier this year, we retracted an article after a former reporter used an AI tool designed to extract verbatim quotes from a source’s blog post—but the tool instead generated fabricated versions of what the source actually wrote. (The latest version of our AI policy can be found here.)

The irony of an author incorporating AI-generated falsehoods into a book on AI’s reality-skewing effects is not lost on Rosenbaum. “I appreciate the book getting some attention, but this would not have been my choice about how to get it,” he said.

While that irony is “uncomfortable,” he’s quick to spin it as “also instructive. The fact that someone writing critically about AI and verification could still encounter these failures tells you how pervasive and persuasive these systems have become.” Rosenbaum’s own issue with AI “demonstrates the problem more vividly than any abstract argument could,” he said.

Perhaps. But if we accept this take, every avoidably obvious mess in the world might be a disguised good because it really helps illuminate the huge mistake. And that can’t be right; sometimes “negligence” is just that.

When asked directly how he could succumb to some of the AI-related problems his own book warns about, Rosenbaum described what sounds like a dysfunctional relationship with a charming charlatan.

“As a writer, AI is often a delightful writing companion,” Rosenbaum told me. “When I say ‘writing companion,’ I don’t use that lightly. It’s strangely creative and crafty and unusual in all these ways… and then it betrays you in ways that are just really quite horrible.”

Throughout our conversation, Rosenbaum frequently cited examples in which obvious AI errors left him enraged and literally cursing at the machine. Those date back to 2022, when Rosenbaum said he started experimenting with AI tools for “little research projects.” At the time, he found AI answers “spectacularly useful” about 8 out of 10 times, with the remainder being confabulations that were “just not true.”

Despite these errors, he kept using the tools in his life and work. When we talked on Tuesday, Rosenbaum said he had recently asked an AI tool to extract his “no changes, verbatim” speaker’s notes out of a slide deck so he could use them for an upcoming presentation. He was about to print those extracted notes when he realized that the LLM had actually rewritten his words despite his “very clear instructions for the robot.”

“And I say to it, ‘Did you rewrite the words?’ And it says, ‘Well, I just made the language a little stronger.’ Well, pardon me, but like, fuck you!” he said.

Even in the face of these kinds of profanity-inducing errors, though, Rosenbaum still believes that AI tools are too efficient not to use.

“The deck was 100 pages,” Rosenbaum said. “To cut and paste page by page, the text from each page would have been an hour’s worth of work, of mindless cutting and pasting. ChatGPT did it in about four seconds.”

To which the obvious retort might be: Yes, it was fast.

But it was also wrong.

Getting off the motorcycle

The efficiency gains might be worth it when the only stakes are personal presentation notes. But The Future of Truth shows how the balance between AI’s reliability and apparent speed should be weighed very differently when it comes to research that ends up in a published book.

As we continued our conversation, I kept coming back to that accuracy/efficiency trade-off, which Rosenbaum seemed to recognize as a problem at some level. Even as he called AI’s research help “magical” and “delightful,” he described dealing with AI’s confabulations and ignored directives as a “pernicious and exhausting” struggle.

“It leaves you… uncomfortable almost any time you’re using it,” he said of its tendency to ignore clear instructions.

“I’ve never fought with tech before this, honestly,” he said at another point. “And I use it extensively.”

I’ve never been in a place where I thought the tech that I was using was both intoxicating and dangerous…

Steven Rosenbaum

Given the issues with his new book, I asked if the risk of introducing inaccuracies that you might not catch was really worth the perceived benefits.

“I don’t do drugs, and I don’t drink, but I presume that that’s kind of the question an addict asks when they’re having one drink too many and they know they are,” Rosenbaum said. “I’ve never been in a place where I thought the tech that I was using was both intoxicating and dangerous. And I wrote the book specifically to raise that concern, so if I end up being the poster child of not being aware of the guardrails, so be it.”

At one point, when discussing the relative risks and rewards of using AI, Rosenbaum noted that he rides a bicycle but wouldn’t ride a motorcycle. “I know a motorcycle gets me places faster. I think it’s dangerous and I might die. And that’s why I don’t own a motorcycle,” he said.

Rosenbaum made it clear that using AI was the relatively safe “bicycle” option in this analogy. I responded that the supercharged efficiency and catastrophic risk inherent in using AI made it feel a bit more like the motorcycle. Rosenbaum said “that might be fair” and thanked me for “sharpening” his analogy.

I then asked the obvious question: Are you going to keep riding the motorcycle?

“Can I get back to you on that?” he said.