Writing is thinking
A common defence one hears for AI-assisted writing is that ‘the idea was mine’, that the large language model just handled the job of expression, as though such expression were a clerical task that follows thought the way typing follows dictation. It’s an account of thinking that suggests ideas sit fully formed in some inner chamber, just waiting for language to come along and get them ready for the public.
But this isn’t how the expression of ideas works, and anyone who takes writing seriously knows that what precedes articulation is something far vaguer and more unstable than the finished sentence retroactively suggests. As the old E.M. Forster quote goes: ‘How can I know what I think till I see what I say?’ (Forster actually borrowed this saying from Graham Wallas). Or as Joan Didion put it: ‘I write entirely to find out what I’m thinking.’
Linda Flower and John R. Hayes, in their hugely influential ‘A Cognitive Process Theory of Writing’ , observe that writers routinely discover their goals in the act of drafting, that the original plan changes because sentences push back, and it is in that pushing that much of the intellectual work gets done. Sondra Perl, building on the work of the psychotherapist Eugene Gendlin, calls the pre-conceptual bodily awareness writers repeatedly consult during composing the ‘felt sense’, a bodily, pre-verbal orientation towards a subject that needs to be tested against emerging sentences. An idea is thus not real until it has been committed to a form that a second party can reliably evaluate.
What the person has before they prompt an LLM is a felt sense, a hunch that something interesting lies in the vicinity of a topic. Don’t get me wrong, hunches are valuable, and every knowledge contribution or creative work begins as one—but a hunch is compatible with thousands of articulations, many of them mutually contradictory, and the work of writing is the work of discovering which of those articulations you can actually defend and which ones turn out on inspection to be two ideas awkwardly fused or no idea at all. When an LLM performs the fleshing out, you receive one articulation from that space of thousands, selected by processes that have nothing to do with your judgement, and you have no way of knowing whether it is the articulation you would have arrived at, because the essential process is one you skipped. The retrospective claim of ownership is unfalsifiable, but in a way that flatters its maker. Sure, the prompt or topic might be ‘yours’, but whether you own the written argument, the thing that can actually make any of it real, that, to me at least, is the real measure of authorship.
Andy Clark and David Chalmers argued in the late 90s for what they called the extended mind, that a notebook consulted reliably enough becomes a genuine part of one’s cognitive system and that we have always thought with tools. LLMs, some might argue, are simply the latest notebook. But Otto’s famous notebook (from Clark and Chalmers’s thought experiment) stores what Otto has already processed through his own cycles of attention and revision, so the tool is scaffolding a process that remains his. An LLM, on the other hand, doesn’t scaffold the process from felt sense to articulated position (ie. where the thinking happens), it just does it.
One might argue that dictation poses a problem for my argument; after all, Milton is understood to have dictated Paradise Lost and Henry James dictated his later novels, and nobody doubts their authorship. But dictation routes composition through the author’s own language production as the sentences are formed, spoken, heard, revised, re-spoken; and the well-documented change in James’ late style—those enormous qualifying clauses that betray the syntax of a man thinking aloud—is if anything evidence for my case, because it shows that the medium of composition reshapes the thought itself, that if you change the process you change what gets thought. Hand the process to a statistical model of everyone else’s prose and what gets thought is an average of all that has come before it.
There is also the reader to consider. Prose is a record of decisions: a transition between paragraphs implies an inferential move someone made just as a particular example implies a survey of possible examples and a selection among them. When we read a piece of writing in good faith, we treat these features as evidence of a person having worked through a problem, and we extend a kind of credit on that basis, assuming the author noticed the obvious objection, weighed it, and has reasons we could ask for. LLM-generated prose claims a history of deliberation that never occurred, and the claim isn’t a harmless because it takes, without warrant, the credit readers extend to writing, and every piece of work that draws on it fraudulently makes it a little harder for the next writer to be believed.
There are plenty of legitimate uses of LLMs, particularly where there are elements of writing where no thinking is displaced: reformatting references, for example. And plenty of human writing was thoughtless long before LLMs emerged; anyone who has read a grant application knows that boilerplate can be produced by hand, on autopilot, by people who would struggle to defend a single sentence of it. ‘Writing is thinking’ can itself curdle into piety if it is taken to mean that all writing is deep thinking, when we all know that, in practice, it isn’t. But that’s not my argument. My argument is that when thinking happens at all in the making of prose, it happens in that making, in the friction between what you meant to say and what the sentence turned out to say, and there is no version of the process in which the thinking is complete before the words are found.
The person who says ‘but it was my idea’ when they used an LLM to do the writing is describing a sensation, and the sensation of having an idea is cheap—everyone has ideas, the challenge is in articulation. Writing is where you find out whether an idea survives contact with its own implications, and if an LLM does the writing, then the machine did the thinking—or not even the thinking, the performance of thinking. That may still be worth publishing, occasionally, but it is worth being honest about what it is (which is not writing).