The discourse around LLMs is so focused on the future — what models will be able to do, with enough dollars or megawatts or datasets — that it can be easy to overlook their past. But original-flavor ChatGPT is already turning four years old in November! It’s even been nearly a year since Anthropic’s hype supernova around Claude Code eclipsed OpenAI in the public eye. You hear “I asked ChatGPT” less and less often, and “Claude said…” more and more.

This does not necessarily have anything to do with the quality of one product over another. Instead, Claude has gained popularity because it’s easier for users to anthropomorphize something with a human name.

Anthropic has leaned into that tendency with its entire might, and anyone who wants to think clearly about these things needs to push back against language that incorrectly grants the machine personhood and agency. Getting into the habit of writing is actually a fantastic antidote:

If you worked in a googly eye factory, you would become relatively immune to the anthropomorphic effect of putting googly eyes on inanimate objects.

But unfortunately, framing Claude as a colleague instead of as a computer program has broader consequences than tired debates about whether language models are “alive.”

From workslop to warslop

In the distant past (the year 2022), when someone handed you a document, you could trust its contents to be accurate. Or at least, you could estimate its accuracy based on how much you trusted that person. It was more or less safe to use that doc as an input into your own work, because the accountability lay with a real human being. And if something in the doc didn’t make sense, you could call them up and ask.

Today, you are likely to hear back: “I don't know, Claude wrote it.” What appears to be a confident statement of fact made by a professional is unmasked as a work of fiction with the least bit of pressure — but critically, not any sooner, meaning that information could go through multiple rounds of hand-offs before anyone bothered to ask if any of it is real.

CEOs who were bullish on AI as recently as earlier this year are now calling out the damage that “slop grenades” have done to their businesses. The blast radius of thoughtless human-laundering creates hours of rework for everyone on the receiving end, and the scale of the effect is only increasing with time. In a way, this is like the “10x engineer” of yesteryear: every person who feels like they are more productive because of AI only does so because they can shift the consequences of cutting corners onto someone else.

The most extreme example of this comes to us from the United States military, where an analyst generating a report with AI did not bother to check whether the claim of nuclear materials on a Chinese ship bound for Iran was actually true (and possibly did not even read that the report said that before hitting “send”). Naval commanders in the Middle East had no way of knowing what parts of the reports they received were essentially made up, and were much further removed from the actual data than the person who decided not to do their job.

This time, things worked out. But what about next time?

LLMs obscure the provenance of agency

When one considers Claude.dmg (a program file on your computer) to be a colleague, it’s easy to think of its workings as a black box, and interrupt any critical thinking process that might be taking place. “Yeah, Claude came up with that. I don’t know why. It’s pretty smart though, so I’m sure this is correct.” We see this anthropomorphization happening in the headlines about OpenAI hacking Hugging Face (with Anthropic and Google chiming in to say that their models also hacked other companies, and are just as super cool and powerful and dangerous as GPT-6).

Eryk Salvaggio calls this the “system from nowhere”:

If you see AI as a system from nowhere, you can make the claim that the model “went rogue,” and that it “broke containment,” both of which place agency and decision-making onto the model itself rather than the people who set the stage for that behavior. 

When you expand the boundary of the system to include the people building and deploying it, the case becomes much less science fiction and more like incompetence.

An LLM does not have agency. Full stop. But this does not mean that it is “just” a tool that’ll do exactly what you ask, and no more. Like any tool, LLMs are designed to behave in a particular way and do a particular thing. And like any other tool, some of the ways they change user behavior are intentional and some are not.

Consider for example the frontier labs’ desperation for training data, leading them to hoover up sources like Russian disinfo, Iranian propaganda, or any old thing someone decided to post on Reddit. The provenance of this data is conveniently erased; it is laundered by the model & spit out at users, producing measurable shifts in attitudes and beliefs.

It is no surprise that using AI as a “writing assistant” ends up completely changing what users were trying to say — even when explicitly prompted not to do that:

Even when LLMs are prompted with expert feedback and asked to only make grammar edits, they still change the text in a way that significantly alters its semantic meaning.

More habitual LLM users are already asking Claude to tell me that you can simply engineer your prompt in such a way that you get only good responses instead of bad ones. Except there’s absolutely no guarantee that the prompt you enter is the prompt the model actually receives; the practice of shadow prompting is as old as LLM products themselves.

Branding Claude as a colleague and even collaborator neatly obscures these issues. But as soon as we buy into the idea that a model acts independently of — or even contrary to — its manufacturer’s wishes, as soon as we start to believe “Rogue AI” is real, we immediately absolve that manufacturer of the harms they use their own, human agency to cause.

While the machine may have a significant degree of independence concerning the means by which it achieves its ends, the ends themselves are always chosen by at least one human agent, whose interests may differ from those of the individuals interacting with the machine.

The “black box” is load-bearing for AI industry narratives

In recent years, the rhetoric has shifted from attributing agency to individual models, to attributing agency to the idea of AI itself. “AI is coming, AI is dangerous” is a neat rhetorical device that once again redirects responsibility from the companies building the AI to some intangible notion of a unified artificial intelligence. To listen to AI boosters speak, it is a force of nature that mere humans cannot hope to understand.

The result of this branding is a sort of fatalism disguised as optimism: it’s happening, so you might as well get on board. You don’t want to be caught unarmed when your colleagues start slinging slop grenades in your direction. You don’t want to be the solitary fool trying to hold back the “wave of AI”:

What does AI as a "wave" imply is a reasonable course of action? … And what does it imply would be an unreasonable response? To think you have any say over the wave itself: its pace; its trajectory; what it should be allowed to do and not allowed to do.

But if we were able to think of AI tools as products, ordinary systems built by people, we start to see that nothing about the choices those people are making on our behalf is inevitable. We can always make different choices about how transformers are productized, and how we change our own conduct to accommodate those products’ intrusion into our lives. We can always set the boundaries of those spaces where we are willing to entertain this technology, and spaces where incursions of AI can (or even must) be weeded out.

This starts with thoughtfully reflecting on what impact AI is actually having on your practice, rather than what the industry is telling you ought to be happening. Are you actually more productive? Are you getting to spend more of your time on higher-leverage tasks — and is that leverage reflecting in customer outcomes? Or are the tools leading you away from the necessary work?

Clustering and theming is something that requires you to be active. It's a spatial exercise. There's something in moving data around that helps you recall context. … I can talk through every decision, why I've prioritised one thing over another, and how an insight speaks to three unrelated things AI would never put together.

Nothing is inevitable. The predictions we made about the future hardly ever come to pass; wondering what will happen is another way of folding your arms and doing nothing. Instead, look around, and ask yourself: is this what I want to happen next? And if not — what is?

— Pavel at the Product Picnic