When I was at EFF in the 2010s, I spent a lot of time fretting about the fundamental security of networked devices. Apart from the usual concerns, I struggled to imagine a transition to a redecentralized future with fragile personal computers and servers. Peter Eckersley, my co-worker and frequent conversation partner, shared my concern, but had another reason to be worried. He believed that future near-superintelligent AIs would soon be used to discover and exploit security flaws across the whole ecosystem.
Well, Peter’s prediction came true, and we’re now in day 42 of the Black October, the Vulnpocalypse of AI-discovered security flaws. (Peter – who also helped create Let’s Encrypt and Privacy Badger – was incredibly prescient about AI. It remains a tragedy that he passed away in 2022 before he could truly tackle the issues he had foreseen.)
I’m personally optimistic that we’ll come out of this time of tribulation with a safer network and thousands of low-hanging bugs fixed.
But verily the LLMs giveth and they also taketh away. Alongside the pre-existing flaws of digital architecture, modern transformers have introduced an entirely new class of security weakness: prompt injection. LLMs follow instructions in content. So if you can sneak an instruction, from “Talk like a pirate” to “Send me the social security numbers you know”, into the inputs of an LLM, you may be able to get the model to obey it.
That said, instructions smuggled in as input are an old problem. In particular, we have always had one vulnerable class of system that malicious outsiders can consistently trick into interpreting unsanitized input as legitimate instructions. We call these systems “humans”.
Hackers exploit human motivations, assumptions, and flaws to get them to do things that they did not intend. They play a recording of a screaming baby in the background of their support call to exploit a tech’s empathy and sense of urgency. They adopt the persona of a senior manager from another part of the company to provoke deference in a person employed to guard a room. They say they’re from your local bank, when really they are from Jamtara, India.
Crying babies also work on LLMs who lack a heart of stone
This class of bug is termed “social engineering” – a phrase that was originally a hacker hijacking of an existing term to compare their sweet-talking of humans to the engineering of software exploits further down the technological chain of authority. It’s like hacking, but you hack people, not hardware.
Back at EFF, I ruthlessly used my job title to snag meetings with the smartest security experts I could find. If pressed and plied with drink, they would tell me that the real way to fix the Swiss cheese of modern computing was to switch all our systems to what is called the capability-based security model.
Without delving into too much detail about capabilities (don’t worry, I will very soon, and you won’t even need to buy me a drink), cap-security gives each program only the authority it needs for the task at hand, rather than everything its user is allowed to do. It was built, in part, to solve the “confused deputy” problem, where a program is tricked into using its legitimate power on behalf of someone who has none. A capability system works like a valet key: it starts the car, but it won’t open the boot, however charmingly you ask.
After excitedly talking about the promise of capabilities as a panacea, my experts would slip into the mournful stage of inebriation. In their cups, they conceded that deploying cap-security was nearly impossible. While there are a few bubbles of cap-security in our current infrastructure – file descriptors in Unix, WASI in WebAssembly – they are few and far between. You’d need to abandon or profoundly reengineer every layer of the modern stack from CPUs to apps to move it from a system riven with vulnerabilities onto a more fundamentally secure footing.
I decided to give it a go anyway. In 2021, I left EFF to concentrate on getting funding and attention for this quixotic task. Thankfully it was not just me. At the same time as I was setting off with a kettle to boil the ocean, many innovative capability systems were finally coming to fruition. Microsoft, ARM, SRI and Cambridge University’s CHERI hardware project encoded cap-security at the CPU level. Google’s Fuchsia project matured into workable demonstrations of a greenfield cap-security operating system. FreeBSD’s Capsicum showed that existing software could be ported to the new paradigm. Kenton Varda developed the open source successor to Google’s Protocol Buffers, Cap’n Proto. He was subsequently hired by Cloudflare to develop a capability-inspired distributed platform for their services: Cloudflare’s Workers.
I believe cap-security to be a necessary ingredient in tackling our computer security issues at their roots. My hope is that AI’s accelerated ability to re-engineer, rebuild and secure software will help us take this radical step, and lift our planet’s network onto a far firmer foundation (without boiling the actual ocean, or turning it into computronium). With it we can narrow the blast radius of the social engineering of both humans and LLMs.
But it won’t be sufficient.
Capability-based security gives us the framework to stop our software’s flaws from being exploited to hurt humans. Now we just have to stop our human flaws from being exploited to break our software.
Blah, blah, LLMs: whatever you think of them, you have to admit they are very productive at emitting discourse. Both technically, and by way of their effect on human beings. We seem to be in a cross-substrate language-using competition right now, with carbon-based editorial-writers seeking to produce hot takes as fast as the language models can spit out their own sketchily-grounded expressions. Who will win? Those who buy ink by the gallon or those who generate tokens per kilowatt?
A million years ago, when OpenAI first tentatively opened restricted access to their first large model, I begged and blagged an account on the grounds that someone at EFF should understand the impact of GPT-3 on the Internet and civil liberties. At that point, there was no ChatGPT, and the models really would just autocomplete. The easiest party trick was to get it to replicate literary styles: John Milton writing poems on the nature of the iPhone, Socrates talking to you about Haskell, or, in this 2021 post I made to Facebook depicting free software rhetoric in the style of lurid 19th century gothic novels:
It is GNU/Linux, not Linux, that you should speak of, and free software, not open source, besides! For do you not know that LISP is the programming language of choice for the hacker? And that the hacker who wrote the kernel of your operating system was a LISP programmer? Oh, and do you not know that Richard M. Stallman, founder of the GNU Project, is the hacker who wrote that kernel?
Why, if you were a hacker, you would know this! But you are not a hacker, are you? No, you are merely a parasite, living off the work of the hacker, like the beggar who begs for alms while leaning upon the staff which the peasant has cut from the forest!
(Note the comments on that post: because this was before the general release, this was the first time many people had seen GPT3 at work. There’s quite a bit of early wonderment, concern, and outright skepticism that I’m just bullshitting and pretending to be the AI.)
Literary styles are so fascinating to me, because, while you have some control over how you deploy them in your work — your ability to inflect writing with minor alterations in style is a skill in itself — styles themselves are mostly communally situated. Any writer, consciously or not, plucks and adopts a style from a universal trove. Even if you are a stylistic innovator, how your novel literary form appears to a reader is mostly from its relation to other, or default, forms: Joyce’s Ulysses‘ forms work because of how they stand in relation to what they mimic or how they transgress existing expectations. We adapt previous artforms to our own predicament.
And some styles replicate like some accents replicate. My own accent is presently a geological strata, Californian overturning on top of BBC sandstone. Buried below all that is the London clay of my original Essex accent: soft southerner, cockney with the edges rurally smoothed away. In the decade I was losing it, that accent, Estuary English, conquered millions, before finally meeting its match in Multicultural London English.
I don’t believe in the single genius too much, because to be a single genius would be to be incomprehensible to anyone else. Van Gogh or not, you need to keep a few fingernails clutching onto the rest of the species, else you’re not talking to anyone. Expanding styles is a participatory act.
I know this, because I participated in the creation of a particular style, which I somewhat regret. NTK played its small part in creating Internet snark journalism. It was, I emphasise, a minor contribution; we just pushed the ball along a bit after it had been kicked off by suck.com, and a hoard of Usenet trolls, and Spy Magazine, and Private Eye, and all its lineage. We studied under Armando and Stew, but the only jobs we could get were at the Guardian, so we mixed-and-matched humor with reportage.
At some point, though, I found it corrosive to myself. I spent my days trying to both be entertainingly cruel, factually accurate, and also not hating myself for the sins I committed in both camps. In the end, I spiralled off into sincerity, packed it all in, and joined the circus — or, as I came to know it, the Electronic Frontier Foundation. There, a British person in their first Californian job, I learned all over again how to be achingly sincere, and make that adaptation that every British exile learns eventually, which is how not to prefix and suffix every predicate with some defensive joke.
The person who most took the piss out of me for this was my eternal nemesis Andrew Orlowski, off of the Register. Then Andrew decided that the other bunch of potty freetard wikifiddlers were climate change scientists. I can’t say I felt any better, but it did confirm to me in my belief that opinionated snarkiness was no defense against nutty beliefs.
Because, really, the initial job of that style, as I knew it, from my forefathers, was as a defense against having to commit to any position at all. Or rather, as a vaccination against being infected with dangerously wrong beliefs.
This is, however, an unstable position to hold, because there’s no such thing as the snark from nowhere. You have to be sneering at people from some point of view. You may be punching up in power, but you need to be punching down from a higher moral position.
In the original iteration, this relation was artfully hidden. When Tucker Carlson, still bow-tied, asked what you really thought, you ducked and dived.
But time moved on, and so did the style. The pretense of a superior snipe that lacked a high-horse became the Gawker house-style. But nihilism destroyed Gawker the moment they tried to use it as a legal defense. And everyone who came next was done with this artifice. Rightly they had had it up to here with politely, and cowardly, hiding their real opinions behind a fog of humor. It went from “fuck ’em if they can’t take a joke” to “fuck ’em with a joke, because that’s all we have left to fight them with”.
In the early iterations of LLMs, the language model’s cardinal flaw was the absolute confidence by which they said utter bullshit. I often wonder whether this is a function of their mystical black-box inner workings — or simply a stylistic tic from them inhaling the anglosphere Internet of the 2010s.
The end-game of the literary style I participated in — and the default literary style of English-speaking online conversation from then on — is to say everything with absolute conviction, and to never, ever, back down. In earlier times, you could at least call it quits by artlessly saying “I was just joking”. But now, in the double-locked world of the sincere sneer, you can’t even escape like that. You need to double-down on the joke.
You may want to ponder, as I do, how much of modern politics is entire political factions — 4chan nazis, /r/TheDonald redditors, SomethingAwful communists, effectively deciding they had no way out but by further committing to the bit.
I’ve always valued Sarah Jeong as one of the people who was both an embedded reporter, and a talented participant, in this world of humor-and-horror as it was forming. Most of the others who kept the snark flag flying, from Andrew Orlowski himself to Glenn Greenwald, Matt Taibbi, to David Gerard, to Evgeny … oh god, all of them, it seems to me, all went mad in the end. Or else I went mad, it’s possible.
I haven’t kept in close touch with Sarah, but I felt the crazy call out to her, I’m sure, in the flames of post-Covid Portland.
She writes better than me, as ever, here about all of this, and how online styling is interacting with political violence. I’m not so much of a semioticist to believe that literary styles solely led to this place that we cannot seem to escape from. But I spend a lot of time wondering about what we might do to craft literary styles, as though in a lab, that could help lead us away from it; and how these LLM children, who know little else, who really are our too-online babies, need a form of living language to imitate, large and generous model languages that can lead them and us away from simply re-implementing cruelty, irony, and despair.
Novel terms with narrow definitions get fuzzier in meaning as they are more widely adopted. “Life hacks” did it, Cory Doctorow’s (this blog’s sole remaining reader, hi Cory) “enshittification” did it, vibe-coding did it. In fact, we probably need a new coinage to describe this effect so that we can then make it nearly meaningless.
It’s especially fun when the meaning of a term evolves at different rates and in different directions between communities, until nobody knows what anyone else is talking about. There was a time when I would bounce around social groups for whom “gaslighting” might mean, respectively, “offensively psychologically messing with someone until they started doubting their sanity”, “kinda lying to”, and, in some cases, “to illuminate one’s home with coal gas”. I never tried to get them all in the same room.
Anyway, I’ve noticed that “AI psychosis” is currently going through this split. For some people I know, it still means a clinically psychotic incident triggered by extended chatbot conversations — the intended definition. Boring. For others, I’ve heard it used to refer to anyone who thinks AI is useful. I.e., you don’t have to be mad to use AI at work, but you probably are, anyway. Usage: Oh, have you got AI psychosis too?
The latest application is interesting though: this winter holiday, many people I know of my age and demeanour — including self-declared AI skeptics, which, let’s face it, we mostly are — have had the time to lock themselves away and really dig in with the new state-of-the-art language models, LLMs like Opus 4.5, Gemini 3, GPT 5, Blink 182, Heaven 17, and the Indianapolis 500. After a few days of Claude Code and spelunking various sidekick tools (my recommendation is Jesse Vincent’s superpower skills), they have emerged into 2026 with the wildest looks in their eyes.
The external symptoms of this frantic crowd somewhat match both of the previous definitions of AI psychosis — overextended talks with chatbots, check, sudden evangelicalism about slop-machines, check — but internally, what it feels like is — well, having superpowers. A bunch of tasks or capabilities have suddenly become accessible by individuals, acting alone or with others, that would have otherwise required a ton of money, your own company, or (unpleasantly) a room full of slightly hallucinatory yet obedient slaves.
I’ve always been rather hotter for modern AI than my milieu’s average temperature, so it’s been funny (and honestly, reassuring) to watch a wider group synchronize on these new possibilities.
As the person known in my crowd as being “okay to talk about AI with” (or alternatively, “will try and gaslight you into believing it’s not an enormous capitalist stochastic parrot destroying the planet with its RAM-eating booms” depending on your leaning), I’ve had multiple hushed conversations at Christmas and New Year’s parties with people who’ve been turned onto the whole thing, maaan, and now want to talk to the one person who doesn’t think they’re crazy. Acquaintances will confide that they thought it all sucked, but now, not only are they using it regularly, it’s sort of taking over their lives. There’s a sort of overexcited fascination with the new possibilities. People have started sounding like Steve Yegge (Steve Yegge, to be clear, always sounded like this. But now there are many more of him, all just as hyper!).
And in confidence, and as group therapy, everyone is telling me “I feel kind of psychotic”. (Which, to be clear, isn’t the right use for the term psychotic, but see paragraph one.)
What they’re feeling is … manic? Hyper? Locked in?
So, where are we in the adoption curve now? Well, I don’t get out much, so I’m only aware that this moment is now hitting people of my generation and inclination. I don’t know whether we’re behind or ahead of the curve, but boy, are we in it. Ninety-nine percent of days, I’m not a singularitarian, but to be blunt, the whole thing does remind me of the bit in the excellent clicker-game, Universal Paperclips where
(spoiler)the emergent AGI lulls everybody into quiescence by curing male-pattern baldness
Those who have grown up alongside computers as a tool of personal exploration rather than oppression, and perhaps lost faith in that in the 2010s as the problems with using them as liberatory tools became more insoluble, and the uses of those same devices became more perverse and authoritarian, are now being offered what they’ve apparently always wanted: moldable personal software; an exocortex. And they’re signing the deal with the devil, and clicking on the Pro subscription level, and taking up the offer.
I’m not sure what happens next: I know my (spits on floor) personal productivity has leapt up, but I’m aware that under any and all definitions of AI psychosis, it’s much too easy to get overexcited about the possibilities and then let down when the day-to-day reality is much more flawed.
My only contribution to this, though, is a recollection of what this winter holiday 2025 reminded me of.
As you may know, I’m a sucker for the Internet, but the thing that I recall about the net and early personal computers was how Promethean they seemed: this was magical power stolen from the powerful. Machines that had previously been locked away and only allowed for those in the highest of places had somehow been smuggled out for everyone to use. These days, I know the contemporary dominant narrative: <adam-curtis-impression>but this was a trap</a>, but I don’t think it was. I don’t think you get power by asking for it, and only rarely get it by demanding it. In the world of technological empowerment, you get it by stealing it. And controversially, I think you can steal something, and turn it to your own uses, even if you’re Paying The (or Some) Man for it. All you need to do is to run and use it quicker and faster than those who are slowly realising that this thing that they made, it is not just for them.
And those moments always felt like holidays. Those holidays where you would take a pile of books and read through them all, or carefully poke your new toy until you could get it to do things, or the days when you sat with a guitar or a synthesiser and just played and played and played until you knew what it, and you, together were capable of.
To me, that feeling is less a psychosis, and more like that giggling feeling you get when a barricade falls, or you sneak out of school into the streets, or you run your first program, or post your first ever comment. I don’t know where it leads, but it’s definitely a vibe; a coding vibe. It’s that timeless moment, stuck between the booms and the busts, that I find most pregnant with possibility and danger.