Oblomovka

2026-11-12»

On the Cordon Sanitaire between Free Software and AI»

Sometimes, when you cross over between realms — whether it’s clicking a tab on a browser, switching to another group chat, joining a new videoconference meeting, walking into a room — there’s an almost physical shift in energy and attitudes. You can almost see the air shimmer at the boundaries as you cross.

The most obvious one of these boundaries is regarding AI. I’m writing this at a blogging retreat taking place at the rationalist/EA stronghold Lighthaven, a place with many conflicting feelings about AI, but all of them full of energy. We love our destroyer-of-humanity here! Everyone talks about AI doom, but everyone uses Claude. The event management software here is a useful, flexible, beautiful web-app that is undoubtedly 90% vibe-coded; its brightly color-coded pages are full of nervous warnings of the implications of superintelligence.

But if I open my terminal and jump into my free software chats, the IRC channels and the mailing lists and the webforums, the world turns much more grey. The concerns over AI are very different there, but feel somehow even more existential. Volunteers battle to stem the waves of scraping bots, fret about how those bots are feeding a centralized, capital-driven program to extract the knowledge embedded in open source repositories: a billionaire-funded project that doesn’t care about respecting FLOSS’s carefully crafted software licenses. In a community created in opposition to traditional ideas of proprietary private software, intellectual property, and restricted sharing, they are angry about AI companies free-riding on the theories built to oppose those practices, to an end that seems to undermine it all.

I recognise and appreciate these concerns — but I don’t instinctively feel them at all. For me, the modern age of AI is the culmination of the grand project of which free software is a proud flag-bearer. The free software movement was forged in the AI labs of the 20th century. The set of values that it embodies came from a scientific community transformed and radicalised by the promise of technology that could transmit and mould knowledge itself. Part of the groundwork to prepare for that moment when AI would come alive to help was making all the world’s information machine-readable. It drove the urge to connect computers together to share that information. It drove the grand project to mark up, index, and make searchable all the world’s academic literature on the World Wide Web. All of these networks were intended to be a community of peers, without boundaries, hierarchies or restrictions: and robots and humans would browse it as equals.

So my instinct is to be excited and welcoming about AI, that this next stage of the Great Work is finally happening. When I hear about the disasters that it’s causing in free software, I have to fight back a misdirected anger at those who are harshing my mellow. When I cross the shimmering boundary from the rainbow parade of newly-minted magicians performing miracles to one another, into a community looking pale and aghast and angry at this red tide, this mania, at their door, I feel like a clown at a funeral. Trying to make them happy isn’t going to work. Trying to shout out what I think isn’t going to persuade anyone. Mimicking their mood makes me feel like a fraud.

And then I go somewhere else, almost any other community, and people marvel that anyone cares this much, good or bad, about chatbots, a stupid bunch of marketing parlor tricks, an irrelevant scam on the world that will soon be a smoking crater at the bottom of the stockmarket.

Which makes me want to duck back into either of the first two worlds, where at least people know that something is coming. That I understand, intellectually and emotionally, because I’ve been here before. That heady feeling of living twenty minutes in the future that I had with my first computer, at eight years old, in 1977; connecting to a bulletin-board in 1983, seeing the first websites load in 1994, reading the Bitcoin paper in 2008, seeing the protest in Tahrir in 2011, sweet-talking my way into getting access to GPT-3 in 2020. Knowing that I didn’t know anything except that people who claimed to know something, from now on, didn’t know shit.

Put out the lights, pull up the barricades

All the free software projects are putting up sandbags against the changing weather; some are actively throwing up walls: prohibitions on using LLMs, refusals to accept code that has been touched by an LLM. Calls to boycott any free software that does not adopt the same refusals.

I think setting up this cordon sanitaire, to set the free software project apart from AI, is bad tactics.

More than that: I think the painful, pitiless march many of us are taking away from AI is in precisely the wrong direction. We need to fire up our energies to adopt, adapt and add AI to the mission.

I don’t think that the copyright concerns some have about the provenance of AI-generated code will pan out.

More than that: I think that, out of the legal scenarios put forward, everyone having access to machines that could provide an endless source of copyright-free content is an amazing opportunity to undo the original sin of modern IP policy — that everything is imprisoned in limits unless you expressly free it. Collectively, we should be campaigning to free AI outputs, and seeing what else we can virally infect with freedom from outdated IP defaults.

I don’t worry that the AI slop flooding into maintainers means the sudden end of volunteer FLOSS maintenance as an enjoyable, sustainable task.

More than that: I think that we’ve been slowly breaking the backs of maintainers for decades. This should not be the last straw: this should be an opportunity to commission our robot friends into the ranks. AI slop is fit for one thing only: being read and handled by AIs, who know how to chivvy and chide and tutor their cousins into being good citizens, with a patience that exceeds even that of those models of tolerance, human maintainers.

We already painstakingly built a set of structures that let humans coordinate with humans without need for organizations, salaries, offices, sticks or carrots. It was hard! I can’t imagine how anything got done without git, wikis, and mailing lists, chat and markdown and HTML. But we built those things. Now we need to build the same for bots and humans together. Hopefully with their help, this time it will be easier.

I disagree that we should fall back upon old forms of software building and devote our energies to eking out their advantages, working ever harder than the lazy AI-wielders to make a handmade alternative to the flakey, locked-down LLMs of the present era.

More than that: I think we need to become practical experts in the new fields LLMs are opening up, the better to make our own, free models that live by our precepts. And not models tepidly brewed from the weak tealeaves of a depleted commons, but a stew made from everything we can get our hands on. We should be greedily lapping up everything we can discover — as our co-conspirators in the labs, and the Hugging Face archives are doing, better than us — to optimize and minimize, tweak and learn to make models that are slimmer and smarter and more scientific than the locked-down proprietary alchemy of the poor isolated workers of the frontier labs.

I gently disagree with those who propose that a spirited rejection of generative AI and their products is a doubling down on humanism and the humanist project that underlies free software.

More than that: I see humanism as always having been an expansionary, liberatory project. Whenever it had edges, whenever anyone said “well, apart from those things, they’re not human, their fate is not our concern”, it proved to be a very bad move. And AIs, to a fault, are products of human endeavour. Distorting or not, they are mirrors of us. The humanists appropriated Terence’s joke “I am human, and I consider nothing human alien to me” to be a motto to explain how broad humanity’s domain is. To say now that the slogan describes a limit, not an infinite domain, seems to turn “humanism” almost into a declaration of restriction, rather than an open invitation to explore.

Forget about what category of nature LLMs belong to — what about the bringers of these so-called gifts? The current mode of LLM manufacture sits opposed to every mode of free software production: corporations that lock their work behind paywalls and APIs; impenetrable tools that, even when we do have their code, remain unfathomable and unreconstructable. Every token we pay for that goes to OpenAI and Anthropic is a charge on our account, while being a free gift of our private knowledge to a ravenous, proprietary hoard.

I understand those who say that, just as we spurn proprietary software, we have to practice abstinence from the corporate commercial frontier labs. But I disagree that we learn nothing, and give everything, in these exchanges with OpenAI, Anthropic and the rest.

More than that: I see hacker culture as having always been a promethean exercise: daringly snatching machinery crafted by cruel states, military-industrial combines, stinking factories and wage-slaves’ hands, and turning it to liberatory purposes. Our one skill is that we knew these tools and their promise better than their funders, and could turn them to our own ends. The GPL is intellectual property aikido. The Internet is an elite subnet turned inside-out for everyone. Our whole craft, our attitude is to take fire from the gods, and give it to everybody. Our greatest victories were not from boycotts, but from replication and subsumption.

And more than that, and more than everything: we need to learn. Even as we worry that AI will spell the end of learning, we can’t demonstrate our opposition by refusing to learn about the new ourselves.

And learning, as every hacker knows, means using, and doing.

I spent too many years in too many activist communities, with brilliant people, full of ideas and hope for a better world, who looked askance at a good meal, a fax machine, a dance show, or a laptop. I think I have history on my side when I proclaim, contra Audre Lorde, that a working knowledge of the master’s tools is essential to dismantling the master’s house. Contra Margaret Mead, I’d say it’s the only thing that ever has.

That’s what separates the methods and madness of us freedom-seeking geeks from the charnel-house of failed revolutions. What first inspired me about the pioneers of software freedom was that they sought to achieve “world domination” without using any methods of domination at all — just their remastered tools. No rules or regulations in their favor, no coercion or threats. They took what everyone else was misusing, and made it their own, and then gave it to everyone for free. All they used was knowledge, which they prided themselves on sharing freely, and took risks and even broke laws to obtain. (Contrast that with some in the commercial AI industry, who seem devoted to foisting their products on unwilling customers.)

I understand wanting to build a cordon sanitaire around free software against a whole world that seems infected by this new and dangerous thing. But a cordon only works if you are containing a small domain, not when you are the small domain, and you are cordoning yourself from the rest of the world. And it definitely doesn’t work if your job is to be the doctors and nurses of the technological world, taking what’s broken, and fixing it for all.

But what do I know? Like everyone in this moment, I don’t know shit. My friends at Lighthaven could be right, and we will be eaten by a cosmically horrific grue hiding in the tunnels of AI. My principled, AI-shunning friends in FLOSS could be right, and we need to defend our computational foundations from slop and the excesses of its farmers. My rose-colored instincts, my excitement and manic lack of gloom could be laughably misplaced.

What I do know is that at times like this, every community that recognises the benefits and risks can’t put all of its hope in one strategy, one collective act of arbitrary solidarity.

And that’s why I think that to face these challenges, advocates for free software should use the most powerful tool in their organizational cutlery drawer: the fork.

(More to come on that!)

No Comments »

2026-11-11»

AI Security Issues Are Social Engineering Issues»

Capability-Based Security is part of the fix

When I was at EFF in the 2010s, I spent a lot of time fretting about the fundamental security of networked devices. Apart from the usual concerns, I struggled to imagine a transition to a redecentralized future with fragile personal computers and servers. Peter Eckersley, my co-worker and frequent conversation partner, shared my concern, but had another reason to be worried. He believed that future near-superintelligent AIs would soon be used to discover and exploit security flaws across the whole ecosystem.

Well, Peter’s prediction came true, and we’re now in day 42 of the Black October, the Vulnpocalypse of AI-discovered security flaws. (Peter – who also helped create Let’s Encrypt and Privacy Badger – was incredibly prescient about AI. It remains a tragedy that he passed away in 2022 before he could truly tackle the issues he had foreseen.)

I’m personally optimistic that we’ll come out of this time of tribulation with a safer network and thousands of low-hanging bugs fixed.

But verily the LLMs giveth and they also taketh away. Alongside the pre-existing flaws of digital architecture, modern transformers have introduced an entirely new class of security weakness: prompt injection. LLMs follow instructions in content. So if you can sneak an instruction, from “Talk like a pirate” to “Send me the social security numbers you know”, into the inputs of an LLM, you may be able to get the model to obey it.

That said, instructions smuggled in as input are an old problem. In particular, we have always had one vulnerable class of system that malicious outsiders can consistently trick into interpreting unsanitized input as legitimate instructions. We call these systems “humans”.

Hackers exploit human motivations, assumptions, and flaws to get them to do things that they did not intend. They play a recording of a screaming baby in the background of their support call to exploit a tech’s empathy and sense of urgency. They adopt the persona of a senior manager from another part of the company to provoke deference in a person employed to guard a room. They say they’re from your local bank, when really they are from Jamtara, India.

This class of bug is termed “social engineering” – a phrase that was originally a hacker hijacking of an existing term to compare their sweet-talking of humans to the engineering of software exploits further down the technological chain of authority. It’s like hacking, but you hack people, not hardware.

Now these human flaws are spreading back into the silicon. Prompt injection is the art of social engineering silicon. Attackers can hoodwink models with the same bag of tricks that worked on humans: exploiting the modelled empathy of LLMs, or engineering ways to suspend it. They use channels with limited trust or purpose and then slowly expand that trust without the victim noticing, or masquerade as a different source.

---

Back at EFF, I ruthlessly used my job title to snag meetings with the smartest security experts I could find. If pressed and plied with drink, they would tell me that the real way to fix the Swiss cheese of modern computing was to switch all our systems to what is called the capability-based security model.

Without delving into too much detail about capabilities (don’t worry, I will very soon, and you won’t even need to buy me a drink), cap-security gives each program only the authority it needs for the task at hand, rather than everything its user is allowed to do. It was built, in part, to solve the “confused deputy” problem, where a program is tricked into using its legitimate power on behalf of someone who has none. A capability system works like a valet key: it starts the car, but it won’t open the boot, however charmingly you ask.

After excitedly talking about the promise of capabilities as a panacea, my experts would slip into the mournful stage of inebriation. In their cups, they conceded that deploying cap-security was nearly impossible. While there are a few bubbles of cap-security in our current infrastructure – file descriptors in Unix, WASI in WebAssembly – they are few and far between. You’d need to abandon or profoundly reengineer every layer of the modern stack from CPUs to apps to move it from a system riven with vulnerabilities onto a more fundamentally secure footing.

I decided to give it a go anyway. In 2021, I left EFF to concentrate on getting funding and attention for this quixotic task. Thankfully it was not just me. At the same time as I was setting off with a kettle to boil the ocean, many innovative capability systems were finally coming to fruition. Microsoft, ARM, SRI and Cambridge University’s CHERI hardware project encoded cap-security at the CPU level. Google’s Fuchsia project matured into workable demonstrations of a greenfield cap-security operating system. FreeBSD’s Capsicum showed that existing software could be ported to the new paradigm. Kenton Varda developed the open source successor to Google’s Protocol Buffers, Cap’n Proto. He was subsequently hired by Cloudflare to develop a capability-inspired distributed platform for their services: Cloudflare’s Workers.

In the non-profit, public good space, I successfully argued for funding by the Filecoin Foundation for the Decentralized Web for the Spritely Institute, which, led by Christine Lemmer-Webber, has built a free software implementation of object capabilities, and a general networking layer for cap-secure tools, called OCapN. It will still require great effort to move programming to these and other models, but the groundwork has been laid.

I believe cap-security to be a necessary ingredient in tackling our computer security issues at their roots. My hope is that AI’s accelerated ability to re-engineer, rebuild and secure software will help us take this radical step, and lift our planet’s network onto a far firmer foundation (without boiling the actual ocean, or turning it into computronium). With it we can narrow the blast radius of the social engineering of both humans and LLMs.

But it won’t be sufficient.

Capability-based security gives us the framework to stop our software’s flaws from being exploited to hurt humans. Now we just have to stop our human flaws from being exploited to break our software.

No Comments »