skip to main bit
a man slumped on his desk, from 'The Sleep of Reason Produces



Archive for September 14th, 2010


Haystack vs How The Internet Works

There’s been a lot of alarming but rather brief statements in the past few days about Haystack, the anti-censorship software connected with the Iranian Green Movement.  Austin Heap, the co-creator of Haystack and co-founder of parent non-profit, the Censorship Research Center, stated that the CRC had “halted ongoing testing of Haystack in Iran”; EFF made a short announcement urging people to stop using the client software;  the Washington Post wrote about unnamed “engineers” who said that “lax security in the Haystack program could hurt users in Iran”.

A few smart people asked the obvious, unanswered question: What exactly happened? Between all those stern statements, there is little public information about why the public view of Haystack switched from it being a “step forward for activists working in repressive environments” that provides “completely uncensored access to the internet from Iran while simultaneously protecting the user’s identity” to being something that no-one should ever consider using.

Obviously, some security flaw in Haystack had become apparent. But why was the flaw not more widely documented? And why now?

As someone who knows a bit of the back story, I’ll  give as much information as I can. Firstly, let me say I am frustrated that I cannot provide all the details. After all, I believe the problem with Haystack all along has been due to explanations denied: either because its creators avoided them, or because those who publicized Haystack failed to demand them. I hope I can convey why we still have one more incomplete explanation to attach to Haystack’s name.

(Those who’d like to read the broader context for what follows should look to the discussions on the Liberation Technology mailing list. It’s an open and public mailing list, but it with moderated subscriptions and with the archives locked for subscribers only. I’m hoping to get permission to publish the core of the Haystack discussion more publicly.)

First, the question that I get asked most often: why make such a fuss, when the word on the street is that a year on from its original announcement, the Haystack service was almost completely nonexistent, a beta product restricted to only a few test users, all of whom were in continuous contact with its creators?

One of the many new facts about Haystack that the large team of external investigators, led by Jacob Appelbaum and Evgeny Morozov, have learned in the past few days is that there were more users of Haystack software than Haystack’s creators knew. Despite the lack of a “public” executable for examination, versions of the Haystack binary were being passed around, just like “unofficial” copies of Windows (or videos of Iranian political violence) get passed around. Copying: it’s how the Internet works.

But the understood structure of Haystack included a centralized, server-based model for providing the final leg of censorship circumvention. We were assured that Haystack had a high granularity of control over usage. Surely those servers blocked rogue copies, and ensured that bootleg Haystacks were excluded from the service?

Apparently not. Last Friday, Jacob Appelbaum approached me with some preliminary concerns about the security of the Haystack system. I brokered a conversation between him, Austin Heap, Haystack developer Dan Colascione and the CEO of CRC CRC’s Director of Development, Babak Siavoshy. Concerned by what Jacob had deduced about the system, Austin announced that he was shutting down Haystack’s central servers, and would keep Haystack down until the problems were resolved.

Shortly after, Jacob obtained a Haystack client binary. On Sunday evening, Jacob was able to conclusively demonstrate to me that he could still use Haystack using this client via Austin’s servers.

When I confronted Austin with proof of this act, on the phone, he denied it was possible. He repeated his statement that Haystack was shut down. He also said that Jacob’s client had been “permanently disabled”. This was all said as I watched Jacob  using Haystack, with his supposedly “disabled” client, using the same Haystack servers Austin claimed were no longer operational.

It appeared that Haystack’s administrator did not or could not effectively track his users and that the methods he believed would lock them out were ineffective. More brutally, it also demonstrated that the CRC did not seem able to adequately monitor nor administrate their half of the live Haystack service.

Rogue clients; no apparent control. This is why I and others decided to make a big noise on Monday: it was not a matter of letting just CRC’s official Haystack testers quietly know of problems; we feared there was a potentially wider and vulnerable pool of users who were background users of Haystack that none of us, including CRC, knew how to directly reach.

Which brings us to the next question: why reach out and tell people to stop using Haystack?

As you might imagine from the above description of  Haystack’s system management, on close and independent examination the Haystack system as a whole, including these untracked binaries, turn out to have very little protection from a high number of potential attacks — including attacks that do not need Haystack server availability. I can’t tell you the details; you’ll have to take it on my word that everyone who learns about them is shocked by their extent.  When I spelled them out to Haystack’s core developer, Dan Colascione late on Sunday, he was shocked too (he resigned from Haystack’s parent non-profit the Censorship Research Center last night, which I believe effectively kills Haystack as a going concern. CRC’s advisory board have also resigned.)

Deciding whether publishing further details of these flaws put Haystack users in danger is not just a technical question. Does the Iranian government have sufficient motivation to hurt Haystack users, even if they’re just curious kids who passed a strange and exotic binary around? There’s no evidence the Iranian government has gone after the users of other censorship circumvention systems. The original branding of Haystack as  “Green Movement” software may increase the apparent value of constructing an attack against Haystack, but Haystack client owners do not have any connection with the sort of high-value targets a government might take an interest in. The average Haystack client owner is probably some bright mischievous kid who snagged it to access Facebook.

Lessons? Well, as many have noted, reporters do need to ask more questions about too-good-to-be-true technology stories.  Coders and architects need to realize (as most do) that you simply can’t build a safe, secure, reliable system without consulting with other people in the field, especially when your real adversary is a powerful and resourceful state-sized actor, and this is your first major project. The Haystack designers lived in deliberate isolation from a large community that repeatedly reached out to try and help them. That too is a very bad idea. Open and closed systems alike need independent security audits.

These are old lessons, repeatedly taught.

New lessons? Well, I’ve  learned that even apparent vaporware can have damaging consequences (I originally got re-involved in investigating Haystack because I was worried the lack of a real Haystack behind the hype might encourage Iranian-government fake Haystack malware — as though such things were even needed!).

Should one be a good cop or a bad cop? I remember sitting in a dark bar in San Francisco back in July of 2009, trying to persuade a blasé Heap to submit Haystack for an independent security audit. I spoke honestly to anyone who contacted me at EFF or CPJ about my concerns, and would prod other human rights activists to share what we knew about Haystack whenever I met them (most of us were skeptical of his operation, but without sufficient evidence to make a public case). I encouraged journalists to investigate the back story to Haystack. I kept a channel open to Austin throughout all of this, which I used to occasionally nudge him toward obtaining an audit of his system, and, finally, get a demonstration that answered some of our questions (and raised many more). Perhaps I should have acted more directly and publicly and sooner?

And I think about Austin Heaps’ own end quote from his Newsweek article in August, surely the height of his fame.”A mischievous kid will show you how the Internet works”, he warns. The Internet is mischievous kids; you try and work around them at your peril. And theirs.


petit disclaimer:
My employer has enough opinions of its own, without having to have mine too.