skip to main bit
a man slumped on his desk, from 'The Sleep of Reason Produces
      Monsters'

Oblomovka

Currently:

Archive for the ‘Committee to Protect Journalists’ Category

2010-09-14

Haystack vs How The Internet Works

There’s been a lot of alarming but rather brief statements in the past few days about Haystack, the anti-censorship software connected with the Iranian Green Movement.  Austin Heap, the co-creator of Haystack and co-founder of parent non-profit, the Censorship Research Center, stated that the CRC had “halted ongoing testing of Haystack in Iran”; EFF made a short announcement urging people to stop using the client software;  the Washington Post wrote about unnamed “engineers” who said that “lax security in the Haystack program could hurt users in Iran”.

A few smart people asked the obvious, unanswered question: What exactly happened? Between all those stern statements, there is little public information about why the public view of Haystack switched from it being a “step forward for activists working in repressive environments” that provides “completely uncensored access to the internet from Iran while simultaneously protecting the user’s identity” to being something that no-one should ever consider using.

Obviously, some security flaw in Haystack had become apparent. But why was the flaw not more widely documented? And why now?

As someone who knows a bit of the back story, I’ll  give as much information as I can. Firstly, let me say I am frustrated that I cannot provide all the details. After all, I believe the problem with Haystack all along has been due to explanations denied: either because its creators avoided them, or because those who publicized Haystack failed to demand them. I hope I can convey why we still have one more incomplete explanation to attach to Haystack’s name.

(Those who’d like to read the broader context for what follows should look to the discussions on the Liberation Technology mailing list. It’s an open and public mailing list, but it with moderated subscriptions and with the archives locked for subscribers only. I’m hoping to get permission to publish the core of the Haystack discussion more publicly.)

First, the question that I get asked most often: why make such a fuss, when the word on the street is that a year on from its original announcement, the Haystack service was almost completely nonexistent, a beta product restricted to only a few test users, all of whom were in continuous contact with its creators?

One of the many new facts about Haystack that the large team of external investigators, led by Jacob Appelbaum and Evgeny Morozov, have learned in the past few days is that there were more users of Haystack software than Haystack’s creators knew. Despite the lack of a “public” executable for examination, versions of the Haystack binary were being passed around, just like “unofficial” copies of Windows (or videos of Iranian political violence) get passed around. Copying: it’s how the Internet works.

But the understood structure of Haystack included a centralized, server-based model for providing the final leg of censorship circumvention. We were assured that Haystack had a high granularity of control over usage. Surely those servers blocked rogue copies, and ensured that bootleg Haystacks were excluded from the service?

Apparently not. Last Friday, Jacob Appelbaum approached me with some preliminary concerns about the security of the Haystack system. I brokered a conversation between him, Austin Heap, Haystack developer Dan Colascione and the CEO of CRC CRC’s Director of Development, Babak Siavoshy. Concerned by what Jacob had deduced about the system, Austin announced that he was shutting down Haystack’s central servers, and would keep Haystack down until the problems were resolved.

Shortly after, Jacob obtained a Haystack client binary. On Sunday evening, Jacob was able to conclusively demonstrate to me that he could still use Haystack using this client via Austin’s servers.

When I confronted Austin with proof of this act, on the phone, he denied it was possible. He repeated his statement that Haystack was shut down. He also said that Jacob’s client had been “permanently disabled”. This was all said as I watched Jacob  using Haystack, with his supposedly “disabled” client, using the same Haystack servers Austin claimed were no longer operational.

It appeared that Haystack’s administrator did not or could not effectively track his users and that the methods he believed would lock them out were ineffective. More brutally, it also demonstrated that the CRC did not seem able to adequately monitor nor administrate their half of the live Haystack service.

Rogue clients; no apparent control. This is why I and others decided to make a big noise on Monday: it was not a matter of letting just CRC’s official Haystack testers quietly know of problems; we feared there was a potentially wider and vulnerable pool of users who were background users of Haystack that none of us, including CRC, knew how to directly reach.

Which brings us to the next question: why reach out and tell people to stop using Haystack?

As you might imagine from the above description of  Haystack’s system management, on close and independent examination the Haystack system as a whole, including these untracked binaries, turn out to have very little protection from a high number of potential attacks — including attacks that do not need Haystack server availability. I can’t tell you the details; you’ll have to take it on my word that everyone who learns about them is shocked by their extent.  When I spelled them out to Haystack’s core developer, Dan Colascione late on Sunday, he was shocked too (he resigned from Haystack’s parent non-profit the Censorship Research Center last night, which I believe effectively kills Haystack as a going concern. CRC’s advisory board have also resigned.)

Deciding whether publishing further details of these flaws put Haystack users in danger is not just a technical question. Does the Iranian government have sufficient motivation to hurt Haystack users, even if they’re just curious kids who passed a strange and exotic binary around? There’s no evidence the Iranian government has gone after the users of other censorship circumvention systems. The original branding of Haystack as  “Green Movement” software may increase the apparent value of constructing an attack against Haystack, but Haystack client owners do not have any connection with the sort of high-value targets a government might take an interest in. The average Haystack client owner is probably some bright mischievous kid who snagged it to access Facebook.

Lessons? Well, as many have noted, reporters do need to ask more questions about too-good-to-be-true technology stories.  Coders and architects need to realize (as most do) that you simply can’t build a safe, secure, reliable system without consulting with other people in the field, especially when your real adversary is a powerful and resourceful state-sized actor, and this is your first major project. The Haystack designers lived in deliberate isolation from a large community that repeatedly reached out to try and help them. That too is a very bad idea. Open and closed systems alike need independent security audits.

These are old lessons, repeatedly taught.

New lessons? Well, I’ve  learned that even apparent vaporware can have damaging consequences (I originally got re-involved in investigating Haystack because I was worried the lack of a real Haystack behind the hype might encourage Iranian-government fake Haystack malware — as though such things were even needed!).

Should one be a good cop or a bad cop? I remember sitting in a dark bar in San Francisco back in July of 2009, trying to persuade a blasé Heap to submit Haystack for an independent security audit. I spoke honestly to anyone who contacted me at EFF or CPJ about my concerns, and would prod other human rights activists to share what we knew about Haystack whenever I met them (most of us were skeptical of his operation, but without sufficient evidence to make a public case). I encouraged journalists to investigate the back story to Haystack. I kept a channel open to Austin throughout all of this, which I used to occasionally nudge him toward obtaining an audit of his system, and, finally, get a demonstration that answered some of our questions (and raised many more). Perhaps I should have acted more directly and publicly and sooner?

And I think about Austin Heaps’ own end quote from his Newsweek article in August, surely the height of his fame.”A mischievous kid will show you how the Internet works”, he warns. The Internet is mischievous kids; you try and work around them at your peril. And theirs.

2010-03-19

what i did next

For a moment, climbing out of the too-fresh sunshine and with the taste of a farewell Guinness still on my tongue, slumping into the creaky old couch in the slightly grimy, Noisebridge to write something from scratch, San Francisco felt like Edinburgh in August, a day before the Festival.

Edinburgh for me was always the randomizer, the place I hitched to every year, camped out in, and came out in some other country, six weeks later, with hungover and overdrawn, with a new skill or passion or someone sadder or more famous or just more fuddled and dumber than ever.

Today was my last day at EFF. Just before our (their? Our.) 20th birthday party in February, where I had the profoundly fannish pleasure to write and barely rehearse a 30 minute sketch starring Adam Savage, Steve Jackson, John Gilmore, me in my underpants, and Barney the Dinosaur, I callously told them I was leaving them all for another non-profit. We commiserated on Thursday, in our dorky way, by playing Settlers of Catan and Set and Hungry Hippos together. They bought me money to buy a new hat. I logged off the intranet, had a drink, and wandered off into a vacation.

In April, after a couple of weeks of … well, catching up on my TV-watching, realistically … I’ll be kickstarting a new position at the Committee to Protect Journalists as Internet Advocacy Coordinator.

I’ve known the CPJ people for a few years now, talking airily to them about the networked world as they grimly recorded the rising numbers of arrested, imprisoned, tortured, threatened and murdered Internet journalists in the world. Bloggers, online editors, uploading videographers. Jail, dead, chased into exile. As newsgathering has gone digital, it’s led to a boom in unmediated expression. But those changes have also disintermediated away the few institutional protections free speech’s front line ever had.

CPJ has incredible resources for dealing with attacks on the free press on every continent: their team assists individuals, lobbies governments at the highest levels, documents and publicizes, names and shames. They were quick to recognize and reconfigure for a digital environment (you have to admire an NGO that knew enough to snag a three letter domain in ’95). Creating a position for tackling the tech, policy and immediate needs of online journalism was the next obvious step.

The question I had for them in my interview was the same that almost everybody I’ve spoken to about this job has asked me so far. On the Internet, how do you (they? We.) define who a journalist is?

The answer made immediate sense. While “journalism” or “newsgathering” or “reportage” as an abstract idea might seem problematic when cut from its familiar institutions, and pasted into the Internet… nonetheless, you know it when you see it. When someone is arrested or threatened or tortured for what they’ve written, if you can pull up what they said in a mailreader or a browser, it really doesn’t take long to identify whether it’s journalism or not.

What’s harder is untangling the slippery facts of the case — whether the journalist was targeted because of their work, or other reasons; whether it was the government or a criminal enterprise that did the deed; where the leverage points are to seek justice or freedom.

In those fuzzier areas, in the same way as EFF uses its legal staff to map the unclear world of the frontier into clear legal lines, CPJ uses its staff’s investigative journalist expertise to uncover what really happened, and then uses the clout of that reinforced and unassailable truth to lobby and expose.

Honestly, I’m still only beginning to map out how I might help in all this. I spent a week last month in New York where CPJ is based, listening to their regional experts talk about every continent, all the dictators, torturers, censors and thugs, all the bloggers and web publishers and whistleblowers.

I know I am starting on that ignorance rollercoaster you get when striking out into new territory. I can tell these people about proxies, AES encryption and SMS security, but I still can’t pronounce Novaya Gazeta, or remember what countries border Kenya. You surprise yourself with how much old knowledge becomes freshly useful, at the same time as you feel stupid for every dumbly obvious fact you fail to grasp.

I think part of my usefulness will come from writing more, and engaging more with the communities here I know well to explain and explore the opportunities and threats their incredible creations are creating today. At the same tie, I’m already resigned to taking a hit in my reputational IQ as I publicly demonstrate my ignorance (my friends in Africa and Russia are already facepalming, I can tell). Hope you’ll forgive me.

In the mean time, I’ll be setting up my monthly donation to EFF. I’ve said it before and I’ll bore you again, EFF are an incredible organization, made up of some of the smartest and most dedicated people I’ve ever met. I smugly joined in 2005 thinking I understood tech policy, and spent the next few years amazed at what it was like to live as the only person who didn’t have an EFF to help me understand what I was looking at and what to do about it. I guess I finally got the hang of juggling five hundred daily emails, a dozen issues refracted through dozens of cultures across the world. And I guess that’s aways the cue to switch tracks and reset to being dumb and ready to learn again.

Incidentally, EFF is looking for an IP attorney right now. I don’t know how many lawyers read this blog, but if you know a smart IP legal person who wants to randomize their life for the opportunity to become even smarter for a good cause, get them to apply. They won’t regret it, not for a minute.

                                                                                                                                                                                                                                                                                                           

petit disclaimer:
My employer has enough opinions of its own, without having to have mine too.