skip to main bit
a man slumped on his desk, from 'The Sleep of Reason Produces
      Monsters'

Oblomovka

Currently:

Archive for August, 2008

2008-08-21

Odlyzko on the Fallacy of Streaming

I’m a great fan of Andrew Odlyzko who I think you can picture as the network economics equivalent of the early Jakob Nielsen, long before he got drunk with power and started challenging beautiful waif-like designers to chicken-wire cage matchs. By that I mean Odlyzko is rather good at explaining what you at some level instinctively understand, but maddeningly no-one else higher than your pay grade does. Handily he explains these characteristics using actual facts that you can clutch onto in the next futile argument you are forced to re-justify over the self-same self-evident truths. You will still lose, because the people who believe the opposite of you are your employers, but you will go down in full righteousness instead of just bleating “but.. but…”

In this recent short note on streaming, Odlyzko explains, as slowly as he can, why content and telecom business executives (erroneously) think that streaming movies need them to demand special network-neutrality-busting queue-barging privileges on the Internet.

It turns out it’s because they think that real-time streaming means that the bytes have to move in exactly real time — that’s to say, if you’re watching a movie encoded at 6Mb/s, you need a constant, unbreakable, 6Mb/s stream over the Internet. In other words, no-one told them about buffering:

I have been asking listeners [at his networking presentations] to raise their hands if they saw any point at all in faster-than-real- time transmission of multimedia … The highest positive response rate I have observed was at a networking seminar at the Royal Institute of Technology in Stockholm, in September 2007. It was about 30%. Twice, at networking seminars at CMU and Stanford, the rate was about 20%. Usually it is far lower, almost always under 10%. And sometimes it is close to zero. I had two similar audi- ences, on two separate continents, of about 100 people in each case, consisting of (mostly non-technical) mid-level telecom managers as well as government research agency staff and others connected with communications, where among the approximately 200 attendees in all, just one hand went up, and that one very tentatively.

Apparently, advocates of streaming have never wondered what that little grey bar that preloads your YouTube clip before you watch it, so you can cope with drops in the download rate, means. Or, more likely, have never actually watched a YouTube clip in their life, and just sit behind their desks wondering what everyone in the open-plan bit of their office is giggling at.

“The purpose of data networks is to satisfy human impatience.”

2008-08-20

on being a bit of an idiot

I love this reassuring advice for contributors to Mozilla projects undergoing their first code review:

Bugzilla reviewers may seem to be harsh, accidentally. They aren’t trying to be harsh or overly critical, they’re just pointing out what needs to be changed, which usually means they’re pointing out what’s wrong with the current code, instead of pointing out what’s right with it. Usually they don’t have lots of extra time in their life for reviews, so they just quickly write what needs to be fixed, without spending too much time thinking about the nicest way to say it. Sometimes they also don’t go into long explanations.

You’re not a terrible programmer or a bad person. All we’re doing is telling you what’s preventing us from checking the code into Bugzilla’s main codebase.

I remember the first time that I, almost accidentally, asked somebody for an opinion on my code, a few years ago. They automatically mailed back with about thirty different suggestions, all correct, and I was mortified. Why had I been such an idiot? And why were they being so cruel and rude?

Actually, thinking back on it, I think the first time that happened was when I wrote some code for .EXE when I was 21, and had to bear about six months of readers writing in with an endless stream of bugs. “Dear .EXE, I spotted yet another error in your ‘Printing in Columns’ program from June”…

I now, at some level, realise that having somebody else go over your code is both humiliating and the best way to learn. And…

You know, this post was going to be about how I thought that was true for most people, and harp on philosophically about how personal and intimate coding is, and how pair programming is fraught with psychological drama. But, if I am honest, as I write this, I must admit that this is mostly a personal bug I’ve had up my arse (weird anglo-american hybrid term) for many years, rather than a common state of humanity. I’ve dodged a lot of things out of terror of criticism in my life. My stand-up career was aborted really before it got started because of an incident with a Scottish lady screaming into my face while standing on a table near the stage. NTK’s wilful obscurity came from, in part, a desire that people not so much be insulted, than as to be unsure what was being said about them, exactly (more on this in a future post). I generally fiddle with projects for months to protect them from even the slightest possibility of a cruel word.

In parallel with this, I’ve noticed something happen online a lot. Some absolute idiot turns up on the Net, and asks absolutely idiotic questions, and generally bumbles loudly to themselves while everyone notes what a clod they are and shoots them down in flames. Then, they start adapting to the criticisms, or facing them down, or disappear entirely and re-appear later in some other guise.

Through any of these processes, they end up transformed. Soon, they’re handing out advice to other idiots, or explaining what was actually utterly arcane to a far wider audience, or alternatively pursuing their dumb ideas to great success. I first noticed this with a guy on Cix, who called himself Nildram, and who was rude, and argumentative, and obtuse and the biggest pain (among a vast community of grumpy patrician carbuncular pains, I should say). At first I patted myself on my back for being quiet and sensitive and afraid to make the kind of humiliating public mistakes he made. And then I watched him just get better and better. I learned from his mistakes too, but at one remove. He ended up being the guy behind Nildram Broadband, which had the reputation of being one of the more responsive and customer-friendly ISPs.

This was what prompted the note I put at the end of Life Hacks, about the value of living some part of your life in public. If you can identify what are the valid criticisms of your code or your ideas or your writing, and who are the trolls, and which are the other people just being temporarily stupid obtuse geniuses-in-waiting, you can parlay all that valid feedback and move from fool to slightly less fool in quick time.

What I don’t know is how to stop the anticipation of that criticism, or the excessive idiocy of some of it, stop good people from becoming better. Today I heard of the existence of a library that is badly needed, but the person who is writing it doesn’t want to release it because “it hasn’t been tested enough”. Release it, and it will be!

I think my biggest encouragement to those who are scared of criticism to enter the public space is that if you don’t, the public space will be filled with people who have no fear of their failings whatsoever. And we all know what fools they are.

2008-08-19

my flatmates’ flotilla launches

My housemates have been away for a few weeks, building various-powered boats to sail in tandem down the Hudson:

I think this means they’ll be home soon. Unless they’re planning on sailing back.

2008-08-18

ORG update; the not-night terrors

I’m having so much fun doing this that I barely spotted that I’d crossed the thirty days mark! Luckily now I have noticed, so I can re-adopt my grudging acceptance of another month’s hard labour, thanks to my new, much despised by me, blog sponsors, who all joined the Open Rights Group to watch me suffer.

I’m now looking for five more donors to make me look the chump in September. If you’re enjoying this (you sadists), do consider joining the Open Rights Group for a fiver a month (cheap), and mail me your reference, and what you’d like the money to go on.

At the risk of sounding like some sort of endless Nerds In Need campaign, I’m amazed we’re up to over 900 members.

My personal goal for this fund-raiser was to hit our original 1000, but I’m afraid I egged the ORGsters to take it up to 1500, because that would let us do a bunch more work, and gain independence from other sources of funding. I find it fascinating how different EFF feels from other non-profits I’ve worked with because of its strong reliance on individual donors (it’s usually around 70% of our income). There’s around 13,000 EFF members from across the world, and all you have to do is say “Hello, I’m an EFF supporter, and…” to get my undivided (if ADD-ridden) attention. I guess I’d describe it as having a huge crowd of incredibly smart, but somewhat vocal, hands-off managers: when they drag me into their office (or just corner me at a conference) usually the start of a very high-level conversation.

I’ve noticed the same with ORG supporters. Oddly, my main involvement with ORG is because it’s part of my EFF job to help out other digital rights groups internationally, so I don’t see myself as an ORG representative per se. But whenever I speak to someone who starts with “So, I’m an ORG member and…”, we usually both end up scribbling madly in notebooks at the end.

I’d be more involved in day-to-day UK issues if it wasn’t for the time difference, which is nasty for a late-night person like me, so the UK falls under the same “all communication via email” shadow as India and Russia.

This may change. Next week, Ada starts kindergarten (the weirdly teutonic name for junior school here. Guards, take zis child to … DAS KINDERGARTEN). As some sort of punishment for living in a 21st century hyperpower, school starts freakishly early here. Ada needs to get to school at 7.40am. That means I have to get up at 6am (2pm UK time).

As some of you remember, the very reason why I started the whole Life Hacks™ thing was because I couldn’t get up in the morning. Now I have to, otherwise I get prosecuted for secondary inducement to truancy or somesuch. I have already warned everyone I know that the consequences will be dire. I will wreak my revenge on the species that plans to keep me a prisoner of the burning day-star. It will be World War Bedtime, let me tell you. Even worse, Ada has inherited my nap rage, so we’ll both be punching milkman in the face and snarling at postal workers for five hours of the “day”. Europe, you may quake now.

inbox seventy-two thousand; email2vcard

Typically my spare time today was spent in the folllowing proportions: 2 hours allotted to emptying my inbox; 3 minutes spent actually emptying my inbox; 1 hour, 57 minutes spent writing a program to theoretically “help” me empty my inbox.

Spreading the procrastination around, here’s that code. It’s for the millions of people who, like me, use good old command line mutt and a graphical contact manager on Unix (although I think it may have more uses than that). It takes a standard email, pulls out the name and email of the person the mail is from, generates a vCard, and then attempts to open that vCard in your nearby contact manager. It uses vobject, hewn from the glowing pits of Chandler, which you can install with easy_install vobject or apt-get install python-vobject. Hours of entertainment.

Things you might be able to glean from this:

2008-08-16

better living through probability: nationmaster and fivethirtyeight

Today involved pancakes, the massively co-ordinated garage sale of Bernal Heights, and preventing (and failing to prevent) small child injuries. Now I’m listening to Americans talking about being kids being made to read hefty classics which are thus ruined, like Moby Dick and Ivanhoe and Hemingway and … Ivanhoe? They make them read Ivanhoe here?

Yes, it’s Saturday, and you catch me racking myself over what to write about in this weeks’ column. I think I’d be more peppy if I hadn’t spent far too long last night browsing NationMaster and playing International Man of Goggling At Slightly Dodgy Statistics. Then I stumbled on Visualizing Economics and after spending a few minutes sniggering a bit too much at their plots which look like visualizations of random number generators, got far far too involved in their archive. I started with this depiction of share of world GDP of various countries from 1500-2000 (population over the same period is a nice complement).

For those of you with a matching love of statpron, and about to get overinvolved in US presidential politics (whether through your own choice or that of your dominant media), I heartily recommend Five Thirty Eight which takes a probabilistic approach to the election, running a simulation system 10,000 times and showing the number of times each split in electoral votes occurred in their dice rolls. The tone is slightly Obama-tilted ( with a 5% margin of snark provided by an ex-You Don’t Know Jack editor).

It is a beautifully quantum-multiverse view of the election — as though on the day I will absolutely confident that there will be thousands of universes out there where Alaska went for Obama, or where McCain lost Ohio but won the war: I just won’t know which one I’ll be navigating to. Oh the tiny fluctuations that lead to such endless points of divergence!

2008-08-15

bandwidth and storage and europe and america

After doing my reading into fiber in San Francisco, I’d learnt a couple of things: firstly, there’s a lot of fiber around, actually, and secondly, a lot of the fiber under me was owned by Astound, who bought it from RCN, San Francisco’s previously weak-as-tea competition to Comcast.

As it happened, he next day I got some flyers through the letter box from Astound offering 10Mbps for $60 a month. As I’ve been tottering along with 4Mbps/1Mbps for $55 with Comcast, I thought I should look into that.

Competition is a marvellous thing. Wherever in the US Comcast has been facing it, I discovered, they have been magically upping their rates to 16Mbps. Simply calling Comcast support and hinting I was going to shift caused them to mention this fact, and five minutes later, I’m running at around 20Mbps/3Mbps for $65 a month. Add to that the $170 terabyte turned up today, and I feel like I’ve just leapt up about twenty countries in some OECD chart. I guess what I should do now is call Astound and see if they will offer to move PAIX into my bedroom cupboard for $65.99.

Up until now, I’ve always assumed that the UK’s consumer bandwidth situation has been rather better than most of the US — a tidbit gleaned from smug Brit slashdotters, and envy-enducing reports from my friends about their DSL deal-shopping. The received opinion is that the US dropped the ball almost immediately after rolling out broadband, and was promptly outgunned by most of the rest of Europe, something that begging outside telcos for ISDN-level speeds in most of Silicon Valley confirmed for me.

Now, after spending a few minutes on Speedtest’s worldwide self-selected statistics, I’m not so sure. I was originally assuming there was some American-bias to the stats, but digging deeper that doesn’t seem to be the case. The US actually does pretty well compared to Europe (except for those bastards in Sweden, etc) these days.

One thing I’ve learnt is that nation-spanning preconceptions like this are often temporarily true, but not for half as long as they hang around. Pleasantly schadenfreuderish viewpoints have a lot of lag to them. Take mobile phone adoption in the US. When I first arrived here, the difference between US cellphone culture and the UK was stark, and I, like many foreign-media journalists, would frequently dine out on the gap. In 2000, you couldn’t actually consistently text people on other networks; nor would it be reasonable to expect a stranger to have a mobile phone at all.

Then, in I think about 2003, I was crunching some stats about the crappy cellphone penetration in the US for my European friends to gawp at. Instead of doing an us-and-them comparison, I did a time-based one. How far behind was the US chronologically from the UK? It turned out that the US had just crossed 50% of households owning a cellphone. Laughably small compared to 2003 Britain, where it was close to 80% (I am surely misremembering these stats, but bear with me). But nonetheless, pretty much exactly the same as 2000 Britain, my original basis for smugly lording it over the Yanks. America’s primitive phone culture was, it turned out, only as primitive as the futuristic super-advanced one I’d left three years ago.

Sure enough, when my family came to stay that year, my usual prattle about how Americans don’t have mobile phones like we do was swiftly undermined. “What are you talking about, Uncle?” said my annoyingly smart niece, “Look around you. They’ve all got mobiles.”. And they had, and my anecdotes were like drinking from yesterday’s half-empty, cigarette-filled party beers.

I think the same thing is happening with broadband. From 2000-2008 there was a much bigger consumer rollout in Europe than in the US, partly because of government-compelled competition in European telecom (and aren’t I a bad libertarian for even suggesting that), and some really terrible decisions both by business and regulators in the United States. But those differences are slowly closing out as both continents start reaching the limits of DSL and the current infrastructure.

I imagine folks will disagree, which is fine. I’m not entirely sure of the position myself. The real question is: how could we test this? Are the Speedtest stats enough on their own?

wuala

Woah, sorry about missing last night: I returned home from work and slept from 8pm to 9am. I blogged in my dreams though.

Briefly, yesterday’s copious free time (ie a few minutes) was spent looking at Wuala (thanks, robwiss!), which is a neat popularisation of some of my pet issues: the infrastructure is a decentralised, fault-tolerant, file storage, with private/public/group access created with a cryptographic filesystem (see the Cryptree paper for details on that, and this Wuala-made Slideshare for a general overview of the tech.) It’s notable for having a user-friendly UI, capability to a run the downloader in a browser via a java client, and therefore have linkability (for instance, in theory you should be able to download the Ogg Vorbis version of the Living in the Edge talk here, once it’s uploaded.) It just went public yesterday, and it’s fun to play around with.

I have a few questions about it, which may be more down to my ignorance than Wuala itself: the source is closed, and so I don’t know yet quite how tied the infrastructure is to Wua.la the company (if Wuala disappeared tomorrow, would the network still exist?), or where the potential weakpoints in overall security might be. On the plus side, Wuala is clearly being used in earnest both for public and private sharing, the user interface does a great job of shielding the crazy cryptopunk shenanigans going on underneath, and it’s cross-platform (albeit via Java, which means it’s not quiite working on my PowerPC Ubuntu server right now).

Tahoe is a lot more transparent, but seems to have a different use case at the moment, which is private nests of stable servers used for distributed backup. But if you wanted to do a free software version of Wuala, that still looks like where you’d start (and Wuala is where you would get your inspiration/learn your lessons from).

2008-08-13

the sort of day a horrible stereotype of me would have

Woke up feeling like the National Sleep Debt, danced with daughter to Fatboy Slim’s remix of magic carpet ride until we were late for school, returned back to struggle (unsuccessfully) with Skype for ORG board meeting call-in, email, run off to record Cranky Geeks, skidaddled to have lunch with Ben Goldacre, who I’d not met before (although am apparently in some big conspiracy with) but I now like a bit too much. Exchanged anecdotes regarding survivalists, on being sued, Americana, assorted data valdez, epidemiology. Had awful suspicion he is as charming as this with everyone. Quick email catch-up, run to pick up daughter. Daughter suggests we explore, get stuck on wrong side of Bernal Heights, shiver as fog closes in, ruefully consider that we may have to survive on only the toy dogs that we find abandoned outside hilltop aromatherapy spas, spend rest of evening trying to find useful link to Fatboy Slim’s remix of “Magic Carpet Ride” to begin this blog with. Wave fist at errant god of consistent metadata and music business models. Talking of which, I am vaguely cheered by continuing leaks of voluntary-collective-license-like systems being considered in UK, although still cautious about technological implementation, privacy issues and other yet unanswered questions, also how close labels always get to this before rearing like frit horses and then galloping off, and, most of all, just how fantastically, painfully, and repetitively smug Andrew Orlowski is going to be if it does come together. Decriminalising millions and saving the music industry will barely make up for it.

Content-free post, so as tradition demands I shall spackle it up with yet another old column from Linux User and Developer, called Danger Dot File. I don’t care what you think, I find these pieces intensely amusing, if only out of amazement that I have lived in an age where one can be paid to write jokes about Unix configuration settings.

2008-08-12

digging tahoe; wattage update; @fontface!

After Sunday’s brunch conversation, where we idly dreamt of a capabilities-based distributed-hash-table fault-tolerant filing system, Jeremy took a closer look at Tahoe, Zooko’s project that I’m continually plugging without completely understanding, and noticed that that is exactly what it is.

Last time I played with Tahoe I blundered into a dark alleyway full of unmet python dependencies and got mugged, but the latest version installed a lot more cleanly, and we now have a mini-grid going. I’m also re-reading RFC’s 34013405 on the Dynamic Delegation Discovery System. I can’t honestly tell you why it feels to me like these are all interlinked, because I’m still at that point where it’s all just pieces looking rather sad on the floor of my garage. I have good instincts, I think, but slow to rationalise, and I backtrack alot. Right now, for instance, my instincts are to put the DiSo people, the AllMyData people, the Zeroconf person, and several bored iPhone hackers in one room, lightly steam them in mild entheogens, and then just when they’re getting somewhere, divert a stampeding herd of straight-edge yet angry and dissolute teenage router-distribution developers into the building. And that can’t be right, so I’ve clearly got some more thinking to do.

I’m going to start dividing up these blog entries when I mention more than one thing, for the love of the metadata. But these are really throwaway items: we discussed how much power a home server draws, and I pulled figures from beyond my arse, but I’ve just run Kill-A-Watt on my old MacMini powerserver (the one running this website), and it does indeed pull down around 17W average. My arse speaks sooth. Of course, I’m not taking into account the cable modem, and the wifi router, so I guess I need to look into that.

In other news, one of the oldest bugs (but not the oldest!) that I watch in Firefox (@fontface support) is getting some new love. Soon, all the major browsers will be able to use downloadable fonts, and then what fresh desktop-publishing linotype hell will we be in?