Oblomovka

2008-07-18»

I never say anything about a performance I’ve just watched until I’m about half a mile from the venue. I’m not entirely sure why. I always used to think it was because I was scared that my scathing appraisal would be overheard by somebody’s doting mother, and she would knife me or tut loudly or somesuch.

But really that’s unlikely to be the real reason, because I’m not even very negative when I’m safely in the car. I start with an insufferable sigh, and then start obsessing about tiny points in the act that no-one else noticed, but which I’ll then blow up into huge theoretical frameworks of the nature of drama. At no point will I really give away whether I had fun or not.

Really, though, at comedy, I almost always have fun. I love going to see terrible comedians, and I love seeing brilliant comedians, and I’m usually canny enough to avoid seeing all the boring stuff in the middle. My favorite gigs are where the terrible comedians suddenly becoming brilliant, as though ridden by Loa, and when brilliant comedians failing miserably, all their clever tricks and mischeviousness failing like corked wine.

Eddie Izzard was about the fifth reason I gave up comedy, maybe 15 years ago, because he was already a master at a style we only haphazardly ever managed to pull off.

Our internal name for it was “barely rehearsed”, which I think was a term we pulled out of a glowing, but confused reviewer who simply could not understand why he was enjoying such rank amateurism.

The trick with being barely rehearsed is to have a firm structure with maybe five perfectly-rehearsed set pieces that you can hop to, with the rest broadly improvising around some key call-backs and themes and set pieces. The two tricks are knowing when to hop out of the improvised bits, and — and here is the HIDEOUS DECEPTIVENESS OF ALL ART — how to suitably disguise which bits are improvised, and which are not.

The beauty of hiding the joins is partly it makes everything seamless and ironically uncontrived, but most of all because you get the constant extra points everyone gets for improvising, and apply them to the multiplier of actually having worked it all out before hand. Some comedians feel that this is a cheat, but every comedian thinks the tricks that others use are cheats, and while their own are perfectly acceptable “comedic shorthand”, so there you go.

I think most people marvel at Eddie Izzard’s ability to go on extended flights of improvised fancy, but I (being special and uniquely insightful) m more admiring of his ability to back out of them as they crash and burn, while still holding the audience’s attention. He’s also fantastic at managing the segue between improvisation and set-piece. Nobody in the audience seemed to notice (only I, being special and brilliantly perceptive, etc) that he wasn’t pulling off either last night that well. It was like those bits where really awesome acrobats actually fall off their perches into the big net; I don’t think it really distracts from the show, but you can sense people having to donate some of their good will to get the show back on tracks.

Anyway, it was a good show which you can see often has better nights. It plays a bit safe with the material, but you know, anyone with ten minutes on Wikipedia, EULAs and iTunes auto-upgrade policy is going to win me over. If he starts doing “Scrappy Doo” material in a panic, just let it ride over you, and concentrate on his brilliant physical gestalts which still rock.

Comments Off on everyone’s a critic: eddie izzard in san francisco

2008-07-17»

Deciding to upgrade to Ubuntu’s alpha-state Intrepid Ibex because of a hope that tiny bug might be fixed: not such a good idea. Deciding that, because your beautiful composite set-up was broken by the Ibex kernel upgrade that you should maybe try out the even-more-alpha intrepid-proposed repository: frankly ill-advised. Struggling with the consequential collapse of all your wireless networking by attempting to remove and re-install dbus from the old Ubuntu: not so much asking for trouble, as drawing a pentagram in your own mother’s blood and hollering in Aramaic for same. So I was without a working laptop for much of today.

Well, as I’ve heard people say in the same circumstances, at least it was a good test of my backup policy. In the end I just threw up my arms, re-installed from an old Ubuntu Huffin’ Heron, and pulled my home directory off the backup drive.

Of course, the painful bit with re-animating an old and familiar set-up is trying to recall all the tiny mods and tweaks that one gave one’s system back in the ice age, then re-implementing them individually on the new system — all without saddling it all with your later, senile wanderings.

I actually do backup my /etc folder, so it wasn’t that bad — Debian is pretty good at keeping most of the configuration files in etc, on pain of maintainer death. But I hadn’t kept a list of the many extra packages I’d installed. Fortunately, Debian/Ubuntu machines, positively trembling with racial knowledge of how badly its users screw up in the past, keep their own backup copies of this list, in /var/backups/dpkg.status.0 .

If you’re ever in the position of checking this list with your current system to try and work out what packages you should install to get to your old state, try package_list. You feed it the dpkg.status backup file as a command line argument, and it spits out the packages you need to install.

Here’s some of what it spat at me. I’m off to see Eddie Izzard now — when I get back, I’ll highlight some of my favourite Ubuntu packages here:

Hello, I am back. I don’t know whether your RSS reader will notice this, but here are my all new summaries of my dpkg list.

These are ones that I snuffled from other, non-official Ubuntu repositories. The SHAME!

amazonmp3

Amazon is a popular book-selling merchant and music distributor. They have a Debian friendly MP3 downloader.

avant-window-navigator-bzr

Avant is in no way a fake MacOS dock applet.

chandler

I tried Chandler for a bit, but now I am all about the Kontact

ec2-ami-tools

Amazon is a popular book-selling merchant and music distributor and virtual machine rental site

isight-firmware-tools

Yes, I run Ubuntu on a Macbook. Somewhere, Steve Jobs is screaming

skype

And I use proprietary software, so that’s Stallman screaming back.

Here are the official Ubuntu packages.

adblock-plus

Who knew you could install Firefox plugins using the Ubuntu package system? Me, clearly, at some point.

alien

Alien lets you convert Redhat RPMs into Debian Ubuntu deb packages. Useful!

apache2

If MacOS laptop owners can run apache locally, then so can I.

apg

More useful! APG is an obscurely named password-generating utility.

avahi-utils

Avahi is the Linux name for Apple’s Bonjour which was the name for Apple’s Rendezvous which was the name for the Internet’s Zeroconf.

blogtk

When somebody says “What can I use instead of Ecto on Linux”, tell them about BloGTK. Then hope they don’t ask any more questions, because it’s not really that polished.

cheese

This is like Photobooth. Not quite as polished, but quite as useless.

compizconfig-settings-manager

This is, by contrast, awesome and lets you turn your poor user-interface into the most tweaked, weird, and customized cube-spinning zoomey wibble-wibble monster imaginable. Makes everyone sick with jealousy. Or vertigo, hard to say.

dillo

Dillo is the rude-sounding super-minimal super-fast browser. Good for checking what your site looks like to Victorians.

discover1

This is discover1, of course.

dovecot-imapd

Dovecot is about as sane an IMAP server as you can muster.

gammu

gnokii

gnome-phone-manager

gnome-vfs-obexftp

These are all for talking to my Nokia phone, and pulling addresses off it, and sending it SMSs and kissing it and hugging it.

gnumeric

I have a soft spot for Gnumeric, which was Gnome’s competitor to Excel, and yet has somehow managed to not become as crazy-ass as Evolution (or OpenOffice, for that matter).

gsynaptics

If you have a Macbook, and you run Linux, this lets you set up your trackpad just the way you like it. Sixty hours later.

hfsplus

If you have a Macbook, and you run Linux, and secretly keep MacOS on it because you’re not entirely crazy, this gives you some utilities to look at them.

idle

Idle is the standard Python editor. It’s sort of funky.

imagemagick

Imagemagick is actually what you need to do all that graphical image conversion.

iodine

This is that program that lets you tunnel IP over DNS, like in Little Brother. The technique was popularised by Dan Kaminsky, who later went on to save the world.

kontact

Even though I use Gnome, Kontact rules my world. I live with the pain of all the background KDE libraries coming in and spilling their strong German beer everywhere.

lynx

Lynx is the text web browser. It’s useful for “lynx -dump http://thiswebpageinasemblenceofatextfile.com/”

midori

Midori is another lightweight browser. It’s useful for when you are tired of only having seven other browsers.

miredo

Miredo gives you IPv6 when other people only give you IPv4. Useful for … well, it’ll be useful one day.

miro

Miro is the new name for Democracy, which has been less popular ever since Iraq.

mnemosyne

I forget. Oh wait, it’s a flashcard memory aid!

mozilla-firefox-locale-en-gb

What can I say? I miss the old country.

mysql-admin

mysql-client

mysql-navigator

mysql-server

ndisgtk

Do you know how long I’ve been scared of relational databases?

nmap

For portscanning the hell out of strange networks and broken machines.

ntop

For working out who the hell is using all the bandwidth on your network.

oolite

GPL’d Elite! For Linux! And MacOS! And SGI Irix!

pandoc

Incredibly useful for converting to and from various markup languages, like HTML, Markdown, RTF, etc. Written in Haskell for extra cred points.

pdftk

For doing hideous things to PDF files.

pidgin-libnotify

For actually telling me when someone is trying to get my attention on IM.

pommed

Handles Macbook hotkeys. From the really useful, somewhat obscure Mac support Ubuntu repository.

powertop

Use fewer watts!

privoxy

Part of my tor setup, natch.

pylint

Horrifically pedantic code style checker.

python-beautifulsoup

Leonard’s damn fine, damn tolerant HTML parser.

python-feedparser

Mark’s damn fine, damn tolerant RSS/Atom parser.

python-mechanize

When you want to webscrape like a pro.

python-nose

Python unit tests handled gently and kindly.

rtorrent

The best of the background, text-based torrent handlers.

sc

The ‘vi’ of spreadsheets. Really.

sox

The ImageMagick of sound files.

squeak

For wasting TOO MUCH TIME in happy Smalltalk land.

squeak-plugin

sshfs

Mount remote systems using just ssh.

swftools

For messing around with Flash files (like extracting images, etc).

terminator

Multiple terminal windows in the same (Python-coded) window.

tidy

Cleans up your (or someone else’s) HTML.

tor

For anonymity and censorship circumvention.

torbutton-extension

trickle

A command line program that will throttle and bandwidth limit almost any other command line program

vlc

For when mplayer won’t cut it.

vrms

Virtual RMS — for nagging you about those non-free programs (see above).

wammu

The graphical bit of gammu, the cellphone software.

wine

For running Windows programs.

wireshark

For monitoring your Net traffic (and pointing out to others how easy it is to monitor theirs).

Phew!

Comments Off on what ubuntu packages did i install again?

2008-07-16»

Curses. Thanks to the irresponsible exuberance of Matt R., David M., Andrew, Diggory and Adewale, I have sentenced myself and the internets to 30 days of blather (folks, if any of you would like links to your sites, send me a mail. Yeah, that’s right kids, I’ll trade my Google juice for your donations).

A few of my bribers mentioned some riders they’d like. First, a copy of my slides from the OpenTech talk, “Living on the Edge”. That seems perfectly reasonable, given that I promised all over the place that I’d do that too. Here’s the original PDF I used. There’s also an OpenOffice presentation file here that has slightly more detail in it.

The blurb I sent OpenTech six months ago is below.

Living on the Edge (of the Network)

When you want to make a private picture or note available only to your friends, why do you hand it over to a multi-national corporation first? What use is a mobile phone running Apache? Does IPv6 really exist? Can we be ecologically-sound and still run our terabyte home servers? Please? These, and other whining rhetorical questions answered by Danny O’Brien, ORG founder and EFF activist.

It was mostly a reworking of these blog entries. There’s been a lot of talk and independent thinking in this area for the last few months, leading to a flurry of public action in the last few weeks as many people come to the same conclusions: that we need to consider a counter-balance to the current move toward centralisation online.

The way I phrase it is that “we’re back at 1984” — not the novel, but the point where Richard Stallman realised that if he was going to preserve the most powerful freedoms of his community into the future, he was going to have to sit down and re-implement Unix with a better license.

We’ve reached the same point with the move to software as a service. If we want people to have the same degree of user autonomy as we’ve come to expect from the world, we may have to sit down and code alternatives to Google Docs, Twitter, and EC3 that can live with us oon the edge, not be run by third parties.

This is the spirit of the Franklin Street Statement and more practically, software like Laconica.

I’m sure to be blathering more on this topic in the next month: if it gets too much, I will consider taking more donations to shut myself up.

(Incidentally, if the slides don’t make sense, I’ll try and get around to recording a slidecast of the whole talk or uploading some video. Crossing ORG’s palms with silver and mailing me will make this more likely… :) )

2 Comments »

2008-07-14»

I saw and heard far too many stirring matters in Europe this month for it to strictly count as a relaxing holiday. It was more like some sort of brisk Victorian tale of moral recuperation, where a malaise-filled city gent, falling asleep at his desk, is shown by his labouring conscience vivid images of model courage struggling against enormous odds while terrible forces swell nearby to depose civilization in the (third) highest corridors of power. Inspired and chastened by his vision he awakes on Sunday morning to sweep himself off to a church revivalist meeting and dedicate himself anew to the cause.

Of course, just as in Victorian times, the inspiration rarely lasts longer than late Sunday afternoon.

But I am determined to be good! For ever and ever! Or at least until mid-August! And you can help me, dear occasional reader!

At OpenTech, I helped the Open Rights Group launch their new membership expansion campaign to double their supporter level from 750 to 1500.

To encourage you to help fund Britain’s own grassroots digital rights group, and to improve my own moral standing, I hereby make a pledge:

If five people reading this sign up for ORG (or increase their subscription from a fiver to a tenner a month), I hereby decree that I will blog every weekday for the next month.

I have lots of things to say, and they are all terrifically interesting, but I am currently too louche and feckless to express them. Your fiver will stiffen my resolve, gall me to action — and support a worthy and fine institution.

Simply send me your ORG “scalp” (the reference code that you get when you sign up), and when I have five, I will start spilling all the beans I have at my disposal.

Or you know, you could just callously click “mark all read”, and move onto the next RSS item.

Oh you wouldn’t. Don’t you dare! I’ll put a javascript spell on you!

Update: Two and half sign-ups! (one of them backdated their increase to the beginning of the year, so I count that as one and a half). You’re so close to obliging me to waste valuable time!

Comments Off on join org *and* the rss reader gets it

2008-01-07»

Scene 1

Ada and Claire, before The Shoe (Surrealist Object Functioning Symbolically) (1974)

Ada: What’s THAT?

Claire: [READING FROM SIGN] It’s … Salv-a-dor Da-li

Ada: Oh.

[BOTH ABRUPTLY RUN OFF]

Ada: [FROM ACROSS GALLERY] BYE SALVADOR DALI!

Scene 2

Before Marcel Duchamp’s The Fountain (1917):

Claire: What’s THAT?

Ada: [PEERING CLOSELY] I think it’s a kind of toilet.

Claire: No it’s not!

Ada: Yes it is. You just don’t see it.

Comments Off on at the san francisco museum of modern art with two four-year old girls

2008-01-01»

I had an awesome New Years, but it sounds like my friends who made it to CCC this year had an awesomer one. They’re all now obsessed by this 1996 Eurovision epic, Surfen Multimedia by the Eurocats. I can’t think of a better monument to Germany’s influence on global cyberculture.

(My Sacrificial New Year’s Resolution is to not say “awesome” so much.)

Comments Off on missing berlin

2007-11-24»

It’s in some degree a little tragic that holidays give me a chance to REALLY GEEK OUT – as though I don’t have endless opportunities at other times. But, honestly, and perhaps equally as tragically, geeking out is what I do when I need to become philosophical and retrospective and moody and irritated at everyone. It also encourages me to drink heavily, all of which I think are the true meanings of festivals in the human condition anyway.

My geeking out this Thanksgiving (apart from a hard-to-fight urge to buy consumer electronics, which I swear to God must be down to morphic resonance, because I’m really not that interested in retail usually, have successfully ad-blocked most of my life, and don’t have any money right now anyway), was a day zero purge of my servers and laptop home directories.

Usually this kind of thing ends up in complete disaster, like deciding one day to take your car apart and put it back together again.

This time it’s been working out rather well. Instead of naked and crying on the floor, I’m clear-headed though a bit chilly.

There wasn’t really any reason for it, except that recently I’ve been very forward-looking: as though I’ve been trembling on the edge of a precipice in a wing-suit. I’ve not been looking down, and I’ve not been jumping, but I’ve certainly not been looking behind me either. So it was about time to push myself over.

rm -rf! goes the urge, and I gave into it. I’m slowly re-introducing (or rewriting) all the scripts that I use, and taking into account the lessons I’ve learnt in the last few years.

It’s really easy to be frozen in the headlights when you delete your existing directory structure and start again, because you end up thinking so much about the future. But you can’t tell anything about the future, so there’s no point to being frozen. You just have to express confidence that it’s not going to be awful, and jump.

I have a fairly concrete aim, which is to see how close I can get to having a setup that is a) replicated everywhere, and where I can b) fall-back to different machines if one breaks, and c) I can throw at an EC2 instance as easily as I can throw it at a Nokia N810 or my Mac laptop, and d) shares as much as I can to the rest of the world.

God knows, I won’t get there, but as this has been my aim for some years, I have some lessons learnt, and some new technology has been rolling along for it. For instance, for years I’ve been following Joey Hess’s Living Life In Subversion credo: version-controlling my home directory, and trying to keep as much of it as public as I could.

It’s a great way to think of your digital life, not because of the fact that it keeps all your documents as revertable backups (like MacOS’s Time Machine) or allows you to sync your home directory across many systems so much, as the discipline of thinking “how much on my computer should be private, and how much should be public?”. Joey keeps a huge chunk of his home directory in a public repository, and it’s incredibly educational – both for readers and for him, I suspect.

Being more public is terrifying, and yet freeing at the same time: apart from anything else, you quickly learn to discriminate between public-because-i-created-it-and-want-to-share and public-because-it’s-not-actually-mine. That’s to say, I have a bunch of free ebooks, say, that I can make public because they’re public documents. But at the same time, I don’t worry too much about backing them up, because I have a world of backups out there already. I’m sure those of you who torrent films or used to file-share music will recognise that feeling. Why should I keep this, when so many others have a copy I could obtain easily enough? My copy of Brian Eno’s albums takes up a few megabytes on my hard drive. Should I back it up? Or should I just keep the receipt, knowing I could get a new copy so quickly if it was lost locally, here?

Thinking about what’s really private is also very clarifying. Passwords are private. Are bookmarks? Which bookmarks? What’s the minimum set of bookmarks I can make private? More people are asking this after their heavy del.icio.us use, I bet.

Here’s my current new directory structure. Like the old one, the public bits will appear sooner or later as a browseable repository on this machine. The private stuff is ghettoised into a single folder – directories that hold things like ssh settings are symlinked into it.

Most of this is (now) kept under Mercurial, a distributed version control system. The scripts are in Python, where I can help it. The structure is replicated across all my machines, with the same contents. Not everything is amenable to version control, but I have some ideas about how to keep the other stuff mirrored across all my machines too.

2007-11-18»

Update! Becky says that the sainted Joseph Rowntree Reform Trust have offered up to 10,000UKP of matching funds for every supporter ORG gets — so if you join for a fiver a month, you’ll be giving ORG ten pounds to play with!

So Join Now! Join Twice!

It’s hard to believe that it’s two years since the Open Rights Group got its first public support from a thousand British Net pioneers. Flicking through their annual report today, it feels like they’ve been around for decades.

What ORG’s staff has done is astounding. When we first sketched out what could be done with a thousand people’s fivers, we thought we could just about pay to have someone clueful on the phone 9-5. That would be enough: redirecting uhh, clue-misdirected journalists to the unheard-from Net users, and real tech experts (not just blowhards) who could explain without the usual fearmongering or special interest hype.

Honestly, I felt that if we just managed to have someone next to the music industry spokesman next time a TV show swallowed the “Internet is full of pirates and criminals, and must be smacked into obeyance” line, ORG would have paid its way.

Instead, here’s what ORG has done with its scant resources. After two years, it’s not just a media clearing house, although it does that too. It battled a celebrity-studded publicity campaign that sought to extend copyright terms. It fought the hype with with clear facts and economics, and won – the first time that has happened anywhere.

Its advice to the Gowers report on Intellectual Property helped give the British Government the most progressive outline of future IP policies in the world.

It organized Britain’s first ever volunteer analysis of electronic voting, and showed that bad e-voting counts could have changed who won in Scotland: a revelation that still shocks me.

Right now, is planning to advise British businesses on how to work with the new norms of copyright. And to guide it, it’s assembled an amazing group of British-based advisory board members, including the coder of Apache’s SSL support, one of Linux’s key figures, the co-founder of the UK’s first commercial Internet provider, and the drummer off of “Blur” – match the names to the reputations. They really are involved in the strategic and technical decisions that ORG makes every day – and it shows.

If you want just a recent example of the sort of in-depth knowledge ORG already shows, check out this GrokLaw interview with Becky Hogge, ORG’s Executive Director. Detailed, smart comments on the BBC iPlayer, a messy but vital part of the UK online debate right now. Now imagine that kind of knowledge being inserted, behind the scenes, in press rooms, Whitehall offices, and TV studios, day in, day out.

Part of the reason it’s been so successful is because of the incredible input of ORG supporters. It’s hard to point to the offline work this incredible team manage, but just give you a taste: If you want the most vibrant, wide-ranging, sensible discussions of IP and privacy online: subscribe to ORG-discuss, a list that has representatives of the music industry, her majesty’s sceptical press, security mavens, and free software advocates. It’s knowledgeable *and* very civil, a minor miracle in itself. You might also want to check out ORG’s equally smart wiki.

Here is where I ask you for moneySo, here’s the most amazing thing. ORG doesn’t do that on a thousand people’s fivers at all. ORG does it on less.

To get our ballpark income, ORG would have had to have converted every single one of the pledge-signers. I think we got around 50%.

So to celebrate two years, I encourage everyone to try and push the membership up to the promised one thousand. No, two thousand.

If you’re an ORG supporter, pressgang two of your friends to join. Find that online pal who is even more fanatical than you in pursuit of digital rights. Tell the blowhards on Digg or Slashdot it’s time to put their pounds where their posts are. Heck, buy one in your mum or niece’s name for Christmas: it’s their Internet too. And check whether your own membership has lapsed (It happens – *blush* mine expired earlier this year, and I missed the memo – I’m back in the black now). Just click here.

Think what ORG can do in the next two years. Think what we can do with 2000 members. Think what we can do with 20,000.

Most of all, think what will happen if we don’t do something.

Comments Off on celebrating org’s second birthday

2007-11-14»

In case you think this piece is more incoherent than usual, I should explain that you’re reading it as I write it. More on that when I finally write the final conclusion to the piece. If you’re looking at this in an RSS reader, and there’s about five million other earlier versions, I apologise. Your aggregator is doing something that I wasn’t expecting, and I think may be a little silly. As the old spammers used to say, “just hit delete!”

So the edge (which is to say, where you live: your home server, your cellphone even, whatever is closest to you on the Internet) seems to be getting more reachable than it was. But what about reliability? If you’ve ever run an important service on your home machine, you’ll know about the Vacation Effect. This is a mysterious force which causes the home computer that to handles your email to crash within hours of you leaving home for a three week vacation, causing you to have to advertise for burglars on Craigslist to break into your home and reboot it.

Even if you can imagine the hardware at your house to be somehow more reliable, there’s always your flakey Net connection, which is up and down like a sine wave’s drawers. Dynamic IPs dynamically change, cable modems reset every few minutes, DSLs are flippantly unplugged by backhoes and disgruntled CO engineers. How could you possible imagine you could run a reliable service on that?

So, as some of you might have guessed, I moved Oblomovka off its co-loc a few weeks ago when I started this series, and transferred the whole website to my Mac Mini (perversely running Ubuntu) that I have in my cupboard here at home. I haven’t heard anyone complain about its unavailabilty, but then again I haven’t invited comments. I’m pretty sure it’s been down a few times, and as its heaviest user, I know I haven’t been able to ping it on occasion.

But it hasn’t been a huge problem. Partly because I’m not really an essential service to anyone else. Oblomovka down? Oh well, I guess I only have a few hundred other blogs to search. Mostly, though, I think it’s down to my first major point:

• The return of polling. Google picks up that I’ve updated my blog within a few seconds of me hitting send (I think because I ping one of the major blog ping servers).

  Judging from the last time I turned on logs, almost everybody reads this site from RSS readers. RSS readers don’t need any stinking five nines. If my server is down, they keep on knocking at the non-existant door until I come back up.

  I used to think that this was a fantastically bad idea, because of course polling doesn’t scale. I wrote a piece for New Scientist in 2004 hinting that this could be a problem for RSS.

  As we now know, I was right, and RSS collapsed as soon as blogs became popular, and brought down with it the rest of the Internet, which is why you’re reading this on an Olivetti typewriter operated by a spirit medium.

  As it happens, RSS is as shonky as the rest of the Net, and just keeps on acting as though it should work. My server doesn’t die under the weight of constant pings. And when my server does die, my readers just have to wait until it’s back up. I just don’t need the reliability.

  That’s lovely for me, but what about, say, Boing Boing, or some other super-popular blog where readers (and advertisers) might get really peeved if I disappeared?

  Well, what we have there isn’t a reliability problem. It’s a scaling problem. You’re getting lots of hits? Then commercialise them and get off your damn home server, you dirty freeloader. Which brings me to my next point:

• Increasingly granular scalability. Good scalability scales down as well as up. Every Net engineer I know has ended abandoning complexity in scaling by collapsing everything into something that just works if you throw more servers or RAM at it, and still works if some of those servers and RAM fry at some future entropic point.

  EC2 and other cloud services appear to be the natural extension of this upwards in an unbounded way. If I need better reliability and bandwidth for my users with a simply-scalable design, I can (ideally) just throw more EC2 servers at them.

  The corollary to this is that if we design edge services well, if they overload the edge, we can throw them up into the cloud again. People can live with higher levels of unreliability than most of us (including the brave, anal, guardians of our system administration class) imagine is possible. The Net is proof of this. It’s shitty, but it’s good enough. And if we want it better, we can usually pay for the privilege.

  What running services at the edge is, in effect, creating services that can scale down to the minimum required. And that in turn, allows those services to scale right up to where they need to be, should they need to be more reliable and with greater loads.

Now, smart readers may have spotted the problem here. I began this discussion because I was confused and worried that we hand over our most private data to companies like Google, and SixApart and Amazon, when really the safest place to keep private data is on your own machine. Am I now suggesting that if somehow it’s okay that your edge server is flakey, because, hey, you can always use Google, and SixApart and Amazon. Aren’t I contradicting myself?

Yes. NO. Hold on. There’s a real difference between holding your data yourself and passing it to others in the event of emergency or changed circumstance. It would be better to think of these cloud services not as where you keep your data, but as temporary caches for the edge. I have an encrypted backup stored somewhere out there. I’m confident that only I have access to that. I’m not sure what I’d do if my home system did go down, but in an ideal world, I’d just feed that backup my key, have it float into operation on an EC2 machine, and then point oblomovka’s DNS to itself. When I had time to get things back to normal at home, I’d evaporate that EC2 machine. Oh, sure, evil gnomes from DoubleClick or the NSA might have pickpocketed that image while it was running, but I would at least have some intimation that was happening (just as the cops might break into my house and feel my pants, but I’d have a hair on the window if I was truly that paranoid). And if what we were talking about was truly just a cache of my current state (like a memcache of the last time my server was around), then it would not expose much deep knowledge of your precious life.

Especially as the benefits of being on the edge grew. For instance, for those of you peering at this from your RSS feed or on the Website before 12:20AM PDT, you’ll have noticed that I’ve been typing this directly into my home server. The words appear almost one by one on the site, as I tweak and update it. I could do that with a remote server, but it’d be a pain: the proximity of the edge to where I am gives the me the low latency and the reactivity to do these things. I feel nearer to the Net when it works like this. It feels more like I thought the Net would be, and more like how I think it will be soon.

Reached this point at:2007-11-15T00:11-0800. Keeping the message at the top to give an idea of how this might have felt to watch being typed at the time. Still under construction for typos and grammar fixes.

|

Comments Off on how many nines does one person need?

2007-11-13»

I realise that I left everybody on tenterhooks in that last post about why NAT and the unreliability of most consumer-quality endlinks. I’m sorry to have kept you waiting for my poorly-thought-out flailings so long. In my defense, my dreams have become far too weird to document here. Suffice to say they have included a dream that was half about falling in love in your mid-twenties, and half about being a prominent insect collector, and one stormer of a revery in which I dreamt that I was writing one of these blog entries about dreams. How self-referential can you get? As self-referential as this paragraph, is the answer.

Anyway, one of the problems with the edge is that these days it is largely behind NATs and unreachable for incoming connections. Usually we get around this by using ICE, STUN, and other horrendous hacks that you don’t want to stare at too long. But is this a problem these days? I think not for a couple of reasons.

• Broadband routers as servers. You know, it boggles me why the home router folk don’t take more advantage of their prime position to seize the home media center high ground. While everyone else like Microsoft and Apple fusses about trying to plug into the existing domestic media assets (aging TVs, HDTVs that will plug into anything anyway, disposable hi-fis, DRM-ridden content devices), router manufacturers should exploit their powerful monopoly: access to an world-addressable IP address. Really the only problem routers have in being kings of the edge is home geography: they’re usually hidden away somewhere close to the DSL/cable spigot, not in your slinky TV center. Someone who starts selling a two-for-one pack of a) a broadband router with NAS and b) a plug-sized wireless extender with video and audio inputs, and then has the gumption to include BitTorrent, PVR and streaming software would probably be able to seize some high value market before Apple had managed to reverse out of the Apple TV cul-de-sac they’ve maneuvred into.

  Oh dear. I do apologise. I appear to have become yet another convergence pundit. Let me come down off the crack a moment, and make the point I was originally aiming for. If you want a edge machine with lots of spare cycles, privacy, and a world-reachable IP address, your standard Linux-powered broadband/wireless router is right in the sweet spot. Plus, such routers are now tipping into that space where they are a bit too powerful for their own good, and are wandering off to find new uses: DD-WRT, OpenWRT.

  I suppose come the IPv4 Apocalypse, even ISPs will be handing out private IP space addresses behind their own monster NATs (some already do), so even routers will become inaccessible for incoming connections. Which brings me to:

• IPv6 (DO NOT LAUGH). While IPv6 is still waiting the moment for it to successfully take over from IPv4 (confidentally estimated to be the day after the United States goes metric, and coincidental with monkeys flying out of the IETF Butt-NG Working Group), it is becoming weirdly easy to build stable low-bandwidth, high reachability, IPv6 nodes on the edge. Most every Windows box from XP on can find itself a tunneled Teredo IPv6 address with one command. Miredo will get you the same with scary ease on Mac or Linux (seriously – try it!). Putting 6to4 on the aforementioned broadband routers will suddenly lurch everybody with a modern OS on your local network into the world of IPv6. Give an IPv6 address to a modern OS, and it just works. It’s just the IPv6 addresses that are (ironically) so impossible to obtain.

  Miredo tunnels through a third-party like Windows or poor old remlab.net, so it’s no good for insecure or high-volume data. But if you’re just trying to reach the edge using standard protocols and none of that tedious mucking around with NAT traversal, it works well. Remember, we’re not talking about file-sharing here, we’re talking about low-volume access to Web and Web-like services.

  Let me intrude here with a single, shocking fact: as Craig notes, a lot (did you say c. 50%, Craig?) of BitTorrent clients are advertising themselves as having IPv6 addresses, even if they don’t use them. It would make total sense if filesharing was what finally bootstrapped the IPv6 world.

Yeah, but what about availability? Aren’t consumer last-mile connections prone to zzzzzzzt NO CARRIER?

Tomorrow! I promise!

(If you enjoy these sorts of discussions, but wish it was a little less one-sided, let me here point you to Zooko’s vintage-yet-riproaring P2P-hackers list, where you can learn from people who actually understand these things.

Comments Off on reachability on the edge