Oblomovka

2007-11-14»

In case you think this piece is more incoherent than usual, I should explain that you're reading it as I write it. More on that when I finally write the final conclusion to the piece. If you're looking at this in an RSS reader, and there's about five million other earlier versions, I apologise. Your aggregator is doing something that I wasn't expecting, and I think may be a little silly. As the old spammers used to say, "just hit delete!"

So the edge (which is to say, where you live: your home server, your cellphone even, whatever is closest to you on the Internet) seems to be getting more reachable than it was. But what about reliability? If you've ever run an important service on your home machine, you'll know about the Vacation Effect. This is a mysterious force which causes the home computer that to handles your email to crash within hours of you leaving home for a three week vacation, causing you to have to advertise for burglars on Craigslist to break into your home and reboot it.

Even if you can imagine the hardware at your house to be somehow more reliable, there's always your flakey Net connection, which is up and down like a sine wave's drawers. Dynamic IPs dynamically change, cable modems reset every few minutes, DSLs are flippantly unplugged by backhoes and disgruntled CO engineers. How could you possible imagine you could run a reliable service on that?

So, as some of you might have guessed, I moved Oblomovka off its co-loc a few weeks ago when I started this series, and transferred the whole website to my Mac Mini (perversely running Ubuntu) that I have in my cupboard here at home. I haven't heard anyone complain about its unavailabilty, but then again I haven't invited comments. I'm pretty sure it's been down a few times, and as its heaviest user, I know I haven't been able to ping it on occasion.

But it hasn't been a huge problem. Partly because I'm not really an essential service to anyone else. Oblomovka down? Oh well, I guess I only have a few hundred other blogs to search. Mostly, though, I think it's down to my first major point:

• The return of polling. Google picks up that I've updated my blog within a few seconds of me hitting send (I think because I ping one of the major blog ping servers).

  Judging from the last time I turned on logs, almost everybody reads this site from RSS readers. RSS readers don't need any stinking five nines. If my server is down, they keep on knocking at the non-existant door until I come back up.

  I used to think that this was a fantastically bad idea, because of course polling doesn't scale. I wrote a piece for New Scientist in 2004 hinting that this could be a problem for RSS.

  As we now know, I was right, and RSS collapsed as soon as blogs became popular, and brought down with it the rest of the Internet, which is why you're reading this on an Olivetti typewriter operated by a spirit medium.

  As it happens, RSS is as shonky as the rest of the Net, and just keeps on acting as though it should work. My server doesn't die under the weight of constant pings. And when my server does die, my readers just have to wait until it's back up. I just don't need the reliability.

  That's lovely for me, but what about, say, Boing Boing, or some other super-popular blog where readers (and advertisers) might get really peeved if I disappeared?

  Well, what we have there isn't a reliability problem. It's a scaling problem. You're getting lots of hits? Then commercialise them and get off your damn home server, you dirty freeloader. Which brings me to my next point:

• Increasingly granular scalability. Good scalability scales down as well as up. Every Net engineer I know has ended abandoning complexity in scaling by collapsing everything into something that just works if you throw more servers or RAM at it, and still works if some of those servers and RAM fry at some future entropic point.

  EC2 and other cloud services appear to be the natural extension of this upwards in an unbounded way. If I need better reliability and bandwidth for my users with a simply-scalable design, I can (ideally) just throw more EC2 servers at them.

  The corollary to this is that if we design edge services well, if they overload the edge, we can throw them up into the cloud again. People can live with higher levels of unreliability than most of us (including the brave, anal, guardians of our system administration class) imagine is possible. The Net is proof of this. It's shitty, but it's good enough. And if we want it better, we can usually pay for the privilege.

  What running services at the edge is, in effect, creating services that can scale down to the minimum required. And that in turn, allows those services to scale right up to where they need to be, should they need to be more reliable and with greater loads.

Now, smart readers may have spotted the problem here. I began this discussion because I was confused and worried that we hand over our most private data to companies like Google, and SixApart and Amazon, when really the safest place to keep private data is on your own machine. Am I now suggesting that if somehow it's okay that your edge server is flakey, because, hey, you can always use Google, and SixApart and Amazon. Aren't I contradicting myself?

Yes. NO. Hold on. There's a real difference between holding your data yourself and passing it to others in the event of emergency or changed circumstance. It would be better to think of these cloud services not as where you keep your data, but as temporary caches for the edge. I have an encrypted backup stored somewhere out there. I'm confident that only I have access to that. I'm not sure what I'd do if my home system did go down, but in an ideal world, I'd just feed that backup my key, have it float into operation on an EC2 machine, and then point oblomovka's DNS to itself. When I had time to get things back to normal at home, I'd evaporate that EC2 machine. Oh, sure, evil gnomes from DoubleClick or the NSA might have pickpocketed that image while it was running, but I would at least have some intimation that was happening (just as the cops might break into my house and feel my pants, but I'd have a hair on the window if I was truly that paranoid). And if what we were talking about was truly just a cache of my current state (like a memcache of the last time my server was around), then it would not expose much deep knowledge of your precious life.

Especially as the benefits of being on the edge grew. For instance, for those of you peering at this from your RSS feed or on the Website before 12:20AM PDT, you'll have noticed that I've been typing this directly into my home server. The words appear almost one by one on the site, as I tweak and update it. I could do that with a remote server, but it'd be a pain: the proximity of the edge to where I am gives the me the low latency and the reactivity to do these things. I feel nearer to the Net when it works like this. It feels more like I thought the Net would be, and more like how I think it will be soon.

Reached this point at:2007-11-15T00:11-0800. Keeping the message at the top to give an idea of how this might have felt to watch being typed at the time. Still under construction for typos and grammar fixes.

2007-11-13»

I realise that I left everybody on tenterhooks in that last post about why NAT and the unreliability of most consumer-quality endlinks. I'm sorry to have kept you waiting for my poorly-thought-out flailings so long. In my defense, my dreams have become far too weird to document here. Suffice to say they have included a dream that was half about falling in love in your mid-twenties, and half about being a prominent insect collector, and one stormer of a revery in which I dreamt that I was writing one of these blog entries about dreams. How self-referential can you get? As self-referential as this paragraph, is the answer.

Anyway, one of the problems with the edge is that these days it is largely behind NATs and unreachable for incoming connections. Usually we get around this by using ICE, STUN, and other horrendous hacks that you don't want to stare at too long. But is this a problem these days? I think not for a couple of reasons.

• Broadband routers as servers. You know, it boggles me why the home router folk don't take more advantage of their prime position to seize the home media center high ground. While everyone else like Microsoft and Apple fusses about trying to plug into the existing domestic media assets (aging TVs, HDTVs that will plug into anything anyway, disposable hi-fis, DRM-ridden content devices), router manufacturers should exploit their powerful monopoly: access to an world-addressable IP address. Really the only problem routers have in being kings of the edge is home geography: they're usually hidden away somewhere close to the DSL/cable spigot, not in your slinky TV center. Someone who starts selling a two-for-one pack of a) a broadband router with NAS and b) a plug-sized wireless extender with video and audio inputs, and then has the gumption to include BitTorrent, PVR and streaming software would probably be able to seize some high value market before Apple had managed to reverse out of the Apple TV cul-de-sac they've maneuvred into.

  Oh dear. I do apologise. I appear to have become yet another convergence pundit. Let me come down off the crack a moment, and make the point I was originally aiming for. If you want a edge machine with lots of spare cycles, privacy, and a world-reachable IP address, your standard Linux-powered broadband/wireless router is right in the sweet spot. Plus, such routers are now tipping into that space where they are a bit too powerful for their own good, and are wandering off to find new uses: DD-WRT, OpenWRT.

  I suppose come the IPv4 Apocalypse, even ISPs will be handing out private IP space addresses behind their own monster NATs (some already do), so even routers will become inaccessible for incoming connections. Which brings me to:

• IPv6 (DO NOT LAUGH). While IPv6 is still waiting the moment for it to successfully take over from IPv4 (confidentally estimated to be the day after the United States goes metric, and coincidental with monkeys flying out of the IETF Butt-NG Working Group), it is becoming weirdly easy to build stable low-bandwidth, high reachability, IPv6 nodes on the edge. Most every Windows box from XP on can find itself a tunneled Teredo IPv6 address with one command. Miredo will get you the same with scary ease on Mac or Linux (seriously - try it!). Putting 6to4 on the aforementioned broadband routers will suddenly lurch everybody with a modern OS on your local network into the world of IPv6. Give an IPv6 address to a modern OS, and it just works. It's just the IPv6 addresses that are (ironically) so impossible to obtain.

  Miredo tunnels through a third-party like Windows or poor old remlab.net, so it's no good for insecure or high-volume data. But if you're just trying to reach the edge using standard protocols and none of that tedious mucking around with NAT traversal, it works well. Remember, we're not talking about file-sharing here, we're talking about low-volume access to Web and Web-like services.

  Let me intrude here with a single, shocking fact: as Craig notes, a lot (did you say c. 50%, Craig?) of BitTorrent clients are advertising themselves as having IPv6 addresses, even if they don't use them. It would make total sense if filesharing was what finally bootstrapped the IPv6 world.

Yeah, but what about availability? Aren't consumer last-mile connections prone to zzzzzzzt NO CARRIER?

Tomorrow! I promise!

(If you enjoy these sorts of discussions, but wish it was a little less one-sided, let me here point you to Zooko's vintage-yet-riproaring P2P-hackers list, where you can learn from people who actually understand these things.