Greg Kroah-Hartman sez: we don’t have a development kernel anymore. He’s a kernel developer for a long time, a kernel maintainer, PCI, USB, drivercore, sysfs… “lotta crap like that”.
Pre-Bitkeeper, this is how they did it: maintainer sends patches to Linus. Wait. Resend if patches dropped. Lather, rinse, repeat.
January 2002 – patch penguin lkml flame fest. Worried about patchs getting dropped. Linus decided to use BitKeeper, 2.5.3 first kernel. “License weenies got their panties in bind, ah feh.” (Greg is like the Vin Diesel of kernel dev: I keep expecting him to shout “I! LOVE! THIS! DEVELOPMENT CYCLE!!!”.)
BitKeeper change things a lot. Bitkeeper maintainers got a lot more work. Linus sucks in all patches, discarded crappy stuff. Non-bk maintainers just sent patches as it goes.
Unexpected consequences: we knew what Linus was doing, so much better feedback. Bk2cvs and bk2svn trees. bkcommits-head mailing list so you could you see what was flying around.
All patches started to go through Andrew Morton; BK stuff goes through Linus.
Result: 2.6.0 = Two years of dev, 27149 different patches, 1.66 patches per hour. At least 916 unique contributors. Top developers handled 6956 patches. Ten patches a day for two years. People should research this data.
2.6.1 538 patches. 2.6.2 = 1472, 2.6.3 = 863 patches, 2.6.4 1385 patches…. keeps going. bY 2.6.7 2306 patches. Something is wrong. This is the stable kernel, but we’re going faster — one million lines added, 700 thousand lines deleted: a third of the kernel. A *lot* of data, and this is *stable*. Feeling uncomfortable yet?
All patches end up in the -mm tree. All bk trees end up iin the -mm tree. 17 different trees, acpi, agpgart, alsa, … So we have a staging area.
And 2.6.7 is the most stable kernel we’ve ever had. So we’re doing something weird, but right.
The -mm tree *is* the development kernel. Patches can go in, and if they suck, they can get out really quickly. Linus gets the good stuff, the -mm tree is the tester’s tree.
Almost everything is tested in the -mm tree before it goes to Linus.
Near future: Linus releases 2.6.x, maintainers flood him with patches, all have proved their stability in the -mm tree. We all recover, fix bugs. One month later, a new 2.6 is released.
Will 2.7 ever happen them? Maybe, if we have big intrusive patches — page clustering, timer core rewrite, stuff that touches everything. Linus’ll fork 2.7, all 2.6 changes will be merged into 2.7 daily. If 2.7 is unworkable, we delete it (so save your 2.7 patches). If 2.7 works, we merge to 2.6, call it 2.8.
Summary: the kernel developers all like how this is working. No stable internal-kernel API, never going to happen, get used to it (syscalls won’t break). Drivers outside of the tree are on their own (quote: “you’re screwed”) (conclusion: get into the kernel, quick before it runs crazily away in a big honking yellow bus of ever-accelerating development.) Proprietary people: if you’re not in the kernel, Greg says he doesn’t know you exist. And if you don’t give back to the community, he doesn’t care, either. (“Just my opinion”)
Everything subject to change at any time. These days, Linus is running the stable tree, -mm is running the unstable tree. When they realise this, they might change their mind about how things are doing.
Q&A: ISV like Oracle are going to go crazy, because they need to catch up with Greg and the kernel developers. They can’t just certify an ancient kernel and stick with that anymore.
Q&A: We don’t want the distros to fork, as happened when SUSE and Redhat settled on differenly patched 2.4.
Q&A: What about automated testing? Big companies have a test suite – hopefully, they’ll start running them every night. Novell has a giant test deparment, and they’re ramping up to do nightly test runs. If they can get that under 24 hours, we have a big honking regression test.
Q&A: (from me) What are the limits on the speed of kernel development now? “We’re going so fast, there’s a huge rate of change. Andrew seems to scale pretty well; I’m maxed out; more people coming in. We’ll find out what the limits are.”
Q&A: How many people working on Kernel at IBM? Answer: don’t know, Greg emphasises that he’s only speaking for himself.
Q&A: When are you going to fix broken stuff, like Firewire? “Well, it should build.” (heckle: is that your testsuite then?) “Yes. We can’t test drivers. You need to write them themselves.”
Q&A: The problem with automated tests for drivers is that it needs hardware. Who is going to pay for that? “I’ve written drivers that I don’t have hardware for; I know someone who has debugged joystick drivers via Irc between Czechoslavakia and Israel. ‘Push the joystick left? Does it work?'” Some stuff isn’t easily testable. We need to fix the infrastructure. OSDL trying to fix this.
Q&A: How can Andrew and Linus do anything but rubber-stamp thousands of patches. Answer: we’re not writing much code these days. It’s a trust network — I trust people who send us stuff. And now we have a blame network, too, so if somebody sends me something that breaks, I know whose fault it is.
Q&A: Is there still a problem with tracing origin of code? With BK, I can resolve all lines to a email address and a name. The lawyers I speak to say that’s okay. All patches get a “signed off by: name and email”. That’s a little more explicit. Not legally binding. Once again, it’s a blame issue. I’ve seen patches flow through five people. More people to blame!
Somebody was sending me code on plug and play, and it just seemed too good: conformed with all the Microsoft specs, worked really well, made me very suspicious. So I made him prove it, send me all the places that he found it the stuff. And I was finally happy with the provenance. Later it turns out he was seventeen years old. You can’t tell where you’re going to get excellent code.
Oh, btw: Greg says there’s a really good piece about this switch in the latest issue of Linux Weekly News. I don’t have a subscription to LWN, for ridiculous personal reasons (I feel odd using paid-for info to compile NTK. Same thing why I don’t like taking journalist freebies: I believe I will be struck by lightning if I do. Previous experious indicates that belief to be justified.) But if I wasn’t so weirdly idiosyncratic, I’d pay their sub rates in a shot.