Holy crapola - they are putting a Lua interpreter in the kernel, and it has access to some limited calls.
This is partly driven by "we need device drivers and no one understands C anymore", partly because it's cool, but mostly because Moores law is still alive and well. They are putting a scripting language in a kernel ! Forget write your app in python / perl / Ruby then optimise in C - the raw power argument is going to overwhelm us all.
I heard a stat the other day - that a greetings card, the kind that plays a silly tune, has more processing power than all computers on the planet in 1960. And we should expect a similar growth in the next 50 years. Even if that's out by two orders of magnitude just let it sink in.
I think we are at a Cambrian explosion period - where the goal is to try out every possible new body shape, as fast as possible and see which ones get the Darwin seal of approval.
New organisation forms are possible, some for the first time in human history, new ways of thinking and communicating - it's stuff like this that makes one realise the water around the frog is getting hotter.
(Still it's worth remembering that most train companies in 19C England failed and the average return for stockholders was 10% - just because the world will change beyond recognition does not mean industry stocks is a great return)
Kernel programming is an eternal struggle between people trying to pull more into the kernel for performance, and people trying to pull stuff back out for safety and modularity. We've all seen the arguments about the in-kernel HTTP daemons and whatnot. I tend to be on the side that says that more things should be pulled out.
> This is partly driven by "we need device drivers and no one understands C anymore"
I don't buy this argument. Who are the people
* Using NetBSD
* having cutting-edge hardware for which drivers don't exist
* deciding to write device drivers
* that have device specs/docs needed to write them
* that don't know/want to use C
* that would do it if only they could write it in a scripting language?
Do these people exist? I assert that the far more common case is that the device specs, documentation, BIOS, etc. aren't accessible. If that's true, they couldn't write the driver regardless of the language, C is hardly the limiting factor here
A far better rationale is that it makes fast prototyping easier. You write three lines of Lua, stuff into the kernel, and run. Did the blinkenlights blink? Yes, good, now translate that code into the C driver.
My worry in situations like that is that the `fast prototype' ends up also being the final driver, because it works well enough and nobody feels the urgent need to rewrite in C.
This is spot on. When you hear NetBSD developers arguing about proposals (or commits, ex post facto, like here), there is often mention of "users". But these are mostly fictional.
I think it's safe to say most NetBSD users are comfortable working with C.
I reckon it's not much different with OpenBSD: most changes they make are for the benefit of the developers themselves, to suit personal preferences, than for the benefit of other "users". (And that's perfectly acceptable, I think.) Of course, these changes can benefit users beyond simply those who make them and those who approve them, but those users are not the ones driving the changes.
This is spot on. When you hear NetBSD developers arguing about proposals (or commits, ex post facto, like here), there is often mention of "users". But these are mostly fictional.
I think this is true of much open source software. Just for the sake of example, a few years ago the Gnome devs removed some printer config options because they decided the "users" didn't need so many options. It's not that Gnome doesn't have users, but the developers seemed to have some notion of a type of user that didn't really exist.
I think more often than not, fictional users are used as justification for the addition of some nice-to-have feature, but of course I'm just speculating.
Novice developers should not be writing drivers in any language. If you don't have a C background, you're not going to be able to contribute much code to any OS. Besides config files and scripts, NetBSD is nearly 100% C.
If you really want to get into OS programming, I recommend you start with "Computer Architecture" by Hennessy & Patterson along with a course in C programming.
"Novice developers should not be writing drivers in any language."
Well, it might not be such a bad thing to write your stuff in Lua and get the lights blinking as a way to work your way up to C and mainline device driver development. If the device drivers interest the novice programmer, let's encourage that. Its not like there aren't a ton of safeguards between the novice and getting their driver in the mainline.
> We've all seen the arguments about the in-kernel HTTP daemons and whatnot. I tend to be on the side that says that more things should be pulled out.
I always wondered if a lot of components like this could be either sped up, or moved out of the kernel, using a zero-copy messaging system based on splice() / vmsplice() in the Linux kernel (which works by page table remapping as far as I understand). I guess the prime candidate would be device drivers.
That's one approach. Another one attempted sometimes is to go the other way and move the networking stack into userland as well. Depends on your bottlenecks etc
I've used lua in kernel space elsewhere. It links easily, it can invoke your C functions easily, and it's great for testing stuff. I could see some clever hooks being out in so you could custom tune some stuff. I bet typically they use it for testing though.
I'm not sure they would be well suited together, NetBSD has a huge list of old hardware they support; I don't think Mike Pall will be porting LuaJIT to m68k anytime soon.
Moore's law will be over by 2025 (We have 12 years or so). Semi-conductor will allow a separation of just 3 - 5 atoms to the conducting part. The belief is that quantum physics will allow the electrons to go through "walls" at that point and the switch will always be closed.
I think we will go back to the days of 640k where there is a ceiling and we will just have to refine for "large" projects. Software and not hardware will be Moore's law BUT only for a extreme circumstances. The Golden Age is coming to an end in our life time.
There were buffer overflows in the Linux kernel in 2012 ( http://www.securityfocus.com/bid/53401 ). Should we not have filesystems in the kernel either?
It's awesome to see this work finally hit the tree. Now, I cannot say I agree with all of the motivations. For example, I do not believe that prototyping should be done in kernel mode at all, nor do I believe that using Lua will shield from incorrect operation or crashes. However, there is already proof that device drivers can be written in Lua (https://github.com/SnabbCo/snabbswitch), and it's great to see more push in that direction.
After all, why use C if you have a working alternative? We just need real world experiments to see if Lua is a working alternative for the kernel.
Anybody has a link that explains their motivations? There's a lot of speculation in this thread but surely NetBSD being an open community there must have been a public discussion before they merged that in?
I write linux drivers for a living and I must say it puzzles be a bit. There are times C falls a bit short (if only by its syntax and idiosyncracies). There are times I wish I could use something like a subset of C++ (for templates and destructors mostly) or maybe Rust these days. But a scripting language?
You can already write drivers in userland (using /dev/mem for instance). You can already write device drivers any way you like that way. One of the main limitations is IRQ handling, but do they propose handling IRQs in a scripting language? Maybe I'm too close minded (or just brainwashed by years of C) but it sounds like a recipe for madness.
I actually used lua to write a small userland test driver (it was convenient because the lua interpreter is very small and easy to compile and cross-compile).
It was quite painful really. There's no builtin bitwise operator (although it's now in the standard library), all integers are stored as double floating point values (floating point in the kernel is hardly ever needed and usually very poorly supported). Using integers and bitwise operator is like all you do in a kernel driver (ok, I'm exaggerating, but not by a lot).
I guess they might have changed the syntax a little bit (and use ints instead of doubles for storing integers) but then it means their lua is not compatible with the "real" lua so you can't prototype your code in userland with any stock lua build.
I dug up my userland test driver code, here's what the naughty bits look like:
function avi_unset(addr, val)
local v = assert(devmem.read32(addr))
v = bit32.band(v, bit32.bnot(val))
avi_write(addr, v)
end
For reference here's what it'd look like in C and its bitwise operators:
void avi_unset(unsigned addr, unsigned val)
{
unsigned v = __raw_readl(addr);
v &= ~val;
__raw_writel(addr, v);
}
Regarding the lack of infix bitwise operators: usually, the functions are cached in locals. It is not as readable as C, but less bulky than what you wrote:
local band, bnot = bit32.band, bit32.bnot
function avi_unset(addr, val)
local v = assert(devmem.read32(addr))
v = band(bnot(v), val)
avi_write(addr, v)
end
If it became apparent that infix operators were needed, adding them to the VM is trivial (There's a power patch for Lua 5.1, and a patch for Lua 5.2 was recently posted in the mailing list). If you want to use the stock VM, you can still use a preprocessor (which makes things a bit more cumbersome).
I remember looking at this LUA_NUMBER but I wondered if changing it to an integer type would break something, it's not explicitly mentioned that you can use an integer type there. I'm sure it can be done but it still means that your custom lua will behave a bit differently than the "stock" lua (especially when it comes to things like roundings and overflows). That might be a bit problematic if you prototype your code on one and then attempt to run on the other.
As for your improved code I didn't know I could do that, it does look much better indeed.
Again, maybe I'm just nitpicking but my main point was that lua, scripting language or not, just didn't seem better suited than C at writing device drivers.
That being said, having a good in-kernel REPL would revolutionize kernel hacking, but it seems very hard to do right. Lots of kernel code is very timing-sensitive and you can't afford to wait for user input when you're in a critical path. Just look at the difficulty to implement proper kernel debugging to get a taste of it.
The thing is, the NetBSD people know that and more and yet they decided to go with it. I would be very interested to know their objective/motivation. I googled a bit around and found this:
The page mentions some use-cases (like power management and packet filtering) but those can already easily be handed to the userland through the sysfs and tun/tap interfaces for instance. I don't get what problem it's trying to solve.
LUA_NUMBER can safely be changed to an integer type, Lua is designed for extreme portablity from the ground up.
If by stock Lua, you mean lua.c, it is partly intended as a minimal demonstration of how you can embed Lua the library. The authors encourage users to modify the VM to better fit their use case, and changing the number type really is trivial.
> Again, maybe I'm just nitpicking but my main point was that lua, scripting language or not, just didn't seem better suited than C at writing device drivers.
As I said, I'm mostly unqualified to answer that. It's harder to shoot yourself in the foot when using Lua, which may make things easier at the prototyping stage.
It's good somebody even tries to break the old myth about C as an essential kernel part. C is _not_ good for drivers: it lacks most of the appropriate abstractions, just as facilities to create them. Once we switch from asm to C - why not go further?
Speed matters, of course, but I suppose that even script speed is enough for many drivers (and, just in case, we have luaJIT).
Don't sure if lua is a right choice, but it's an interesting experiment - and this is what NetBSD is good for.
It's just legacy. C as infrastructure is good, C as a language is bad. There's a whole industry and millions of lines of code in visual basic, but I never met somebody who says that VB is good.
Is there a way to say "do this under spinlock"?
Is there a way to handle errors not mixing it with main logic? Not wrapping nearly every line with if/goto?
Is there a way to create a generic collection? No, linux list.h is not that: it requires manual initialization, it does not work with elements, but with their special parts, it's macroses requires logically unneeded information (such as 'member' in list_for_each_entry), and so on. And this is the best you can do in C.
C is too unobvious, too. Google for "wait4 backdoor".
C works, of course. It just requires too much boilerplate code and too much programmer resources.
The industry of million lines of code exists, because UNIX and its system programming language, C, spread into the enterprise when the university students brought it from their labs into the enterprise.
Thankfully the industry is slowly moving to other languages.
Actually the best example is OS/400 mainframe system from IBM.
The kernel is a mixture of Assembly, C++ and Modula-2, with a JIT.
All user space code compiles to bytecode.
The kernel JIT compiles the applications on first execution. Afterwards the application is validated against generated native code, and re-JITed again if the bytecode has changed between executions.
> Marc Balmer comes across as some sort of Lua zealot. Lua is a great little language but c'mon. He is like a broken record. Why does he want this so badly?
Quoting bullet points from his FOSDEM 2013 slides:
• Modifying software written in C is hard for users
• Give users the power to modify and extend the system
• Let users explore the system in an easy way
• "Rapid Application Development" approach to driver/kernel development
Maybe this argument is less useful for BSD users, but wouldn't most users be using the stock kernel? So it's up to the whims of the distributors whether to include it or not.
It's pretty common in the BSD world to build your own kernel. The entire build system, imo, is much more streamlined and well thought out than linux, so relying on the distributor is not needed. Also, one of the missions of NetBSD is to get it working on everything, so as others have stated, being able to poke at the kernel in lua, both for developing drivers, and for pulling state out of the kernel for general debugging, is very useful.
I don't think these things (Lua zealot, above points) are mutually exclusive.
I think this is mostly for prototyping though. I can't imagine running this in a production system. If it fosters exploration (via Lua, or the Lua interface to the kernel), it's a net win (no pun intended).
This is partly driven by "we need device drivers and no one understands C anymore", partly because it's cool, but mostly because Moores law is still alive and well. They are putting a scripting language in a kernel ! Forget write your app in python / perl / Ruby then optimise in C - the raw power argument is going to overwhelm us all.
I heard a stat the other day - that a greetings card, the kind that plays a silly tune, has more processing power than all computers on the planet in 1960. And we should expect a similar growth in the next 50 years. Even if that's out by two orders of magnitude just let it sink in.
I think we are at a Cambrian explosion period - where the goal is to try out every possible new body shape, as fast as possible and see which ones get the Darwin seal of approval.
New organisation forms are possible, some for the first time in human history, new ways of thinking and communicating - it's stuff like this that makes one realise the water around the frog is getting hotter.
(Still it's worth remembering that most train companies in 19C England failed and the average return for stockholders was 10% - just because the world will change beyond recognition does not mean industry stocks is a great return)