I'm concerned about the gradual move from GCC to LLVM. The lack of copyleft protections on LLVM means that it's much more dependent on corporate sponsorship, and means that there's a risk that major improvements to LLVM compilers will only become available as proprietary products.
People underestimate the role of copyleft licenses in preserving long-running FOSS products like GCC, Linux, etc.
> I'm concerned about the gradual move from GCC to LLVM. The lack of copyleft protections on LLVM means that it's much more dependent on corporate sponsorship, and means that there's a risk that major improvements to LLVM compilers will only become available as proprietary products.
As someone who works within LLVM professionally, I don't think this is particularly likely -- compilers are massive and complicated beasts, and the "moat" for proprietary improvements is small: they're either niche and therefore not of interest to the majority of programmers, or they're sufficiently general and easiest to maintain by releasing back upstream (where Apple, Google, and Microsoft will pay a small army of compiler engineers to keep them working).
Your concern is the one that kept GCC from stabilizing its various intermediate representations for decades, which is why virtually all program analysis research happens in LLVM these days.
Edit: To elaborate on the above: neither Apple, nor Google, nor Microsoft wants to individually maintain LLVM. Microsoft appears (to this outsider) to be actively looking to replace (parts of) MSVC/cl with the LLVM ecosystem, because they're tired on maintaining their own optimizing compiler.
That is a weird way to say "your concerns were entirely correct". Every single proprietary mobile graphics driver uses LLVM for its shader compiler and the lack of copyleft has severely curtailed progress on open-source drivers.
Of course none of it goes back to LLVM because updating your production copy of LLVM is just as messy and broken as the non-stable GCC IR you complain about and the fact that OSS development is still fundamentally incompatible with 1-year industry release schedules that shift all the time.
> Every single proprietary mobile graphics driver uses LLVM for its shader compiler and the lack of copyleft has severely curtailed progress on open-source drivers.
And you think if they didn't have the option of using LLVM they would have released an open source driver instead? That makes no sense to me.
Yes. Developing a state of the art optimizing compiler is hard. If their only choices are developing their own compiler or basing off of gcc and releasing their work, they'd choose gcc.
> If their only choices are developing their own compiler or basing off of gcc and releasing their work, they'd choose gcc.
But that's the exact choice Apple faced in 2005 and they did not choose gcc. They paid Chris Lattner and his team to develop an alternative compiler. Excerpt from wikipedia[1]:
>"Finally, GCC is licensed under the terms of GNU General Public License (GPL) version 3, which requires developers who distribute extensions for, or modified versions of, GCC to make their source code available, whereas LLVM has a BSD-like license which does not have such a requirement.
>Apple chose to develop a new compiler front end from scratch, supporting C, Objective-C and C++. This "clang" project was open-sourced in July 2007."
For some reason, the enthusiastic focus on the benefits of GPL principles seems to ignore the actual game theoretic behavior of actors in the real world to not choose GPL at all.
Sure, if you have the resources of Apple, you can make other choices. I'd be willing to bet that Apple has spent more than $1B on clang, llvm, tooling, testing, integration, et cetera.
That's the wrong question. How much resources does someone have who decides to write an entirely new compiler when a working one already exists?
Of course the answer then reveals that this was only possible because there was comfy GCC to fall back on all along. They started in 2005, Clang became default in XCode with the 4.2 release in October 2011.
Ask yourself if you can sell your manager on 6 years of effort for no functional difference, likely even inferior.
Nah, they license a proprietary one. There are a whole bunch of them (ICC, Digital Mars, PGI before they were acquired by Nvidia...). They generally aren't as good as GCC, but they keep your code private and copy-left licenses away from private codebases.
I don't consider the improvements that ARM, Apple, Sony, CodePlay, NVidia, AMD, among others, don't upstream due to IP considerations or revealing of hardware secrets, niche.
But if they have IP considerations or trade secrets to protect, they wouldn't contribute them to any other open source project either. This way, they can at least upstream everything, which doesn't fall under these restrictions.
> They don't upstream everything, while reducing their development costs, that is the point.
_ph_ meant to write "This way, they can at least upstream everything which doesn't fall under these restrictions." (note the deleted comma before "which"), i.e., they can at least upstream something instead of nothing.
They don't upstream what they never would be up streamed. But they probably they upstream a lot which they wouldn't have, if they couldn't use the open source tool at all. Also, a lot of companies reduce their development costs by using open source software, gpl or other, without ever upstreaming. So "saving development cost" doesn't sound like an argument to me.
With GPL there is a legal tool to force contribution, though.
As mentioned I don't care, after university my main UNIX platforms were HP-UX, Aix and Solaris with their respective system compilers anyway.
Linux is already getting replacement candidates in IoT space via Zephyr, NuttX, mbed, RTOS, Azure RTOS, and who knows if Fuchsia will eventually get out of the lab, so be it.
With GPL there is a legal tool to force contribution, though.
No, for practical purposes, there is not. You can only enforce delivery of the source when a product based on GPL software gets delivered. But of course companies, which create software, know about this. Any usage of GPL software for delivered products only happens after the decision to publish the created software has been made. In doubt, companies tend to not use GPL software as part of deliveries.
In practice companies are more likely to upstream non-gpl not less. With GPL they decide to use it without allowing any change, or they don't use it at all. With more liberal licenses they upstream anything that isn't core value because the cost of maintaining a fork is higher the more they are different.
I would be happily using HP-UX, Solaris and Aix to this day, so don't mistake my philosophical questions as a GPL advocate, just don't be surprised how the IT landscape will look like when GCC and Linux are no longer around.
On the age of cloud based runtimes and OS agnostic containers, Linux is an implementation detail of the server.
On Android Linux is an implementation detail, most of it isn't exposed to userspace, not even on the NDK as Linux APIs aren't part of the stable interface specification.
Upstreaming non-GPL code is much easier for these companies to upstream GPL code, because a GPL upstream can "accidentally" leak all the IP of that company by requiring that company to publish all of its software.
So the current options are: these companies don't open-source anything (which is what you get with GPL), or they open-source something (which is what you get with BSD, and in practice they open source a lot).
The claim that with the GPL they would open source _everything_ isn't true; they would just not open source anything instead. I also don't see any current legal framework that would allow them to opens ource everything without loosing a significant competitive advantage, and am very skeptical that such a framework can be conceived.
> On the contrary, the claim is that without the GPL, commercial UNIXes would still be around.
I'm not sure this is true. See LLVM as an example. GCC existing and being GPL only meant that, e.g., Apple couldn't properly use it. Apple could have bought LLVM and kept it private, or develop their own proprietary solution and not make it open source, or fork the open source project into a private project and never contribute anything back, etc. There were many options.
Open source software has always existed, if anything, the GPL demonstrated that a particular open source model does not work well for a big part of the industry, while it works well for other parts, and non industrial usage.
Linux being successful seems incidental to it being GPL'ed, at least to me. Other open source OSes, like BSD 4.x, have also been quite successfull (powering the whole MacOS and iOS ecosystems, after a significant frankenstransform into Mach). Maybe Linux would have been even more succesfull with a BSD license, or maybe it would be dead.
Exactly because I see LLVM as an example, given that not everyone using it contributes 100% back upstream.
I wouldn't consider BSD layer from NeXTSTEP an example of success for BSD's market adoption, given that not everything goes upstream, by now it hardly represents the current state of affairs.
If anything it represents what would have happened without Linux, all major UNIX vendors would continue to take pieces of BSD and not necessarily contribute anything back.
> I wouldn't consider BSD layer from NeXTSTEP an example of success for BSD's market adoption, given that not everything goes upstream, by now it hardly represents the current state of affairs.
Its essentially what happens if (1) only one company wants to use the open source product, and (2) the open source product has a tiny community where no development happens.
In that scenario, there is no benefit from anybody forking the open source project (a private company or an individual) for upstreaming anything. It just costs time, but adds no value for them.
Linux and GCC never was like this (not even early days), and I think this is independent of its license. LLVM never was like this either.
In fact, there are many private companies that maintain a GCC fork, like arm, and due to the GPL need to provide its sources with a copy, which they do. But they are not required to reintegrate anything upstream, which they very often don't, and ARM support in gcc-arm from ARM is much better than on GCC upstream. A volunteer can't merge (review, rebase, ...) a 200k LOC patch on their free time, so once these forks diverge, its essentially game over. You'll need a team of volunteers equal in man power to what the private company provides.
So while the license affects which companies can use an open source project in practice, and what they can or cannot contribute. The GPL2 and GPL3 licenses are not a good tool to actually allow everybody to benefit from those contributions.
Maybe a GPL4 could require companies to upstream and get their contributions accepted, but IMO that would just get even less companies to use those projects.
1/3 of the us internet traffic, not counting Juniper, then add all PS2/3/4 Consoles, MacOS's, FreeNas, EMC-San's:
Biggest German Online Seller, Sony Japan, Checkpoint and so on...search for yourself.
BTW: Market-share means nothing, you know Android is NOT Gnu/Linux
EDIT: Maybe your too young..but do you remember that SCO/Microsoft thingy with Linux
They did not...the opposite is true, they shat their pants because shortly before they said "yes" to Linux, that's why SCO came...first time you had really big money (IBM) behind Linux, please don't change the timeline...it's kind of important.
>And as information, during the early days, the Internet ran on commercial UNIXes.
What do you wanna say with that? During the early days of Smartphones they ran on commercial OS's??
What if those bigcorps don't merge some improvements for getting an edge over the competitors? And, what if every one of those bigcorps do the same? How would that impact the project in the future?
They pay the costs of maintaining their fork. Thus there will be constant internal chatter of is the non-merged stuff really valuable enough considering the cost of maintenance. Sometimes yes, but sometimes no as well. So overall that is a long term win.
> or they're sufficiently general and easiest to maintain by releasing back upstream
Exactly; maintaining a fork is a big pain. The ongoing cost of keeping it up-to-date is a pretty big incentive to merge it, even if there's not a legal requirement.
But Intel put proprietary work in ICC and their math libs, so there is something on periphery that’s going to keep the worry alive even if it doesn’t have a base.
Well, to be clear, both ICC and MKL are fully proprietary. They do indeed justify the worry, but they're not in the same "open source but may be susceptible to proprietary creep" category.
> they're sufficiently general and easiest to maintain by releasing back upstream (where Apple, Google, and Microsoft will pay a small army of compiler engineers to keep them working).
Are you under the impression that C and C++ code within Apple, Microsoft, and Google is fundamentally different from C and C++ code elsewhere? Because it isn’t. Google’s engineers maintain LLVM’s ASan for their purposes, but those purposes happen to be everybody else’s as well.
It may not be fundamentally different but it may only use a subset of features (e.g., no exceptions in C++ code) and/or target a subset of platforms. Without pointing fingers, I am sitting on (what looks to be) a codegen bug in Clang that is not of a high priority I am pretty sure due to the above reasons.
This is not to say that I am not appreciative of all the work Google/Apple/etc engineers do in LLVM (I will be eternally grateful for the Clang targeting MSVC work).
Right, the reality is that if you want to be a profitable tech company today, you have to leverage open source. "Their use cases" includes, like, everything in a Linux distro. (And yes, this applies to Apple and Microsoft as well as Google.)
The necessity of open source for industry has both negative and positive implications for those of us who care about open source / free software as an ideal and not simply a tool of capitalism. The negative one (which GCC's leadership failed to really internalize) is that the number of engineer-hours at the command of for-profit companies is much higher than the number of engineer-hours at the command of community-driven projects. If you deliberately build a worse product to prevent corporations from using it for profit, given enough time, the corporations will replace it. The positive one, though, is that those engineer-hours are generally more valuable when pointed at some common cross-company codebase unless it is the specific thing that makes you money, and FOSS as an ideal provides a well-accepted model under which they can organize cross-company work (doubly so when they employ idealists like us as implementors). A compiler makes very few people money directly. It's generally a tool that you want to work well, and it's helpful to have other people run into the majority of problems and fix them before they cause trouble for the things that do make money.
So it's not surprising that LLVM is catching up with GCC, nor is it surprising that LLVM is and remains open source. If you are concerned about the LLVM monoculture, build a competitor such that it is cheaper / more profitable for companies to work on your competitor than to either work on LLVM or build their own compiler. Figure out what will make companies want to contribute and encourage it. (GCC did not do this, but it is perhaps slowly relaxing here.) If you are concerned about LLVM becoming proprietary, make it so that is cheaper / more profitable for companies to release their changes instead of holding onto them; that is, figure out what will make companies feel like they must contribute and encourage it. (One common strategy, used by Linux itself, is to maintain a high level of internal API churn coupled with genuinely good improvements in that churn and a policy that people must update in-tree callers; at that point, the more API surface you use in your private fork, the farther behind you get, and you'll watch competing companies outpace you.)
> One common strategy, used by Linux itself, is to maintain a high level of internal API churn coupled with genuinely good improvements in that churn
Interesting angle that I had never thought of as deliberate. As someone who works on a (bespoke) integration/embedding of Chromium, I could say exactly the same thing about it too.
It's not churn for churn's sake. But having a policy of having drivers in-tree and explicitly not caring about out-of-tree stuff allows them a relatively free hand to improve the internals. Which is then seen as churn by out-of-tree code.
Meaning, internal things in chromium change so fast, so you'd slightly wish you could have your custom changes merged into upstream? (I.e into chromium)
So that the ppl at Google would keep your code working? And you didn't need to spend time resolving git merge conflicts, and compilation errors?
But you cannot, because some of those bespoke changes are "secret" and what you make money from? (And maybe some changes are off topic to Google)
I wonder how much time does it take to merge a new chromium version into your repo? Like, hours? Days? Weeks
Also, the fact that Chromium is a high profile security critical software with occasional emergency security updates for bugs exploited in the wild, doesn't help at all when you want to maintain your own fork.
Personally I see this as a good counterargument for "just fork it." Imagine if chrome decided to crank up churn with the intention to exhaust forks. Maybe all forks would join together to fight google or more likely they would just fall behind.
That makes me sad as well, but I also think it was largely self-inflicted. GCC took much too long to realize that people needed a common code-generation framework, and be willing to support that and work towards it. LLVM was designed from the start to be such a framework.
This seems all too common in the software industry. A company creates a product, but after a while, ignores the changing needs of the market until it’s too late.
I don't see how, personally. There are tons of thriving open source non-copyleft projects. And there are tons of GPL violators who go on business as usual. I think perhaps copyleft licenses are overattributed to the success of such projects.
Most of the thriving non-GPL OSS projects are supported and funded by large corporate sponsors--not by communities or foundations (e.g. React.js by Facebook, Blink by Chrome).
Furthermore, many non-GPL OSS projects have eventually gone proprietary (e.g. MongoDB's switch from AGPL to SSPL).
I think this gets the cause and effect backwards. No corporate lawyer prefers the GPL, so when companies start new open source projects, they don’t pick the GPL. Once a project has been started, GPLing it doesn’t really make it any easier to get it non-corporate support.
> Furthermore, many non-GPL OSS projects have eventually gone proprietary
Every time it happens people are upset and then forget everything a day later. To me non-GPL + corporate involved is a huge flag, especially if it's a more complex or niche project that'd be difficult to fork.
Those GPL'd(eg. GCC, Linux) projects predate their current corporate sponsors. The sponsors may go away anytime if they feel the project won't add any business value for them. I highly suspect if they would sponsor support any new GPL project.
But the direction of Linux as a GPL project is largely controlled by corporations e.g. Redhat now a.k.a. IBM, Intel, etc. GPL is not a cure all, it is a tool. Some licenses are better for some purposes. Use the right tool for the right job. It is nice that we have compilers (and OSes) under both licenses, it shows both strengths and weaknesses of each type of license. There are strengths, and weaknesses, to both models, and to other models, such as Intel's optimizing compiler.
Half the linux foundation's funding comes from 6 companies. Not to mention the full time employees paid to work on it. And I find that there's similar patterns in a lot of other widely used gpl software. Similarly, MIT/Apache software gets tons of contributions from non-corp people too.
I think GCC’s resistance to modularisation and the resulting rise of LLVM has proven beyond doubt that the greater risk is that by putting up barriers to corporate contribution there will not be any major improvements in first place.
LLVM is just a proprietary-able version of GCC. And thus compiler authors mostly focus on benchmarks while not caring about other metrics such as compilation speed, correctness and codebase quality / ease of contribution. There appears to be some tribal knowledge requirement to add a new target.
With all the hype and apple funding LLVM gets, it could have been better.
It isn't just that. It's also a compiler framework, somewhat usable as a library without being part of its codebase. And I really do mean somewhat usable; the LLVM experience for use as a library is not great, but it's wildly better than GCC's. I think that, not just the license, is a big part of what made LLVM successful.
Developing a GCC frontend is a massive PITA, and the worst of all it was made purposefully so complicated because of RMS' concerns about closed compilers. He actually brought LLVM's existence onto himself, because it was somewhat clear no one would have bothered writing a new compiler if it had been possible to use it's frontend for other things (even from free software it's hard) and they kept the license as "GPLv2 or later" instead of pushing for v3 that arguably makes 0 sense with GCC (it's not like it's going to be put in a closed box any time soon).
Sometimes in life it's better to be a bit heterodox, but have influence and leverage, than being alone on your high castle. GCC can be as open as it gets, but pushing people away in the name of freedom has actually only given reasons to the industry to move away from it, which is sad and could be avoided.
The fear of GPLv3 is just an excuse to pull things into proprietary. They became aware of the success of GPL and started to actively lobby against it with what they were able to come up with.
Sometimes in order to achieve your goals you must compromise or risk loosing the footing you already have, because you have close to zero chances of succeeding.
If RMS had compromised on GCC in the '00s, i.e. if GNU had spun off the C/C++/ObjC front end as a separate GPLv2-or-later library upon which something like clangd could have been made, Clang would have never existed. Yes, LLVM would have still been present, but as a special-case backend that used GCC as its frontend instead than its own, as they were planning to do since the beginning. All these improvements you talk about would have been done under the GPL, and not BSD licenses or proprietary. IDEs like Xcode and such would still be proprietary like they are nowadays, because there was a 0% chance of getting Apple or whomever to release them under the GPL.
It doesn't matter how much strong your moral principles are, or how much you value integrity. The world is definitely more pragmatic about software and values different things than RMS; while this might or might not be beneficial to our overall society, that's the way it is. Ignoring it is myopic, if it does not outright amount to shooting yourself in the foot.
The change also means libgcc is now GPLv3 as well, making licensing for anything embedded that uses glibc (which has a hard dependency on libgcc_s) a royal PITA to figure out since GCC’s linking exception. Okay, so a proprietary program can link against libgcc without coming afoul of the GPL, but do we still have to make libgcc_s itself replaceable according the the anti-tivo clause?
The FSF is shooting themselves in the foot with the way they handle many of their projects today, GCC is no exception. Free software purity is a great goal and all, but what use is it when nobody uses it or it falls behind more permissively licenses projects.
The point is, these things would have happened no matter what. RMS simply overplayed its leverage, and the magic that kept GCC at the centre of everything for decades broke.
If he could just stop being a zealot for a split second and actually tried to understand the situation, he could have maybe been able to foresee that Apple had the people, resources and will to reimplement a whole compiler infrastructure that could threaten GCC's dominance, but he neglected it (it famously didn't care about Apple's offer to merge LLVM into GCC under the GPL).
Sometimes even the best of intents are shadowed by people's inability to compromise.
All of those frontends live inside the GCC source tree. Which requires copyright assignment to contribute to. And for a long time, it was difficult to "casually contribute" to GCC; you had to work towards "commit access", and it wasn't anyone's job to review other people's patches, so often it just didn't happen. And there are various other barriers to contribution as well.
LLVM's license was certainly a factor, and I'd never suggest otherwise, but there were many other reasons that GCC couldn't help kick off a wave of language design and experimentation the way LLVM did.
Sure, you could also fork all of GCC as part of your frontend. That's still a massive undertaking compared to a framework that's designed to be used as a library.
Having contributed to both (nothing major but not trivial one-liners either), my feeling is GCC definitely has a higher quality codebase and a more rigorous review process. While the latter doesn't always translate to the ease of contribution, I found that if you are persistent enough, your patches to GCC will be reviewed. On the other hand, I had my patches to LLVM ignored forever. Just my 2c based on experience.
A GCC back-end for Rust is incredibly important for FLOSS. Even if the world doesn't switch from C to Rust overnight, Rust creates a negative halo effect for GCC, casting doubt on GCC's future relevance. Rust support in GCC would be a huge win for both projects.
All the bits for the core of the system are open, it’s just not very exciting without the proprietary bits like WindowServer, AppKit, etc. - all of which would have been equally easy to keep closed if much of Darwin was based on GPL’d projects.
> All the bits for the core of the system are open
No, some bits are open, and some bits track what's actually being shipped. Much of what is open sourced lags behind what is running on end user machines by several years. iOS modifications to XNU and Darwin were never made open source.
Apple is also dropping GPL internals in favor of open source but not copyleft licenses so that they don't have to release source code changes to the software they release.
Although this is probably not the best example because this is sold by the company that largely funds the open source project. Although that is a big conflict of interest.
Yes, we are short of getting back into the days of Shareware and PD, and then the anti-GPL crowd will be happy with the outcome.
As commercial software user, I don't have much issue with it, after all I have been coding since the early 80's.
And in spite of it and my occasional Linux rants, I am thankful for GPL, because without it there wouldn't be a UNIX to install at home to get my university work from DG/UX done without having to spend one hour traveling into the campus and fighting for a terminal.
Without Linux + GNU based userspace, the commercial UNIXes would all be around.
LMAO none of the BSD's have not been superior to Linux in over 20 years. Linux has been faster, more stable and more secure then any of the *BSD's for over a decade now.
GCC changed from GPLv2 to GPLv3 which is a stupid license for a compiler (Thanks FSF). I'm happy that LLVM exists, happy that Microsoft, Apple, Linux, and BSD's can use it without that GPLv3 BS.
That would eliminate the need for Rust. Just imagine, if your C code can be converted into Rust, would there ever be a case to start a fresh project in Rust? It will permanently become an Intermediate language.
How would that make things any easier? The kernel supports being built with clang now, and there’s no issue linking an object built with another compiler into a module either.
Even if it is possible to build the kernel with clang, presumably that doesn't mean it is acceptable to suddenly require the kernel to be built with clang?
One of the challenges to building an entirely new kernel is the vast amount of hardware support in the form of drivers and the features the existing kernel exports to userland in the form of syscalls.
Would it be possible for a hypothetical new kernel, presumably written in Rust, to run an existing kernel such as Linux in a VM, just to tap into its drivers and emulate its syscalls? As more drivers are ported to the base kernel, the reliance on the donor kernel would shrink over time.
Edit: This was an aside. Of course I realize that the conference was about adding Rust code to the Linux kernel, and not about rewriting any large, important, and functional codebase in language-of-the-day™. Hence my speculative language: "hypothetical" new kernel, "presumably" written in Rust.
The point of this LPC session was how to incrementally introduce Rust in the existing Linux kernel, with all its existing drivers and syscalls and similar. A rewrite would indeed be a daunting and problematic proposition.
There are kernels written in Rust, such as Redox, but those are separate projects, and that's not what this conference session or article are talking about.
Standing reminder: the Rust project is not a fan of rabid Rust over-evangelism (e.g. "Rewrite It In Rust", "Rust Evangelism Strike Force"), and sees it as damaging and unhelpful. We discourage that kind of thing wherever we see it. We're much more in favor of a measured, cautious approach.
Man, thank you for writing the last paragraph. I looked at Rust, I quite like the language, I think it gets a million things right and is a substantial improvement over existing languages, but ... every time I point out that Rust does not magically solve all security problems, I need to spend two days dealing with people with less security expertise than me yelling at me that I am wrong and rewriting the world in Rust will deliver enlightenment. That particular fringe of the Rust community almost turned me away from the language.
So reading your last paragraph makes me feel welcome as someone that says "Rust is great - but rewriting everything in Rust will not solve security". Thanks. <3
Unless the OP changed his comment, this is an overreaction to a harmless, hypothetical, technical question. I've seen a lot of annoying Rust evangelism, but this ain't it.
BTW, I'm not a Rust dev, and it would appear the OP isn't a Rust developer either. They don't even seem aware of Redox. Seems odd to accuse them of "Rust evangelism".
I was not accusing the OP or anyone here of such evangelism; rather, because the OP was talking about writing/rewriting a kernel in Rust, I was stating that we weren't looking to "rewrite it in Rust" or similar. We've seen a lot of people suggesting that the work on putting Rust in Linux was trying to "rewrite the kernel", and it seemed appropriate to provide such a disclaimer.
> Standing reminder: the Rust project is not a fan of rabid Rust over-evangelism (e.g. "Rewrite It In Rust", "Rust Evangelism Strike Force"), and sees it as damaging and unhelpful.
Agreed. If you want to rewrite everything in Rust, then please help out with a project such as Redox where the goal is indeed to write everything in Rust. But don't bother people by talking about that goal, just write some code!
> the Rust project is not a fan of rabid Rust over-evangelism (e.g. "Rewrite It In Rust", "Rust Evangelism Strike Force"), and sees it as damaging and unhelpful.
I was really caught off guard by this stuff recently. I was around the rust subreddit a lot back in 2013-2015 and never saw stuff like that. Stepped out while working on other stuff and now it seems common. Super strange.
As an occasional rust programmer: you don't see this kind of behavior from the rust community. You see it when a random person opens an issue on a bugtracker you are subscribed to. Then it's usually prompty closed by a maintainer because the issue is very low effort, and the author doesn't offer to do any work themselves.
Re-reading my comment I realize "you don't see this kind of behavior from the rust community" was unclear. I meant that people who are part of rust community don't see it, because it happens outside.
I was thinking about something similar recently - if the hardware drivers could somehow be abstracted from the rest of the linux kernel (as if...), then suddenly a ton of experimental kernels could have widespread hardware availability.
It would probably shake up the OS ecosystem to no small degree, but I imagine the fresh ideas that could be experimented with by non-experts would be hugely beneficial. Isn't that the big argument against so many new projects? "It looks cool, but the hardware support isn't there."
FreeBSD might be a better target though, since that's specifically designed to be compatible with everything.
ESX once ran this way, it had native drivers and Linux drivers, where it would run Linux as a vm and have it handle drivers for unsupported hardware. theres some more documentation on Wikipedia[1], it later switched to using vmklinux as a shim for those drivers.
The idea of that project was to turn Linux into a user mode application run by a microkernel; rewriting that microkernel in Rust would essentially give you what you're suggesting.
this is a very old idea. ndiswrapper did this for network drivers back in 2003, and freebsd implements shim layers for linux graphics drivers, linux applications, some solaris kernel modules (dtrace, firewall, others?). the main problem is that nowadays, modern simple hardware has mostly standardized on a fairly small set of APIs (e.g. SATA/SCSI/NVMe for the majority of disks, UVC for the majority of cameras), and modern complex hardware requires complex shims. see: the amount of work put into Linux graphics kernel APIs.
> "One of the challenges to building an entirely new kernel is the vast amount of hardware support in the form of drivers [..]"
This reminds me of an interview with Linus Torvalds from many many years ago, when he was still very young. The interviewer asked him if he was afraid of getting replaced by someone young and hungry. Torvalds shrugged it of with something along the lines of: Nah, no one likes to do driver development.
I might be wrong, but isn't there already a precedent for something similar in the form of the XNU kernel? From a cursory glance, it is basically a BSD and a Mach kernel running side by side.
I'm not sure why you'd need a VM for this. It should be much easier to implement bridges for major kernel apis like character devices, as I understand some BSDs already do?
> The ubiquitous kmalloc() function, for instance, is defined as __always_inline, meaning that it is inlined into all of its callers and no kmalloc() symbol exists in the kernel symbol table for Rust to link against. This problem can be easily worked around — one can define a kmalloc_for_rust() symbol containing an un-inlined version but performing these workarounds by hand would result in a large amount of manual work and duplicated code.
That's why there exists crates like https://docs.rs/cpp/ which allow to embed C++ (and thus C) directly into the rust source code, simplifying the manual work and reducing the duplicated code.
Yup - that got a very brief mention in the article as "Google is working on automated C++ conversions"; there's also https://github.com/google/autocxx which builds on top of it.
C++ is a richer language than C and has more information about ownership (e.g., if you return a std::string you know how ownership works; if you have two arguments that are a char * and a length it's much less clear), but additional annotations in the kernel's C headers to convey this level of information in a machine-parseable way would be awesome.
Sounds like they have the same problem Apple's Swift does for calling into C/Obj-C.
Two examples: My understanding is that there's a ton of code and added complexity in Swift itself to support Obj-C interoperation. And while you can call C and Obj-C APIs, as-is, there was/is basically a company-wide effort to write API wrappers (aka. overlays) for existing system APIs.
For the second part, maybe if Rust for Linux kernel programming gets to the level of popularity as, say, TypeScript in the JavaScript community, the Linux community will end up creating the needed API wrappers as an organic, group effort.
This is mostly just a random tidbit but one of the fascinating things to me about Swift <-> ObjC interop is that it's not just that Swift was modified for ObjC, but ObjC was modified for Swift as well. In general it works pretty well and I wonder how much of that is because Apple can modify ObjC whenever it wants.
.Net is a much simpler integration, all of the support is in the CLR because COM defines a limited ABI that can be called from any language that can walk a few pointers in a C++ vtable given an interface definition.
One can even completely ignore the RCW/CCW support in .Net entirely and just hack together some struct types that match the vtable layout.
The WinRT ABI does implement some new features, yes - but they’re all still implementable with nothing more than standard C# code with no special support from WinRT itself (in fact, the WinRT ABI integration is being stripped out of the runtime and cswinrt is the way forward).
C++/CLI ultimately still uses the same features the runtime uses for p/invoke calls, though it uses slightly different IL to transition to native code since it doesn’t have to worry about an external ABI within a mixed mode assembly. Sure, there’s a whole compiler that supports this weird mixed mode world from a language perspective - but it’s really boring at runtime.
All of that doesn't change the fact that there is tooling in place, and language features (e.g. dynamic for COM, upcoming C#9 improved native FFI) to ease interoperability between managed and some level of C++.
Something that no UNIX does, and we only find similar ideas in Swift/Objective-C++, IBM language environments, and the old OS/2 SOM (Smalltalk/C++).
While everyone else just writes glue code with C like interfaces, as if the world has hardly changed from UNIX V6.
Purists may hate it, but as a toy project I’ve been working on making a stripped-down version of the WinRT concepts that runs on Linux (well, anything with a c++17 compiler and a dlopen function provided by a linker). The guy behind cppwinrt started work on the Xlang project in a similar vein, but it’s currently not being worked on so I figured I’d give it a shot.
I agree that C as the lowest common denominator sucks.
The same is true of Swift, and Swift can also call into C whenever it likes. The difference is that Swift can do the same for Objective-C at full fidelity, including support the full Objective-C feature set (ARC, arbitrary selectors, interfaces, properties, you name it).
Probably a crazy unpopular opinion, but I kind of agree with the Hyperbola (GNU / formerly Linux-libre, in future using a fork of the OpenBSD kernel https://itsfoss.com/hyperbola-linux-bsd/) devs that Rust in Linux is not necessarily a good thing, for the reasons they mentioned when they announced they were switching kernels (https://www.hyperbola.info/news/announcing-hyperbolabsd-road...) (Edit: and the LLVM dependency).
The vast majority of the text in that post is inaccurate and hyperbolic. Linux is not "forcing adoption of HDCP" (it's just something Linux has a driver for), Rust does not require internet access to use (and the trademark comments are not particularly accurate either), the Kernel Self Protection Project is alive and well and making new improvements in every new version of the kernel (see Kees Cook's regular blog posts for details), systemd is not spelled "SystemD", gettext has no dependency on Java (and the link in that post explains that). And in general, decisions about what technologies to use are made by those who show up and do the work to build viable solutions, not by people who snark.
It's a rant by an obscure one-developer distribution that formerly used the Linux kernel. Some people are never happy no matter what you do, and if you make the mistake of treating their rants as useful feedback, you get a list of 30 more demands before they'd consider gracing your little project with the honor of their all-important usage again.
It is well supported by the project itself, we offer a variety of tools so that you do not need internet access, and this was also a hard requirement of a lot of our early important users, like Firefox and various distros.
If crates.io was down tomorrow there would be a major disruption to you if you were using Rust for development. That is their point, as well as potentially exploring other reasons.
As someone who uses a source-based distro I hate language package managers. They always end up inferior to a distro package manager and create extra work for everyone, including the language developers, who tolerate it because it is in their favorite language.
It would only be an issue if I wanted to use crates that weren’t in my local cache already. And that’s gonna be true of any build system. It would also be true of any distro package manager.
Most other software is more dispersed and has plenty of mirrors, e.g. the Debian project. The same criticism of crates.io is typically levied against GitHub.
I think you were being "inaccurate and hyperbolic".
> Linux is not "forcing adoption of HDCP" (it's just something Linux has a driver for)
What they actually said in the article:
> Historically, some features began as optional ones until they reached total functionality. Then they became forced and difficult to patch out. Even if this does not happen in the case of HDCP, we remain cautious about such implementations.
My comment was entirely based on the second link. Direct quote from that:
> Linux kernel forcing adaption of DRM, including HDCP.
(Which linked to a patch adding driver support for handling HDCP, with absolutely no possibility of it somehow being forced.)
Regarding the quote from that interview: sure, and there's also no guarantee that the Linux kernel won't drop support for every architecture except SPARC, except of course for all the people involved not putting up with it. In what possible universe would Linux developers decide it was a good idea to mandate HDCP? It's completely unfounded and unsupported hyperbole and fearmongering, without even a hint of potential truth.
It's also roughly consistent with what I'd expect from someone who thinks that gettext daring to have support for Java and C# format strings is a horrific problem that should be ripped out by the roots.
I don’t know about Linux, but Mozilla has mandated many things that I would have hoped it never would have, like XUL depreciation, and especially added many anti-features to their largest product, deciding each was not their hill to die on https://news.ycombinator.com/item?id=24124954. I left Firefox and have never regretted it. The Hyperbola devs did the same, and I will be happy to try their OS and support them fighting on every hill they find, no matter how tiny or unimportant it seems to larger organizations.
Oops. This is why I try not to rush HN comments. This one I did and tried to remove my partial list of complaints about Mozilla and left one element, that’s not the most important one.¹
Mozilla markets Firefox as ‘the only browser made for people, not profit’. But they’ve done the exact opposite time and time again. Virtually the single most prominent UI element, the omnibar, has been sold out to the single largest threat to the open web, the gatekeeper to the entire internet to the vast majority of its users. I don’t want to spend all day railing against Mozilla, as easily as I could, but I can easily find a couple other examples off the top of my head. The Mr. Robot scandal is another example of selling out and (not necessarily directly harming but very significantly) alienating users. And it’s been a few years since I’ve used FF as my main browser but I still remember every time I installed it on a new device going into settings and changing default after default to revert it back from a profit-seeking product to a personal tool, disabling telemetry, etc.
1: Although it does make a difference to me. I’d use pre-Chromium Edge if it was FLOSS, on my OS, and let me use Pentadactyl. The excuses for XUL’s removal — security, stability, and speed — don’t concern me: https://yoric.github.io/post/why-did-mozilla-remove-xul-addo... seems to claim it would allow someone to somehow steal my passwords. I’ve been using Palemoon for years and I don’t have any weird activity on any of my accounts, and I keep my browser sandboxed (something Firefox should do better by default, one way it’s behind Chromium) so I don’t have any other security worries. I have no stability or speed issues, and maintainability seems like a non-issue given that an apparently one-man main team keeps Palemoon running.
I was going to comment similarly. The comment could be described as “inaccurate and hyperbolic”: if spelling counts, the second link does not say Linux was “forcing adoption [emphasis added] of HDCP.” Systemd stinks and I don’t care how it’s spelled as long as it’s sent to … nvm. And the Hyperbola devs have shown up, done the work, and ported pacman to their kernel, not making the mistake of listening to hyperbolic haters.
it involves the "patches-must-be-approved-by-upstream" to still be called Rust clause that Rust inherits from the Mozilla Trademark policy. Its the same policy that led to the creation of IceWeasel/Waterfox etc.
The root issue with Iceweasel was the logo, in my understanding. It was also resolved years ago. Rust does not inherit these problems, and even then, Mozilla will not be holding these trademarks anymore, even though I do not expect our policy will change once ownership changes.
For a desktop app it's understandable, but tons of scripts rely on the existence of rustc and cargo binaries. If the fork has to rename it to not-rustc just because it added e.g. gcc support, then that's going to break all the scripts.
I'm generally in support of rust in the kernel, but in the interest of playing devils advocate: Would adding more and more dependence on rust mean that it ties the kernel to a single set of compilers?
The Linux kernel would only compile with GCC for... decades? I think? So it's not really a new thing. It would be really nice if the kernel did have first-class support for LLVM and Clang directly (rather than having to set a bunch of environment variables to set the compiler, the linker to use with that compiler, various variables, etc.
That said, it would certainly be uncomfortable to move from "it compiles with Clang, though it's a pain, or GCC, which works fine" to "it compiles with GCC, if you don't use any of this functionality, or LLVM only, if you want any of these features or anything that depends on them".
Originally, we thought that Linux maintainers were asking us to only support kernels compiled with Clang, because they didn't want to support code built with two different compilers. However, in the session, Greg Kroah-Hartman specifically said that if it works to build Rust code with the LLVM-based rustc and C code with GCC and link the two together, and there aren't any problems in practice, then that's perfectly fine.
So, we're currently expecting that supporting Rust will not place any requirements on the C compiler you use, unless you want to use cross-language LTO (which we'll eventually want to do, but it won't be a hard requirement for a working kernel).
There was a talk right after ours about LTO! I only half paid attention to it because I was catching up on the chat, but it sounds like Google is doing kernel LTO in prod for their internal systems (and maybe also for Android and CrOS?):
At the moment, I believe the Linux kernel can do LTO with some additional patches, but there are still regular issues. It isn't nearly as well supported as Clang or LLD yet.
Linux currently only compiles on GCC and Clang, and the latter is a recent development. rustc uses LLVM as a backend, just like Clang, and people are working on other backends. Linux maintainers didn't express any concerns about this being an issue.
Also it would be kind of funny. Where the rust parts can only be compiled by the llvm based rust compiler, but clang(llvm) can't build the C parts.
And gcc can only build the C parts but not the rust parts :)
Mozilla’s trademark policies have been annoying enough in the past that Debian used to ship a fork with the branding stripped. Hyperbola continued this practice for a while, but stopped for a variety of reasons in the It’s Foss interview, and switched to a UXP-based browser. I’m writing this from a UXP-based browser because I find it much better than Firefox and I dislike Mozilla in general for more than one reason.
But no, there are other reasons outside of trademarks, some technical ones are in submisson’s article.
> Some users have correctly mentioned that many other software packages have trademarks, do we plan to remove them all? No. We are not against all trademarks, only those which explicitly prohibit normal use, patching, and modification.
> As an example, neither Python PSF nor Perl Trademarks currently prohibit patching the code without prior approval. They do prohibit abuse of their trademarks, e.g. you cannot create a company called “Python”, but this does not affect your ability to modify their free software and/or apply patches.
> Due to the anti-modification clause, Rust is a non-permissive trademark that violates user freedom.
I suspect whomever wrote that hasn't actually read the trademark policies in detail, or conferred with lawyers about it. Neither Python's nor Perl's agreements explicitly cover the case in detail, but Perl does mention this:
> Licensed redistributors of Perl code are permitted by their licenses to use the Perl logo in connection with their distribution services, on product packaging, and in promotional materials.
IANAL, but that text would especially suggest to me that you would have no right to use the trademark even to distribute an unmodified binary of Perl without approval.
I suspect the real reason is Debian and Mozilla had a very public, well-known spat over licensing, and Python and Perl have not had very public, well-known spats with anybody.
The article mentions possible ABI compatibility challenges and I also think it’s not a good thing for a language in such critical applications to only have a single major implementation.
The linux kernel still can only be compiled with GCC. Of course, it's a single patch away from clang compatibility, but the kernel has never been compilable with much else but GCC.
Yes, clang compatibility is a single patch away. Google, unlike the vast majority of companies out there, is capable of maintaining it's own forks of the kernel.
They can be used by the original authors, if the original authors stop intentionally licensing their code so weakly that anyone can make changes that no one else can even see.
Maybe a GNU Rust compiler without the trademark is in order? The state of KSPP is not nice as well. I wish Ada was more popular. From what I know, it has much of the security guarantees rust has and it has been battle tested.
GNU can either cut a deal with the Rust Foundation or use fair-use previsions afforded by trademark law. I think there is some leeway for cases like these.
How do you pronounce GNR? It think it needs to be a vowel, or an R.
Some attempts: GRR: Gnu Renames Rust, GAR: Gnu assimilates rust, GER: Gnu non Est Rust (gnu is not rust, but in latin to get an E), GIR: Gnu Isn't Rust, GOR: Gnu Oxidizes Rust, GUR: Gnu Unseats Rust.
I'd be more concerned about the fact that the Rust Evangelism Strike Force is hellbent on getting everybody to depend on their exhausts-all-32-bit-address-space-to-build compiler and toolchain -- in addition to whatever toolchain the project currently uses, at least as long as the strangler pattern is being applied to migrate the code base to 100% Rust. GNU software in particular has always been such that the core stuff is bootstrappable from just C.
But it seems, from an RESF standpoint, that the fact that any C code is out there running at all is a crisis-level problem.
Standing reminder: the Rust project is not a fan of rabid Rust over-evangelism (e.g. "Rewrite It In Rust", "Rust Evangelism Strike Force"), and sees it as damaging and unhelpful. We discourage that kind of thing wherever we see it. We're much more in favor of a measured, cautious approach.
There is no doubt a lot of work here, but C <-> Rust interoperability is a lot cleaner than C++ <-> Rust.
And for what it's worth, for almost all langauges C <-> X interoperability is a lot cleaner than C++ <-> X. Since almost all languages (including rust) "speak C", both in terms of compiler support and and being able to map every important C concept to a concept in their language. The same isn't true with C++.
One thing to worry about with adding any language (including rust) to the kernel, is that the next good looking language will probably interoperate well with C and not interoperate well with Rust. So there is a much higher cost the second time you try to add a language.
> One thing to worry about with adding any language (including rust) to the kernel, is that the next good looking language will probably interoperate well with C and not interoperate well with Rust. So there is a much higher cost the second time you try to add a language.
This is something that Rust is acutely aware of. There have been many discussions about the idea of a higher-level "safe ABI", such that languages with notions like counted strings or bounded buffers could interoperate with each other without having to go by way of an unsafe C interface.
Most of the chrome teams issues stemmed from C++'s lack of an ABI, not wanting to deal with using unsafe around the C++ interfaces & not wanting to have to annotate c++ to export it to rust.
I'd say the biggest questions for Linux that may be blocking here relate to LLVM not being available on nearly as many platforms as GCC (many of which will likely never be supported cause they're legacy systems that are only supported in the sense that they aren't purposefully broken) which limits it's use in any core-ish kernel systems.
Can you elaborate re: issues and lots of work required? I think it's generally to be expected that non-C language support in the kernel would be hard...
Ah, right, I'd read that - but it's for C++, so it's a bit different than the kernel situation, and I actually had a much more positive reaction to it ("wow, seems like C++<->Rust is actually pretty far along"). Guess both are reasonable takeaways.
I won't get tired of repeating the same comment in every topic that suggests Rust to be a great replacement of C in existing projects, that it isn't.
The safety guarantees that Rust provides are neither unique nor complete, and if we discuss the amount of effort necessary for bringing new interfaces like the one this article mentions ("one can define a kmalloc_for_rust() symbol containing an un-inlined version"), we should compare it to other existing solutions, like ATS [1], that are designed to be seamlessly interoperable with C codebases without giving up on the safety side of the argument.
Just a week ago there was a link in ATS reddit channel that advertised ATS Linux [2][3] initiative, that may be of great interest for the same people who are interested in bringing more memory sefety to Kernel development, without giving up on existing C interfaces and the toolchain[4].
But if you had seen the previous comments, that is, if you'd experienced repetition instead of new information, then they'd have been less curious and less useful. Curiosity wants to find new things, learn new things—it basically wants diffs because diffs are what it finds interesting. My source? Self-observation and observation of this community. Is it really a controversial point?
Even when we mark stories as dupes there are often users who say it was new to them and they appreciated the post. That makes perfect sense. No one sees everything, and if you haven't encountered the earlier elements of a repetitive sequence, then for you there is no repetition. I'm always reminded of a second-hand clothing store in my home town called "New to You".
But at a site-wide level it's not hard to understand that this is a stochastic process and we have to manage it globally. The alternative would be not to moderate repetition at all, and the community would definitely not prefer that.
What would be the proper way of informing those who are interested in the topic about available alternatives? I assume not everyone follows the same topics and there may be people who see the presented information and the relevant references for the first time.
You know the sort of person who comments 'Just install Linux' in every thread about more or less any kind of computer problem? You don't want to be that sort of person. If you're starting your comments with 'I'm not going to stop telling you to install Linux', it's probably a good time to check if you're not turning into the sort of person who's always telling you to install Linux.
If I ommited that first part of the sentence the points made afterwards would remain the same. I did so to indicate that it's not the first time the Rust community suggests that something should be (re-)written in Rust for memory-safety reasons, in a project that has a well-established C-codebase. I'd argue it's the same "Just install Linux" situation that you mention, and that "Linux" may not be the answer.
And 'ghostwriter stated elsewhere in this thread that the previous round of these comments was in reply to an article talking about how QEMU should start to develop new device backends as out-of-process Rust programs - which also was not about outsiders proposing a complete rewrite, either.
Perhaps the most long-lasting harm of the Rust Evangelism Strike Force has been the rise of the Anything But Rust Counterinsurgency, who sees that someone, somewhere, is thinking about Rust, and needs to convince them that they're making a mistake.
At the very least, please include links to your previous comments, so that any refutations of your previous comments can be seen by others - otherwise we will all have to repeat ourselves until there is no room for any new discussion. (One of the other replies pointed out that you had not incorporated an objection that someone made to your previous comment.)
I think if you absorb the responses from other users here, and adapt your future comments more to the specifics of a given story, that should help a lot. geofft makes a good point about linking to past points rather than repeating them. That's the way to handle repetition on HN. Don't inline generic material from before—refer to it via links.
Yes, I've seen this comment a number of times now.
ATS is a very interesting language! But there are a wide range of options for formal verification of systems programs, including the techniques used for the formally verified seL4 kernel, ATS, recent Ada work, TLA+, etc. Some of these tools have been very successful for certain projects.
But the more powerful proof systems involve tradeoffs. Frequently, you need to invest significant amounts of developer effort in exchange for very rigorously proven code. I've played with a number of these systems over the years, and they're great. But the costs would be hard to justify for many projects.
Rust occupies a slightly different niche. It focuses on preventing several classes of memory errors and data races. It's certainly harder to learn than Python, but probably easier than some common subsets of C++.
So the right tool for the job depends on what you're trying to do. If you want to formally verify your kernel's security model, or your distributed protocol, Rust would be a poor choice. If you want a language that makes it easy to work within sight of the metal, and that takes pains to make expensive operations visible, Rust can be a reasonable choice.
I agree with your comment, and I think we are mentioning the same question here, that is essential for the kind of discussion that this article raises - whether or not a suggested amount of effort to bring Rust into existing and well-established C codebase is a great idea and is justified on a technical merit basis, compared to the effort of brining more sophisticated formal proof assistants that aim at non-intrusive interfacing with C while being productive for the cause of promoting safe and formally-verified programming.
Python's gradual typing with MyPy demonstrated a great utility of static type checking that could be brought to existing project through many small and non-desruptive iterations. From my observation an learning of ATS, I expect even greater utility of gradual formal proofs that ATS enables for C. There are tons of successful C codebases, they are used in production around the world and contain a great amount of battle-tested knowledge. They just miss formal proofs that now could be gradually written for them.
Your evidence (that there are potentially other alternatives) doesn't support your claim (that Rust isn't a great C replacement).
There can be multiple great C replacements.
Not being "perfect" in terms of the guarantees you provide doesn't mean you aren't a great improvement. Nor would not providing any guarantees at all mean that a language is necessarily not a great improvement.
That said, I would be very interested in seeing a comparison between C, Rust, and ATS in practical code. This is the first I'm hearing of ATS and it sounds interesting.
> Your evidence (that there are potentially other alternatives) doesn't support your claim (that Rust isn't a great C replacement).
I didn't claim that Rust isn't a great replacement overall, I was specifically mentioning that it's not great for existing C codebases, by the fact that it requires significant work to replace internal interfaces that may be important for performance-, backward-compatibility- and conventional tooling reasons. It doesn't interoperate with C as well as other alternatives are capable of, whilst not being objectively better at safety guarantees either.
How does ATS provide safety guarantees without additional semantic information about the C code it’s linking to? If it does require that, then you’re in the same boat as Rust: your choices are trivial interoperation with no safety at the interfaces or encoding the semantics of the C interface in the new language. The only difference I can think of is built in ability to inline C from headers (based on your comment about kmalloc), is that the case for ATS?
It's not only inclusion of the headers. ATS compiles to C, and it can inline C without FFI. The benefit that you receive is the ability to introduce gradual automated proofs around unchanged stable interfaces that are known to be safe. You start with inlining everything but the small core that is proven to be safe. Then you can iterate indefinitely at the desired pace to bring new layers of formally verified API in places of previously unverified C calls without breaking backward compatibility and expected runtime profiles (safe pointer-based memory access [1] and stack-allocated closures [2] as an example)
This may need fewer code changes (if the code in question is correct and the interfaces can possible be implemented in a safe way, that is) but it still requires developers to write proofs in a dependent type theory. I’ve only looked at ATS briefly, but I’m familiar with a lot of the other languages in the space (Idris, Fstar, etc.). Are ATS’s proof tactics drastically more productive/brief than those (F* in particular, which has great aliasing reasoning)? If not I think you’re radically overestimating the approachability of this route compared to Rust.
I’ve used ATS, Idris and Rust, and I find ATS way more cumbersome to use for writing proofs than idris.
I’ve never used ATS to do what the GP is suggesting of incrementally wrapping C with proofs, but I can’t imagine this being simpler than just rewriting the C in Rust. You wouldn’t get the same proofs, but you would get memory and thread safety, which for many apps would be an incremental improvement.
I haven’t taught ATS, but I found it harder to learn than Idris, and I can’t imagine C programmers which have a hard time with Rust learning it quicker than they would learn Idris or Rust.
> but I can’t imagine this being simpler than just rewriting the C in Rust.
sometimes it's impossible without falling back to unsafe Rust, which kind of undermines the initial incentive. C codebases heavily utilise pointer arithmetic programming.
"Sometimes" and "kind of" are doing a lot of work there.
In reality most Rust code hardly ever needs to be unsafe and the existence of unsafe code in libraries you use hardly ever has any impact on the security and stability of what you ship, because there just isn't very much of it compared to the safe code. The "trophy cases" for Rust fuzzing bear witness to this.
> C codebases heavily utilise pointer arithmetic programming.
You don't write unsafe Rust code everywhere C code would use pointer arithmetic. You use safe Rust idioms and APIs instead.
If it turns out that writing Linux drivers in Rust requires writing a lot of unsafe Rust code in each driver, then that would certainly be a failure. I don't see any reason to believe that will be the case.
ok, I'm fine with that, the question is why it is technically better for Linux Kernel than a proper formally-verified implementation that doesn't require Rust toolchain?
My experience may be too specific to reason about productiveness with ATS when it comes to other developers. I knew some Haskell and I've already read the Idris book when I discovered ATS, and I knew some C from university courses. The proofs require the same mental model, totality check principles are the same, resource utilisation principles are the same, so for me the difference when comparing these two langauges is mostly about the things available apart from the common features: ATS prelude has many predefined constraint functions that facilitate low-level programming with pointers over unboxed data.
I haven't tried F*, but from quick googling of "aliasing", it seems that a similar kind of checks can be achieved with view-changes in ATS (but please correct me if I'm missing the point) - http://ats-lang.sourceforge.net/DOCUMENT/INT2PROGINATS/HTML/...
I'm not sure what you mean by "FFI" (I would apply the term "FFI" to the way ATS interoperates with C, too), but you can certainly get cross-language LTO between C and Rust, since they both compile to LLVM IR.
> you can iterate indefinitely at the desired pace to bring new layers of formally verified API in places of previously unverified C calls without breaking backward compatibility and expected runtime profiles
I meant not having to think about wrapping/boxing/unboxing, as they have exactly the same native data representation, so it's slightly different than LTO. But mine using "FFI" there was probably too vague and frivolous, I agree.
> You can do this in Rust, too.
not always within boundaries of safe Rust though (assuming each iteration is allowed to brings zero unsafe/unverified code), which undermines the initial safety argument.
> wrapping/boxing/unboxing, as they have exactly the same native data representation
Rust's Box<T> and Option<Box<T>> have the same representation as a C pointer to the same type, so this is true of Rust too.
You do have to wrap the type in source code to indicate what the ownership guarantees/contracts are. (Is this not also true of ATS? That is, if ATS code calls a C function that returns a pointer, how do you know how long that pointer is valid for and whether you're expected to free it?) But there is no runtime overhead/conversion.
> which undermines the initial safety argument
No, adding unsafe Rust does not undermine the safety argument (at least, no more than having any C somewhere in your address space does). See my reply in the comment section of TFA: https://lwn.net/Articles/830158/
(It's unfortunate that the Rust language used the "unsafe" keyword to mark blocks where you can access raw pointers, because it causes people to think that using such code is unsafe. Something like "manually_verified_safe" would have been better - the whole point of putting an "unsafe" block on a safe function is to write code that is safe to use that cannot benefit from the Rust compiler's checks. But at the boundary with C, it is impossible to have automated checks, because the relevant typing/safety information doesn't exist in C in the first place. It's no different from ATS code using $UN where you've manually verified that you're using a C function or C-created data in the way it's intended to be used.)
> Is this not also true of ATS? That is, if ATS code calls a C function that returns a pointer, how do you know how long that pointer is valid for and whether you're expected to free it?
the C implementation needs to be known, then the signature (interface) of the extern-annotated C function would define a pointer that will be associated with a proof object. Every ATS program that compiles must consume all instantiated proofs (with a proper proof-consuming function), and the proof itself can indicate whether a pointer is stored/taken care of inside the C function or it should be taken care of outside the function (the latter is indicated by a prefixed "!" to the pointer type). An example could be found here - https://bluishcoder.co.nz/2012/08/30/safer-handling-of-c-mem...
> ring buffers in Rust use "unsafe" and at some point there was a logical CVE in vec_deque that broke one invariant of the unsafe block
First, you're talking about two different things here. Originally you were talking about incrementally adding Rust bindings to C code and how that obligated you to write "unsafe". Now you are talking about pure-Rust implementations of data structures that use "unsafe" for optimization purposes. The second one is much more dangerous.
Second, it does not undermine the safety argument - quite the opposite, as demonstrated by the fact that code that used VecDeque remained unchanged after the bug fix. Everything that the Rust compiler proved about code that used VecDeque, on the assumption that VecDeque was sound, remained just as proven after the bug was fixed and VecDeque was changed to be actually sound. There was no ex-falso-quodlibet problem. There was a clear delineation in the safety argument: on one side, a human was obligated to check that VecDeque was implemented soundly, and on the other side, the Rust compiler checked that every user of VecDeque did so soundly. The first part was done incorrectly and then fixed; the second part remained done correctly.
> the proof itself can indicate whether a pointer is stored/taken care of inside the C function or it should be taken care of outside the function (the latter is indicated by a prefixed "!" to the pointer type).
Then this is an axiom as input to the proof, not a proof itself. That is, ATS is no more capable than Rust of magically determining what the unstated invariants of C code are; it can only prove that certain things logically follow from certain assumptions. This is precisely what Rust does, too, except it is clearer than ATS because any use of such unchecked assumptions are clearly marked with the "unsafe" keyword.
In the linked ATS code, I see no indication, as a human reviewer, about which parts I should carefully audit (e.g., the annotation of the "extern fun") and which parts ATS has checked for me (e.g., the implementation of "string_to_base64"). In Rust, you can very quickly search a codebase for "unsafe" and see what needs auditing. (And in fact this technique empirically works well for finding unsoundness in production Rust code, and I can point to multiple examples of it. I wonder what people do when reviewing production ATS code.)
Again, I'm not saying ATS isn't cool. I'm not saying it wouldn't be great to have ATS support in the Linux kernel. Please work on this. There are, in fact, lots of things that ATS can do that Rust cannot do, and they are helpful to have. But I think the specific things you are claiming are things that Rust can do just fine or that ATS in fact cannot do.
> Then this is an axiom as input to the proof, not a proof itself. That is, ATS is no more capable than Rust of magically determining what the unstated invariants of C code are
I've never stated that there's such a capability. I was saying that unlike Rust, ATS can be used to gradually rewrite every C call into a safe and formally-verified equivalent function that has exactly the same performance and memory charactersitics as the original C function.
> I won't get tired of repeating the same comment in every topic that suggests Rust to be a great replacement of C in existing projects, that it isn't.
That's not what this is about, so even though you aren't tired of posting this, I'm not sure it's warranted.
This is about the kernel supporting other languages in select places where it makes sense, such as kernel modules, where there's already an API in place to conform to. It's also not limited to Rust, it's about providing support for multiple other languages that could be beneficial in those locations.
> I won't get tired of repeating the same comment in every topic that suggests Rust to be a great replacement of C in existing projects, that it isn't.
Nobody is giving up on existing C interfaces. They are simply making sure those interfaces are not actively hostile to languages that aren't C by nature of how they are implemented in C.
To be clear, the "kmalloc_for_rust" was mentioned as one possible way forward. Another would be a more advanced Rust bindgen tool that can determine how to deal with it automatically. Neither affect ATS, based on your description (although a "kmalloc_for_extern" might generally help any non-C language, which would probably be beneficial).
If ATS doesn't require any changes to C to interoperate with it, this whole announcement should have zero impact on ATS, other than possibly making it easier to get an ATS module in-kernel since they're trying to make it easier in general for non-C modules.
You can treat this as a zero-sum gain, where Rust or Go's gain is ATS' loss, in which case you might as well pack up your bags now, since the writing is on the wall based on publicity, or you can treat it as a rising tide raises all boats type situation, where more general acceptance of alternatives to C is still beneficial to ATS actually being allowed in-kernel, even if it doesn't play exactly to ATS' strengths. I know which one I would put my money behind as being a better strategy in the end.
> It's also not limited to Rust, it's about providing support for multiple other languages that could be beneficial in those locations
But what are these benefits that other languages bring on the table? From the article it appears that proponents of the new toolchain mention memory-safety as the primary concern:
> They focused on security concerns, citing work showing that around two-thirds of the kernel vulnerabilities that were assigned CVEs in both Android and Ubuntu stem from memory-safety issues. Rust, in principle, can completely avoid this error class via safer APIs enabled by its type system and borrow checker.
So the natural follow-up question to that concern, it seems to me, would be whether there's something in the existing C/GCC toolchain (or around it) that can resolve the raised concerns without adopting a whole new toolchain (Rust/LLVM) and without having to extend the existing interface with language-specific suffixes like "kmalloc_for_rust".
> But what are these benefits that other languages bring on the table? From the article it appears that proponents of the new toolchain mention memory-safety as the primary concern:
Of the venn diagram of languages that provide more safety than C (and that is, either require it or make it hard to bypass), and languages that people know know or are likely to know in the semi-near future, Rust and Go are probably the big contenders.
ATS may be easier to interface with the kernel in, but is random peripheral company writing a kernel module driver likely to know it and use it? Theoretically, everyone can write most or all of their modules in Coq, right? So why don't they? Why is ATS different?
> So the natural follow-up question to that concern, it seems to me, would be whether there's something in the existing C/GCC toolchain (or around it) that can resolve the raised concerns without adopting a whole new toolchain (Rust/LLVM)
You're mistaking the aims of a project to specifically adapt Rust to the linux kernel and what they focused on. This note is frm a Rust initiative, not a kernel initiative, why would they focus on doing something in a different language?
> and without having to extend the existing interface with language-specific suffixes like "kmalloc_for_rust".
I addressed this in the last comment. Did you have something new to bring to this aspect of the discussion? It feels like you're ignoring how I noted that this is but one option people are discussing, and using it as a reason why the whole thing is a bad idea. That doesn't make sense.
> So the natural follow-up question to that concern, it seems to me, would be whether there's something in the existing C/GCC toolchain (or around it) that can resolve the raised concerns without adopting a whole new toolchain (Rust/LLVM)
The answer to this question appears to be: no.
Memory errors sit somewhere around 2/3 to 3/4 of errors irrespective of codebase when C is involved (this has been consistent in things like the Linux kernel as well as the Windows codebase). C++ hasn't seemed to have changed that, either.
Remember, Rust IS Mozilla's answer to this question.
> without having to extend the existing interface with language-specific suffixes like "kmalloc_for_rust".
The requirement to do that is pointing out the that kernel API is poor and is too intimately tied to the build chain. We have seen this before--when Linux got ported to Alpha, for example.
The point is to fix the API so that more languages than just Rust can be used.
I suspect that more "system programming" languages may appear once Rust does the difficult groundwork of breaking this kind of intimate toolchain dependence. As things currently stand, why develop a system programming language given that no one will ever use it?
C++ is not like C. Putting some C++ objects in a C codebase would, indeed, not change it much, but if you write in modern C++, some memory errors will literally disappear, and some become less likely.
Caveat: This is based on language mechanics, standard library facilities and recommended idioms; of course people can write unsafe C++ if they want to.
You are partially correct but mostly incorrect, I would say.
It's true that writing safe code requires some study and some understanding. But you don't need to "know what's safe". That _is_ complicated - there are a lot of complex depths to the language. The thing is, the non-expert user can avoid them.
For example, a green developer can start with "Avoid using new and delete directly; use vectors, other containers, or unique_ptr's instead. Maybe shared_ptr's in some cases." If at some point they really need to write their own allocating class, they'll be told to be very careful with the allocation, and take into account the possibility of exceptions, concurrent execution by other threads etc.
Also, when you're not sure of yourself, there's a good enough reference in the form of the C++ Core Guidelines [1]. (That's a long document, but you don't need to read through it.)
Finally, a whole lot of memory error hazards / unsafe code can be statically identified and discouraged. Gradually, IDEs and other tools have started doing so.
Some of us are in the managed language world now, the problem is that C++ still has the best integration story with Java and .NET codebases, which Rust still needs to catch up to.
So modern C++ will have to do for the few cases that require going out of them, e.g. runtime plugins for native agents, COM/UWP with access to OS APIs, IDE mixed language debugging, GUI components.
Also there is the whole issue that GPGPU shaders and Apple/Google/MS driver stacks are mostly about C++.
So one either does managed language + C++, or managed language + C++ + additional glue effort + Rust, not much to win currently.
* Dangling references from lambda captures are no different than dangling references or pointers passed without lambdas, and not more likely to be used.
* spans are no worse, and usually better, than using a raw pointer and a size; they don't make memory errors more likely.
* string_view - indeed, it potentially introduces a dangling pointer. But - that only happens if you store it. Which is why I qualified my claim to "using recommended idioms". If you replace your `const char*` or `const std::string&` with `std::string_view` you have not introduced a potential memory error.
More generally - new language/library construct which do not manage their own memory can introduce memory errors. But, luckily, C++ now has a better "harness" of protecting you from many of these without costing you and performance nor introducing now memory issues.
So I stand by my claim. Having said that - C++ is not a safety-guaranteed language, and if that's what you need then it is indeed not the language to use.
2. Linux is already moving in the direction of having easier-to-use internal interface over small, fast, and tricky ones. For kmalloc in particular, the memory management docs https://www.kernel.org/doc/html/latest/core-api/memory-alloc... recommends kzalloc (which zeroes data) and suggests a function called kvmalloc which calls either kmalloc or vmalloc depending on the size of the allocation. It unconditionally recommends kvfree (which handles alloations from either kmalloc or vmalloc) instead of asking you to use kfree or vfree as appropriate.
3. In this particular case, kmalloc is inlined because it calls krealloc(NULL, ...). This could almost certainly be equivalently handled by inlining the other side of it. (Again, benchmarking.)
The Linux kernel is fairly modular, I could definitely imagine rewriting subsystems by subsystems with fairly good ergonomics and safety. Like rewrite serial_core.c in Rust and then let people write serial drivers in Rust. Then the SPI code. Then the I2C. Then USB etc...
Easier said than done of course, and definitely a huge effort especially for such a huge project as the Linux kernel, but there's also a significant amount of resources that could be allocated to that task by several large institutions.
I don't know if it's a good idea, or if ATS or something else could be better, but I think you're overly pessimistic.
Also "the safety guarantees that Rust provides are neither unique nor complete", while technically true, is also borderline FUD-y. At least you should expand on what you have in mind here, because I doubt that there exists a programing language that deals with low level hardware configuration that could ever provide "complete guarantees" about anything.
And beyond that, keep in mind that Rust provides a lot more than just safety guarantees, its syntax is lot more advanced that good old C in particular for anything dealing with types (including destructuring, matching, inference etc...). Even if you don't get "full" safety right away it can still be a much nicer language to code in (and I say that as somebody who knows C well and quite like it overall).
> Also "the safety guarantees that Rust provides are neither unique nor complete", while technically true, is also borderline FUD-y. At least you should expand on what you have in mind here, because I doubt that there exists a programing language that deals with low level hardware configuration that could ever provide "complete guarantees" about anything.
I meant inability to implement within the boundaries of safe Rust certain data structures and algorithms that ATS can prove to be safe. To name a few: stack-allocated closures, and safe pointer arithmetic programming [1]. For instance, ring buffers in Rust use "unsafe" and at some point there was a logical CVE in vec_deque that broke one invariant of the unsafe block [2]. Safe implementation of the ring buffer in ATS could be found in [1]
You can't directly prove the pointer arithmetic safe with the compiler, but we routinely write a small amount of unsafe code which the programmer is confident in (and depending on the project that can mean an informal proof), and then re-use the data-structure that unsafe code created millions of times.
A ring buffer is a great example. Vecdeque is implemented once in the standard library, everyone who uses Vecdeque can do so in completely safe code provided the small amount of code in Vecdeque is safe.
This is orders of magnitude better than C, where every user of the ring buffer might make a mistake and cause a bug (or at least the API has no way of communicating that that is not the case, and in general the type system isn't expressive enough to make APIs that force that to be the case). It's not clear that taking the next step to proving the ring buffer implementation itself is correct to is actually worth the effort (it's not clear that it's not, but you need to make that cost benefit analysis, and I highly doubt it would be anywhere close to uncontroversially the case).
They are implemented once in a crate on crates.io and similarly re-used. Or in a code base like Linux with some unique requirements they'll be implemented once there and re-used many times there.
As it turns out most code isn't datastructures with unique memory requirements (i.e. that can't be implemented in terms of other datastructures), most code just re-uses existing datastructures.
> they are implemented once in a crate on crates.io and similarly re-used.
this doesn't automatically prevent the code from containing CVE, does it?
> As it turns out most code isn't datastructures with unique memory requirements
most code that has been written in Rust so far. But when you enter a territory of OS kernels, unique memory requirements and data structures such as ring buffers and dynamic mutable trees are the things that have to be implemented according to the requirements, and we want them to be implemented as efficiently as C is capable of, and correctly. Falling back to "unsafe" is an option, but it's hardly an argument that supports the effort of bringing safety to such a kernel.
But let's say that we've solved pointer issues. What about dependent types and totality checking? ATS has that [1] [2]
> this doesn't automatically prevent the code from containing CVE, does it?
There are certainly tools to detect any known CVEs, and tools to help you audit the unsafe code that you are using. They don't provide rigorous proofs automatically of course, the same way the standard library is generally not rigorously proved to be correct.
> But when you enter a territory of OS kernels,
Nothing really changes, see redox as an example of a fairly "complete" kernel written in rust with a reasonably low amount of unsafe.
> we want them to be implemented as efficiently as C is capable of,
This is already the case with Rust's typical library system. Generic libraries in rust are not less efficient.
In the real world rust datastructures are typically more efficient in my experience because the clean separation makes it easy to optimize the hell out of them.
> What about dependent types
That's a tool not a problem - you have yet to provide an argument that they are necessary and real world experience says that Rust achieves a meaningful amount of safety and convenience without them.
> totality checking
That's also a tool not a problem, and you could argue that rust's ADTs (enums) give you a weak version of it.
> There are certainly tools to detect any known CVEs, and tools to help you audit the unsafe code that you are using.
so they don't automatically prevent the code from containing new CVEs.
> That's a tool not a problem - you have yet to provide an argument that they are necessary and real world experience says that Rust achieves a meaningful amount of safety and convenience without them.
I can argue along the same line that Rust is a tool not a problem, and that you need to prove that Rust has a "meaningful" amount of safety. When you say that something has "meaningful amount of sth" you should add to whom it is meaningful and based on what criteria. Or, alternatively, you can stay on a pure technical aspect of the matter and conclude that the safety part of the language is not complete.
> That's also a tool not a problem, and you could argue that rust's ADTs (enums) give you a weak version of it.
totality checking is not exhaustiveness checking, there are two more properties that enums are unable to express in any "weak" form - termination and productiveness.
ATS is also harder to understand[1] (in my experience) and has a fraction of a fraction of the adoption rust managed to get, despite being older. To me that suggests that rust is more approachable, has better documentation, and would be used by more people if supported by the kernel.
[1] and really, I'm an OCaml developer and I can write rust; yet ATS' syntax looks terrible and very complicated. Its website and tooling make it appear like a research language made by one person. I don't know if that's really the case.
ATS certainly looks interestinb, but it's an academic language (of which there are many) that most people probably haven't heard of... At some point, the momentum of a new programming language is just as important -- in practical terms -- as its formal attributes and qualities.
Popularity and momentum translate into (and are proxies for) important things like: library availability, long-term maintenance and support, more edge cases are explored (so less “research”, breaking new ground and bugs when going off the beaten track), tooling and even availability of teaching material like documentation and tutorials.
In case of ATS, any C library can be treated as a "unsafe-marked" ATS library, so there's no problem with library support, the problem is with their formal verification. And Rust libraries regularly suffer from the same lack of formally-verified and proven-to-be-safe APIs.
That helps with the "library availability" sub-point specifically, yes. However, there's a more than that when evaluating a language, as suggested by my comment above.
In any case, if one is discussing modern and safe languages, there's a dramatic difference between using a C library and using a library native to the language. Even ignoring safety, the ergonomics/developer experience is dramatically different and the impedance mismatch can be very frustrating (for a "best case" example of this, Swift puts a lot of effort into exposing Objective-C interfaces as "swiftier" APIs automatically, but this benefits significantly from a relatively opinionated set of idioms in the source language, something arbitrary C libraries do not have).
You're right that being able to import a C library directly is a very nice feature.
> And Rust libraries regularly suffer from the same lack of formally-verified and proven-to-be-safe APIs.
Right... but having a larger community means there's a much higher chance of a safe or easier-to-use library existing for any particular task. In particular, I think it means there's almost certainly more libraries in total, with more safe libraries and more unsafe libraries overall, so simply comparing number (or proportion) of low-safety libraries is misleading.
(Formally verified is shifting the goal posts here: the bar is just "has a native library".)
> (Formally verified is shifting the goal posts here: the bar is just "has a native library".)
It is, but it is for a good reason - ATS formally verifies your safe functions, so you only need to implement your proof once and make a compiler agree with you, and then you can save on a community-driven code-review process. Rust, on the other hand, provides safety guarantees only for a subset of safe guarantees that ATS provides. Pointer manipulations have to reside in "unsafe" blocks, and that leads to CVEs - https://gts3.org/2019/cve-2018-1000657.html
It's moving the goal posts and changing the point of the discussion. We can equally well say that that "Rust formally verifies your safe functions", and that this reduces how much code review is required... it's a matter of degree and specifics.
Focusing on pointer manipulations is also misleading: it's entirely true that it's dangerous, but most Rust code does not need to do any sort of raw pointer manipulation. For instance, there's extensive work on operating systems and other low level code that involves both inherent unsafety due to hardware specifics (that is, ATS almost certainly does not model it natively) and safe Rust wrappers (i.e. proofs of safety) for the unsafety at a surprisingly low level:
> We can equally well say that that "Rust formally verifies your safe functions", and that this reduces how much code review is required... it's a matter of degree and specifics.
You cannot say that while this kind of CVE is possible https://gts3.org/2019/cve-2018-1000657.html and as long as Rust is not capable of performing safe pointer manipulations. The above issue may be solved for VecDeque in stdlib, but what about other data structures and algorithms in the wild?
> Focusing on pointer manipulations is also misleading
Remember that the discussion is happening in the topic about brining Rust into well-established C codebase, and C uses pointer-based arithmetics all the time, sometimes the whole algorithms are implemented this way because it brings efficiency to lower-level interfaces that are supposed to be very fast. If the motivation to bring a new tool is pronounced as "let's make it safe", why should we allow the argument to fallback into the "unsafe Rust" territory?
> For instance, there's extensive work on operating systems and other low level code that involves both inherent unsafety due to hardware specifics
I'm not going argue against that, because it's not related to the current topic related to Linux Kernel.
it's not more, it's one specific example. Shall we now enumerate all the things that both languages are able to prove? I think we should, to make it clear what the differences are and what is possible to prove safe. That's exactly my point about comparing brining Rust into existing C codebases with alternative tooling available specifically for C codebases.
"In the wild" one hardly ever has to write unsafe Rust data structures and algorithms.
> If the motivation to bring a new tool is pronounced as "let's make it safe", why should we allow the argument to fallback into the "unsafe Rust" territory?
Because going from 0% of code proven safe by the compiler to 99% is very valuable. Verifying the remaining code is desirable but not as valuable as what Rust already provides.
Having said that, it would indeed be great to have a proof system for verifying properties of unsafe Rust code. Much work has been done in that area: https://alastairreid.github.io/rust-verification-tools/ Something to look forward to.
> "In the wild" one hardly ever has to write unsafe Rust data structures and algorithms.
one has to do it all the time if cyclic mutable graphs or specific buffers/caches are involved.
> Verifying the remaining code is desirable but not as valuable as what Rust already provides.
How do we know that? What are the criteria and the thresholds that lead us to that conclusion? Is it true for all fields where the language can be used? What should we do about inability to express more precise constraints at compile time? Shall we stop on Rust, or try to embrace more powerful tools that already support Dependent Types in low-level systems programming? These checks enable a whole new world of expressive powers and correctness guarantees, even compared to the cool borrow-checker. ATS supports them today [1]
"Specific buffers/caches" is ambiguous. For embedded systems there are crates that provide safe interfaces to memory-mapped hardware.
You will likely argue that using unsafe code in a library is just as bad as writing unsafe code, even if that library is used and tested by a lot of people and the unsafety is corralled behind a safe API. You would be wrong.
> How do we know that? What are the criteria and the thresholds that lead us to that conclusion?
My current project is 170K lines of Rust code, and has 225 uses of unsafe. That's about 1.3 uses of 'unsafe' per 1000 lines of code. If I could write C++ code and introduce less than 2 exploitable vulnerabilities per 1000 lines of code I'd have an even higher opinion of myself than I already do.
> Is it true for all fields where the language can be used?
I'm sure we're both imaginative enough to dream up some "field" narrow enough to disprove any universally quantified proposition.
> What should we do about inability to express more precise constraints at compile time?
We should adopt proof systems that let us verify safety properties for those little bits of unsafe Rust code. We should not, however, make that a precondition for writing that vast majority of code that can be written in safe Rust in safe Rust.
> Shall we stop on Rust, or try to embrace more powerful tools that already support Dependent Types in low-level systems programming?
That is a false dichotomy.
Safe Rust is a sweet spot where the compiler and tools can verify a strong set of safety properties without the developer having to deal with proof systems and dependent types, with lots of engineering to produce helpful messages when things go wrong. Plus a large library ecosystem that, among other things, provides lots of safe abstractions over unsafe code. Trying to put the brakes on Rust and get everyone to buy into ATS instead is putting the needs of the few over the needs of the many.
> I can have cyclic graphs with mutable data without writing any unsafe code
You can, because the libraries that you use fallback to unsafe blocks. How about those who want to write these libraries without having to use unsafe?
> You will likely argue that using unsafe code in a library is just as bad as writing unsafe code, even if that library is used and tested by a lot of people and the unsafety is corralled behind a safe API. You would be wrong.
just because you think I'd be wrong doesn't prove me being wrong. Why should I rely on unsafe if I can implement the same algorithms and data structures with proper safety guarantees, but not in Rust? Why should I stick to Rust specifically for that matter?
> My current project is 170K lines of Rust code, and has 225 uses of unsafe. That's about 1.3 uses of 'unsafe' per 1000 lines of code. If I could write C++ code and introduce less than 2 exploitable vulnerabilities per 1000 lines of code I'd have an even higher opinion of myself than I already do.
yeah, it shows your personal commitment to Rust toolchain. This is good. But what if you've already acquired some knowledge of more powerful tooling out there, and you're given a task to estimate applicability of these two toolchains to the development of Linux Kernel?
> We should adopt proof systems that let us verify safety properties for those little bits of unsafe Rust code. We should not, however, make that a precondition for writing that vast majority of code that can be written in safe Rust in safe Rust.
Why shouldn't we do it specifically for Linux Kernel? Why shouldn't we use compile-time checks for element incusion, length-based non-emptiness of containers instead of implementing another set of runtime validators?
> Safe Rust is a sweet spot where the compiler and tools can verify a strong set of safety properties without the developer having to deal with proof systems and dependent types
How do you define a sweet spot? Is it indeed a sweet spot when it comes to kernel development? Why does this spot happen to be at the level of Rust type system, and not somewhere else?
> Trying to put the brakes on Rust and get everyone to buy into ATS instead is putting the needs of the few over the needs of the many.
I'm actually arguing here from a position of technical merit of two tools in the context of Linux Kernel development, but for some reason you bring vague and non-technical definitions of "sweet spot", "lots of people using something", "needs of the few vs needs of many", which I've not mentioned anywhere in the thread.
> In any case, if one is discussing modern and safe languages, there's a dramatic difference between using a C library and using a library native to the language.
The dramatic nature of that difference is whether that library exists or not. Odds are, if exists then it's written in C. Thus this point is mute with regards to Rust because at best it's relegated to a nice-to-have, in the sense you can enjoy the same features that are already available in C but with language-specific assertions.
Ah, furthermore, looking at a comment below, it seems ATS's support for C libraries is almost identical to Rust's: specify a list of functions and their signatures on the ATS/Rust side. From your adoring descriptions, I had been assuming it simplified the process properly, by allowing importing a C header directly (like Swift can).
(The main difference seems to be ATS allows inline C, since it looks to be tied to C as a compilation target.)
Choosing an obscure language with little community support imposes real-world development costs: it's harder to find or ramp up new developers, there are fewer eyes identifying bugs in the implementation, tooling support can be subpar, documentation and blog posts are harder to find, etc.
(btw I know nothing about ATS so I'm not saying this is a good description of that language in particular.)
I agree with your points regarding tooling and dev support in general. ATS, in that regard, benefits from being a "frontend compiler" to GCC - all the tooling for GCC can be used equally well for ATS codebases. Developers need to know C well (not an issue with all the learning material available), and at least one language from ML family (OCaml, Standard ML, F#, Haskell to some extent) to quickly pick up the syntax and common recursive constructs. There are three great books [1] available online, and a collection of common system programming examples that demonstrate how data types and algorithms can leverage safe language features [2]
Popularity is not quite the same as PR. But yes, it's about the number of engineers and projects using it, and the growth of such adoption. If technical merits were the only criteria for making a difference, I think it's fair to say we would have a very different technology ecosystem today. :)
Yes, it really does if you're talking about success. Just like with Betamax vs VHS, and many, many other examples over the years. Technical merit is only one small part of a success story, and it's not the most important one.
I’m glad that there other efforts for memory-safety going on concurrently, and we as a software community should absolutely continue to invest in those. I also agree that rust is not suited for all circumstances that people try to use it in - but I would contend that the set of situations where rust is absolutely the right choice is non-empty. I say this as part of a ~20 person team, rewriting business-critical software in Rust, where most of the engineers I talk with are glad they’re writing rust.
> but I would contend that the set of situations where rust is absolutely the right choice is non-empty
Maybe I should've highlighted it in my initial comment, because I definitely agree that this list is non-empty for Rust. I was mentioning specifically existing C codebases that are heavily used in production around the world (last time a similar topic was related to QUEMU project).
> I won't get tired of repeating the same comment in every topic that suggests Rust [...] we should compare it to other existing solutions, like ATS [1]
One of the problems with repeating comments is that the previous replies are not incorporated into the discussion; for example, mine:
>Back when it was on the PL shootout, ATS was consistently in the top five for performance. However, it was also consistently the worst in program size
I looked into learning ATS some years ago before Rust existed, and shied away due to the complexity. It's been around much longer than Rust, and I think there's a reason it hasn't caught on: it's just too onerous. ATS might be appropriate for formally verified systems, but language ergonomics does matter, especially for a project with as many contributors as the Linux kernel.
> The safety guarantees that Rust provides are neither unique nor complete
That sounds like "LEDs shouldn't be used for traffic lights because they don't melt snow", while ignoring all the benefits. Just add a heater where it is required.
Rust does add benefits. Are these worth the extra effort? It seems that some people think it's worth it.
The question is more specific - does it add more benefit to the existing well-established C codebases, compared to other tooling particularly aiming at helping existing C codebases.
To me the benefit of Rust is less safety, than better abstractions and modularity.
Linux has been growing in complexity without much of a breaking point, and the discipline and separation of concerns high level languages could help with is a huge long-term benefit.
(For example, say we could one day make a bunch of basic Linux drivers somewhat kernel agnostic? That would be amazing.)
Now, ATS is I hear a fine language, but I think the ability to write good, and especially abstract, ATS is sadly too far outside of most kernel dev's skill set. Even among PL researchers, there is a tendency to write monolithic Coq---to wit, is there a Hackage or crates.io for Coq?---because abstraction are hard and the academic paper economy doesn't really reward it.
Rust is just easy enough, and with enough crates.io momentum, that I hope the relative cost of reusable vs unreusable code will be less.
And finally, it takes a lot of political will to introduce a new language in any existing project, let alone one as big and storied as Linux. Like it or not, but general popularity of the language absolutely does help with that.
Reading through your various replies to comments, it seems your argument boils down to the FFI issue, and I find that a little bizarre. Sure, ATS can inline library calls due to the lack of an FFI, and maybe it has a cleaner interface to those libraries, but now you've introduced a completely different set of problems: now you have to tell the compiler how to prove your code.
In my experience, learning the fundamental concepts behind theorem proving is a lot harder of a problem than writing an FFI interface to a library. In fact, that problem is so easily solved that there are tools that do 99% of the work for you.
And sure, maybe rust will never be able to inline C code, but inlining function calls is such a trivially small performance gap that most applications would never know the difference.
I'm not convinced that there are better alternatives for contributing to or incrementally rewriting an existing code base. FFI is for most purposes a trivial impediment compared to the alternatives.
Hmm, I think my point is less about "native" FFI and more about safety guarantees and compatibility. Not every algorithm is portable to safe Rust without loosing on runtime/performance characteristics of the code, and falling back to unsafe Rust for that matter undermines the initial "rust is memory-safe" argument. C codebases are heavily invested in pointer-based algorithms and optimisations. ATS is able to verify them, Rust isn't.
It honestly doesn’t matter a whole lot what makes people excited about contributing to Linux. I think it’s worth embracing whatever does as long as it’s at worst harmless.
Edit: I think what I’m trying to say is, let’s embrace what excites people.
How do you know unless you try? I don't know why people are so against it. If you want to see vitriol see any discussion of rust on the now impotent slashdot community. If Linus is okay with it, I'm okay with it. He knows a lot more about the kernel than me and probably 99.99999% of of the people on the planet. I'm sure there are some awesome Einstein IQ people that know it even better than Linux, but I'll probably never meet them.
There's an extremely high cost to introduce and maintain another language in a stack such as the Linux kernel. Everyone involved would be permanently impacted regardless of their opinion on the matter. It's not just a flag you can toggle.
Otherwise Rust community would have already forked the Linux kernel instead of bugging their maintainers to RIIR.
On top of that Rust compilation is very costly. It would render some devices that currently can compile the kernel unable to do so.
As one of the presenters of this session, I'd like to reply by saying, this is cool! I'm hopeful it shows some good results! I'm not wedded to Rust in particular - it just seems to have the right balance of solving the particular problems the kernel wants to solve (there are plenty of other great languages like Go and Java and Python and Haskell and C++ that aren't suitable), easy to learn for people who have been working with C and have picked up an intuition for things like lifetimes and ownership that matter in C but aren't concrete, and popular (which is important because it means there are good learning resources, communities building things, etc.).
It looks like the "ATS Linux" work is happening on a Git fork at http://git.bejocama.org/ats-linux-5.7.2 . However, I don't see any commits there when I clone it - it looks like it's just the v5.7.2 tag from upstream. Do you know if there's any sample code that's been written demonstrating kernel code in ATS?
I think that, empirically, we have built something that works. You can check it out (https://github.com/fishinabarrel/linux-kernel-module-rust - see tests/* in particular). I'd be delighted to see folks who are excited about other languages build something in their language, too.
I'd like to correct two specific things about your comment:
> if we discuss the amount of effort necessary for bringing new interfaces like the one this article mentions ("one can define a kmalloc_for_rust() symbol containing an un-inlined version"), we should compare it to other existing solutions, like ATS
Here is the amount of effort necessary to work around that:
I took a semester-long MIT class on a proof assistant (Coq) and I thought it was great but I'm still not fully comfortable with it. I realize that Coq and ATS are different languages, but I would humbly submit that if you compare the amount of effort spent calling krealloc instead of kmalloc vs. learning ATS, the former is probably lower.
> without giving up on existing C interfaces and the toolchain
Looking at Chapter 8 "Interaction with C" from the ATS book http://ats-lang.github.io/DOCUMENT/INT2PROGINATS/HTML/c2016.... , it seems like ATS's interoperability with C is pretty comparable to Rust's https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html#usin... - I don't think either of them require giving up on existing C interfaces or the C toolchain. One of the specific reasons to use Rust in this context instead of Go (which is a great language too) is that Rust is designed to fit in very closely with the C toolchain. The first paragraph of that chapter, which discusses how ATS's data structures can be zero-overhead bridged with C and you can view ATS as a better-typed frontend for C, seems to equally well apply to Rust.
(In fact, as mentioned in the article, we'd love to automatically generate bindings from existing C interfaces. It turns out that most interfaces lack documentation - certainly machine-parseable docs, but often human-readable docs too - that indicates their ownership properties and locking constraints and so forth, but definitely the best approach is to get this sort of information into C annotations instead of having manual bindings in Rust. As a bonus, if it turns out that another language like ATS or Sing# or whatever is a better fit than Rust, the information is right there.)
>The safety guarantees that Rust provides are neither unique nor complete,
Well no language can ever have "complete" safety guarantees. But I would like to know what other languages offer the same set of guarantees without GC overhead. This would include:
* the usual memory safety
* no nulls
* no undefined behavior
* no data races
Yes, ATS can guarantee all those things and more. In particular, ATS allows you to write "unsafe" code, which you then prove safe with a mechanically verified proof. This is in contrast to unsafe Rust, which depends on programmer discipline, and has no (intra-language) mechanism for verification beyond the basic type system. You could say that Rust is all-or-nothing, while ATS allows gradual verification, as well as proving safe much more complex code than you can in Rust. I'm not going to claim that ATS is overall better than Rust - it's not - but it's a far more powerful language than Rust for safe systems programming. It's just that most people are not willing or able to pay the very high price in ergonomics. I recommend watching this presentation for a good idea of what ATS programming is like: https://www.youtube.com/watch?v=zt0OQb1DBko In short, it's about as user-friendly as an overheating nuclear reactor.
Compilation in Cargo has been split into metadata pass and code generation pass. Metadata part has interdependencies, but is relatively fast (it's roughly like a C header generation). This allowed more codegen to be done in parallel.
To be honest, it was a modest improvement, because compilation has been decently parallelized already (compiler uses incremental builds, parallel codegen, and ThinLTO even within a single crate)
It's not about safety, if that was the real argument we should start with obvious improvements like checked C
https://github.com/microsoft/checkedc
The C++ lifetime checker or Cyclone
Checked C addresses a small number of memory safety problems - it addresses bounds checking / buffer overflows, but not use-after-frees, double frees, data races / insufficient locking, and several other things that fall into the general category of "memory safety," nor more general "safety" things like type confusion.
The kernel is not written in C++ and is unlikely to use C++, for many reasons - Rust is a more natural C-but-better for the kernel than C is.
Cyclone hasn't been maintained for over a decade; it was a research language, and it largely served its purpose as a research language in inspiring production-supported languages like Rust.
What kind of performance hit is rewriting part of the kernel in Rust going to entail when it comes to compiling the kernel? How many platforms will cease to be able to compile their own kernels? The article says there will be some, but doesn't give numbers.
More worryingly, the article also says that only some platforms will be "Rusted" in this manner, splitting the codebase into "mainline Rust" and "obscure platform C" as far as these sections are concerned; this seems like it will fracture the development community between people who know Rust and those who don't, and those who can work on mainstream platforms and those who can't. Fewer eyes on this code seems like the wrong way to go.
People underestimate the role of copyleft licenses in preserving long-running FOSS products like GCC, Linux, etc.