Why the Asahi Linux lead developer resigned

Let's take a step back.

I decided to begin this story in 2019, during the Linux Security Summit.

Developers Alex Gaynor and Geoffrey Thomas decided to hold a talk called Writing Linux Kernel Modules in Safe Rust. In it, they made the case for adopting Rust in the Linux kernel. According to them, that would improve security, as two-thirds of vulnerabilities were due to memory-safety issues.

A few months later, in June 2020, Linus expressed interest in this option during an interview with The Register. He said, and I quote,

[The kernel team is looking at] having interfaces to do those, for example, in Rust... I'm convinced it's going to happen. It might not be Rust. But it is going to happen that we will have different models for writing these kinds of things, and C won't be the only one.

Then, in August 2020, the Linux Plumbers Conference was held (virtually). The event's most attended session was about open questions on accepting Rust upstream in the Linux kernel. It featured both Thomas and Gaynor, but also Rust language team co-leader Josh Triplett and many others.

Finally, the Rust for Linux - or, in short, r4l - project was announced on the Linux mailing list. The goal is not to rewrite the entire kernel in Rust, but to allow new code to be added over time.

A few months later, Apple announced that they would be switching away from Intel x86 processors. I was quite shocked at the announcement, and the reviews quickly showed that we were dealing with a very well-performing device with a great battery life. Of course, there was no chance to run Linux out of the box on these.

In November 2020, Linus was asked what he thought of this new kind of device in the Real World Technologies forum. He replied,

I'd absolutely love to have one, if it just ran Linux... [...] And I don't have the time to tinker with it, or the inclination to fight companies that don't want to help.

Someone else picked up the challenge – Hector Martin. He was part of the Team Twiizers (later called fail0verflow), which allowed him to run unofficial applications on the Nintendo Wii. In 2016 he also ported Linux to the PlayStation 4, just after being sued for hacking the PlayStation 3.

However, he burned out. A major factor were users; quote,

Most people using our software just wanted to play pirated games (something we did not support, condone, or directly enable). We kept playing a cat and mouse game with the manufacturer to keep the platform open, only to see our efforts primarily used by people who just wanted to steal other people’s work, and very loudly felt entitled to it.

When Apple released the M1, he thought that could be a much more interesting challenge – there would be no entitled users who want to pirate software, plus no necessity for a jailbreak either.

He thus announced the Asahi Linux project, asking for funds to get it started. The name, by the way, comes from the Japanese name for McIntosh apple.

He started a Patreon with a 4k€ monthly donations goal, which was reached within days, and surpassed too. He now had both the money and the determination to bring Linux to M1 Macbooks.

When deciding what language to use to write drivers with, the project bet on the newly-announced Rust for Linux initiative and decided to go with that. According to Hector,

Rust is the entire reason our GPU driver was able to succeed in the time it did. We have two more Rust drivers in our downstream tree now, and a third one on track to be rewritten from C to Rust, because Rust is simply much better suited to the unique challenges we face, and the C driver is becoming unmaintainable.

Of course, this meant tying the long-term success of Asahi to the r4l project, at least to some extent.

In the short term, this bet paid off. In a couple of years, they were able to have great support for the M1 MacBook Air, and Linux 5.19 was even released by Linus from an M2 MacBook Air running Asahi.

Even better, during the Open Source Summit of 2022, Torvalds was interviewed by Steven Vaughan-Nichols of ZDNet. In the interview, Linus said that he would like the first signs of Rust infrastructure to be merged in Linux 6.0 and that he was cautiously optimistic about this goal.

Indeed, Linus was a bit too optimistic here. The merge request containing initial Rust support in the kernel was only merged in 6.1, and overall the r4l project was not advancing at the pace that Torvalds was expecting, or hoping, to see.

Indeed, during the 2024 Maintainers Summit (a 30 people invite-only event), Linus said that

nothing depends on Rust in the kernel now, and nothing will for some time yet. What is important is to make forward progress, so developers should "steam right ahead" and not worry about these problems for now.

Here's another important quote from Jonathan Corbet's report of the event:

Linus Torvalds admonished the group that he did not want to talk about every subsystem supporting Rust at this time; getting support into some of them is sufficient for now. When Airlie asked what would happen when some subsystem blocks progress, Torvalds answered "that's my job".

This is important, as Linus is directly taking some level of responsibility on this topic. Hector Martin was not part of the Maintainers Summit, but he did send an email to Linus to say he was available to talk, given his experience with using Rust in a high-profile downstream kernel. However, Torvalds never replied.

In a 2024 interview by the very same Steven Vaughan-Nichols, Linus expressed disappointment that the introduction of Rust was not going faster.

I was expecting updates to be faster, but part of the problem is that old-time kernel developers are used to C and don't know Rust. They're not exactly excited about having to learn a new language that is, in some respects, very different. So there's been some pushback on Rust.

This is an issue for the Asahi project, which relies on r4l to work. It's not the only one, though; Martin also started having issues with entitled users again.

According to him, users would repeat over and over the same questions and criticisms of the project: When is Thunderbolt coming? What about external monitors via USB-C? Battery life sucks compared to MacOS! …and so on.

If I may very quickly share a personal opinion in this otherwise somewhat objective piece, I would like to say that I've had the same issues before. I've implemented some features and received criticism for certain decisions behind them, or bugs that they caused; I've also received pushback when I instead chose not to implement this or that. This used to drag me down quite a bit, and it did have some impact on my mental health; I'm now starting to believe that it's part of what I do, and the feeling of being surrounded by entitled users is more a matter of how we community-driven developers approach our job rather than the world around us. But that's just my opinion.

Back to Martin, he also noticed that donations were decreasing month after month; the more features he implemented, the fewer donations he received. He interpreted this as a decreasing interest in Asahi.

Also, a lot of development time was spent rebasing Asahi Linux changes on top of the latest versions of the kernel. He tried to upstream code, but he found the experience to be "incredibly frustrating".

All of this was on top of some personal issues that he encountered in 2024, and which he prefers not to share publicly.

All of this gets us to this merge request, which aims to "add dma coherent allocator abstraction", in Rust. Now, I wish I could tell you what that means, but I haven't the slightest clue. The important part is that this was written in Rust (or at least uses Rust to some extent).

The patch was rejected by maintainer Hellwig, with the single sentence "No rust code in kernel/dma, please".

To be fair, Danilo, who had proposed the patch, did try to argue why the usage of Rust was justified here, and he did say he was willing to take care of the Rust abstraction that it needed to work.

Hellwig again replied with something along the lines of "don't force me to deal with Rust", to which Daniel replied with "We're not asking you to, we're doing an abstraction as you asked us"; Hellwig did not further reply, which Daniel interpreted as green lights from him. Other developers also pitched in saying that the level of abstraction seemed reasonable.

A couple of weeks later, Hellwig again replied by blocking the patch, and specifically saying that Danilo was trying to "spread this cancer to the core subsystem", where "cancer" refers to using multiple languages in the same codebase.

And, a few emails later, he explained that he opposed crosslanguage codebases (and, thus, the r4l project as a whole) so much that he would "do everything [he] can do to stop this". This seems to be a good example of the maintainer issues that Linus was talking about.

As you might've noticed, this entire thread went on without Hector Martin pitching in. However, as mentioned, the Asahi project relies on r4l, and this patch was needed for two of the three drivers they developed.

Martin expressed his frustration on Mastodon, pointing out that the "I will do everything I can do to stop this" can be interpreted as an attempt to sabotage the r4l project as a whole, and that he considers the usage of the word "cancer" to refer to r4l a Code of Conduct violation.

Now, he was deeply criticized for this post, and he has deleted it since then. However, I would like to defend him here. Hellwig words seem to be extremely hostile towards the r4l project, which has existed for years now with the explicit support of Torvalds. Though I might not agree with the wording Martin chose, or the Code of Conduct accusation, I believe the frustration is fully understandable, and ranting on his personal social media account is normal here. I don't take issue with this specific choice of his.

And, to be clear, he also sent a more detailed reply to the mailing list. He also added Linus to the thread – a fair action, I think, given how he has some responsibility over this development. He also suggested to Rust folks to "not waste your time on drama like this".

Now, this is not the first time Hector Martin has vented on Mastodon about the pushback from certain developers to the r4l project; as an example, here's another post that is pretty, uh, strong:

In the patch list thread, a few developers argued that this sort of attitude is toxic too. To quote them,

To back up Sima here, we don't need grandstanding, brigading, playing to the crowd, streamer drama creation or any of that in discussions around this. [...] Being toxic on the right side of an argument is still toxic.

And, finally, as a response to these last few messages, Martin replied with a exhausted message, explaining all of his frustration with the current processes in the kernel. He complains about outdated tooling, nitpicking, unnecessary delays, bikeshedding, and more.

This is where Torvalds stepped in, but he did not say a word about the patch itself and only commented on this last vent by Martin. To quote him,

How about you accept the fact that maybe the problem is you. You think you know better. But the current process works. [...] The social media brigading just makes me not want to have anything at all to do with your approach.

I feel like this accusation is somewhat unfair. Hector Martin did not organize any kind of brigading, which is when a community purposefully targets another space to damage it, flood it, or otherwise change its working. This did not take place at all: Hector Martin shared his frustration on social media, which I believe to be an understandable reaction. Linux news sites picked up on the accusation and some buzz arose, but nothing was organized, nor was the patch mailing list hijacked. Linus is pretty much saying, if you have a problem with how we do things, you are not allowed to vent about it on social media, otherwise I don't want to have anything at all to do with your approach.

Evidently, this last exchange raised some thoughts in Martin, who - given all of the context that I've given you, and not just this discussion specifically - decided he was no longer willing to give up his free time and hobbies to work on Asahi. He stepped down to seek better mental health, which is something I respect.

He has closed his Patreon and OpenCollective and asked his GitHub sponsors to unsubscribe from him.

However, the Asahi Linux project won't die off. A team of seven people - including, by the way, Neal Gompa, hi Neal! - will continue his work, focusing on upstreaming changes, implementing DP alt mode, sparse images, and getting internal microphones working.

You can donate to the Asahi Linux project directly, as they have now set up their own OpenCollective account that is independent of Martin. They say,

Our community will miss him. Still, with your support, the project has a bright future to come.

Before I go, did you know that this video took hours to make and yet it is not sponsored? I also have to pay servers, equipment, and collaborators – which renders this channel a net negative on my finances, right now. You can help me change that by donating on Patreon, Liberapay, Ko-Fi or PayPal – links in the video description.

But, you can also subscribe to or become a member of thelibre.news, the website which contains all of these videos but in an article format. There will soon be some extra content over there, I'm working on it.