Why OBS and Bottles got in a fight with Fedora

Let's start with Bottles, and then ease to OBS.

Bottles is an application that allows you to run Windows software.

It relies on Wine, thus the name, and it makes it easy to create environments with specific Windows prefixes, settings, and dependencies. It lets you select various components to run those applications and comes with an integrated package Windows manager.

Many people bring the project forward, two of them - which you will often see in this story - are Hani Rana, a.k.a. theEvilSkeleton, and Mirko.

Now, Bottles does not have a set release schedule; on top of that, its dependencies change really fast, sometimes they are even custom-made, and a reliable environment is necessary to triage any issues that might arise.

Due to these reasons, packaging Bottles has been a somewhat complex and time-consuming task; and, due to not enough developer time around it, the Fedora RPM package of Bottles quickly fell out of date.

This led users to continue reporting already-addressed bugs to Bottles developers; and, on top of that, the packaging - according to Rana - was done incorrectly, introducing some bugs of its own. According to them, these bugs account for the majority of reports to Bottles developers.

This causes many issues for Bottles developers; not only do they have to deal with an increased bug-triaging workload, but their brand is damaged by these issues. As a direct example, The Linux Cast wanted to do a video about the application, but - due to an RPM packaging issue - he was unable to even start the application.

Oh, and this is not relevant to the story, but Rana points out that "In the example I provided above, we were really lucky for being aware of it, only because we were pinged".

And that's me! I pinged them! I'm in this story, too!

Oh, and let me quickly mention that if you prefer reading over listening, this entire video is also available as an article at thelibre.news; link in the video description. But if you prefer the video, keep watching.

Anyhow, as a result of all of these issues with packaging, the Bottles team asked Fedora to remove the RPM package, so that Bottles would only be shipped via flatpak. This would also improve security, as Wine can introduce some vulnerabilities that are addressed by Flatpak's sandboxing.

Though the Fedora people maintaining this project initially agreed to retire the package, they had a change of mind and decided to make sure it was not outdated for the next major release, and then to start looking for other maintainers for the package.

The Fedora team here is trying to say: "we know how to deal with irregular release schedules and changing dependencies; we saw interest in this package, so we'd like to keep working on it".

According to Fabio, from the Fedora side, the reason that the Bottles package was out of that is that one of Bottle's dependencies, orjson, could cause some segfaults, and they were trying to push a fix to the project. To quote him,

Again, Bottles developers might not care, but this is the reason why linux distributions exist. Because *we need to care* about these nitty-gritty low-level things to *make things work*, even if upstream developers don't (or cannot).

However, Bottle developers quickly asked Bottles not to be shipped as a Fedora package; understandably, they had lost trust that Fedora could keep up with the quickly changing requirements of the project and preferred the stability that Flatpaks was giving them.

Another point that they raised is that the Fedora maintainer for the package was sort of asking the developers to work with them, to some extent, to make sure the package was done properly. This was not something Bottles developers had the time or interest to do.

However, it seems like these requests did not stop Fedora from still intending to ship the Bottles package:

As a result, Mirko decided to retire the AUR package as well, and make sure to disable support links for unofficially distributed Bottles version, along with a warning and request to use the official one. On top of that, he says that "in the event of the removal of these controls or falsification of the product's identity, we will consider implementing restrictions on Bottles repositories for unofficial packages".

Personally, I felt like the Fedora person here missed the mark of the comment by replying with, and I quote,

The above sounds to me like it's one step short of moving to a restrictive none FOSS license. But, seeing that most of you are involved in open source in some way or another, I'm sure you are aware that doing so would close the doors for your audience.

That was the end of the story; Bottles developers did end up feeling highly frustrated and disappointed in their experience with Fedora packaging. However, there are a couple of other examples to bring up concerning this, but they require some additional context.

You're probably aware of flatpaks and their sandboxing benefits. I believe most people associate them with Flathub, where most flatpaks are distributed.

To get an application on Flathub you have to respect many guidelines, with the goal of "ensuring that the applications hosted on Flathub are safe to use and integrate well in the desktop experience.".

On top of that, some Flathub applications are verified, which means that the official developers (or authorized third party) claimed ownership of that specific flatpak. This tells you that you're downloading an application as it the developers meant it to be.

Though Fedora offers both Flathub and RPM packages, they also maintain their own flatpak remote, called - unsurprisingly - Fedora Flatpaks. Though it was meant to be used on Fedora, it can also be used on other distributions.

Now, would it shock you to discover that the announcement that I've just shown you comes directly from Hari Rana, the Bottles developer? The Open Source world is a small place, I guess.

Fedora Flatpaks has a few differences from Flathub. Firstly, they do not allow for any proprietary application to be shipped there; all flatpaks are fully open source.

Secondly, Flathub does not impose any kind of limitation on which source the flatpak re-uses to be shipped; this means that sometimes it's a deb package, sometimes an appimage, or a snap package, or an archive… Fedora Flatpaks all consistently re-use RPM packages.

According to Matt, the Fedora Project Leader, Flathub rules are quite lax and there's not much of a review, without checking that it was built with good practices. Fedora Flatpaks are instead built with their secure pipeline so that the user can have more trust that the package is going to be what it says it is. By the way, he says this in a 40 minute interview with Brodie Robertson, which I would suggest that you go check out.

As a result of all of this, according to Neal Gompa, a few distros are considering to use Fedora Flatpaks over Flathub.

However, this new system also has its own issues; namely, sometimes applications are shipped with more bugs compared to upstream.

Take Fractal as an example. Recently, a missing dependency issue went completely unnoticed; as a result, the Fedora Flatpak version lost the ability to see images.

As a developer points out,

The upstream Flatpak on Flathub was properly tested by the developers, and included all the components needed for correct functionality. During the period when this broken Flatpak was available, a number of people came to the Fractal matrix channel looking for help. The response given to them was almost universally "The Fedora Flatpak is broken, switch to Flathub and report a bug to Fedora".

This is not an isolated example. Fedora codec policies also mean that Kdenlive and media players will not be able to edit or play media, without a clear indication on why this is happening in the first place.

A similar situation was happening with OBS, the program that I'm using right now to record this video.

The Fedora Flatpak version of OBS is shipped with multiple issues: missing hardware acceleration, crashes on X11, VLC plugin not working out of the box, and they also included third-party plugins out-of-the-box.

On top of that, due to either technical or legal reasons, that package also lacked a Twitch/YouTube panel.

The above issues (and many others, such as this one) were regularly reported to OBS directly, which often had to close them and point out that the Fedora Flatpak version of their program was unofficial.

Over time, the usage Fedora Flatpaks raised some concerns. As an example, one developer - Jordan Petridis - decided to propose GNOME Software to always use the verified Flathub application over any non-verified one. In practice, this would mean that if an application is shipped to Flathub by the developers themselves, then Software would always install that one, instead of using other remotes such as Fedora Flatpaks. This change was eventually accepted and merged.

Another discussion that's still ongoing is the ability to set a default remote to download applications from. Right now you can only select it on a per-application basis, and there's no UI way to say: hey, I always want to download from Flathub instead of Fedora Flatpaks.

Oh, by the way, KDE Discover does have this functionality out of the box. This does not matter, but I just had to point this out. I'm a KDE developer if you didn't know.

Oh, and since I drifted off topic anyway, let me mention that this video is not sponsored by anyone, and yet it took time and money in equipment and contractors to make it. You can help support this channel via a donation on Patreon, Liberapay, Ko-Fi, and Paypal. Anyhow, back to the topic.

The discussion did not just happen over GNOME, but an issue was opened in Fedora too, asking to "Deprioritize or remove Fedora Flatpaks, and prioritize Flathub in GNOME Software", due to "quality issues when using Fedora's flatpak apps".

One of the people who disagreed with this proposal was Yakoov Selkowitz, also the maintainer of the OBS Fedora flatpak. According to him,

Many of these complaints would apply equally to the RPMs. I see this as no different than the Bottles fiasco from earlier. If software is FOSS then by definition we must have the right to redistribute it as we please, and upstreams cannot restrict that and remain FOSS. As such, I don't think we should need to entertain any such "requests" by upstream to not distribute "their" software in whichever form it may be. (Personally, I take a very dim view on such "requests".)

Which, by the way, is why I started this story by talking about Bottles: I believe you should now start to see a pattern emerging. Another interesting quote from his message is:

Based on what I've seen in discussions along these lines, what we have is a new era in which upstreams now believe that, thanks to Flathub etc., they don't need distributions anymore for their software to reach users, no longer see the value in distributions, and simply wish to cut out the "middleman" entirely. However, that does not give them the right or power to do so.

The discussion continued for pages and pages; just a couple of weeks ago, developer Michael Catanzaro said that, after a long discussion during a Workstation Working Group meeting, there was some level of agreement that "Fedora Flatpaks are generally the worst software source for users to install" and that "Good Flathub should be preferred over Fedora Flatpaks".

During all of this, four weeks ago, Joel Bethke from the OBS project created an issue to ask the Fedora OSB flatpak to be either removed, or to make it clear that it is a third-party package. Given the issues presented above, this can only seem like a reasonable request.

Discussion about this was happening in the other thread I mentioned, but it was more general. Around the same time, Georges Stavracas also raised the issue, saying that overriding the only official version of OBS with a Fedora-specific flatpak was unacceptable.

However, in all of this, all the issues of the OBS Fedora flatpak were being reported to OBS, not Fedora, which meant that Yaakov was completely unaware of the issues; thus, he just asked for those issues to be raised to Fedora, a request that Stavracas deemed absurd.

Eventually, when told that Fedora does have the right to redistribute OBS, Stavracas asked for a rebrand to make sure it was clear that the project was unofficial. Overall, this came out to be yet another bad experience with Fedora packaging.

As a final bit of context, during the conversation on whether to deprioritize Fedora Flatpaks in favor of Flathub, it was pointed out that OBS was currently relying on a EOL (End Of Life) version of Qt, the graphical toolkit to display the interface due to regressions with more modern Qt versions.

My understanding is that Fedora Flatpak maintainers decided that this was unacceptable, and decided to ship OBS with a more recent Qt version; one one hand, I can understand the choice to avoid End Of Life runtimes; on the other hand, you're intentionally shipping something that was deemed too broken to work by OBS developers.

As a result, Michael Catanzaro pointed out that allowing this to happen was terrible maintainership on OBS side, and showed that Fedora Flatpaks did have an added value compared to Flathub (since it used a more up-to-date Qt version).

The words "terrible maintainership", plus Bethke's request to remove the OBS flatpak getting completely ignored, prompted OBS to take stronger measures. They thus threatened Fedora with legal action if they kept using their branding for the unofficial flatpak, asking for a response within 7 business days.

It's worth noting that the very first like that this comment got was by Hari Rana, who had been previously disappointed by Fedora handling of upstream "remove us" requests.

Immediately Neal Gompa, the maintainer of the RPM package of OBS, reached out to the project to try to do damage control.

He created an issue to remove the obs-studio flatpak from the registry, immediately. This raised some discussion on how exactly to do that: if you just remove it, it will still stay on the users computers, and the only result is that their version will slowly go unmaintained until they delete and re-install the application. However, this discussion was finally taking place.

The option to rebrand the flatpak to avoid using any of OBS branding was also briefly considered, but ultimately none of this ended up being necessary.

As another way to immediately address the situation, Yakoov marked the OBS flatpak as End Of Life, within hours of OBS's request.

Nonetheless, a great amount of discussion was raised in the thread. Hari Rana pointed out that previous response on these issues were "shallow and completely inconsiderate of our point of view", and specifically mentioned Yakoov's "unwillingness to cooperate until the trademark enforcement occurred".

Bethke agreed with Hari, mentioning a "lack of willingness to work with upstream".

Yakoov decided to apologize for the lack of responses in the initial thread and offered to have a direct talk with Bethke. This seemed to have worked, as he then retracted his request to remove the flatpak immediately and said that he felt like now he could work with Fedora to tackle the issues presented in the Fedora OBS flatpak. He said, and I quote,

The discussion was positive and they are actively working to resolve those issues as well, which should hopefully only affect a small number of users.

Thus, I can only praise the good damage control of the Fedora project here, especially - I think - by Neal, but also Yakoov and others. Of course, it would've been better to avoid this conflict altogether.

This does not solve the broader issue by any means, though. The Fedora Flatpaks still exist, discussion about them is still active, and it's not yet clear when users should be directed towards them, and when they should use Flathub instead. Matt, the Fedora Project Leader, has stressed that Fedora cares a lot about good collaboration with upstream projects, and I can believe that these were just a few exceptions where the discussion got heated; however, I'm still not fully comfortable with a Fedora maintainer saying that upstream does not have the right to ask Fedora not to re-package their application. I don't like that.