For once I feel a little out of touch after I took a bit of a break from following the news to focus on studying, and suddenly everyone is talking about immutable distributions. What are they exactly? What are the benefits and the disadvantages of immutable systems?
Have you used Android? Has it ever failed an update or break due to an app install in a way that can’t be fixed by uninstalling it or factory resetting? Android is an immutable Linux OS. Its system files are stored on a read-only partition. They’re only mounted read-write during update. (That’s a lie, this is no longer the case, but it used to be, these days there are two partitions and the whole inactive partition is written during an update, or a volume snapshot pretending to be a partition is created and then merged, but functionally it’s consistent with the lie.) Apps are also stored in read-only form. One implication of this is that upon update, the partition/files you want to update are always in a predictable, unchanged state. That guarantees successful updates. It also allows trivial diff updates. The other implication of these facts is that you can always delete the mutable part of the OS, where your data and the apps’ data is stored, and you’ll always end up with a clean, working OS in a factory state. On Android you can also do this per-app by tapping “Clear data”.
Wouldn’t it be nice if you desktop or server behaved like this? Some folks think so and are trying to implement it.
There are few disadvantages beyond having to change how some systems work to accomodate this model. There’s typically more space wasted.
Thank you for this explanation.
With tbis update system, I don’t understand why you can’t use your phone while doing it.
You can. Google pixel updates are just a reboot. Sadly many OEMs don’t do A/B updates, like samsung, so your phone can’t be used while updating the system partition
It’s worth mentioning that they often do this to reduce flash size, i.e. save 💰. Virtual AB was introduced to help with this but it’s relatively new. With it, there’s no need to reserve the space for 2x system partition. The needed extra space during update is taken from /data and released post success. There’s also a compressed virtual AB scheme now which helps reduce the space needed from /data to enable really space-crippled devices like CCwGTV and other Android TV things like Sony and other TVs.
the whole concept of immutable is focused on stability and safety of your system – yes, it is still possible to break an immutable distro, but it’s a LOT harder and takes some actual effort – there’s also a few concepts wrapped up into the “immutable” phrasing:
- immutable filesystem – the root filesystem is set as read only, updates are queued up and applied during an upgrade (some distros require a reboot, some don’t)
- VanillaOS keeps two copies of the root system (ABroot), upgrades the inactive copy and then swaps them out
- NixOS has everything defined in a master config file and keeps an archive of previous generations of the config file allowing you to boot into whichever generation you want
- atomicity – updates are applied individually and checked, if the update breaks then it’s reverted to the previous working state (ie. you are never left with a borked system)
- containerized apps – user space apps isolated or sandboxed in some way like Flatpaks or Docker containers or OCI so if they break, they don’t take anything else down with them
- declarative systems – the whole system (and packages and configs) are defined (declared) in one master config file – back up that config file and if something happens to your system, you just need that one file to do a full rebuild (or make an identical copy of your system on another computer) – NixOS and GNU Guix are the two more well-known in this space
- EDIT: minor side-effect of this is you can easily tell exactly what packages are installed on your system at any given time – no hunting through
historyor trying to remember what you installed last month when you were testing out video players
- EDIT: minor side-effect of this is you can easily tell exactly what packages are installed on your system at any given time – no hunting through
I like the idea of containerised apps, especially if you can have multiples of the same app.
A few years ago, when I was pretty much fully on Windows, I used PortableApps to manage a few websites and their emails, and having a simple way to do the same thing is one of the few reasons I’m still stuck with Windows.
On top of that, I like trying out new apps and programs, but uninstalling them doesn’t always get rid of the extras, like new dependencies.
- immutable filesystem – the root filesystem is set as read only, updates are queued up and applied during an upgrade (some distros require a reboot, some don’t)
My understanding is “immutable” is a bit of a misnomer and avoids the “point” of using these distros.
“Layered Distros” is a better terminology, where you can imagine the OS as multiple layers, and you can swap 1 layer out for another without modifying the others and still have a functional operational machine.
Now some of those layers have to be immutable ones at runtime for this concept to work, so thats where that part of the name comes from, but thats an implicit result from the actual point/use case of these distros, not the selling point.
So you can swap versions/releases of your OS very cleanly at boot, without modifying userland, and it will continue to function just the same. This lets you do stuff at the admin level like broadly releases a version update merely by having users just reboot their machines, and next time they boot up their machine will now be running on the new OS layer, with their local “user” layer being unchanged.
They also separate concerns better than classical distros. Executable binaries & libraries are separate from configuration which is separate from data. It makes backups much simpler, makes configuring new machines easier than something like Ansible, etc.
I especially look forward to replacing Ansible :)
Tried to do so with Guix though and I have to say I found it quite difficult, but I hope it catches on and becomes easier to use.NixOS is a tad simpler IMO
I’m not so sure about simpler. I really don’t like the Nix language and I’d rather use Guile, but the amount of packages and configuration options drew me to NixOS.
Immutable distros are locked-down versions of the traditional operating systems. Literally.
In the normal traditional world, you can use the package manager to remove python as a system package and all its dependencies if you so wished. You could rm -rf / too.
With immutable, the whole filesystem is on ro mode. Every system program that a user needs is bundled by the OS and no other changes can be made without breaking the model.
This model of having the OS immutable means less chances of malware getting persistence, high systems availability and reproducible environments since the OS controls the state of the versions available and makes this state available to all users of the distro.
####For an in the wild example, Steam Deck.
Steam Deck runs Steam OS, which is a heavily customized version of Arch Linux by Valve. But unlike normal Arch installs, Valve has locked the System partition(/) to read-only.
The reasons they’ve done this is two fold. First, there’s actually two copies of the system partition. The reason for this is that when an update is downloaded it’s actually written to the other partition, not the one you’re currently using. This way the update happens in the background, and then you just need to reboot for it to switch partitions and do some house cleaning. What this means though, is that any changes you might’ve normally made to the system partition, disappear, as they are now on the other partition you aren’t using. So if any changes you make won’t matter, not much point in letting people make them in the first place. Using Flatpaks, any applications users install would instead be stored in the user partition, and never touch the OS itself.
The second major reason for doing this, is consistency. If people are discouraged or blocked from modifying the system partition, then any bugs or issues that crop up are, 99% of the time, Valve’s fault. And because of that, Valve can more easily diagnose and try to reproduce any reported issues, because theoretically, the user’s Steam Deck should be configured exactly the same as one at Valve HQ. All that’s needed is for a user to report what they were doing when they encountered the issue, and Valve can follow those same steps and hopefully encounter the same issue, get detailed logs, and hopefully quickly push out a patch as needed.
And that’s one version of Immutability. Valve doesn’t go the full nine-yards here, just enough for their use case. In other Immutable Distros, versioning is taken even further, where you can control multiple versions and reset the OS state on the fly as needed, keeping any changes to a minimum and in controlled sand boxes. There’s a ton of use cases for these, but the most obvious benefits are for enterprise and mass-market solutions, where a single configuration is multiplied across a large amount of servers, or end-user devices, allowing for easier diagnoses of issues and pushing of patches. For end-user clients especially, if they aren’t expected to be customizing the OS to begin with, it makes support much easier for IT.
For your average Linux user, the benefits aren’t as large, as you’ll often want to be able to tweak things to your liking. But your Average Joe that just wants a computer that can surf the web and install some apps that can be found as Flatpaks, an immutable OS that they can’t easily screw up is a plus.
I just have a big shell script that will redo all my changes after updates. I only broke my install once :>
/jk
In the case of NixOS at least, ‘immutable’ doesn’t mean you can’t change the system at all.
It just means you cannot change the currently installed set of packages and services (generation in NixOS parlance); all you can do is create new ones and delete old ones.
Basically every update might as well be a complete reinstall of
/usr,/etcand whatnot if you compare it to traditional distros.If you can’t change etc, how do you configure your software?
Unlike a traditionally managed system, you manually keep an install list of all used system packages (including current version). It’s conceptually similar to how conventional package managers might have a mirror configuration list. Within this list, you’re also sprinkling in individual package options which Nix will automatically apply for you (the options that Nix “sees” are manually coded in by whoever did the bundling of that particular Nix package).
In other words: you’re treating packages as if they were configuration and bundling that in with the actual configuration under a unified framework. This basically means that you can think of Nix packages as wrappers bundling the base software plus instructions for a config file generator. That’s effectively the “gimmick” of Nix: config files are routinely regenerated, so in everyday use they’re conceptually equivalent to an abstract interface between the Nix package manager and the software proper.
Worth mentioning at this point is the fact that the core Nix ecosystem is all about managing system-level software; Nix handles sudoer stuff and steers clear of interfering with user-level dotfiles. Using the hyprland wayland compositor as an example, you can see that the module only provides flags for things like NVIDIA patches and allowing xwayland – options for shadows/colors/animations are omitted entirely because they should (under the Nix philosophy) be configured with user dotfiles.
Now with that being said, Nix itself is merely a package manager, so it’s entirely possible to splash in as a companion to your distro’s main package manager (similar to how you can globally install npm/pip packages). This is handy because Nix’s ecosystem is highly invested in enabling users to write their own packages (i.e.: Nix Flakes). In fact, some Nix users exclusively use it as a framework for creating and managing custom packages whenever the need arises to install something not already provided in their distro’s package repository.
Of course, the most vocal Nix users are all-in and use NixOS. With NixOS, Nix is your native system package manager. Everything system-level becomes Nix-managed, though the same leave-userspace-to-dotfiles philosophy still applies on the user level. This strategy enables cool stuff like seamlessly syncing your system environment between multiple workstations and having guaranteed safe restore points through the use of nothing but a handful of config files.
Beyond that, for the total Nix freaks, there’s the Home Manager Nix package, which allows you to install and manage userspace applications Nix-style. This is handled separately from Nix/NixOS proper because userspace configuration is a much larger and faster-moving target. If you use Home Manager to fully capture your userspace, it’s nigh-guaranteed that you’ll need to get your hands dirty by writing custom Nix modules. As you may imagine, this process can be a big pain… though it comes with the benefit of being able to seamlessly carry user applications & dotfiles across any distro that supports the Nix package manager (including Darwin).
Thank you for this comment! I’ve been flirting with Nix for a few months now, and this helped me understand a lot of the core concepts better.
By having the right configuration file there as part of the package’s options, like:
globalProgram.doFoo = true;or something likeglobalProgram.extraConfig = "barCount=4567";Are those changes system-wide or stored in user space? Where are those files stored?
It’s system-wide (unless home-manager is involved).
They’re a part of the immutable install, whose components reside in
/nix/storeand are symlinked to/etc.Example from my computer:
$ realpath /etc/sddm.conf /nix/store/slkq2k8vc4rx4ag55zf8ssl7qd9ry49v-sddm.conf
Using NixOS options
[This comment has been deleted by an automated system]
I love Universal Blue.
It’s OCI cloud image based Fedora Silverblue/Kinoite/Serica with extra steps/batteries included.
“The reliability of a Chromebook, but with the flexibility and power of a traditional Linux desktop.”
But also probably an easier way for
Nvidia Fedorausers to game on Linux:Easily roll back deployments or 📌 one and rebase to something else easy peasy. (So many different choices) Test betas with no fear!
I’ve actually been gaming on Bazzite for two weeks now:
Jorge’s Blog:
Media:
- High-Tech Lo-Life on YouTube:
- “As Seen on TechSpot”
- “As Seen on GamingOnLinux”
- “As Seen on tom’sHARDWARE”
If you wanna simply make your own image to share with friends/family:
Universal Blue isn’t a distro. It’s more of a reimplementation/enhancement of
ImmutableOCI Cloud Based Images of Fedora.People literally made a distro spin that’s dedicated to rolling back nvidia drivers.
Classic nvidia moment right there.
But Universal Blue does look very interesting, I need to try and use it with distrobox and see if I can hit any walls that aren’t there with a classic setup.
Nvidia is just a specific pain point, it’s nice to be able to roll back to a specific version of any given deployment.
It’s just more obvious for out-of-tree drivers since that’s usually a worse user experience.
It does take some adjusting- the pitfalls you’d encounter with Distrobox on Universal Blue are the same as Distrobox on any other distro, so first I’d say to try moving your workflow to Flatpak and Distrobox on your current system or a VM and see how it works out. Generally Flatpak is preferred to a rootless Distrobox which is preferred to a rootful one, but sometimes there’s not a Flatpak for something (especially command line tools) and you need access to hardware or system level stuff that only a rootful one can do properly.
Flatpaks are already my preferred way of installing random crap, but I did run into a few walls with that. VSCodium for example is unusable because it throws random errors about running out of space or not finding files that are definitely there even after giving it all the permissions via flatseal.
Proton has a similar thing where windows apps don’t detect the amount of free space properly and see 4GB instead, so I guess it’s inherent to containers.
I’ll definitely try distrobox on my arch machine, is there anything I need to consider beforehand to not shoot myself in the foot?
Not particularly, the workflow on your Arch system will be the same as any other distro, that’s the nice thing about Distrobox.
I would highly recommend looking into the
distrobox-assemblecommand, though: it lets you declaratively build distroboxes with the packages and config you need on them. I have a personal box which operates as my primary terminal that’s automatically destroyed and recreated on every boot. This way, the packages I always use in a terminal are available, and I can add something I need temporarily with no issue without worrying about forgetting about that package being there down the line and causing some weird update failure or general bloat.
uBlue is great. After using Fedora Silverblue for more than a year I used it to have the same OS on my laptop and desktop. It’s works great and is quite simple if your already familiar with building containers. But the constant reboots and rebuilding an image taking minutes made me switch to NixOS.
The advantage of uBlue over NixOS is imo that the former is configured like any other Linux by placing files in the traditional file system hierarchy (e.g. binaries in /usr/local/bin). NixOS throws most of that over board and makes use of it’s own configuration language and package manager. Getting started with uBlue is definitely easier, while NixOS is a time-consuming rabbit hole (not that uBlue isn’t…). For a tiling wm setup I definitely think NixOS is the better choice, since changing core system components is quicker.
Here is an alternative Piped link(s): https://piped.video/watch?v=aaeRk8_i1Ds
Piped is a privacy-respecting open-source alternative frontend to YouTube.
I’m open-source, check me out at GitHub.
As far as use cases where immutable distros would be more or less convenient, it appears to me that due to the security and reproducibility factors, immutable distros are better for server or enterprise environments where updates need to be rolled out quickly and smoothly, which are most actual Linux systems out there and where Linux companies make their money (hence why they seem like the hot new thing right now), and for desktops where people are primarily concerned with not wanting to “break” anything. But a classical distro, IMO, would be better for folks like me who do want the control to change or customize things down to the core parts of the OS. As far as I’m aware, you don’t have much freedom to tinker with an immutable distro until it’s wiped away with the next update.
NixOS/Guix still give users the control to change anything at any time. That’s because they aren’t image based and instead they achieve immutability with the Nix package manager (symlinks and a declarative system configuration file).
I agree with immutable OS on servers being great, but I also believe immutable systems are a good choice for desktops. Especially managed desktops (eg. my moms) work well with an image-based OS. Flatpak is often enough for those few gui apps and there’s less risk of automatic updates failing (eg. pulling power plug while updating).
My understanding is that most immutable distributions store the root filesystem in a read-only partition separate from userspace. If you want to install something that would modify this partition, it is applied temporarily until reboot. On reboot, the operating system attempts to resolve your changes, and if they break something it will reject them.
Each OS might implement immutability differently, but I believe that’s the general idea. macOS is another example of an immutable operating system, as of Big Sur.
Read only root and was they way unix was installed for decades. Some things, like nfs roots, somewhat rely on it (multiple machines would run a single static install of unix, with user partitions being mounted on demand on each machine).
People don’t like dealing with partitions though so modern distros tend towards a single root paritition with everything in it. It’s funny to go back to the old way.
Hmm… I didn’t know that. Thanks!
Security, Stability
TheLinuxExperiment has a good video about it: https://tilvids.com/w/3RjSzdS9jjK2y1nP3M6oJD
deleted by creator
Removed by mod
Honestly every time I mention NixOS I get someone telling me it’s not a beginner friendly distro
Provided you get to understand the config file which only took a day of fiddling it’s so much easier to use than mutable distros
I add something to my package list and it just works, and nothing else breaks and even if I do manage to screw up my entire system so badly that I can’t boot it anymore I just boot the last working configuration and undo my changes
I believe people have even built UIs around the config to make it more user friendly
https://github.com/nix-gui/nix-gui https://github.com/vlinkz/nix-software-center
I agree, when I was on other distributions I would gradually customize things until something broken; then it was easier to start from scratch than to fix my mistakes. With NixOS it still gives me the ability to tinker with things, but makes it much easier than starting from scratch when I inevitably mess it up.
I find it’s better than that, don’t even need to start from scratch because it’s so difficult to break it to the point you can’t just revert to a previous version
You know, there was Ansible, containers, ZFS and BTRFS that provided all the required immutability needed already but someone decided to transform regular machines into MIPS-style shitty devices that have a read-only OSes and a separate partition for configs. :D
TBF the unix model originally was read only mounted / and /usr - typically in a separate partition - so they were immutable for decades. It’s only later when home users started using single partition models that that really broke.
But the separation is built right in… /etc for config, /home for user stuff, etc. so there’s really no need for another layer… it’s not like windows were it’s common to store the configuration in the same directory as the binaries… so not really seeing what the current fad is about. Presumably something will come out of it, maybe even a standard…
Yeah I’m with you, I want more reliability but immutable is a bit of a long way to go to get it.
I get why people are willing to make the tradeoff though, docker taught us os configuration should be done programmatically, my workflow is just used to constantly fidgeting things till they feel right, to each their own.
I worked at the big cloud guys, I get the point.
But this isn’t a cloud host, this is a workstation that I’m happily tinkering with, I enjoy bending and breaking the os for fun and development.
My data is usually not on the workstation, so clearly it out is no real loss, and I get to experiment more.
