I want to reset my server soon and I’m toying with the idea of using a different operating system. I am currently using Ubuntu Server LTS. However, I have been toying with the idea of using Fedora Server (I use Fedora on my laptop and made good experiences with it) or even Fedora CoreOS. I also recently installed NixOS on my desktop computer and find the declarativeness pretty cool (but I’m still a complete beginner) and could imagine that it would fit well into a server setup.
I have quite a few services running on my server, such as Nextcloud, Conduit (Matrix), Jellyfin, etc. and all in containers. I would also rather not install programs without containers, because 1. compose is super easy to maintain and set up, 2. it remains very clear with containers (and compose) and 3. I believe that containers are more secure. But since I also want to make the services inside the containers available, I currently have Nginx installed as a reverse proxy (not in the container, but on the system) and always create certificates with certbot so that I can use HTTPS encryption.
In the paragraph above I actually described exactly the use-case of Fedora CoreOS, but I have no experience with the system and how it works. That’s why I’m still a bit hesitant at considering the OS at the moment. I can imagine that NixOS with its declarative nature seems well suited, since, as I have heard, you can configure containers as well as Nginx and with Nginx also https certificates declaratively. But I could also use a base system like before (Fedora Server or Ubuntu Server) and simply install podman, nginx and certbot and manage everything that way.
Have you had any experience with Fedora Server, Fedora CoreOS, NixOS or a completely different operating system for servers and what are/were your impressions with this setup? Or do you just want to share your knowledge here? I would be delighted.
My $0.02:
NixOS is excellent, and actually pretty easy if you’re not trying to do anything fancy (running all services under a single user, etc.). Personally this is my pick because I primarily host services for myself, so down time in exchange for learning a new thing is acceptable.
As I mentioned elsewhere, Debian + Incus is a great minimal and rock solid solution for longer standing services. Although, it’s not
composeable :(More directly to your preferences, I would also recommend considering Rocky. Being in the RHEL ecosystem has its perks (especially with rootless support for
podmanandpodman-compose). I’m also generally a fan of SELinux. Rocky is a little less bleeding edge than Fedora with many of the same conveniences and recent packages. In my mind, for my purposes, that makes it a better choice than Fedora for a server OS.I use Fedora CoreOS on my homeserver and a bunch of VPSs. Migrated the homeserver just recently, but I’ve migrated the first VPSs a bit more than a year ago. So far, I had no problems with it. There’s a low-traffic mailing list where the devs inform about security issues and breaking changes to the whole container stack.
I used debian before for some years, but at some point became tired of manually updating the system (which is probably one of the biggest benefit of FCOS). It takes, however, quite some time to put your first Ignition config together, and debugging is tedious as you have to redeploy to see if a bug / error is now gone (I’ve used a VM for that).
I use podman on some, Docker on other servers (you can’t use both at the same time). Both have been working well so far.
I’d recommend it, but would also recommend taking a look at Flatcar Linux which is more or less the same without the IBM dependency (which makes my stomach hurt sometimes).
I’d recommend it, but would also recommend taking a look at Flatcar Linux which is more or less the same without the IBM dependency (which makes my stomach hurt sometimes).
Why exactly are the IBM dependencies a problem for you?
I used debian before for some years, but at some point became tired of manually updating the system (which is probably one of the biggest benefit of FCOS). It takes, however, quite some time to put your first Ignition config together, and debugging is tedious as you have to redeploy to see if a bug / error is now gone (I’ve used a VM for that).
I can’t really find good resources on how FCOS is working and what are the benefits. Is it updating the system/kernel automatically as well as the containers? And what are generally, in your opinion, the advantages of FCOS?
Why exactly are the IBM dependencies a problem for you?
I guess I just like independent, community-driven distros, since there’s less space for financially motivated enshittification. Just shortly after I decided to go with FCOS, RedHat / IBM decided to close down CentOS, for example.
I can’t really find good resources on how FCOS is working and what are the benefits. Is it updating the system/kernel automatically as well as the containers?
The system & kernel yes. The whole system is basically a read-only system “image” for which the devs make sure all the packages play nicely together. Packages are not updated individually, but whole system “image” are released periodically, which the system then downloads automatically and reboots (you decide when it actually reboots through the config). If anything goes wrong, the system is rolled back to the previous “image”.
When you go with podman, there’s a systemd service you can enable which will update the containers (i.e. pull the specified image tag). I’m not aware of a similar mechanism for Docker, which is why I use watchtower for that which has been working smoothly so far.
Edit:
And what are generally, in your opinion, the advantages of FCOS?
For me, it’s the (quite safely designed) auto-updates of the base system (I just feel like having to do less repetitive work), infrastructure-as-code aspect, and the container mindset (as I containerize everything anyways). Also I just have a weakness for new, fancy stuff.
Thanks, I might give it a try. I am not settled yet but FCOS sounds very promising. We will see.
Debian
You might want to consider just Dockerising everything. That way, the underlying OS really doesn’t matter to the applications running.
I’ve got a few Raspberry Pi’s running Debian, and on top of that, they’re running a kubernetes cluster with K3s. I host a bunch of different services, all in their own containers (effectively their own OS) and I don’t have to care. If I want to change the underlying OS, the containers don’t know either. It’s pretty great.
See you back on Debian in a few months
Can’t be hit by new backdoors when your packages haven’t had updates for years 😉
In all seriousness Debian makes solid choices that makes everything as low maintenance as it can get for self hosting.
For someone who recently lost a bunch of their free time, that is amazing to not have to mess with stuff.
deleted by creator
Proxmox is just Debian 12 with additional software preinstalled
deleted by creator
Or, better yet, LXD/Incus.
I really don’t see any advantages in your post for choices other than NixOS. I’m sure you’ll improve quickly by necessity! :D
Give Gentoo a shot. It’s super stable and you will understand everything in your system. Also it now supports binary packages
And by now you mean for the past decade at least.
Huh?
Portage has supported binary packages since forever, back in 2012 I had some binary packages on my system, I clearly remember because it was a pain in the ass to compile certain things, for those I installed the binary version. It’s like Debian supporting source packages, it’s been there since forever but people don’t know about it.
The point is that they have recently focused on better binary package availability. Sure they always had support for binary packages but most software needed to be compiled.
I mean it’s had -k/-K since mid 2000s from what I remember but it’s changed
Proxmox.
Each service becomes an LXC. Docker containers can be migrated to LXC, or be contained within an LXC dedicated to docker.
Running out of processing power? Add another server, add to a cluster, and migrate services (LXC or VM) over.
Having run Fedora, Debian, Ubuntu, CentOS, RHEL, slack, even Oracle Linux - Proxmox is what I run for myself (and some clients).
I don’t know if the use-case you describes fit into my problem. I only have one server and its a physical server. I’m also not really able to extend the number of servers, as I don’t really have the budget.
Proxmox is a server OS based on Debian which is oriented on running virtual machines and Linux containers.
The physical server runs proxmox. The services can all be individual containers (LXC’s).
Adding to the number of servers (and migrating containers later) is a benefit of Proxmox, since you can buy another PC to be a server later, and easily expand as you go.
I tend to not use the webui, so I prefer the similarly useful combination of Debian + Incus (spawned from the LXC project).
Sure, HA isn’t baked into Incus (to my knowledge) but similar to OP I only have one physical box and don’t necessarily care to manage multiple.
That being said, Proxmox is a good solution in the scheme of things and generally a good recommendation.
I’ve got a small fleet of tmm’s, so HA is just practical for me, but yeah that works to with a single machine. Especially if you were sharing desktop use on it.
Slackware is a great, simple OS that does what it does and does it well. There will be some getting used to, but when it clicks, it makes sense and doesn’t do anything you wouldn’t expect. It is great if you want to use containers as it provides you with the stable, simple base to run all your containers on top of.
You next OS will be… Debian. Because you care about your time and you want stuff to be stable.
I really really like Fedora Server, but any RHEL derivative is my go to for servers. I use Rocky Linux when I need something closer to RHEL, and Fedora server for pretty much everything else. I highly recommend Cockpit as well (main reason I like Fedora server) as it has allowed me to so easily manage all of my servers from a single point.
I’m using FreeBSD now and I have been blown away at how well it just works and gets out of your way. I am using appjail templates to script containerisation of my services
Yes yes yes. It’s great to see other FreeBSD fans here with the same opinion.
I was using Debian as a server OS for more than twenty years with short escapades to other distros but then I discovered FreeBSD and there was no way back. ;)
What services do you run on FreeBSD? Does using FreeBSD limit you in the number of apps you can have, as most of them target Linux?
I am also curious. FreeBSD is, in my opinion, is such an unorthodox choice.
If I remember correctly when Microsoft bought Hotmail years ago, it was run on FreeBSD and SUN Solaris (And it took Microsoft a really long time to migrate it to Windows servers, but that’s another thing).
Netflix is also hosted from freebsd
No, I haven’t found anything that I haven’t been able to host.
I have Jellyfin, silverbullet, nginx web server with certbot etc, java game servers, samba and nfs shares, syncthing, qbittorrent, etc.
Could you explain more about how you found FreeBSD to be superior to Debian for a server OS?
Anything you want me to touch on specifically?
Semantically superior OS components, performance, QoL improvements in networking and storage stacks please
Fedora Server works well, and the Podman integration is great.
I guess it is the boring option, but probably the best when coming from Ubuntu.
I went with Fedora on my VPS because I was also planning to use rootless Podman. Quadlets and running everything through systemd with SELinux enabled is working pretty well for me.
always . freaking . debian
Do you have any reasons for wanting to switch your server OS, or is it more to learn something new? Either way is fine, but it might change what is more interesting to you.
I used centos forever, but only recently started slowly migrating everything to NixOS. I use NixOS for the OS and a few common things like VPN, monitoring, etc. For all of my actual services, I deploy them using Hashicorp Nomad with docker.
I’m not sure i would recommend defining docker containers using NixOS. It’d be fine for a couple servers, but not great for a cluster where services can move around.
