It's been fashionable to hate on systemd, but I definitely prefer it to the hodgepodge of inconsistent init scripts, logging hacks, and bespoke process monitors that preceded it.
init.d was familiar to graybeards (like myself), but I'm loathe to consider them a superior solution.
100%. My toes hurt every time Lennart Poettering steps on them, but every time I dig into the "why" it turns out to not only have been a good reason but a couldn't have worked any other way reason. If someone hadn't dragged Linux kicking and screaming into the 21st century, the arbitrary choices made in the definition of a config file format 50 years ago would cripple the ability to deliver basic features 50 years later in a manner that was robust/isolated/performant/etc.
I've been looking for these for years: the reasons that might justify the removal of basic functionality and interoperability with other tools. Got any sources?
From what I can see, the dislike exists for far more practical reasons.
In my experience, it has a tendency to break in ways that the numerous other init systems I've used over the decades never did.
When it does break, it can easily prevent a computer from booting, which makes that computer pretty much useless until the problem is fixed.
I've found that it has often been far more difficult, and takes far more effort, to debug and fix problems with it than with other init systems.
Its tendency to become more invasive over time has also opened the door for even more problems involving it, well beyond its role as an init system.
Given the large number of bug reports, questions, complaints, and so on about it that I've seen over the years, people don't dislike it just to be "fashionable". They dislike it because it's making their lives unnecessarily painful.
I've liked it from the beginning (coming from upstart), but I think my one frustration is that you can't run systemd in an unprivileged container— and this is not simply an implementation gap, it's an intentional design choice:
You can argue that a process manager for an unprivileged container should be more like supervisor or systemd user services. However, it would be really convenient to be able to globally install unit files (say, using a deb package), and have them work whether it's bare metal, a VM, or a container.
I started on initrc but came to maturity on systemd, and though systemd does simplify the process of inspecting running tasks, its sheer lack-of-fun of getting those processes running in the first place always puts me off hacking my system to better suit my whims.
It forces you to think of stability instead of creativity, and that might be great for people who value predictable things in their lives, but it stands rigidly in the way of fun.
It’s an init system and a process manager. How exactly can it be fun?
Technically, I guess it’s interesting to design if you are an expert but as a user, having to use creativity is the last thing I want from low level pieces of my OS. If you want to tinker with OS design, you can always install Minix in a VM.
> It’s an init system and a process manager. How exactly can it be fun?
I don't know about fun, but different people find different things fun.
I find systemd to be often irritating, and sometimes infuriating. It has not made my life easier at all. Just the opposite. So, in that sense anyway, I'd say systemd is much less fun.
As someone who had to mess with init scripts for a couple of distros for someone who requested it, this was not far from my idea of pure anti-fun.
Every distro did init scripts in different, exciting ways, so I had to start over from reading the not-inconsequential docs for each one. The init systems come with some high-level helpers, but only if you want to use the shell, which is very unfun in its own right.
"Operating Debian without systemd is a pleasure and every time one of my friends has some systemd-induced lossage I get to feel smug."
How common is a systemd-induced loss? I've never had one, and I've never seen or heard of any. Is this a larger problem that I've somehow missed? (This is an honest question, I was just surprised to read this)
I've never had data loss (assuming that's what "lossage" means), but here's an incomplete list of problems I've experienced on systems running systemd:
* System randomly taking 5+ minutes to boot
* Daemons sometimes not starting on boot
* Logging sometimes just stops (I confirmed this was caused by a systemd bug)
* Logging in over SSH starts taking 2 minutes and can only be fixed by reboot (I confirmed this was caused by a systemd bug)
It's not that sysvinit scripts are without bugs, especially race conditions and daemons inheriting environment when you invoke a restart script, but systemd appears to be distinct in that a seemingly-quiescent system can just suddenly stop working correctly. They had a low bar to clear to improve on the reliability of sysvinit systems, and yet they failed to clear it. That's not surprising given its excessively complex design.
"systemd-analyze blame" can help identify issues relating to boot time
* Daemons sometimes not starting on boot
This most often occurs for me when services don't start quickly enough (too much contention on spinning rust and aggressive default timeout settings) so I have to adjust startup timeouts. "systemctl --state=failed" along with "systemctl status X" can usually help narrow down why your services failed to start.
* Logging sometimes just stops (I confirmed this was caused by a systemd bug)
I've found journald can get forcibly restarted if there is a lot of contention for disk iops and it can't checkpoint. I frequently run into this issue on a server that is used for backups for about a dozen or so clients that relies on filesystem snapshots.
* Logging in over SSH starts taking 2 minutes and can only be fixed by reboot (I confirmed this was caused by a systemd bug)
I've sometimes had this happen and for me it seems to be related to systemd-resolved deciding to stop responding to queries and the only solution I've found so far that actually works is to restart the system.
Despite the bugs I still prefer systemd over sysv-init - being able to run something like "systemctl status" and "systemctl --state=failed" to inspect the state of all services on a system has been invaluable to me.
> I've sometimes had this happen and for me it seems to be related to systemd-resolved deciding to stop responding to queries and the only solution I've found so far that actually works is to restart the system.
I hit this all the time on my Arch laptop and was a major impetus to switch to Void.
I really wish I could reproduce the thing on demand. I'd file a bug report but it has happened only a handful of times to me in the past 2 years and only on a single server. :-/
Heh I absolutely love Arch, and probably would never have made it to the point of being able to do Void or FreeBSD installs if it wasn't for Arch and its excellent docs. But yeah, some kinda annoying stuff with it lately.
Hackers used to say "win" to mean succeeding at some technical task, particularly if you gain some desired advantage, and "lose" to mean failure or being put at a disadvantage. Windows 11 is a losing operating system. Tree-sitter is a huge win for code editors. Etc.
"Lossage" means "the consequences of a lose" even if no data loss occurs. Some people consider systemd with its complexity and tightly coupled components to be a lose.
I'm still livid at this Luca guy gaslighting people on the "correct" behavior of a feature people were using for years already. They re-purposed an existing sleep mode and changed it entirely to mean something else. Originally you would set a timeout that your laptop would suspend and then hibernate. They changed it to mean that your laptop suspends until your battery hits 5% and then hibernates. As if people want to open their laptop the next morning only to have no battery left.
Ironically, the stated reason for this change was to "prevent data loss". Guess what they did? They introduced a bug in this "fix" that guaranteed your laptop would not wake from hibernate. You had to reboot the damn laptop each time you woke from hibernate because it would get stuck on a black screen, thus "losing" your data. I "lost" more data from their stupid fix than anything else.
They admit the original feature was broken because it didn't take into consideration for battery percent. Fine. But there was absolutely no reason to change the behavior. You can do both! The only reason I can think of is you just like being an incredible asshole to people. Which is obvious from that thread. Do they even use a laptop? Why would I want a dead battery next time I open the lid???
> I'm still livid at this Luca guy gaslighting people on the "correct" behavior of a feature people were using for years already.
As a fellow (though not lately very active there) systemd maintainer, reading his comments in that issue is a real bummer.
I'm not sure I'd call it "gaslighting", I don't think he was even bothering to grok what people were asking for. Not until it became a pile-on of additional systemd members @yuwata and @DaanDeMeyer.
Maybe he's checked out a bit with all the drama going on over at BlueHat?
I’ve never heard of one either. While systemd is certainly more complex than old school init scripts, and arguably deviates quite a bit from the “Unix philosophy,” it seems some people just have an irrational hatred towards it. To the point of making up problems with it that I’ve never heard anyone actually having.
There are a few reasons why systemd left a bad taste in the mouths of many Linux enthusiasts:
Systemd replaces (or tries to replace) whole swaths of OS functionality all at once, instead of addressing each piece individually. As a result, all of the systemd pieces are tightly integrated and nigh-inseparable, and critics claim this was intentional so that distributions could not easily pick and choose the parts they wanted. My understanding is that this is better now. In my experience, some pieces are better than others. (Systemd-timesyncd is absolute rubbish compared to chrony, for example.)
Although the goal of writing a better init system is laudable, a lot of people who looked at it early on not impressed with the implementation (a lot of NIH wheel-reinventing) and mediocre code quality.
It was introduced into Fedora rather suddenly with seemingly little warning, community discussion, or press coverage. In the Linux world where big things change slowly if at all, other popular distributions seemed to adopt it fairly quickly, one after the other. This raised a lot of conspiracy-like speculation about how that could have possibly happened.
The developers are famously obstinate about their technical decisions and often argue in, close, or ignore bugs asking them to consider a different direction for specific low-level issues. There have been a few public disagreements over "correct behavior" when integrating with other projects (e.g. the Linux kernel).
Had a problem with systemd-resolved yet again the other day. Blatted resolv.conf and job done, but between systemd and related things (network manager etc) it just breaks the way things have worked for decades and gives many people no benefit.
This isn’t all systemd of course, it’s the distros that choose to use it.
Two systemd things that I disable straight away on every box: systemd-resolved, and systemd-timesyncd. The former tries to be "smart" and therefore tends to mess up DNS in very simple scenarios (such as a box in a datacenter with a static IP) and the latter is just plain rubbish compared to chrony.
It's been a blessing for me. It's so easy to see the process hierarchy, easy to create units and timers and dependencies, and with services being combined into cgroups the concept of a zombie process was basically eliminated.
I guess I get that init was simple, but it didn't care about the process lifecycle. You always needed to have an incredibly smart init script or watchdog service to manage that.
I did have systemd induced problems but not recently. It was beta quality software when it was first introduced. Even on Debian which wasn't the first distro to ship it. Problems were mostly related to the different naming schemes they introduced to solve problems where the name of a disk or a network interface could randomly change and cause issues. Incidentally that's exactly the areas where it bit me. On a remote system the issues could have been very serious. I did not have such problems in at least 10 years. Ubuntu's new networking changes are causing a similar problem on the latest LTS upgrade. The package manager does not complain about anything nor migrate the configuration but systems end up with no network. These issues are a pain to resolve on remote bare metal servers, thus unacceptable. Yet we accept them. (Not really. There's already an exodus from Ubuntu, due to Snap)
I used Debian without systemd for a long time until it was no longer possible by upgrading. (I believe they make an effort to optionally run without systemd nowadays). In the meantime systemd got better. I still don't like it. It's like how people won't give btrfs another chance because it ate their data once 15 years ago. I think this is completely understandable. Thinking about the past, looking at the history, there's no reason to trust neither the technical skills or ethics of the people pushing systemd (or snaps) either.
Agreed, I'd say it's first on the system that isn't handling exiting/shutdown cleanly, and secondly on the distro config timeout value. I get this quite a bit on some machines (I use Fedora) and it is super annoying. I have yet to find which service is resonsible though as anytime I'm shutting the system down is inherently not a good time for debugging :-)
I have had one issue related to systemd; at one point when doing a routine update there was some dangling symlink issue that caused systemd to crash. This wasn't catastrophic; a lot of things didn't work, but I was able to cleanly save and exit out of applications, manually run a sync to make sure everything was flushed to disk, and then hard reboot.
systemd hasn't been without issue, but I've definitely had fewer issues with systemd than I did dealing with broken init scripts or upstart prior to moving to systemd.
I've never suffered data loss from systemd, but I certainly have encountered loss of functionality. There are a number of things that I just can't accomplish with systemd that I can accomplish without it.
Sure. My most recent headache is with using a USB wifi dongle. During system startup, I want to attach some remote drives over wifi, so I need to be able to do that after the wifi connection is established.
Systemd does not appear to be able to tell when that happens. It thinks the network is established before it actually is, and then the mount calls cause the startup process to pause for 3 minutes for those mount calls to time out before continuing.
The funny thing is that sounds like a perfect case for systemd: By eating so many parts of the OS (here, device management, filesystem mounting, networking, and service management), it should be able to create very explicit dependency graphs - "this mount depends on this network which depends on this device". I wonder where the problem is - can it not properly handle something between the device and network? Or is it hitting the "network is up" goal too early?
Thank you. I wish to add bluetooth dongle and expecting some headaches there. It seems setting up a full bluetooth stack would be challenging to say the least and I wander if systemd would contribute more to the challenge or to the solution.
While "lossage" in the context of an OS implies other things, in this statement it could mean time. I've had plenty of time wasted by systemd, as has pretty much everyone I know who has had to do more than basic tasks with systemd (and some people who were just trying to do basic things).
Very uncommon. I'm using it since it became default on Arch and never had any problems. Systemd is actually the best thing to happen to Linux. Luddites gonna luddit.
Systemd is umbrella project for many software components, one of which is pid1 init. Almost all of the components are optional and many of them integrate through well-defined interfaces.
> Also, it has 1.2 million lines of code. It's massive for an init system. It increases the chance of potential bugs which can be exploited.
You're willfully conflating the systemd project repository with the init system.
I use systemd the init system, but in-tree components like resolved and networkd have never run on any of my machines.
Surely you can understand that a monorepo with code reuse across myriad components has its advantages, and that systemd the init system can be just a subset of that tree.
One can argue the systemd project is simply following in the Linux kernel's footsteps here. It too is a monorepo chock full of all kernel functionality one might ever need, with the expectation that use cases will pick and choose what snowflake suits them best.
> One can argue the systemd project is simply following in the Linux kernel's footsteps here. It too is a monorepo chock full of all kernel functionality one might ever need, with the expectation that use cases will pick and choose what snowflake suits them best.
Linux is openly a monolithic kernel that builds everything in one tree. Are you sure that's the analogy you want to pick?
I hate systemd with a passion, but I won't fiddle with Debian to get rid of it.
I would love if a major distro emerges which works with human-readable scripts and logs and stays away from any "binary" formats for system configuration and logging.
Good news for you! You have slackware [0], void [1] and alpine [3], which are widely-used non-systemd distributions with sane scripts. They are well-maintained rolling releases which allow you to use much newer versions of the kernel and packages than your typical ubuntu/debian installs. I don't particularly care about systemd, but these distros are great by themselves!
Of those, I think only Void is fully rolling release; Slackware, as you note, is very much not rolling, and Alpine has regular releases and you'd have to go to the "edge" version to get rolling releases.
I've just started playing with Void because I want to use runit on both my dev machines (Linux) and servers (FreeBSD). I've really been liking it a lot, although the batteries-not-included approach did not phase me since I've been on Arch for awhile. Another thing that prompted the switch was annoying intermittent failures of systemd-resolved.
It will always be a fringe effort since the vast majority of the people doing the actual work are fine with it. It's been what, ten years by now? If instead of complaining all that time systemd haters cooperated and put in the work, they could have produced something (supposedly) much better by now.
Alpine is probably your best bet, it's pretty widely used. All of the others are a minority of a minority of a minority.
As per other sibling comments, there are already many efforts underway. Void Linux in particular, which I've used, has a surprising number of packages.
Unfortunately alpine ships with a lot of init scripts that don't work. To make it worse it ships with init scripts that could never have worked. I don't know if discovering this should make me stay away from it as a distro or send patches (which is not trivial. I tried a few times). I do like < 3 minutes full distro version upgrades though.
Obviously it’s a different OS and possibly isn’t relevant to you but I’ve found solace in OpenBSD (and other BSDs) in the search of “simplicity” of “the old days”.
This is a vastly under-rated opinion. I found FreeBSD to be surprisingly superior in my use-cases to Linux as a server OS. From stability, sensible defaults, config files where-I-can-find-them, and more.
My only real headache with FreeBSD has been SAMBA. It seems that every major upgrade bjorked the SAMBA user/account DB, and would cause much wailing and gnashing of teeth. So I learned to deal with that.
I now work for a K8s shop, and find myself migrating back into the linux/systemd fold. There's frustration with systemd'isms, and I truly despise the binary logs; but I'm learning.
I tried FreeBSD a few times, but after ~4 hours of reading docs, searching the forums etc, I was not able to create a fully featured systemd like service. With stop, start, restart and auto restart support. something was always missing or not working correctly.
But what I found was lots of hate and stupid talkin against systemd, that they all hate it and that FreeBSD will "never have this $hit"
I thought to myself, yeah just hate systemd but not able to produce an easy to use and full fledged service manager. The arrogance was staggering, and I stopped my use of freebsd since then, in almost all aspects linux is superior anyhow.
Such a fully featured systemd service is easily done in 6 - 8 lines of configuration.
>major distro emerges
have you tried gentoo? Really fits your requirement of 'emerges' lol
They also have a pretty good page on their wiki that explains the current popular alternative init systems to systemd(0), even ones gentoo doesn't support.
I cut my teeth on OpenRC and I will die with OpenRC.
Actually, I don't really care about the init system, as far as I can tell. When I started using Gentoo, the default was OpenRC, so that is all I have ever known.
I have not the faintest idea what benefit I would get from using Systemd compared to OpenRC, so I have never tried changing. I have used Debian with Systemd, and I can't particularly tell the difference from an everyday-user perspective. I used Ubuntu a few times when I think it had upstart maybe? Again, no idea what the difference is.
All I know is that there are things I can do with the rest of Gentoo that I have not easily been able to do with other distros, so that is where I stay.
Anyway, for anyone who hates systemd for whatever reason, Gentoo is definitely an alternative. It is a little weird to me that any distro would have a hard dependency on a particular init system, when it is clearly not necessary. More work to be compatible with multiple, I guess? At the same time, I don't really understand the hate. Just ... use something else?
I get the silliness of journald not being clear text, you can configure it to be, but it's not the default, which is really weird. If you have the scale where you need binary logs for performance, then you also have remote log hosts. The remote log hosts will frequently also store logs in a binary format, like logstash, Humio, Splunk and what have you, but they are shipped a text, so the binary is just an unnecessary extra step. Sure you can use journald as a log collector, but I've never seen anyone do that... Kinda cool, but not frequently used.
Systemd unit files though, those are much much better than any other init system I've ever used. They are easy to write, easy to understand and just overall nice to work with. Timers are awesome as well, same easy syntax as startup scripts, easy to test, easy to debug.
I'm neutral on systemd, just give me something that works and I'll be ok with it.
Anyway, if you're after a distro that doesn't use systemd and is a joy to use and tinker with, take a look at Alpine [0]. It is a lot more compact and faster compared to other distros for using Musl instead of Glibc, which means you may find software that has not been ported yet, however so far I've installed it also on mini PCs and laptops with great results.
Text files are generally more resistant to corruption, especially when parsed by a human, and more repairable if they do get corruption. (A one byte error probably won't matter in a text file; it could wreck parsing rigid structure.)
They more easily work with a lot of un-specialized tools, or tools specialized for text, which is a very general problem with very transferable learning.
Perhaps better to distinguish between "structured" and "unstructured" or "machine optimized" vs "human optimized".
A different beast at the end of the day, whatever name or distinction you use.
It is unfortunately trivial to argue that Devuan is without point, as Debian have proven themselves (per the article) entirely amenable to retaining the ability to run without systemd. As, indeed, they said they allow for would when they decided to adopt it.
If anything, the presence of Devuan cements systemd even more tightly into position -- those who care about not running systemd are distracted by forking an entire distribution, rather than putting their effort into the much smaller problem space that is retaining functionality that Debian is quite happy to retain if its proponents are willing to put in the effort.
So Debian is going to integrate more with systemd than it would if Devuan didn't exist, and I suppose for that I should thank the folk behind Devuan. But it's always disappointing when people expend their effort against something, rather than for something.
Very happy Debian and systemd user here, in case you couldn't tell.
>"as Debian have proven themselves (per the article) entirely amenable to retaining the ability to run without systemd"
There's obviously more to this (and to Devuan) than just uninstalling systemd and installing sysvinit. Devuan's package repo (for which Debian is the upstream) contains a lot of customizations to allow certain packages to function with a sysvinit foundation.
And my understanding is that Debian would be quite happy to have those customisations too -- maybe with a little more upfront effort, as they have to coexist with systemd support. But with less ongoing effort, as the code is at least present to be able to tell if a change will break it.
Ultimately though it's entirely up to the developers where they put their effort. If folk prefer to work on a fork, they're entitled to do that. And it is a bit refreshing to see people putting at least some effort into not-systemd, rather than complaining about being "forced" to use it.
Note that under the "Init Software" section (scroll down on page!), the following two choices (amongst numerous others, perhaps too many!) are included:
[ ] systemd
[ ] Not systemd
In other words, here, the User, regardless of their opinion, regardless of their political ideology (or lack thereof!) -- has a happy choice!
They can choose whatever they themselves believe to be the correct choice in the matter!
Or they can choose to leave both checkmarks blank -- and let Distrowatch's Advanced Search -- decide for them, and perhaps "surprise" them! :-)
You know, like you like Coke, I like Pepsi...
You like Windows, I like Linux (or vice versa!)...
You might think that it's a floor wax, I might think that it's a dessert topping -- but we could both be right!
Does Bookworm then ship two init "script" for every single service? A sysv init script and a systemd unit file. That seems like a lot of work, especially the sysv script.
init.d was familiar to graybeards (like myself), but I'm loathe to consider them a superior solution.