Created attachment 127297 [details] [review]
dbus.service.in: Allow D-Bus to start during early boot

Patch against current master which relaxes the dependencies. With that, our earlyresolve.service now does not experience the timeout any more:

Okt 14 13:01:53 pid1 systemd[1]: Starting earlyresolve.service...
Okt 14 13:01:53 pid1 systemctl[563]: ● dbus.socket - D-Bus System Message Bus Socket
[...]
Okt 14 13:01:53 pid1 ping[566]: ping: www.freedesktop.org: Temporary failure in name resolution
Okt 14 13:01:53 pid1 systemd[1]: Started earlyresolve.service.

I added After=local-fs.target mostly for cautiousness -- it should actually suffice to wait for systemd-remount-fs.service, but I'd rather stick to target names as they are more stable than the individual components of it.

I added RequiresMountsFor=/var/lib/dbus in case /var is on a remote file system (although this has rarely ever been a practical thing). For local file systems this is already implied by local-fs.target, so this should actually be a no-op but it can't hurt to keep it I think.

Wrt. these two from the patch:

+Conflicts=shutdown.target
+Before=shutdown.target

I mostly added those as this the other effect of DefaultDependencies=yes. But there is actually not much reason to stop D-Bus during shutdown, and in fact doing that causes a lot of hangs (see bug 89847). That's why in Ubuntu we actually modified dbus.service to not stop at all during shutdown, which is a lot more fault tolerant than trying to chase down everything during shutdown that tries to talk to it still. So we could even drop these two lines to make D-Bus available for an even longer time.

(In reply to Martin Pitt from comment #0)
> When introducing resolved into Ubuntu we stumbled over a deadlock. The
> "resolve" NSS module talks to systemd-resolved over D-Bus. There are
> services such as cloud-init which try to resolve names during early boot (i.
> e. before basic.target), and these currently run into a 25s D-Bus timeout
> during boto (https://launchpad.net/bugs/1629797).

I'm fairly sure the D-Bus system bus is not meant to be an early-boot service. I'm not convinced NSS or resolved is either. NSS modules often require /var, for instance. I think last time this was suggested, Lennart said it was a very bad idea for dbus-daemon to (try to) run that early.

What is cloud-init doing that it needs to run that early, and can it be made to not do that, or bypass NSS and resolved and do its own simpler name-resolution, or something?

Now that major distributions all start with /usr mounted[1], the remaining issues with starting dbus-daemon early are:

* The canonical path for the machine ID and the socket is in /var, which might not be mounted yet
* Loading <policy> requires full-fat username and group resolution (the equivalent of `getent passwd foo` and `getent group bar`), which can use arbitrarily complex NSS modules too (up to and including ones that need D-Bus :-P )
* Traditional (non-systemd) activation can happen any time after dbus-daemon starts up, but 99% of activated services won't work (and shouldn't try) during early boot

The first is trivial, and could be avoided by limiting the supported configurations. AIUI, kdbus was carefully designed to dodge the others.

[1] except for when they don't: Debian with no initramfs and a separate /usr is still something people complain about when it doesn't work, although I think the official line is now "if you don't have an initramfs you get to keep both pieces"

(In reply to Martin Pitt from comment #2)
> But there is actually not much reason to stop D-Bus during shutdown, and in
> fact doing that causes a lot of hangs (see bug 89847). That's why in Ubuntu
> we actually modified dbus.service to not stop at all during shutdown

... which I'll note the systemd maintainer (who is also a dbus maintainer) specifically asked you not to do.

(In reply to Martin Pitt from comment #0)
> But dbus.service itself currently only starts in late boot (after
> basic.target). So the D-Bus client triggers the start of dbus.service which
> needs to wait for the D-Bus client to finish first and eventually runs into
> the D-Bus timeout (25s).

If the 25s timeout is not appropriate for your use case, please stop passing -1 to D-Bus libraries when asked for a timeout.

(In reply to Simon McVittie from comment #3)
> I'm fairly sure the D-Bus system bus is not meant to be an early-boot
> service. I'm not convinced NSS or resolved is either. NSS modules often
> require /var, for instance.

We have no choice about NSS -- we can't suddenly forbid early-boot (< basic.target) services to not use NSS any more. They *must* be able to resolve user names, and even  for the "hosts:" part specifically it is not an unreasonable requirement to be able to resolve "localhost". Note that the above reproducer works even with "ping -c1 localhost", provided you don't have an /etc/hosts which will already resolve "localhost" via "files".

> I think last time this was suggested, Lennart said it was a very bad idea for dbus-daemon to (try to) run that early.

Where was that said? If that is the case, then we need to find and document an agreement what D-Bus should be:

 * An application like apache, display manager, or cups, i. e. stuff that starts in multi-user.target and are more or less the top-level services of the OS. Or

 * Infrastructure that the above applications need, similarly to file systems, consoles, networking, etc.

To me, D-Bus as the provider of more or less "the" Linux IPC falls into the latter category. Also, I see just about zero reason why it could *not* start early -- with Upstart that has happened for about 10 years now ("start on filesystems" which is basically the equivalent of "DefaultDependencies=no; Requires=local-fs.target"), and it really does not have many dependencies.

But if you/Lennart say that it should be in the former category, then this basically rules out D-Bus for things like resolved (and using dnsmasq as local resolver at the same time, as that also uses D-Bus). But then we should stop pretending that they can and either drop dbus.socket, or move it from sockets.target into multi-user.target to avoid the deadlock.

FTR: We already discussed pretty much all that in bug 89847, and e. g. in https://bugs.freedesktop.org/show_bug.cgi?id=89847#c3 ("The fewer constraints we have to apply to startup, the better") it seems that we already agreed on this in principle. So this is the next step of trying to clean this up more properly.

> What is cloud-init doing that it needs to run that early, and can it be made
> to not do that, or bypass NSS and resolved and do its own simpler
> name-resolution, or something?

It's checking for various possible config providers (local mounts, network sources, etc.). But this isn't really a cloud-init specific problem, it just happened to be the first thing that exposed this more general problem -- anything which will try to connect to D-Bus or do host name resolution (including "localhost") before basic.target will cause this.

> * The canonical path for the machine ID and the socket is in /var, which
> might not be mounted yet

That dependency is declared with

+RequiresMountsFor=/var/lib/dbus

> * Loading <policy> requires full-fat username and group resolution (the
> equivalent of `getent passwd foo` and `getent group bar`), which can use
> arbitrarily complex NSS modules too (up to and including ones that need
> D-Bus :-P )

I think we can safely rule that out -- you need to be able to resolve user names to start D-Bus as you say (even with a completely empty policy the "messagebus" system user must be resolved), so this would never have worked and cloud never work.

> * Traditional (non-systemd) activation can happen any time after dbus-daemon
> starts up, but 99% of activated services won't work (and shouldn't try)
> during early boot

Indeed this wouldn't be a problem under systemd as their .service files would be in multi-user.target (and the systemd unit is all that this patch touches). But indeed his situation causes trouble right now in early boot: as dbus.socket is already running, client libraries *already* try to activate the service, but are guaranteed to time out as dbus.service is blocked/not running. So in the worst case, running D-Bus itself earlier would merely shift the timeout from connecting to D-Bus itself to waiting for the service to activate, but for cases like trying  to talk to resolved, to systemd itself ("systemctl") or to networkd in the future, it would then just work.

> [1] except for when they don't: Debian with no initramfs and a separate /usr
> is still something people complain about when it doesn't work, although I
> think the official line is now "if you don't have an initramfs you get to
> keep both pieces"

Right, and this has been broken for a while already due to completely different reasons. Also, there has been another attempt to finally enable merged /usr by default now.

> (In reply to Martin Pitt from comment #2)
> > But there is actually not much reason to stop D-Bus during shutdown, and in
> > fact doing that causes a lot of hangs (see bug 89847). That's why in Ubuntu
> > we actually modified dbus.service to not stop at all during shutdown
> 
> ... which I'll note the systemd maintainer (who is also a dbus maintainer)
> specifically asked you not to do.

Right, and with this it should not be necessary any more, as dbus.service would then also stop much later, after everything in multi-user, instead of getting stopped as the first thing during shutdown.

(In reply to Simon McVittie from comment #4)
> If the 25s timeout is not appropriate for your use case, please stop passing
> -1 to D-Bus libraries when asked for a timeout.

This cannot be addressed via tweaking timeouts -- the timeout needs to allow D-Bus itself and the activated service (resolved in this case) to start, so lowering it much below 5s will introduce failures on slow systems (spinning rust or ARM). But at the same time, increasing boot time by 5 s is pretty much unacceptable if you otherwise measure container boot speed in ms and even modern laptops need only 10s until they arrive at the display manager.

(In reply to Martin Pitt from comment #5)
> To me, D-Bus as the provider of more or less "the" Linux IPC falls into the
> latter category. Also, I see just about zero reason why it could *not* start
> early -- with Upstart that has happened for about 10 years now ("start on
> filesystems" which is basically the equivalent of "DefaultDependencies=no;
> Requires=local-fs.target"), and it really does not have many dependencies.

I think this is a question for the people who define basic.target (hello Lennart): is D-Bus messaging something that makes sense in the basic target?

As soon as dbus-daemon has started, services that use traditional D-Bus activation can get auto-started. Everywhere that isn't booting Upstart, we have traditionally (whether deliberately or accidentally) ensured that normal D-Bus system services can't start until basic system infrastructure - systemd's DefaultDependencies, or sysvinit's rcS.d - is up. Most, perhaps all, of those services are going to fall over if asked to start without basic infrastructure.

If there is consensus that D-Bus *messaging* should be an early-boot service, then I think we should arrange for D-Bus *activation* to not be an early-boot facility in general. One way to do that might be by giving dbus-daemon a command-line option to start up with traditional (non-systemd) activation blocked, adding that command-line option to dbus.service, and also adding a oneshot unit with the DefaultDependencies that tells dbus-daemon to unblock traditional activation.

If a subset of system services are actually fine to be early boot services themselves, then either they can be systemd-activated, or they can have some flag in their D-Bus .service files that is the equivalent of DefaultDependencies=no.

(Now that I think more about this idea, perhaps the oneshot unit that unblocks traditional activation should also trigger a policy reload in the hope that a larger subset of NSS has become available?)

(In reply to Martin Pitt from comment #5)
> We have no choice about NSS -- we can't suddenly forbid early-boot (<
> basic.target) services to not use NSS any more. They *must* be able to
> resolve user names

Is this true in general? (nss-ldap, NIS, etc.)

I think the recommendation in sites that use nss-ldap and similar has typically been to store system usernames like root and messagebus locally, and fetch "real human user" names from LDAP?

> and even  for the "hosts:" part specifically it is not
> an unreasonable requirement to be able to resolve "localhost"

I would suggest nss-myhostname for this, tbh. It's stateless, which seems like an obvious win for VM/container/embedded use.

> But if you/Lennart say that it should be [late-boot], then this
> basically rules out D-Bus for things like resolved (and using dnsmasq as
> local resolver at the same time, as that also uses D-Bus).

For what it's worth, dnsmasq (as shipped in Debian; other distros may vary) has the DefaultDependencies, and its sysvinit service is correspondingly in rc[2-5].d. systemd-resolved does the same, and so do NetworkManager and wicd.

On the other side, ConnMan and systemd-networkd have DefaultDependencies=no (but also an explicit dependency on dbus.service, so they'd benefit from your proposed change). Similarly, the traditional Debian networking service (ifupdown) is an early-boot service (DefaultDependencies=no and rcS).

So, dbus.socket shouldn't be upped unless dbus-daemon can actually be started up, too.

If dbus.socket is started already at the time cloud-init runs, but cloud-init blocks dbus-daemon, then that's a deadlock that cloud-init should really fix (specifically: it should run before sysinit.target, as dbus.socket runs only after).

The code in nss-resolve is written in a way that it will fail cleanly if dbus isn't around, chainloading nss-dns in this case.

I don't think it makes sense to move dbus-daemon as it is right now to early boot. It's not ready for that. For that we'd need the ability to individually schedule when bus names become activatable. Right now dbus-daemon doesn't support that.

Long story short: I doubt there's anything to fix in dbus-daemon here: cloud-init didn't get the ordering deps right, that's all i see here to fix...

In your example, this is not right:

    After=local-fs.target sockets.target
    Before=basic.target

if you want to be an early boot service, then you should use something like this:

    Before=sysinit.target

And that's all.

Inserting yourself between the sockets and the regular services (which your suggested deps do) is highly problematic, if you actually intend to make use of the sockets, as then you will make the system hang, as to fulfill your requests you need the services you are delaying...

(In reply to Lennart Poettering from comment #8)
> So, dbus.socket shouldn't be upped unless dbus-daemon can actually be
> started up, too.

Right, that was my alternate proposal: move dbus.socket from sockets.target to multi-user.target so that this deadlock can never happen in the first place.

Ack, thanks for the heads-up. I'll pass this on to the cloud-init devs, to replace their

  Before=basic.target
  Before=dbus.socket

with

  Before=sysinit.target

So is the conclusion here that dbus-daemon is not doing anything wrong, and this should be NOTOURBUG, and fixed in cloud-init instead?

i would say so, yes

> move dbus.socket from sockets.target to multi-user.target so that this deadlock can never happen in the first place.

I didn't close the bug yet because of the above -- this would plug the hole. It's not very important once you know how to fix it, but apparently people do fall into it.

(In reply to Martin Pitt from comment #13)
> > move dbus.socket from sockets.target to multi-user.target so that this
> > deadlock can never happen in the first place.
> 
> I didn't close the bug yet because of the above -- this would plug the hole.
> It's not very important once you know how to fix it, but apparently people
> do fall into it.

Wouldn't that be a compat break? It would mean that system services whose Type is not dbus, but that do incidentally use the system bus, would all have to add After=dbus.socket or they couldn't guarantee to be started after it. At the moment they get this implicitly, via DefaultDependencies and sockets.target.

(Concrete examples include dnsmasq and systemd-resolved.)

Good point, thanks. Closing then.