Bug 33421

Summary: Deadlock on shutdown due to triggered automount unit
Product: systemd Reporter: Andrei Borzenkov <arvidjaar>
Component: generalAssignee: Lennart Poettering <lennart>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: medium CC: hp4everything
Version: unspecified   
Hardware: Other   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:
Attachments: Console log of current GIT

Description Andrei Borzenkov 2011-01-24 10:52:10 UTC
Reported for Mandriva as https://qa.mandriva.com/show_bug.cgi?id=62101
Console log that illustrates timeout on /etc/init.d/alsa: https://qa.mandriva.com/attachment.cgi?id=18176
Console log with "set -x" for /etc/init.d/alsa: https://qa.mandriva.com/attachment.cgi?id=18179

/etc/init.d/alsa runs "find /dev ..." as part of shutdown. This triggers /dev/mqueue automount as long as it was not yet mounted. This never completes.

Setting Conflicts=shutdown.target for dev-mqueue.automount "fixes" it but I do not understand why - according to debug log dev-mqueue.automount/stop is executed after alsa.service/stop completes.

If using this Conflict is correct, it probably should be set (by default?) for every automount unit.
Comment 1 Lennart Poettering 2011-01-25 17:47:41 UTC
Hmm, so adding this to shutdown.target is not really the right solution, the mount and automount points are pulled down by umount.target, not shutdown.target.

So, the easy fix is to just get rid of that ALSA script. The systemd support that is now in alsa-utils upstream is everything needed to make ALSA work nicely. All that crap of iterating through /dev or /proc is a recipe to slow things down and make things ugly. It will trigger all automount points for no reason.

That said we probably should fix systemd so that this deadlock becomes impossible.

Most likely this is simply an ordering problem: the alsa shutdown script waits for the /dev/mqueue fs, which however is ordered to run after the alsa shutdown script. A possible fix might be to fail the triggering of an automount unit if a shutdown request for it is queued. That should be easy to do.
Comment 2 Lennart Poettering 2011-01-25 17:58:47 UTC
Hmm, I made some changes to systemd git now, to be more verbose about when automount is triggered during shutdown. THere's now a debug log msg printed if ths happens when we go down.

Could you install systemd git and generate me an outout like the one you already generated but with "systemd.log_level=debug" and "systemd.log_target=kmsg" on the kernel command line? This should give us enough information to figure out the real reason for the deadlock.
Comment 3 Andrei Borzenkov 2011-01-26 10:51:02 UTC
Created attachment 42545 [details]
Console log of current GIT

See attached. Your debug output does not fire. Actually /dev/mqueue is happily mounted ... it just happens shortly before power off :)

3 minutes timeout is there

systemd[1]: alsa.service stopping timed out. Terminating.
systemd[1]: alsa.service changed stop -> stop-sigterm
Comment 4 Andrei Borzenkov 2011-01-28 23:12:24 UTC
The reason for deadlock is local-fs.target dependency implicitly added to every mount unit.

Before=local-fs.target
After=dev-mqueue.automount -.mount

local-fs.target is being stopped on shutdown due to dependency on shutdown.target:

systemd[1]: Installed new job poweroff.target/start as 112
systemd[1]: Installed new job poweroff.service/start as 113
systemd[1]: Installed new job shutdown.target/start as 114
systemd[1]: Installed new job systemd-update-utmp-shutdown.service/start as 115
systemd[1]: Installed new job local-fs.target/stop as 195
...

When later mount request for /dev/mqueue comes, dev-mqueue.mount waits for local-fs.target to be stopped according to rules:

systemd[1]: Trying to enqueue job dev-mqueue.mount/start/replace
systemd[1]: Installed new job dev-mqueue.mount/start as 216
systemd[1]: Enqueued job dev-mqueue.mount/start as 216

But local-fs.target can be stopped only after all services are stopped due to (start) ordering:

local-fs.target => sysinit.target => basic.target => foo.service 

So this is typical A - B - C - A deadlock. local-fs.target (implicitly) waits for alsa.service, alsa.service (implicitly) waits for dev-mqueue.mount and dev-mqueue.mount waits for local-fs.target.

This is second deadlock encountered due to implicit ordering rules (and one more was reported but not investigated due to lack of debugging information).
Comment 5 Lennart Poettering 2016-06-07 10:28:36 UTC
This has been fixed a long time ago, we don't permit triggering of autofs anymore during shutdown.
Comment 6 Lennart Poettering 2016-06-10 12:59:34 UTC
*** Bug 75535 has been marked as a duplicate of this bug. ***

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.