Bug 86707

Summary: Hang on shutdown because wpa_supplicant is stopped before that umount a _netdev mount point
Product: systemd Reporter: Javi Barroso <javibarroso>
Component: generalAssignee: systemd-bugs
Status: RESOLVED FIXED QA Contact: systemd-bugs
Severity: major    
Priority: medium CC: alexandre, hutou01, javibarroso, mike
Version: unspecified   
Hardware: Other   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:
Attachments: Add network-online.target as dependency to _netdev mount points
systemctl-status after boot
dmesg booting, to know which order and which services are being executed
Summarize what happens when system try to halt

Description Javi Barroso 2014-11-25 16:36:28 UTC
Created attachment 110009 [details]
Add network-online.target as dependency to _netdev mount points

Hello,

I was having trouble with a curlftpfs mount point [1], that did not umount at shutdown.

Debugging the issue (at Debian did not work /usr/lib/systemd/system-shutdown/debug.sh hack, but yes the crash shell console at tty9):

[  473.352374] systemd[1]: mnt-usb.mount changed mounted -> unmounting
[  473.356270] systemd[1]: wpa_supplicant.service changed running -> stop-sigterm
[  473.387815] systemd[1]: Failed to send unit change signal for wpa_supplicant.service: Transport endpoint is not connected
[  473.387847] systemd[1]: Failed to send unit change signal for mnt-usb.mount: Transport endpoint is not connected
[  473.703928] systemd[1]: Child 1564 (wpa_supplicant) died (code=exited, status=0/SUCCESS)

/mnt/usb is the mount point which is failing to umount (it is the router usb mounted on my laptop)

I'm not sure if curlftpfs should do some thing better when it tries to umount when there is not connection..

To solve this issue I modified fstab-generator to add "After and Requires" depends to "network-online.target" to .mount units which are network mounts. I'm attaching the patch

What do you think is the best solution for my problem ?

Thank you

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=769186
Comment 1 Javi Barroso 2014-11-25 16:53:29 UTC
Sorry forgot muy patch, i was testing with a reboot and not with shutdown. It does not work

Sorry for the inconveniente

Regards
Comment 2 Alexandre Detiste 2014-12-01 16:44:23 UTC
Hi,

I had the same kind of problem with a / filesystem hosted in a file on a NTFS partition (/boot is on an usb key).

Here is how solve it:

https://github.com/koniiiik/oi-live/blob/master/oi-live/ntfs-3g-initrd-wrapper.c

http://0pointer.de/blog/projects/systemd-for-admins-2.html
Comment 3 Lennart Poettering 2014-12-08 23:23:12 UTC
_netdev mounts already get "network.target" dependency. However, as _netdev is a userspace option, this only works if the file systems are mounted from userspace.

My guess is that wpa_supplicant.service should really have a Before=network.target dependency. Does it have that on your system?
Comment 4 Javi Barroso 2014-12-09 22:22:53 UTC
Created attachment 110652 [details]
systemctl-status after boot
Comment 5 Javi Barroso 2014-12-09 22:23:53 UTC
Created attachment 110653 [details]
dmesg booting, to know which order and which services are being executed
Comment 6 Javi Barroso 2014-12-09 22:24:51 UTC
Created attachment 110654 [details]
Summarize what happens when system try to halt
Comment 7 Javi Barroso 2014-12-09 22:30:07 UTC
(In reply to Alexandre Detiste from comment #2)
> Hi,
> 
> I had the same kind of problem with a / filesystem hosted in a file on a
> NTFS partition (/boot is on an usb key).
> 
> Here is how solve it:
> 
> https://github.com/koniiiik/oi-live/blob/master/oi-live/ntfs-3g-initrd-
> wrapper.c
> 
> http://0pointer.de/blog/projects/systemd-for-admins-2.html

Hello Alexandre,

Thank you for pointing your solution.

I think this problem should be solved at wpa_supplicant upstream (if somebody can confirm it happens too outside Debian)

If we are able to solve this issue, I will give this reference bug to wpa_supplicant developpers (seems active people), so they can change its wpa_supplicant.service

Maybe if the problem is not solved soon (weeks?) I will ask to them about this issue at their mail list / bugtracker

Thank you
Comment 8 Javi Barroso 2014-12-09 22:37:14 UTC
(In reply to Lennart Poettering from comment #3)
> _netdev mounts already get "network.target" dependency. However, as _netdev
> is a userspace option, this only works if the file systems are mounted from
> userspace.
> 
> My guess is that wpa_supplicant.service should really have a
> Before=network.target dependency. Does it have that on your system?

Hello,

At my (Debian Sid) system wpa_supplicant.service has not such depend.

Seems like at Debian wpa_supplicant.service is not changed from upstream:
http://w1.fi/cgit/hostap/tree/wpa_supplicant/systemd/wpa_supplicant.service.in

I tried to add Before attribute? so it can read:

$ systemctl cat  wpa_supplicant.service 
# /lib/systemd/system/wpa_supplicant.service
[Unit]
Description=WPA supplicant
Before=network.target

[Service]
Type=dbus
BusName=fi.epitest.hostap.WPASupplicant
ExecStart=/sbin/wpa_supplicant -u -s -O /run/wpa_supplicant

[Install]
WantedBy=multi-user.target
Alias=dbus-fi.epitest.hostap.WPASupplicant.service

But it didn't work.

I'm uploading files which I think can help to solve the issue.

At systemctl-status-booting.txt you can see two things:
 - curlftps is a dependency from NetworkManager-dispatcher.service
 - mnt-usb.mount fails the first try (see Process field at systemctl status -f mnt-usb.mount) (Error Connecting to FTP)

At dmesg-booting.txt.gz can be seen dmesg after booting

Finally at halting.txt I try to summarize what happen when system is halting:
 - start umount /mnt/usb
 - wpa_supplicant deauthenticate
 - umount will not finish until it is killed by systemd after a timeout

I really appreciate your help. Please tell me if you need some additional info.

Thank you very much
Comment 9 Javi Barroso 2014-12-09 22:41:06 UTC
I'm not sure if I should change status field. Sorry if that field should be only changed by Systemd developpers
Comment 10 Mike Nerone 2014-12-28 09:54:27 UTC
In support of the fix per Lennart Poettering in #3, I had a similar issue with my media server failing to exit at shutdown, which I eventually was able to isolate to wpa_supplicant.service having stopped too early. I installed "Before=network.target" (which makes perfect sense to me) via a drop-in for wpa_supplicant.service, and it did, indeed, resolve the issue.
Comment 11 Javi Barroso 2014-12-28 23:32:23 UTC
Hello,

I have just send next mail to debian bug tracker system:

=== COMMENT BEGIN ===
What I am see on my Debian Sid system:

1. systemd-fstab-generator generate mnt-usb.mount
2. NetworkManager-dispatcher is really who mount /mnt/usb directory

After I generate a /etc/systemd/system/mnt-usb.mount with an incorrect
syntax, My system shutdown well after a timeout of 10 seconds. I added
Before=network.target to wpa_supplicant.service as indicated at
upstream bugtracker.

I tried to disabling nm-dispatcher (chmod -x), but mnt-usb.mount was
not able to manage to mount (I guess it tries before than the network
is online (a ftp connection error is showed at systemctl status
mnt-usb.mount)

Which is the correct way to make this work ?

How can I disable mnt-usb.mount ? (I think I  tried without success
systemctl mask)

=== COMMENT END ===

Which is the preferred method to use only one service to such mount point (dispatcher or fstab-generator)?

Thank you very much
Comment 12 Lennart Poettering 2015-02-04 16:52:32 UTC
Note that systemd git is now able to read the _netdev option from userspace for mounts that are done on the shell, without /etc/fstab. With that in place systemd is now able to order all mounts after network.target automatically, regardless if configured in /etc/fstab or done manually, as long as they have _netdev  specified. Then, if wpa_supplicant.service contains After=network.target all should be good. As the shutdown order between two units is always the reverse of the startup order this means the mount units are unmounted first, before wpa_supplicant is terminated.

Hence, closing, everything else needs to be done with the right After=network.target dep in wpa_supplicant.service
Comment 13 Javi Barroso 2015-02-04 17:12:53 UTC
Hello,

Can you say in which version start systemd to have such behaviour ?

Thank you
PD: I will forward your comments to wpa_supplicant bug tracker / debian bug tracker
Comment 14 Lennart Poettering 2015-02-04 17:16:44 UTC
This is fixed in systemd git, will be part of the next systemd release, which is hopefully out within two weeks or so.
Comment 15 Javi Barroso 2015-02-04 17:31:52 UTC
(In reply to Lennart Poettering from comment #14)
> This is fixed in systemd git, will be part of the next systemd release,
> which is hopefully out within two weeks or so.

Thank you !
Comment 16 Lennart Poettering 2015-02-04 18:44:21 UTC
*** Bug 82061 has been marked as a duplicate of this bug. ***
Comment 17 Michael Biebl 2015-03-16 00:32:48 UTC
Has someone forwarded the issue to wpasupplicant upstream to get Before=network.target added to wpa_supplicant.service.in?
Comment 18 Javi Barroso 2015-03-16 07:18:52 UTC
(In reply to Michael Biebl from comment #17)
> Has someone forwarded the issue to wpasupplicant upstream to get
> Before=network.target added to wpa_supplicant.service.in?
Hello,

I tried to forward, but not body replied:
http://readlist.com/lists/shmoo.com/hostap/3/18622.html

Regards,

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.