Bug 89383

Summary: systemd-219 can't mount devices
Product: systemd Reporter: Tomasz Paweł Gajc <tpgxyz>
Component: generalAssignee: systemd-bugs
Status: RESOLVED FIXED QA Contact: systemd-bugs
Severity: critical    
Priority: highest CC: bugs.freedesktop.org, devurandom, egorov_egor, grey, ian.kumlien, radek
Version: unspecified   
Hardware: All   
OS: Linux (All)   
URL: https://bugzilla.gnome.org/show_bug.cgi?id=743891
See Also: https://bugs.gentoo.org/show_bug.cgi?id=541402
https://bugzilla.redhat.com/show_bug.cgi?id=1196452
http://bugs.debian.org/779612
https://launchpad.net/bugs/1432265
Whiteboard:
i915 platform: i915 features:
Attachments: Proposed patch

Description Tomasz Paweł Gajc 2015-03-01 21:25:59 UTC
Log story short.

After update to systemd-219, dracut during live iso boot was not able to mount devices for further switch root, whereby booting iso in live mode was not possible. An example:

LOOPDEV=$( losetup -f )
losetup -r $LOOPDEV /live/media/LiveOS/squashfs.img
mount -n -t squashfs -o ro $LOOPDEV /live/distrib

After a deep digging, and getting system to boot into a real root simple mount command was not mounting anything.

mount -n -t squashfs -o ro /media/OpenMandriva_2015.0/LiveOS/squashfs.img /media/test

Applying those patches fixed issue:
https://bugzilla.gnome.org/show_bug.cgi?id=743891
Comment 1 Tomasz Paweł Gajc 2015-03-03 14:39:39 UTC
Bump. System is unbootable with 219 release. Raising importance.
Comment 2 Tomasz Paweł Gajc 2015-03-05 10:44:24 UTC
Looks like that applying this patch:
https://abf.io/openmandriva/systemd/blob/master/0001-Revert-core-mount-add-dependencies-to-dynamically-mo.patch

allows to mount devices.
Comment 3 Tomasz Paweł Gajc 2015-04-03 10:50:49 UTC
Ping anyone ?
Comment 4 Zuyi Hu 2015-04-19 04:51:36 UTC
(In reply to Tomasz Paweł Gajc from comment #2)
> Looks like that applying this patch:
> https://abf.io/openmandriva/systemd/blob/master/0001-Revert-core-mount-add-
> dependencies-to-dynamically-mo.patch
> 
> allows to mount devices.

Thank you very much, this patch helps me, I am doubt why upstream don't fix this problem.
Comment 5 Grey Christoforo 2015-04-22 13:28:34 UTC
This is a bad bug. 

Please see my report on the Arch Linux bug tracker on it here: https://bugs.archlinux.org/task/44658#comment134665

This report is using systemd from Arch's [testing] repo which seems to be version 219 with some patches applied by the Arch maintainers which try to address this bug: systemd 219-6, https://www.archlinux.org/packages/testing/x86_64/systemd/

In summary, even when the bug has been "fixed" (allowing mounts to be made again) there are cases when systemd is unmounting things it should not be when the user runs umount (seems like it unmounts ALL mounts associated with a device whenever any mount associated with that device is umounted).

A bug that breaks filesystem mouting/unmounting seems particularly critical. Whatever new logic 219 introduced that created this bug should probably be reverted from the systemd release until all the mounting and unmounting corner cases can be tested and covered.
Comment 6 Bastien Traverse 2015-04-23 16:02:20 UTC
Another use case to complement the report:

- system is a home server running Arch linux, full disk encrypted with dmcrypt/LUKS and remotely unlocked via dropbear_initrd_encrypt [1]
- it was updated today with the latest systemd packages ({lib}systemd{-sysvcompat} 219-5) and latest linux-lts (3.14.39), therefore the initramfs was rebuilt with mkinitcpio.

What happens: after rebooting I am locked out at the initramfs stage with the following error messages:

- on client trying to ssh in dropbear_initrd:
    Device /dev/disk/by-id/wwn-<redacted_id>-part3 doesn't exist or access denied.

- on server:
   Running systemd 219
(dropbear initialization sequence, everything is ok)
    Starting dropbear
    [123] Apr 23 09:43:56 Running in background
(try to connect remotely via SSH)
    Pubkey auth succeeded for 'root' with key xxx from 192.xxx
    syslogin_perform_logout: logout(pts/0) returned an error: No such file or directory
    Exit (root) Disconnect received
    ERROR: device '/dev/mapper/lvm-archroot' not found. Skipping fsck.
    ERROR: Unable to find rot device '/dev/mapper/lvm-archroot'
    You are being dropped to a recovery shell

If I don't try to login via SSH there is a 15 seconds delay between [123] line and the first ERROR; there is no mean whatsoever to unlock dropbear locally as used to be the case ("enter passphrase for /dev/disk/by-id/wwn-<redacted_id>-part" used to be displayed).

My kernel command line is BOOT_IMAGE=/vmlinuz-linux-lts root=/dev/mapper/lvm-archroot rw quiet cryptdevice=/dev/disk/by-id/wwn-<redacted_id>-part3:crypt ip=192.168.1.xxx::192.168.1.254::arch-medion:eth0:none

[1] https://wiki.archlinux.org/index.php/Dm-crypt/Specialties#Remote_unlocking_of_the_root_.28or_other.29_partition
Comment 7 Tomasz Paweł Gajc 2015-04-23 16:10:40 UTC
Created attachment 115298 [details] [review]
Proposed patch

With this patch systemd does not umounts manually mounted devices
Comment 8 Martin Pitt 2015-04-27 13:25:10 UTC
Lennart recently fixed the recognition of state "tenative" devices, which might help with this, i. e. not declaring such devices as "dead" and trying to unmount them: http://cgit.freedesktop.org/systemd/systemd/commit/?id=f62009410 

It would be great if you could try this patch, on top of the v219-stable branch.
Comment 9 Tomasz Paweł Gajc 2015-04-27 13:28:49 UTC
Thanks for the hint. I'll try systemd wit this patch applied today and give feedback.
Comment 10 Tomasz Paweł Gajc 2015-05-05 08:05:51 UTC
I was unable to test this patch, as it is not a simple "oneliner" second this commit does not apply on systemd 219. Would be nice if someone backported this commit and related to it other commits to systemd-stable/219
Comment 11 Lennart Poettering 2015-05-15 14:39:56 UTC
As mentioned, this should be fixed in git since a while ago. Closing hence.
Comment 12 Lennart Poettering 2015-05-15 14:46:18 UTC
*** Bug 89768 has been marked as a duplicate of this bug. ***
Comment 13 Lennart Poettering 2015-05-15 14:51:51 UTC
*** Bug 90131 has been marked as a duplicate of this bug. ***

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.