Bug 90913

Summary: Failed to mount /sysroot - Nightly live images - LiveCD/DVD
Product: systemd Reporter: poma <pomidorabelisima>
Component: generalAssignee: systemd-bugs
Status: RESOLVED FIXED QA Contact: systemd-bugs
Severity: critical    
Priority: highest CC: bero, harald, juliux.pigface, kzak, leho, teg, tobias.hunger, zbyszek
Version: unspecified   
Hardware: x86-64 (AMD64)   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:
Attachments: Failed to mount /sysroot screenshoot
related rdsosreport

Description poma 2015-06-09 11:31:39 UTC
...
systemd[1]: Mounting /sysroot...
mount[558]: mount: unknown filesystem type 'nfs'
systemd[1]: sysroot.mount: Mount process exited, code=exited status=32
systemd[1]: Failed to mount /sysroot.
systemd[1]: Dependency failed for Initrd Root File System.
...

"Failed to mount /sysroot" occurs with recent Rawhide Live images

Live image list in koji
http://koji.fedoraproject.org/koji/tasks?state=all&view=tree&method=livecd&order=-id


Tested:
- Fedora-Live-Xfce-x86_64-rawhide-20150531.iso
- Fedora-Live-Xfce-x86_64-rawhide-20150602.iso
- Fedora-Live-Xfce-x86_64-rawhide-20150608.iso
- Rawhide-Live-Xfce-sd-git-220-7.iso

systemd 220-x are all broken in this sense - Failed to mount /sysroot
- systemd-220-5.fc23
- systemd-220-3.fc23
- systemd-220-2.fc23
- systemd-220-1.fc23

systemd git is also broken in this sense - Failed to mount /sysroot
- systemd-220-7.gitfba3d90.fc23
Comment 1 poma 2015-06-09 11:32:52 UTC
Created attachment 116391 [details]
Failed to mount /sysroot screenshoot
Comment 2 poma 2015-06-09 11:33:44 UTC
Created attachment 116392 [details]
related rdsosreport
Comment 3 poma 2015-06-09 11:34:57 UTC
Last "bootable" is systemd-219-15.fc23 - as part of
- Test compilation:
  http://goo.gl/Gm4ffO
  ISO/Rawhide-Live-Xfce.iso
Comment 5 Lennart Poettering 2015-06-09 12:54:14 UTC
Well, this is the mount tool saying that it doesn't known the "nfs" file system type. That might have a number of different reasons, such as mount.nfs or the kernel nfs modules missing from the initrd or the kernel in general. Either way it's unlikely to be a systemd issue. Please ask the initrd, util-linux or kernel maintainers for help on this.
Comment 6 Karel Zak 2015-06-09 14:17:37 UTC
mount: unknown filesystem type 'nfs'

... means ENODEV from mount(2) syscall and it's obviously wrong, for NFS we use  /sbin/mount.nfs helper. The helper is probably not available (IMHO initrd setup mistake)
Comment 7 poma 2015-06-12 14:59:12 UTC
Related:

"fstab-generator: don't accept missing root=, but accept root=none"
http://cgit.freedesktop.org/systemd/systemd/commit/?id=093c2cfe3b1ae6081f12927ae7906d90d6623534
Comment 9 Tobias Hunger 2015-06-12 15:11:34 UTC
I have no idea how the fedora cd actually boots, so I am just guessing here...

I would think that the systemd code used to bomb out with a return 0 and a debug message and fedora did some special magic afterwards. Now we are trying to mount and fail on the unexpected syntax. Labels, uuids and whatnot are all supported, but I don't think we handle live:CDLABEL=, but I might be wrong there.

If that is indeed the case then I'd argue that fedora is doing it wrong and should hand us something that we can actually mount...
Comment 10 poma 2015-06-12 15:33:26 UTC
I look forward to all your comments.
Comment 11 Tobias Hunger 2015-06-12 15:34:31 UTC
(In reply to Tobias Hunger from comment #9)
> I have no idea how the fedora cd actually boots, so I am just guessing
> here...
> 
> I would think that the systemd code used to bomb out with a return 0 and a
> debug message and fedora did some special magic afterwards. Now we are
> trying to mount and fail on the unexpected syntax. Labels, uuids and whatnot
> are all supported, but I don't think we handle live:CDLABEL=, but I might be
> wrong there.
> 
> If that is indeed the case then I'd argue that fedora is doing it wrong and
> should hand us something that we can actually mount...

... or disable the generator entirely.
Comment 12 Adam Williamson 2015-06-12 19:23:06 UTC
"I would think that the systemd code used to bomb out with a return 0 and a debug message and fedora did some special magic afterwards. Now we are trying to mount and fail on the unexpected syntax. Labels, uuids and whatnot are all supported, but I don't think we handle live:CDLABEL=, but I might be wrong there."

Yes, this is basically what I'm saying. The offending commit is http://cgit.freedesktop.org/systemd/systemd/commit/?id=093c2cfe3b1ae6081f12927ae7906d90d6623534 . Before that commit, systemd would not try and mount root at all; it would read the 'root=live:CDLABEL=...' , conclude it didn't understand it, and pass on it. Which was fine. That's what we wanted it to do. There is a dracut module which parses it later and knows how to interpret it properly. After that commit, systemd *does* try and mount root - as an NFS share, which is not at all what it is, so it fails.

I can't comment on what the 'correct' fix is, but it's clearly a systemd behaviour change that caused the problem here.
Comment 13 Tobias Hunger 2015-06-12 19:25:13 UTC
So why do you even run that generator in your initrd when you do not want to see it used?
Comment 14 Tomasz Paweł Gajc 2015-06-14 12:31:28 UTC
Hi,

i've investigated this a bit.

In my case i'm booting live iso by special dracut module which does:
1. mount iso
2. mount squashfs
3. mount aufs combined with tmpfs and squashfs to /union dir (plus fstab entry)
4. mout rbind /union to /sysroot
5. dracut does the pivot

In case of systemd-219 this was working, but with 220 not.

Problem lies with systemd-fstab-generator as in case of 219 there is no /run/systemf/generator/sysroot.mount even if i run /lib/systemd/system-generator/systemd-fstab-generator. In this case system-fstab-generator creates file:

cat /run/systemd/generator/run-initramfs-union.mount

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
What=aufs
Where=/run/initramfs/union
Type=aufs

 
While in 220 there is no /run/systemd/generator/run-initramfs-union.mount but this:

cat /run/systemd/generator/sysroot.mount

[Unit]
SourcePath=/proc/cmdline
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=initrd-root-fs.target

[Mount]
What=live:UUID=2015-06-13-09-16-55-00
Where=/sysroot
Options=ro

So sysroot.mount tries to mount ISO with false output:
mount: unknown filesystem type 'nfs'
Comment 16 poma 2015-06-15 16:52:17 UTC
[liveuser@localhost ~]$ cat /proc/cmdline 
BOOT_IMAGE=vmlinuz0 initrd=initrd0.img root=live:CDLABEL=Rawhide-Live-Base-043 rootfstype=auto ro rd.live.image

$ rpm -q anaconda dracut systemd
anaconda-23.10-1.fc23.x86_64
dracut-043-1.fc23.x86_64
systemd-220-9.fc23.x86_64

$ uname -a
Linux localhost 4.1.0-0.rc8.git0.1.fc23.x86_64 #1 SMP Mon Jun 15 13:14:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.