Bug 75512

Created attachment 94740 [details]
Contextual Info and results of various commands under emergency shell

Hi,

I have been using systemd-208 without issue since its release on source-based i686-pc-linux-gnu systems (under gcc-4.8.2, glibc-2.19, binutils-2.24, and linux-3.13.5.) which I maintain.

On upgrading from systemd-208 to systemd-209 (and then to systemd-210), my development
system no longer is able to mount non-root disk partitions listed in /etc/fstab at boot time. On this system, I'm using MBR (not GPT) disk partitions, and not using lvm.

My grub2 boot kernel line is:

  linux   /boot/vmlinuz-3.13.5 root=/dev/sda7 ro raid=noautodetect quiet \
          acpi_os_name="Microsoft Windows NT" \
          systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M

I have attached a number of text files with detailed contextual infomation, but here I'll simply summarize. At boot, I get the messages:

systemd-fsck[62]: /sbin/fsck.xfs: XFS file system.
[ * ] (1 of 4) A start job is running for dev-sda10.device. (1min 26s / 1min 30s)

The root partition is fsck'd and mounted properly, but 4 other partitions specified in /etc/fstab, /dev/sda{1,6,9,10}, are not (/dev/sda1 is a swap partition). For each of /dev/sda{1,6,9,10}, I get journal messages like:

Feb 24 05:01:58 b-movie systemd[1]: Job dev-sda10.device/start timed out.
Feb 24 05:01:58 b-movie systemd[1]: Timed out waiting for device dev-sda10.device.
Feb 24 05:01:58 b-movie systemd[1]: Dependency failed for /home.
Feb 24 05:01:58 b-movie systemd[1]: Dependency failed for Local File Systems.
Feb 24 05:01:58 b-movie systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
Feb 24 05:01:58 b-movie systemd[1]: Dependency failed for File System Check on /dev/sda10.

After the 1m 30s timeout, I'm greeted with an emergency shell; I login, then issue 'systemctl daemon-reload', and then, 'exit'.

Immediately after issuing 'exit', booting resumes, the fstab entries are fsck'd and mounted, and all appears to be Ok.

Note that the hardware is fine as I am able to boot on the system without issue under systemd-208 on a different partition which also shares the use of /dev/sda{1,6,9,10}.

I'm unable to run systemd-analyze blame in the emergency shell as dbus.service/socket is stopped on entry to the emergency shell, and /run/dbus is thus no longer present; I get

systemd-analyze blame
Failed to create bus connection: No such file or directory

The absent file (from strace) is:
connect(3, {sa_family=AF_LOCAL, sun_path="/run/dbus/system_bus_socket"}, 29) = -1 ENOENT (No such file or directory)

In addition to the disk device issue, I also have a (probably related) problem with network device unit creation/detection: I have a udev rule file (in /etc/udev/rules.d) for network activation which, while a bit hackish, works just fine under systemd-208, with the following line:

ACTION=="add", SUBSYSTEM=="net", NAME=="en*|wl*", TAG+="systemd", ENV{SYSTEMD_WANTS}+="dhcpcd@$name.service"

and in dhcpcd@.service, I have the lines:

BindsTo = sys-subsystem-net-devices-%i.device
After   = sys-subsystem-net-devices-%i.device

My Ethernet interface is enp2s2; in systemd-208 the dhcpcd@enp2s2.service is started without issue, while in systemd-209/210, it fails due to sys-subsystem-net-devices-enp2s2.device never appearing.

Again, if I issue 'systemctl daemon-reload' in the emergency shell, then immediately after this, sys-subsystem-net-devices-enp2s2.device becomes 'active (plugged)', and I can manually start dhcpcd.

So, it appears to me that the 'Timed out waiting for device dev-sda10.device' message is the key here; for some reason the device unit creation is failing at boot time for non-'/' disk partitions, and for network devices.

I have attached a zip archive (systemd-209_210-boot-issue.zip) containing number of text files providing contextual information from within the emergency shell:

unzip -l systemd-209_210-boot-issue.zip

Archive:  systemd-209_210-boot-issue.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  02-25-2014 18:47   systemd-209_210-boot-issue/
     2471  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:dev-disk-contents
      335  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:df
   100456  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:dmesg
      268  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:etc-fstab
   118487  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:journalctl_xb
     1355  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:lsblk_f
     1269  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:lsmod
     2309  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:ps_xa
      928  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:run-systemd-contents
     4588  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:summary
       17  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:systemctl_list-jobs
     1932  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:systemd-build-info
     3774  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:systemd_list-units
     1962  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:udevadm_info_qall_p-sda1
     1961  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:udevadm_info_qall_p-sda10
     1953  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:udevadm_info_qall_p-sda6
     1953  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:udevadm_info_qall_p-sda7
     1952  02-25-2014 18:43   systemd-209_210-boot-issue/00-boot-issue:udevadm_info_qall_p-sda9
---------                     -------
   247970                     19 files

Unfortunately, I'm not at all familiar with systemd internals, so I'm not sure where to begin looking at the code to root out this issue. I can, however, provide additional info, run tests, test patches, etc. Any ideas/help would be greatly appreciated.

thanks much,
John