Bug 49112

Summary: systemd-nspawn can't start systemd system
Product: systemd Reporter: Maksim Melnikau <maxposedon>
Component: generalAssignee: Lennart Poettering <lennart>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: medium    
Version: unspecified   
Hardware: x86-64 (AMD64)   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:

Description Maksim Melnikau 2012-04-24 09:27:47 UTC
1. I prepared small gentoo system with systemd in kvm.
2. It boots successfully with kvm
3. I commented lines in fstab
4. Trying use systemd-nspawn
5. System doesn't boots

# this is how I switching testing same data with kvm and systemd-nspawn
umount /media/gentoo-mobile
kpartx -d gentoo-mobile.img
qemu-kvm -machine pc,accel=kvm -drive file=gentoo-mobile.img,if=virtio,media=disk -m 1G -vga qxl -spice disable-ticketing,port=7777
kpartx -a gentoo-mobile.img
mount /dev/mapper/loop0p2 /media/gentoo-mobile

# full log, including manual killing systemd
m_melnikau-vaio kvms # systemd-nspawn -D /media/gentoo-mobile/ /sbin/init
Spawning namespace container on /media/gentoo-mobile (console is /dev/pts/8).
systemd 44 running in system mode. (+PAM +LIBWRAP -AUDIT -SELINUX -SYSVINIT -LIBCRYPTSETUP; gentoo)

Welcome to Gentoo Linux!

Set hostname to <m_melnikau-vaio>.
Netlink failure for request 1: Operation not permitted
Failed to configure loopback device: Operation not permitted
Failed to enable ctrl-alt-del handling: Operation not permitted
Failed to open /dev/tty0: No such file or directory
Failed to open /dev/autofs: No such file or directory
Failed to initialize automounter: No such file or directory
Unit proc-sys-fs-binfmt_misc.automount entered failed state.
Starting File System Check on Root Device...
Started Runtime Directory                                                                                                  [  OK  ]
Starting Setup Virtual Console...
Started Set Up Additional Binary Formats                                                                                   [  OK  ]
Started Huge Pages File System                                                                                             [  OK  ]
Starting POSIX Message Queue File System...
Starting Debug File System...
Started Security File System                                                                                               [  OK  ]
Started Load Kernel Modules                                                                                                [  OK  ]
Starting Configuration File System...
Starting FUSE Control File System...
Starting Apply Kernel Variables...
Started udev Kernel Device Manager                                                                                         [  OK  ]
Starting udev Coldplug all Devices...
Starting Journal Service...
Started Journal Service                                                                                                    [  OK  ]
Starting Remount API VFS...
Started File System Check on Root Device                                                                                   [  OK  ]
Started Media Directory                                                                                                    [  OK  ]
systemd-vconsole-setup.service: main process exited, code=exited, status=1
Failed to start Setup Virtual Console                                                                                      [FAILED]
See 'systemctl status systemd-vconsole-setup.service' for details.
Unit systemd-vconsole-setup.service entered failed state.
Started POSIX Message Queue File System                                                                                    [  OK  ]
Started Debug File System                                                                                                  [  OK  ]
Started Configuration File System                                                                                          [  OK  ]
Started FUSE Control File System                                                                                           [  OK  ]
systemd-sysctl.service: main process exited, code=exited, status=1
Failed to start Apply Kernel Variables                                                                                     [FAILED]
See 'systemctl status systemd-sysctl.service' for details.
Unit systemd-sysctl.service entered failed state.
systemd-journald.service: main process exited, code=exited, status=205
Unit systemd-journald.service entered failed state.
Started Remount API VFS                                                                                                    [  OK  ]
Started udev Coldplug all Devices                                                                                          [  OK  ]
Starting Remount Root FS...
Starting Journal Service...
Started Journal Service                                                                                                    [  OK  ]
systemd-journald.service: main process exited, code=exited, status=205
Started Remount Root FS                                                                                                    [  OK  ]
Starting Load Random Seed...
Starting Recreate Volatile Files and Directories...
Started Load Random Seed                                                                                                   [  OK  ]
Started Recreate Volatile Files and Directories                                                                            [  OK  ]
Starting D-Bus System Message Bus...
Starting Login Service...
Starting Permit User Sessions...
Starting Journal Service...
Started Journal Service                                                                                                    [  OK  ]
systemd-logind.service: main process exited, code=exited, status=205
Failed to start Login Service                                                                                              [FAILED]
See 'systemctl status systemd-logind.service' for details.
Unit systemd-logind.service entered failed state.
Started Permit User Sessions                                                                                               [  OK  ]
systemd-journald.service: main process exited, code=exited, status=205
Unit systemd-journald.service entered failed state.
Started D-Bus System Message Bus                                                                                           [  OK  ]
Starting Console Shell...
Started Console Shell                                                                                                      [  OK  ]
Failed at step EXEC spawning /sbin/sulogin: No such file or directory
Starting Cleanup of Temporary Directories...
Started Cleanup of Temporary Directories                                                                                   [  OK  ]
Starting Journal Service...
Started Journal Service                                                                                                    [  OK  ]
systemd-journald.service: main process exited, code=exited, status=205
Unit systemd-journald.service entered failed state.
Starting Journal Service...
Started Journal Service                                                                                                    [  OK  ]
systemd-journald.service: main process exited, code=exited, status=205
Unit systemd-journald.service entered failed state.
Starting Journal Service...
Started Journal Service                                                                                                    [  OK  ]
Stopping /proc/sys...
Stopping /etc/localtime...
Stopping /dev/console...
Stopping D-Bus System Message Bus...
Stopping Console Shell...
Starting Save Random Seed...
systemd-journald.service: main process exited, code=exited, status=205
Unit systemd-journald.service entered failed state.
Stopped /proc/sys                                                                                                          [  OK  ]
Stopped D-Bus System Message Bus                                                                                           [  OK  ]
Stopped /etc/localtime                                                                                                     [  OK  ]
dev-console.mount mount process exited, code=exited status=1
Stopped /dev/console                                                                                                       [  OK  ]
Started Save Random Seed                                                                                                   [  OK  ]
Starting Journal Service...
systemd-journald.service start request repeated too quickly, refusing to start.
Unit systemd-journald.socket entered failed state.
Failed to start Journal Service                                                                                            [FAILED]
See 'systemctl status systemd-journald.service' for details.
Stopped Console Shell                                                                                                      [  OK  ]
Stopping Permit User Sessions...
systemd-user-sessions.service: control process exited, code=exited status=209
Stopped Permit User Sessions                                                                                               [  OK  ]
Unit systemd-user-sessions.service entered failed state.
Starting Notify Audit System and Update UTMP about System Runlevel Changes...
Stopping Remount API VFS...
Stopped Remount API VFS                                                                                                    [  OK  ]
Stopping Remount Root FS...
Stopped Remount Root FS                                                                                                    [  OK  ]
systemd-update-utmp-runlevel.service: main process exited, code=exited, status=209
Unit systemd-update-utmp-runlevel.service entered failed state.
media.mount mount process exited, code=exited status=209
systemd-update-utmp-shutdown.service: main process exited, code=exited, status=209
Unit systemd-update-utmp-shutdown.service entered failed state.
poweroff.service: main process exited, code=exited, status=209
Job poweroff.target/start failed with result 'dependency'.
Unit poweroff.service entered failed state.

^C'/sbin/init terminated by signal KILL.
Comment 1 Maksim Melnikau 2012-04-24 09:37:46 UTC
root@m_melnikau-vaio ~ # systemctl status systemd-vconsole-setup.service
systemd-vconsole-setup.service - Setup Virtual Console
          Loaded: loaded (/usr/lib64/systemd/system/systemd-vconsole-setup.service; static)
          Active: failed (Result: exit-code) since Tue, 24 Apr 2012 19:34:57 +0300; 33s ago
         Process: 5 ExecStart=/usr/lib/systemd/systemd-vconsole-setup (code=exited, status=1/FAILURE)
          CGroup: name=systemd:/system/kdm@.service/tty7/nspawn-15815/system/systemd-vconsole-setup.service
Comment 2 Maksim Melnikau 2012-04-24 11:27:16 UTC
(I sumlinked /sbin/sulogin to /bin/bash for now)

system also starts in RO mode, may be I need some fstab entry?
root@m_melnikau-vaio ~ # touch a
touch: a: Read-only file system
Comment 3 Maksim Melnikau 2012-04-24 11:34:49 UTC
(In reply to comment #2)
> (I sumlinked /sbin/sulogin to /bin/bash for now)
> 
> system also starts in RO mode, may be I need some fstab entry?
> root@m_melnikau-vaio ~ # touch a
> touch: a: Read-only file system
looks like it because of 2nd run, systemd-nspawn after unsuccessful run remounts dir RO
Comment 4 Lennart Poettering 2012-05-02 18:07:31 UTC
Could you retest this with systemd git on the host and in the container? With systemd git I tested Debian and Fedora in the container and they now boot entirely cleanly on a Fedora host.
Comment 5 Maksim Melnikau 2012-05-03 04:10:01 UTC
(In reply to comment #4)
> Could you retest this with systemd git on the host and in the container? With
> systemd git I tested Debian and Fedora in the container and they now boot
> entirely cleanly on a Fedora host.
All failures exactly the same (with and without commenting lines in /etc/fstab)

This is image, with which I do experiments:
http://dl.dropbox.com/u/8793962/gentoo-systemd.img.lzma

With such script:
umount /media/gentoo-mobile
kpartx -d gentoo-systemd.img
qemu-kvm -machine pc,accel=kvm -drive file=gentoo-systemd.img,if=virtio,media=disk -m 64M -vga qxl -spice disable-ticketing,port=7777
kpartx -a gentoo-systemd.img
mount /dev/mapper/loop0p2 /media/gentoo-mobile

Boot works via kvm, and doesn't work via nspawn.
Comment 6 Lennart Poettering 2012-05-03 05:21:18 UTC
(In reply to comment #5)
> (In reply to comment #4)
> > Could you retest this with systemd git on the host and in the container? With
> > systemd git I tested Debian and Fedora in the container and they now boot
> > entirely cleanly on a Fedora host.
> All failures exactly the same (with and without commenting lines in /etc/fstab)

Hmm, exactly the same? Can't really believe that. At least the block of early messages "Failed to configure loopback device..." and so on should be gone?

Are you sure you installed systemd git both on the host and in the container, and also made sure you dropped all old unit files from earlier installations that might still be lurking?

If so, could you please paste an updated boot log?
Comment 7 Maksim Melnikau 2012-05-03 06:18:23 UTC
Sorry, I didn't updated guest systemd. Will check in a day.
Comment 8 Maksim Melnikau 2012-05-04 00:15:08 UTC
Unstall properly systemd (in Gentoo) requires bug 49459 fix.
Comment 9 Maksim Melnikau 2012-05-09 14:34:08 UTC
(In reply to comment #6)
> Hmm, exactly the same? Can't really believe that. At least the block of early
> messages "Failed to configure loopback device..." and so on should be gone?
> 
> Are you sure you installed systemd git both on the host and in the container,
> and also made sure you dropped all old unit files from earlier installations
> that might still be lurking?
> 
> If so, could you please paste an updated boot log?
Today I updated guest systemd and running systemd-nspawn from systemd master. (Don't update host systemd - a bit afraid of problems).

There is 3 minor problems now:
(empty fstab)
1. Failed to start Save Random Seed. See 'systemctl status systemd-random-seed-save.service' for details.
2. After systemd-nspawn and poweroff from guest:
/dev/mapper/loop0p2 on /media/gentoo-mobile type btrfs (rw,relatime,space_cache)
becomes
/dev/mapper/loop0p2 on /media/gentoo-mobile type btrfs (ro,relatime,space_cache)
(non empty fstab, with entries for boot from kvm)
3. Failed to open /dev/autofs: No such file or directory

What is expected behavior of systemd-nspawn in such cases?

New full log with empty fstab.
====================================
# ./systemd-nspawn -D /media/gentoo-mobile /sbin/init 
Spawning namespace container on /media/gentoo-mobile (console is /dev/pts/6).
systemd 182 running in system mode. (+PAM +LIBWRAP -AUDIT -SELINUX +IMA -SYSVINIT -LIBCRYPTSETUP; gentoo)

Welcome to Gentoo Linux!

Set hostname to <m_melnikau-vaio>.
Starting Temporary Directory...                                                                                                    
Starting POSIX Message Queue File System...                                                                                        
Starting Debug File System...                                                                                                      
Starting Configuration File System...                                                                                              
Starting FUSE Control File System...                                                                                               
Starting Journal Service...                                                                                                        
Started Journal Service.                                                                                                   [  OK  ]
Starting Remount Root and Kernel File Systems...                                                                                   
Started Temporary Directory.                                                                                               [  OK  ]
Started POSIX Message Queue File System.                                                                                   [  OK  ]
Started Debug File System.                                                                                                 [  OK  ]
Started Configuration File System.                                                                                         [  OK  ]
Started FUSE Control File System.                                                                                          [  OK  ]
Started Remount Root and Kernel File Systems.                                                                              [  OK  ]
Starting Load Random Seed...                                                                                                       
Starting Recreate Volatile Files and Directories...                                                                                
Started Load Random Seed.                                                                                                  [  OK  ]
Started Recreate Volatile Files and Directories.                                                                           [  OK  ]
Starting Login Service...                                                                                                          
Starting Permit User Sessions...                                                                                                   
Starting D-Bus System Message Bus...                                                                                               
Started Permit User Sessions.                                                                                              [  OK  ]
Starting Console Getty...                                                                                                          
Started Console Getty.                                                                                                     [  OK  ]
Starting Cleanup of Temporary Directories...                                                                                       
Started D-Bus System Message Bus.                                                                                          [  OK  ]
Started Cleanup of Temporary Directories.                                                                                  [  OK  ]
Started Login Service.                                                                                                     [  OK  ]


This is m_melnikau-vaio.unknown_domain (Linux x86_64 3.3.0-gentoo) 00:16:32

m_melnikau-vaio login: root
Password: 
m_melnikau-vaio ~ # poweroff 
Stopping Login Service...                                                                                                          
Stopping Console Getty...                                                                                                          
Stopping D-Bus System Message Bus...                                                                                               
Starting Save Random Seed...                                                                                                       
Hangup
m_melnikau-vaio ~ # Stopped Login Service.                                                                                                     [  OK  ]
Stopped Console Getty.                                                                                                     [  OK  ]
Stopped D-Bus Syst\em Message Bus.                                                                                          [  OK  ]
Failed to start Save Random Seed.                                                                                          [FAILED]
See 'systemctl status systemd-random-seed-save.service' for details.                                                               
Stopping Permit User Sessions...                                                                                                   
Stopped Permit User Sessions.                                                                                              [  OK  ]
Starting Update UTMP about System Shutdown...                                                                                      
Stopping Temporary Directory...                                                                                                    
Stopping Remount Root and Kernel File Systems...                                                                                   
Stopped Remount Root and Kernel File Systems.                                                                              [  OK  ]
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounted /sys/fs/fuse/connections.
Unmounted /sys/kernel/config.
Unmounted /sys/kernel/debug.
Unmounted /dev/mqueue.
Detaching loop devices.
Exiting container.
====================================

New full log with non-empty fstab.
====================================
m_melnikau-vaio systemd-45 # ./systemd-nspawn -D /media/gentoo-mobile /sbin/init
Spawning namespace container on /media/gentoo-mobile (console is /dev/pts/6).
systemd 182 running in system mode. (+PAM +LIBWRAP -AUDIT -SELINUX +IMA -SYSVINIT -LIBCRYPTSETUP; gentoo)

Welcome to Gentoo Linux!

Set hostname to <m_melnikau-vaio>.
Starting Temporary Directory...                                                                                                    
Starting POSIX Message Queue File System...                                                                                        
Starting Debug File System...                                                                                                      
Starting Configuration File System...                                                                                              
Starting FUSE Control File System...                                                                                               
Starting Journal Service...                                                                                                        
Started Journal Service.                                                                                                   [  OK  ]
Starting Remount Root and Kernel File Systems...                                                                                   
Failed to open /dev/autofs: No such file or directory
Failed to initialize automounter: No such file or directory
Job local-fs.target/start failed with result 'dependency'.
Triggering OnFailure= dependencies of local-fs.target.
Job graphical.target/start failed with result 'dependency'.
Job multi-user.target/start failed with result 'dependency'.
Job console-getty.service/start failed with result 'dependency'.
Job getty@tty1.service/start failed with result 'dependency'.
Job systemd-user-sessions.service/start failed with result 'dependency'.
Job systemd-logind.service/start failed with result 'dependency'.
Job dbus.service/start failed with result 'dependency'.
Job basic.target/start failed with result 'dependency'.
Job systemd-tmpfiles-clean.timer/start failed with result 'dependency'.
Job dbus.socket/start failed with result 'dependency'.
Job update-etc-systemd-symlinks.path/start failed with result 'dependency'.
Job boot.mount/start failed with result 'dependency'.
Unit boot.automount entered failed state.
Comment 10 Maksim Melnikau 2012-06-22 11:55:44 UTC
Kind of new problem: systemd-185(host and guest) couldn't start dbus.service in guest.

==============
localhost kvms # systemctl --version                                            
systemd 185
gentoo
+PAM +LIBWRAP -AUDIT -SELINUX +IMA -SYSVINIT -LIBCRYPTSETUP
localhost kvms # systemd-nspawn -D /media/gentoo-mobile systemctl --version
Spawning namespace container on /media/gentoo-mobile (console is /dev/pts/7).
systemd 185
gentoo
+PAM +LIBWRAP -AUDIT -SELINUX +IMA -SYSVINIT -LIBCRYPTSETUP


localhost kvms # systemd-nspawn -D /media/gentoo-mobile -b                 
Spawning namespace container on /media/gentoo-mobile (console is /dev/pts/7).
systemd 185 running in system mode. (+PAM +LIBWRAP -AUDIT -SELINUX +IMA -SYSVINIT -LIBCRYPTSETUP; gentoo)

Welcome to Gentoo Linux!

Set hostname to <m_melnikau-vaio>.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Reached target Swap.
[  OK  ] Listening on Journal Socket.
         Starting Remount Root and Kernel File Systems...
         Mounting FUSE Control File System...
         Mounting Debug File System...
         Mounting Configuration File System...
         Mounting POSIX Message Queue File System...
         Starting Journal Service...
[  OK  ] Started Journal Service.
[  OK  ] Started Remount Root and Kernel File Systems.
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Mounted Debug File System.
[  OK  ] Mounted Configuration File System.
[  OK  ] Mounted POSIX Message Queue File System.
         Starting Load Random Seed...
[  OK  ] Reached target Local File Systems (Pre).
         Mounting Temporary Directory...
[  OK  ] Started Load Random Seed.
[  OK  ] Mounted Temporary Directory.
[  OK  ] Reached target Local File Systems.
         Starting Recreate Volatile Files and Directories...
[  OK  ] Started Recreate Volatile Files and Directories.
[  OK  ] Reached target System Initialization.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting Permit User Sessions...
         Starting Login Service...
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Permit User Sessions.
         Starting Console Getty...
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
         Starting Cleanup of Temporary Directories...
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Cleanup of Temporary Directories.
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
         Starting D-Bus System Message Bus...
[FAILED] Failed to start D-Bus System Message Bus.
         See 'systemctl status dbus.service' for details.


This is m_melnikau-vaio.unknown_domain (Linux x86_64 3.4.3-gentoo) 21:53:39

m_melnikau-vaio login: [FAILED] Failed to start Login Service.
         See 'systemctl status systemd-logind.service' for details.
[  OK  ] Reached target Multi-User.
[  OK  ] Reached target Graphical Interface.


This is m_melnikau-vaio.unknown_domain (Linux x86_64 3.4.3-gentoo) 21:53:54

m_melnikau-vaio login: root
Password: 
m_melnikau-vaio ~ # systemctl status dbus.service
dbus.service - D-Bus System Message Bus
         Loaded: loaded (/usr/lib64/systemd/system/dbus.service; static)
         Active: failed (Result: exit-code) since Fri, 22 Jun 2012 21:53:34 +0300; 28s ago
        Process: 23 ExecStart=/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation (code=exited, status=206/OOM_ADJUST)
         CGroup: name=systemd:/user/root/c2/nspawn-30821/system/dbus.service

m_melnikau-vaio ~ # /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
Failed to start message bus: No socket received.
m_melnikau-vaio ~ # ls -la /run/    
dbus/    lock/    log/     systemd/ user/    utmp     
m_melnikau-vaio ~ # ls -la /run/dbus/system_bus_socket 
srw-rw-rw- 1 root root 0 Jun 22 21:53 /run/dbus/system_bus_socket
Comment 11 Lennart Poettering 2012-07-20 12:36:47 UTC
The OOM adjust issue should be fixed now. Could you retest please?
Comment 12 Maksim Melnikau 2012-07-22 16:00:28 UTC
(In reply to comment #11)
> The OOM adjust issue should be fixed now. Could you retest please?
systemd-nspawn works nice for me now.

But there is few minor systemd's issues:
1. if guest rootfs mounted rw, it will be ro after systemd-nspawn, e.g.:

After systemd-nspawn -D /media/gentoo-mobile and poweroff from guest:
/dev/mapper/loop0p2 on /media/gentoo-mobile type btrfs (rw,relatime,space_cache)
becomes
/dev/mapper/loop0p2 on /media/gentoo-mobile type btrfs (ro,relatime,space_cache)
(non empty fstab, with entries for boot from kvm)

2. if systemd runned from systemd-nspawn, and domain name not available boot paused before login prompt for a long time, e.g.:

# time hostname -d
hostname -d  0.00s user 0.00s system 0% cpu 40.038 total

This is kvm-gentoo-mobile.<PAUSE FOR ~30 seconds>unknown_domain (Linux x86_64 3.4.4-gentoo) 18:58:21

kvm-gentoo-mobile login:

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.