Which distro is this?

If this is something more exotic: are you sure the udev support in the LVM utils has been enabled during compilation?

I'm running gentoo linux with gentoo-kernel 2.6.36-r5 (which is a lightly patched 2.6.36.2 kernel)

Lvm2 (2.02.73) should be compiled with udev, the configure parameters the ebuild has set at least include a lot of udev related stuff. 

./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --enable-readline --disable-selinux --enable-pkgconfig --libdir=/lib64 --with-usrlibdir=/usr/lib64 --enable-udev_rules --enable-udev_sync --with-udevdir=/lib64/udev/rules.d/ --enable-dmeventd --enable-cmdlib --enable-applib --enable-fsadm --enable-static_link --with-mirrors=internal --with-snapshots=internal --with-lvm1=internal --with-clvmd=none --with-cluster=none --sbindir=/sbin --with-staticdir=/sbin --with-dmeventd-path=/sbin/dmeventd CLDFLAGS=-Wl,-O1 -Wl,--as-needed

As configure --help lists, all udev related options are enabled (I guess --enable-udev-rules, --enable-udev_sync and --enable-dmeventd are candidates in question)



Here is some more info about my environment:

$ cat /proc/version 
Linux version 2.6.36-gentoo-r5 (root@Ganymed) (gcc version 4.4.4 (Gentoo 4.4.4-r2 p1.3, pie-0.4.5) ) #4 SMP Mon Jan 24 01:04:27 CET 2011

$ lvm version
  LVM version:     2.02.73(2) (2010-09-18)
  Library version: 1.02.54 (2010-08-18)
  Driver version:  4.18.0


$ udevd --version
164

$ systemctl --version
systemd 16
gentoo
+PAM +LIBWRAP -AUDIT -SELINUX +SYSVINIT +LIBCRYPTSETUP

I'm getting closer in solving that issue. Your note about udev<->lvm pointed me in the right direction:

Although udev support was compiled into the lvm tools vgscan and vgchanged invoked in my lvm.service file did not call udev while activating the logical volume group. I found out that my /dev nodes were only created, because I used the --mknodes switch on vgscan. Vgscan then detected that udev did noch create the /dev/mapper/raid* nodes and fell back on creating it on its own. Without the swith the nodes wouldn't have been created at all. This was my lvm.service file:

---
[Unit]
Description=Linux Volume Manager
DefaultDependencies=no
Before=fsck-root.service local-fs.target
After=udev.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/vgscan --mknodes --ignorelockingfailure
ExecStart=/sbin/vgchange -ay raid --sysinit
ExecStop=/sbin/vgchange -an raid

[Install]
WantedBy=sysinit.target
---

As I found out here: http://bugs.gentoo.org/show_bug.cgi?id=330651, the reason for udev not creating this nodes is, that I use an initrd that activates this volume group before systemd is started (because root itself is on a LV in this VG), therefore the volume group is already active when vgchange is called the second time while booting and it assumes the node do not need to be created. However, vgscan checks for the nodes and, if invoked with --mknodes, creates them on its own. Therefore also no TAGS=:systemd:, and therefore no monitoring of systemd of this devices, thus mounting failes.

I fixed the situation with udev not creating the device nodes by removing the --mknodes parameter and add a manual trigger for udev to the service-file, so that it deals with the already-active vg - as it is discussed as a possible solution for this in http://bugs.gentoo.org/show_bug.cgi?id=330651, like this:

...
 [Service]
 Type=oneshot
 RemainAfterExit=yes
+ExecStart=/sbin/udevadm trigger --action=change --attr-match=dm/name
-ExecStart=/sbin/vgscan --mknodes --ignorelockingfailure 
+ExecStart=/sbin/vgscan --ignorelockingfailure 
 ExecStart=/sbin/vgchange -ay raid --sysinit
 ExecStop=/sbin/vgchange -an raid
...

Now udev created the device nodes. Yet my problem is not completely solved as this seems to not include that udev applies the same rules it would, if it was triggered by a real activation of an LV or VG. 

The udev attributes for a device created like this look like this:

$ udevadm info --query all --name=/dev/mapper/raid-data

P: /devices/virtual/block/dm-3
N: dm-3
S: mapper/raid-data
S: block/253:3
E: UDEV_LOG=3
E: DEVPATH=/devices/virtual/block/dm-3
E: MAJOR=253
E: MINOR=3
E: DEVNAME=/dev/dm-3
E: DEVTYPE=disk
E: SUBSYSTEM=block
E: DM_SBIN_PATH=/sbin
E: DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1
E: DM_UDEV_DISABLE_DISK_RULES_FLAG=1
E: DM_UDEV_DISABLE_OTHER_RULES_FLAG=1
E: DM_NAME=raid-data
E: DM_UUID=LVM-CV3wblfNWXped1XylxUQYfLbhcy0vgb6EPCMiCly98ivfxKrO8OIf3TtuLlvwatr
E: DM_SUSPENDED=0
E: DM_UDEV_RULES_VSN=2
E: DM_VG_NAME=raid
E: DM_LV_NAME=data
E: DEVLINKS=/dev/mapper/raid-data /dev/block/253:3
E: UDISKS_PRESENTATION_NOPOLICY=1

The systemd rule to set the "TAGS=:systemd:" was not applied, so systemd is not monitoring this device, thus cannot mount it if requested.

If I deactive an LV and reactivate it again however udev applies the correct rules:

$ lvchange -a n /dev/raid/data
$ lvchange -a y /dev/raid/data

$ udevadm info --query all --name=/dev/mapper/raid-data

P: /devices/virtual/block/dm-3
N: dm-3
S: mapper/raid-data
S: raid/data
S: disk/by-id/dm-name-raid-data
S: disk/by-id/dm-uuid-LVM-CV3wblfNWXped1XylxUQYfLbhcy0vgb6EPCMiCly98ivfxKrO8OIf3TtuLlvwatr
S: disk/by-uuid/9ce58e77-f9a1-4a6f-a8b6-5b4539c2f692
S: block/253:3
E: UDEV_LOG=3
E: DEVPATH=/devices/virtual/block/dm-3
E: MAJOR=253
E: MINOR=3
E: DEVNAME=/dev/dm-3
E: DEVTYPE=disk
E: SUBSYSTEM=block
E: DM_SBIN_PATH=/sbin
E: DM_UDEV_PRIMARY_SOURCE_FLAG=1
E: DM_NAME=raid-data
E: DM_UUID=LVM-CV3wblfNWXped1XylxUQYfLbhcy0vgb6EPCMiCly98ivfxKrO8OIf3TtuLlvwatr
E: DM_SUSPENDED=0
E: DM_UDEV_RULES_VSN=2
E: DM_VG_NAME=raid
E: DM_LV_NAME=data
E: DEVLINKS=/dev/mapper/raid-data /dev/raid/data /dev/disk/by-id/dm-name-raid-data /dev/disk/by-id/dm-uuid-LVM-CV3wblfNWXped1XylxUQYfLbhcy0vgb6EPCMiCly98ivfxKrO8OIf3TtuLlvwatr /dev/disk/by-uuid/9ce58e77-f9a1-4a6f-a8b6-5b4539c2f692 /dev/block/253:3
E: ID_FS_UUID=9ce58e77-f9a1-4a6f-a8b6-5b4539c2f692
E: ID_FS_UUID_ENC=9ce58e77-f9a1-4a6f-a8b6-5b4539c2f692
E: ID_FS_VERSION=1.0
E: ID_FS_TYPE=ext4
E: ID_FS_USAGE=filesystem
E: UDISKS_PRESENTATION_NOPOLICY=1
E: UDISKS_DM_TARGETS_COUNT=1
E: UDISKS_DM_TARGETS_TYPE=striped
E: UDISKS_DM_TARGETS_START=0
E: UDISKS_DM_TARGETS_LENGTH=419438592
E: TAGS=:systemd:

Now with the tag set I did a "systemd daemon-reload" to be safe and voià, the volumes get mounted.

Now I only have to figure out the proper way to tell udev to apply the rules upon boot up, as if the already activated vg is activated then.

I finally solved it! So here is what was to be done do initiate LVM correctly, if it a VG has already been activated previously in an initrd:

I didn't like my "call udevadm manually" approach i explained in the last post anyway, because it seemed very hacky to me - I thought, this should be handled by the lvm tools or at least with the correct udev rules themselves, not by any manually synthesised udev events. As it turned out, the lvm tools can do this quite well.

As I describes in my last post, what I tried to accomplish by calling vgscan / vgchange twice (once in the inird to get root, a second time while booting) was not to activate the VG, as this is already done in the initrd, but to re-create the nodes and re-read the data by udev. (this is as far as I know most distributions do it in their initscripts as well - at least gentoo does). So, this got me thinking: This does not seem right. Why doing something like this twice? Shouldn't there be a way to just re-gather the data from the already activated VG? And yes, there is: 

The lvm tools also include a tool called vgmknodes, which actually does NOT what I first expected (just creating the dev nodes while by-passing udev), but actually exactly the thing I needed: asking udev to do that (and only fall back to direct node creating, if that fails). So I removed all the vgscan, vgchange and udevadm calls from my lvm.service file and just call vgmknodes. That did the same as before - udev rules still were incomplete. So I added the "--refresh" parameter, that seems to do exacly what is to be done here: re-read the metadata from dem VGs/LVs and hand that over to udevd, which applies all neccessary rules (including creating the device nodes). So my script simply looks like this then:

$ cat /etc/systemd/system/lvm.service

[Unit]
Description=Linux Volume Manager
DefaultDependencies=no
Before=fsck-root.service local-fs.target 
After=udev.service

[Service]
Type=oneshot
RemainAfterExit=yes

ExecStart=/sbin/vgmknodes --refresh

ExecStop=/sbin/lgchange -an 
ExecStop=/sbin/vgchange -an 

[Install]
WantedBy=sysinit.target

(The changed part is the ExecStart call - i also added "ExecStop=/sbin/lvchange -an" before vgchage, so if a vg cannot be deactivated entirely (as in my case, as root itself is an LV) at least the other lvs are deactivated - but that has nothing to do with the problem here).


So this approach does not only look more like doing only what is neccessary to be done, it also solves my problem. After vgmknodes --refresh is called, udev applied all its rules (including adding the systemd rule to add the "systemd" tag), so now systemd can finally mount all LVs - or automount it on demand.

I marked the bug report as resolved / Notabug, as this is not an issue with systemd and there also is no bug. 


Note: If someone wants to use this service file: This of course is only the way to do it if you use an initrd which activates all your VGs before calling systemd. If someone wants to use my service-file but has no initrd, the right approach would be the default "vgscan ; vgchange -ay" call. If you only activate one of your VGs in initrd, and want to activate all of them at boot, you should do both. Also note, that if lvm is called before root is mount rw (my way with "Before=fsck-root.service" assures that), you have to call "vgscan --ignorelockingfailure" and "vgchange -ay --sysinit", otherwise it will fail as root isn't writable. This of course is neccessary in my case, as root is an LV itself which needs to be active before it can be fsck'ed.

If one would think of creating an lvm.service file for genereal use, one should probably call a script that checks if a VG has already been activated, re-reads this, and activate all other remaining vgs.