Previous versions of udisks had partial support for MD RAID devices. New version of udisks is missing this functionality. Intel would like to offer help in bringing back the MD RAID support in udisks and to add support for MD RAID with IMSM metadata. This would require defining new API for udisks with the respect of IMSM specific operations (like including idea of containers) and reimplementing the old support in newer versions.
Sounds great to me - I also mentioned MD-RAID in my latest blog entry [1] about udisks/gnome-disk-utility so this is great timing :-) [1] : http://davidz25.blogspot.com/2012/08/disks-improvements-in-gnome-36.html
Just for reference, the corresponding bug for using this functionality in GNOME Disks is this one https://bugzilla.gnome.org/show_bug.cgi?id=682352 The way I'd like us to work on this is that once we have a concrete idea of what the GUI should look (UI mockups etc.) like we can start talking about what API additions we want in the udisks D-Bus interfaces. Then when we have that, we can start talking about implementing it. Sounds OK? Thanks!
Mostly cosmetic low-hanging fruit but just added textual descriptions for IMSM members, see http://cgit.freedesktop.org/udisks/commit/?id=463ee2dd5cf9c5ffdbd519f183d1eae4dbb35bb6 http://people.freedesktop.org/~david/imsm_before.png http://people.freedesktop.org/~david/imsm_after.png More importantly, I managed to install Fedora on a system with a IMSM container (4 x 500GB disks) and a single 100GB RAID-5 volume. This will be my main testing/development system for this feature.
I've just created wip/mdraid branches in udisks and gnome-disk-utility repositories with very rudimentary code - still lots of work left! http://cgit.freedesktop.org/udisks/log/?h=wip/mdraid http://git.gnome.org/browse/gnome-disk-utility/log/?h=wip/mdraid
While working on this, I realized that one thing we need from mdadm(8) is to export the size of the array. We need this so we can show e.g. "12 TB RAID-5 Array" in the UI instead of just "RAID-5 Array". For the case where the array is running, we can just get this from the mapped device (e.g. /dev/md0) but in the case where it's not running we need "mdadm --examine" to print it out. Currently this prints out this # /sbin/mdadm --examine --export /dev/loop0 MD_LEVEL=raid5 MD_DEVICES=3 MD_NAME=thinkpad:raid-t MD_UUID=7dc6c18d:62b514f3:434af505:30591adb MD_UPDATE_TIME=1345895318 MD_DEV_UUID=1b0c5901:5bdf50ac:ba23c484:1a6d9ce5 MD_EVENTS=18 It's not as simple as just looking at MD_LEVEL, MD_DEVICES and the size of each member (they are the same) because this doesn't take into account the fact that part of the device is used for metadata. In fact, this is already printed out when not using --export, see # /sbin/mdadm --examine /dev/loop0 /dev/loop0: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 7dc6c18d:62b514f3:434af505:30591adb Name : thinkpad:raid-t (local to host thinkpad) Creation Time : Sat Aug 25 07:27:21 2012 Raid Level : raid5 Raid Devices : 3 Avail Dev Size : 195184 (95.32 MiB 99.93 MB) Array Size : 194560 (190.03 MiB 199.23 MB) Used Dev Size : 194560 (95.02 MiB 99.61 MB) Data Offset : 128 sectors Super Offset : 8 sectors State : clean Device UUID : 1b0c5901:5bdf50ac:ba23c484:1a6d9ce5 Update Time : Sat Aug 25 07:48:38 2012 Checksum : fff9cc9f - correct Events : 18 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 0 Array State : AAA ('A' == active, '.' == missing) so it's not far-fetched to have e.g. MD_ARRAY_SIZE with --export which would be the same as "Array Size" above (except that it would print number of bytes instead of sectors). Maciej, is this something you can look at adding to mdadm(8)? Thanks!
Sure, I will add this ! The only question I have is whether there is any information, other than array's size, which we would need to extract from 'mdadm --examine --export' or 'mdadm --detail --export'? udisks needs all the information about the state of array (status of operations, clean/rebuilding) and the disks belonging to the array. Of course, we can get this kind of information by simply parsing the /proc/mdstat, but it doesn't sound like a good idea :) So, unless you already have another option for reading such information, I think that --export should provide this data. Yesterday I worked on adding the --export to 'mdadm --detail-platform', and I asked myself how much information do we need to extract. I guess all the data below may become useful at some point, so I decided to add all of the possible values. # /sbin/mdadm --detail-platform -e imsm Platform : Intel(R) Matrix Storage Manager Version : 9.5.0.1037 RAID Levels : raid0 raid1 raid10 raid5 Chunk Sizes : 4k 8k 16k 32k 64k 128k 2TB volumes : supported 2TB disks : not supported Max Disks : 7 Max Volumes : 2 per array, 4 per controller I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2 (SATA) # /sbin/mdadm --detail-platform -e imsm --export IMSM_VERSION=9.5.0.1037 IMSM_RAID_LEVELS=raid0,raid1,raid10,raid5 IMSM_CHUNK_SIZES=4k,8k,16k,32k,64k,128k IMSM_2TB_VOLUMES=yes IMSM_2TB_DISKS=no IMSM_MAX_DISKS=7 IMSM_MAX_VOLUMES_PER_ARRAY=2 IMSM_MAX_VOLUMES_PER_CONTROLLER=4 IMSM_SATA_CONTROLLER=/sys/devices/pci0000:00/0000:00:1f.2 IMSM_SAS_CONTROLLER=none Is it fine? Is there any other information we would like to get with --detail-platform --export? Patches should be ready by Wednesday after some small tests, I'll post info here when I send them upstream!
(In reply to comment #6) > Sure, I will add this ! The only question I have is whether there is any > information, other than array's size, which we would need to extract from > 'mdadm --examine --export' or 'mdadm --detail --export'? I think MD_ARRAY_SIZE is the only thing we need for now. > udisks needs all the information about the state of array (status of > operations, clean/rebuilding) and the disks belonging to the array. Of course, > we can get this kind of information by simply parsing the /proc/mdstat, but it > doesn't sound like a good idea :) So, unless you already have another option > for reading such information, I think that --export should provide this data. Actually all this information is available in /sys/block/md0/md/* and /sys/block/md0/slaves/* - I just added part of it yesterday with this commit: http://cgit.freedesktop.org/udisks/commit/?h=wip/mdraid&id=78b8781a4c738dc42dc03f9363ec7779d28e9884 As we add more stuff to the UI, see http://people.freedesktop.org/~david/gnome-disks-mdraid-20120907-1.png for the current state, we'll probably end up adding more D-Bus properties. > Yesterday I worked on adding the --export to 'mdadm --detail-platform', and I > asked myself how much information do we need to extract. I guess all the data > below may become useful at some point, so I decided to add all of the possible > values. > > # /sbin/mdadm --detail-platform -e imsm > Platform : Intel(R) Matrix Storage Manager > Version : 9.5.0.1037 > RAID Levels : raid0 raid1 raid10 raid5 > Chunk Sizes : 4k 8k 16k 32k 64k 128k > 2TB volumes : supported > 2TB disks : not supported > Max Disks : 7 > Max Volumes : 2 per array, 4 per controller > I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2 (SATA) > > # /sbin/mdadm --detail-platform -e imsm --export > IMSM_VERSION=9.5.0.1037 > IMSM_RAID_LEVELS=raid0,raid1,raid10,raid5 > IMSM_CHUNK_SIZES=4k,8k,16k,32k,64k,128k > IMSM_2TB_VOLUMES=yes > IMSM_2TB_DISKS=no > IMSM_MAX_DISKS=7 > IMSM_MAX_VOLUMES_PER_ARRAY=2 > IMSM_MAX_VOLUMES_PER_CONTROLLER=4 > IMSM_SATA_CONTROLLER=/sys/devices/pci0000:00/0000:00:1f.2 > IMSM_SAS_CONTROLLER=none > > Is it fine? Is there any other information we would like to get with > --detail-platform --export? Actually, I think it would be nicer if one could do # /sbin/mdadm --detail-platform --sysfs-path /sys/devices/pci0000:00/0000:00:1f.2 --export because then we'd just include this udev rule in 80-udisks2.rules SUBSYSTEM=="pci", ACTION=="add|change", ATTR{class}=="0x01*", IMPORT{program}="/sbin/mdadm --detail-platform --sysfs-path %p --export" This udev rule would then run "mdadm --detail-platform" on any PCI device of class 1 e.g. all "Mass storage controller" PCI devices. I'd expect the output on an ISMS-capable controller to be MD_FIRWMARE_TYPE=imsm IMSM_VERSION=9.5.0.1037 IMSM_RAID_LEVELS=raid0,raid1,raid10,raid5 IMSM_CHUNK_SIZES=4k,8k,16k,32k,64k,128k IMSM_2TB_VOLUMES=yes IMSM_2TB_DISKS=no IMSM_MAX_DISKS=7 IMSM_MAX_VOLUMES_PER_ARRAY=2 IMSM_MAX_VOLUMES_PER_CONTROLLER=4 where the MD_FIRWMARE_TYPE property is new and the value "isms" means that information will be in the IMSM_* properties. The IMSM_{SATA,SAS}_CONTROLLER properties are not needed, it's implicit since all properties belong to the PCI device in question (the application (e.g. udisks) can determine for itself whether it's a SAS or SATA device). So, with this setup, the udev database would simply just contain up-to-date information about the capabilities of HBAs/controllers with support for firmware-raid. In udisks, we'd simply just monitor the udev database for HBAs the same way we already monitor it for block devices. Doing it this way also easily solves the problem of a) multiple adapters b) hotplug c) new firmware formats so applications won't have to pass e.g. "-e imsm" (e.g. just set MD_FIRMWARE_TYPE=xyz and XYZ_* properties) Thoughts? > Patches should be ready by Wednesday after some small tests, I'll post info > here when I send them upstream! Sounds great, thanks!
> > udisks needs all the information about the state of array (status of > > operations, clean/rebuilding) and the disks belonging to the array. Of course, > > we can get this kind of information by simply parsing the /proc/mdstat, but it > > doesn't sound like a good idea :) So, unless you already have another option > > for reading such information, I think that --export should provide this data. > > Actually all this information is available in /sys/block/md0/md/* and > /sys/block/md0/slaves/* - I just added part of it yesterday with this commit: > > http://cgit.freedesktop.org/udisks/commit/?h=wip/mdraid&id=78b8781a4c738dc42dc03f9363ec7779d28e9884 > That's great ! We also mentioned the /sys/* during the call but I must have misunderstood something- anyway, this way is much better. > > Actually, I think it would be nicer if one could do > > # /sbin/mdadm --detail-platform --sysfs-path > /sys/devices/pci0000:00/0000:00:1f.2 --export > > because then we'd just include this udev rule in 80-udisks2.rules > > SUBSYSTEM=="pci", ACTION=="add|change", ATTR{class}=="0x01*", > IMPORT{program}="/sbin/mdadm --detail-platform --sysfs-path %p --export" > > This udev rule would then run "mdadm --detail-platform" on any PCI device of > class 1 e.g. all "Mass storage controller" PCI devices. > Sounds good, this also should be ready by Wednesday. > > So, with this setup, the udev database would simply just contain up-to-date > information about the capabilities of HBAs/controllers with support for > firmware-raid. In udisks, we'd simply just monitor the udev database for HBAs > the same way we already monitor it for block devices. > > Doing it this way also easily solves the problem of > > a) multiple adapters > b) hotplug > c) new firmware formats so applications won't have to pass e.g. > "-e imsm" (e.g. just set MD_FIRMWARE_TYPE=xyz and XYZ_* properties) > > Thoughts? > Actually you don't have to add the '-e imsm', mdadm automatically scans for present formats just with 'mdadm --detail-platform'. At the moment, IMSM is the only platform-dependent format, so trying to '--detail-platform -v -e [format]' any other format just gives '[format] is not platform dependent'. Nevertheless, I added MD_FIRMWARE_TYPE to make it all clear !
Btw, can we also change it so it's IMSM_RAID_LEVELS=raid0 raid1 raid10 raid5 instead of IMSM_RAID_LEVELS=raid0,raid1,raid10,raid5 Not super important but it makes it easier to access the device properties using g_udev_device_get_property_as_strv() from libgudev http://www.freedesktop.org/software/systemd/gudev/GUdevDevice.html#g-udev-device-get-property-as-strv Thanks!
If it's the preferred way, then I'll change the separators to spaces - both for raid levels and chunk sizes. What's more, I also changed the '--sysfs-path' option to '--controller-path' to be more precise about the functionality. The patches are currently being tested, they will be sent to Neil on Thursday / Friday. I can also send them to you for testing purposes, or prepare an RPM if you prefer :)
(In reply to comment #10) > If it's the preferred way, then I'll change the separators to spaces - both for > raid levels and chunk sizes. > > What's more, I also changed the '--sysfs-path' option to '--controller-path' to > be more precise about the functionality. The patches are currently being > tested, they will be sent to Neil on Thursday / Friday. I can also send them to > you for testing purposes, or prepare an RPM if you prefer :) Sure, I'd like to see the patch and test it - no need to prepare an RPM, it's not a problem for me to apply it myself. Thanks!
Created attachment 67039 [details] [review] Patch adding --export option for --detail-platform Not fully tested nor sent to Neil yet.
Created attachment 67040 [details] [review] Patch adding --controller-path option for --detail-platform Not fully tested nor sent to Neil yet.
David - I'll send the patches upstream once you say they're all right. We tested mdadm a bit here and there and everything looks fine.
Hey, haven't had a chance to test this yet - my only system with Intel RAID is at home and I've been at the office a lot lately. I'll try to test it later today. Thanks! Btw, did you write the MD_ARRAY_SIZE patch yet?
Heh, I must have forgotten about it :) I'll prepare it tomorrow and post it here as well!
(In reply to comment #12) > Created attachment 67039 [details] [review] [review] > Patch adding --export option for --detail-platform > > Not fully tested nor sent to Neil yet. Looks good to me, this is what I get on my system $ sudo ./mdadm --detail-platform --export MD_FIRMWARE_TYPE=imsm IMSM_VERSION=10.6.0.1091 IMSM_SUPPORTED_RAID_LEVELS=raid0 raid1 raid10 raid5 IMSM_SUPPORTED_CHUNK_SIZES=4k 8k 16k 32k 64k 128k IMSM_2TB_VOLUMES=yes IMSM_2TB_DISKS=yes IMSM_MAX_DISKS=7 IMSM_MAX_VOLUMES_PER_ARRAY=2 IMSM_MAX_VOLUMES_PER_CONTROLLER=4
(In reply to comment #13) > Created attachment 67040 [details] [review] [review] > Patch adding --controller-path option for --detail-platform > > Not fully tested nor sent to Neil yet. This is almost right - it prints the information that was requested sudo ./mdadm --detail-platform --export --controller-path /sys/devices/pci0000\:00/0000\:00\:1f.2 MD_FIRMWARE_TYPE=imsm IMSM_VERSION=10.6.0.1091 IMSM_SUPPORTED_RAID_LEVELS=raid0 raid1 raid10 raid5 IMSM_SUPPORTED_CHUNK_SIZES=4k 8k 16k 32k 64k 128k IMSM_2TB_VOLUMES=yes IMSM_2TB_DISKS=yes IMSM_MAX_DISKS=7 IMSM_MAX_VOLUMES_PER_ARRAY=2 IMSM_MAX_VOLUMES_PER_CONTROLLER=4 [davidz@phobos mdadm]$ echo $? 1 However the exit code should be 0 (not 1) since the operation succeeded. Another problem is when it doesn't find anything [davidz@phobos mdadm]$ sudo ./mdadm --detail-platform --export --controller-path /sys/devices/pci0000\:00/0000\:00\:1f.3 mdadm: no active Intel(R) RAID controller found under /sys/devices/pci0000:00/0000:00:1f.3 [davidz@phobos mdadm]$ echo $? 1 In this case the exit code is correct (e.g. non-zero) but it should not print anything on stdout/stderr.
(In reply to comment #18) > (In reply to comment #13) > > Created attachment 67040 [details] [review] [review] [review] > > Patch adding --controller-path option for --detail-platform > > > > Not fully tested nor sent to Neil yet. > > This is almost right - it prints the information that was requested > > sudo ./mdadm --detail-platform --export --controller-path > /sys/devices/pci0000\:00/0000\:00\:1f.2 > MD_FIRMWARE_TYPE=imsm > IMSM_VERSION=10.6.0.1091 > IMSM_SUPPORTED_RAID_LEVELS=raid0 raid1 raid10 raid5 > IMSM_SUPPORTED_CHUNK_SIZES=4k 8k 16k 32k 64k 128k > IMSM_2TB_VOLUMES=yes > IMSM_2TB_DISKS=yes > IMSM_MAX_DISKS=7 > IMSM_MAX_VOLUMES_PER_ARRAY=2 > IMSM_MAX_VOLUMES_PER_CONTROLLER=4 > > [davidz@phobos mdadm]$ echo $? > 1 > > However the exit code should be 0 (not 1) since the operation succeeded. > Thanks, I'll fix it tomorrow. > Another problem is when it doesn't find anything > > [davidz@phobos mdadm]$ sudo ./mdadm --detail-platform --export > --controller-path /sys/devices/pci0000\:00/0000\:00\:1f.3 > mdadm: no active Intel(R) RAID controller found under > /sys/devices/pci0000:00/0000:00:1f.3 > > [davidz@phobos mdadm]$ echo $? > 1 > > In this case the exit code is correct (e.g. non-zero) but it should not print > anything on stdout/stderr. Actually that was intended, but - since you probably don't want any output there - I'll make it appear in '--verbose' only.
(In reply to comment #19) > Actually that was intended, The concern is that it may show up in the udev log files. > but - since you probably don't want any output > there - I'll make it appear in '--verbose' only. Sounds good, thanks!
I think David is right. In '--export' mode mdadm should only print 'KEY=VALUE' messages or nothing and return proper exit code. I don't think option '--verbose' is not needed here.
Errr... is needed here ;)
Created attachment 67162 [details] [review] [1/6] add --export for --detail-platform Not tested yet
Created attachment 67163 [details] [review] [2/6] add --controller-path option for --detail-platform Not tested yet
Created attachment 67164 [details] [review] [3/6] fix return code for --detail-platform Not tested yet
Created attachment 67165 [details] [4/6] Synchronize size calculation in human_size and human_size_readable Not tested yet
Created attachment 67166 [details] [review] [5/6] Display size with human_size_brief with a chosen prefix Not tested yet
Created attachment 67167 [details] [review] [4/6] Synchronize size calculation in human_size and human_size_readable Not tested yet
Created attachment 67168 [details] [review] [6/6] Add MD_ARRAY_SIZE for --examine --export Not tested yet
Created attachment 67173 [details] [review] [1/6] add --export for --detail-platform Not tested yet
Created attachment 67174 [details] [review] [2/6] add --controller-path option for --detail-platform Not tested yet
Created attachment 67245 [details] [review] [4/6] Synchronize size calculation in human_size and human_size_readable Not tested yet
I've now merged wip/mdraid into master - development continues on master. This feature will land in udisks 2.1 and gnome-disk-utility 3.8 - both scheduled for March 2013, see https://live.gnome.org/ThreePointSeven for details
It's great to hear it ! I also managed to get 5 of 6 patches to upstream, Neil has also accepted the 6th one so it should be there in the next few days. The only difference between what is in the upstream and what was prepared here is the lack of --controller-path option. If we want 'mdadm --detail-platform' to scan only a specific controller, we just add the fullpath afterwards. So instead of $ mdadm --detail-platform --controller-path=/sys/devices/pci0000\:00/0000\:00\:1f.2 we use $ mdadm --detail-platform /sys/devices/pci0000\:00/0000\:00\:1f.2
David, I'd just like to confirm that all the patches are already in mdadm's upstream.
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.