We have some systems with a lot (2000+) services that we automatically spin down when they're not being used and then back up when they're needed. Each one uses PrivateTmp and other options that require a namespace. Unfortunately, these namespaces don't seem to get cleaned up by systemd when the service stops. Without cleanup, it doesn't take long to run into the 32K subdirectory limit in /tmp for ext3 or the 64K one for ext4. I'm currently seeing this on systemd 37 on Fedora 15, but I'm happy to try a newer combination if there's been progress on this front. I didn't notice anything in the commit log. Here's what I see: [root@david-onebox tmp]# ls systemd-namespace-* systemd-namespace-QPIEhS: inaccessible root [root@david-onebox tmp]# systemctl restart php_fpm_fcf89238355f4fd389ab99f4b07f7829.service [root@david-onebox tmp]# ls systemd-namespace-* systemd-namespace-q1TrzY: inaccessible root systemd-namespace-QPIEhS: inaccessible root [root@david-onebox tmp]# systemctl restart php_fpm_fcf89238355f4fd389ab99f4b07f7829.service [root@david-onebox tmp]# ls systemd-namespace-* systemd-namespace-q1TrzY: inaccessible root systemd-namespace-QPIEhS: inaccessible root systemd-namespace-SL8O27: inaccessible root ...and so on.
Hmm, normally the tmpfiles aging logic should clean this up. We had a bug where it never cleaned up directories, but only normal files. This should be fixed since quite a while however. Does this problem still exist with new systemd versions?
David, does this issue still exist?
I just pinged one of our ops developers to update the /tmp namespace directory issue. We can only speak for whether it's an issue on F16 because we haven't moved the affected class of servers to F17 yet.
Hi Lennart, I checked some of our servers and we are still seeing this issue. If there's anything in particular you want me to check or provide, please let me know. Here are some things I see: 1) most of the directories contain a single dir 'inaccessible'. Some also contain a 'root' dir: systemd-namespace-BfILsd: inaccessible systemd-namespace-BG8w4H: inaccessible systemd-namespace-bGb7nT: inaccessible root As a cleanup, I've been deleting old dirs that do not contain the root subdir. 2) Ownership differs across the dirs. Most are "root:<some_valid_group". Some are "root:root", others are "root:<gid of group that no longer exists>". I've been removing the dirs without a valid group as a cleanup task. drwx------ 3 root 1e1d4507872b4b7c8de8fab5fda8e021 4096 Aug 18 23:03 systemd-namespace-ZsjgRJ drwx------ 3 root root 4096 Jul 28 23:03 systemd-namespace-ZsMUKt drwx------ 3 root e4c8360b868a4d5d86c6bccd20ae58b2 4096 Aug 18 23:03 systemd-namespace-zSqHH9 drwx------ 3 root 10182 4096 Jun 24 23:03 systemd-namespace-ZtKbPD
some additional info: - found another server today. this one is f16 running 'systemd-37-19.fc16.x86_64' Best method I've found to clean this up is to look for any /tmp/systemd-namespace-* dirs that do not contain a 'root' subdir: find /tmp -name systemd-namespace-\* -type d '!' -exec test -e "{}/root" ";" -print0 | xargs -0 -- rm -rf
Hey, Seeing this behavior again on F17, mostly with 'systemd-inaccessible' directories. [nick@endpointff828748 ~]$ cat /etc/issue Fedora release 17 (Beefy Miracle) Kernel \r on an \m (\l) [nick@endpointff828748 ~]$ systemctl --version systemd 44 fedora +PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP [nick@endpointff828748 ~]$ ls -l /tmp | grep 'systemd-inaccessible' | wc -l 25440 [nick@endpointff828748 ~]$ sudo find /tmp -name systemd-inaccessible-\* -type d '!' -exec test -e "{}/root" ";" -print0 | sudo xargs -0 -- rm -rf [nick@endpointff828748 ~]$ ls -l /tmp | grep systemd | wc -l 1
/tmp/systemd-inaccessible-* is gone in favour of /run/systemd/inaccessible. It would be great if you could retest with systemd-199 or later.
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.