Bug 46857

Summary:	systemd does not properly clean up namespaces after service restart or stop
Product:	systemd	Reporter:	David Strauss <david>
Component:	general	Assignee:	Lennart Poettering <lennart>
Status:	RESOLVED FIXED	QA Contact:
Severity:	major
Priority:	medium	CC:	colin, david, joeym, nathaniel.chen
Version:	unspecified
Hardware:	x86-64 (AMD64)
OS:	Linux (All)
Whiteboard:
i915 platform:		i915 features:

Description David Strauss 2012-03-01 17:47:44 UTC

We have some systems with a lot (2000+) services that we automatically spin down when they're not being used and then back up when they're needed. Each one uses PrivateTmp and other options that require a namespace.

Unfortunately, these namespaces don't seem to get cleaned up by systemd when the service stops. Without cleanup, it doesn't take long to run into the 32K subdirectory limit in /tmp for ext3 or the 64K one for ext4.

I'm currently seeing this on systemd 37 on Fedora 15, but I'm happy to try a newer combination if there's been progress on this front. I didn't notice anything in the commit log.

Here's what I see:

[root@david-onebox tmp]# ls systemd-namespace-*
systemd-namespace-QPIEhS:
inaccessible  root
[root@david-onebox tmp]# systemctl restart php_fpm_fcf89238355f4fd389ab99f4b07f7829.service
[root@david-onebox tmp]# ls systemd-namespace-*
systemd-namespace-q1TrzY:
inaccessible  root

systemd-namespace-QPIEhS:
inaccessible  root
[root@david-onebox tmp]# systemctl restart php_fpm_fcf89238355f4fd389ab99f4b07f7829.service
[root@david-onebox tmp]# ls systemd-namespace-*
systemd-namespace-q1TrzY:
inaccessible  root

systemd-namespace-QPIEhS:
inaccessible  root

systemd-namespace-SL8O27:
inaccessible  root

...and so on.

Comment 1 Lennart Poettering 2012-07-20 14:02:47 UTC

Hmm, normally the tmpfiles aging logic should clean this up. We had a bug where it never cleaned up directories, but only normal files. This should be fixed since quite a while however. Does this problem still exist with new systemd versions?

Comment 2 Lennart Poettering 2012-09-13 18:00:32 UTC

David, does this issue still exist?

Comment 3 David Strauss 2012-09-13 18:39:18 UTC

I just pinged one of our ops developers to update the /tmp namespace directory issue. We can only speak for whether it's an issue on F16 because we haven't moved the affected class of servers to F17 yet.

Comment 4 joe miller 2012-09-13 19:22:24 UTC

Hi Lennart,

I checked some of our servers and we are still seeing this issue.

If there's anything in particular you want me to check or provide, please let me know. Here are some things I see:

1) most of the directories contain a single dir 'inaccessible'. Some also contain a 'root' dir:

systemd-namespace-BfILsd:
inaccessible

systemd-namespace-BG8w4H:
inaccessible

systemd-namespace-bGb7nT:
inaccessible    root

As a cleanup, I've been deleting old dirs that do not contain the root subdir.

2) Ownership differs across the dirs.  Most are "root:<some_valid_group". Some are "root:root", others are "root:<gid of group that no longer exists>".  I've been removing the dirs without a valid group as a cleanup task.

drwx------  3 root   1e1d4507872b4b7c8de8fab5fda8e021     4096 Aug 18 23:03 systemd-namespace-ZsjgRJ
drwx------  3 root   root                                 4096 Jul 28 23:03 systemd-namespace-ZsMUKt
drwx------  3 root   e4c8360b868a4d5d86c6bccd20ae58b2     4096 Aug 18 23:03 systemd-namespace-zSqHH9
drwx------  3 root                              10182     4096 Jun 24 23:03 systemd-namespace-ZtKbPD

Comment 5 joe miller 2012-10-07 16:32:20 UTC

some additional info:

- found another server today. this one is f16 running 'systemd-37-19.fc16.x86_64'

Best method I've found to clean this up is to look for any /tmp/systemd-namespace-* dirs that do not contain a 'root' subdir:

find /tmp -name systemd-namespace-\* -type d '!' -exec test -e "{}/root" ";" -print0 | xargs -0 -- rm -rf

Comment 6 nick.stielau 2013-03-21 16:07:04 UTC

Hey,

Seeing this behavior again on F17, mostly with 'systemd-inaccessible' directories.

[nick@endpointff828748 ~]$ cat /etc/issue
Fedora release 17 (Beefy Miracle)
Kernel \r on an \m (\l)
[nick@endpointff828748 ~]$ systemctl --version
systemd 44
fedora
+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP

[nick@endpointff828748 ~]$ ls -l /tmp | grep 'systemd-inaccessible'  | wc -l
25440

[nick@endpointff828748 ~]$ sudo find /tmp -name systemd-inaccessible-\* -type d '!' -exec test -e "{}/root" ";" -print0 | sudo xargs -0 -- rm -rf
[nick@endpointff828748 ~]$ ls -l /tmp | grep systemd | wc -l
1

Comment 7 Zbigniew Jedrzejewski-Szmek 2013-04-05 14:22:11 UTC

/tmp/systemd-inaccessible-* is gone in favour of /run/systemd/inaccessible. It would be great if you could retest with systemd-199 or later.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.