I am running gentoo ~amd64 with 2GB /var mounted off lvm volume. Journalctl slowly eats up all free space on /var with logs going up to 1.6GB. I have to delete them, and after some time free space runs out. According to manual, this shouldn't happen even without any special configuration. I've tried setting up /etc/systemd/journald.conf like so : [Journal] Storage=auto Compress=yes SystemMaxUse=50M RuntimeMaxUse=20M And the situation repeated, with journal's log dir growing to 1.6GB over the course of few weeks. Systemd version used is always latest one from http://packages.gentoo.org/package/sys-apps/systemd
Please specify the exact systemd versions you are seeing this with, and the output from 'journalctl -u systemd-journald.service'.
some of the latter parts of the log : -- Reboot -- sie 15 19:46:22 box systemd-journal[1597]: Runtime journal is using 120.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 15 19:46:22 box systemd-journal[1597]: Runtime journal is using 124.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 15 19:46:22 box systemd-journal[1597]: Journal started sie 15 19:46:28 box systemd-journal[1597]: Permanent journal is using 1.3G (max 50.0M, leaving 292.7M of free 0B, current limit 0B). -- Reboot -- sie 16 06:25:22 box systemd-journal[1599]: Runtime journal is using 120.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 16 06:25:22 box systemd-journal[1599]: Runtime journal is using 124.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 16 06:25:22 box systemd-journal[1599]: Journal started sie 16 06:25:27 box systemd-journal[1599]: Permanent journal is using 111.7M (max 50.0M, leaving 292.7M of free 1.1G, current limit 50.0M). sie 16 06:59:20 box systemd-journal[1599]: Journal stopped -- Reboot -- sie 16 18:39:45 box systemd-journal[1595]: Runtime journal is using 120.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 16 18:39:45 box systemd-journal[1595]: Runtime journal is using 124.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 16 18:39:45 box systemd-journal[1595]: Journal started sie 16 18:39:50 box systemd-journal[1595]: Permanent journal is using 112.5M (max 50.0M, leaving 292.7M of free 1.1G, current limit 50.0M). ~ # journalctl --disk-usage Journals take up 130.2M on disk. I am currently running systemd 206 with patches from here : http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/sys-apps/systemd/files/ I will try completely erasing log directory and see how it grows over time.
Hm, can you do 'ls -l /var/log/journal/*/'? Please don't delete the directory yet, rather rename it.
-rw-r----- 1 root systemd-journal 4001792 08-16 18:39 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000000001-0004e408faa04cca.journal -rw-r----- 1 root systemd-journal 3964928 08-16 18:54 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000001ac3-0004e4133d658d5b.journal -rw-r----- 1 root systemd-journal 3977216 08-16 19:09 system@64f47d6b6bb14549aabec620cc5b1e18-00000000000037e2-0004e413725e649b.journal -rw-r----- 1 root systemd-journal 3977216 08-16 19:25 system@64f47d6b6bb14549aabec620cc5b1e18-000000000000557a-0004e413a8cfb9a7.journal -rw-r----- 1 root systemd-journal 3981312 08-16 19:40 system@64f47d6b6bb14549aabec620cc5b1e18-000000000000731a-0004e413df411304.journal -rw-r----- 1 root systemd-journal 3981312 08-16 19:55 system@64f47d6b6bb14549aabec620cc5b1e18-00000000000090cc-0004e41415d1f06c.journal -rw-r----- 1 root systemd-journal 3977216 08-16 20:10 system@64f47d6b6bb14549aabec620cc5b1e18-000000000000ae7c-0004e4144c62db53.journal -rw-r----- 1 root systemd-journal 3981312 08-16 20:26 system@64f47d6b6bb14549aabec620cc5b1e18-000000000000cc1e-0004e41482d4813c.journal -rw-r----- 1 root systemd-journal 3985408 08-16 20:41 system@64f47d6b6bb14549aabec620cc5b1e18-000000000000e9ce-0004e414b9655631.journal -rw-r----- 1 root systemd-journal 3981312 08-16 20:56 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000010785-0004e414eff5cf96.journal -rw-r----- 1 root systemd-journal 3981312 08-16 21:11 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000012535-0004e4152688331f.journal -rw-r----- 1 root systemd-journal 4030464 08-16 21:16 system@64f47d6b6bb14549aabec620cc5b1e18-00000000000142dd-0004e4155d1745ab.journal -rw-r----- 1 root systemd-journal 4046848 08-16 21:19 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000016019-0004e4156d0dac8a.journal -rw-r----- 1 root systemd-journal 4042752 08-16 21:23 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000017df7-0004e4157a058ae6.journal -rw-r----- 1 root systemd-journal 4046848 08-16 21:27 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000019bc7-0004e41586ee4c6a.journal -rw-r----- 1 root systemd-journal 4042752 08-16 21:30 system@64f47d6b6bb14549aabec620cc5b1e18-000000000001b9a5-0004e41593e693b0.journal -rw-r----- 1 root systemd-journal 4046848 08-16 21:34 system@64f47d6b6bb14549aabec620cc5b1e18-000000000001d776-0004e415a0d70cf5.journal -rw-r----- 1 root systemd-journal 4046848 08-16 21:38 system@64f47d6b6bb14549aabec620cc5b1e18-000000000001f559-0004e415ade15254.journal -rw-r----- 1 root systemd-journal 4116480 08-17 19:12 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000021337-0004e415bacfb605.journal -rw-r----- 1 root systemd-journal 3928064 08-17 21:22 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000022f0b-0004e427d19a3269.journal -rw-r----- 1 root systemd-journal 3993600 08-17 21:27 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000024bab-0004e429a1c041a2.journal -rw-r----- 1 root systemd-journal 3989504 08-17 21:32 system@64f47d6b6bb14549aabec620cc5b1e18-000000000002699a-0004e429b2d6f0a5.journal -rw-r----- 1 root systemd-journal 3985408 08-17 21:36 system@64f47d6b6bb14549aabec620cc5b1e18-000000000002877c-0004e429c3ce80b4.journal -rw-r----- 1 root systemd-journal 3981312 08-17 21:39 system@64f47d6b6bb14549aabec620cc5b1e18-000000000002a553-0004e429d335d5af.journal -rw-r----- 1 root systemd-journal 3989504 08-17 21:44 system@64f47d6b6bb14549aabec620cc5b1e18-000000000002c32f-0004e429dcf412ae.journal -rw-r----- 1 root systemd-journal 3993600 08-17 21:48 system@64f47d6b6bb14549aabec620cc5b1e18-000000000002e110-0004e429edebfbc1.journal -rw-r----- 1 root systemd-journal 3989504 08-17 21:53 system@64f47d6b6bb14549aabec620cc5b1e18-000000000002ff06-0004e429ff02cd1e.journal -rw-r----- 1 root systemd-journal 3993600 08-17 21:58 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000031ce9-0004e42a0ffa966e.journal -rw-r----- 1 root systemd-journal 3989504 08-17 22:03 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000033ad7-0004e42a20f28e55.journal -rw-r----- 1 root systemd-journal 3993600 08-17 22:07 system@64f47d6b6bb14549aabec620cc5b1e18-00000000000358b4-0004e42a31ea56fb.journal -rw-r----- 1 root systemd-journal 3989504 08-17 22:12 system@64f47d6b6bb14549aabec620cc5b1e18-00000000000376a3-0004e42a42e9a279.journal -rw-r----- 1 root systemd-journal 3993600 08-17 22:17 system@64f47d6b6bb14549aabec620cc5b1e18-0000000000039484-0004e42a53e14cb6.journal -rw-r----- 1 root systemd-journal 3989504 08-17 22:22 system@64f47d6b6bb14549aabec620cc5b1e18-000000000003b276-0004e42a64f06406.journal -rw-r----- 1 root systemd-journal 3993600 08-17 22:26 system@64f47d6b6bb14549aabec620cc5b1e18-000000000003d057-0004e42a75e82b4d.journal -rw-r----- 1 root systemd-journal 3989504 08-17 22:29 system@64f47d6b6bb14549aabec620cc5b1e18-000000000003ee45-0004e42a86dfe26a.journal -rw-r----- 1 root systemd-journal 1392640 08-18 10:04 system.journal ----------------------------------------------------------------------- [root@box ~]# journalctl --disk-usage Journals take up 134.9M on disk. this is after cleaning the journals few days ago to <30mb Ok, i'll try renaming it then.
The logs are up to ~430mb as of now, log entries since last time : ================================================================== sie 16 21:40:01 box systemd-journal[1595]: Journal stopped -- Reboot -- sie 17 08:29:48 box systemd-journal[1597]: Runtime journal is using 120.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 17 08:29:48 box systemd-journal[1597]: Runtime journal is using 124.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 17 08:29:48 box systemd-journal[1597]: Journal started sie 17 08:29:53 box systemd-journal[1597]: Permanent journal is using 71.1M (max 50.0M, leaving 292.7M of free 1.2G, current limit 50.0M). sie 17 09:07:03 box systemd-journal[1597]: Journal stopped -- Reboot -- sie 17 11:01:25 box systemd-journal[1599]: Runtime journal is using 120.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 17 11:01:25 box systemd-journal[1599]: Runtime journal is using 124.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 17 11:01:25 box systemd-journal[1599]: Journal started sie 17 11:01:39 box systemd-journal[1599]: Permanent journal is using 71.8M (max 50.0M, leaving 292.7M of free 1.2G, current limit 50.0M). sie 17 18:29:33 box systemd-journal[1599]: Journal stopped -- Reboot -- sie 17 19:12:51 box systemd-journal[1596]: Runtime journal is using 120.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 17 19:12:51 box systemd-journal[1596]: Runtime journal is using 124.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 17 19:12:51 box systemd-journal[1596]: Journal started sie 17 19:12:56 box systemd-journal[1596]: Permanent journal is using 72.4M (max 50.0M, leaving 292.7M of free 1.2G, current limit 50.0M). sie 17 22:29:41 box systemd-journal[1596]: Journal stopped -- Reboot -- sie 18 08:44:36 box systemd-journal[1596]: Runtime journal is using 120.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 18 08:44:36 box systemd-journal[1596]: Runtime journal is using 124.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 18 08:44:36 box systemd-journal[1596]: Journal started sie 18 08:44:41 box systemd-journal[1596]: Permanent journal is using 134.3M (max 50.0M, leaving 292.7M of free 1.1G, current limit 50.0M). sie 18 15:33:16 box systemd-journal[2626]: Permanent journal is using 135.4M (max 50.0M, leaving 292.7M of free 1.1G, current limit 50.0M). sie 18 15:33:16 box systemd-journal[2626]: Missed 375 kernel messages sie 18 15:33:16 box systemd-journal[2626]: Journal started sie 18 15:33:16 box systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE sie 18 20:11:58 box systemd-journal[2626]: Journal stopped -- Reboot -- sie 18 21:41:51 box systemd-journal[1594]: Runtime journal is using 120.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 18 21:41:51 box systemd-journal[1594]: Runtime journal is using 124.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 18 21:41:51 box systemd-journal[1594]: Journal started sie 18 21:41:57 box systemd-journal[1594]: Permanent journal is using 369.4M (max 50.0M, leaving 292.7M of free 1.0G, current limit 50.0M). sie 18 21:42:51 box systemd-journal[3043]: Permanent journal is using 370.6M (max 50.0M, leaving 292.7M of free 1.0G, current limit 50.0M). sie 18 21:42:51 box systemd-journal[3043]: Missed 4934 kernel messages sie 18 21:42:51 box systemd-journal[3043]: Journal started sie 18 21:42:51 box systemd[1]: systemd-journald.service: main process exited, code=exited, status=1/FAILURE sie 18 22:12:09 box systemd-journal[3043]: Journal stopped -- Reboot -- sie 19 06:31:51 box systemd-journal[1597]: Runtime journal is using 120.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 19 06:31:51 box systemd-journal[1597]: Runtime journal is using 124.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 19 06:31:51 box systemd-journal[1597]: Journal started sie 19 06:31:56 box systemd-journal[1597]: Permanent journal is using 419.1M (max 50.0M, leaving 292.7M of free 991.4M, current limit 50.0M). sie 19 07:00:28 box systemd-journal[1597]: Journal stopped -- Reboot -- sie 19 17:47:46 box systemd-journal[1599]: Runtime journal is using 120.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 19 17:47:46 box systemd-journal[1599]: Runtime journal is using 124.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 19 17:47:46 box systemd-journal[1599]: Journal started sie 19 17:47:52 box systemd-journal[1599]: Permanent journal is using 419.9M (max 50.0M, leaving 292.7M of free 990.6M, current limit 50.0M). sie 19 18:10:59 box systemd-journal[1599]: Journal stopped -- Reboot -- sie 19 20:00:02 box systemd-journal[1596]: Runtime journal is using 120.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 19 20:00:02 box systemd-journal[1596]: Runtime journal is using 124.0K (max 20.0M, leaving 207.1M of free 1.3G, current limit 20.0M). sie 19 20:00:02 box systemd-journal[1596]: Journal started sie 19 20:00:07 box systemd-journal[1596]: Permanent journal is using 420.5M (max 50.0M, leaving 292.7M of free 989.7M, current limit 50.0M). ==========================================================================
Logs got back to 1.4 GB atm. I'll move them aside and will provide any info if necessary.
(In reply to comment #6) > Logs got back to 1.4 GB atm. I'll move them aside and will provide any info > if necessary. It's pretty hard to see what's the problem, because the logging capabilities are a bit lacking :) I've added some more logging: http://cgit.freedesktop.org/systemd/systemd/commit/?id=289f910e and there are bugfixes regarding the deletion of empty journal files: http://cgit.freedesktop.org/systemd/systemd/commit/?id=629bfc5a http://cgit.freedesktop.org/systemd/systemd/commit/?id=48979861 629bfc5a might fix your issue... It would be great if you could try with those patches, or simply with latest git.
systemd-207 works properly, and my logs cleaned up themselves gradually. If the aforementioned patches are included in that release, it might be considered resolved.
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.