Bug 58549

Summary: journald doesn't rotate uncleanly closed files
Product: systemd Reporter: Dave Reisner <d>
Component: generalAssignee: systemd-bugs
Status: RESOLVED INVALID QA Contact: systemd-bugs
Severity: normal    
Priority: medium    
Version: unspecified   
Hardware: x86-64 (AMD64)   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:

Description Dave Reisner 2012-12-20 01:34:39 UTC
Copypasta of my original reply on the ML:
http://lists.freedesktop.org/archives/systemd-devel/2012-November/007585.html

---------8<-----------------------------------------------------------
I'm not sure when it happened, but rotation used to be "off" by a
difference of whatever the MaxFileSize option was, e.g. if you had a
200M MaxUse and a 50M MaxFileSize, then you'd see usage up to 250M
before old logs were expired. I reported this, and there was a TODO item
added.

At least with 196 (maybe earlier), this no longer holds true any more.
The journal appears to just ignore settings in journald.conf and grow
without bound:

$ grep -E 'SystemMax(Use|FileSize)' /etc/systemd/journald.conf
SystemMaxUse=200M
SystemMaxFileSize=50M

$ journalctl --disk-usage
Journals take up 436.0M on disk.

Forcing rotation opens a new file, but old logs are not expired, leaving
disk usage way above the set maximum. There also seems to be a recent
issue with more uncleanly closed journal files showing up. Unless I
umount my root at shutdown (with initramfs magic), I can very reliably
expect that the journal will mark the last used file as unclean on the
next bootup. My suspicion is that the journal doesn't take into account
these uncleanly closed files when calculating disk usage for rotation,
but I've not had time to confirm that.
-----------------------------------------------------------8<---------

And to add, I've gone back and confirmed that rotation still happens,
and that actual maximum "managed" usage is:

 SystemMaxUse + (NumFilesOpen * SystemMaxFileUseSize)

But there indeed seems to be no attention paid to the tilde suffixed
files, which inflate this amount above the aforementioned maximum.
Comment 1 Adam Williamson 2013-06-14 07:50:11 UTC
Looks like I can confirm this up to Fedora 19 at least, after I hit the massive journald/rsyslog fail that showed up today, I noticed I had journal~ files going back to March in my /var/log/journal, but nothing that old without the tilde.
Comment 2 Zbigniew Jedrzejewski-Szmek 2013-06-25 01:31:24 UTC
I went over the vacuuming code, and and I don't see any basis for corrupted journal files to be treated differently. There was a buglet in space reporting
[1], and I made a change to delete "empty" (entry-less) journal files, which makes the usage reported by journalctl and by systemd-journald agree [2],
but the code paths for .journal and journal~ files are basically the same.

Can you make sure that the journal directory is actually over SystemMaxUse or the free space is below SystemKeepFree, and if yes, show /etc/systemd/journald.conf, and the output from 'journalctl -b --full -u systemd-journald', and 'du -a /var/log/journal/'? The version of systemd-journald from git would be best [3].

Thanks,
Zbyszek

[1] http://cgit.freedesktop.org/systemd/systemd/commit/?id=670b110
[2] http://cgit.freedesktop.org/systemd/systemd/commit/?id=9d64774
[3] I use: make -C build -j5 && sudo rm /usr/lib/systemd/systemd-journald && sudo cp build/.libs/systemd-journald /usr/lib/systemd/ -v  && sudo systemctl restart systemd-journald
Comment 3 Dave Reisner 2013-06-25 01:46:59 UTC
I can't reproduce this anymore in 204 -- my original test case passes, so I'm going to close this as invalid.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.