Created attachment 109734 [details] #systemd IRC log discussing issue On CoreOS I'm running Java apps in Docker and logging to stdout. Large logs (eg. stacktraces) get split over multiple lines which is far from ideal. This only seems to affect logs passed in via fd as discussed in IRC (attached) and not those sent to the journald socket. Quick testcase Docker app at: https://github.com/stuart-warren/testbigdockerlogs Or: ``perl -e "print 'a'x4096, 'b'" | systemd-cat``
Just to clarify, the long Java log exceptions are serialised into JSON to avoid the new lines and provide extra context for Logstash.
We cannot accept unbounded log lines. We currently add a hard break after LINE_MAX (as defined by glibc), which is at 2K. I am not sure what we really can do about this. We must enforce something. We can of course play games and bump it to 4K or 16K, but I am not convinced this would be that beneficial, really.
Could you just make it the same limit as the socket entrypoint so that it's consistent?
hmm, "socket entrypoint"? What do you mean by that?
I believe it is located at /run/systemd/journal/socket: https://github.com/intgr/systemd/blob/master/units/systemd-journald.socket#L21 This can accept much larger messages as shown in this example: https://github.com/stuart-warren/bigjournallogs It was discussed in the IRC log attached above.
Not following. The socket has a simple buffer inside, you can store a number of lines in it. Really not following what you are asking me to do?
I got around this limitation/feature by creating a wrapper around the docker container that sucks up stdout/stderr and sends it directly to the journald socket. While this works like a champ, it is not really an optimal solution, why do we need to do a break at LINE_MAX? This is not consistent with the maximum log line size and is not the behaviour that you would 'expect'. Can we not just remove the LINE_MAX break code entirely?
I observed this issue as well. It is really puzzling why perl -e "print 'MESSAGE=', 'a'x4096, 'b'" | logger --journald generates single log message, perl -e "print 'a'x4096, 'b'" | systemd-cat generates 3 and perl -e "print 'a'x4096, 'b'" | logger generates 5 log lines. Whatever the technical reasons to restrict the log message length, at least the limits should be consistent.
I checked that on Fedora 21 perl -e "print 'MESSAGE=', 'a'x100000, 'b'" | logger --journald happily sends to the journal as a single line with 1e5 characters without splitting anything. So the current restriction of 2K from stdout is pretty arbitrary and restrictive indeed.
We're struggling with line truncation at 2048 chars as well. We log JSON to stdout in Docker containers on CoreOS, much like SW described. Newline placement is important for Logstash and other tools to handle the logs correctly. What's the status of this issue?
We're having this same problem. It seems the reasonable solution is to have a sane default, while providing the ability to override via configuration. Any opinions on this?
We had been using this patch/fork of systemd-docker to hack around the problem on CoreOS https://github.com/sysboy/systemd-docker/commit/87516f34b4fe6bc6ab0cdb07f2a1c17ceb9b882a Probably not up to date, not something I'm personally using...
We've just hit this too. Looking at the workarounds, although they do allow longer lines, they are not satisfactory for other reasons. Does anyone have a view on what the correct fix for allowing longer lines is? A 16K limit would fix 100% of the cases I've seen but as Lennart correctly pointed out, this is not a proper fix. I have to say it does seem inconsistent that the 2048 limit doesn't apply when using "logger --journald"
Any suggestion on what the proper fix should be? I'm happy to have a go at fixing this if there is agreement on the approach.
For all those using CoreOS/Docker, we just added `--log-driver=journald` to our `docker run` command and we're not affected any more. In addition, we also threw in `StandardOutput=null` to our unit files so logs wouldn't be duplicated.
This issue is causing me emotional distress. 2k is not a reasonable limit for log messages. If there must be a limit let's make it 4x or 8x larger. Is this a systemd project decision or a distribution decision?
we use LINE_MAX for lack of a better value, which glibc defines to 2K. I figure we could enforce a limit of 64K or so in journald instead, but we have to enforce something. Please consider filing a PR that introduces a new limit define and sets it higher.
Do you know whether somebody has planed to fix this issue?
There's a PR, but it needs reworking.
Thanks for this. Just for sake of completeness, this PR can be found here : https://github.com/systemd/systemd/pull/4875
I have a very small proxy that pipes stdin to the journald socket. https://git.reform.hmcts.net/roby/journald-proxy We're planning on using this in our service unit files to address the problem, is that an awful idea?
(In reply to Rob from comment #21) > I have a very small proxy that pipes stdin to the journald socket. > https://git.reform.hmcts.net/roby/journald-proxy > > We're planning on using this in our service unit files to address the > problem, is that an awful idea? Whoops, that should be https://github.com/robyoung/journald-proxy
Fixed in https://github.com/systemd/systemd/commit/ec20fe5ffb8a00469bab209fff6c069bb93c6db2.
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.