Bug 86465 - journal logs from service stdout/stderr split at 2048 chars
Summary: journal logs from service stdout/stderr split at 2048 chars
Alias: None
Product: systemd
Classification: Unclassified
Component: general (show other bugs)
Version: unspecified
Hardware: x86-64 (AMD64) Linux (All)
: medium normal
Assignee: systemd-bugs
QA Contact: systemd-bugs
Depends on:
Reported: 2014-11-19 14:57 UTC by SW
Modified: 2017-09-22 08:24 UTC (History)
5 users (show)

See Also:
i915 platform:
i915 features:

#systemd IRC log discussing issue (2.24 KB, text/plain)
2014-11-19 14:57 UTC, SW

Description SW 2014-11-19 14:57:38 UTC
Created attachment 109734 [details]
#systemd IRC log discussing issue

On CoreOS I'm running Java apps in Docker and logging to stdout.
Large logs (eg. stacktraces) get split over multiple lines which is far from ideal.

This only seems to affect logs passed in via fd as discussed in IRC (attached) and not those sent to the journald socket.

Quick testcase Docker app at: 


``perl -e "print 'a'x4096, 'b'" | systemd-cat``
Comment 1 SW 2014-11-20 17:15:39 UTC
Just to clarify, the long Java log exceptions are serialised into JSON to avoid the new lines and provide extra context for Logstash.
Comment 2 Lennart Poettering 2014-12-08 23:26:21 UTC
We cannot accept unbounded log lines. We currently add a hard break after LINE_MAX (as defined by glibc), which is at 2K. I am not sure what we really can do about this. We must enforce something. We can of course play games and bump it to 4K or 16K, but I am not convinced this would be that beneficial, really.
Comment 3 SW 2014-12-09 10:23:08 UTC
Could you just make it the same limit as the socket entrypoint so that it's consistent?
Comment 4 Lennart Poettering 2014-12-09 13:01:55 UTC
hmm, "socket entrypoint"? What do you mean by that?
Comment 5 SW 2014-12-09 13:11:29 UTC
I believe it is located at /run/systemd/journal/socket:

This can accept much larger messages as shown in this example:

It was discussed in the IRC log attached above.
Comment 6 Lennart Poettering 2014-12-09 13:13:31 UTC
Not following. The socket has a simple buffer inside, you can store a number of lines in it. Really not following what you are asking me to do?
Comment 7 Steve 2014-12-15 09:51:40 UTC
I got around this limitation/feature by creating a wrapper around the docker container that sucks up stdout/stderr and sends it directly to the journald socket.

While this works like a champ, it is not really an optimal solution, why do we need to do a break at LINE_MAX? This is not consistent with the maximum log line size and is not the behaviour that you would 'expect'.

Can we not just remove the LINE_MAX break code entirely?
Comment 8 Igor Bukanov 2015-02-03 08:24:53 UTC
I observed this issue as well. It is really puzzling why

    perl -e "print 'MESSAGE=', 'a'x4096, 'b'" | logger --journald

generates single log message, 

   perl -e "print 'a'x4096, 'b'" | systemd-cat

generates 3 and

    perl -e "print 'a'x4096, 'b'" | logger

generates 5 log lines.

Whatever the technical reasons to restrict the log message length, at least the limits should be consistent.
Comment 9 Igor Bukanov 2015-02-03 08:31:08 UTC
I checked that on Fedora 21 
  perl -e "print 'MESSAGE=', 'a'x100000, 'b'" | logger --journald
happily sends to the journal as a single line with 1e5 characters without splitting anything. So the current restriction of 2K from stdout is pretty arbitrary and restrictive indeed.
Comment 10 Mike Craig 2015-03-04 15:01:28 UTC
We're struggling with line truncation at 2048 chars as well. We log JSON to stdout in Docker containers on CoreOS, much like SW described. Newline placement is important for Logstash and other tools to handle the logs correctly.

What's the status of this issue?
Comment 11 Matt Walters 2015-05-20 19:10:42 UTC
We're having this same problem. It seems the reasonable solution is to have a sane default, while providing the ability to override via configuration. 

Any opinions on this?
Comment 12 SW 2015-05-21 08:46:13 UTC
We had been using this patch/fork of systemd-docker to hack around the problem on CoreOS


Probably not up to date, not something I'm personally using...
Comment 13 Martin Garton 2015-07-01 16:06:42 UTC
We've just hit this too.

Looking at the workarounds, although they do allow longer lines, they are not satisfactory for other reasons.

Does anyone have a view on what the correct fix for allowing longer lines is?

A 16K limit would fix 100% of the cases I've seen but as Lennart correctly pointed out, this is not a proper fix.

I have to say it does seem inconsistent that the 2048 limit doesn't apply when using "logger --journald"
Comment 14 Martin Garton 2015-07-16 12:56:33 UTC
Any suggestion on what the proper fix should be?  I'm happy to have a go at fixing this if there is agreement on the approach.
Comment 15 francis.x.fitzpatrick 2016-02-16 22:06:27 UTC
For all those using CoreOS/Docker, we just added `--log-driver=journald` to our `docker run` command and we're not affected any more.

In addition, we also threw in `StandardOutput=null` to our unit files so logs wouldn't be duplicated.
Comment 16 Theodore Cowan 2016-06-11 13:39:19 UTC
This issue is causing me emotional distress.  2k is not a reasonable limit for log messages.  If there must be a limit let's make it 4x or 8x larger.  Is this a systemd project decision or a distribution decision?
Comment 17 Lennart Poettering 2016-06-14 12:36:57 UTC
we use LINE_MAX for lack of a better value, which glibc defines to 2K.

I figure we could enforce a limit of 64K or so in journald instead, but we have to enforce something. Please consider filing a PR that introduces a new limit define and sets it higher.
Comment 18 Jérôme Laforge 2016-12-09 10:46:50 UTC
Do you know whether somebody has planed to fix this issue?
Comment 19 Zbigniew Jedrzejewski-Szmek 2016-12-17 18:51:59 UTC
There's a PR, but it needs reworking.
Comment 20 Jérôme Laforge 2016-12-28 12:33:12 UTC
Thanks for this. 
Just for sake of completeness, this PR can be found here :
Comment 21 Rob 2017-05-09 10:31:17 UTC
I have a very small proxy that pipes stdin to the journald socket.

We're planning on using this in our service unit files to address the problem, is that an awful idea?
Comment 22 Rob 2017-05-09 10:37:53 UTC
(In reply to Rob from comment #21)
> I have a very small proxy that pipes stdin to the journald socket.
> https://git.reform.hmcts.net/roby/journald-proxy
> We're planning on using this in our service unit files to address the
> problem, is that an awful idea?

Whoops, that should be https://github.com/robyoung/journald-proxy
Comment 23 Zbigniew Jedrzejewski-Szmek 2017-09-22 08:24:41 UTC
Fixed in https://github.com/systemd/systemd/commit/ec20fe5ffb8a00469bab209fff6c069bb93c6db2.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.