Bug 89102 - RFE: Support custom fields in systemd-journal-upload
Summary: RFE: Support custom fields in systemd-journal-upload
Status: NEW
Alias: None
Product: systemd
Classification: Unclassified
Component: general (show other bugs)
Version: unspecified
Hardware: Other All
: medium normal
Assignee: systemd-bugs
QA Contact: systemd-bugs
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-02-12 10:44 UTC by Duncan Innes
Modified: 2015-02-17 10:16 UTC (History)
0 users

See Also:
i915 platform:
i915 features:


Attachments

Description Duncan Innes 2015-02-12 10:44:08 UTC
Expand the options for systemd-journal-upload to allow export of log data in JSON format (for example), and to provide optional key-value pair data that will be added to the exported data.

Reason:
Receive logs from hundreds of systems on a central log consolidation server.
There will be little to differentiate any of these log entries from another.
If a client can insert (for example):
"Environment": "Dev",
"Project": "MainWebsite",
"Type": "AppServer",

into the exported data, then a user can parse the consolidated logs and search for data in a more meaningful way:

Show all Firewall Denials on Production WebServers.
Show all Failed Login Attempts on Production AppServers.
Comment 1 Zbigniew Jedrzejewski-Szmek 2015-02-12 12:31:43 UTC
Your request should be split into two or ever three parts.

First, adding custom fields to the log event stream. This has been requested before, and should be done.

Second, we might consider adding fields from /usr/lib/os-release automatically, things like description of the OS like CPE_NAME, BUILD_ID, and stuff from /etc/machine-info, DEPLOYMENT, LOCATION, CHASSIS. I can see how this would be useful, but I'm not sure if it is worth the overhead.

Third, JSON format. I'm not sure why you would s-j-upload specifically to support JSON. journalctl -o json/json-pretty gives you JSON already. s-j-upload is really a tool to communicate with s-j-remote, and the binary and HTTP protocols it already has should be enough.
Comment 2 Duncan Innes 2015-02-12 13:32:35 UTC
(In reply to Zbigniew Jedrzejewski-Szmek from comment #1)
> Your request should be split into two or ever three parts.
> 

Sure - no problem doing this.

>
> First, adding custom fields to the log event stream. This has been requested
> before, and should be done.
> 

Could you point me to the request.  I did search for similar, but turned up nothing.  If there's already an RFE, I'll just track that one.

>
> Second, we might consider adding fields from /usr/lib/os-release
> automatically, things like description of the OS like CPE_NAME, BUILD_ID,
> and stuff from /etc/machine-info, DEPLOYMENT, LOCATION, CHASSIS. I can see
> how this would be useful, but I'm not sure if it is worth the overhead.
> 

Sure - I didn't realise /etc/machine-info was able to store user data.  It's limited to PRETTY_HOSTNAME, ICON_NAME, CHASSIS, and DEPLOYMENT though.  I think it would be a big benefit to large estates to be able to specify custom fields as well here.  Would the overhead not be limited to reading this file at s-j-upload startup, then inserting the data into each exported entry?  A flag on s-j-upload command could indicate whether or not to add extra fields.  Default to NO.  User then chooses whether to increase system load & network traffic by adding extra data.  I might be over-simplifying this though.

>
> Third, JSON format. I'm not sure why you would s-j-upload specifically to
> support JSON. journalctl -o json/json-pretty gives you JSON already.
> s-j-upload is really a tool to communicate with s-j-remote, and the binary
> and HTTP protocols it already has should be enough.

I'll leave this ticket for the supporting of extra fields in the log data.  I'll open a separate ticket for adding the export in JSON format.
Comment 3 Zbigniew Jedrzejewski-Szmek 2015-02-12 15:09:53 UTC
(In reply to Duncan Innes from comment #2)
> Could you point me to the request.  I did search for similar, but turned up
> nothing.  If there's already an RFE, I'll just track that one.
Might now have been public. Chances are that it'll get implemented in
a week or two.

> > Second, we might consider adding fields from /usr/lib/os-release
> > automatically, things like description of the OS like CPE_NAME, BUILD_ID,
> > and stuff from /etc/machine-info, DEPLOYMENT, LOCATION, CHASSIS. I can see
> > how this would be useful, but I'm not sure if it is worth the overhead.
>
> Sure - I didn't realise /etc/machine-info was able to store user data.  It's
> limited to PRETTY_HOSTNAME, ICON_NAME, CHASSIS, and DEPLOYMENT though.  I
> think it would be a big benefit to large estates to be able to specify
> custom fields as well here.  Would the overhead not be limited to reading
> this file at s-j-upload startup, then inserting the data into each exported
> entry?  A flag on s-j-upload command could indicate whether or not to add
> extra fields.  Default to NO.  User then chooses whether to increase system
> load & network traffic by adding extra data.  I might be over-simplifying
> this though.
I'm thinking of something like this in /etc/systemd/journal-upload.conf:
SendFields=DEPLOYMENT CPE_NAME CUSTOM1=value
where fields without a value specified would be extracted from machine-info
and os-release, and fields with a value would be send with the given value.
Comment 4 Duncan Innes 2015-02-17 10:16:13 UTC
(In reply to Zbigniew Jedrzejewski-Szmek from comment #3)
> I'm thinking of something like this in /etc/systemd/journal-upload.conf:
> SendFields=DEPLOYMENT CPE_NAME CUSTOM1=value
> where fields without a value specified would be extracted from machine-info
> and os-release, and fields with a value would be send with the given value.

Sounds ideal to me.  Can't see the need for too many custom fields.
Would it be acceptable to enter custom fields in /etc/machine-info?  Then the fields could simply be referenced in your config file.


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.