Ok, I'm being told "systemd.log_target=null" takes care of the second part of the issue.

(In reply to comment #0)
> Supplying "debug" on the kernel command line gets parsed by systemd. If
> a random assertion in it fires
> it floods dmesg and I cannot log into the machine anymore.I have to
> reboot and remove "debug" from the command line so that the machine can
> boot. BUT(!)

That is the expected current behaviour, "debug" can cause "too many" messages to be useful anymore if things are broken.

> I want to see kernel debug messages so I have to be able
> to supply "debug" *without* systemd interfering.

Then use the kernels loglevel= parameter for the kernel only, not affecting Base OS tools.

> What it should do, instead, is *not* parse "debug" but something else
> with a namespace, i.e. "systemd.debug" or whatever as "debug" is a
> kernel cmdline parameter and not aimed for systemd.

The generic term "debug" is read by Base OS tools too.

> And while we're at it, we'd need a configurable option to tell systemd
> *not* to log to dmesg for when one wants to see only output from the
> kernel in the kernel ring buffer, i.e. something like "systemd.nodmesg"
> or so.

Like for the kernel, there are options to fin-grain control systemd's logging behaviour; just do not use the generic term "debug" which is a convenience shortcut for the kernel AND the Base OS.

(In reply to comment #2)
> Then use the kernels loglevel= parameter for the kernel only, not
> affecting Base OS tools.

Actually, I'd like to continue using "debug" - a workflow a lot of
kernel people have been doing for years - without anything interfering,
systemd including.

> The generic term "debug" is read by Base OS tools too.

... which was wrong to begin with, without a namespace prefix.

> Like for the kernel, there are options to fin-grain control systemd's
> logging behaviour; just do not use the generic term "debug" which is a
> convenience shortcut for the kernel AND the Base OS.

And yet, it was working exactly as expected until now.

Is there any other way to tone down systemd logging as much as possible
without having to supply "systemd.log_target=null" on the command line
of each box I'm booting? I'm guessing you're going to say to go edit
/etc/systemd/system.conf each time I'm booting the box for the first
time...

Thanks.

(In reply to comment #3)
> > The generic term "debug" is read by Base OS tools too.
> 
> ... which was wrong to begin with, without a namespace prefix.

Which we do not agree. Generic terms are generic, not the first user
owns them.

> Is there any other way to tone down systemd logging as much as possible
> without having to supply "systemd.log_target=null" on the command line
> of each box I'm booting? I'm guessing you're going to say to go edit
> /etc/systemd/system.conf each time I'm booting the box for the first
> time...

Systemd does not "log" to the console, or to the kernel by default.

If it is about the console status output, use "quiet", just like it disables
the kernel boot log.

(In reply to comment #4)
> Which we do not agree. Generic terms are generic, not the first user
> owns them.

Right, and breaking existing use cases by hijacking generic terms is
also ok. I was right to be very skeptical when considering opening a bug
here.

> If it is about the console status output, use "quiet", just like it
> disables the kernel boot log.

No, I want to say "debug" on the kernel command line and *not* see
systemd output at all but kernel output only. Is this possible?

Thanks.

(In reply to comment #4)
> (In reply to comment #3)
> > > The generic term "debug" is read by Base OS tools too.
> > 
> > ... which was wrong to begin with, without a namespace prefix.
> 
> Which we do not agree. Generic terms are generic, not the first user
> owns them.
> 

Come on Kai. The kernel is the primary user of its own command line and has a reasonable expectation that it's the default namespace for parameters. In cases where that is not true (such as modules), the parameter get prefixed by the name of the module affected like i915.foo. There may be base utils that parse the command line (they really shouldn't) but even if they do, their reaction should not be to scribble all over the kernel log.

systemds default debug behaviour should not be to hijack the kernel log. I can see cases where you want to do that if you need the output from a serial console but that should be explicitly stated with a systemd.log_target.

Bugzilla is not a discussion forum, and repeating the same

(In reply to comment #7)
> Bugzilla is not a discussion forum, and repeating the same

So do not discuss and bring arguments.

(In reply to comment #7)
> Bugzilla is not a discussion forum, and repeating the same

Ok, the bug is that a *kernel* parameter compels a *userspace* program to trash the *kernels* log with enough information to break boot due to timeouts.

(In reply to comment #9)
> (In reply to comment #7)
> > Bugzilla is not a discussion forum ...

Again, move discussions to the mailing list; this is a bug tracker, but
there is no bug to track or fix here.

(In reply to comment #10)
> Again, move discussions to the mailing list; this is a bug tracker,
> but there is no bug to track or fix here.

Dude, are you for real? Are you actually actively trying to be a d*ck or
you're this way by default. We're telling you there's a serious issue
and we're even being constructive by giving suggestions and you're
deflecting.

You have got to be f*cking kidding me!

Hmm, a user adds to the *kernel* command line "debug" and systemd starts spitting out so much crap that the system doesn't boot anymore? That sounds like a major regression to me. Note this is a kernel command line, not a systemd command line. Userspace tools should not be using the same kernel parameters that are defined by the kernel. That's just broken and wrong.

This bugzilla is the poster child of why people hate systemd and do not trust the developers that work on it.

This is utterly ridiculous, and it matches what I have observed that the system becomes undebuggable on a dracut/systemd system.  There are a lot of things I genuinely like about systemd, but this aspect is a disaster.

I believe this report deserves a little bit more serious architectural discussion and consideration than what it has been given.

2014-04-02 08:57 UTC - reported
2014-04-02 10:19 UTC - bug closed with a not a bug resolution

While for systemd it may make sense to parse debug the kernel has been using it for ages. Even still, for system it may make sense but interpreting it this way means we may run into other generic namespace kernel parameters and the expectations seems to be to close all of those issues as a non bug.

This seems rather unreasonable given we can't realistically expect to allow all userspace application to take on over the kernel ring buffer interpreting the generic namespace as they see fit. If anything they should honor the kernel for the generic namespace and since systemd is the new application and it is very conscientiousness about namespaces it can provide its own.

just an fyi, comment by torvalds at http://lkml.iu.edu/hypermail/linux/kernel/1404.0/01331.html

(In reply to comment #10)
> (In reply to comment #9)
> > (In reply to comment #7)
> > > Bugzilla is not a discussion forum ...
> 
> Again, move discussions to the mailing list; this is a bug tracker, but
> there is no bug to track or fix here.

If the system is broken or otherwise not performing properly, then enabling debug is the way to determine the cause. If enabling debug breaks the system, then that is a bug by definition.

In this case, I'm inclined to state that the very existence of systemd is a big.

Kay, Please go die in a fire along with Lennart. Your type is the cancer that is killing any semblence of usability Linux once had.

(In reply to comment #0)
> Supplying "debug" on the kernel command line gets parsed by systemd. If
> a random assertion in it fires, something like this, for example:
> 
> [  150.308000] systemd-journald[1559]: Assertion
> 'dual_timestamp_is_set(&e->timestamp)' failed at
> src/libsystemd/sd-event/sd-event.c:2191, function
> sd_event_get_now_monotonic(). Ignoring.
> [  150.308000] systemd-journald[1559]: Assertion
> 'dual_timestamp_is_set(&e->timestamp)' failed at
> src/libsystemd/sd-event/sd-event.c:2191, function
> sd_event_get_now_monotonic(). Ignoring.
> 
> it floods dmesg and I cannot log into the machine anymore. I have to
> reboot and remove "debug" from the command line so that the machine can
> boot. BUT(!), I want to see kernel debug messages so I have to be able
> to supply "debug" *without* systemd interfering.
Yep, this looks like a bug, and the assertion should be fixed. If this is still occuring, can you file a new bug specifying what systemd version you are using, or post to the mailing list?

> What it should do, instead, is *not* parse "debug" but something else
> with a namespace, i.e. "systemd.debug" or whatever as "debug" is a
> kernel cmdline parameter and not aimed for systemd.
Yes, systemd.log_level is that.

> And while we're at it, we'd need a configurable option to tell systemd
> *not* to log to dmesg for when one wants to see only output from the
> kernel in the kernel ring buffer, i.e. something like "systemd.nodmesg"
> or so.
Yes, use loglevel=, and or just systemd.log_level=.

(In reply to comment #5)
> No, I want to say "debug" on the kernel command line and *not* see
> systemd output at all but kernel output only. Is this possible?
Yes, use 'loglevel=...'. Or .debug systemd.log_target=...., or .debug systemd.log_level=...'.

As has been said, plain 'debug' turns more than just the kernel debugging, and this is extremely useful for general boot problems, where it is not clear what is a bug in the kernel, and what is a but in systemd, or plymouth, or elsewhere.
Available options are listed in http://www.freedesktop.org/software/systemd/man/systemd.html#Kernel%20Command%20Line. If a combination of options which implements the functionality you need is not available, please holler.

(In reply to comment #17)
> (In reply to comment #0)
> 
> > What it should do, instead, is *not* parse "debug" but something else
> > with a namespace, i.e. "systemd.debug" or whatever as "debug" is a
> > kernel cmdline parameter and not aimed for systemd.
> Yes, systemd.log_level is that.
> 
No, systemd.log_level is NOT that. If it were, then adding debug to the KERNEL's command line would not have resulting in systemd spamming the kernel to death. Systemd should be looking only at parameters that are in its namespace. If it already looks at systemd.log_level, then it should definitely NOT be looking at debug as that is redundant to the log_level parameter specifically for it, which can be used to turn up the logging to whatever level of insanity is desired.

(In reply to comment #18)
Get a non-anonymous account if you want to post.

(In reply to comment #19)
> Get a non-anonymous account if you want to post.

Give technical arguments if you want to post.

Do we have a commit SHA already, so that we can actually close this?

I'm very disappointed with the attitude of the systemd developers.

They clearly don't care about systemd, let alone Linux.

I strongly believe systemd should be forked to avoid dealing with these bad developers, or the distributions that are adopting systemd should reconsider adopting this project.

There seems to be quite a lot of misunderstanding here. systemd has used "debug" to set its own level to debug for quite some time, at least v208, which is from mid-2013.

The issue causing the acute pain was an broken assertion, a bug that should have been fixed before the thread on LKML even started. No one claimed that assertion  failure wasn't a bug, and no one hesitated to fix it (see comment #17). So, "debug" should already be usable without also setting systemd.log_level, which should have fully worked around the issue even with the assertion bug.

In short, if everything was fine for you as of weeks or months ago, it should already be back to being fine.

The question now is the semantics of the kernel command line, specifically the scope which "debug" affects. I've personally given the +1 to Greg's patch, which would switch it to "systemd.debug", but there isn't consensus to do that.

The primary argument against the namespacing is catering to end users trying to troubleshoot who are unfamiliar with the kernel/udev/systemd split of responsibilities. Obviously, the counter-argument is to namespace in order to fully isolate systemd's debug mode from the kernel's (without requiring systemd.log_level) and avoid a similar assertion bug affecting developers the same way.

Because this is now a discussion about what *intended* functionality should be, most discussion is happening on the mailing list in the thread started by Greg's patch. I encourage you to post there if you would like to contribute. If you're merely going to flame, especially in a way showing that you don't understand the assertion fix vs. debug option distinction, then please find somewhere else to post outside of systemd Bugzilla and the mailing list.

Lennart Poettering and Kay Sievers: worst developers of the FOSS community.

Systemd needs to be forked NOW.

Linus explained it well already:

Kay and Lennart: please just go away, disappear from the FOSS community, we don't need you and your crap.

03:20:36 <davidstrauss> Is there any way to block/ban problem users from Bugzilla?

Douchebag.

The systemd developers are not just idiots who can't code, but also dictators who want to censor what people say about their shitty project.

systemd is the worst thing that ever have happened to Linux and FOSS.

Fork now and fix it, boycott their idiot developers.

(In reply to comment #25)
> 03:20:36 <davidstrauss> Is there any way to block/ban problem users from
> Bugzilla?
> 
> Douchebag.
> 
> The systemd developers are not just idiots who can't code, but also
> dictators who want to censor what people say about their shitty project.
> 
> systemd is the worst thing that ever have happened to Linux and FOSS.
> 
> Fork now and fix it, boycott their idiot developers.

Why fork it? There is no part of systemd that is worthwhile. It would be better to start fresh, but there is no need. There are already numerous sane alternatives. OpenRC and Upstart are two that come to mind that are ready to go, in use by major distros, and not braindamaged. They are not the only alternatives.

The real problem is systemd specific crap getting pushed into other project (e.g. Xorg). That propagation of systemd dependency is a malicious action which harms not only other distros which have chosen not to use systemd, but also completely other OSes which also use that code (*BSD, Solaris/illumOS, OS X, etc).

If it was mere bad behavior of the systemd developers, the clueful distros could avoid this mess of a project and clueful users could avoid the distros foolish enough to use this crap. It is the forced dependency that is the most malicious act of the systemd developers. Hopefully the FOSS community will wake up and eject these fucktards.

(In reply to comment #22)
> There seems to be quite a lot of misunderstanding here. systemd has used
> "debug" to set its own level to debug for quite some time, at least v208,
> which is from mid-2013.
> 
> The issue causing the acute pain was an broken assertion, a bug that should
> have been fixed before the thread on LKML even started. No one claimed that
> assertion  failure wasn't a bug, and no one hesitated to fix it (see comment
> #17). So, "debug" should already be usable without also setting
> systemd.log_level, which should have fully worked around the issue even with
> the assertion bug.
> 
> In short, if everything was fine for you as of weeks or months ago, it
> should already be back to being fine.
> 
> The question now is the semantics of the kernel command line, specifically
> the scope which "debug" affects. I've personally given the +1 to Greg's
> patch, which would switch it to "systemd.debug", but there isn't consensus
> to do that.
> 
> The primary argument against the namespacing is catering to end users trying
> to troubleshoot who are unfamiliar with the kernel/udev/systemd split of
> responsibilities. Obviously, the counter-argument is to namespace in order
> to fully isolate systemd's debug mode from the kernel's (without requiring
> systemd.log_level) and avoid a similar assertion bug affecting developers
> the same way.
> 
> Because this is now a discussion about what *intended* functionality should
> be, most discussion is happening on the mailing list in the thread started
> by Greg's patch. I encourage you to post there if you would like to
> contribute. If you're merely going to flame, especially in a way showing
> that you don't understand the assertion fix vs. debug option distinction,
> then please find somewhere else to post outside of systemd Bugzilla and the
> mailing list.

Thank you for being sane and stating the true situation clearly. As has been said on LKML, if Kay wasn't being a dick and had merely stated the broken assertion was a known and resolved issue, there would not have been any problem. It was the attitude in his response that has made so many in the community finally stand up and tell Kay (and Lennart) to simply fuck off for good.

(In reply to comment #25)
> 03:20:36 <davidstrauss> Is there any way to block/ban problem users from
> Bugzilla?
> 
> Douchebag.
> 
> The systemd developers are not just idiots who can't code, but also
> dictators who want to censor what people say about their shitty project.
>
> systemd is the worst thing that ever have happened to Linux and FOSS.
>
> Fork now and fix it, boycott their idiot developers.

Censorship is preventing people from saying what they want to say, regardless of place or a government preventing publication of something. We haven't done that at all. Please find your own place to post what you want to say; there are plenty. Calling to shun some people entirely is also rather more hostile than trying to keep things on-topic and free of personal attacks in a bug tracker.

And that's where you are: a systemd bug tracker, in an issue discussing using the use of the "debug" option by the kernel and systemd (and implications thereof, like the assert issue spilling over). Your last few posts have been empty of anything but insults and calls to fork. It's a waste of my time as a systemd maintainer to come back to Bugzilla repeatedly and only see another flame post from you.

I don't know where you learned this behavior, but it's not productive here. It's also rather "pot calling the kettle black" to keep posting off-topic flames to a bug tracker about why you think some of our developers are bad for the FOSS community.

Seriously, find somewhere else to vent.

(In reply to comment #28)
> (In reply to comment #25)
> > 03:20:36 <davidstrauss> Is there any way to block/ban problem users from
> > Bugzilla?
> > 
> > Douchebag.
> > 
> > The systemd developers are not just idiots who can't code, but also
> > dictators who want to censor what people say about their shitty project.
> >
> > systemd is the worst thing that ever have happened to Linux and FOSS.
> >
> > Fork now and fix it, boycott their idiot developers.
> 
> Censorship is preventing people from saying what they want to say,
> regardless of place or a government preventing publication of something. We
> haven't done that at all. Please find your own place to post what you want
> to say; there are plenty. Calling to shun some people entirely is also
> rather more hostile than trying to keep things on-topic and free of personal
> attacks in a bug tracker.
> 
> And that's where you are: a systemd bug tracker, in an issue discussing
> using the use of the "debug" option by the kernel and systemd (and
> implications thereof, like the assert issue spilling over). Your last few
> posts have been empty of anything but insults and calls to fork. It's a
> waste of my time as a systemd maintainer to come back to Bugzilla repeatedly
> and only see another flame post from you.
> 
> I don't know where you learned this behavior, but it's not productive here.
> It's also rather "pot calling the kettle black" to keep posting off-topic
> flames to a bug tracker about why you think some of our developers are bad
> for the FOSS community.
> 
> Seriously, find somewhere else to vent.

OK I'm sorry about that, I actually like systemd and I shouldn't have insulted anyone, I'm very sorry about that, I deeply regret and apologize.

I'm just sad about what have happened with all this problem between systemd and the kernel and especially about Kay Sievers making problems actually harder to solve with his bad attitude.

(In reply to comment #29)
> OK I'm sorry about that, I actually like systemd and I shouldn't have
> insulted anyone, I'm very sorry about that, I deeply regret and apologize.
> 
> I'm just sad about what have happened with all this problem between systemd
> and the kernel and especially about Kay Sievers making problems actually
> harder to solve with his bad attitude.

Thank you for that. It means a lot. Seeing things progressing to civility is a step forward, even if getting consensus is still a ways off.

(In reply to comment #24)
> Systemd needs to be forked NOW.
> 
> Linus explained it well already:
> 
> Kay and Lennart: please just go away, disappear from the FOSS community, we
> don't need you and your crap.

Please cite that quote as I've not seen it on LKML and didn't find it with a quick search, so I'm interested in where else this topic is being discussed.

(In reply to comment #22)
> 
> The primary argument against the namespacing is catering to end users trying
> to troubleshoot who are unfamiliar with the kernel/udev/systemd split of
> responsibilities. Obviously, the counter-argument is to namespace in order
> to fully isolate systemd's debug mode from the kernel's (without requiring
> systemd.log_level) and avoid a similar assertion bug affecting developers
> the same way.
> 

Is this a valid argument, though?  Surely any end user sophisticated enough to attempt to troubleshoot a problem at this level will also be sufficiently knowledgeable to able to type systemd.debug on the kernel command line.

Being able to sandbox debugging output with something as simple as namespacing seems like it would be great feature, not a problem; i.e.

  ---debug
  +++systemd.debug

makes systemd better, as far as I can tell.

(In reply to comment #23)
> Lennart Poettering and Kay Sievers: worst developers of the FOSS community.

Ad hominem attacks are completely out of place on a bug report comment list.  You're disrupting FOSS community in a very negative way by doing things like this.  Save this kind of thing for your blog.

Thanks.

(In reply to comment #33)
> (In reply to comment #23)
> > Lennart Poettering and Kay Sievers: worst developers of the FOSS community.
> 
> Ad hominem attacks are completely out of place on a bug report comment list.
> You're disrupting FOSS community in a very negative way by doing things like
> this.  Save this kind of thing for your blog.
> 
> Thanks.

Sorry about that, I don't actually believe what I said when I wrote that. I don't believe that Lennart and Kay are the "worst developers" as I quite like systemd and I enjoy using it every day.

I was angry about the whole situation with Kay not being helpful enough with others and making things harder to solve.

But that's not an excuse for my bad behavior and I don't have excuses, it won't happen again and I'm sorry.

I only hope the developers involved will put their egos aside and work together and improve things for a better future.

Again, my apologies.

(In reply to comment #31)
> (In reply to comment #24)
> > Systemd needs to be forked NOW.
> > 
> > Linus explained it well already:
> > 
> > Kay and Lennart: please just go away, disappear from the FOSS community, we
> > don't need you and your crap.
> 
> Please cite that quote as I've not seen it on LKML and didn't find it with a
> quick search, so I'm interested in where else this topic is being discussed.

Linus Torvalds wrote:

+Paul Morgan I think what you (and others) seem to miss is that the systemd people made the "debug" option that we introduced not just do something - but do something useless that actively broke other peoples use of that option.

It doesn't matter who "owns" it, the fact is, they broke it.

Ok, fine. Bugs happen, and that's not what makes people upset.

What makes me (and others) upset is that when the bug is reported, with explanations and a suggestion for how to fix it, Kay just closed the bug-report, claiming it wasn't a bug.

Seriously? You want to debug kernel stuff, using the kernel command line command "debug" that makes the kernel more verbose, and now the systemd people say "sorry, we stole your thing and made it useless, and it's not a bug because you didn't call shot-gun".

Now, if this was an isolated incident, I personally would let it go. There are bad engineers out there, it's not worth worrying about. Ignore them and move on.

But this is not an isolated incident. This is how Kay has treated other bugs in the past. Literally months of stalling, closing bug-reports, and blaming other people and projects for problems that he caused, telling others how they should change their projects because he broke something, and obviously it can't be his fault.

And that is a problem.

https://plus.google.com/+TheodoreTso/posts/K7ijdmxJ8PF

systemd in git since a while will turn its debug output on if the kernel command line cotnains "debug", this will stay that way. However, it has been changed to not dump things to kmsg as soon as the journal is up. Up to that point we will still dump things to kmsg howerver, since there's no other nice place to put this.

Closing.