Bug 94508 - dbus-1.10 (1.10.6 and 1.0.8 tries) breaks gnome 3.18 logout leaving running processes
Summary: dbus-1.10 (1.10.6 and 1.0.8 tries) breaks gnome 3.18 logout leaving running p...
Status: RESOLVED MOVED
Alias: None
Product: dbus
Classification: Unclassified
Component: core (show other bugs)
Version: 1.10
Hardware: Other All
: medium major
Assignee: D-Bus Maintainers
QA Contact: D-Bus Maintainers
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-03-12 08:31 UTC by Pacho Ramos
Modified: 2018-10-12 21:26 UTC (History)
5 users (show)

See Also:
i915 platform:
i915 features:


Attachments
systemd-cgls output (6.38 KB, text/plain)
2016-04-18 08:41 UTC, Pacho Ramos
Details

Description Pacho Ramos 2016-03-12 08:31:11 UTC
For some days I am suffering an strange behavior: when I logout, lots of processes from the gnome session are kept running. This causes problems when other users try to login in their accounts.

Googling a bit, I saw the same problem on Arch (I am using Gentoo):
https://bbs.archlinux.org/viewtopic.php?id=204307

And I have seen that this is caused by dbus-1.10.x upgrade, if I switch back 1.8.x releases all go back to normal and processes die when logging out

Thanks a lot for your help :)
Comment 1 Pacho Ramos 2016-03-13 15:57:18 UTC
Or, if this is a fault in other part (gnome-session?, systemd?) please let me know because I am completely lose about how to solve this issue
Comment 2 Simon McVittie 2016-03-25 10:37:18 UTC
Sorry, for some reason I didn't get email about this new bug being filed. My response is basically the same as to your equivalent GNOME bug <https://bugzilla.gnome.org/show_bug.cgi?id=764146>:

(In reply to Pacho Ramos from comment #0)
> For some days I am suffering an strange behavior: when I logout, lots of
> processes from the gnome session are kept running.

dbus-daemon 1.10 has a new --enable-user-session mode. Upstream, this is opt-in behaviour, but some distributions (Arch, Fedora) have enabled this unconditionally. In Debian, I set it up as a separate package (dbus-user-session_*.deb) which can be installed in addition to the normal dbus packages.

The user bus works alongside the `systemd --user` represented by user@1000.service (or whatever your numeric uid is, if not 1000). It should terminate if and only if the `systemd --user` does: I specifically don't want it to have its own special D-Bus life-cycle. If the hypothetical kdbus future ever happens, that isn't likely to have its own special life cycle either.

If you want user@1000.service to terminate with your last login session, enable KillUserProcesses=yes in /etc/logind.conf (see logind.conf(5)). Note that this will break screen(1) and similar tools, unless/until they are modified to register themselves as a systemd user-session in their own right.

By adjusting KillUserProcesses, you can choose user-services that persist, or user-services that don't persist. I don't think it makes sense to have an arbitrary distinction between user-services that do and don't use D-Bus for IPC.
Comment 3 Simon McVittie 2016-03-25 10:52:15 UTC
(In reply to Simon McVittie from comment #2)
> The user bus works alongside the `systemd --user` represented by
> user@1000.service (or whatever your numeric uid is, if not 1000). It should
> terminate if and only if the `systemd --user` does: I specifically don't
> want it to have its own special D-Bus life-cycle.

See <https://lists.freedesktop.org/archives/systemd-devel/2015-January/027711.html> for context and terminology.

If OS integrators (i.e. distributions) don't want this behaviour, then they should not --enable-user-session. This is really a decision about how you build operating systems, not a decision about how you do IPC.

If distributions intend to support both the login-session-centric and user-session-centric views of the world, and their D-Bus maintainers are willing to take on the extra maintenance effort to support both, then they should do what I did in Debian, and split out /usr/lib/systemd/user/dbus.{socket,service} into a separate binary package:

dbus (source)
\- libdbus-1-3 (binary)
    \- libdbus-1.so.3, etc.
\- libdbus-dev (binary)
    \- libdbus-1.a, etc.
\- dbus (binary)
    \- dbus-daemon, etc.
    \- [/usr]/lib/systemd/system
\- dbus-user-session (binary) (new with 1.10)
    \- /usr/lib/systemd/user

In source-based distributions like Gentoo where you install everything or nothing from a source package, it could be a USE flag or something.

> This causes problems when other users try to login in their accounts

If there are specific services that behave particularly poorly with multiple instances, I think this should be considered to be a bug in those services: they should "get out of the way" when their user does not have any surviving login-sessions. For instance, PulseAudio should release audio devices when it becomes inactive, if systemd does not already forcibly reclaim them.
Comment 4 Simon McVittie 2016-03-25 11:28:20 UTC
(In reply to Simon McVittie from comment #3)
> If there are specific services that behave particularly poorly with multiple
> instances, I think this should be considered to be a bug in those services:
> they should "get out of the way" when their user does not have any surviving
> login-sessions.

I've opened a systemd feature request https://github.com/systemd/systemd/issues/2900 which would make this a lot easier.
Comment 5 Pacho Ramos 2016-03-28 08:37:22 UTC
(In reply to Simon McVittie from comment #2)
[...]
> dbus-daemon 1.10 has a new --enable-user-session mode. Upstream, this is
> opt-in behaviour, but some distributions (Arch, Fedora) have enabled this
> unconditionally. In Debian, I set it up as a separate package
> (dbus-user-session_*.deb) which can be installed in addition to the normal
> dbus packages.
> 

It's also unconditional on Gentoo as we don't pass any configure switch for this and, by default, user-session is enabled then

> The user bus works alongside the `systemd --user` represented by
> user@1000.service (or whatever your numeric uid is, if not 1000). It should
> terminate if and only if the `systemd --user` does: I specifically don't
> want it to have its own special D-Bus life-cycle. If the hypothetical kdbus
> future ever happens, that isn't likely to have its own special life cycle
> either.

I will check what is keeping systemd --user running then. In some setups it was deluged (that was kept running on purpose as it's meant to keep downloading things even if I logout... it worked nice with old dbus but now :/)

> 
> If you want user@1000.service to terminate with your last login session,
> enable KillUserProcesses=yes in /etc/logind.conf (see logind.conf(5)). Note
> that this will break screen(1) and similar tools, unless/until they are
> modified to register themselves as a systemd user-session in their own right.

Yeah, I am aware of the option but, by default, it's not enabled by systemd upstream and, then, people will get this strange behavior of processes running forever by default. Also, as we in Gentoo try to keep close to upstream and this feature would break some apps (screen is used extensively by our users to, for example, upgrade their systems, as they can take a long time to update) I wonder if we could switch that default setting in our side :(

> 
> By adjusting KillUserProcesses, you can choose user-services that persist,
> or user-services that don't persist. I don't think it makes sense to have an
> arbitrary distinction between user-services that do and don't use D-Bus for
> IPC.

I am not sure if I understand properly, sorry. KillUserProcesses looks to allow me to only set it to true/false and, then, kill all user processes or none, but it doesn't seem to let me to configure what processes I could keep running (like screen, deluged...)

Also, I wonder how "correct" is that the tool that is launching dbus at login time is not taking care or shutting it down at logout (by correct I mean that is looks a bit strange to me from a "user" point of view... as probably there are some reason I don't know for this being an "advantage" :/ It seems that kdbus will only support user-session... but last time I checked, kdbus was a bit "stalled" because it was going to be rethink or similar)

(In reply to Simon McVittie from comment #3)
> (In reply to Simon McVittie from comment #2)
> > The user bus works alongside the `systemd --user` represented by
> > user@1000.service (or whatever your numeric uid is, if not 1000). It should
> > terminate if and only if the `systemd --user` does: I specifically don't
> > want it to have its own special D-Bus life-cycle.
> 
> See
> <https://lists.freedesktop.org/archives/systemd-devel/2015-January/027711.
> html> for context and terminology.
> 
> If OS integrators (i.e. distributions) don't want this behaviour, then they
> should not --enable-user-session. This is really a decision about how you
> build operating systems, not a decision about how you do IPC.

OK thanks, but, please, take care dbus upstream is enabling by default the user-session switch. I say that because, then, distributions like Gentoo and probably Arch are getting this enabled without knowing the downsides... like those exposed in the first paragraph of "What this means for D-Bus".

As I understand it, it seems that we need additional configuration for properly support users sessions and, then, maybe it would be better to now enable it by default until those unit files are collected in some "official" dbus repo or similar that we could refer to (and that could allow us to learn how to use ;)


> 
> If distributions intend to support both the login-session-centric and
> user-session-centric views of the world, and their D-Bus maintainers are
> willing to take on the extra maintenance effort to support both, then they
> should do what I did in Debian, and split out
> /usr/lib/systemd/user/dbus.{socket,service} into a separate binary package:
> 
> dbus (source)
> \- libdbus-1-3 (binary)
>     \- libdbus-1.so.3, etc.
> \- libdbus-dev (binary)
>     \- libdbus-1.a, etc.
> \- dbus (binary)
>     \- dbus-daemon, etc.
>     \- [/usr]/lib/systemd/system
> \- dbus-user-session (binary) (new with 1.10)
>     \- /usr/lib/systemd/user
> 
> In source-based distributions like Gentoo where you install everything or
> nothing from a source package, it could be a USE flag or something.

Thanks for pointing to the concrete file we would need to take care :)

Well, the problem is that, even if we can use a USE flag for letting people to switch from one model to another, it wouldn't be too "fair" to allow that until we know how to fix the problems people could get when enabling user-session, then, in a short term, we will probably need to hard disable it :/ (even if we have other mechanisms for allowing people to pass their preferred configure flags and, then, allowing them to shot them in their foot ;)

> 
> > This causes problems when other users try to login in their accounts
> 
> If there are specific services that behave particularly poorly with multiple
> instances, I think this should be considered to be a bug in those services:
> they should "get out of the way" when their user does not have any surviving
> login-sessions. For instance, PulseAudio should release audio devices when
> it becomes inactive, if systemd does not already forcibly reclaim them.

Most of the users have complained about random issues with keyring not being accessible, I remember to also have suffered that issue myself until I figured our that this was the problem. Also I get issues with external devices needing to get root password to get mounted for some reason :/

A big problem I see about not killing dbus-daemon and the processes depending on it as that the list of leftover processes is a bit random depending on each setup and, then, it is really unpredictable to know how each application will behave when previous instance is still running or multiple instances from multiple users are running :/ Maybe we can start to dig into each app that fails, but I am nearly sure that we will see some random apps (some outside gnome world) that will probably behave wrongly and we won't notice the root cause until we don't figure out that maybe it's caused by a leftover process :|

(In reply to Simon McVittie from comment #4)
> (In reply to Simon McVittie from comment #3)
> > If there are specific services that behave particularly poorly with multiple
> > instances, I think this should be considered to be a bug in those services:
> > they should "get out of the way" when their user does not have any surviving
> > login-sessions.
> 
> I've opened a systemd feature request
> https://github.com/systemd/systemd/issues/2900 which would make this a lot
> easier.

Thanks, I will follow it... hopefully it will help me to learn more about this as it has always being a hell for me to understand it properly :)
Comment 6 Simon McVittie 2016-04-11 12:15:23 UTC
(In reply to Pacho Ramos from comment #5)
> I will check what is keeping systemd --user running then. In some setups it
> was deluged (that was kept running on purpose as it's meant to keep
> downloading things even if I logout... it worked nice with old dbus but now
> :/)

When you have a system in this state, please use systemd-cgls to list what is running within the context of a particular login session, and what is running outside login sessions; and use `loginctl` and `loginctl session-status` to list what logind thinks is going on. Please report this information here or on <https://github.com/systemd/systemd/issues/2900>. In particular, the State of each login session and the processes remaining in each login session are likely to be important.

It isn't clear from the original report precisely what "lots of processes" means, and systemd upstream would like to know what is going on.

Examples from a running (not-logged-out) Debian system where I am logged in via gdm and a tty:

% systemd-cgls
...
└─user.slice
  ├─user-1000.slice
  │ ├─user@1000.service                <- me, outside any particular session
  │ │ ├─gvfs-daemon.service
      ...
  │ │ ├─dbus.service
  │ │ │ ├─2857 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopi
        ...
  │ │ │ ├─4088 /usr/lib/gnome-terminal/gnome-terminal-server
  │ │ │ ├─6248 zsh
  │ │ │ ├─6535 systemd-cgls
  │ │ │ └─6536 pager
...
  │ ├─session-9.scope                  <- me, on the tty
  │ │ ├─6077 /bin/login --     
  │ │ └─6087 -zsh
  │ └─session-4.scope                  <- me, in X11 via gdm
  │   ├─2798 gdm-session-worker [pam/gdm-password]
  │   ├─2817 /usr/lib/gdm3/gdm-x-session --run-script default
  │   ├─2819 /usr/lib/xorg/Xorg vt2 -displayfd 3 -auth /run/user/1000/gdm/...
  │   ├─2831 /usr/lib/gnome-session/gnome-session-binary
      ...
  └─user-119.slice                     <- the system user for gdm
    ├─session-c1.scope
    │ ├─1383 gdm-session-worker [pam/gdm-launch-environment]
    │ ├─1441 /usr/lib/xorg/Xorg vt7 -displayfd 3 -auth /run/user/119/gdm/...
    │ ├─1575 /usr/lib/gnome-session/gnome-session-binary ...
      ...
    └─user@119.service
      ├─dbus.service
      │ ├─1610 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopi
      │ ├─1621 /usr/lib/at-spi2-core/at-spi-bus-launcher
      │ ├─1626 /usr/bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.con
      │ └─1629 /usr/lib/at-spi2-core/at-spi2-registryd --use-gnome-session
      └─init.scope
        ├─1426 /lib/systemd/systemd --user
        └─1429 (sd-pam)         

% loginctl                 
   SESSION        UID USER             SEAT
         4       1000 smcv             seat0
         9       1000 smcv             seat0
        c1        119 Debian-gdm       seat0
% loginctl session-status 4
4 - smcv (1000)
...
            Seat: seat0; vc2
             TTY: /dev/tty2
         Service: gdm-password; type x11; class user
           State: active
            Unit: session-4.scope
                  ├─2798 gdm-session-worker [pam/gdm-password]
                  ...
% loginctl session-status 9
9 - smcv (1000)
           Since: Mon 2016-04-11 13:02:08 BST; 6min ago
          Leader: 6077 (login)
            Seat: seat0; vc5
             TTY: /dev/tty5
         Service: login; type tty; class user
           State: online
            Unit: session-9.scope
                  ├─6077 /bin/login --     
                  └─6087 -zsh

> OK thanks, but, please, take care dbus upstream is enabling by default the
> user-session switch.

We aren't. It's off by default.

If your OS integrator is not behaving as you would prefer, please raise it with them or choose a different OS integrator.

> until we know how to fix the problems people could get when enabling
> user-session, then, in a short term, we will probably need to hard disable
> it

That's the OS integrator's decision. If that is what they choose to do for the next few months or even years, that's their call.
Comment 7 Pacho Ramos 2016-04-15 14:19:03 UTC
(In reply to Simon McVittie from comment #6)
> When you have a system in this state, please use systemd-cgls to list what
> is running within the context of a particular login session, and what is
> running outside login sessions; and use `loginctl` and `loginctl
> session-status` to list what logind thinks is going on. Please report this
> information here or on <https://github.com/systemd/systemd/issues/2900>. In
> particular, the State of each login session and the processes remaining in
> each login session are likely to be important.
> 
> It isn't clear from the original report precisely what "lots of processes"
> means, and systemd upstream would like to know what is going on.

Umm, sorry, I reported it downstream but not here. 

The remaining processes for the case of a session WITHOUT deluged (and that, without user-sessions, end with all processes dying at logout) are many like pulseaudio, gconf, gvfs... 

the output is here:
https://577416.bugs.gentoo.org/attachment.cgi?id=428490 

> We aren't. It's off by default.
> 
> If your OS integrator is not behaving as you would prefer, please raise it
> with them or choose a different OS integrator.

We fixed it a few days later sorry (it's a headache to have the information spread in multiple bug trackers :( )
Comment 8 Simon McVittie 2016-04-15 18:31:35 UTC
(In reply to Pacho Ramos from comment #7)
> The remaining processes for the case of a session WITHOUT deluged (and that,
> without user-sessions, end with all processes dying at logout) are many like
> pulseaudio, gconf, gvfs... 

That's part of the information I asked for, but not all of it.

When the system is in this state, what do `loginctl` and `loginctl session-status` say is going on? Use `loginctl` with no arguments to list sessions, then `loginctl session-status 123` for each of the relevant sessions (in the simplest case there will only be one) to list its status.

In what you posted on the Gentoo bugzilla, you had one session, and its session ID was 1:

  ├─user-1000.slice
  │ ├─session-1.scope
  │ │ ├─548 /usr/bin/gnome-keyring-daemon --daemonize --login
  │ │ ├─681 ibus-daemon --xim --panel disable
  │ │ ├─686 /usr/libexec/ibus-dconf
  │ │ ├─783 /usr/libexec/ibus-engine-simple
  │ │ ├─820 python2.7 /usr/bin/hp-systray -x
  │ │ └─821 python2.7 /usr/bin/hp-systray -x

I'm guessing it had the "closing" status, and would have gone away altogether if you killed those 6 processes.

gnome-keyring-daemon should ideally become a systemd user unit in its own right, like dbus and gvfs-daemon.

As far as I know, ibus and hp-systray are both graphical (X11) things, so I don't know why they didn't terminate when they lost their connection to X11...
Comment 9 Pacho Ramos 2016-04-18 08:41:06 UTC
Created attachment 123023 [details]
systemd-cgls output
Comment 10 Pacho Ramos 2016-04-18 08:44:55 UTC
All this outputs are taken just now after logout and seeing all the remaining processes still running. I hope it was the information you needed, if not, please don't hesitate to ask me :)

loginctl output:
   SESSION        UID USER             SEAT            
        c1        117 gdm              seat0           
         2          0 root             seat0           
         1       1000 pacho            seat0           

3 sessions listed.

(root is the console session I am running to get all this information and check the remaining processes, gdm is the one from the DM and "pacho" is the session I have logged out)

session-status:
2 - root (0)
           Since: lun 2016-04-18 10:36:54 CEST; 1min 20s ago
          Leader: 276 (login)
            Seat: seat0; vc1
             TTY: /dev/tty1
         Service: login; type tty; class user
           State: active
            Unit: session-2.scope
                  ├─ 276 /bin/login --     
                  ├─1295 /usr/bin/gnome-keyring-daemon --daemonize --login
                  ├─1297 -bash
                  └─1342 loginctl session-status

abr 18 10:36:54 optiplex760 login[1297]: ROOT LOGIN  on '/dev/tty1'

session-status 1:
1 - pacho (1000)
	   Since: lun 2016-04-18 10:34:22 CEST; 5min ago
	  Leader: 533
	    Seat: seat0; vc8
	     TTY: /dev/tty8
	 Service: gdm-password; type x11; class user
	   State: closing
	    Unit: session-1.scope
		  ├─689 ibus-daemon --xim --panel disable
		  ├─694 /usr/libexec/ibus-dconf
		  ├─799 python2.7 /usr/bin/hp-systray -x
		  ├─800 python2.7 /usr/bin/hp-systray -x
		  └─801 /usr/libexec/ibus-engine-simple

abr 18 10:36:47 optiplex760 /usr/libexec/gdm-x-session[555]: (II) UnloadModule: "evdev"
abr 18 10:36:47 optiplex760 /usr/libexec/gdm-x-session[555]: (II) systemd-logind: releasing fd for 13:66
abr 18 10:36:48 optiplex760 /usr/libexec/gdm-x-session[555]: (II) evdev: Power Button: Close
abr 18 10:36:48 optiplex760 /usr/libexec/gdm-x-session[555]: (II) UnloadModule: "evdev"
abr 18 10:36:48 optiplex760 /usr/libexec/gdm-x-session[555]: (II) systemd-logind: releasing fd for 13:64
abr 18 10:36:48 optiplex760 /usr/libexec/gdm-x-session[555]: (II) evdev: Power Button: Close
abr 18 10:36:48 optiplex760 /usr/libexec/gdm-x-session[555]: (II) UnloadModule: "evdev"
abr 18 10:36:48 optiplex760 /usr/libexec/gdm-x-session[555]: (II) systemd-logind: releasing fd for 13:65
abr 18 10:36:48 optiplex760 /usr/libexec/gdm-x-session[555]: (II) Server terminated successfully (0). Closing log file.
abr 18 10:36:48 optiplex760 gdm-password][533]: pam_unix(gdm-password:session): session closed for user pacho

(it shows as "closing" forever :/)

session-status 2:
2 - root (0)
	   Since: lun 2016-04-18 10:36:54 CEST; 1min 48s ago
	  Leader: 276 (login)
	    Seat: seat0; vc1
	     TTY: /dev/tty1
	 Service: login; type tty; class user
	   State: active
	    Unit: session-2.scope
		  ├─ 276 /bin/login --     
		  ├─1295 /usr/bin/gnome-keyring-daemon --daemonize --login
		  ├─1297 -bash
		  └─1348 loginctl session-status 2

abr 18 10:36:54 optiplex760 login[1297]: ROOT LOGIN  on '/dev/tty1'

session-status c1:
c1 - gdm (117)
	   Since: lun 2016-04-18 10:34:06 CEST; 4min 26s ago
	  Leader: 308 (gdm-session-wor)
	    Seat: seat0; vc7
	     TTY: /dev/tty7
	 Service: gdm-launch-environment; type x11; class greeter
	   State: online
	    Unit: session-c1.scope
		  ├─308 gdm-session-worker [pam/gdm-launch-environment]
		  ├─342 /usr/libexec/gdm-x-session /usr/bin/gnome-session --autostart /usr/share/gdm/greeter/autostart
		  ├─344 /usr/bin/Xorg vt7 -displayfd 3 -auth /run/user/117/gdm/Xauthority -nolisten tcp -background none -noreset -keeptty -verbose 3
		  ├─359 /usr/libexec/gnome-session-binary --autostart /usr/share/gdm/greeter/autostart
		  ├─385 /usr/libexec/gnome-settings-daemon
		  ├─435 gnome-shell --mode=gdm
		  ├─440 /usr/bin/pulseaudio --start --log-target=syslog
		  ├─450 /usr/libexec/pulse/gconf-helper
		  ├─476 ibus-daemon --xim --panel disable
		  ├─481 /usr/libexec/ibus-dconf
		  ├─485 /usr/libexec/ibus-x11 --kill-daemon
		  └─528 /usr/libexec/ibus-engine-simple

abr 18 10:36:48 optiplex760 /usr/libexec/gdm-x-session[342]: Errors from xkbcomp are not fatal to the X server
abr 18 10:36:49 optiplex760 /usr/libexec/gdm-x-session[342]: (II) AIGLX: Suspending AIGLX clients for VT switch
abr 18 10:36:49 optiplex760 /usr/libexec/gdm-x-session[342]: (II) systemd-logind: got pause for 226:0
abr 18 10:36:49 optiplex760 /usr/libexec/gdm-x-session[342]: (II) systemd-logind: got pause for 13:65
abr 18 10:36:49 optiplex760 /usr/libexec/gdm-x-session[342]: (II) systemd-logind: got pause for 13:67
abr 18 10:36:49 optiplex760 /usr/libexec/gdm-x-session[342]: (II) systemd-logind: got pause for 13:68
abr 18 10:36:49 optiplex760 /usr/libexec/gdm-x-session[342]: (II) systemd-logind: got pause for 13:64
abr 18 10:36:49 optiplex760 /usr/libexec/gdm-x-session[342]: (II) systemd-logind: got pause for 13:66
abr 18 10:36:49 optiplex760 gnome-session[359]: (gnome-shell:435): Clutter-CRITICAL **: clutter_input_device_get_device_id: assertion 'CLUTTER_IS_INPUT_DEVICE (device)' failed
abr 18 10:36:49 optiplex760 gnome-session[359]: (gnome-shell:435): Clutter-CRITICAL **: clutter_input_device_get_device_id: assertion 'CLUTTER_IS_INPUT_DEVICE (device)' failed


Thanks :)
Comment 11 Simon McVittie 2016-04-18 11:36:18 UTC
(In reply to Pacho Ramos from comment #10)
> 	   State: closing
> 	    Unit: session-1.scope
> 		  ├─689 ibus-daemon --xim --panel disable
> 		  ├─694 /usr/libexec/ibus-dconf
> 		  ├─799 python2.7 /usr/bin/hp-systray -x
> 		  ├─800 python2.7 /usr/bin/hp-systray -x
> 		  └─801 /usr/libexec/ibus-engine-simple

OK, I think this confirms my guesses: these three ibus-related processes and two hp-systray processes are what is holding your session open.

You could confirm this further as follows:

* Get your system back into this state
* Log in as root on a text-mode console or via ssh
* Do not be logged in to this system with uid 1000 "pacho", other than
  the session that is in closing state (no text-mode logins, no ssh, no
  new X11/Wayland/Mir sessions)
* As root, kill those processes one at a time
* As root, after killing each process, look at the session state again

I believe you will find that when the last of those processes is killed, the login-session terminates; and when the login-session terminates, everything else running as uid 1000 "pacho" also terminates, including "dbus-daemon --session" and "systemd --user".

Presumably these processes were previously relying on being terminated by getting disconnected from the session bus, but now there's a circular dependency: they won't terminate until the session bus exits, and the session bus won't exit until they terminate.

I see several ways this could be resolved:

* You (or systemd upstream, or your distro) could use KillUserProcesses.
  This would break the deadlock by terminating those processes early.
* These processes could connect to the X11 display, and exit when that
  connection is broken. This would break the deadlock by terminating those
  processes early.
* These processes could be activated D-Bus services, which would appear
  under dbus.service in systemd-cgls, and would not prevent the session
  from ending. This would break the deadlock by having those processes
  not prevent the login-session from ending.
* These processes could be systemd user services (possibly also activated
  D-Bus services), which would appear in their own cgroups (like gvfs
  does). This would break the deadlock by having those processes not
  prevent the login-session from ending.
Comment 12 Pacho Ramos 2016-04-18 12:40:38 UTC
(In reply to Simon McVittie from comment #11)
> (In reply to Pacho Ramos from comment #10)
> > 	   State: closing
> > 	    Unit: session-1.scope
> > 		  ├─689 ibus-daemon --xim --panel disable
> > 		  ├─694 /usr/libexec/ibus-dconf
> > 		  ├─799 python2.7 /usr/bin/hp-systray -x
> > 		  ├─800 python2.7 /usr/bin/hp-systray -x
> > 		  └─801 /usr/libexec/ibus-engine-simple
> 
> OK, I think this confirms my guesses: these three ibus-related processes and
> two hp-systray processes are what is holding your session open.

Indeed, killing ibus-daemon and hp-systray makes the session to finally exit and get all the other processes killed. But I have a few doubts:
- Why are processes under session-1.scope causing all the other unrelated processes to be kept alive?
- The list of processes under session-1.scope changes a bit depending on the time. For example, when I have retried just now, I noticed that gnome-keyring processes is still running there under session-1.scope for some seconds (well, probably more than 1 minute). That is probably causing other issues people were hitting in my maintained machines of keyring being locked when users where switching or reloggining in again "instantly", just after loggining out
- The list of processes prone to behave like ibus-daemon and hp-systray (from hplip) is a bit unpredictable. For example, in other machine it was deluged daemon that, as it is expected because the way deluged is designed to work, it keeps running when it is launched to keep downloading/uploading things. The same will probably apply to many other apps designed as daemons that, once they are launched, will cause the logout process to behave in this ugly way :/

[...]
> I see several ways this could be resolved:
> 
> * You (or systemd upstream, or your distro) could use KillUserProcesses.
>   This would break the deadlock by terminating those processes early.

Well, that is the workaround I was applying until I discovered that disabling user-sessions were solving this. The problem of this approach is that it does not only kill apps like screen, ever some users complained because it was also killing the processes they were running with nohup from a terminal :O (at least the case they experienced: they loggining in as normal user, open gnome-terminal and becoming root with "su -", launching a process with nohup.
Comment 13 Simon McVittie 2016-04-18 13:43:04 UTC
(In reply to Pacho Ramos from comment #12)
> - Why are processes under session-1.scope causing all the other unrelated
> processes to be kept alive?

Because your login-session isn't over yet, and until your login-session is over, killing your background services could (and probably will) break processes in the login-session.

systemd can't know the difference between hp-systray (presumably unimportant to you) and screen/tmux (which would presumably be important to you), unless you give it that information somehow; at the moment, there is no difference between them other than your expectations.

If you want the intepretation of "is my login-session over?" to change to just "is the session leader (gnome-session) still running?", i.e. making systemd-logind aggressively terminate sessions in the "closing" state, that's exactly what KillUserProcesses=true does.

> - The list of processes under session-1.scope changes a bit depending on the
> time. For example, when I have retried just now, I noticed that
> gnome-keyring processes is still running there under session-1.scope for
> some seconds

The list of processes under session-1.scope is "every child or descendant of session 1's gnome-session". Exactly what that means is dependent on what you did during the session - more or fewer processes will have been started on-demand, depending on your activity.

> - The list of processes prone to behave like ibus-daemon and hp-systray
> (from hplip) is a bit unpredictable. For example, in other machine it was
> deluged daemon that, as it is expected because the way deluged is designed
> to work, it keeps running when it is launched to keep downloading/uploading
> things. The same will probably apply to many other apps designed as daemons
> that, once they are launched, will cause the logout process to behave in
> this ugly way :/

systemd does not have any way to tell the difference between "this part of the login-session is still running but it shouldn't be" and "this part of the login-session is still running and that's by design".

KillUserProcesses=true makes it assume that everything still running after gnome-session exits is a mistake and should be killed.

KillUserProcesses=false makes it assume that everything still running after gnome-session exits is deliberate and should keep running.

What machine-readable fact do you think systemd should use to distinguish between intentionally and unintentionally surviving processes? It can't read your mind, unfortunately.

As long as an "intentional daemon" like deluged is running, the only conservative assumption is that it might be relying on per-user background services (gvfs, dconf, dbus-daemon), so they are not terminated either.

> Well, that is the workaround I was applying until I discovered that
> disabling user-sessions were solving this. The problem of this approach is
> that it does not only kill apps like screen, ever some users complained
> because it was also killing the processes they were running with nohup from
> a terminal :O (at least the case they experienced: they loggining in as
> normal user, open gnome-terminal and becoming root with "su -", launching a
> process with nohup.

Yes, a process run with nohup from a terminal is in exactly the category of: this was started within a login-session, and yet you want it to persist after the login-session ends, even though you may want other members of the same login-session to be terminated.

Please prefer systemd-run(1) over the combination of "su -" and nohup: this is what it's for. That command runs the started service outside any particular session, making it immune to KillUserProcesses (more precisely, it was never a user process in the first place).

The reason that X11-autolaunched D-Bus did approximately what you expected is that in the "D-Bus session per login-session" view of the world, the bus is terminated with the X11 server (even if the same uid has other parallel sessions!). It happens to have historically been true that most (but not all) background processes that connected to D-Bus  were in the category that should be terminated, and most (but not all) background processes that did not connect to D-Bus were in the category that should not.

The purpose of --enable-user-session is exactly that it makes D-Bus available to non-session-specific processes, so terminating the dbus-daemon on logout would be counterproductive; that would be no better than if we had never implemented the option.

In Debian, the units that make up the difference between --disable-user-session and --enable-user-session are packaged separately, in a dbus-user-session binary package; if that package is not installed you effectively get --disable-user-session, and if it is installed, you effectively get --enable-user-session. If you believe your distribution should offer this choice, please take that up with your distribution; in Gentoo I suspect that would be implemented as USE="dbus-user-session" or something. Don't necessarily be surprised if they are unwilling to support both modes, though.

Alternatively, "systemctl --user mask dbus.socket" (as a user) or "systemctl --user --global mask dbus.socket" (as root) would disable it per-user or per-machine.
Comment 14 Pacho Ramos 2016-04-18 14:24:15 UTC
Ah, no problem about how to solve it in Gentoo, we are already allowing it to be configurable for some days :)

But I was wondering about how could we solve this issues to not need to always rely on non-user-session approach :/ I will probably try to report at least to ibus upstream (and I will need to check how to report to hplip people). But I am still really unsure about we being able to solve this ever. I am sure many more random apps will need to be fixed to die themselves when X die... and we will only notice them when people gets all the drawbacks of finding a lot of leftover processes in their systems :S

On the other hand others like deluged, are done on purpose to allow them to run forever without needing X as daemons... ever if they were launched from a gnome-session, then, I wonder what their upstreams could do :(

That is the reason I was trying to understand how all the processes are being launched on a login in sequence, how some processes are falling under session-X.scope, others under dbus.service (and those depending later on session-X.scope to be cleaned before exiting) and other have their own .service (like gvfs)

My idea was that, maybe , if I was able to know what app is launching dbus.service when login in a graphical session, maybe it could take care of killing it at logout time as it would know that that dbus.service was launched by him and, hence, killing it at logout should be save enough.

We can live without user-sessions currently sure... but, if at some point you decide to hard enable it, we will be lost :S
Comment 15 Simon McVittie 2016-04-18 17:14:46 UTC
(In reply to Pacho Ramos from comment #14)
> That is the reason I was trying to understand how all the processes are
> being launched on a login in sequence, how some processes are falling under
> session-X.scope, others under dbus.service (and those depending later on
> session-X.scope to be cleaned before exiting) and other have their own
> .service (like gvfs)

I'm assuming GNOME here, but the same basic structure applies regardless.

The login manager (gdm or whatever) makes a subprocess representing the login session. That subprocess logs in, using PAM modules. One of those is pam_systemd, which tells systemd-logind about the new login-session. systemd-logind creates a session (let's say 123) and tells systemd (pid 1) to put the subprocess in session-123.scope - it is the first process in the new session.

If this is the first login session, systemd starts user@1000.service, which is systemd --user. That service runs any user-services that are configured to start in default.target or its dependencies, notably including dbus.socket if enabled.

After login, the login manager subprocess execs gnome-session (or its equivalent in a non-GNOME desktop, like startkde) - same pid, different executable. It is still in session-123.scope and cannot leave. When it runs child processes - via /etc/xdg/autostart, or by executing desktop-environment components like GNOME Shell - those processes are also in session-123.

If you are *not* using --enable-user-session, gnome-session starts dbus-launch, which starts dbus-daemon, as child processes of session 123. If you are not using GNOME either, maybe nothing starts dbus-launch until something eventually uses the session bus, at which point X11 autolaunching kicks in, and the new dbus-launch and dbus-daemon will still be child processes of session 123.

When GNOME Shell runs child processes (Alt+F2) they are *also* in session 123.

If you *are* using --enable-user-session, eventually something probably connects to the session bus. systemd --user sees the new connection to the socket that it created for dbus.socket, and starts the corresponding user service, dbus.service. dbus.service is *not* part of session 123.

Whichever code path you are on, you now have a session bus. Whenever something communicates with an activatable D-Bus service on the session bus, dbus-daemon arranges for that service to be started. There are three code paths:

* If you didn't --enable-user-session, dbus-daemon forks and execs the
  service directly. It is a child of dbus-daemon, so it's in the same
  cgroup, which in this case is session-123.scope.

* If you did --enable-user-session, but the service does not have
  SystemdService=whatever in its .service file (like the dconf-service
  in your systemd-cgls output), dbus-daemon still forks and execs the
  service directly. It is a child of dbus-daemon, so it's in the same
  cgroup, which in this case is dbus.service.

* If you did --enable-user-session *and* the service has
  SystemdService=whatever in its .service file, dbus-daemon asks
  systemd --user to start it. systemd puts the service in its own
  cgroup, like the various gvfs processes in your systemd-cgls.

During logout, gnome-session exits. This triggers a monitoring process started by pam_systemd, which tells systemd-logind that the session is closing. The session's state moves to "closing". If systemd was configured to KillUserProcesses, it kills everything that is still running in session-123.

Meanwhile, when gnome-session exits, the X server also exits, which disconnects all its clients. If you still have Firefox open, for instance, this is what causes it to exit.

If you didn't --enable-user-session, one of the X server's clients was dbus-launch, which notices that its X connection has gone away, and responds by terminating dbus-daemon. Go to [1], below.

If you did --enable-user-session, time passes. Eventually, everything in session-123 has exited, or been killed by systemd, or been killed manually by you (in Comment #12). Now the session-123.scope cgroup is empty, and systemd cleans it up. At the transition from 1 to 0 login sessions, systemd (pid 1) terminates user@1000.service (systemd --user), which in turn terminates all its services, notably including dbus.service (dbus-daemon).

[1]

When dbus-daemon is terminated, all its clients get disconnected. They normally respond to this by exiting.

If everything has gone well, you don't have any more processes by now.

> My idea was that, maybe , if I was able to know what app is launching
> dbus.service when login in a graphical session, maybe it could take care of
> killing it at logout time as it would know that that dbus.service was
> launched by him and, hence, killing it at logout should be save enough.

It is systemd that launches dbus.service. It *already does* kill dbus.service when appropriate. However, the reason systemd does not kill dbus-service immediately at X11 logout is that while your session is in "closing" state, *it has not really exited* - it still has processes that could be using dbus-daemon.

It is also not true that "started at session 123 login" should always imply "killed at session 123 logout"! Consider this session:

13:00: log in to X11, session-123.scope
13:30: log in via getty (text mode), session-c124.scope
14:00: log in via ssh, session-c125.scope
15:00: cron job starts, session-c126.scope
15:10: cron job finishes, session-c126.scope
15:30: log out of X11, session-123.scope
16:00: log out from getty, session-c124.scope
16:30: log out from ssh, session-c125.scope

There are up to 4 login-sessions here, depending whether you count the cron job. I hope you agree that terminating per-user services like gvfs and dconf would be wrong at 15:10, would be wrong at 15:30, would *still* be wrong at 16:00, but would be OK at 16:30.

Please understand that the issue here is not "the component that starts dbus.service should know how to kill it when appropriate". *It already does that*. The issue here is that what you think "when appropriate" means is not the same as what systemd's authors think it means. I happen to think they're correct here.

> We can live without user-sessions currently sure... but, if at some point
> you decide to hard enable it, we will be lost :S

I don't intend to make user-session mode mandatory. Like I said in the initial email about this stuff, it's a question for OS builders, not D-Bus authors. If you're a Gentoo maintainer, it's your decision (where "refuse to make a decision and require your users to make it instead" is itself a decision - unfortunately that's the one we usually get in Debian).

Other upstreams (GNOME? dconf?) might want to require it, and declare that if you disable it, they won't support the resulting system; that's their decision, not mine.

I'm more likely to delete dbus-launch than I am to make user-session mode mandatory, and the number of people who would come after me with pitchforks if I did that any time in the next few years is likely to be quite substantial :-)
Comment 16 Pacho Ramos 2016-04-19 08:16:48 UTC
OK, thanks a lot for the explanation :)

For now we are in Gentoo allowing people to optionally enable user-sessions... but printing a warning about this possible issues they could get. I will probably report to ibus and hplip maintainers about they trying to make their processes die when X exits instead of relying on dbus-daemon dying. Regarding how other processes under dbus.service not dying when X exits, maybe the situation is different but, well, I guess the relevant upstream will tell and teach me (for example gnome-shell-calendar-server being still running without X even when the full gnome-shell has already exited)

Best regards!
Comment 17 Simon McVittie 2016-04-19 11:15:06 UTC
(In reply to Pacho Ramos from comment #16)
> I will probably report to ibus and hplip maintainers about they
> trying to make their processes die when X exits instead of relying on
> dbus-daemon dying. Regarding how other processes under dbus.service not
> dying when X exits, maybe the situation is different but, well, I guess the
> relevant upstream will tell and teach me (for example
> gnome-shell-calendar-server being still running without X even when the full
> gnome-shell has already exited)

The "big picture" for how I see this working in 5 years, at least on systems with systemd and --enable-user-session, goes something like this:

* Per-login-session processes (like gnome-shell) may remain in
  session-123.scope. If they do, they exit on logout from *their* login
  session, even if there are other concurrent logins (either because they
  are disconnected from X11/Wayland/Mir, or they exit voluntarily, or
  systemd --user kills them via KillUserProcesses).

* Per-user services (dbus-daemon, gvfs, dconf, mpd...) are under
  user@1000.service; many of them are also D-Bus-activatable. They are
  terminated by systemd --user when there are no more login sessions
  using them.

* "Applications" may be in either of those categories (traditionally
  they've been the first category, but increasingly they're moving
  to the second). Either way, they should normally exit when
  disconnected from X11/Wayland/Mir.

* Per-user background processes (cron jobs, screen, tmux) should probably
  communicate with PAM/systemd-logind to arrange to be treated as not
  part of any particular session, and keep user@1000.service alive as
  long as they are running.

gnome-shell-calendar-server seems like it could potentially fall into the same category as dconf - it's a non-GUI service used by gnome-shell.
Comment 18 Colin Walters 2016-05-30 15:25:22 UTC
I think I mentioned this elsewhere, but my strawman proposal on this was that processes have to opt-in somehow to get the user bus, and by default on logout of all sessions, dbus closes the connections of processes which didn't opt in.

Having process lifecycle tied to dbus was an intentional point of its design.

I think even though systemd now defaults to KillUserProcesses, this design would be worth implementing to avoid bugs for the sites/users which haven't upgraded or disable it.
Comment 19 Simon McVittie 2016-05-30 16:55:04 UTC
(In reply to Colin Walters from comment #18)
> I think I mentioned this elsewhere, but my strawman proposal on this was
> that processes have to opt-in somehow to get the user bus, and by default on
> logout of all sessions, dbus closes the connections of processes which
> didn't opt in.

I still think having a user bus in addition to a session bus (with distinct bus-name namespaces, no ordering guarantee, etc.) would have been an interoperability disaster, and I'm glad we didn't do it.

If you want to propose a concrete mechanism by which (user|session)-bus services can declare that they intentionally live longer than the union of all login sessions, together with support for having dbus-daemon monitor the login sessions and disconnect everything that didn't opt-in to that mechanism, go ahead. I don't think 1.10 should do that - we've had several 1.10 releases with broadly compatible behaviour, after all - but it could be in 1.11.
Comment 20 Colin Walters 2016-06-09 16:44:09 UTC
(In reply to Simon McVittie from comment #19)

> If you want to propose a concrete mechanism by which (user|session)-bus
> services can declare that they intentionally live longer than the union of
> all login sessions, together with support for having dbus-daemon monitor the
> login sessions and disconnect everything that didn't opt-in to that
> mechanism, go ahead.

I think the API would be something like g_bus_get_sync_with_flags (G_BUS_TYPE_SESSION, G_BUS_LIFECYCLE_USER, cancellable, error);

Under the hood, during the connection we'd send a NEGOTIATE_LIFECYCLE_USER.

Then yes, there'd have to be some tight integration with logind.

The high level is there is still one bus, but applications still need to request a way to live beyond the session.

To be clear I'm not going to work on this, but if someone did it seems like it's a viable path forward, right?
Comment 21 GitLab Migration User 2018-10-12 21:26:50 UTC
-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/dbus/dbus/issues/147.


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.