Bug 29069 - polkitd memory leaks
polkitd memory leaks
Status: RESOLVED FIXED
Product: PolicyKit
Classification: Unclassified
Component: daemon
unspecified
x86-64 (AMD64) Linux (All)
: medium normal
Assigned To: David Zeuthen (not reading bugmail)
David Zeuthen (not reading bugmail)
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-07-14 21:13 UTC by Petr Mrázek
Modified: 2010-08-20 11:29 UTC (History)
0 users

See Also:
i915 platform:
i915 features:


Attachments
polkitd DoS utility (6.24 KB, patch)
2010-07-14 21:13 UTC, Petr Mrázek
Details | Splinter Review
Backtraces from stalled polkitd and pkdos (5.37 KB, text/plain)
2010-07-15 19:45 UTC, Petr Mrázek
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Petr Mrázek 2010-07-14 21:13:23 UTC
Created attachment 37058 [details] [review]
polkitd DoS utility

I first noticed the problem when I tried to integrate KAuth (KDE's new authorization framework, uses polkit indirectly through polkit-qt) into Kiosk (KDE's old authorization framework). The integration was simple, but the synchronous queries for authorization eventually stopped returning.

polkitd is the problem here -- it locks up and consumes a lot of resources while doing that. Kiosk is used heavily all over KDE and it looks like the volume of requests is way above what polkit was tested with. This leads to KDE being completely unresponsive, because everything is waiting on polkitd.

I added a simple test program to the polkit sources called 'pkdos'. This simply repeats the same query for a non-existent action. Run a few instances and wait a minute or two -- polkitd ends up consuming over 600MB of private memory and hogging the CPU. No further requests from any user are processed after that, even if the pkdos instances are terminated.

The enclosed patch was made on top of the previous one:
https://bugs.freedesktop.org/show_bug.cgi?id=29051
Comment 1 David Zeuthen (not reading bugmail) 2010-07-15 06:34:32 UTC
FWIW, I can only make the daemon leak - it stops using CPU as soon as I kill pkdos. top(1) looks approx like this when it's running

 1297 root      20   0  320m 151m 1508 S 51.1  7.8   0:48.38 polkitd            
  901 dbus      20   0 22712 1788  596 S 35.9  0.1   0:50.46 dbus-daemon        
29682 davidz    20   0 51996  15m 2068 S 19.4  0.8   0:05.38 pkdos

and even while I'm doing all this polkit works just fine - e.g. I can run 'pkexec bash' just fine from another terminal. Sure, we should fix the leak, definitely, but apart from that I don't see anything weird here - I mean, this is not different from other kinds of attacks (e.g. fork-bomb).

Any if you are concerned that your user can make polkitd consume a lot of CPU, then the way to deal with that is to play with <limit> directives for the system message bus (see the dbus-daemon(1) man page).

Maybe the problem with polkitd keeping consuming CPU after you killed pkdos(1) is that you are doing this on a low-memory box and it ends you up in swap-hell? Any chance you can attach to polkitd with strace(1) to figure out what's going on. Thanks.
Comment 2 David Zeuthen (not reading bugmail) 2010-07-15 06:37:38 UTC
(In reply to comment #0)
> polkitd is the problem here -- it locks up and consumes a lot of resources
> while doing that. Kiosk is used heavily all over KDE and it looks like the
> volume of requests is way above what polkit was tested with. This leads to KDE
> being completely unresponsive, because everything is waiting on polkitd.

Btw, if KDE keeps asking polkit questions then it's buggy and needs to be fixed - e.g. use the Changed() signal

 http://hal.freedesktop.org/docs/polkit/eggdbus-interface-org.freedesktop.PolicyKit1.Authority.html#eggdbus-signal-org.freedesktop.PolicyKit1.Authority::Changed

and caching.
Comment 3 David Zeuthen (not reading bugmail) 2010-07-15 06:39:06 UTC
Readjusting title - at least until we've established there's a denial of service going on here.
Comment 4 Petr Mrázek 2010-07-15 19:45:44 UTC
Created attachment 37097 [details]
Backtraces from stalled polkitd and pkdos

I got some backtraces from the stalled processes. Seems like the more pkdos instances I start, the faster polkitd stops responding. I ran four this time. 

Using polkit through KAuth leads to many processes making many requests all at the same time. The session won't even start properly.
Comment 5 Petr Mrázek 2010-07-15 20:19:56 UTC
(In reply to comment #1)
> and even while I'm doing all this polkit works just fine - e.g. I can run
> 'pkexec bash' just fine from another terminal. Sure, we should fix the leak,
> definitely, but apart from that I don't see anything weird here - I mean, this
> is not different from other kinds of attacks (e.g. fork-bomb).
Well, not different, but I can trigger this with just a bunch of KDE apps that work fine with Kiosk.

> Maybe the problem with polkitd keeping consuming CPU after you killed pkdos(1)
> is that you are doing this on a low-memory box and it ends you up in swap-hell?
8GB of ram, mostly empty. I also don't have any swap.

> Any chance you can attach to polkitd with strace(1) to figure out what's going
> on. Thanks.
Tried it with logging strace output into a file... 650MB big.
After around 7 million lines, it contains only things like this:
poll([{fd=5, events=POLLIN}], 1, 25000) = 1 ([{fd=5, revents=POLLIN}])
Fits the backtrace.

(In reply to comment #2)
>Btw, if KDE keeps asking polkit questions then it's buggy and needs to be fixed
> - e.g. use the Changed() signal and caching.
Sounds like a good idea. I worked with KAuth mostly and I'm still working my way down the stack of libraries between my code and polkit... if anywhere, this should be in polkit-qt.
Comment 6 David Zeuthen (not reading bugmail) 2010-08-03 08:19:31 UTC
FWIW, these leaks are now fixed in the gdbus branch of polkit

http://cgit.freedesktop.org/PolicyKit/log/?h=gdbus

You will need glib master especially this patch

http://git.gnome.org/browse/glib/commit/?id=3940cc9a11fde063bb9f83cc362e575e5f378609

otherwise pkdos.c will terminate very quickly.

I'm going to close this bug when the gdbus branch has been merged to master. Thanks.
Comment 7 Petr Mrázek 2010-08-03 19:24:55 UTC
Yes. With the gdbus polkit and a build of glib from git master, the daemon seems to be stable and not leaking. This is good news indeed :)
Comment 8 David Zeuthen (not reading bugmail) 2010-08-20 11:29:46 UTC
Seems to be resolved in master and the released 0.98 version. I can't make the daemon leak anymore. Closing.