When max_incomplete_connections is reached, new connections fail and we disconnect the oldest unauthenticated connection. The code contains a "FIXME" explaining the risk with a suggestion of a better solution. We should investigate if we can do better than this.
> /* And we might also disconnect ourselves here, but again it
> * only takes effect on return to main loop.
> if (connections->n_incomplete >
> bus_context_get_max_incomplete_connections (connections->context))
> _dbus_verbose ("Number of incomplete connections exceeds max, dropping oldest one\n");
> _dbus_assert (connections->incomplete != NULL);
> /* Disconnect the oldest unauthenticated connection. FIXME
> * would it be more secure to drop a *random* connection? This
> * algorithm seems to mean that if someone can create new
> * connections quickly enough, they can keep anyone else from
> * completing authentication. But random may or may not really
> * help with that, a more elaborate solution might be required.
> dbus_connection_close (connections->incomplete->data);
Note: max_incomplete_connections is 64 by default (the system bus uses the default) and 10000 on the session bus.
When Alban investigated this he realized that there was a denial of service vulnerability, Bug #80919.
That bug was fixed in a different way in 1.8.8, and the algorithm suggested in the comment turned out not to help, so this is WONTFIX.