Bug 21937 - Tests shouldn't take 5 minutes to run
Summary: Tests shouldn't take 5 minutes to run
Status: NEW
Alias: None
Product: Telepathy
Classification: Unclassified
Component: gabble (show other bugs)
Version: unspecified
Hardware: Other All
: medium enhancement
Assignee: Telepathy bugs list
QA Contact: Telepathy bugs list
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-05-26 04:27 UTC by Will Thompson
Modified: 2011-06-03 05:01 UTC (History)
1 user (show)

See Also:
i915 platform:
i915 features:


Attachments
DO NOT MERGE: cProfile for accept-private-stream-tube.py (1.08 KB, patch)
2011-06-03 05:01 UTC, Will Thompson
Details | Splinter Review
DO NOT MERGE: disable introspection for accept-private-stream-tube.py (3.94 KB, patch)
2011-06-03 05:01 UTC, Will Thompson
Details | Splinter Review
cProfile output for accept-private-stream-tube.py (202.10 KB, application/octet-stream)
2011-06-03 05:01 UTC, Will Thompson
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Will Thompson 2009-05-26 04:27:37 UTC
Gabble's test suite now takes 5 minutes to run.

This puts at least myself and Guillaume off running the whole test suite; I tend only to run the bit that tests what I'm editing. Occasionally, this means we merge code with failing tests.

The bulk of the time is from running all the tests that use bytestreams against 8 different bytestream implementations/scenarios. I suspect we could make them faster, if we looked into why they're so slow.
Comment 1 Dafydd Harries 2009-08-20 03:28:22 UTC
Ideas:

 - Eliminate reactor.iterate() call, either by switching to a new API, or using some cunning producer/consumer system. I have a branch working on the former (test-api-3); I tried the latter once, but had problems with serializing events.
 - Profile with gprof/oprofile.
 - Run tests in parallel. Each test definitely needs a connection to itself, and maybe a CM to itself (which would mean a bus to itself).

I thought maybe that Python startup time (including imports etc.) was a bottleneck, but I wrote a test runner that in theory made all startup costs a one-off cost and it only shaved off ~30s.

What's our goal, assuming the current number of tests? One minute? Two?
Comment 2 Will Thompson 2011-06-02 09:29:27 UTC
I did a bit of profiling recently. I shaved a few seconds off by disabling introspection for the ConnectionManager objects constructed by the test suite, and a few more by reducing the debug output by 10%. I think turning off automatic introspection across the board would make a significant difference—cProfile confirmed it, but I've lost the logs—but unfortunately most of the tests depend on it. dbus-python doesn't make it very easy to shove in known types without subclassing dbus.Interface() all over the shop. We can do that but it's not a five-minute thing.
Comment 3 Simon McVittie 2011-06-02 09:44:11 UTC
If you don't mind depending on the latest version of dbus-python, putting signature='blah' on all method calls should work now (I fixed Bug #36206 in 0.84).
Comment 4 Will Thompson 2011-06-03 05:00:42 UTC
Hmm. Rather embarrassingly it turns out that the vast, vast majority of the time spent parsing XML is not spent parsing introspection XML. It is spent parsing XMPP XML.

Patches I used to profile one of the Tubes tests follow, along with a cProfile output file. They do indeed show spinning the reactor to be extremely costly: but I'm not sure how much of that is just busy-looping in time that would be used anyway. Profiling Gabble itself would almost certainly yield interesting facts.
Comment 5 Will Thompson 2011-06-03 05:01:01 UTC
Created attachment 47471 [details] [review]
DO NOT MERGE: cProfile for accept-private-stream-tube.py
Comment 6 Will Thompson 2011-06-03 05:01:06 UTC
Created attachment 47472 [details] [review]
DO NOT MERGE: disable introspection for accept-private-stream-tube.py
Comment 7 Will Thompson 2011-06-03 05:01:36 UTC
Created attachment 47473 [details]
cProfile output for accept-private-stream-tube.py


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.