Bug 54716

Created attachment 69267 [details] [review]
[PATCH] Combine copying and  merging of flags to avoid repeated traversals


When combining the flags from all the packages together, each flags list
was being copied and then concatenated to then end of the combined list.
This was extremely inefficient because it caused the combined list to be
traversed multiple times to find the end. Instead, combine the copying
and merging of the flags together so the last element is always tracked
and can easily be appended to.

Freedesktop #54716 (https://bugs.freedesktop.org/show_bug.cgi?id=54716)
---
 pkg.c |   38 +++++++++++++++++++++++---------------
 1 files changed, 23 insertions(+), 15 deletions(-)

After getting all the pc files together to actually get pkg-config to resolve, I found one spot that was definitely inefficient. I have to say that this is the most extreme usage of pkg-config I've come across. pkg-config uses a lot of single-linked lists, which is no big deal when the lists of packages and arguments are relatively small. When the lists become large like this, it becomes very painful to find the list endpoint.

The attached patch gets rid of the worst bottleneck. It got me from not being able to finish your test in 1 minute to about 1 second of runtime. That's still pretty bad, but obviously an improvement from useless to usable. Can you give it a try?

It is much better. Still about 2x slower than pykg-config when running the test case but good enough to be usable when there's a large amount of .pc files.

I'm unable to run pkg-config on my project at the moment due to an incompatibility between pkg-config and pykg-config in handling --variable switch.

Created attachment 69303 [details]
.pc files and test script showing problem

.pc files with all external dependencies copied in to demonstrate slowness. Unpack, cd to directory, and run ./test.sh. pkg-config is expected to be in the parent directory, but can be set via PKG_CONFIG environment variable, too.

(In reply to comment #3)
> It is much better. Still about 2x slower than pykg-config when running the
> test case but good enough to be usable when there's a large amount of .pc
> files.

Yeah, I suspect we're still losing a lot of time traversing these large single-linked lists in your case. A project for another day is to trade size for speed and convert all the GSLists to GLists and make them double-linked. Then again, I just need to attach a proper profiler and see where the hot spots are.

> I'm unable to run pkg-config on my project at the moment due to an
> incompatibility between pkg-config and pykg-config in handling --variable
> switch.

What's the issue there? Could you open a separate bug about this if there isn't one already?

This bug should be fixed in commit 90e0ec0 after minor refactoring and rewording of the commit message.

(In reply to comment #5)
> > I'm unable to run pkg-config on my project at the moment due to an
> > incompatibility between pkg-config and pykg-config in handling --variable
> > switch.
> 
> What's the issue there? Could you open a separate bug about this if there
> isn't one already?

I'm not sure whether it's a bug or a feature.

pkg-config will print variables defined only directly in the queried package's .pc file while pykg-config will print them also if they're defined in .pc files listed in "Requires" section. I need the recursive behaviour of pykg-config.