Bug 105572

This issue is more subtle than it first appears, so I'm going to try to summarize how we got here.

(In reply to Xavier Claessens from comment #0)
> If I understand correctly the reason is to support case where libfoo exposes
> libbar in its API but does not need to link to it. So there are 2 valid use
> cases but only one of them is possible to express.

As discussed in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=340904 back in 2006, there are three valid use cases, but the combination of Requires and Requires.private can only ever express two of them. I'll refer to the use cases, in order from least-private to most-private, as *critical dependencies*, *visible dependencies* and *invisible dependencies*.

Critical dependencies: (Almost) any use of libfoo requires use of libbar, and the maintainer of libfoo has chosen to make a pkg-config check for libfoo imply access to the entire API and ABI of libbar. Example: libfoo = GDK, libbar = GObject: you cannot use a GDK object correctly without making use of GObject APIs.

Visible dependencies: Types from libbar are exposed in some header files of libfoo, but only a subset of users of libfoo will need to call functions from libbar. If they do, they are expected to carry out their own pkg-config check for libbar. Example: libfoo = GDK, libbar = libX11.

Invisible dependencies: There is no mention of libbar in the libfoo header files, and it is completely an implementation detail of libfoo. The libfoo maintainer could drop the dependency at any time and library users would be unaffected, with one exception: static linking (see below). Example: libfoo = GDK, libbar = libepoxy.

In each case, libfoo.pc needs to mention libbar somehow, so that users of libfoo can do the right thing.

In the critical dependency case, it's very simple: `pkg-config --cflags libfoo` must return -I options for both libfoo's headers and libbar's headers, and `pkg-config [--static] --libs libfoo` must return both -lfoo and -lbar (plus -L options to find them if necessary). libfoo Requires libbar.

In the visible dependency case, <foo/foo-bar.h> has #include <bar.h> to get the required "typedef struct _BarObject BarObject" so that it can declare "BarObject *foo_object_get_bar_object (FooObject *self);" or similar. This means it is not possible to include all libfoo headers without having libbar's headers in the -I path, so `pkg-config --cflags libfoo` still needs to return -I options for both libfoo's headers and libbar's headers.

Binary distributions like Debian and Fedora often need to deal with library transitions, where libbar.so.3 is replaced by a libbar.so.4 that is not entirely compatible (an ABI break). During a library transition, every binary with a runtime dependency on libbar.so.3 (a DT_NEEDED ELF header) needs to be rebuilt against libbar.so.4. The more packages you rebuild during a library transition, the more likely it is that something will go wrong (like a package failing to build from source), stalling the whole transition; so distributions want to ensure that a binary has a DT_NEEDED dependency on a library if and only if it uses symbols from that library. So in the visible dependency case, we want `pkg-config --libs libfoo` to include -lfoo but *not* -lbar. The final binary still loads libbar at runtime, because libfoo depends on it, but it is always the right version of libbar for this particular libfoo.

There is another point of complexity here, which is static linking. Static libraries (.a files) are basically just an archive (like a zip file) containing all the objects (.o files) for the library: they do not have metadata that can be parsed by gcc/ld, and they do not have dependencies. So when you are linking your binary to libfoo statically, you *do* need to include both -lfoo and -lbar in the link line - hence `pkg-config --static --libs`, which does that (typical usage would be gcc -ohello hello.c -static `pkg-config --static --libs libfoo`).

Finally, invisible dependencies, like libsoup's use of libxml2. Because libbar is just an implementation detail of libfoo, in an ideal world, `pkg-config --cflags libfoo` would include the necessary -I options for libfoo but, unlike visible dependencies, *not* libbar. The link flags are the same as for visible dependencies, and for the same reasons: `pkg-config --libs libfoo` must return -lfoo, while `pkg-config --static --libs libfoo` must return both -lfoo and -lbar.

The observant will notice that we have three use cases with different semantics here (critical, visible and invisible dependencies), but pkg-config currently only has two dependency types (Requires and Requires.private). So we have to either introduce a third dependency type, or choose two of the use cases and conflate them. There are two possibilities for conflating the dependency types, both of which are imperfect trade-offs.

If we treat critical and visible dependencies as the same thing, then libraries with visible dependencies result in overlinking: binaries that depend on libfoo have an explicit runtime dependency (DT_NEEDED) on libbar, which means binary distributions have to recompile a lot more packages during an ABI transition, significantly increasing the probability of a build failure that will stall the transition. This is what was done before 2006, until the behaviour changed as a result of <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=340904> (see that bug for extensive reasoning).

If, instead, we treat visible and invisible dependencies as the same thing, then we get too many CFLAGS for invisible dependencies, instead of too many DT_NEEDED entries for visible dependencies. This is still somewhat problematic, but seems (to me) less bad than too many DT_NEEDED entries: it can only cause problems at compile-time, not at runtime. This is what pkg-config has done since 2006. It's a trade-off, paying the price of having too many CFLAGS as a way to encourage library maintainers to make more use of Requires.private, which in turn results in fewer DT_NEEDED entries and less painful ABI transitions.

Going back to the pre-2006 behaviour would reintroduce the problems mentioned in Debian bug 340904, but more importantly it would be an incompatible change: many libraries rely being able to use Requires.private for visible dependencies. So I think we should rule that out. If there is a behaviour change, then it must always be in the direction of more dependencies, because having too many dependencies is a non-critical bug that does not prevent the use of a library, while having not enough dependencies is a critical bug that breaks linking.

On Debian bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892956, Adrian Bunk asserts that "tons of unnecessary -I flags" are a real problem. In further discussion of that bug on IRC, there were a few examples given:

* The development package for libfoo (libfoo-dev in Debian, foo-devel in Fedora, etc.) has to have a hard dependency on the development package for libbar, even if bar is merely an implementation detail of foo, because otherwise you can't even link a trivial "hello world" program with gcc hello.c `pkg-config --cflags --libs libfoo`. This results in developer systems having to install more -dev packages than they should. (Mitigation: If you want to support static linking, then these dependencies need to exist *anyway*, to make gcc -static hello.c `pkg-config --static --cflags --libs libfoo` work as intended.)

* If you indirectly depend on both (for example) libjpeg.so.62 and libjpeg.so.9, that actually works fine because they have versioned ELF symbols; but you can only have one <jpeglib.h> come first in the search path, so the other one is not going to work. (Mitigation: you can sometimes dodge this by choosing the order of dependencies.)

* If two of your dependencies both include header files with namespace-polluting names like <config.h>, then only one of them can work as intended. (Mitigation: design your libraries so they don't have namespace-polluting header files. I think the real answer to this has to be "don't do that", but unfortunately there are many libraries in Debian that unwisely install a config.h, and removing it would itself be an incompatible change.)

I think the only compatible options here are:

* Add a third dependency type, as Xavier advocated in Comment #0, to represent invisible dependencies;

* Declare that we don't care about excess developer dependencies and excess CFLAGS because they're a sufficiently small price to pay for not having excess runtime dependencies, and keep the current design

(In reply to Xavier Claessens from comment #0)
> 2) Add Requires.cflags to mean only pull cflags from those deps, not libs.
> Maybe pull their libs with --static?

Requires.cflags would be a misleading name for the current (post-2006) behaviour of Requires.private, because you are right to say that recursing through transitive dependencies' library flags *is* required when using --static.

Honestly, I think the Requires.private name should just stay - it's perhaps slightly misleading if there is a new Requires.very-private or something, but library authors mistakenly treating invisible dependencies as though they were visible dependencies is not the end of the world. We've had the current behaviour for over a decade and it wasn't a disaster.

Thanks Simon, much appreciated. I'm working on a patch to add Require.internal to cover the 3rd case of invisible dependencies.

(In reply to Xavier Claessens from comment #2)
> I'm working on a patch to add
> Require.internal to cover the 3rd case of invisible dependencies.

Thanks - Requires.internal seems a better name for that concept than anything that I could think of.

See also:

<https://bugzilla.redhat.com/show_bug.cgi?id=224148> is an older version of essentially the same bug/feature-request as this. It has some more interesting links and background, including Bug #4738 and <https://people.freedesktop.org/~dbn/pkg-config-guide.html>.

I do not agree with you on everything here, but I do appreciate your summary.

(In reply to Simon McVittie from comment #1)
>...
> On Debian bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892956,
> Adrian Bunk asserts that "tons of unnecessary -I flags" are a real problem.
> In further discussion of that bug on IRC, there were a few examples given:
>...

These are just examples for the some of problems when exposing implementation details of all libraries used directly or indirectly.

Looking at existing .pc files one easily finds other problems like e.g. https://sources.debian.org/src/libsecret/0.18.5-6/libsecret/libsecret-unstable.pc.in/#L13

It is unnecessary to expose such implementation details in cflags, and the way forward should be to reduce that - not increase that.

> I think the only compatible options here are:
> 
> * Add a third dependency type, as Xavier advocated in Comment #0, to
> represent invisible dependencies;
> 
> * Declare that we don't care about excess developer dependencies and excess
> CFLAGS because they're a sufficiently small price to pay for not having
> excess runtime dependencies, and keep the current design
>...

A third dependency type already exists and is widely used: Libs.private

Plenty existing software already uses something like "Libs.private: @PRIVATE_LIBS@" or even "Libs.private: @LIBS@" to generate a list of libraries from their pkg-config dependencies.

The only open question is IMHO whether the .pc format should be expanded to provide a new option that is better than Libs.private for static linking.

I've read through the various summaries, and we don't need to overthink this, really.

There are three kinds of dependencies a project can have on a specific library (`somelib`):

1. Includes required during compilation (when doing `#include <somelib.h>`)
2. Libraries required during dynamic linking (DT_NEEDED)
3. Libraries required during static linking (recursive DT_NEEDED)

Note that (1) and (2) are not the same. A library's headers might use prototypes or defines from another library in its headers, but that doesn't mean that library's symbols are needed at link time.

This combined with --as-needed not really existing back in 2006 meant that the decision was made to gather includes from `Requires.private:` for compilation, even though that list should only be used to gather libraries for linking.

This decision was wrong. We need to differentiate between Requires for compilation, Requires for dynamic linking, and Requires for static linking. Otherwise, pc files with correct lists of Requires.private will require the whole world even when doing dynamic linking.

The migration path is:

1) Deprecate Requires.private
2) Continue using Requires for dynamic linking and compilation
3) Create Requires.cflags for compilation (must not duplicate names from Requires)
4) Create Requires.private-libs for static linking (must not duplicate names from Requires, *must* duplicate names from Requires.cflags)

I think I've been convinced that Requires and Requires.private actually both make sense and we just need a 3rd Requires.internal which will be needed purely for static linking.

(In reply to Adrian Bunk from comment #4)
> A third dependency type already exists and is widely used: Libs.private
> 
> Plenty existing software already uses something like "Libs.private:
> @PRIVATE_LIBS@" or even "Libs.private: @LIBS@" to generate a list of
> libraries from their pkg-config dependencies.

Adding manually to Libs.private only works if I know all internal dependencies used by the libs I depend on and they never change. For example if I add Libs.private: -lglib-2.0 I also have to know that glib uses libffi and add -lffi. Otherwise I won't be able to static link.

(In reply to Xavier Claessens from comment #7)
> (In reply to Adrian Bunk from comment #4)
> > A third dependency type already exists and is widely used: Libs.private
> > 
> > Plenty existing software already uses something like "Libs.private:
> > @PRIVATE_LIBS@" or even "Libs.private: @LIBS@" to generate a list of
> > libraries from their pkg-config dependencies.
> 
> Adding manually to Libs.private only works if I know all internal
> dependencies used by the libs I depend on and they never change. For example
> if I add Libs.private: -lglib-2.0 I also have to know that glib uses libffi
> and add -lffi. Otherwise I won't be able to static link.

For -lglib-2.0 you do not need -lffi, only for -lgobject-2.0.

And you don't have to know, you ask pkg-config:

$ pkg-config --static --libs gobject-2.0
-lgobject-2.0 -lffi -lglib-2.0 -pthread  -lpcre    -lpcre -pthread  
$ 

See the "Libs.private: @LIBFFI_LIBS@" in https://gitlab.gnome.org/GNOME/glib/blob/master/gobject-2.0.pc.in#L11

This is how things are done today, and it works reasonably well in practice.

Note that "reasonably well" != "100% correct and guaranteed to work in all cases", there is no disagreement that a Requires.internal instead of hardcoding the libraries would be better here.

(In reply to Xavier Claessens from comment #6)
> I think I've been convinced that Requires and Requires.private actually both
> make sense and we just need a 3rd Requires.internal which will be needed
> purely for static linking.

Well, if we want to change the meaning of Requires.private to refer to include requirements, that's fine too. I don't mind too much that the name would be inaccurate. We just need to update the documentation:

https://people.freedesktop.org/~dbn/pkg-config-guide.html#faq

Quote:

> My library z uses libx internally, but does not expose libx data types in its public API. What do I put in my z.pc file?
> Again, add the module to Requires.private

We should also give the pkgconf maintainers a heads-up and get their opinion. I'll try reaching out to them.

A related thing is how static dependencies should be handled. Currently if you add --static, the only difference is that in addition to -lmainlib it would add -ldeplib. If you have both libmainlib.so and libmainlib.a installed pkg-config does not in any way instruct the linker to pick up either library. The only way is to give -Wl,-static to the linker manually but it is a global option like -L.

A reasonable thing to do is to link every other dependency shared but some one thing statically. So to get it to actually work, pkgconfig should add flags like the following:

-L<> -Wl,-push-staticness -Wl,-static -lfoo -lbar -Wl,-pop-staticness

The first and last flags obviously don't exist. An alternative would be to do something like:

/path/to/libfoo.a /other/path/to/libbar.a

For the more general case aside of distros and jhbuild like alternate monolithic prefixes there is also the problem, that "Invisible dependencies" are only invisible when they are found in the search path of the linker(ld).

If they are in a different library directory ld insists that it is told of that directory. So in the general case the build system has to generate "-rpath-link" options. This is similar to static linking, only that "-l" are ignored and "-L" is changed to "-rpath-link". (I believe this information is the same that build systems that have an uninstalled rpath would use as rpath, and installed rpath is something that pkg-config should not need to know about).

But done naively this would cause the same dependency chain problems with regard to .pc files that partially started this whole discussion... Not sure how that could be solved realistically...

Created attachment 138185 [details] [review]
Code refactoring

Created attachment 138186 [details] [review]
Add Require.internal

Obviously missing doc & tests, but that gives an idea. (Hm, and why is my git bz broken with fdo?)

(In reply to Jussi Pakkanen from comment #10)
> The first and last flags obviously don't exist. An alternative would be to
> do something like:
> 
> /path/to/libfoo.a /other/path/to/libbar.a

That's what meson already does and it works well, AFAIK.

After talking to the pkgconf maintainers, it turns out that this whole thing is a problem only for Debian (and derivatives). Every other distro generates the -dev dependency list for a package automatically using both Requires and Requires.private so that static linking works correctly.

The only effect on apk/rpm distros would be that more -dev packages would get pulled in, which makes sense because otherwise you'd have to install those packages manually while linking statically.

So the only effect of this would be that Debian would be allowed to maintain the status-quo and not have -dev packages that properly support static linking. Fedora and other rpm-based distros would have to parse Requires.internal to get the full list of -dev packages anyway.

I've been told that there have been patches to debhelper to make it parse the pkg-config file and generate -dev dependencies automatically, but they were rejected for non-technical reasons.

(In reply to Jussi Pakkanen from comment #10)
> A related thing is how static dependencies should be handled. Currently if
> you add --static, the only difference is that in addition to -lmainlib it
> would add -ldeplib. If you have both libmainlib.so and libmainlib.a
> installed pkg-config does not in any way instruct the linker to pick up
> either library. 

I believe this is a feature. 

pkg-config cannot know what linker you are using, it cannot know how your libraries are named, and it cannot know which libraries you want to link to statically. All it can do is give you a full list of libraries you might need.

pkg-config maintains dependency information, and resolves it. It's on a totally different layer of abstraction. Only the build system (like meson) can ascertain the user's intent and transform those -lfoo lines.

(In reply to Nirbheek Chauhan from comment #16)
> After talking to the pkgconf maintainers, it turns out that this whole thing
> is a problem only for Debian (and derivatives). Every other distro generates
> the -dev dependency list for a package automatically using both Requires and
> Requires.private so that static linking works correctly.

I also think that in the current situation meson does the right thing and it's debian's problem to add more -dev to their packages. It's not a big deal.

However, the use-case to reduce build deps is valid, so debian will be able to reduce their -dev list once Requires.internal gets merged and used. They could distro-patch if they really care about the number of build deps.

(In reply to Xavier Claessens from comment #17)
> (In reply to Nirbheek Chauhan from comment #16)
> > After talking to the pkgconf maintainers, it turns out that this whole thing
> > is a problem only for Debian (and derivatives). Every other distro generates
> > the -dev dependency list for a package automatically using both Requires and
> > Requires.private so that static linking works correctly.
> 
> I also think that in the current situation meson does the right thing and
> it's debian's problem to add more -dev to their packages. It's not a big
> deal.
> 
> However, the use-case to reduce build deps is valid, so debian will be able
> to reduce their -dev list once Requires.internal gets merged and used. They
> could distro-patch if they really care about the number of build deps.

What's interesting is that this makes static linking a second-class citizen because users have to manually install those -dev packages. I don't have a strong opinion about that; we can probably handle that in Meson by prompting for installation.

As an aside, the pkgconf maintainers intend to merge the Requires.internal list with Requires.private, so nothing will change for distros that use it.

In fact, I already implemented that:

https://github.com/pkgconf/pkgconf/commit/f03ec3ff90ee619a21826d23609adb5409dbe9a2

A note from irc. The pkgconf implementation also outputs Requires.internal in the result of --print-requires-private. As far as i understand this was made to keep packaging infrastructure that uses --print-requires and --print-requires-private to determine package dependencies / metadata to work without changes.

It of course would be nice to also have a way to only get Requires.internal via command line options.

(In reply to Martin Hostettler from comment #20)
> A note from irc. The pkgconf implementation also outputs Requires.internal
> in the result of --print-requires-private.

I think it makes sense, yes. pkg-config --help says:
--print-requires-private                print which packages the package requires for static linking

My patch also add Requires.internal into --print-requires-private, I don't see a usecase where --print-requires-internal is needed, can always be added later if/when needed.

Created attachment 138201 [details] [review]
Simplify requires lists

- Lists of names are useless, better keep list of RequireVersion to
always have the context.
- Since we keep lists of RequireVersion we no longer need an hash table
to map package name to their context.
- No longer append requires to requires_private

This allows to factor out log of duplicated code and will simplify
adding a 3rd category of requires.

Created attachment 138202 [details] [review]
Add Require.internal field

This field is to be used for purely internal dependencies that are not
exposed into the API at all. Those packages are there only for static
link.

Created attachment 138203 [details] [review]
Add test for Requires.internal

Created attachment 138204 [details] [review]
Update the guide to include Requires.internal

Who's reviewing pkg-config patches nowadays?

(In reply to Adrian Bunk from comment #4)
> A third dependency type already exists and is widely used: Libs.private

Libs.private can certainly be used as a substitute for a third dependency type (with some caveats, mainly around the need to hard-code dependency chains into a dependent library, as already discussed), but I don't think that's how it was intended to be used. The pkg-config man page suggests that Libs.private is meant to be the equivalent of Requires.private for libraries that don't have a .pc file - or at least, that's how I've always interpreted it.

(In reply to Jussi Pakkanen from comment #10)
> A related thing is how static dependencies should be handled. Currently if
> you add --static, the only difference is that in addition to -lmainlib it
> would add -ldeplib.

I believe the design was that this is considered to be outside pkg-config's scope: if you want to produce a statically linked binary, then you're expected to pass -static to the compiler driver yourself (or use something like "-Wl,--push-state, -Wl,-static $(pkg-config --static --libs foo) -Wl,--pop-state" if you want to direct the linker to link only a particular dependency statically). It's documented as "Output libraries suitable for static linking", not as "Output complete linker arguments suitable for static linking".

(In reply to Martin Hostettler from comment #11)
> For the more general case aside of distros and jhbuild like alternate
> monolithic prefixes there is also the problem, that "Invisible dependencies"
> are only invisible when they are found in the search path of the linker(ld).

If a library libfoo depends on a dependency libbar, and libbar is not in the linker's default search path, shouldn't libfoo have a DT_RUNPATH or DT_RPATH header that causes the linker to find libbar in the appropriate place automatically? (GNU ld(1) says those headers are respected at link time by "a native ELF linker".)

(In reply to Nirbheek Chauhan from comment #16)
> After talking to the pkgconf maintainers, it turns out that this whole thing
> is a problem only for Debian (and derivatives). Every other distro generates
> the -dev dependency list for a package automatically using both Requires and
> Requires.private so that static linking works correctly.

It's true that the broken dependency due to missing -dev dependencies is only a practical problem for Debian (and derivatives), but Adrian's concern about excess CFLAGS affects everyone. Perhaps you don't consider that concern to be valid - I'm not sure how important it is myself - but if it's being dismissed, it should be dismissed for the right reasons.

> I've been told that there have been patches to debhelper to make it parse
> the pkg-config file and generate -dev dependencies automatically, but they
> were rejected for non-technical reasons.

Do you have a reference for those patches? If someone (maybe me) has time to pick them up and improve them, that seems a good approach.

(Earlier in this bug's history I did think that Debian's Lintian tool should check that there aren't missing -dev dependencies, but then I realised that isn't possible in general, because Lintian can't know which .pc file is found in which -dev package. A debhelper tool that runs during the package's build can normally rely on the dependencies all being installed, though.)

(In reply to Nirbheek Chauhan from comment #18)
> What's interesting is that this makes static linking a second-class citizen
> because users have to manually install those -dev packages.

Some libraries don't support static linking at all: I'm told Mesa is an example of a library stack whose upstream maintainers specifically don't support static linking. libinput, which is the library that triggered all this, doesn't install a libinput.a, contrary to normal Debian policy (I don't know whether that was a decision made by its upstream maintainers or by its Debian maintainers). For those libraries, static linking is already impossible, so there's little point in supporting pkg-config --static.

(In reply to Simon McVittie from comment #27)
> libinput, which is the library that triggered all
> this, doesn't install a libinput.a, contrary to normal Debian policy (I
> don't know whether that was a decision made by its upstream maintainers or
> by its Debian maintainers).

I think that's because Meson does not support (yet) compiling both static and shared at the same time. I have a PR pending for that and once it's merged debian should enable it for their builds. However I see libinput uses explicitly shared_library() so that won't work for them. I don't know if that's intentional.

Created attachment 138281 [details] [review]
Ignore Requires.internal if we only want cflags

`pkg-config --cflags --libs dep` should not fail if internal packages
are missing.

Created attachment 138282 [details] [review]
m4: Fix autoreconf

Patch copied from debian package.

(In reply to Simon McVittie from comment #27)
> (In reply to Nirbheek Chauhan from comment #16)
> > I've been told that there have been patches to debhelper to make it parse
> > the pkg-config file and generate -dev dependencies automatically, but they
> > were rejected for non-technical reasons.
> 
> Do you have a reference for those patches? If someone (maybe me) has time to
> pick them up and improve them, that seems a good approach.

I don't think the debhelper maintainers ever pronounced themselves over that bug
report, TBH.
 
> (Earlier in this bug's history I did think that Debian's Lintian tool should
> check that there aren't missing -dev dependencies, but then I realised that
> isn't possible in general, because Lintian can't know which .pc file is
> found in which -dev package. A debhelper tool that runs during the package's
> build can normally rely on the dependencies all being installed, though.)

I think this tool belongs one layer below, in the dpkg-dev package. I started
implementing support for .pc files some time ago, but got distracted by other
stuff. I'd be happy if you'd like to continue that work. Otherwise I might get
around to it eventually. Draft branch with bug references:

https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/dpkg-pkgconfigdeps

Although I'm not sure if we should instead go into a dpkg-gendeps tool that
would handle pluggable dependencies from .shlibs/.symbols files, .pc files,
and .so links from this other branch:

https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=pu/dpkg-solinkdeps

In any case let's move the discussion to the debian-dpkg mailing list, if
there's interest.

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/pkg-config/pkg-config/issues/7.