When doing antialiasing, fontconfig-based renderers do not take gamma into account and assume a linear color space. This make black on white text difficult to read at small font sizez.
For example, with the font spec "FreeMono:pixelsize=10:autohint:hintstyle=3:rgba=none", used in the following test case:
xterm -fa $fa -geometry 20x2 -bg black -fg white -e 'echo foobar; sleep 10000'
xterm -fa $fa -geometry 20x2 -bg white -fg black -e 'echo foobar; sleep 10000'
The black on white version is quite readable on my screen, while the white on black version is almost unreadable.
The reason is that the stems of the glyphs are thinner than a whole pixel. Therefore, they get a fractionnal value. For example, the pixels on the lower part of the stem of the 'f' get the pixel value 151/255 in black on white, and 104/255 in white on black (and 104+151=255). With the usual 2.2 gamma, this makes respectively 32% and 14%, which gives a contrast of 68% for black on white, and 14% for white on black.
Obviously, the problem is not in fontconfig itself, but components that should do gamma correction need a gamma value, and this is fontconfig task to provide it.
Fontconfig has nothing to do with presenting the glyphs to the user, it simply selects the fonts. The bug you are seeing (and, yes, I agree that it is a bug even if white text on a black background is wrong) is due to limitations in various rendering libraries, like Xrender, cairo et al.