I was trying to install Xfbdev (from Xorg-server-1.2.0) on a 32-bits framebuffer with 8 bits masks for each channel. It is due to hardware specification.
31..24 -> b
23..16 -> g
15..8 -> r
7..0 -> a
I didn't have any problem at install, but a Segfault while building colormap at execution.
Indeed, in fbdev.c, we can see a bloc #ifdef FBDEV_KLUDGE_FORMAT which modifies color masks (especially for red where mask becomes 0xFFFF instead of OxFF00).
This modification implies an out of memory acces in dix/colormap.c around line 313. The 'size' value, which should be red depth (ie 256) is 65536 because of previous modification.
If a delete the block between #ifdef FBDEV_KLUDGE_FORMAT, output initialization goes on successfully, but I got a SegFault in Xrender (as explained in fbdev.c)
It seems to be a real incompatibility with 32bits framebuffers with alpha channel on low-bits, but I don't have time to study Xrender and submit a patch.
Sorry and thank you!
Is this still an issue for you on a newer xserver? If so, can you provide a log with backtrace from the segfault to help with fixing this issue? Maybe a patch to show exactly what you did?
Xfbdev isn't a thing anymore.