SProcShmCreatePixmap() in file programs/Xserver/Xext/shm.c swaps all the multi-byte members of the xShmCreatePixmapReq struct except for pid. If the application and the X server are running in different endianness, calls to XShmCreatePixmap will fail. This is difficult to reproduce unless you happen to have a locally running X client in a different endianness mode to the server, but if you have and you try to use XShmCreatePixmap the X server will return an error. I have a patch which fixes this bug, I will attach it to this report. The bug was found in release 6.8.2, but the problem exists in CVS HEAD as well.
Created attachment 3530 [details] [review] Patch to byteswap xShmCreatePixmapReq.pid in SProcShmCreatePixmap(). The attached patch corrects the problem I'm seeing with the endian-mismatched client and server. In the common case for shared memory (ie. that in which the server and client are the same endianness) this shouldn't change anything, as the data should not be byteswapped anyway.
Doesn't this patch break compatibility for old clients connecting to a new server?
(In reply to comment #2) > Doesn't this patch break compatibility for old clients connecting to a new server? I don't believe so, unless the client has been hacked to reverse the pid itself. If the endianness of client and server is the same, which it nearly always is, then the swap won't have any effect. If the endianness is different, it just won't work as is; the server won't recognise the pid as a valid id.
lib/Xext/XShm.c does not byte-swap the pid element of the request (nor should it), so Neil's patch is correct. since this change only affects the SProc path it will not break normal (same-endianness) MIT-SHM clients. applied to head, will be in 7.0RC2. thanks!
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.