SProcShmCreatePixmap() in file programs/Xserver/Xext/shm.c swaps all the
multi-byte members of the xShmCreatePixmapReq struct except for pid. If the
application and the X server are running in different endianness, calls to
XShmCreatePixmap will fail.
This is difficult to reproduce unless you happen to have a locally running X
client in a different endianness mode to the server, but if you have and you try
to use XShmCreatePixmap the X server will return an error.
I have a patch which fixes this bug, I will attach it to this report. The bug
was found in release 6.8.2, but the problem exists in CVS HEAD as well.
Created attachment 3530 [details] [review]
Patch to byteswap xShmCreatePixmapReq.pid in SProcShmCreatePixmap().
The attached patch corrects the problem I'm seeing with the endian-mismatched
client and server. In the common case for shared memory (ie. that in which the
server and client are the same endianness) this shouldn't change anything, as
the data should not be byteswapped anyway.
Doesn't this patch break compatibility for old clients connecting to a new server?
(In reply to comment #2)
> Doesn't this patch break compatibility for old clients connecting to a new server?
I don't believe so, unless the client has been hacked to reverse the pid itself.
If the endianness of client and server is the same, which it nearly always is,
then the swap won't have any effect. If the endianness is different, it just
won't work as is; the server won't recognise the pid as a valid id.
lib/Xext/XShm.c does not byte-swap the pid element of the request (nor should
it), so Neil's patch is correct. since this change only affects the SProc path
it will not break normal (same-endianness) MIT-SHM clients.
applied to head, will be in 7.0RC2. thanks!