Bug 4730

Summary: SProcShmCreatePixmap() fails to byte-swap 'pid'
Product: xorg Reporter: Neil Campbell <batneil>
Component: Server/GeneralAssignee: Xorg Project Team <xorg-team>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: high CC: batneil
Version: git   
Hardware: All   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:
Attachments:
Description Flags
Patch to byteswap xShmCreatePixmapReq.pid in SProcShmCreatePixmap(). none

Description Neil Campbell 2005-10-10 02:00:25 UTC
SProcShmCreatePixmap() in file programs/Xserver/Xext/shm.c swaps all the
multi-byte members of the xShmCreatePixmapReq struct except for pid.  If the
application and the X server are running in different endianness, calls to
XShmCreatePixmap will fail.

This is difficult to reproduce unless you happen to have a locally running X
client in a different endianness mode to the server, but if you have and you try
to use XShmCreatePixmap the X server will return an error.

I have a patch which fixes this bug, I will attach it to this report.  The bug
was found in release 6.8.2, but the problem exists in CVS HEAD as well.
Comment 1 Neil Campbell 2005-10-10 02:45:03 UTC
Created attachment 3530 [details] [review]
Patch to byteswap xShmCreatePixmapReq.pid in SProcShmCreatePixmap().

The attached patch corrects the problem I'm seeing with the endian-mismatched
client and server.  In the common case for shared memory (ie. that in which the
server and client are the same endianness) this shouldn't change anything, as
the data should not be byteswapped anyway.
Comment 2 Matthieu Herrb 2005-10-10 03:45:18 UTC
Doesn't this patch break compatibility for old clients connecting to a new server?
Comment 3 Neil Campbell 2005-10-10 03:49:47 UTC
(In reply to comment #2)
> Doesn't this patch break compatibility for old clients connecting to a new server?
 
I don't believe so, unless the client has been hacked to reverse the pid itself.
 If the endianness of client and server is the same, which it nearly always is,
then the swap won't have any effect.  If the endianness is different, it just
won't work as is; the server won't recognise the pid as a valid id.
Comment 4 Adam Jackson 2005-10-21 11:50:25 UTC
lib/Xext/XShm.c does not byte-swap the pid element of the request (nor should
it), so Neil's patch is correct.  since this change only affects the SProc path
it will not break normal (same-endianness) MIT-SHM clients.

applied to head, will be in 7.0RC2.  thanks!

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.