If you build an extension outside of the xorg-server build system (e.g. following http://www.x.org/wiki/X11_Extension_tutorial), the resulting extension will fail on 64 bit systems with an BadLength error.

The reason the following: Outside of the xorg-server build system, the _XSERVER64 macro is not defined (it usually come from include/dix-config.h). In this case, the Mask type becomes 64 bit, although 32 bit is correct even on 64 bit systems.

Mask is used in the ClientRec struct. But due to the wrong size of Mask, the member alignment is broken: in the 3rdparty extension the Mask variables cause a different binary layout of the ClientRec struct than in the internal extensions.

Test case:

  #define _XSERVER64

  #include <X11/X.h>
  #include <X11/Xproto.h>
  #include <xorg/misc.h>
  #include <xorg/extnsionst.h>
  #include <xorg/dixstruct.h>

  int main(int argc, char **argv) {
    printf("%d\n", sizeof(Mask));
    printf("%d\n", sizeof(ClientRec));
  }

You get 4 and 304.

Now remove the _XSERVER64 definition. You get 8 and 312.