verified with git master. 

Problem is that dix-config.h defines X_BYTE_ORDER without having a definition for X_LITTLE_ENDIAN or X_BIG_ENDIAN. A subsequent include of Xarch.h then doesn't define it because it believes them to be defined already. This would affect all files that include dix-config.h before any other header files (i.e. most if not all).

Two possible fixes - either include Xarch.h before dix-config.h (lots of changes!) or somehow define L/B_ENDIAN in dix-config.h.

Alan: you have more experience than me with this, can you please have a look at that?

actually, can you give us the output of the preprocessor?

cd dix/
make colormap.o

then copy the full gcc command (without the if/then wrapper around it) and append a -E. Run it, and please rename and attach the output file (colormap.o -> colormap.txt) to this bug report. Thx.

Created attachment 17105 [details]
Preprocessed output of colormap.c

compiler call:

powerpc-unknown-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I../include -I../Xprint -DHAVE_DIX_CONFIG_H -Wall -Wpointer-arith -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fno-strict-aliasing -D_BSD_SOURCE -DHAS_FCHOWN -DHAS_STICKY_DIR_BIT -DDBUS_API_SUBJECT_TO_CHANGE -I/usr/include/freetype2 -I/usr/include/pixman-1 -I/usr/include/hal -I/usr/include/dbus-1.0 -I/usr/lib/dbus-1.0/include -I../include -I../include -I../Xext -I../composite -I../damageext -I../xfixes -I../Xi -I../mi -I../miext/shadow -I../miext/damage -I../render -I../randr -I../fb "-DVENDOR_NAME=\"The X.Org Foundation\"" "-DVENDOR_RELEASE=(((1) * 10000000) + ((4) * 100000) + ((2) * 1000) + 0)" -mcpu=7400 -mtune=7400 -maltivec -mabi=altivec -O2 -pipe -MT colormap.lo -MD -MP -MF .deps/colormap.Tpo -c colormap.c  -fPIC -DPIC -E > colormap.txt

Since i'm bothered by the problem for quite a while now (just didn't have the time to look into it closer) i'll provide the preprocessed output and hope for a quick solution ;)
The problem occurs on a ppc32 system (iBook G4) running Gentoo linux.

Actually, since it's the preprocessor that fails during compile attempt it's not really preprocessed output (the file specified using -o doesn't exist/is empty), just something the compiler wrote to stdout on the aforementioned call.

The sanest thing should be avoid this completely and just include Xarch.h in that header.

(or make sure _X_BYTE_ORDER doesn't get defined as nothing as now happens.

> --- Comment #6 from Luca Barbato <lu_zero@gentoo.org>  2008-06-14 19:05:31 PST ---
> The sanest thing should be avoid this completely and just include Xarch.h in
> that header.

That's the problem. Xarch.h is included.
http://cgit.freedesktop.org/xorg/xserver/tree/include/colormapst.h#n51

Created attachment 17120 [details]
a not exactly nice but effective fix.

the "problem" is that dix-config.h has some brain damage added so instead of using Xarch.h to get the endianess or using the autotool facility to get it. Tries to put a simplified Xarch.h for apple and tries to overwrite the value using configure. The thing doesn't work for a reason I didn't tracked, but just sticking #include <X11/Xarch.h> inside dix-config.h.in did the job (now I'm tracking glibc-2.8 related fixes)

Probably just removing all endianess reference from dix-config.h should work as well but I'd rather be safe than sorry.

Peter & I discussed on IRC last week - I can't see any way that you
can #include Xarch.h and not end up with X_LITTLE_ENDIAN & X_BIG_ENDIAN
undefined.

I'm not sure what the added MacOS X code is trying to do there though.

_X_BYTE_ORDER is defined as nothing in the dix-config.h

X_BYTE_ORDER gets defined as nothing, thus making the macro eval as
( == X_LITTLE_ENDIAN)

since X_BYTE_ORDER is defined Xarch.h doesn't reset it.

either removing everything from dix-config.h or putting there 

#include <X11/Xarch.h>

solves the issue (and will work for generating fat binaries in macosx/darwin.

Why is X_BYTE_ORDER getting set to nothing?  Shouldn't configure.ac be setting it appropriately?

The added Apple code there is trying to get around the problem that the same header file is used to compile for both little endian and big endian at the exact same time, so we can't just ./configure for one target... we need to ./configure for multiple targets (ppc, ppc64, i386, x86_64) at the same time.

xorg-server-1.4.0.90, little-endian:
# grep _X_BYTE_ORDER /usr/include/{xorg,X11} -r
/usr/include/xorg/xorg-server.h:#define _X_BYTE_ORDER X_LITTLE_ENDIAN
/usr/include/xorg/xorg-server.h:#define X_BYTE_ORDER _X_BYTE_ORDER

xorg-server-1.4.2, big-endian (PPC64-32ul):
# grep _X_BYTE_ORDER /usr/include/{xorg,X11} -r
/usr/include/xorg/xorg-server.h:/* #undef _X_BYTE_ORDER */
/usr/include/xorg/xorg-server.h:#define X_BYTE_ORDER _X_BYTE_ORDER

xorg-server-1.4.2, little-endian:
# grep _X_BYTE_ORDER /usr/include/{xorg,X11} -r
/usr/include/xorg/xorg-server.h:/* #undef _X_BYTE_ORDER */
/usr/include/xorg/xorg-server.h:#define X_BYTE_ORDER _X_BYTE_ORDER

The source of the issue for xorg is this definition in configure.ac:
AC_C_BIGENDIAN([ENDIAN="X_BIG_ENDIAN"], [ENDIAN="X_LITTLE_ENDIAN"])
while AC_C_BIGENDIAN can take 4 inputs: true, false, failure, universal.

But it doesn't end there...

And autoconf-2.6.2 has a broken detection of universal on linux-PPC:
http://www.mail-archive.com/bug-autoconf@gnu.org/msg01680.html

AC_C_BIGENDIAN takes the universal path when autoconf-2.6.2 is installed and we are on ppclinux, so $ENDIAN does not get defined, so _X_BYTE_ORDER ends up as empty (and Luca tried to patch that part out).

There hasn't been any fix posted to the autoconf list yet however :-(.

Created attachment 18574 [details] [review]
better patch to work around autoconf macro problem

Attachment #17120 [details] didn't work for me -- build failed later because X11/Xarch.h includes features.h before _GNU_SOURCE gets set, so things don't work right.  Switching it around so Xarch.h gets included after settting _GNU_SOURCE allowed the build to succeed, but then generated an xorg-config.h with incorrect endianness detection/macros, which broke the nouveau driver.

This patch works for me, though I'm not entirely sure I need it anymore, as when I ran configure, it actually detected as 'big endian' this time, rather than 'universal'.  So maybe the autoconf bug has been fixed since the last time I tried.

I imagine this patch might also break things on systems that actually should be correctly detected as 'universal'... though maybe xorg doesn't support them anyway.

(In reply to comment #14)
> ... though I'm not entirely sure I need it anymore, as
> when I ran configure, it actually detected as 'big endian' this time, rather
> than 'universal'.  So maybe the autoconf bug has been fixed since the last time
> I tried.

Ah, no, it looks like Gentoo has a patched version of autoconf 2.62 that reverts the changes in AC_C_BIGENDIAN() so the macro is effectively the same as in 2.61.

So I can't be completely sure that my patch fixes the issue (since I'm apparently being correctly identified as big endian again), but I think it should.  The other patch almost certainly doesn't work properly, and can make the problems worse.

Autoconf 2.63, released 2008-09-09, fixed this bug. You can read the report here :
http://lists.gnu.org/archive/html/autoconf/2008-09/msg00065.html.

Since this only affects autoconf versions two+ years old, can this bug just be closed?

Fixed by now.