References:
http://access1.sun.com/techarticles/stack-overflow.html

Let's clarify a few things first.

-xcheck=stkovf flag protects against a stack overflow but not a buffer overflow.
The difference is that a stack overflow can occur even in a non-buggy program
that simply uses more stack space than the actual stack size, but a buffer
overflow occurs due to a program bug that sometimes can overwrite the
non-current stack frame. So compiling Xserver and X11 applications with
-xcheck=stkovf will help debugging a potential stack overflow problems, but
that's not likely to make them secure from the buffer overflow attack.
The former alone might be valuable enough to trade the performance hit of
-xcheck=stkovf, but I'm not so sure.

The only stack overflows (other than buffer overflows on the stack which 
get fixed other ways) I remember seeing in X servers on Solaris have been 
attempts to allocate way too much memory via alloca() - personally I'd rather
solve those by removing alloca() as its lack of any way to check errors is
not a good design.

We get a limited amount of protection against overflow exploits on SPARC & AMD64 
platforms by linking with mapfiles that mark stack & data segments as
non-executable (see /usr/lib/ld/map.noexstk & on Solaris 10 
/usr/lib/ld/map.noexdata - limited because it doesn't prevent all classes of
exploits, just the easier ones).

gcc systems using the ProPolice stack protector require a few extra symbols to
be exported by the module loader, at least for elfloader configurations.  i
don't know how -xcheck=stkovf works, but since similar methods require ABI
extensions this one might too.  just something to be aware of.

ajax, -xcheck=stkovf is not what you think it is (stack overflow != buffer
overflow on stack). Please read the link Roland provided if you're interested in
more detail. BTW, -xcheck=stkovf does not require any ABI change.

(In reply to comment #3)
> The only stack overflows (other than buffer overflows on the stack which 
> get fixed other ways) I remember seeing in X servers on Solaris have been 
> attempts to allocate way too much memory via alloca() - personally I'd rather
> solve those by removing alloca() as its lack of any way to check errors is
> not a good design.

OK... should be close this bug as WONTFIX then ?

> We get a limited amount of protection against overflow exploits on SPARC & AMD64 
> platforms by linking with mapfiles that mark stack & data segments as
> non-executable (see /usr/lib/ld/map.noexstk & on Solaris 10 
> /usr/lib/ld/map.noexdata - limited because it doesn't prevent all classes of
> exploits, just the easier ones).

Which compiler switch is needed to get that working in Solaris 10 ?

To use the mapfiles mentioned, the flags would be:
-Wl,-M,/usr/lib/ld/map.noexstk -Wl,-M,/usr/lib/ld/map.noexdata

The Xorg sun.cf already sets the first flag on Solaris 9 and later - see the
NonExecStackLoadFlag setting.   I've never gotten around to adding the second
for Solaris 10 - it would have to be x86 only since it will break the SPARC PLT
as noted in the comments in the file.   (When building the binaries distributed
with Solaris we actually use custom mapfiles which contain the commands noted
in the above, plus those from the /usr/lib/ld/map.pagealign and on SPARC
the non-executable bss flag described in the comments in the map.noexdata.)

For autoconf use, the STSF configure.ac shows an example of one way to detect
and use these:
case "$target" in
        *-*-solaris*)
                AC_CHECK_FILE([/usr/lib/ld/map.noexstk],[LDFLAGS="${LDFLAGS}
-Wl,-M/usr/lib/ld/map.noexstk"])
                AC_CHECK_FILE([/usr/lib/ld/map.pagealign],[LDFLAGS="${LDFLAGS}
-Wl,-M/usr/lib/ld/map.pagealign" ; SHAREDLIB_LDFLAGS="${SHAREDLIB_LDFLAGS}
-Wl,-M/usr/lib/ld/map.pagealign"])
                ;;
esac

I checked in changes to the sun.cf Imake config file in CVS on 8/23/2005 to 
use the noexstk & noexdata mapfiles where possible.

Since the -xcheck=stkovf flag should mainly be used by adding to the flags when
debugging and not for general use, I'm closing this bug as WONTFIX as Roland
suggested a while ago.