First of all, I have a bit of an unusual config, I think: Linux, kernel 2.4.29 glibc-2.3.4, built with ../glibc-2.3.4/configure " '--prefix=/usr' '--enable-add-ons=linuxthreads,libidn' '--enable-kernel=2.4.18' This means that glibc is built with linuxthreads and TLS. Basically, the X server segfaults within the libint10.a code, or just after (hard to tell, of course). I tracked this down, and think I know what is going on. The problem is that the vm86old syscall which is executed by vm86_rep clobbers the gs segment register. But TLS (at least with linuxthreads) uses the gs register to keep track of where the thread local storage is (which is evidently used even when you don't use threads...). Thus, the next time malloc is called, we segfault and die. My simple solution was to push the gs register before the int $0x80 call and restore it after (gcc didn't seem to recognize the gs register for a clobber spec). From looking on Google, there is at least one other person with this same problem, so my crazy configuration is not completely unique...
Created attachment 1797 [details] [review] Patch to save the ds register before the vm86old syscall
wow, good catch. i'll take a look at this one.
The patch looks sane. Ajax, I've already prepared everything in my tree to commit this. If you haven't done that already I can commit it.
go for it.
Committed to HEAD 2005-02-07 Egbert Eich <eich-at-freedesktop-dot-org> * programs/Xserver/hw/xfree86/os-support/linux/int10/vm86/linux_vm86.c: (vm86_rep): Save gs register before calling the vm86_old syscall thru int 0x80. This is required for linuxthreads as the TLS uses this register to keep track of local thread storage (Bugzilla #2431, J. Scott Berg).
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.