Bug 3412 - Xvfb with the composite extension gets segmentation fault
Summary: Xvfb with the composite extension gets segmentation fault
Alias: None
Product: xorg
Classification: Unclassified
Component: Server/DDX/Xvfb (show other bugs)
Version: 6.8.2
Hardware: x86 (IA32) Linux (All)
: high normal
Assignee: Xorg Project Team
QA Contact:
Depends on:
Reported: 2005-05-27 10:40 UTC by Irek Szczesniak
Modified: 2005-07-30 23:23 UTC (History)
1 user (show)

See Also:
i915 platform:
i915 features:

a proposed fix (297 bytes, patch)
2005-06-03 10:44 UTC, Irek Szczesniak
no flags Details | Splinter Review

Description Irek Szczesniak 2005-05-27 10:40:28 UTC
First I execute:

> Xvfb :1 +extension Composite -ac

Then start some X application with DISPLAY=:1 and quit the application.  Xvfb
gets the segmentation fault when the application disconnects from it.

This is the output of Xvfb:

> Fri May 27, 11:10:55, ijs@some
> ~/local/X11/usr/X11R6/bin >./Xvfb :1 +extension Composite -ac
> _XSERVTransSocketOpenCOTSServer: Unable to open socket for inet6
> _XSERVTransOpen: transport open failed for inet6/some.example.com:1
> _XSERVTransMakeAllCOTSServerListeners: failed to open listener for inet6
> Could not init font path element /usr/X11R6/lib/X11/fonts/TTF/, removing from
> Could not init font path element /usr/X11R6/lib/X11/fonts/Type1/, removing >
from list!
> Could not init font path element /usr/X11R6/lib/X11/fonts/CID/, removing from
> Segmentation fault

This is the backtrace I get with the gdb debugger:

> This is the backtrace taken from the dumped core:
> Program received signal SIGSEGV, Segmentation fault.
> 0x080595f9 in FreeColormap ()
> (gdb) backtrace
> #0  0x080595f9 in FreeColormap ()
> #1  0x0804f4db in FreeClientResources ()
> #2  0x0804f5b0 in FreeAllResources ()
> #3  0x0804db97 in main ()
> #4  0x400bc62d in __libc_start_main () from /lib/libc.so.6

When Xvfb receives SIGTERM (when I enter ctrl+c or send SIGTERM with the kill
command), it gets the segmentation fault in the same place.

I compiled Xvfb on AIX 5.2 with the IBM tools, and the problem is there too. 
This is the output of the dbx debugger:

> Segmentation fault in FreeColormap at 0x10088f68 ($t1)
> 0x10088f68 (FreeColormap+0xd0) a0830002        lhz   r4,0x2(r3)
> (dbx) where
> FreeColormap() at 0x10088f68
> FreeClientResources_33_13() at 0x10027438
> FreeAllResources() at 0x1002635c
> main() at 0x10000a68
> (dbx) 

PLATFORMS TESTED: GNU/Linux 2.4.21-3, GNU/Linux 2.6.11-1.27_FC3, AIX 5.2
Comment 1 Adam Jackson 2005-05-29 21:30:16 UTC
possible dup of bug #3043.
Comment 2 Irek Szczesniak 2005-05-30 08:35:54 UTC
I got this additional information on GNU/Linux 2.6.11-1.27_FC3.

I get this bug when I compile the whole X.org tree with:
make World

However, Xvfb works without the bug if I compile the tree with CDEBUGFLAGS=-g:
make CDEBUGFLAGS=-g World

To make sure that the debug symbols do not influence the bug, I did:
strip Xvfb

The stripped Xvfb (previously compiled with the debug symbols) works without the
bug, so it seems like a workaroud for this problem, and I believe this can be my
last resort.

For both versions of Xvfb (the clean and the dirty one) I did:
strace -o output.txt ./Xvfb -ac :1 +extension Composite

Then I diffed the outputs and got:

< brk(0)                                  = 0x84c2000
> brk(0)                                  = 0x83a1000
< brk(0)                                  = 0x84c2000
< brk(0x84e3000)                          = 0x84e3000
> brk(0)                                  = 0x83a1000
> brk(0x83c2000)                          = 0x83c2000
< getpgrp()                               = 5194
> getpgrp()                               = 5226
< getpid()                                = 5195
< write(0, "      5195\n", 11)            = 11
> getpid()                                = 5227
> write(0, "      5227\n", 11)            = 11
< rt_sigaction(SIGALRM, {0x808fb8e, [ALRM], SA_RESTORER, 0x5d28c8}, NULL, 8) = 0
> rt_sigaction(SIGALRM, {0x807ba44, [ALRM], SA_RESTORER, 0x5d28c8}, NULL, 8) = 0
< rt_sigaction(SIGHUP, {0x808ddd9, [HUP], SA_RESTORER, 0x5d28c8}, {SIG_DFL}, 8) = 0
< rt_sigaction(SIGINT, {0x808de05, [INT], SA_RESTORER, 0x5d28c8}, {SIG_DFL}, 8) = 0
< rt_sigaction(SIGTERM, {0x808de05, [TERM], SA_RESTORER, 0x5d28c8}, {SIG_DFL},
8) = 0
> rt_sigaction(SIGHUP, {0x807a4d4, [HUP], SA_RESTORER, 0x5d28c8}, {SIG_DFL}, 8) = 0
> rt_sigaction(SIGINT, {0x807a4f4, [INT], SA_RESTORER, 0x5d28c8}, {SIG_DFL}, 8) = 0
> rt_sigaction(SIGTERM, {0x807a4f4, [TERM], SA_RESTORER, 0x5d28c8}, {SIG_DFL},
8) = 0
< getppid()                               = 5194
< gettimeofday({1117465649, 345025}, NULL) = 0
> getppid()                               = 5226
> gettimeofday({1117465837, 960885}, NULL) = 0
< gettimeofday({1117465649, 345963}, NULL) = 0
> gettimeofday({1117465837, 961411}, NULL) = 0
< time(NULL)                              = 1117465649
> time(NULL)                              = 1117465837
< brk(0x8505000)                          = 0x8505000
> brk(0x83e4000)                          = 0x83e4000
< brk(0x8526000)                          = 0x8526000
< gettimeofday({1117465649, 771449}, NULL) = 0
> brk(0x8405000)                          = 0x8405000
> gettimeofday({1117465838, 98478}, NULL) = 0
< brk(0x8547000)                          = 0x8547000
> brk(0x8426000)                          = 0x8426000
< brk(0x856a000)                          = 0x856a000
> brk(0x8449000)                          = 0x8449000
< brk(0x858b000)                          = 0x858b000
> brk(0x846a000)                          = 0x846a000
< brk(0x85ad000)                          = 0x85ad000
> brk(0x848c000)                          = 0x848c000
< brk(0x85db000)                          = 0x85db000
> brk(0x84ba000)                          = 0x84ba000
< gettimeofday({1117465649, 826551}, NULL) = 0
< gettimeofday({1117465649, 826713}, NULL) = 0
< gettimeofday({1117465649, 826883}, NULL) = 0
> gettimeofday({1117465838, 129578}, NULL) = 0
> gettimeofday({1117465838, 129615}, NULL) = 0
> gettimeofday({1117465838, 129668}, NULL) = 0
< close(0)                                = 0
< close(1)                                = 0
< unlink("/tmp/.X11-unix/X1")             = 0
< unlink("/tmp/.X1-lock")                 = 0
< munmap(0xb7ea9000, 1314816)             = 0
< exit_group(0)                           = ?
> --- SIGSEGV (Segmentation fault) @ 0 (0) ---
> +++ killed by SIGSEGV +++

We can see that both versions call the same system funcions and in the same
order.  I believe the different arguments to those system calls are ok. 
Therefore the bug is not related to system calls.
Comment 3 Irek Szczesniak 2005-05-31 08:55:00 UTC
I was wandering whether the "-g" option passed with CDEBUGFLAGS influences the
source files, i.e. whether there are some "#define" statements that depend on
"-g".  Therefore I had to compare the output of the preprocessor.  I modified
the build system, so that whenever a file is compiled, first a preprocessor is
run on this file, and the output is written to a file with some extension (and I
just chose irek, my name).  Then I built X.org with and without the "-g" option.
 As a result I got a lot of *.irek files, which I then diffed.  The problem was
that the differences were huge, and they produced a huge file.  Only the diff on
xc/programs/Xserver/composite/ gave me a file 80KB large.  When the differences
are so large, you don't know what to look for.

Bottom line: this method is not useful for debugging this problem.
Comment 4 Irek Szczesniak 2005-05-31 13:24:32 UTC
More information for GNU/Linux: I extracted X.org into two separate directories.
 I built one with "make World", the other with "make CDEBUGFLAGS=-g World". 
After the built were complete I had a look at the differences between the text
files in the two directories, and found that there were only some unimportant
differences.  I was hoping to see some differences in Makefiles, but found none.
Comment 5 Irek Szczesniak 2005-06-01 21:59:55 UTC
More information for GNU/Linux: the -O2 flag during the compilation of X.org
causes Xvfb to crash.

If you compile X.org with "make CDEBUGFLAGS=-g World", then the "-g" flag is
used.  If you compile with "make World", the default flags for compiling sources
are "-O2 -fno-strength-reduce -fno-strict-aliasing" among other options.

I played with these three options, and noticed that if I remove "-O2", the
compiled Xvfb doesn't crash.
Comment 6 Irek Szczesniak 2005-06-02 15:13:14 UTC
A workaround that works for me on GNU/Linux is putting this host.def file into
xc/config/cf with this contents:

#  define DefaultGcc2i386Opt -fno-strength-reduce GccAliasingArgs

This file override this setting:

#  define DefaultGcc2i386Opt -O2 -fno-strength-reduce GccAliasingArgs

That is, it removes the optimization flag.
Comment 7 Irek Szczesniak 2005-06-03 09:21:36 UTC
Using the -g flag or dropping the -O2 flag is not a solution, they simply
influence the problem.  When I change the code in some place, I get the crash
even though I compiled Xvfb with -g or without -O2.  My guess is that these
flags arrange some data/code in memory differently, so that even when we
reference some wrong memory (it's semgmentation fault), we are lucky to
reference memory that belongs to us anyway.  Otherwise (i.e. when I modifiy the
code, or play with the -g -O2 flags), we reference some memory owned by others
and Xvfb gets SIGSEGV.

At least I got Xvfb with debug symbol that crashes, so that I get the exact line
of code, which causes SIGSEGV: xc/programs/Xserver/dix/colormap.c:450

IT'S HERE:      if (--pent->co.shco.red->refcnt == 0)
                if (--pent->co.shco.green->refcnt == 0)
                if (--pent->co.shco.blue->refcnt == 0)

Comment 8 Irek Szczesniak 2005-06-03 10:44:51 UTC
Created attachment 2820 [details] [review]
a proposed fix
Comment 9 Irek Szczesniak 2005-06-03 10:45:56 UTC
This is the comment to the bug fix above.

I found one way of fixing the bug.  I am not sure that this fix is alright and
that it won't cause other problems.  When I compile and run Xvfb, it doesn't crash.

THE FIX: I added this line:
  pent->fShared = FALSE;

at xc/programs/Xserver/dix/colormap.c:456

I am attaching the patch for this file.  I came up with this fix when I saw
nearly the same block of code at xc/programs/Xserver/dix/colormap.c:735, but
these two blocks differ only with the line.  I added the line, and it works.
Comment 10 Irek Szczesniak 2005-06-26 10:12:07 UTC
The bug fix I submitted is wrong.  Unfortunately I don't have a bug fix.  But
when I demand the 16 bit color depth, the problem is gone, because the colormaps
are handled differently and Xvfb doesn't crash.  For me it's good enough.
Comment 11 Adam Jackson 2005-08-01 09:23:14 UTC
i can no longer reproduce this with CVS head.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.