Could you provide a stack trace from when Xnest aborts?

Are you citing the version of Xnest, or the version of the real server that Xnest is connecting to, or both? This assert has previously caught a server bug, so both versions may matter.

Did Xnest abort immediately on startup, or did you connect some client to it first? Please provide more details about the circumstances. Thanks!

Both Xnest and the real server are 1.7.6. Xnest abort happens
immediately.

stacktrace:
---------------------------------------------------
Program received signal SIGABRT, Aborted.
0x00007ffff4efdc45 in raise () from /lib/libc.so.6
(gdb) bt
#0  0x00007ffff4efdc45 in raise () from /lib/libc.so.6
#1  0x00007ffff4eff811 in abort () from /lib/libc.so.6
#2  0x00007ffff4ef6d79 in __assert_fail () from /lib/libc.so.6
#3  0x00007ffff78e2653 in _XReply () from /usr/lib/libX11.so.6
#4  0x00007ffff78bf80f in XGetKeyboardControl () from /usr/lib/libX11.so.6
#5  0x000000000042081e in xnestKeyboardProc (pDev=0x9be200, onoff=0)
    at Keyboard.c:189
#6  0x000000000055aae0 in ActivateDevice (dev=0x9be200, sendevent=1 '\001')
    at devices.c:470
#7  0x000000000055b005 in InitAndStartDevices () at devices.c:640
#8  0x00000000005a803b in main (argc=2, argv=0x7fffffffe318, 
    envp=0x7fffffffe330) at main.c:258
---------------------------------------------------

I use my own hacked keyboard. (That's why I use Xnest, instead of
Xephyr which doesn't accept keyboard configuration.) The real server
works ok with it, but when I invoke Xnest, the real server startx
reports:
----------------------------------------------------
> Warning:          Duplicate shape name ""
>                   Using last definition
(repeated many times)
> Warning:          Multiple doodads named ""
>                   Using first definition
(repeated many times)
Errors from xkbcomp are not fatal to the X server
----------------------------------------------------

setxkbmap and xkbcomp are new, too. My keyboard driver is evdev, and
in Gentoo Linux which I use, evdev configuration is passed from
hal. Upgrading of xorg-server is done in Gentoo way, so I'm not going
my own cowboy way.

Sorry if my bug report is clumsy, I'm not skilled. (I though it's
better go upstream directly for such delicate issue, rather than
Gentoo.)

Thanks a lot for your kind reply.

It still fails with xorg-server 1.9.2, with libX11-1.4.0.

And in OpenOffice, assertions fail in _XReply too. (This is the first
try of OpenOffice in a few years.) It reproduces always, but it
depends where an assertion fails among OOo components. For example,
the call of XGetWindowProperty breaks in oowriter, XGrabPointer in
oocalc.

It's peculiar that similar reports can't be found on the web. (Many
OpenOffice failure reports in _XReply are done, but not recent, and
they're different from mine.)

I'd appreciate any suggestion very much. Thanks.

Created attachment 42958 [details]
Full backtrace

`XkbGetKeyboardByName' seems to be the culprit, both in Xnest and
OpenOffice.

In xnest, in the function xnestKeyboardProc() in hw/xnest/Keyboard.c,
XkbGetKeyboard is called, and it returns NULL. Then immediately jump to
"XkbError:" with goto, and XGetKeyboardControl() is called, then it
breaks. Below I pasted the "bt full" result for both function calls.
(XkbGetKeyboard is a wrapper for XkbGetKeyboardByName.)

In OpenOffice too, XkbGetKeyboardByName seems to trigger the failure.
Within the (seemingly) next 1-3 calls of xlib functions after
XkbGetKeyboardByName, it crashes. More precisely, I set _XReply as a
breakpoint after the first call of XkbGetKeyboardByName, and it happens
after first several calls of _XReply.

What can I do to investigate more? I'd appreciate any help / suggestion.

With best regards.

I found the cause.

I think the bug is still valid and it's better to fix it, but I apologize you that the direct cause is my personal setting, and I
reported it without a thorough check.

Now the description: if files in xkb/types/* lack "level_name" entries for corresponding "map" item, a client will crash when it calls an xlib function after a call of XkbGetKeyboardByName. X server itself starts, and doesn't crash. See comment #4 for more.

For example, I have xkb/types/my which reads

partial default xkb_types "default" {
   type "Lv3_Control" {
       modifiers = Shift+Control;
       map[None] = Level1;
       map[Shift] = Level2;
       map[Control] = Level3;
       map[Control+Shift] = Level4;
       // Withot the four lines below, it will fail.
       level_name[Level1] = "Level1";
       level_name[Level2] = "Level2";
       level_name[Level3] = "Level3";
       level_name[Level4] = "Level4";
   };
};

It's a regression, since it used to work without "level_name" at the beginning of year 2010.

I modified the title of this bugzilla item to describe the bug better. (It refered to xnest, but it's not related.)

I myself develop an open source, and I really thank you for developing X. With best regards.

Removing XCB list since it's in libX11's XKB internals, I guess. Thanks for tracking the problem down further. Daniel's already on the Cc list and I'm going to hope he takes care of it. :-)

Mass closure: This bug has been untouched for more than six years, and is not obviously still valid. Please file a new report if you continue to experience issues with a current server.