I have found that sometimes xorg-server crashes when qemu grabs mouse and keyboard input (it happens when you click in its window). I didn't find any other conditions which cause crash, so I suppose that it occurs randomly. I have observed this crash twice, once xorg-server printed a backtrace, and once it didn't. But both times there was the following error reported:

Fatal server error:
bogus pointer event from ddx

The backtrace:

Backtrace:
0: /usr/bin/X(xf86SigHandler+0x80) [0x80bf650]
1: [0xb7f13420]
2: /usr/bin/X(XkbRemoveResourceClient+0xb8) [0x8187bb8]
3: /usr/bin/X [0x807eb72]
4: /usr/bin/X(CloseDownDevices+0x38) [0x807edd8]
5: /usr/bin/X(AbortServer+0x23) [0x81b4be3]
6: /usr/bin/X(FatalError+0x67) [0x81b5107]
7: /usr/bin/X(CoreProcessPointerEvent+0x1c5) [0x80923a5]
8: /usr/bin/X(ProcessPointerEvent+0x6e) [0x81895be]
9: /usr/bin/X(mieqProcessInputEvents+0x137) [0x8116777]
10: /usr/bin/X(ProcessInputEvents+0x2c) [0x80bff2c]
11: /usr/bin/X(Dispatch+0x71) [0x80866f1]
12: /usr/bin/X(main+0x495) [0x806dda5]
13: /lib/libc.so.6(__libc_start_main+0xd8) [0xb7c56838]
14: /usr/bin/X(FontFileCompleteXLFD+0xa1) [0x806d0c1]

FatalError re-entered, aborting
Caught signal 11.  Server aborting

I'm using xorg-server-1.4-r2 from gentoo with this patch (which fixes broken keyboard leds) applied:
https://bugs.freedesktop.org/show_bug.cgi?id=12434#c17