Created attachment 68429 [details]
Xorg.0.log

Created attachment 68430 [details]
dmesg when run xonotic

That Xorg.log doesn't match the versions you stated, which I note do not include the ddx...

Sorry, I make a mistake. The right environment is as following which matches the Xorg.log. 

---------------------------------------------------------
Libdrm:(master)libdrm-2.4.39-18-g844d75e5a0b3b8f3466a24256955e886275fb298
Mesa:  (9.0)8e73273cb95626d9def012eeb508160e4022ff60
Xserver:(server-1.12-branch)xorg-server-1.12.4-2-g78c77356c5d88 Xf86_video_intel:(master)2.20.9-47-gd73f5b5bb1a81421f1fdc3ac3
Kernel:	(drm-intel-fixes) 5713a6cfd335dd665400de95fb645ea3d45f27f5

Just to clarify my point, from the Xorg.0.log:

48.867] (II) LoadModule: "intel"
[    48.867] (II) Loading /opt/X11R7/lib/xorg/modules/drivers/intel_drv.so
[    48.870] (II) Module intel: vendor="X.Org Foundation"
[    48.870] 	compiled for 1.11.4, module version = 2.17.0
[    48.870] 	Module class: X.Org Video Driver
[    48.870] 	ABI class: X.Org Video Driver, version 11.0

which is definitely not the version you think you are testing.

Run xonotic by xonotic-linux64-glx or xonotic-linux64-sdl(bug 55829).
This problem can be only reproduced by xonotic-linux64-glx. 
Please see xonotic_Xorg.0.log with above version.

Created attachment 68596 [details]
Xorg.0.log of xonotic

Created attachment 68597 [details]
Xorg.0.log when run xonotic

Well that is just bizarre. Care to attach an xtrace so that we can get a dump of the commands sent?

Created attachment 68893 [details]
xtrace for xonotic( xtrace -n -D:9 ./xonotic-linux64-glx)

Fantastic, the XVidMode extension doesn't bother to fill in the error correctly. So a couple of patches to apply to the xserver and then repeat the test with xtrace...

Created attachment 68907 [details] [review]
Fix BadValue if no currentMode set

Created attachment 68908 [details] [review]
Use errorValue to report failing test

Created attachment 68938 [details]
xtrace for xonotic when xserver with above patches

Ok, that means that we found a valid mode, but the actual modeset failed (VidModeSwitchMode). Now to dig deeper to see how that could fail silently.

No obvious failures that might correspond with the silence in the Xorg.log. Can you attach gdb to Xorg and set a breakpoint on VidModeSwitchMode and step through until you see the return FALSE?

(In reply to comment #16)

Hi.It seems that Xorg don't run to breakpoint VidModeSwitchMode.
-----------------------------
gdb /opt/X11R7/bin/Xorg
(gdb) b VidModeSwitchMode
Breakpoint 1 at 0x486160: file xf86VidMode.c, line 327.
(gdb) r
...
[tcsetpgrp failed in terminal_inferior: Operation not permitted]
Detaching after fork from child process 1163.
Detaching after fork from child process 1164.

Hmm, you may have had different line numbers to me. :|

Try breaking on ProcXF86VidModeSwitchToMode instead and stepping until you hit a FALSE.

(In reply to comment #18)

Try to break on ProcXF86VidModeSwitchToMode(Breakpoint 1 at 0x4a0500: file xf86vmode.c, line 1078), but the same as above.

That's a little unexpected as you successfully patched that function to change the reporting of the Error...

So far, the indication is that SwitchMode is failing due to DGA being active (for no apparent reason).

I think a bisection on the xserver is in order; there were a few changes made to DGA in the last 6 months or so that might be responsible.

(In reply to comment #21)

The problem still exists on xorg-server-1.10.3
----------------------------------------------
Libdrm:  (master)2.4.26-11-gc82ef03e4c92017bf5644f294ea04e30500f8d4c
Mesa:    (7.11)b9c7773e0dbf75b9bf5142382fa4174a9f2f569f
Xserver: (server-1.10-branch)xorg-server-1.10.3
Xf86_video_intel: (master)2.16.0-115-gc5414ec992d935e10156a2b513d5ec2dded2f689
Cairo:   (master)34f507a919b0709caa2c0be30e43719356293dd1
Kernel:  (drm-intel-fixes)b3f33cbf7ace8fc149993ee35e0d0fd57f41d6d8

Ok, how about we first confirm our diagnosis with:

diff --git a/hw/xfree86/common/xf86Cursor.c b/hw/xfree86/common/xf86Cursor.c
index 65a9e82..31cebab 100644
--- a/hw/xfree86/common/xf86Cursor.c
+++ b/hw/xfree86/common/xf86Cursor.c
@@ -206,7 +206,7 @@ xf86SwitchMode(ScreenPtr pScreen, DisplayModePtr mode)
         return FALSE;
 
 #ifdef XFreeXDGA
-    if (DGAActive(pScr->scrnIndex))
+    if (0 && DGAActive(pScr->scrnIndex))
         return FALSE;
 #endif

Test xorg-server-1.13.0 with above patch, pls see xtrace attached.
When set a breakpoint on VidModeSwitchMode, bt as following:
216    if (mode->HDisplay > pScr->virtualX || mode->VDisplay > pScr->virtualY)
217    return FALSE;
-------------------------------
#0  xf86SwitchMode (pScreen=0x1b921a0, mode=0x1b8c990) at xf86Cursor.c:217
#1  0x00000000004b4819 in VidModeSwitchMode (scrnIndex=0, mode=0x1b8c990) at xf86VidMode.c:340
#2  0x00000000004caddd in ProcXF86VidModeSwitchToMode (client=0x1edd550) at xf86vmode.c:1168
#3  0x00000000004cc198 in ProcXF86VidModeDispatch (client=0x1edd550) at xf86vmode.c:1673
#4  0x0000000000434726 in Dispatch () at dispatch.c:428
#5  0x000000000049916d in main (argc=2, argv=0x7fff2c2ac538, envp=0x7fff2c2ac550) at main.c:295

Created attachment 69592 [details]
xtrace for xonotic with above patch

Are we confident that it is indeed now returning FALSE due to the check of the request against the screen size?

What does xrandr report whilst xonotic is running? And afterwards?

(In reply to comment #26)
When run xonotic, display resolution is 1024x768(xonotic pre-set resolution). After set 1280x1080 in xonotic, game crash and display is still 1024x768.

I need to know what xrandr says the screen size is in addition to the display size.

Created attachment 70054 [details]
xrandr report whilst xonotic is running

Created attachment 70055 [details]
xrandr when crash

The problem only exists on gnome-session, not xinit and gnome-shell.

Ok, we have an explanation at last:

  if (mode->HDisplay > pScr->virtualX || mode->VDisplay > pScr->virtualY) return FALSE;

is the winner as gnome-shell (or one of its helpers) is adjusting the screen size upon the VidMode change. So the bug lies somewhere between VidMode not handling screen size in conjunction with display sizes, Xonotic using the legacy VidMode rather than RandR and gnome-shell's overly helpful behaviour of changing the screen size.

Should also file a bug against gnome-shell for them to investigate their behaviour.

This problem still exists on environment.
----------------------------
Kernel_version:     3.8
Libdrm:             2.4.42
Mesa:               (9.1)9.1-rc2
Xserver:            (server-1.13-branch)xorg-server-1.13.2
Xf86_video_intel:   (master)2.21.0
Cairo:              (master)1.12.12
Libva:              staging-20130205
Libva_intel_driver: staging-20130205

The problem doesn't exists on the driver.
-----------------
Mesa:        mesa-9.2
Kernel :     3.11
Libdrm:     2.4.46
Xf86_video_intel:    2.99.902
Libva:         libva-1.2.1
Libva_intel_driver:    (master)git-9f3fd62
Cairo:         1.12.16
Xserver:     xorg-server-1.14.2.902