Bug 73459

Summary: nv10 get_rt_format: Assertion `0' failed.
Product: Mesa Reporter: Diego Nieto Cid <dnietoc>
Component: Drivers/DRI/nouveauAssignee: Nouveau Project <nouveau>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: medium    
Version: git   
Hardware: x86 (IA32)   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:
Attachments: Test program which reproduces the crash
implement buffer validation callback

Description Diego Nieto Cid 2014-01-10 03:39:35 UTC 
Created attachment 91805 [details]
Test program which reproduces the crash

I found this error when trying to run DirectX programs in wine. During initialization the following error is reported:

----------------------------------------------------
    nv10_state_fb.c:51: get_rt_format: Assertion `0' failed.
----------------------------------------------------

Wine shows the following stack trace:

----------------------------------------------------
    Backtrace:
    =>0 0xb77c7424 __kernel_vsyscall+0x10() in [vdso].so (0x7c83e5f0)
      1 0xb742b5f6 gsignal+0x45() in libc.so.6 (0x7c83e5f0)
      2 0xb742cd83 abort+0x142() in libc.so.6 (0x7c83e5f0)
      3 0xb7424877 __assert_fail_base+0x156() in libc.so.6 (0x7c83e5f0)
      4 0xb7424927 __assert_fail+0x56() in libc.so.6 (0x7c83e5f0)
      5 0x7d77b664 nv10_emit_framebuffer+0x683() in nouveau_vieux_dri.so (0x7c83e5f0)
      6 0x7d76ec47 nouveau_state_emit+0x46() in nouveau_vieux_dri.so (0x7ca39ee8)
      7 0x7d4b0646 _mesa_update_state_locked+0x195() in nouveau_vieux_dri.so (0x00c10008)
      8 0x7d4b0ce4 _mesa_update_state+0x23() in nouveau_vieux_dri.so (0x00000000)
      9 0x7d429cd8 _mesa_Clear+0x257() in nouveau_vieux_dri.so (0x00000000)
      10 0x7e0db196 check_fbo_compat+0xacc(gl_info=0x1251a0, format=0x12eacc) [/home/diego/floss/wine/dlls/wined3d/utils.c:1204] in wined3d (0x0032f748)
      11 0x7e0dc625 init_format_fbo_compat_info+0x69a(gl_info=0x1251a0) [/home/diego/floss/wine/dlls/wined3d/utils.c:1438] in wined3d (0x0032f7e8)
      12 0x7e0de6c7 wined3d_adapter_init_format_info+0x7e(adapter=0x125190) [/home/diego/floss/wine/dlls/wined3d/utils.c:1984] in wined3d (0x0032f818)
      13 0x7e043952 wined3d_adapter_init+0x49f(adapter=0x125190, ordinal=0) [/home/diego/floss/wine/dlls/wined3d/directx.c:5115] in wined3d (0x0032fc28)
      14 0x7e043d87 wined3d_init+0xbc(wined3d=0x125180, version=0x8, flags=0x5) [/home/diego/floss/wine/dlls/wined3d/directx.c:5190] in wined3d (0x0032fc58)
      15 0x7e0ec0ca wined3d_create+0xd0(version=<couldn't compute location>, flags=<couldn't compute location>) [/home/diego/floss/wine/dlls/wined3d/wined3d_main.c:105] in wined3d (0x0032fcc8)
      16 0x7e162fdd d3d8_init+0x45(d3d8=0x122578) [/home/diego/floss/wine/dlls/d3d8/directx.c:413] in d3d8 (0x0032fd08)
      17 0x7e1575d1 Direct3DCreate8+0xb6(sdk_version=<couldn't compute location>) [/home/diego/floss/wine/dlls/d3d8/d3d8_main.c:47] in d3d8 (0x0032fd68)
      18 0x0042a55f in crimsonland (+0x2a55e) (0x0032fe20)
      19 0x7b86353c call_process_entry+0xb() in kernel32 (0x0032fe38)
      20 0x7b863689 start_process+0x14a(peb=<couldn't compute location>) [/home/diego/floss/wine/dlls/kernel32/process.c:1097] in kernel32 (0x0032fe98)
      21 0x7bc8557c call_thread_func_wrapper+0xb() in ntdll (0x0032feb8)
      22 0x7bc855c5 call_thread_func+0x3e(entry=0x7b86353e, arg=0x7ffdf000, frame=0x32ffb8) [/home/diego/floss/wine/dlls/ntdll/signal_i386.c:2602] in ntdll (0x0032ff98)
      23 0x7bc8555a call_thread_entry_point+0x11() in ntdll (0x0032ffb8)
      24 0x7bc58ff6 start_process+0x23(kernel_start=0x7b86353e) [/home/diego/floss/wine/dlls/ntdll/loader.c:2762] in ntdll (0x0032ffe8)
      25 0xb75f6f8d wine_call_on_stack+0x1c() in libwine.so.1 (0x00000000)
      26 0xb75f6f6b wine_switch_to_stack+0x2a(func=0x7bc58fd2, arg=0x7b86353e, stack=0x330000) [/home/diego/floss/wine/libs/wine/port.c:59] in libwine.so.1 (0xbfb7d9c8)
      27 0x7bc59323 LdrInitializeThunk+0x32c(kernel_start=<couldn't compute location>, unknown2=<couldn't compute location>, unknown3=<couldn't compute location>, unknown4=<couldn't compute location>) [/home/diego/floss/wine/dlls/ntdll/loader.c:2818] in ntdll (0xbfb7da58)
      28 0x7b863ecc __wine_kernel_init+0x67d() [/home/diego/floss/wine/dlls/kernel32/process.c:1269] in kernel32 (0xbfb7e918)
      29 0x7bc59b05 __wine_process_init+0x156() [/home/diego/floss/wine/dlls/ntdll/loader.c:3027] in ntdll (0xbfb7e978)
      30 0xb75f5800 wine_init+0x140(argc=0x2, argv=0xbfb7ee84, error="", error_size=0x400) [/home/diego/floss/wine/libs/wine/loader.c:952] in libwine.so.1 (0xbfb7e9b8)
      31 0x7bf01144 main+0x11e(argc=0x2, argv=0xbfb7ee84) [/home/diego/floss/wine/loader/main.c:237] in <wine-loader> (0xbfb7ede8)
      32 0xb7416993 __libc_start_main+0xf2() in libc.so.6 (0x00000000)
----------------------------------------------------

To get some information about how get_rt_format was called I applied the following patch to mesa:

----------------------------------------------------
    commit ccd7c55c5ea814a431d592865de490527528ba51
    Author: Diego Nieto Cid <dnietoc@gmail.com>
    Date:   Thu Dec 19 19:21:22 2013 -0300

        Print format before assert.

    diff --git a/src/mesa/drivers/dri/nouveau/nv10_state_fb.c b/src/mesa/drivers/dri/nouveau/nv10_state_fb.c
    index 87ae1d6..91b4ee7 100644
    --- a/src/mesa/drivers/dri/nouveau/nv10_state_fb.c
    +++ b/src/mesa/drivers/dri/nouveau/nv10_state_fb.c
    @@ -47,6 +47,7 @@ get_rt_format(gl_format format)
            case MESA_FORMAT_Z24_S8:
                    return NV10_3D_RT_FORMAT_DEPTH_Z24S8;
            default:
    +               printf("Unsupported format %d\n", format);
                    assert(0);
            }
     }
----------------------------------------------------

The output after running wine with the patched libraries now includes the following line:

----------------------------------------------------
    Unsupported format 16
----------------------------------------------------

Debugging wine I was able to get a trace of GL calls and I wrote a little GLUT program which succesfully reproduce the crash. It's source file is attached to the bug report.
Comment 1 Ilia Mirkin 2014-01-10 06:57:48 UTC 
Well, 32855 corresponds to GL_RGB5_A1. And mesa format 16 corresponds to MESA_FORMAT_ARGB1555, which isn't one of the supported render target formats. So that's all good. (So to speak.) Although it _is_ a supported texture format.

But I'm guessing there should be some logic somewhere that errors that out or creates some sort of conversion thing. Or glFramebufferTexture2D should fail.

Looking at the code, _mesa_FramebufferTexture2D calls framebuffer_texture, which in turn checks a few things about the params but never calls out to the driver, and then just sets the attachment. And all the completeness stuff largely ignores this stuff. I don't see how the driver is supposed to refuse... maybe it has to be done a different way.
Comment 2 Ilia Mirkin 2014-01-10 08:10:58 UTC 
Created attachment 91807 [details] [review]
implement buffer validation callback

Well, this "fixes" your test program. I haven't really tested whether this actually ever works... but give it a shot.
Comment 3 Diego Nieto Cid 2014-01-10 19:00:15 UTC 
Ilia thanks for the fast response. Yes, applying the patch prevents wine from crashing and at a quick glance games appear to work correctly.

There's an error message:

      Mesa: User error: GL_INVALID_ENUM in glDisable(GL_COLOR_SUM)

but I guess it's unrelated.
Comment 4 Ilia Mirkin 2014-01-10 19:05:38 UTC 
Awesome, I'll submit the patch for wider review. Out of curiousity, what games? Everything I tried (ok, like 2 things) on my NV18 failed pretty miserably. As for the glDisable(GL_COLOR_SUM) thing -- the application (well, I guess wine) is using that enum even though it's not enabled. I guess GL_COLOR_SUM only appeared with opengl-later-than-1.2 and/or some extension that's not enabled.
Comment 5 Diego Nieto Cid 2014-01-10 19:13:18 UTC 
I tried Fallout and Crimsonland. I got to the main menu but didn't actually played them.
The last one has some issues with the cursor, some region which should be transparent isn't. No idea why
Comment 6 Ilia Mirkin 2014-01-10 20:24:43 UTC 
Here's a fix for the GL_COLOR_SUM thing: http://lists.freedesktop.org/archives/mesa-dev/2014-January/051248.html
Comment 7 Diego Nieto Cid 2014-01-14 00:26:27 UTC 
Nice, I don't get the error anymore. Thanks Ilia!
Comment 8 Ilia Mirkin 2014-01-15 22:21:10 UTC 
A different version of the patch has been commited to mesa master. Please double-check that it also works for you (I tested against your test app, but not a real game). The GL_COLOR_SUM is also on its way to be checked in, and it's pretty minor for you, since it's only getting disabled (and it's off by default).

Feel free to reopen if this is still an issue.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.