If additional diagnostics is needed - don't hesitate to contact me debian AT onerussian.com
Thanks in advance

Created attachment 16860 [details] [review]
split client vertex array

The vertex buffer is too large and the memory manager fails to upload the vertex buffer.
This patch tries to split the vertex buffer on 965.

Hi, Halchenko  
    Could you try this patch?
   
Thanks.
Haihao

Created attachment 16873 [details]
backtraces after applying 16860: split client vertex array

patched mesa sources, installed patched packages onto my debian system, but problem persisted and it seems that arguments (from visual inspection) stay the same in a fresh backtrace... I am pretty sure that I patched mesa properly before building... could I be wrong?

Seems it didn't split the vertex buffer after applying this patch. Could you try a smaller upload buffer?

modify the following in the patch:
brw->vb.upload.max_size = brw->intel.intelScreen->tex.size >> 4.

>modify the following in the patch:
>brw->vb.upload.max_size = brw->intel.intelScreen->tex.size >> 4.
I am sorry -- I might be slow, but modify that to what? the line is already like that:
*$> grep 'vb.upload.max_size =' patch1.patch   mesa-7.0.3/src/mesa/drivers/dri/i965/brw_draw.c
patch1.patch:+   brw->vb.upload.max_size = brw->intel.intelScreen->tex.size >> 4;
mesa-7.0.3/src/mesa/drivers/dri/i965/brw_draw.c:   brw->vb.upload.max_size = brw->intel.intelScreen->tex.size >> 4;

or what it should be modified to?

(In reply to comment #5)
> >modify the following in the patch:
> >brw->vb.upload.max_size = brw->intel.intelScreen->tex.size >> 4.
> I am sorry -- I might be slow, but modify that to what? the line is already
> like that:
> *$> grep 'vb.upload.max_size =' patch1.patch  
> mesa-7.0.3/src/mesa/drivers/dri/i965/brw_draw.c
> patch1.patch:+   brw->vb.upload.max_size = brw->intel.intelScreen->tex.size >>
> 4;
> mesa-7.0.3/src/mesa/drivers/dri/i965/brw_draw.c:   brw->vb.upload.max_size =
> brw->intel.intelScreen->tex.size >> 4;
> 
> or what it should be modified to?
> 
I mean you can replace 4 with 6 (or 7, 8 etc.)

ok -- I tried it with 6 and 8: now it got a bit further: it rendered image once and then stall at the same point. traceback will follow below. But let me bring post-mortem debug analysis since it might be helpful:

(gdb) p size
$20 = 11796480
(gdb) p element_size
$21 = 12
(gdb) p map
$23 = (GLubyte *) 0x0
(gdb) print brw->vb.upload.max_size
$25 = 131072
If wonder if map=0x0 is the one to blame?
for the cause I am leaving it up to you to figure out since I have no clue in what I am doing... or should I try >> 9, >> 10, etc?

ok -- backtrace (top part)

#0  0x00002b026c489dbb in copy_array_to_vbo_array (brw=0x1f39ae0, 
    i=<value optimized out>, array=0x1f69f78, element_size=12, count=983040)
    at brw_draw_upload.c:263
        map = (GLubyte *) 0x0
        ctx = <value optimized out>
        size = 11796480
        vbo = (struct gl_buffer_object *) 0x265f470
        offset = <value optimized out>
        __PRETTY_FUNCTION__ = "copy_array_to_vbo_array"
#1  0x00002b026c48a2fa in brw_upload_vertices (brw=0x1f39ae0, min_index=0, 
    max_index=983039) at brw_draw_upload.c:478
        input = (struct brw_vertex_element *) 0x1f4e168
        tmp = 0
        vep = {header = {length = 0, opcode = 0}, ve = {{ve0 = {
        src_offset = 0, pad = 0, src_format = 0, pad0 = 0, valid = 0, 
        vertex_buffer_index = 0}, ve1 = {dst_offset = 0, pad = 0, 
        vfcomponent3 = 0, vfcomponent2 = 0, vfcomponent1 = 0, 
        vfcomponent0 = 0}} <repeats 18 times>}}
        vbp = {header = {bits = {length = 0, opcode = 0}, dword = 0}, vb = {{
      vb0 = {bits = {pitch = 0, pad = 0, access_type = 0, vb_index = 0}, 
        dword = 0}, buffer = 0x0, offset = 0, max_index = 0, 
      instance_data_step_rate = 0} <repeats 17 times>}}
        i = 2
        ptr = (const void *) 0x2b02709d7060
        interleave = 0
        enabled = {0x1f4e128, 0x1f4e168, 0x1f4e188, 0xeffff00000000, 
  0x2b026c489960, 0x7fff52b00c90, 0xdfff200000404, 0x1000e, 0x0, 
  0x2b026c493330, 0x24, 0x1f4e8ec, 0x1f4dad8, 0x8, 0x7, 0x2b026c4933f7, 0x0, 
  0x0, 0x0, 0x2b026c493330, 0x20, 0x7fff52b00b40, 0x1f4d778, 0x0, 
  0x2b026c890d40, 0x2b026c4933f7, 0x0, 0x0, 0x0, 0x12c000000000, 0x1ff0c40, 
  0x1f39ae0}
        nr_enabled = 3
        upload = {0x1f4e128, 0x1f4e168, 0x1f4e188, 0x2b026c4899cf, 0x1000d, 
  0x2b026c489960, 0x7fff52b00ab0, 0xfffffffc00000004, 0x0, 0x7fff52b00a8c, 
  0x100000001, 0x7fff52b00a20, 0x1, 0x1000e, 0xf0000, 0x7fff52b00cf0, 0xf0000, 
  0x2b026c5219e0, 0x1ff000effff, 0x1, 0xfff2, 0x2b026c521e0d, 0x4000e0012, 
  0x8382f8, 0x7fff52b00c10, 0x7fff52b00bf0, 0x7fff52b00c2c, 0x7fff52b00c28, 
  0x7fff52b00a20, 0x1000e00000001, 0x1f39ae0, 0x1f93ba8}
        nr_uploads = 3
        __PRETTY_FUNCTION__ = "brw_upload_vertices"
#2  0x00002b026c489779 in brw_try_draw_prims (ctx=0x1f39ae0, arrays=0x1f93ba8, 
    prim=0x7fff52b00cf0, nr_prims=1, ib=0x0, min_index=0, max_index=983039)
    at brw_draw.c:319
        intel = <value optimized out>
        brw = (struct brw_context *) 0x0
        retval = <value optimized out>
        i = <value optimized out>
        j = <value optimized out>
        __PRETTY_FUNCTION__ = "brw_try_draw_prims"
        __FUNCTION__ = "brw_try_draw_prims"
#3  0x00002b026c489a4e in brw_draw_prims (ctx=0x1f39ae0, arrays=0x1f93ba8, 
    prim=0x7fff52b00cf0, nr_prims=1, ib=0x0, min_index=0, max_index=983039)
    at brw_draw.c:478
        intel = <value optimized out>
        retval = <value optimized out>
#4  0x00002b026c51f9bb in vbo_exec_DrawArrays (mode=4, start=0, count=983040)
    at vbo/vbo_exec_array.c:264
        ctx = (GLcontext *) 0x0
        prim = {{mode = 4, indexed = 0, begin = 1, end = 1, weak = 0, pad = 0, 
    start = 0, count = 983040}}
#5  0x00002b025a566295 in __glXDisp_DrawArrays (
    pc=0x2b02709d7060 "@\021@a\027IBѿ\200\031\026?\222N?#*\207=>>>\021@HB\0375\032?QK?GW=>>>k\035@[zHB}L\027?3\031N?ZO=>>>@\021@a\027IBѿ5\032?QK?GW=>>>k\035@[zHB}M6\032?\231\035L?\030=>>>H+\030@\224HB^俠L\027?3\031N?"...)
    at ../../../GL/glx/render2.c:248
        datatype = 5126
        numVals = 3
        numComponents = 3
        primType = 4
        stride = 36
        i = 3
#6  0x00002b025a547da6 in DoRenderLarge (cl=0x135b620, 
    pc=0x2911160 "\212ɾ\201O:$ݾ", do_swap=0)
    at ../../../GL/glx/glxcmds.c:2055
        proc = (__GLXdispatchRenderProcPtr) 0xa0
        client = (ClientPtr) 0x1482380
        dataBytes = 3780

probably you know that already, but without DRI (option "DRI" "off") the bug is not exercised. Is there anything else I could help troubleshoot?

Setting INTEL_DEBUG=buf in your environment will detail how to allocate graphics memory  for all objects, Could you try with it and attach the result?

ok - did export INTEL_DEBUG=buf in debian/rules. But then what to look for? 

Shouldn't it be enabled with some other debug options (e.g. DEBUG_DRI, DEBUG_TEXTURE) since judging from first look at ifdefs INTEL_DEBUG is never capable to trigger additional actions alone:

*$> grep -h -r INTEL_DEBUG * | sed -e 's/^ *//g' -e 's/ *$//g' | sort | uniq 
#define DBG(...)  do { if (INTEL_DEBUG & DEBUG_BUFMGR) _mesa_printf(__VA_ARGS__); } while(0)
#define DBG(...)  do { if (INTEL_DEBUG & FILE_DEBUG_FLAG) _mesa_printf(__VA_ARGS__); } while(0)
#define INTEL_DEBUG             0
extern int INTEL_DEBUG;
I810_DEBUG |= driParseDebugString( getenv( "INTEL_DEBUG" ),
if (intel->aub_file && (INTEL_DEBUG & DEBUG_SYNC)) {
if (INTEL_DEBUG) {
if (INTEL_DEBUG & DEBUG_BATCH)
if (INTEL_DEBUG & DEBUG_BLIT)
if (INTEL_DEBUG & DEBUG_BUFMGR)
if (INTEL_DEBUG & DEBUG_DMA)
if (INTEL_DEBUG & DEBUG_DRI)
if (INTEL_DEBUG&DEBUG_DRI)
if(INTEL_DEBUG&DEBUG_DRI)
if (INTEL_DEBUG & DEBUG_FALLBACKS)
if (INTEL_DEBUG & DEBUG_IOCTL)
if (INTEL_DEBUG & DEBUG_LOCK)
         if (INTEL_DEBUG & DEBUG_PIXEL)
if (INTEL_DEBUG & DEBUG_PIXEL)
if (INTEL_DEBUG & DEBUG_PRIMS)
if (INTEL_DEBUG & DEBUG_SINGLE_THREAD)
if (INTEL_DEBUG & DEBUG_STATE)
if (INTEL_DEBUG & DEBUG_STATS)
if (INTEL_DEBUG & DEBUG_STATS || intel->stats_wm)
if (INTEL_DEBUG & DEBUG_SYNC) {
               if (INTEL_DEBUG & DEBUG_TEXTURE)
                  if (INTEL_DEBUG & DEBUG_TEXTURE)
if (INTEL_DEBUG & DEBUG_TEXTURE)
if(INTEL_DEBUG&DEBUG_TEXTURE)
if (INTEL_DEBUG & DEBUG_URB)
         if (INTEL_DEBUG & (DEBUG_URB|DEBUG_FALLBACKS))
if (INTEL_DEBUG & DEBUG_VERTS)
if (INTEL_DEBUG & DEBUG_VS) {
if (INTEL_DEBUG & DEBUG_WM) {
#ifndef INTEL_DEBUG
INTEL_DEBUG |= driParseDebugString( getenv( "INTEL_DEBUG" ),
INTEL_DEBUG  = driParseDebugString( getenv( "INTEL_DEBUG" ),
INTEL_DEBUG = driParseDebugString(getenv("INTEL_DEBUG"), debug_control);
int INTEL_DEBUG = (0);

So, now it crashed as usual with
#0  0x00007fdde7d0ec04 in copy_array_to_vbo_array (brw=0x3179160, i=<value optimized out>, 
    array=0x31a0258, element_size=12, count=983040) at brw_draw_upload.c:263
        map = (GLubyte *) 0x0
        ctx = <value optimized out>
        size = 11796480
        vbo = (struct gl_buffer_object *) 0x3522a20
        offset = <value optimized out>
        __PRETTY_FUNCTION__ = "copy_array_to_vbo_array"

Full bt is at 
http://www.onerussian.com/Linux/bugs/xorg.crash/Xorg.INTEL_DEBUG.log

I built it with all the changes we already went through previousely but on top of debian's 7.0.3-5 release of mesa which listed following changes in the last releases (since -1)

 * Disable the i915tex driver, it doesn't build against libdrm 2.3.1.
  * Pull from mesa_7_0_branch (27425708).
mesa (7.0.3-4) unstable; urgency=low
  * Pull from mesa_7_0_branch (2ac4919d).
  * Put back our configs/ changes into the .diff.gz since choose-configs
    needs them before quilt is invoked. Put 04_cleanup-osmesa-configs.patch
    there as well for #485161.
mesa (7.0.3-3) unstable; urgency=low
  * Pull from mesa_7_0_branch (718724de).
    + Fix intel_batchbuffer_space on i965, closes: #455817.
    + Fix busy error in i915_wait_irq for real now, closes: #467319.
  * Move our configs/ changes from the .diff.gz into our quilt patches,
    with 04_cleanup-osmesa-configs.patch renamed into 04_debian-configs.patch,
    closes: #485161.
mesa (7.0.3-2) unstable; urgency=low
  * Pull from mesa_7_0_branch (03447de3).
  * Set right cliprects for the current draw region on Intel, closes: #467319.
  * Use BRW_TEXCOORDMODE_CLAMP instead of BRW_TEXCOORDMODE_CLAMP_BORDER
    to implement GL_CLAMP on i965, closes: #478880.
  * Fix segment fault with BASE_LEVEL set to 5 for MipMap on i915,
    closes: #451339.
  * Disable low impact fallback on r300 by default, closes: #440868.


Full diff (from release 7.0.3) for my build is available from
http://www.onerussian.com/Linux/bugs/xorg.crash/mesa_7.0.3-6~1.diff.gz

actually I missed the single point where DEBUG_INTEL is used alone in 
mesa/drivers/dri/i965/brw_state_upload.c ;-) but I guess if it detected anything I would see assertion failed (not segfault). Also is there a way to doublecheck if DEBUG_INTEL=buf was in effect? any log message anywhere?

full fresh xorg log is at
http://www.onerussian.com/Linux/bugs/xorg.crash/Xorg.0.log

not sure if it would be of any help but
http://www.onerussian.com/Linux/bugs/xorg.crash/Xorg.valgrind.log
shows valgrind output for me starting X with
 sudo valgrind /usr/bin/Xorg :0 -ac

and then 
1. opening xterm (there are some msgs from valgrind on that action).
2. closing xterm (followed with fresh Wacom General ...)
3. opening freesurfer which leads to fresh reports and segfault at the end with bt you already familiar with:

==11090== Invalid write of size 1
==11090==    at 0x1962FC04: copy_array_to_vbo_array (brw_draw_upload.c:263)
==11090==    by 0x1962FEFE: brw_upload_vertices (brw_draw_upload.c:478)
==11090==    by 0x1962F61F: brw_try_draw_prims (brw_draw.c:319)
==11090==    by 0x1962F91E: brw_draw_prims (brw_draw.c:478)
==11090==    by 0x196D0227: vbo_exec_DrawArrays (vbo_exec_array.c:264)
==11090==    by 0x789599B: __glXDisp_DrawArrays (render2.c:248)
==11090==    by 0x7879D24: DoRenderLarge (glxcmds.c:2055)
==11090==    by 0x787D874: __glXDispatch (glxext.c:561)
==11090==    by 0x44F801: Dispatch (dispatch.c:502)
==11090==    by 0x436BD4: main (main.c:452)
==11090==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

ok... a bit more... according to valgrind segfault happens at line 263 which points to the the body of the loop within copy_strided_array:
    for (i = 0; i < count; i++) {
     for (j = 0; j < size; j++)
        *dest++ = *src++;
     src += (stride - size);
      }


stride value got optimized but from where it is called it seems to correspond to
array->StrideA
and here is the reason for the problem:

(gdb) p array->StrideB
$18 = 36
(gdb) p size
$19 = 1572840

so -- we are shifting through 'src' backwards since stride is much smaller than size for some reason and hence sooner or later we arrive to 0x0 address:

src is opt out from 
(gdb) p &(array->Ptr)
$28 = (const GLubyte **) 0x19a12a8

actually we should need just
p (int)&(array->Ptr) / (size-array->StrideB)
steps to get there ;-) ... although I must confess I am not entirely sure since it might be trickery of optimization since after segfault values seems to be sane... trying to build the beast without optimization now

ok -- it has nothing to do with sizes explicitely... without optimization I get:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f4ddd39e6e0 (LWP 5000)]
0x00007f4dc890822a in copy_strided_array (dest=0x0, 
    src=0x7f4dc21ab054 ">>>@\021@a\027IBѿ\200\031\026?\222N?#*\207=>>>\021@HB\0375\032?QK?GW=>>>k\035@[zHB}L\027?3\031N?ZO=>>>@\021@a\027IBѿ5\032?QK?GW=>>>k\035@[zHB}M6\032?\231\035L?\030=>>>H+\030@\224HB"..., size=12, stride=36, count=983040) at brw_draw_upload.c:263

so dest is 0x0 right from the beginning

FWIW: I added simple return on dest==0x0 and then it crashes with
#0  0x00007fca2449d1d5 in raise () from /lib/libc.so.6
No symbol table info available.
#1  0x00007fca2449e680 in abort () from /lib/libc.so.6
No symbol table info available.
#2  0x00007fca2449675f in __assert_fail () from /lib/libc.so.6
No symbol table info available.
#3  0x00007fca1143d520 in intel_bufferobj_unmap (ctx=0x2127f50, target=34962, obj=0x2424ae0) at intel_buffer_objects.c:183
        intel = (struct intel_context *) 0x2127f50
        intel_obj = (struct intel_buffer_object *) 0x2424ae0
        __PRETTY_FUNCTION__ = "intel_bufferobj_unmap"
#4  0x00007fca1145859b in copy_array_to_vbo_array (brw=0x2127f50, i=2, array=0x214d358, element_size=12, count=983040)
    at brw_draw_upload.c:356
        map = (GLubyte *) 0x0
        ctx = (GLcontext *) 0x2127f50
        vbo_array = (struct gl_client_array *) 0x213bff0
        size = 11796480
        vbo = (struct gl_buffer_object *) 0x2424ae0
        offset = 0
        new_stride = 12
        __PRETTY_FUNCTION__ = "copy_array_to_vbo_array"


assert seems to be at
   assert(obj->Pointer);

in
static GLboolean intel_bufferobj_unmap( GLcontext *ctx,
                    GLenum target,
                    struct gl_buffer_object *obj )
{
   struct intel_context *intel = intel_context(ctx);
   struct intel_buffer_object *intel_obj = intel_buffer_object(obj);

   assert(intel_obj);
   assert(intel_obj->buffer);
   assert(obj->Pointer);
   bmUnmapBufferAUB(intel, intel_obj->buffer, 0, 0);
   obj->Pointer = NULL;
   return GL_TRUE;
}

I guess you are running some manager such compiz. Could you start X by hand? 
Then run your apps,  eg glxgears

export INTEL_DEBUG=buf
glxgears 2>&1 | tee buf.log

It will log all information about graphics memory in buf.log

doh ... why did I think DEBUG_INTEL should be set at build time? :)
but I do not run compiz (kde + wmaker)... will do your test on my problematic software tomorrow -- thanks for the hint

uff... apparently with this application it never gets to any of debug outputs for INTEL_DEBUG=buf. At first I thought that may be apps redirects stderr somewhere, but that is not the reason -- I've set breakpoints at printf and _mesa_printf, which get triggered whenever I run glxgears with INTEL_DEBUG=buf, but for the application of interest -- it never gets there -- thus nothing is printed

May be your apps create a indirect context for rendering, and the backtrace you attached also tell us the rendering is done by X.

may be -- I have not knowledge about internals of the software, but the crash happens within DRI intel module -- what other troubleshooting I could do to provide you with information to boil down the problem?

This issue should go away with mesa master. 
Mark it as 'WONTFIX' because it only happens with 7_0 branch.

how do you know that it doesn't happen with mesa master????

On Wed, 10 Dec 2008, bugzilla-daemon@freedesktop.org wrote:

> http://bugs.freedesktop.org/show_bug.cgi?id=16056

> --- Comment #21 from haihao <haihao.xiang@intel.com>  2008-12-10 23:06:52 PST ---
> This issue should go away with mesa master. 
> Mark it as 'WONTFIX' because it only happens with 7_0 branch.

Maybe I should say "This issue should go away with GEM". 

Under legacy mode, only 32M graphics memories are allocated for 3D operations, including vertex buffers, texture buffers, state buffers... Obviously  the fake memory manager fails to get free space for this vertex buffer which is too large (Some memories are allocated for other objects) in this case.  Under GEM mode, the limitation of 32M doesn't exist any more, so I am sure this issue should go away. Could you try it with GEM enabled? If it still exists, feel free to reopen it.