Bug 53111

Created attachment 65108 [details]
dmesg of piglit r600.test crash

I also have the same issue, here is the dmesg of the crash I get when running the piglit test case r600.test.  This is with virtual address enabled and the patches from bug 45018 applied.

Small note to whoever could come here and was not following bug 45018:

Bisecting identified the following commit as culprit:

bb1f0cf3508630a9a93512c79badf8c493c46743 is the first bad commit
commit bb1f0cf3508630a9a93512c79badf8c493c46743
Author: Jerome Glisse <jglisse@redhat.com>
Date:   Fri Dec 2 10:20:29 2011 -0500

    r600g: add support for virtual address space on cayman v11

FWIW, r600.tests should no longer be used in favour of quick-driver.tests. I assume it still happens with the latter though, so here's some debugging tips:

For isolating a single piglit test that locks up, it may help to run piglit-run.py with -c 0 to prevent several tests from running in parallel.

For isolating the cause of a lockup, it may help to add some debugging output about virtual addresses to the r600g driver, and compare that to the fault address in the VM_CONTEXT1_PROTECTION_FAULT_ADDR register.

(In reply to comment #3)
> FWIW, r600.tests should no longer be used in favour of quick-driver.tests. I
> assume it still happens with the latter though, so here's some debugging tips:
> 
> For isolating a single piglit test that locks up, it may help to run
> piglit-run.py with -c 0 to prevent several tests from running in parallel.
> 
> For isolating the cause of a lockup, it may help to add some debugging output
> about virtual addresses to the r600g driver, and compare that to the fault
> address in the VM_CONTEXT1_PROTECTION_FAULT_ADDR register.

Your info will be helpful for piglit tests, I'll try that later. For the debug calls, I'll let someone else propose a patch so it is at the right spot.

Tested running one piglit test at a time (thanks Michel) and it always locks on "texturing/depthstencil-render-miplevels 146 s=z24_s8_d=z32f_s8". It locks hard, resets, stays locked and usually restarts the computer.

The fault address in the VM_CONTEXT1_PROTECTION_FAULT_ADDR register is less than the start of the virtual address area, unless that is due to the bug?

(In reply to comment #6)
> The fault address in the VM_CONTEXT1_PROTECTION_FAULT_ADDR register is less
> than the start of the virtual address area, unless that is due to the bug?

Sorry, should have mentioned that the address in VM_CONTEXT1_PROTECTION_FAULT_ADDR is shifted right by 12 bits (i.e. it's the page frame number).

Is there a way to use apitrace in combination with piglit? I'd like to trace the problematic test.

(In reply to comment #8)
> Is there a way to use apitrace in combination with piglit? I'd like to trace
> the problematic test.

The first step would be to reproduce the problem by manually running the problematic test from the piglit/bin directory. Then you should be able to apitrace it just like any other app.

Well, it seems running it through qapitrace doesn't lock. But running only this single test in a terminal does.

One thing though: when using qapitrace and looking up state, framebuffer under surfaces is pretty much garbage whatever stage I look at. I don't know if this is expected fom depthstencil-render-miplevels 146 s=z24_s8_d=z32f_s8.

Created attachment 65723 [details]
apitrace

I tried to trace RenderFeatTest (one of the other applications locking my system). It did as  with the piglit test: it didn't crash. However, the rendering is corrupted where it locks when launched from a terminal. Trace is 75MB when compressed if you want me to upload it somewhere.

(In reply to comment #12)
> I tried to trace RenderFeatTest (one of the other applications locking my
> system). It did as  with the piglit test: it didn't crash. However, the
> rendering is corrupted where it locks when launched from a terminal. Trace is
> 75MB when compressed if you want me to upload it somewhere.

I forgot to say: it doesn't lock anymore at all. I should have written "... where it locked when launched from a terminal". It was locking at test 7. I'm attaching a screenshot from that test.

I'll bisect to see if I can find which commit "fixed" the lock.

Created attachment 65813 [details]
bad rendereing on test 7, where it used to lock

(In reply to comment #10)
> Well, it seems running it through qapitrace doesn't lock.

The apitrace looks incomplete: it doesn't contain any actual rendering operations.

(In reply to comment #15)
> (In reply to comment #10)
> > Well, it seems running it through qapitrace doesn't lock.
> 
> The apitrace looks incomplete: it doesn't contain any actual rendering
> operations.

I'll rerun it at home tonight.

(In reply to comment #13)
> (In reply to comment #12)
> > I tried to trace RenderFeatTest (one of the other applications locking my
> > system). It did as  with the piglit test: it didn't crash. However, the
> > rendering is corrupted where it locks when launched from a terminal. Trace is
> > 75MB when compressed if you want me to upload it somewhere.
> 
> I forgot to say: it doesn't lock anymore at all. I should have written "...
> where it locked when launched from a terminal". It was locking at test 7. I'm
> attaching a screenshot from that test.
> 
> I'll bisect to see if I can find which commit "fixed" the lock.

I was not able to figure out the combination that fixed the thing. Well, let's focus on the piglit test that locks the beast.

(In reply to comment #16)
> (In reply to comment #15)
> > (In reply to comment #10)
> > > Well, it seems running it through qapitrace doesn't lock.
> > 
> > The apitrace looks incomplete: it doesn't contain any actual rendering
> > operations.
> 
> I'll rerun it at home tonight.

You were right, I had missed a ";" between the arguments. Bam, locked. I was unable to retrieve a trace. Well, I may try to run it in debug mode to see where it stops later this week.

So about this locking piglit test (depthstencil-render-miplevels 146 s=z24_s8_d=z32f_s8), I've been able to track it down to:
line 218: 		piglit_report_result(PIGLIT_SKIP);

I don't know if we are supposed to be hitting this path, but either way, it seems piglit_report_result(PIGLIT_SKIP) locks. I suppose this function must be releasing resources before exiting, but something wrong is happening in there.

By the way, I'm now running kernel 3.6.0-rc3 with latest drm and mesa.

(In reply to comment #19)
> So about this locking piglit test (depthstencil-render-miplevels 146
> s=z24_s8_d=z32f_s8), I've been able to track it down to:
> line 218:         piglit_report_result(PIGLIT_SKIP);

How did you determine that? It's weird, I wouldn't expect a skipped test to produce any actual GPU rendering.

(In reply to comment #20)
> (In reply to comment #19)
> > So about this locking piglit test (depthstencil-render-miplevels 146
> > s=z24_s8_d=z32f_s8), I've been able to track it down to:
> > line 218:         piglit_report_result(PIGLIT_SKIP);
> 
> How did you determine that? It's weird, I wouldn't expect a skipped test to
> produce any actual GPU rendering.

I used gdb and step into the code until it locked. It gets out at level 0, after going through:

 /**
 * Attach the proper miplevel of each texture to the framebuffer
 */
void
set_up_framebuffer_for_miplevel(int level)...

Before this call, there is a framebuffer initialization:
	GLuint fbo;
	glGenFramebuffers(1, &fbo);
	glBindFramebuffer(GL_DRAW_FRAMEBUFFER, fbo);
	glBindFramebuffer(GL_READ_FRAMEBUFFER, fbo);

	for (int level = 0; level <= max_miplevel; ++level) {
		set_up_framebuffer_for_miplevel(level);

It seems Marek has more weight than me about lockups related to VM on Cayman(problem first reported as bug 45018). Patch by Marek to disable VM by default for Cayman: http://lists.freedesktop.org/archives/mesa-dev/2012-August/026590.html

If you have any news on the subject, feel free to add info in the current bug. To Marek: are you experiencing the same first lockup in the piglit tests as reported in comment 10. I'm sure have seen a previous comment from another dev who was also experiencing lockups on Cayman, but I can't find who that was.

(In reply to comment #22)
> It seems Marek has more weight than me about lockups related to VM on
> Cayman(problem first reported as bug 45018). 

Well, we were hoping to get this resolved in time for 9.0, but as it's getting pretty close now, it's probably better to disable it at least for the 9.0 release.  The problem is, when it's disabled, there's not much chance of anyone testing it, so it's not likely to ever get properly fixed.  Also, SI only supports VM, so we can't disable VM for SI.

(In reply to comment #23)
> (In reply to comment #22)
> > It seems Marek has more weight than me about lockups related to VM on
> > Cayman(problem first reported as bug 45018). 
> 
> Well, we were hoping to get this resolved in time for 9.0, but as it's getting
> pretty close now, it's probably better to disable it at least for the 9.0
> release.  The problem is, when it's disabled, there's not much chance of anyone
> testing it, so it's not likely to ever get properly fixed.  Also, SI only
> supports VM, so we can't disable VM for SI.

Meanwhile, since fixes committed for bug 45018 helped me a lot, I'll gladly keep VM activated to test it. After all, my desktop is now usable now, I've been running for 3 days without any lockup, while I was previously only able to run for a couple of hours before restarting. So, if you have any patches you want to test that could help, ask me.

I'll have to confirm it later today by disabling VM, but I'm pretty sure I experienced a lock (can be reproduced every time) related to VM when testing with Unigine Tropics. It loaded, the demo began and then it locked when the island appeared at the horizon (I guess that's what it is since it was the first time I was running this demo).

From the retrieved logs, I could only identify a GPU lock with a reset that failed to reset rings properly.

As I mentiond in bug 55416 I received a new lockup due to VA being enabled, however, the lockups only started occuring after commit c8b06dccff9cb89e20378664f3cbc202876a180f.  Disabling VA also prevents the lockups, so it may be similar to what was mentioned in comment 25.  I will check if that piglit test still locks up for me.

(In reply to comment #21)
> (In reply to comment #20)
> > (In reply to comment #19)
> > > So about this locking piglit test (depthstencil-render-miplevels 146
> > > s=z24_s8_d=z32f_s8), I've been able to track it down to:
> > > line 218:         piglit_report_result(PIGLIT_SKIP);
> > 
> > How did you determine that? It's weird, I wouldn't expect a skipped test to
> > produce any actual GPU rendering.
> 
> I used gdb and step into the code until it locked. It gets out at level 0,
> after going through:
> 
>  /**
>  * Attach the proper miplevel of each texture to the framebuffer
>  */
> void
> set_up_framebuffer_for_miplevel(int level)...
> 
> Before this call, there is a framebuffer initialization:
> 	GLuint fbo;
> 	glGenFramebuffers(1, &fbo);
> 	glBindFramebuffer(GL_DRAW_FRAMEBUFFER, fbo);
> 	glBindFramebuffer(GL_READ_FRAMEBUFFER, fbo);
> 
> 	for (int level = 0; level <= max_miplevel; ++level) {
> 		set_up_framebuffer_for_miplevel(level);

It seems that with latest mesa, drm, xf86 and kernel 3.7.0-rc7-71633-g3b6b59b from drm-next, it doesn't fail on this test anymore. It does lock however on a different one. I'll debug it and see where it locks.

I'm closing this bug, the original triggering application is not doing it anymore. Also many things changed since then. I'll reopen it if for any reason I can still point to this exact problem, but I think it was more like an umbrella bug: many others were hidden under it.