I have test it with geforce GTX 670. This is the Link to the trace file https://drive.google.com/file/d/0B3ZWj0J3q6uRcHpJYkNPeTk1dVE/view?usp=sharing

Reassigning to nouveau for now until we can verify where the issue lies.

I'm able to reproduce with Mesa 11.1.2, but not with the current mesa master. Would be nice to figure out what fixed it, but I don't really have the patience to do a reverse bisect (trace takes a long time, my GPU is slow).

I bisected it, here is my git bisect log:

git bisect start
# bad: [5dbb57a99482e2e04302ec8ef8cb4130e4ddb882] nouveua_compiler stuff
git bisect bad 5dbb57a99482e2e04302ec8ef8cb4130e4ddb882
# good: [9f21fdd8e697a1a0868f4ec4573235325294b7b3] Update version to 11.2.0-rc3
git bisect good 9f21fdd8e697a1a0868f4ec4573235325294b7b3
# good: [4cd5e5b48e24a8b8ff7255022208d3e5fe6557d8] nouveau: update the Makefile.sources list
git bisect good 4cd5e5b48e24a8b8ff7255022208d3e5fe6557d8
# good: [b4b50b074beae9b679e5acdbb4b49193e539576d] mesa: Make glGet queries initialize ctx->Debug when necessary.
git bisect good b4b50b074beae9b679e5acdbb4b49193e539576d
# good: [60c08aa90bce4c8766a747c8517f7ff6987937f0] gallium/radeon: disable CMASK on handle export if sharing doesn't allow it (v2)
git bisect good 60c08aa90bce4c8766a747c8517f7ff6987937f0
# good: [8be9efcce70f671afcaf3cfc76c624b3467b7834] radeon/uvd: handle HEVC main 10 decode
git bisect good 8be9efcce70f671afcaf3cfc76c624b3467b7834
# bad: [c1e4a6bfbf015801c6a8b0ae694482421a22c2d9] nv50,nvc0: handle SQRT lowering inside the driver
git bisect bad c1e4a6bfbf015801c6a8b0ae694482421a22c2d9
# good: [a651bc027d5ed4150bb5240fc9f46a6ca569f665] nvc0: fix blit triangle size to fully cover FB's > 8192x8192
git bisect good a651bc027d5ed4150bb5240fc9f46a6ca569f665
# good: [b3e7fb52349848b24f005c07859bc43691bd64bd] nv50/ir: avoid folding mul + add if the mul has a dnz
git bisect good b3e7fb52349848b24f005c07859bc43691bd64bd
# first bad commit: [c1e4a6bfbf015801c6a8b0ae694482421a22c2d9] nv50,nvc0: handle SQRT lowering inside the driver

OK, so looks like it was the sqrt change that fixed it (commit c1e4a6bfbf01) - thanks Karol. I'm a bit scared of backporting that to 11.2 at such a late date, so I'd rather leave this alone for now. It should be fixed in 11.3 though.

Bug is still present in mesa 11.2.2

(In reply to Heiko Ernst from comment #6)
> Bug is still present in mesa 11.2.2

The patch is not present in 11.2.2, but it should be there in 11.3/12.0, as Ilia said. Please retry when 11.3/12.0 is released, or you could grab master from Mesa and compile it.

This is a GLSL compiler bug. Removing the RSQ+RCP -> SQRT transformation fixes it.

Bug is closed in mesa 12.0.2

Closing as per comment #9. Thanks for pointing this out.