Can you debug this at all?  I don't know what format is causing this failure.  I've updated the code to print the format in the error message (and will attach the patch).

Created attachment 40205 [details] [review]
emit the format name in the error message

By inspection I found a few pixel formats that weren't handled in that function.  I've added them now and committed to git.  If you can test with Mesa from git, that'd be great.

(In reply to comment #1)
> Can you debug this at all?  I don't know what format is causing this failure. 
> I've updated the code to print the format in the error message (and will attach
> the patch).

With the patch applied to the current Fedora 14 package (mesa-dri-drivers-7.9-2.fc14.x86_64 with Mesa from 2010-10-14), I now get

Mesa 7.9 implementation error: bad format MESA_FORMAT_NONE in _mesa_format_to_type_and_comps

Will try Mesa from git next.

Hmmm, we shouldn't even be in that function if format=MESA_FORMAT_NONE.  Can you run w/ gdb, set a breakpoint on _mesa_problem() and get a stack trace?

I am having some build problems with the sources from git.  Presumably, to get the stack trace, the current Fedora sources are OK?

I need a few hours to get a debug environment set up and deal with some other unrelated issues.

(In reply to comment #6)
> I am having some build problems with the sources from git.  Presumably, to get
> the stack trace, the current Fedora sources are OK?

Sure.


> I need a few hours to get a debug environment set up and deal with some other
> unrelated issues.

OK, thanks for digging into this!

Here's the top of the stack.  I'll attach to full-backtrace.  Please let me know if there is any other information I can provide.

#0  _mesa_problem (ctx=0x0, fmtString=
    0x7f5b436a4d95 "bad format %s in _mesa_format_to_type_and_comps")
    at main/imports.c:933
#1  0x00007f5b43562b92 in _mesa_format_to_type_and_comps (
    format=<value optimized out>, datatype=0x7fffcafad76c, comps=
    0x7fffcafad768) at main/formats.c:1336
#2  0x00007f5b43580b14 in _mesa_generate_mipmap (ctx=0x296a720, target=3553,
    texObj=0x331cb50) at main/mipmap.c:1582
#3  0x00007f5b43526ba4 in radeon_generate_mipmap (ctx=0x296a720, target=3553,
    texObj=0x331cb50) at radeon_texture.c:257
#4  radeonGenerateMipmap (ctx=0x296a720, target=3553, texObj=0x331cb50)
    at radeon_texture.c:300
#5  0x00007f5b43561087 in _mesa_GenerateMipmapEXT (target=3553)
    at main/fbobject.c:2126
#6  0x00007f5b57f330c4 in KWin::GLTexture::enableFilter() ()
   from /usr/lib64/libkwineffects.so.1
#7  0x00007f5b5865b881 in ?? () from /usr/lib64/libkdeinit4_kwin.so
#8  0x00007f5b57f32f99 in KWin::GLTexture::setWrapMode(unsigned int) ()
   from /usr/lib64/libkwineffects.so.1
#9  0x00007f5b5865efda in ?? () from /usr/lib64/libkdeinit4_kwin.so
#10 0x00007f5b58661ddc in ?? () from /usr/lib64/libkdeinit4_kwin.so
#11 0x00007f5b5865229f in ?? () from /usr/lib64/libkdeinit4_kwin.so

Created attachment 40210 [details]
Full backtrace (partial symbols)

Could you go up the stack into _mesa_generate_mipmap() then do:

(gdb) p *texObj
(gdb) p *texObj->Image[0][texObj->BaseLevel]

(gdb) up
#2  0x00007f7388223b14 in _mesa_generate_mipmap (ctx=0x22f6750, target=3553, 
    texObj=0x2d9f930) at main/mipmap.c:1582
1582       _mesa_format_to_type_and_comps(convertFormat, &datatype, &comps);
(gdb)  p *texObj
$1 = {Mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
      __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
    __size = '\000' <repeats 39 times>, __align = 0}, RefCount = 3, Name = 23, 
  Target = 3553, Priority = 1, BorderColor = {f = {0, 0, 0, 0}, ui = {0, 0, 0, 
    0}, i = {0, 0, 0, 0}}, WrapS = 10497, WrapT = 10497, WrapR = 10497, 
  MinFilter = 9987, MagFilter = 9729, MinLod = -1000, MaxLod = 1000, LodBias = 
    0, BaseLevel = 0, MaxLevel = 1000, MaxAnisotropy = 1, CompareMode = 0, 
  CompareFunc = 515, CompareFailValue = 0, DepthMode = 6409, _MaxLevel = 0, 
  _MaxLambda = 0, CropRect = {0, 0, 0, 0}, Swizzle = {6403, 6404, 6405, 6406}, 
  _Swizzle = 1672, GenerateMipmap = 0 '\000', _Complete = 0 '\000', 
  _RenderToTexture = 1 '\001', Purgeable = 0 '\000', Image = {{0x2d947d0, 
    0x0 <repeats 12 times>}, {0x0 <repeats 13 times>}, {
    0x0 <repeats 13 times>}, {0x0 <repeats 13 times>}, {
    0x0 <repeats 13 times>}, {0x0 <repeats 13 times>}}, Palette = {
    InternalFormat = 0, _BaseFormat = 0, Size = 0, TableF = 0x0, TableUB = 
    0x0, RedSize = 0 '\000', GreenSize = 0 '\000', BlueSize = 0 '\000', 
    AlphaSize = 0 '\000', LuminanceSize = 0 '\000', IntensitySize = 0 '\000'}, 
  DriverData = 0x0}
(gdb)  p *texObj->Image[0][texObj->BaseLevel]
$2 = {InternalFormat = 4, _BaseFormat = 6408, TexFormat = 0, Border = 0, 
  Width = 1093, Height = 52, Depth = 1, Width2 = 1093, Height2 = 52, Depth2 = 
    1, WidthLog2 = 10, HeightLog2 = 5, DepthLog2 = 0, MaxLog2 = 10, 
  WidthScale = 1093, HeightScale = 52, DepthScale = 1, IsClientData = 
    0 '\000', _IsPowerOfTwo = 0 '\000', TexObject = 0x2d9f930, FetchTexelc = 
    0, FetchTexelf = 0, RowStride = 1152, ImageOffsets = 0x2c4e670, Data = 
    0x0, DriverData = 0x0}

Created attachment 40216 [details] [review]
additional assertion for radeon driver

It should never be the case that a texture image has a non-zero width and height with TexFormat==0.  This assertion might catch the root problem.  Let me know what happens.

There are significant changes to the radeon_texture.c file between the 7.9 and master branches.  The Fedora package stems from 7.9 as of 2010-10-14.  The patch does not apply against that version.  I will try to resolve my build problems with git master and retest against it.

I retested against git master with the patch applied.  There is no discernible difference in behavior or gdb output.  The assertion was not triggered.

Created attachment 40217 [details] [review]
set texImage->TexFormat field in radeonSetTexBuffer2()

Here's another patch to try (for 7.9).

The patch applied cleanly but unfortunately it doesn't seem to have any impact.  

However, I am not sure if I am doing this correctly.  I am testing with a Radeon HD 3650 card using the r600_dri.so driver.  The radeonSetTexBuffer2 symbol does not appear in this driver; it does exist in the radeon_dri.so driver.

Created attachment 40233 [details] [review]
set texImage->TexFormat field in r600SetTexBuffer2()

OK, here's a patch for the r600 driver.  Same idea as previous patch.

With the r600 patch, kwin aborts when compositing is enabled:

drmRadeonCmdBuffer: -22. Kernel failed to parse or rejected command stream. See dmesg for more info.


From dmesg:

[82568.581911] radeon 0000:01:00.0: r600_cs_track_check:280 mask 0x0000000F | 0x0000000F no cb for 0
[82568.581913] radeon 0000:01:00.0: r600_packet3_check:1108 invalid cmd stream 502
[82568.581915] [drm:radeon_cs_ioctl] *ERROR* Invalid command stream !

Hmmm, I don't know what's going on there.  Maybe one of the r600 developers can take a look.  In the mean time, can you try the r600SetTexBuffer2() patch on Mesa/master from git?

The command stream error also happens with git master.

Taking a slightly different tack, I have tried to narrow down which desktop effects cause the failure.  Many work just fine, e.g., basic compositing, translucency, thumbnails.  It seems that effects that distort windows such as "wobbly windows" and "magic lamp" trigger the problem.

If I disable those effects, the unpatched Fedora package also works for me.

At this point I think my patch which set's the texImage->TexFormat field in r600SetTexBuffer2() is the right fix but now you're hitting another r600 bug.  One of the r600 developers will probably have to help you with that.

I'll commit the above patches for radeon/r600.

(In reply to comment #21)
> I'll commit the above patches for radeon/r600.

Please patch r200 and r300 similarly.

(In reply to comment #22)
> (In reply to comment #21)
> > I'll commit the above patches for radeon/r600.
> 
> Please patch r200 and r300 similarly.

Yup, just did.

Three questions:

1. Can we come up with a test case that would have triggered this bug?

2. Are all of these patches in 7.9 and 7.10?

3. Assuming the answer to #2 is "yes," can this bug be closed?

(In reply to comment #24)
> Three questions:
> 
> 1. Can we come up with a test case that would have triggered this bug?

I think I can add some code in Mesa to check that all cases are handled in the _mesa_format_to_type_and_comps() function.


> 2. Are all of these patches in 7.9 and 7.10?

Yes.  Commit 97805cfedd6fbf344f70f187021c337665135466 in 7.9

> 3. Assuming the answer to #2 is "yes," can this bug be closed?

Yes.  Norbert's last issue should be filed as a new bug, if it's still happening.