Created attachment 40964 [details] [review]
Fix for the levels array overflow
extremetuxracer 0.5 beta2 shows up a bug in r600c.
The levels array in radeon_mipmap_tree structure is defined with a size of RADEON_MIPTREE_MAX_TEXTURE (ie 13). Though in radeon_try_alloc_miptree the size of numLevels can overflow this size.
Then in calculate_miptree_layout_r300 the loop write out of the array.
Which leads to calloc failure in bo_open from radeon_gem_bo due to corrupted memory.
This patch fixes this by setting the numLevels max to RADEON_MIPTREE_MAX_TEXTURE.
Created attachment 40965 [details]
backtrace of the calloc failure in radeon drm due to this previous corruption
Created attachment 40967 [details]
valgrind output from such an etracer run which shows the overflow
the overflow happens in in radeon_mipmap_tree.c:195 ie calculate_miptree_layout_r300 and has origin radeon_try_alloc_miptree .
This should be fixed in:
User reported the issue as fixed. Can be closed. Thank you