Bug 32277

Summary: overflow in calculate_miptree_layout_r300
Product: Mesa Reporter: Alban Browaeys <prahal>
Component: Drivers/DRI/R600Assignee: Default DRI bug account <dri-devel>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: medium    
Version: git   
Hardware: Other   
OS: All   
Whiteboard:
i915 platform: i915 features:
Attachments: Fix for the levels array overflow
backtrace of the calloc failure in radeon drm due to this previous corruption
valgrind output from such an etracer run which shows the overflow

Description Alban Browaeys 2010-12-09 14:10:12 UTC 
Created attachment 40964 [details] [review]
Fix for the levels array overflow

extremetuxracer 0.5 beta2 shows up a bug in r600c. 

The levels array in radeon_mipmap_tree structure is defined with a size of RADEON_MIPTREE_MAX_TEXTURE (ie 13). Though in radeon_try_alloc_miptree the size of numLevels can overflow this size.
Then in calculate_miptree_layout_r300 the loop write out of the array.
Which leads to calloc failure in bo_open from radeon_gem_bo due to corrupted memory.

This patch fixes this by setting the numLevels max to RADEON_MIPTREE_MAX_TEXTURE.
Comment 1 Alban Browaeys 2010-12-09 14:11:16 UTC 
Created attachment 40965 [details]
backtrace of the calloc failure in radeon drm due to this previous corruption
Comment 2 Alban Browaeys 2010-12-09 14:14:53 UTC 
Created attachment 40967 [details]
valgrind output from such an etracer run which shows the overflow

the overflow happens in  in radeon_mipmap_tree.c:195 ie calculate_miptree_layout_r300 and has origin radeon_try_alloc_miptree .
Comment 4 Alban Browaeys 2010-12-20 07:06:40 UTC 
User reported the issue as fixed. Can be closed. Thank you

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.