Bug 22512

Summary: SIGFPE in _mesa_sse_transform_points3_3d_no_rot
Product: Mesa Reporter: Arthur Huillet <arthur.huillet>
Component: Mesa coreAssignee: mesa-dev
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: medium CC: arthur.huillet
Version: unspecified   
Hardware: Other   
OS: All   
Whiteboard:
i915 platform: i915 features:
Attachments: Properly zero high quadword of %xmm0 before issuing a vector multiplication

Description Arthur Huillet 2009-06-27 10:43:00 UTC
Hi,

MESA 7.4 crashes with a SIGFPE in _mesa_sse_transform_points3_3d_no_rot.
This can be reproduced with the FreedroidRPG game, and maybe other applications.

Here is the relevant information:

Program received signal SIGFPE, Arithmetic exception.
[Switching to Thread 0xa7a6c6d0 (LWP 5471)]
0xa6f7d2e3 in _mesa_sse_transform_points3_3d_no_rot () from /usr/lib/xorg/modules/dri/libdricore.so
(gdb) print $xmm0
$1 = {v4_float = {556, 148, -nan(0x32b2b2), -nan(0x32b2b2)}, v2_double = {1407375168946176, -1.3130212884391347e+307}, v16_int8 = {
    0, 0, 11, 68, 0, 0, 20, 67, -78, -78, -78, -1, -78, -78, -78, -1}, v8_int16 = {0, 17419, 0, 17172, -19790, -78, -19790, -78}, 
  v4_int32 = {1141571584, 1125384192, -5066062, -5066062}, v2_int64 = {4833488301216956416, -21758566319607118}, 
  uint128 = 0xffb2b2b2ffb2b2b243140000440b0000}
(gdb) bt       
#0  0xa6f7d2e3 in _mesa_sse_transform_points3_3d_no_rot () from /usr/lib/xorg/modules/dri/libdricore.so
#1  0xa6edb732 in run_vertex_stage () from /usr/lib/xorg/modules/dri/libdricore.so
#2  0xa6ecfc53 in _tnl_run_pipeline () from /usr/lib/xorg/modules/dri/libdricore.so
#3  0xa702fda9 in intelRunPipeline () from /usr/lib/xorg/modules/dri/i915_dri.so
#4  0xa6ed096c in _tnl_draw_prims () from /usr/lib/xorg/modules/dri/libdricore.so
#5  0xa6ec7374 in vbo_exec_vtx_flush () from /usr/lib/xorg/modules/dri/libdricore.so
#6  0xa6ec307e in vbo_exec_wrap_buffers () from /usr/lib/xorg/modules/dri/libdricore.so
#7  0xa6ec320c in vbo_exec_fixup_vertex () from /usr/lib/xorg/modules/dri/libdricore.so
#8  0xa6ec3dd0 in vbo_Color4f () from /usr/lib/xorg/modules/dri/libdricore.so
#9  0xa6e0f21f in loopback_Color4ub_f () from /usr/lib/xorg/modules/dri/libdricore.so

As you can see, %xmm0 has two of its values at NaN.
The crash occurs on the following instruction :

0xa6f7d2e3 <_mesa_sse_transform_points3_3d_no_rot+103>: mulps  %xmm1,%xmm0


As you can see, we are trying to multiply the *full* xmm0 register with xmm1, but we have only loaded half of it, and the upper quadword is left to whatever values were before. Multiplying four floats when only two were loaded is dangerous.

I believe _mesa_sse_transform_points3_3d_no_rot should carefully zero the upper quadword of xmm0 instead of relying on it not containing invalid values.

I cheat in the application by zeroing out the xmm0 register so it contains no NaN, but it's obviously a dirty workaround.

Thanks.
Comment 1 Arthur Huillet 2009-06-27 11:31:53 UTC
Hi, 

there is the exact same problem in _mesa_sse_transform_points2_3d_no_rot.

I patched my binary libdricore.so so the two functions do a PXOR %xmm0, %xmm0.
I confirm it fixes the crash I have been encountering.

I can prepare an actual (source) patch if necessary.

Comment 2 Brian Paul 2009-06-29 07:16:24 UTC
Yes, please provide a source patch if possible.  Thanks.
Comment 3 Arthur Huillet 2009-06-30 04:35:25 UTC
Created attachment 27260 [details] [review]
Properly zero high quadword of %xmm0 before issuing a vector multiplication

Here is my proposed patch.
My attempt at a commit message is probably poor, sorry about that.
Comment 4 Arthur Huillet 2009-06-30 04:35:50 UTC
Hi, 

patch submitted.
Comment 5 Brian Paul 2009-06-30 07:43:15 UTC
Thanks.  Committed to Mesa: 7d55cd8765abe3385028815b06e1d3ececda7fb9
It'll go into 7.5 and later and I'll also cherry-pick to the 7.4.x branch

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.