Bug 23581 - Stack overflow when rendering with a Picture that has an alpha_map cycle
Summary: Stack overflow when rendering with a Picture that has an alpha_map cycle
Status: RESOLVED FIXED
Alias: None
Product: xorg
Classification: Unclassified
Component: Server/General (show other bugs)
Version: git
Hardware: Other All
: medium normal
Assignee: Xorg Project Team
QA Contact: Xorg Project Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-08-28 12:58 UTC by Aaron Plattner
Modified: 2010-08-10 10:20 UTC (History)
0 users

See Also:
i915 platform:
i915 features:


Attachments
Test case (780 bytes, text/x-csrc)
2009-08-28 12:59 UTC, Aaron Plattner
no flags Details
Do not recurse looking for an alpha map (2.97 KB, patch)
2010-06-05 19:52 UTC, Keith Packard
no flags Details | Splinter Review

Description Aaron Plattner 2009-08-28 12:58:48 UTC
Compile the attached program with "gcc -o render-alphamap-cycle render-alphamap-cycle.c -lX11 -lXrender" and then run it.  X crashes due to a stack overflow:

Program received signal SIGSEGV, Segmentation fault.
0xfffffd7fff257b5d in cleanfree () from /usr/lib/amd64/libc.so.1
(gdb) bt 20
#0  0xfffffd7fff257b5d in cleanfree () from /usr/lib/amd64/libc.so.1
#1  0xfffffd7fff256de1 in _malloc_unlocked () from /usr/lib/amd64/libc.so.1
#2  0xfffffd7fff256d06 in malloc () from /usr/lib/amd64/libc.so.1
#3  0xfffffd7ffe9402d4 in allocate_image () from /usr/lib/amd64/libpixman-1.so.0
#4  0xfffffd7ffe94081c in pixman_image_create_bits () from /usr/lib/amd64/libpixman-1.so.0
#5  0xfffffd7ffa421e3e in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#6  0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#7  0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#8  0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#9  0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#10 0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#11 0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#12 0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#13 0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#14 0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#15 0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#16 0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#17 0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#18 0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
#19 0xfffffd7ffa42208a in wfb_image_from_pict () from /usr/X11/lib/modules/amd64//libwfb.so
(More stack frames follow...)

Idea courtesy of Mark Vojkovich.
Comment 1 Aaron Plattner 2009-08-28 12:59:18 UTC
Created attachment 28987 [details]
Test case
Comment 2 Søren Sandmann Pedersen 2009-08-28 13:35:21 UTC
My immediate reaction is that it should be a BadMatch if you try to set a picture as its own alpha map, and also if you try to set an alpha map that itself has an alpha map already. Those two rules together seem like they would prevent cycles.

There is also the case where a picture and its alpha map point to the same underlying drawable, but I don't think that will cause other issues than unpredictable painting, at least in the software implementation.
Comment 3 Keith Packard 2010-06-05 19:52:45 UTC
Created attachment 36080 [details] [review]
Do not recurse looking for an alpha map

This patch makes the alpha map lookup non-recursive, stopping after the first alpha map.
Comment 4 Adam Jackson 2010-08-10 10:20:58 UTC
Fixed in 1.9.


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.