The overflows happen in 64-bit variants of pdf operators where two 'comp1_t' (uint16_t) values are promoted to 'int' type and multiplied with the actual intention to get 'uint64_t' product:
    http://cgit.freedesktop.org/pixman/tree/pixman/pixman-combine.c.template?id=pixman-0.24.0#n525

The reduced testcase may look like this:

/*****************/

#include <inttypes.h>
#include <stdio.h>

int main (void)
{
    volatile uint16_t a = 0xFFFF, b = 0xFFFF;
    uint64_t c = a * b;
    printf("%" PRIX64 "\n", c);
    return 0;
}

/*****************/

Because signed overflows are undefined behavior in C, different compilers may produce different results here. And looks like the results produced by svn version of clang already differ.

To hunt down at least some of such signed overflow bugs, -ftrapv option can be used in either gcc or clang. Or alternatively something even more advanced can be tried: http://embed.cs.utah.edu/ioc/

Created attachment 54537 [details] [review]
Partial fix

Thanks, good catch. 

The attached file is a partial fix, but there is still at least one remaining signed overflow when compiled with -ftrapv. That one basically boils down to this:

    int
    main ()
    {
        int64_t ne = 590261328917LL;
        int nx = 100245;
        int dy = 5888203;

        printf ("%lld\n", (int64_t)ne - (int64_t)nx * (int64_t) dy);

        (int32_t)(((int64_t)ne - (int64_t)nx * (int64_t) dy));
    }

which generates a trap when compiled with -ftrapv -O0 by gcc 4.3.2 on x86-32.

But I can't see anything wrong with this code, and the computed value (-1580818) easily fits in a 32 bit integer, so the case to int32_t should be fine too. I'm beginning to suspect a compiler bug, but maybe I'm missing something.

Apparently, ftrapv is considered "totally broken" by the GCC develoeprs: 

    http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35412

It looks like signed overflows just get more attention from llvm/clang side and clang is becoming more eager to cut some corners for better performance. So the ones who are relying on the "it is probably safe even though it does not strictly conform to the C standard" assumption, are likely going to be punished:
    http://www.gnu.org/s/hello/manual/autoconf/Signed-Overflow-Examples.html

It is interesting how different compilers generate different code:

===== C source ======

#include <inttypes.h>

uint64_t f(uint16_t a, uint16_t b)
{
    return a * b;
}

===== gcc 4.5.3 (-O2 -c) ====

0000000000000000 <f>:
   0:	0f b7 c6             	movzwl %si,%eax
   3:	0f b7 ff             	movzwl %di,%edi
   6:	0f af c7             	imul   %edi,%eax
   9:	48 98                	cltq   
   b:	c3                   	retq   

==== clang 3.0 (-O2 -c) ====

0000000000000000 <f>:
   0:	0f af fe             	imul   %esi,%edi
   3:	48 63 c7             	movslq %edi,%rax
   6:	c3                   	retq   

==== clang svn (-O2 -c) ====

0000000000000000 <f>:
   0:	48 0f af f7          	imul   %rdi,%rsi
   4:	48 89 f0             	mov    %rsi,%rax
   7:	c3                   	retq   


BTW, clang-3.0 fails to compile pixman correctly, I just was a bit too late to try clang release candidates:
    http://llvm.org/bugs/show_bug.cgi?id=11438

Proposed patches here:

    http://lists.freedesktop.org/archives/pixman/2011-December/001633.html

The IOC tool is really effective, although it's a bit of a pain to build.

Fixed in master.