Bug 91686

Created attachment 117779 [details]
patch to fix the core dump

Pieter van der Eems produced a PDF where
 pdftoppm -cropbox -f 0 -l 0 -r 150 /tmp/memory_issue_pdf.pdf x
core dumps with a memory corruption error like
  *** Error in `/usr/bin/pdftoppm': double free or corruption (out): 0x00000000011145b0 ***
or
  *** Error in `./pdftoppm': free(): invalid next size (normal): 0x000000000107cf80 ***

valgrind reports
Invalid read of size 1
   at 0x47AF9C: Splash::pipeRun(SplashPipe*) (Splash.cc:504)
   by 0x488D97: Splash::drawPixel(SplashPipe*, int, int, bool) (Splash.cc:1414)
   by 0x47F96A: Splash::arbitraryTransformMask(bool (*)(void*, unsigned char*), void*, int, int, double*, bool) (Splash.cc:3242)
   by 0x48331B: Splash::fillImageMask(bool (*)(void*, unsigned char*), void*, int, int, double*, bool) (Splash.cc:2980)
   by 0x40EE72: SplashOutputDev::setSoftMaskFromImageMask(GfxState*, Object*, Stream*, int, int, bool, bool, double*) (SplashOutputDev.cc:2845)
   by 0x42529A: Gfx::doPatternImageMask(Object*, Stream*, int, int, bool, bool) (Gfx.cc:2091)
   by 0x42611A: Gfx::doImage(Object*, Stream*, bool) (Gfx.cc:4380)
   by 0x4268C9: Gfx::opBeginImage(Object*, int) (Gfx.cc:4987)
   by 0x421BC9: Gfx::go(bool) (Gfx.cc:763)
   by 0x422054: Gfx::display(Object*, bool) (Gfx.cc:729)
   by 0x4BF027: Page::displaySlice(OutputDev*, double, double, int, bool, bool, int, int, int, int, bool, bool (*)(void*), void*, bool (*)(Annot*, void*), void*, ...
   by 0x40AAF3: savePageSlice(PDFDoc*, SplashOutputDev*, int, int, int, int, int, double, double, char*) (pdftoppm.cc:225)
 Address 0x58d82af is 1 bytes before a block of size 2,112 alloc'd
   at 0x4A06BBD: malloc (vg_replace_malloc.c:296)
   by 0x469F6C: gmalloc(unsigned long, bool) (gmem.cc:110)
   by 0x489A8F: SplashBitmap::SplashBitmap(int, int, int, SplashColorMode, bool, bool, GooList*) (SplashBitmap.cc:113)
   by 0x40EDC9: SplashOutputDev::setSoftMaskFromImageMask(GfxState*, Object*, Stream*, int, int, bool, bool, double*) (SplashOutputDev.cc:2839)
   by 0x42529A: Gfx::doPatternImageMask(Object*, Stream*, int, int, bool, bool) (Gfx.cc:2091)
   by 0x42611A: Gfx::doImage(Object*, Stream*, bool) (Gfx.cc:4380)
   by 0x4268C9: Gfx::opBeginImage(Object*, int) (Gfx.cc:4987)
   by 0x421BC9: Gfx::go(bool) (Gfx.cc:763)
   by 0x422054: Gfx::display(Object*, bool) (Gfx.cc:729)
   by 0x4BF027: Page::displaySlice(OutputDev*, double, double, int, bool, bool, int, int, int, int, bool, bool (*)(void*), void*, bool (*)(Annot*, void*), void*, ...
   by 0x40AAF3: savePageSlice(PDFDoc*, SplashOutputDev*, int, int, int, int, int, double, double, char*) (pdftoppm.cc:225)
   by 0x40A348: main (pdftoppm.cc:532)

Debug code shows that Splash::arbitraryTransformMask() was setting xa to -1 and then running
  for (x = xa; x < xb; x++) {
    drawPixel(&pipe, x, y, clipRes2 == splashClipAllInside);
  }
With xa of -1, it was corrupting internal malloc data in front of the buffer.
valgrind warns "Address 0x58d82af is 1 bytes before a block of size 2,112 alloc'd"

A later write to the byte corrupts memory, and then pdftoppm will eventually crash.

Invalid write of size 1
  at 0x47AAD1: Splash::pipeRun(SplashPipe*) (Splash.cc:827)
  by 0x488D97: Splash::drawPixel(SplashPipe*, int, int, bool) (Splash.cc:1414)
  by 0x47F96A: Splash::arbitraryTransformMask(bool (*)(void*, unsigned char*), void*, int, int, double*, bool) (Splash.cc:3242)
  by 0x48331B: Splash::fillImageMask(bool (*)(void*, unsigned char*), void*, int, int, double*, bool) (Splash.cc:2980)

The patch ensures that xa is >= 0.

There might be a deeper problem that splashFloor() can return a negative value.
splashFloor() is implemented in SplashMath.h.