Bug 50992

I don't see the problem of Albert's statement in https://bugs.kde.org/show_bug.cgi?id=185908 that none of our backends is thread safe in practice, at least if we speak about concurrent rendering of pages in splash. In theory we have a problem with the static globalParam variable on the one hand and with the call of exit if we run out of memory on the other, but if the calling application allocate globalParam only once and is compiled in 64 bit mode, these problems should be minimal.

Hi Thomas,

Thanks for the swift reply.

To be honest, I haven't investigate the details of how okular calls poppler, but my request to the Okular developers that they improve the code's concurrently was rejects due to lack of support in poppler (specifically, lack of thread-safeness) .. hence this bug.  [I had hoped they would open the ticket, rather than rejecting it, but hey ho...]

If you say that the poppler API is thread-safe, then I'll reopen the okular ticket saying that they're reason for closing it is flawed (or my understand of it is wrong).

Cheers,

Paul.

Hi, Okular and Poppler maintainer here.
Didn't open a bug to myself since well, it's a well known thing that you can't use more than one thread on poppler.

Thomas, I can't really parse your answer :D Not sure if you are saying that you think poppler is thread safe or not, but it's clearly not, do this and see the explosions

* Create doc
* Create thread that renders page 1 of doc
* Create thread that renders page 2 of doc

(In reply to comment #3)
> Hi, Okular and Poppler maintainer here.
> Didn't open a bug to myself since well, it's a well known thing that you can't
> use more than one thread on poppler.
> 
> Thomas, I can't really parse your answer :D Not sure if you are saying that you
> think poppler is thread safe or not, but it's clearly not, do this and see the
> explosions
> 
With splash?
> * Create doc
> * Create thread that renders page 1 of doc
> * Create thread that renders page 2 of doc
With the same splashoutput device? Of course You can't do that: You need a different splashOut for every thread at least because of the state. And I forgot one other point: if You use little cms the poppler isn't thread safe, it will crashes in that area. One of the points I want to solve somewhere in futur.

(In reply to comment #4)
> (In reply to comment #3)
> > Hi, Okular and Poppler maintainer here.
> > Didn't open a bug to myself since well, it's a well known thing that you can't
> > use more than one thread on poppler.
> > 
> > Thomas, I can't really parse your answer :D Not sure if you are saying that you
> > think poppler is thread safe or not, but it's clearly not, do this and see the
> > explosions
> > 
> With splash?

Yes

> > * Create doc
> > * Create thread that renders page 1 of doc
> > * Create thread that renders page 2 of doc
> With the same splashoutput device? 

Nope

> Of course You can't do that: You need a
> different splashOut for every thread at least because of the state. And I
> forgot one other point: if You use little cms the poppler isn't thread safe, > it
> will crashes in that area. One of the points I want to solve somewhere in
> futur.

Last time i tried it crashed in the Parser (since you are sharing the same doc/parser/lexer from different threads)

(In reply to comment #5)
> Last time i tried it crashed in the Parser (since you are sharing the same
> doc/parser/lexer from different threads)
Hmmh, parser and lexer are components of Gfx, and a new Gfx is created on Page::displaySlice. So the only object which is shared should be the doc.
A problem there could be the XRef if it is reconstructed during reading the document.
But I must admit that I never tested that scenario, so I'll give it a try in near future...

Ups, Albert, You're true, poppler isn't thread safe, although the reason is slightly different: After thinking a little bit about it I figured out the reason: It's not parser or the lexer itself, but the sharing of the underlying file stream.
But it should be quite easy to implement a thread safe Page::displaySliceMT which dup's the file stream used by xref component. 
And because You're also the Okular maintainer: Should we give it a try?

Sure, feel free to try to fix it, i never had time to look at it properly, so basically we have just one thread at a time accessing poppler in okular

Success! It was how I thought: the problem was fetching objects over xref with the underlying file stream which cannot be shared. But at least under windows we cannot use dup and fdopen to get a thread safe reading, because it seems to share the underlying buffering, so I had to change a little bit more.
And because the document xref cannot be shared, it also means a few changes in the output devices, i.e. on reading embedded fonts. Therefore I put the patch at the end of my queue, because patch committing is not thread safe, too :-)

Great stuff, sorry i'm a bit slow reviewing your patches, swamped with work and with some health problems in elbow, hand, etc due to RSI

(In reply to comment #10)
> Great stuff, sorry i'm a bit slow reviewing your patches, swamped with work and
> with some health problems in elbow, hand, etc due to RSI

Don't apologize for anything, it's Your and my spare time, nothing else. On the contrary, we have to thank for Your great work and patience with us.
But the reason for my additional comment: I just made a few more tests: Multi-threading pdftoppm on my dual core laptop under 32 bit Windows XP reduces the rendering time to round about 58 percent for a 20 page document. Of course there is some overhead for thread handling, and another reason is that font are no more globally cached, but I think this is a great improvement.

If these changes were implemented, would this apply only to the internal API or would this be usable through the frontends as well, e.g. would libpoppler-qt4's Poppler::Page::renderToImage become reentrant when using the Splash output device?

(In reply to comment #12)
> If these changes were implemented, would this apply only to the internal API or
> would this be usable through the frontends as well, e.g. would libpoppler-qt4's
> Poppler::Page::renderToImage become reentrant when using the Splash output
> device?

My upcoming patch applies only to the internal API but will offer a thread safe method for rendering pages. But anyone who is interested could call it in Poppler::Page::renderToImage, it should be quite easy. I'm not able (or willing :-) to test that, therefore I've made no changes in that area.

(In reply to comment #13)
> (In reply to comment #12)
> > If these changes were implemented, would this apply only to the internal API or
> > would this be usable through the frontends as well, e.g. would libpoppler-qt4's
> > Poppler::Page::renderToImage become reentrant when using the Splash output
> > device?
> 
> My upcoming patch applies only to the internal API but will offer a thread safe
> method for rendering pages. But anyone who is interested could call it in
> Poppler::Page::renderToImage, it should be quite easy. I'm not able (or willing
> :-) to test that, therefore I've made no changes in that area.

Thanks for the clarification. I will wait until your patch hits master then and maybe try to work out a patch to expose the added functionality in libpoppler-qt4 afterwards.

Created attachment 65138 [details] [review]
Makes page rendering in splash thread safe

Here now the patch which makes page rendering thread safe. It especially offers a new parameter fMT (multi thread) for PDFDoc::displayPageSlice which must be set to gTrue if it should use in different threads with a shared PDFDoc object.
Only splash is really thread safe with this patch, because I haven't time to test other output devices (and therefore didn't made the necessary changes there). But it should be quite easy to make also other devices thread safe now: just assure that nowhere the doc->getXRef() is used, use the new parameter given by startPage.

Created attachment 65139 [details] [review]
testcase

The attached patch is not designed to be committed, it just changed pdftoppm using threads if compiled with -DUSE_THREADS. I use it to test my implementation under Windows and under Ubuntu.

(In reply to comment #15)
> Created attachment 65138 [details] [review] [review]
> Makes page rendering in splash thread safe
> 
> Here now the patch which makes page rendering thread safe. It especially offers
> a new parameter fMT (multi thread) for PDFDoc::displayPageSlice which must be
> set to gTrue if it should use in different threads with a shared PDFDoc object.
> Only splash is really thread safe with this patch, because I haven't time to
> test other output devices (and therefore didn't made the necessary changes
> there). But it should be quite easy to make also other devices thread safe now:
> just assure that nowhere the doc->getXRef() is used, use the new parameter
> given by startPage.

Hello Thomas,

I have problems compiling Poppler after applying your patch. Some places in the code seem to call things like SplashOutputDev::startPage(0, NULL) with a missing XRef argument. Would changing this to SplashOutputDev::startPage(0, NULL, NULL) be ok?

Also the ArthurOutputDev does not seem to reimplement the correct method signature. Does this pose a problem?

Regards, Adam.

Ok, I tested this using your patch and writing my own to expose it in the Qt4 frontend and at least using qpdfview with locking reduced from per-document to per-page, it seems to work like a charm! Thanks for the great work.

P.S.: Should I try to attach my patch based on yours here as well or should I rather wait till this hits master to rebase?

(In reply to comment #17)
> Hello Thomas,
> 
> I have problems compiling Poppler after applying your patch. Some places in the
> code seem to call things like SplashOutputDev::startPage(0, NULL) with a
> missing XRef argument. Would changing this to SplashOutputDev::startPage(0,
> NULL, NULL) be ok?

Should be. But if You have a gfx handle, a startPage(0, NULL, gfx->getXRef()) would be the better choice to ensure thread safeness.
 
> 
> Also the ArthurOutputDev does not seem to reimplement the correct method
> signature. Does this pose a problem?

I didn't got anny compiler errors, so probably ArthurOutputDev isn't compiled in my environment. Just change it like the other output devices, i.e. look at the changes of ImageOutputDev.cc / *.h.

Cheers,
Thomas

(In reply to comment #18)
> Ok, I tested this using your patch and writing my own to expose it in the Qt4
> frontend and at least using qpdfview with locking reduced from per-document to
> per-page, it seems to work like a charm! Thanks for the great work.

Thanks for the compliment, hope it has the desired enhancements in Your enevironment and doesn't just work.

> 
> P.S.: Should I try to attach my patch based on yours here as well or should I
> rather wait till this hits master to rebase?

In my opinion it should be okay to attach it here, but Alfred is the one who decides it.

Thomas

Thomas: http://t1.gstatic.com/images?q=tbn:ANd9GcSG-ijR0OY9XPKeYnsm8nRErxmsIsFie8VxYTKzyelIDsqDE8iL5iyxLQ !=http://t2.gstatic.com/images?q=tbn:ANd9GcTmjyBfBw7iO0JiC9cUZt5sRhVa0qKIM1ZE82X_TGiGM3N6sebJyrYvpTU ;-)

Some random quick comments from looking at the code:
 * Seems "most" of the "let make it thread safe" is adding the 
      xref = (fCopy) ? doc->getXRef()->copy() : doc->getXRef();
    stuff, have you measured how slow is copying the xref? Because if it's not "expensive" we may just do it all the time

 * To make the "thread-safety" better we should remove PDFDoc::getXRef() and force everyone to use Gfx:::getXRef() or ask for an XRef an input, right?

 * Should we remove the xref member in Page and use gfx->getXRef() everywhere?

 * The mutex locking seems to be very "rendering" oriented, e.g. if you are locking  XRef::copy, why not lock XRef::add or XRef::setModifiedObject ?

 * Why do you need UniqueFileStream ?

(In reply to comment #22)
> Some random quick comments from looking at the code:
>  * Seems "most" of the "let make it thread safe" is adding the 
>       xref = (fCopy) ? doc->getXRef()->copy() : doc->getXRef();
>     stuff, have you measured how slow is copying the xref? Because if it's not
> "expensive" we may just do it all the time

I made just one test: I rendered "cud10nebsiaol.pdf" (from bug 52488) with and without the patch, it took 1 second more with the patch (without threads, nearly half the timee with threads). But that could propably be a measuring error.
 
> 
>  * To make the "thread-safety" better we should remove PDFDoc::getXRef() and
> force everyone to use Gfx:::getXRef() or ask for an XRef an input, right?

Yes, but without knowing anything about the consequences

> 
>  * Should we remove the xref member in Page and use gfx->getXRef() everywhere?

Same answer

> 
>  * The mutex locking seems to be very "rendering" oriented, e.g. if you are
> locking  XRef::copy, why not lock XRef::add or XRef::setModifiedObject ?

It is rendering oriented. Until yesterday it even wasn't used, but I got some problems in my last tests, probably later solved later in FileStream::makeSubStream later, but I have forgot to remove it again and therefore also didn't test that.
> 
>  * Why do you need UniqueFileStream ?

Probably could done in FileStream itself. But the difference is that UniqueFileStream closes it filehandle in the destructor, FileStream doesn't.

(In reply to comment #23)
> (In reply to comment #22)
> > Some random quick comments from looking at the code:
> >  * Seems "most" of the "let make it thread safe" is adding the 
> >       xref = (fCopy) ? doc->getXRef()->copy() : doc->getXRef();
> >     stuff, have you measured how slow is copying the xref? Because if it's not
> > "expensive" we may just do it all the time
> 
> I made just one test: I rendered "cud10nebsiaol.pdf" (from bug 52488) with and
> without the patch, it took 1 second more with the patch (without threads,
> nearly half the timee with threads). But that could propably be a measuring
> error.

Ouch, that seems quite a lot, i'll do a test if i have time

> > 
> >  * To make the "thread-safety" better we should remove PDFDoc::getXRef() and
> > force everyone to use Gfx:::getXRef() or ask for an XRef an input, right?
> 
> Yes, but without knowing anything about the consequences

Sure, but having half PDFDoc::getXRef and half Gfx::getXRef is just asking for trouble in my opinion

> 
> > 
> >  * Should we remove the xref member in Page and use gfx->getXRef() everywhere?
> 
> Same answer
> 
> > 
> >  * The mutex locking seems to be very "rendering" oriented, e.g. if you are
> > locking  XRef::copy, why not lock XRef::add or XRef::setModifiedObject ?
> 
> It is rendering oriented. Until yesterday it even wasn't used, but I got some
> problems in my last tests, probably later solved later in
> FileStream::makeSubStream later, but I have forgot to remove it again and
> therefore also didn't test that.

Well, i understand that having rendering work is great, but if we are going to make it thread-safe it should be really thread-safe everywhere, i.e. can i ask for the fonts of the document while it's rendering? That's the typical use case were okular now blocks

> > 
> >  * Why do you need UniqueFileStream ?
> 
> Probably could done in FileStream itself. But the difference is that
> UniqueFileStream closes it filehandle in the destructor, FileStream doesn't.

Sure, i see the difference in the code, but why is that behaviour difference needed?

(In reply to comment #24)
> 
> Ouch, that seems quite a lot, i'll do a test if i have time

1 second in comparison that it took 2:40 minutes to render it at all? And I haven't really measured it, just made a date; pdftoppm ...; date

> 
> Well, i understand that having rendering work is great, but if we are going to
> make it thread-safe it should be really thread-safe everywhere, i.e. can i ask
> for the fonts of the document while it's rendering? That's the typical use case
> were okular now blocks

When every thread has it's own XRef (Gfx) it shouldn't be need. It depends how "You" ask for fonts

> 
> Sure, i see the difference in the code, but why is that behaviour difference
> needed?

The PDFDoc gives the file handle to FileStream but closes the handle itself, so I can't close it in the FileStream destructor without breaking that. But I need to close every additional file handle I need to open to get it thread safe, and the easiest way to ensure that is doing it in a destructor.

(In reply to comment #25)
> (In reply to comment #24)
> > 
> > Ouch, that seems quite a lot, i'll do a test if i have time
> 
> 1 second in comparison that it took 2:40 minutes to render it at all? And I
> haven't really measured it, just made a date; pdftoppm ...; date
> 
> > 
> > Well, i understand that having rendering work is great, but if we are going to
> > make it thread-safe it should be really thread-safe everywhere, i.e. can i ask
> > for the fonts of the document while it's rendering? That's the typical use case
> > were okular now blocks
> 
> When every thread has it's own XRef (Gfx) it shouldn't be need. It depends how
> "You" ask for fonts

Superficially seems so: I just tried calling Poppler::Document::fonts while several threads were rendering and it did not lead to any problems or errors. (I still block for the ::fonts call, it is really just Poppler::Page::renderToImage the lives in a different thread than the GUI.)

> > 
> > Sure, i see the difference in the code, but why is that behaviour difference
> > needed?
> 
> The PDFDoc gives the file handle to FileStream but closes the handle itself, so
> I can't close it in the FileStream destructor without breaking that. But I need
> to close every additional file handle I need to open to get it thread safe, and
> the easiest way to ensure that is doing it in a destructor.

Created attachment 65145 [details] [review]
changes to the qt4 frontend to use multi-thread by default

Might be useful if the changes to the internal API are implemented. Makes Poppler::Page::renderToImage always create a new SplashOutputDev and enables multi-threading by default. Also removes other code that becomes unnecessary because of the changes.

Here just the try of an additonal explanation of my solution:
Even if it seems so that the essential part of the implementation is the XRef part, it is substantial the UniqueFileStream, even if this has only a few lines of code.
To clearify that I simplify a little bit what the problem is:
Getting a PDF stream object is done over XRef and FileStream with a FILE pointer, which doesn't read always the stream object in one step but expects that nobody changes the file position in then meantime when reading the next portion of an object stream. But this is not true if we share the FILE pointer between different threads.
So we need a unique FILE pointer for every thread, and that is done by the implementation of UniqueFileStream. Only because that the stream is a member of an XRef instance, we need a private instance of XRef for every thread and therefore I spend the XRef::copy functionality.

Of course there are other possible solutions for the problem, i.e. buffer each (object) stream as MemStream in XRef and use mutex locking there, but this would mean a basically change of the core stuff and could cause memory problems if there are for example quite huge images. 

Hope this clearifies it a little bit for everyone who is interested in my solution.

(In reply to comment #23)
> (In reply to comment #22)
> > Some random quick comments from looking at the code:
> >  * Seems "most" of the "let make it thread safe" is adding the 
> >       xref = (fCopy) ? doc->getXRef()->copy() : doc->getXRef();
> >     stuff, have you measured how slow is copying the xref? Because if it's not
> > "expensive" we may just do it all the time
> 
> I made just one test: I rendered "cud10nebsiaol.pdf" (from bug 52488) with and
> without the patch, it took 1 second more with the patch (without threads,
> nearly half the timee with threads). But that could propably be a measuring
> error.

Yeah, I was wondering exactly the same when I looked at the code. Copying xref doesn't look too expensive, I guess it depends on how many times the xref is copied. Maybe we could use a global setting (or per page/document, I don't know) for that instead of passing just another parameter to displaySliceMethods?

(In reply to comment #24)

> 
> Well, i understand that having rendering work is great, but if we are going to
> make it thread-safe it should be really thread-safe everywhere, i.e. can i ask
> for the fonts of the document while it's rendering? That's the typical use case
> were okular now blocks

Exactly, and the reason why Evince uses a single worker thread.

(In reply to comment #30)
> (In reply to comment #24)
> 
> > 
> > Well, i understand that having rendering work is great, but if we are going to
> > make it thread-safe it should be really thread-safe everywhere, i.e. can i ask
> > for the fonts of the document while it's rendering? That's the typical use case
> > were okular now blocks
> 
> Exactly, and the reason why Evince uses a single worker thread.

Sorry, but I fear that "it should be really thread-safe everywhere" is the wrong statement: It's thread safe is always a question of design, a function or a method can be thread-safe in the meaning that it can be called / used in different threads simultaneous or a c++ object can be thread-safe in the meaning that it can shared in different threads.
So You first need to determine which objects / methods should be shared. So once again it looks like as if we miss a documentation which is internal and what is external, external calls should then be thread-safe!

(In reply to comment #25)
> (In reply to comment #24)
> > 
> > Ouch, that seems quite a lot, i'll do a test if i have time
> 
> 1 second in comparison that it took 2:40 minutes to render it at all? And I
> haven't really measured it, just made a date; pdftoppm ...; date

Oh, i thought you mean 1 second per page, if it's 1 second over 2:40 that's really "nothing"

(In reply to comment #30)
> (In reply to comment #24)
> 
> > 
> > Well, i understand that having rendering work is great, but if we are going to
> > make it thread-safe it should be really thread-safe everywhere, i.e. can i ask
> > for the fonts of the document while it's rendering? That's the typical use case
> > were okular now blocks
> 
> Exactly, and the reason why Evince uses a single worker thread.

Which means you block if the user requests two things at the same time (like render a page and tell me the fonts it uses ;-))

(In reply to comment #31)
> Sorry, but I fear that "it should be really thread-safe everywhere" is the
> wrong statement: It's thread safe is always a question of design, a function or
> a method can be thread-safe in the meaning that it can be called / used in
> different threads simultaneous or a c++ object can be thread-safe in the
> meaning that it can shared in different threads.
> So You first need to determine which objects / methods should be shared. So
> once again it looks like as if we miss a documentation which is internal and
> what is external, external calls should then be thread-safe!

Well, that's easy, the externall calls are the ones that can be called from the frontends ;-)

(In reply to comment #34)
> 
> Well, that's easy, the externall calls are the ones that can be called from the
> frontends ;-)

Great: Then it should be easy for You to provide me :-)
a) a list of the external calls
b) a list of the external calls which uses locking in the moment (as far as I know all external calls are already thread safe, otherwise we don't need the MULTHREADED compiler switch) and could be a candidate for improvement, i.e. remove the locks there and make the internal functions they call thread safe instead.
c) tell me how I can test b) or tell me someone who want to test together with me.

Then I'm willing to have a look at necessary code changes how I did it for the rendering of pages :-)

Created attachment 65478 [details] [review]
Correction for thread safe page rendering

I was curious: even if I thought, that on my dual core processor the regtest with and without the testcase patch from comment 16 doesn't make any difference, because regtesting is already split in two tasks (rendering odd and even page numbers), I started the regtest for bug 52488 together with this patch and the testcase on top.
I encountered several problem with the former thread-safe patch (a big problem were annotations: they are fetched over the xref member of PDFDoc!), but I was able to solve the problems all, therefore here now a new patch with a better implementation.
And now it comes: the regtest over my PDF Suite reduces from
3 hours, 40 minutes and 50 seconds
to
3 hours, 2 minutes and 6 seconds
I think, that's a big enhancement, and perhaps we should think about to cleanup the testcase and commit it, too!

Created attachment 66013 [details] [review]
revised patch

I upload here a revised patch because of several reasons:
1. The final changes of the last patch I tested just under Windows, and I once again forgot that it is not a problem to recursive lock the same mutex with the same thread under Windows where it runs into a dead lock under Unix.
2. I missed some crashes of the former patch because I just search for failed tests but not for crashes. I encountered that the crashes were forced by the shared resource dict which still uses the xref of the shared document. This is repaired now, too.
3. I took a deeper look now into failed tests on damaged PDF documents and encountered that I missed to copy some private members of the XRef class.

This patch runs now completely over my PDF suite on Windows and Unix (with my test case) and has no more crashes and only a very few failed tests (13 of 1133). Two or three of these failed tests have changes because a different external font was taken (because external fonts are now cashed on page base and no more document base, and fetching external fonts with fontconfig is not completely exact) and the other tests are so corrupted PDF that even the page components cannot be fetched for all pages.

How should we continue now (and should we continue or leave it as a study only)?

1. The just uploaded patch is complete and regtested for page rendering with splash. What is with other devices? I could implement it for cairo (if the cairo library is thread safe itself and does i.e. not use static variables) and test if with a testcase for pdftocairo, but I'm not sure about other devices, and i.e. in my opinion it makes no sense to have a thread safe page rendering in PSOutputDev.

2. We still have the open question if we copy the xref all the time and do not use doc->getXRef() anymore. I think it's not too expensive to copy it all the time and would implement it if all agree. I hadn't done that until now because it was easier to compare differences.

3. I think that a PDFDoc instance is what should be able to share in different threads, so I offer to have a additional look into the PDFDoc implementation which methods are changing private members and lock these methods, but I have no idea how to test such changes.

Any comments are welcome, even if the main opinion would be to leave this as a study. It was more work than I thought, but until now I did it because of theoretical interests.

(In reply to comment #38)
> How should we continue now (and should we continue or leave it as a study
> only)?

Speaking as a user of the library:

> 1. The just uploaded patch is complete and regtested for page rendering with
> splash. What is with other devices? I could implement it for cairo (if the
> cairo library is thread safe itself and does i.e. not use static variables) and
> test if with a testcase for pdftocairo, but I'm not sure about other devices,
> and i.e. in my opinion it makes no sense to have a thread safe page rendering
> in PSOutputDev.

I'd say that this is most important for the output devices that are generically used in interactive applications, so I don't see to much use of it in PSOutputDev either.

> 2. We still have the open question if we copy the xref all the time and do not
> use doc->getXRef() anymore. I think it's not too expensive to copy it all the
> time and would implement it if all agree. I hadn't done that until now because
> it was easier to compare differences.

I'd say that the overhead is warranted if this improved thread safety can be expanded upon later. Currently, it seems like concurrent execution in all its forms will only become more important. (I would also consider cleaner code a benefit that can be weighted against performance, but I am speaking as a user.)

> 3. I think that a PDFDoc instance is what should be able to share in different
> threads, so I offer to have a additional look into the PDFDoc implementation
> which methods are changing private members and lock these methods, but I have
> no idea how to test such changes.
> 
> Any comments are welcome, even if the main opinion would be to leave this as a
> study. It was more work than I thought, but until now I did it because of
> theoretical interests.

I would be very interested in this for practical reasons because with complicated documents, this increases the perceived performance of interactive applications very much. (Especially when combined with some form of prefetching.) Even if this introduces some overhead in the single-threaded case, the latency of an interface would go down and overall rendering speed would improve (at least on regular desktop systems).

Patch does not apply anymore, could you rebase it?

Also answering what i'd like to have working lock free i guess my answer is that you'd compile okular and remove all the userMutex->lock and unlock calls, but that's a lot of work :D

Created attachment 67403 [details] [review]
rebased patch

(In reply to comment #40)
> Patch does not apply anymore, could you rebase it?

Here it is. Do You need / want also a rebased testcase?

> 
> Also answering what i'd like to have working lock free i guess my answer is
> that you'd compile okular and remove all the userMutex->lock and unlock calls,
> but that's a lot of work :D

I never looked into okular, I even haven't it installed. But I will have a look into it in near futur and come back with an answer if I'm able (and willing :-) ) to help.

Created attachment 67537 [details] [review]
patch to support qt4, too

Trying to compile okular I encountered that I never compiled qt4. Trying to compile qt4 I encountered that my patch needs some adjustments. I did it (without being really threadsafe in qt4), here the resulting patch.

Created attachment 67636 [details] [review]
more changes to get qt library thread safe

This patch also inlcudes the changes from Adam. I went through the code of okular and tried to make all calls of the poppler qt library thread safe.

Created attachment 67637 [details] [review]
Patch for okular to use the thread safe qt library

Even if a okular patch is a little bit off topic for a poppler bug, I upload it here "as a testcase". I replaced nearly every lock/unlock. From my point of view only the annotation stuff is open.
Funny: because okular renders with this patch and can search parallel, the search may be jump to a white page, and when the rendering for this page finishes the former searched text is marked :-)
@Albert: it would be nice if You can test this patch BEFORE having a deeper look into it. I haven't found any error, but I'm not familiar with okular, and at least I definitely have not tested 
a) PDF with JavaScript
b) PDF with links and following links
c) PDF with movies
It should work, as far as I understand (my own) code, but it would be nice if I can get a confirmation from You in near futur and not until when I already forget what I did :-)

(In reply to comment #43)
> Created attachment 67636 [details] [review] [review]
> more changes to get qt library thread safe
> 
> This patch also inlcudes the changes from Adam. I went through the code of
> okular and tried to make all calls of the poppler qt library thread safe.

Just two question whether I understand this correctly: This patch aims to make all (or just some and if so which) methods exposed through the Qt4 frontend by Poppler::Document and Poppler::Page thread-safe, i.e. I could share my document instance over several threads concurrently calling stuff like renderToImage and search?

And I am supposed to check the POPPLER_QT4_THREADSAFE macro to find out whether I need to lock document access? So for example:

#ifndef POPPLER_QT4_THREADSAFE
// locking goes here
#endif

// make Poppler do something

Best regards, Adam.

P.S.: The documentation comments have the "since 0.22" tag, is there a chance this will make it into 0.22 even though 0.21 was already released?

(In reply to comment #45)
> (In reply to comment #43)
> > Created attachment 67636 [details] [review] [review] [review]
> > more changes to get qt library thread safe
> > 
> > This patch also inlcudes the changes from Adam. I went through the code of
> > okular and tried to make all calls of the poppler qt library thread safe.
> 
> Just two question whether I understand this correctly: This patch aims to
> make all (or just some and if so which) methods exposed through the Qt4
> frontend by Poppler::Document and Poppler::Page thread-safe, i.e. I could
> share my document instance over several threads concurrently calling stuff
> like renderToImage and search?

The goal is all (sharing document over threads, NOT pages), and if I haven't missed one, I'm quite close. But I'm still working on the annotation stuff.

> 
> And I am supposed to check the POPPLER_QT4_THREADSAFE macro to find out
> whether I need to lock document access? So for example:
> 
> #ifndef POPPLER_QT4_THREADSAFE
> // locking goes here
> #endif

Yes. Have a look at the okular patch. It calls lockDocument / releaseDocument which completely locks / unlocks if the macro isn't set but just calls incRefCnt / decRefCnt to protect it against deletion otherwise.

> 
> // make Poppler do something
> 
> Best regards, Adam.
> 
> P.S.: The documentation comments have the "since 0.22" tag, is there a
> chance this will make it into 0.22 even though 0.21 was already released?

I'm not sure, we need to ask Albert. If we aren't able to commit it before 0.21.2 we have to change the tag to 0.24 :-)

(In reply to comment #43)
> Created attachment 67636 [details] [review] [review]
> more changes to get qt library thread safe
> 
> This patch also inlcudes the changes from Adam. I went through the code of
> okular and tried to make all calls of the poppler qt library thread safe.

What are the incref and decref in Document supposed to do?

The poppler patch fails to apply on master.

(In reply to comment #48)
> The poppler patch fails to apply on master.

Did you try "git apply --ignore-space-change"?

(In reply to comment #47)
> (In reply to comment #43)
> > Created attachment 67636 [details] [review] [review] [review]
> > more changes to get qt library thread safe
> > 
> > This patch also inlcudes the changes from Adam. I went through the code of
> > okular and tried to make all calls of the poppler qt library thread safe.
> 
> What are the incref and decref in Document supposed to do?

I think they are supposed to the delete the document instance after the last thread that uses it has finished which might be after the thread that create it. Otherwise, the main, e.g. GUI, thread would have to wait for all work threads to finish before deletion.

Created attachment 67707 [details] [review]
more changes to get qt library thread safe

(In reply to comment #48)
> The poppler patch fails to apply on master.

More precise: it cannot applied anymore, because You and Carlos made additional commits in the meantime. So here my next try of a rebased patch :-)

(In reply to comment #49)
> (In reply to comment #47)
> > (In reply to comment #43)
> > > Created attachment 67636 [details] [review] [review] [review] [review]
> > > more changes to get qt library thread safe
> > > 
> > > This patch also inlcudes the changes from Adam. I went through the code of
> > > okular and tried to make all calls of the poppler qt library thread safe.
> > 
> > What are the incref and decref in Document supposed to do?
> 
> I think they are supposed to the delete the document instance after the last
> thread that uses it has finished which might be after the thread that create
> it. Otherwise, the main, e.g. GUI, thread would have to wait for all work
> threads to finish before deletion.

Or in other words: if You share an instance of an object (not even over threads) the call of the destructor is no more safe. You have to count the references to the instance and can call the destructor when the reference count become 0. The style I did it is how it is also done in several other places in poppler.

In practise: If You start printing a PDF in background and call "Close" in the main thread before printing has finished You have to delete the instance of the document when printing has finished. That is what incRefCnt and decRefCnt guarantees.

Created attachment 67723 [details] [review]
more changes to get qt library thread safe

Sorry, I made a mistake during creation of the rebased patch today, here's a better one.

Created attachment 67733 [details] [review]
todays 2nd try for a rebase patch

rebased patch after last commits

Some quick comments before i have a change to try it in 2 hours or so.

The Annot.cc change does not belong here, right?

The recursion in Stream::addFilters does not belong here either i'd say, no?

GBool fCopy seems a bad name for something that decides if the xref is going to be copied, can we call it copyXRef? Same for fMT

Seems a bit weird leaving the non thread safe versions of getDocInfo and getDocInfoNF there, no? What's the use of them?

In the qt4 frontend we don't need the incRef and decRef, that's poor's man solution. Users of qt4 frontend can put the Document * in a QSharedPointer to get the reference counting properly done (which by the way we should not need in Okular since the code ensure the PDF generator will only be deleted once all the threads have finished)

When opening PDFReference17_reference.pdf with okular it gets stuck in a Catalog problem with trying to lock a mutex you already own

#2  0x00007ffff3089eba in __pthread_mutex_lock (mutex=0x28fae90) at pthread_mutex_lock.c:61
#3  0x00007fffe074120c in Catalog::getDests (this=0x28fad80) at /home/tsdgeos/devel/poppler/poppler/Catalog.cc:903
#4  0x00007fffe073f991 in Catalog::findDest (this=0x28fad80, name=0x4995950) at /home/tsdgeos/devel/poppler/poppler/Catalog.cc:441
#5  0x00007fffe0b8b20f in PDFDoc::findDest (this=0x28fd4f0, name=0x4995950) at /home/tsdgeos/devel/poppler/poppler/PDFDoc.h:182

findDest does lock the mutex and getDests too which is a no go

(In reply to comment #54)
> In the qt4 frontend we don't need the incRef and decRef, that's poor's man
> solution. Users of qt4 frontend can put the Document * in a QSharedPointer
> to get the reference counting properly done (which by the way we should not
> need in Okular since the code ensure the PDF generator will only be deleted
> once all the threads have finished)

As a user of the Qt4 frontend, I agree that the reference counting should not be done by Poppler itself.

(In reply to comment #54)
> Some quick comments before i have a change to try it in 2 hours or so.
> 
> The Annot.cc change does not belong here, right?

Yes and no: I encountered that problem when I tried to get page rendering thread safe and still had problems with it. BUT it's definitely safer to have it in.

> 
> The recursion in Stream::addFilters does not belong here either i'd say, no?

On contrary, it does belong here: Stream::addFilters can be called by XRef::fetch and dictLookup can call XRef::fetch, and because the goo mutex implementation does not know a recursive mutex we will run in a dead lock without it.

> 
> GBool fCopy seems a bad name for something that decides if the xref is going
> to be copied, can we call it copyXRef? Same for fMT

I'll change it.

> 
> Seems a bit weird leaving the non thread safe versions of getDocInfo and
> getDocInfoNF there, no? What's the use of them?

I think I miss something. What do You mean?

> 
> In the qt4 frontend we don't need the incRef and decRef, that's poor's man
> solution. Users of qt4 frontend can put the Document * in a QSharedPointer
> to get the reference counting properly done (which by the way we should not
> need in Okular since the code ensure the PDF generator will only be deleted
> once all the threads have finished)

I wasn't really lucky with it, so if not needed I'm glad to remove it again.

(In reply to comment #55)
> When opening PDFReference17_reference.pdf with okular it gets stuck in a
> Catalog problem with trying to lock a mutex you already own
> 
> #2  0x00007ffff3089eba in __pthread_mutex_lock (mutex=0x28fae90) at
> pthread_mutex_lock.c:61
> #3  0x00007fffe074120c in Catalog::getDests (this=0x28fad80) at
> /home/tsdgeos/devel/poppler/poppler/Catalog.cc:903
> #4  0x00007fffe073f991 in Catalog::findDest (this=0x28fad80, name=0x4995950)
> at /home/tsdgeos/devel/poppler/poppler/Catalog.cc:441
> #5  0x00007fffe0b8b20f in PDFDoc::findDest (this=0x28fd4f0, name=0x4995950)
> at /home/tsdgeos/devel/poppler/poppler/PDFDoc.h:182
> 
> findDest does lock the mutex and getDests too which is a no go

Upps, I missed that. I'll have an additional look into it and change it in the next patch. But it will probably again more or less a guess, I'm on a business trip this week and so I haven't this PDF here, but I want to finish the annotation iplementation, seems to be quite easy,

Created attachment 67775 [details] [review]
complete solution for testing

Here now my complete solution to make at least the okular usages of the qt library thread safe.
1. I made the changes of comment 57.
2. I made a few changes to solve comment 55 (I found a PDF on my local harddisk which had the same problem)
3. I made some additional changes to make the save of a document thread safe (In this case I need to lock the doc->xref against any further changes until the save is complete)
4. I implement the annotation stuff.
I made several tests, and didn't run into any problems, but I want to regtest the changes at least with pdftoppm at the weekend. But perhaps You'll find also some time to make some tests.

Created attachment 67776 [details] [review]
okular testcase

I removed the incRef and decRef again, but I haven't put the Document * in a QSharedPointer (I'm not familiar enough with QT). And I changed the annotations stuff, too.
BTW, I think it could be a nice idea to render the thumbnails in an extra thread. This is possible now, too, but that's up to the okular developpers if they agree.

Created attachment 67852 [details] [review]
final thread safe patch

When I uploaded my patch on thursday I still wasn't completely satisfied with the catalog implementation: Even if it works, I saw possible gaps. So I worked on a more complete version today morning.
After that I made several valgrind tests and found three memory leaks, two in my implementation of this patch, which I solved, and one, which doesn't depend on this patch but where I already uploaded a patch to bug 54702.
After that I run a regtest with pdftoppm (with my testcase, which uses threads) and one with pdftocairo (without using threads) and didn't find any further problems.
I also made again okular tests, following content, add annotations, save as PDF with filled forms, everything worked fine. The only problem I had was when opening a PDF with sound, it didn't play. But I think it is because I use okular under ubuntu without having the correct player installed (or isn't it possible at all?).
So I upload here my (first) final patch.

(In reply to comment #54)
> 
> Seems a bit weird leaving the non thread safe versions of getDocInfo and
> getDocInfoNF there, no? What's the use of them?
> 
After reviewing my own patch :-) I now also understand what You mean, it wasn't in my mind anymore that I made an additional, thread safe version of getDocInfo when I answered the last time.
So here now the attempt to explain why it is necessary: an instance of XRef is not thread safe by nature and therefore cannot be shared over threads. The reason is the file stream member of XRef which is used to fetch objects.
But the non thread safe versions of getDocInfo and getDocInfoNF returns a PDFObject which could have references to an instance of XRef (i.e. a Dict which points to other objects which then will be fetched over the xref member), so they are also not thread safe. It would be a hard work and a lot of changes to make these functions thread safe without changing the parameters, a copy of xref is needed than in the objects referenced by the info object, too, and this copy must be deleted in the destructors of every object where the copy is created.
So the thread safe version has an additional paramter, the thread own xref pointer, and this xref pointer is then injected in all objects referenced by the info object, and so it's quite easy to create an xref copy in the thread if necessary and delete this xref pointer when no more needed. That's what I did in the qt-library in poppler-document.cc.
Last but not least I cannot remove the non thread safe versions, they are public and therefore probably used in a lot of programs which use poppler.

(In reply to comment #62)
> (In reply to comment #54)
> > 
> > Seems a bit weird leaving the non thread safe versions of getDocInfo and
> > getDocInfoNF there, no? What's the use of them?
> > 
After explaining it, my brain began to work on it. And I think, I found a possible solution to remove the thread safe version and make the other calls thread safe instead of it. If You wish that, I'll do it. On the other hand, there are a lot of system calls which are commented in the form "this call is not thread safe, if Xou need that, call ... otherwise". And as a last statement: My work on this thread begin to exceed the time where I do it only experimentally. I'll continue only if the community wants to have a solution for it.

(In reply to comment #63)
> [...] And as a last
> statement: My work on this thread begin to exceed the time where I do it
> only experimentally. I'll continue only if the community wants to have a
> solution for it.

I am not sure what "this thread" refers to: If you mean the "getDocInfo" stuff, then I would just add that it seems to be part of the internal Poppler API which has no stability guarantees at all. Hence you could remove the old call and if someone used it, that is his own fault. (At least this is how I understood the mailing list discussions concerning such topics.)

If you refer to the whole threading work, as maintainer of qpdfview and hence Poppler user, I am very much interested in seeing this resolved. I was just testing your patch again yesterday and the perceived difference is rather noticeable. Especially when searching the document as the rendering does not stall the search any more. (Btw., I always have to apply with "--ignore-space-changes".)

Your work on this is very much appreciated. Thanks!

(In reply to comment #64)
> (In reply to comment #63)
> > [...] And as a last
> > statement: My work on this thread begin to exceed the time where I do it
> > only experimentally. I'll continue only if the community wants to have a
> > solution for it.
> 

Sorry, I was probably misunderstandable: What I meant was that it is hard to keep these changes appliable to git master. It would be much easier to have it committed and then make these smaller changes later...

> I am not sure what "this thread" refers to: If you mean the "getDocInfo"
> stuff, then I would just add that it seems to be part of the internal
> Poppler API which has no stability guarantees at all. Hence you could remove
> the old call and if someone used it, that is his own fault. (At least this
> is how I understood the mailing list discussions concerning such topics.)

Regarding getDocInfo: I think I made it TOO complicate. It can probably removed in PDFDoc.cc / *.h and we can call 

    XRef *xref = m_doc->doc->getXRef()->copy();
    Object *infoObj = xref->getDocInfo(&info);

in poppler-document.cc at the appropriate places. This is thread safe, too.

> 
> If you refer to the whole threading work, as maintainer of qpdfview and
> hence Poppler user, I am very much interested in seeing this resolved. I was
> just testing your patch again yesterday and the perceived difference is
> rather noticeable. Especially when searching the document as the rendering
> does not stall the search any more. (Btw., I always have to apply with
> "--ignore-space-changes".)
> 
> Your work on this is very much appreciated. Thanks!

Don't thanks for nothing :-)

helgrind (a valgrind tool) gives me warnings like

==32238== ----------------------------------------------------------------
==32238==
==32238== Possible data race during read of size 4 at 0x1B0929C8 by thread #10
==32238== Locks held: none
==32238==    at 0x1CDC1D02: Array::decRef() (Array.h:50)
==32238==    by 0x1CDC1802: Object::free() (Object.cc:133)
==32238==    by 0x1CD52085: Dict::~Dict() (Dict.cc:113)
==32238==    by 0x1CD62564: GfxResources::~GfxResources() (Gfx.cc:390)
==32238==    by 0x1CD7C713: Gfx::popResources() (Gfx.cc:5371)
==32238==    by 0x1CD7A5DD: Gfx::drawForm(Object*, Dict*, double*, double*, bool, bool, GfxColorSpace*, bool, bool, bool, Function*, GfxColor*) (Gfx.cc:4872)
==32238==    by 0x1CD7BDF1: Gfx::drawAnnot(Object*, AnnotBorder*, AnnotColor*, double, double, double, double) (Gfx.cc:5261)
==32238==    by 0x1CD3411B: AnnotWidget::draw(Gfx*, bool) (Annot.cc:4957)
==32238==    by 0x1CDC70D3: Page::displaySlice(OutputDev*, double, double, int, bool, bool, int, int, int, int, bool, bool (*)(void*), void*, bool (*)(Annot*, void*), void*, bool) (Page.cc:556)
==32238==    by 0x1CDCA689: PDFDoc::displayPageSlice(OutputDev*, int, double, double, int, bool, bool, bool, int, int, int, int, bool (*)(void*), void*, bool (*)(Annot*, void*), void*, bool) (PDFDoc.cc:518)
==32238==    by 0x1C9F59D5: Poppler::Page::textList(Poppler::Page::Rotation) const (poppler-page.cc:491)
==32238==    by 0x1C74B177: PDFGenerator::textPage(Okular::Page*) (generator_pdf.cpp:968)
==32238==
==32238== This conflicts with a previous write of size 4 by thread #9
==32238== Locks held: none
==32238==    at 0x1CDC1CEA: Array::incRef() (Array.h:49)
==32238==    by 0x1CDC16CA: Object::copy(Object*) (Object.cc:99)
==32238==    by 0x1CD51E95: Dict::Dict(Dict*) (Dict.cc:88)
==32238==    by 0x1CD51EE2: Dict::copy(XRef*) (Dict.cc:93)
==32238==    by 0x1CD6221E: GfxResources::GfxResources(XRef*, Dict*, GfxResources*) (Gfx.cc:333)
==32238==    by 0x1CD7C6CE: Gfx::pushResources(Dict*) (Gfx.cc:5364)
==32238==    by 0x1CD79F5A: Gfx::drawForm(Object*, Dict*, double*, double*, bool, bool, GfxColorSpace*, bool, bool, bool, Function*, GfxColor*) (Gfx.cc:4789)
==32238==    by 0x1CD7BDF1: Gfx::drawAnnot(Object*, AnnotBorder*, AnnotColor*, double, double, double, double) (Gfx.cc:5261)
==32238==
==32238== Address 0x1B0929C8 is 24 bytes inside a block of size 32 alloc'd
==32238==    at 0x4C2B377: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==32238==    by 0x1CDC1530: Object::initArray(XRef*) (Object.cc:66)
==32238==    by 0x1CDC87E9: Parser::getObj(Object*, bool, unsigned char*, CryptAlgorithm, int, int, int, int, bool) (Parser.cc:91)
==32238==    by 0x1CDC8AA9: Parser::getObj(Object*, bool, unsigned char*, CryptAlgorithm, int, int, int, int, bool) (Parser.cc:119)
==32238==    by 0x1CDE3A6B: XRef::fetch(int, int, Object*, int) (XRef.cc:1161)
==32238==    by 0x1CDC175E: Object::fetch(XRef*, Object*, int) (Object.cc:121)
==32238==    by 0x1CD5250D: Dict::lookup(char const*, Object*, int) (Dict.cc:220)
==32238==    by 0x1CD3BC6A: Object::dictLookup(char const*, Object*, int) (in /home/kdeunstable/instalado/lib/libpoppler.so.29.0.0)
==32238==    by 0x1CD5889B: Form::Form(PDFDoc*, Object*) (Form.cc:1329)
==32238==    by 0x1CD44235: Catalog::getForm() (Catalog.cc:920)
==32238==    by 0x1CD3AF7C: Annots::createAnnot(Dict*, Object*) (Annot.cc:6568)
==32238==    by 0x1CD3A7AF: Annots::Annots(PDFDoc*, int, Object*) (Annot.cc:6478)

Do we need some more locks or helgrind is just reporting wrong?

Created attachment 68025 [details] [review]
lock inc/decRef final patch two

(In reply to comment #66)
> 
> Do we need some more locks or helgrind is just reporting wrong?

helgrind is reporting right, we need some more locks (even if I cannot reconstruct it: if I start okular with helgrind, I get tons of error messages in the qt area, but I can't find a report like Yours).
I'm not completely sure if in this case it would be better to copy the array, too. But I haven't found a testcase for it, and locks are needed in either case, not only for arrays, but for dicts and streams, too, because it is possible that they are shared over threads in some cases.
BTW, because I had to create a new patch anyway, I also changed the getDocInfo stuff now.

Created attachment 68840 [details] [review]
a new rebased patch

I just encountered that the patch is no more appliable. So I created a rebased patch. I also changed the preprocessor directive to POPPLER_QT4_OVERPRINT_PREVIEW how Adam mentioned.

(In reply to comment #68)
> Created attachment 68840 [details] [review] [review]
> a new rebased patch
> 
> I just encountered that the patch is no more appliable. So I created a
> rebased patch. I also changed the preprocessor directive to
> POPPLER_QT4_OVERPRINT_PREVIEW how Adam mentioned.

Limited testing of parallel rendering using this latest patch does seem to work as expected.

Concerning a more fundamental question: It is a big patch, so it takes very much to review and I understand that it does not get into master easily. But it seems a lot of work to continuous rebase it. From looking at the code, it seems unlikely to me that it will introduce regressions in the _single-threaded_ usage of Poppler, i.e. programs that do not take advantage of these changes should not be affected. (Aside from the slight performance decrease because of less sharing that is.)

I really do not want to be presumptuous. I am not the maintainer, so I do not have to make the though calls. However, my personal opinion is that it would probably make sense to be bold and push this even if we find a few problems by long-time usage. (It could also receive more testing in the Poppler prereleases leading to 0.22 this way.)

Created attachment 69482 [details] [review]
only rebased, not tested again

(In reply to comment #70)
> Created attachment 69482 [details] [review] [review]
> only rebased, not tested again

Tested using qpdfview with my limited collection of mostly TeX-generated documents. And just for fun using overprint preview on the GWG output suite. :-) No obvious regressions spotted.

Created attachment 69495 [details] [review]
Last try for this weekend :-)

I was thinking since this is growing bigger and bigger (with my request to try to improve the okular usecase) maybe we could go back to the code that "just" enabled multi-threaded rendering, review that, commit that and once that is in build up from there?

Thomas do you have the patch that "just" did the multi-threaded rendering?

Created attachment 69504 [details] [review]
make page rendering thread-safe

(In reply to comment #73)
> I was thinking since this is growing bigger and bigger (with my request to
> try to improve the okular usecase) maybe we could go back to the code that
> "just" enabled multi-threaded rendering, review that, commit that and once
> that is in build up from there?
> 
> Thomas do you have the patch that "just" did the multi-threaded rendering?

Here it is.

It seems that you miss a few gDestroyMutex, calls, no?

Also there seems to be a huge lock-contention (at leat with your pdftoppm test), running the patched pdftoppm with USE_THREADS and MAX_THREADS of 8 i never got cpu usage over 100% (where 100% is one core used) of my 8 cores (i expected it to go close to 700% at least) and the number of pages per second also dropped, in a given PDF with lots of pages and 20 seconds of rendering it went down from around 250 to around 170 pages rendered when using multithreading.

Any idea why that might be happening?

(In reply to comment #76)
> Also there seems to be a huge lock-contention (at leat with your pdftoppm
> test), running the patched pdftoppm with USE_THREADS and MAX_THREADS of 8 i
> never got cpu usage over 100% (where 100% is one core used) of my 8 cores (i
> expected it to go close to 700% at least) and the number of pages per second
> also dropped, in a given PDF with lots of pages and 20 seconds of rendering
> it went down from around 250 to around 170 pages rendered when using
> multithreading.
> 
> Any idea why that might be happening?

Probably a problem of using sleep(), which has one second as minimal value when all threads are used. Can You try to increase the MAX_THREADS? I used 5 for my dual core during testing, and it became faster!

(In reply to comment #75)
> It seems that you miss a few gDestroyMutex, calls, no?

Yes, You're true: I haven't thought about that because I normaly using code where it is destroyed automatlically in the destructor (like You are doing in okular). I'll insert the calls coming weekend.

Right, changing the sleep(1) to usleep(100000) helped.

Created attachment 70178 [details] [review]
thread safe page rendering with gDestroyMutex

(In reply to comment #75)
> It seems that you miss a few gDestroyMutex, calls, no?

Here the rebased and changed patch. I tested it only basically, because the changes are quite simple.

  if (xref != NULL) {
    xref = xrefA;
  }

Shoud the if be xrefA and not xref?

I found a deadlock when running your pdftoppm over the pdf at http://ompldr.org/vOWlsbQ

Here the backtraces of all the 9 threads

Thread 9 (Thread 0x7fffed7fa700 (LWP 11535)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x7fffcc006e88) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6e84 in XRef::fetch (this=0x7fffcc006de0, num=49, gen=0, obj=0x7fffed7f96e0, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7ad1271 in arrayGet (obj=0x7fffed7f96e0, i=1, this=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:282
#5  Stream::addFilters (this=0x7fffcc024060, dict=<optimized out>, recursion=0) at /home/tsdgeos/devel/poppler/poppler/Stream.cc:191
#6  0x00007ffff7ac399b in Parser::makeStream (this=this@entry=0x7fffcc0239d0, dict=dict@entry=0x7fffed7f9950, fileKey=fileKey@entry=0x0, encAlgorithm=encAlgorithm@entry=(unknown: 1146103040), keyLength=keyLength@entry=774974790, objNum=objNum@entry=50, 
    objGen=objGen@entry=0, recursion=recursion@entry=1, strict=strict@entry=false) at /home/tsdgeos/devel/poppler/poppler/Parser.cc:274
#7  0x00007ffff7ac3d14 in Parser::getObj (this=0x7fffcc0239d0, obj=0x7fffed7f9950, simpleOnly=<optimized out>, fileKey=0x0, encAlgorithm=(unknown: 1146103040), keyLength=774974790, objNum=50, objGen=0, recursion=0, strict=false)
    at /home/tsdgeos/devel/poppler/poppler/Parser.cc:129
#8  0x00007ffff7ad7306 in XRef::fetch (this=0x7fffcc006de0, num=50, gen=0, obj=0x7fffed7f9950, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1164
#9  0x00007ffff7a76a78 in dictLookup (obj=0x7fffed7f9950, key=0x7fffcc022c30 "XIPLAYER0", this=<optimized out>, recursion=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:309
#10 GfxResources::lookupXObject (this=<optimized out>, name=name@entry=0x7fffcc022c30 "XIPLAYER0", obj=obj@entry=0x7fffed7f9950) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:412
#11 0x00007ffff7a85d14 in Gfx::opXObject (this=0x7fffcc007b10, args=<optimized out>, numArgs=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:4109
#12 0x00007ffff7a81ac4 in Gfx::go (this=this@entry=0x7fffcc007b10, topLevel=topLevel@entry=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:722
#13 0x00007ffff7a81f30 in Gfx::display (this=0x7fffcc007b10, obj=<optimized out>, topLevel=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:688
#14 0x00007ffff7ac2e15 in Page::displaySlice (this=0x7fffcc007a40, out=0x7fffcc0008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1755, sliceH=1240, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:538
#15 0x0000000000402c57 in savePageSlice (doc=0x636030, splashOut=0x7fffcc0008c0, pg=8, x=0, y=0, w=1755, h=1240, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x7fffcc007f70 "foo-008.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#16 0x0000000000403260 in processPage (pageInfo=0x6061d0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#17 0x00007ffff7764e9a in start_thread (arg=0x7fffed7fa700) at pthread_create.c:308
#18 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fffedffb700 (LWP 11534)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x7fffd8006e88) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6e84 in XRef::fetch (this=0x7fffd8006de0, num=42, gen=0, obj=0x7fffedffa6e0, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7ad1271 in arrayGet (obj=0x7fffedffa6e0, i=1, this=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:282
#5  Stream::addFilters (this=0x7fffd8023c80, dict=<optimized out>, recursion=0) at /home/tsdgeos/devel/poppler/poppler/Stream.cc:191
#6  0x00007ffff7ac399b in Parser::makeStream (this=this@entry=0x7fffd80235f0, dict=dict@entry=0x7fffedffa950, fileKey=fileKey@entry=0x0, encAlgorithm=encAlgorithm@entry=(unknown: 1146103040), keyLength=keyLength@entry=774974790, objNum=objNum@entry=43, 
    objGen=objGen@entry=0, recursion=recursion@entry=1, strict=strict@entry=false) at /home/tsdgeos/devel/poppler/poppler/Parser.cc:274
#7  0x00007ffff7ac3d14 in Parser::getObj (this=0x7fffd80235f0, obj=0x7fffedffa950, simpleOnly=<optimized out>, fileKey=0x0, encAlgorithm=(unknown: 1146103040), keyLength=774974790, objNum=43, objGen=0, recursion=0, strict=false)
    at /home/tsdgeos/devel/poppler/poppler/Parser.cc:129
#8  0x00007ffff7ad7306 in XRef::fetch (this=0x7fffd8006de0, num=43, gen=0, obj=0x7fffedffa950, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1164
#9  0x00007ffff7a76a78 in dictLookup (obj=0x7fffedffa950, key=0x7fffd8022850 "XIPLAYER0", this=<optimized out>, recursion=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:309
#10 GfxResources::lookupXObject (this=<optimized out>, name=name@entry=0x7fffd8022850 "XIPLAYER0", obj=obj@entry=0x7fffedffa950) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:412
#11 0x00007ffff7a85d14 in Gfx::opXObject (this=0x7fffd8007730, args=<optimized out>, numArgs=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:4109
#12 0x00007ffff7a81ac4 in Gfx::go (this=this@entry=0x7fffd8007730, topLevel=topLevel@entry=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:722
#13 0x00007ffff7a81f30 in Gfx::display (this=0x7fffd8007730, obj=<optimized out>, topLevel=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:688
#14 0x00007ffff7ac2e15 in Page::displaySlice (this=0x7fffd8007660, out=0x7fffd80008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1755, sliceH=1240, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:538
#15 0x0000000000402c57 in savePageSlice (doc=0x636030, splashOut=0x7fffd80008c0, pg=7, x=0, y=0, w=1755, h=1240, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x7fffd8007b90 "foo-007.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#16 0x0000000000403260 in processPage (pageInfo=0x606190) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#17 0x00007ffff7764e9a in start_thread (arg=0x7fffedffb700) at pthread_create.c:308
#18 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fffee7fc700 (LWP 11533)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x7fffd4006e88) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6e84 in XRef::fetch (this=0x7fffd4006de0, num=35, gen=0, obj=0x7fffee7fb6e0, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7ad1271 in arrayGet (obj=0x7fffee7fb6e0, i=1, this=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:282
#5  Stream::addFilters (this=0x7fffd4023ca0, dict=<optimized out>, recursion=0) at /home/tsdgeos/devel/poppler/poppler/Stream.cc:191
#6  0x00007ffff7ac399b in Parser::makeStream (this=this@entry=0x7fffd4023610, dict=dict@entry=0x7fffee7fb950, fileKey=fileKey@entry=0x0, encAlgorithm=encAlgorithm@entry=(unknown: 1146103040), keyLength=keyLength@entry=774974790, objNum=objNum@entry=36, 
    objGen=objGen@entry=0, recursion=recursion@entry=1, strict=strict@entry=false) at /home/tsdgeos/devel/poppler/poppler/Parser.cc:274
#7  0x00007ffff7ac3d14 in Parser::getObj (this=0x7fffd4023610, obj=0x7fffee7fb950, simpleOnly=<optimized out>, fileKey=0x0, encAlgorithm=(unknown: 1146103040), keyLength=774974790, objNum=36, objGen=0, recursion=0, strict=false)
    at /home/tsdgeos/devel/poppler/poppler/Parser.cc:129
#8  0x00007ffff7ad7306 in XRef::fetch (this=0x7fffd4006de0, num=36, gen=0, obj=0x7fffee7fb950, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1164
#9  0x00007ffff7a76a78 in dictLookup (obj=0x7fffee7fb950, key=0x7fffd4022870 "XIPLAYER0", this=<optimized out>, recursion=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:309
#10 GfxResources::lookupXObject (this=<optimized out>, name=name@entry=0x7fffd4022870 "XIPLAYER0", obj=obj@entry=0x7fffee7fb950) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:412
#11 0x00007ffff7a85d14 in Gfx::opXObject (this=0x7fffd4007750, args=<optimized out>, numArgs=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:4109
#12 0x00007ffff7a81ac4 in Gfx::go (this=this@entry=0x7fffd4007750, topLevel=topLevel@entry=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:722
#13 0x00007ffff7a81f30 in Gfx::display (this=0x7fffd4007750, obj=<optimized out>, topLevel=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:688
#14 0x00007ffff7ac2e15 in Page::displaySlice (this=0x7fffd4007680, out=0x7fffd40008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1755, sliceH=1240, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:538
#15 0x0000000000402c57 in savePageSlice (doc=0x636030, splashOut=0x7fffd40008c0, pg=6, x=0, y=0, w=1755, h=1240, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x7fffd4007bb0 "foo-006.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#16 0x0000000000403260 in processPage (pageInfo=0x606150) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#17 0x00007ffff7764e9a in start_thread (arg=0x7fffee7fc700) at pthread_create.c:308
#18 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fffeeffd700 (LWP 11532)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x7fffe0006e88) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6e84 in XRef::fetch (this=0x7fffe0006de0, num=28, gen=0, obj=0x7fffeeffc6e0, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7ad1271 in arrayGet (obj=0x7fffeeffc6e0, i=1, this=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:282
#5  Stream::addFilters (this=0x7fffe0023c80, dict=<optimized out>, recursion=0) at /home/tsdgeos/devel/poppler/poppler/Stream.cc:191
#6  0x00007ffff7ac399b in Parser::makeStream (this=this@entry=0x7fffe00235f0, dict=dict@entry=0x7fffeeffc950, fileKey=fileKey@entry=0x0, encAlgorithm=encAlgorithm@entry=(unknown: 1146103040), keyLength=keyLength@entry=774974790, objNum=objNum@entry=29, 
    objGen=objGen@entry=0, recursion=recursion@entry=1, strict=strict@entry=false) at /home/tsdgeos/devel/poppler/poppler/Parser.cc:274
#7  0x00007ffff7ac3d14 in Parser::getObj (this=0x7fffe00235f0, obj=0x7fffeeffc950, simpleOnly=<optimized out>, fileKey=0x0, encAlgorithm=(unknown: 1146103040), keyLength=774974790, objNum=29, objGen=0, recursion=0, strict=false)
    at /home/tsdgeos/devel/poppler/poppler/Parser.cc:129
#8  0x00007ffff7ad7306 in XRef::fetch (this=0x7fffe0006de0, num=29, gen=0, obj=0x7fffeeffc950, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1164
#9  0x00007ffff7a76a78 in dictLookup (obj=0x7fffeeffc950, key=0x7fffe0022850 "XIPLAYER0", this=<optimized out>, recursion=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:309
#10 GfxResources::lookupXObject (this=<optimized out>, name=name@entry=0x7fffe0022850 "XIPLAYER0", obj=obj@entry=0x7fffeeffc950) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:412
#11 0x00007ffff7a85d14 in Gfx::opXObject (this=0x7fffe0007730, args=<optimized out>, numArgs=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:4109
#12 0x00007ffff7a81ac4 in Gfx::go (this=this@entry=0x7fffe0007730, topLevel=topLevel@entry=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:722
#13 0x00007ffff7a81f30 in Gfx::display (this=0x7fffe0007730, obj=<optimized out>, topLevel=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:688
#14 0x00007ffff7ac2e15 in Page::displaySlice (this=0x7fffe0007660, out=0x7fffe00008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1755, sliceH=1240, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:538
#15 0x0000000000402c57 in savePageSlice (doc=0x636030, splashOut=0x7fffe00008c0, pg=5, x=0, y=0, w=1755, h=1240, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x7fffe0007b90 "foo-005.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#16 0x0000000000403260 in processPage (pageInfo=0x606110) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#17 0x00007ffff7764e9a in start_thread (arg=0x7fffeeffd700) at pthread_create.c:308
#18 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fffef7fe700 (LWP 11531)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x7fffdc006e88) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6e84 in XRef::fetch (this=0x7fffdc006de0, num=21, gen=0, obj=0x7fffef7fd6e0, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7ad1271 in arrayGet (obj=0x7fffef7fd6e0, i=1, this=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:282
#5  Stream::addFilters (this=0x7fffdc023ca0, dict=<optimized out>, recursion=0) at /home/tsdgeos/devel/poppler/poppler/Stream.cc:191
#6  0x00007ffff7ac399b in Parser::makeStream (this=this@entry=0x7fffdc023610, dict=dict@entry=0x7fffef7fd950, fileKey=fileKey@entry=0x0, encAlgorithm=encAlgorithm@entry=(unknown: 1146103040), keyLength=keyLength@entry=774974790, objNum=objNum@entry=22, 
    objGen=objGen@entry=0, recursion=recursion@entry=1, strict=strict@entry=false) at /home/tsdgeos/devel/poppler/poppler/Parser.cc:274
#7  0x00007ffff7ac3d14 in Parser::getObj (this=0x7fffdc023610, obj=0x7fffef7fd950, simpleOnly=<optimized out>, fileKey=0x0, encAlgorithm=(unknown: 1146103040), keyLength=774974790, objNum=22, objGen=0, recursion=0, strict=false)
    at /home/tsdgeos/devel/poppler/poppler/Parser.cc:129
#8  0x00007ffff7ad7306 in XRef::fetch (this=0x7fffdc006de0, num=22, gen=0, obj=0x7fffef7fd950, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1164
#9  0x00007ffff7a76a78 in dictLookup (obj=0x7fffef7fd950, key=0x7fffdc022870 "XIPLAYER0", this=<optimized out>, recursion=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:309
#10 GfxResources::lookupXObject (this=<optimized out>, name=name@entry=0x7fffdc022870 "XIPLAYER0", obj=obj@entry=0x7fffef7fd950) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:412
#11 0x00007ffff7a85d14 in Gfx::opXObject (this=0x7fffdc007750, args=<optimized out>, numArgs=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:4109
#12 0x00007ffff7a81ac4 in Gfx::go (this=this@entry=0x7fffdc007750, topLevel=topLevel@entry=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:722
#13 0x00007ffff7a81f30 in Gfx::display (this=0x7fffdc007750, obj=<optimized out>, topLevel=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:688
#14 0x00007ffff7ac2e15 in Page::displaySlice (this=0x7fffdc007680, out=0x7fffdc0008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1755, sliceH=1240, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:538
#15 0x0000000000402c57 in savePageSlice (doc=0x636030, splashOut=0x7fffdc0008c0, pg=4, x=0, y=0, w=1755, h=1240, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x7fffdc007bb0 "foo-004.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#16 0x0000000000403260 in processPage (pageInfo=0x6060d0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#17 0x00007ffff7764e9a in start_thread (arg=0x7fffef7fe700) at pthread_create.c:308
#18 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7fffeffff700 (LWP 11530)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x7fffe4006e88) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6e84 in XRef::fetch (this=0x7fffe4006de0, num=14, gen=0, obj=0x7fffefffe6e0, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7ad1271 in arrayGet (obj=0x7fffefffe6e0, i=1, this=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:282
#5  Stream::addFilters (this=0x7fffe4024690, dict=<optimized out>, recursion=0) at /home/tsdgeos/devel/poppler/poppler/Stream.cc:191
#6  0x00007ffff7ac399b in Parser::makeStream (this=this@entry=0x7fffe4024000, dict=dict@entry=0x7fffefffe950, fileKey=fileKey@entry=0x0, encAlgorithm=encAlgorithm@entry=(unknown: 1146103040), keyLength=keyLength@entry=774974790, objNum=objNum@entry=15, 
    objGen=objGen@entry=0, recursion=recursion@entry=1, strict=strict@entry=false) at /home/tsdgeos/devel/poppler/poppler/Parser.cc:274
#7  0x00007ffff7ac3d14 in Parser::getObj (this=0x7fffe4024000, obj=0x7fffefffe950, simpleOnly=<optimized out>, fileKey=0x0, encAlgorithm=(unknown: 1146103040), keyLength=774974790, objNum=15, objGen=0, recursion=0, strict=false)
    at /home/tsdgeos/devel/poppler/poppler/Parser.cc:129
#8  0x00007ffff7ad7306 in XRef::fetch (this=0x7fffe4006de0, num=15, gen=0, obj=0x7fffefffe950, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1164
#9  0x00007ffff7a76a78 in dictLookup (obj=0x7fffefffe950, key=0x7fffe4023260 "XIPLAYER0", this=<optimized out>, recursion=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:309
#10 GfxResources::lookupXObject (this=<optimized out>, name=name@entry=0x7fffe4023260 "XIPLAYER0", obj=obj@entry=0x7fffefffe950) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:412
#11 0x00007ffff7a85d14 in Gfx::opXObject (this=0x7fffe4008140, args=<optimized out>, numArgs=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:4109
#12 0x00007ffff7a81ac4 in Gfx::go (this=this@entry=0x7fffe4008140, topLevel=topLevel@entry=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:722
#13 0x00007ffff7a81f30 in Gfx::display (this=0x7fffe4008140, obj=<optimized out>, topLevel=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:688
#14 0x00007ffff7ac2e15 in Page::displaySlice (this=0x7fffe4008070, out=0x7fffe40008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1755, sliceH=1240, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:538
#15 0x0000000000402c57 in savePageSlice (doc=0x636030, splashOut=0x7fffe40008c0, pg=3, x=0, y=0, w=1755, h=1240, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x7fffe40085a0 "foo-003.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#16 0x0000000000403260 in processPage (pageInfo=0x606090) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#17 0x00007ffff7764e9a in start_thread (arg=0x7fffeffff700) at pthread_create.c:308
#18 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7ffff48e7700 (LWP 11529)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x7fffe8006ec8) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6e84 in XRef::fetch (this=0x7fffe8006e20, num=7, gen=0, obj=0x7ffff48e66e0, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7ad1271 in arrayGet (obj=0x7ffff48e66e0, i=1, this=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:282
#5  Stream::addFilters (this=0x7fffe8023260, dict=<optimized out>, recursion=0) at /home/tsdgeos/devel/poppler/poppler/Stream.cc:191
#6  0x00007ffff7ac399b in Parser::makeStream (this=this@entry=0x7fffe8022bd0, dict=dict@entry=0x7ffff48e6950, fileKey=fileKey@entry=0x0, encAlgorithm=encAlgorithm@entry=(unknown: 1146103040), keyLength=keyLength@entry=774974790, objNum=objNum@entry=8, 
    objGen=objGen@entry=0, recursion=recursion@entry=1, strict=strict@entry=false) at /home/tsdgeos/devel/poppler/poppler/Parser.cc:274
#7  0x00007ffff7ac3d14 in Parser::getObj (this=0x7fffe8022bd0, obj=0x7ffff48e6950, simpleOnly=<optimized out>, fileKey=0x0, encAlgorithm=(unknown: 1146103040), keyLength=774974790, objNum=8, objGen=0, recursion=0, strict=false)
    at /home/tsdgeos/devel/poppler/poppler/Parser.cc:129
#8  0x00007ffff7ad7306 in XRef::fetch (this=0x7fffe8006e20, num=8, gen=0, obj=0x7ffff48e6950, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1164
#9  0x00007ffff7a76a78 in dictLookup (obj=0x7ffff48e6950, key=0x7fffe8021e30 "XIPLAYER0", this=<optimized out>, recursion=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:309
#10 GfxResources::lookupXObject (this=<optimized out>, name=name@entry=0x7fffe8021e30 "XIPLAYER0", obj=obj@entry=0x7ffff48e6950) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:412
#11 0x00007ffff7a85d14 in Gfx::opXObject (this=0x7fffe8006cd0, args=<optimized out>, numArgs=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:4109
#12 0x00007ffff7a81ac4 in Gfx::go (this=this@entry=0x7fffe8006cd0, topLevel=topLevel@entry=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:722
#13 0x00007ffff7a81f30 in Gfx::display (this=0x7fffe8006cd0, obj=<optimized out>, topLevel=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:688
#14 0x00007ffff7ac2e15 in Page::displaySlice (this=0x7fffe4007680, out=0x7fffe80008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1755, sliceH=1240, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:538
#15 0x0000000000402c57 in savePageSlice (doc=0x636030, splashOut=0x7fffe80008c0, pg=2, x=0, y=0, w=1755, h=1240, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x7fffe8006cb0 "foo-002.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#16 0x0000000000403260 in processPage (pageInfo=0x606050) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#17 0x00007ffff7764e9a in start_thread (arg=0x7ffff48e7700) at pthread_create.c:308
#18 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7ffff50e8700 (LWP 11528)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x7ffff0006e88) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6e84 in XRef::fetch (this=0x7ffff0006de0, num=1814, gen=0, obj=0x7ffff50e76e0, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7ad1271 in arrayGet (obj=0x7ffff50e76e0, i=1, this=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:282
#5  Stream::addFilters (this=0x7ffff002cb30, dict=<optimized out>, recursion=0) at /home/tsdgeos/devel/poppler/poppler/Stream.cc:191
#6  0x00007ffff7ac399b in Parser::makeStream (this=this@entry=0x7ffff001be30, dict=dict@entry=0x7ffff50e7950, fileKey=fileKey@entry=0x0, encAlgorithm=encAlgorithm@entry=(unknown: 1146103040), keyLength=keyLength@entry=774974790, objNum=objNum@entry=1815, 
    objGen=objGen@entry=0, recursion=recursion@entry=1, strict=strict@entry=false) at /home/tsdgeos/devel/poppler/poppler/Parser.cc:274
#7  0x00007ffff7ac3d14 in Parser::getObj (this=0x7ffff001be30, obj=0x7ffff50e7950, simpleOnly=<optimized out>, fileKey=0x0, encAlgorithm=(unknown: 1146103040), keyLength=774974790, objNum=1815, objGen=0, recursion=0, strict=false)
    at /home/tsdgeos/devel/poppler/poppler/Parser.cc:129
#8  0x00007ffff7ad7306 in XRef::fetch (this=0x7ffff0006de0, num=1815, gen=0, obj=0x7ffff50e7950, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1164
#9  0x00007ffff7a76a78 in dictLookup (obj=0x7ffff50e7950, key=0x7ffff001b710 "XIPLAYER0", this=<optimized out>, recursion=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:309
#10 GfxResources::lookupXObject (this=<optimized out>, name=name@entry=0x7ffff001b710 "XIPLAYER0", obj=obj@entry=0x7ffff50e7950) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:412
#11 0x00007ffff7a85d14 in Gfx::opXObject (this=0x7ffff0008fd0, args=<optimized out>, numArgs=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:4109
#12 0x00007ffff7a81ac4 in Gfx::go (this=this@entry=0x7ffff0008fd0, topLevel=topLevel@entry=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:722
#13 0x00007ffff7a81f30 in Gfx::display (this=0x7ffff0008fd0, obj=<optimized out>, topLevel=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:688
#14 0x00007ffff7ac2e15 in Page::displaySlice (this=0x7ffff0008f00, out=0x7ffff00008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1240, sliceH=1755, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:538
#15 0x0000000000402c57 in savePageSlice (doc=0x636030, splashOut=0x7ffff00008c0, pg=1, x=0, y=0, w=1240, h=1755, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x7ffff0009430 "foo-001.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#16 0x0000000000403260 in processPage (pageInfo=0x606010) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#17 0x00007ffff7764e9a in start_thread (arg=0x7ffff50e8700) at pthread_create.c:308
#18 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ffff7fb3740 (LWP 11525)):
#0  0x00007ffff6e5e83d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007ffff6e8c774 in usleep (useconds=<optimized out>) at ../sysdeps/unix/sysv/linux/usleep.c:33
#2  0x0000000000402152 in main (argc=3, argv=<optimized out>) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:849

(In reply to comment #81)
>   if (xref != NULL) {
>     xref = xrefA;
>   }
> 
> Shoud the if be xrefA and not xref?

I think, You mean the occurence in CairoOutputDev::startPage?  Yes, I think that I made a mistake.

Created attachment 70196 [details] [review]
Try to solve #82

(In reply to comment #82)
> I found a deadlock when running your pdftoppm over the pdf at
> http://ompldr.org/vOWlsbQ
> 

First of all: Thanks that You start testing it. In the meantime I thought that there is no really interest at all. But now to this comment:
I'm wondering why this can happens: GfxResources should be copied for every thread. So a dead lock shouldn't occur never here.
But instead of searching where I missed to copy it, I locked GfxResources in the this patch too. I hope, it helps, but I'll have a deeper look into it tomorrow. Would be nice if You are able to regtest it in the meantime, I'm not really sure if I can provide a better solution tomorrow.

(In reply to comment #84)
> Created attachment 70196 [details] [review] [review]
> Try to solve #82
> 
> (In reply to comment #82)
> > I found a deadlock when running your pdftoppm over the pdf at
> > http://ompldr.org/vOWlsbQ
> > 
> 
> First of all: Thanks that You start testing it. In the meantime I thought
> that there is no really interest at all. But now to this comment:
> I'm wondering why this can happens: GfxResources should be copied for every
> thread. So a dead lock shouldn't occur never here.
> But instead of searching where I missed to copy it, I locked GfxResources in
> the this patch too. I hope, it helps, but I'll have a deeper look into it
> tomorrow. Would be nice if You are able to regtest it in the meantime, I'm
> not really sure if I can provide a better solution tomorrow.

Sorry, this patch probably doesn't help. I fear, that the new dict->copy() is not deep enough, because it copies just the dict entries. But I'm too tired today evening to upload a better solution. I'll try to do it tomorrow afternoon, but I can't promise that, I'll have several dates tomorrow!

Created attachment 70209 [details] [review]
solves comment #84

Damnish :-(. Just having a quick look into it today morning at breakfast, I encountered that this problem was already solved in the complete patch, but I missed to put the Parser.cc changes into the page render only stuff. and it was only one but important line :-(

Hmm, maybe you uploaded the wrong diff? It still deadlocks in that very same file here when using https://bugs.freedesktop.org/attachment.cgi?id=70209

Created attachment 70249 [details] [review]
solves comment #82

(In reply to comment #87)
> Hmm, maybe you uploaded the wrong diff? It still deadlocks in that very same
> file here when using https://bugs.freedesktop.org/attachment.cgi?id=70209

No, it was the right diff, but I haven't tested it with Your sample. I was already confused on saturday evening when I saw the arrayGet in the backtrace, but after I found the missing line in Parser.cc it seemed to be clear for me that I already solved that problem, but of course I should have tested it again.

The period I work on this becomes more and more too long, I already begin to forget a few things. But that's not Your problem, it's just a reminder to me that I should be more careful. So if this patch will not help (but it's now tested again the sample file), I'll do a complete regtest next weekend again.

We are supposed to release Beta 2 (the point we don't accept more features) this thursday.

What do you think we should do?
 * Release without the feature
 * Delay a few days to try to get it in?
 * Something else :D

(In reply to comment #89)
> We are supposed to release Beta 2 (the point we don't accept more features)
> this thursday.
> 
> What do you think we should do?
>  * Release without the feature
>  * Delay a few days to try to get it in?
>  * Something else :D

If You're able to run a regtest 'til thursday and it succeeds, please commit this part (or delay a few days if You need more time to review it).
If You're not able to run the regtest or the regtest still fails, release Beta 2 without that feature.
I promise that I have a thicker skin than Stefan :-)

Oh, i didn't see that comment #88 had a new patch, sure i'll test this one.

Ran the last patch and i found that running the "thread patched" pdftoppm gave me crashes in the following files

09_r100pdf_a.pdf
bug167299.pdf
bug243990.pdf
bug306818.pdf
bug-poppler10898.pdf
bug-poppler10910.pdf

And then I stopped regtesting and tried to have a look at fixing it myself but didn't get much far.

(In reply to comment #92)
> Ran the last patch and i found that running the "thread patched" pdftoppm
> gave me crashes in the following files
> 
> 09_r100pdf_a.pdf
> bug167299.pdf
> bug243990.pdf
> bug306818.pdf
> bug-poppler10898.pdf
> bug-poppler10910.pdf
> 
> And then I stopped regtesting and tried to have a look at fixing it myself
> but didn't get much far.

Sorry, I read that at work, and in my office I'm only available to get bug-poppler10898.pdf and bug-poppler10910.pdf. Both are corrupt PDFs, so I wouldn't really care about them for a first patch, but I tried them, but on my laptop the thread patched pdftoppm didn't crash.
I searched for the others in the internet, but I couldn't find them, so I'm not able to look at this problem at all without any further hints.

Thomas they should be part of the pdf's I gave you when you downloaded my test suite, do you have them around still?

(In reply to comment #94)
> Thomas they should be part of the pdf's I gave you when you downloaded my
> test suite, do you have them around still?

Probably I have them, but not in my office, only at home. And unfortunately just this week I hadn't the time to look at the problem in the evening. Therefore I asked for but probably not clear enough:
a) are all PDF You listed damaged?
b) what do You mean with crash? I didn't got a crash for the last two one.

If a) is the case, couldn't we commit the patch and solve this bug later? Nobody uses in the moment parallel rendering, and if we don't commit the patch today we have (according to Your time schedule) to wait at least until next year.
And if You then again start to have a look at it just four days before we froze it again we probably never can get it in... (no criticism, just a fact)

Here the backtrace from 09_r100pdf_a.pdf (which i'll attach now)

Note how Thread 3 tries to inc the ref of a 0x0 Annots

Thread 7 (Thread 0x7fffeeffd700 (LWP 29811)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x636bb8) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6f74 in XRef::fetch (this=0x636b10, num=1111, gen=0, obj=0x7fffeeffc830, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7a6be41 in fieldLookup (field=field@entry=0x7fffdc137b50, key=key@entry=0x7ffff7b312f2 "V", obj=obj@entry=0x7fffeeffc8e0, usedParents=usedParents@entry=0x7fffeeffc880) at /home/tsdgeos/devel/poppler/poppler/Form.cc:1499
#5  0x00007ffff7a6bed6 in Form::fieldLookup (field=field@entry=0x7fffdc137b50, key=key@entry=0x7ffff7b312f2 "V", obj=obj@entry=0x7fffeeffc8e0) at /home/tsdgeos/devel/poppler/poppler/Form.cc:1518
#6  0x00007ffff7a6cefe in FormFieldText::FormFieldText (this=0x7fffdc137a00, docA=<optimized out>, aobj=<optimized out>, ref=..., parent=<optimized out>, usedParents=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Form.cc:1003
#7  0x00007ffff7a6d3c4 in Form::createFieldFromDict (obj=0x7fffeeffc9c0, docA=0x636030, pref=..., parent=0x7fffdc1345d0, usedParents=0x7fffeeffca00) at /home/tsdgeos/devel/poppler/poppler/Form.cc:1529
#8  0x00007ffff7a6c60e in FormField::FormField (this=this@entry=0x7fffdc1345d0, docA=<optimized out>, aobj=<optimized out>, aref=..., parentA=<optimized out>, usedParents=0x7fffeeffcb60, ty=ty@entry=formText) at /home/tsdgeos/devel/poppler/poppler/Form.cc:510
#9  0x00007ffff7a6ce27 in FormFieldText::FormFieldText (this=0x7fffdc1345d0, docA=<optimized out>, aobj=<optimized out>, ref=..., parent=<optimized out>, usedParents=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Form.cc:971
#10 0x00007ffff7a6d3c4 in Form::createFieldFromDict (obj=obj@entry=0x7fffeeffcb40, docA=0x636030, pref=..., parent=parent@entry=0x0, usedParents=usedParents@entry=0x7fffeeffcb60) at /home/tsdgeos/devel/poppler/poppler/Form.cc:1529
#11 0x00007ffff7a6d6f6 in Form::Form (this=0x7fffdc071e80, docA=<optimized out>, acroFormA=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Form.cc:1456
#12 0x00007ffff7a5cee8 in Catalog::getForm (this=0x636d40) at /home/tsdgeos/devel/poppler/poppler/Catalog.cc:890
#13 0x00007ffff7a59abc in Annots::createAnnot (this=this@entry=0x7fffdc001010, dict=0x7fffdc019a50, obj=obj@entry=0x7fffeeffcc50) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6608
#14 0x00007ffff7a59c66 in Annots::Annots (this=0x7fffdc001010, docA=<optimized out>, page=6, annotsObj=0x7fffeeffcc90) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6518
#15 0x00007ffff7ac1deb in Page::getAnnots (this=this@entry=0x7fffdc006de0) at /home/tsdgeos/devel/poppler/poppler/Page.cc:364
#16 0x00007ffff7ac2f1f in Page::displaySlice (this=0x7fffdc006de0, out=0x7fffdc0008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1241, sliceH=1755, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:548
#17 0x0000000000402c17 in savePageSlice (doc=0x636030, splashOut=0x7fffdc0008c0, pg=6, x=0, y=0, w=1241, h=1755, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#18 0x0000000000403394 in processPage (pageInfo=0x606150) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:340
#19 0x00007ffff7764e9a in start_thread (arg=0x7fffeeffd700) at pthread_create.c:308
#20 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#21 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fffef7fe700 (LWP 29810)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x636bb8) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6f74 in XRef::fetch (this=0x636b10, num=747, gen=0, obj=0x7fffef7fdb00, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7a51315 in AnnotWidget::initialize (this=0x7fffd8151fc0, docA=<optimized out>, dict=0x7fffd80ceac0) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:3800
#5  0x00007ffff7a69e5c in FormWidget::createWidgetAnnotation (this=0x7fffd80cee00) at /home/tsdgeos/devel/poppler/poppler/Form.cc:106
#6  0x00007ffff7a6a424 in FormField::createWidgetAnnotations (this=0x7fffd80cfad0) at /home/tsdgeos/devel/poppler/poppler/Form.cc:637
#7  0x00007ffff7a6a3f4 in FormField::createWidgetAnnotations (this=0x7fffd80ce9d0) at /home/tsdgeos/devel/poppler/poppler/Form.cc:640
#8  0x00007ffff7a6bb7b in Form::postWidgetsLoad (this=0x7fffd801dfd0) at /home/tsdgeos/devel/poppler/poppler/Form.cc:1550
#9  0x00007ffff7a5cef4 in Catalog::getForm (this=0x636d40) at /home/tsdgeos/devel/poppler/poppler/Catalog.cc:892
#10 0x00007ffff7a59abc in Annots::createAnnot (this=this@entry=0x7fffd801e080, dict=0x7fffd801e0c0, obj=obj@entry=0x7fffef7fdc50) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6608
#11 0x00007ffff7a59c66 in Annots::Annots (this=0x7fffd801e080, docA=<optimized out>, page=5, annotsObj=0x7fffef7fdc90) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6518
#12 0x00007ffff7ac1deb in Page::getAnnots (this=this@entry=0x7fffd80093a0) at /home/tsdgeos/devel/poppler/poppler/Page.cc:364
#13 0x00007ffff7ac2f1f in Page::displaySlice (this=0x7fffd80093a0, out=0x7fffd80008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1241, sliceH=1755, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:548
#14 0x0000000000402c17 in savePageSlice (doc=0x636030, splashOut=0x7fffd80008c0, pg=5, x=0, y=0, w=1241, h=1755, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#15 0x0000000000403394 in processPage (pageInfo=0x606110) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:340
#16 0x00007ffff7764e9a in start_thread (arg=0x7fffef7fe700) at pthread_create.c:308
#17 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#18 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fffeffff700 (LWP 29809)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x636bb8) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6f74 in XRef::fetch (this=0x636b10, num=1073, gen=0, obj=0x7fffefffeb00, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7a51350 in AnnotWidget::initialize (this=0x7fffe4147600, docA=<optimized out>, dict=0x7fffe40b4de0) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:3807
#5  0x00007ffff7a69e5c in FormWidget::createWidgetAnnotation (this=0x7fffe40b5380) at /home/tsdgeos/devel/poppler/poppler/Form.cc:106
#6  0x00007ffff7a6a424 in FormField::createWidgetAnnotations (this=0x7fffe40b5e40) at /home/tsdgeos/devel/poppler/poppler/Form.cc:637
#7  0x00007ffff7a6a3f4 in FormField::createWidgetAnnotations (this=0x7fffe40b4cf0) at /home/tsdgeos/devel/poppler/poppler/Form.cc:640
#8  0x00007ffff7a6bb7b in Form::postWidgetsLoad (this=0x7fffe40189a0) at /home/tsdgeos/devel/poppler/poppler/Form.cc:1550
#9  0x00007ffff7a5cef4 in Catalog::getForm (this=0x636d40) at /home/tsdgeos/devel/poppler/poppler/Catalog.cc:892
#10 0x00007ffff7a59abc in Annots::createAnnot (this=this@entry=0x7fffe403a0a0, dict=0x7fffe4018ef0, obj=obj@entry=0x7fffefffec50) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6608
#11 0x00007ffff7a59c66 in Annots::Annots (this=0x7fffe403a0a0, docA=<optimized out>, page=4, annotsObj=0x7fffefffec90) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6518
#12 0x00007ffff7ac1deb in Page::getAnnots (this=this@entry=0x7fffd8008860) at /home/tsdgeos/devel/poppler/poppler/Page.cc:364
#13 0x00007ffff7ac2f1f in Page::displaySlice (this=0x7fffd8008860, out=0x7fffe40008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1241, sliceH=1755, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:548
#14 0x0000000000402c17 in savePageSlice (doc=0x636030, splashOut=0x7fffe40008c0, pg=4, x=0, y=0, w=1241, h=1755, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#15 0x0000000000403394 in processPage (pageInfo=0x6060d0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:340
#16 0x00007ffff7764e9a in start_thread (arg=0x7fffeffff700) at pthread_create.c:308
#17 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#18 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7ffff48e7700 (LWP 29808)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff7767065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff7766eba in __pthread_mutex_lock (mutex=0x636bb8) at pthread_mutex_lock.c:61
#3  0x00007ffff7ad6f74 in XRef::fetch (this=0x636b10, num=1045, gen=0, obj=0x7ffff48e6b00, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1103
#4  0x00007ffff7a51350 in AnnotWidget::initialize (this=0x7fffe0132f10, docA=<optimized out>, dict=0x7fffe005f820) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:3807
#5  0x00007ffff7a69e5c in FormWidget::createWidgetAnnotation (this=0x7fffe005ee70) at /home/tsdgeos/devel/poppler/poppler/Form.cc:106
#6  0x00007ffff7a6a424 in FormField::createWidgetAnnotations (this=0x7fffe005eb80) at /home/tsdgeos/devel/poppler/poppler/Form.cc:637
#7  0x00007ffff7a6a3f4 in FormField::createWidgetAnnotations (this=0x7fffe005e720) at /home/tsdgeos/devel/poppler/poppler/Form.cc:640
#8  0x00007ffff7a6bb7b in Form::postWidgetsLoad (this=0x7fffe00396a0) at /home/tsdgeos/devel/poppler/poppler/Form.cc:1550
#9  0x00007ffff7a5cef4 in Catalog::getForm (this=0x636d40) at /home/tsdgeos/devel/poppler/poppler/Catalog.cc:892
#10 0x00007ffff7a59abc in Annots::createAnnot (this=this@entry=0x7fffe0018650, dict=0x7fffe0018ef0, obj=obj@entry=0x7ffff48e6c50) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6608
#11 0x00007ffff7a59c66 in Annots::Annots (this=0x7fffe0018650, docA=<optimized out>, page=3, annotsObj=0x7ffff48e6c90) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6518
#12 0x00007ffff7ac1deb in Page::getAnnots (this=this@entry=0x7fffd8007d20) at /home/tsdgeos/devel/poppler/poppler/Page.cc:364
#13 0x00007ffff7ac2f1f in Page::displaySlice (this=0x7fffd8007d20, out=0x7fffe00008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1241, sliceH=1755, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:548
#14 0x0000000000402c17 in savePageSlice (doc=0x636030, splashOut=0x7fffe00008c0, pg=3, x=0, y=0, w=1241, h=1755, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#15 0x0000000000403394 in processPage (pageInfo=0x606090) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:340
#16 0x00007ffff7764e9a in start_thread (arg=0x7ffff48e7700) at pthread_create.c:308
#17 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#18 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fffec8e7700 (LWP 29807)):
#0  Annot::incRefCnt (this=this@entry=0x0) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:1519
#1  0x00007ffff7a59b00 in Annots::createAnnot (this=this@entry=0x7fffe8018460, dict=0x7fffe8018ef0, obj=obj@entry=0x7fffec8e6c50) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6613
#2  0x00007ffff7a59c66 in Annots::Annots (this=0x7fffe8018460, docA=<optimized out>, page=2, annotsObj=0x7fffec8e6c90) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6518
#3  0x00007ffff7ac1deb in Page::getAnnots (this=this@entry=0x7fffd8006de0) at /home/tsdgeos/devel/poppler/poppler/Page.cc:364
#4  0x00007ffff7ac2f1f in Page::displaySlice (this=0x7fffd8006de0, out=0x7fffe80008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1241, sliceH=1755, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:548
#5  0x0000000000402c17 in savePageSlice (doc=0x636030, splashOut=0x7fffe80008c0, pg=2, x=0, y=0, w=1241, h=1755, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#6  0x0000000000403394 in processPage (pageInfo=0x606050) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:340
#7  0x00007ffff7764e9a in start_thread (arg=0x7fffec8e7700) at pthread_create.c:308
#8  0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7ffff50e8700 (LWP 29806)):
#0  FileStream::getChar (this=0x7ffff01a6d70) at /home/tsdgeos/devel/poppler/poppler/Stream.h:453
#1  0x00007ffff7ab9847 in streamGetChar (this=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Object.h:342
#2  getChar (comesFromLook=<optimized out>, this=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Lexer.cc:125
#3  Lexer::getChar (this=0x7ffff01a7010, comesFromLook=true) at /home/tsdgeos/devel/poppler/poppler/Lexer.cc:115
#4  0x00007ffff7ab98ea in lookChar (this=0x7ffff01a7010) at /home/tsdgeos/devel/poppler/poppler/Lexer.cc:146
#5  Lexer::lookChar (this=0x7ffff01a7010) at /home/tsdgeos/devel/poppler/poppler/Lexer.cc:141
#6  0x00007ffff7ab9998 in Lexer::getObj (this=0x7ffff01a7010, obj=0x7ffff01a6f28, objNum=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Lexer.cc:565
#7  0x00007ffff7ac3cb3 in Parser::getObj (this=this@entry=0x7ffff01a6f00, obj=obj@entry=0x7ffff50e79e0, simpleOnly=simpleOnly@entry=false, fileKey=fileKey@entry=0x636b80 "\202GHL7\261c\363\322T\275\200\320", <incomplete sequence \357\220>, 
    encAlgorithm=encAlgorithm@entry=cryptRC4, keyLength=keyLength@entry=16, objNum=objNum@entry=1079, objGen=objGen@entry=0, recursion=recursion@entry=1, strict=strict@entry=false) at /home/tsdgeos/devel/poppler/poppler/Parser.cc:177
#8  0x00007ffff7ac4135 in Parser::getObj (this=0x7ffff01a6f00, obj=0x7ffff50e7b00, simpleOnly=<optimized out>, fileKey=0x636b80 "\202GHL7\261c\363\322T\275\200\320", <incomplete sequence \357\220>, encAlgorithm=cryptRC4, keyLength=16, objNum=1079, objGen=0, recursion=0, 
    strict=false) at /home/tsdgeos/devel/poppler/poppler/Parser.cc:119
#9  0x00007ffff7ad73f6 in XRef::fetch (this=0x636b10, num=1079, gen=0, obj=0x7ffff50e7b00, recursion=0) at /home/tsdgeos/devel/poppler/poppler/XRef.cc:1164
#10 0x00007ffff7a51350 in AnnotWidget::initialize (this=0x7ffff01a6720, docA=<optimized out>, dict=0x7ffff011f030) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:3807
#11 0x00007ffff7a69e5c in FormWidget::createWidgetAnnotation (this=0x7ffff011f2d0) at /home/tsdgeos/devel/poppler/poppler/Form.cc:106
#12 0x00007ffff7a6a424 in FormField::createWidgetAnnotations (this=0x7ffff011fdd0) at /home/tsdgeos/devel/poppler/poppler/Form.cc:637
#13 0x00007ffff7a6a3f4 in FormField::createWidgetAnnotations (this=0x7ffff0113930) at /home/tsdgeos/devel/poppler/poppler/Form.cc:640
#14 0x00007ffff7a6bb7b in Form::postWidgetsLoad (this=0x7ffff0009f40) at /home/tsdgeos/devel/poppler/poppler/Form.cc:1550
#15 0x00007ffff7a5cef4 in Catalog::getForm (this=0x636d40) at /home/tsdgeos/devel/poppler/poppler/Catalog.cc:892
#16 0x00007ffff7a59abc in Annots::createAnnot (this=this@entry=0x7ffff003ce50, dict=0x7ffff0031870, obj=obj@entry=0x7ffff50e7c50) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6608
#17 0x00007ffff7a59c66 in Annots::Annots (this=0x7ffff003ce50, docA=<optimized out>, page=1, annotsObj=0x7ffff50e7c90) at /home/tsdgeos/devel/poppler/poppler/Annot.cc:6518
#18 0x00007ffff7ac1deb in Page::getAnnots (this=this@entry=0x7ffff0006de0) at /home/tsdgeos/devel/poppler/poppler/Page.cc:364
#19 0x00007ffff7ac2f1f in Page::displaySlice (this=0x7ffff0006de0, out=0x7ffff00008c0, hDPI=<optimized out>, vDPI=<optimized out>, rotate=0, useMediaBox=true, crop=<optimized out>, sliceX=0, sliceY=0, sliceW=1241, sliceH=1755, printing=false, abortCheckCbk=0x0, 
    abortCheckCbkData=0x0, annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:548
#20 0x0000000000402c17 in savePageSlice (doc=0x636030, splashOut=0x7ffff00008c0, pg=1, x=0, y=0, w=1241, h=1755, pg_w=<optimized out>, pg_h=<optimized out>, ppmFile=0x0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#21 0x0000000000403394 in processPage (pageInfo=0x606010) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:340
#22 0x00007ffff7764e9a in start_thread (arg=0x7ffff50e8700) at pthread_create.c:308
#23 0x00007ffff6e92cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#24 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ffff7fb3740 (LWP 29803)):
#0  0x00007ffff6e5e83d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007ffff6e5e6dc in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
#2  0x000000000040340a in FinishAllThreads (thread=thread@entry=0x7fffffffe528) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:654
#3  0x00000000004021c1 in main (argc=2, argv=<optimized out>) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:908

Created attachment 70419 [details]
The said 09_r100pdf_a.pdf file

(In reply to comment #97)
> Created attachment 70419 [details]
> The said 09_r100pdf_a.pdf file

Just a short guess, I'll look at it right now: The reason is probably something which You didn't want to have in this patch, s. comment 54:

>>>>
The Annot.cc change does not belong here, right?
<<<<

:-)

Created attachment 70420 [details] [review]
solves comment #96

This is (only) the missing annot stuff which solves the crash in 09_r100pdf_a.pdf, not a full page render patch.
And I was true :-)

Sure, that if might fix the crash, but the question of why that variable is null (unlike when not using multiple threads that has a correct value) still remains, no?

(In reply to comment #100)
> Sure, that if might fix the crash, but the question of why that variable is
> null (unlike when not using multiple threads that has a correct value) still
> remains, no?

May be. But in a diff of the rendered output I didn't see any changes, as far as I remember. And that is what the patch should do: parallel rendering pages without changing output.
And one other comment to it: why could widget->getWidgetAnnotation() return a NULL pointer at all?
But I think it's even too late to commit this patch, so I'll have time enough to look at it.

Yes, having the same rendering is important, but to be honest i like to understand why the code is there and why it does what it does.

If you want to delay the release just propose it on the mailing list, i have no problem, it's just random dates i throw up and stick because no one complains.

(In reply to comment #102)
> Yes, having the same rendering is important, but to be honest i like to
> understand why the code is there and why it does what it does.

And You really understand every part of poppler code :-) ? Okay, kidding aside, I had a first deeper look into it: Annots are not thread safe because the Catalog is not thread safe because I removed that part from the patch when I create this page render only stuff. And I removed it because it wasn't there when I uploaded my first thread safe patch for page rendering only (which was regtested by me long time ago). Of course I could insert now the catalog part again and see what happens, but then we'll probably be back to my complete patch in a few hours.

> 
> If you want to delay the release just propose it on the mailing list, i have
> no problem, it's just random dates i throw up and stick because no one
> complains.

The problem in the code of poppler is the usage of doc and xref and doc->getXRef() everywhere. As I mentioned: xref is not thread safe and need therefore to be copied. I don't know a good way to make poppler thread safe in smaller steps without any compromises. If we want to do that we have probably to rewrite the complete code.
Therefore a delay of a few days will probably not help. Do You have any other suggestions?

I made some additional tests: I tried it without the Annot change but inserted the thread safe Catalog stuff again, but it didn't help, even with a few more changes.
So to get the annotation stuff thread safe we need a lot of more changes in the Annot-implementation, working only with the copied xref from Page. This is because of the tricky way annotations are created, see i.e. the comment in Form.cc:

<<<<
  // We create the widget annotations associated to
  // every form widget here, because the AnnotWidget constructor
  // needs the form object that gets from the catalog. When constructing
  // a FormWidget the Catalog is still creating the form object
>>>>

It's possible to do reimplement it, but not in a few hours and even not in a few days (everything must be regtested again). So we have either to live with the compromise or move this stuff to the next year.

(In reply to comment #103)
> (In reply to comment #102)
> > Yes, having the same rendering is important, but to be honest i like to
> > understand why the code is there and why it does what it does.
> 
> And You really understand every part of poppler code :-) ? 

No, but i understand that the guy sending the code should understand it ;-)

> Therefore a delay of a few days will probably not help. Do You have any
> other suggestions?

As said in coment #22 i think to make the "thread-safety" better we should remove PDFDoc::getXRef() and force everyone to use Gfx:::getXRef() or ask for an XRef an input, right?

Of course that is a long term-ish solution.

If you really want this in, i can add the "if" solution with a comment explaining that that "breaks" in the multithreaded rendering case but still gives proper rendering and that should be properly fixed in the future.

I'm still going to release beta 2 today, will ask the list how they feel about adding a new beta 3 that let's us add more features.

Does this work for you?

(In reply to comment #105)
> (In reply to comment #103)
> > (In reply to comment #102)
> > > Yes, having the same rendering is important, but to be honest i like to
> > > understand why the code is there and why it does what it does.
> > 
> > And You really understand every part of poppler code :-) ? 
> 
> No, but i understand that the guy sending the code should understand it ;-)

Hopefully, but sometimes I just use a "best guess" :-)
> 
> > Therefore a delay of a few days will probably not help. Do You have any
> > other suggestions?
> 
> As said in coment #22 i think to make the "thread-safety" better we should
> remove PDFDoc::getXRef() and force everyone to use Gfx:::getXRef() or ask
> for an XRef an input, right?

Agree

> 
> Of course that is a long term-ish solution.

Agree

> 
> If you really want this in, i can add the "if" solution with a comment
> explaining that that "breaks" in the multithreaded rendering case but still
> gives proper rendering and that should be properly fixed in the future.

The XRef-changes are the base for futur code changes! And hopefully the steps for correct poppler code to be thread safe at all are smaller if we have that in. One idea I got already yesterday to make the annot stuff thread safe in removing the doc references and using instead a xref reference. This hopefully changes only a few file (i.e. Page, Annot, Form and perhaps Catalog).

So another idea: if the patch (together with the small Annot change) doesn't break single thread page rendering and renders the page almost correctly with multi thread rendering, can we commit it with the hint, that multi thread rendering is still experimental? (and of course remove the QT4_MULTITHREAD directive? Than we and especially I have a base where I can continue working on, and if possible, we can add the Annot changes I'll make in the very next futur as bug fix?

> 
> I'm still going to release beta 2 today, will ask the list how they feel
> about adding a new beta 3 that let's us add more features.
> 
> Does this work for you?

Yes, of course!

Created attachment 70555 [details] [review]
solves the annotation problem

After running the complete day in the wrong direction I found now a clean solution to solve the annotation problem. I tested this patch with all files from comment 92 beside bug306818.pdf, that is not in my PDF suite.

Created attachment 70556 [details] [review]
solves the annotation problem

Sorry, during reviewing I encountered that I once again the gDestroyMutex in Catalog.cc, so here a new patch with the call

You can find bug306818.pdf at http://bugs.kde.org/306818

Can you have a look at bug-poppler13514.pdf -> https://bugs.freedesktop.org/attachment.cgi?id=12963 ?

I know the document is "broken" but when running the "old" pdftoppm it works and if i use the "new" one (with your testcase patch) it breaks and the top left paragraph of letters is not there.

Created attachment 70599 [details]
Output by pdftoppm -png -cropbox bug-poppler13514.pdf bug-poppler13514

Sorry, I can't see what You mean. I attach the output, produced on my laptop with 5 threads and my test case. The output is the same as with the head revision and what acrobat on ubuntu shows.

I don't get the "Life investors" and other sentences thing on the top left :-S

I'll give it another try now.

Don't know, the unpatched version gives me

tsdgeos@xps:~/devel/poppler/build-old$ ./utils/pdftoppm -png ~/okularfiles/pdf/bug-poppler13514.pdf borrame
Syntax Error: Couldn't find trailer dictionary
Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 9: reading configurations from ~/.fonts.conf is deprecated.

and a proper rendering, the patched version gives me

tsdgeos@xps:~/devel/poppler/build-new$ ./utils/pdftoppm -png ~/okularfiles/pdf/bug-poppler13514.pdf borrame
Syntax Error: Couldn't find trailer dictionary
Starting thread no '0'
Starting thread no '1'
Starting thread no '2'
Stopping thread no '0'
Syntax Error: font resource is not a dictionary
Syntax Error: font resource is not a dictionary
Syntax Error: font resource is not a dictionary
Syntax Error: font resource is not a dictionary
Syntax Error: font resource is not a dictionary
Syntax Error: font resource is not a dictionary
Syntax Error: font resource is not a dictionary
Syntax Error: Couldn't find trailer dictionary
Syntax Error: Unknown font tag 'F003'
Syntax Error (186900): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (186969): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187141): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187204): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187267): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187325): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187385): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187453): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187512): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187576): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187635): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187699): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187761): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187828): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187892): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (187955): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188015): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188084): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188141): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188212): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188272): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188379): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188437): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188496): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188557): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188616): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188674): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188738): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188799): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188859): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188920): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (188983): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (189057): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (189117): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (189177): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (189236): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error: Unknown font tag 'F003'
Syntax Error (189304): No font in show
Syntax Error (3667): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error: Unknown font tag 'F003'
Syntax Error (189364): No font in show
Syntax Error (3767): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error: Unknown font tag 'F003'
Syntax Error (189428): No font in show
Syntax Error (3865): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error: Unknown font tag 'F003'
Syntax Error (189487): No font in show
Syntax Error (3953): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error: Unknown font tag 'F003'
Syntax Error (189553): No font in show
Syntax Error (4054): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (189615): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (189675): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (189735): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (189902): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (189961): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190018): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190074): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190134): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190201): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190261): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190322): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190380): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190440): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190501): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190564): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190634): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190694): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190752): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190822): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190887): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (190952): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191020): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191081): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191143): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191203): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191261): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191320): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191383): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191446): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191504): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191563): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191633): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191693): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191753): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191819): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191878): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (191940): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192010): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192071): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192141): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192206): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192272): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192331): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192399): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192457): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192521): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192584): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192647): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192710): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192770): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192831): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192890): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (192955): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193014): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193072): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193133): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193192): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193252): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193311): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193371): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193435): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193500): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193559): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193619): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193680): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193739): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193800): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193858): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (193991): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194049): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194109): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194168): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194230): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194292): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194351): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194408): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194472): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194531): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194589): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194651): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194712): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194777): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194840): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194900): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (194969): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195029): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195089): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195154): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195214): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195273): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195334): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195398): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195462): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195521): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195582): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195644): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195705): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195765): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195825): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195887): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (195947): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196006): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196067): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196131): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196194): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196256): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196323): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196382): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196444): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196507): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196567): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196635): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196750): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196819): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196879): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (196936): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197000): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197063): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197123): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197182): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197246): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197305): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197363): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197427): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197487): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197551): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197613): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197672): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197732): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197791): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197851): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197915): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (197978): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198048): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198108): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198167): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198230): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198288): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198347): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198413): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198471): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198530): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198590): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198649): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198711): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198770): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198830): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198892): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (198952): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199011): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199075): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199141): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199206): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199267): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199326): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199390): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199455): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199514): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199573): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199636): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199697): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199755): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199817): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199876): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199938): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (199997): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200063): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200123): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200184): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200244): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200307): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200366): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200424): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200490): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200550): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200617): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200684): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200744): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (200873): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201021): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (4356): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201156): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201216): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201285): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201349): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201412): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201472): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201533): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error: Unknown font tag 'F003'
Syntax Error (201599): No font in show
Syntax Error (172251): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error: Unknown font tag 'F003'
Syntax Error (201659): No font in show
Syntax Error (172320): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201723): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201793): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201852): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201912): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (201977): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (202038): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (202097): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (202162): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (202225): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (202285): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (202440): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (4706): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (4797): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (4853): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (4909): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (5034): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (5093): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (5151): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (5208): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (5265): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (5334): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (5399): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (5466): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (5522): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (5578): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (5701): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (5783): No font in show
Syntax Error: Unknown font tag 'F004'
Syntax Error (5994): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error: Unknown font tag 'F003'
Syntax Error (6137): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (6240): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (6296): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (6407): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (6526): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (6642): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (6793): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (7032): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (7088): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (7144): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (7200): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (7263): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (7390): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (7446): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (7533): No font in show
Syntax Error: Unknown font tag 'F007'
Syntax Error (7756): No font in show
Syntax Error: Unknown font tag 'F007'
Syntax Error (7869): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (8022): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (8163): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (8225): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (8347): No font in show
Syntax Error: Unknown font tag 'F008'
Syntax Error (8403): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (172686): No font in show
Syntax Error: Unknown font tag 'F006'
Syntax Error (8826): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (172859): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (172919): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (173035): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (174130): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (174208): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (175597): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (175654): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (175713): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (175774): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (175833): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (175893): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (175957): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176021): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176084): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176144): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176205): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176264): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176329): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176388): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176446): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176507): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176566): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176626): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176691): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176751): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176811): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176869): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176927): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (176991): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177049): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177106): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177168): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177232): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177292): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177354): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177414): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177475): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177539): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177602): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177662): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177730): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177790): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177851): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177916): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (177977): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178036): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178100): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178159): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178220): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178335): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178393): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178456): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178516): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178576): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178633): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178700): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178761): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178819): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178880): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (178941): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179001): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179069): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179128): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179189): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179251): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179311): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179372): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179430): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179497): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179557): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179627): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179692): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179752): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179810): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179873): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179931): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (179997): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180060): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180128): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180186): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180253): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180314): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180378): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180437): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180501): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180561): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180625): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180686): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180745): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180809): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180870): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180929): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (180993): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181063): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181125): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181185): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181248): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181307): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181367): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181431): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181494): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181553): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181616): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181675): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181740): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181800): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181859): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181922): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (181982): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182052): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182111): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182176): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182233): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182292): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182355): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182414): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182477): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182536): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182595): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182659): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182718): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182786): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182847): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182907): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (182971): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183030): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183088): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183147): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183207): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183272): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183330): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183398): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183459): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183517): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183631): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183693): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183751): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183811): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183871): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183935): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (183993): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184050): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184115): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184174): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184238): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184303): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184368): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184427): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184488): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184552): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184611): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184672): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184733): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184792): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184852): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184913): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (184975): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185038): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185099): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185157): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185217): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185281): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185343): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185402): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185462): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185524): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185583): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185654): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185714): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185780): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185840): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185903): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (185965): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (186028): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (186087): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (186153): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (186213): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (186274): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (186334): No font in show
Syntax Error: Unknown font tag 'F003'
Syntax Error (186469): No font in show
Terminating thread_id '139726626637568'
Terminating thread_id '139726643422976'
Stopping thread no '1'
Terminating thread_id '139726635030272'

And as the warnings say, some fonts are missing :-/

(In reply to comment #113)
> Don't know, the unpatched version gives me
> 
> tsdgeos@xps:~/devel/poppler/build-old$ ./utils/pdftoppm -png
> ~/okularfiles/pdf/bug-poppler13514.pdf borrame
> Syntax Error: Couldn't find trailer dictionary
> Fontconfig warning: "/etc/fonts/conf.d/50-user.conf", line 9: reading
> configurations from ~/.fonts.conf is deprecated.
> 
> and a proper rendering, the patched version gives me
> 
> tsdgeos@xps:~/devel/poppler/build-new$ ./utils/pdftoppm -png
> ~/okularfiles/pdf/bug-poppler13514.pdf borrame
> Syntax Error: Couldn't find trailer dictionary

It seems as if we use different PDF files: I don't get that "Syntax Error: Couldn't find trailer dictionary", neither with head revision nor with my patch. The output is the same for both:

date;./utils/pdftoppm -png -cropbox /media/Data/Testdaten/kunden/bugzilla/merge/crash/bug-poppler13514.pdf bug-poppler13514; date
Di 27. Nov 10:39:16 CET 2012
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Syntax Warning: Mismatch between font type and embedded font file
Di 27. Nov 10:39:17 CET 2012

So can You mail me Your PDF in private?

Created attachment 70657 [details] [review]
solves comment #110

Uppps: Repair was done with a copy of the XRef (in Catalog constructor!), so none of the threads got the "repaired" xref. This was a very, very early design error (when I also wanted to use a copy of the Catalog and not to share the Catalog).
Quite easy to solve (and works now with the said PDF), hopefully it has no side effects.

Before anything, yes, i've been out this week too, i suck.

Did regtest the last patch and it still causes some strange differences in rendering, e.g. in bug-poppler26776.pdf in page 4 i get the attached renderings.

Created attachment 70850 [details]
Rendering of page4 of bug-poppler26776 with the new code

Created attachment 70851 [details]
Rendering of page4 of bug-poppler26776 with the old code

Created attachment 70852 [details]
Difference of endering of page4 of bug-poppler26776 between new and old code

First of all, yes, I run a new regtest today and got the same strange changes. But I'm not completely sure if it comes through the code changes by the patch or if the rendering changes because there are any changes in code. Once again You pick up a PDF which procudes tons of error messages like

Syntax Warning: Bad bounding box in Type 3 glyph

What is rendered if we get an error like this? I'll have a deeper look into it tomorrow, but as I encountered You have already close the 0.21, haven't You? So there is a lot of time looking into it, but isn't it a waste of time continue working on this thread?

(In reply to comment #120)
> First of all, yes, I run a new regtest today and got the same strange
> changes. But I'm not completely sure if it comes through the code changes by
> the patch 

Well, the changes in this patch are the only different between my old and new rendering, so yes, it would seem the changes in this patch are the ones causing this.

> I'll have a deeper look into
> it tomorrow, but as I encountered You have already close the 0.21, haven't
> You? 

If you mean i released Beta 3 and we don't accept new features for 0.22, yes you are right

> So there is a lot of time looking into it, but isn't it a waste of time
> continue working on this thread?

Why would it be a waste of time? Isn't this a feature we want?

(In reply to comment #119)
> Created attachment 70852 [details]
> Difference of endering of page4 of bug-poppler26776 between new and old code

After a lot of work the last weekends, often went one step forward and then two steps back, optimizing the code, fixing some areas and restructuring others, I now get now a solution which looks complete in my eyes, but I wasn't able to solve this problem. After looking hours into this PDF I encountered that it isn't a problem of the thread-safe page rendering, but a problem of the type 3 font cache in splash: this cache isn't exact, quite the contrary, it seems as if it produces random output.
You can see that if You just render page 4 (with -f 4 -l 4) with the old code: You'll get the same output as with the new code.
So a deeper look in the implementation of this cache is necessary to produce an exact or nearly exact output, indifferent if the char is taken from the cache or not. But this is another matter alltogether.
BTW, examining the left failed tests of my solution I encountered that we have a simular problem with the "normal" font cache using fontconfig: because it is not exact either sometimes the algorithm decides to take a cached font which output is other than if the font would be fetched new with fontconfig, in other words, the cached font file is another font file as if searched fresh with fontconfig.

Created attachment 72218 [details] [review]
a restructured solution for thread safe page rendering

Well, actually I wanted to wait until 0.22 is released before uploading a new patch. But after it looks complete in my eyes I wanted to run also a regtest with the cairo backend, even if I haven't created a test case for it (cairo should be thread-safe in page rendering now, too), just to see if I made some mistakes in the adjustments in the cairo code.
And so, here is the solution diff based on actual git master. Perhaps You find the time to have a look into it before 0.22 and add Your own comments.
Of course I'll create a rebased patch after 0.22 is released and You'll give me a go.

Just to be sure: Here a listing of the failed tests and my comments to it:

Algorithmics - The Spirit of Computing, 3rd Ed.pdf: page 284 & page 288: tiling pattern fill, but no visible differences
Essentials of English Grammar - www.ielts4u.blogfa.com.pdf, page 4, 5 & 16: uses different (not embedded!) fonts: font caching is no more global, only page based 
Radio Propagation and Adaptive Antennas for Wireless Communication Links (2006).pdf, page 140, 141, 183, 186, 211: tiling pattern fill, but no visible differences
SPU_ISA_v10.f8.pdf: stops rendering after page 117: head revision does the same, but instead of outputting an empty page, it duplicates page 117. If rendering only page 117, it outputs an empty page, too!
bug-poppler10898.pdf: page 30 not rendered (same as SPU_ISA_v10.f8.pdf)
bug-poppler10910.pdf: stops rendering after page 26 (same as SPU_ISA_v10.f8.pdf)
bug-poppler21754.pdf: page 115 & 117: problem with fonts (Bad bounding box in Type 3 glyph: type 3 fonts are cached in SplashOutputDev, which is global in head revision but page based in thread based rendering): just rendering page 115 or page 117 with head and thread revision has the same output!
bug-poppler26533.pdf: invisible differences, again tiling pattern fill
bug-poppler26776.pdf: several pages: problem with fonts (Bad bounding box in Type 3 glyph), same as bug-poppler21754.pdf
bug-poppler28357.pdf: invisible differences (only 1 page!), again tiling pattern fill
bug-poppler28806.pdf: only page 1 rendered (same as SPU_ISA_v10.f8.pdf)
dc-15-muentz-WP.f9.pdf: page 5 not rendered  (same as SPU_ISA_v10.f8.pdf)
political_world.pdf: invisible differences (only 1 page!), assume once again tiling pattern fill 
sinatr4c.f5.pdf: stops rendering after page 100 (same as SPU_ISA_v10.f8.pdf)
slls185d.f6.pdf: stops rendering after page 3 (same as SPU_ISA_v10.f8.pdf)
tauya.f8.pdf: some chars missing on almost all pages, but because it is really damaged and the result is not readable with either version, I like to ignore this output

I forgot that I wanted to tell:

Total 1131 tests
: : :
Tests run in 38 minutes and 15 seconds

Okay, that's together with the new regtest, which starts 8 worker threads on my new laptop, but it reduces together with my test case regtesting time awfully :-)

bug-poppler10898.pdf is still crashing here

time while [ true ]; do /home/tsdgeos/devel/poppler/build-debug/utils/pdftoppm -r 72 -cropbox -png ../bug-poppler10898.pdf render/foo; if [ $? -ne 0 ]; then break; fi; done

Will eventually crash (usually in less than 10 seconds, never more than 30s) when pdftoppm is the pdftoppm patched to work on threads with the code from this bugreport. With poppler 0.22 it won't crash (had it running for 5 minutes).

I know bug-poppler10898.pdf is broken, but the fact that pdftoppm is crashing randomly seems to imply there's something wrong, maybe something even thread-racy in the code.

Can you reproduce the issue? If not maybe i can give you access to my machine so you can give it a try?

(In reply to comment #126)
> bug-poppler10898.pdf is still crashing here
> 
> time while [ true ]; do
> /home/tsdgeos/devel/poppler/build-debug/utils/pdftoppm -r 72 -cropbox -png
> ../bug-poppler10898.pdf render/foo; if [ $? -ne 0 ]; then break; fi; done
> 
> Will eventually crash (usually in less than 10 seconds, never more than 30s)
> when pdftoppm is the pdftoppm patched to work on threads with the code from
> this bugreport. With poppler 0.22 it won't crash (had it running for 5
> minutes).
> 
> I know bug-poppler10898.pdf is broken, but the fact that pdftoppm is
> crashing randomly seems to imply there's something wrong, maybe something
> even thread-racy in the code.
> 
> Can you reproduce the issue? If not maybe i can give you access to my
> machine so you can give it a try?

It doesn't crashes on my laptop after 5 minutes, but what I can see on the output and looking again at comment 124 is that page 30 has only 14 bytes, and when I remember right from experiences in the past, there can be crashes in the png library with incomplete images (what means there is no really a bitmap). With pdftoppm without threads the output from page 30 is identical to page 29 (so it uses the former produced bitmap) and therefore will never crash there.
But to attest it, can You give me 
1. the backtrace of the crashing thread
2. the backtrace of all other threads
?
And perhaps You can also try Your loop with poppler 0.22 but only rendering page 30 with -f 30 -l 30? If I'm right, this should crashes then, too!

Adding -f 30 -l 30 actually makes it not crash anymore.

Note i have a slight change over your pdftoppm code, i've changed the Sleep(1) in the linux code to usleep(100000), that may be introducing the races you don't see.

backtraces looks a bit weird to be honest and they don't match either

BACKTRACE 1

Program received signal SIGILL, Illegal instruction.
Switching to Thread 12 (Thread 0x7ffff4849700 (LWP 24157))


Thread 13 (Thread 0x7ffff504a700 (LWP 24158)):
#0  _int_malloc (av=0x7ffff0000020, bytes=360) at malloc.c:3552
#1  0x00007ffff6d83fb5 in __GI___libc_malloc (bytes=360) at malloc.c:2924
#2  0x00007ffff741b60d in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x0000000000402e0b in processPage (pageInfo=0x6070d0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:297
#4  0x00007ffff76c6e9a in start_thread (arg=0x7ffff504a700) at pthread_create.c:308
#5  0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7ffff4849700 (LWP 24157)):
#0  0x00007ffff6a72af2 in ?? () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#1  0x00007ffff6a7789d in ft_mem_qrealloc () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#2  0x00007ffff6a7796b in ft_mem_realloc () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#3  0x00007ffff6a78a96 in FT_CMap_New () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#4  0x00007ffff6a93f7e in ?? () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#5  0x00007ffff6a78e51 in ?? () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#6  0x00007ffff6a7b662 in FT_Open_Face () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#7  0x00007ffff6a7c64b in FT_New_Face () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#8  0x00007ffff7b06673 in SplashFTFontFile::loadType1Font (engineA=0x7fffe8004bf0, idA=0x7fffe8010470, src=0x7fffe801bf50, encA=0x7fffe801c2e8) at /home/tsdgeos/devel/poppler/splash/SplashFTFontFile.cc:49
#9  0x00007ffff7b0632c in SplashFTFontEngine::loadType1Font (this=0x7fffe8004bf0, idA=0x7fffe8010470, src=0x7fffe801bf50, enc=0x7fffe801c2e8) at /home/tsdgeos/devel/poppler/splash/SplashFTFontEngine.cc:86
#10 0x00007ffff7b079e2 in SplashFontEngine::loadType1Font (this=0x7fffe8019150, idA=0x7fffe8010470, src=0x7fffe801bf50, enc=0x7fffe801c2e8) at /home/tsdgeos/devel/poppler/splash/SplashFontEngine.cc:147
#11 0x00007ffff7adea5c in SplashOutputDev::doUpdateFont (this=0x7fffe80008c0, state=0x7fffe80237d0) at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:1875
#12 0x00007ffff7adffd8 in SplashOutputDev::drawChar (this=0x7fffe80008c0, state=0x7fffe80237d0, x=70.799999999999997, y=802.08000000000004, dx=11.040000000000001, dy=0, originX=0, originY=0, code=87, nBytes=1, u=0x7fffe80066dc, uLen=1)
    at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:2207
#13 0x00007ffff7a3298c in Gfx::doShowText (this=0x7fffe8018d70, s=0x7fffe8019e50) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:3999
#14 0x00007ffff7a31036 in Gfx::opShowText (this=0x7fffe8018d70, args=0x7ffff4848950, numArgs=1) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:3742
#15 0x00007ffff7a200fe in Gfx::execOp (this=0x7fffe8018d70, cmd=0x7ffff4848910, args=0x7ffff4848950, numArgs=1) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:859
#16 0x00007ffff7a1f959 in Gfx::go (this=0x7fffe8018d70, topLevel=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:718
#17 0x00007ffff7a1f745 in Gfx::display (this=0x7fffe8018d70, obj=0x7ffff4848c90, topLevel=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:684
#18 0x00007ffff7a838b2 in Page::displaySlice (this=0x7fffe8004d40, out=0x7fffe80008c0, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, printing=false, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:574
#19 0x00007ffff7a87086 in PDFDoc::displayPageSlice (this=0x636fb0, out=0x7fffe80008c0, page=11, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, printing=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/PDFDoc.cc:522
#20 0x0000000000402b61 in savePageSlice (doc=0x636fb0, splashOut=0x7fffe80008c0, pg=11, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffe8005140 "render/foo-11.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#21 0x0000000000403282 in processPage (pageInfo=0x607090) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#22 0x00007ffff76c6e9a in start_thread (arg=0x7ffff4849700) at pthread_create.c:308
#23 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#24 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7fffeffff700 (LWP 24156)):
#0  0x00007ffff6833ebc in ?? () from /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
#1  0x00007ffff683496a in ?? () from /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
#2  0x00007ffff68358a8 in FcConfigSubstituteWithPat () from /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
#3  0x00007ffff7a64573 in GlobalParams::findSystemFontFile (this=0x607250, font=0x7fffe4002c80, type=0x7fffefffe2c8, fontNum=0x7fffefffe2cc, substituteFontName=0x0, base14Name=0x7fffe401f0b0) at /home/tsdgeos/devel/poppler/poppler/GlobalParams.cc:1189
#4  0x00007ffff7a643a7 in GlobalParams::findBase14FontFile (this=0x607250, base14Name=0x7fffe401f0b0, font=0x7fffe4002c80) at /home/tsdgeos/devel/poppler/poppler/GlobalParams.cc:1159
#5  0x00007ffff7a3bb32 in GfxFont::locateFont (this=0x7fffe4002c80, xref=0x7fffe40151c0, ps=false) at /home/tsdgeos/devel/poppler/poppler/GfxFont.cc:697
#6  0x00007ffff7ade882 in SplashOutputDev::doUpdateFont (this=0x7fffe40008c0, state=0x7fffe400c190) at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:1845
#7  0x00007ffff7adffd8 in SplashOutputDev::drawChar (this=0x7fffe40008c0, state=0x7fffe400c190, x=96.719999999999999, y=36.240000000000002, dx=7.6699200000000003, dy=0, originX=0, originY=0, code=83, nBytes=1, u=0x7fffe4003fcc, uLen=1)
    at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:2207
#8  0x00007ffff7a3298c in Gfx::doShowText (this=0x7fffe401e500, s=0x7fffe401f080) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:3999
#9  0x00007ffff7a31036 in Gfx::opShowText (this=0x7fffe401e500, args=0x7fffefffe950, numArgs=1) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:3742
#10 0x00007ffff7a200fe in Gfx::execOp (this=0x7fffe401e500, cmd=0x7fffefffe910, args=0x7fffefffe950, numArgs=1) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:859
#11 0x00007ffff7a1f959 in Gfx::go (this=0x7fffe401e500, topLevel=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:718
#12 0x00007ffff7a1f745 in Gfx::display (this=0x7fffe401e500, obj=0x7fffefffec90, topLevel=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:684
#13 0x00007ffff7a838b2 in Page::displaySlice (this=0x7fffe4015a00, out=0x7fffe40008c0, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, printing=false, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:574
#14 0x00007ffff7a87086 in PDFDoc::displayPageSlice (this=0x636fb0, out=0x7fffe40008c0, page=10, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, printing=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/PDFDoc.cc:522
#15 0x0000000000402b61 in savePageSlice (doc=0x636fb0, splashOut=0x7fffe40008c0, pg=10, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffe4015e00 "render/foo-10.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#16 0x0000000000403282 in processPage (pageInfo=0x607050) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#17 0x00007ffff76c6e9a in start_thread (arg=0x7fffeffff700) at pthread_create.c:308
#18 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#19 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7fffef7fe700 (LWP 24155)):
#0  0x00007ffff6ad456d in ?? () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#1  0x00007ffff6a91b90 in ?? () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#2  0x00007ffff6a906a1 in ?? () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#3  0x00007ffff6a9368b in ?? () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#4  0x00007ffff6a93cc8 in ?? () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#5  0x00007ffff6a78e51 in ?? () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#6  0x00007ffff6a7b662 in FT_Open_Face () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#7  0x00007ffff6a7c64b in FT_New_Face () from /usr/lib/x86_64-linux-gnu/libfreetype.so.6
#8  0x00007ffff7b06673 in SplashFTFontFile::loadType1Font (engineA=0x7fffdc05c670, idA=0x7fffdc04b160, src=0x7fffdc04b790, encA=0x7fffdc01d468) at /home/tsdgeos/devel/poppler/splash/SplashFTFontFile.cc:49
#9  0x00007ffff7b0632c in SplashFTFontEngine::loadType1Font (this=0x7fffdc05c670, idA=0x7fffdc04b160, src=0x7fffdc04b790, enc=0x7fffdc01d468) at /home/tsdgeos/devel/poppler/splash/SplashFTFontEngine.cc:86
#10 0x00007ffff7b079e2 in SplashFontEngine::loadType1Font (this=0x7fffdc010000, idA=0x7fffdc04b160, src=0x7fffdc04b790, enc=0x7fffdc01d468) at /home/tsdgeos/devel/poppler/splash/SplashFontEngine.cc:147
#11 0x00007ffff7adea5c in SplashOutputDev::doUpdateFont (this=0x7fffdc0008c0, state=0x7fffdc05f640) at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:1875
#12 0x00007ffff7adffd8 in SplashOutputDev::drawChar (this=0x7fffdc0008c0, state=0x7fffdc05f640, x=96.719999999999999, y=36.240000000000002, dx=7.6699200000000003, dy=0, originX=0, originY=0, code=83, nBytes=1, u=0x7fffdc01e74c, uLen=1)
    at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:2207
#13 0x00007ffff7a3298c in Gfx::doShowText (this=0x7fffdc05d3c0, s=0x7fffdc00fe30) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:3999
#14 0x00007ffff7a31036 in Gfx::opShowText (this=0x7fffdc05d3c0, args=0x7fffef7fd950, numArgs=1) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:3742
#15 0x00007ffff7a200fe in Gfx::execOp (this=0x7fffdc05d3c0, cmd=0x7fffef7fd910, args=0x7fffef7fd950, numArgs=1) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:859
#16 0x00007ffff7a1f959 in Gfx::go (this=0x7fffdc05d3c0, topLevel=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:718
#17 0x00007ffff7a1f745 in Gfx::display (this=0x7fffdc05d3c0, obj=0x7fffef7fdc90, topLevel=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:684
#18 0x00007ffff7a838b2 in Page::displaySlice (this=0x7fffdc05c7c0, out=0x7fffdc0008c0, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, printing=false, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:574
#19 0x00007ffff7a87086 in PDFDoc::displayPageSlice (this=0x636fb0, out=0x7fffdc0008c0, page=9, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, printing=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/PDFDoc.cc:522
#20 0x0000000000402b61 in savePageSlice (doc=0x636fb0, splashOut=0x7fffdc0008c0, pg=9, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffdc05cbc0 "render/foo-09.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#21 0x0000000000403282 in processPage (pageInfo=0x607010) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#22 0x00007ffff76c6e9a in start_thread (arg=0x7fffef7fe700) at pthread_create.c:308
#23 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#24 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fffed7fa700 (LWP 24154)):
#0  0x00007ffff58abccb in ?? () from /lib/x86_64-linux-gnu/libz.so.1
#1  0x00007ffff58ac833 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
#2  0x00007ffff5f553b4 in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#3  0x00007ffff5f5587b in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#4  0x00007ffff5f59434 in png_write_row () from /lib/x86_64-linux-gnu/libpng12.so.0
#5  0x00007ffff5f59658 in png_write_image () from /lib/x86_64-linux-gnu/libpng12.so.0
#6  0x00007ffff79c049a in PNGWriter::writePointers (this=0x7fffcc007010, rowPointers=0x7fffcc017040, rowCount=841) at /home/tsdgeos/devel/poppler/goo/PNGWriter.cc:159
#7  0x00007ffff7b02ee5 in SplashBitmap::writeImgFile (this=0x7fffcc000a30, writer=0x7fffcc007010, f=0x7fffcc000f40, hDPI=72, vDPI=72) at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:652
#8  0x00007ffff7b01fbd in SplashBitmap::writeImgFile (this=0x7fffcc000a30, format=splashFormatPng, f=0x7fffcc000f40, hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:410
#9  0x00007ffff7b01e0b in SplashBitmap::writeImgFile (this=0x7fffcc000a30, format=splashFormatPng, fileName=0x7fffcc0071e0 "render/foo-08.png", hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:346
#10 0x0000000000402bbd in savePageSlice (doc=0x636fb0, splashOut=0x7fffcc0008c0, pg=8, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffcc0071e0 "render/foo-08.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:210
#11 0x0000000000403282 in processPage (pageInfo=0x6071d0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#12 0x00007ffff76c6e9a in start_thread (arg=0x7fffed7fa700) at pthread_create.c:308
#13 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fffedffb700 (LWP 24153)):
#0  0x00007ffff58aaa9f in ?? () from /lib/x86_64-linux-gnu/libz.so.1
#1  0x00007ffff58abcc8 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
#2  0x00007ffff58ac833 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
#3  0x00007ffff5f553b4 in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#4  0x00007ffff5f5587b in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#5  0x00007ffff5f59434 in png_write_row () from /lib/x86_64-linux-gnu/libpng12.so.0
#6  0x00007ffff5f59658 in png_write_image () from /lib/x86_64-linux-gnu/libpng12.so.0
#7  0x00007ffff79c049a in PNGWriter::writePointers (this=0x7fffd8008260, rowPointers=0x7fffd8021c10, rowCount=841) at /home/tsdgeos/devel/poppler/goo/PNGWriter.cc:159
#8  0x00007ffff7b02ee5 in SplashBitmap::writeImgFile (this=0x7fffd8000a30, writer=0x7fffd8008260, f=0x7fffd8000f40, hDPI=72, vDPI=72) at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:652
#9  0x00007ffff7b01fbd in SplashBitmap::writeImgFile (this=0x7fffd8000a30, format=splashFormatPng, f=0x7fffd8000f40, hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:410
#10 0x00007ffff7b01e0b in SplashBitmap::writeImgFile (this=0x7fffd8000a30, format=splashFormatPng, fileName=0x7fffd80087e0 "render/foo-07.png", hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:346
#11 0x0000000000402bbd in savePageSlice (doc=0x636fb0, splashOut=0x7fffd80008c0, pg=7, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffd80087e0 "render/foo-07.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:210
#12 0x0000000000403282 in processPage (pageInfo=0x607190) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#13 0x00007ffff76c6e9a in start_thread (arg=0x7fffedffb700) at pthread_create.c:308
#14 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#15 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fffee7fc700 (LWP 24152)):
#0  0x00007ffff58aab27 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
#1  0x00007ffff58abcc8 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
#2  0x00007ffff58ac833 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
#3  0x00007ffff5f553b4 in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#4  0x00007ffff5f5587b in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#5  0x00007ffff5f59434 in png_write_row () from /lib/x86_64-linux-gnu/libpng12.so.0
#6  0x00007ffff5f59658 in png_write_image () from /lib/x86_64-linux-gnu/libpng12.so.0
#7  0x00007ffff79c049a in PNGWriter::writePointers (this=0x7fffd4006db0, rowPointers=0x7fffd4015390, rowCount=841) at /home/tsdgeos/devel/poppler/goo/PNGWriter.cc:159
#8  0x00007ffff7b02ee5 in SplashBitmap::writeImgFile (this=0x7fffd4000a30, writer=0x7fffd4006db0, f=0x7fffd4000f40, hDPI=72, vDPI=72) at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:652
#9  0x00007ffff7b01fbd in SplashBitmap::writeImgFile (this=0x7fffd4000a30, format=splashFormatPng, f=0x7fffd4000f40, hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:410
#10 0x00007ffff7b01e0b in SplashBitmap::writeImgFile (this=0x7fffd4000a30, format=splashFormatPng, fileName=0x7fffd4006cb0 "render/foo-06.png", hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:346
#11 0x0000000000402bbd in savePageSlice (doc=0x636fb0, splashOut=0x7fffd40008c0, pg=6, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffd4006cb0 "render/foo-06.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:210
#12 0x0000000000403282 in processPage (pageInfo=0x607150) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#13 0x00007ffff76c6e9a in start_thread (arg=0x7fffee7fc700) at pthread_create.c:308
#14 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#15 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fffeeffd700 (LWP 24151)):
#0  0x00007ffff58aaa88 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
#1  0x00007ffff58abcc8 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
#2  0x00007ffff58ac833 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
#3  0x00007ffff5f553b4 in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#4  0x00007ffff5f5587b in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#5  0x00007ffff5f59434 in png_write_row () from /lib/x86_64-linux-gnu/libpng12.so.0
#6  0x00007ffff5f59658 in png_write_image () from /lib/x86_64-linux-gnu/libpng12.so.0
#7  0x00007ffff79c049a in PNGWriter::writePointers (this=0x7fffe0006db0, rowPointers=0x7fffe0016ad0, rowCount=841) at /home/tsdgeos/devel/poppler/goo/PNGWriter.cc:159
#8  0x00007ffff7b02ee5 in SplashBitmap::writeImgFile (this=0x7fffe0000a30, writer=0x7fffe0006db0, f=0x7fffe0057bf0, hDPI=72, vDPI=72) at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:652
#9  0x00007ffff7b01fbd in SplashBitmap::writeImgFile (this=0x7fffe0000a30, format=splashFormatPng, f=0x7fffe0057bf0, hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:410
#10 0x00007ffff7b01e0b in SplashBitmap::writeImgFile (this=0x7fffe0000a30, format=splashFormatPng, fileName=0x7fffe0006cb0 "render/foo-05.png", hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:346
#11 0x0000000000402bbd in savePageSlice (doc=0x636fb0, splashOut=0x7fffe00008c0, pg=5, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffe0006cb0 "render/foo-05.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:210
#12 0x0000000000403282 in processPage (pageInfo=0x607110) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#13 0x00007ffff76c6e9a in start_thread (arg=0x7fffeeffd700) at pthread_create.c:308
#14 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#15 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ffff7fb3740 (LWP 24146)):
#0  0x00007ffff6dc083d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007ffff6dee774 in usleep (useconds=<optimized out>) at ../sysdeps/unix/sysv/linux/usleep.c:33
#2  0x0000000000403d49 in main (argc=3, argv=0x7fffffffe618) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:849


BACKTRACE 2

Program received signal SIGSEGV, Segmentation fault.
Switching to Thread 11 (Thread 0x7fffef7fe700 (LWP 24170))
0x00007ffff7a7e3aa in Array::decRef (this=0x2b1029d01d501c1) at /home/tsdgeos/devel/poppler/poppler/Array.h:50


Thread 14 (Thread 0x7ffff4849700 (LWP 24173)):
#0  0x00007ffff6842327 in ?? () from /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
#1  0x00007ffff684269b in ?? () from /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
#2  0x00007ffff6843238 in FcFontSetSort () from /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
#3  0x00007ffff68437b2 in FcFontSort () from /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
#4  0x00007ffff7a645aa in GlobalParams::findSystemFontFile (this=0x607250, font=0x7fffe801db40, type=0x7ffff48482c8, fontNum=0x7ffff48482cc, substituteFontName=0x0, base14Name=0x7fffe8250290) at /home/tsdgeos/devel/poppler/poppler/GlobalParams.cc:1191
#5  0x00007ffff7a643a7 in GlobalParams::findBase14FontFile (this=0x607250, base14Name=0x7fffe8250290, font=0x7fffe801db40) at /home/tsdgeos/devel/poppler/poppler/GlobalParams.cc:1159
#6  0x00007ffff7a3bb32 in GfxFont::locateFont (this=0x7fffe801db40, xref=0x7fffe8005490, ps=false) at /home/tsdgeos/devel/poppler/poppler/GfxFont.cc:697
#7  0x00007ffff7ade882 in SplashOutputDev::doUpdateFont (this=0x7fffe80008c0, state=0x7fffe8012190) at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:1845
#8  0x00007ffff7adffd8 in SplashOutputDev::drawChar (this=0x7fffe80008c0, state=0x7fffe8012190, x=98.159999999999997, y=770.88, dx=3.3000000000000003, dy=0, originX=0, originY=0, code=32, nBytes=1, u=0x7fffe801ef00, uLen=1)
    at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:2207
#9  0x00007ffff7a3298c in Gfx::doShowText (this=0x7fffe800eee0, s=0x7fffe8010150) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:3999
#10 0x00007ffff7a31036 in Gfx::opShowText (this=0x7fffe800eee0, args=0x7ffff4848950, numArgs=1) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:3742
#11 0x00007ffff7a200fe in Gfx::execOp (this=0x7fffe800eee0, cmd=0x7ffff4848910, args=0x7ffff4848950, numArgs=1) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:859
#12 0x00007ffff7a1f959 in Gfx::go (this=0x7fffe800eee0, topLevel=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:718
#13 0x00007ffff7a1f745 in Gfx::display (this=0x7fffe800eee0, obj=0x7ffff4848c90, topLevel=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:684
#14 0x00007ffff7a838b2 in Page::displaySlice (this=0x7fffe8004ef0, out=0x7fffe80008c0, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, printing=false, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:574
#15 0x00007ffff7a87086 in PDFDoc::displayPageSlice (this=0x636fb0, out=0x7fffe80008c0, page=13, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, printing=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/PDFDoc.cc:522
#16 0x0000000000402b61 in savePageSlice (doc=0x636fb0, splashOut=0x7fffe80008c0, pg=13, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffe8005290 "render/foo-13.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#17 0x0000000000403282 in processPage (pageInfo=0x607190) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#18 0x00007ffff76c6e9a in start_thread (arg=0x7ffff4849700) at pthread_create.c:308
#19 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#20 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7ffff504a700 (LWP 24172)):
#0  0x00007ffff7af9ee7 in Splash::compositeBackground (this=0x7ffff001b8a0, color=0x7ffff0000943 "\377\377\377") at /home/tsdgeos/devel/poppler/splash/Splash.cc:5182
#1  0x00007ffff7add6b7 in SplashOutputDev::endPage (this=0x7ffff00008c0) at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:1429
#2  0x00007ffff7a1f51d in Gfx::~Gfx (this=0x7ffff0018840, __in_chrg=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:649
#3  0x00007ffff7a839ef in Page::displaySlice (this=0x7ffff0004d30, out=0x7ffff00008c0, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, printing=false, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:601
#4  0x00007ffff7a87086 in PDFDoc::displayPageSlice (this=0x636fb0, out=0x7ffff00008c0, page=12, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, printing=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/PDFDoc.cc:522
#5  0x0000000000402b61 in savePageSlice (doc=0x636fb0, splashOut=0x7ffff00008c0, pg=12, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7ffff0005130 "render/foo-12.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#6  0x0000000000403282 in processPage (pageInfo=0x6070d0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#7  0x00007ffff76c6e9a in start_thread (arg=0x7ffff504a700) at pthread_create.c:308
#8  0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7fffeffff700 (LWP 24171)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007ffff76c9065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007ffff76c8eba in __pthread_mutex_lock (mutex=0x607370) at pthread_mutex_lock.c:61
#3  0x00007ffff7a641c4 in GlobalParams::findFontFile (this=0x607250, fontName=0x7fffe4044770) at /home/tsdgeos/devel/poppler/poppler/GlobalParams.cc:1117
#4  0x00007ffff7a3bf34 in GfxFont::locateFont (this=0x7fffe4004290, xref=0x7fffe40151c0, ps=false) at /home/tsdgeos/devel/poppler/poppler/GfxFont.cc:775
#5  0x00007ffff7ade882 in SplashOutputDev::doUpdateFont (this=0x7fffe40008c0, state=0x7fffe400c190) at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:1845
#6  0x00007ffff7adffd8 in SplashOutputDev::drawChar (this=0x7fffe40008c0, state=0x7fffe400c190, x=70.799999999999997, y=770.88, dx=5.04, dy=0, originX=0, originY=0, code=51, nBytes=1, u=0x7fffe400555c, uLen=1)
    at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:2207
#7  0x00007ffff7a3298c in Gfx::doShowText (this=0x7fffe401e500, s=0x7fffe401f080) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:3999
#8  0x00007ffff7a31036 in Gfx::opShowText (this=0x7fffe401e500, args=0x7fffefffe950, numArgs=1) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:3742
#9  0x00007ffff7a200fe in Gfx::execOp (this=0x7fffe401e500, cmd=0x7fffefffe910, args=0x7fffefffe950, numArgs=1) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:859
#10 0x00007ffff7a1f959 in Gfx::go (this=0x7fffe401e500, topLevel=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:718
#11 0x00007ffff7a1f745 in Gfx::display (this=0x7fffe401e500, obj=0x7fffefffec90, topLevel=true) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:684
#12 0x00007ffff7a838b2 in Page::displaySlice (this=0x7fffe4015a00, out=0x7fffe40008c0, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, printing=false, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:574
#13 0x00007ffff7a87086 in PDFDoc::displayPageSlice (this=0x636fb0, out=0x7fffe40008c0, page=11, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, printing=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/PDFDoc.cc:522
#14 0x0000000000402b61 in savePageSlice (doc=0x636fb0, splashOut=0x7fffe40008c0, pg=11, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffe4015e00 "render/foo-11.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#15 0x0000000000403282 in processPage (pageInfo=0x607090) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#16 0x00007ffff76c6e9a in start_thread (arg=0x7fffeffff700) at pthread_create.c:308
#17 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#18 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7fffef7fe700 (LWP 24170)):
#0  0x00007ffff7a7e3aa in Array::decRef (this=0x2b1029d01d501c1) at /home/tsdgeos/devel/poppler/poppler/Array.h:50
#1  0x00007ffff7a7deab in Object::free (this=0x7ffff0227800) at /home/tsdgeos/devel/poppler/poppler/Object.cc:133
#2  0x00007ffff79f8058 in Array::~Array (this=0x7ffff0007690, __in_chrg=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Array.cc:51
#3  0x00007ffff7a7ded1 in Object::free (this=0x7fffdc025a08) at /home/tsdgeos/devel/poppler/poppler/Object.cc:134
#4  0x00007ffff7a0d948 in Dict::~Dict (this=0x7fffdc0259a0, __in_chrg=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Dict.cc:126
#5  0x00007ffff7a1e685 in GfxResources::~GfxResources (this=0x7fffdc0258d0, __in_chrg=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:390
#6  0x00007ffff7a3880a in Gfx::popResources (this=0x7fffdc025780) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:5364
#7  0x00007ffff7a1f588 in Gfx::~Gfx (this=0x7fffdc025780, __in_chrg=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:658
#8  0x00007ffff7a839ef in Page::displaySlice (this=0x7fffdc014c90, out=0x7fffdc0008c0, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, printing=false, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:601
#9  0x00007ffff7a87086 in PDFDoc::displayPageSlice (this=0x636fb0, out=0x7fffdc0008c0, page=10, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, printing=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/PDFDoc.cc:522
#10 0x0000000000402b61 in savePageSlice (doc=0x636fb0, splashOut=0x7fffdc0008c0, pg=10, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffdc015090 "render/foo-10.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#11 0x0000000000403282 in processPage (pageInfo=0x607050) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#12 0x00007ffff76c6e9a in start_thread (arg=0x7fffef7fe700) at pthread_create.c:308
#13 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7fffedffb700 (LWP 24169)):
#0  0x00007ffff7af9ee4 in Splash::compositeBackground (this=0x7fffd8011f20, color=0x7fffd8000943 "\377\377\377") at /home/tsdgeos/devel/poppler/splash/Splash.cc:5182
#1  0x00007ffff7add6b7 in SplashOutputDev::endPage (this=0x7fffd80008c0) at /home/tsdgeos/devel/poppler/poppler/SplashOutputDev.cc:1429
#2  0x00007ffff7a1f51d in Gfx::~Gfx (this=0x7fffd8011000, __in_chrg=<optimized out>) at /home/tsdgeos/devel/poppler/poppler/Gfx.cc:649
#3  0x00007ffff7a839ef in Page::displaySlice (this=0x7fffd8013060, out=0x7fffd80008c0, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, printing=false, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/Page.cc:601
#4  0x00007ffff7a87086 in PDFDoc::displayPageSlice (this=0x636fb0, out=0x7fffd80008c0, page=9, hDPI=72, vDPI=72, rotate=0, useMediaBox=false, crop=false, printing=false, sliceX=0, sliceY=0, sliceW=595, sliceH=841, abortCheckCbk=0x0, abortCheckCbkData=0x0, 
    annotDisplayDecideCbk=0x0, annotDisplayDecideCbkData=0x0, copyXRef=true) at /home/tsdgeos/devel/poppler/poppler/PDFDoc.cc:522
#5  0x0000000000402b61 in savePageSlice (doc=0x636fb0, splashOut=0x7fffd80008c0, pg=9, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffd8013460 "render/foo-09.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:204
#6  0x0000000000403282 in processPage (pageInfo=0x607010) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#7  0x00007ffff76c6e9a in start_thread (arg=0x7fffedffb700) at pthread_create.c:308
#8  0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fffed7fa700 (LWP 24168)):
#0  0x00007ffff58aaa9f in ?? () from /lib/x86_64-linux-gnu/libz.so.1
#1  0x00007ffff58abcc8 in ?? () from /lib/x86_64-linux-gnu/libz.so.1
#2  0x00007ffff58ac833 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
#3  0x00007ffff5f553b4 in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#4  0x00007ffff5f5587b in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#5  0x00007ffff5f59434 in png_write_row () from /lib/x86_64-linux-gnu/libpng12.so.0
#6  0x00007ffff5f59658 in png_write_image () from /lib/x86_64-linux-gnu/libpng12.so.0
#7  0x00007ffff79c049a in PNGWriter::writePointers (this=0x7fffcc007010, rowPointers=0x7fffcc017040, rowCount=841) at /home/tsdgeos/devel/poppler/goo/PNGWriter.cc:159
#8  0x00007ffff7b02ee5 in SplashBitmap::writeImgFile (this=0x7fffcc000a30, writer=0x7fffcc007010, f=0x7fffcc000f40, hDPI=72, vDPI=72) at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:652
#9  0x00007ffff7b01fbd in SplashBitmap::writeImgFile (this=0x7fffcc000a30, format=splashFormatPng, f=0x7fffcc000f40, hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:410
#10 0x00007ffff7b01e0b in SplashBitmap::writeImgFile (this=0x7fffcc000a30, format=splashFormatPng, fileName=0x7fffcc0071e0 "render/foo-08.png", hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:346
#11 0x0000000000402bbd in savePageSlice (doc=0x636fb0, splashOut=0x7fffcc0008c0, pg=8, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffcc0071e0 "render/foo-08.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:210
#12 0x0000000000403282 in processPage (pageInfo=0x6071d0) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#13 0x00007ffff76c6e9a in start_thread (arg=0x7fffed7fa700) at pthread_create.c:308
#14 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#15 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fffee7fc700 (LWP 24166)):
#0  0x00007ffff58aba3f in ?? () from /lib/x86_64-linux-gnu/libz.so.1
#1  0x00007ffff58ac833 in deflate () from /lib/x86_64-linux-gnu/libz.so.1
#2  0x00007ffff5f553b4 in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#3  0x00007ffff5f5587b in ?? () from /lib/x86_64-linux-gnu/libpng12.so.0
#4  0x00007ffff5f59434 in png_write_row () from /lib/x86_64-linux-gnu/libpng12.so.0
#5  0x00007ffff5f59658 in png_write_image () from /lib/x86_64-linux-gnu/libpng12.so.0
#6  0x00007ffff79c049a in PNGWriter::writePointers (this=0x7fffd4006db0, rowPointers=0x7fffd4011510, rowCount=841) at /home/tsdgeos/devel/poppler/goo/PNGWriter.cc:159
#7  0x00007ffff7b02ee5 in SplashBitmap::writeImgFile (this=0x7fffd4000a30, writer=0x7fffd4006db0, f=0x7fffd4077dd0, hDPI=72, vDPI=72) at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:652
#8  0x00007ffff7b01fbd in SplashBitmap::writeImgFile (this=0x7fffd4000a30, format=splashFormatPng, f=0x7fffd4077dd0, hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:410
#9  0x00007ffff7b01e0b in SplashBitmap::writeImgFile (this=0x7fffd4000a30, format=splashFormatPng, fileName=0x7fffd4006cb0 "render/foo-06.png", hDPI=72, vDPI=72, compressionString=0x404340 "") at /home/tsdgeos/devel/poppler/splash/SplashBitmap.cc:346
#10 0x0000000000402bbd in savePageSlice (doc=0x636fb0, splashOut=0x7fffd40008c0, pg=6, x=0, y=0, w=595, h=841, pg_w=595, pg_h=841, ppmFile=0x7fffd4006cb0 "render/foo-06.png") at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:210
#11 0x0000000000403282 in processPage (pageInfo=0x607150) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:337
#12 0x00007ffff76c6e9a in start_thread (arg=0x7fffee7fc700) at pthread_create.c:308
#13 0x00007ffff6df4cbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ffff7fb3740 (LWP 24160)):
#0  0x00007ffff6dc083d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007ffff6dee774 in usleep (useconds=<optimized out>) at ../sysdeps/unix/sysv/linux/usleep.c:33
#2  0x0000000000403d49 in main (argc=3, argv=0x7fffffffe618) at /home/tsdgeos/devel/poppler/utils/pdftoppm.cc:849

(In reply to comment #128)
> Adding -f 30 -l 30 actually makes it not crash anymore.
> 
> Note i have a slight change over your pdftoppm code, i've changed the
> Sleep(1) in the linux code to usleep(100000), that may be introducing the
> races you don't see.
> 
Hmmh, I ignored that at the beginning, because I changed it, too, a long time ago, but to usleep(10000). But if I'm giving it a Zero more, I get also these suspicious crashes.

I already looked into it with valgrind, but the only thing it detects is that I forgot a "delete pageDict" at the end of Page::replaceXRef.

And I can also see that the second backtrace is definitely in one of my changes, but I can't imagine, why.

So I decide now to give clang a chance to help me. Unfortunately I got a new laptop after solving the Coldwind problems, therefore this means a lot of more work as just doing it on my old laptop :-(

Created attachment 72729 [details] [review]
Enhancements for damaged PDFs

clang gave me a hint, unfortunately not the solution. And working on a solution I had more than one moment where I wanted to give up or at least planned to change the solution that it isn't allowed to reconstruct xref tables in the thread based solution.
But finally I succeeded: the attached patch doesn't crash (or at least it seems to be) anymore on broken PDF without getting new problems. I stopped my tests either with the clang or with the normal compiled version after running 5 minutes without any crashes.

Can you run the new code against folien-ts.pdf ?

It seems to deadlock/hang here, i.e.
 pdftoppm -r 72 -cropbox -png folien-ts.pdf render
just hangs forever with no cpu usage

(In reply to comment #131)
> Can you run the new code against folien-ts.pdf ?
> 
> It seems to deadlock/hang here, i.e.
>  pdftoppm -r 72 -cropbox -png folien-ts.pdf render
> just hangs forever with no cpu usage

I don't have that PDF

Created attachment 72857 [details]
The folien-ts.pdf file

Created attachment 72952 [details] [review]
Revised solution for bug-poppler10898.pdf

I had to revert the former patch for bug-poppler10898.pdf, because it causes the problems with folien-ts.pdf.
When I started to look again on it today morning it took just 5 minutes to see the really reason for the crash, and it was so obviously that I can't understand anymore why I spent hours over hours looking at the code without seeing it before. Today it took much more time to create the new patch and to regtest it than to fix it.
But the good news is that also the solution is so obvious that I'm sure that at least the reason for the crashes is now definitely solved with this new patch!

With attachment.cgi?id=72952 bug-poppler10898.pdf is crashing on me again with the while loop

Created attachment 72996 [details] [review]
Additional locks for bug-poppler10898.pdf

(In reply to comment #135)
> With attachment.cgi?id=72952 bug-poppler10898.pdf is crashing on me again
> with the while loop

When I read Your comment today early morning I immediately started a new while-loop test. After half a hour I started writing a new comment, but just before I wanted to press the submit button, the test crashed. Unfortunately I forgot to allow core dumps, so I couldn't figure out the reason.
The former reason for crashes is definitely solved: two threads were calling Dict::copy on the same dict at the same time, but because Dict:copy also modifies the dict entry object, this caused the crashes. I solved that by inserting lock/unlock pairs in the method. 
Because I have no core dump and You also didn't attach backtraces, I've to guess now: In the crashes before there was always also an array involved, but there are no lock/unlock pairs in the array implementation. I tried to proof that idea, starting the while-loop test with the clang compiled version 2 hours ago.
In the meantime I inserted the lock/unlock pairs in the array implementation, run a complete regtest on this new version, but the clang version is still running.
So I upload the new patch now and ask You to test it, too, and provide a backtrace if it still crashes. I'll stop now the clang test and start a normal test with core dumps enables, too.

Doh, i'm a stupid and was running from a build folder that contained the old version, that's why it still crashed. Been running attachment.cgi?id=72952 properly for 2 hours and can't get it to crash anymore. Wonder if we should go with that one for now instead of attachment.cgi?id=72996 that you say that is something like a "blind shot" and might not really fix stuff?.

What do you think?

(In reply to comment #137)
> Doh, i'm a stupid and was running from a build folder that contained the old
> version, that's why it still crashed. Been running attachment.cgi?id=72952
> properly for 2 hours and can't get it to crash anymore. Wonder if we should
> go with that one for now instead of attachment.cgi?id=72996 that you say
> that is something like a "blind shot" and might not really fix stuff?.
> 
> What do you think?

As I already told, I had one crash with attachment.cgi?id=72952 today morning, and in that patch definitely the array locking was missing. And because arrays are used by reference, I would prefer You use attachment.cgi?id=72996. It is regtested by me, and also the loop test runs now stable for over two hours.

I just stopped my loop test with attachment.cgi?id=72996 after running it over 4 hours without any problems:

thomas@w7-thomas:~/Entwicklung/xpdfmerge/pgrender/poppler$ date; while [ true ]; do ./utils/pdftoppm -r 72 -cropbox /media/thomas/HD-PCTU3/PDF\ Suite/bug-poppler10898.pdf output/foo 2>error.out; if [ $? -ne 0 ]; then break; fi; done; date
Mo 14. Jan 11:02:46 CET 2013
^C
thomas@w7-thomas:~/Entwicklung/xpdfmerge/pgrender/poppler$ date
Mo 14. Jan 15:15:11 CET 2013

Test Regression is good, horray!

Now let's look at the code :-)

First the small easy stuff ;-)

In the qt4 frotend there's this change

-       m_doc->doc->getDocInfo( &info );
-       if ( !info.isDict() )
+       XRef *xref = m_doc->doc->getXRef()->copy();
+       Object *infoObj = xref->getDocInfo(&info);
+       if ( !infoObj->isDict() ) {
            return QStringList();
+       }
 
-       Dict *infoDict = info.getDict();
+       Dict *infoDict = infoObj->getDict();


Any reason we can not keep using info? i.e. something like

-       m_doc->doc->getDocInfo( &info );
+       XRef *xref = m_doc->doc->getXRef()->copy();
+       xref->getDocInfo(&info);
        if ( !info.isDict() )
            return QStringList();
 
        Dict *infoDict = info.getDict();



Rest of the code looks good, i'd introduce two improvements, but let's do it *LATER*, one introducing something similar to the concept of the mutexlocker that unlocks the mutex on destruction (making it much easier to deal if early returns, etc) and second changing some of the lock booleans to enums so 
   page = doc->getCatalog()->findPage (ref.num, ref.gen, gFalse);
looks like
   page = doc->getCatalog()->findPage (ref.num, ref.gen, Poppler::DoNotLock);
making the code a bit easier to read when you go around

I'm happy with the current state expect the minor question about the qt4 frontend.

Anyone else has any comment?

(In reply to comment #140)
> Test Regression is good, horray!
> 
> Now let's look at the code :-)
> 
> First the small easy stuff ;-)
> 
> In the qt4 frotend there's this change
> 
> -       m_doc->doc->getDocInfo( &info );
> -       if ( !info.isDict() )
> +       XRef *xref = m_doc->doc->getXRef()->copy();
> +       Object *infoObj = xref->getDocInfo(&info);
> +       if ( !infoObj->isDict() ) {
>             return QStringList();
> +       }
>  
> -       Dict *infoDict = info.getDict();
> +       Dict *infoDict = infoObj->getDict();
> 
> 
> Any reason we can not keep using info? i.e. something like
> 
> -       m_doc->doc->getDocInfo( &info );
> +       XRef *xref = m_doc->doc->getXRef()->copy();
> +       xref->getDocInfo(&info);
>         if ( !info.isDict() )
>             return QStringList();
>  
>         Dict *infoDict = info.getDict();
> 

No idea why I thought such complicate.Your code looks definitely smarter.

Merged.

Thomas do you want to do the mutexlocker and true/false->enum stuff? Or should i open a bug to remember about it?

(In reply to comment #142)
> Merged.
> 
> Thomas do you want to do the mutexlocker and true/false->enum stuff? Or
> should i open a bug to remember about it?

I'll do it (if I understand correctly what You meant: what You point in comment 140). And there are still some issues to do to get at least the qt library completely thread safe, therefore I always left attachment.cgi?id=69495 open. But I would prefer to do it in an additional bug, we have now 143 comments here, it becomes unreadable, or not?

Yes comment 140, and yes, let's go to different bugs