From my reply to the original email:

I'm not sure what the fix is, but the line with the error is:
   <text top="632" left="152" width="58" height="0"
font="7">¥§¦©¨   ¥§    ¦ ¨   ¦</text>
and firefox gives:
   <text top="632" left="152" width="58" height="0"
font="7">¥§¦©¨   ¥§    ¦ ¨   ¦</text>
   ---------------------------------------------------------------^
(that is -- it is choking on the [00|11] character; there are also
other chatacters in the latin-1 control character range (c < 0x20)).

This will cause any xml parser to choke, as the characters are
invalid. What I don't know is why/how these are appearing in
pdftohtml.

Looking at the PDF in okular (which appears to render the PDF
correctly there), shows a mathematical equation for the faulty lines,
specifically:

<text top="606" left="101" width="173" height="10" font="6">Digital
signal processing basic formula:</text>
<text top="632" left="101" width="25" height="10" font="6">y(t) =</text>
<text top="626" left="133" width="0" height="0" font="7"> </text>
<text top="631" left="133" width="0" height="0" font="7">¡</text>
<text top="647" left="128" width="0" height="0" font="7">¢</text>
<text top="646" left="134" width="11" height="0" font="7"> ¤£</text>
<text top="632" left="152" width="58" height="0"
font="7">¥§¦©¨   ¥§    ¦ ¨   ¦</text>

should be (in the proper math layout for this formula):

   y(t) = integral [above: inf, below: -inf] h(u)x(t - u)du

where the h(u)x(t - u)du bit is in the stylised script used in maths.

My initial thought is that the characters are referencing the Unicode
codepoints (e.g. in the U+2100 range). However, these all appear to be
in the ascii range (i.e. not multi-byte UTF-8 as the encoding
suggests, but I may be wrong as there look to be more characters than
what is displayed).

Instead, they look like they are codepoints into a special
mathematical font (e.g. Symbol(?) in Windows (I don't have as Windows
box to hand at the moment, so can't verify the font name)). This would
make sense given the font="7" attribute and the seemingly random
characters. And given the greater number of characters, this looks to
be using a non-URF8 multi-byte encoding.

In the generated html and/or xml file (the html output is similarly affected), control characters (and other invalid xml characters) are being written out to the file.

The XML version will cause an XML parser to generate an error when it encounters these characters (it first chokes on a 0x11 control character).

The HTML version loads the page, but displays garbage instead of the integral equation (same as viewing either file in a text editor).

a/ My initial thought was that the characters (such as the integral sign) in the integral part of the equation were being written as Unicode. For example, U+222B is the Unicode code point for the integral sign (∫) [1]. If that were the case, the UTF-8 forms would be written out and it would form valid XML and HTML output.

b/ My next thought was that the control characters in the ascii and Unicode code pages corresponded to the correct glyphs when using font 7 (e.g. if the font was a special mathematical font). This is not right, as the font is the Times font (looking at the font defiinition), and the number of characters in that element don't match the number of characters in the equation (thus indicating a multi-byte encoding is being used).

c/ Another possibility that has occurred to me is that what is shown is the raw UTF-8 byte sequence, and that is being UTF-8 encoded! Removing the UTF-8 encoding in the html version, I get " ¤£" instead of " ¤£" for one of the outputted text nodes.

My current thinking (without looking at the code yet, just from examining the behaviour) is that (c) is the most probable cause of this error.

[1] http://en.wikipedia.org/wiki/Integral_symbol

pdftotext (and likely the other utilities) is affected. It gives:

    £ ¤  ¢ ¦ §©§¦¥  ¡ ¨¦  ¥ ¨

for the equation (it even misses out the `y(t) =` part).

The text is being written out on line 651 of utils/HtmlOuputDev.cc for xml (there are other cases for simple and complex html outputs, not to mention the places in the other output formatters). The line is:

      fputs(str1->getCString(),f);

where str1 is a GooString. This assumes that what is returned from getCString() is in the correct encoding (i.e. the one that has been requested by the output stream).

Has any progress been made on this issue? I am attempting to extract the text with position from a large number of PDF files, and I have encountered this issue several times so far.

Confirming the issue with latest Poppler 0.38

For example, with the following file:
http://www.ikea.com/us/en/assembly_instructions/dioder-led-multi-use-light__AA-241740-1_pub.PDF

The output for page 2 includes the ASCII control characters 0x01, 0x12, 0x0c, 0x0d, 0x0e.

The file mentioned in Comment 6 exhibits another bug:
The xml for page 4 opens some <b> tags and doesn't close them.

(In reply to Ori Avtalion from comment #7)
> The file mentioned in Comment 6 exhibits another bug:
> The xml for page 4 opens some <b> tags and doesn't close them.

Another problem in the code I'm noticing:
When text is both bold and italic, in some cases, it outputs <b><i> and sometimes <i><b>. It should be consistent, and the CloseTags function should close it in the reverse order.

Sorry, but for some reason Bugzilla failed to generate valid XML to export this bug when migrating to GitLab. As a result we've been unable to duplicate it there. Please file a new issue on https://gitlab.freedesktop.org/poppler/poppler/issues/new/ referencing this one. Thanks, and sorry again for the inconvenience.

Oh, the irony. So now I need to open a bug about Bugzilla producing invalid XML.