Created attachment 24919 [details]
encrypted PDF file (48 bit); owner pw "o", user pw "u"

PDF files encrypted with more than 40 bits key length ("revision 3 security handler") cannot be decrypted with the owner password unless the key length is exactly 128 bits.

The attached PDF file shows this behaviour. The owner password is "o" and the user password "u" (without quotes). Acrobat Reader and GS accept both, evince (and xpdf) only the user password.

I'm pretty sure that bug is caused by the following loop in the function Decrypt::makeFileKey, where the owner password is treated:

    if (encRevision == 3) {
      for (i = 0; i < 50; ++i) {
        md5(test, 16, test);
      }
    }

The corresponding code for the user password in Decrypt::makeFileKey2 reads as follows:

  if (encRevision == 3) {
    for (i = 0; i < 50; ++i) {
      md5(fileKey, keyLength, fileKey);
    }
  }

The difference is that the in the second loop the hash is truncated to the key length. This truncation (which does nothing if the key is exactly 16 bytes) is not mentioned in the PDF spec, but it seems to be the behaviour of Acrobat Reader ...