Currently, using the utils in a windows console does not work with unicode, either for the command line arguments, or for the output. I looked into the possible solutions for this. The command line arguments are easy to fix. There is GetCommandLineW() to get the unicode command line. Getting unicode output on the console is a bit more work. There are three ways: - Set the code page to UTF-8 then printf with UTF-8 should just work - except for when it doesn't. The UTF-8 support is a second class citizen in windows and very buggy. Just redirecting the output to "more" crashed it. - Use the wide IO functions eg wprintf(). This only works if the IO mode is set to 16-bit. ie _setmode(_fileno(stdout), _O_U16TEXT). The only problems is once the mode is changed to 16-bit, the 8-bit IO functions no longer work. printf() crashes. - Use the WriteConsoleW() function. This seems to be the best solution as it can be mixed with printf. I did some testing and windows flushes stdio after each end of line so printf and WriteConsoleW() can be interleaved providing they are not used for the same line. The next problem is finding the easiest way to support this in poppler. For the command line argument I create a Win32Console class that converts the args to UTF-8. eg int main(int argc, char **argv) { Win32Console win32Console(&argc, &argv); ... } On windows it will get the unicode args, convert to UTF-8 and store in argc/argv. On other platforms it is a no-op. For the output there a 3 ways it could be done: - Create a DLL that overrides the printf and related functions. I couldn't get this to work. I seems to depend on the order of linking and MSVC links the system libraries first. - Write our own utf8_printf that calls WriteConsoleW on windows or printf on other platforms. I would prefer not to use non standard functions that need to be used everywhere in the code. - #define printf to a replacement function on windows. I ended up going the #define solution. The Win32Console header redefines printf/fprintf/fputc on windows to use WriteConsoleW if outputting to the console.
Created attachment 135409 [details] [review] Move UTF8.h to UnicodeMapFuncs.h
Created attachment 135410 [details] [review] sort encodings
Created attachment 135411 [details] [review] fix some mingw warnings
Created attachment 135412 [details] [review] support unicode on windows console
One problem I did find while testing is this doesn't work the pdftotext outputing to stdout because the TextOuputDev printf is in the poppler lib, not in pdftotext.cc. However, I don't think this a much of an issue as you would not normally output pdftotext to the console due to the size of the output. Normally you would pipe it through "more". The unicode console output is intended more for things like pdfinfo, and file names.
The next thing I want to look at is the windows filename support in poppler. Currently is is a mess of duplicated code and #ifdefs everywhere. I'd like to simplify it to pass UTF-8 filenames through the code and just convert to wchar_t at the point where the file is opened (in goo/gfile.cc). goo/gfile.cc curently has: - openFile() takes UTF-8 filenames, has it's own UTF-8 to wchar_t conversion. - GooFile::open() has separate UTF-8 and wchar_t variants. - the GdirEntry code does not support unicode on windows.
You need to copy the license text from the decoder, not just link to the web. Are you sure we're not breaking people's code by changing the PDFDoc constructor behaviour?
Created attachment 135422 [details] [review] support unicode on windows console v2 Include license.
(In reply to Albert Astals Cid from comment #7) > Are you sure we're not breaking people's code by changing the PDFDoc > constructor behaviour? Old behavior is on win32 it calls GooFile::open(const GooString *fileName) which uses CreateFileA. This only supports ASCII. So new code is backwards compatible.
Having to have Win32Console win32Console(&argc, &argv); in each and every of the apps is a bit meh, but i guess there's no other way around it. About the improvements, do you want to finish them before commiting or you want to commit what you already have?
I'd like to commit this first. I've got some other work in progress that I'd like to finish first then I'll open a new bug for the windows file/path unicode improvements. I noticed also that the code has a mixture of mostly gmalloc/gfree but sometimes new/delete.Do you have a preference for one or the other in new code?
(In reply to Adrian Johnson from comment #11) > I'd like to commit this first. I've got some other work in progress that I'd > like to finish first then I'll open a new bug for the windows file/path > unicode improvements. I don't see anything obviously wrong there and since it's mostly windows changes (which personally is less important for me) i guess you can go on. > > I noticed also that the code has a mixture of mostly gmalloc/gfree but > sometimes new/delete.Do you have a preference for one or the other in new > code? The only benefit i see in gmalloc is the checkoverflow variants, otherwise i'd say use new/delete ?
pushed
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.