Bug 73049

HI Andis,

Can confirm this. Thanks.

Tested with 32 bits Ubuntu 
4.1.3.2 and 4.2.0.rc1, and - by the way - 4.2.0beta2 show the same problem.

4.0.6.2 does open it (takes a while)
4.1.0. beta1 opens it fast
4.1.0. beta2 opens it slowwww
4.1.0. rc1 opens it slowwww
4.1.0. rc3 opens it slowwww

4.1.1 rc2 is the first one that fails for me. 
I do not have 4.1.1.1 to test, but set to that version...

This is an out of memory problem and not a normal crash.

On pc Debian x86-64 with master sources updated today (debug mode also)
PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM     TIME+ COMMAND
18575 julien    20   0 4932m 2,3g 119m S   1,7 40,8   1:43.49 soffice.bin
Markus: Indeed about the memory pb! :-)

Created attachment 91370 [details]
valgrind trace

Valgrind trace from file opening to its closing.

Added similar reports to 'See Also'.

The problem persists in Libreoffice 4.2.6 It's really hopeless to open 10 MB Calc file (Ubuntu 14.04, 32 bit), in spite it can be opened and manipulated in Openoffice.org on the same computer.

When I changed memory settings to default values in Openoffice.org, I was able to lead files and even do changes in Pivot tables. The default values are:
Undo 
Number of steps - 100
Graphics cache
Use for Libreoffice 20
Memory per object 5.2
Remove from memory after 00:10
Cache for inserted objects
Number of objects 20

Probably the first step would be to test different memory options.

With 4.2.6 LO Debian package with i5 + 6GB, it gives: 
real	0m14.427s
user	0m12.560s
sys	0m0.820s

with this command:
time soffice --calc MPS\ Augsnes\ pretestiba\ kopsanas\ parauglaukumos.ods 

For the test, could you rename your LO directory profile (see https://wiki.documentfoundation.org/UserProfile#GNU.2FLinux) and give it a new try?

(In reply to comment #8)
> With 4.2.6 LO Debian package with i5 + 6GB, it gives: 
> real	0m14.427s
> user	0m12.560s
> sys	0m0.820s
> 
> with this command:
> time soffice --calc MPS\ Augsnes\ pretestiba\ kopsanas\ parauglaukumos.ods 
> 
> For the test, could you rename your LO directory profile (see
> https://wiki.documentfoundation.org/UserProfile#GNU.2FLinux) and give it a
> new try?

Libreoffice 4.2.6 crashed with these messages

(soffice:28742): Gtk-WARNING **: Error loading theme icon 'dialog-warning' for stock: Unable to load image-loading module: /usr/lib/i386-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-png.so: /usr/lib/i386-linux-gnu/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-png.so: failed to map segment from shared object: Cannot allocate memory

(soffice:28742): Gdk-WARNING **: shmat failed: error 12 (Cannot allocate memory)

real	14m12.912s
user	0m50.132s
sys	0m3.564s

Libreoffice 4.3.0.4 crashed with this message

real	5m53.196s
user	0m0.340s
sys	0m0.208s

When I closed most of the applications, Libreoffice 4.3 was able to load document

real	5m36.688s
user	0m0.024s
sys	0m0.096s

System: Ubuntu 14.04, 32 bit, Kernel 3.13.0-32-generic, Memory 3.0 GiB, Processor 2 x Pentium(R) Dual-Core CPU T4500 @ 2,3 GHz

Thank you Andis for your feedback, I put it back to NEW.

Hello!

I tried in Libreoffice 4.3.1.2. It's impossible to open 7,9 MB calc file with libreoffice calc, if the file contains array or complex formula. The program just crash with:
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

I guess it became worse since 4.1, and in Openoffice.org 4.1 the same file opens rather easy.

Together with bug on slow scrolling the situation became really terrible for research use of libreoffice.

Do you think you could bibisect the bug? https://wiki.documentfoundation.org/QA/HowToBibisect

That would help quite a bit.

Andis: just to be sure, did you rename your LO directory profile (as indicated in my previous comment)?
You gave results but didn't confirm you had renamed it.

Hi!

I tried it without installation by extraction using command 

for i in ../*.deb; do dpkg-deb -x $i . ; done

and with it's own user directory

[Bootstrap]
BaseInstallation=${OOO_BASE_DIR}
InstallMode=<installmode>
ProductKey=OOo-dev 3.0
UserInstallation=$ORIGIN/..

I have one desktop PC with 8 GB of RAM and it is not better.

(In reply to comment #12)
> Do you think you could bibisect the bug?
> https://wiki.documentfoundation.org/QA/HowToBibisect
> 
> That would help quite a bit.

I have 32 bit OS, not an option to me

@Andis - can you provide the 7.9 meg file?

(In reply to comment #16)
> @Andis - can you provide the 7.9 meg file?

Hello!

Unfortunately that particular file contains classified information. I can send to you link to that file by e-mail (mine is andis.lazdins@gmail.com).

You can try this one below. I was able to open this (initially xls) file and even do some edits at the beginning of 2013 with libreoffice 3.X. Now it can't be opened either with libreoffice and openoffice. However, I'm not sure if there are some integrity errors in the file, because I don't have possibility to check it. The original xls file was also very hard to open for Excel.

https://drive.google.com/file/d/0Bxv4jQ_04jXZcXVIazkyMEw0R2s/edit?usp=sharing

Here is another example. The file initially created in libreoffice 3.X, now I can open it only with openoffice. With libreoffice it constantly crashes. The same happens also with much smaller files. And it is impossible to work even with smaller files because auto saving is very common status of the files.

https://drive.google.com/file/d/0Bxv4jQ_04jXZUk5qeEZBVnZXb1E/edit?usp=sharing

Here is another example. It can be opened with libreoffice 4.2.6.3, but last time it took about 15 minutes and any changes in formulas are really painful in terms of time consuming.

https://drive.google.com/file/d/0Bxv4jQ_04jXZV1J6bHpZZkstVTA/edit?usp=sharing

I hope it will help, because performance of calc, in spite of excellent improvements, becomes worse with every major release.

If there are any hope to solve this and similar terrible Calc CPU use regressions or the only way is to move back to OpenOffice.org? Unfortunately it becomes worse with every next major releases and gap between Libreoffice and OpenOffice.org Calc performance increases. There are nice new features in Libreoffice Calc, like improved pivot tables, but they cost nothing, if the most of files, which could be easily operated with 3rd version branch now can't be opened at all or it takes minutes to do any changes in documents. I'm not mentioning Excel, which is much worse in terms of functionality, but much faster with those big files.

One example https://drive.google.com/file/d/0Bxv4jQ_04jXZRHZUeDlxSDRHNTQ/edit?usp=sharing

With 4.2.6.3 it can be opened in 54 seconds (production version, extensions etc.)
With 4.3.2.2 - 55 seconds
With 4.4.0.0 - 49 seconds
With Openoffice.org 4.1.1. - 18 seconds (production version, only 50% of CPU load in contrast to 100 % of all Libreoffice versions)
Excel 2007 through wine (file exported to xlsx format - 4 seconds (converted file https://drive.google.com/file/d/0Bxv4jQ_04jXZWGxOLUlIRnU2cUU/edit?usp=sharing)

Something really should be done here.

Kohei/Markus/Eike: thought you might be interested in this one.

(In reply to comment #18)
> [...]
> One example
> https://drive.google.com/file/d/0Bxv4jQ_04jXZRHZUeDlxSDRHNTQ/edit?usp=sharing
> 
> With 4.2.6.3 it can be opened in 54 seconds (production version, extensions
> etc.)
> With 4.3.2.2 - 55 seconds

Hmmm, 8 seconds for me with version 4.3.3.0.0+ x86-64 build at home under Ubuntu 14.04 (LO being closed, select the file on the desktop, press enter key at hh:mm:10, file ready to work with at hh:mm:18, sometimes hh:mm:16)

Best regards. JBF

(In reply to comment #20)
> (In reply to comment #18)
> > [...]
> > One example
> > https://drive.google.com/file/d/0Bxv4jQ_04jXZRHZUeDlxSDRHNTQ/edit?usp=sharing
> > 
> > With 4.2.6.3 it can be opened in 54 seconds (production version, extensions
> > etc.)
> > With 4.3.2.2 - 55 seconds
> 
> Hmmm, 8 seconds for me with version 4.3.3.0.0+ x86-64 build at home under
> Ubuntu 14.04 (LO being closed, select the file on the desktop, press enter
> key at hh:mm:10, file ready to work with at hh:mm:18, sometimes hh:mm:16)
> 
> Best regards. JBF

The results mentioned in comment 18 are from Ubuntu 14.04, 32 bit, Kernel 3.13.0-32-generic, Memory 3.0 GiB, Processor 2 x Pentium(R) Dual-Core CPU T4500 @ 2,3 GHz

It is the best result from computers we are using in our office. It's actually very complicated to say to employees, that the problem will be solved in some time (repeat it with every upgrade of Libreoffice).

I think it is important sometimes to imagine, that people are not laying just to get more attention, but to put yourself in place of those trying to advocate open-source solutions, donating personal money and having to negotiate with employees, which are really unhappy with the software and have good reason to sit in the office and look to the monitor doing nothing, because the program is hanged again in spite the problematic file can be opened on another nearly 10 years old celeron with MS Office 2003, but our policy is not to use MS Office.

(In reply to comment #21)
> [...]
> The results mentioned in comment 18 are from Ubuntu 14.04, 32 bit, Kernel
> 3.13.0-32-generic, Memory 3.0 GiB, Processor 2 x Pentium(R) Dual-Core CPU
> T4500 @ 2,3 GHz

It would be interesting if you could try with 64 bits versions of Ubuntu 14.04 and LibreOffice.
Did you observe an heavy use of the swap when using LibreOffice ? Ubuntu can be tweaked to reduce the use of the swap.

Note: I only try to find what is the cause of the difference in performance.

Best regards. JBF

(In reply to comment #22)

> 
> It would be interesting if you could try with 64 bits versions of Ubuntu
> 14.04 and LibreOffice.
> Did you observe an heavy use of the swap when using LibreOffice ? Ubuntu can
> be tweaked to reduce the use of the swap.
> 
> Note: I only try to find what is the cause of the difference in performance.
> 
> Best regards. JBF

We are using here only 32 bit OS, all of them are different kind of Ubuntu versions, mostly 12.04 and 14.04.

Swap is not used, on laptop with 3GB RAM is utilized to about 1-2 GB, the same on PC with 8 GB RAM. Only processor is loaded to 100 %. I'll try to test this evening on laptop with 64 bit windows.

Ubuntu 13.10, Kernel 3.11-0-26-generic 64 bit, Mate 1.6.0
RAM 7.7 GiB
Processor 4 x Intel(R) Core(TM) i3-3110M CPU @ 2.40 GHz
Libreoffice 4.1.6.2 (used in production)
7 sec
Libreoffice 4.3.1.2 (fresh installation)
16 sec
For testing first open libreoffice dashboard, then file - open the specified document.

I tried to open the problematic file (https://drive.google.com/file/d/0Bxv4jQ_04jXZaVR2ZHZGR0JLVFE/edit?usp=sharing) on 64 bit Lubuntu 14.04 loaded from USB stick. Loading time is a bit longer on 64 bit system (about 70 sec. and 50 sec with Libreoffice 4.3.2.2, 8 GB RAM and 2 x 1,6 GHz CPU), but probably it is because the system is loaded from USB stick.

(In reply to comment #25)
> I tried to open the problematic file
> (https://drive.google.com/file/d/0Bxv4jQ_04jXZaVR2ZHZGR0JLVFE/
> edit?usp=sharing) on 64 bit Lubuntu 14.04 loaded from USB stick. Loading
> time is a bit longer on 64 bit system (about 70 sec. and 50 sec with
> Libreoffice 4.3.2.2, 8 GB RAM and 2 x 1,6 GHz CPU), but probably it is
> because the system is loaded from USB stick.

Installed fresh 64 bit Ubuntu 14.04 on that machine and got the same results, about 1 minute to open complex calc file with 4.2.6, respectively there are no difference between 32 and 64 bit versions on that particular machine.

unable to bibisect with bibisect-43all (git checkout oldest - LO3.5 freeze )

Results from bibisect-43all:
(Unsure if this is the only relevant performance regression, but the below is one significant transition from acceptable speed/memory use to extreme slowness and high memory use)

89740762f0af849e492932bd71e59149cdcd5a00 is the first bad commit
commit 89740762f0af849e492932bd71e59149cdcd5a00
Author: Bjoern Michaelsen <bjoern.michaelsen@canonical.com>
Date:   Mon Dec 10 01:57:45 2012 +0000

    source-hash-06f20d73da21342046a480a6b22af69901351328
    
    commit 06f20d73da21342046a480a6b22af69901351328
    Author:     Stephan Bergmann <sbergman@redhat.com>
    AuthorDate: Fri Jul 20 11:10:05 2012 +0200
    Commit:     Stephan Bergmann <sbergman@redhat.com>
    CommitDate: Fri Jul 20 11:10:05 2012 +0200
    
        Fix SAL_LOG area usage
    
        Change-Id: If8acc5e9fee2730796637dfb505e0c514f96f1a3


# bad: [2e0fa432485d1db6abd355dad8ccb06f0b97e4fb] source-hash-ce90f99a2d66c2b998ad3f9f028e2ea623a757f5
# good: [a71a4447320f177181c9cff9f7c6fd93802cbd8e] source-hash-9afb6e1e38c362a768e8e981f7b03cf8bcaf22cf
git bisect start 'last40onmaster' 'last36onmaster'
# bad: [bf9969effb2f759d95ecbb1a688e25f75a78da16] source-hash-8638f1e72a3fe830c0e8dcc1bd847d4fb9e599ee
git bisect bad bf9969effb2f759d95ecbb1a688e25f75a78da16
# good: [7742d3f486b624b5467b51abdccf658dbbafc505] source-hash-836822522a2e9f009c0870cbbcd48d45bbd3c622
git bisect good 7742d3f486b624b5467b51abdccf658dbbafc505
# bad: [2e349599ef946cf01cfe40929509254c596fdca3] source-hash-cf2bdd65945d2a02af44db535cf1964d4d09ae20
git bisect bad 2e349599ef946cf01cfe40929509254c596fdca3
# bad: [173f32b96a0224f28f311adf21d65f4d4e98dfa1] source-hash-22cf0759547aa1803f77dbd3ee91774600dadc6f
git bisect bad 173f32b96a0224f28f311adf21d65f4d4e98dfa1
# good: [33fcd09137f9589d4f5f619e1b64347aa0beaef5] source-hash-36170cd0dbc3409270cf3cf998805a790ee6228f
git bisect good 33fcd09137f9589d4f5f619e1b64347aa0beaef5
# bad: [89740762f0af849e492932bd71e59149cdcd5a00] source-hash-06f20d73da21342046a480a6b22af69901351328
git bisect bad 89740762f0af849e492932bd71e59149cdcd5a00
# good: [ce3917fd9ea98ae2089cf4dca9cc742c10935e7f] source-hash-b0f170d7df9cff12535d2ecfd146b32b745a8ef8
git bisect good ce3917fd9ea98ae2089cf4dca9cc742c10935e7f
# first bad commit: [89740762f0af849e492932bd71e59149cdcd5a00] source-hash-06f20d73da21342046a480a6b22af69901351328

(In reply to Matthew Francis from comment #28)
> Results from bibisect-43all:
> (Unsure if this is the only relevant performance regression, but the below
> is one significant transition from acceptable speed/memory use to extreme
> slowness and high memory use)

In our group we can use Libreoffice versions above 4.1 only on computers with fast CPU, initially obtained for GIS data processing. On other computers we are using Libreoffice 4.1.x or 4.1.x in combination with Openoffice.org 4.1 for calc data processing, because performance of Openoffice in such operations is still better. 

Increase of RAM on general purpose notebooks doesn't change anything. The problem is load of CPU.

I hope it will be solved somehow.