You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@poi.apache.org by bu...@apache.org on 2019/05/30 09:54:01 UTC
[Bug 63474] New: Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
https://bz.apache.org/bugzilla/show_bug.cgi?id=63474
Bug ID: 63474
Summary: Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
Product: POI
Version: unspecified
Hardware: PC
OS: Linux
Status: NEW
Severity: regression
Priority: P2
Component: HSSF
Assignee: dev@poi.apache.org
Reporter: andrea.stacchiotti@acus.eu
Target Milestone: ---
We are observing the regression in the title with
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
the relevant JVM flags are "-Xmx3584m -Xms256m"
on Linux [...] 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux
the same regression does NOT happen on Windows, under a variety of java
versions and stack/heap space (all java 8).
-------------
Detailed description:
When using `WorkbookFactory.create()` on this XLS file:
https://drive.google.com/file/d/10XOpyz2imxJ6Q3cItqhboab8EWdUEQR_/view?usp=sharing
the entire JVM crashes because it's trying to request too much memory from the
OS, with the following message:
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory([...],
[...], 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map [...] bytes for committing
reserved memory.
# An error report file with more information is saved as:
# [...]
this does not happen in v3.13 and lower, but starts happening v3.14-beta1, I've
tested several versions, including 3.14, 3.17, 4.1.0 and others I've probably
forgotten.
Note that this is different from the usual OOM errors that happen while reading
large XLS files, as those do not cause the whole JVM to crash.
The files that trigger this error are rare, but I've seen more than one
example.
Please let me know if there is any other test of information I can do/give.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 63474] Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63474
--- Comment #4 from andrea.stacchiotti@acus.eu ---
I'm not exactly sure how exactly to investigate it, the JVM straight crashes,
it's not a regular OOM.
I also don't have a linux dev environment right now, but I can acquire it.
If you can suggest a testing procedure, I can try it.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 63474] Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63474
--- Comment #3 from PJ Fanning <fa...@yahoo.com> ---
Any chance you could take the lead on investigating the memory issue by using a
java profiler and testing with POI 3.13 and POI 4.1 (best to use POI 4.1
because if we fix something it will be in that version)?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 63474] Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63474
--- Comment #9 from andrea.stacchiotti@acus.eu ---
The system has plenty of memory leftover, I easily have 5 free GBs over the max
size of java's heap.
I am trying to bisect the 3.13 - 3.14 cycle to find the problematic commit, but
I'm having some issues.
I can easily replicate the issue on a separate project with the two artifacts
from maven, but I can't get the library junit tests to help me.
I've added a junit test doing exactly what I do in my standalone program, but
whatever tag I checkout, `ant test-main` always fails at my test with an OOM
(either heap space or GM limit, NOT the horrible crash a standalone program
does), no matter what heap space I give with ANT_OPTS.
I could bisect by building the jars each time and importing them in my toy
project, but that seems very very laborious, do you have better ideas?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 63474] Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63474
--- Comment #7 from andrea.stacchiotti@acus.eu ---
I understand that giving the program more space would solve the issue, but what
I wanted to point out is that it does not happen in 3.13, at all.
I guess I can try to bisect the 3.13 - 3.14-beta1 cycle and see what commit is
the culprit, if you guys cannot do so.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 63474] Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63474
PJ Fanning <fa...@yahoo.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |INVALID
--- Comment #1 from PJ Fanning <fa...@yahoo.com> ---
Could you move this to the dev list?
Reading xlsx files using XSSFWorkbook takes a lot of memory.
In my mind you have options like
* stay with POI 3.13
* increase your JVM heap
* rewrite your code to use a streaming xlsx parser (eg
https://svn.apache.org/repos/asf/poi/trunk/src/examples/src/org/apache/poi/xssf/eventusermodel/XLSX2CSV.java)
* try to patch POI 4.1 to use less memory
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 63474] Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63474
--- Comment #8 from Dominik Stadler <do...@gmx.at> ---
What I was trying to say was that the error you see actually indicates that
Java itself did not use too much memory yet, but allocating the memory from the
operating system failed, i.e. like if you have a system with only 2GB or RAM,
but give Java -Xmx3g, at some point it would not be able to aquire more memory
from the operating system.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 63474] Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63474
--- Comment #5 from Andreas Beeker <ki...@apache.org> ---
If I don't know where and when an exception / crash happens, I usually add log
/ system.out statements to the source. So you can iteratively narrow the error
corridor.
A different strategy is https://git-scm.com/docs/git-bisect, to find out which
commit caused the trouble.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 63474] Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63474
--- Comment #6 from Dominik Stadler <do...@gmx.at> ---
Please note that a crash with "Cannot allocate memory" indicates that your
system runs out of physical available memory overall, not just for the single
Java process. So it seems your Xmx is higher than the amount of memory
available on your system.
You might simply test on a different machine with more available physical
memory or try stopping some memory-hungry processes on this machine to see if
that makes it work. Also reducing the Xmx might show the usual OutOfMemory
exception again.
In general, if the JVM crashes, the problem is usually outside of the Java
program itself.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 63474] Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63474
--- Comment #2 from andrea.stacchiotti@acus.eu ---
Do note that this is an XLS file, I expect WorkbookFactory.create to create a
HSSFWorkbook under the hood.
XLSX files are much less memory heavy, and there are effective streaming
parsers for it, but no such options exist for XLS files.
I imagine something in the 4.14-beta dev cycle raised by a lot the memory
needed for an XLS file, I hope there is a way to fix it or disable it from the
user side.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 63474] Workbook creation fails with a JVM memory error in
3.14-beta1 and later, but succeeds in 3.13 and earlier
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63474
--- Comment #10 from Dominik Stadler <do...@gmx.at> ---
You could use git repo and use git-biset to do the tedious work for you if you
can come up with a script which does the full building-jars-and-run-application
cycle with a result, see https://git-scm.com/docs/git-bisect for details. I
have used this a number of times successfully in the past.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org