You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@poi.apache.org by bu...@apache.org on 2014/04/23 09:53:25 UTC
[Bug 56447] New: IllegalArgumentException when initializing
NPOIFSFileSystem object
https://issues.apache.org/bugzilla/show_bug.cgi?id=56447
Bug ID: 56447
Summary: IllegalArgumentException when initializing
NPOIFSFileSystem object
Product: POI
Version: 3.10
Hardware: Macintosh
Status: NEW
Severity: normal
Priority: P2
Component: POIFS
Assignee: dev@poi.apache.org
Reporter: mskan@dtu.dk
I am using POIFS to extract data from OLE2 files generated by an X-ray scanner.
Opening small files works fine with both POIFSFileSystem and NPOIFSFileSystem.
However, when I try to open large files (say, a few GB or larger), I get an
OutOfMemory exception with POIFSFileSystem, and with NPOIFSFileSystem, I get
the following exception:
java.lang.IllegalArgumentException
at java.nio.ByteBuffer.allocate(ByteBuffer.java:330)
at
org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.java:275)
Could this be a bug in NPOIFSFileSystem?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 56447] IllegalArgumentException when initializing
NPOIFSFileSystem object
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56447
Nick Burch <ap...@gagravarr.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #3 from Nick Burch <ap...@gagravarr.org> ---
The NPOIFSFileSystem constructor that takes an InputStream buffers the whole
file into memory. So, your heap space needs to be at least the size of the
file, plus a bit extra. Nothing we can do to help there - you just have to
increase your heap
Alternately, NPOFSFileSystem has a constructor that takes a File, that has a
much much lower memory footprint as a File allows for Random Access
(InputStream does not)
If you increase your heap to something like 15% bigger than the file, does it
work with an InputStream?
If you switch to a File, does that fix it without a bigger heap?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 56447] IllegalArgumentException when initializing
NPOIFSFileSystem object
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56447
Nick Burch <ap...@gagravarr.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #5 from Nick Burch <ap...@gagravarr.org> ---
Looks like we might have one or more ints that need to be a long.
Are you able to generate a file that's >2gb, but data mostly 0s so it can be
easily compressed down to something small? That can then be used in unit tests
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 56447] IllegalArgumentException when initializing
NPOIFSFileSystem object
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56447
mskan@dtu.dk changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #2 from mskan@dtu.dk ---
I have now created a minimal example that throws the following exception:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
at
org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.java:273)
at TestApp.main(TestApp.java:16)
The values in NPOIFS are:
_header.getBATCount() = 493
maxSize = 2067795968
The size of the file that I am opening is 3,365,793,792 bytes.
(In reply to Nick Burch from comment #1)
> Any chance you could attach a debugger, and check the values in NPOIFS of
> maxSize (line 274) and _header.getBATCount()?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 56447] IllegalArgumentException when initializing
NPOIFSFileSystem object
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56447
Nick Burch <ap...@gagravarr.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
OS| |All
--- Comment #1 from Nick Burch <ap...@gagravarr.org> ---
Any chance you could attach a debugger, and check the values in NPOIFS of
maxSize (line 274) and _header.getBATCount()?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 56447] IllegalArgumentException when initializing
NPOIFSFileSystem object
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56447
mskan@dtu.dk changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #4 from mskan@dtu.dk ---
I realize that the numbers that I gave you before were for a file of size
2,066,718,720 bytes. Increasing the heap size (using -Xmx3G) helps when opening
the file as an InputStream, and it also works with the standard heap size when
I use File instead of InputStream. However, when I try opening the file that is
3,365,793,792 bytes, I get this exception when passing a File object to
NPOIFSFileSystem:
Exception in thread "main" java.lang.IllegalArgumentException
at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:275)
at
org.apache.poi.poifs.nio.FileBackedDataSource.read(FileBackedDataSource.java:57)
at
org.apache.poi.poifs.filesystem.NPOIFSFileSystem.getBlockAt(NPOIFSFileSystem.java:426)
at
org.apache.poi.poifs.filesystem.NPOIFSFileSystem.readBAT(NPOIFSFileSystem.java:402)
at
org.apache.poi.poifs.filesystem.NPOIFSFileSystem.readCoreContents(NPOIFSFileSystem.java:377)
at
org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.java:201)
at
org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.java:162)
at
org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.java:143)
Could this be because of integer overflow? If I use the InputStream and
increase the heap size to 4GB, then maxSize is equal to -926937088 (and
_header.getBATCount() = 803), and I get the following exception:
Exception in thread "main" java.lang.IllegalArgumentException
at java.nio.ByteBuffer.allocate(ByteBuffer.java:330)
at
org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.java:273)
at ReadTXRM.main(ReadTXRM.java:19)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 56447] IllegalArgumentException when initializing
NPOIFSFileSystem object
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56447
--- Comment #9 from mskan@dtu.dk ---
I found the nightly build (poi-3.11-beta1-20140424). It works! Now I can read
large data files using File. Thanks for all your help, I really appreciate it!
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 56447] IllegalArgumentException when initializing
NPOIFSFileSystem object
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56447
--- Comment #8 from mskan@dtu.dk ---
Sure, I can try it. Where do I find the new version? Can you send me a jar
file?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 56447] IllegalArgumentException when initializing
NPOIFSFileSystem object
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56447
--- Comment #7 from Nick Burch <ap...@gagravarr.org> ---
Can you try now?
For an InputStream, you should now get a helpful IllegalArgumentException on a
>2gb file, because ByteBuffer has a 2gb limit
For a File, we ought to be able to go bigger. I've switched a couple of ints to
longs, can you see if that helps?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 56447] IllegalArgumentException when initializing
NPOIFSFileSystem object
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56447
Nick Burch <ap...@gagravarr.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org
[Bug 56447] IllegalArgumentException when initializing
NPOIFSFileSystem object
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56447
mskan@dtu.dk changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #6 from mskan@dtu.dk ---
Unfortunately I don't think that I can create such a file, but I can give you
the file that causes the problem (3.37 GB uncompressed and 2.16 GB compressed
with bzip2).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org