You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2013/08/10 17:22:47 UTC
[jira] [Comment Edited] (LUCENE-5164) Remove the OOM cactching in
SimpleFSDirectory and NIOFSDirectory
[ https://issues.apache.org/jira/browse/LUCENE-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735936#comment-13735936 ]
Uwe Schindler edited comment on LUCENE-5164 at 8/10/13 3:21 PM:
----------------------------------------------------------------
Here is the code of RandomAccessFile's native part that explains why the OOM happens there: [http://hg.openjdk.java.net/jdk6/jdk6/jdk/file/dffaa68042cd/src/share/native/java/io/io_util.c]
In fact, when the byte[] is larger than 8 Kilobytes it mallocs a buffer. Otherwise it use a buffer from call stack. This malloc has the same effect as the direct buffer in FileChannel.read().
So we really need the same chunking for write and set the chunk size to 8 Kilobytes W8hcih is done in Robert's previous commit), just not explicit! So we should insert 8192 as number into FSDirectory.
was (Author: thetaphi):
Here is the code of RandomAccessFile's native part that explains why the OOM happens there: [http://hg.openjdk.java.net/jdk6/jdk6/jdk/file/dffaa68042cd/src/share/native/java/io/io_util.c]
In fact, when the byte[] is larger than 8 Kilobytes it mallocs a buffer. Otherwise it use a buffer from heap. This malloc has the same effect as the direct buffer in FileChannel.read().
So we really need the same chunking for write and set the chunk size to 8 Kilobytes W8hcih is done in Robert's previous commit), just not explicit! So we should insert 8192 as number into FSDirectory.
> Remove the OOM cactching in SimpleFSDirectory and NIOFSDirectory
> ----------------------------------------------------------------
>
> Key: LUCENE-5164
> URL: https://issues.apache.org/jira/browse/LUCENE-5164
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Uwe Schindler
> Fix For: 5.0, 4.5
>
>
> Followup from LUCENE-5161:
> In former times we added the OOM cactching in NIOFSDir and SimpleFSDir because nobody understand why the OOM could happen on FileChannel.read() or SimpleFSDir.read(). By reading the Java code its easy to understand (it allocates direct buffers with same size as the requested length to read). As we have chunking now reduce to a few kilobytes it cannot happen anymore that we get spurious OOMs.
> In fact we might hide a *real* OOM! So we should remove it.
> I am also not sure if we should make chunk size configureable in FSDirectory at all! It makes no sense to me (it was in fact only added for people that hit the OOM to fine-tune).
> In my opinion we should remove the setter in trunk and keep it deprecated in 4.x. The buf size is then in trunk equal to the defaults from LUCENE-5161.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org