You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Erick Erickson (Commented) (JIRA)" <ji...@apache.org> on 2011/12/20 16:27:32 UTC
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173247#comment-13173247 ]
Erick Erickson commented on LUCENE-3659:
----------------------------------------
>From the dev list, didn't want to lose this background (or make Uwe type it again <G>)
The idea was to maybe replace RAMDirectory by a “clone” of MMapDirectory that uses large DirectByteBuffers outside the JVM heap. The current RAMDirectory is very limited (buffersize hardcoded to 8 KB, if you have a 50 Gigabyte Index in this RAMDirectory, your GC simply drives crazy – we investigated this several times for customers. RAMDirectory was in fact several times slower than a simple disk-based MMapDir). Also the locking on the RAMFile class is horrible, as for large indexes you have to change buffer several times when seeking/reading/…, which does heavily locking. In contrast, MMapDir is completely lock-free!
Until there is no replacement we will not remove it, but the current RAMDirectory is not useable for large indexes. That’s a limitation and the design of this class does not support anything else. It’s currently unfixable and instead of putting work into fixing it, the time should be spent in working on a new ByteBuffer-based RAMDir with larger blocs/blocs that merge or IOContext helping to calculate the file size before writing it (e.g. when triggering a merge you know the approximate size of the file before, so you can allocate a buffer that’s better than 8 Kilobytes). Also directByteBuffer helps to make GC happy, as the RAMdir is outside JVM heap.....
RAMdir uses more time for switching buffers than reading the data. The problem is that MMapDir does not support *writing* and that why we plan to improve this. Have you tried MMapDir for read access in comparison to RAMDirectory for a larger index, it outperforms several times (depending on OS and if file data is in FS cache already). The new directory will simply mimic the MMapIndexInput, add MMapIndexOutput, but not based on a mmapped buffer, instead a in-memory (Direct)ByteBuffer (outside or inside JVM heap – both will be supported). This simplifies code a lot.
The discussions about the limitations of crappy RAMDirectory were discussed on conferences, sorry. We did *not*decide to remove it (without a patch/replacement). The whole “message” on the issue was that RAMDirectory is a bad idea. The recommended approach at the moment to handle large in-ram directories would be to use a tmpfs on Linux/Solaris and use MMapDir on top (for larger indexes). The MMap would then directly map the RAM of the underlying tmpfs.....
> Improve Javadocs of RAMDirectory to document its limitations
> ------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org