You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (Created) (JIRA)" <ji...@apache.org> on 2011/12/20 16:15:30 UTC
[jira] [Created] (LUCENE-3659) Improve Javadocs of RAMDirectory to
document its limitations
Improve Javadocs of RAMDirectory to document its limitations
------------------------------------------------------------
Key: LUCENE-3659
URL: https://issues.apache.org/jira/browse/LUCENE-3659
Project: Lucene - Java
Issue Type: Task
Affects Versions: 3.5, 4.0
Reporter: Uwe Schindler
Assignee: Uwe Schindler
Fix For: 3.6, 4.0
Spinoff from several dev@lao issues:
- [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/browser]
- issue LUCENE-3653
The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations and add improvements to make it more GC
friendly on large indexes
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238441#comment-13238441 ]
Uwe Schindler commented on LUCENE-3659:
---------------------------------------
Robert: That was the first idea that came to my mind, too. I think thats a good idea. It especially strange that the segments_xx file (which is not part of the current segment) is written with MERGE/FLUSH context. It should be written with a standard context? Or do I miss something? (This was the reason why I added the file name check). Initially I was expecting that writing the commit is done with a separate IOContext, but it isn't - the noisy debugging helps.
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3659) Improve Javadocs of RAMDirectory to
document its limitations and add improvements to make it more GC friendly
on large indexes
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3659:
----------------------------------
Attachment: LUCENE-3659.patch
More improvements:
- If you use new RAMDirectory(existingDir), the RAMFiles in the created RAMDirectory will have the original fileSize (if less then 1L << 30 bytes) as bufferSize, as we know the file size upfront.
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Allow per-RAMFile buffer sizes
based on IOContext and source of data (e.g. copy from another directory)
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239407#comment-13239407 ]
Uwe Schindler commented on LUCENE-3659:
---------------------------------------
{quote}
I also don't think RAMFile should be public, ie, the only way to make
changes to a file stored in a RAMDir is via RAMOutputStream. We can
do this separately...
{quote}
RAMFile's public ctor without directory is only used by PrefixCodedTerms, itself used only by FrozenBufferedDeletes. I don't really see the real use case to do this like that. We can maybe replace that using a FST (it is already sorted by BytesRef) or using PagesBytes? Alternatively replace the whole thing with OutputStreamDataOutput(new ByteArrayOutputStream())?
> Allow per-RAMFile buffer sizes based on IOContext and source of data (e.g. copy from another directory)
> -------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Sub-task
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations and add improvements to make it more GC
friendly on large indexes
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238451#comment-13238451 ]
Michael McCandless commented on LUCENE-3659:
--------------------------------------------
I'm torn on the binary "metadata" idea... not all files cleanly fall into one category?
Eg what about live doc bits? It can easily be tiny (we write a sparse set sparsely).
Indices w/ immense docs will also start to look like they have tiny files that are not metadata (eg, fdx file, if they don't store fields).
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (LUCENE-3659) Improve Javadocs of
RAMDirectory to document its limitations and add improvements to make it
more GC friendly on large indexes
Posted by "Uwe Schindler (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238441#comment-13238441 ]
Uwe Schindler edited comment on LUCENE-3659 at 3/26/12 2:46 PM:
----------------------------------------------------------------
Robert: That was the first idea that came to my mind, too. I think thats a good idea. It especially strange that the segments_xx/segments.gen file (which is not part of the current segment) is written with MERGE/FLUSH context. It should be written with a standard context? Or do I miss something? (This was the reason why I added the file name check). Initially I was expecting that writing the commit is done with a separate IOContext, but it isn't - the noisy debugging helps.
was (Author: thetaphi):
Robert: That was the first idea that came to my mind, too. I think thats a good idea. It especially strange that the segments_xx file (which is not part of the current segment) is written with MERGE/FLUSH context. It should be written with a standard context? Or do I miss something? (This was the reason why I added the file name check). Initially I was expecting that writing the commit is done with a separate IOContext, but it isn't - the noisy debugging helps.
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3659) Allow per-RAMFile buffer sizes based
on IOContext and source of data (e.g. copy from another directory)
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3659:
----------------------------------
Affects Version/s: (was: 3.5)
Fix Version/s: (was: 3.6)
Summary: Allow per-RAMFile buffer sizes based on IOContext and source of data (e.g. copy from another directory) (was: Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes)
> Allow per-RAMFile buffer sizes based on IOContext and source of data (e.g. copy from another directory)
> -------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Sub-task
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations
Posted by "Erick Erickson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173247#comment-13173247 ]
Erick Erickson commented on LUCENE-3659:
----------------------------------------
>From the dev list, didn't want to lose this background (or make Uwe type it again <G>)
The idea was to maybe replace RAMDirectory by a “clone” of MMapDirectory that uses large DirectByteBuffers outside the JVM heap. The current RAMDirectory is very limited (buffersize hardcoded to 8 KB, if you have a 50 Gigabyte Index in this RAMDirectory, your GC simply drives crazy – we investigated this several times for customers. RAMDirectory was in fact several times slower than a simple disk-based MMapDir). Also the locking on the RAMFile class is horrible, as for large indexes you have to change buffer several times when seeking/reading/…, which does heavily locking. In contrast, MMapDir is completely lock-free!
Until there is no replacement we will not remove it, but the current RAMDirectory is not useable for large indexes. That’s a limitation and the design of this class does not support anything else. It’s currently unfixable and instead of putting work into fixing it, the time should be spent in working on a new ByteBuffer-based RAMDir with larger blocs/blocs that merge or IOContext helping to calculate the file size before writing it (e.g. when triggering a merge you know the approximate size of the file before, so you can allocate a buffer that’s better than 8 Kilobytes). Also directByteBuffer helps to make GC happy, as the RAMdir is outside JVM heap.....
RAMdir uses more time for switching buffers than reading the data. The problem is that MMapDir does not support *writing* and that why we plan to improve this. Have you tried MMapDir for read access in comparison to RAMDirectory for a larger index, it outperforms several times (depending on OS and if file data is in FS cache already). The new directory will simply mimic the MMapIndexInput, add MMapIndexOutput, but not based on a mmapped buffer, instead a in-memory (Direct)ByteBuffer (outside or inside JVM heap – both will be supported). This simplifies code a lot.
The discussions about the limitations of crappy RAMDirectory were discussed on conferences, sorry. We did *not*decide to remove it (without a patch/replacement). The whole “message” on the issue was that RAMDirectory is a bad idea. The recommended approach at the moment to handle large in-ram directories would be to use a tmpfs on Linux/Solaris and use MMapDir on top (for larger indexes). The MMap would then directly map the RAM of the underlying tmpfs.....
> Improve Javadocs of RAMDirectory to document its limitations
> ------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3659) Improve Javadocs of RAMDirectory to
document its limitations and add improvements to make it more GC friendly
on large indexes
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3659:
----------------------------------
Summary: Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes (was: Improve Javadocs of RAMDirectory to document its limitations)
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations and add improvements to make it more GC
friendly on large indexes
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238453#comment-13238453 ]
Robert Muir commented on LUCENE-3659:
-------------------------------------
But also codecs that write their own private tiny metadata files (like .per from PerFieldPostingsFormat)
should set this in the context.
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3659) Improve Javadocs of RAMDirectory to
document its limitations and add improvements to make it more GC friendly
on large indexes
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3659:
----------------------------------
Attachment: LUCENE-3659.patch
A played a little bit around and implemented the IOContext / filename dependent buffer sizes for RAMFiles.
The code currently prints out lot's of size infornation (like buffer sizes) on RAMDirectory.close(). This is just for debugging and to show what happens.
To catually see real-world use cases, execute tests with ant test -Dtests.directory=RAMDirectory -Dtests.nightly=true
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3659) Improve Javadocs of RAMDirectory to
document its limitations and add improvements to make it more GC friendly
on large indexes
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3659:
----------------------------------
Attachment: (was: LUCENE-3659.patch)
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173252#comment-13173252 ]
Uwe Schindler commented on LUCENE-3659:
---------------------------------------
It's even worse, it uses a buffer size of 1 Kilobyte:
{code:java}
public class RAMOutputStream extends IndexOutput {
static final int BUFFER_SIZE = 1024;
{code}
A 50 Gigabyte file means 52428800 byte[] arrays :-(
> Improve Javadocs of RAMDirectory to document its limitations
> ------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations and add improvements to make it more GC
friendly on large indexes
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238402#comment-13238402 ]
Michael McCandless commented on LUCENE-3659:
--------------------------------------------
This looks great Uwe!
I'm a little worried about the tiny file case; you're checking for
SEGMENTS_* now, but many other files can be much smaller than 1/64th
of the estimated segment size.
I wonder if we should "improve" IOContext to hold the [rough]
estimated file size (not just overall segment size)... the thing is
that's sort of a hassle on codec impls.
Or: maybe, on closing the ROS/RAMFile, we can downsize the final
buffer (yes, this means copying the bytes, but that cost is vanishingly
small as the RAMDir grows). Then tiny files stay tiny, though they
are still [relatively] costly to create...
I don't this RAMDir.createOutput should publish the RAMFile until the
ROS is closed? Ie, you are not allowed to openInput on something
still opened with createOutput in any Lucene Dir impl..? This would
allow us to make RAMFile frozen (eg if ROS holds its own buffers and
then creates RAMFile on close), that requires no sync when reading?
I also don't think RAMFile should be public, ie, the only way to make
changes to a file stored in a RAMDir is via RAMOutputStream. We can
do this separately...
Maybe we should pursue a growing buffer size...? Ie, where each newly
added buffer is bigger than the one before (like ArrayUtil.oversize's
growth function)... I realize that adds complexity
(RAMInputStream.seek is more fun), but this would let tiny files use
tiny RAM and huge files use few buffers. Ie, RAMDir would scale up
and scale down well.
Separately: I noticed we still have IndexOutput.setLength, but, nobody
calls it anymore I think? (In 3.x we call this when creating a CFS).
Maybe we should remove it...
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations and add improvements to make it more GC
friendly on large indexes
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237927#comment-13237927 ]
Robert Muir commented on LUCENE-3659:
-------------------------------------
Honestly I don't even have the time to review the patch: I'm sure Uwe's changes (as always)
are very nice and thorough.
I just want to propose the idea of a javadocs-only fix for 3.6: I am afraid of any .store changes,
except serious bugfixes (with serious tests to go with them) this close to release.
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3659) Improve Javadocs of RAMDirectory to
document its limitations and add improvements to make it more GC friendly
on large indexes
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3659:
----------------------------------
Attachment: LUCENE-3659.patch
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations and add improvements to make it more GC
friendly on large indexes
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238407#comment-13238407 ]
Robert Muir commented on LUCENE-3659:
-------------------------------------
{quote}
I'm a little worried about the tiny file case; you're checking for
SEGMENTS_* now, but many other files can be much smaller than 1/64th
of the estimated segment size.
I wonder if we should "improve" IOContext to hold the [rough]
estimated file size (not just overall segment size)... the thing is
that's sort of a hassle on codec impls.
{quote}
Maybe its enough for IOContext to specify that its writing a 'metadata'
file? These are all the tiny ones (fieldinfos, segmentinfos, .cfe, etc),
as opposed to 'real files' like frq or prx that are expected to be possibly huge.
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3659) Improve Javadocs of RAMDirectory to
document its limitations
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3659:
----------------------------------
Description:
Spinoff from several dev@lao issues:
- [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
- issue LUCENE-3653
The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
was:
Spinoff from several dev@lao issues:
- [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/browser]
- issue LUCENE-3653
The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
> Improve Javadocs of RAMDirectory to document its limitations
> ------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations
Posted by "Shay Banon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174946#comment-13174946 ]
Shay Banon commented on LUCENE-3659:
------------------------------------
Uwe: Hey, have you looked at LUCENE-2292, this can be a good candidate to use to replace RAM directory. I just "refreshed" it and it seems to work find (all tests pass).
> Improve Javadocs of RAMDirectory to document its limitations
> ------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Allow per-RAMFile buffer sizes
based on IOContext and source of data (e.g. copy from another directory)
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239463#comment-13239463 ]
Robert Muir commented on LUCENE-3659:
-------------------------------------
oops: I did that, sorry.
it just wants a thing that combined byte[] slices that you can
get datainput from, so it seemed like the right thing?
> Allow per-RAMFile buffer sizes based on IOContext and source of data (e.g. copy from another directory)
> -------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Sub-task
> Affects Versions: 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations
Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188008#comment-13188008 ]
Simon Willnauer commented on LUCENE-3659:
-----------------------------------------
should we make this a blocker for 4.0?
> Improve Javadocs of RAMDirectory to document its limitations
> ------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188111#comment-13188111 ]
Uwe Schindler commented on LUCENE-3659:
---------------------------------------
If I remember the misinformation about using RAMDir going on, yes, it's a blocker - just look at java-user@lao last week, those mails about people saying "I want to copy my 20 Gigabyte FSDir to a RAMDir because it's faster as it has the word 'RAM' in it" looks more like a XY problem than a good reason for using it). MMapDir and even NIOFSDir work mostly from RAM, as the OS will cache for you.
On the other hand, it's just documentation we can do it always.
Fixing the default buffer size to something more suitable for real-world use cases is something that can be done with a one-line-patch. I would prefer 64 Kilobytes buffer size.
> Improve Javadocs of RAMDirectory to document its limitations
> ------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3659) Improve Javadocs of RAMDirectory to
document its limitations and add improvements to make it more GC friendly
on large indexes
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3659:
----------------------------------
Issue Type: Sub-task (was: Task)
Parent: LUCENE-3924
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Sub-task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3659) Improve Javadocs of RAMDirectory to
document its limitations
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3659:
----------------------------------
Attachment: LUCENE-3659.patch
I started to work on this, here is just a first step (trunk). This patch removes the BUFFER_SIZE constant and moves it up to RAMDirectory (but for now only as default, see below!). RAMDirectory inherits the default buffersize for now to its RAMFile childs (newRAMFile() method), but this can likely change (see below).
As every RAMFile has its own buffer size, optimizations are possible:
- when you open an IndexOutput, in trunk we get the IOContext, which may contain a Merge/Flush desc containing the complete segment size (unfortunately the *complete* segment size). But this number can be used as a order of magnitude for specifiing the buffer size.
The patch does not yet implement that, but an idea would be to maybe allocate 1/32 of the segment size as buffer size. By that the buffer size does not get too big, but on the other hand the number of slices has an upper limit (approx 32 slices per merged segment). Currently a merged segment with a size of say 32 Gigabytes would have 32 million byte[] arrays, after the change only 32 byte[] arrays with a size of 1 Gigabyte each. This should make GC happy.
When backporting to 3.x, the IOContext is not yet available and RAMDirectory always uses the default buffer size (maybe randomize in tests). Rainsing the buffer size should bring improvements here.
We should still add some warnings into the Javadocs, that for *large* indexes it is often preferable to use MMapDir, especially when you store it on disk. We should also peple tell that new RAMDirectoty(OtherDirectory) maybe a bad idea...
The new default buffer size was raised from 1024 to 8192.
> Improve Javadocs of RAMDirectory to document its limitations
> ------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations and add improvements to make it more GC
friendly on large indexes
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238454#comment-13238454 ]
Robert Muir commented on LUCENE-3659:
-------------------------------------
Live docs aren't a metadata. I think you are conflating 'tiny' with 'metadata'.
I'm saying we should declare its metadata, thats all. This is pretty black and white!
IF a directory wants to, as a heuristic, interpret metadata == tiny, then thats fine,
but thats separate.
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173255#comment-13173255 ]
Robert Muir commented on LUCENE-3659:
-------------------------------------
I actually think heap versus direct is just an optimization (and ideally would just be an option, in my opinion not the default).
The problem is mostly tiny buffers.
I think a good idea is to rename MMapIndexInput to ByteBufferIndexInput, it does not really care if something is mapped or not, it has all the logic for dealing with multiple fixed-size buffers.
And RAMDirectory could then just use 1MB shift by default (normal heap array-backed buffers). Sure it wastes at most 1MB for tiny files but the RAMFile is wasteful today too.
Down the road we could optimize this: e.g. add IOContext.METADATA for tiny files (segments.gen, .fnm, segments_N, .per, etc), and RAMDir could use say a 256KB shift there and 4MB otherwise. This iocontext flag could also be used if someone didnt want to MMap tiny-files too.
> Improve Javadocs of RAMDirectory to document its limitations
> ------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations
Posted by "Uwe Schindler (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235083#comment-13235083 ]
Uwe Schindler commented on LUCENE-3659:
---------------------------------------
I think we should fix at least JavaDocs for 3.6 and maybe raise buffersize. I will provide a patch tomorrow.
> Improve Javadocs of RAMDirectory to document its limitations
> ------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3659) Improve Javadocs of RAMDirectory to
document its limitations and add improvements to make it more GC friendly
on large indexes
Posted by "Uwe Schindler (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-3659:
----------------------------------
Attachment: LUCENE-3659.patch
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3659) Improve Javadocs of RAMDirectory
to document its limitations and add improvements to make it more GC
friendly on large indexes
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238448#comment-13238448 ]
Robert Muir commented on LUCENE-3659:
-------------------------------------
I think if we were to implement it this way, its not a burden on codecs.
By default, somewhere in lucene core inits the codec APIs with a context always.
For example SegmentInfos.write():
{code}
infosWriter.writeInfos(directory, segmentFileName, codec.getName(), this, IOContext.DEFAULT);
{code}
and DocFieldProcessor/SegmentMerger for fieldinfos:
{code}
infosWriter.write(state.directory, state.segmentName, state.fieldInfos, IOContext.DEFAULT);
{code}
These guys would just set this in the IOContext. Most/All codecs just pass this along.
If a codec wants to ignore the IOContext and lie about it, thats its own choice.
So I think its an easy change.
> Improve Javadocs of RAMDirectory to document its limitations and add improvements to make it more GC friendly on large indexes
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3659
> URL: https://issues.apache.org/jira/browse/LUCENE-3659
> Project: Lucene - Java
> Issue Type: Task
> Affects Versions: 3.5, 4.0
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3659.patch, LUCENE-3659.patch, LUCENE-3659.patch
>
>
> Spinoff from several dev@lao issues:
> - [http://mail-archives.apache.org/mod_mbox/lucene-dev/201112.mbox/%3C001001ccbf1c%2471845830%24548d0890%24%40thetaphi.de%3E]
> - issue LUCENE-3653
> The use cases for RAMDirectory are very limited and to prevent users from using it for e.g. loading a 50 Gigabyte index from a file on disk, we should improve the javadocs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org