You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Shay Banon (JIRA)" <ji...@apache.org> on 2010/03/02 23:48:27 UTC
[jira] Created: (LUCENE-2292) ByteBuffer Directory - allowing to
store the index outside the heap
ByteBuffer Directory - allowing to store the index outside the heap
-------------------------------------------------------------------
Key: LUCENE-2292
URL: https://issues.apache.org/jira/browse/LUCENE-2292
Project: Lucene - Java
Issue Type: New Feature
Components: Store
Reporter: Shay Banon
A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2292) ByteBuffer Directory - allowing to
store the index outside the heap
Posted by "Shay Banon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840394#action_12840394 ]
Shay Banon commented on LUCENE-2292:
------------------------------------
By the way, an implementation note. I thought about preallocating a large direct buffer and then splicing it into chunks, but currently I think that the complexity (and overhead in maintaining splicing locations) is not really needed and the current caching should do the trick (with the ability to control both the buffer size and the cache size).
> ByteBuffer Directory - allowing to store the index outside the heap
> -------------------------------------------------------------------
>
> Key: LUCENE-2292
> URL: https://issues.apache.org/jira/browse/LUCENE-2292
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
> Reporter: Shay Banon
> Attachments: LUCENE-2292.patch, LUCENE-2292.patch
>
>
> A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2292) ByteBuffer Directory - allowing to
store the index outside the heap
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840373#action_12840373 ]
Uwe Schindler commented on LUCENE-2292:
---------------------------------------
Hi,
looks interesting as a replacement for RAMDirectory.
Your patch uses a "sun." internal package. If you want to do something similar to MMapDirectory to release the buffer without waiting for GC, do it in the same way using reflection like in MMapDirectory.
> ByteBuffer Directory - allowing to store the index outside the heap
> -------------------------------------------------------------------
>
> Key: LUCENE-2292
> URL: https://issues.apache.org/jira/browse/LUCENE-2292
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
> Reporter: Shay Banon
> Attachments: LUCENE-2292.patch
>
>
> A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2292) ByteBuffer Directory - allowing to
store the index outside the heap
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840382#action_12840382 ]
Uwe Schindler commented on LUCENE-2292:
---------------------------------------
There are also other non Sun JREs on the market (IBM, Harmony,...). And e.g. the forceful unmap of MMap dirs is not working on all of them. It is simply hack. Performance is no problem for Directory as close() is seldom called, so there is no method cache in MMapDirectory.
> ByteBuffer Directory - allowing to store the index outside the heap
> -------------------------------------------------------------------
>
> Key: LUCENE-2292
> URL: https://issues.apache.org/jira/browse/LUCENE-2292
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
> Reporter: Shay Banon
> Attachments: LUCENE-2292.patch
>
>
> A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2292) ByteBuffer Directory - allowing to
store the index outside the heap
Posted by "Shay Banon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shay Banon updated LUCENE-2292:
-------------------------------
Attachment: LUCENE-2292.patch
> ByteBuffer Directory - allowing to store the index outside the heap
> -------------------------------------------------------------------
>
> Key: LUCENE-2292
> URL: https://issues.apache.org/jira/browse/LUCENE-2292
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
> Reporter: Shay Banon
> Attachments: LUCENE-2292.patch
>
>
> A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2292) ByteBuffer Directory - allowing to
store the index outside the heap
Posted by "Shay Banon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840379#action_12840379 ]
Shay Banon commented on LUCENE-2292:
------------------------------------
Hi,
>> looks interesting as a replacement for RAMDirectory.
This class uses ByteBuffer, which has its overhead over simple byte[], though using the same logic (if you verify it) can be used to improve the concurrency in RAMDirectory (just use byte[[).
>> Your patch uses a "sun." internal package. If you want to do something similar to MMapDirectory to release the buffer without waiting for GC, do it in the same way using reflection like in MMapDirectory.
>From what I know, it was there in all JDKs I worked with (its like sun.misc.Unsafe). Have you seen otherwise? If so, its a simple change (though I am not sure about the access control thingy in MMapDirectory, its a performance killer, and caching of the Method(s) make sense).
> ByteBuffer Directory - allowing to store the index outside the heap
> -------------------------------------------------------------------
>
> Key: LUCENE-2292
> URL: https://issues.apache.org/jira/browse/LUCENE-2292
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
> Reporter: Shay Banon
> Attachments: LUCENE-2292.patch
>
>
> A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2292) ByteBuffer Directory - allowing to
store the index outside the heap
Posted by "Shay Banon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shay Banon updated LUCENE-2292:
-------------------------------
Attachment: LUCENE-2292.patch
Attached new patch, does not use sun.* package. I still cache Method since cleaning a buffer is not only done on close of the directory.
> ByteBuffer Directory - allowing to store the index outside the heap
> -------------------------------------------------------------------
>
> Key: LUCENE-2292
> URL: https://issues.apache.org/jira/browse/LUCENE-2292
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
> Reporter: Shay Banon
> Attachments: LUCENE-2292.patch, LUCENE-2292.patch
>
>
> A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org