You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Vitaly Funstein <vf...@gmail.com> on 2012/07/10 04:59:37 UTC
Direct memory footprint of NIOFSDirectory
Hello,
I have recently run into the situation when there was not a sufficient
amount of direct memory available for IndexWriter to work. This was
essentially caused by the embedding application making heavy use of JVM's
direct memory buffers and not leaving enough headroom for NIOFSDirectory to
operate. So what are the approximate guidelines, if any, in terms of JVM
configuration for this choice of Directory to operate safely? Basically,
what I am looking for is a rough estimate of direct memory usage per GB of
indexed data, or per directory/writer instance, if applicable.
Thanks,
-V
Re: Direct memory footprint of NIOFSDirectory
Posted by Vitaly Funstein <vf...@gmail.com>.
Lance,
With all due respect - I am aware of the existence of MMapDirectory,
but it does not answer my original question. Switching over to a
different implementation from a well-tested one to work around a
poorly understood issue carries non-zero risks in a production
environment.
Vitaly
On Thu, Jul 12, 2012 at 2:34 PM, Lance Norskog <go...@gmail.com> wrote:
> You can choose another directory implementation.
>
> On Thu, Jul 12, 2012 at 1:42 PM, Vitaly Funstein <vf...@gmail.com> wrote:
>> Just thought I'd bump this. To clarify - for reasons outside my
>> control, I can't just run the JVM hosting Lucene-enabled application
>> with -XX:MaxDirectMemorySize=100G or some other huge value for the
>> ceiling and never worry about this. Due to preallocation and other
>> restrictions, this parameter has to be fairly close to the actual size
>> used by the app (padded for Lucene and possibly other consumers).
>>
>> On Mon, Jul 9, 2012 at 7:59 PM, Vitaly Funstein <vf...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> I have recently run into the situation when there was not a sufficient amount of direct memory available for IndexWriter to work. This was essentially caused by the embedding application making heavy use of JVM's direct memory buffers and not leaving enough headroom for NIOFSDirectory to operate. So what are the approximate guidelines, if any, in terms of JVM configuration for this choice of Directory to operate safely? Basically, what I am looking for is a rough estimate of direct memory usage per GB of indexed data, or per directory/writer instance, if applicable.
>>>
>>> Thanks,
>>> -V
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Direct memory footprint of NIOFSDirectory
Posted by Lance Norskog <go...@gmail.com>.
You can choose another directory implementation.
On Thu, Jul 12, 2012 at 1:42 PM, Vitaly Funstein <vf...@gmail.com> wrote:
> Just thought I'd bump this. To clarify - for reasons outside my
> control, I can't just run the JVM hosting Lucene-enabled application
> with -XX:MaxDirectMemorySize=100G or some other huge value for the
> ceiling and never worry about this. Due to preallocation and other
> restrictions, this parameter has to be fairly close to the actual size
> used by the app (padded for Lucene and possibly other consumers).
>
> On Mon, Jul 9, 2012 at 7:59 PM, Vitaly Funstein <vf...@gmail.com> wrote:
>>
>> Hello,
>>
>> I have recently run into the situation when there was not a sufficient amount of direct memory available for IndexWriter to work. This was essentially caused by the embedding application making heavy use of JVM's direct memory buffers and not leaving enough headroom for NIOFSDirectory to operate. So what are the approximate guidelines, if any, in terms of JVM configuration for this choice of Directory to operate safely? Basically, what I am looking for is a rough estimate of direct memory usage per GB of indexed data, or per directory/writer instance, if applicable.
>>
>> Thanks,
>> -V
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--
Lance Norskog
goksron@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Direct memory footprint of NIOFSDirectory
Posted by Vitaly Funstein <vf...@gmail.com>.
Just thought I'd bump this. To clarify - for reasons outside my
control, I can't just run the JVM hosting Lucene-enabled application
with -XX:MaxDirectMemorySize=100G or some other huge value for the
ceiling and never worry about this. Due to preallocation and other
restrictions, this parameter has to be fairly close to the actual size
used by the app (padded for Lucene and possibly other consumers).
On Mon, Jul 9, 2012 at 7:59 PM, Vitaly Funstein <vf...@gmail.com> wrote:
>
> Hello,
>
> I have recently run into the situation when there was not a sufficient amount of direct memory available for IndexWriter to work. This was essentially caused by the embedding application making heavy use of JVM's direct memory buffers and not leaving enough headroom for NIOFSDirectory to operate. So what are the approximate guidelines, if any, in terms of JVM configuration for this choice of Directory to operate safely? Basically, what I am looking for is a rough estimate of direct memory usage per GB of indexed data, or per directory/writer instance, if applicable.
>
> Thanks,
> -V
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org