You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Ganesh <em...@yahoo.co.in> on 2009/01/08 09:38:04 UTC
Writer memory released after flush
I am maintaing multiple indexes and all writers will be opened. Based on
some criteria, the document is added to a particular index. I am using 50 MB
of RAM buffer size. Whether the buffer will be released once flush or
optinization done or the writer will maintain the memory and recyle it.
Regards
Ganesh
Send instant messages to your online friends http://in.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Writer memory released after flush
Posted by Ganesh <em...@yahoo.co.in>.
I tried deletion using readers, but as the writers are opened the deletion
got failed. I am sharding the index based on datetime and the writer needs
to be opened for certain days. Till the writers are opened, i can't do
deletion using reader.
I think i need to close and open a new reader.
Regards
Ganesh
----- Original Message -----
From: "Michael McCandless" <lu...@mikemccandless.com>
To: <ja...@lucene.apache.org>
Sent: Friday, January 09, 2009 4:47 PM
Subject: Re: Writer memory released after flush
>
> Calling writer.setRamBufferSizeMB(X) won't immediately free any held RAM;
> on making the next change to the index (deletion or addition) the RAM
> should be freed. The workaround is to close & open a new writer.
>
> Can you open readers to do deletion?
>
> Mike
>
> Ganesh wrote:
>
>> I am sharding my index database and as i move on to different database,
>> the memory acquired by the writer is not released and shortly the app is
>> crashed due to out of memory.
>>
>> During indexing a particular folder, i want the RAM buffer set to 50MB,
>> thereafter i should do only deletions and the RAM buffer should be set
>> to 10MB or even less 5MB. I need to someway release the memory hold by
>> the writer. I am refreshing my searcher often. I need to keep open all
>> my writers to do deletion.
>>
>> writer.setRamBufferSizeMB(50)
>> .....
>> writer.setRamBufferSizeMB(5) //Whether this call will reset the buffer.
>> OR closing and opening writer will reset the buffer
>>
>> Please let me know, is there any other way?
>>
>> Regards
>> Ganesh
>>
>>
>>
>> ----- Original Message ----- From: "Michael McCandless"
>> <lucene@mikemccandless.com
>> >
>> To: <ja...@lucene.apache.org>
>> Sent: Thursday, January 08, 2009 6:44 PM
>> Subject: Re: Writer memory released after flush
>>
>>
>>>
>>> Opening a writer should be a very fast operation, but closing may be
>>> costly since that does a commit() to make the changes permanent in the
>>> IO system.
>>>
>>> Still... if you don't need to refresh your searchers you may want to
>>> just leave the writers open, but perhaps lower the RAM buffer if you
>>> don't have enough memory dedicated to the JRE?
>>>
>>> Mike
>>>
>>> Ganesh wrote:
>>>
>>>> Whether it will be a good idea to open writer only when required.
>>>> Whether opening writer will take some time as like searcher?
>>>> Whether to initialize writer will take some time?
>>>>
>>>> Regards
>>>> Ganesh
>>>>
>>>> ----- Original Message ----- From: "Michael McCandless"
>>>> <lucene@mikemccandless.com
>>>> >
>>>> To: <ja...@lucene.apache.org>
>>>> Sent: Thursday, January 08, 2009 3:52 PM
>>>> Subject: Re: Writer memory released after flush
>>>>
>>>>
>>>>>
>>>>> IndexWriter holds onto the memory & recycles it. It's best to close
>>>>> the writer if you need to release the memory.
>>>>>
>>>>> Mike
>>>>>
>>>>> Ganesh wrote:
>>>>>
>>>>>> I am maintaing multiple indexes and all writers will be opened.
>>>>>> Based on some criteria, the document is added to a particular
>>>>>> index. I am using 50 MB of RAM buffer size. Whether the buffer
>>>>>> will be released once flush or optinization done or the writer
>>>>>> will maintain the memory and recyle it.
>>>>>>
>>>>>> Regards
>>>>>> Ganesh
>>>>>>
>>>>>>
>>>>>> Send instant messages to your online friends
>>>>>> http://in.messenger.yahoo.com
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>> Send instant messages to your online friends
>>>> http://in.messenger.yahoo.com
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> Send instant messages to your online friends
>> http://in.messenger.yahoo.com
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Send instant messages to your online friends http://in.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Writer memory released after flush
Posted by Michael McCandless <lu...@mikemccandless.com>.
Calling writer.setRamBufferSizeMB(X) won't immediately free any held
RAM; on making the next change to the index (deletion or addition) the
RAM should be freed. The workaround is to close & open a new writer.
Can you open readers to do deletion?
Mike
Ganesh wrote:
> I am sharding my index database and as i move on to different
> database, the memory acquired by the writer is not released and
> shortly the app is crashed due to out of memory.
>
> During indexing a particular folder, i want the RAM buffer set to
> 50MB, thereafter i should do only deletions and the RAM buffer
> should be set to 10MB or even less 5MB. I need to someway release
> the memory hold by the writer. I am refreshing my searcher often. I
> need to keep open all my writers to do deletion.
>
> writer.setRamBufferSizeMB(50)
> .....
> writer.setRamBufferSizeMB(5) //Whether this call will reset the
> buffer. OR closing and opening writer will reset the buffer
>
> Please let me know, is there any other way?
>
> Regards
> Ganesh
>
>
>
> ----- Original Message ----- From: "Michael McCandless" <lucene@mikemccandless.com
> >
> To: <ja...@lucene.apache.org>
> Sent: Thursday, January 08, 2009 6:44 PM
> Subject: Re: Writer memory released after flush
>
>
>>
>> Opening a writer should be a very fast operation, but closing may
>> be costly since that does a commit() to make the changes permanent
>> in the IO system.
>>
>> Still... if you don't need to refresh your searchers you may want
>> to just leave the writers open, but perhaps lower the RAM buffer
>> if you don't have enough memory dedicated to the JRE?
>>
>> Mike
>>
>> Ganesh wrote:
>>
>>> Whether it will be a good idea to open writer only when required.
>>> Whether opening writer will take some time as like searcher?
>>> Whether to initialize writer will take some time?
>>>
>>> Regards
>>> Ganesh
>>>
>>> ----- Original Message ----- From: "Michael McCandless" <lucene@mikemccandless.com
>>> >
>>> To: <ja...@lucene.apache.org>
>>> Sent: Thursday, January 08, 2009 3:52 PM
>>> Subject: Re: Writer memory released after flush
>>>
>>>
>>>>
>>>> IndexWriter holds onto the memory & recycles it. It's best to
>>>> close the writer if you need to release the memory.
>>>>
>>>> Mike
>>>>
>>>> Ganesh wrote:
>>>>
>>>>> I am maintaing multiple indexes and all writers will be
>>>>> opened. Based on some criteria, the document is added to a
>>>>> particular index. I am using 50 MB of RAM buffer size. Whether
>>>>> the buffer will be released once flush or optinization done or
>>>>> the writer will maintain the memory and recyle it.
>>>>>
>>>>> Regards
>>>>> Ganesh
>>>>>
>>>>>
>>>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Writer memory released after flush
Posted by Ganesh <em...@yahoo.co.in>.
I am sharding my index database and as i move on to different database, the
memory acquired by the writer is not released and shortly the app is crashed
due to out of memory.
During indexing a particular folder, i want the RAM buffer set to 50MB,
thereafter i should do only deletions and the RAM buffer should be set to
10MB or even less 5MB. I need to someway release the memory hold by the
writer. I am refreshing my searcher often. I need to keep open all my
writers to do deletion.
writer.setRamBufferSizeMB(50)
.....
writer.setRamBufferSizeMB(5) //Whether this call will reset the buffer. OR
closing and opening writer will reset the buffer
Please let me know, is there any other way?
Regards
Ganesh
----- Original Message -----
From: "Michael McCandless" <lu...@mikemccandless.com>
To: <ja...@lucene.apache.org>
Sent: Thursday, January 08, 2009 6:44 PM
Subject: Re: Writer memory released after flush
>
> Opening a writer should be a very fast operation, but closing may be
> costly since that does a commit() to make the changes permanent in the IO
> system.
>
> Still... if you don't need to refresh your searchers you may want to just
> leave the writers open, but perhaps lower the RAM buffer if you don't
> have enough memory dedicated to the JRE?
>
> Mike
>
> Ganesh wrote:
>
>> Whether it will be a good idea to open writer only when required.
>> Whether opening writer will take some time as like searcher? Whether
>> to initialize writer will take some time?
>>
>> Regards
>> Ganesh
>>
>> ----- Original Message ----- From: "Michael McCandless"
>> <lucene@mikemccandless.com
>> >
>> To: <ja...@lucene.apache.org>
>> Sent: Thursday, January 08, 2009 3:52 PM
>> Subject: Re: Writer memory released after flush
>>
>>
>>>
>>> IndexWriter holds onto the memory & recycles it. It's best to close
>>> the writer if you need to release the memory.
>>>
>>> Mike
>>>
>>> Ganesh wrote:
>>>
>>>> I am maintaing multiple indexes and all writers will be opened. Based
>>>> on some criteria, the document is added to a particular index. I am
>>>> using 50 MB of RAM buffer size. Whether the buffer will be released
>>>> once flush or optinization done or the writer will maintain the
>>>> memory and recyle it.
>>>>
>>>> Regards
>>>> Ganesh
>>>>
>>>>
>>>> Send instant messages to your online friends
>>>> http://in.messenger.yahoo.com
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> Send instant messages to your online friends
>> http://in.messenger.yahoo.com
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Send instant messages to your online friends http://in.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Writer memory released after flush
Posted by Michael McCandless <lu...@mikemccandless.com>.
Opening a writer should be a very fast operation, but closing may be
costly since that does a commit() to make the changes permanent in the
IO system.
Still... if you don't need to refresh your searchers you may want to
just leave the writers open, but perhaps lower the RAM buffer if you
don't have enough memory dedicated to the JRE?
Mike
Ganesh wrote:
> Whether it will be a good idea to open writer only when required.
> Whether opening writer will take some time as like searcher?
> Whether to initialize writer will take some time?
>
> Regards
> Ganesh
>
> ----- Original Message ----- From: "Michael McCandless" <lucene@mikemccandless.com
> >
> To: <ja...@lucene.apache.org>
> Sent: Thursday, January 08, 2009 3:52 PM
> Subject: Re: Writer memory released after flush
>
>
>>
>> IndexWriter holds onto the memory & recycles it. It's best to
>> close the writer if you need to release the memory.
>>
>> Mike
>>
>> Ganesh wrote:
>>
>>> I am maintaing multiple indexes and all writers will be opened.
>>> Based on some criteria, the document is added to a particular
>>> index. I am using 50 MB of RAM buffer size. Whether the buffer
>>> will be released once flush or optinization done or the writer
>>> will maintain the memory and recyle it.
>>>
>>> Regards
>>> Ganesh
>>>
>>>
>>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Writer memory released after flush
Posted by Ganesh <em...@yahoo.co.in>.
Whether it will be a good idea to open writer only when required. Whether
opening writer will take some time as like searcher? Whether to initialize
writer will take some time?
Regards
Ganesh
----- Original Message -----
From: "Michael McCandless" <lu...@mikemccandless.com>
To: <ja...@lucene.apache.org>
Sent: Thursday, January 08, 2009 3:52 PM
Subject: Re: Writer memory released after flush
>
> IndexWriter holds onto the memory & recycles it. It's best to close the
> writer if you need to release the memory.
>
> Mike
>
> Ganesh wrote:
>
>> I am maintaing multiple indexes and all writers will be opened. Based on
>> some criteria, the document is added to a particular index. I am using
>> 50 MB of RAM buffer size. Whether the buffer will be released once flush
>> or optinization done or the writer will maintain the memory and recyle
>> it.
>>
>> Regards
>> Ganesh
>>
>>
>> Send instant messages to your online friends
>> http://in.messenger.yahoo.com
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Send instant messages to your online friends http://in.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Writer memory released after flush
Posted by Michael McCandless <lu...@mikemccandless.com>.
IndexWriter holds onto the memory & recycles it. It's best to close
the writer if you need to release the memory.
Mike
Ganesh wrote:
> I am maintaing multiple indexes and all writers will be opened.
> Based on some criteria, the document is added to a particular index.
> I am using 50 MB of RAM buffer size. Whether the buffer will be
> released once flush or optinization done or the writer will maintain
> the memory and recyle it.
>
> Regards
> Ganesh
>
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org