You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Shay Banon <ki...@gmail.com> on 2010/06/05 04:10:14 UTC
Index Writer, RawPostingList and Memory Management
Hi,
I have a question on how IndexWriter manages its memory when it comes to
RawPostingList. Its pretty late, so sorry if the question is obvious, but
the question is when does the RawPostingList array gets reclaimed or
shrinked? I assume they get factored in the ram balancing process, but, is
there a chance that they won't be reclaimed? Can I force them to be
reclaimed, I think that IndexWriter#commit does not affect that... . Here is
a link to a sample heap dump I took (I do call flush):
http://skitch.com/kimchy/dg3e4/eclipse-memory-analyzer.
Thanks,
Shay
Re: Index Writer, RawPostingList and Memory Management
Posted by Michael McCandless <lu...@mikemccandless.com>.
Previously, they were not freed, just reused.
As of the tips of 29x, 30x, 3x, trunk, they are now fully freed.
Mike
On Sat, Jun 5, 2010 at 7:33 AM, Shay Banon <ki...@gmail.com> wrote:
> I will check, just wanted to verify though, do they get freed when calling
> commit, or do they still hang around? Other pools are freed when calling
> commit, right?
>
> -shay.banon
>
> On Sat, Jun 5, 2010 at 12:55 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> They are reclaimed, but, you may be hitting one of the memory leak
>> bugs recently fixed but not yet (though soon to be) released.
>>
>> Can you try running with the tip of whichever release you are using?
>>
>> Mike
>>
>> On Fri, Jun 4, 2010 at 10:10 PM, Shay Banon <ki...@gmail.com> wrote:
>> > Hi,
>> >
>> > I have a question on how IndexWriter manages its memory when it comes to
>> > RawPostingList. Its pretty late, so sorry if the question is obvious, but
>> > the question is when does the RawPostingList array gets reclaimed or
>> > shrinked? I assume they get factored in the ram balancing process, but,
>> is
>> > there a chance that they won't be reclaimed? Can I force them to be
>> > reclaimed, I think that IndexWriter#commit does not affect that... . Here
>> is
>> > a link to a sample heap dump I took (I do call flush):
>> > http://skitch.com/kimchy/dg3e4/eclipse-memory-analyzer.
>> >
>> > Thanks,
>> > Shay
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Index Writer, RawPostingList and Memory Management
Posted by Shay Banon <ki...@gmail.com>.
I will check, just wanted to verify though, do they get freed when calling
commit, or do they still hang around? Other pools are freed when calling
commit, right?
-shay.banon
On Sat, Jun 5, 2010 at 12:55 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> They are reclaimed, but, you may be hitting one of the memory leak
> bugs recently fixed but not yet (though soon to be) released.
>
> Can you try running with the tip of whichever release you are using?
>
> Mike
>
> On Fri, Jun 4, 2010 at 10:10 PM, Shay Banon <ki...@gmail.com> wrote:
> > Hi,
> >
> > I have a question on how IndexWriter manages its memory when it comes to
> > RawPostingList. Its pretty late, so sorry if the question is obvious, but
> > the question is when does the RawPostingList array gets reclaimed or
> > shrinked? I assume they get factored in the ram balancing process, but,
> is
> > there a chance that they won't be reclaimed? Can I force them to be
> > reclaimed, I think that IndexWriter#commit does not affect that... . Here
> is
> > a link to a sample heap dump I took (I do call flush):
> > http://skitch.com/kimchy/dg3e4/eclipse-memory-analyzer.
> >
> > Thanks,
> > Shay
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Index Writer, RawPostingList and Memory Management
Posted by Michael McCandless <lu...@mikemccandless.com>.
They are reclaimed, but, you may be hitting one of the memory leak
bugs recently fixed but not yet (though soon to be) released.
Can you try running with the tip of whichever release you are using?
Mike
On Fri, Jun 4, 2010 at 10:10 PM, Shay Banon <ki...@gmail.com> wrote:
> Hi,
>
> I have a question on how IndexWriter manages its memory when it comes to
> RawPostingList. Its pretty late, so sorry if the question is obvious, but
> the question is when does the RawPostingList array gets reclaimed or
> shrinked? I assume they get factored in the ram balancing process, but, is
> there a chance that they won't be reclaimed? Can I force them to be
> reclaimed, I think that IndexWriter#commit does not affect that... . Here is
> a link to a sample heap dump I took (I do call flush):
> http://skitch.com/kimchy/dg3e4/eclipse-memory-analyzer.
>
> Thanks,
> Shay
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org