You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Otis Gospodnetic <ot...@yahoo.com> on 2006/05/12 00:40:41 UTC
LUCENE-436
I'm not at home with some of the things mentioned in LUCENE-436, so I'm not applying any of the various patches provided there, but it looks like something that deserves attention. I think it has been brought up a while back, too.
http://issues.apache.org/jira/browse/LUCENE-436
Otis
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: LUCENE-436
Posted by Robert Engels <re...@ix.netcom.com>.
There is no "memory leak" per se - just the propensity to use more memory
than would seem to be needed (based on index size).
Using the FixedThreadLocal along with the modified TermInfosReader (that
uses it), the memory problem is resolved. If you are not seeing that, then
you have some other memory leak in your application.
-----Original Message-----
From: Fernando Padilla [mailto:fern@alum.mit.edu]
Sent: Friday, May 12, 2006 1:35 PM
To: java-dev@lucene.apache.org
Subject: Re: LUCENE-436
hmm. So what you're saying is that there is a "memory leak", but very
very noticeable with large RAMDirectories (like what we have)...
With a 5M directories, it seems to be leaking atleast 5M per hour,
depending on queries.. even on our 1500M VM they run our of memory over
24 hours.
So I guess we have no choice but to use FSDirectories?
The FixedThreadLocal patch doesn't seem to have helped afterall..
Robert Engels wrote:
> As stated many times, it is SIGNIFICANT if using RAMdirectories to hold
> entire indexes. If not, then it is not such a big deal.
>
> Rather than using FixedThreadLocal, a more involved solution using a
runtime
> property to determine which thread local impl to use is possible. In lieu
of
> that, RAMDirectories are either broken, or everyone takes a performance
hit.
>
>
> -----Original Message-----
> From: Fernando Padilla [mailto:fern@alum.mit.edu]
> Sent: Thursday, May 11, 2006 9:53 PM
> To: java-dev@lucene.apache.org
> Subject: Re: LUCENE-436
>
>
> so... what do you think?
>
> We just took the patch through QA and there was a noticeable memory
> increase through time, and once we applied the patch, the memory didn't
> increase..
>
> So if you don't like the solution.. what are some alternatives?
>
> fernando
>
> ps - www.protrade.com
>
>
> Otis Gospodnetic wrote:
>
>>I'm not at home with some of the things mentioned in LUCENE-436, so I'm
>
> not applying any of the various patches provided there, but it looks like
> something that deserves attention. I think it has been brought up a while
> back, too.
>
>>http://issues.apache.org/jira/browse/LUCENE-436
>>
>>Otis
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: LUCENE-436
Posted by Fernando Padilla <fe...@alum.mit.edu>.
hmm. So what you're saying is that there is a "memory leak", but very
very noticeable with large RAMDirectories (like what we have)...
With a 5M directories, it seems to be leaking atleast 5M per hour,
depending on queries.. even on our 1500M VM they run our of memory over
24 hours.
So I guess we have no choice but to use FSDirectories?
The FixedThreadLocal patch doesn't seem to have helped afterall..
Robert Engels wrote:
> As stated many times, it is SIGNIFICANT if using RAMdirectories to hold
> entire indexes. If not, then it is not such a big deal.
>
> Rather than using FixedThreadLocal, a more involved solution using a runtime
> property to determine which thread local impl to use is possible. In lieu of
> that, RAMDirectories are either broken, or everyone takes a performance hit.
>
>
> -----Original Message-----
> From: Fernando Padilla [mailto:fern@alum.mit.edu]
> Sent: Thursday, May 11, 2006 9:53 PM
> To: java-dev@lucene.apache.org
> Subject: Re: LUCENE-436
>
>
> so... what do you think?
>
> We just took the patch through QA and there was a noticeable memory
> increase through time, and once we applied the patch, the memory didn't
> increase..
>
> So if you don't like the solution.. what are some alternatives?
>
> fernando
>
> ps - www.protrade.com
>
>
> Otis Gospodnetic wrote:
>
>>I'm not at home with some of the things mentioned in LUCENE-436, so I'm
>
> not applying any of the various patches provided there, but it looks like
> something that deserves attention. I think it has been brought up a while
> back, too.
>
>>http://issues.apache.org/jira/browse/LUCENE-436
>>
>>Otis
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: LUCENE-436
Posted by Robert Engels <re...@ix.netcom.com>.
As stated many times, it is SIGNIFICANT if using RAMdirectories to hold
entire indexes. If not, then it is not such a big deal.
Rather than using FixedThreadLocal, a more involved solution using a runtime
property to determine which thread local impl to use is possible. In lieu of
that, RAMDirectories are either broken, or everyone takes a performance hit.
-----Original Message-----
From: Fernando Padilla [mailto:fern@alum.mit.edu]
Sent: Thursday, May 11, 2006 9:53 PM
To: java-dev@lucene.apache.org
Subject: Re: LUCENE-436
so... what do you think?
We just took the patch through QA and there was a noticeable memory
increase through time, and once we applied the patch, the memory didn't
increase..
So if you don't like the solution.. what are some alternatives?
fernando
ps - www.protrade.com
Otis Gospodnetic wrote:
> I'm not at home with some of the things mentioned in LUCENE-436, so I'm
not applying any of the various patches provided there, but it looks like
something that deserves attention. I think it has been brought up a while
back, too.
> http://issues.apache.org/jira/browse/LUCENE-436
>
> Otis
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: LUCENE-436
Posted by Fernando Padilla <fe...@alum.mit.edu>.
so... what do you think?
We just took the patch through QA and there was a noticeable memory
increase through time, and once we applied the patch, the memory didn't
increase..
So if you don't like the solution.. what are some alternatives?
fernando
ps - www.protrade.com
Otis Gospodnetic wrote:
> I'm not at home with some of the things mentioned in LUCENE-436, so I'm not applying any of the various patches provided there, but it looks like something that deserves attention. I think it has been brought up a while back, too.
> http://issues.apache.org/jira/browse/LUCENE-436
>
> Otis
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org