You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Chris Hostetter <ho...@fucit.org> on 2006/12/01 00:03:48 UTC
Re: 2.1-dev memory leak?
: IndexSearchers open. The other ones I "let go" without an explicit
: close() call. The assumption is that the old IndexSearchers "expire",
: that they get garbage collected, as I'm no longer holding references to
: them.
yeah ... that just seems really bad in general, i would try to explicitly
close any searcher your purge from your cache.
: getting GCed as before. I did not change my logic for creating new
: IndexSearchers (inlined in my previous email). On the other hand, this
: app has recently started getting a lot more search action, so perhaps
: it's just that the GC is not cleaning things up fast enough....
: I happen to have an lsof output from the same system from July. I see
: the same thing there - a number of FDs open and pointing to the same
: .cfs index file. Perhaps it's just that the JVM GC was able to clean
: things up then, and now it can't, because the CPU is maxed out....
: really maxed out.
yeah ... i think you may just be running ino a GC quirk ... with the old
code you didn't need to explicitly close things because they got GCed fast
enough, and now ... not so much. There was a bug in the last few days
looking at calling close() in a finalizer where Michael noticed some
really odd GC/finalizer behavior just depending on wether a boolean that
was never used was set in a finalizer or not ... ghosts in the machine
dude, gremlins and things.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 2.1-dev memory leak?
Posted by Yonik Seeley <yo...@apache.org>.
On 11/30/06, Michael McCandless <lu...@mikemccandless.com> wrote:
> I tested this in 1.9.1, 1.9.2, 2.0.0, and trunk, and all of these
> versions would run out of descriptors. So I'm at a loss so far on
> where the regression is here ...
Right... no regression, just a Java limitation.
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 2.1-dev memory leak?
Posted by Michael McCandless <lu...@mikemccandless.com>.
Yonik Seeley wrote:
> On 11/30/06, Chris Hostetter <ho...@fucit.org> wrote:
>> : IndexSearchers open. The other ones I "let go" without an explicit
>> : close() call. The assumption is that the old IndexSearchers "expire",
>> : that they get garbage collected, as I'm no longer holding references to
>> : them.
>>
>> yeah ... that just seems really bad in general, i would try to explicitly
>> close any searcher your purge from your cache.
>
> Yeah, GC may kick off when memory gets low, but the GC system
> unfortunately doesn't know anything about when file descriptors get
> low.
I actually ran some stress tests to explore this. Admittedly, they
are very stressful tests (open searcher, do 10 searches, don't close
it, repeat). And I definitely found GC would not kick in "often"
enough because I would quickly exhaust file descriptors.
Then, the scary thing is, when I called System.gc explicitly after
creating each searcher, I still exhausted descriptors, which is really
weird! I think System.gc is not guaranteed to have called all
finalizers before returning, but, I expected the explicit calls to
at least close enough of them to stay under the limit.
I tested this in 1.9.1, 1.9.2, 2.0.0, and trunk, and all of these
versions would run out of descriptors. So I'm at a loss so far on
where the regression is here ...
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 2.1-dev memory leak?
Posted by Yonik Seeley <yo...@apache.org>.
On 11/30/06, Chris Hostetter <ho...@fucit.org> wrote:
> : IndexSearchers open. The other ones I "let go" without an explicit
> : close() call. The assumption is that the old IndexSearchers "expire",
> : that they get garbage collected, as I'm no longer holding references to
> : them.
>
> yeah ... that just seems really bad in general, i would try to explicitly
> close any searcher your purge from your cache.
Yeah, GC may kick off when memory gets low, but the GC system
unfortunately doesn't know anything about when file descriptors get
low.
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org