You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Chris Bamford <Ch...@scalix.com> on 2009/08/26 15:18:44 UTC
Lucene gobbling file descriptors
Hi there,
I wonder if someone can help? We have a successful Lucene app deployed on Tomcat which works well. As far as we can tell, our developers have observed all the guidelines in the Lucene FAQ, but on some of our installations, Tomcat eventually runs out of file descriptors and needs a restart to clear it. We know Lucene is the culprit because use lsof -p <java PID> and the vast majority (usually tens of thousands) of files reported are Lucene index files.
I am hoping to get some tips on how this can be avoided. Is it simply the case that as time goes by, more and more descriptors are left open and no matter how high ulimit is set, you will run out? Or is there a policy of recycling that we are failing to utilise properly?
I am happy to provide more information, just don't know what at this point! Please ask....
Thanks in advance
- Chris
Chris Bamford
Senior Development Engineer
Scalix
chris.bamford@scalix.com
Tel: +44 (0)1344 381814
www.scalix.com
Re: Lucene gobbling file descriptors
Posted by Andrzej Bialecki <ab...@getopt.org>.
Erick Erickson wrote:
> Note that for search speed reasons, you really, really want to share your
> readers and NOT open/close for every request.
> FWIW
> Erick
>
> On Thu, Aug 27, 2009 at 9:10 AM, Chris Bamford <Ch...@scalix.com>wrote:
>
>> I'm glad its not normal. That means we can fix it! I will conduct a
>> review of IndexReader/Searcher open/close ops.
If you use Lucene 2.4.1 you can also use the LucidGaze to actually see
how many IndexReader and IndexSearcher instances you have in your
application when running.
See this link for downloads and docs:
http://www.lucidimagination.com/Downloads/LucidGaze-for-Lucene
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene gobbling file descriptors
Posted by Erick Erickson <er...@gmail.com>.
Note that for search speed reasons, you really, really want to share your
readers and NOT open/close for every request.
FWIW
Erick
On Thu, Aug 27, 2009 at 9:10 AM, Chris Bamford <Ch...@scalix.com>wrote:
> I'm glad its not normal. That means we can fix it! I will conduct a
> review of IndexReader/Searcher open/close ops.
>
> Thanks!
>
> Chris
>
> ----- Original Message -----
> From: Michael McCandless <lu...@mikemccandless.com>
> Sent: Wed, 26/8/2009 2:26pm
> To: java-user@lucene.apache.org
> Subject: Re: Lucene gobbling file descriptors
>
> This is not normal. As long as you are certain you close every
> IndexReader/Searcher that you opened, the number of file descriptors
> should stay "contained".
>
> Though: how many files are there in your index directory?
>
> Mike
>
> On Wed, Aug 26, 2009 at 9:18 AM, Chris Bamford<Ch...@scalix.com>
> wrote:
> > Hi there,
> >
> > I wonder if someone can help? We have a successful Lucene app deployed
> on Tomcat which works well. As far as we can tell, our developers have
> observed all the guidelines in the Lucene FAQ, but on some of our
> installations, Tomcat eventually runs out of file descriptors and needs a
> restart to clear it. We know Lucene is the culprit because use lsof -p
> <java PID> and the vast majority (usually tens of thousands) of files
> reported are Lucene index files.
> >
> > I am hoping to get some tips on how this can be avoided. Is it simply
> the case that as time goes by, more and more descriptors are left open and
> no matter how high ulimit is set, you will run out? Or is there a policy of
> recycling that we are failing to utilise properly?
> >
> > I am happy to provide more information, just don't know what at this
> point! Please ask....
> >
> > Thanks in advance
> >
> > - Chris
> >
> > Chris Bamford
> > Senior Development Engineer
> > Scalix
> > chris.bamford@scalix.com
> > Tel: +44 (0)1344 381814
> > www.scalix.com
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Lucene gobbling file descriptors
Posted by Chris Bamford <Ch...@scalix.com>.
I'm glad its not normal. That means we can fix it! I will conduct a review of IndexReader/Searcher open/close ops.
Thanks!
Chris
----- Original Message -----
From: Michael McCandless <lu...@mikemccandless.com>
Sent: Wed, 26/8/2009 2:26pm
To: java-user@lucene.apache.org
Subject: Re: Lucene gobbling file descriptors
This is not normal. As long as you are certain you close every
IndexReader/Searcher that you opened, the number of file descriptors
should stay "contained".
Though: how many files are there in your index directory?
Mike
On Wed, Aug 26, 2009 at 9:18 AM, Chris Bamford<Ch...@scalix.com> wrote:
> Hi there,
>
> I wonder if someone can help? We have a successful Lucene app deployed on Tomcat which works well. As far as we can tell, our developers have observed all the guidelines in the Lucene FAQ, but on some of our installations, Tomcat eventually runs out of file descriptors and needs a restart to clear it. We know Lucene is the culprit because use lsof -p <java PID> and the vast majority (usually tens of thousands) of files reported are Lucene index files.
>
> I am hoping to get some tips on how this can be avoided. Is it simply the case that as time goes by, more and more descriptors are left open and no matter how high ulimit is set, you will run out? Or is there a policy of recycling that we are failing to utilise properly?
>
> I am happy to provide more information, just don't know what at this point! Please ask....
>
> Thanks in advance
>
> - Chris
>
> Chris Bamford
> Senior Development Engineer
> Scalix
> chris.bamford@scalix.com
> Tel: +44 (0)1344 381814
> www.scalix.com
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene gobbling file descriptors
Posted by Michael McCandless <lu...@mikemccandless.com>.
This is not normal. As long as you are certain you close every
IndexReader/Searcher that you opened, the number of file descriptors
should stay "contained".
Though: how many files are there in your index directory?
Mike
On Wed, Aug 26, 2009 at 9:18 AM, Chris Bamford<Ch...@scalix.com> wrote:
> Hi there,
>
> I wonder if someone can help? We have a successful Lucene app deployed on Tomcat which works well. As far as we can tell, our developers have observed all the guidelines in the Lucene FAQ, but on some of our installations, Tomcat eventually runs out of file descriptors and needs a restart to clear it. We know Lucene is the culprit because use lsof -p <java PID> and the vast majority (usually tens of thousands) of files reported are Lucene index files.
>
> I am hoping to get some tips on how this can be avoided. Is it simply the case that as time goes by, more and more descriptors are left open and no matter how high ulimit is set, you will run out? Or is there a policy of recycling that we are failing to utilise properly?
>
> I am happy to provide more information, just don't know what at this point! Please ask....
>
> Thanks in advance
>
> - Chris
>
> Chris Bamford
> Senior Development Engineer
> Scalix
> chris.bamford@scalix.com
> Tel: +44 (0)1344 381814
> www.scalix.com
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene gobbling file descriptors
Posted by Shai Erera <se...@gmail.com>.
That's strange ... how do you execute your searches - each search opens up
an IndexReader? Do you make sure to close them? Maybe those are file
descriptors of files you index?
Forgive the silly questions, but I've never seen Lucene run into
out-of-files handles ...
Shai
On Wed, Aug 26, 2009 at 4:18 PM, Chris Bamford <Ch...@scalix.com>wrote:
> Hi there,
>
> I wonder if someone can help? We have a successful Lucene app deployed on
> Tomcat which works well. As far as we can tell, our developers have
> observed all the guidelines in the Lucene FAQ, but on some of our
> installations, Tomcat eventually runs out of file descriptors and needs a
> restart to clear it. We know Lucene is the culprit because use lsof -p
> <java PID> and the vast majority (usually tens of thousands) of files
> reported are Lucene index files.
>
> I am hoping to get some tips on how this can be avoided. Is it simply the
> case that as time goes by, more and more descriptors are left open and no
> matter how high ulimit is set, you will run out? Or is there a policy of
> recycling that we are failing to utilise properly?
>
> I am happy to provide more information, just don't know what at this point!
> Please ask....
>
> Thanks in advance
>
> - Chris
>
> Chris Bamford
> Senior Development Engineer
> Scalix
> chris.bamford@scalix.com
> Tel: +44 (0)1344 381814
> www.scalix.com
>
>