You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Chris Bamford <Ch...@scalix.com> on 2009/09/01 12:32:12 UTC
RE: Lucene gobbling file descriptors
Hi Erick,
>>Note that for search speed reasons, you really, really want to share your
>>readers and NOT open/close for every request.
I have often wondered about this - I hope you can help me understand it better in the context of our app, which is an email client:
When one of our users receives email we index and store it so he (and only he) can search on it. This means a separate index per user. On large customer sites this can mean hundreds/thousands of indexes. Sharing readers seems counter-intuitive, unless I am missing something. What we do instead is that once a user performs a search, we keep his IndexReader open in case he searches again. At present, we have no expiry on this mechanism, so they stay open indefinitely. I'm a bit hazy on the underlying details but we have observed that the number of open fds jumps by around 10 each time a new user performs a search. What would be a good strategy for managing this in your opinon? Does it really make sense to keep the IndexReader open? Would performance suffer that much if we did an open/close for each search? Or would it perhaps be better to close open readers after a period of inactivity?
Thanks for any wisdom / thoughts/ ideas.
- Chris
----- Original Message -----
From: Erick Erickson <er...@gmail.com>
Sent: Thu, 27/8/2009 4:49pm
To: java-user@lucene.apache.org
Subject: Re: Lucene gobbling file descriptors
Note that for search speed reasons, you really, really want to share your
readers and NOT open/close for every request.
FWIW
Erick
On Thu, Aug 27, 2009 at 9:10 AM, Chris Bamford <Ch...@scalix.com>wrote:
> I'm glad its not normal. That means we can fix it! I will conduct a
> review of IndexReader/Searcher open/close ops.
>
> Thanks!
>
> Chris
>
> ----- Original Message -----
> From: Michael McCandless <lu...@mikemccandless.com>
> Sent: Wed, 26/8/2009 2:26pm
> To: java-user@lucene.apache.org
> Subject: Re: Lucene gobbling file descriptors
>
> This is not normal. As long as you are certain you close every
> IndexReader/Searcher that you opened, the number of file descriptors
> should stay "contained".
>
> Though: how many files are there in your index directory?
>
> Mike
>
> On Wed, Aug 26, 2009 at 9:18 AM, Chris Bamford<Ch...@scalix.com>
> wrote:
> > Hi there,
> >
> > I wonder if someone can help? We have a successful Lucene app deployed
> on Tomcat which works well. As far as we can tell, our developers have
> observed all the guidelines in the Lucene FAQ, but on some of our
> installations, Tomcat eventually runs out of file descriptors and needs a
> restart to clear it. We know Lucene is the culprit because use lsof -p
> <java PID> and the vast majority (usually tens of thousands) of files
> reported are Lucene index files.
> >
> > I am hoping to get some tips on how this can be avoided. Is it simply
> the case that as time goes by, more and more descriptors are left open and
> no matter how high ulimit is set, you will run out? Or is there a policy of
> recycling that we are failing to utilise properly?
> >
> > I am happy to provide more information, just don't know what at this
> point! Please ask....
> >
> > Thanks in advance
> >
> > - Chris
> >
> > Chris Bamford
> > Senior Development Engineer
> > Scalix
> > chris.bamford@scalix.com
> > Tel: +44 (0)1344 381814
> > www.scalix.com
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Question about IndexCommit
Posted by Chris Hostetter <ho...@fucit.org>.
: Subject: Question about IndexCommit
: In-Reply-To: <9a...@mail.gmail.com>
http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/Thread_hijacking
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Question about IndexCommit
Posted by Ted Stockwell <em...@yahoo.com>.
That's excellent.
Thanks very much for the explanations....
----- Original Message ----
> From: Michael McCandless <lu...@mikemccandless.com>
> To: java-user@lucene.apache.org
> Sent: Tuesday, September 1, 2009 8:26:45 AM
> Subject: Re: Question about IndexCommit
>
> Further, when IndexWriter writes new .del files, it's always to a new
> (next generation) filename, so that the old .del file remains present.
> This means if a fresh IndexReader is opened, it will load the old
> .del file, and still not see any of IndexWriter's pending changes.
>
> Mike
>
> On Tue, Sep 1, 2009 at 9:04 AM, Shai Erera wrote:
> > If I'm not mistaken, IndexReader reads the .del file into memory, and
> > therefore subsequent updates to it won't be visible to it.
> >
> > Shai
> >
> > On Tue, Sep 1, 2009 at 3:54 PM, Ted Stockwell wrote:
> >
> >> Hi All,
> >>
> >> I am interested in using Lucene to index RDF (Resource Description Format)
> >> data.
> >> Ultimately I want to create a transactional interface to the data with
> >> proper transaction isolation.
> >> Therefore I am trying to educate myself on the details of index readers and
> >> writers, I am using v2.9rc2.
> >>
> >> One thing I have noticed is that it seems that Lucene is designed with the
> >> intent that changes to the index by a writer should not be visible to index
> >> readers until the writer commits its changes.
> >> However, one thing that I have noticed in the code is that when documents
> >> are deleted by a writer the writers change the .del (deleted documents)
> >> files in the segment in which the document exists.
> >> Thus, it appears that document deletions may become visible to readers
> >> *before* a write actually commits its changes.
> >>
> >> Am I correct about how document deletions work?
> >> Since I'm new to Lucene I'm wondering if I'm missing something...
> >>
> >>
> >> Thanks,
> >> ted stockwell
> >>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Question about IndexCommit
Posted by Michael McCandless <lu...@mikemccandless.com>.
Further, when IndexWriter writes new .del files, it's always to a new
(next generation) filename, so that the old .del file remains present.
This means if a fresh IndexReader is opened, it will load the old
.del file, and still not see any of IndexWriter's pending changes.
Mike
On Tue, Sep 1, 2009 at 9:04 AM, Shai Erera<se...@gmail.com> wrote:
> If I'm not mistaken, IndexReader reads the .del file into memory, and
> therefore subsequent updates to it won't be visible to it.
>
> Shai
>
> On Tue, Sep 1, 2009 at 3:54 PM, Ted Stockwell <em...@yahoo.com> wrote:
>
>> Hi All,
>>
>> I am interested in using Lucene to index RDF (Resource Description Format)
>> data.
>> Ultimately I want to create a transactional interface to the data with
>> proper transaction isolation.
>> Therefore I am trying to educate myself on the details of index readers and
>> writers, I am using v2.9rc2.
>>
>> One thing I have noticed is that it seems that Lucene is designed with the
>> intent that changes to the index by a writer should not be visible to index
>> readers until the writer commits its changes.
>> However, one thing that I have noticed in the code is that when documents
>> are deleted by a writer the writers change the .del (deleted documents)
>> files in the segment in which the document exists.
>> Thus, it appears that document deletions may become visible to readers
>> *before* a write actually commits its changes.
>>
>> Am I correct about how document deletions work?
>> Since I'm new to Lucene I'm wondering if I'm missing something...
>>
>>
>> Thanks,
>> ted stockwell
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Question about IndexCommit
Posted by Shai Erera <se...@gmail.com>.
If I'm not mistaken, IndexReader reads the .del file into memory, and
therefore subsequent updates to it won't be visible to it.
Shai
On Tue, Sep 1, 2009 at 3:54 PM, Ted Stockwell <em...@yahoo.com> wrote:
> Hi All,
>
> I am interested in using Lucene to index RDF (Resource Description Format)
> data.
> Ultimately I want to create a transactional interface to the data with
> proper transaction isolation.
> Therefore I am trying to educate myself on the details of index readers and
> writers, I am using v2.9rc2.
>
> One thing I have noticed is that it seems that Lucene is designed with the
> intent that changes to the index by a writer should not be visible to index
> readers until the writer commits its changes.
> However, one thing that I have noticed in the code is that when documents
> are deleted by a writer the writers change the .del (deleted documents)
> files in the segment in which the document exists.
> Thus, it appears that document deletions may become visible to readers
> *before* a write actually commits its changes.
>
> Am I correct about how document deletions work?
> Since I'm new to Lucene I'm wondering if I'm missing something...
>
>
> Thanks,
> ted stockwell
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Question about IndexCommit
Posted by Ted Stockwell <em...@yahoo.com>.
Hi All,
I am interested in using Lucene to index RDF (Resource Description Format) data.
Ultimately I want to create a transactional interface to the data with proper transaction isolation.
Therefore I am trying to educate myself on the details of index readers and writers, I am using v2.9rc2.
One thing I have noticed is that it seems that Lucene is designed with the intent that changes to the index by a writer should not be visible to index readers until the writer commits its changes.
However, one thing that I have noticed in the code is that when documents are deleted by a writer the writers change the .del (deleted documents) files in the segment in which the document exists.
Thus, it appears that document deletions may become visible to readers *before* a write actually commits its changes.
Am I correct about how document deletions work?
Since I'm new to Lucene I'm wondering if I'm missing something...
Thanks,
ted stockwell
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene gobbling file descriptors
Posted by Michael McCandless <lu...@mikemccandless.com>.
In this approach it's expected you'll run out of file descriptors,
when "enough" users attempt to search at the same time.
You can reduce the number of file descriptors required per IndexReader
by 1) using compound file format (it's the default for IndexWriter),
and 2) optimizing the index before opening it (though, since you have
updates trickling in, that could get costly). Yet, if enough users
try to search, you'll run out of descriptors.
If performance is OK, I think you should in fact open IndexReader, do
search, close IndexReader, per request. Or maybe reuse IndexReader
for the "biggest" indexes. This reduces your "max file descriptors
envelope". Still, you can run of descriptors with the "perfect storm"
of usage.
Also make sure you're giving the JRE the max open file descriptors
allowed by the OS.
A bigger change would be to aggregate multiple users into a single
index, and use filtering to apply the entitlements constraints. But
that's got its own set of tradeoffs... eg, scoring will be different,
respelling is dangerous (entitlements can "leak" through), it's less
"secure", etc.
Mike
On Tue, Sep 1, 2009 at 6:32 AM, Chris Bamford<Ch...@scalix.com> wrote:
> Hi Erick,
>
>>>Note that for search speed reasons, you really, really want to share your
>>>readers and NOT open/close for every request.
> I have often wondered about this - I hope you can help me understand it better in the context of our app, which is an email client:
>
> When one of our users receives email we index and store it so he (and only he) can search on it. This means a separate index per user. On large customer sites this can mean hundreds/thousands of indexes. Sharing readers seems counter-intuitive, unless I am missing something. What we do instead is that once a user performs a search, we keep his IndexReader open in case he searches again. At present, we have no expiry on this mechanism, so they stay open indefinitely. I'm a bit hazy on the underlying details but we have observed that the number of open fds jumps by around 10 each time a new user performs a search. What would be a good strategy for managing this in your opinon? Does it really make sense to keep the IndexReader open? Would performance suffer that much if we did an open/close for each search? Or would it perhaps be better to close open readers after a period of inactivity?
>
> Thanks for any wisdom / thoughts/ ideas.
>
> - Chris
>
>
>
> ----- Original Message -----
> From: Erick Erickson <er...@gmail.com>
> Sent: Thu, 27/8/2009 4:49pm
> To: java-user@lucene.apache.org
> Subject: Re: Lucene gobbling file descriptors
>
> Note that for search speed reasons, you really, really want to share your
> readers and NOT open/close for every request.
> FWIW
> Erick
>
> On Thu, Aug 27, 2009 at 9:10 AM, Chris Bamford <Ch...@scalix.com>wrote:
>
>> I'm glad its not normal. That means we can fix it! I will conduct a
>> review of IndexReader/Searcher open/close ops.
>>
>> Thanks!
>>
>> Chris
>>
>> ----- Original Message -----
>> From: Michael McCandless <lu...@mikemccandless.com>
>> Sent: Wed, 26/8/2009 2:26pm
>> To: java-user@lucene.apache.org
>> Subject: Re: Lucene gobbling file descriptors
>>
>> This is not normal. As long as you are certain you close every
>> IndexReader/Searcher that you opened, the number of file descriptors
>> should stay "contained".
>>
>> Though: how many files are there in your index directory?
>>
>> Mike
>>
>> On Wed, Aug 26, 2009 at 9:18 AM, Chris Bamford<Ch...@scalix.com>
>> wrote:
>> > Hi there,
>> >
>> > I wonder if someone can help? We have a successful Lucene app deployed
>> on Tomcat which works well. As far as we can tell, our developers have
>> observed all the guidelines in the Lucene FAQ, but on some of our
>> installations, Tomcat eventually runs out of file descriptors and needs a
>> restart to clear it. We know Lucene is the culprit because use lsof -p
>> <java PID> and the vast majority (usually tens of thousands) of files
>> reported are Lucene index files.
>> >
>> > I am hoping to get some tips on how this can be avoided. Is it simply
>> the case that as time goes by, more and more descriptors are left open and
>> no matter how high ulimit is set, you will run out? Or is there a policy of
>> recycling that we are failing to utilise properly?
>> >
>> > I am happy to provide more information, just don't know what at this
>> point! Please ask....
>> >
>> > Thanks in advance
>> >
>> > - Chris
>> >
>> > Chris Bamford
>> > Senior Development Engineer
>> > Scalix
>> > chris.bamford@scalix.com
>> > Tel: +44 (0)1344 381814
>> > www.scalix.com
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene gobbling file descriptors
Posted by Erick Erickson <er...@gmail.com>.
Late reply, but do what Michael said. I didn't understand that you hadso
many indexes, but opening/closing makes sense to me now, reusing
them wouldn't be very useful. Although I *can* imagine a "keep each open
for 5 minute" sort of rule being useful on the assumption that a user might
search several ways in a short interval, but I sure wouldn't go there unless
and until I had a performance issue to deal with....
In fact, I'd leave all the complex stuff (like, perhaps, queueing requests
after there were X outstanding ones being serviced) until I could
demonstrate
a need.
Although I would put in a bit of logging showing the max number of
concurrent requests just to try to get ahead of the "perfect storm"
scenario Michael mentioned.
Erick
On Tue, Sep 1, 2009 at 6:32 AM, Chris Bamford <Ch...@scalix.com>wrote:
> Hi Erick,
>
> >>Note that for search speed reasons, you really, really want to share your
> >>readers and NOT open/close for every request.
> I have often wondered about this - I hope you can help me understand it
> better in the context of our app, which is an email client:
>
> When one of our users receives email we index and store it so he (and only
> he) can search on it. This means a separate index per user. On large
> customer sites this can mean hundreds/thousands of indexes. Sharing readers
> seems counter-intuitive, unless I am missing something. What we do instead
> is that once a user performs a search, we keep his IndexReader open in case
> he searches again. At present, we have no expiry on this mechanism, so they
> stay open indefinitely. I'm a bit hazy on the underlying details but we
> have observed that the number of open fds jumps by around 10 each time a new
> user performs a search. What would be a good strategy for managing this in
> your opinon? Does it really make sense to keep the IndexReader open? Would
> performance suffer that much if we did an open/close for each search? Or
> would it perhaps be better to close open readers after a period of
> inactivity?
>
> Thanks for any wisdom / thoughts/ ideas.
>
> - Chris
>
>
>
> ----- Original Message -----
> From: Erick Erickson <er...@gmail.com>
> Sent: Thu, 27/8/2009 4:49pm
> To: java-user@lucene.apache.org
> Subject: Re: Lucene gobbling file descriptors
>
> Note that for search speed reasons, you really, really want to share your
> readers and NOT open/close for every request.
> FWIW
> Erick
>
> On Thu, Aug 27, 2009 at 9:10 AM, Chris Bamford <Chris.Bamford@scalix.com
> >wrote:
>
> > I'm glad its not normal. That means we can fix it! I will conduct a
> > review of IndexReader/Searcher open/close ops.
> >
> > Thanks!
> >
> > Chris
> >
> > ----- Original Message -----
> > From: Michael McCandless <lu...@mikemccandless.com>
> > Sent: Wed, 26/8/2009 2:26pm
> > To: java-user@lucene.apache.org
> > Subject: Re: Lucene gobbling file descriptors
> >
> > This is not normal. As long as you are certain you close every
> > IndexReader/Searcher that you opened, the number of file descriptors
> > should stay "contained".
> >
> > Though: how many files are there in your index directory?
> >
> > Mike
> >
> > On Wed, Aug 26, 2009 at 9:18 AM, Chris Bamford<Ch...@scalix.com>
> > wrote:
> > > Hi there,
> > >
> > > I wonder if someone can help? We have a successful Lucene app deployed
> > on Tomcat which works well. As far as we can tell, our developers have
> > observed all the guidelines in the Lucene FAQ, but on some of our
> > installations, Tomcat eventually runs out of file descriptors and needs a
> > restart to clear it. We know Lucene is the culprit because use lsof -p
> > <java PID> and the vast majority (usually tens of thousands) of files
> > reported are Lucene index files.
> > >
> > > I am hoping to get some tips on how this can be avoided. Is it simply
> > the case that as time goes by, more and more descriptors are left open
> and
> > no matter how high ulimit is set, you will run out? Or is there a policy
> of
> > recycling that we are failing to utilise properly?
> > >
> > > I am happy to provide more information, just don't know what at this
> > point! Please ask....
> > >
> > > Thanks in advance
> > >
> > > - Chris
> > >
> > > Chris Bamford
> > > Senior Development Engineer
> > > Scalix
> > > chris.bamford@scalix.com
> > > Tel: +44 (0)1344 381814
> > > www.scalix.com
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>