You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jamie <ja...@stimulussoft.com> on 2010/10/01 07:41:13 UTC
Re: File Handle Leaks During Lucene 3.0.2 Merge
Hi Mike
I managed to get hold of a copy of your book through Safari Books.
Quite an impressive online reading system they have there! I integrated
your SearchManager class into our code, but I am still seeing file
handles marked deleted in the index directory. I am running the
following command on Linux:
sudo watch -n 0 "lsof | grep /var/index | grep deleted | wc -l"
Every 0.1s: lsof | grep /var/index | grep deleted |... Fri Oct 1
09:37:36 2010
54
The deleted file handles fluctuate up and down. 54 -> 102 -> 64 -> 32,
etc. They seem stable though. Is this to be expected when using NRT search?
I am pretty certain that all Searchers are released at the end of
every search. I double checked it at least twenty times.
Jamie
On 2010/09/30 11:56 PM, Michael McCandless wrote:
> On Thu, Sep 30, 2010 at 5:59 AM, Jamie<ja...@stimulussoft.com> wrote:
>> Hi Michael / Uwe
>>
>>> It's good to cache the reader, but, finalize would worry me too since
>>> you have no control over when GC gets around to calling it... you risk
>>> tying up resources for longer than necessary.
>> I did it this way, as I didn't want to over complicate the code by
>> introducing mechanisms to track the number of search threads using a shared
>> indexreader. Admittedly, its not a very clean solution but in my case it
>> does work. Is there a particular technique for knowing when to a close a
>> reader when there are multiple search threads using that reader? Should I
>> keep some kind of counter and override the close method of the reader such
>> that the underlying reader is only closed when everyone's done with it?
> See Uwe's response (or SearcherManager).
>
>>> IndexWriter has a reader pool, internally, where it holds open
>>> SegmentReaders for the still-live segments in the index. This is used
>>> by IndexReader.reopen to share open SegmentReaders.
>>>
>>> But the open files should correspond only to segments still "live" in
>>> the index. After segments are merged away, these readers are dropped.
>>> Is this what you are seeing?
>>>
>> I dont fully understand your explanation/question. When I run lsof, I am
>> seeing the following:
>>
>> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyr.cfs
>> (deleted)
>> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyp.cfs
>> (deleted)
>>
>> I assume these are left by the OS after the merge operation tried to delete
>> old segments. The OS is unable to delete the files. I think its because our
>> new code never closes the indexwriter, but rather uses the
>> indexwriter.commit() method to apply the changes. Is this correct?
> Ahh I see they are deleted but held open... hmmm.
>
> Though this is also what you'd see if there were still a reader open.
> Are you certain all readers were closed (finalized) when you ran lsof?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: File Handle Leaks During Lucene 3.0.2 Merge
Posted by Thomas Rewig <tr...@mufin.com>.
Hello,
please excuse that I hijack this old thread but I have the same problem
with the deleted file handles, so I think this is the right place for.
I also integrated the searchManager in our Code and see the file handles
fluctuate up and down. At first glance the situation seems stable but
the count of the deleted file handles never drops to zero and I think
the deleted file handles are constantly rising. About +50 handles stay
per day by refreshing the searchers every 2 minutes and committing after
10 minutes. I think some deleted handles remain open after the
committing, but I could not figure out why. The max value of deleted
handles until now was about 250, because the Index will be changed after
a few days. Jamie could you achieve that all deleted handles have been
removed? Do you have some information how to repair this or is that an
expected behavior and the amount of the deleted file handles will be
stable - but first at a amount of 500 or so.
Thanks in advance
Thomas
>> I integrated your
>> SearchManager class into our code, but I am still seeing file handles marked
>> deleted in the index directory. I am running the following command on Linux:
>>
>> sudo watch -n 0 "lsof | grep /var/index | grep deleted | wc -l"
>>
>> Every 0.1s: lsof | grep /var/index | grep deleted |... Fri Oct 1 09:37:36
>> 2010
>>
>> 54
>>
>> The deleted file handles fluctuate up and down. 54 -> 102 -> 64 -> 32, etc.
>> They seem stable though. Is this to be expected when using NRT search?
>>
>> I am pretty certain that all Searchers are released at the end of every
>> search. I double checked it at least twenty times.
> SearcherManager always keeps one IndexReader open (the current "live"
> one), so it's expected that this reader is holding open references to
> deleted files if in fact IndexWriter has completed merges since the
> reader was opened. Though.. I would expect you to sometimes see 0
> deleted files in your lsof...
>
> Maybe try this: 1) pause your indexing, 2) reopen the reader (call
> SearcherManager.reopen), and then 3) run lsof again, at which point
> you should see no deleted files still held open.
>
> But note that to truly pause your indexing, you should switch
> (temporarily, for this test) your IndexWriter to the
> SerialMergeScheduler, and then ensure you stop calling IndexWriter ops
> while you run steps 2 and 3.
>
> Mike
>
Re: File Handle Leaks During Lucene 3.0.2 Merge
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Oct 1, 2010 at 1:41 AM, Jamie <ja...@stimulussoft.com> wrote:
> I managed to get hold of a copy of your book through Safari Books. Quite an
> impressive online reading system they have there!
Neat! I had never heard of Safari Books until now, but it looks cool.
> I integrated your
> SearchManager class into our code, but I am still seeing file handles marked
> deleted in the index directory. I am running the following command on Linux:
>
> sudo watch -n 0 "lsof | grep /var/index | grep deleted | wc -l"
>
> Every 0.1s: lsof | grep /var/index | grep deleted |... Fri Oct 1 09:37:36
> 2010
>
> 54
>
> The deleted file handles fluctuate up and down. 54 -> 102 -> 64 -> 32, etc.
> They seem stable though. Is this to be expected when using NRT search?
>
> I am pretty certain that all Searchers are released at the end of every
> search. I double checked it at least twenty times.
SearcherManager always keeps one IndexReader open (the current "live"
one), so it's expected that this reader is holding open references to
deleted files if in fact IndexWriter has completed merges since the
reader was opened. Though.. I would expect you to sometimes see 0
deleted files in your lsof...
Maybe try this: 1) pause your indexing, 2) reopen the reader (call
SearcherManager.reopen), and then 3) run lsof again, at which point
you should see no deleted files still held open.
But note that to truly pause your indexing, you should switch
(temporarily, for this test) your IndexWriter to the
SerialMergeScheduler, and then ensure you stop calling IndexWriter ops
while you run steps 2 and 3.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: File Handle Leaks During Lucene 3.0.2 Merge
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Jamie,
YES, ist expected for the reasons described above (segments are still
referenced by the open IndexReaders, but files were already deleted by
IndexWriter). The approx. number of open, but already deleted files should
be approx. stable.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Jamie [mailto:jamie@stimulussoft.com]
> Sent: Friday, October 01, 2010 7:41 AM
> To: java-user@lucene.apache.org
> Subject: Re: File Handle Leaks During Lucene 3.0.2 Merge
>
> Hi Mike
>
> I managed to get hold of a copy of your book through Safari Books.
> Quite an impressive online reading system they have there! I integrated
your
> SearchManager class into our code, but I am still seeing file handles
marked
> deleted in the index directory. I am running the following command on
Linux:
>
> sudo watch -n 0 "lsof | grep /var/index | grep deleted | wc -l"
>
> Every 0.1s: lsof | grep /var/index | grep deleted |... Fri Oct 1
> 09:37:36 2010
>
> 54
>
> The deleted file handles fluctuate up and down. 54 -> 102 -> 64 -> 32,
etc. They
> seem stable though. Is this to be expected when using NRT search?
>
> I am pretty certain that all Searchers are released at the end of every
search.
> I double checked it at least twenty times.
>
> Jamie
>
>
>
> On 2010/09/30 11:56 PM, Michael McCandless wrote:
> > On Thu, Sep 30, 2010 at 5:59 AM, Jamie<ja...@stimulussoft.com> wrote:
> >> Hi Michael / Uwe
> >>
> >>> It's good to cache the reader, but, finalize would worry me too
> >>> since you have no control over when GC gets around to calling it...
> >>> you risk tying up resources for longer than necessary.
> >> I did it this way, as I didn't want to over complicate the code by
> >> introducing mechanisms to track the number of search threads using a
> >> shared indexreader. Admittedly, its not a very clean solution but in
> >> my case it does work. Is there a particular technique for knowing
> >> when to a close a reader when there are multiple search threads using
> >> that reader? Should I keep some kind of counter and override the
> >> close method of the reader such that the underlying reader is only
closed
> when everyone's done with it?
> > See Uwe's response (or SearcherManager).
> >
> >>> IndexWriter has a reader pool, internally, where it holds open
> >>> SegmentReaders for the still-live segments in the index. This is
> >>> used by IndexReader.reopen to share open SegmentReaders.
> >>>
> >>> But the open files should correspond only to segments still "live"
> >>> in the index. After segments are merged away, these readers are
dropped.
> >>> Is this what you are seeing?
> >>>
> >> I dont fully understand your explanation/question. When I run lsof, I
> >> am seeing the following:
> >>
> >> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyr.cf
> >> s
> >> (deleted)
> >> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyp.cf
> >> s
> >> (deleted)
> >>
> >> I assume these are left by the OS after the merge operation tried to
> >> delete old segments. The OS is unable to delete the files. I think
> >> its because our new code never closes the indexwriter, but rather
> >> uses the
> >> indexwriter.commit() method to apply the changes. Is this correct?
> > Ahh I see they are deleted but held open... hmmm.
> >
> > Though this is also what you'd see if there were still a reader open.
> > Are you certain all readers were closed (finalized) when you ran lsof?
> >
> > Mike
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org