You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Matt Magoffin <ap...@msqr.us> on 2005/11/18 08:57:58 UTC
RE: References to deleted file handles in long-running server
application
I'm updating nearly continuously (probably average about every 10
seconds). I don't explicitly close the IndexSearcher objects I create, as
I share them across threads, but do leave them to be garbage collected. I
ran into index corruption issues when I explicitly closed them, since I
don't have any code keeping track of how many threads might be using a
given IndexSearcher.
I wonder if it's worth the effort of adding that code in, or can I be
ensured that when the IndexSearcher objects are garbage collected the file
handles will be released?
-- m@
> How often are you updating your index? Are you closing your old
> IndexSearchers after switching over to the new index? You'll need to
> close the searchers in order to release the file handle. This was the
> same issue I was experiencing:
>
> http://mail-archives.apache.org/mod_mbox/lucene-java-user/200504.mbox/%3
> c200504281819640.SM01836@chicago%3e
>
> - Monsur
>
>
>
>> -----Original Message-----
>> From: Matt Magoffin [mailto:apache.org@msqr.us]
>> Sent: Thursday, November 17, 2005 11:05 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: References to deleted file handles in
>> long-running server application
>>
>>
>> I've been watching our servers today, and now there are 2500
>> "deleted" file handles open like this. Seems to be quite
>> large. Still don't know why there are so many. I'm using the
>> compound index format already to reduce the number of open files.
>>
>> -- m@
>>
>> > Hello, I use Lucene in a long-running server application on a Linux
>> > server, and the other day I got the "Too many open files"
>> exception.
>> > I've increased the number of allowed file handles, but was checking
>> > out the open file handles using "lsof", and see about 300
>> files listed
>> > like the
>> > following:
>> >
>> > java 1458 jboss 436r REG 8,2 3945 6258825
>> > /var/lucene-index/_1o6hl.cfs (deleted)
>> > java 1458 jboss 437r REG 8,2 3467 6258829
>> > /var/lucene-index/_1o6hp.cfs (deleted)
>> > java 1458 jboss 438r REG 8,2 2743 6258826
>> > /var/lucene-index/_1o6ht.cfs (deleted)
>> > java 1458 jboss 439r REG 8,2 4069 6258817
>> > /var/lucene-index/_1o6hx.cfs (deleted)
>> > java 1458 jboss 440r REG 8,2 4098 6258830
>> > /var/lucene-index/_1o6i1.cfs (deleted)
>> > java 1458 jboss 441r REG 8,2 1817 6258821
>> > /var/lucene-index/_1o6i3.cfs (deleted)
>> >
>> > I haven't been monitoring this long enough to tell if the number
>> > simply continues to grow over time or levels off at some point.
>> >
>> > I was wondering if this behavior to be expected? When will the
>> > application release these deleted file handles (if ever)? Perhaps
>> > during garbage collection? Or might I be handling the index
>> > incorrectly in some way?
>> >
>> > Any thoughts appreciated.
>> > -- m@
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: References to deleted file handles in long-running server application
Posted by Paul Smith <ps...@aconex.com>.
I have found exactly the same problem. You should consider adding an
'expired" IndexSearcher to some sort of Buffer that closes the
IndexSearcher after some default timeout. I have not found in
practice that allowing the GC to do the work does quite what you
think it might do (particularly on Linux).
It could be to do with the different constructors of IndexSearcher
(String vs IndexReader), but my advise is to mark the IndexSearcher
as ready to be closed by placing it in a buffer, and after some
sensible timeout where you would consider any existing queries that
may have been running at the time of 'close' should be cancelled by
force closing the IndexSearcher. If you were to do this every X
minutes, then any running Searcher that was still being used after
this time is probably a candidate for an interrupt. (In a web
search, the user has probably navigated away and given up on the long
running search anyway).
Paul Smith
On 18/11/2005, at 6:57 PM, Matt Magoffin wrote:
> I'm updating nearly continuously (probably average about every 10
> seconds). I don't explicitly close the IndexSearcher objects I
> create, as
> I share them across threads, but do leave them to be garbage
> collected. I
> ran into index corruption issues when I explicitly closed them,
> since I
> don't have any code keeping track of how many threads might be using a
> given IndexSearcher.
>
> I wonder if it's worth the effort of adding that code in, or can I be
> ensured that when the IndexSearcher objects are garbage collected
> the file
> handles will be released?
>
> -- m@
>
>> How often are you updating your index? Are you closing your old
>> IndexSearchers after switching over to the new index? You'll need to
>> close the searchers in order to release the file handle. This was
>> the
>> same issue I was experiencing:
>>
>> http://mail-archives.apache.org/mod_mbox/lucene-java-user/
>> 200504.mbox/%3
>> c200504281819640.SM01836@chicago%3e
>>
>> - Monsur
>>
>>
>>
>>> -----Original Message-----
>>> From: Matt Magoffin [mailto:apache.org@msqr.us]
>>> Sent: Thursday, November 17, 2005 11:05 PM
>>> To: java-user@lucene.apache.org
>>> Subject: Re: References to deleted file handles in
>>> long-running server application
>>>
>>>
>>> I've been watching our servers today, and now there are 2500
>>> "deleted" file handles open like this. Seems to be quite
>>> large. Still don't know why there are so many. I'm using the
>>> compound index format already to reduce the number of open files.
>>>
>>> -- m@
>>>
>>>> Hello, I use Lucene in a long-running server application on a Linux
>>>> server, and the other day I got the "Too many open files"
>>> exception.
>>>> I've increased the number of allowed file handles, but was checking
>>>> out the open file handles using "lsof", and see about 300
>>> files listed
>>>> like the
>>>> following:
>>>>
>>>> java 1458 jboss 436r REG 8,2 3945 6258825
>>>> /var/lucene-index/_1o6hl.cfs (deleted)
>>>> java 1458 jboss 437r REG 8,2 3467 6258829
>>>> /var/lucene-index/_1o6hp.cfs (deleted)
>>>> java 1458 jboss 438r REG 8,2 2743 6258826
>>>> /var/lucene-index/_1o6ht.cfs (deleted)
>>>> java 1458 jboss 439r REG 8,2 4069 6258817
>>>> /var/lucene-index/_1o6hx.cfs (deleted)
>>>> java 1458 jboss 440r REG 8,2 4098 6258830
>>>> /var/lucene-index/_1o6i1.cfs (deleted)
>>>> java 1458 jboss 441r REG 8,2 1817 6258821
>>>> /var/lucene-index/_1o6i3.cfs (deleted)
>>>>
>>>> I haven't been monitoring this long enough to tell if the number
>>>> simply continues to grow over time or levels off at some point.
>>>>
>>>> I was wondering if this behavior to be expected? When will the
>>>> application release these deleted file handles (if ever)? Perhaps
>>>> during garbage collection? Or might I be handling the index
>>>> incorrectly in some way?
>>>>
>>>> Any thoughts appreciated.
>>>> -- m@
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>