File Handle Resource Usage

[an...@teleord.co.uk]

Hi,

I have been using Lucene in a production enviroment now for about a year,
using version 1.1.    I  had moved across to the 1.2 release, but as the
result of this the number of file handles that were needed increased
dramatically.

A quick look at some trace from the OS indicated that when the Index was
opened, all the files in the directory were opened.   I suspect that this
is done to improve performance for various searches etc.

Does anyone have any suggestions as a way around. I am already aware of the
obvious things:

Increasing the No Available to the Program.
Reducing the no of files created ( mergeFactor etc)
Optimising the index more often.

Andrew

"if the will of the majority of the citizens were ever to become
ascertainable, no body of representatives but public opinion would not only
reign but govern."



--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>

Re: File Handle Resource Usage

[Hamish Carpenter <ha...@catalyst.net.nz>]

We had a similar problem (under linux) but have had no issues during 
testing on windows.

We did two things to solve the problem:
1) Increase the file limit for the program to 4096.  The alleviated but 
did not completely solve the problem.  Under stress testing we would 
still fall short of the maximum number of file handles.
2) Implemented a cached IndexSearcher.  This is using Peter Halacsy's 
IndexSearcherCache slightly modified to be a singleton and return cached 
searchers for a given directory. This code was found in the list archives.

More optimisation and a lower marge factor will help things but the 
files open will skyrocket when you have simultaneous searches each using 
its own IndexSearcher.

HTH

Hamish

andrewp@teleord.co.uk wrote:

>Hi,
>
>I have been using Lucene in a production enviroment now for about a year,
>using version 1.1.    I  had moved across to the 1.2 release, but as the
>result of this the number of file handles that were needed increased
>dramatically.
>
>A quick look at some trace from the OS indicated that when the Index was
>opened, all the files in the directory were opened.   I suspect that this
>is done to improve performance for various searches etc.
>
>Does anyone have any suggestions as a way around. I am already aware of the
>obvious things:
>
>Increasing the No Available to the Program.
>Reducing the no of files created ( mergeFactor etc)
>Optimising the index more often.
>
>Andrew
>



--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>