You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Dmitry Serebrennikov <dm...@earthlink.net> on 2004/10/03 01:37:31 UTC
Re: Too many Open Files + lucene 1.4.1 + Linux O/s
Karthik N S wrote:
>Hi Luceners,
>
>
>Apologies.........
>
>
>Other day was Trying to Search using the "Luceneweb" version
>with Lucene1-4-1.zip and O/s = Linux, J2SDK version "1.4.2_03-b02"
>
>With Roughly around 500 Documents (715116 kb ) Indexed using
>Lucene1.4-final.jar and writer.setUseCompoundFile(true);
>
>
Here are a couple of possibilities:
- the setUseCompoundFile(true) will only apply to indexes created (or
optimized) after the option is set.
All pre-existing indexes will still be in the multi-file format.
- number of documents does not directly impact the number of files
needed by Lucene. If the index is
really in a compound file format (see above), and is optimized, you
will need a fixed number of file handles.
Even if the index is in a multi-file format, the number of files
needed depends on the number of indexed *fields* in the index (not
documents).
- do you get the error on the first and every search or only once in a
while? Perhaps where there are lots of
concurrent users? Perhaps after you've done X searchers?
- check your OS-level setting for the number of open files. This is
shell/system-dependent somewhat, but
"ulimit -a" should get you started. The number of open files should
be large enough to allow for all files
and sockets that your application needs to open. In a typical
server-side Java app setting this value should
be around 8000. Defaults are much smaller, so unless you have changed
this, this may be the answer.
- look into "lsof" utility. It can display all file handles in use by a
given process. This is a good tool to
troubleshoot "too many open files" issues.
Good luck.
Dmitry.
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Too many Open Files + lucene 1.4.1 + Linux O/s
Posted by Karthik N S <ka...@controlnet.co.in>.
Hi
Apologies for the Long wait..........
My Linux system on ulimit -a respresent
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1983
virtual memory (kbytes, -v) unlimited
The Problem of "Too many Open Files" happens on every 2nd Search being
done
I think as u say "open files (-n) 1024" should be
increased...........
More Advises is Accepted greatefully....
Thx in advance
-----Original Message-----
From: Dmitry Serebrennikov [mailto:dmitrys@earthlink.net]
Sent: Sunday, October 03, 2004 5:08 AM
To: Lucene Users List
Subject: Re: Too many Open Files + lucene 1.4.1 + Linux O/s
Karthik N S wrote:
>Hi Luceners,
>
>
>Apologies.........
>
>
>Other day was Trying to Search using the "Luceneweb" version
>with Lucene1-4-1.zip and O/s = Linux, J2SDK version "1.4.2_03-b02"
>
>With Roughly around 500 Documents (715116 kb ) Indexed using
>Lucene1.4-final.jar and writer.setUseCompoundFile(true);
>
>
Here are a couple of possibilities:
- the setUseCompoundFile(true) will only apply to indexes created (or
optimized) after the option is set.
All pre-existing indexes will still be in the multi-file format.
- number of documents does not directly impact the number of files
needed by Lucene. If the index is
really in a compound file format (see above), and is optimized, you
will need a fixed number of file handles.
Even if the index is in a multi-file format, the number of files
needed depends on the number of indexed *fields* in the index (not
documents).
- do you get the error on the first and every search or only once in a
while? Perhaps where there are lots of
concurrent users? Perhaps after you've done X searchers?
- check your OS-level setting for the number of open files. This is
shell/system-dependent somewhat, but
"ulimit -a" should get you started. The number of open files should
be large enough to allow for all files
and sockets that your application needs to open. In a typical
server-side Java app setting this value should
be around 8000. Defaults are much smaller, so unless you have changed
this, this may be the answer.
- look into "lsof" utility. It can display all file handles in use by a
given process. This is a good tool to
troubleshoot "too many open files" issues.
Good luck.
Dmitry.
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org