You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by karl wettin <ka...@snigel.dnsalias.net> on 2004/03/04 21:36:10 UTC
Too many open files
I'm getting this exception, and I can't explain it. It only occurs when
calling my method that retrieves the content from my index. I get the
same exception in 1.2 as in 1.3-final, been searching the web all over
the place but can't find anything else than the same problem described,
no solution.
It takes some 50-100 calls to the method until the exception in thrown.
Don't I close the IndexReader correct? Is there anything else I need to
close?
Greatful for any hints.
----> exception snippet
java.io.FileNotFoundException: /home/kalle/projects/snigel/egdelon_data/lucene/_2ur.fdx (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:200)
at org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.java:389)
at org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:418)
at org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:291)
at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:80)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:139)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:120)
at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:122)
at org.apache.lucene.store.Lock$With.run(Lock.java:148)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:111)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:99)
at org.apache.lucene.search.IndexSearcher.<init>(IndexSearcher.java:75)
at se.snigel.egdelon.document.indexer.LuceneIndexer.getSegmentContents(LuceneIndexer.java:48)
row 48 >> s = new IndexSearcher(Egdelon.getSetting("lucene.path"));
----> full method
public String getSegmentContents(Segment segment)
throws IOException {
File path = new File(Egdelon.getSetting("lucene.path"));
if (!IndexReader.indexExists(path))
return null;
Searcher s = null;
try {
Analyzer a = new StandardAnalyzer();
Query q = QueryParser.parse(String.valueOf(segment.getController().getPrimaryKey().getPK()),
"segment.pk", a);
s = new IndexSearcher(Egdelon.getSetting("lucene.path"));
Hits hits = s.search(q);
if (hits.length() == 1)
return hits.doc(0).getField("contents").stringValue();
s.close();
return null;
} catch (ParseException e) {
e.printStackTrace();
if (s != null)
s.close();
return null;
}
}
--
karl
--
kalle
--
kalle
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Too many open files
Posted by karl wettin <ka...@snigel.dnsalias.net>.
On Thu, 04 Mar 2004 15:32:53 -0700
Dmitry Serebrennikov <dm...@earthlink.net> wrote:
> karl wettin wrote:
>
> >I'm getting this exception, and I can't explain it. It only occurs
> >when calling my method that retrieves the content from my index. I
> >get the same exception in 1.2 as in 1.3-final, been searching the web
> >all over the place but can't find anything else than the same problem
> >described, no solution.
> >
> >
> >
> >It takes some 50-100 calls to the method until the exception in
> >thrown. Don't I close the IndexReader correct? Is there anything else
> >I need to close?
> >
> >Greatful for any hints.
> You can tell how many file handles you are allowed to open by your OS
> (looks like some flavor of Unix from the paths that you have
> included). One way to reduce the number of files Lucene opens is to
> use compound indexes (where each index segment uses a single file).
> Look for this flag on IndexWriter object. You will have to optimize
> existing indexes to convert them.
My file limit was set to "unlimited" when I got the problem, so I
started using .setUseCompoundFile(true) in order to resolve my problem.
Even though I only add new data (no delete) to the index, I started
getting the exception "Lock obtain timed out" after the change above.
Passing on "-DdisableLuceneLocks=true" to java "solved" this problem,
but it does not feel right. I only access Lucene from one JVM, and I
make sure I never write and delete at the same time. Do I have anyhting
to worry about?
--
karl
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Too many open files
Posted by Dmitry Serebrennikov <dm...@earthlink.net>.
You can tell how many file handles you are allowed to open by your OS
(looks like some flavor of Unix from the paths that you have included).
One way to reduce the number of files Lucene opens is to use compound
indexes (where each index segment uses a single file). Look for this
flag on IndexWriter object. You will have to optimize existing indexes
to convert them.
If you need to know how to tell how many files you are allowed to open
on Unix, the command is ulimit. Try ulimit -Ha and look for
"nofiles(descriptors)" - this works on Solaris.
There are usually two limits for each variable - hard and soft. Soft
limit you can control with the ulimit command before you start your
process. You can rase it as far as the hard limit. The hard limit is
given in the /etc/system file (again for Solaris) and is generally not
changeable except by root. You will find many references on the web that
say not to increase this parameter in /etc/system beyound 2000 or so,
but in reality any significant J2EE installation on Solaris will have
8000 to 9000 setting.
Good luck.
Dmitry.
karl wettin wrote:
>I'm getting this exception, and I can't explain it. It only occurs when
>calling my method that retrieves the content from my index. I get the
>same exception in 1.2 as in 1.3-final, been searching the web all over
>the place but can't find anything else than the same problem described,
>no solution.
>
>
>
>It takes some 50-100 calls to the method until the exception in thrown.
>Don't I close the IndexReader correct? Is there anything else I need to
>close?
>
>Greatful for any hints.
>
>
>----> exception snippet
>
>java.io.FileNotFoundException: /home/kalle/projects/snigel/egdelon_data/lucene/_2ur.fdx (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:200)
> at org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.java:389)
> at org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:418)
> at org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:291)
> at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:80)
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:139)
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:120)
> at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:122)
> at org.apache.lucene.store.Lock$With.run(Lock.java:148)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:111)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:99)
> at org.apache.lucene.search.IndexSearcher.<init>(IndexSearcher.java:75)
> at se.snigel.egdelon.document.indexer.LuceneIndexer.getSegmentContents(LuceneIndexer.java:48)
>
>row 48 >> s = new IndexSearcher(Egdelon.getSetting("lucene.path"));
>
>----> full method
>
> public String getSegmentContents(Segment segment)
> throws IOException {
> File path = new File(Egdelon.getSetting("lucene.path"));
>
> if (!IndexReader.indexExists(path))
> return null;
>
> Searcher s = null;
>
> try {
> Analyzer a = new StandardAnalyzer();
> Query q = QueryParser.parse(String.valueOf(segment.getController().getPrimaryKey().getPK()),
> "segment.pk", a);
> s = new IndexSearcher(Egdelon.getSetting("lucene.path"));
>
> Hits hits = s.search(q);
>
> if (hits.length() == 1)
> return hits.doc(0).getField("contents").stringValue();
>
> s.close();
>
> return null;
> } catch (ParseException e) {
> e.printStackTrace();
>
> if (s != null)
> s.close();
>
> return null;
> }
> }
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org