You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by James Ricci <ja...@riccinursery.com> on 2002/05/23 21:05:29 UTC
Too many file?
Anyone,
I am trying to evaluate Lucene for use in our company. I tried the simple
test below. Before it finished creating the index, I got the exception
exception below. Examining the directory where the index was created, there
were more than 10,000 files created. The error I got was, not surprisingly,
too many files open. What have I done wrong here? Should the directory
contain a single file for each document indexed? Is there a way to limit
this?
James
static public void createLuceneIndex()
{
try
{
IndexWriter iw = new IndexWriter("c:\\temp\\lutest", new
StandardAnalyzer(), true);
for (int i = 0; i < 100000; i++)
{
Document df = new Document();
for (int j = 0; j < 3; j++)
{
Field f = new Field("FIELD_" + (i + 1),
Integer.toString(i), true, true, true);
df.add(f);
}
iw.addDocument(df);
System.out.print('.');
}
}
catch(Throwable t)
{
t.printStackTrace();
}
}
java.io.FileNotFoundException: C:\temp\lutest\_2kk.f27 (Too many
open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
at
java.io.RandomAccessFile.<init>(RandomAccessFile.java:143)
at
org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.java:254
)
at
org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:262)
at
org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:211)
at
org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:259)
at
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:114)
at
org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:304)
at
org.apache.lucene.index.IndexWriter.maybeMergeSegments(IndexWriter.java:283)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:185)
at
ch.abacus.lib.db.net.test.TestMain.createLuceneIndex(TestMain.java:147)
at ch.abacus.lib.db.net.test.TestMain.main(TestMain.java:61)
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Too many file?
Posted by "Nader S. Henein" <ns...@bayt.net>.
Optimize the index after finishing the indexing process, this consolidates
the
number of files in the index, the reason why you're getting this error is
that
when you don't optimize the index, you get hundreds of files in the index
and the
OS runs out of file handles before opening them all for searching or
updating.
Please keep me posted if you have any other questions.
Nader Henein
-----Original Message-----
From: James Ricci [mailto:james@riccinursery.com]
Sent: Thursday, May 23, 2002 11:05 PM
To: 'lucene-user@jakarta.apache.org'
Cc: Claudio Hintermann; Mario Castillo
Subject: Too many file?
Anyone,
I am trying to evaluate Lucene for use in our company. I tried the simple
test below. Before it finished creating the index, I got the exception
exception below. Examining the directory where the index was created, there
were more than 10,000 files created. The error I got was, not surprisingly,
too many files open. What have I done wrong here? Should the directory
contain a single file for each document indexed? Is there a way to limit
this?
James
static public void createLuceneIndex()
{
try
{
IndexWriter iw = new IndexWriter("c:\\temp\\lutest", new
StandardAnalyzer(), true);
for (int i = 0; i < 100000; i++)
{
Document df = new Document();
for (int j = 0; j < 3; j++)
{
Field f = new Field("FIELD_" + (i + 1),
Integer.toString(i), true, true, true);
df.add(f);
}
iw.addDocument(df);
System.out.print('.');
}
}
catch(Throwable t)
{
t.printStackTrace();
}
}
java.io.FileNotFoundException: C:\temp\lutest\_2kk.f27 (Too many
open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
at
java.io.RandomAccessFile.<init>(RandomAccessFile.java:143)
at
org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.java:254
)
at
org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:262)
at
org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:211)
at
org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:259)
at
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:114)
at
org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:304)
at
org.apache.lucene.index.IndexWriter.maybeMergeSegments(IndexWriter.java:283)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:185)
at
ch.abacus.lib.db.net.test.TestMain.createLuceneIndex(TestMain.java:147)
at ch.abacus.lib.db.net.test.TestMain.main(TestMain.java:61)
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Too many file?
Posted by Peter Carlson <ca...@bookandhammer.com>.
This seems a little strange.
I index over 100K documents on a windows machine and I don't get this
problem.
One way to solve it though is to optimize periodically. Just call
Iw.optimize(); every 50000 records or so. This will consolidate all the
individual files into a single file (or set of files).
Hope that helps.
--Peter
On 5/23/02 12:05 PM, "James Ricci" <ja...@riccinursery.com> wrote:
> Anyone,
>
> I am trying to evaluate Lucene for use in our company. I tried the simple
> test below. Before it finished creating the index, I got the exception
> exception below. Examining the directory where the index was created, there
> were more than 10,000 files created. The error I got was, not surprisingly,
> too many files open. What have I done wrong here? Should the directory
> contain a single file for each document indexed? Is there a way to limit
> this?
>
> James
>
> static public void createLuceneIndex()
> {
> try
> {
> IndexWriter iw = new IndexWriter("c:\\temp\\lutest", new
> StandardAnalyzer(), true);
>
> for (int i = 0; i < 100000; i++)
> {
> Document df = new Document();
>
> for (int j = 0; j < 3; j++)
> {
> Field f = new Field("FIELD_" + (i + 1),
> Integer.toString(i), true, true, true);
> df.add(f);
> }
>
> iw.addDocument(df);
> System.out.print('.');
> }
> }
> catch(Throwable t)
> {
> t.printStackTrace();
> }
> }
>
>
> java.io.FileNotFoundException: C:\temp\lutest\_2kk.f27 (Too many
> open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
>
> at
> java.io.RandomAccessFile.<init>(RandomAccessFile.java:143)
> at
> org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.java:254
> )
> at
> org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:262)
> at
> org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:211)
> at
> org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:259)
> at
> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:114)
> at
> org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:304)
>
> at
> org.apache.lucene.index.IndexWriter.maybeMergeSegments(IndexWriter.java:283)
>
> at
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:185)
> at
> ch.abacus.lib.db.net.test.TestMain.createLuceneIndex(TestMain.java:147)
>
> at ch.abacus.lib.db.net.test.TestMain.main(TestMain.java:61)
>
>
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>