You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Michael McCandless <lu...@mikemccandless.com> on 2009/12/31 13:45:06 UTC
Re: Lucene 2.9: IOException from IndexReader.reopen() - Real time
search
It sounds like you may be running out of file descriptors -- how many
segments are in your index?
The reopen logic looks correct (you are closing the old reader). Is
there anything else that may be holding files open?
Have you changed any of IW's settings, eg mergeFactor?
Mike
On Wed, Dec 30, 2009 at 3:21 PM, Kumaravel Kandasami
<ku...@gmail.com> wrote:
> I am getting IOException when I am doing a "Real-time" search, i.e. I am
> creating a Index using the Index Writer and also opening the Index using
> Index Reader (writer.getReader()) to make sure the document does not exist
> prior adding to the Index file.
>
> The code works perfect fine multiple time indexed across couple of GB data,
> however, on a TB it fails with the exception listed below.
> Any suggestions appreciated. Thanks. Kumar
>
> Code Snippet:
>
> IndexReader reader2 = reader.reopen();//get
>
> if (reader2 != reader) {
>
> reader.close();
> reader = reader2;
> searcher = new IndexSearcher(reader);
> reader2 = null;
> }
>
>
> Exception:
>
> Exception in thread "main" java.io.IOException: directory
> '/Users/Kumar/Dev/Cust/index/values' exists and is a directory, but cannot
> be listed: list() returned null
> at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516)
> at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
> at
> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307)
> at
> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300)
> at
> org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:470)
> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
> at
> org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:384)
> at
> org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:348)
>
> Kumar _/|\_
> www.saisk.com
> kumar@saisk.com
> "making a profound difference with knowledge and creativity..."
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 2.9: IOException from IndexReader.reopen() - Real time
search
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Dec 31, 2009 at 12:34 PM, Kumaravel Kandasami
<ku...@gmail.com> wrote:
> Identified the problem.
>
> reader.close() was not getting called in a specific logic flow.
Phew :) Thanks for bringing closure.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 2.9: IOException from IndexReader.reopen() - Real time
search
Posted by Kumaravel Kandasami <ku...@gmail.com>.
Identified the problem.
reader.close() was not getting called in a specific logic flow.
Thank You.
Kumar _/|\_
www.saisk.com
kumar@saisk.com
"making a profound difference with knowledge and creativity..."
On Thu, Dec 31, 2009 at 11:11 AM, Kumaravel Kandasami <
kumaravel.kandasami@gmail.com> wrote:
> Thanks Mike.
>
> I think it is something to do with the merge factor.
> Modified the code to do optimize in the finally block the following error
> message was thrown.
>
> Code Snippet:
> nameWriter.optimize(); // errors here
> nameWriter.close();
> valueWriter.optimize(); //I am using multiple writers
> simultaneously. Writing to different directory.
> valueWriter.close()
>
>
> Index Directory Files:
> _1o.cfs
> _1p.cfs
> _1q.cfs
> _1r.cfs
> _1s.cfs
> _1t.fnm
> segments_1
> segments.gen
> write.lock
>
>
> Exception in thread "main" java.io.IOException: background merge hit
> exception: _1o:c55 _1p:c1 _1q:c1 _1r:c1 _1s:c1 into _1t [optimize]
> [mergeDocStores]
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2854)
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2789)
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2759)
> at
> com.hli.vis.index.VisualSoftIndexWriter.commitChanges(VSIndexWriter.java:46)
> at com.hli.vis.index.Main.main(Main.java:39)
> Caused by: java.io.FileNotFoundException:
> /Users/Kumar/Dev/Cust/index/names/_1t.fdx (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:186)
> at
> org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:76)
> at
> org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:82)
> at
> org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:334)
> at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:153)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5029)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
> Java Result: 1
>
>
>
> Kumar _/|\_
> www.saisk.com
> kumar@saisk.com
> "making a profound difference with knowledge and creativity..."
>
>
> On Thu, Dec 31, 2009 at 6:45 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> It sounds like you may be running out of file descriptors -- how many
>> segments are in your index?
>>
>> The reopen logic looks correct (you are closing the old reader). Is
>> there anything else that may be holding files open?
>>
>> Have you changed any of IW's settings, eg mergeFactor?
>>
>> Mike
>>
>> On Wed, Dec 30, 2009 at 3:21 PM, Kumaravel Kandasami
>> <ku...@gmail.com> wrote:
>> > I am getting IOException when I am doing a "Real-time" search, i.e. I am
>> > creating a Index using the Index Writer and also opening the Index using
>> > Index Reader (writer.getReader()) to make sure the document does not
>> exist
>> > prior adding to the Index file.
>> >
>> > The code works perfect fine multiple time indexed across couple of GB
>> data,
>> > however, on a TB it fails with the exception listed below.
>> > Any suggestions appreciated. Thanks. Kumar
>> >
>> > Code Snippet:
>> >
>> > IndexReader reader2 = reader.reopen();//get
>> >
>> > if (reader2 != reader) {
>> >
>> > reader.close();
>> > reader = reader2;
>> > searcher = new IndexSearcher(reader);
>> > reader2 = null;
>> > }
>> >
>> >
>> > Exception:
>> >
>> > Exception in thread "main" java.io.IOException: directory
>> > '/Users/Kumar/Dev/Cust/index/values' exists and is a directory, but
>> cannot
>> > be listed: list() returned null
>> > at
>> org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516)
>> > at
>> org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
>> > at
>> >
>> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307)
>> > at
>> >
>> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300)
>> > at
>> > org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
>> > at
>> org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
>> > at
>> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:470)
>> > at
>> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
>> > at
>> >
>> org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:384)
>> > at
>> > org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:348)
>> >
>> > Kumar _/|\_
>> > www.saisk.com
>> > kumar@saisk.com
>> > "making a profound difference with knowledge and creativity..."
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: Lucene 2.9: IOException from IndexReader.reopen() - Real time
search
Posted by Kumaravel Kandasami <ku...@gmail.com>.
Thanks Mike.
I think it is something to do with the merge factor.
Modified the code to do optimize in the finally block the following error
message was thrown.
Code Snippet:
nameWriter.optimize(); // errors here
nameWriter.close();
valueWriter.optimize(); //I am using multiple writers
simultaneously. Writing to different directory.
valueWriter.close()
Index Directory Files:
_1o.cfs
_1p.cfs
_1q.cfs
_1r.cfs
_1s.cfs
_1t.fnm
segments_1
segments.gen
write.lock
Exception in thread "main" java.io.IOException: background merge hit
exception: _1o:c55 _1p:c1 _1q:c1 _1r:c1 _1s:c1 into _1t [optimize]
[mergeDocStores]
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2854)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2789)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2759)
at
com.hli.vis.index.VisualSoftIndexWriter.commitChanges(VSIndexWriter.java:46)
at com.hli.vis.index.Main.main(Main.java:39)
Caused by: java.io.FileNotFoundException:
/Users/Kumar/Dev/Cust/index/names/_1t.fdx (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:186)
at
org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:76)
at org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:82)
at
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:334)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:153)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5029)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
Java Result: 1
Kumar _/|\_
www.saisk.com
kumar@saisk.com
"making a profound difference with knowledge and creativity..."
On Thu, Dec 31, 2009 at 6:45 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> It sounds like you may be running out of file descriptors -- how many
> segments are in your index?
>
> The reopen logic looks correct (you are closing the old reader). Is
> there anything else that may be holding files open?
>
> Have you changed any of IW's settings, eg mergeFactor?
>
> Mike
>
> On Wed, Dec 30, 2009 at 3:21 PM, Kumaravel Kandasami
> <ku...@gmail.com> wrote:
> > I am getting IOException when I am doing a "Real-time" search, i.e. I am
> > creating a Index using the Index Writer and also opening the Index using
> > Index Reader (writer.getReader()) to make sure the document does not
> exist
> > prior adding to the Index file.
> >
> > The code works perfect fine multiple time indexed across couple of GB
> data,
> > however, on a TB it fails with the exception listed below.
> > Any suggestions appreciated. Thanks. Kumar
> >
> > Code Snippet:
> >
> > IndexReader reader2 = reader.reopen();//get
> >
> > if (reader2 != reader) {
> >
> > reader.close();
> > reader = reader2;
> > searcher = new IndexSearcher(reader);
> > reader2 = null;
> > }
> >
> >
> > Exception:
> >
> > Exception in thread "main" java.io.IOException: directory
> > '/Users/Kumar/Dev/Cust/index/values' exists and is a directory, but
> cannot
> > be listed: list() returned null
> > at
> org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516)
> > at
> org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
> > at
> >
> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307)
> > at
> >
> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300)
> > at
> > org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
> > at
> org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
> > at
> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:470)
> > at
> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
> > at
> >
> org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:384)
> > at
> > org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:348)
> >
> > Kumar _/|\_
> > www.saisk.com
> > kumar@saisk.com
> > "making a profound difference with knowledge and creativity..."
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>