You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by bart_212 <wo...@gmail.com> on 2011/03/09 20:44:32 UTC
reopen with optimize and FileNotFoundException
Hi,
I have two web applications that uses lucene 2.3.2. Both share the same
index and can write or read. Writing is synchronized based on file system to
allow only one IndexWriter to work at the moment. There can be multiple
IndexReader. IndexReader is stored and before searching, there is done
#reopen on it. IndexWriter can add/delete group of documents and can
optimize the index while searching is performed (multiple threads can
search).
I'm wondering if these operations are thread safe and don't corrupt the
index? I have observed the following exception when one IndexWriter was
adding and one IndexReader was reopened to search:
java.io.FileNotFoundException: /tmp/test/index/_7y.tis
RandomAccessFile.open(String, int) line: not available [native method]
FSDirectory$FSIndexInput$Descriptor(RandomAccessFile).(File, String) line:
212
FSDirectory$FSIndexInput$Descriptor.(File, String) line: 506
FSDirectory$FSIndexInput.(File, int) line: 536
FSDirectory.openInput(String, int) line: 445
TermInfosReader.(Directory, String, FieldInfos, int) line: 61
SegmentReader.initialize(SegmentInfo, int, boolean) line: 317
SegmentReader.get(Directory, SegmentInfo, SegmentInfos, boolean, boolean,
int, boolean) line: 262
SegmentReader.get(SegmentInfo) line: 197
MultiSegmentReader.(Directory, SegmentInfos, boolean, SegmentReader[],
int[], Map) line: 109
MultiSegmentReader.doReopen(SegmentInfos) line: 203
DirectoryIndexReader$2.doBody(String) line: 98
DirectoryIndexReader$2(SegmentInfos$FindSegmentsFile).run() line: 636
MultiSegmentReader(DirectoryIndexReader).reopen() line: 92
IndexReaderManager.reopenReader(FileProxy, IndexReader) line: 92
IndexReaderManager.getIndexReader(FileProxy) line: 76
IndexSearcherFactory.createIndexSearcher(FileProxy) line: 66
This exception is caught inside, however is it ok that the file is missing
or maybe there is some problem with usage? Please clarify.
--
View this message in context: http://lucene.472066.n3.nabble.com/reopen-with-optimize-and-FileNotFoundException-tp2656875p2656875.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: reopen with optimize and FileNotFoundException
Posted by Ian Lea <ia...@gmail.com>.
Usage sounds OK, but missing files on IndexReader.reopen definitely
doesn't sound OK.
2.3.2 is ancient and there have been many improvements since then.
I'd upgrade if possible.
You could also try losing the optimizes. On recent releases you don't
really need to use it. Not sure about 2.3.2 though.
--
Ian.
On Wed, Mar 9, 2011 at 7:44 PM, bart_212 <wo...@gmail.com> wrote:
> Hi,
> I have two web applications that uses lucene 2.3.2. Both share the same
> index and can write or read. Writing is synchronized based on file system to
> allow only one IndexWriter to work at the moment. There can be multiple
> IndexReader. IndexReader is stored and before searching, there is done
> #reopen on it. IndexWriter can add/delete group of documents and can
> optimize the index while searching is performed (multiple threads can
> search).
> I'm wondering if these operations are thread safe and don't corrupt the
> index? I have observed the following exception when one IndexWriter was
> adding and one IndexReader was reopened to search:
>
> java.io.FileNotFoundException: /tmp/test/index/_7y.tis
> RandomAccessFile.open(String, int) line: not available [native method]
> FSDirectory$FSIndexInput$Descriptor(RandomAccessFile).(File, String) line:
> 212
> FSDirectory$FSIndexInput$Descriptor.(File, String) line: 506
> FSDirectory$FSIndexInput.(File, int) line: 536
> FSDirectory.openInput(String, int) line: 445
> TermInfosReader.(Directory, String, FieldInfos, int) line: 61
> SegmentReader.initialize(SegmentInfo, int, boolean) line: 317
> SegmentReader.get(Directory, SegmentInfo, SegmentInfos, boolean, boolean,
> int, boolean) line: 262
> SegmentReader.get(SegmentInfo) line: 197
> MultiSegmentReader.(Directory, SegmentInfos, boolean, SegmentReader[],
> int[], Map) line: 109
> MultiSegmentReader.doReopen(SegmentInfos) line: 203
> DirectoryIndexReader$2.doBody(String) line: 98
> DirectoryIndexReader$2(SegmentInfos$FindSegmentsFile).run() line: 636
> MultiSegmentReader(DirectoryIndexReader).reopen() line: 92
> IndexReaderManager.reopenReader(FileProxy, IndexReader) line: 92
> IndexReaderManager.getIndexReader(FileProxy) line: 76
> IndexSearcherFactory.createIndexSearcher(FileProxy) line: 66
>
> This exception is caught inside, however is it ok that the file is missing
> or maybe there is some problem with usage? Please clarify.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/reopen-with-optimize-and-FileNotFoundException-tp2656875p2656875.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: reopen with optimize and FileNotFoundException
Posted by bart_212 <wo...@gmail.com>.
I've tried different lucene locks to use, however I always get FNFE during
huge index update while IR reopens index. To prevent two IW at the same
time, I synchronize this operation using external locking mechanism based on
atomic filesystem operation like creating directory. So just before indexing
I lock manually ,then I create IW ,do indexing, then release the lock. It
works fine, I have always only one IW during indexing. So in my case lucene
locks are irrelevant unless IR during searching needs to lock?
I've noticed that I get this exception only once, then it is caught inside
and some segment seems to be just omitted. I haven't checked if index is
healthy, however other reopens seems to succeed. BTW, it occurs when one app
is indexing and the second is searching.
Please clarify, what component (IR or IW) is responsible for creating
"*.tis" file? Sometimes it is also some "*.frq" file...
I suspected that it can be caused of merging segments by IW, however you
claim it should be fine...
What do you think?
--
View this message in context: http://lucene.472066.n3.nabble.com/reopen-with-optimize-and-FileNotFoundException-tp2655555p2662149.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: reopen with optimize and FileNotFoundException
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Wed, Mar 9, 2011 at 2:44 PM, bart_212 <wo...@gmail.com> wrote:
> Hi,
> I have two web applications that uses lucene 2.3.2. Both share the same
> index and can write or read. Writing is synchronized based on file system to
> allow only one IndexWriter to work at the moment. There can be multiple
> IndexReader. IndexReader is stored and before searching, there is done
> #reopen on it. IndexWriter can add/delete group of documents and can
> optimize the index while searching is performed (multiple threads can
> search).
> I'm wondering if these operations are thread safe and don't corrupt the
> index? I have observed the following exception when one IndexWriter was
> adding and one IndexReader was reopened to search:
This usage is [supposed to be] perfectly fine. Any ops you do with IW
while IRs are in use, IR is being opened/reopened, is fine. You can
even open a new IW with create=true while IRs are in use.
> java.io.FileNotFoundException: /tmp/test/index/_7y.tis
> RandomAccessFile.open(String, int) line: not available [native method]
> FSDirectory$FSIndexInput$Descriptor(RandomAccessFile).(File, String) line:
> 212
> FSDirectory$FSIndexInput$Descriptor.(File, String) line: 506
> FSDirectory$FSIndexInput.(File, int) line: 536
> FSDirectory.openInput(String, int) line: 445
> TermInfosReader.(Directory, String, FieldInfos, int) line: 61
> SegmentReader.initialize(SegmentInfo, int, boolean) line: 317
> SegmentReader.get(Directory, SegmentInfo, SegmentInfos, boolean, boolean,
> int, boolean) line: 262
> SegmentReader.get(SegmentInfo) line: 197
> MultiSegmentReader.(Directory, SegmentInfos, boolean, SegmentReader[],
> int[], Map) line: 109
> MultiSegmentReader.doReopen(SegmentInfos) line: 203
> DirectoryIndexReader$2.doBody(String) line: 98
> DirectoryIndexReader$2(SegmentInfos$FindSegmentsFile).run() line: 636
> MultiSegmentReader(DirectoryIndexReader).reopen() line: 92
> IndexReaderManager.reopenReader(FileProxy, IndexReader) line: 92
> IndexReaderManager.getIndexReader(FileProxy) line: 76
> IndexSearcherFactory.createIndexSearcher(FileProxy) line: 66
This is definitely not good. Is the index otherwise healthy? Like
this error is transient and later on opening a reader succeeds?
What locking are you using to ensure the two IWs don't open the index
at the same time? Just the default locking for your Directory impl?
Which Directory impl are you using...?
What filesystem is the index on? It looks like /tmp which is
presumably a local filesystem right?
--
Mike
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org