You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Paul Taylor <pa...@fastmail.fm> on 2009/04/24 17:57:00 UTC
no segments* file found: files: Error on opening index
Hi I was using a RAMDirectory and this was working fine but have now
moved over to a filesystem directory to preserve space, the directory
is just initialized once
directory = new RAMDirectory();
directory =
FSDirectory.getDirectory(Platform.getPlatformLicenseFolder()+ "/" +
TAG_BROWSER_INDEX);
but I create an IndexReader as required
IndexReader ir = IndexReader.open(directory);
and then close it when Ive finished with it.
but sometimes it is failing with
no segments* file found: files:
interestingly no file is specified in the error message whereas other
reporters of this message show a reference to an actual file in the message.
My code is multithreaded so Im wondering if the problem is that two
thread have both opened an IndexReader, or one thread is using an
IndexWriter to add a document, whilst another is trying to read. But I
wasnt aware either of these cases was not allowed, any ideas ?
thanks Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: no segments* file found: files: Error on opening index
Posted by Paul Taylor <pa...@fastmail.fm>.
Michael McCandless wrote:
> Are you sure you can't make the reader reopen block on a reindex? Or
> skip reopen if reindex is in process?
>
> (Because that's the simplest solution).
>
Thats what Im suggesting in principle, I just need to work out the best
way to do it because the reader reopen has no idea whether a reindex is
occurring athe moment, I have to introduce some concurrency code to
achive this.
> If not, I think the next best solution is likely to allow multiple
> commit points in the index. You'll need a custom deletion policy that
> always keeps the "last commit point that's safe to reopen" in the
> index. Then a reindex can gleefully proceed since the last "safe to
> reopen" commit point will not be deleted. If a reader wants to
> reopen, you ask it to reopen on that "last safe commit point".
>
> But these APIs (reopen to a commit point) are only available in 2.9
> trunk. You can emulate this in prior releases by using separate
> indexing directories in the filesystem.
>
I had already upgraded to 2.9, so I'll have a look at this as well. For
me I would want it to work so that before a reindex starts it creates a
commit point on the existing index, so a that any index readers created
in the meantime will work on the last one
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: no segments* file found: files: Error on opening index
Posted by Michael McCandless <lu...@mikemccandless.com>.
Are you sure you can't make the reader reopen block on a reindex? Or
skip reopen if reindex is in process?
(Because that's the simplest solution).
If not, I think the next best solution is likely to allow multiple
commit points in the index. You'll need a custom deletion policy that
always keeps the "last commit point that's safe to reopen" in the
index. Then a reindex can gleefully proceed since the last "safe to
reopen" commit point will not be deleted. If a reader wants to
reopen, you ask it to reopen on that "last safe commit point".
But these APIs (reopen to a commit point) are only available in 2.9
trunk. You can emulate this in prior releases by using separate
indexing directories in the filesystem.
Mike
On Wed, Apr 29, 2009 at 5:53 AM, Paul Taylor <pa...@fastmail.fm> wrote:
> Michael McCandless wrote:
>>
>> Lucene doesn't have anything builtin to handle this.
>>
>> It's probably best to put synchronization into your code in such a
>> case? It's presumably also not great if your IndexReader opens an
>> empty index since searches will find no results. Ie, you should
>> probably only reopen the IndexReader after all docs have been
>> reindexed?
>>
>> Or you could catch a failure to open the IndexReader and retry, but
>> that's not great since differentiating a true failure isn't easy.
>>
>> Mike
>>
>>
>
> I can synchronize the reindex method but I dont think that will help because
> the problem is with the readers, the trouble is that I have various threads,
> and sometimes IndexReaders are being opened in reponse to something done on
> the GUI, without being aware that a reindex is being done. I think I need to
> get any IndexReader opens to wait for any reindexing to complete using
> something in the Concurrent Package. Or alternatively I was wondering if the
> reindex could be done on a different folder, so Indexreaders opened in the
> meantime work on the old index, than once the reindexd has completed I can
> swap the directory that the reader looks at to the new one.
>
>
> Which do you think would work ?
> Paul
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: no segments* file found: files: Error on opening index
Posted by Paul Taylor <pa...@fastmail.fm>.
Michael McCandless wrote:
> Lucene doesn't have anything builtin to handle this.
>
> It's probably best to put synchronization into your code in such a
> case? It's presumably also not great if your IndexReader opens an
> empty index since searches will find no results. Ie, you should
> probably only reopen the IndexReader after all docs have been
> reindexed?
>
> Or you could catch a failure to open the IndexReader and retry, but
> that's not great since differentiating a true failure isn't easy.
>
> Mike
>
>
I can synchronize the reindex method but I dont think that will help
because the problem is with the readers, the trouble is that I have
various threads, and sometimes IndexReaders are being opened in reponse
to something done on the GUI, without being aware that a reindex is
being done. I think I need to get any IndexReader opens to wait for any
reindexing to complete using something in the Concurrent Package. Or
alternatively I was wondering if the reindex could be done on a
different folder, so Indexreaders opened in the meantime work on the old
index, than once the reindexd has completed I can swap the directory
that the reader looks at to the new one.
Which do you think would work ?
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: no segments* file found: files: Error on opening index
Posted by Paul Taylor <pa...@fastmail.fm>.
Michael McCandless wrote:
> It's fine if one thread is changing the index (w/ IndexWriter) while
> other threads are opening IndexReaders.
>
> Though, if you have IndexWriter opening with "create=true" at the same
> time that an IndexReader is attempting to open the index, that would
> explain this. Is it possible that's happening?
>
> The fact that you see no files listed in the exception's message means
> the directory was empty at the time IndexReader was being openened.
>
> Mike
>
Hi Mike,
Its now occurred to me that the IndexWriter code might have decided that
so many records needed modifying that it would be better to recreate the
index from scratch rather than modify it, so if an Indexreader accessed
it at the time it would be empty. is there a way I can get the
IndexReader to wait for the IndexWriter to complete.
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: no segments* file found: files: Error on opening index
Posted by Michael McCandless <lu...@mikemccandless.com>.
It's fine if one thread is changing the index (w/ IndexWriter) while
other threads are opening IndexReaders.
Though, if you have IndexWriter opening with "create=true" at the same
time that an IndexReader is attempting to open the index, that would
explain this. Is it possible that's happening?
The fact that you see no files listed in the exception's message means
the directory was empty at the time IndexReader was being openened.
Mike
On Mon, Apr 27, 2009 at 6:01 AM, Paul Taylor <pa...@fastmail.fm> wrote:
> Paul Taylor wrote:
>>
>> Hi I was using a RAMDirectory and this was working fine but have now moved
>> over to a filesystem directory to preserve space, the directory is just
>> initialized once
>> directory = new RAMDirectory();
>> directory =
>> FSDirectory.getDirectory(Platform.getPlatformLicenseFolder()+ "/" +
>> TAG_BROWSER_INDEX);
>>
>> but I create an IndexReader as required
>> IndexReader ir = IndexReader.open(directory);
>> and then close it when Ive finished with it.
>>
>> but sometimes it is failing with
>> no segments* file found: files:
>> interestingly no file is specified in the error message whereas other
>> reporters of this message show a reference to an actual file in the message.
>>
>> My code is multithreaded so I'm wondering if the problem is that two
>> thread have both opened an IndexReader, or one thread is using an
>> IndexWriter to add a document, whilst another is trying to read. But I wasnt
>> aware either of these cases was not allowed, any ideas ?
>>
>> thanks Paul
>>
> Hi, anybody have an idea about this, Im sure Im not the only one to
> encounter this.
>
> Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: no segments* file found: files: Error on opening index
Posted by Paul Taylor <pa...@fastmail.fm>.
Paul Taylor wrote:
> Hi I was using a RAMDirectory and this was working fine but have now
> moved over to a filesystem directory to preserve space, the directory
> is just initialized once
> directory = new RAMDirectory();
> directory =
> FSDirectory.getDirectory(Platform.getPlatformLicenseFolder()+ "/" +
> TAG_BROWSER_INDEX);
>
> but I create an IndexReader as required
> IndexReader ir = IndexReader.open(directory);
> and then close it when Ive finished with it.
>
> but sometimes it is failing with
> no segments* file found: files:
> interestingly no file is specified in the error message whereas other
> reporters of this message show a reference to an actual file in the
> message.
>
> My code is multithreaded so I'm wondering if the problem is that two
> thread have both opened an IndexReader, or one thread is using an
> IndexWriter to add a document, whilst another is trying to read. But I
> wasnt aware either of these cases was not allowed, any ideas ?
>
> thanks Paul
>
Hi, anybody have an idea about this, Im sure Im not the only one to
encounter this.
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org