You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Paul Taylor <pa...@fastmail.fm> on 2009/04/24 17:57:00 UTC

no segments* file found: files: Error on opening index

Hi I was using a RAMDirectory and this was working fine but have now 
moved over to a  filesystem directory to preserve space, the directory 
is just initialized once
    directory = new RAMDirectory();
    directory = 
FSDirectory.getDirectory(Platform.getPlatformLicenseFolder()+ "/" + 
TAG_BROWSER_INDEX);

but I create an IndexReader as required
     IndexReader ir = IndexReader.open(directory);
and then close it when Ive finished with it.

but sometimes it is failing with
    no segments* file found: files:
interestingly no file is specified in the error message whereas other 
reporters of this message show a reference to an actual file in the message.

My code is multithreaded so Im wondering if the problem is that two 
thread have both opened an IndexReader, or one thread is using an 
IndexWriter to add a document, whilst another is trying to read. But I 
wasnt aware either of these cases was not allowed, any ideas ?

thanks Paul



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: no segments* file found: files: Error on opening index

Posted by Paul Taylor <pa...@fastmail.fm>.
Michael McCandless wrote:
> Are you sure you can't make the reader reopen block on a reindex?  Or
> skip reopen if reindex is in process?
>
> (Because that's the simplest solution).
>   
Thats what Im suggesting in principle, I just need to work out the best 
way to do it because the reader reopen has no idea whether a reindex is 
occurring athe moment, I have to introduce some concurrency code to 
achive this.
> If not, I think the next best solution is likely to allow multiple
> commit points in the index.  You'll need a custom deletion policy that
> always keeps the "last commit point that's safe to reopen" in the
> index.  Then a reindex can gleefully proceed since the last "safe to
> reopen" commit point will not be deleted.  If a reader wants to
> reopen, you ask it to reopen on that "last safe commit point".
>
> But these APIs (reopen to a commit point) are only available in 2.9
> trunk.  You can emulate this in prior releases by using separate
> indexing directories in the filesystem.
>   
I had already upgraded to 2.9, so I'll have a look at this as well. For 
me I would want it to work so that before a reindex starts it creates a 
commit point on the existing index, so a that any index readers created 
in the meantime will work on the last one

Paul


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: no segments* file found: files: Error on opening index

Posted by Michael McCandless <lu...@mikemccandless.com>.
Are you sure you can't make the reader reopen block on a reindex?  Or
skip reopen if reindex is in process?

(Because that's the simplest solution).

If not, I think the next best solution is likely to allow multiple
commit points in the index.  You'll need a custom deletion policy that
always keeps the "last commit point that's safe to reopen" in the
index.  Then a reindex can gleefully proceed since the last "safe to
reopen" commit point will not be deleted.  If a reader wants to
reopen, you ask it to reopen on that "last safe commit point".

But these APIs (reopen to a commit point) are only available in 2.9
trunk.  You can emulate this in prior releases by using separate
indexing directories in the filesystem.

Mike

On Wed, Apr 29, 2009 at 5:53 AM, Paul Taylor <pa...@fastmail.fm> wrote:
> Michael McCandless wrote:
>>
>> Lucene doesn't have anything builtin to handle this.
>>
>> It's probably best to put synchronization into your code in such a
>> case?  It's presumably also not great if your IndexReader opens an
>> empty index since searches will find no results.  Ie, you should
>> probably only reopen the IndexReader after all docs have been
>> reindexed?
>>
>> Or you could catch a failure to open the IndexReader and retry, but
>> that's not great since differentiating a true failure isn't easy.
>>
>> Mike
>>
>>
>
> I can synchronize the reindex method but I dont think that will help because
> the problem is with the readers, the trouble is that I have various threads,
> and sometimes IndexReaders are being opened in reponse to something done on
> the GUI, without being aware that a reindex is being done. I think I need to
> get any IndexReader opens to wait for any reindexing to complete using
> something in the Concurrent Package. Or alternatively I was wondering if the
> reindex could be done on a different folder, so Indexreaders opened in the
> meantime work on the old index, than once the reindexd has completed I can
> swap the directory that the reader looks at to the new one.
>
>
> Which do you think would work ?
> Paul
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: no segments* file found: files: Error on opening index

Posted by Paul Taylor <pa...@fastmail.fm>.
Michael McCandless wrote:
> Lucene doesn't have anything builtin to handle this.
>
> It's probably best to put synchronization into your code in such a
> case?  It's presumably also not great if your IndexReader opens an
> empty index since searches will find no results.  Ie, you should
> probably only reopen the IndexReader after all docs have been
> reindexed?
>
> Or you could catch a failure to open the IndexReader and retry, but
> that's not great since differentiating a true failure isn't easy.
>
> Mike
>
>   
I can synchronize the reindex method but I dont think that will help 
because the problem is with the readers, the trouble is that I have 
various threads, and sometimes IndexReaders are being opened in reponse 
to something done on the GUI, without being aware that a reindex is 
being done. I think I need to get any IndexReader opens to wait for any 
reindexing to complete using something in the Concurrent Package. Or 
alternatively I was wondering if the reindex could be done on a 
different folder, so Indexreaders opened in the meantime work on the old 
index, than once the reindexd has completed I can swap the directory 
that the reader looks at to the new one.


Which do you think would work ?
Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: no segments* file found: files: Error on opening index

Posted by Paul Taylor <pa...@fastmail.fm>.
Michael McCandless wrote:
> It's fine if one thread is changing the index (w/ IndexWriter) while
> other threads are opening IndexReaders.
>
> Though, if you have IndexWriter opening with "create=true" at the same
> time that an IndexReader is attempting to open the index, that would
> explain this.  Is it possible that's happening?
>
> The fact that you see no files listed in the exception's message means
> the directory was empty at the time IndexReader was being openened.
>
> Mike
>   
Hi Mike,

Its now occurred to me that the IndexWriter code might have decided that 
so many records needed modifying that it would be better to recreate the 
index from scratch  rather than modify it, so if an Indexreader accessed 
it at the time it would be empty. is there a way I can get the 
IndexReader to wait for the IndexWriter to complete.

Paul


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: no segments* file found: files: Error on opening index

Posted by Michael McCandless <lu...@mikemccandless.com>.
It's fine if one thread is changing the index (w/ IndexWriter) while
other threads are opening IndexReaders.

Though, if you have IndexWriter opening with "create=true" at the same
time that an IndexReader is attempting to open the index, that would
explain this.  Is it possible that's happening?

The fact that you see no files listed in the exception's message means
the directory was empty at the time IndexReader was being openened.

Mike

On Mon, Apr 27, 2009 at 6:01 AM, Paul Taylor <pa...@fastmail.fm> wrote:
> Paul Taylor wrote:
>>
>> Hi I was using a RAMDirectory and this was working fine but have now moved
>> over to a  filesystem directory to preserve space, the directory is just
>> initialized once
>>   directory = new RAMDirectory();
>>   directory =
>> FSDirectory.getDirectory(Platform.getPlatformLicenseFolder()+ "/" +
>> TAG_BROWSER_INDEX);
>>
>> but I create an IndexReader as required
>>    IndexReader ir = IndexReader.open(directory);
>> and then close it when Ive finished with it.
>>
>> but sometimes it is failing with
>>   no segments* file found: files:
>> interestingly no file is specified in the error message whereas other
>> reporters of this message show a reference to an actual file in the message.
>>
>> My code is multithreaded so I'm wondering if the problem is that two
>> thread have both opened an IndexReader, or one thread is using an
>> IndexWriter to add a document, whilst another is trying to read. But I wasnt
>> aware either of these cases was not allowed, any ideas ?
>>
>> thanks Paul
>>
> Hi, anybody have an idea about this, Im sure Im not the only one to
> encounter this.
>
> Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: no segments* file found: files: Error on opening index

Posted by Paul Taylor <pa...@fastmail.fm>.
Paul Taylor wrote:
> Hi I was using a RAMDirectory and this was working fine but have now 
> moved over to a  filesystem directory to preserve space, the directory 
> is just initialized once
>    directory = new RAMDirectory();
>    directory = 
> FSDirectory.getDirectory(Platform.getPlatformLicenseFolder()+ "/" + 
> TAG_BROWSER_INDEX);
>
> but I create an IndexReader as required
>     IndexReader ir = IndexReader.open(directory);
> and then close it when Ive finished with it.
>
> but sometimes it is failing with
>    no segments* file found: files:
> interestingly no file is specified in the error message whereas other 
> reporters of this message show a reference to an actual file in the 
> message.
>
> My code is multithreaded so I'm wondering if the problem is that two 
> thread have both opened an IndexReader, or one thread is using an 
> IndexWriter to add a document, whilst another is trying to read. But I 
> wasnt aware either of these cases was not allowed, any ideas ?
>
> thanks Paul
>
Hi, anybody have an idea about this, Im sure Im not the only one to 
encounter this.

Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org