You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Nikhil Goel <ni...@gmail.com> on 2006/03/13 17:18:38 UTC
IndexSearcher and IndexWriter in conjuction
Hi,
Can someone please explain how does IndexSearcher and IndexWriter works in
conjuction. As far as i know after reading all the posts in newgroup, it
seems everything works fine if we have one IndexWriter thread and multiple
IndexSearcher thread. But my doubt here is, looking at IndexSearcher class,
it seems it first reads the segments file and then one by one go to the
respective .fnm files in the index...hence can occur a case, where it has
read segments file but in the meantime IndexWriter thread has updated the
index and the corresponding .fnm file doesnt exist in the index and this
will give us the error .fnm doesn't exist and we will get an IOException.
Am I missing something in making sure that there can be multiple
IndexSearcher thread and one IndexWriter Thread and still everything works
fine.
thanks
-Nikhil
Re: IndexSearcher and IndexWriter in conjuction
Posted by Chris Hostetter <ho...@fucit.org>.
: The trick is that once segment files are written, they are never
: modified (except for the "segments" file itself). New documents are
: added to new segments, not existing segments. When segments are
: merged, a new bigger segment is created. This way, the view of the
: index for a specific IndexReader will never change.
one thing to add here - when a segment is removed (beecuase it's contents
have been merged into a new segment) the file is deleted -- but any
existing reader/searcher that needs using that segment will still have it
open, so even though the file is no longer present in the directory, the
data will still exist and be available to the serachers that are using it.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher and IndexWriter in conjuction
Posted by Yonik Seeley <ys...@gmail.com>.
On 3/13/06, Nikhil Goel <ni...@gmail.com> wrote:
> Can someone please explain how does IndexSearcher and IndexWriter works in
> conjuction.
The trick is that once segment files are written, they are never
modified (except for the "segments" file itself). New documents are
added to new segments, not existing segments. When segments are
merged, a new bigger segment is created. This way, the view of the
index for a specific IndexReader will never change.
-Yonik
http://incubator.apache.org/solr Solr, The Open Source Lucene Search Server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexSearcher and IndexWriter in conjuction
Posted by Nikhil Goel <ni...@gmail.com>.
Hi Patrick,
thanks for writing back but my question is:- do we really need to write
something new to achieve what I want to achieve. By going thru Lucene
Tutorials, i dont think there is a need to do such a thing:-
http://blog.danbartels.com/archive/2004/09/09/186.aspx
Indexing and searching are not only thread safe, but process safe. (Multiple
processes can work with the Lucene index files at the same time).
Generally the index searcher is thread safe and the index writer is thread
safe. The query parser is *not* thread safe (so each thread using the index
should have its own query parser).
The index writer is thread safe, so you can update the index while people
are searching it. However, you then have to make sure that the threads with
open index searchers close them and open new ones, to get the newly updated
data.
Thanks
-Nikhil
On 3/13/06, Patrick Kimber <ma...@gmail.com> wrote:
>
> Hi Nikhil
> We are using the index accessor contribution. For more information see:
>
> http://www.nabble.com/Fwd%3A-Contribution%3A-LuceneIndexAccessor-t17416.html#a47049
> This should help you to co-ordinate the IndexSearcher and IndexWriter.
> Patrick
>
> On 13/03/06, Nikhil Goel <ni...@gmail.com> wrote:
> > Hi,
> >
> > Can someone please explain how does IndexSearcher and IndexWriter works
> in
> > conjuction. As far as i know after reading all the posts in newgroup, it
> > seems everything works fine if we have one IndexWriter thread and
> multiple
> > IndexSearcher thread. But my doubt here is, looking at IndexSearcher
> class,
> > it seems it first reads the segments file and then one by one go to the
> > respective .fnm files in the index...hence can occur a case, where it
> has
> > read segments file but in the meantime IndexWriter thread has updated
> the
> > index and the corresponding .fnm file doesnt exist in the index and this
> > will give us the error .fnm doesn't exist and we will get an
> IOException.
> >
> > Am I missing something in making sure that there can be multiple
> > IndexSearcher thread and one IndexWriter Thread and still everything
> works
> > fine.
> >
> > thanks
> > -Nikhil
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: IndexSearcher and IndexWriter in conjuction
Posted by Patrick Kimber <ma...@gmail.com>.
Hi Nikhil
We are using the index accessor contribution. For more information see:
http://www.nabble.com/Fwd%3A-Contribution%3A-LuceneIndexAccessor-t17416.html#a47049
This should help you to co-ordinate the IndexSearcher and IndexWriter.
Patrick
On 13/03/06, Nikhil Goel <ni...@gmail.com> wrote:
> Hi,
>
> Can someone please explain how does IndexSearcher and IndexWriter works in
> conjuction. As far as i know after reading all the posts in newgroup, it
> seems everything works fine if we have one IndexWriter thread and multiple
> IndexSearcher thread. But my doubt here is, looking at IndexSearcher class,
> it seems it first reads the segments file and then one by one go to the
> respective .fnm files in the index...hence can occur a case, where it has
> read segments file but in the meantime IndexWriter thread has updated the
> index and the corresponding .fnm file doesnt exist in the index and this
> will give us the error .fnm doesn't exist and we will get an IOException.
>
> Am I missing something in making sure that there can be multiple
> IndexSearcher thread and one IndexWriter Thread and still everything works
> fine.
>
> thanks
> -Nikhil
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org