You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by vasu shah <va...@yahoo.com> on 2006/07/25 15:11:26 UTC
IndexReader / IndexWriter Synchronization
Hi,
I went through the IndexModifier class. It says that - Although an instance of this class can be used from more than one thread, you will not get the best performance. You might want to use IndexReader and IndexWriter directly for that (but you will need to care about synchronization yourself then).
If I use IndexReader and IndexWriter class for inserts/updates, then I need to handle the threading issues myself. Is there any other class (even in nightly build) that I can use without having to take care of synchronization.
Any help/pointers would be appreciated.
Thanks,
-Vasu
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: IndexReader / IndexWriter Synchronization
Posted by vasu shah <va...@yahoo.com>.
Thanks Mike. Your explanation was really helpful.
I would use the IndexModifier class till the new IndexWriter class comes up.
Thanks once again.
-Vasu
Michael McCandless <lu...@mikemccandless.com> wrote:
> I am not very good at threading. So I was looking if there is any api class (even in nightly builds) on top of the IndexReader/IndexWriter that takes care of concurrency rules.
This is exactly why IndexModifier was created (so you wouldn't have to
worry about the details of closing/opening IndexReader/IndexWriter).
There is also additional locking under the hood of those two classes
that is abstracting away concurrency details so you don't have to worry
about them.
> Every developer must be facing this problem of concurrency while programming.
Definitely. This is indeed one of the common sources of confusion about
Lucene: the fact that you have to carefully open/close these two classes
to do common operations is odd; the fact that you need an IndexReader to
do deletes is "unexpected" (one would expect IndexWriter to do that).
There is work underway to enable IndexWriter to [efficiently] handle
deletes -- there's been alot of lively discussion on lucene-dev &
several iterations on possible patches -- see here:
http://issues.apache.org/jira/browse/LUCENE-565
But this is very much "in process" now. I don't know of anything else
in the nightly builds (since 2.0) that address these issues yet.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------
Yahoo! Messenger with Voice. Make PC-to-Phone Calls to the US (and 30+ countries) for 2ยข/min or less.
Re: IndexReader / IndexWriter Synchronization
Posted by Michael McCandless <lu...@mikemccandless.com>.
> I am not very good at threading. So I was looking if there is any api class (even in nightly builds) on top of the IndexReader/IndexWriter that takes care of concurrency rules.
This is exactly why IndexModifier was created (so you wouldn't have to
worry about the details of closing/opening IndexReader/IndexWriter).
There is also additional locking under the hood of those two classes
that is abstracting away concurrency details so you don't have to worry
about them.
> Every developer must be facing this problem of concurrency while programming.
Definitely. This is indeed one of the common sources of confusion about
Lucene: the fact that you have to carefully open/close these two classes
to do common operations is odd; the fact that you need an IndexReader to
do deletes is "unexpected" (one would expect IndexWriter to do that).
There is work underway to enable IndexWriter to [efficiently] handle
deletes -- there's been alot of lively discussion on lucene-dev &
several iterations on possible patches -- see here:
http://issues.apache.org/jira/browse/LUCENE-565
But this is very much "in process" now. I don't know of anything else
in the nightly builds (since 2.0) that address these issues yet.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexReader / IndexWriter Synchronization
Posted by vasu shah <va...@yahoo.com>.
Thanks Mike for the reply. I will look into Lucene in Action.
I am not very good at threading. So I was looking if there is any api class (even in nightly builds) on top of the IndexReader/IndexWriter that takes care of concurrency rules.
Every developer must be facing this problem of concurrency while programming.
Thanks again.
-Vasu
Michael McCandless <lu...@mikemccandless.com> wrote:
> If I use IndexReader and IndexWriter class for inserts/updates, then I need to handle the threading issues myself. Is there any other class (even in nightly build) that I can use without having to take care of synchronization.
All this means is your code must ensure only one "writer" (IndexWriter
adding docs or IndexReader removing docs) is open at a time. If you
fail to do so, then the second "writer" you try to create will throw a
"Lock obtain timed out" IOException.
Otherwise these classes are fully thread safe.
The performance caveat in IndexModifier (that you should batch your
deletes and batch your adds) still applies, because you will need to
open/close an IndexReader for the former and an IndexWriter for the
latter.
The Lucene in Action book does a great job describing these
concurrency rules.
Also, you should first test IndexModifier to be really sure its
performance is not sufficient for your needs.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------
See the all-new, redesigned Yahoo.com. Check it out.
Re: IndexReader / IndexWriter Synchronization
Posted by Michael McCandless <lu...@mikemccandless.com>.
> If I use IndexReader and IndexWriter class for inserts/updates, then I need to handle the threading issues myself. Is there any other class (even in nightly build) that I can use without having to take care of synchronization.
All this means is your code must ensure only one "writer" (IndexWriter
adding docs or IndexReader removing docs) is open at a time. If you
fail to do so, then the second "writer" you try to create will throw a
"Lock obtain timed out" IOException.
Otherwise these classes are fully thread safe.
The performance caveat in IndexModifier (that you should batch your
deletes and batch your adds) still applies, because you will need to
open/close an IndexReader for the former and an IndexWriter for the
latter.
The Lucene in Action book does a great job describing these
concurrency rules.
Also, you should first test IndexModifier to be really sure its
performance is not sufficient for your needs.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org