You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by jchang <jc...@gmail.com> on 2010/01/13 00:27:12 UTC
Lucene 2.9.0 Near Real Time Indexing and lock timeouts
Hello,
I am using Lucene 2.4.0 and am getting
org.apache.lucene.store.LockObtainFailedException's when I have a backed up
queue of items to index (with multiple concurrent writers). Of course, if I
throttle all my writer threads to 1, I don't get the exception, but I'm
hoping to write faster than that. Also, when I have multiple index writer
threads, it's not that much faster anyhow, I assume because the bottle neck
is on one the index file.
Will 2.9.0 help? I've read about the near real time indexing. Will this
allow me to have:
1) Multiple threads indexing items all at the same time, achieving
appreciably faster throughput than only one thread?
2) And avoid getting org.apache.lucene.store.LockObtainFailedException's?
3) Have this all near real time?
And generally, if anybody has any advice on high-throughput indexing with
Lucene and what kind of numbers I can acheive, I'd welcome the feedback.
Thanks,
John
--
View this message in context: http://old.nabble.com/Lucene-2.9.0-Near-Real-Time-Indexing-and-lock-timeouts-tp27136743p27136743.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Lucene 2.9.0 Near Real Time Indexing and lock timeouts
Posted by Sanne Grinovero <sa...@gmail.com>.
A common error I see is that people assume the IndexWriter to be not
threadsafe, and open several different instances.
You should use just one IndexWriter, keep it open and flush
periodically (not commit at each add operation), and read the Lucene
wiki pages about the IndexWriter settings like ramBufferSize. That why
there's only one lock, no contention from different threads.
There's an explanation of the fastest design I could get here:
http://in.relation.to/Bloggers/HibernateSearch32FastIndexRebuild
It's describing the procedure used by Hibernate Search for rebuilding
the Lucene index from an Hibernate mapped database.
While I recommend reading for newcomers, I'd also appreciate feedback
and comments from Lucene experts and developers :-)
Regards,
Sanne
2010/1/14 Michael McCandless <lu...@mikemccandless.com>:
> Calling commit after every addition will drastically slow down your
> indexing throughput, and concurrency (commit is internally
> synchronized), but should not create lock timeouts, unless you are
> also opening a new IndexWriter for every addition?
>
> Mike
>
> On Thu, Jan 14, 2010 at 12:15 PM, jchang <jc...@gmail.com> wrote:
>>
>> With only 10 concurrent consumers, I do get lock problems. However, I am
>> calling commit() at the end of each addition. Could I expect better
>> concurrency without timeouts if I did not commit as often?
>>
>> --
>> View this message in context: http://old.nabble.com/Lucene-2.9.0-Near-Real-Time-Indexing-and-lock-timeouts-tp27136743p27164797.html
>> Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Lucene 2.9.0 Near Real Time Indexing and lock timeouts
Posted by Michael McCandless <lu...@mikemccandless.com>.
Calling commit after every addition will drastically slow down your
indexing throughput, and concurrency (commit is internally
synchronized), but should not create lock timeouts, unless you are
also opening a new IndexWriter for every addition?
Mike
On Thu, Jan 14, 2010 at 12:15 PM, jchang <jc...@gmail.com> wrote:
>
> With only 10 concurrent consumers, I do get lock problems. However, I am
> calling commit() at the end of each addition. Could I expect better
> concurrency without timeouts if I did not commit as often?
>
> --
> View this message in context: http://old.nabble.com/Lucene-2.9.0-Near-Real-Time-Indexing-and-lock-timeouts-tp27136743p27164797.html
> Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Lucene 2.9.0 Near Real Time Indexing and lock timeouts
Posted by jchang <jc...@gmail.com>.
With only 10 concurrent consumers, I do get lock problems. However, I am
calling commit() at the end of each addition. Could I expect better
concurrency without timeouts if I did not commit as often?
--
View this message in context: http://old.nabble.com/Lucene-2.9.0-Near-Real-Time-Indexing-and-lock-timeouts-tp27136743p27164797.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Lucene 2.9.0 Near Real Time Indexing and lock timeouts
Posted by Otis Gospodnetic <ot...@yahoo.com>.
John,
Yes, you should get 2.9.0 or 3.0.0, their indexing is faster. Still, even with 2.4.0 you shouldn't run into problems if you are really using just 1 IndexWriter. Still, I'd try upgrading first. Oh, and java-user is the place to ask.
Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
----- Original Message ----
> From: jchang <jc...@gmail.com>
> To: java-dev@lucene.apache.org
> Sent: Tue, January 12, 2010 6:27:12 PM
> Subject: Lucene 2.9.0 Near Real Time Indexing and lock timeouts
>
>
> Hello,
>
> I am using Lucene 2.4.0 and am getting
> org.apache.lucene.store.LockObtainFailedException's when I have a backed up
> queue of items to index (with multiple concurrent writers). Of course, if I
> throttle all my writer threads to 1, I don't get the exception, but I'm
> hoping to write faster than that. Also, when I have multiple index writer
> threads, it's not that much faster anyhow, I assume because the bottle neck
> is on one the index file.
>
> Will 2.9.0 help? I've read about the near real time indexing. Will this
> allow me to have:
> 1) Multiple threads indexing items all at the same time, achieving
> appreciably faster throughput than only one thread?
> 2) And avoid getting org.apache.lucene.store.LockObtainFailedException's?
> 3) Have this all near real time?
>
> And generally, if anybody has any advice on high-throughput indexing with
> Lucene and what kind of numbers I can acheive, I'd welcome the feedback.
>
> Thanks,
> John
> --
> View this message in context:
> http://old.nabble.com/Lucene-2.9.0-Near-Real-Time-Indexing-and-lock-timeouts-tp27136743p27136743.html
> Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Lucene 2.9.0 Near Real Time Indexing and lock timeouts
Posted by Jason Rutherglen <ja...@gmail.com>.
> And generally, if anybody has any advice on high-throughput indexing with
> Lucene and what kind of numbers I can acheive, I'd welcome the feedback.
I believe it's directly related to how often IW.getReader is called.
The longer the duration between calls, larger the resultant new
segment is (which reduces subsequent merging).
On Tue, Jan 12, 2010 at 3:27 PM, jchang <jc...@gmail.com> wrote:
>
> Hello,
>
> I am using Lucene 2.4.0 and am getting
> org.apache.lucene.store.LockObtainFailedException's when I have a backed up
> queue of items to index (with multiple concurrent writers). Of course, if I
> throttle all my writer threads to 1, I don't get the exception, but I'm
> hoping to write faster than that. Also, when I have multiple index writer
> threads, it's not that much faster anyhow, I assume because the bottle neck
> is on one the index file.
>
> Will 2.9.0 help? I've read about the near real time indexing. Will this
> allow me to have:
> 1) Multiple threads indexing items all at the same time, achieving
> appreciably faster throughput than only one thread?
> 2) And avoid getting org.apache.lucene.store.LockObtainFailedException's?
> 3) Have this all near real time?
>
> And generally, if anybody has any advice on high-throughput indexing with
> Lucene and what kind of numbers I can acheive, I'd welcome the feedback.
>
> Thanks,
> John
> --
> View this message in context: http://old.nabble.com/Lucene-2.9.0-Near-Real-Time-Indexing-and-lock-timeouts-tp27136743p27136743.html
> Sent from the Lucene - Java Developer mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org