You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Li Bing <lb...@gmail.com> on 2009/10/29 06:56:37 UTC
Exceptions When Indexing with Multi-threading
Dear all,
Today I got exceptions when indexing with multi-threading. It says
that "deletable.new" or "segments.new" are accessed by other
processes. So it cannot rename ...
However, I have used "ReaderWriterSlim" lock to avoid the relevant
folders to be accessed by multiple threads at the same time. For
example, if the indexing is done in D:\Test\, I use "ReaderWriterSlim"
to prevent other threads from writing in the folder. I think the above
exceptions should not happen if I do that. However, it is NOT true.
Why?
Any other ways to avoid the problem?
Thanks so much!
LB
Re: Exceptions When Indexing with Multi-threading
Posted by Li Bing <lb...@gmail.com>.
Dear Michael,
Thanks so much for your help! I will try your solution and let you
know if it works fine.
Best,
LB
On Fri, Oct 30, 2009 at 11:05 AM, Michael Garski
<mg...@myspace-inc.com> wrote:
> You won't need to synchronize the writers yourself as there are locks internally, but I would avoid multiple threads sharing a writer for performance reasons.
>
> Michael
>
> -----Original Message-----
> From: Li Bing [mailto:lblabs@gmail.com]
> Sent: Thursday, October 29, 2009 8:02 PM
> To: Michael Garski
> Cc: lucene-net-user@incubator.apache.org
> Subject: Re: Exceptions When Indexing with Multi-threading
>
> Dear Michael,
>
> Yes, each thread in my system has its own IndexWriter. But all of them
> are synchronized by a unique lock. So I think the exception should not
> occur, right? If the threads share a single IndexWriter, what about
> the performance? I don't need to synchronize by myself if so, right?
>
> I appreciate so much for your help!
>
> LB
>
> On Fri, Oct 30, 2009 at 12:33 AM, Michael Garski
> <mg...@myspace-inc.com> wrote:
>> Does each thread have its own IndexWriter? This could cause the issue
>> you describe.
>>
>> IndexWriter is thread-safe so that multiple threads can use the same
>> instance, however you will see a performance degradation compared to
>> only having one thread modify the index. We've done quite a bit of
>> testing for that case a while back and only have one thread modifying an
>> index at a time.
>>
>> Michael
>>
>> -----Original Message-----
>> From: Li Bing [mailto:lblabs@gmail.com]
>> Sent: Wednesday, October 28, 2009 10:57 PM
>> To: lucene-net-user@incubator.apache.org
>> Cc: Li Bing - Gmail
>> Subject: Exceptions When Indexing with Multi-threading
>>
>> Dear all,
>>
>> Today I got exceptions when indexing with multi-threading. It says
>> that "deletable.new" or "segments.new" are accessed by other
>> processes. So it cannot rename ...
>>
>> However, I have used "ReaderWriterSlim" lock to avoid the relevant
>> folders to be accessed by multiple threads at the same time. For
>> example, if the indexing is done in D:\Test\, I use "ReaderWriterSlim"
>> to prevent other threads from writing in the folder. I think the above
>> exceptions should not happen if I do that. However, it is NOT true.
>> Why?
>>
>> Any other ways to avoid the problem?
>>
>> Thanks so much!
>> LB
>>
>>
>>
>
>
>
RE: Exceptions When Indexing with Multi-threading
Posted by Michael Garski <mg...@myspace-inc.com>.
You won't need to synchronize the writers yourself as there are locks internally, but I would avoid multiple threads sharing a writer for performance reasons.
Michael
-----Original Message-----
From: Li Bing [mailto:lblabs@gmail.com]
Sent: Thursday, October 29, 2009 8:02 PM
To: Michael Garski
Cc: lucene-net-user@incubator.apache.org
Subject: Re: Exceptions When Indexing with Multi-threading
Dear Michael,
Yes, each thread in my system has its own IndexWriter. But all of them
are synchronized by a unique lock. So I think the exception should not
occur, right? If the threads share a single IndexWriter, what about
the performance? I don't need to synchronize by myself if so, right?
I appreciate so much for your help!
LB
On Fri, Oct 30, 2009 at 12:33 AM, Michael Garski
<mg...@myspace-inc.com> wrote:
> Does each thread have its own IndexWriter? This could cause the issue
> you describe.
>
> IndexWriter is thread-safe so that multiple threads can use the same
> instance, however you will see a performance degradation compared to
> only having one thread modify the index. We've done quite a bit of
> testing for that case a while back and only have one thread modifying an
> index at a time.
>
> Michael
>
> -----Original Message-----
> From: Li Bing [mailto:lblabs@gmail.com]
> Sent: Wednesday, October 28, 2009 10:57 PM
> To: lucene-net-user@incubator.apache.org
> Cc: Li Bing - Gmail
> Subject: Exceptions When Indexing with Multi-threading
>
> Dear all,
>
> Today I got exceptions when indexing with multi-threading. It says
> that "deletable.new" or "segments.new" are accessed by other
> processes. So it cannot rename ...
>
> However, I have used "ReaderWriterSlim" lock to avoid the relevant
> folders to be accessed by multiple threads at the same time. For
> example, if the indexing is done in D:\Test\, I use "ReaderWriterSlim"
> to prevent other threads from writing in the folder. I think the above
> exceptions should not happen if I do that. However, it is NOT true.
> Why?
>
> Any other ways to avoid the problem?
>
> Thanks so much!
> LB
>
>
>
Re: Exceptions When Indexing with Multi-threading
Posted by Li Bing <lb...@gmail.com>.
Dear Michael,
Yes, each thread in my system has its own IndexWriter. But all of them
are synchronized by a unique lock. So I think the exception should not
occur, right? If the threads share a single IndexWriter, what about
the performance? I don't need to synchronize by myself if so, right?
I appreciate so much for your help!
LB
On Fri, Oct 30, 2009 at 12:33 AM, Michael Garski
<mg...@myspace-inc.com> wrote:
> Does each thread have its own IndexWriter? This could cause the issue
> you describe.
>
> IndexWriter is thread-safe so that multiple threads can use the same
> instance, however you will see a performance degradation compared to
> only having one thread modify the index. We've done quite a bit of
> testing for that case a while back and only have one thread modifying an
> index at a time.
>
> Michael
>
> -----Original Message-----
> From: Li Bing [mailto:lblabs@gmail.com]
> Sent: Wednesday, October 28, 2009 10:57 PM
> To: lucene-net-user@incubator.apache.org
> Cc: Li Bing - Gmail
> Subject: Exceptions When Indexing with Multi-threading
>
> Dear all,
>
> Today I got exceptions when indexing with multi-threading. It says
> that "deletable.new" or "segments.new" are accessed by other
> processes. So it cannot rename ...
>
> However, I have used "ReaderWriterSlim" lock to avoid the relevant
> folders to be accessed by multiple threads at the same time. For
> example, if the indexing is done in D:\Test\, I use "ReaderWriterSlim"
> to prevent other threads from writing in the folder. I think the above
> exceptions should not happen if I do that. However, it is NOT true.
> Why?
>
> Any other ways to avoid the problem?
>
> Thanks so much!
> LB
>
>
>
RE: Exceptions When Indexing with Multi-threading
Posted by Michael Garski <mg...@myspace-inc.com>.
Does each thread have its own IndexWriter? This could cause the issue
you describe.
IndexWriter is thread-safe so that multiple threads can use the same
instance, however you will see a performance degradation compared to
only having one thread modify the index. We've done quite a bit of
testing for that case a while back and only have one thread modifying an
index at a time.
Michael
-----Original Message-----
From: Li Bing [mailto:lblabs@gmail.com]
Sent: Wednesday, October 28, 2009 10:57 PM
To: lucene-net-user@incubator.apache.org
Cc: Li Bing - Gmail
Subject: Exceptions When Indexing with Multi-threading
Dear all,
Today I got exceptions when indexing with multi-threading. It says
that "deletable.new" or "segments.new" are accessed by other
processes. So it cannot rename ...
However, I have used "ReaderWriterSlim" lock to avoid the relevant
folders to be accessed by multiple threads at the same time. For
example, if the indexing is done in D:\Test\, I use "ReaderWriterSlim"
to prevent other threads from writing in the folder. I think the above
exceptions should not happen if I do that. However, it is NOT true.
Why?
Any other ways to avoid the problem?
Thanks so much!
LB
RE: Exceptions When Indexing with Multi-threading
Posted by Digy <di...@gmail.com>.
Firstly, which version of Lucene.NET are you using? if 2.9.0, it hasn't been
thoroughly tested yet.
Secondly, you don't have to use any synchronization mechanism, since
Lucene.NET is thread-safe. So you can share the same instance of IndexWriter
or IndexSearcher among threads. Using a write-lock, it also hinders a second
IndexWriter instance to be opened with the same index.
DIGY
-----Original Message-----
From: Li Bing [mailto:lblabs@gmail.com]
Sent: Thursday, October 29, 2009 7:57 AM
To: lucene-net-user@incubator.apache.org
Cc: Li Bing - Gmail
Subject: Exceptions When Indexing with Multi-threading
Dear all,
Today I got exceptions when indexing with multi-threading. It says
that "deletable.new" or "segments.new" are accessed by other
processes. So it cannot rename ...
However, I have used "ReaderWriterSlim" lock to avoid the relevant
folders to be accessed by multiple threads at the same time. For
example, if the indexing is done in D:\Test\, I use "ReaderWriterSlim"
to prevent other threads from writing in the folder. I think the above
exceptions should not happen if I do that. However, it is NOT true.
Why?
Any other ways to avoid the problem?
Thanks so much!
LB