You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Eran Sevi <Er...@nice.com> on 2008/03/19 11:16:44 UTC
Multi process writer access to an index
Hi,
Using the latest Lucene.Net release I encounter exceptions when trying to write to the same index from different processes/threads.
The errors are about accessing the deletable file which is being used by another process.
I don't perform any specific locking because as I understand it there should be file-specific locking mechanism used by lucene API. This doesn't seem to be working as I am able to instantiate an IndexWriter on the same index simultaneously, but I encounter problems when I try to close/optimize it .
Here's an example of what I do in each process/thread:
writer = new IndexWriter(fi, new StandardAnalyzer(threadData.stopwords), true);
while (currentFile < threadData.files.Length)
{
//Add document
AddDocument(threadData.files[currentFile], writer);
writer.Close();
currentFile++;
}
Please help me understand if lucene doesn't support this kind of access, and what's the proper way to write to the same index from two or more different processes/computers.
Thanks,
Eran S.
RE: Multi process writer access to an index
Posted by Digy <di...@gmail.com>.
1- Getting the exception "....which is being used by another process." is a
lucene bug, but
getting "Lock obtain timeout" is an expected behaviour. Two
processes/threads can not
access the same index for writing at the same time.
2- To set the timeout value -->
Lucene.Net.Index.IndexWriter.SetDefaultWriteLockTimeout(1000*60);
(you can't use the instance of IndexWriter since it times out at the
constructor).
DIGY
-----Original Message-----
From: Eran Sevi [mailto:Eran.Sevi@nice.com]
Sent: Thursday, March 20, 2008 4:06 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Multi process writer access to an index
I moved to 2.1 although it's not declared as "final".
Still, I get the following exception:
Lock obtain timed out: SimpleFSLock@D:\Temp\xml\Lucene\MTIndex\write.lock
at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout) in
D:\C#\src\Lucene.Net\Store\Lock.cs:line 85
at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean
create, Boolean closeDir) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line
437
at Lucene.Net.Index.IndexWriter.Init(FileInfo path, Analyzer a, Boolean
create) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line 421
at Lucene.Net.Index.IndexWriter..ctor(FileInfo path, Analyzer a, Boolean
create) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line 297
at Nice.Analytics.Lucene.AAIndexer.RunThread(Object data) in
D:\LuceneWrapper\Indexer.cs:line 158
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
I run 2 threads which doesn't share any object between them. Both succeed at
creating an instance of a new IndexWriter on the same folder but fail when
trying to Close the index.
I tried setting the lockWaitTimeout to 300000 but the exception is thrown
immediately.
-----Original Message-----
From: Digy [mailto:digydigy@gmail.com]
Sent: Wednesday, March 19, 2008 18:37 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Multi process writer access to an index
Which version do you use?.This bug exists in version 2.0.
You can get the latest version 2.1 from svn.
DIGY.
-----Original Message-----
From: Eran Sevi [mailto:Eran.Sevi@nice.com]
Sent: Wednesday, March 19, 2008 12:17 PM
To: lucene-net-user@incubator.apache.org
Subject: Multi process writer access to an index
Hi,
Using the latest Lucene.Net release I encounter exceptions when trying to
write to the same index from different processes/threads.
The errors are about accessing the deletable file which is being used by
another process.
I don't perform any specific locking because as I understand it there should
be file-specific locking mechanism used by lucene API. This doesn't seem to
be working as I am able to instantiate an IndexWriter on the same index
simultaneously, but I encounter problems when I try to close/optimize it .
Here's an example of what I do in each process/thread:
writer = new IndexWriter(fi, new StandardAnalyzer(threadData.stopwords),
true);
while (currentFile < threadData.files.Length)
{
//Add document
AddDocument(threadData.files[currentFile], writer);
writer.Close();
currentFile++;
}
Please help me understand if lucene doesn't support this kind of access, and
what's the proper way to write to the same index from two or more different
processes/computers.
Thanks,
Eran S.
RE: Multi process writer access to an index
Posted by Digy <di...@gmail.com>.
1- Getting the exception "....which is being used by another process." is a
lucene bug, but
getting "Lock obtain timeout" is an expected behaviour. Two
processes/threads can not
access the same index for writing at the same time.
2- To set the timeout value -->
Lucene.Net.Index.IndexWriter.SetDefaultWriteLockTimeout(1000*60);
(you can't use the instance of IndexWriter since it times out at the
constructor).
DIGY
-----Original Message-----
From: Eran Sevi [mailto:Eran.Sevi@nice.com]
Sent: Thursday, March 20, 2008 4:06 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Multi process writer access to an index
I moved to 2.1 although it's not declared as "final".
Still, I get the following exception:
Lock obtain timed out: SimpleFSLock@D:\Temp\xml\Lucene\MTIndex\write.lock
at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout) in
D:\C#\src\Lucene.Net\Store\Lock.cs:line 85
at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean
create, Boolean closeDir) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line
437
at Lucene.Net.Index.IndexWriter.Init(FileInfo path, Analyzer a, Boolean
create) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line 421
at Lucene.Net.Index.IndexWriter..ctor(FileInfo path, Analyzer a, Boolean
create) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line 297
at Nice.Analytics.Lucene.AAIndexer.RunThread(Object data) in
D:\LuceneWrapper\Indexer.cs:line 158
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
I run 2 threads which doesn't share any object between them. Both succeed at
creating an instance of a new IndexWriter on the same folder but fail when
trying to Close the index.
I tried setting the lockWaitTimeout to 300000 but the exception is thrown
immediately.
-----Original Message-----
From: Digy [mailto:digydigy@gmail.com]
Sent: Wednesday, March 19, 2008 18:37 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Multi process writer access to an index
Which version do you use?.This bug exists in version 2.0.
You can get the latest version 2.1 from svn.
DIGY.
-----Original Message-----
From: Eran Sevi [mailto:Eran.Sevi@nice.com]
Sent: Wednesday, March 19, 2008 12:17 PM
To: lucene-net-user@incubator.apache.org
Subject: Multi process writer access to an index
Hi,
Using the latest Lucene.Net release I encounter exceptions when trying to
write to the same index from different processes/threads.
The errors are about accessing the deletable file which is being used by
another process.
I don't perform any specific locking because as I understand it there should
be file-specific locking mechanism used by lucene API. This doesn't seem to
be working as I am able to instantiate an IndexWriter on the same index
simultaneously, but I encounter problems when I try to close/optimize it .
Here's an example of what I do in each process/thread:
writer = new IndexWriter(fi, new StandardAnalyzer(threadData.stopwords),
true);
while (currentFile < threadData.files.Length)
{
//Add document
AddDocument(threadData.files[currentFile], writer);
writer.Close();
currentFile++;
}
Please help me understand if lucene doesn't support this kind of access, and
what's the proper way to write to the same index from two or more different
processes/computers.
Thanks,
Eran S.
RE: Multi process writer access to an index
Posted by Eran Sevi <Er...@nice.com>.
What I really want to do is add documents to the same index from several different computers/processes so I can't use any shared memory or resources.
As a result I can't use any normal locking mechanism. I thought since Lucene uses file-based locking this might just work.
Am I missing something here?
-----Original Message-----
From: Jeff [mailto:jeff.lucene@gmail.com]
Sent: Thursday, March 20, 2008 16:31 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: Multi process writer access to an index
IndexWriter is not threadsafe. What you can do though is index the files on
seperate threads and put them into a RAMDirectory[] and use a single
IndexWriter to add them using AddIndexes(Directory[]).
On Thu, Mar 20, 2008 at 10:06 AM, Eran Sevi <Er...@nice.com> wrote:
> I moved to 2.1 although it's not declared as "final".
>
> Still, I get the following exception:
>
> Lock obtain timed out: SimpleFSLock@D:\Temp\xml\Lucene\MTIndex\write.lock
> at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout) in
> D:\C#\src\Lucene.Net\Store\Lock.cs:line 85
> at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean
> create, Boolean closeDir) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line
> 437
> at Lucene.Net.Index.IndexWriter.Init(FileInfo path, Analyzer a, Boolean
> create) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line 421
> at Lucene.Net.Index.IndexWriter..ctor(FileInfo path, Analyzer a, Boolean
> create) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line 297
> at Nice.Analytics.Lucene.AAIndexer.RunThread(Object data) in
> D:\LuceneWrapper\Indexer.cs:line 158
> at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
> at System.Threading.ExecutionContext.Run(ExecutionContext
> executionContext, ContextCallback callback, Object state)
> at System.Threading.ThreadHelper.ThreadStart(Object obj)
>
> I run 2 threads which doesn't share any object between them. Both succeed
> at creating an instance of a new IndexWriter on the same folder but fail
> when trying to Close the index.
> I tried setting the lockWaitTimeout to 300000 but the exception is thrown
> immediately.
>
>
> -----Original Message-----
> From: Digy [mailto:digydigy@gmail.com]
> Sent: Wednesday, March 19, 2008 18:37 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Multi process writer access to an index
>
> Which version do you use?.This bug exists in version 2.0.
> You can get the latest version 2.1 from svn.
>
> DIGY.
>
>
> -----Original Message-----
> From: Eran Sevi [mailto:Eran.Sevi@nice.com]
> Sent: Wednesday, March 19, 2008 12:17 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Multi process writer access to an index
>
> Hi,
> Using the latest Lucene.Net release I encounter exceptions when trying to
> write to the same index from different processes/threads.
>
> The errors are about accessing the deletable file which is being used by
> another process.
>
> I don't perform any specific locking because as I understand it there
> should
> be file-specific locking mechanism used by lucene API. This doesn't seem
> to
> be working as I am able to instantiate an IndexWriter on the same index
> simultaneously, but I encounter problems when I try to close/optimize it .
>
> Here's an example of what I do in each process/thread:
>
> writer = new IndexWriter(fi, new StandardAnalyzer(threadData.stopwords),
> true);
> while (currentFile < threadData.files.Length)
> {
> //Add document
> AddDocument(threadData.files[currentFile], writer);
>
> writer.Close();
>
> currentFile++;
> }
>
> Please help me understand if lucene doesn't support this kind of access,
> and
> what's the proper way to write to the same index from two or more
> different
> processes/computers.
>
> Thanks,
> Eran S.
>
>
>
Re: Multi process writer access to an index
Posted by Jeff <je...@gmail.com>.
IndexWriter is not threadsafe. What you can do though is index the files on
seperate threads and put them into a RAMDirectory[] and use a single
IndexWriter to add them using AddIndexes(Directory[]).
On Thu, Mar 20, 2008 at 10:06 AM, Eran Sevi <Er...@nice.com> wrote:
> I moved to 2.1 although it's not declared as "final".
>
> Still, I get the following exception:
>
> Lock obtain timed out: SimpleFSLock@D:\Temp\xml\Lucene\MTIndex\write.lock
> at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout) in
> D:\C#\src\Lucene.Net\Store\Lock.cs:line 85
> at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean
> create, Boolean closeDir) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line
> 437
> at Lucene.Net.Index.IndexWriter.Init(FileInfo path, Analyzer a, Boolean
> create) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line 421
> at Lucene.Net.Index.IndexWriter..ctor(FileInfo path, Analyzer a, Boolean
> create) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line 297
> at Nice.Analytics.Lucene.AAIndexer.RunThread(Object data) in
> D:\LuceneWrapper\Indexer.cs:line 158
> at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
> at System.Threading.ExecutionContext.Run(ExecutionContext
> executionContext, ContextCallback callback, Object state)
> at System.Threading.ThreadHelper.ThreadStart(Object obj)
>
> I run 2 threads which doesn't share any object between them. Both succeed
> at creating an instance of a new IndexWriter on the same folder but fail
> when trying to Close the index.
> I tried setting the lockWaitTimeout to 300000 but the exception is thrown
> immediately.
>
>
> -----Original Message-----
> From: Digy [mailto:digydigy@gmail.com]
> Sent: Wednesday, March 19, 2008 18:37 PM
> To: lucene-net-user@incubator.apache.org
> Subject: RE: Multi process writer access to an index
>
> Which version do you use?.This bug exists in version 2.0.
> You can get the latest version 2.1 from svn.
>
> DIGY.
>
>
> -----Original Message-----
> From: Eran Sevi [mailto:Eran.Sevi@nice.com]
> Sent: Wednesday, March 19, 2008 12:17 PM
> To: lucene-net-user@incubator.apache.org
> Subject: Multi process writer access to an index
>
> Hi,
> Using the latest Lucene.Net release I encounter exceptions when trying to
> write to the same index from different processes/threads.
>
> The errors are about accessing the deletable file which is being used by
> another process.
>
> I don't perform any specific locking because as I understand it there
> should
> be file-specific locking mechanism used by lucene API. This doesn't seem
> to
> be working as I am able to instantiate an IndexWriter on the same index
> simultaneously, but I encounter problems when I try to close/optimize it .
>
> Here's an example of what I do in each process/thread:
>
> writer = new IndexWriter(fi, new StandardAnalyzer(threadData.stopwords),
> true);
> while (currentFile < threadData.files.Length)
> {
> //Add document
> AddDocument(threadData.files[currentFile], writer);
>
> writer.Close();
>
> currentFile++;
> }
>
> Please help me understand if lucene doesn't support this kind of access,
> and
> what's the proper way to write to the same index from two or more
> different
> processes/computers.
>
> Thanks,
> Eran S.
>
>
>
RE: Multi process writer access to an index
Posted by Eran Sevi <Er...@nice.com>.
I moved to 2.1 although it's not declared as "final".
Still, I get the following exception:
Lock obtain timed out: SimpleFSLock@D:\Temp\xml\Lucene\MTIndex\write.lock
at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout) in D:\C#\src\Lucene.Net\Store\Lock.cs:line 85
at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean create, Boolean closeDir) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line 437
at Lucene.Net.Index.IndexWriter.Init(FileInfo path, Analyzer a, Boolean create) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line 421
at Lucene.Net.Index.IndexWriter..ctor(FileInfo path, Analyzer a, Boolean create) in D:\C#\src\Lucene.Net\Index\IndexWriter.cs:line 297
at Nice.Analytics.Lucene.AAIndexer.RunThread(Object data) in D:\LuceneWrapper\Indexer.cs:line 158
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
I run 2 threads which doesn't share any object between them. Both succeed at creating an instance of a new IndexWriter on the same folder but fail when trying to Close the index.
I tried setting the lockWaitTimeout to 300000 but the exception is thrown immediately.
-----Original Message-----
From: Digy [mailto:digydigy@gmail.com]
Sent: Wednesday, March 19, 2008 18:37 PM
To: lucene-net-user@incubator.apache.org
Subject: RE: Multi process writer access to an index
Which version do you use?.This bug exists in version 2.0.
You can get the latest version 2.1 from svn.
DIGY.
-----Original Message-----
From: Eran Sevi [mailto:Eran.Sevi@nice.com]
Sent: Wednesday, March 19, 2008 12:17 PM
To: lucene-net-user@incubator.apache.org
Subject: Multi process writer access to an index
Hi,
Using the latest Lucene.Net release I encounter exceptions when trying to
write to the same index from different processes/threads.
The errors are about accessing the deletable file which is being used by
another process.
I don't perform any specific locking because as I understand it there should
be file-specific locking mechanism used by lucene API. This doesn't seem to
be working as I am able to instantiate an IndexWriter on the same index
simultaneously, but I encounter problems when I try to close/optimize it .
Here's an example of what I do in each process/thread:
writer = new IndexWriter(fi, new StandardAnalyzer(threadData.stopwords),
true);
while (currentFile < threadData.files.Length)
{
//Add document
AddDocument(threadData.files[currentFile], writer);
writer.Close();
currentFile++;
}
Please help me understand if lucene doesn't support this kind of access, and
what's the proper way to write to the same index from two or more different
processes/computers.
Thanks,
Eran S.
RE: Multi process writer access to an index
Posted by Digy <di...@gmail.com>.
Which version do you use?.This bug exists in version 2.0.
You can get the latest version 2.1 from svn.
DIGY.
-----Original Message-----
From: Eran Sevi [mailto:Eran.Sevi@nice.com]
Sent: Wednesday, March 19, 2008 12:17 PM
To: lucene-net-user@incubator.apache.org
Subject: Multi process writer access to an index
Hi,
Using the latest Lucene.Net release I encounter exceptions when trying to
write to the same index from different processes/threads.
The errors are about accessing the deletable file which is being used by
another process.
I don't perform any specific locking because as I understand it there should
be file-specific locking mechanism used by lucene API. This doesn't seem to
be working as I am able to instantiate an IndexWriter on the same index
simultaneously, but I encounter problems when I try to close/optimize it .
Here's an example of what I do in each process/thread:
writer = new IndexWriter(fi, new StandardAnalyzer(threadData.stopwords),
true);
while (currentFile < threadData.files.Length)
{
//Add document
AddDocument(threadData.files[currentFile], writer);
writer.Close();
currentFile++;
}
Please help me understand if lucene doesn't support this kind of access, and
what's the proper way to write to the same index from two or more different
processes/computers.
Thanks,
Eran S.