You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Shai Erera <se...@gmail.com> on 2009/12/02 13:35:33 UTC
Best Locking approach (Directory Lock)
Hi
We've run into problems w/ LockFactory usage on our system. The problem is
that the system can be such that the index is configured on a local file
system, or a remote, shared one. If remote/share, the protocol is sometimes
SMB 1.0/2.0, Samba, NFS 3/4 etc. In short, we have no control on the FS type
used or OS (Linux, Windows 2003/2008, Unix, AIX, iSeries etc.).
We currently use NativeFSLockFactory, because we want to ensure the lock
file is deleted upon JVM exit, whether by clean exit, or "kill". However,
the more I read the documentation of the class, and the inline documentation
in the class, I get a feeling that this choice, in our environment, is error
prone. Some file systems do not support native FS locking, some may take too
long to obtain the lock, in which case the Lock obtain method will timeout
etc. The current problem is that the index is configured on a Samba share,
and we hit LockObtainFailedExceptions (timeouts, or Access Denied once).
When the system is configured on a Windows share (I think SMB 1.0),
everything works fine.
So I'm not sure what to do. Should we always use SimpleFSLockFactory? We
know for sure only one node in the cluster will access the shared file
system for writing (not in a given time, but always). Therefore I was
tempted to move to NoLockFactory. But this move scares me a bit, and I'd
appreciate if someone can confirm my understanding of Lock being used to
protect against multiple IndexWriters from different JVMs *only*, or does it
have other uses as well? What about multiple IndexWriters from the same JVM?
If NoLockFactory is not safe, then we can move to SimpleFSLockFactory, and
since we can guarantee by other means that only one IndexWriter attempts to
write to the index, we can IndexWriter.unlock(Dir) if the lock exists.
Also, the code of NativeFSLockFactory mentions that on NFS if not configured
properly, the lock obtaining may take 35 seconds. What sort of configuration
is needed? Just out of curiosity.
Any advice will be appreciated.
Shai
Re: Best Locking approach (Directory Lock)
Posted by Shai Erera <se...@gmail.com>.
Thanks Mike.
I'm pretty sure about our mechanism to control one JVM opening the index for
write, but bugs can always be discovered. That's why we used
NativeFSLockFactory - because even if the JVM is killed, the lock file
should be removed. So if we do find the lock, it's fishy (Siamese or not).
But if I use SimpleFSLockFactory and find the lock, it will be harder for me
to tell if it's because of JVM kill, or because there is a bug and another
IndexWriter opens the index for write.
I think I'll go w/ NoLockFactory then ...
Shai
On Wed, Dec 2, 2009 at 3:45 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> Currently the sole usage of the LockFactory within Lucene is for the
> write.lock, ie, to prevent more than one writer (= IndexWriter, or,
> IndexReader trying to do deletions or set norms) from being open on
> the index at once.
>
> If your app has external mechanisms that can ensure, without fail,
> that only one writer will be open at once, then NoLockFactory is
> perfectly safe.
>
> But you have to be certain: one slipup will quickly corrupt the index.
> IndexWriters are like the male Siamese fighting fish (betta fish) --
> they will fight to the death [of the index]!
>
> Mike
>
> On Wed, Dec 2, 2009 at 8:24 AM, Shai Erera <se...@gmail.com> wrote:
> > I have multiple JVMs on different machines accessing the shared file
> system.
> > I don't really have multiple IndexWriters on the same JVM, I asked this
> just
> > out of curiosity.
> >
> > So I don't understand from your reply if it's safe to use NoLockFactory,
> or
> > I should use SimpleFSLockFactory and unlock if needed?
> >
> > Shai
> >
> > On Wed, Dec 2, 2009 at 3:03 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> >
> >> Hi Shai,
> >>
> >> > have other uses as well? What about multiple IndexWriters from the
> same
> >> > JVM?
> >>
> >> This also needs a lock. But if it is guaranteed to only have one JVM
> >> accessing the index, you can use an in-JVM-Locking mechanism which is
> >> provided by
> >>
> >>
> http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn
> >> stanceLockFactory.html<
> http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn%0AstanceLockFactory.html
> >
> >>
> >> > If NoLockFactory is not safe, then we can move to SimpleFSLockFactory,
> >> and
> >> > since we can guarantee by other means that only one IndexWriter
> attempts
> >> > to
> >> > write to the index, we can IndexWriter.unlock(Dir) if the lock exists.
> >>
> >> If you can guarantee that you only have *one* JVM and there may be
> multiple
> >> IndexWriters, use above solution.
> >>
> >> Uwe
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Best Locking approach (Directory Lock)
Posted by Michael McCandless <lu...@mikemccandless.com>.
Currently the sole usage of the LockFactory within Lucene is for the
write.lock, ie, to prevent more than one writer (= IndexWriter, or,
IndexReader trying to do deletions or set norms) from being open on
the index at once.
If your app has external mechanisms that can ensure, without fail,
that only one writer will be open at once, then NoLockFactory is
perfectly safe.
But you have to be certain: one slipup will quickly corrupt the index.
IndexWriters are like the male Siamese fighting fish (betta fish) --
they will fight to the death [of the index]!
Mike
On Wed, Dec 2, 2009 at 8:24 AM, Shai Erera <se...@gmail.com> wrote:
> I have multiple JVMs on different machines accessing the shared file system.
> I don't really have multiple IndexWriters on the same JVM, I asked this just
> out of curiosity.
>
> So I don't understand from your reply if it's safe to use NoLockFactory, or
> I should use SimpleFSLockFactory and unlock if needed?
>
> Shai
>
> On Wed, Dec 2, 2009 at 3:03 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> Hi Shai,
>>
>> > have other uses as well? What about multiple IndexWriters from the same
>> > JVM?
>>
>> This also needs a lock. But if it is guaranteed to only have one JVM
>> accessing the index, you can use an in-JVM-Locking mechanism which is
>> provided by
>>
>> http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn
>> stanceLockFactory.html<http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn%0AstanceLockFactory.html>
>>
>> > If NoLockFactory is not safe, then we can move to SimpleFSLockFactory,
>> and
>> > since we can guarantee by other means that only one IndexWriter attempts
>> > to
>> > write to the index, we can IndexWriter.unlock(Dir) if the lock exists.
>>
>> If you can guarantee that you only have *one* JVM and there may be multiple
>> IndexWriters, use above solution.
>>
>> Uwe
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Best Locking approach (Directory Lock)
Posted by Shai Erera <se...@gmail.com>.
I have multiple JVMs on different machines accessing the shared file system.
I don't really have multiple IndexWriters on the same JVM, I asked this just
out of curiosity.
So I don't understand from your reply if it's safe to use NoLockFactory, or
I should use SimpleFSLockFactory and unlock if needed?
Shai
On Wed, Dec 2, 2009 at 3:03 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi Shai,
>
> > have other uses as well? What about multiple IndexWriters from the same
> > JVM?
>
> This also needs a lock. But if it is guaranteed to only have one JVM
> accessing the index, you can use an in-JVM-Locking mechanism which is
> provided by
>
> http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn
> stanceLockFactory.html<http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn%0AstanceLockFactory.html>
>
> > If NoLockFactory is not safe, then we can move to SimpleFSLockFactory,
> and
> > since we can guarantee by other means that only one IndexWriter attempts
> > to
> > write to the index, we can IndexWriter.unlock(Dir) if the lock exists.
>
> If you can guarantee that you only have *one* JVM and there may be multiple
> IndexWriters, use above solution.
>
> Uwe
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Best Locking approach (Directory Lock)
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Shai,
> have other uses as well? What about multiple IndexWriters from the same
> JVM?
This also needs a lock. But if it is guaranteed to only have one JVM
accessing the index, you can use an in-JVM-Locking mechanism which is
provided by
http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn
stanceLockFactory.html
> If NoLockFactory is not safe, then we can move to SimpleFSLockFactory, and
> since we can guarantee by other means that only one IndexWriter attempts
> to
> write to the index, we can IndexWriter.unlock(Dir) if the lock exists.
If you can guarantee that you only have *one* JVM and there may be multiple
IndexWriters, use above solution.
Uwe
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org