You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Francesco Bellomi <fb...@libero.it> on 2004/04/20 19:04:50 UTC
Locking problems with NFS
Dear Lucene users,
we are experiencing some difficulties in using Lucene with a NFS filesystem.
Basically, locking seems not to work properly, since it appears that
attempted concurring writing on the index (from different VMs) are not
blocked, and this often causes the index to be corrupted.
I found a reference to this topic in the archives:
http://issues.apache.org/eyebrowse/ReadMsg?listName=lucene-user@jakarta.apac
he.org&msgId=784959
[Doug Cutting]
> So long as all access is read-only, there should not be a problem. Keep
> in mind however that lock files are known to not work correctly over NFS.
Does anybody have a solution for this problem? Does anyone have some
suggestions about the implementation of some alternative approach for
locking, in order to make Lucene compatible with NFS?
We are using 1.4rc2.
Thanks in advance,
Francesco
-
Francesco Bellomi
"Use truth to show illusion,
and illusion to show truth."
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Locking problems with NFS
Posted by Doug Cutting <cu...@apache.org>.
Francesco Bellomi wrote:
> The only problem is that, as lucene 1.4rc2, FSDirectory is 'final'.
Please submit a patch to lucene-dev to make FSDirectory non-final.
> In fact, a third architectural approach would be to define an API for
> "pluggable" lock implementations: IMHO that would be more robust to changes
> in the Directory interface and implementations.
If you have a proposal, submit a patch to lucene-dev to make locking
pluggable.
FYI, I just ran across another NFS-safe locking approach:
# In NFS, the O_CREAT|O_EXCL isn't guaranteed to be atomic.
# So we create a temp file that is probably unique in space
# and time ($folder.lock.$time.$pid.$host).
# Then we use link to create the real lock file. Since link
# is atomic across nfs, this works.
This is from:
http://search.cpan.org/src/MARKOV/Mail-Box-2.053/lib/Mail/Box/Locker/NFS.pm
Unfortunately links are not supported by Java's io APIs, so I think the
dotlock+fcntl() is probably best for Lucene.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Locking problems with NFS
Posted by Francesco Bellomi <fb...@libero.it>.
Doug, thanks for your suggestions!
I will implement your solution and make it public.
We already use a 1.4 JVM, so we have no problems with that.
The only problem is that, as lucene 1.4rc2, FSDirectory is 'final'; I would
go for the subclassing solution if you can make it not final; otherwise, if
someone has good arguments for the single 'fat' FSDirectory solution, just
let me know it.
In fact, a third architectural approach would be to define an API for
"pluggable" lock implementations: IMHO that would be more robust to changes
in the Directory interface and implementations.
Thanks again!
Francesco
Doug Cutting <cu...@apache.org> wrote:
> Francesco Bellomi wrote:
>> we are experiencing some difficulties in using Lucene with a NFS
>> filesystem. Basically, locking seems not to work properly, since it
>> appears that
>> attempted concurring writing on the index (from different VMs) are
>> not
>> blocked, and this often causes the index to be corrupted.
>>
>> Does anybody have a solution for this problem? Does anyone have some
>> suggestions about the implementation of some alternative approach for
>> locking, in order to make Lucene compatible with NFS?
>
> I just found the following article, which might point the way:
>
> http://www.spinnaker.de/linux/nfs-locking.html
>
> FSDirectory.lock currently implements what this article calls
> "dotlocking". One can implement fcntl() locking using nio, but this
> requires a 1.4 JVM, which Lucene does not (yet).
>
>
http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileChannel.html#t
ryLock()
>
> A couple of alternatives:
>
> 1. One could implement a 1.4-only subclass of FSDirectory in the
> Sandbox
> that implements dotlocking plus fcntl() locking.
>
> 2. We could add this directly into FSDirectory so that it will only
> compile in 1.4+ JDKs, but that it will run in earlier JDKs by only
> performing fcntl() locking when in a 1.4+ JVM. Some folks would
> complain.
>
> Thoughts?
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
-
Francesco Bellomi
"Use truth to show illusion,
and illusion to show truth."
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Locking problems with NFS
Posted by Doug Cutting <cu...@apache.org>.
Francesco Bellomi wrote:
> we are experiencing some difficulties in using Lucene with a NFS filesystem.
> Basically, locking seems not to work properly, since it appears that
> attempted concurring writing on the index (from different VMs) are not
> blocked, and this often causes the index to be corrupted.
>
> Does anybody have a solution for this problem? Does anyone have some
> suggestions about the implementation of some alternative approach for
> locking, in order to make Lucene compatible with NFS?
I just found the following article, which might point the way:
http://www.spinnaker.de/linux/nfs-locking.html
FSDirectory.lock currently implements what this article calls
"dotlocking". One can implement fcntl() locking using nio, but this
requires a 1.4 JVM, which Lucene does not (yet).
http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileChannel.html#tryLock()
A couple of alternatives:
1. One could implement a 1.4-only subclass of FSDirectory in the Sandbox
that implements dotlocking plus fcntl() locking.
2. We could add this directly into FSDirectory so that it will only
compile in 1.4+ JDKs, but that it will run in earlier JDKs by only
performing fcntl() locking when in a 1.4+ JVM. Some folks would complain.
Thoughts?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org