Issues with locked indices

[Frederik Van Hoyweghen <fr...@chapoo.com>]

Hi,

We are currently experiencing some issues where the lock on our index isn't
being released properly (for some reason). Our indexing process ends with
close() being called on all the writers that were used in the indexing
process.

We currently use the default NativeFSLockFactory
<https://lucene.apache.org/core/4_3_0/core/org/apache/lucene/store/NativeFSLockFactory.html>
but
are considering trying one of the other ones.
I am looking at SimpleFSlockFactory
<https://lucene.apache.org/core/4_3_0/core/org/apache/lucene/store/SimpleFSLockFactory.html>
where
the only code change would be to unlock the index directory before indexing
to account for any abnormal JVM exits.

Is this something that would function well in both a local storage/NFS
based environment?
This is my only worry since some of our servers use NFS while others use
local storage.

For good measure, the exception I am getting is:
LockObtainFailedException: Lock obtain timed out: NativeFSLock@
/home/DEV01/index/1/write.lock

This is on a non-NFS system.


Kind regards,
Frederik

RE: Issues with locked indices

[Uwe Schindler <uw...@thetaphi.de>]

Hi,

It is just simple, Lucene locking and commits do not work correct on NFS file systems because they are not fully POSIX conformant. Because of this you may also produce corrupt indexes, as commits don't work and corrupt concurrently open files. Also you may see JVM crushes if memory mapped files are unmapped because of network failures and cause SIGSEGV.

If you want to use Lucene on NFS mounts, you have 2 possibilities:
- Change to CIFS/Samba mounts (CIFS conforms to POSIX standards like delete on last close and also supports correct locking with NativeFSLockFactory) -- or move to local disks!
- Use a special deletion policy (https://lucene.apache.org/core/6_5_1/core/org/apache/lucene/index/SnapshotDeletionPolicy.html) to make the commits not corrupt you open IndexSearchers because of suddenly disappearing files (Lucene deletes files while they are open, as POSIX has delete-on-last-close) and use SimpleFSLockFactory. But SimpleFSLockFactory may hit stale lock files issues on killed JVMs. Also don't use MMapDirectory for file storage as this wil likely crush your JVM on network problems! 

Some background: The original and recommended lock system works correct with killed VMs, as the existence of the lock file has nothing to do with the "state" of being locked. The lock file is just a placeholder to actually have a file instance to do the locking. There is no solution for mixed NFS and non-NFS directories. So get either rid of them or add your own logic to choose right lock and deletion policy depending on the file system. You may use Java 7+'s Path/Files API to get all mount points. Memory mapping is risky with NFS, as a no-longer reachable file may suddenly unmap the cache buffers from process space and the next access will segmentation fault your JVM. The Snapshot deletion policy keeps the last commits available on disk, so the "delete-on-last-close" behaviour by POSIX is not required. But you have to take care to delete snapshots when you have closed all readers.

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: Frederik Van Hoyweghen
> [mailto:frederik.vanhoyweghen@chapoo.com]
> Sent: Tuesday, June 6, 2017 12:36 PM
> To: java-user@lucene.apache.org
> Subject: Issues with locked indices
> 
> Hi,
> 
> We are currently experiencing some issues where the lock on our index isn't
> being released properly (for some reason). Our indexing process ends with
> close() being called on all the writers that were used in the indexing
> process.
> 
> We currently use the default NativeFSLockFactory
> <https://lucene.apache.org/core/4_3_0/core/org/apache/lucene/store/Nati
> veFSLockFactory.html>
> but
> are considering trying one of the other ones.
> I am looking at SimpleFSlockFactory
> <https://lucene.apache.org/core/4_3_0/core/org/apache/lucene/store/Simp
> leFSLockFactory.html>
> where
> the only code change would be to unlock the index directory before indexing
> to account for any abnormal JVM exits.
> 
> Is this something that would function well in both a local storage/NFS
> based environment?
> This is my only worry since some of our servers use NFS while others use
> local storage.
> 
> For good measure, the exception I am getting is:
> LockObtainFailedException: Lock obtain timed out: NativeFSLock@
> /home/DEV01/index/1/write.lock
> 
> This is on a non-NFS system.
> 
> 
> Kind regards,
> Frederik


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org