You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Shlomit Rosen <SH...@il.ibm.com> on 2014/09/08 11:08:32 UTC

IOExceptions during search

Hello :)

We have a customer who's system keeps crashing at certain queries. 
At default we use Mmap directory for search as it usually gives us the 
best performance. 
Although the heap usage seemed normal, we asked them to switch to 
NIOFSDirectory to make sure it's not a memory issue... 
The system stopped crashing, but now they see the following errors occur 
in some of the searches: 

java.io.IOException: Input/output error
      sun.nio.ch.FileDispatcher.pread0(Native Method)
      sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:43)
      sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:207)
      sun.nio.ch.IOUtil.read(IOUtil.java:183)
      sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:642)
 
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:162)
 
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:156)
 
org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:275)
 
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:270)
 
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:40)
      org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)
 
org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:217)
 org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:121)
 
org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:103)
      org.apache.lucene.search.spans.TermSpans.next(TermSpans.java:49)

AND - 


java.io.IOException: Stale NFS file handle
      sun.nio.ch.FileDispatcher.pread0(Native Method)
      sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:43)
      sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:207)
      sun.nio.ch.IOUtil.read(IOUtil.java:183)
      sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:642)
 
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:162)
 
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:156)
 
org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:275)
 
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:270)
 
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:40)
      org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)
 
org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:217)
      org.apache.lucene.index.TermBuffer.read(TermBuffer.java:64)
 org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:133)
 org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:174)
 org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:236)
 org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:304)
      org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:473)
 
org.apache.lucene.index.DirectoryReader$MultiTermEnum.&lt;init&gt;(DirectoryReader.java:1086)
 org.apache.lucene.index.DirectoryReader.terms(DirectoryReader.java:698)
 
org.apache.lucene.search.TermRangeTermEnum.&lt;init&gt;(TermRangeTermEnum.java:93)
 org.apache.lucene.search.TermRangeQuery.getEnum(TermRangeQuery.java:135)


The customer is running on Linux 64, OS version: 2.6.18-371.8.1.el5
I saw in different forums that this might have to do with the lock 
factory, but we are using SimpleFSLockFactory....
Currently they are running on lucene 3.6.0 but previously they were on 
lucene 3.0.2/ lucene 3.0.3, so the collections they are searching might 
have been created with an older version. 

We have asked them to run checkIndex to rule out corruption, but we 
haven't heard back yet. 

Any idea what might be wrong here, or how can we help them? 

Thanks in advance !
Shlomit

RE: IOExceptions during search

Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,

NFS is not a filesystem that works reliable and correctly with Lucene. Please consider using another file system, preferably on local disks or (if it needs to be networked) using iSCSI or similar.
MMap crushes the whole JVM easily if NFS connections drop. NIOFSDir works better, but the whole issue are those stale file handles, which makes commits in Lucene unreliable.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Shlomit Rosen [mailto:SHLOMITR@il.ibm.com]
> Sent: Monday, September 08, 2014 11:09 AM
> To: java-user@lucene.apache.org
> Subject: IOExceptions during search
> 
> Hello :)
> 
> We have a customer who's system keeps crashing at certain queries.
> At default we use Mmap directory for search as it usually gives us the best
> performance.
> Although the heap usage seemed normal, we asked them to switch to
> NIOFSDirectory to make sure it's not a memory issue...
> The system stopped crashing, but now they see the following errors occur in
> some of the searches:
> 
> java.io.IOException: Input/output error
>       sun.nio.ch.FileDispatcher.pread0(Native Method)
>       sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:43)
>       sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:207)
>       sun.nio.ch.IOUtil.read(IOUtil.java:183)
>       sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:642)
> 
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIO
> FSDirectory.java:162)
> 
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput
> .java:156)
> 
> org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(
> CompoundFileReader.java:275)
> 
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
> 270)
> 
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.
> java:40)
>       org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)
> 
> org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.j
> ava:217)
> 
> org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:1
> 21)
> 
> org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPosition
> s.java:103)
>       org.apache.lucene.search.spans.TermSpans.next(TermSpans.java:49)
> 
> AND -
> 
> 
> java.io.IOException: Stale NFS file handle
>       sun.nio.ch.FileDispatcher.pread0(Native Method)
>       sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:43)
>       sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:207)
>       sun.nio.ch.IOUtil.read(IOUtil.java:183)
>       sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:642)
> 
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIO
> FSDirectory.java:162)
> 
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput
> .java:156)
> 
> org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(
> CompoundFileReader.java:275)
> 
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
> 270)
> 
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.
> java:40)
>       org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)
> 
> org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.j
> ava:217)
>       org.apache.lucene.index.TermBuffer.read(TermBuffer.java:64)
> 
> org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:
> 133)
> 
> org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.ja
> va:174)
>  org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:236)
> 
> org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:304
> )
> 
> org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:473)
> 
> org.apache.lucene.index.DirectoryReader$MultiTermEnum.&lt;init&gt;(Direc
> toryReader.java:1086)
>  org.apache.lucene.index.DirectoryReader.terms(DirectoryReader.java:698)
> 
> org.apache.lucene.search.TermRangeTermEnum.&lt;init&gt;(TermRangeTer
> mEnum.java:93)
> 
> org.apache.lucene.search.TermRangeQuery.getEnum(TermRangeQuery.java
> :135)
> 
> 
> The customer is running on Linux 64, OS version: 2.6.18-371.8.1.el5 I saw in
> different forums that this might have to do with the lock factory, but we are
> using SimpleFSLockFactory....
> Currently they are running on lucene 3.6.0 but previously they were on
> lucene 3.0.2/ lucene 3.0.3, so the collections they are searching might have
> been created with an older version.
> 
> We have asked them to run checkIndex to rule out corruption, but we
> haven't heard back yet.
> 
> Any idea what might be wrong here, or how can we help them?
> 
> Thanks in advance !
> Shlomit


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org