You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by liat oren <or...@gmail.com> on 2010/04/15 09:31:54 UTC

Exception in thread "Lucene Merge Thread #3" MergePolicy$MergeException: java.io.IOException: Data error (cyclic redundancy check)

Hi All,
I got the following error while trying to optimize index sized 31 GB:
Exception in thread "Lucene Merge Thread #3"
P.Lucene.Expert.index.MergePolicy$MergeException: java.io.IOException: Data
error (cyclic redundancy check)
        at
P.Lucene.Expert.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:369)
        at
P.Lucene.Expert.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:338)
Caused by: java.io.IOException: Data error (cyclic redundancy check)
        at java.io.RandomAccessFile.readBytes(Native Method)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
        at P.Lucene.Expert.store.FSDirectory$FSIndexInput

When could be the problem here?
Since it threw the application, the index stayed open. Is there a
possibility to close it properly? to save the data?

Many thanks,
Liat

Re: Exception in thread "Lucene Merge Thread #3" MergePolicy$MergeException: java.io.IOException: Data error (cyclic redundancy check)

Posted by liat oren <or...@gmail.com>.
Ok, thank you!
No, the full traceback is:
Exception in thread "Lucene Merge Thread #3"
P.Lucene.Expert.index.MergePolicy$MergeException: java.io.IOException: Data
error (cyclic redundancy check)
        at
P.Lucene.Expert.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:369)
        at
P.Lucene.Expert.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:338)
Caused by: java.io.IOException: Data error (cyclic redundancy check)
        at java.io.RandomAccessFile.readBytes(Native Method)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
        at
P.Lucene.Expert.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:727)
        at
P.Lucene.Expert.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:168)
        at
P.Lucene.Expert.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:271)
        at
P.Lucene.Expert.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:168)
        at
P.Lucene.Expert.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:109)
        at
P.Lucene.Expert.index.SegmentReader.norms(SegmentReader.java:1118)
        at
P.Lucene.Expert.index.SegmentMerger.mergeNorms(SegmentMerger.java:936)
        at P.Lucene.Expert.index.SegmentMerger.merge(SegmentMerger.java:145)
        at
P.Lucene.Expert.index.IndexWriter.mergeMiddle(IndexWriter.java:4894)
        at P.Lucene.Expert.index.IndexWriter.merge(IndexWriter.java:4469)
        at
P.Lucene.Expert.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:238)
        at
P.Lucene.Expert.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:305)
Exception in thread "main" java.io.IOException: background merge hit
exception:
_u:c83887 _15:c52319->_v _16:c5884->_v _17:c5856->_v _18:c5593->_v
_19:c6474->_v _1a:c5788->_v _1b:c6265->_v _1c:c8108->_v into _1d [optimize]
        at P.Lucene.Expert.index.IndexWriter.optimize(IndexWriter.java:2539)
        at P.Lucene.Expert.index.IndexWriter.optimize(IndexWriter.java:2468)
        at P.Lucene.Expert.index.IndexWriter.optimize(IndexWriter.java:2446)
        at P.index.SynonymIndex.generateSynonyms(SynonymIndex.java:140)
        at
P.indexGenerator.SynonymsIndexGenerator.generateIndex(SynonymsIndexGenerator.java:61)
        at
P.indexGenerator.SynonymsIndexGenerator.main(SynonymsIndexGenerator.java:32)
Caused by: java.io.IOException: Data error (cyclic redundancy check)
        at java.io.RandomAccessFile.readBytes(Native Method)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
        at
P.Lucene.Expert.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:727)
        at
P.Lucene.Expert.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:168)
        at
P.Lucene.Expert.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:271)
        at
P.Lucene.Expert.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:168)
        at
P.Lucene.Expert.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:109)
        at
P.Lucene.Expert.index.SegmentReader.norms(SegmentReader.java:1118)
        at
P.Lucene.Expert.index.SegmentMerger.mergeNorms(SegmentMerger.java:936)
        at P.Lucene.Expert.index.SegmentMerger.merge(SegmentMerger.java:145)
        at
P.Lucene.Expert.index.IndexWriter.mergeMiddle(IndexWriter.java:4894)
        at P.Lucene.Expert.index.IndexWriter.merge(IndexWriter.java:4469)
        at
P.Lucene.Expert.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:238)
        at
P.Lucene.Expert.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:305)


I will check it out.
Thanks!

On 15 April 2010 12:23, Michael McCandless <lu...@mikemccandless.com>wrote:

> The traceback is truncated?
>
> Still, this looks like your filesystem (and, specifically at least one
> of the files it's holding for Lucene) is corrupted -- Lucene was
> trying to read bytes from a file, in order to merge segments, and the
> OS/IO system threw that exception.
>
> You should run CheckIndex (which should detect & describe the
> corruption), and then try to recover the index by re-running
> CheckIndex with -fix.  But, more importantly, you need to figure out
> why your filesystem has developed corruption because it will likely
> happen again!
>
> Mike
>
> On Thu, Apr 15, 2010 at 3:31 AM, liat oren <or...@gmail.com> wrote:
> > Hi All,
> > I got the following error while trying to optimize index sized 31 GB:
> > Exception in thread "Lucene Merge Thread #3"
> > P.Lucene.Expert.index.MergePolicy$MergeException: java.io.IOException:
> Data
> > error (cyclic redundancy check)
> >        at
> >
> P.Lucene.Expert.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:369)
> >        at
> >
> P.Lucene.Expert.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:338)
> > Caused by: java.io.IOException: Data error (cyclic redundancy check)
> >        at java.io.RandomAccessFile.readBytes(Native Method)
> >        at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
> >        at P.Lucene.Expert.store.FSDirectory$FSIndexInput
> >
> > When could be the problem here?
> > Since it threw the application, the index stayed open. Is there a
> > possibility to close it properly? to save the data?
> >
> > Many thanks,
> > Liat
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Exception in thread "Lucene Merge Thread #3" MergePolicy$MergeException: java.io.IOException: Data error (cyclic redundancy check)

Posted by Michael McCandless <lu...@mikemccandless.com>.
The traceback is truncated?

Still, this looks like your filesystem (and, specifically at least one
of the files it's holding for Lucene) is corrupted -- Lucene was
trying to read bytes from a file, in order to merge segments, and the
OS/IO system threw that exception.

You should run CheckIndex (which should detect & describe the
corruption), and then try to recover the index by re-running
CheckIndex with -fix.  But, more importantly, you need to figure out
why your filesystem has developed corruption because it will likely
happen again!

Mike

On Thu, Apr 15, 2010 at 3:31 AM, liat oren <or...@gmail.com> wrote:
> Hi All,
> I got the following error while trying to optimize index sized 31 GB:
> Exception in thread "Lucene Merge Thread #3"
> P.Lucene.Expert.index.MergePolicy$MergeException: java.io.IOException: Data
> error (cyclic redundancy check)
>        at
> P.Lucene.Expert.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:369)
>        at
> P.Lucene.Expert.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:338)
> Caused by: java.io.IOException: Data error (cyclic redundancy check)
>        at java.io.RandomAccessFile.readBytes(Native Method)
>        at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
>        at P.Lucene.Expert.store.FSDirectory$FSIndexInput
>
> When could be the problem here?
> Since it threw the application, the index stayed open. Is there a
> possibility to close it properly? to save the data?
>
> Many thanks,
> Liat
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org