You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by 董自鸣 <dz...@gmail.com> on 2016/07/20 10:57:10 UTC
org.apache.lucene.index.CorruptIndexException: checksum failed
I use ubuntu to run lucene(java8, lucene6.0.0, not ES), and nfs to mount
disk(actually is rocketstor) of macmini. I get this error constantly. I
have already used org.apache.lucene.index.CheckIndex, but will still crash
after restart program.
Exception in thread "main" org.apache.lucene.store.AlreadyClosedException:
this IndexWriter is closed at
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724) at
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738) at
org.apache.lucene.index.IndexWriter.numDocs(IndexWriter.java:1198) at
xxxx.xxxxx.search.XxxxxxxxIndexer.close(XxxxxxxxIndexer.java:184) at
xxxx.xxxxx.search.ThreadedXxxxxxxxIndexer.close(ThreadedXxxxxxxxIndexer.java:59)
at
xxxx.xxxxx.search.ThreadedXxxxxxxxIndexer.main(ThreadedXxxxxxxxIndexer.java:136)
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
(hardware problem?) : expected=51fbdb5c actual=6e964d17
(resource=BufferedChecksumIndexInput(MMapIndexInput(path="/mnt/HPT8_56T/xxxxxxxxx-index/index1/_mq.cfs")
[slice=_mq_Lucene50_0.pos])) at
org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:365) at
org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:469)
at
org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.checkIntegrity(Lucene50PostingsReader.java:1286)
at
org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:336)
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:317)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96) at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:211) at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:96) at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4099) at
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3679) at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
--
*suiyuan*
Re: org.apache.lucene.index.CorruptIndexException: checksum failed
Posted by Ziming Dong <dz...@gmail.com>.
I upgraded lucene to 6.1.0, installed rocketstor 6318A driver, then used
checkIndex to rescue index, but my program still crashed. I find that if
just one computer builds index, everything is fine, but if I start second
computer to build index, one of this two program will crash several hours
later. these two computer mount the same directory of a macmini use nfs,
but their index directory is different. It seems nfs is the problem.
On Thu, Jul 21, 2016 at 12:49 PM, Ziming Dong <dz...@gmail.com>
wrote:
> I upgrade lucene to 6.1.0, hope it works, thank you very much!
>
> the mount( contain all data directories and index directories ) is shared
> by four different computers, but each index directory can be accessed only
> by one computer, we have four index directories.
>
> 2016-07-21 5:13 GMT+08:00 Michael McCandless <lu...@mikemccandless.com>:
>
>> This means that something is flipping bits from when Lucene wrote this
>> file, to when Lucene wanted to merge it.
>>
>> NFS is dangerous to use with Lucene: it doesn't support
>> delete-on-final-close semantics, its client side directory listing caches
>> can be stale, locking is iffy, etc. If you are using a single computer to
>> both write and read to/from the index, then it should be OK, though not
>> heavily tested, but if you share the mount across different computers
>> taking turns doing indexing and searching, there will be problems.
>>
>> Finally, even if things are working OK over NFS, it will likely be very
>> slow performance, compared to a local mount.
>>
>> All that being said, the known problems with NFS should not lead to this
>> corruption, I think. Something else is wrong. Maybe run an fsck on the
>> rocketsor disks? Maybe try eliminating variables, e.g. network card/cables
>> between your macmini and the rocketstor, swap in different hard drives in
>> the rocketstor, etc. Does the problem still happen with Lucene 6.1.0?
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Wed, Jul 20, 2016 at 6:57 AM, 董自鸣 <dz...@gmail.com> wrote:
>>
>>> I use ubuntu to run lucene(java8, lucene6.0.0, not ES), and nfs to mount
>>> disk(actually is rocketstor) of macmini. I get this error constantly. I
>>> have already used org.apache.lucene.index.CheckIndex, but will still
>>> crash
>>> after restart program.
>>>
>>>
>>> Exception in thread "main"
>>> org.apache.lucene.store.AlreadyClosedException:
>>> this IndexWriter is closed at
>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724) at
>>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738) at
>>> org.apache.lucene.index.IndexWriter.numDocs(IndexWriter.java:1198) at
>>> xxxx.xxxxx.search.XxxxxxxxIndexer.close(XxxxxxxxIndexer.java:184) at
>>>
>>> xxxx.xxxxx.search.ThreadedXxxxxxxxIndexer.close(ThreadedXxxxxxxxIndexer.java:59)
>>> at
>>>
>>> xxxx.xxxxx.search.ThreadedXxxxxxxxIndexer.main(ThreadedXxxxxxxxIndexer.java:136)
>>> Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
>>> (hardware problem?) : expected=51fbdb5c actual=6e964d17
>>>
>>> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/mnt/HPT8_56T/xxxxxxxxx-index/index1/_mq.cfs")
>>> [slice=_mq_Lucene50_0.pos])) at
>>> org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:365) at
>>> org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:469)
>>> at
>>>
>>> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.checkIntegrity(Lucene50PostingsReader.java:1286)
>>> at
>>>
>>> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:336)
>>> at
>>>
>>> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:317)
>>> at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
>>> at
>>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:211)
>>> at
>>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:96) at
>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4099) at
>>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3679) at
>>>
>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
>>> at
>>>
>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
>>>
>>> --
>>> *suiyuan*
>>>
>>
>>
>
>
> --
> *http://suiyuan2009.github.io/ <http://suiyuan2009.github.io/>*
> *suiyuan2009*
>
--
Ziming Dong
*http://suiyuan2009.github.io/ <http://suiyuan2009.github.io/>*
Re: org.apache.lucene.index.CorruptIndexException: checksum failed
Posted by Ziming Dong <dz...@gmail.com>.
I upgrade lucene to 6.1.0, hope it works, thank you very much!
the mount( contain all data directories and index directories ) is shared
by four different computers, but each index directory can be accessed only
by one computer, we have four index directories.
2016-07-21 5:13 GMT+08:00 Michael McCandless <lu...@mikemccandless.com>:
> This means that something is flipping bits from when Lucene wrote this
> file, to when Lucene wanted to merge it.
>
> NFS is dangerous to use with Lucene: it doesn't support
> delete-on-final-close semantics, its client side directory listing caches
> can be stale, locking is iffy, etc. If you are using a single computer to
> both write and read to/from the index, then it should be OK, though not
> heavily tested, but if you share the mount across different computers
> taking turns doing indexing and searching, there will be problems.
>
> Finally, even if things are working OK over NFS, it will likely be very
> slow performance, compared to a local mount.
>
> All that being said, the known problems with NFS should not lead to this
> corruption, I think. Something else is wrong. Maybe run an fsck on the
> rocketsor disks? Maybe try eliminating variables, e.g. network card/cables
> between your macmini and the rocketstor, swap in different hard drives in
> the rocketstor, etc. Does the problem still happen with Lucene 6.1.0?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Wed, Jul 20, 2016 at 6:57 AM, 董自鸣 <dz...@gmail.com> wrote:
>
>> I use ubuntu to run lucene(java8, lucene6.0.0, not ES), and nfs to mount
>> disk(actually is rocketstor) of macmini. I get this error constantly. I
>> have already used org.apache.lucene.index.CheckIndex, but will still crash
>> after restart program.
>>
>>
>> Exception in thread "main" org.apache.lucene.store.AlreadyClosedException:
>> this IndexWriter is closed at
>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724) at
>> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738) at
>> org.apache.lucene.index.IndexWriter.numDocs(IndexWriter.java:1198) at
>> xxxx.xxxxx.search.XxxxxxxxIndexer.close(XxxxxxxxIndexer.java:184) at
>>
>> xxxx.xxxxx.search.ThreadedXxxxxxxxIndexer.close(ThreadedXxxxxxxxIndexer.java:59)
>> at
>>
>> xxxx.xxxxx.search.ThreadedXxxxxxxxIndexer.main(ThreadedXxxxxxxxIndexer.java:136)
>> Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
>> (hardware problem?) : expected=51fbdb5c actual=6e964d17
>>
>> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/mnt/HPT8_56T/xxxxxxxxx-index/index1/_mq.cfs")
>> [slice=_mq_Lucene50_0.pos])) at
>> org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:365) at
>> org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:469)
>> at
>>
>> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.checkIntegrity(Lucene50PostingsReader.java:1286)
>> at
>>
>> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:336)
>> at
>>
>> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:317)
>> at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
>> at
>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:211)
>> at
>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:96) at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4099) at
>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3679) at
>>
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
>> at
>>
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
>>
>> --
>> *suiyuan*
>>
>
>
--
*http://suiyuan2009.github.io/ <http://suiyuan2009.github.io/>*
*suiyuan2009*
Re: org.apache.lucene.index.CorruptIndexException: checksum failed
Posted by Michael McCandless <lu...@mikemccandless.com>.
This means that something is flipping bits from when Lucene wrote this
file, to when Lucene wanted to merge it.
NFS is dangerous to use with Lucene: it doesn't support
delete-on-final-close semantics, its client side directory listing caches
can be stale, locking is iffy, etc. If you are using a single computer to
both write and read to/from the index, then it should be OK, though not
heavily tested, but if you share the mount across different computers
taking turns doing indexing and searching, there will be problems.
Finally, even if things are working OK over NFS, it will likely be very
slow performance, compared to a local mount.
All that being said, the known problems with NFS should not lead to this
corruption, I think. Something else is wrong. Maybe run an fsck on the
rocketsor disks? Maybe try eliminating variables, e.g. network card/cables
between your macmini and the rocketstor, swap in different hard drives in
the rocketstor, etc. Does the problem still happen with Lucene 6.1.0?
Mike McCandless
http://blog.mikemccandless.com
On Wed, Jul 20, 2016 at 6:57 AM, 董自鸣 <dz...@gmail.com> wrote:
> I use ubuntu to run lucene(java8, lucene6.0.0, not ES), and nfs to mount
> disk(actually is rocketstor) of macmini. I get this error constantly. I
> have already used org.apache.lucene.index.CheckIndex, but will still crash
> after restart program.
>
>
> Exception in thread "main" org.apache.lucene.store.AlreadyClosedException:
> this IndexWriter is closed at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724) at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738) at
> org.apache.lucene.index.IndexWriter.numDocs(IndexWriter.java:1198) at
> xxxx.xxxxx.search.XxxxxxxxIndexer.close(XxxxxxxxIndexer.java:184) at
>
> xxxx.xxxxx.search.ThreadedXxxxxxxxIndexer.close(ThreadedXxxxxxxxIndexer.java:59)
> at
>
> xxxx.xxxxx.search.ThreadedXxxxxxxxIndexer.main(ThreadedXxxxxxxxIndexer.java:136)
> Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
> (hardware problem?) : expected=51fbdb5c actual=6e964d17
>
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/mnt/HPT8_56T/xxxxxxxxx-index/index1/_mq.cfs")
> [slice=_mq_Lucene50_0.pos])) at
> org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:365) at
> org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:469)
> at
>
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.checkIntegrity(Lucene50PostingsReader.java:1286)
> at
>
> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:336)
> at
>
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:317)
> at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96) at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:211) at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:96) at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4099) at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3679) at
>
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
> at
>
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
>
> --
> *suiyuan*
>