Oak - Corrupt lucene index update

[Roy Teeuwen <ro...@teeuwen.be>]

Hey all,

We are facing the a reindex issue with with Oak (1.4.8) when the async reindex gets triggered, this has been going on for 3 days now and because of this the datastore has grown to 1TB of data because it keeps on trying to reindex the Lucene index. Could anyone help me in what would be the best approach to fix this? The repo itself is 100GB big, so we are trying to stall deleting the entire index because it would take one day for an entire reindex, making the production environment unaccessible, while not even being sure if that would fix it.

The exception being thrown:

20.09.2017 09:08:07.348 *WARN* [aysnc-index-update-async] org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate [async] The index update is still failing
org.apache.jackrabbit.oak.api.CommitFailedException: OakLucene0005: Failed to remove the index entries of the removed subtree /var/eventing/distribution/7ac221b3-0775-4072-a764-c396707c38bf/2017/9/14
    at org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.childNodeDeleted(LuceneIndexEditor.java:281)
    at org.apache.jackrabbit.oak.spi.commit.CompositeEditor.childNodeDeleted(CompositeEditor.java:135)
    at org.apache.jackrabbit.oak.spi.commit.VisibleEditor.childNodeDeleted(VisibleEditor.java:114)
    at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:166)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:515)
    at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:531)
    at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:531)
    at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:531)
    at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
    at org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:399)
    at org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:440)
    at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:483)
    at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:432)
    at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:390)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:584)
    at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
    at org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:399)
    at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:456)
    at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:390)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:584)
    at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
    at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:456)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:584)
    at org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
    at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:588)
    at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:463)
    at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:345)
    at org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:118)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException: reached end of stream after reading 0 bytes; 1047552 bytes expected
    at com.google.common.io.ByteStreams.readFully(ByteStreams.java:697)
    at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.loadBlob(OakDirectory.java:360)
    at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.readBytes(OakDirectory.java:413)
    at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.readBytes(OakDirectory.java:532)
    at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.readByte(OakDirectory.java:539)
    at org.apache.lucene.store.DataInput.readVInt(DataInput.java:108)
    at org.apache.lucene.store.CompoundFileDirectory.readEntries(CompoundFileDirectory.java:137)
    at org.apache.lucene.store.CompoundFileDirectory.<init>(CompoundFileDirectory.java:104)
    at org.apache.lucene.index.SegmentReader.readFieldInfos(SegmentReader.java:202)
    at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:818)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:770)
    at org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext.getWriter(LuceneIndexEditorContext.java:199)
    at org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.childNodeDeleted(LuceneIndexEditor.java:275)
    ... 34 common frames omitted

Thanks!
Roy