You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-issues@jackrabbit.apache.org by "Julian Sedding (Jira)" <ji...@apache.org> on 2022/08/24 12:00:00 UTC
[jira] [Comment Edited] (OAK-9785) Tar SegmentStore can be corrupted during compaction
[ https://issues.apache.org/jira/browse/OAK-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584215#comment-17584215 ]
Julian Sedding edited comment on OAK-9785 at 8/24/22 11:59 AM:
---------------------------------------------------------------
[~adulceanu] unfortunately I had a hardware failure a little while ago and I lost the original logs. However, if you look at the last stack-trace in the attached log file, you'll notice that [{{AbstractBlob.equals()}} wraps an {{IOException}} in an {{IllegalStateException}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9/oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/memory/AbstractBlob.java#L68] before throwing it. The thread-name has then changed to {{sling-oak-observation-7}}, but a look into [{{SafeRunnable}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/SafeRunnable.java#L63-L74] easily explains this. The {{SafeRunnable}}, [initialized in the {{FileStore}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L278-L284], would never catch an exception if compaction was properly aborted.
{noformat}
09.06.2022 02:00:18.755 *ERROR* [sling-oak-observation-7] org.apache.jackrabbit.oak.segment.file.SafeRunnable Uncaught exception in TarMK revision gc [/opt/aem/crx-quickstart/repository/segmentstore]
java.lang.IllegalStateException: Blob equality check failed
at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:68) [org.apache.jackrabbit.oak-store-spi:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:205) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at com.google.common.base.Objects.equal(Objects.java:55)
at org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:59) [org.apache.jackrabbit.oak-store-spi:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:664) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:558) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:598) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:444) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord$3.childNodeChanged(MapRecord.java:430) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compareBranch(MapRecord.java:595) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:420) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:444) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CheckpointCompactor.compactWithCache(CheckpointCompactor.java:249) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CheckpointCompactor.compact(CheckpointCompactor.java:172) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CheckpointCompactor.compact(CheckpointCompactor.java:119) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.AbstractCompactionStrategy.compact(AbstractCompactionStrategy.java:150) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.TailCompactionStrategy.compact(TailCompactionStrategy.java:51) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.FallbackCompactionStrategy.compact(FallbackCompactionStrategy.java:37) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.CleanupFirstCompactionStrategy.compact(CleanupFirstCompactionStrategy.java:149) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.AbstractGarbageCollectionStrategy.run(AbstractGarbageCollectionStrategy.java:134) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.CleanupFirstGarbageCollectionStrategy.run(CleanupFirstGarbageCollectionStrategy.java:53) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.AbstractGarbageCollectionStrategy.collectTailGarbage(AbstractGarbageCollectionStrategy.java:72) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.AbstractGarbageCollectionStrategy.collectGarbage(AbstractGarbageCollectionStrategy.java:58) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.SynchronizedGarbageCollectionStrategy.collectGarbage(SynchronizedGarbageCollectionStrategy.java:38) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.GarbageCollector.run(GarbageCollector.java:281) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.FileStore.lambda$getGCRunner$7(FileStore.java:280) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.SafeRunnable.run(SafeRunnable.java:67) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.spi.state.RevisionGC$1.call(RevisionGC.java:99) [org.apache.jackrabbit.oak-store-spi:1.22.9]
at org.apache.jackrabbit.oak.spi.state.RevisionGC$1.call(RevisionGC.java:96) [org.apache.jackrabbit.oak-store-spi:1.22.9]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record 72755da5cfc4eb9f973133795016dcfe385ef2b0913e93760fdce8acd3a64d38 does not exist
at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob$1.read(BlobStoreBlob.java:57) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
at java.io.InputStream.read(InputStream.java:170)
at com.google.common.io.ByteStreams.read(ByteStreams.java:828)
at com.google.common.io.ByteSource.contentEquals(ByteSource.java:303)
at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:66) [org.apache.jackrabbit.oak-store-spi:1.22.9]
... 86 common frames omitted
Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record 72755da5cfc4eb9f973133795016dcfe385ef2b0913e93760fdce8acd3a64d38 does not exist
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:874) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:448) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:50) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:263) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:84) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.openStream(AbstractBlob.java:43) [org.apache.jackrabbit.oak-store-spi:1.22.9]
at com.google.common.io.ByteSource.contentEquals(ByteSource.java:300)
... 87 common frames omitted
Caused by: org.apache.jackrabbit.core.data.DataStoreException: Record 72755da5cfc4eb9f973133795016dcfe385ef2b0913e93760fdce8acd3a64d38 does not exist
at org.apache.jackrabbit.core.data.AbstractDataStore.getRecord(AbstractDataStore.java:59) [org.apache.jackrabbit.jackrabbit-data:2.20.2]
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:883) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:868) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
... 93 common frames omitted
{noformat}
was (Author: jsedding):
[~adulceanu] unfortunately I had a hardware failure a little while ago and I lost the original logs. However, if you look at the last stack-trace in the attached log file, you'll notice that [{{AbstractBlob.equals()}} wraps an {{IOException}} in an {{IllegalStateException}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9//oak-store-spi/src/main/java/org/apache/jackrabbit/oak/plugins/memory/AbstractBlob.java#L68] before throwing it. The thread-name has then changed to {{sling-oak-observation-7}}, but a look into [{{SafeRunnable}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9//oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/SafeRunnable.java#L63-L74] easily explains this. The {{SafeRunnable}}, [initialized in the {{FileStore}}|https://github.com/apache/jackrabbit-oak/blob/jackrabbit-oak-1.22.9/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/FileStore.java#L278-L284], would never catch an exception if compaction was properly aborted.
{noformat}
09.06.2022 02:00:18.755 *ERROR* [sling-oak-observation-7] org.apache.jackrabbit.oak.segment.file.SafeRunnable Uncaught exception in TarMK revision gc [/opt/aem/crx-quickstart/repository/segmentstore]
java.lang.IllegalStateException: Blob equality check failed
at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:68) [org.apache.jackrabbit.oak-store-spi:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:205) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at com.google.common.base.Objects.equal(Objects.java:55)
at org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:59) [org.apache.jackrabbit.oak-store-spi:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:664) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:558) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:598) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:444) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord$3.childNodeChanged(MapRecord.java:430) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compareBranch(MapRecord.java:595) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:420) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:444) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.childNodeChanged(Compactor.java:217) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:517) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor$CompactDiff.diff(Compactor.java:165) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.Compactor.compact(Compactor.java:123) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CheckpointCompactor.compactWithCache(CheckpointCompactor.java:249) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CheckpointCompactor.compact(CheckpointCompactor.java:172) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.CheckpointCompactor.compact(CheckpointCompactor.java:119) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.AbstractCompactionStrategy.compact(AbstractCompactionStrategy.java:150) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.TailCompactionStrategy.compact(TailCompactionStrategy.java:51) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.FallbackCompactionStrategy.compact(FallbackCompactionStrategy.java:37) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.CleanupFirstCompactionStrategy.compact(CleanupFirstCompactionStrategy.java:149) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.AbstractGarbageCollectionStrategy.run(AbstractGarbageCollectionStrategy.java:134) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.CleanupFirstGarbageCollectionStrategy.run(CleanupFirstGarbageCollectionStrategy.java:53) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.AbstractGarbageCollectionStrategy.collectTailGarbage(AbstractGarbageCollectionStrategy.java:72) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.AbstractGarbageCollectionStrategy.collectGarbage(AbstractGarbageCollectionStrategy.java:58) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.SynchronizedGarbageCollectionStrategy.collectGarbage(SynchronizedGarbageCollectionStrategy.java:38) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.GarbageCollector.run(GarbageCollector.java:281) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.FileStore.lambda$getGCRunner$7(FileStore.java:280) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.file.SafeRunnable.run(SafeRunnable.java:67) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.spi.state.RevisionGC$1.call(RevisionGC.java:99) [org.apache.jackrabbit.oak-store-spi:1.22.9]
at org.apache.jackrabbit.oak.spi.state.RevisionGC$1.call(RevisionGC.java:96) [org.apache.jackrabbit.oak-store-spi:1.22.9]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record 72755da5cfc4eb9f973133795016dcfe385ef2b0913e93760fdce8acd3a64d38 does not exist
at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob$1.read(BlobStoreBlob.java:57) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
at java.io.InputStream.read(InputStream.java:170)
at com.google.common.io.ByteStreams.read(ByteStreams.java:828)
at com.google.common.io.ByteSource.contentEquals(ByteSource.java:303)
at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:66) [org.apache.jackrabbit.oak-store-spi:1.22.9]
... 86 common frames omitted
Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record 72755da5cfc4eb9f973133795016dcfe385ef2b0913e93760fdce8acd3a64d38 does not exist
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:874) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:448) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:50) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:263) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:84) [org.apache.jackrabbit.oak-segment-tar:1.22.9]
at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.openStream(AbstractBlob.java:43) [org.apache.jackrabbit.oak-store-spi:1.22.9]
at com.google.common.io.ByteSource.contentEquals(ByteSource.java:300)
... 87 common frames omitted
Caused by: org.apache.jackrabbit.core.data.DataStoreException: Record 72755da5cfc4eb9f973133795016dcfe385ef2b0913e93760fdce8acd3a64d38 does not exist
at org.apache.jackrabbit.core.data.AbstractDataStore.getRecord(AbstractDataStore.java:59) [org.apache.jackrabbit.jackrabbit-data:2.20.2]
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:883) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:868) [org.apache.jackrabbit.oak-blob-plugins:1.22.9]
... 93 common frames omitted
{noformat}
> Tar SegmentStore can be corrupted during compaction
> ---------------------------------------------------
>
> Key: OAK-9785
> URL: https://issues.apache.org/jira/browse/OAK-9785
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segment-tar
> Affects Versions: 1.42.0
> Reporter: Julian Sedding
> Priority: Major
> Attachments: error.log.2022-06-09
>
>
> There is a scenario where a segment store can become corrupted, leading to {{SegmentNotFoundExceptions}} with very "young" {{SegmentIds}}, i.e. in the 1-2 digit millisecond range. E.g. {{SegmentId age=2ms}}.
> The scenario I observed looks as follows:
> - a blob is "lost" from the external blob store (presumably due to incorrect cloning of the instance, most likely only happens with unfortunate timing)
> - a tail revision GC run is performed (not sure if it matters that this was a tail compaction)
> -- the missing blob is encountered during compaction
> -- an exception other than an {{IOException}} (IIRC it was a {{{}IllegalArgumentException{}}}) is thrown due to the missing blob
> -- revision GC fails WITHOUT properly being aborted, and thus the partially written revision of the compaction run is not removed
> - more data is written on the instance
> - a full revision GC run is performed
> -- a referenced segment is removed due to the incorrect/confused revision data
> - the {{SegmentNotFoundException}} is first observed either during the remainder of the compaction run or when the respective node is requested the next time, usually during a traversal
> The root cause is in [{{AbstractCompactionStrategy}}|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/file/AbstractCompactionStrategy.java#L233], where only {{IOExceptions}} are caught.
> In order to improve the robustness of the code, I think we need to catch all {{Throwables}}. Otherwise we cannot guarantee that compaction is correctly aborted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)