You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-issues@jackrabbit.apache.org by "Thomas Mueller (JIRA)" <ji...@apache.org> on 2014/09/15 15:40:33 UTC
[jira] [Comment Edited] (OAK-2049) ArrayIndexOutOfBoundsException
in Segment.getRefId()
[ https://issues.apache.org/jira/browse/OAK-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133891#comment-14133891 ]
Thomas Mueller edited comment on OAK-2049 at 9/15/14 1:40 PM:
--------------------------------------------------------------
Possibly the problem is in SegmentWriter.prepare, where "rootcount--" is called in a loop. In cases where the same record is referenced multiple times from the current record, and this record was so far a root record (for example the last record that was written, but not strictly necessary), I think this can end up with a wrong "rootcount", such that the segment is not flushed even thought it should be.
Proposed patch (just the fix, no test case yet):
{noformat}
Index: src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentWriter.java
===================================================================
--- src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentWriter.java (revision 1624000)
+++ src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentWriter.java (working copy)
@@ -46,6 +46,7 @@
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
+import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -291,14 +292,32 @@
refcount -= idcount;
Set<SegmentId> segmentIds = newIdentityHashSet();
+
+ // The set of old record ids in this segment
+ // that were previously root record ids, but will no longer be,
+ // because the record to be written references them.
+ // This needs to be a set, because the list of ids can
+ // potentially reference the same record multiple times
+ Set<RecordId> notRoots = new HashSet<RecordId>();
+
for (RecordId recordId : ids) {
SegmentId segmentId = recordId.getSegmentId();
if (segmentId != segment.getSegmentId()) {
segmentIds.add(segmentId);
} else if (roots.containsKey(recordId)) {
- rootcount--;
+
+ // log to verify OAK-2049 was hit
+ // TODO this can be removed later on
+ if (log.isDebugEnabled()) {
+ if (notRoots.contains(recordId)) {
+ log.debug("Multiple pointers to the same record");
+ }
+ }
+
+ notRoots.add(recordId);
}
}
+ rootcount -= notRoots.size();
if (!segmentIds.isEmpty()) {
for (int refid = 1; refid < refcount; refid++) {
segmentIds.remove(segment.getRefId(refid));
{noformat}
was (Author: tmueller):
Possibly the problem is in SegmentWriter.prepare, where "rootcount--" is called in a loop. In cases where the same record is referenced multiple times from the current record, and this record was written just before (the last record that was written), I think this can end up with a wrong "rootcount", such that the segment is not flushed even thought it should be.
Proposed patch (just the fix, no test case yet):
{noformat}
Index: src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentWriter.java
===================================================================
--- src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentWriter.java (revision 1624000)
+++ src/main/java/org/apache/jackrabbit/oak/plugins/segment/SegmentWriter.java (working copy)
@@ -46,6 +46,7 @@
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
+import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -291,14 +292,32 @@
refcount -= idcount;
Set<SegmentId> segmentIds = newIdentityHashSet();
+
+ // The set of old record ids in this segment
+ // that were previously root record ids, but will no longer be,
+ // because the record to be written references them.
+ // This needs to be a set, because the list of ids can
+ // potentially reference the same record multiple times
+ Set<RecordId> notRoots = new HashSet<RecordId>();
+
for (RecordId recordId : ids) {
SegmentId segmentId = recordId.getSegmentId();
if (segmentId != segment.getSegmentId()) {
segmentIds.add(segmentId);
} else if (roots.containsKey(recordId)) {
- rootcount--;
+
+ // log to verify OAK-2049 was hit
+ // TODO this can be removed later on
+ if (log.isDebugEnabled()) {
+ if (notRoots.contains(recordId)) {
+ log.debug("Multiple pointers to the same record");
+ }
+ }
+
+ notRoots.add(recordId);
}
}
+ rootcount -= notRoots.size();
if (!segmentIds.isEmpty()) {
for (int refid = 1; refid < refcount; refid++) {
segmentIds.remove(segment.getRefId(refid));
{noformat}
> ArrayIndexOutOfBoundsException in Segment.getRefId()
> ----------------------------------------------------
>
> Key: OAK-2049
> URL: https://issues.apache.org/jira/browse/OAK-2049
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: core
> Affects Versions: 1.1, 1.0.4
> Reporter: Andrew Khoury
> Assignee: Jukka Zitting
> Priority: Critical
> Fix For: 1.0.6
>
>
> It looks like there is some SegmentMK bug that causes the {{Segment.getRefId()}} to throw an {{ArrayIndexOutOfBoundsException}} in some fairly rare corner cases.
> The data was originally migrated into oak via the crx2oak tool mentioned here: http://docs.adobe.com/docs/en/aem/6-0/deploy/upgrade.html
> That tool uses *oak-core-1.0.0* creating an oak instance.
> Similar to OAK-1566 this system was using FileDataStore with SegmentNodeStore.
> In this case the error is seen when running offline compaction using oak-run-1.1-SNAPSHOT.jar (latest).
> {code:none}
> > java -Xmx4096m -jar oak-run-1.1-SNAPSHOT.jar compact /oak/crx-quickstart/repository/segmentstore
> Apache Jackrabbit Oak 1.1-SNAPSHOT
> Compacting /wcm/cq-author/crx-quickstart/repository/segmentstore
> before [data00055a.tar, data00064a.tar, data00045b.tar, data00005a.tar, data00018a.tar, data00022a.tar, data00047a.tar, data00037a.tar, data00049a.tar, data00014a.tar, data00066a.tar, data00020a.tar, data00058a.tar, data00065a.tar, data00069a.tar, data00012a.tar, data00009a.tar, data00060a.tar, data00041a.tar, data00016a.tar, data00072a.tar, data00048a.tar, data00061a.tar, data00053a.tar, data00038a.tar, data00001a.tar, data00034a.tar, data00003a.tar, data00052a.tar, data00006a.tar, data00027a.tar, data00031a.tar, data00056a.tar, data00035a.tar, data00063a.tar, data00068a.tar, data00008v.tar, data00010a.tar, data00043b.tar, data00021a.tar, data00017a.tar, data00024a.tar, data00054a.tar, data00051a.tar, data00057a.tar, data00059a.tar, data00036a.tar, data00033a.tar, data00019a.tar, data00046a.tar, data00067a.tar, data00004a.tar, data00044a.tar, data00013a.tar, data00070a.tar, data00026a.tar, data00002a.tar, data00011a.tar, journal.log, data00030a.tar, data00042a.tar, data00025a.tar, data00062a.tar, data00023a.tar, data00071a.tar, data00032b.tar, data00040a.tar, data00015a.tar, data00029a.tar, data00050a.tar, data00000a.tar, data00007a.tar, data00028a.tar, data00039a.tar]
> -> compacting
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 206
> at org.apache.jackrabbit.oak.plugins.segment.Segment.getRefId(Segment.java:191)
> at org.apache.jackrabbit.oak.plugins.segment.Segment.internalReadRecordId(Segment.java:299)
> at org.apache.jackrabbit.oak.plugins.segment.Segment.readRecordId(Segment.java:295)
> at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getTemplateId(SegmentNodeState.java:69)
> at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getTemplate(SegmentNodeState.java:78)
> at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getProperties(SegmentNodeState.java:150)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:154)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:124)
> at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
> at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:395)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java:80)
> at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:85)
> at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.compact(FileStore.java:438)
> at org.apache.jackrabbit.oak.run.Main.compact(Main.java:311)
> at org.apache.jackrabbit.oak.run.Main.main(Main.java:133)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)