You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jackrabbit.apache.org by Tobias Frech <to...@frech.info> on 2015/11/13 13:44:40 UTC

SegmentNotFoundException after online compaction with OOM-Error

Hi there!

We are using Adobe Experience Manager 6.0 which includes Apache OAK
1.0.x and are looking into ways to shrink the size of our repository.
Since offline compaction comes with additonal ops burden online
compaction seems to be a better solution for us.
I enabled online compaction on a test system and triggered a compaction
run. After a while the JVM (-Xmx6g) ran into an OutOfMemory-Error
(Heap). I tried to shutdown AEM cleanly but had to kill the process
after waiting for an hour or so.
Now after trying to start AEM again it does not fully start, some
service seems not to be started anymore. But no error message is logged
by AEM.

Trying to check the repository I ran the following command and got the
exception below:

[cq@xxxxxxx crx-quickstart]$ java -jar oak-run-1.0.19.jar backup
./repository/segmentstore /opt/cq/backup/
Apache Jackrabbit Oak 1.0.19
05:54:27.495 [main] WARN  o.a.j.o.p.segment.file.TarReader - Unexpected
size 1024 of tar file data01186a.tar
05:54:27.510 [main] WARN  o.a.j.o.p.segment.file.TarReader - Could not
find a valid tar index in [./repository/segmentstore/data01186a.tar],
recovering...
06:09:05.078 [main] ERROR o.a.j.o.p.segment.SegmentTracker - Segment not
found: 11f36041-cece-48d6-b95c-029be7c7a91c. Creation date delta is 23 ms.
org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException:
Segment 11f36041-cece-48d6-b95c-029be7c7a91c not found
	at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:855)
~[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:134)
~[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:101)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.java:52)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream.java:168)
[oak-run-1.0.19.jar:1.0.19]
	at com.google.common.io.ByteStreams.read(ByteStreams.java:828)
[oak-run-1.0.19.jar:1.0.19]
	at com.google.common.io.ByteSource.contentEquals(ByteSource.java:303)
[oak-run-1.0.19.jar:1.0.19]
	at com.google.common.io.ByteStreams.equal(ByteStreams.java:661)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:214)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:263)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:219)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor.access$100(Compactor.java:50)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAdded(Compactor.java:140)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:155)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:434)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java:93)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:98)
[oak-run-1.0.19.jar:1.0.19]
	at
org.apache.jackrabbit.oak.plugins.backup.FileStoreBackup.backup(FileStoreBackup.java:78)
[oak-run-1.0.19.jar:1.0.19]
	at org.apache.jackrabbit.oak.run.Main.backup(Main.java:193)
[oak-run-1.0.19.jar:1.0.19]
	at org.apache.jackrabbit.oak.run.Main.main(Main.java:119)
[oak-run-1.0.19.jar:1.0.19]
Exception in thread "main"
org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException:
Segment 11f36041-cece-48d6-b95c-029be7c7a91c not found
	at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:855)
	at
org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:134)
	at
org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:101)
	at
org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
	at
org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.java:52)
	at
org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream.java:168)
	at com.google.common.io.ByteStreams.read(ByteStreams.java:828)
	at com.google.common.io.ByteSource.contentEquals(ByteSource.java:303)
	at com.google.common.io.ByteStreams.equal(ByteStreams.java:661)
	at
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67)
	at
org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:214)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:263)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:219)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor.access$100(Compactor.java:50)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAdded(Compactor.java:140)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:155)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
	at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
	at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:434)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java:93)
	at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:98)
	at
org.apache.jackrabbit.oak.plugins.backup.FileStoreBackup.backup(FileStoreBackup.java:78)
	at org.apache.jackrabbit.oak.run.Main.backup(Main.java:193)
	at org.apache.jackrabbit.oak.run.Main.main(Main.java:119)


What can we do now to get the repository back into a clean state?

Thanks in advance,
Tobias Frech

-- 
Frech IT GmbH / Am Brünnele 7 / 71642 Ludwigsburg
phone : +49-(0)7141-9113037 / HR B 744851 / AG Stuttgart
--------------------------------------------------------
Geschäftsführer: Tobias Frech
mobile: +49-(0)172-7112352  / email: tobias@frech.info


Re: SegmentNotFoundException after online compaction with OOM-Error

Posted by Andrew Khoury <ak...@adobe.com>.
Hi Tobias,
I copied some documentation [1] below that was written by Michael Duerig.

If the steps below don¹t work for you then please submit an AEM support
ticket so we can have Adobe engineering team assist further.

[1]
Reverting the Oak repository to fix ³SegmentNotFoundException"
------------------------------------
The check run-mode [2] of oak-run can be used to determine the last known
good revision of a segment store, which can subequently be restored
manually. This utility can be used to revert a corrupt segment store to
its latest good revision change into Oak's working directory (the one
containing the crx-quickstart folder) and backup all files in
./crx-quickstart/repository/segmentstore/.

1. Run the consistency check,
java -jar oak-run-*.jar check -d1 --bin=-1 -p
crx-quickstart/repository/segmentstore/


it will search backwards through the revisions until it finds a consistent
one:
14:00:30.783 [main] INFO  o.a.j.o.p.s.f.t.ConsistencyChecker - Found
latest good revision afdb922d-ba53-4a1b-aa1b-1cb044b535cf:234880


2. Revert the repository to this revision by editing
./crx-quickstart/repository/segmentstore/journal.log and deleting all
lines after the line containing the latest good revision.

3. Remove all ./crx-quickstart/repository/segmentstore/*.bak files.
4. Run checkpoint clean-up to remove orphaned checkpoints:
java -jar oak-run-*.jar checkpoints
./crx-quickstart/repository/segmentstore rm-unreferenced

5. Finally compact the repository:
java -jar oak-run-*.jar compact ./crx-quickstart/repository/segmentstore/

[2] 
https://github.com/apache/jackrabbit-oak/blob/trunk/oak-run/README.md#check


Thanks,
Andrew

On 11/13/15, 4:44 AM, "Tobias Frech" <to...@frech.info> wrote:

>Hi there!
>
>We are using Adobe Experience Manager 6.0 which includes Apache OAK
>1.0.x and are looking into ways to shrink the size of our repository.
>Since offline compaction comes with additonal ops burden online
>compaction seems to be a better solution for us.
>I enabled online compaction on a test system and triggered a compaction
>run. After a while the JVM (-Xmx6g) ran into an OutOfMemory-Error
>(Heap). I tried to shutdown AEM cleanly but had to kill the process
>after waiting for an hour or so.
>Now after trying to start AEM again it does not fully start, some
>service seems not to be started anymore. But no error message is logged
>by AEM.
>
>Trying to check the repository I ran the following command and got the
>exception below:
>
>[cq@xxxxxxx crx-quickstart]$ java -jar oak-run-1.0.19.jar backup
>./repository/segmentstore /opt/cq/backup/
>Apache Jackrabbit Oak 1.0.19
>05:54:27.495 [main] WARN  o.a.j.o.p.segment.file.TarReader - Unexpected
>size 1024 of tar file data01186a.tar
>05:54:27.510 [main] WARN  o.a.j.o.p.segment.file.TarReader - Could not
>find a valid tar index in [./repository/segmentstore/data01186a.tar],
>recovering...
>06:09:05.078 [main] ERROR o.a.j.o.p.segment.SegmentTracker - Segment not
>found: 11f36041-cece-48d6-b95c-029be7c7a91c. Creation date delta is 23 ms.
>org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException:
>Segment 11f36041-cece-48d6-b95c-029be7c7a91c not found
>	at
>org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileS
>tore.java:855)
>~[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(Segmen
>tTracker.java:134)
>~[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.j
>ava:101)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82
>)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.jav
>a:52)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream
>.java:168)
>[oak-run-1.0.19.jar:1.0.19]
>	at com.google.common.io.ByteStreams.read(ByteStreams.java:828)
>[oak-run-1.0.19.jar:1.0.19]
>	at com.google.common.io.ByteSource.contentEquals(ByteSource.java:303)
>[oak-run-1.0.19.jar:1.0.19]
>	at com.google.common.io.ByteStreams.equal(ByteStreams.java:661)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.j
>ava:67)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.j
>ava:214)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java
>:263)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java
>:219)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor.access$100(Compactor.j
>ava:50)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAd
>ded(Compactor.java:140)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:155)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstB
>aseState(SegmentNodeState.java:434)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java
>:93)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java
>:98)
>[oak-run-1.0.19.jar:1.0.19]
>	at
>org.apache.jackrabbit.oak.plugins.backup.FileStoreBackup.backup(FileStoreB
>ackup.java:78)
>[oak-run-1.0.19.jar:1.0.19]
>	at org.apache.jackrabbit.oak.run.Main.backup(Main.java:193)
>[oak-run-1.0.19.jar:1.0.19]
>	at org.apache.jackrabbit.oak.run.Main.main(Main.java:119)
>[oak-run-1.0.19.jar:1.0.19]
>Exception in thread "main"
>org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException:
>Segment 11f36041-cece-48d6-b95c-029be7c7a91c not found
>	at
>org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileS
>tore.java:855)
>	at
>org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(Segmen
>tTracker.java:134)
>	at
>org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.j
>ava:101)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82
>)
>	at
>org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.jav
>a:52)
>	at
>org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream
>.java:168)
>	at com.google.common.io.ByteStreams.read(ByteStreams.java:828)
>	at com.google.common.io.ByteSource.contentEquals(ByteSource.java:303)
>	at com.google.common.io.ByteStreams.equal(ByteStreams.java:661)
>	at
>org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.j
>ava:67)
>	at
>org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.j
>ava:214)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java
>:263)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java
>:219)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor.access$100(Compactor.j
>ava:50)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAd
>ded(Compactor.java:140)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:155)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA
>dded(Compactor.java:167)
>	at
>org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt
>yState(EmptyNodeState.java:160)
>	at
>org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstB
>aseState(SegmentNodeState.java:434)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java
>:93)
>	at
>org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java
>:98)
>	at
>org.apache.jackrabbit.oak.plugins.backup.FileStoreBackup.backup(FileStoreB
>ackup.java:78)
>	at org.apache.jackrabbit.oak.run.Main.backup(Main.java:193)
>	at org.apache.jackrabbit.oak.run.Main.main(Main.java:119)
>
>
>What can we do now to get the repository back into a clean state?
>
>Thanks in advance,
>Tobias Frech
>
>-- 
>Frech IT GmbH / Am Brünnele 7 / 71642 Ludwigsburg
>phone : +49-(0)7141-9113037 / HR B 744851 / AG Stuttgart
>--------------------------------------------------------
>Geschäftsführer: Tobias Frech
>mobile: +49-(0)172-7112352  / email: tobias@frech.info
>