You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-issues@jackrabbit.apache.org by "Julian Reschke (JIRA)" <ji...@apache.org> on 2018/11/22 13:25:00 UTC

[jira] [Closed] (OAK-7755) deadlock TarMK flush, lucene

     [ https://issues.apache.org/jira/browse/OAK-7755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Julian Reschke closed OAK-7755.
-------------------------------

> deadlock TarMK flush, lucene
> ----------------------------
>
>                 Key: OAK-7755
>                 URL: https://issues.apache.org/jira/browse/OAK-7755
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: segment-tar
>    Affects Versions: 1.6.1
>         Environment: apache sling9, oak 1.6.1
>            Reporter: Guido Dunker
>            Assignee: Francesco Mari
>            Priority: Blocker
>              Labels: deadlock
>             Fix For: 1.6.15
>
>         Attachments: OAK-7755-01.patch
>
>
> We are getting the following deadlock. Please help! (Production environment)
> I have already annotated possible locks and synchronized blocks in between:
> {noformat}
> "TarMK flush [/opt/condat/epet9/sling/repository/segmentstore]":
>   waiting to lock Monitor@0x00007fedfc00cc28 (Object@0x00000004795519a8, a org/apache/jackrabbit/oak/segment/SegmentId),
>   which is held by "oak-lucene-14"
> "oak-lucene-14":
>  waiting for ownable synchronizer 0x00000003c13818c0, (a java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync),
>  which is held by "TarMK flush [/opt/condat/epet9/sling/repository/segmentstore]"
> Thread 28883: (state = BLOCKED)
>  - org.apache.jackrabbit.oak.segment.SegmentId.getSegment() @bci=12, line=121 (Compiled frame)
>         synchronized (this) 
>  - org.apache.jackrabbit.oak.segment.Record.getSegment() @bci=4, line=70 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.BlockRecord.read(int, byte[], int, int) @bci=49, line=57 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentStream.read(byte[], int, int) @bci=314, line=189 (Compiled frame)
>  - com.google.common.io.ByteStreams.read(java.io.InputStream, byte[], int, int) @bci=43, line=828 (Compiled frame)
>  - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[], int, int) @bci=4, line=695 (Compiled frame)
>  - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[]) @bci=5, line=676 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentStream.getString() @bci=93, line=103 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.Segment.readString(int) @bci=189, line=524 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(org.apache.jackrabbit.oak.segment.Segment, int) @bci=15, line=212 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentBlob.readBlobId(org.apache.jackrabbit.oak.segment.Segment, int) @bci=37, line=167 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.AbstractFileStore$4.consume(int, org.apache.jackrabbit.oak.segment.RecordType, int) @bci=24, line=354 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.Segment.forEachRecord(org.apache.jackrabbit.oak.segment.Segment$RecordConsumer) @bci=48, line=716 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.AbstractFileStore.populateTarBinaryReferences(org.apache.jackrabbit.oak.segment.Segment, org.apache.jackrabbit.oak.segment.file.TarWriter) @bci=25, line=349 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore.writeSegment(org.apache.jackrabbit.oak.segment.SegmentId, byte[], int, int) @bci=136, line=657 (Compiled frame)
>         fileStoreLock.writeLock().lock(); Zeile 639
>         bis: populateTarBinaryReferences
>  - org.apache.jackrabbit.oak.segment.SegmentBufferWriter.flush() @bci=383, line=383 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.flush() @bci=165, line=148 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentWriter.flush() @bci=4, line=143 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore$7.call() @bci=7, line=373 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore$7.call() @bci=1, line=370 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.TarRevisions.doFlush(java.util.concurrent.Callable) @bci=25, line=224 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.TarRevisions.flush(java.util.concurrent.Callable) @bci=42, line=212 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore.flush() @bci=20, line=370 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore$2.run() @bci=15, line=233 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.SafeRunnable.run() @bci=21, line=67 (Compiled frame)
>  - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Compiled frame)
>  - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1, line=180 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=37, line=294 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
> Locked ownable synchronizers:
>     - <0x00000003c1361228>, (a java/util/concurrent/locks/ReentrantLock$NonfairSync)
>     - <0x00000003c13818c0>, (a java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync)
>     - <0x00000003c1c3a0d8>, (a java/util/concurrent/ThreadPoolExecutor$Worker)
> Thread 31035: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int) @bci=83, line=967 (Interpreted frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int) @bci=10, line=1283 (Compiled frame)
>  - java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock() @bci=5, line=727 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore$8.call() @bci=158, line=567 (Compiled frame)
>     fileStoreLock.readLock().lock();
>  - org.apache.jackrabbit.oak.segment.file.FileStore$8.call() @bci=1, line=542 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(org.apache.jackrabbit.oak.segment.SegmentId, java.util.concurrent.Callable) @bci=1, line=95 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(org.apache.jackrabbit.oak.segment.SegmentId) @bci=14, line=542 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentId.getSegment() @bci=38, line=125 (Compiled frame)
>     synchronized (this) 
>  - org.apache.jackrabbit.oak.segment.Record.getSegment() @bci=4, line=70 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.BlockRecord.read(int, byte[], int, int) @bci=49, line=57 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentStream.read(byte[], int, int) @bci=314, line=189 (Compiled frame)
>  - com.google.common.io.ByteStreams.read(java.io.InputStream, byte[], int, int) @bci=64, line=833 (Compiled frame)
>  - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[], int, int) @bci=4, line=695 (Compiled frame)
>  - com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[]) @bci=5, line=676 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentStream.getString() @bci=93, line=103 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.Segment.readString(int) @bci=189, line=524 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(org.apache.jackrabbit.oak.segment.Segment, int) @bci=15, line=212 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentBlob.length() @bci=124, line=115 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.<init>(java.lang.String, org.apache.jackrabbit.oak.spi.state.NodeBuilder, java.lang.String, org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$BlobFactory) @bci=204, line=409 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.<init>(java.lang.String, org.apache.jackrabbit.oak.spi.state.NodeBuilder, java.lang.String, org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$BlobFactory) @bci=25, line=589 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory.fileLength(java.lang.String) @bci=64, line=176 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.copyFilesToLocal(org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory$CORFileReference, boolean, boolean) @bci=195, line=214 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.prefetchIndexFiles() @bci=96, line=170 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.<init>(org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier, org.apache.lucene.store.Directory, org.apache.lucene.store.Directory, boolean, java.lang.String, java.util.concurrent.Executor) @bci=85, line=81 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier.wrapForRead(java.lang.String, org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition, org.apache.lucene.store.Directory, java.lang.String) @bci=35, line=122 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.reader.DefaultIndexReaderFactory.createReader(org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition, org.apache.jackrabbit.oak.spi.state.NodeState, java.lang.String, java.lang.String, java.lang.String) @bci=61, line=102 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.reader.DefaultIndexReaderFactory.createReaders(org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition, org.apache.jackrabbit.oak.spi.state.NodeState, java.lang.String) @bci=20, line=61 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(java.lang.String, org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.plugins.index.lucene.reader.LuceneIndexReaderFactory, org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.NRTIndexFactory) @bci=17, line=68 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker$1.leave(org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState) @bci=30, line=132 (Compiled frame)
>  - org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(java.lang.String, org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState) @bci=66, line=153 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.MapRecord.compare(org.apache.jackrabbit.oak.segment.MapRecord, org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=197, line=415 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=909, line=608 (Compiled frame)
>  - org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(java.lang.String, org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState) @bci=43, line=148 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.MapRecord.compare(org.apache.jackrabbit.oak.segment.MapRecord, org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=400, line=457 (Compiled frame)
>  - org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=909, line=608 (Compiled frame)
>  - org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(org.apache.jackrabbit.oak.spi.commit.Editor, org.apache.jackrabbit.oak.spi.state.NodeState, org.apache.jackrabbit.oak.spi.state.NodeState) @bci=34, line=52 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.diffAndUpdate(org.apache.jackrabbit.oak.spi.state.NodeState) @bci=140, line=142 (Compiled frame)
>  - org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.update(org.apache.jackrabbit.oak.spi.state.NodeState) @bci=36, line=113 (Compiled frame)
>  - org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call() @bci=79, line=135 (Compiled frame)
>  - org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call() @bci=1, line=128 (Compiled frame)
>  - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=748 (Compiled frame)
> Locked ownable synchronizers:
>     - <0x0000000476194f30>, (a java/util/concurrent/ThreadPoolExecutor$Worker)   
> {noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)