You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "Hugo Arès (JIRA)" <ji...@apache.org> on 2017/10/25 17:07:00 UTC
[jira] [Comment Edited] (SSHD-779) Deadlock in ChannelOutputStream
[ https://issues.apache.org/jira/browse/SSHD-779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16219087#comment-16219087 ]
Hugo Arès edited comment on SSHD-779 at 10/25/17 5:06 PM:
----------------------------------------------------------
When this happens, the session hangs forever and won't unblock since the 2 threads are in a deadlock. This is only happening sometimes which can probably be explained by the fact that both thread must be at specific place for the deadlock to happen.
In the example I gave, thread "sshd-SshServer[4c110f87]-nio2-thread-20" locked object 0x00007fb450addb38(AbstractSession) and it trying to lock object 0x00007fb450ade608 (ChannelOutputStream). Thread "SSH-Stream-Worker-20" locked object 0x00007fb450ade608 (ChannelOutputStream) and trying to object 0x00007fb450addb38(AbstractSession).
Each thread is waiting to lock an object which is locked by the other.
This happened under heavy load but we did use the NIO2 backend long enough to confirm if it's happening without heavy load. As soon as we saw the deadlocks, we tried the MINA backend and problem did not happen again.
was (Author: hugares):
When this happens, the session hangs forever and won't unblock since the 2 threads are in a deadlock. This is only happening sometimes which can probably be explained by the fact that both thread must be at specific place for the deadlock to happen.
In the example I gave, thread "sshd-SshServer[4c110f87]-nio2-thread-20" locked object 0x00007fb450addb38(AbstractSession) and it trying to lock object 0x00007fb450ade608 (ChannelOutputStream). Thread "SSH-Stream-Worker-20" locked object 0x00007fb450ade608 (ChannelOutputStream) and trying to object 0x00007fb450addb38(AbstractSession).
Each thread is waiting to lock an object which is locked by the other.
> Deadlock in ChannelOutputStream
> -------------------------------
>
> Key: SSHD-779
> URL: https://issues.apache.org/jira/browse/SSHD-779
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 1.4.0, 1.6.0
> Reporter: Hugo Arès
>
> Since Gerrit upgraded to 1.4.0, the following deadlock is happening with NI02:
> Found one Java-level deadlock:
> =============================
> "SSH-Stream-Worker-20":
> waiting to lock monitor 0x00007fac34021cf8 (object 0x00007fb450addb38, a java.lang.Object),
> which is held by "sshd-SshServer[4c110f87]-nio2-thread-20"
> "sshd-SshServer[4c110f87]-nio2-thread-20":
> waiting to lock monitor 0x00007fad20048678 (object 0x00007fb450ade608, a org.apache.sshd.common.channel.ChannelOutputStream),
> which is held by "SSH-Stream-Worker-20"
> Java stack information for the threads listed above:
> ===================================================
> "SSH-Stream-Worker-20":
> at org.apache.sshd.common.session.helpers.AbstractSession.doWritePacket(AbstractSession.java:1056)
> - waiting to lock <0x00007fb450addb38> (a java.lang.Object)
> at org.apache.sshd.common.session.helpers.AbstractSession.writePacket(AbstractSession.java:1005)
> at org.apache.sshd.common.channel.AbstractChannel.writePacket(AbstractChannel.java:766)
> at org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:219)
> - locked <0x00007fb450ade608> (a org.apache.sshd.common.channel.ChannelOutputStream)
> at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
> at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
> - locked <0x00007fb450ae2eb0> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
> at java.io.BufferedWriter.flush(BufferedWriter.java:254)
> - locked <0x00007fb450ae2eb0> (a java.io.OutputStreamWriter)
> at java.io.PrintWriter.flush(PrintWriter.java:320)
> - locked <0x00007fb450ae2e70> (a java.io.BufferedWriter)
> at java.io.PrintWriter.checkError(PrintWriter.java:357)
> at com.google.gerrit.sshd.commands.StreamEvents.writeEvents(StreamEvents.java:234)
> at com.google.gerrit.sshd.commands.StreamEvents.access$000(StreamEvents.java:53)
> at com.google.gerrit.sshd.commands.StreamEvents$1.run(StreamEvents.java:100)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:418)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> "sshd-SshServer[4c110f87]-nio2-thread-20":
> at org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:241)
> - waiting to lock <0x00007fb450ade608> (a org.apache.sshd.common.channel.ChannelOutputStream)
> at org.apache.sshd.common.util.io.IoUtils.closeQuietly(IoUtils.java:137)
> at org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:201)
> at org.apache.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:81)
> at org.apache.sshd.common.channel.AbstractChannel.close(AbstractChannel.java:548)
> at org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(ParallelCloseable.java:61)
> at org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63)
> at org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:45)
> at org.apache.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:81)
> at org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(ParallelCloseable.java:61)
> at org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63)
> at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:55)
> at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:45)
> at org.apache.sshd.common.util.closeable.SequentialCloseable.doClose(SequentialCloseable.java:68)
> at org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63)
> at org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:45)
> at org.apache.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:81)
> at org.apache.sshd.common.session.helpers.AbstractSession.exceptionCaught(AbstractSession.java:862)
> at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.exceptionCaught(AbstractSessionIoHandler.java:56)
> at org.apache.sshd.common.io.nio2.Nio2Session.exceptionCaught(Nio2Session.java:164)
> at org.apache.sshd.common.io.nio2.Nio2Session.handleWriteCycleFailure(Nio2Session.java:386)
> at org.apache.sshd.common.io.nio2.Nio2Session$2.onFailed(Nio2Session.java:348)
> at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$failed$1(Nio2CompletionHandler.java:46)
> at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$318/1992024659.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:45)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:128)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
> at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
> at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
> at org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:334)
> at org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:318)
> at org.apache.sshd.common.io.nio2.Nio2Session.write(Nio2Session.java:148)
> at org.apache.sshd.common.session.helpers.AbstractSession.doWritePacket(AbstractSession.java:1063)
> - locked <0x00007fb450addb38> (a java.lang.Object)
> at org.apache.sshd.common.session.helpers.AbstractSession.writePacket(AbstractSession.java:1005)
> at org.apache.sshd.common.session.helpers.AbstractConnectionService.sendGlobalResponse(AbstractConnectionService.java:694)
> at org.apache.sshd.common.session.helpers.AbstractConnectionService.globalRequest(AbstractConnectionService.java:662)
> at org.apache.sshd.common.session.helpers.AbstractConnectionService.process(AbstractConnectionService.java:353)
> at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:564)
> at org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:497)
> - locked <0x00007fb450ad9c48> (a java.lang.Object)
> at org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1406)
> at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:458)
> - locked <0x00007fb450addb48> (a java.lang.Object)
> at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:67)
> at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:277)
> at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:257)
> at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:254)
> at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
> at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$182/2116066305.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker$2.run(Invoker.java:218)
> at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)