You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "Roberto Deandrea (JIRA)" <ji...@apache.org> on 2018/09/17 10:08:00 UTC
[jira] [Created] (SSHD-844) Deadlock detected in Apache SSHD client
code
Roberto Deandrea created SSHD-844:
-------------------------------------
Summary: Deadlock detected in Apache SSHD client code
Key: SSHD-844
URL: https://issues.apache.org/jira/browse/SSHD-844
Project: MINA SSHD
Issue Type: Bug
Affects Versions: 2.0.1
Environment: AIX 7.2
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 8.0.5.7 - pap6480sr5fp7-20171216_01(SR5 FP7))
IBM J9 VM (build 2.9, JRE 1.8.0 AIX ppc64-64 Compressed References 20171215_373586 (JIT enabled, AOT enabled)
OpenJ9 - 5aa401f
OMR - 101e793
IBM - b4a79bf)
JCL - 20171214_01 based on Oracle jdk8u151-b12
Reporter: Roberto Deandrea
Attachments: javacore.20180913.153332.14221746.0001.txt, jca457.jar
I found a deadlock in Apache SSHD client code running integration test for a local port forwarding application. Attached the IBM javacore with information about the deadlock.
Please run $java -jar jca457.jar to load the javacore file and display thread stack traces.
[^javacore.20180913.153332.14221746.0001.txt]
[^jca457.jar]
There are 2 threads in DEADLOCK belonging to client pool's threads, that blocked the other pool threads from working.
The locks involved are LOCK1 (object ChannelOutputStream) and LOCK2 Object encodeLock of AbstractSession class.
The below thread is waiting on LOCK2 and owns LOCK1 (that is ChannelOutputStream object)
The thread sshd-SshClient[61e9f863]-nio2-thread-6 has the following stack trace :
at org/apache/sshd/common/session/helpers/AbstractSession.doWritePacket(AbstractSession.java:1189(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.writePacket(AbstractSession.java:1137(Compiled Code))
at org/apache/sshd/common/channel/AbstractChannel.writePacket(AbstractChannel.java:773(Compiled Code))
at org/apache/sshd/common/channel/ChannelOutputStream.flush(ChannelOutputStream.java:227(Compiled Code))
at org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.messageReceived(DefaultForwardingFilter.java:1063(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2Session.handleReadCycleCompletion(Nio2Session.java:339(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:318(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:315(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$254/0000000023F76250.run(Bytecode PC:12(Compiled Code))
at java/security/AccessController.doPrivileged(AccessController.java:638(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37(Compiled Code))
at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:138(Compiled Code))
at sun/nio/ch/Invoker$2.run(Invoker.java:230(Compiled Code))
at sun/nio/ch/AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:124(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java/lang/Thread.run(Thread.java:811)
This thread is waiting for LOCK1 and owns LOCK2.
The thread sshd-SshClient[61e9f863]-nio2-thread-10 has the following stack trace:
at org/apache/sshd/common/channel/ChannelOutputStream.close(ChannelOutputStream.java:249(Compiled Code))
at org/apache/sshd/common/util/io/IoUtils.closeQuietly(IoUtils.java:151(Compiled Code))
at org/apache/sshd/client/channel/AbstractClientChannel.lambda$getInnerCloseable$1(AbstractClientChannel.java:203(Compiled Code))
at org/apache/sshd/client/channel/AbstractClientChannel$$Lambda$314/00000000252C5370.run(Bytecode PC:4(Compiled Code))
at org/apache/sshd/common/util/closeable/Builder$1.doClose(Builder.java:47(Compiled Code))
at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled Code))
at org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled Code))
at org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled Code))
at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled Code))
at org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled Code))
at org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled Code))
at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled Code))
at org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseGracefully(AbstractInnerCloseable.java:40(Compiled Code))
at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:98(Compiled Code))
at org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.sessionClosed(DefaultForwardingFilter.java:1043(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2Session.doCloseImmediately(Nio2Session.java:266(Compiled Code))
at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled Code))
at org/apache/sshd/common/util/closeable/ParallelCloseable.doClose(ParallelCloseable.java:65(Compiled Code))
at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled Code))
at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled Code))
at org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled Code))
at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled Code))
at org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled Code))
at org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled Code))
at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled Code))
at org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled Code))
at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled Code))
at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled Code))
at org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled Code))
at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled Code))
at org/apache/sshd/common/util/closeable/ParallelCloseable.doClose(ParallelCloseable.java:65(Compiled Code))
at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled Code))
at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled Code))
at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled Code))
at org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled Code))
at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.exceptionCaught(AbstractSession.java:988)
at org/apache/sshd/client/session/ClientSessionImpl.exceptionCaught(ClientSessionImpl.java:121)
at org/apache/sshd/common/session/helpers/AbstractSessionIoHandler.exceptionCaught(AbstractSessionIoHandler.java:53)
at org/apache/sshd/common/io/nio2/Nio2Session.exceptionCaught(Nio2Session.java:184(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2Session.handleWriteCycleFailure(Nio2Session.java:460)
at org/apache/sshd/common/io/nio2/Nio2Session$2.onFailed(Nio2Session.java:415)
at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$failed$1(Nio2CompletionHandler.java:46(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$324/00000000252CCA50.run(Bytecode PC:12(Compiled Code))
at java/security/AccessController.doPrivileged(AccessController.java:638(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.failed(Nio2CompletionHandler.java:45(Compiled Code))
at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:140(Compiled Code))
at sun/nio/ch/Invoker.invokeDirect(Invoker.java:169(Compiled Code))
at sun/nio/ch/UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:749(Compiled Code))
at sun/nio/ch/AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:394(Compiled Code))
at sun/nio/ch/AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:411(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2Session.doWriteCycle(Nio2Session.java:401(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2Session.startWriting(Nio2Session.java:386(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2Session.writePacket(Nio2Session.java:167(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.doWritePacket(AbstractSession.java:1196(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.writePacket(AbstractSession.java:1137(Compiled Code))
at org/apache/sshd/common/channel/AbstractChannel.writePacket(AbstractChannel.java:773(Compiled Code))
at org/apache/sshd/common/channel/ChannelOutputStream.flush(ChannelOutputStream.java:227(Compiled Code))
at org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.lambda$messageReceived$1(DefaultForwardingFilter.java:1069)
at org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler$$Lambda$327/00000000246D5DB0.operationComplete(Bytecode PC:20)
at org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled Code))
at org/apache/sshd/common/future/DefaultSshFuture.notifyListeners(DefaultSshFuture.java:217(Compiled Code))
at org/apache/sshd/common/future/DefaultSshFuture.setValue(DefaultSshFuture.java:112(Compiled Code))
at org/apache/sshd/client/future/DefaultOpenFuture.setOpened(DefaultOpenFuture.java:65(Compiled Code))
at org/apache/sshd/client/channel/AbstractClientChannel.handleOpenSuccess(AbstractClientChannel.java:352(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractConnectionService.channelOpenConfirmation(AbstractConnectionService.java:423(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractConnectionService.process(AbstractConnectionService.java:333(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.doHandleMessage(AbstractSession.java:626(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.handleMessage(AbstractSession.java:559(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.decode(AbstractSession.java:1542(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.messageReceived(AbstractSession.java:520(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:63(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2Session.handleReadCycleCompletion(Nio2Session.java:339(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:318(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:315(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$254/0000000023F76250.run(Bytecode PC:12(Compiled Code))
at java/security/AccessController.doPrivileged(AccessController.java:638(Compiled Code))
at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37(Compiled Code))
at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:138(Compiled Code))
at sun/nio/ch/Invoker$2.run(Invoker.java:230(Compiled Code))
at sun/nio/ch/AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:124(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java/lang/Thread.run(Thread.java:811)
Can you find a solution to avoid this deadlock ?
Thanks in advance
Roberto
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)