You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Milap Wadhwa (JIRA)" <ji...@apache.org> on 2016/07/15 07:07:20 UTC

[jira] [Assigned] (IGNITE-433) Add async message processing in grid thin client and fix deadlock in TcpRoute

     [ https://issues.apache.org/jira/browse/IGNITE-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Milap Wadhwa reassigned IGNITE-433:
-----------------------------------

    Assignee: Milap Wadhwa

> Add async message processing in grid thin client and fix deadlock in TcpRoute 
> ------------------------------------------------------------------------------
>
>                 Key: IGNITE-433
>                 URL: https://issues.apache.org/jira/browse/IGNITE-433
>             Project: Ignite
>          Issue Type: Bug
>            Reporter: Yakov Zhdanov
>            Assignee: Milap Wadhwa
>            Priority: Minor
>              Labels: Muted_test
>
> # Router receives message from server/client and forwards it to client/server in the same thread which is NIO worker thread which wrong (see dumps below)
> # Also need to check if client futures returned to user are completed in NIO threads. If so, it is wrong, since listeners will be executed (if any) in the same thread which slows down IO.
> # uncomment test {{TcpSslRouterSelfTest}}
> {code}
> [11:50:48]W:		 [org.apache.ignite:ignite-clients] Thread [name="grid-nio-worker-0-#10203%routerClient%", id=102408, state=WAITING, blockCnt=2, waitCnt=3]
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]     Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@3732762, ownerName=grid-nio-worker-1-#10196%router-77155b53-a9a1-44ab-b675-16371b760033%, ownerId=102400]
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at sun.misc.Unsafe.park(Native Method)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.ssl.GridNioSslFilter.onSessionWrite(GridNioSslFilter.java:242)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:114)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioCodecFilter.onSessionWrite(GridNioCodecFilter.java:86)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:114)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain$TailFilter.onSessionWrite(GridNioFilterChain.java:259)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain.onSessionWrite(GridNioFilterChain.java:184)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioSessionImpl.send(GridNioSessionImpl.java:102)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.router.impl.GridTcpRouterNioListenerAdapter$1.onDone(GridTcpRouterNioListenerAdapter.java:111)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.impl.GridClientFutureAdapter$DoneCallback.proceed(GridClientFutureAdapter.java:291)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.impl.GridClientFutureAdapter.fireDone(GridClientFutureAdapter.java:244)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.impl.GridClientFutureAdapter.onDone(GridClientFutureAdapter.java:178)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.impl.GridClientFutureAdapter.onDone(GridClientFutureAdapter.java:140)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.impl.connection.GridClientNioTcpConnection.handleResponse(GridClientNioTcpConnection.java:479)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.impl.connection.GridClientConnectionManagerAdapter$NioListener.onMessage(GridClientConnectionManagerAdapter.java:601)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:269)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:106)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.ssl.GridNioSslFilter.onMessageReceived(GridNioSslFilter.java:289)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:2012)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:172)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:670)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeys(GridNioServer.java:1331)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1273)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1160)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.lang.Thread.run(Thread.java:722)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients] 
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]     Locked synchronizers:
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         java.util.concurrent.locks.ReentrantLock$NonfairSync@4e479454
> [11:50:48]W:		 [org.apache.ignite:ignite-clients] Thread [name="grid-nio-worker-1-#10196%router-77155b53-a9a1-44ab-b675-16371b760033%", id=102400, state=WAITING, blockCnt=7, waitCnt=5]
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]     Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@4e479454, ownerName=grid-nio-worker-0-#10203%routerClient%, ownerId=102408]
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at sun.misc.Unsafe.park(Native Method)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.ssl.GridNioSslFilter.onSessionWrite(GridNioSslFilter.java:242)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:114)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioCodecFilter.onSessionWrite(GridNioCodecFilter.java:86)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:114)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain$TailFilter.onSessionWrite(GridNioFilterChain.java:259)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain.onSessionWrite(GridNioFilterChain.java:184)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioSessionImpl.send(GridNioSessionImpl.java:102)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.impl.connection.GridClientNioTcpConnection.makeRequest(GridClientNioTcpConnection.java:418)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.impl.connection.GridClientNioTcpConnection.forwardMessage(GridClientNioTcpConnection.java:852)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.router.impl.GridRouterClientImpl.forwardMessage(GridRouterClientImpl.java:108)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.router.impl.GridTcpRouterNioListenerAdapter.onMessage(GridTcpRouterNioListenerAdapter.java:103)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.router.impl.GridTcpRouterNioListenerAdapter.onMessage(GridTcpRouterNioListenerAdapter.java:38)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:269)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:106)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.ssl.GridNioSslFilter.onMessageReceived(GridNioSslFilter.java:289)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:2012)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:172)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:670)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeys(GridNioServer.java:1331)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1273)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1160)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.lang.Thread.run(Thread.java:722)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients] 
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]     Locked synchronizers:
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         java.util.concurrent.locks.ReentrantLock$NonfairSync@3732762
> [11:50:48]
> [11:50:48]W:		 [org.apache.ignite:ignite-clients] Thread [name="grid-nio-worker-3-#10198%router-77155b53-a9a1-44ab-b675-16371b760033%", id=102402, state=WAITING, blockCnt=7, waitCnt=2]
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]     Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@4e479454, ownerName=grid-nio-worker-0-#10203%routerClient%, ownerId=102408]
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at sun.misc.Unsafe.park(Native Method)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.ssl.GridNioSslFilter.onSessionWrite(GridNioSslFilter.java:242)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:114)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioCodecFilter.onSessionWrite(GridNioCodecFilter.java:86)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedSessionWrite(GridNioFilterAdapter.java:114)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain$TailFilter.onSessionWrite(GridNioFilterChain.java:259)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain.onSessionWrite(GridNioFilterChain.java:184)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioSessionImpl.send(GridNioSessionImpl.java:102)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.impl.connection.GridClientNioTcpConnection.makeRequest(GridClientNioTcpConnection.java:418)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.impl.connection.GridClientNioTcpConnection.forwardMessage(GridClientNioTcpConnection.java:852)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.router.impl.GridRouterClientImpl.forwardMessage(GridRouterClientImpl.java:108)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.router.impl.GridTcpRouterNioListenerAdapter.onMessage(GridTcpRouterNioListenerAdapter.java:103)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.client.router.impl.GridTcpRouterNioListenerAdapter.onMessage(GridTcpRouterNioListenerAdapter.java:38)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:269)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:106)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.ssl.GridNioSslFilter.onMessageReceived(GridNioSslFilter.java:289)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:2012)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:172)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:670)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeys(GridNioServer.java:1331)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1273)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1160)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         at java.lang.Thread.run(Thread.java:722)
> [11:50:48]W:		 [org.apache.ignite:ignite-clients] 
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]     Locked synchronizers:
> [11:50:48]W:		 [org.apache.ignite:ignite-clients]         java.util.concurrent.locks.ReentrantLock$NonfairSync@7d1e94b2
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)