You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by hulitao198758 <hu...@lenovo.com> on 2020/02/21 02:43:11 UTC
Connection reset by peer
^-- System thread pool [active=0, idle=6, qSize=0]
[16:57:36,455][SEVERE][grid-nio-worker-tcp-comm-3-#50%SubScriptionCluster%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=3,
bytesRcvd=4878431, bytesSent=2691434, bytesRcvd0=0, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-3,
igniteInstanceName=SubScriptionCluster, finished=false, hashCode=1064068859,
interrupted=false,
runner=grid-nio-worker-tcp-comm-3-#50%SubScriptionCluster%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=1316, resendCnt=0, rcvCnt=1426,
sentCnt=1316, reserved=true, lastAck=1426, nodeLeft=false,
node=TcpDiscoveryNode [id=f019bbc8-7c8e-469c-ae4b-a89b19afbbed,
addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.16.7.19],
sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /172.16.7.19:0], discPort=0,
order=40, intOrder=24, lastExchangeTime=1581909203338, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=3, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=1316, resendCnt=0, rcvCnt=1426,
sentCnt=1316, reserved=true, lastAck=1426, nodeLeft=false,
node=TcpDiscoveryNode [id=f019bbc8-7c8e-469c-ae4b-a89b19afbbed,
addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.16.7.19],
sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /172.16.7.19:0], discPort=0,
order=40, intOrder=24, lastExchangeTime=1581909203338, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=3, pairedConnections=false],
super=GridNioSessionImpl [locAddr=/10.122.64.127:48339,
rmtAddr=/10.122.23.172:57818, createTime=1581971404618, closeTime=0,
bytesSent=1128312, bytesRcvd=849644, bytesSent0=0, bytesRcvd0=0,
sndSchedTime=1582217852002, lastSndTime=1582196198388,
lastRcvTime=1582217852002, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@6b3aff0d, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1250)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
[16:57:36,457][WARNING][grid-nio-worker-tcp-comm-3-#50%SubScriptionCluster%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
The ignite server is 3 nodes. It is deployed on the server as a database to
use and enable persistence. The client node is deployed in a k8s container.
This error occurs after running for a while. Why?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Connection reset by peer
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
[01:07:25,966][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/172.16.99.27:47100,
failureDetectionTimeout=100000]
Failed to send message (node
may have left the grid or TCP connection cannot be established due to
firewall issues)
Looks like massive network problems to me, or very long GC.
Maybe your firewall is too restrictive? Please note that we expect that all
client and server nodes can open connections to each other.
Regards,
--
Ilya Kasnacheev
пт, 21 февр. 2020 г. в 06:16, hulitao198758 <hu...@lenovo.com>:
> Detailed error message：
>
> Established outgoing communication connection [locAddr=/
> 10.122.64.129:40870,
> rmtAddr=/10.122.64.128:48339]
>
> [01:01:59,775][SEVERE][grid-nio-worker-tcp-comm-1-#48%SubScriptionCluster%][TcpCommunicationSpi]
> Failed to process selector key [ses=GridSelectorNioSessionImpl
> [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=1,
> bytesRcvd=1151627, bytesSent=2289496, bytesRcvd0=64, bytesSent0=537,
> select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1,
> igniteInstanceName=SubScriptionCluster, finished=false, hashCode=440471613,
> interrupted=false,
> runner=grid-nio-worker-tcp-comm-1-#48%SubScriptionCluster%]]],
> writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
> readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
> inRecovery=GridNioRecoveryDescriptor [acked=2651, resendCnt=0, rcvCnt=2752,
> sentCnt=2652, reserved=true, lastAck=2752, nodeLeft=false,
> node=TcpDiscoveryNode [id=09e38b03-ea63-47bf-a27b-661a0f0ab8d8,
> addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.16.99.27],
> sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /172.16.99.27:0],
> discPort=0, order=44, intOrder=26, lastExchangeTime=1581939704719,
> loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=true],
> connected=true,
> connectCnt=0, queueLimit=4096, reserveCnt=225, pairedConnections=false],
> outRecovery=GridNioRecoveryDescriptor [acked=2651, resendCnt=0,
> rcvCnt=2752,
> sentCnt=2652, reserved=true, lastAck=2752, nodeLeft=false,
> node=TcpDiscoveryNode [id=09e38b03-ea63-47bf-a27b-661a0f0ab8d8,
> addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.16.99.27],
> sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /172.16.99.27:0],
> discPort=0, order=44, intOrder=26, lastExchangeTime=1581939704719,
> loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=true],
> connected=true,
> connectCnt=0, queueLimit=4096, reserveCnt=225, pairedConnections=false],
> super=GridNioSessionImpl [locAddr=/10.122.64.129:48339,
> rmtAddr=/10.122.23.179:33052, createTime=1582183927690, closeTime=0,
> bytesSent=33096, bytesRcvd=29254, bytesSent0=211, bytesRcvd0=0,
> sndSchedTime=1582246566604, lastSndTime=1582246918763,
> lastRcvTime=1582246566604, readsPaused=false,
> filterChain=FilterChain[filters=[GridNioCodecFilter
> [parser=o.a.i.i.util.nio.GridDirectParser@7419c93, directMode=true],
> GridConnectionBytesVerifyFilter], accepted=true]]]
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> at
>
> org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1250)
> at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
> at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
> at
>
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:748)
>
> [01:01:59,776][WARNING][grid-nio-worker-tcp-comm-1-#48%SubScriptionCluster%][TcpCommunicationSpi]
> Closing NIO session because of unhandled exception [cls=class
> o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
>
> [01:02:14,897][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Connect timed out (consider increasing 'failureDetectionTimeout'
> configuration property) [addr=/172.16.99.27:47100,
> failureDetectionTimeout=100000]
>
> [01:02:14,898][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Connect timed out (consider increasing 'failureDetectionTimeout'
> configuration property) [addr=/0:0:0:0:0:0:0:1%lo:47100,
> failureDetectionTimeout=100000]
>
> [01:02:14,898][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Connect timed out (consider increasing 'failureDetectionTimeout'
> configuration property) [addr=/127.0.0.1:47100,
> failureDetectionTimeout=100000]
>
>
> [01:02:14,898][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Failed to connect to a remote node (make sure that destination node is
> alive
> and operating system firewall is disabled on local and remote hosts)
> [addrs=[/172.16.99.27:47100, /0:0:0:0:0:0:0:1%lo:47100, /127.0.0.1:47100]]
>
> [01:02:14,899][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:02:14,917][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=19ms]
> [01:02:14,985][WARNING][grid-timeout-worker-#45%SubScriptionCluster%][G]
> >>>
> Possible starvation in striped pool.
> Thread name: sys-stripe-3-#4%SubScriptionCluster%
> Queue: []
> Deadlock: false
> Completed: 96
> Thread [name="sys-stripe-3-#4%SubScriptionCluster%", id=19,
> state=TIMED_WAITING, blockCnt=1, waitCnt=215]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7629)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1896)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1840)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1822)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1240)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1177)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:890)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:85)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:430)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:376)
> at
>
> o.a.i.i.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1080)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:652)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:583)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:464)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:505)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitAsync(GridDhtTxLocal.java:526)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:735)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:104)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
>
> o.a.i.i.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
> at
> o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
> at
> o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
> at
>
> o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at
> o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at
>
> o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture$MiniFuture.onResult(GridDhtTxPrepareFuture.java:1870)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onResult(GridDhtTxPrepareFuture.java:549)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareResponse(IgniteTxHandler.java:708)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler.access$500(IgniteTxHandler.java:97)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:207)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:205)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> at
>
> o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
> at
>
> o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
> at
>
> o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
> at
> o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
> at
> o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
> at java.lang.Thread.run(Thread.java:748)
>
>
> [01:02:16,011][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:02:16,012][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=4ms]
>
> [01:02:26,883][INFO][grid-timeout-worker-#45%SubScriptionCluster%][IgniteKernal%SubScriptionCluster]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=bb0d277b, name=SubScriptionCluster, uptime=25:20:25.180]
> ^-- H/N/C [hosts=5, nodes=5, CPUs=32]
> ^-- CPU [cur=0.87%, avg=0.93%, GC=0%]
> ^-- PageMemory [pages=59694]
> ^-- Heap [used=1111MB, free=81.9%, comm=6144MB]
> ^-- Non heap [used=101MB, free=86.42%, comm=103MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=6, qSize=0]
> [01:02:44,991][WARNING][grid-timeout-worker-#45%SubScriptionCluster%][G]
> >>>
> Possible starvation in striped pool.
> Thread name: sys-stripe-3-#4%SubScriptionCluster%
> Queue: []
> Deadlock: false
> Completed: 96
> Thread [name="sys-stripe-3-#4%SubScriptionCluster%", id=19, state=WAITING,
> blockCnt=1, waitCnt=218]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
> at
> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
> at
>
> o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2882)
> at
>
> o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2692)
> at
>
> o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2651)
> at
> o.a.i.i.managers.communication.GridIoManager.send(GridIoManager.java:1643)
> at
>
> o.a.i.i.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1863)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1869)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1840)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1822)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1240)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1177)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:890)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:85)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:430)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:376)
> at
>
> o.a.i.i.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1080)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:652)
>
> at
> o.a.i.i.managers.communication.GridIoManager.send(GridIoManager.java:1643)
> at
>
> o.a.i.i.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1863)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1869)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1840)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1822)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1240)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1177)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:890)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:85)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:430)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:376)
> at
>
> o.a.i.i.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1080)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:652)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:583)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:464)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:505)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitAsync(GridDhtTxLocal.java:526)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:735)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:104)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
>
> o.a.i.i.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
> at
> o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
> at
> o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
> at
>
> o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at
> o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at
>
> o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture$MiniFuture.onResult(GridDhtTxPrepareFuture.java:1870)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onResult(GridDhtTxPrepareFuture.java:549)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareResponse(IgniteTxHandler.java:708)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler.access$500(IgniteTxHandler.java:97)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:207)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:205)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> at
>
> o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
> at
>
> o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
> at
>
> o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
> at
> o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
> at
> o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
> at java.lang.Thread.run(Thread.java:748)
>
>
> [01:02:47,071][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:02:47,072][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=9ms]
>
>
> [01:02:47,071][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:02:47,072][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=9ms]
> [01:03:15,000][WARNING][grid-timeout-worker-#45%SubScriptionCluster%][G]
> >>>
> Possible starvation in striped pool.
> Thread name: sys-stripe-3-#4%SubScriptionCluster%
> Queue: []
> Deadlock: false
> Completed: 96
> Thread [name="sys-stripe-3-#4%SubScriptionCluster%", id=19, state=WAITING,
> blockCnt=1, waitCnt=220]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
> at
> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
> at
>
> o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2882)
> at
>
> o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2692)
> at
>
> o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2651)
> at
> o.a.i.i.managers.communication.GridIoManager.send(GridIoManager.java:1643)
> at
>
> o.a.i.i.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1863)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1869)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1840)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1822)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1240)
> at
>
> o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1177)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:890)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:85)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:430)
> at
>
> o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:376)
> at
>
> o.a.i.i.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1080)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:652)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:583)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:464)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:505)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitAsync(GridDhtTxLocal.java:526)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:735)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:104)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
>
> o.a.i.i.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
> at
> o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
> at
> o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
> at
>
> o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at
> o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at
>
> o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture$MiniFuture.onResult(GridDhtTxPrepareFuture.java:1870)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onResult(GridDhtTxPrepareFuture.java:549)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareResponse(IgniteTxHandler.java:708)
>
> 59619,9
>
>
> at
> o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture$MiniFuture.onResult(GridDhtTxPrepareFuture.java:1870)
> at
>
> o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onResult(GridDhtTxPrepareFuture.java:549)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareResponse(IgniteTxHandler.java:708)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler.access$500(IgniteTxHandler.java:97)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:207)
> at
>
> o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:205)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> at
>
> o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> at
>
> o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
> at
>
> o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
> at
>
> o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
> at
> o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
> at
> o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
> at java.lang.Thread.run(Thread.java:748)
>
>
> [01:03:26,885][INFO][grid-timeout-worker-#45%SubScriptionCluster%][IgniteKernal%SubScriptionCluster]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=bb0d277b, name=SubScriptionCluster, uptime=25:21:25.186]
> ^-- H/N/C [hosts=5, nodes=5, CPUs=32]
> ^-- CPU [cur=0.9%, avg=0.93%, GC=0%]
> ^-- PageMemory [pages=59694]
> ^-- Heap [used=1137MB, free=81.48%, comm=6144MB]
> ^-- Non heap [used=101MB, free=86.42%, comm=103MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=6, qSize=0]
>
> [01:04:34,595][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:04:34,597][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=5ms]
>
> [01:05:37,760][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:05:37,761][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=8ms]
>
> [01:05:38,855][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:05:38,856][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=2ms]
>
> [01:05:45,888][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:05:45,889][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=4ms]
>
> [01:07:25,966][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Connect timed out (consider increasing 'failureDetectionTimeout'
> configuration property) [addr=/172.16.99.27:47100,
> failureDetectionTimeout=100000]
>
> [01:07:25,967][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Connect timed out (consider increasing 'failureDetectionTimeout'
> configuration property) [addr=/0:0:0:0:0:0:0:1%lo:47100,
> failureDetectionTimeout=100000]
>
> [01:07:25,967][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Connect timed out (consider increasing 'failureDetectionTimeout'
> configuration property) [addr=/127.0.0.1:47100,
> failureDetectionTimeout=100000]
>
> [01:07:25,968][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Failed to connect to a remote node (make sure that destination node is
> alive
> and operating system firewall is disabled on local and remote hosts)
> [addrs=[/172.16.99.27:47100, /0:0:0:0:0:0:0:1%lo:47100, /127.0.0.1:47100]]
>
> [01:07:27,270][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:07:27,272][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=7ms]
>
> [01:07:28,299][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:07:28,300][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=4ms]
>
> [01:07:28,321][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> 59
>
> [01:07:25,966][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Connect timed out (consider increasing 'failureDetectionTimeout'
> configuration property) [addr=/172.16.99.27:47100,
> failureDetectionTimeout=100000]
>
> [01:07:25,967][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Connect timed out (consider increasing 'failureDetectionTimeout'
> configuration property) [addr=/0:0:0:0:0:0:0:1%lo:47100,
> failureDetectionTimeout=100000]
>
> [01:07:25,967][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Connect timed out (consider increasing 'failureDetectionTimeout'
> configuration property) [addr=/127.0.0.1:47100,
> failureDetectionTimeout=100000]
>
> [01:07:25,968][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
> Failed to connect to a remote node (make sure that destination node is
> alive
> and operating system firewall is disabled on local and remote hosts)
> [addrs=[/172.16.99.27:47100, /0:0:0:0:0:0:0:1%lo:47100, /127.0.0.1:47100]]
>
> [01:07:27,270][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:07:27,272][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=7ms]
>
> [01:07:28,299][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:07:28,300][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=4ms]
>
> [01:07:28,321][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
>
> [01:07:28,323][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
> time=7ms]
>
> [01:07:29,340][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
> Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
> [01:07:29,340][SEVERE][sys-stripe-3-#4%SubScriptionCluster%][query] Failed
> to send event notification to node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
> class org.apache.ignite.IgniteCheckedException: Failed to send message
> (node
> may have left the grid or TCP connection cannot be established due to
> firewall issues) [node=TcpDiscoveryNode
> [id=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, addrs=[0:0:0:0:0:0:0:1%lo,
> 127.0.0.1, 172.16.99.27], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0,
> /172.16.99.27:0], discPort=0, order=44, intOrder=26,
> lastExchangeTime=1581939704719, loc=false,
> ver=2.6.0#20180710-sha1:669feacc,
> isClient=true], topic=T4 [topic=TOPIC_CACHE,
> id1=651b3302-d586-38ee-89b9-965a111ee94e,
> id2=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, id3=0], msg=GridContinuousMessage
> [type=MSG_EVT_NOTIFICATION, routineId=d807ddd1-7fdf-4792-9324-6fffa78ad24c,
> data=null, futId=null], policy=2]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1651)
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1863)
> at
>
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1869)
> at
>
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1840)
> at
>
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1822)
> at
>
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1240)
> at
>
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1177)
> at
>
> org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:890)
> at
>
> org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:85)
> at
>
> org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:430)
> at
>
> org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:376)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1080)
> at
>
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:652)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:583)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:464)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:505)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitAsync(GridDhtTxLocal.java:526)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:735)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:104)
> at
>
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
>
> org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
> at
>
> org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
> at
>
> org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
> at
>
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
>
>
>
> Then the node keeps reporting errors and the node cannot be used.
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Connection reset by peer
Posted by hulitao198758 <hu...@lenovo.com>.
Detailed error message：
Established outgoing communication connection [locAddr=/10.122.64.129:40870,
rmtAddr=/10.122.64.128:48339]
[01:01:59,775][SEVERE][grid-nio-worker-tcp-comm-1-#48%SubScriptionCluster%][TcpCommunicationSpi]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=1,
bytesRcvd=1151627, bytesSent=2289496, bytesRcvd0=64, bytesSent0=537,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1,
igniteInstanceName=SubScriptionCluster, finished=false, hashCode=440471613,
interrupted=false,
runner=grid-nio-worker-tcp-comm-1-#48%SubScriptionCluster%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=2651, resendCnt=0, rcvCnt=2752,
sentCnt=2652, reserved=true, lastAck=2752, nodeLeft=false,
node=TcpDiscoveryNode [id=09e38b03-ea63-47bf-a27b-661a0f0ab8d8,
addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.16.99.27],
sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /172.16.99.27:0],
discPort=0, order=44, intOrder=26, lastExchangeTime=1581939704719,
loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=225, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=2651, resendCnt=0, rcvCnt=2752,
sentCnt=2652, reserved=true, lastAck=2752, nodeLeft=false,
node=TcpDiscoveryNode [id=09e38b03-ea63-47bf-a27b-661a0f0ab8d8,
addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.16.99.27],
sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0, /172.16.99.27:0],
discPort=0, order=44, intOrder=26, lastExchangeTime=1581939704719,
loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=true], connected=true,
connectCnt=0, queueLimit=4096, reserveCnt=225, pairedConnections=false],
super=GridNioSessionImpl [locAddr=/10.122.64.129:48339,
rmtAddr=/10.122.23.179:33052, createTime=1582183927690, closeTime=0,
bytesSent=33096, bytesRcvd=29254, bytesSent0=211, bytesRcvd0=0,
sndSchedTime=1582246566604, lastSndTime=1582246918763,
lastRcvTime=1582246566604, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@7419c93, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1250)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
[01:01:59,776][WARNING][grid-nio-worker-tcp-comm-1-#48%SubScriptionCluster%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
[01:02:14,897][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/172.16.99.27:47100,
failureDetectionTimeout=100000]
[01:02:14,898][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/0:0:0:0:0:0:0:1%lo:47100,
failureDetectionTimeout=100000]
[01:02:14,898][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/127.0.0.1:47100,
failureDetectionTimeout=100000]
[01:02:14,898][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Failed to connect to a remote node (make sure that destination node is alive
and operating system firewall is disabled on local and remote hosts)
[addrs=[/172.16.99.27:47100, /0:0:0:0:0:0:0:1%lo:47100, /127.0.0.1:47100]]
[01:02:14,899][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:02:14,917][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=19ms]
[01:02:14,985][WARNING][grid-timeout-worker-#45%SubScriptionCluster%][G] >>>
Possible starvation in striped pool.
Thread name: sys-stripe-3-#4%SubScriptionCluster%
Queue: []
Deadlock: false
Completed: 96
Thread [name="sys-stripe-3-#4%SubScriptionCluster%", id=19,
state=TIMED_WAITING, blockCnt=1, waitCnt=215]
at java.lang.Thread.sleep(Native Method)
at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7629)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1896)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1840)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1822)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1240)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1177)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:890)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:85)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:430)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:376)
at
o.a.i.i.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1080)
at
o.a.i.i.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:652)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:583)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:464)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:505)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitAsync(GridDhtTxLocal.java:526)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:735)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:104)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
at
o.a.i.i.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
at
o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
at
o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
at
o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
at
o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
at
o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture$MiniFuture.onResult(GridDhtTxPrepareFuture.java:1870)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onResult(GridDhtTxPrepareFuture.java:549)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareResponse(IgniteTxHandler.java:708)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler.access$500(IgniteTxHandler.java:97)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:207)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:205)
at
o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
at
o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at
o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at
o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at
o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
at java.lang.Thread.run(Thread.java:748)
[01:02:16,011][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:02:16,012][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=4ms]
[01:02:26,883][INFO][grid-timeout-worker-#45%SubScriptionCluster%][IgniteKernal%SubScriptionCluster]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=bb0d277b, name=SubScriptionCluster, uptime=25:20:25.180]
^-- H/N/C [hosts=5, nodes=5, CPUs=32]
^-- CPU [cur=0.87%, avg=0.93%, GC=0%]
^-- PageMemory [pages=59694]
^-- Heap [used=1111MB, free=81.9%, comm=6144MB]
^-- Non heap [used=101MB, free=86.42%, comm=103MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
[01:02:44,991][WARNING][grid-timeout-worker-#45%SubScriptionCluster%][G] >>>
Possible starvation in striped pool.
Thread name: sys-stripe-3-#4%SubScriptionCluster%
Queue: []
Deadlock: false
Completed: 96
Thread [name="sys-stripe-3-#4%SubScriptionCluster%", id=19, state=WAITING,
blockCnt=1, waitCnt=218]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
at
o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2882)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2692)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2651)
at
o.a.i.i.managers.communication.GridIoManager.send(GridIoManager.java:1643)
at
o.a.i.i.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1863)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1869)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1840)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1822)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1240)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1177)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:890)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:85)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:430)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:376)
at
o.a.i.i.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1080)
at
o.a.i.i.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:652)
at
o.a.i.i.managers.communication.GridIoManager.send(GridIoManager.java:1643)
at
o.a.i.i.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1863)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1869)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1840)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1822)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1240)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1177)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:890)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:85)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:430)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:376)
at
o.a.i.i.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1080)
at
o.a.i.i.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:652)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:583)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:464)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:505)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitAsync(GridDhtTxLocal.java:526)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:735)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:104)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
at
o.a.i.i.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
at
o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
at
o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
at
o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
at
o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
at
o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture$MiniFuture.onResult(GridDhtTxPrepareFuture.java:1870)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onResult(GridDhtTxPrepareFuture.java:549)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareResponse(IgniteTxHandler.java:708)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler.access$500(IgniteTxHandler.java:97)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:207)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:205)
at
o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
at
o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at
o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at
o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at
o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
at java.lang.Thread.run(Thread.java:748)
[01:02:47,071][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:02:47,072][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=9ms]
[01:02:47,071][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:02:47,072][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=9ms]
[01:03:15,000][WARNING][grid-timeout-worker-#45%SubScriptionCluster%][G] >>>
Possible starvation in striped pool.
Thread name: sys-stripe-3-#4%SubScriptionCluster%
Queue: []
Deadlock: false
Completed: 96
Thread [name="sys-stripe-3-#4%SubScriptionCluster%", id=19, state=WAITING,
blockCnt=1, waitCnt=220]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
at
o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2882)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2692)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2651)
at
o.a.i.i.managers.communication.GridIoManager.send(GridIoManager.java:1643)
at
o.a.i.i.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1863)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1869)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1840)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1822)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1240)
at
o.a.i.i.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1177)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:890)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:85)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:430)
at
o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:376)
at
o.a.i.i.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1080)
at
o.a.i.i.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:652)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:583)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:464)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:505)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxLocal.commitAsync(GridDhtTxLocal.java:526)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:735)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:104)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
at
o.a.i.i.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
at
o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
at
o.a.i.i.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
at
o.a.i.i.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
at
o.a.i.i.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
at
o.a.i.i.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture$MiniFuture.onResult(GridDhtTxPrepareFuture.java:1870)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onResult(GridDhtTxPrepareFuture.java:549)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareResponse(IgniteTxHandler.java:708)
59619,9
at
o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture$MiniFuture.onResult(GridDhtTxPrepareFuture.java:1870)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onResult(GridDhtTxPrepareFuture.java:549)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareResponse(IgniteTxHandler.java:708)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler.access$500(IgniteTxHandler.java:97)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:207)
at
o.a.i.i.processors.cache.transactions.IgniteTxHandler$6.apply(IgniteTxHandler.java:205)
at
o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
at
o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at
o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at
o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at
o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
at java.lang.Thread.run(Thread.java:748)
[01:03:26,885][INFO][grid-timeout-worker-#45%SubScriptionCluster%][IgniteKernal%SubScriptionCluster]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=bb0d277b, name=SubScriptionCluster, uptime=25:21:25.186]
^-- H/N/C [hosts=5, nodes=5, CPUs=32]
^-- CPU [cur=0.9%, avg=0.93%, GC=0%]
^-- PageMemory [pages=59694]
^-- Heap [used=1137MB, free=81.48%, comm=6144MB]
^-- Non heap [used=101MB, free=86.42%, comm=103MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
[01:04:34,595][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:04:34,597][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=5ms]
[01:05:37,760][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:05:37,761][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=8ms]
[01:05:38,855][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:05:38,856][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=2ms]
[01:05:45,888][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:05:45,889][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=4ms]
[01:07:25,966][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/172.16.99.27:47100,
failureDetectionTimeout=100000]
[01:07:25,967][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/0:0:0:0:0:0:0:1%lo:47100,
failureDetectionTimeout=100000]
[01:07:25,967][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/127.0.0.1:47100,
failureDetectionTimeout=100000]
[01:07:25,968][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Failed to connect to a remote node (make sure that destination node is alive
and operating system firewall is disabled on local and remote hosts)
[addrs=[/172.16.99.27:47100, /0:0:0:0:0:0:0:1%lo:47100, /127.0.0.1:47100]]
[01:07:27,270][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:07:27,272][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=7ms]
[01:07:28,299][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:07:28,300][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=4ms]
[01:07:28,321][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
59
[01:07:25,966][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/172.16.99.27:47100,
failureDetectionTimeout=100000]
[01:07:25,967][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/0:0:0:0:0:0:0:1%lo:47100,
failureDetectionTimeout=100000]
[01:07:25,967][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Connect timed out (consider increasing 'failureDetectionTimeout'
configuration property) [addr=/127.0.0.1:47100,
failureDetectionTimeout=100000]
[01:07:25,968][WARNING][tcp-comm-worker-#1%SubScriptionCluster%][TcpCommunicationSpi]
Failed to connect to a remote node (make sure that destination node is alive
and operating system firewall is disabled on local and remote hosts)
[addrs=[/172.16.99.27:47100, /0:0:0:0:0:0:0:1%lo:47100, /127.0.0.1:47100]]
[01:07:27,270][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:07:27,272][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=7ms]
[01:07:28,299][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:07:28,300][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=4ms]
[01:07:28,321][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:07:28,323][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Finished node ping [nodeId=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, res=true,
time=7ms]
[01:07:29,340][INFO][tcp-comm-worker-#1%SubScriptionCluster%][TcpDiscoverySpi]
Pinging node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
[01:07:29,340][SEVERE][sys-stripe-3-#4%SubScriptionCluster%][query] Failed
to send event notification to node: 09e38b03-ea63-47bf-a27b-661a0f0ab8d8
class org.apache.ignite.IgniteCheckedException: Failed to send message (node
may have left the grid or TCP connection cannot be established due to
firewall issues) [node=TcpDiscoveryNode
[id=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 172.16.99.27], sockAddrs=[/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0,
/172.16.99.27:0], discPort=0, order=44, intOrder=26,
lastExchangeTime=1581939704719, loc=false, ver=2.6.0#20180710-sha1:669feacc,
isClient=true], topic=T4 [topic=TOPIC_CACHE,
id1=651b3302-d586-38ee-89b9-965a111ee94e,
id2=09e38b03-ea63-47bf-a27b-661a0f0ab8d8, id3=0], msg=GridContinuousMessage
[type=MSG_EVT_NOTIFICATION, routineId=d807ddd1-7fdf-4792-9324-6fffa78ad24c,
data=null, futId=null], policy=2]
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1651)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1863)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1869)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1840)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1822)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1240)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1177)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:890)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$600(CacheContinuousQueryHandler.java:85)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$2.onEntryUpdated(CacheContinuousQueryHandler.java:430)
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:376)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1080)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:652)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:795)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:583)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:464)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:505)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitAsync(GridDhtTxLocal.java:526)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:735)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:104)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
at
org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
at
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
at
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
Then the node keeps reporting errors and the node cannot be used.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/