You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by wangsan <wq...@gmail.com> on 2019/01/22 12:38:26 UTC
ignite zk: Possible starvation in striped pool
10:38:31.577 [grid-timeout-worker-#55%DAEMON-NODE-10-153-106-16-8991%] WARN
o.a.ignite.internal.util.typedef.G - >>> Possible starvation in striped
pool.
Thread name: sys-stripe-9-#10%DAEMON-NODE-10-153-106-16-8991%
Queue: []
Deadlock: false
Completed: 17156
Thread [name="sys-stripe-9-#10%DAEMON-NODE-10-153-106-16-8991%", id=38,
state=RUNNABLE, blockCnt=0, waitCnt=17103]
at sun.nio.ch.Net.poll(Native Method)
at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:954)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:110)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3262)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2958)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2841)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2692)
at
o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2651)
at
o.a.i.i.managers.communication.GridIoManager.send(GridIoManager.java:1643)
at
o.a.i.i.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1715)
at
o.a.i.i.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1160)
at
o.a.i.i.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1199)
at
o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicAbstractUpdateFuture.sendDhtRequests(GridDhtAtomicAbstractUpdateFuture.java:466)
at
o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicAbstractUpdateFuture.map(GridDhtAtomicAbstractUpdateFuture.java:423)
at
o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1805)
at
o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1628)
at
o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3056)
at
o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:130)
at
o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:266)
at
o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:261)
at
o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
at
o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at
o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at
o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at
o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
at java.lang.Thread.run(Thread.java:748)
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: ignite zk: Possible starvation in striped pool
Posted by wangsan <wq...@gmail.com>.
Thank you！
I see that this is communication spi not discovery spi. But in other
nodes there are many zk session timeout message or zk reconnect fail
message. And the starvation message only print in the node which have the zk
server(not cluster,just three zk node in one machine) in the same machine.
In the log:"Completed: 17156 and blockCnt=0, waitCnt=17103 ". Why there
are so many waitCnt? Is this a countdownlatch?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: ignite zk: Possible starvation in striped pool
Posted by Denis Mekhanikov <dm...@gmail.com>.
This message is printed, when a thread in striped pool doesn't have any
progress for some time.
As far as I can see from the thread dump, a TCP connection with another
node cannot be established for some reason.
It's probably caused by network problems or long GC on one of the nodes.
This is about communication SPI, and doesn't have anything to do with
ZooKeeper.
Denis
вт, 22 янв. 2019 г. в 15:38, wangsan <wq...@gmail.com>:
> 10:38:31.577 [grid-timeout-worker-#55%DAEMON-NODE-10-153-106-16-8991%]
> WARN
> o.a.ignite.internal.util.typedef.G - >>> Possible starvation in striped
> pool.
> Thread name: sys-stripe-9-#10%DAEMON-NODE-10-153-106-16-8991%
> Queue: []
> Deadlock: false
> Completed: 17156
> Thread [name="sys-stripe-9-#10%DAEMON-NODE-10-153-106-16-8991%", id=38,
> state=RUNNABLE, blockCnt=0, waitCnt=17103]
>
> at sun.nio.ch.Net.poll(Native Method)
>
>
> at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:954)
>
>
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:110)
>
>
> at
>
> o.a.i.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3262)
> at
> o.a.i.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2958)
>
> at
> o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2841)
>
> at
> o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2692)
>
> at
> o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2651)
>
> at
> o.a.i.i.managers.communication.GridIoManager.send(GridIoManager.java:1643)
>
>
> at
> o.a.i.i.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1715)
>
>
> at
> o.a.i.i.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1160)
>
>
> at
> o.a.i.i.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1199)
>
>
> at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicAbstractUpdateFuture.sendDhtRequests(GridDhtAtomicAbstractUpdateFuture.java:466)
>
> at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicAbstractUpdateFuture.map(GridDhtAtomicAbstractUpdateFuture.java:423)
>
> at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1805)
>
> at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1628)
>
> at
>
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3056)
> at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:130)
>
> at
>
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:266)
> at
> o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:261)
>
> at
> o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
>
> at
> o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
>
>
> at
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
>
>
> at
> o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
>
>
> at
> o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
>
>
> at
> o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
>
>
> at
> o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
>
>
> at
> o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
>
> at
> o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
>
>
> at
> o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
>
>
> at
> o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
>
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>