You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by swara <sw...@ekaplus.com> on 2021/03/04 05:42:44 UTC
Ignite hanging with log "TcpCommunicationSpi - TCP client created"
Hi,
In our application suddenly ignite is getting hanged(threads blocked) for 1
to 5 hours and then all threads are releasing at the same time and working
normally.
The logs which we are getting repeatedly at that time is
log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112],
duration=10143ms]
Ignite version: 2.8.1
Please tell me the solution to avoid hanging.
Thank You
Swara
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite hanging with log "TcpCommunicationSpi - TCP client created"
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Can you please provide a runnable reproducer for this behavior?
Regards,
--
Ilya Kasnacheev
чт, 1 апр. 2021 г. в 12:54, swara <sw...@ekaplus.com>:
>
> Thank you for your response.
>
> we are querying the same cache object concurrently and executing the
> cache.query() eagerly. The second request is invoking the cursor.iterator()
> without the first request getting completed and all the next subsequent
> requests are getting stuck.
>
> ***The first request got stuck for 4hours now with these errors.
>
>
> org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:92)
>
> org.apache.ignite.internal.processors.cache.query.RegisteredQueryCursor.iter(RegisteredQueryCursor.java:73)
>
> org.apache.ignite.internal.processors.cache.QueryCursorImpl.iter(QueryCursorImpl.java:102)
>
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$7.iterator(IgniteH2Indexing.java:1731)
>
> org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:608)
>
> org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.awaitAllReplies(GridReduceQueryExecutor.java:947)
> org.apache.ignite.internal.util.IgniteUtils.await(IgniteUtils.java:7697)
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> sun.misc.Unsafe.park(Native Method)
>
> ***Second request logs
>
> log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
> [client=null, node addrs=[/(*.*.*.*:48112, /127.0.0.1:48112],
> duration=209ms]
> log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
> [client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112],
> duration=206ms]
> log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
> [client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112],
> duration=208ms]
> log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
> [client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112],
> duration=209ms]
>
> What should we do to kill these processes in ignite as the application is
> getting stuck for hours with no response and getting released on its own ??
>
> Thank you
> Swara
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite hanging with log "TcpCommunicationSpi - TCP client
created"
Posted by swara <sw...@ekaplus.com>.
Thank you for your response.
we are querying the same cache object concurrently and executing the
cache.query() eagerly. The second request is invoking the cursor.iterator()
without the first request getting completed and all the next subsequent
requests are getting stuck.
***The first request got stuck for 4hours now with these errors.
org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:92)
org.apache.ignite.internal.processors.cache.query.RegisteredQueryCursor.iter(RegisteredQueryCursor.java:73)
org.apache.ignite.internal.processors.cache.QueryCursorImpl.iter(QueryCursorImpl.java:102)
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$7.iterator(IgniteH2Indexing.java:1731)
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:608)
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.awaitAllReplies(GridReduceQueryExecutor.java:947)
org.apache.ignite.internal.util.IgniteUtils.await(IgniteUtils.java:7697)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
sun.misc.Unsafe.park(Native Method)
***Second request logs
log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/(*.*.*.*:48112, /127.0.0.1:48112],
duration=209ms]
log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112], duration=206ms]
log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112], duration=208ms]
log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
[client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112], duration=209ms]
What should we do to kill these processes in ignite as the application is
getting stuck for hours with no response and getting released on its own ??
Thank you
Swara
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite hanging with log "TcpCommunicationSpi - TCP client created"
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
It's too hard to answer since I do not have context for some of these
questions.
I recommend closing cursors when they are no longer needed, then checking
if you still have issues.
Regards,
--
Ilya Kasnacheev
пт, 19 мар. 2021 г. в 13:43, swara <sw...@ekaplus.com>:
> Thank you for your response.
>
> Is actual SQL request handling means cursor iterations?
>
> Please explain what is long retransmits means?
>
> We are not closing the cursor is it the reason for hanging ignite for a few
> hours?
>
>
> Thank you
> Swara
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite hanging with log "TcpCommunicationSpi - TCP client
created"
Posted by swara <sw...@ekaplus.com>.
Thank you for your response.
Is actual SQL request handling means cursor iterations?
Please explain what is long retransmits means?
We are not closing the cursor is it the reason for hanging ignite for a few
hours?
Thank you
Swara
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite hanging with log "TcpCommunicationSpi - TCP client created"
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
SqlFieldsQuery does not have connection pool limit, since it is handled by
GridGain nodes and it is multiplexed asynchronously.
The actual SQL request handling takes place in query thread pool,
which indeed has default size of max(8, total number of cores)
Regards,
--
Ilya Kasnacheev
ср, 17 мар. 2021 г. в 14:12, swara <sw...@ekaplus.com>:
> Thank you for your response.
>
> Based on your analysis we figured out that there is a cursor close issue.
> On
> a related note could you please suggest on What is the default connection
> pool limit for running queries using SqlFieldsQuery?
>
> Does it depend on the number of cores alone?
> If we have 8 core processor then how many connections will we get to
> ignite,
> is there a way to increase the pool size to 100 as we have many users using
> the ignite for join queries?
>
> By referring
>
> https://ignite.apache.org/docs/latest/perf-and-troubleshooting/thread-pools-tuning
>
> /The default pool size is max(8, total number of cores). Use
> IgniteConfiguration.setQueryThreadPoolSize(…) or a similar API from your
> programming language to change the pool size./
>
> Thank You
> Swara
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite hanging with log "TcpCommunicationSpi - TCP client
created"
Posted by swara <sw...@ekaplus.com>.
Thank you for your response.
Based on your analysis we figured out that there is a cursor close issue. On
a related note could you please suggest on What is the default connection
pool limit for running queries using SqlFieldsQuery?
Does it depend on the number of cores alone?
If we have 8 core processor then how many connections will we get to ignite,
is there a way to increase the pool size to 100 as we have many users using
the ignite for join queries?
By referring
https://ignite.apache.org/docs/latest/perf-and-troubleshooting/thread-pools-tuning
/The default pool size is max(8, total number of cores). Use
IgniteConfiguration.setQueryThreadPoolSize(…) or a similar API from your
programming language to change the pool size./
Thank You
Swara
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite hanging with log "TcpCommunicationSpi - TCP client created"
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
This looks like the case of half-open connections with very long
retransmits. One host acknowledges that connection has failed when the
other does not.
Please provide complete logs from both directions.
Regards,
--
Ilya Kasnacheev
чт, 4 мар. 2021 г. в 08:42, swara <sw...@ekaplus.com>:
> Hi,
>
>
> In our application suddenly ignite is getting hanged(threads blocked) for 1
> to 5 hours and then all threads are releasing at the same time and working
> normally.
>
> The logs which we are getting repeatedly at that time is
>
> log info: o.a.i.s.c.tcp.TcpCommunicationSpi - TCP client created
> [client=null, node addrs=[/*.*.*.*:48112, /127.0.0.1:48112],
> duration=10143ms]
>
> Ignite version: 2.8.1
>
> Please tell me the solution to avoid hanging.
>
> Thank You
> Swara
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>