You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Kamlesh Joshi <Ka...@ril.com> on 2020/10/09 12:55:31 UTC
RE: [External]Re: Failed to process selector key frequently at client
side
Hi Ilya,
We could see idleConnTimeout value is set at the server level as below. Do we have to set this value on both client and server?
TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@560348e6<ma...@560348e6>, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=47100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=60000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@1df8b5b8[Count<mailto:ctxInitLatch=java.util.concurrent.CountDownLatch@1df8b5b8[Count> = 1], stopping=false],
Thanks and Regards,
Kamlesh Joshi
From: Ilya Kasnacheev <il...@gmail.com>
Sent: 09 October 2020 17:14
To: user@ignite.apache.org
Subject: [External]Re: Failed to process selector key frequently at client side
The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin.
Hello!
It seems that the network that you are using will close idle connections. You can try sidestepping it by specifying TcpCommunicationSpi.idleConnectionTimeout to some value such as 25000 ms.
In this case Ignite will be first to close these connections explicitly.
Regards,
--
Ilya Kasnacheev
пт, 9 окт. 2020 г. в 09:37, Kamlesh Joshi <Ka...@ril.com>>:
Hi Igniters,
We have three server nodes, there is firewall between clients and servers.
Opened 47500 and 47100 series ports, but still below errors are printing at client side frequently. But client application able to get data from cluster, could please help us how to avoid these errors at client side.
We are using Ignite 2.7.0 version.
[2020-10-08T00:26:23,389][ERROR][grid-nio-worker-tcp-comm-0-#24%JBPLCustomer%][TcpCommunicationSpi] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=84608, bytesSent=1725593, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=JBPLCustomer, finished=false, heartbeatTs=1602096980980, hashCode=1228592538, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#24%JBPLCustomer%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=48, resendCnt=0, rcvCnt=58, sentCnt=48, reserved=true, lastAck=58, nodeLeft=false, node=TcpDiscoveryNode [id=ef6723ee-f70d-4515-89b9-7a3e85ab5072, addrs=[0:0:0:0:0:0:0:1%lo, clientIP, 127.0.0.1, 2405:200:a60:fd04:20c:29ff:fe63:629f%eth0], sockAddrs=[/clientIP:0, 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0<https://protect2.fireeye.com/v1/url?k=29480a65-756b19ca-2949809a-00259087546c-f07172281c791ac1&q=1&e=ae47e4b2-c5cf-41b8-9e41-a11ed2e2ec12&u=http%3A%2F%2F127.0.0.1%3A0%2F>, 2405:200:a60:fd04:20c:29ff:fe63:629f%eth0:0], discPort=0, order=72, intOrder=64, lastExchangeTime=1602089183465, loc=false, ver=2.7.0#20181201-sha1:256ae401, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=48, resendCnt=0, rcvCnt=58, sentCnt=48, reserved=true, lastAck=58, nodeLeft=false, node=TcpDiscoveryNode [id=ef6723ee-f70d-4515-89b9-7a3e85ab5072, addrs=[0:0:0:0:0:0:0:1%lo, clientIP, 127.0.0.1, 2405:200:a60:fd04:20c:29ff:fe63:629f%eth0], sockAddrs=[/clientIP:0, 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0<https://protect2.fireeye.com/v1/url?k=ef3f61e8-b31c7247-ef3eeb17-00259087546c-d45fc1450e7efc2c&q=1&e=ae47e4b2-c5cf-41b8-9e41-a11ed2e2ec12&u=http%3A%2F%2F127.0.0.1%3A0%2F>, 2405:200:a60:fd04:20c:29ff:fe63:629f%eth0:0], discPort=0, order=72, intOrder=64, lastExchangeTime=1602089183465, loc=false, ver=2.7.0#20181201-sha1:256ae401, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/Server1IP:47100, rmtAddr=/clientIP:52303, createTime=1602089184427, closeTime=0, bytesSent=248963, bytesRcvd=1559, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1602096401587, lastSndTime=1602089666831, lastRcvTime=1602096401587, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@18323d50, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_151]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_151]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_151]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.8.0_151]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_151]
at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275) ~[ignite-core-2.7.0.jar:2.7.0]
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389) [ignite-core-2.7.0.jar:2.7.0]
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156) [ignite-core-2.7.0.jar:2.7.0]
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) [ignite-core-2.7.0.jar:2.7.0]
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.7.0.jar:2.7.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Thanks and Regards,
Kamlesh Joshi
"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system.
Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
"Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email.
and delete this message and any attachments from your system.
Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
Re: [External]Re: Failed to process selector key frequently at client side
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I guess you will have to set it on all nodes in cluster, including client.
Also, 60s may be too long - maybe your network will start closing
connections earlier than that.
Regards,
--
Ilya Kasnacheev
пт, 9 окт. 2020 г. в 15:55, Kamlesh Joshi <Ka...@ril.com>:
> Hi Ilya,
>
>
>
> We could see *idleConnTimeout* value is set at the server level as below.
> Do we have to set this value on both client and server?
>
>
>
>
>
> TcpCommunicationSpi [connectGate=null,
> connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@560348e6,
> enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null,
> locHost=null, locPort=47100, locPortRange=100, shmemPort=-1,
> directBuf=true, directSndBuf=false, *idleConnTimeout=600000*,
> connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768,
> sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null,
> shmemSrv=null, usePairedConnections=false, connectionsPerNode=1,
> tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32,
> unackedMsgsBufSize=0, sockWriteTimeout=60000, boundTcpPort=-1,
> boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null,
> ctxInitLatch=java.util.concurrent.CountDownLatch@1df8b5b8[Count = 1],
> stopping=false],
>
>
>
>
>
>
>
> *Thanks and Regards,*
>
> *Kamlesh Joshi*
>
>
>
> *From:* Ilya Kasnacheev <il...@gmail.com>
> *Sent:* 09 October 2020 17:14
> *To:* user@ignite.apache.org
> *Subject:* [External]Re: Failed to process selector key frequently at
> client side
>
>
>
> The e-mail below is from an external source. Please do not open
> attachments or click links from an unknown or suspicious origin.
>
> Hello!
>
>
>
> It seems that the network that you are using will close idle connections.
> You can try sidestepping it by specifying
> TcpCommunicationSpi.idleConnectionTimeout to some value such as 25000 ms.
>
>
>
> In this case Ignite will be first to close these connections explicitly.
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> пт, 9 окт. 2020 г. в 09:37, Kamlesh Joshi <Ka...@ril.com>:
>
> Hi Igniters,
>
>
>
> We have three server nodes, there is firewall between clients and servers.
>
>
>
> Opened 47500 and 47100 series ports, but still below errors are printing
> at client side frequently. But client application able to get data from
> cluster, could please help us how to avoid these errors at client side.
>
>
>
> We are using Ignite 2.7.0 version.
>
>
>
> [2020-10-08T00:26:23,389][ERROR][grid-nio-worker-tcp-comm-0-#24%JBPLCustomer%][TcpCommunicationSpi]
> Failed to process selector key [ses=GridSelectorNioSessionImpl
> [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0,
> bytesRcvd=84608, bytesSent=1725593, bytesRcvd0=0, bytesSent0=0,
> select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0,
> igniteInstanceName=JBPLCustomer, finished=false, heartbeatTs=1602096980980,
> hashCode=1228592538, interrupted=false,
> runner=grid-nio-worker-tcp-comm-0-#24%JBPLCustomer%]]],
> writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
> readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
> inRecovery=GridNioRecoveryDescriptor [acked=48, resendCnt=0, rcvCnt=58,
> sentCnt=48, reserved=true, lastAck=58, nodeLeft=false,
> node=TcpDiscoveryNode [id=ef6723ee-f70d-4515-89b9-7a3e85ab5072,
> addrs=[0:0:0:0:0:0:0:1%lo, clientIP, 127.0.0.1,
> 2405:200:a60:fd04:20c:29ff:fe63:629f%eth0], sockAddrs=[/clientIP:0,
> 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0
> <https://protect2.fireeye.com/v1/url?k=29480a65-756b19ca-2949809a-00259087546c-f07172281c791ac1&q=1&e=ae47e4b2-c5cf-41b8-9e41-a11ed2e2ec12&u=http%3A%2F%2F127.0.0.1%3A0%2F>,
> 2405:200:a60:fd04:20c:29ff:fe63:629f%eth0:0], discPort=0, order=72,
> intOrder=64, lastExchangeTime=1602089183465, loc=false,
> ver=2.7.0#20181201-sha1:256ae401, isClient=true], connected=true,
> connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false],
> outRecovery=GridNioRecoveryDescriptor [acked=48, resendCnt=0, rcvCnt=58,
> sentCnt=48, reserved=true, lastAck=58, nodeLeft=false,
> node=TcpDiscoveryNode [id=ef6723ee-f70d-4515-89b9-7a3e85ab5072,
> addrs=[0:0:0:0:0:0:0:1%lo, clientIP, 127.0.0.1,
> 2405:200:a60:fd04:20c:29ff:fe63:629f%eth0], sockAddrs=[/clientIP:0,
> 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0
> <https://protect2.fireeye.com/v1/url?k=ef3f61e8-b31c7247-ef3eeb17-00259087546c-d45fc1450e7efc2c&q=1&e=ae47e4b2-c5cf-41b8-9e41-a11ed2e2ec12&u=http%3A%2F%2F127.0.0.1%3A0%2F>,
> 2405:200:a60:fd04:20c:29ff:fe63:629f%eth0:0], discPort=0, order=72,
> intOrder=64, lastExchangeTime=1602089183465, loc=false,
> ver=2.7.0#20181201-sha1:256ae401, isClient=true], connected=true,
> connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false],
> super=GridNioSessionImpl [locAddr=/Server1IP:47100,
> rmtAddr=/clientIP:52303, createTime=1602089184427, closeTime=0,
> bytesSent=248963, bytesRcvd=1559, bytesSent0=0, bytesRcvd0=0,
> sndSchedTime=1602096401587, lastSndTime=1602089666831,
> lastRcvTime=1602096401587, readsPaused=false,
> filterChain=FilterChain[filters=[GridNioCodecFilter
> [parser=o.a.i.i.util.nio.GridDirectParser@18323d50, directMode=true],
> GridConnectionBytesVerifyFilter], accepted=true, markedForClose=false]]]
>
> java.io.IOException: Connection reset by peer
>
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> ~[?:1.8.0_151]
>
> at
> sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_151]
>
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> ~[?:1.8.0_151]
>
> at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.8.0_151]
>
> at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_151]
>
> at
> org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1275)
> ~[ignite-core-2.7.0.jar:2.7.0]
>
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2389)
> [ignite-core-2.7.0.jar:2.7.0]
>
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2156)
> [ignite-core-2.7.0.jar:2.7.0]
>
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> [ignite-core-2.7.0.jar:2.7.0]
>
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> [ignite-core-2.7.0.jar:2.7.0]
>
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
>
>
>
>
>
> *Thanks and Regards,*
>
> *Kamlesh Joshi*
>
>
>
>
> "*Confidentiality Warning*: This message and any attachments are intended
> only for the use of the intended recipient(s), are confidential and may be
> privileged. If you are not the intended recipient, you are hereby notified
> that any review, re-transmission, conversion to hard copy, copying,
> circulation or other use of this message and any attachments is strictly
> prohibited. If you are not the intended recipient, please notify the sender
> immediately by return email and delete this message and any attachments
> from your system.
>
> *Virus Warning:* Although the company has taken reasonable precautions to
> ensure no viruses are present in this email. The company cannot accept
> responsibility for any loss or damage arising from the use of this email or
> attachment."
>
>
> "*Confidentiality Warning*: This message and any attachments are intended
> only for the use of the intended recipient(s), are confidential and may be
> privileged. If you are not the intended recipient, you are hereby notified
> that any review, re-transmission, conversion to hard copy, copying,
> circulation or other use of this message and any attachments is strictly
> prohibited. If you are not the intended recipient, please notify the sender
> immediately by return email and delete this message and any attachments
> from your system.
>
> *Virus Warning:* Although the company has taken reasonable precautions to
> ensure no viruses are present in this email. The company cannot accept
> responsibility for any loss or damage arising from the use of this email or
> attachment."
>