You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by "Ilya Suntsov (JIRA)" <ji...@apache.org> on 2017/12/15 11:43:00 UTC
[jira] [Created] (IGNITE-7212) Load stoped after server node kill
Ilya Suntsov created IGNITE-7212:
------------------------------------
Summary: Load stoped after server node kill
Key: IGNITE-7212
URL: https://issues.apache.org/jira/browse/IGNITE-7212
Project: Ignite
Issue Type: Bug
Components: general
Affects Versions: 2.4
Reporter: Ilya Suntsov
Priority: Critical
Scenario:
* Start 4 servers
* Start load tasks on 5 clients
* Kill 1 server
* Waiting for rebalancing
* Kill 1 server
Result:
After the kill of second servers node load stoped.
In servers logs I see messages like this:
{noformat}
[2017-12-15 11:23:50][DEBUG][grid-nio-worker-tcp-comm-0-#41] Remote client closed connection: GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=130952565, bytesSent=131203245, bytesRcvd0=3069200, bytesSent0=3068083, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=null, finished=false, hashCode=1748650517, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#41]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=1024, resendCnt=0, rcvCnt=1026, sentCnt=1029, reserved=true, lastAck=1024, nodeLeft=false, node=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=1024, resendCnt=0, rcvCnt=1026, sentCnt=1029, reserved=true, lastAck=1024, nodeLeft=false, node=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/172.31.23.220:41732, rmtAddr=ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47100, createTime=1513335774008, closeTime=0, bytesSent=131203245, bytesRcvd=130952565, bytesSent0=3068083, bytesRcvd0=3069200, sndSchedTime=1513335774008, lastSndTime=1513337029027, lastRcvTime=1513337029027, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@11ae7d3b, directMode=true], GridConnectionBytesVerifyFilter], accepted=false]]
[2017-12-15 11:23:50][WARN ][tcp-disco-msg-worker-#2] Failed to send message to next node [msg=TcpDiscoveryConnectionCheckMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=6c7f6d95061-c3cf9fe4-ab13-4d95-ace3-84a54cd73e08, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryConnectionCheckMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=6c7f6d95061-c3cf9fe4-ab13-4d95-ace3-84a54cd73e08, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, order=1, addr=[127.0.0.1, 172.31.20.3], daemon=false]]]
[2017-12-15 11:23:50][DEBUG][grid-nio-worker-tcp-comm-0-#41] Session was closed but there are unacknowledged messages, will try to reconnect [rmtNode=b7cfaa4e-b3b7-4485-a421-c731d9ed869d]
[2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Recovery reconnect [rmtNode=b7cfaa4e-b3b7-4485-a421-c731d9ed869d]
[2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Creating NIO client to node: TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false]
[2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Addresses resolved from attributes [rmtNode=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47100, /127.0.0.1:47100], isRmtAddrsExist=true]
[2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Client creation failed [addr=ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47100, err=java.net.ConnectException: Connection refused]
[2017-12-15 11:23:50][WARN ][tcp-comm-worker-#1] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47100, failureDetectionTimeout=10000]
[2017-12-15 11:23:50][WARN ][disco-event-worker-#61] Node FAILED: TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false]
[2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Skipping local address [addr=/127.0.0.1:47100, locAddrs=[172.31.20.3, 127.0.0.1], node=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false]]
[2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Skipping local address [addr=/127.0.0.1:47100, locAddrs=[172.31.20.3, 127.0.0.1], node=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 172.31.20.3], sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, isClient=false]]
{noformat}
Logs and configs was attached to this ticket.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)