You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Jason <fq...@outlook.com> on 2016/08/03 03:55:42 UTC
Re: Failed to wait for initial partition map exchange
hi Val,
seems that when there's assertion or OOM in one node, it doesn't exit,
right? so it still sends heartbeat to others, then the whole cluster are
waiting for it to recover (hanging).
Is there any easy way to let one node restart immediately when encounter
unrecoverable errors, like OOM or severe assertion?
Thanks,
-Jason
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p6691.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by ndipiazza3565 <ni...@lucidworks.com>.
No. Persistence is disabled in my case.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to wait for initial partition map exchange
Posted by eugene miretsky <eu...@gmail.com>.
Do you have persistence enabled?
On Wed, Sep 12, 2018 at 6:31 PM ndipiazza3565 <
nicholas.dipiazza@lucidworks.com> wrote:
> I'm trying to build up a list of possible causes for this issue.
>
> I'm only really interested in the issues that occur after successful
> production deployments. Meaning the environment has been up for some time
> successfully, but then later on our ignite nodes will not start and stick
>
> But as of now, a certain bad behavior from a single node in the ignite
> cluster can cause a deadlock
>
> * Anything that causes one of the ignite nodes to become unresponsive
> * oom
> * high gc
> * high cpu
> * high disk usage
> * Network issues?
>
> I'm trying to get a list of the causes for this issue so I can troubleshoot
> further.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Failed to wait for initial partition map exchange
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Regarding PME problems.
OOM will cause this. High GC could cause this under some circumstances.
High CPU or Disk usage should not cause this. Network inavailability (such
as closed communication port) could also cause it.
But the prime cause is programming errors. Either those are errors on
Apache Ignite side (caused by some strange circumstances since all normal
cases should be normally tested), or they are in your code.
Such as deadlocks. If you have deadlocks in your code exposed to Apache
Ignite, or you are manage to lock up Apache Ignite in other ways
(listeners, invokes and continuous queries are notorious for that, since
there are limitations on operations you can use from within them), you can
catch infinite PME very easily.
However, it's hard to say without reviewing logs and thread dumps./
Regards,
--
Ilya Kasnacheev
чт, 13 сент. 2018 г. в 1:31, ndipiazza3565 <nicholas.dipiazza@lucidworks.com
>:
> I'm trying to build up a list of possible causes for this issue.
>
> I'm only really interested in the issues that occur after successful
> production deployments. Meaning the environment has been up for some time
> successfully, but then later on our ignite nodes will not start and stick
>
> But as of now, a certain bad behavior from a single node in the ignite
> cluster can cause a deadlock
>
> * Anything that causes one of the ignite nodes to become unresponsive
> * oom
> * high gc
> * high cpu
> * high disk usage
> * Network issues?
>
> I'm trying to get a list of the causes for this issue so I can troubleshoot
> further.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Failed to wait for initial partition map exchange
Posted by ndipiazza3565 <ni...@lucidworks.com>.
I'm trying to build up a list of possible causes for this issue.
I'm only really interested in the issues that occur after successful
production deployments. Meaning the environment has been up for some time
successfully, but then later on our ignite nodes will not start and stick
But as of now, a certain bad behavior from a single node in the ignite
cluster can cause a deadlock
* Anything that causes one of the ignite nodes to become unresponsive
* oom
* high gc
* high cpu
* high disk usage
* Network issues?
I'm trying to get a list of the causes for this issue so I can troubleshoot
further.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to wait for initial partition map exchange
Posted by vkulichenko <va...@gmail.com>.
This looks like a network issue. Please check that connections can be
established between any nodes in topology and nothing is blocked by
firewall. Note that connectivity must be bidirectional, even for clients
(i.e. server node can establish TCP connection with client node). Also note
that both discovery and communication are required to work and they use
different ports (47500 and 47100 by default).
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p14283.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Vladimir <vl...@yandex.ru>.
INFO 2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[OS: Linux
4.1.12-61.1.18.el7uek.x86_64 amd64]]
INFO 2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[PID: 7608]]
[20:36:22] VM information: Java(TM) SE Runtime Environment 1.8.0_102-b14
Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.102-b14
INFO 2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Language runtime: Java
Platform API Specification ver. 1.8]]
INFO 2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[VM information: Java(TM) SE
Runtime Environment 1.8.0_102-b14 Oracle Corporation Java HotSpot(TM) 64-Bit
Server VM 25.102-b14]]
INFO 2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[VM total memory: 2.1GB]]
INFO 2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Remote Management [restart:
off, REST: on, JMX (remote: off)]]]
INFO 2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[System cache's MemoryPolicy
size is configured to 40 MB. Use MemoryConfiguration.systemCacheMemorySize
property to change the setting.]]
INFO 2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Configured caches [in
'default' memoryPolicy: ['ignite-sys-cache', 'ignite-atomics-sys-cache']]]]
INFO 2017-06-13 20:36:22 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Local node user attribute
[IgSupport_LogicClusterGroups=com.bpcbt.common.support.ignite.beans.IgLogicClusterGroups@0]]]
WARN 2017-06-13 20:36:22 [pub-#14%svip%]
org.apache.ignite.internal.GridDiagnostic - [[Initial heap size is 154MB
(should be no less than 512MB, use -Xms512m -Xmx512m).]]
[20:36:22] Initial heap size is 154MB (should be no less than 512MB, use
-Xms512m -Xmx512m).
[20:36:23] Configured plugins:
INFO 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.processors.plugin.IgnitePluginProcessor -
[[Configured plugins:]]
[20:36:23] ^-- None
INFO 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.processors.plugin.IgnitePluginProcessor - [[ ^--
None]]
[20:36:23]
INFO 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.processors.plugin.IgnitePluginProcessor - [[]]
INFO 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[Successfully
bound communication NIO server to TCP port [port=9343, locHost=/127.0.0.1,
selectorsCnt=4, selectorSpins=0, pairedConn=false]]]
WARN 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[Message
queue limit is set to 0 which may lead to potential OOMEs when running cache
operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth
on sender and receiver sides.]]
[20:36:23] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
WARN 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.spi.checkpoint.noop.NoopCheckpointSpi - [[Checkpoints are
disabled (to enable configure any GridCheckpointSpi implementation)]]
WARN 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.managers.collision.GridCollisionManager -
[[Collision resolution is disabled (all jobs will be activated upon
arrival).]]
[20:36:23] Security status [authentication=off, tls/ssl=off]
INFO 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Security status
[authentication=off, tls/ssl=off]]]
INFO 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestProtocol
- [[Command protocol successfully started [name=TCP binary,
host=0.0.0.0/0.0.0.0, port=11214]]]
INFO 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Non-loopback local IPs:
192.168.122.1, 192.168.209.65]]
INFO 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.internal.IgniteKernal%svip - [[Enabled local MACs:
0021F6321229, 5254000A6937]]
INFO 2017-06-13 20:36:23 [localhost-startStop-1]
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - [[Successfully bound
to TCP port [port=9463, localHost=/127.0.0.1,
locNodeId=7fc5ea30-913d-4ff5-932b-62d81c6027db]]]
INFO 2017-06-13 20:36:24 [localhost-startStop-1]
org.apache.ignite.internal.processors.cache.GridCacheProcessor - [[Started
cache [name=ignite-sys-cache, memoryPolicyName=sysMemPlc, mode=REPLICATED]]]
INFO 2017-06-13 20:36:24 [localhost-startStop-1]
org.apache.ignite.internal.processors.cache.GridCacheProcessor - [[Started
cache [name=ignite-atomics-sys-cache, memoryPolicyName=sysMemPlc,
mode=REPLICATED]]]
INFO 2017-06-13 20:36:24 [grid-nio-worker-tcp-comm-0-#17%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[Established
outgoing communication connection [locAddr=/127.0.0.1:47331,
rmtAddr=/127.0.0.1:9340]]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Failed to wait for partition map exchange
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0],
node=7fc5ea30-913d-4ff5-932b-62d81c6027db]. Dumping pending objects that
might be the cause: ]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Ready affinity version: AffinityTopologyVersion [topVer=-1,
minorTopVer=0]]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false,
forcePreload=false, reassign=false, discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=7fc5ea30-913d-4ff5-932b-62d81c6027db,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62,
intOrder=33, lastExchangeTime=1497353794662, loc=true,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], topVer=62,
nodeId8=7fc5ea30, msg=null, type=NODE_JOINED, tstamp=1497353784308],
crd=TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=62, minorTopVer=0], nodeId=7fc5ea30, evt=NODE_JOINED], added=false,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true,
hash=1130708319], init=true, lastVer=null, partReleaseFut=GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null,
futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false,
clientOnlyExchange=false, initTs=1497353784601, centralizedAff=false,
changeGlobalStateE=null, exchangeOnChangeGlobalState=false,
forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353794662, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending exchange futures:]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last 10 exchange futures (total: 1):]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[>>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353794662, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353794662, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transactions:]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending explicit locks:]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending cache futures:]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending atomic cache futures:]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending data streamer futures:]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transaction deadlock detection futures:]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi -
[[Communication SPI recovery descriptors:
[key=ConnectionKey [nodeId=f03fe6d2-8858-4b32-9ac2-260d1a79693a, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=3, msgsRcvd=2, lastAcked=0,
reserveCnt=1, descIdHash=2015302722]
Communication SPI clients:
[node=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker
[selector=sun.nio.ch.EPollSelectorImpl@3e96a835, idx=0, bytesRcvd=45363,
bytesSent=1305, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-0, igniteInstanceName=svip, finished=false,
hashCode=518525437, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#17%svip%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/127.0.0.1:47331, rmtAddr=/127.0.0.1:9340,
createTime=1497353784631, closeTime=0, bytesSent=1305, bytesRcvd=45363,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1497353784631,
lastSndTime=1497353784897, lastRcvTime=1497353787145, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5e7030df, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]],
super=GridAbstractCommunicationClient [lastUsed=1497353784642, closed=false,
connIdx=0]]]
]]
WARN 2017-06-13 20:36:34 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[NIO server
statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0]]]
WARN 2017-06-13 20:36:34 [grid-nio-worker-tcp-comm-3-#20%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:36:34 [grid-nio-worker-tcp-comm-1-#18%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=1, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:36:34 [grid-nio-worker-tcp-comm-2-#19%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:36:34 [grid-nio-worker-tcp-comm-0-#17%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=0, keysCnt=1, bytesRcvd=45363, bytesRcvd0=0,
>> bytesSent=1305, bytesSent0=0]
Connection info [in=false, rmtAddr=/127.0.0.1:9340,
locAddr=/127.0.0.1:47331, msgsSent=3, msgsAckedByRmt=3,
descIdHash=2015302722, msgsRcvd=2, lastAcked=0, descIdHash=2015302722,
bytesRcvd=45363, bytesRcvd0=0, bytesSent=1305, bytesSent0=0, opQueueSize=0,
msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0,
stack=[StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
null, null, null, null, null, null]]], msgReader=DirectMessageReader
[state=DirectMessageState [pos=0, stack=[StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013892784, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null,
null, null, null, null, null, null]], lastRead=true]]
]]
WARN 2017-06-13 20:36:44 [localhost-startStop-1]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Failed to wait for initial partition map exchange. Possible reasons are:
^-- Transactions in deadlock.
^-- Long running transactions (ignore if this is the case).
^-- Unreleased explicit locks.]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Failed to wait for partition map exchange
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0],
node=7fc5ea30-913d-4ff5-932b-62d81c6027db]. Dumping pending objects that
might be the cause: ]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Ready affinity version: AffinityTopologyVersion [topVer=-1,
minorTopVer=0]]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false,
forcePreload=false, reassign=false, discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=7fc5ea30-913d-4ff5-932b-62d81c6027db,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62,
intOrder=33, lastExchangeTime=1497353804668, loc=true,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], topVer=62,
nodeId8=7fc5ea30, msg=null, type=NODE_JOINED, tstamp=1497353784308],
crd=TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=62, minorTopVer=0], nodeId=7fc5ea30, evt=NODE_JOINED], added=false,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true,
hash=1130708319], init=true, lastVer=null, partReleaseFut=GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null,
futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false,
clientOnlyExchange=false, initTs=1497353784601, centralizedAff=false,
changeGlobalStateE=null, exchangeOnChangeGlobalState=false,
forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353804668, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending exchange futures:]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last 10 exchange futures (total: 1):]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[>>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353804668, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353804668, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transactions:]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending explicit locks:]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending cache futures:]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending atomic cache futures:]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending data streamer futures:]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transaction deadlock detection futures:]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi -
[[Communication SPI recovery descriptors:
[key=ConnectionKey [nodeId=f03fe6d2-8858-4b32-9ac2-260d1a79693a, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=3, msgsRcvd=2, lastAcked=0,
reserveCnt=1, descIdHash=2015302722]
Communication SPI clients:
[node=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker
[selector=sun.nio.ch.EPollSelectorImpl@3e96a835, idx=0, bytesRcvd=45363,
bytesSent=1305, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-0, igniteInstanceName=svip, finished=false,
hashCode=518525437, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#17%svip%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/127.0.0.1:47331, rmtAddr=/127.0.0.1:9340,
createTime=1497353784631, closeTime=0, bytesSent=1305, bytesRcvd=45363,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1497353784631,
lastSndTime=1497353784897, lastRcvTime=1497353787145, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5e7030df, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]],
super=GridAbstractCommunicationClient [lastUsed=1497353784642, closed=false,
connIdx=0]]]
]]
WARN 2017-06-13 20:36:44 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[NIO server
statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0]]]
WARN 2017-06-13 20:36:44 [grid-nio-worker-tcp-comm-0-#17%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=0, keysCnt=1, bytesRcvd=45363, bytesRcvd0=0,
>> bytesSent=1305, bytesSent0=0]
Connection info [in=false, rmtAddr=/127.0.0.1:9340,
locAddr=/127.0.0.1:47331, msgsSent=3, msgsAckedByRmt=3,
descIdHash=2015302722, msgsRcvd=2, lastAcked=0, descIdHash=2015302722,
bytesRcvd=45363, bytesRcvd0=0, bytesSent=1305, bytesSent0=0, opQueueSize=0,
msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0,
stack=[StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
null, null, null, null, null, null]]], msgReader=DirectMessageReader
[state=DirectMessageState [pos=0, stack=[StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013892784, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null,
null, null, null, null, null, null]], lastRead=true]]
]]
WARN 2017-06-13 20:36:44 [grid-nio-worker-tcp-comm-2-#19%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:36:44 [grid-nio-worker-tcp-comm-3-#20%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:36:44 [grid-nio-worker-tcp-comm-1-#18%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=1, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Failed to wait for partition map exchange
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0],
node=7fc5ea30-913d-4ff5-932b-62d81c6027db]. Dumping pending objects that
might be the cause: ]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Ready affinity version: AffinityTopologyVersion [topVer=-1,
minorTopVer=0]]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false,
forcePreload=false, reassign=false, discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=7fc5ea30-913d-4ff5-932b-62d81c6027db,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62,
intOrder=33, lastExchangeTime=1497353814702, loc=true,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], topVer=62,
nodeId8=7fc5ea30, msg=null, type=NODE_JOINED, tstamp=1497353784308],
crd=TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=62, minorTopVer=0], nodeId=7fc5ea30, evt=NODE_JOINED], added=false,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true,
hash=1130708319], init=true, lastVer=null, partReleaseFut=GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null,
futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false,
clientOnlyExchange=false, initTs=1497353784601, centralizedAff=false,
changeGlobalStateE=null, exchangeOnChangeGlobalState=false,
forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353814702, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending exchange futures:]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last 10 exchange futures (total: 1):]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[>>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353814702, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353814702, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transactions:]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending explicit locks:]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending cache futures:]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending atomic cache futures:]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending data streamer futures:]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transaction deadlock detection futures:]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi -
[[Communication SPI recovery descriptors:
[key=ConnectionKey [nodeId=f03fe6d2-8858-4b32-9ac2-260d1a79693a, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=3, msgsRcvd=2, lastAcked=2,
reserveCnt=1, descIdHash=2015302722]
Communication SPI clients:
[node=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker
[selector=sun.nio.ch.EPollSelectorImpl@3e96a835, idx=0, bytesRcvd=45363,
bytesSent=1315, bytesRcvd0=0, bytesSent0=10, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-0, igniteInstanceName=svip, finished=false,
hashCode=518525437, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#17%svip%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/127.0.0.1:47331, rmtAddr=/127.0.0.1:9340,
createTime=1497353784631, closeTime=0, bytesSent=1315, bytesRcvd=45363,
bytesSent0=10, bytesRcvd0=0, sndSchedTime=1497353784631,
lastSndTime=1497353813114, lastRcvTime=1497353787145, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5e7030df, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]],
super=GridAbstractCommunicationClient [lastUsed=1497353784642, closed=false,
connIdx=0]]]
]]
WARN 2017-06-13 20:36:54 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[NIO server
statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0]]]
WARN 2017-06-13 20:36:54 [grid-nio-worker-tcp-comm-0-#17%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=0, keysCnt=1, bytesRcvd=45363, bytesRcvd0=0,
>> bytesSent=1315, bytesSent0=10]
Connection info [in=false, rmtAddr=/127.0.0.1:9340,
locAddr=/127.0.0.1:47331, msgsSent=3, msgsAckedByRmt=3,
descIdHash=2015302722, msgsRcvd=2, lastAcked=2, descIdHash=2015302722,
bytesRcvd=45363, bytesRcvd0=0, bytesSent=1315, bytesSent0=10, opQueueSize=0,
msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0,
stack=[StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
null, null, null, null, null, null]]], msgReader=DirectMessageReader
[state=DirectMessageState [pos=0, stack=[StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013892784, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null,
null, null, null, null, null, null]], lastRead=true]]
]]
WARN 2017-06-13 20:36:54 [grid-nio-worker-tcp-comm-1-#18%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=1, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:36:54 [grid-nio-worker-tcp-comm-2-#19%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:36:54 [grid-nio-worker-tcp-comm-3-#20%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:04 [localhost-startStop-1]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Still waiting for initial partition map exchange
[fut=GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353824361, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353824361, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Failed to wait for partition map exchange
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0],
node=7fc5ea30-913d-4ff5-932b-62d81c6027db]. Dumping pending objects that
might be the cause: ]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Ready affinity version: AffinityTopologyVersion [topVer=-1,
minorTopVer=0]]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false,
forcePreload=false, reassign=false, discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=7fc5ea30-913d-4ff5-932b-62d81c6027db,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62,
intOrder=33, lastExchangeTime=1497353824726, loc=true,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], topVer=62,
nodeId8=7fc5ea30, msg=null, type=NODE_JOINED, tstamp=1497353784308],
crd=TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=62, minorTopVer=0], nodeId=7fc5ea30, evt=NODE_JOINED], added=false,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true,
hash=1130708319], init=true, lastVer=null, partReleaseFut=GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null,
futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false,
clientOnlyExchange=false, initTs=1497353784601, centralizedAff=false,
changeGlobalStateE=null, exchangeOnChangeGlobalState=false,
forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353824726, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending exchange futures:]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last 10 exchange futures (total: 1):]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[>>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353824726, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353824726, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transactions:]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending explicit locks:]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending cache futures:]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending atomic cache futures:]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending data streamer futures:]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transaction deadlock detection futures:]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi -
[[Communication SPI recovery descriptors:
[key=ConnectionKey [nodeId=f03fe6d2-8858-4b32-9ac2-260d1a79693a, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=3, msgsRcvd=2, lastAcked=2,
reserveCnt=1, descIdHash=2015302722]
Communication SPI clients:
[node=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker
[selector=sun.nio.ch.EPollSelectorImpl@3e96a835, idx=0, bytesRcvd=45363,
bytesSent=1315, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-0, igniteInstanceName=svip, finished=false,
hashCode=518525437, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#17%svip%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/127.0.0.1:47331, rmtAddr=/127.0.0.1:9340,
createTime=1497353784631, closeTime=0, bytesSent=1315, bytesRcvd=45363,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1497353818946,
lastSndTime=1497353813114, lastRcvTime=1497353818946, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5e7030df, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]],
super=GridAbstractCommunicationClient [lastUsed=1497353784642, closed=false,
connIdx=0]]]
]]
WARN 2017-06-13 20:37:04 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[NIO server
statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0]]]
WARN 2017-06-13 20:37:04 [grid-nio-worker-tcp-comm-0-#17%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=0, keysCnt=1, bytesRcvd=45363, bytesRcvd0=0,
>> bytesSent=1315, bytesSent0=0]
Connection info [in=false, rmtAddr=/127.0.0.1:9340,
locAddr=/127.0.0.1:47331, msgsSent=3, msgsAckedByRmt=3,
descIdHash=2015302722, msgsRcvd=2, lastAcked=2, descIdHash=2015302722,
bytesRcvd=45363, bytesRcvd0=0, bytesSent=1315, bytesSent0=0, opQueueSize=0,
msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0,
stack=[StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
null, null, null, null, null, null]]], msgReader=DirectMessageReader
[state=DirectMessageState [pos=0, stack=[StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013892784, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null,
null, null, null, null, null, null]], lastRead=true]]
]]
WARN 2017-06-13 20:37:04 [grid-nio-worker-tcp-comm-2-#19%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:04 [grid-nio-worker-tcp-comm-3-#20%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:04 [grid-nio-worker-tcp-comm-1-#18%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=1, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Failed to wait for partition map exchange
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0],
node=7fc5ea30-913d-4ff5-932b-62d81c6027db]. Dumping pending objects that
might be the cause: ]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Ready affinity version: AffinityTopologyVersion [topVer=-1,
minorTopVer=0]]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false,
forcePreload=false, reassign=false, discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=7fc5ea30-913d-4ff5-932b-62d81c6027db,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62,
intOrder=33, lastExchangeTime=1497353834746, loc=true,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], topVer=62,
nodeId8=7fc5ea30, msg=null, type=NODE_JOINED, tstamp=1497353784308],
crd=TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=62, minorTopVer=0], nodeId=7fc5ea30, evt=NODE_JOINED], added=false,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true,
hash=1130708319], init=true, lastVer=null, partReleaseFut=GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null,
futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false,
clientOnlyExchange=false, initTs=1497353784601, centralizedAff=false,
changeGlobalStateE=null, exchangeOnChangeGlobalState=false,
forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353834746, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending exchange futures:]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last 10 exchange futures (total: 1):]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[>>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353834746, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353834746, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transactions:]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending explicit locks:]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending cache futures:]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending atomic cache futures:]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending data streamer futures:]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transaction deadlock detection futures:]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi -
[[Communication SPI recovery descriptors:
[key=ConnectionKey [nodeId=f03fe6d2-8858-4b32-9ac2-260d1a79693a, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=3, msgsRcvd=2, lastAcked=2,
reserveCnt=1, descIdHash=2015302722]
Communication SPI clients:
[node=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker
[selector=sun.nio.ch.EPollSelectorImpl@3e96a835, idx=0, bytesRcvd=45363,
bytesSent=1315, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-0, igniteInstanceName=svip, finished=false,
hashCode=518525437, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#17%svip%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/127.0.0.1:47331, rmtAddr=/127.0.0.1:9340,
createTime=1497353784631, closeTime=0, bytesSent=1315, bytesRcvd=45363,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1497353818946,
lastSndTime=1497353813114, lastRcvTime=1497353818946, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5e7030df, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]],
super=GridAbstractCommunicationClient [lastUsed=1497353784642, closed=false,
connIdx=0]]]
]]
WARN 2017-06-13 20:37:14 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[NIO server
statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0]]]
WARN 2017-06-13 20:37:14 [grid-nio-worker-tcp-comm-0-#17%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=0, keysCnt=1, bytesRcvd=45363, bytesRcvd0=0,
>> bytesSent=1315, bytesSent0=0]
Connection info [in=false, rmtAddr=/127.0.0.1:9340,
locAddr=/127.0.0.1:47331, msgsSent=3, msgsAckedByRmt=3,
descIdHash=2015302722, msgsRcvd=2, lastAcked=2, descIdHash=2015302722,
bytesRcvd=45363, bytesRcvd0=0, bytesSent=1315, bytesSent0=0, opQueueSize=0,
msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0,
stack=[StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
null, null, null, null, null, null]]], msgReader=DirectMessageReader
[state=DirectMessageState [pos=0, stack=[StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013892784, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null,
null, null, null, null, null, null]], lastRead=true]]
]]
WARN 2017-06-13 20:37:14 [grid-nio-worker-tcp-comm-1-#18%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=1, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:14 [grid-nio-worker-tcp-comm-2-#19%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:14 [grid-nio-worker-tcp-comm-3-#20%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:24 [localhost-startStop-1]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Still waiting for initial partition map exchange
[fut=GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353844387, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353844387, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Failed to wait for partition map exchange
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0],
node=7fc5ea30-913d-4ff5-932b-62d81c6027db]. Dumping pending objects that
might be the cause: ]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Ready affinity version: AffinityTopologyVersion [topVer=-1,
minorTopVer=0]]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false,
forcePreload=false, reassign=false, discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=7fc5ea30-913d-4ff5-932b-62d81c6027db,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62,
intOrder=33, lastExchangeTime=1497353844791, loc=true,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], topVer=62,
nodeId8=7fc5ea30, msg=null, type=NODE_JOINED, tstamp=1497353784308],
crd=TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=62, minorTopVer=0], nodeId=7fc5ea30, evt=NODE_JOINED], added=false,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true,
hash=1130708319], init=true, lastVer=null, partReleaseFut=GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null,
futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false,
clientOnlyExchange=false, initTs=1497353784601, centralizedAff=false,
changeGlobalStateE=null, exchangeOnChangeGlobalState=false,
forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353844791, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending exchange futures:]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last 10 exchange futures (total: 1):]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[>>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353844791, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353844791, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transactions:]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending explicit locks:]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending cache futures:]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending atomic cache futures:]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending data streamer futures:]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transaction deadlock detection futures:]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi -
[[Communication SPI recovery descriptors:
[key=ConnectionKey [nodeId=f03fe6d2-8858-4b32-9ac2-260d1a79693a, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=3, msgsRcvd=2, lastAcked=2,
reserveCnt=1, descIdHash=2015302722]
Communication SPI clients:
[node=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker
[selector=sun.nio.ch.EPollSelectorImpl@3e96a835, idx=0, bytesRcvd=45363,
bytesSent=1315, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-0, igniteInstanceName=svip, finished=false,
hashCode=518525437, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#17%svip%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/127.0.0.1:47331, rmtAddr=/127.0.0.1:9340,
createTime=1497353784631, closeTime=0, bytesSent=1315, bytesRcvd=45363,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1497353818946,
lastSndTime=1497353813114, lastRcvTime=1497353818946, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5e7030df, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]],
super=GridAbstractCommunicationClient [lastUsed=1497353784642, closed=false,
connIdx=0]]]
]]
WARN 2017-06-13 20:37:24 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[NIO server
statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0]]]
WARN 2017-06-13 20:37:24 [grid-nio-worker-tcp-comm-0-#17%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=0, keysCnt=1, bytesRcvd=45363, bytesRcvd0=0,
>> bytesSent=1315, bytesSent0=0]
Connection info [in=false, rmtAddr=/127.0.0.1:9340,
locAddr=/127.0.0.1:47331, msgsSent=3, msgsAckedByRmt=3,
descIdHash=2015302722, msgsRcvd=2, lastAcked=2, descIdHash=2015302722,
bytesRcvd=45363, bytesRcvd0=0, bytesSent=1315, bytesSent0=0, opQueueSize=0,
msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0,
stack=[StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
null, null, null, null, null, null]]], msgReader=DirectMessageReader
[state=DirectMessageState [pos=0, stack=[StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013892784, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null,
null, null, null, null, null, null]], lastRead=true]]
]]
WARN 2017-06-13 20:37:24 [grid-nio-worker-tcp-comm-1-#18%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=1, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:24 [grid-nio-worker-tcp-comm-2-#19%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:24 [grid-nio-worker-tcp-comm-3-#20%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
INFO 2017-06-13 20:37:34 [ignite-update-notifier-timer]
org.apache.ignite.internal.processors.cluster.GridUpdateNotifier - [[Update
status is not available.]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Failed to wait for partition map exchange
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0],
node=7fc5ea30-913d-4ff5-932b-62d81c6027db]. Dumping pending objects that
might be the cause: ]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Ready affinity version: AffinityTopologyVersion [topVer=-1,
minorTopVer=0]]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false,
forcePreload=false, reassign=false, discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=7fc5ea30-913d-4ff5-932b-62d81c6027db,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62,
intOrder=33, lastExchangeTime=1497353854788, loc=true,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], topVer=62,
nodeId8=7fc5ea30, msg=null, type=NODE_JOINED, tstamp=1497353784308],
crd=TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=62, minorTopVer=0], nodeId=7fc5ea30, evt=NODE_JOINED], added=false,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true,
hash=1130708319], init=true, lastVer=null, partReleaseFut=GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null,
futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false,
clientOnlyExchange=false, initTs=1497353784601, centralizedAff=false,
changeGlobalStateE=null, exchangeOnChangeGlobalState=false,
forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353854788, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending exchange futures:]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last 10 exchange futures (total: 1):]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[>>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353854788, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353854788, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transactions:]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending explicit locks:]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending cache futures:]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending atomic cache futures:]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending data streamer futures:]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transaction deadlock detection futures:]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi -
[[Communication SPI recovery descriptors:
[key=ConnectionKey [nodeId=f03fe6d2-8858-4b32-9ac2-260d1a79693a, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=3, msgsRcvd=2, lastAcked=2,
reserveCnt=1, descIdHash=2015302722]
Communication SPI clients:
[node=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker
[selector=sun.nio.ch.EPollSelectorImpl@3e96a835, idx=0, bytesRcvd=45363,
bytesSent=1315, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-0, igniteInstanceName=svip, finished=false,
hashCode=518525437, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#17%svip%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/127.0.0.1:47331, rmtAddr=/127.0.0.1:9340,
createTime=1497353784631, closeTime=0, bytesSent=1315, bytesRcvd=45363,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1497353848962,
lastSndTime=1497353813114, lastRcvTime=1497353848962, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5e7030df, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]],
super=GridAbstractCommunicationClient [lastUsed=1497353784642, closed=false,
connIdx=0]]]
]]
WARN 2017-06-13 20:37:34 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[NIO server
statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0]]]
WARN 2017-06-13 20:37:34 [grid-nio-worker-tcp-comm-0-#17%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=0, keysCnt=1, bytesRcvd=45363, bytesRcvd0=0,
>> bytesSent=1315, bytesSent0=0]
Connection info [in=false, rmtAddr=/127.0.0.1:9340,
locAddr=/127.0.0.1:47331, msgsSent=3, msgsAckedByRmt=3,
descIdHash=2015302722, msgsRcvd=2, lastAcked=2, descIdHash=2015302722,
bytesRcvd=45363, bytesRcvd0=0, bytesSent=1315, bytesSent0=0, opQueueSize=0,
msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0,
stack=[StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
null, null, null, null, null, null]]], msgReader=DirectMessageReader
[state=DirectMessageState [pos=0, stack=[StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013892784, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null,
null, null, null, null, null, null]], lastRead=true]]
]]
WARN 2017-06-13 20:37:34 [grid-nio-worker-tcp-comm-1-#18%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=1, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:34 [grid-nio-worker-tcp-comm-2-#19%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:34 [grid-nio-worker-tcp-comm-3-#20%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:44 [localhost-startStop-1]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Still waiting for initial partition map exchange
[fut=GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353864442, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353864442, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Failed to wait for partition map exchange
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0],
node=7fc5ea30-913d-4ff5-932b-62d81c6027db]. Dumping pending objects that
might be the cause: ]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Ready affinity version: AffinityTopologyVersion [topVer=-1,
minorTopVer=0]]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false,
forcePreload=false, reassign=false, discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=7fc5ea30-913d-4ff5-932b-62d81c6027db,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62,
intOrder=33, lastExchangeTime=1497353864796, loc=true,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], topVer=62,
nodeId8=7fc5ea30, msg=null, type=NODE_JOINED, tstamp=1497353784308],
crd=TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=62, minorTopVer=0], nodeId=7fc5ea30, evt=NODE_JOINED], added=false,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true,
hash=1130708319], init=true, lastVer=null, partReleaseFut=GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null,
futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false,
clientOnlyExchange=false, initTs=1497353784601, centralizedAff=false,
changeGlobalStateE=null, exchangeOnChangeGlobalState=false,
forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353864796, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending exchange futures:]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last 10 exchange futures (total: 1):]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[>>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353864796, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353864796, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transactions:]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending explicit locks:]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending cache futures:]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending atomic cache futures:]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending data streamer futures:]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transaction deadlock detection futures:]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi -
[[Communication SPI recovery descriptors:
[key=ConnectionKey [nodeId=f03fe6d2-8858-4b32-9ac2-260d1a79693a, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=3, msgsRcvd=2, lastAcked=2,
reserveCnt=1, descIdHash=2015302722]
Communication SPI clients:
[node=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker
[selector=sun.nio.ch.EPollSelectorImpl@3e96a835, idx=0, bytesRcvd=45363,
bytesSent=1315, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-0, igniteInstanceName=svip, finished=false,
hashCode=518525437, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#17%svip%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/127.0.0.1:47331, rmtAddr=/127.0.0.1:9340,
createTime=1497353784631, closeTime=0, bytesSent=1315, bytesRcvd=45363,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1497353848962,
lastSndTime=1497353813114, lastRcvTime=1497353848962, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5e7030df, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]],
super=GridAbstractCommunicationClient [lastUsed=1497353784642, closed=false,
connIdx=0]]]
]]
WARN 2017-06-13 20:37:44 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[NIO server
statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0]]]
WARN 2017-06-13 20:37:44 [grid-nio-worker-tcp-comm-0-#17%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=0, keysCnt=1, bytesRcvd=45363, bytesRcvd0=0,
>> bytesSent=1315, bytesSent0=0]
Connection info [in=false, rmtAddr=/127.0.0.1:9340,
locAddr=/127.0.0.1:47331, msgsSent=3, msgsAckedByRmt=3,
descIdHash=2015302722, msgsRcvd=2, lastAcked=2, descIdHash=2015302722,
bytesRcvd=45363, bytesRcvd0=0, bytesSent=1315, bytesSent0=0, opQueueSize=0,
msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0,
stack=[StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
null, null, null, null, null, null]]], msgReader=DirectMessageReader
[state=DirectMessageState [pos=0, stack=[StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013892784, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null,
null, null, null, null, null, null]], lastRead=true]]
]]
WARN 2017-06-13 20:37:44 [grid-nio-worker-tcp-comm-1-#18%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=1, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:44 [grid-nio-worker-tcp-comm-2-#19%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:44 [grid-nio-worker-tcp-comm-3-#20%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Failed to wait for partition map exchange
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0],
node=7fc5ea30-913d-4ff5-932b-62d81c6027db]. Dumping pending objects that
might be the cause: ]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Ready affinity version: AffinityTopologyVersion [topVer=-1,
minorTopVer=0]]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false,
forcePreload=false, reassign=false, discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=7fc5ea30-913d-4ff5-932b-62d81c6027db,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62,
intOrder=33, lastExchangeTime=1497353874839, loc=true,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], topVer=62,
nodeId8=7fc5ea30, msg=null, type=NODE_JOINED, tstamp=1497353784308],
crd=TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=62, minorTopVer=0], nodeId=7fc5ea30, evt=NODE_JOINED], added=false,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true,
hash=1130708319], init=true, lastVer=null, partReleaseFut=GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null,
futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false,
clientOnlyExchange=false, initTs=1497353784601, centralizedAff=false,
changeGlobalStateE=null, exchangeOnChangeGlobalState=false,
forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353874839, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending exchange futures:]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Last 10 exchange futures (total: 1):]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[>>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353874839, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=62, nodeId8=7fc5ea30, msg=null, type=NODE_JOINED,
tstamp=1497353784308], crd=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=62, minorTopVer=0], nodeId=7fc5ea30,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1130708319], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1497353784601, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[a7a74608-3b2d-440f-910d-3ef3537d907f,
a4a43486-2322-475c-9b25-aef7907500dc, f03fe6d2-8858-4b32-9ac2-260d1a79693a],
srvNodes=[TcpDiscoveryNode [id=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=a4a43486-2322-475c-9b25-aef7907500dc, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=2, intOrder=2,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=a7a74608-3b2d-440f-910d-3ef3537d907f,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=60,
intOrder=32, lastExchangeTime=1497353784020, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=7fc5ea30-913d-4ff5-932b-62d81c6027db, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=62, intOrder=33,
lastExchangeTime=1497353874839, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=277178513]]]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transactions:]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending explicit locks:]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending cache futures:]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending atomic cache futures:]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending data streamer futures:]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager
- [[Pending transaction deadlock detection futures:]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi -
[[Communication SPI recovery descriptors:
[key=ConnectionKey [nodeId=f03fe6d2-8858-4b32-9ac2-260d1a79693a, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=3, msgsRcvd=2, lastAcked=2,
reserveCnt=1, descIdHash=2015302722]
Communication SPI clients:
[node=f03fe6d2-8858-4b32-9ac2-260d1a79693a,
client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker
[selector=sun.nio.ch.EPollSelectorImpl@3e96a835, idx=0, bytesRcvd=45363,
bytesSent=1315, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-0, igniteInstanceName=svip, finished=false,
hashCode=518525437, interrupted=false,
runner=grid-nio-worker-tcp-comm-0-#17%svip%]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false],
outRecovery=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=2,
sentCnt=3, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode
[id=f03fe6d2-8858-4b32-9ac2-260d1a79693a, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1497353784020, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], connected=true, connectCnt=1, queueLimit=4096,
reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl
[locAddr=/127.0.0.1:47331, rmtAddr=/127.0.0.1:9340,
createTime=1497353784631, closeTime=0, bytesSent=1315, bytesRcvd=45363,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1497353848962,
lastSndTime=1497353813114, lastRcvTime=1497353848962, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@5e7030df, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]],
super=GridAbstractCommunicationClient [lastUsed=1497353784642, closed=false,
connIdx=0]]]
]]
WARN 2017-06-13 20:37:54 [exchange-worker-#29%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[NIO server
statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0]]]
WARN 2017-06-13 20:37:54 [grid-nio-worker-tcp-comm-0-#17%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=0, keysCnt=1, bytesRcvd=45363, bytesRcvd0=0,
>> bytesSent=1315, bytesSent0=0]
Connection info [in=false, rmtAddr=/127.0.0.1:9340,
locAddr=/127.0.0.1:47331, msgsSent=3, msgsAckedByRmt=3,
descIdHash=2015302722, msgsRcvd=2, lastAcked=2, descIdHash=2015302722,
bytesRcvd=45363, bytesRcvd0=0, bytesSent=1315, bytesSent0=0, opQueueSize=0,
msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0,
stack=[StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013860000, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false],
null, null, null, null, null, null]]], msgReader=DirectMessageReader
[state=DirectMessageState [pos=0, stack=[StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0],
StateItem [stream=DirectByteBufferStreamImplV2
[buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
baseOff=139765013892784, arrOff=-1, tmpArrOff=0, tmpArrBytes=0,
msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false,
readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0,
uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem
[stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], baseOff=139765013892784, arrOff=-1, tmpArrOff=0,
tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1,
keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0,
uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null,
null, null, null, null, null, null]], lastRead=true]]
]]
WARN 2017-06-13 20:37:54 [grid-nio-worker-tcp-comm-1-#18%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=1, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:54 [grid-nio-worker-tcp-comm-2-#19%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
WARN 2017-06-13 20:37:54 [grid-nio-worker-tcp-comm-3-#20%svip%]
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi - [[
>> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0,
>> bytesSent0=0]
]]
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p13645.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by vkulichenko <va...@gmail.com>.
Please attach full logs.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p13487.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Vladimir <vl...@yandex.ru>.
Unfortunately, it’s not a warning. New node cannot join the cluster. And
there is no heavy load on the cluster, no CPU/memory consumption. No any
network problem because I tested all the nodes on single machine.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p13458.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by vkulichenko <va...@gmail.com>.
Vladimir,
As I told before, this message is very generic and can be caused by multiple
reasons (sometimes it's not even an issue BTW - it's warning, not an error).
First things to check is memory consumption, GC and network.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p13439.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Vladimir <vl...@yandex.ru>.
I have met similar problem too. A node cannot start:
WARN 16:08:36.158 [main]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager:
Still waiting for initial partition map exchange
[fut=GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=fa2d0f11-80cf-4891-9533-7a36392157f6, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=8, intOrder=6,
lastExchangeTime=1496754516085, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], topVer=8, nodeId8=fa2d0f11, msg=null, type=NODE_JOINED,
tstamp=1496754274106], crd=TcpDiscoveryNode
[id=7751c8dc-9bf6-4a21-9ea5-376f2be83913, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1, intOrder=1,
lastExchangeTime=1496754273954, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], nodeId=fa2d0f11,
evt=NODE_JOINED], added=false, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=true, hash=1259192920], init=true,
lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true,
true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
initTs=1496754276137, centralizedAff=false, changeGlobalStateE=null,
exchangeOnChangeGlobalState=false, forcedRebFut=null, evtLatch=0,
remaining=[bf7a4e48-4867-437b-9cc5-dd598b8795f3,
7751c8dc-9bf6-4a21-9ea5-376f2be83913, 373fd690-4171-4503-ab1b-d622c3ec6fc7],
srvNodes=[TcpDiscoveryNode [id=7751c8dc-9bf6-4a21-9ea5-376f2be83913,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9460], discPort=9460, order=1,
intOrder=1, lastExchangeTime=1496754273954, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=bf7a4e48-4867-437b-9cc5-dd598b8795f3, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9461], discPort=9461, order=6, intOrder=4,
lastExchangeTime=1496754273954, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false], TcpDiscoveryNode [id=373fd690-4171-4503-ab1b-d622c3ec6fc7,
addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:9462], discPort=9462, order=7,
intOrder=5, lastExchangeTime=1496754273954, loc=false,
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], TcpDiscoveryNode
[id=fa2d0f11-80cf-4891-9533-7a36392157f6, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:9463], discPort=9463, order=8, intOrder=6,
lastExchangeTime=1496754516085, loc=true, ver=2.0.0#20170430-sha1:d4eef3c6,
isClient=false]], super=GridFutureAdapter [ignoreInterrupts=false,
state=INIT, res=null, hash=1079801970]]]
WARN 16:08:36.499 [exchange-worker-#37%svip%]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager:
Failed to wait for partition map exchange [topVer=AffinityTopologyVersion
[topVer=8, minorTopVer=0], node=fa2d0f11-80cf-4891-9533-7a36392157f6].
Dumping pending objects that might be the cause:
What can it be?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p13414.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by vkulichenko <va...@gmail.com>.
Generally, you can't add new node while holding a lock. If you're holding
locks for long period of time, I would recommend you to revisit the
architecture. In any case, make sure that you properly release all the
locks. Also I think you should to upgrade to the latest version, I believe
there were some fixes in this area.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p12960.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by jaipal <da...@gmail.com>.
Val,
It doesn't look like a memory issue.We are using explicit locking (lock
entry , do processing and then unlock).Under high load, When we bring down
one of the Ignite Server nodes and then again add it back to the cluster,We
are getting this issue.We observed the exceptions like Failed to wait for
partition map exchange.sometimes it also gives Failed to wait for partition
release future and then dumps the pending cache futures,exchange futures and
transactions.Once we encounter this issue,Cluster hangs with out processing.
We need to clean start all Ignite Nodes to resolve this.We are observing
this issue quite often and it is preventing us from scaling up and down.
Please find the attached file that contains the ignite configuration we
used.Is something wrong with cache configuration or with way we are using
locking or with Ignite ?
ignite.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/n12957/ignite.xml>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p12957.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by vkulichenko <va...@gmail.com>.
Jai,
So what is the result of investigation? Does it look like memory issue or
not? As I said earlier, the issue itself doesn't have generic solution, you
need to find out the reason.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p12890.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by jaipal <da...@gmail.com>.
Hii val,
I tried recommended settings and configurations. still I observe this issue
and it can be reproduced easily.
Is there any fix available to this issue.
Regards
Jai
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p12885.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
It seems that the only way is making checkpoints more often by decreasing
checkpointFrequency value (specified in msec).
Smaller WAL - faster startup.
Regards,
--
Ilya Kasnacheev
сб, 19 янв. 2019 г. в 09:42, Justin Ji <bi...@foxmail.com>:
> Ilya -
>
> Thank for your reply.
>
> Is there any configuration can help us reduce the WAL time in 2.6.0.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Failed to wait for initial partition map exchange
Posted by Justin Ji <bi...@foxmail.com>.
Ilya -
Thank for your reply.
Is there any configuration can help us reduce the WAL time in 2.6.0.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to wait for initial partition map exchange
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I think that 2.7 had WAL reading speed improvements. However, starting
persistent node can still take a prolonged period of time.
Regards.
--
Ilya Kasnacheev
пт, 18 янв. 2019 г. в 12:59, Justin Ji <bi...@foxmail.com>:
> I also have this problem with ignite 2.6.0.
> Spend more than 350 seconds to restart a ignite node.
>
> Here is the system usage during partition map exchange:
>
> The CPU is at a low level and memory is enough.
>
> Here is the logs, we can find that applying WAL changes takes a long time,
> about 375664ms.
>
> >>> __________ ________________
> >>> / _/ ___/ |/ / _/_ __/ __/
> >>> _/ // (7 7 // / / / / _/
> >>> /___/\___/_/|_/___/ /_/ /___/
> >>>
> >>> ver. 2.6.0#20180710-sha1:669feacc
> >>> 2018 Copyright(C) Apache Software Foundation
> >>>
> >>> Ignite documentation: http://ignite.apache.org
>
> 2019-01-18 09:17:27:979 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - Config URL: file:/opt/ignite/config/ignite-config-benchmark-cn.xml
> 2019-01-18 09:17:27:996 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - IgniteConfiguration [igniteInstanceName=ignite-server, pubPoolSize=8,
> svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=16,
> mgmtPoolSize=4, igfsPoolSize=8, dataStreamerPoolSize=8,
> utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2,
> qryPoolSize=16, igniteHome=/opt/ignite/apache-ignite-fabric,
> igniteWorkDir=/opt/ignite/apache-ignite-fabric/work,
> mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94fa3e,
> nodeId=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> marsh=org.apache.ignite.internal.binary.BinaryMarshaller@72ade7e3,
> marshLocJobs=false, daemon=false, p2pEnabled=false, netTimeout=5000,
> sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000,
> metricsUpdateFreq=2000, metricsExpTime=9223372036854775807,
> discoSpi=ZookeeperDiscoverySpi [zkRootPath=/ignite/discovery,
> zkConnectionString=10.0.230.12:2181,10.0.223.106:2181,10.0.247.246:2181,
> joinTimeout=10000, sesTimeout=30000, clientReconnectDisabled=false,
> internalLsnr=null,
>
> stats=org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryStatistics@560348e6
> ],
> segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true,
> allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi
> [connectGate=null, connPlc=null, enableForcibleNodeKill=false,
> enableTroubleshootingLog=false,
>
> srvLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2@1df8b5b8
> ,
> locAddr=null, locHost=null, locPort=47174, locPortRange=100, shmemPort=-1,
> directBuf=true, directSndBuf=false, idleConnTimeout=600000,
> connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768,
> sockRcvBuf=32768, msgQueueLimit=1024, slowClientQueueLimit=0, nioSrvr=null,
> shmemSrv=null, usePairedConnections=false, connectionsPerNode=1,
> tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32,
> unackedMsgsBufSize=0, sockWriteTimeout=2000, lsnr=null, boundTcpPort=-1,
> boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null,
> ctxInitLatch=java.util.concurrent.CountDownLatch@23202fce[Count = 1],
> stopping=false,
>
> metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@7b993c65
> ],
> evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@37911f88,
> colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [lsnr=null],
> indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@4ea5b703,
> addrRslvr=null, clientMode=false, rebalanceThreadPoolSize=4,
> txCfg=org.apache.ignite.configuration.TransactionConfiguration@2a7ed1f,
> cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED,
> p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100,
> timeSrvPortRange=100, failureDetectionTimeout=10000,
> clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null,
>
> connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@3fa247d1
> ,
> odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration
> [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null,
> grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null,
> binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration
> [sysRegionInitSize=41943040, sysCacheMaxSize=104857600, pageSize=0,
> concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default,
> maxSize=3119855206, initSize=268435456, swapPath=null,
> pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100,
> metricsEnabled=false, metricsSubIntervalCount=5,
> metricsRateTimeInterval=60000, persistenceEnabled=false,
> checkpointPageBufSize=0], storagePath=/data/ignite/persistence,
> checkpointFreq=360000, lockWaitTime=10000, checkpointThreads=4,
> checkpointWriteOrder=SEQUENTIAL, walHistSize=10, walSegments=10,
> walSegmentSize=67108864, walPath=/ignite-wal/ignite/wal,
> walArchivePath=/ignite-wal/ignite/wal/archive, metricsEnabled=false,
> walMode=BACKGROUND, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000,
> walFsyncDelay=1000, walRecordIterBuffSize=67108864,
> alwaysWriteFullPages=false,
>
> fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@4f4c4b1a
> ,
> metricsSubIntervalCnt=5, metricsRateTimeInterval=60000,
> walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=true,
> walCompactionEnabled=false], activeOnStart=true, autoActivation=true,
> longQryWarnTimeout=3000, sqlConnCfg=null,
> cliConnCfg=ClientConnectorConfiguration [host=null, port=10800,
> portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true,
> maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0,
> jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false,
> useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null],
> authEnabled=false, failureHnd=null,
> commFailureRslvr=DefaultCommunicationFailureResolver []]
> 2019-01-18 09:17:27:997 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - Daemon mode: off
> 2019-01-18 09:17:27:998 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - OS: Linux 3.10.0-862.14.4.el7.x86_64 amd64
> 2019-01-18 09:17:27:998 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - OS user: root
> 2019-01-18 09:17:27:999 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - PID: 103
> 2019-01-18 09:17:28:000 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - Language runtime: Java Platform API Specification ver. 1.8
> 2019-01-18 09:17:28:000 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - VM information: OpenJDK Runtime Environment 1.8.0_171-b11 Oracle
> Corporation OpenJDK 64-Bit Server VM 25.171-b11
> 2019-01-18 09:17:28:002 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - VM total memory: 3.0GB
> 2019-01-18 09:17:28:003 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - Remote Management [restart: on, REST: on, JMX (remote: on, port: 49113,
> auth: off, ssl: off)]
> 2019-01-18 09:17:28:003 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - Logger: Log4J2Logger [quiet=false, config=log4j2.xml]
> 2019-01-18 09:17:28:004 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - IGNITE_HOME=/opt/ignite/apache-ignite-fabric
> 2019-01-18 09:17:28:004 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - VM arguments: [-Xms3g, -Xmx3g, -XX:+AlwaysPreTouch, -XX:+UseG1GC,
> -XX:+ScavengeBeforeFullGC, -XX:+DisableExplicitGC,
> -XX:+PrintGCApplicationStoppedTime, -XX:+PrintGCApplicationConcurrentTime,
> -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/mnt/heapdump/ignite,
> -XX:+ExitOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCTimeStamps,
> -XX:+PrintGCDateStamps, -XX:+UseGCLogFileRotation,
> -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=100M,
> -Xloggc:/mnt/logs/apps/ignite/gc.log, -DIGNITE_QUIET=true,
>
> -DIGNITE_SUCCESS_FILE=/opt/ignite/apache-ignite-fabric/work/ignite_success_8719edab-bf99-496e-add5-309d532f9c9b,
> -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=49113,
> -Dcom.sun.management.jmxremote.authenticate=false,
> -Dcom.sun.management.jmxremote.ssl=false,
> -DIGNITE_HOME=/opt/ignite/apache-ignite-fabric,
> -DIGNITE_PROG_NAME=/opt/ignite/apache-ignite-fabric/bin/ignite.sh]
> 2019-01-18 09:17:28:005 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - System cache's DataRegion size is configured to 40 MB. Use
> DataStorageConfiguration.systemCacheMemorySize property to change the
> setting.
> 2019-01-18 09:17:28:017 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']]
> 2019-01-18 09:17:28:022 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - 3-rd party licenses can be found at:
> /opt/ignite/apache-ignite-fabric/libs/licenses
> 2019-01-18 09:17:28:114 [main] INFO o.a.i.i.p.p.IgnitePluginProcessor:478
> -
> Configured plugins:
> 2019-01-18 09:17:28:114 [main] INFO o.a.i.i.p.p.IgnitePluginProcessor:478
> -
> ^-- None
> 2019-01-18 09:17:28:114 [main] INFO o.a.i.i.p.p.IgnitePluginProcessor:478
> -
> 2019-01-18 09:17:28:116 [main] INFO o.a.i.i.p.f.FailureProcessor:478 -
> Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
> [tryStop=false, timeout=0]]
> 2019-01-18 09:17:28:187 [main] INFO o.a.i.s.c.t.TcpCommunicationSpi:478 -
> Successfully bound communication NIO server to TCP port [port=47174,
> locHost=0.0.0.0/0.0.0.0, selectorsCnt=4, selectorSpins=0,
> pairedConn=false]
> 2019-01-18 09:17:28:213 [main] WARN o.a.i.s.c.n.NoopCheckpointSpi:488 -
> Checkpoints are disabled (to enable configure any GridCheckpointSpi
> implementation)
> 2019-01-18 09:17:28:248 [main] WARN o.a.i.i.m.c.GridCollisionManager:488 -
> Collision resolution is disabled (all jobs will be activated upon
> arrival).
> 2019-01-18 09:17:28:249 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - Security status [authentication=off, tls/ssl=off]
> 2019-01-18 09:17:28:379 [main] INFO o.a.i.i.p.c.p.f.PdsFoldersResolver:478
> - Successfully locked persistence storage folder
> [/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330]
> 2019-01-18 09:17:28:380 [main] INFO o.a.i.i.p.c.p.f.PdsFoldersResolver:478
> - Consistent ID used for local node is
> [4fa65519-a24b-4a05-a143-d39a39b9c330] according to persistence data
> storage
> folders
> 2019-01-18 09:17:28:380 [main] INFO
> o.a.i.i.p.c.b.CacheObjectBinaryProcessorImpl:478 - Resolved directory for
> serialized binary metadata:
> /opt/ignite/apache-ignite-fabric/work/binary_meta/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
>
> 2019-01-18 09:17:28:745 [main] INFO
> o.a.i.i.p.c.p.f.FilePageStoreManager:478 - Resolved page store work
> directory:
> /data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
> 2019-01-18 09:17:28:746 [main] INFO
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log
> work
> directory:
> /ignite-wal/ignite/wal/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
> 2019-01-18 09:17:28:746 [main] INFO
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log
> archive directory:
> /ignite-wal/ignite/wal/archive/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
> 2019-01-18 09:17:28:772 [main] INFO
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Started write-ahead log
> manager [mode=BACKGROUND]
> 2019-01-18 09:17:28:792 [main] WARN
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:488 - Page eviction mode set
> for [3G_DataRegion] data will have no effect because the oldest pages are
> evicted automatically if Ignite persistence is enabled.
> 2019-01-18 09:17:28:821 [main] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Read checkpoint status
>
> [startMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-START.bin,
> endMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-END.bin]
>
> 2019-01-18 09:17:28:841 [main] INFO o.a.i.i.p.c.p.p.PageMemoryImpl:478 -
> Started page memory [memoryAllocated=100.0 MiB, pages=24808, tableSize=1.9
> MiB, checkpointBuffer=100.0 MiB]
> 2019-01-18 09:17:28:842 [main] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checking memory state
> [lastValidPos=FileWALPointer [idx=508, fileOff=43025380, len=14626],
> lastMarked=FileWALPointer [idx=508, fileOff=43025380, len=14626],
> lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
> 2019-01-18 09:17:28:874 [main] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Found last checkpoint
> marker [cpId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8, pos=FileWALPointer
> [idx=508, fileOff=43025380, len=14626]]
> 2019-01-18 09:17:34:366 [main] WARN
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager:488 - WAL segment tail is reached.
> [ Expected next state: {Index=540,Offset=11065132}, Actual state :
> {Index=9024937041086329393,Offset=23034} ]
> 2019-01-18 09:17:34:405 [main] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Applying lost cache
> updates since last checkpoint record [lastMarked=FileWALPointer [idx=508,
> fileOff=43025380, len=14626],
> lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
> 2019-01-18 09:17:39:766 [main] WARN
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager:488 - WAL segment tail is reached.
> [ Expected next state: {Index=540,Offset=11065132}, Actual state :
> {Index=9024937041086329393,Offset=23034} ]
> 2019-01-18 09:17:39:767 [main] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Finished applying WAL
> changes [updatesApplied=0, time=5366ms]
> 2019-01-18 09:17:39:816 [main] INFO
> o.a.i.i.p.c.GridClusterStateProcessor:478 - Restoring history for
> BaselineTopology[id=2]
> 2019-01-18 09:17:39:967 [main] INFO
> o.a.i.i.p.o.ClientListenerProcessor:478
> - Client connector processor has started on TCP port 10800
> 2019-01-18 09:17:40:047 [main] INFO
> o.a.i.i.p.r.p.t.GridTcpRestProtocol:478
> - Command protocol successfully started [name=TCP binary,
> host=0.0.0.0/0.0.0.0, port=11211]
> 2019-01-18 09:17:40:559 [main] INFO
> o.a.i.i.p.r.p.h.j.GridJettyRestProtocol:478 - Command protocol successfully
> started [name=Jetty REST, host=/0.0.0.0, port=8080]
> 2019-01-18 09:17:40:595 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - Non-loopback local IPs: 10.0.230.117, 172.17.0.1,
> fe80:0:0:0:48a:c9ff:fe24:4b10%ens3
> 2019-01-18 09:17:40:596 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - Enabled local MACs: 024212517DB3, 068AC9244B10
> 2019-01-18 09:17:40:605 [main] INFO o.a.i.s.d.z.ZookeeperDiscoverySpi:478
> -
> Start Zookeeper discovery
> [zkConnectionString=10.0.230.12:2181,10.0.223.106:2181,10.0.247.246:2181,
> sessionTimeout=30000, zkRootPath=/ignite/discovery]
> 2019-01-18 09:17:40:753 [zk-ignite-server-EventThread] INFO
> o.a.i.s.d.z.i.ZookeeperClient:478 - ZooKeeper client state changed
> [prevState=Disconnected, newState=Connected]
> 2019-01-18 09:17:40:783 [main] INFO
> o.a.i.s.d.z.i.ZookeeperDiscoveryImpl:478 - Node started join
> [nodeId=67815159-8a6d-4c3a-b626-a82e5bea1a65, instanceName=ignite-server,
> zkSessionId=0x167775ea5d90428, joinDataSize=13674,
> consistentId=4fa65519-a24b-4a05-a143-d39a39b9c330, initTime=56,
> nodePath=/ignite/discovery/n/ebd81838-6ce4-4335-8eba-52b1842972d6:67815159-8a6d-4c3a-b626-a82e5bea1a65:80|0000001109]
>
> 2019-01-18 09:17:40:793 [zk-ignite-server-EventThread] INFO
> o.a.i.s.d.z.i.ZookeeperDiscoveryImpl:478 - Discovery coordinator already
> exists, watch for previous server node
> [locId=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> watchPath=b179406c-8f94-4bc1-a400-9731c169b66b:23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb:80|0000001107]
>
> 2019-01-18 09:17:40:852 [zk-ignite-server-EventThread] INFO
> o.a.i.s.d.z.i.ZookeeperDiscoveryImpl:478 - Local join event data:
> ZkJoinedNodeData [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, order=18]]
> 2019-01-18 09:17:40:963 [main] WARN o.a.i.i.IgniteKernal%ignite-server:488
> - Nodes started on local machine require more than 20% of physical RAM what
> can lead to significant slowdown due to swapping (please decrease JVM heap
> size, data region size or checkpoint buffer size) [required=17667MB,
> available=14876MB]
> 2019-01-18 09:17:40:982 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.e.time:478 - Started exchange init [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], crd=false, evt=NODE_JOINED,
> evtNode=67815159-8a6d-4c3a-b626-a82e5bea1a65, customEvt=null,
> allowMerge=true]
> 2019-01-18 09:17:40:982 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.f.FilePageStoreManager:478 - Resolved page store work
> directory:
> /data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
> 2019-01-18 09:17:40:983 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log
> work
> directory:
> /ignite-wal/ignite/wal/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
> 2019-01-18 09:17:40:983 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log
> archive directory:
> /ignite-wal/ignite/wal/archive/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
> 2019-01-18 09:17:40:986 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Started write-ahead log
> manager [mode=BACKGROUND]
> 2019-01-18 09:17:41:000 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
> [memoryAllocated=100.0 MiB, pages=24808, tableSize=1.9 MiB,
> checkpointBuffer=100.0 MiB]
> 2019-01-18 09:17:41:004 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
> [memoryAllocated=100.0 MiB, pages=24808, tableSize=1.9 MiB,
> checkpointBuffer=100.0 MiB]
> 2019-01-18 09:17:41:013 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
> [memoryAllocated=1.0 GiB, pages=254104, tableSize=19.8 MiB,
> checkpointBuffer=256.0 MiB]
> 2019-01-18 09:17:41:045 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
> [memoryAllocated=5.0 GiB, pages=1270520, tableSize=98.8 MiB,
> checkpointBuffer=2.0 GiB]
> 2019-01-18 09:17:42:442 [grid-nio-worker-tcp-comm-0-#25%ignite-server%]
> INFO
> o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
> connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.251.89:33706]
> 2019-01-18 09:17:42:912 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Read checkpoint status
>
> [startMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-START.bin,
> endMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-END.bin]
>
> 2019-01-18 09:17:42:913 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checking memory state
> [lastValidPos=FileWALPointer [idx=508, fileOff=43025380, len=14626],
> lastMarked=FileWALPointer [idx=508, fileOff=43025380, len=14626],
> lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
> 2019-01-18 09:17:42:930 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Found last checkpoint
> marker [cpId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8, pos=FileWALPointer
> [idx=508, fileOff=43025380, len=14626]]
> 2019-01-18 09:17:48:321 [exchange-worker-#42%ignite-server%] WARN
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager:488 - WAL segment tail is reached.
> [ Expected next state: {Index=540,Offset=11065132}, Actual state :
> {Index=9024937041086329393,Offset=23034} ]
> 2019-01-18 09:17:48:333 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resuming logging to WAL
> segment
>
> [file=/ignite-wal/ignite/wal/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/0000000000000000.wal,
> offset=11065132, ver=2]
> 2019-01-18 09:17:48:397 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.GridClusterStateProcessor:478 - Writing BaselineTopology[id=2]
> 2019-01-18 09:17:48:859 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.GridCacheProcessor:478 - Started cache [name=ignite-sys-cache,
> id=-2100569601, memoryPolicyName=sysMemPlc, mode=REPLICATED,
> atomicity=TRANSACTIONAL, backups=2147483647]
> 2019-01-18 09:17:48:872 [exchange-worker-#42%ignite-server%] INFO
> c.t.i.c.c.DataSourceConfiguration:29 - Begin to load external
> configuration...
> 2019-01-18 09:17:48:876 [exchange-worker-#42%ignite-server%] INFO
> c.t.i.c.c.DataSourceConfiguration:50 - Initializing configuration
> url=jdbc:mysql://
> tuya-shadows-cluster-1.cluster-cmu36ckfhd7h.rds.cn-northwest-1.amazonaws.com.cn:3306/tuya_shadows?characterEncoding=utf-8&useUnicode=true&autoReconnect=true&allowMultiQueries=true&rewriteBatchedStatements=true,username=tuya_shadows_rw,dns
> cache time=30
> 2019-01-18 09:17:48:983 [exchange-worker-#42%ignite-server%] WARN
> o.a.i.i.p.c.s.CacheOsStoreManager:488 - <data_point_new> Both Ignite native
> persistence and CacheStore are configured for cache 'data_point_new'. This
> configuration does not guarantee strict consistency between CacheStore and
> Ignite data storage upon restarts. Consult documentation for more details.
> 2019-01-18 09:17:49:086 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.GridCacheProcessor:478 - Started cache [name=data_point_new,
> id=-2047491044, memoryPolicyName=5G_DataRegion, mode=PARTITIONED,
> atomicity=ATOMIC, backups=1]
> 2019-01-18 09:17:49:110 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting
> for
> partition release future [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], waitTime=0ms, futInfo=NA]
> 2019-01-18 09:17:49:112 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting
> for
> partition release future [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], waitTime=0ms, futInfo=NA]
> 2019-01-18 09:17:49:113 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Read checkpoint status
>
> [startMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-START.bin,
> endMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-END.bin]
>
> 2019-01-18 09:17:49:114 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Applying lost cache
> updates since last checkpoint record [lastMarked=FileWALPointer [idx=508,
> fileOff=43025380, len=14626],
> lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
> 2019-01-18 09:17:50:972 [ignite-update-notifier-timer] WARN
> o.a.i.i.p.c.GridUpdateNotifier:488 - New version is available at
> ignite.apache.org: 2.7.0
> 2019-01-18 09:18:00:978 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Failed to wait for
> initial partition map exchange. Possible reasons are:
> ^-- Transactions in deadlock.
> ^-- Long running transactions (ignore if this is the case).
> ^-- Unreleased explicit locks.
> 2019-01-18 09:18:20:980 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:18:40:981 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:19:40:986 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:20:00:988 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:20:40:992 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:21:00:993 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:21:20:994 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:21:40:996 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:22:00:997 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:22:20:998 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:22:40:999 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:23:01:001 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:23:41:003 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:24:01:004 [main] WARN
> o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
> initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
> [firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
> [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
> topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
> tstamp=1547803060924], crd=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
> loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
> type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
> added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
> res=null, hash=1766610307], init=false, lastVer=null,
> partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], DataStreamerReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
> LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]], AllTxReleaseFuture
> [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
> [startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
> resetParts=null, stateChangeRequest=null], affChangeMsg=null,
> initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
> changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
> remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> 23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
> [ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
> 2019-01-18 09:24:04:775 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Finished applying WAL
> changes [updatesApplied=1320211, time=375664ms]
> 2019-01-18 09:24:04:881 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.e.time:478 - Finished exchange init [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=0], crd=false]
> 2019-01-18 09:24:04:917 [sys-#48%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Received full
> message, will finish exchange [node=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]]
> 2019-01-18 09:24:05:002 [sys-#48%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finish exchange
> future [startVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], err=null]
> 2019-01-18 09:24:05:057 [grid-nio-worker-tcp-comm-1-#26%ignite-server%]
> INFO
> o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
> connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.216.155:51920]
> 2019-01-18 09:24:05:265 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.GridCachePartitionExchangeManager:478 - Rebalancing scheduled
> [order=[ignite-sys-cache, data_point_new]]
> 2019-01-18 09:24:05:267 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.GridCachePartitionExchangeManager:478 - Rebalancing started
> [top=AffinityTopologyVersion [topVer=18, minorTopVer=0], evt=NODE_JOINED,
> node=67815159-8a6d-4c3a-b626-a82e5bea1a65]
> 2019-01-18 09:24:05:268 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
> [grp=ignite-sys-cache, mode=SYNC,
> fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, partitionsCount=51,
> topology=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> rebalanceId=1]
> 2019-01-18 09:24:05:285 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
> [grp=ignite-sys-cache, mode=SYNC,
> fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb, partitionsCount=49,
> topology=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> rebalanceId=1]
> 2019-01-18 09:24:05:791 [db-checkpoint-thread-#63%ignite-server%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checkpoint started
> [checkpointId=d1455b4d-cee7-4909-8417-fdadc515db7e, startPtr=FileWALPointer
> [idx=564, fileOff=60228974, len=12726], checkpointLockWait=0ms,
> checkpointLockHoldTime=53ms, walCpRecordFsyncDuration=219ms, pages=145395,
> reason='timeout']
> 2019-01-18 09:24:20:905 [grid-nio-worker-tcp-comm-2-#27%ignite-server%]
> INFO
> o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
> connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.251.89:33710]
> 2019-01-18 09:24:20:907 [grid-nio-worker-tcp-comm-2-#27%ignite-server%]
> INFO
> o.a.i.s.c.t.TcpCommunicationSpi:478 - Received incoming connection when
> already connected to this node, rejecting
> [locNode=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> rmtNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4]
> 2019-01-18 09:24:21:111 [grid-nio-worker-tcp-comm-3-#28%ignite-server%]
> INFO
> o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
> connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.251.89:33712]
> 2019-01-18 09:24:24:201 [grid-nio-worker-tcp-comm-0-#25%ignite-server%]
> INFO
> o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
> connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.216.155:51946]
> 2019-01-18 09:24:24:203 [grid-nio-worker-tcp-comm-0-#25%ignite-server%]
> INFO
> o.a.i.s.c.t.TcpCommunicationSpi:478 - Received incoming connection when
> already connected to this node, rejecting
> [locNode=67815159-8a6d-4c3a-b626-a82e5bea1a65,
> rmtNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb]
> 2019-01-18 09:24:24:406 [grid-nio-worker-tcp-comm-1-#26%ignite-server%]
> INFO
> o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
> connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.216.155:51948]
> 2019-01-18 09:25:30:367 [db-checkpoint-thread-#63%ignite-server%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checkpoint finished
> [cpId=d1455b4d-cee7-4909-8417-fdadc515db7e, pages=145395,
> markPos=FileWALPointer [idx=564, fileOff=60228974, len=12726],
> walSegmentsCleared=37, markDuration=993ms, pagesWrite=24067ms,
> fsync=60505ms, total=85565ms]
> 2019-01-18 09:26:42:954 [utility-#71%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalancing
> [fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> cacheOrGroup=ignite-sys-cache, topology=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], time=157682 ms]
> 2019-01-18 09:26:42:968 [utility-#74%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed (final)
> rebalancing [fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb,
> cacheOrGroup=ignite-sys-cache, topology=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], time=157692 ms]
> 2019-01-18 09:26:42:969 [utility-#74%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalance
> future:
> RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache],
> topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
> 2019-01-18 09:26:42:970 [utility-#74%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
> [grp=data_point_new, mode=ASYNC,
> fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, partitionsCount=337,
> topology=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> rebalanceId=1]
> 2019-01-18 09:26:42:971 [utility-#74%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
> [grp=data_point_new, mode=ASYNC,
> fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb, partitionsCount=330,
> topology=AffinityTopologyVersion [topVer=18, minorTopVer=0],
> rebalanceId=1]
> 2019-01-18 09:26:43:142 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - Performance suggestions for grid 'ignite-server' (fix if possible)
> 2019-01-18 09:26:43:142 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
> 2019-01-18 09:26:43:143 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - ^-- Disable grid events (remove 'includeEventTypes' from
> configuration)
> 2019-01-18 09:26:43:143 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - ^-- Set max direct memory size if getting 'OOME: Direct buffer memory'
> (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
> 2019-01-18 09:26:43:143 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - ^-- Decrease number of backups (set 'backups' to 0)
> 2019-01-18 09:26:43:146 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - Refer to this page for more performance suggestions:
> https://apacheignite.readme.io/docs/jvm-and-system-tuning
> 2019-01-18 09:26:43:146 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> -
> 2019-01-18 09:26:43:146 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> - To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
> 2019-01-18 09:26:43:147 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> -
> 2019-01-18 09:26:43:149 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
> -
>
> >>>
> +----------------------------------------------------------------------+
> >>> Ignite ver.
> 2.6.0#20180710-sha1:669feacc5d3a4e60efcdd300dc8de99780f38eed
> >>>
> +----------------------------------------------------------------------+
> >>> OS name: Linux 3.10.0-862.14.4.el7.x86_64 amd64
> >>> CPU(s): 8
> >>> Heap: 3.0GB
> >>> VM name: 103@nx-s-ignite-002
> >>> Ignite instance name: ignite-server
> >>> Local node [ID=67815159-8A6D-4C3A-B626-A82E5BEA1A65, order=18,
> >>> clientMode=false]
> >>> Local node addresses: [172.17.0.1/172.17.0.1,
> >>> nx-s-ignite-002/10.0.230.117, /0:0:0:0:0:0:0:1%lo, /127.0.0.1]
> >>> Local ports: TCP:8080 TCP:10800 TCP:11211 TCP:47174
>
> 2019-01-18 09:26:43:155 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
> Topology snapshot [ver=18, servers=3, clients=1, CPUs=32, offheap=36.0GB,
> heap=17.0GB]
> 2019-01-18 09:26:43:155 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478
> -
> ^-- Node [id=67815159-8A6D-4C3A-B626-A82E5BEA1A65, clusterState=ACTIVE]
> 2019-01-18 09:26:43:156 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478
> -
> ^-- Baseline [id=2, size=3, online=3, offline=0]
> 2019-01-18 09:26:43:156 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
> Data Regions Configured:
> 2019-01-18 09:26:43:158 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478
> -
> ^-- default [initSize=256.0 MiB, maxSize=2.9 GiB,
> persistenceEnabled=false]
> 2019-01-18 09:26:43:159 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478
> -
> ^-- 1G_DataRegion [initSize=512.0 MiB, maxSize=1.0 GiB,
> persistenceEnabled=true]
> 2019-01-18 09:26:43:159 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478
> -
> ^-- 3G_DataRegion [initSize=512.0 MiB, maxSize=3.0 GiB,
> persistenceEnabled=false]
> 2019-01-18 09:26:43:160 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478
> -
> ^-- 5G_DataRegion [initSize=512.0 MiB, maxSize=5.0 GiB,
> persistenceEnabled=true]
> 2019-01-18 09:26:47:855 [grid-nio-worker-tcp-comm-2-#27%ignite-server%]
> INFO
> o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
> connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.214.219:36298]
> 2019-01-18 09:27:43:141 [grid-timeout-worker-#23%ignite-server%] INFO
> o.a.i.i.IgniteKernal%ignite-server:478 -
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=67815159, name=ignite-server, uptime=00:01:00.005]
> ^-- H/N/C [hosts=4, nodes=4, CPUs=32]
> ^-- CPU [cur=70.6%, avg=12.89%, GC=0.53%]
> ^-- PageMemory [pages=1130803]
> ^-- Heap [used=2230MB, free=27.39%, comm=3072MB]
> ^-- Non heap [used=75MB, free=-1%, comm=77MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=6, idle=10, qSize=0]
> 2019-01-18 09:28:10:913 [sys-#43%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalancing
> [fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> cacheOrGroup=data_point_new,
> topology=AffinityTopologyVersion [topVer=18, minorTopVer=0], time=87942
> ms]
> 2019-01-18 09:28:15:627 [sys-#46%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed (final)
> rebalancing [fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb,
> cacheOrGroup=data_point_new, topology=AffinityTopologyVersion [topVer=18,
> minorTopVer=0], time=92658 ms]
> 2019-01-18 09:28:15:628 [sys-#46%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalance
> future:
> RebalanceFuture [grp=CacheGroupContext [grp=data_point_new],
> topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
> 2019-01-18 09:28:17:202 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.e.time:478 - Started exchange init [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=1], crd=false, evt=DISCOVERY_CUSTOM_EVT,
> evtNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> customEvt=CacheAffinityChangeMessage
> [id=fcc5eef5861-b0e299ac-bcd4-4ecf-855c-af770da317dc,
> topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], exchId=null,
> partsMsg=null, exchangeNeeded=true], allowMerge=false]
> 2019-01-18 09:28:17:209 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting
> for
> partition release future [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=1], waitTime=0ms, futInfo=NA]
> 2019-01-18 09:28:17:666 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting
> for
> partitions release latch: ClientLatch [coordinator=ZookeeperClusterNode
> [id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
> 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
> ackSent=true, super=CompletableLatch [id=exchange,
> topVer=AffinityTopologyVersion [topVer=18, minorTopVer=1]]]
> 2019-01-18 09:28:17:666 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting
> for
> partition release future [topVer=AffinityTopologyVersion [topVer=18,
> minorTopVer=1], waitTime=0ms, futInfo=NA]
> 2019-01-18 09:28:17:678 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.e.time:478 - Finished exchange init [topVer=AffinityTopologyVersion
> [topVer=18, minorTopVer=1], crd=false]
> 2019-01-18 09:28:17:685 [sys-#49%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Received full
> message, will finish exchange [node=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
> resVer=AffinityTopologyVersion [topVer=18, minorTopVer=1]]
> 2019-01-18 09:28:17:686 [sys-#49%ignite-server%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finish exchange
> future [startVer=AffinityTopologyVersion [topVer=18, minorTopVer=1],
> resVer=AffinityTopologyVersion [topVer=18, minorTopVer=1], err=null]
> 2019-01-18 09:28:17:688 [exchange-worker-#42%ignite-server%] INFO
> o.a.i.i.p.c.GridCachePartitionExchangeManager:478 - Skipping rebalancing
> (nothing scheduled) [top=AffinityTopologyVersion [topVer=18,
> minorTopVer=1],
> evt=DISCOVERY_CUSTOM_EVT, node=bea075f4-a7b2-409d-8c78-5ea420d4ebd4]
> 2019-01-18 09:28:19:008 [flusher-0-#50%ignite-server%] INFO
> c.t.i.c.s.DataPointCacheStore:43 - begin write records to
> database:smart_datapoint
>
>
> And I also reviewed the GC log, the GC became more frequent(1 time per
> second), but only took 10~50ms.
>
> Can anyone tell me how to decrease the stop time of ignite cluster.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Failed to wait for initial partition map exchange
Posted by Justin Ji <bi...@foxmail.com>.
I also have this problem with ignite 2.6.0.
Spend more than 350 seconds to restart a ignite node.
Here is the system usage during partition map exchange:
The CPU is at a low level and memory is enough.
Here is the logs, we can find that applying WAL changes takes a long time,
about 375664ms.
>>> __________ ________________
>>> / _/ ___/ |/ / _/_ __/ __/
>>> _/ // (7 7 // / / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 2.6.0#20180710-sha1:669feacc
>>> 2018 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org
2019-01-18 09:17:27:979 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Config URL: file:/opt/ignite/config/ignite-config-benchmark-cn.xml
2019-01-18 09:17:27:996 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- IgniteConfiguration [igniteInstanceName=ignite-server, pubPoolSize=8,
svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=16,
mgmtPoolSize=4, igfsPoolSize=8, dataStreamerPoolSize=8,
utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2,
qryPoolSize=16, igniteHome=/opt/ignite/apache-ignite-fabric,
igniteWorkDir=/opt/ignite/apache-ignite-fabric/work,
mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94fa3e,
nodeId=67815159-8a6d-4c3a-b626-a82e5bea1a65,
marsh=org.apache.ignite.internal.binary.BinaryMarshaller@72ade7e3,
marshLocJobs=false, daemon=false, p2pEnabled=false, netTimeout=5000,
sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000,
metricsUpdateFreq=2000, metricsExpTime=9223372036854775807,
discoSpi=ZookeeperDiscoverySpi [zkRootPath=/ignite/discovery,
zkConnectionString=10.0.230.12:2181,10.0.223.106:2181,10.0.247.246:2181,
joinTimeout=10000, sesTimeout=30000, clientReconnectDisabled=false,
internalLsnr=null,
stats=org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryStatistics@560348e6],
segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true,
allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi
[connectGate=null, connPlc=null, enableForcibleNodeKill=false,
enableTroubleshootingLog=false,
srvLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2@1df8b5b8,
locAddr=null, locHost=null, locPort=47174, locPortRange=100, shmemPort=-1,
directBuf=true, directSndBuf=false, idleConnTimeout=600000,
connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768,
sockRcvBuf=32768, msgQueueLimit=1024, slowClientQueueLimit=0, nioSrvr=null,
shmemSrv=null, usePairedConnections=false, connectionsPerNode=1,
tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32,
unackedMsgsBufSize=0, sockWriteTimeout=2000, lsnr=null, boundTcpPort=-1,
boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null,
ctxInitLatch=java.util.concurrent.CountDownLatch@23202fce[Count = 1],
stopping=false,
metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@7b993c65],
evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@37911f88,
colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [lsnr=null],
indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@4ea5b703,
addrRslvr=null, clientMode=false, rebalanceThreadPoolSize=4,
txCfg=org.apache.ignite.configuration.TransactionConfiguration@2a7ed1f,
cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED,
p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100,
timeSrvPortRange=100, failureDetectionTimeout=10000,
clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null,
connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@3fa247d1,
odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration
[seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null,
grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null,
binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration
[sysRegionInitSize=41943040, sysCacheMaxSize=104857600, pageSize=0,
concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default,
maxSize=3119855206, initSize=268435456, swapPath=null,
pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100,
metricsEnabled=false, metricsSubIntervalCount=5,
metricsRateTimeInterval=60000, persistenceEnabled=false,
checkpointPageBufSize=0], storagePath=/data/ignite/persistence,
checkpointFreq=360000, lockWaitTime=10000, checkpointThreads=4,
checkpointWriteOrder=SEQUENTIAL, walHistSize=10, walSegments=10,
walSegmentSize=67108864, walPath=/ignite-wal/ignite/wal,
walArchivePath=/ignite-wal/ignite/wal/archive, metricsEnabled=false,
walMode=BACKGROUND, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000,
walFsyncDelay=1000, walRecordIterBuffSize=67108864,
alwaysWriteFullPages=false,
fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@4f4c4b1a,
metricsSubIntervalCnt=5, metricsRateTimeInterval=60000,
walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=true,
walCompactionEnabled=false], activeOnStart=true, autoActivation=true,
longQryWarnTimeout=3000, sqlConnCfg=null,
cliConnCfg=ClientConnectorConfiguration [host=null, port=10800,
portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true,
maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0,
jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false,
useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null],
authEnabled=false, failureHnd=null,
commFailureRslvr=DefaultCommunicationFailureResolver []]
2019-01-18 09:17:27:997 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Daemon mode: off
2019-01-18 09:17:27:998 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- OS: Linux 3.10.0-862.14.4.el7.x86_64 amd64
2019-01-18 09:17:27:998 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- OS user: root
2019-01-18 09:17:27:999 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- PID: 103
2019-01-18 09:17:28:000 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Language runtime: Java Platform API Specification ver. 1.8
2019-01-18 09:17:28:000 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- VM information: OpenJDK Runtime Environment 1.8.0_171-b11 Oracle
Corporation OpenJDK 64-Bit Server VM 25.171-b11
2019-01-18 09:17:28:002 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- VM total memory: 3.0GB
2019-01-18 09:17:28:003 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Remote Management [restart: on, REST: on, JMX (remote: on, port: 49113,
auth: off, ssl: off)]
2019-01-18 09:17:28:003 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Logger: Log4J2Logger [quiet=false, config=log4j2.xml]
2019-01-18 09:17:28:004 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- IGNITE_HOME=/opt/ignite/apache-ignite-fabric
2019-01-18 09:17:28:004 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- VM arguments: [-Xms3g, -Xmx3g, -XX:+AlwaysPreTouch, -XX:+UseG1GC,
-XX:+ScavengeBeforeFullGC, -XX:+DisableExplicitGC,
-XX:+PrintGCApplicationStoppedTime, -XX:+PrintGCApplicationConcurrentTime,
-XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/mnt/heapdump/ignite,
-XX:+ExitOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCTimeStamps,
-XX:+PrintGCDateStamps, -XX:+UseGCLogFileRotation,
-XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=100M,
-Xloggc:/mnt/logs/apps/ignite/gc.log, -DIGNITE_QUIET=true,
-DIGNITE_SUCCESS_FILE=/opt/ignite/apache-ignite-fabric/work/ignite_success_8719edab-bf99-496e-add5-309d532f9c9b,
-Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=49113,
-Dcom.sun.management.jmxremote.authenticate=false,
-Dcom.sun.management.jmxremote.ssl=false,
-DIGNITE_HOME=/opt/ignite/apache-ignite-fabric,
-DIGNITE_PROG_NAME=/opt/ignite/apache-ignite-fabric/bin/ignite.sh]
2019-01-18 09:17:28:005 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- System cache's DataRegion size is configured to 40 MB. Use
DataStorageConfiguration.systemCacheMemorySize property to change the
setting.
2019-01-18 09:17:28:017 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']]
2019-01-18 09:17:28:022 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- 3-rd party licenses can be found at:
/opt/ignite/apache-ignite-fabric/libs/licenses
2019-01-18 09:17:28:114 [main] INFO o.a.i.i.p.p.IgnitePluginProcessor:478 -
Configured plugins:
2019-01-18 09:17:28:114 [main] INFO o.a.i.i.p.p.IgnitePluginProcessor:478 -
^-- None
2019-01-18 09:17:28:114 [main] INFO o.a.i.i.p.p.IgnitePluginProcessor:478 -
2019-01-18 09:17:28:116 [main] INFO o.a.i.i.p.f.FailureProcessor:478 -
Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0]]
2019-01-18 09:17:28:187 [main] INFO o.a.i.s.c.t.TcpCommunicationSpi:478 -
Successfully bound communication NIO server to TCP port [port=47174,
locHost=0.0.0.0/0.0.0.0, selectorsCnt=4, selectorSpins=0, pairedConn=false]
2019-01-18 09:17:28:213 [main] WARN o.a.i.s.c.n.NoopCheckpointSpi:488 -
Checkpoints are disabled (to enable configure any GridCheckpointSpi
implementation)
2019-01-18 09:17:28:248 [main] WARN o.a.i.i.m.c.GridCollisionManager:488 -
Collision resolution is disabled (all jobs will be activated upon arrival).
2019-01-18 09:17:28:249 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Security status [authentication=off, tls/ssl=off]
2019-01-18 09:17:28:379 [main] INFO o.a.i.i.p.c.p.f.PdsFoldersResolver:478
- Successfully locked persistence storage folder
[/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330]
2019-01-18 09:17:28:380 [main] INFO o.a.i.i.p.c.p.f.PdsFoldersResolver:478
- Consistent ID used for local node is
[4fa65519-a24b-4a05-a143-d39a39b9c330] according to persistence data storage
folders
2019-01-18 09:17:28:380 [main] INFO
o.a.i.i.p.c.b.CacheObjectBinaryProcessorImpl:478 - Resolved directory for
serialized binary metadata:
/opt/ignite/apache-ignite-fabric/work/binary_meta/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:28:745 [main] INFO
o.a.i.i.p.c.p.f.FilePageStoreManager:478 - Resolved page store work
directory:
/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:28:746 [main] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log work
directory:
/ignite-wal/ignite/wal/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:28:746 [main] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log
archive directory:
/ignite-wal/ignite/wal/archive/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:28:772 [main] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Started write-ahead log
manager [mode=BACKGROUND]
2019-01-18 09:17:28:792 [main] WARN
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:488 - Page eviction mode set
for [3G_DataRegion] data will have no effect because the oldest pages are
evicted automatically if Ignite persistence is enabled.
2019-01-18 09:17:28:821 [main] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Read checkpoint status
[startMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-START.bin,
endMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-END.bin]
2019-01-18 09:17:28:841 [main] INFO o.a.i.i.p.c.p.p.PageMemoryImpl:478 -
Started page memory [memoryAllocated=100.0 MiB, pages=24808, tableSize=1.9
MiB, checkpointBuffer=100.0 MiB]
2019-01-18 09:17:28:842 [main] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checking memory state
[lastValidPos=FileWALPointer [idx=508, fileOff=43025380, len=14626],
lastMarked=FileWALPointer [idx=508, fileOff=43025380, len=14626],
lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
2019-01-18 09:17:28:874 [main] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Found last checkpoint
marker [cpId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8, pos=FileWALPointer
[idx=508, fileOff=43025380, len=14626]]
2019-01-18 09:17:34:366 [main] WARN
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:488 - WAL segment tail is reached.
[ Expected next state: {Index=540,Offset=11065132}, Actual state :
{Index=9024937041086329393,Offset=23034} ]
2019-01-18 09:17:34:405 [main] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Applying lost cache
updates since last checkpoint record [lastMarked=FileWALPointer [idx=508,
fileOff=43025380, len=14626],
lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
2019-01-18 09:17:39:766 [main] WARN
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:488 - WAL segment tail is reached.
[ Expected next state: {Index=540,Offset=11065132}, Actual state :
{Index=9024937041086329393,Offset=23034} ]
2019-01-18 09:17:39:767 [main] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Finished applying WAL
changes [updatesApplied=0, time=5366ms]
2019-01-18 09:17:39:816 [main] INFO
o.a.i.i.p.c.GridClusterStateProcessor:478 - Restoring history for
BaselineTopology[id=2]
2019-01-18 09:17:39:967 [main] INFO o.a.i.i.p.o.ClientListenerProcessor:478
- Client connector processor has started on TCP port 10800
2019-01-18 09:17:40:047 [main] INFO o.a.i.i.p.r.p.t.GridTcpRestProtocol:478
- Command protocol successfully started [name=TCP binary,
host=0.0.0.0/0.0.0.0, port=11211]
2019-01-18 09:17:40:559 [main] INFO
o.a.i.i.p.r.p.h.j.GridJettyRestProtocol:478 - Command protocol successfully
started [name=Jetty REST, host=/0.0.0.0, port=8080]
2019-01-18 09:17:40:595 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Non-loopback local IPs: 10.0.230.117, 172.17.0.1,
fe80:0:0:0:48a:c9ff:fe24:4b10%ens3
2019-01-18 09:17:40:596 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Enabled local MACs: 024212517DB3, 068AC9244B10
2019-01-18 09:17:40:605 [main] INFO o.a.i.s.d.z.ZookeeperDiscoverySpi:478 -
Start Zookeeper discovery
[zkConnectionString=10.0.230.12:2181,10.0.223.106:2181,10.0.247.246:2181,
sessionTimeout=30000, zkRootPath=/ignite/discovery]
2019-01-18 09:17:40:753 [zk-ignite-server-EventThread] INFO
o.a.i.s.d.z.i.ZookeeperClient:478 - ZooKeeper client state changed
[prevState=Disconnected, newState=Connected]
2019-01-18 09:17:40:783 [main] INFO
o.a.i.s.d.z.i.ZookeeperDiscoveryImpl:478 - Node started join
[nodeId=67815159-8a6d-4c3a-b626-a82e5bea1a65, instanceName=ignite-server,
zkSessionId=0x167775ea5d90428, joinDataSize=13674,
consistentId=4fa65519-a24b-4a05-a143-d39a39b9c330, initTime=56,
nodePath=/ignite/discovery/n/ebd81838-6ce4-4335-8eba-52b1842972d6:67815159-8a6d-4c3a-b626-a82e5bea1a65:80|0000001109]
2019-01-18 09:17:40:793 [zk-ignite-server-EventThread] INFO
o.a.i.s.d.z.i.ZookeeperDiscoveryImpl:478 - Discovery coordinator already
exists, watch for previous server node
[locId=67815159-8a6d-4c3a-b626-a82e5bea1a65,
watchPath=b179406c-8f94-4bc1-a400-9731c169b66b:23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb:80|0000001107]
2019-01-18 09:17:40:852 [zk-ignite-server-EventThread] INFO
o.a.i.s.d.z.i.ZookeeperDiscoveryImpl:478 - Local join event data:
ZkJoinedNodeData [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, order=18]]
2019-01-18 09:17:40:963 [main] WARN o.a.i.i.IgniteKernal%ignite-server:488
- Nodes started on local machine require more than 20% of physical RAM what
can lead to significant slowdown due to swapping (please decrease JVM heap
size, data region size or checkpoint buffer size) [required=17667MB,
available=14876MB]
2019-01-18 09:17:40:982 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.e.time:478 - Started exchange init [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], crd=false, evt=NODE_JOINED,
evtNode=67815159-8a6d-4c3a-b626-a82e5bea1a65, customEvt=null,
allowMerge=true]
2019-01-18 09:17:40:982 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.f.FilePageStoreManager:478 - Resolved page store work
directory:
/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:40:983 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log work
directory:
/ignite-wal/ignite/wal/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:40:983 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log
archive directory:
/ignite-wal/ignite/wal/archive/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:40:986 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Started write-ahead log
manager [mode=BACKGROUND]
2019-01-18 09:17:41:000 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
[memoryAllocated=100.0 MiB, pages=24808, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
2019-01-18 09:17:41:004 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
[memoryAllocated=100.0 MiB, pages=24808, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
2019-01-18 09:17:41:013 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
[memoryAllocated=1.0 GiB, pages=254104, tableSize=19.8 MiB,
checkpointBuffer=256.0 MiB]
2019-01-18 09:17:41:045 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
[memoryAllocated=5.0 GiB, pages=1270520, tableSize=98.8 MiB,
checkpointBuffer=2.0 GiB]
2019-01-18 09:17:42:442 [grid-nio-worker-tcp-comm-0-#25%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.251.89:33706]
2019-01-18 09:17:42:912 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Read checkpoint status
[startMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-START.bin,
endMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-END.bin]
2019-01-18 09:17:42:913 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checking memory state
[lastValidPos=FileWALPointer [idx=508, fileOff=43025380, len=14626],
lastMarked=FileWALPointer [idx=508, fileOff=43025380, len=14626],
lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
2019-01-18 09:17:42:930 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Found last checkpoint
marker [cpId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8, pos=FileWALPointer
[idx=508, fileOff=43025380, len=14626]]
2019-01-18 09:17:48:321 [exchange-worker-#42%ignite-server%] WARN
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:488 - WAL segment tail is reached.
[ Expected next state: {Index=540,Offset=11065132}, Actual state :
{Index=9024937041086329393,Offset=23034} ]
2019-01-18 09:17:48:333 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resuming logging to WAL
segment
[file=/ignite-wal/ignite/wal/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/0000000000000000.wal,
offset=11065132, ver=2]
2019-01-18 09:17:48:397 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridClusterStateProcessor:478 - Writing BaselineTopology[id=2]
2019-01-18 09:17:48:859 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridCacheProcessor:478 - Started cache [name=ignite-sys-cache,
id=-2100569601, memoryPolicyName=sysMemPlc, mode=REPLICATED,
atomicity=TRANSACTIONAL, backups=2147483647]
2019-01-18 09:17:48:872 [exchange-worker-#42%ignite-server%] INFO
c.t.i.c.c.DataSourceConfiguration:29 - Begin to load external
configuration...
2019-01-18 09:17:48:876 [exchange-worker-#42%ignite-server%] INFO
c.t.i.c.c.DataSourceConfiguration:50 - Initializing configuration
url=jdbc:mysql://tuya-shadows-cluster-1.cluster-cmu36ckfhd7h.rds.cn-northwest-1.amazonaws.com.cn:3306/tuya_shadows?characterEncoding=utf-8&useUnicode=true&autoReconnect=true&allowMultiQueries=true&rewriteBatchedStatements=true,username=tuya_shadows_rw,dns
cache time=30
2019-01-18 09:17:48:983 [exchange-worker-#42%ignite-server%] WARN
o.a.i.i.p.c.s.CacheOsStoreManager:488 - <data_point_new> Both Ignite native
persistence and CacheStore are configured for cache 'data_point_new'. This
configuration does not guarantee strict consistency between CacheStore and
Ignite data storage upon restarts. Consult documentation for more details.
2019-01-18 09:17:49:086 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridCacheProcessor:478 - Started cache [name=data_point_new,
id=-2047491044, memoryPolicyName=5G_DataRegion, mode=PARTITIONED,
atomicity=ATOMIC, backups=1]
2019-01-18 09:17:49:110 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting for
partition release future [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], waitTime=0ms, futInfo=NA]
2019-01-18 09:17:49:112 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting for
partition release future [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], waitTime=0ms, futInfo=NA]
2019-01-18 09:17:49:113 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Read checkpoint status
[startMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-START.bin,
endMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-END.bin]
2019-01-18 09:17:49:114 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Applying lost cache
updates since last checkpoint record [lastMarked=FileWALPointer [idx=508,
fileOff=43025380, len=14626],
lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
2019-01-18 09:17:50:972 [ignite-update-notifier-timer] WARN
o.a.i.i.p.c.GridUpdateNotifier:488 - New version is available at
ignite.apache.org: 2.7.0
2019-01-18 09:18:00:978 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Failed to wait for
initial partition map exchange. Possible reasons are:
^-- Transactions in deadlock.
^-- Long running transactions (ignore if this is the case).
^-- Unreleased explicit locks.
2019-01-18 09:18:20:980 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:18:40:981 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:19:40:986 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:20:00:988 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:20:40:992 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:21:00:993 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:21:20:994 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:21:40:996 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:22:00:997 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:22:20:998 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:22:40:999 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:23:01:001 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:23:41:003 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:24:01:004 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:24:04:775 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Finished applying WAL
changes [updatesApplied=1320211, time=375664ms]
2019-01-18 09:24:04:881 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.e.time:478 - Finished exchange init [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], crd=false]
2019-01-18 09:24:04:917 [sys-#48%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Received full
message, will finish exchange [node=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]]
2019-01-18 09:24:05:002 [sys-#48%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finish exchange
future [startVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], err=null]
2019-01-18 09:24:05:057 [grid-nio-worker-tcp-comm-1-#26%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.216.155:51920]
2019-01-18 09:24:05:265 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridCachePartitionExchangeManager:478 - Rebalancing scheduled
[order=[ignite-sys-cache, data_point_new]]
2019-01-18 09:24:05:267 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridCachePartitionExchangeManager:478 - Rebalancing started
[top=AffinityTopologyVersion [topVer=18, minorTopVer=0], evt=NODE_JOINED,
node=67815159-8a6d-4c3a-b626-a82e5bea1a65]
2019-01-18 09:24:05:268 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
[grp=ignite-sys-cache, mode=SYNC,
fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, partitionsCount=51,
topology=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:24:05:285 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
[grp=ignite-sys-cache, mode=SYNC,
fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb, partitionsCount=49,
topology=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:24:05:791 [db-checkpoint-thread-#63%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checkpoint started
[checkpointId=d1455b4d-cee7-4909-8417-fdadc515db7e, startPtr=FileWALPointer
[idx=564, fileOff=60228974, len=12726], checkpointLockWait=0ms,
checkpointLockHoldTime=53ms, walCpRecordFsyncDuration=219ms, pages=145395,
reason='timeout']
2019-01-18 09:24:20:905 [grid-nio-worker-tcp-comm-2-#27%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.251.89:33710]
2019-01-18 09:24:20:907 [grid-nio-worker-tcp-comm-2-#27%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Received incoming connection when
already connected to this node, rejecting
[locNode=67815159-8a6d-4c3a-b626-a82e5bea1a65,
rmtNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4]
2019-01-18 09:24:21:111 [grid-nio-worker-tcp-comm-3-#28%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.251.89:33712]
2019-01-18 09:24:24:201 [grid-nio-worker-tcp-comm-0-#25%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.216.155:51946]
2019-01-18 09:24:24:203 [grid-nio-worker-tcp-comm-0-#25%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Received incoming connection when
already connected to this node, rejecting
[locNode=67815159-8a6d-4c3a-b626-a82e5bea1a65,
rmtNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb]
2019-01-18 09:24:24:406 [grid-nio-worker-tcp-comm-1-#26%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.216.155:51948]
2019-01-18 09:25:30:367 [db-checkpoint-thread-#63%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checkpoint finished
[cpId=d1455b4d-cee7-4909-8417-fdadc515db7e, pages=145395,
markPos=FileWALPointer [idx=564, fileOff=60228974, len=12726],
walSegmentsCleared=37, markDuration=993ms, pagesWrite=24067ms,
fsync=60505ms, total=85565ms]
2019-01-18 09:26:42:954 [utility-#71%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalancing
[fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
cacheOrGroup=ignite-sys-cache, topology=AffinityTopologyVersion [topVer=18,
minorTopVer=0], time=157682 ms]
2019-01-18 09:26:42:968 [utility-#74%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed (final)
rebalancing [fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb,
cacheOrGroup=ignite-sys-cache, topology=AffinityTopologyVersion [topVer=18,
minorTopVer=0], time=157692 ms]
2019-01-18 09:26:42:969 [utility-#74%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalance future:
RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache],
topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:26:42:970 [utility-#74%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
[grp=data_point_new, mode=ASYNC,
fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, partitionsCount=337,
topology=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:26:42:971 [utility-#74%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
[grp=data_point_new, mode=ASYNC,
fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb, partitionsCount=330,
topology=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:26:43:142 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Performance suggestions for grid 'ignite-server' (fix if possible)
2019-01-18 09:26:43:142 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
2019-01-18 09:26:43:143 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- ^-- Disable grid events (remove 'includeEventTypes' from configuration)
2019-01-18 09:26:43:143 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- ^-- Set max direct memory size if getting 'OOME: Direct buffer memory'
(add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
2019-01-18 09:26:43:143 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- ^-- Decrease number of backups (set 'backups' to 0)
2019-01-18 09:26:43:146 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
2019-01-18 09:26:43:146 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
-
2019-01-18 09:26:43:146 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
2019-01-18 09:26:43:147 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
-
2019-01-18 09:26:43:149 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
-
>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.6.0#20180710-sha1:669feacc5d3a4e60efcdd300dc8de99780f38eed
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 3.10.0-862.14.4.el7.x86_64 amd64
>>> CPU(s): 8
>>> Heap: 3.0GB
>>> VM name: 103@nx-s-ignite-002
>>> Ignite instance name: ignite-server
>>> Local node [ID=67815159-8A6D-4C3A-B626-A82E5BEA1A65, order=18,
>>> clientMode=false]
>>> Local node addresses: [172.17.0.1/172.17.0.1,
>>> nx-s-ignite-002/10.0.230.117, /0:0:0:0:0:0:0:1%lo, /127.0.0.1]
>>> Local ports: TCP:8080 TCP:10800 TCP:11211 TCP:47174
2019-01-18 09:26:43:155 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
Topology snapshot [ver=18, servers=3, clients=1, CPUs=32, offheap=36.0GB,
heap=17.0GB]
2019-01-18 09:26:43:155 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- Node [id=67815159-8A6D-4C3A-B626-A82E5BEA1A65, clusterState=ACTIVE]
2019-01-18 09:26:43:156 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- Baseline [id=2, size=3, online=3, offline=0]
2019-01-18 09:26:43:156 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
Data Regions Configured:
2019-01-18 09:26:43:158 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- default [initSize=256.0 MiB, maxSize=2.9 GiB, persistenceEnabled=false]
2019-01-18 09:26:43:159 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- 1G_DataRegion [initSize=512.0 MiB, maxSize=1.0 GiB,
persistenceEnabled=true]
2019-01-18 09:26:43:159 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- 3G_DataRegion [initSize=512.0 MiB, maxSize=3.0 GiB,
persistenceEnabled=false]
2019-01-18 09:26:43:160 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- 5G_DataRegion [initSize=512.0 MiB, maxSize=5.0 GiB,
persistenceEnabled=true]
2019-01-18 09:26:47:855 [grid-nio-worker-tcp-comm-2-#27%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.214.219:36298]
2019-01-18 09:27:43:141 [grid-timeout-worker-#23%ignite-server%] INFO
o.a.i.i.IgniteKernal%ignite-server:478 -
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=67815159, name=ignite-server, uptime=00:01:00.005]
^-- H/N/C [hosts=4, nodes=4, CPUs=32]
^-- CPU [cur=70.6%, avg=12.89%, GC=0.53%]
^-- PageMemory [pages=1130803]
^-- Heap [used=2230MB, free=27.39%, comm=3072MB]
^-- Non heap [used=75MB, free=-1%, comm=77MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=6, idle=10, qSize=0]
2019-01-18 09:28:10:913 [sys-#43%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalancing
[fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, cacheOrGroup=data_point_new,
topology=AffinityTopologyVersion [topVer=18, minorTopVer=0], time=87942 ms]
2019-01-18 09:28:15:627 [sys-#46%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed (final)
rebalancing [fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb,
cacheOrGroup=data_point_new, topology=AffinityTopologyVersion [topVer=18,
minorTopVer=0], time=92658 ms]
2019-01-18 09:28:15:628 [sys-#46%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalance future:
RebalanceFuture [grp=CacheGroupContext [grp=data_point_new],
topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:28:17:202 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.e.time:478 - Started exchange init [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=1], crd=false, evt=DISCOVERY_CUSTOM_EVT,
evtNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
customEvt=CacheAffinityChangeMessage
[id=fcc5eef5861-b0e299ac-bcd4-4ecf-855c-af770da317dc,
topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], exchId=null,
partsMsg=null, exchangeNeeded=true], allowMerge=false]
2019-01-18 09:28:17:209 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting for
partition release future [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=1], waitTime=0ms, futInfo=NA]
2019-01-18 09:28:17:666 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting for
partitions release latch: ClientLatch [coordinator=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=18, minorTopVer=1]]]
2019-01-18 09:28:17:666 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting for
partition release future [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=1], waitTime=0ms, futInfo=NA]
2019-01-18 09:28:17:678 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.e.time:478 - Finished exchange init [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=1], crd=false]
2019-01-18 09:28:17:685 [sys-#49%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Received full
message, will finish exchange [node=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
resVer=AffinityTopologyVersion [topVer=18, minorTopVer=1]]
2019-01-18 09:28:17:686 [sys-#49%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finish exchange
future [startVer=AffinityTopologyVersion [topVer=18, minorTopVer=1],
resVer=AffinityTopologyVersion [topVer=18, minorTopVer=1], err=null]
2019-01-18 09:28:17:688 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridCachePartitionExchangeManager:478 - Skipping rebalancing
(nothing scheduled) [top=AffinityTopologyVersion [topVer=18, minorTopVer=1],
evt=DISCOVERY_CUSTOM_EVT, node=bea075f4-a7b2-409d-8c78-5ea420d4ebd4]
2019-01-18 09:28:19:008 [flusher-0-#50%ignite-server%] INFO
c.t.i.c.s.DataPointCacheStore:43 - begin write records to
database:smart_datapoint
And I also reviewed the GC log, the GC became more frequent(1 time per
second), but only took 10~50ms.
Can anyone tell me how to decrease the stop time of ignite cluster.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to wait for initial partition map exchange
Posted by Justin Ji <bi...@foxmail.com>.
I also have this problem with ignite 2.6.0.
Spend more than 350 seconds to restart a ignite node.
Here is the system usage during partition map exchange:
<http://apache-ignite-users.70518.x6.nabble.com/file/t2000/1.png>
The CPU is at a low level and memory is enough.
Here is the logs, we can find that applying WAL changes takes a long time,
about 375664ms.
>>> __________ ________________
>>> / _/ ___/ |/ / _/_ __/ __/
>>> _/ // (7 7 // / / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 2.6.0#20180710-sha1:669feacc
>>> 2018 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org
2019-01-18 09:17:27:979 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Config URL: file:/opt/ignite/config/ignite-config-benchmark-cn.xml
2019-01-18 09:17:27:996 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- IgniteConfiguration [igniteInstanceName=ignite-server, pubPoolSize=8,
svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=16,
mgmtPoolSize=4, igfsPoolSize=8, dataStreamerPoolSize=8,
utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2,
qryPoolSize=16, igniteHome=/opt/ignite/apache-ignite-fabric,
igniteWorkDir=/opt/ignite/apache-ignite-fabric/work,
mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94fa3e,
nodeId=67815159-8a6d-4c3a-b626-a82e5bea1a65,
marsh=org.apache.ignite.internal.binary.BinaryMarshaller@72ade7e3,
marshLocJobs=false, daemon=false, p2pEnabled=false, netTimeout=5000,
sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000,
metricsUpdateFreq=2000, metricsExpTime=9223372036854775807,
discoSpi=ZookeeperDiscoverySpi [zkRootPath=/ignite/discovery,
zkConnectionString=10.0.230.12:2181,10.0.223.106:2181,10.0.247.246:2181,
joinTimeout=10000, sesTimeout=30000, clientReconnectDisabled=false,
internalLsnr=null,
stats=org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryStatistics@560348e6],
segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true,
allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi
[connectGate=null, connPlc=null, enableForcibleNodeKill=false,
enableTroubleshootingLog=false,
srvLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2@1df8b5b8,
locAddr=null, locHost=null, locPort=47174, locPortRange=100, shmemPort=-1,
directBuf=true, directSndBuf=false, idleConnTimeout=600000,
connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768,
sockRcvBuf=32768, msgQueueLimit=1024, slowClientQueueLimit=0, nioSrvr=null,
shmemSrv=null, usePairedConnections=false, connectionsPerNode=1,
tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32,
unackedMsgsBufSize=0, sockWriteTimeout=2000, lsnr=null, boundTcpPort=-1,
boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null,
ctxInitLatch=java.util.concurrent.CountDownLatch@23202fce[Count = 1],
stopping=false,
metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@7b993c65],
evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@37911f88,
colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [lsnr=null],
indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@4ea5b703,
addrRslvr=null, clientMode=false, rebalanceThreadPoolSize=4,
txCfg=org.apache.ignite.configuration.TransactionConfiguration@2a7ed1f,
cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED,
p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100,
timeSrvPortRange=100, failureDetectionTimeout=10000,
clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null,
connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@3fa247d1,
odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration
[seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null,
grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null,
binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration
[sysRegionInitSize=41943040, sysCacheMaxSize=104857600, pageSize=0,
concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default,
maxSize=3119855206, initSize=268435456, swapPath=null,
pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100,
metricsEnabled=false, metricsSubIntervalCount=5,
metricsRateTimeInterval=60000, persistenceEnabled=false,
checkpointPageBufSize=0], storagePath=/data/ignite/persistence,
checkpointFreq=360000, lockWaitTime=10000, checkpointThreads=4,
checkpointWriteOrder=SEQUENTIAL, walHistSize=10, walSegments=10,
walSegmentSize=67108864, walPath=/ignite-wal/ignite/wal,
walArchivePath=/ignite-wal/ignite/wal/archive, metricsEnabled=false,
walMode=BACKGROUND, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000,
walFsyncDelay=1000, walRecordIterBuffSize=67108864,
alwaysWriteFullPages=false,
fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@4f4c4b1a,
metricsSubIntervalCnt=5, metricsRateTimeInterval=60000,
walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=true,
walCompactionEnabled=false], activeOnStart=true, autoActivation=true,
longQryWarnTimeout=3000, sqlConnCfg=null,
cliConnCfg=ClientConnectorConfiguration [host=null, port=10800,
portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true,
maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0,
jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false,
useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null],
authEnabled=false, failureHnd=null,
commFailureRslvr=DefaultCommunicationFailureResolver []]
2019-01-18 09:17:27:997 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Daemon mode: off
2019-01-18 09:17:27:998 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- OS: Linux 3.10.0-862.14.4.el7.x86_64 amd64
2019-01-18 09:17:27:998 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- OS user: root
2019-01-18 09:17:27:999 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- PID: 103
2019-01-18 09:17:28:000 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Language runtime: Java Platform API Specification ver. 1.8
2019-01-18 09:17:28:000 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- VM information: OpenJDK Runtime Environment 1.8.0_171-b11 Oracle
Corporation OpenJDK 64-Bit Server VM 25.171-b11
2019-01-18 09:17:28:002 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- VM total memory: 3.0GB
2019-01-18 09:17:28:003 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Remote Management [restart: on, REST: on, JMX (remote: on, port: 49113,
auth: off, ssl: off)]
2019-01-18 09:17:28:003 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Logger: Log4J2Logger [quiet=false, config=log4j2.xml]
2019-01-18 09:17:28:004 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- IGNITE_HOME=/opt/ignite/apache-ignite-fabric
2019-01-18 09:17:28:004 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- VM arguments: [-Xms3g, -Xmx3g, -XX:+AlwaysPreTouch, -XX:+UseG1GC,
-XX:+ScavengeBeforeFullGC, -XX:+DisableExplicitGC,
-XX:+PrintGCApplicationStoppedTime, -XX:+PrintGCApplicationConcurrentTime,
-XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/mnt/heapdump/ignite,
-XX:+ExitOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCTimeStamps,
-XX:+PrintGCDateStamps, -XX:+UseGCLogFileRotation,
-XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=100M,
-Xloggc:/mnt/logs/apps/ignite/gc.log, -DIGNITE_QUIET=true,
-DIGNITE_SUCCESS_FILE=/opt/ignite/apache-ignite-fabric/work/ignite_success_8719edab-bf99-496e-add5-309d532f9c9b,
-Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=49113,
-Dcom.sun.management.jmxremote.authenticate=false,
-Dcom.sun.management.jmxremote.ssl=false,
-DIGNITE_HOME=/opt/ignite/apache-ignite-fabric,
-DIGNITE_PROG_NAME=/opt/ignite/apache-ignite-fabric/bin/ignite.sh]
2019-01-18 09:17:28:005 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- System cache's DataRegion size is configured to 40 MB. Use
DataStorageConfiguration.systemCacheMemorySize property to change the
setting.
2019-01-18 09:17:28:017 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']]
2019-01-18 09:17:28:022 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- 3-rd party licenses can be found at:
/opt/ignite/apache-ignite-fabric/libs/licenses
2019-01-18 09:17:28:114 [main] INFO o.a.i.i.p.p.IgnitePluginProcessor:478 -
Configured plugins:
2019-01-18 09:17:28:114 [main] INFO o.a.i.i.p.p.IgnitePluginProcessor:478 -
^-- None
2019-01-18 09:17:28:114 [main] INFO o.a.i.i.p.p.IgnitePluginProcessor:478 -
2019-01-18 09:17:28:116 [main] INFO o.a.i.i.p.f.FailureProcessor:478 -
Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0]]
2019-01-18 09:17:28:187 [main] INFO o.a.i.s.c.t.TcpCommunicationSpi:478 -
Successfully bound communication NIO server to TCP port [port=47174,
locHost=0.0.0.0/0.0.0.0, selectorsCnt=4, selectorSpins=0, pairedConn=false]
2019-01-18 09:17:28:213 [main] WARN o.a.i.s.c.n.NoopCheckpointSpi:488 -
Checkpoints are disabled (to enable configure any GridCheckpointSpi
implementation)
2019-01-18 09:17:28:248 [main] WARN o.a.i.i.m.c.GridCollisionManager:488 -
Collision resolution is disabled (all jobs will be activated upon arrival).
2019-01-18 09:17:28:249 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Security status [authentication=off, tls/ssl=off]
2019-01-18 09:17:28:379 [main] INFO o.a.i.i.p.c.p.f.PdsFoldersResolver:478
- Successfully locked persistence storage folder
[/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330]
2019-01-18 09:17:28:380 [main] INFO o.a.i.i.p.c.p.f.PdsFoldersResolver:478
- Consistent ID used for local node is
[4fa65519-a24b-4a05-a143-d39a39b9c330] according to persistence data storage
folders
2019-01-18 09:17:28:380 [main] INFO
o.a.i.i.p.c.b.CacheObjectBinaryProcessorImpl:478 - Resolved directory for
serialized binary metadata:
/opt/ignite/apache-ignite-fabric/work/binary_meta/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:28:745 [main] INFO
o.a.i.i.p.c.p.f.FilePageStoreManager:478 - Resolved page store work
directory:
/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:28:746 [main] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log work
directory:
/ignite-wal/ignite/wal/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:28:746 [main] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log
archive directory:
/ignite-wal/ignite/wal/archive/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:28:772 [main] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Started write-ahead log
manager [mode=BACKGROUND]
2019-01-18 09:17:28:792 [main] WARN
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:488 - Page eviction mode set
for [3G_DataRegion] data will have no effect because the oldest pages are
evicted automatically if Ignite persistence is enabled.
2019-01-18 09:17:28:821 [main] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Read checkpoint status
[startMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-START.bin,
endMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-END.bin]
2019-01-18 09:17:28:841 [main] INFO o.a.i.i.p.c.p.p.PageMemoryImpl:478 -
Started page memory [memoryAllocated=100.0 MiB, pages=24808, tableSize=1.9
MiB, checkpointBuffer=100.0 MiB]
2019-01-18 09:17:28:842 [main] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checking memory state
[lastValidPos=FileWALPointer [idx=508, fileOff=43025380, len=14626],
lastMarked=FileWALPointer [idx=508, fileOff=43025380, len=14626],
lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
2019-01-18 09:17:28:874 [main] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Found last checkpoint
marker [cpId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8, pos=FileWALPointer
[idx=508, fileOff=43025380, len=14626]]
2019-01-18 09:17:34:366 [main] WARN
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:488 - WAL segment tail is reached.
[ Expected next state: {Index=540,Offset=11065132}, Actual state :
{Index=9024937041086329393,Offset=23034} ]
2019-01-18 09:17:34:405 [main] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Applying lost cache
updates since last checkpoint record [lastMarked=FileWALPointer [idx=508,
fileOff=43025380, len=14626],
lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
2019-01-18 09:17:39:766 [main] WARN
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:488 - WAL segment tail is reached.
[ Expected next state: {Index=540,Offset=11065132}, Actual state :
{Index=9024937041086329393,Offset=23034} ]
2019-01-18 09:17:39:767 [main] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Finished applying WAL
changes [updatesApplied=0, time=5366ms]
2019-01-18 09:17:39:816 [main] INFO
o.a.i.i.p.c.GridClusterStateProcessor:478 - Restoring history for
BaselineTopology[id=2]
2019-01-18 09:17:39:967 [main] INFO o.a.i.i.p.o.ClientListenerProcessor:478
- Client connector processor has started on TCP port 10800
2019-01-18 09:17:40:047 [main] INFO o.a.i.i.p.r.p.t.GridTcpRestProtocol:478
- Command protocol successfully started [name=TCP binary,
host=0.0.0.0/0.0.0.0, port=11211]
2019-01-18 09:17:40:559 [main] INFO
o.a.i.i.p.r.p.h.j.GridJettyRestProtocol:478 - Command protocol successfully
started [name=Jetty REST, host=/0.0.0.0, port=8080]
2019-01-18 09:17:40:595 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Non-loopback local IPs: 10.0.230.117, 172.17.0.1,
fe80:0:0:0:48a:c9ff:fe24:4b10%ens3
2019-01-18 09:17:40:596 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Enabled local MACs: 024212517DB3, 068AC9244B10
2019-01-18 09:17:40:605 [main] INFO o.a.i.s.d.z.ZookeeperDiscoverySpi:478 -
Start Zookeeper discovery
[zkConnectionString=10.0.230.12:2181,10.0.223.106:2181,10.0.247.246:2181,
sessionTimeout=30000, zkRootPath=/ignite/discovery]
2019-01-18 09:17:40:753 [zk-ignite-server-EventThread] INFO
o.a.i.s.d.z.i.ZookeeperClient:478 - ZooKeeper client state changed
[prevState=Disconnected, newState=Connected]
2019-01-18 09:17:40:783 [main] INFO
o.a.i.s.d.z.i.ZookeeperDiscoveryImpl:478 - Node started join
[nodeId=67815159-8a6d-4c3a-b626-a82e5bea1a65, instanceName=ignite-server,
zkSessionId=0x167775ea5d90428, joinDataSize=13674,
consistentId=4fa65519-a24b-4a05-a143-d39a39b9c330, initTime=56,
nodePath=/ignite/discovery/n/ebd81838-6ce4-4335-8eba-52b1842972d6:67815159-8a6d-4c3a-b626-a82e5bea1a65:80|0000001109]
2019-01-18 09:17:40:793 [zk-ignite-server-EventThread] INFO
o.a.i.s.d.z.i.ZookeeperDiscoveryImpl:478 - Discovery coordinator already
exists, watch for previous server node
[locId=67815159-8a6d-4c3a-b626-a82e5bea1a65,
watchPath=b179406c-8f94-4bc1-a400-9731c169b66b:23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb:80|0000001107]
2019-01-18 09:17:40:852 [zk-ignite-server-EventThread] INFO
o.a.i.s.d.z.i.ZookeeperDiscoveryImpl:478 - Local join event data:
ZkJoinedNodeData [id=67815159-8a6d-4c3a-b626-a82e5bea1a65, order=18]]
2019-01-18 09:17:40:963 [main] WARN o.a.i.i.IgniteKernal%ignite-server:488
- Nodes started on local machine require more than 20% of physical RAM what
can lead to significant slowdown due to swapping (please decrease JVM heap
size, data region size or checkpoint buffer size) [required=17667MB,
available=14876MB]
2019-01-18 09:17:40:982 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.e.time:478 - Started exchange init [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], crd=false, evt=NODE_JOINED,
evtNode=67815159-8a6d-4c3a-b626-a82e5bea1a65, customEvt=null,
allowMerge=true]
2019-01-18 09:17:40:982 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.f.FilePageStoreManager:478 - Resolved page store work
directory:
/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:40:983 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log work
directory:
/ignite-wal/ignite/wal/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:40:983 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resolved write ahead log
archive directory:
/ignite-wal/ignite/wal/archive/node00-4fa65519-a24b-4a05-a143-d39a39b9c330
2019-01-18 09:17:40:986 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Started write-ahead log
manager [mode=BACKGROUND]
2019-01-18 09:17:41:000 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
[memoryAllocated=100.0 MiB, pages=24808, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
2019-01-18 09:17:41:004 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
[memoryAllocated=100.0 MiB, pages=24808, tableSize=1.9 MiB,
checkpointBuffer=100.0 MiB]
2019-01-18 09:17:41:013 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
[memoryAllocated=1.0 GiB, pages=254104, tableSize=19.8 MiB,
checkpointBuffer=256.0 MiB]
2019-01-18 09:17:41:045 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.p.PageMemoryImpl:478 - Started page memory
[memoryAllocated=5.0 GiB, pages=1270520, tableSize=98.8 MiB,
checkpointBuffer=2.0 GiB]
2019-01-18 09:17:42:442 [grid-nio-worker-tcp-comm-0-#25%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.251.89:33706]
2019-01-18 09:17:42:912 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Read checkpoint status
[startMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-START.bin,
endMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-END.bin]
2019-01-18 09:17:42:913 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checking memory state
[lastValidPos=FileWALPointer [idx=508, fileOff=43025380, len=14626],
lastMarked=FileWALPointer [idx=508, fileOff=43025380, len=14626],
lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
2019-01-18 09:17:42:930 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Found last checkpoint
marker [cpId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8, pos=FileWALPointer
[idx=508, fileOff=43025380, len=14626]]
2019-01-18 09:17:48:321 [exchange-worker-#42%ignite-server%] WARN
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:488 - WAL segment tail is reached.
[ Expected next state: {Index=540,Offset=11065132}, Actual state :
{Index=9024937041086329393,Offset=23034} ]
2019-01-18 09:17:48:333 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.w.FileWriteAheadLogManager:478 - Resuming logging to WAL
segment
[file=/ignite-wal/ignite/wal/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/0000000000000000.wal,
offset=11065132, ver=2]
2019-01-18 09:17:48:397 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridClusterStateProcessor:478 - Writing BaselineTopology[id=2]
2019-01-18 09:17:48:859 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridCacheProcessor:478 - Started cache [name=ignite-sys-cache,
id=-2100569601, memoryPolicyName=sysMemPlc, mode=REPLICATED,
atomicity=TRANSACTIONAL, backups=2147483647]
2019-01-18 09:17:48:872 [exchange-worker-#42%ignite-server%] INFO
c.t.i.c.c.DataSourceConfiguration:29 - Begin to load external
configuration...
2019-01-18 09:17:48:876 [exchange-worker-#42%ignite-server%] INFO
c.t.i.c.c.DataSourceConfiguration:50 - Initializing configuration
url=jdbc:mysql://tuya-shadows-cluster-1.cluster-cmu36ckfhd7h.rds.cn-northwest-1.amazonaws.com.cn:3306/tuya_shadows?characterEncoding=utf-8&useUnicode=true&autoReconnect=true&allowMultiQueries=true&rewriteBatchedStatements=true,username=tuya_shadows_rw,dns
cache time=30
2019-01-18 09:17:48:983 [exchange-worker-#42%ignite-server%] WARN
o.a.i.i.p.c.s.CacheOsStoreManager:488 - <data_point_new> Both Ignite native
persistence and CacheStore are configured for cache 'data_point_new'. This
configuration does not guarantee strict consistency between CacheStore and
Ignite data storage upon restarts. Consult documentation for more details.
2019-01-18 09:17:49:086 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridCacheProcessor:478 - Started cache [name=data_point_new,
id=-2047491044, memoryPolicyName=5G_DataRegion, mode=PARTITIONED,
atomicity=ATOMIC, backups=1]
2019-01-18 09:17:49:110 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting for
partition release future [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], waitTime=0ms, futInfo=NA]
2019-01-18 09:17:49:112 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting for
partition release future [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], waitTime=0ms, futInfo=NA]
2019-01-18 09:17:49:113 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Read checkpoint status
[startMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-START.bin,
endMarker=/data/ignite/persistence/node00-4fa65519-a24b-4a05-a143-d39a39b9c330/cp/1547802739471-37b53ea8-1fc7-4e05-901b-6a9e32b477a8-END.bin]
2019-01-18 09:17:49:114 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Applying lost cache
updates since last checkpoint record [lastMarked=FileWALPointer [idx=508,
fileOff=43025380, len=14626],
lastCheckpointId=37b53ea8-1fc7-4e05-901b-6a9e32b477a8]
2019-01-18 09:17:50:972 [ignite-update-notifier-timer] WARN
o.a.i.i.p.c.GridUpdateNotifier:488 - New version is available at
ignite.apache.org: 2.7.0
2019-01-18 09:18:00:978 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Failed to wait for
initial partition map exchange. Possible reasons are:
^-- Transactions in deadlock.
^-- Long running transactions (ignore if this is the case).
^-- Unreleased explicit locks.
2019-01-18 09:18:20:980 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:18:40:981 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:19:40:986 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:20:00:988 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:20:40:992 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:21:00:993 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:21:20:994 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:21:40:996 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:22:00:997 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:22:20:998 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:22:40:999 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:23:01:001 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:23:41:003 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:24:01:004 [main] WARN
o.a.i.i.p.c.GridCachePartitionExchangeManager:488 - Still waiting for
initial partition map exchange [fut=GridDhtPartitionsExchangeFuture
[firstDiscoEvt=DiscoveryEvent [evtNode=ZookeeperClusterNode
[id=67815159-8a6d-4c3a-b626-a82e5bea1a65, addrs=[172.17.0.1, 10.0.230.117,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18, loc=true, client=false],
topVer=18, nodeId8=67815159, msg=null, type=NODE_JOINED,
tstamp=1547803060924], crd=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=ZookeeperClusterNode [id=67815159-8a6d-4c3a-b626-a82e5bea1a65,
addrs=[172.17.0.1, 10.0.230.117, 0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=18,
loc=true, client=false], topVer=18, nodeId8=67815159, msg=null,
type=NODE_JOINED, tstamp=1547803060924], nodeId=67815159, evt=NODE_JOINED],
added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT,
res=null, hash=1766610307], init=false, lastVer=null,
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], futures=[ExplicitLockReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], DataStreamerReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], futures=[]],
LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]], AllTxReleaseFuture
[topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=0], futures=[]]]]]], exchActions=ExchangeActions
[startCaches=null, stopCaches=null, startGrps=[], stopGrps=[],
resetParts=null, stateChangeRequest=null], affChangeMsg=null,
initTs=1547803060975, centralizedAff=false, forceAffReassignment=false,
changeGlobalStateE=null, done=false, state=SRV, evtLatch=0,
remaining=[bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=34788805]]]
2019-01-18 09:24:04:775 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Finished applying WAL
changes [updatesApplied=1320211, time=375664ms]
2019-01-18 09:24:04:881 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.e.time:478 - Finished exchange init [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=0], crd=false]
2019-01-18 09:24:04:917 [sys-#48%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Received full
message, will finish exchange [node=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]]
2019-01-18 09:24:05:002 [sys-#48%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finish exchange
future [startVer=AffinityTopologyVersion [topVer=18, minorTopVer=0],
resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], err=null]
2019-01-18 09:24:05:057 [grid-nio-worker-tcp-comm-1-#26%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.216.155:51920]
2019-01-18 09:24:05:265 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridCachePartitionExchangeManager:478 - Rebalancing scheduled
[order=[ignite-sys-cache, data_point_new]]
2019-01-18 09:24:05:267 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridCachePartitionExchangeManager:478 - Rebalancing started
[top=AffinityTopologyVersion [topVer=18, minorTopVer=0], evt=NODE_JOINED,
node=67815159-8a6d-4c3a-b626-a82e5bea1a65]
2019-01-18 09:24:05:268 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
[grp=ignite-sys-cache, mode=SYNC,
fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, partitionsCount=51,
topology=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:24:05:285 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
[grp=ignite-sys-cache, mode=SYNC,
fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb, partitionsCount=49,
topology=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:24:05:791 [db-checkpoint-thread-#63%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checkpoint started
[checkpointId=d1455b4d-cee7-4909-8417-fdadc515db7e, startPtr=FileWALPointer
[idx=564, fileOff=60228974, len=12726], checkpointLockWait=0ms,
checkpointLockHoldTime=53ms, walCpRecordFsyncDuration=219ms, pages=145395,
reason='timeout']
2019-01-18 09:24:20:905 [grid-nio-worker-tcp-comm-2-#27%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.251.89:33710]
2019-01-18 09:24:20:907 [grid-nio-worker-tcp-comm-2-#27%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Received incoming connection when
already connected to this node, rejecting
[locNode=67815159-8a6d-4c3a-b626-a82e5bea1a65,
rmtNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4]
2019-01-18 09:24:21:111 [grid-nio-worker-tcp-comm-3-#28%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.251.89:33712]
2019-01-18 09:24:24:201 [grid-nio-worker-tcp-comm-0-#25%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.216.155:51946]
2019-01-18 09:24:24:203 [grid-nio-worker-tcp-comm-0-#25%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Received incoming connection when
already connected to this node, rejecting
[locNode=67815159-8a6d-4c3a-b626-a82e5bea1a65,
rmtNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb]
2019-01-18 09:24:24:406 [grid-nio-worker-tcp-comm-1-#26%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.216.155:51948]
2019-01-18 09:25:30:367 [db-checkpoint-thread-#63%ignite-server%] INFO
o.a.i.i.p.c.p.GridCacheDatabaseSharedManager:478 - Checkpoint finished
[cpId=d1455b4d-cee7-4909-8417-fdadc515db7e, pages=145395,
markPos=FileWALPointer [idx=564, fileOff=60228974, len=12726],
walSegmentsCleared=37, markDuration=993ms, pagesWrite=24067ms,
fsync=60505ms, total=85565ms]
2019-01-18 09:26:42:954 [utility-#71%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalancing
[fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
cacheOrGroup=ignite-sys-cache, topology=AffinityTopologyVersion [topVer=18,
minorTopVer=0], time=157682 ms]
2019-01-18 09:26:42:968 [utility-#74%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed (final)
rebalancing [fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb,
cacheOrGroup=ignite-sys-cache, topology=AffinityTopologyVersion [topVer=18,
minorTopVer=0], time=157692 ms]
2019-01-18 09:26:42:969 [utility-#74%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalance future:
RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache],
topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:26:42:970 [utility-#74%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
[grp=data_point_new, mode=ASYNC,
fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, partitionsCount=337,
topology=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:26:42:971 [utility-#74%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Starting rebalancing
[grp=data_point_new, mode=ASYNC,
fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb, partitionsCount=330,
topology=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:26:43:142 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Performance suggestions for grid 'ignite-server' (fix if possible)
2019-01-18 09:26:43:142 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
2019-01-18 09:26:43:143 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- ^-- Disable grid events (remove 'includeEventTypes' from configuration)
2019-01-18 09:26:43:143 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- ^-- Set max direct memory size if getting 'OOME: Direct buffer memory'
(add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
2019-01-18 09:26:43:143 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- ^-- Decrease number of backups (set 'backups' to 0)
2019-01-18 09:26:43:146 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
2019-01-18 09:26:43:146 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
-
2019-01-18 09:26:43:146 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
- To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
2019-01-18 09:26:43:147 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
-
2019-01-18 09:26:43:149 [main] INFO o.a.i.i.IgniteKernal%ignite-server:478
-
>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.6.0#20180710-sha1:669feacc5d3a4e60efcdd300dc8de99780f38eed
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 3.10.0-862.14.4.el7.x86_64 amd64
>>> CPU(s): 8
>>> Heap: 3.0GB
>>> VM name: 103@nx-s-ignite-002
>>> Ignite instance name: ignite-server
>>> Local node [ID=67815159-8A6D-4C3A-B626-A82E5BEA1A65, order=18,
>>> clientMode=false]
>>> Local node addresses: [172.17.0.1/172.17.0.1,
>>> nx-s-ignite-002/10.0.230.117, /0:0:0:0:0:0:0:1%lo, /127.0.0.1]
>>> Local ports: TCP:8080 TCP:10800 TCP:11211 TCP:47174
2019-01-18 09:26:43:155 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
Topology snapshot [ver=18, servers=3, clients=1, CPUs=32, offheap=36.0GB,
heap=17.0GB]
2019-01-18 09:26:43:155 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- Node [id=67815159-8A6D-4C3A-B626-A82E5BEA1A65, clusterState=ACTIVE]
2019-01-18 09:26:43:156 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- Baseline [id=2, size=3, online=3, offline=0]
2019-01-18 09:26:43:156 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
Data Regions Configured:
2019-01-18 09:26:43:158 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- default [initSize=256.0 MiB, maxSize=2.9 GiB, persistenceEnabled=false]
2019-01-18 09:26:43:159 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- 1G_DataRegion [initSize=512.0 MiB, maxSize=1.0 GiB,
persistenceEnabled=true]
2019-01-18 09:26:43:159 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- 3G_DataRegion [initSize=512.0 MiB, maxSize=3.0 GiB,
persistenceEnabled=false]
2019-01-18 09:26:43:160 [main] INFO o.a.i.i.m.d.GridDiscoveryManager:478 -
^-- 5G_DataRegion [initSize=512.0 MiB, maxSize=5.0 GiB,
persistenceEnabled=true]
2019-01-18 09:26:47:855 [grid-nio-worker-tcp-comm-2-#27%ignite-server%] INFO
o.a.i.s.c.t.TcpCommunicationSpi:478 - Accepted incoming communication
connection [locAddr=/10.0.230.117:47174, rmtAddr=/10.0.214.219:36298]
2019-01-18 09:27:43:141 [grid-timeout-worker-#23%ignite-server%] INFO
o.a.i.i.IgniteKernal%ignite-server:478 -
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=67815159, name=ignite-server, uptime=00:01:00.005]
^-- H/N/C [hosts=4, nodes=4, CPUs=32]
^-- CPU [cur=70.6%, avg=12.89%, GC=0.53%]
^-- PageMemory [pages=1130803]
^-- Heap [used=2230MB, free=27.39%, comm=3072MB]
^-- Non heap [used=75MB, free=-1%, comm=77MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=6, idle=10, qSize=0]
2019-01-18 09:28:10:913 [sys-#43%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalancing
[fromNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, cacheOrGroup=data_point_new,
topology=AffinityTopologyVersion [topVer=18, minorTopVer=0], time=87942 ms]
2019-01-18 09:28:15:627 [sys-#46%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed (final)
rebalancing [fromNode=23fe0ad0-2b6d-4b28-bc6b-c9cb84a8becb,
cacheOrGroup=data_point_new, topology=AffinityTopologyVersion [topVer=18,
minorTopVer=0], time=92658 ms]
2019-01-18 09:28:15:628 [sys-#46%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionDemander:478 - Completed rebalance future:
RebalanceFuture [grp=CacheGroupContext [grp=data_point_new],
topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1]
2019-01-18 09:28:17:202 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.e.time:478 - Started exchange init [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=1], crd=false, evt=DISCOVERY_CUSTOM_EVT,
evtNode=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
customEvt=CacheAffinityChangeMessage
[id=fcc5eef5861-b0e299ac-bcd4-4ecf-855c-af770da317dc,
topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], exchId=null,
partsMsg=null, exchangeNeeded=true], allowMerge=false]
2019-01-18 09:28:17:209 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting for
partition release future [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=1], waitTime=0ms, futInfo=NA]
2019-01-18 09:28:17:666 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting for
partitions release latch: ClientLatch [coordinator=ZookeeperClusterNode
[id=bea075f4-a7b2-409d-8c78-5ea420d4ebd4, addrs=[172.17.0.1, 10.0.251.89,
0:0:0:0:0:0:0:1%lo, 127.0.0.1], order=3, loc=false, client=false],
ackSent=true, super=CompletableLatch [id=exchange,
topVer=AffinityTopologyVersion [topVer=18, minorTopVer=1]]]
2019-01-18 09:28:17:666 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finished waiting for
partition release future [topVer=AffinityTopologyVersion [topVer=18,
minorTopVer=1], waitTime=0ms, futInfo=NA]
2019-01-18 09:28:17:678 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.e.time:478 - Finished exchange init [topVer=AffinityTopologyVersion
[topVer=18, minorTopVer=1], crd=false]
2019-01-18 09:28:17:685 [sys-#49%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Received full
message, will finish exchange [node=bea075f4-a7b2-409d-8c78-5ea420d4ebd4,
resVer=AffinityTopologyVersion [topVer=18, minorTopVer=1]]
2019-01-18 09:28:17:686 [sys-#49%ignite-server%] INFO
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture:478 - Finish exchange
future [startVer=AffinityTopologyVersion [topVer=18, minorTopVer=1],
resVer=AffinityTopologyVersion [topVer=18, minorTopVer=1], err=null]
2019-01-18 09:28:17:688 [exchange-worker-#42%ignite-server%] INFO
o.a.i.i.p.c.GridCachePartitionExchangeManager:478 - Skipping rebalancing
(nothing scheduled) [top=AffinityTopologyVersion [topVer=18, minorTopVer=1],
evt=DISCOVERY_CUSTOM_EVT, node=bea075f4-a7b2-409d-8c78-5ea420d4ebd4]
2019-01-18 09:28:19:008 [flusher-0-#50%ignite-server%] INFO
c.t.i.c.s.DataPointCacheStore:43 - begin write records to
database:smart_datapoint
And I also reviewed the GC log, the GC became more frequent(1 time per
second), but only took 10~50ms.
Can anyone tell me how to decrease the stop time of ignite cluster.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to wait for initial partition map exchange
Posted by Valentin Kulichenko <va...@gmail.com>.
Usually it's caused by GC or network issues. First, I would recommend to
check if you're not running out of memory, and then collect and investigate
GC logs:
https://apacheignite.readme.io/docs/jvm-and-system-tuning#debugging-memory-usage-issues-and-gc-pauses
-Val
On Tue, Apr 18, 2017 at 2:10 PM, jaipal <da...@gmail.com> wrote:
> We are facing the similar issue with 1.9 version..Is there any recommended
> configuration parametres to overccome this issue
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Failed-to-wait-for-initial-partition-
> map-exchange-tp6252p12028.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Failed to wait for initial partition map exchange
Posted by jaipal <da...@gmail.com>.
We are facing the similar issue with 1.9 version..Is there any recommended
configuration parametres to overccome this issue
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p12028.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by vkulichenko <va...@gmail.com>.
Hi Yuci,
1.8 in in final testing/bug fixing stage, so it should become available
soon. However, "Failed to wait for initial partition map exchange" is a very
generic error and can be caused by different circumstances. I would
recommend to build from 1.8 branch and run your tests on it ti check if the
issue is reproduced or not.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p9338.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by yucigou <yu...@gmail.com>.
Hi Vladislav,
We have the same issue, "Failed to wait for initial partition map exchange",
when stopping or restarting one of the nodes in our cluster. In other words,
when stopping or restarting just one of the nodes, the whole cluster screws
up, i.e., the whole cluster stops working.
Just had a look at the ticket you mentioned,
https://issues.apache.org/jira/browse/IGNITE-3748, and it as said has been
fixed for version 1.8 Wonder if you know when version 1.8 would be released?
This has been in our production, and we really want this fix to be applied
as soon as possible.
Many thanks,
Yuci
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p9333.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Jason <fq...@outlook.com>.
No fix or workaround for this so far.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p7536.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Ignitebie <an...@gmail.com>.
Jason
Scenario that you shared is same as ours and we face similar issue for large
heap servers grid where one of node in restart keeps waiting on initial
partition map exchange.
Have you received a fix or were able to hack a solution. Please share.
Thanks.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p7535.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Thanks Jason, for the information.
I have created Jira ticket[1].
[1]: https://issues.apache.org/jira/browse/IGNITE-3748
On Fri, Aug 19, 2016 at 6:27 PM, Jason <fq...@outlook.com> wrote:
> Thanks Vladislav.
>
> Attached the logs & thread_dumps for the new change. Please take a look.
>
> Apache.config
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7171/Apache.config>
> default-config.xml
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7171/default-config.xml>
> logs.zip
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7171/logs.zip>
> thread_dump.zip
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7171/thread_dump.zip
> >
>
> Thanks,
> -Jason
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Failed-to-wait-for-initial-partition-
> map-exchange-tp6252p7171.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Vladislav Pyatkov
Re: Failed to wait for initial partition map exchange
Posted by Jason <fq...@outlook.com>.
Thanks Vladislav.
Attached the logs & thread_dumps for the new change. Please take a look.
Apache.config
<http://apache-ignite-users.70518.x6.nabble.com/file/n7171/Apache.config>
default-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/n7171/default-config.xml>
logs.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/n7171/logs.zip>
thread_dump.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/n7171/thread_dump.zip>
Thanks,
-Jason
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p7171.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Also, try to get dump (using jstack) from the node, which adding to the
topology.
On Fri, Aug 19, 2016 at 3:39 PM, Vladislav Pyatkov <vl...@gmail.com>
wrote:
> Jason,
>
> Please, add system property -DIGNITE_THREAD_DUMP_ON_EXCHANGE_TIMEOUT=true
> and attach new logs.
> It prints dump on log when message "Failed to wait for partition eviction"
> appeared.
>
> On Thu, Aug 18, 2016 at 6:11 PM, Jason <fq...@outlook.com> wrote:
>
>> Thanks for your suggestion, Vladislav.
>>
>> After use them, the same issue still happens.
>> There must be some logic which causes the race contention between eviction
>> and cache partition re-balance.
>>
>> Attached the logs for all the server nodes and all the config files.
>> please
>> help take a look, if any further info, don't hesitate to ask for.
>>
>> Apache.config
>> <http://apache-ignite-users.70518.x6.nabble.com/file/n7153/Apache.config>
>> cache_config_in_client.cs
>> <http://apache-ignite-users.70518.x6.nabble.com/file/n7153/
>> cache_config_in_client.cs>
>> CO3SCH010254814_ignite-c90e3d10.log
>> <http://apache-ignite-users.70518.x6.nabble.com/file/n7153/
>> CO3SCH010254814_ignite-c90e3d10.log>
>> CO3SCH050481640_ignite-ae9e33dc.log
>> <http://apache-ignite-users.70518.x6.nabble.com/file/n7153/
>> CO3SCH050481640_ignite-ae9e33dc.log>
>> CO3SCH050500219_ignite-da04136c.log
>> <http://apache-ignite-users.70518.x6.nabble.com/file/n7153/
>> CO3SCH050500219_ignite-da04136c.log>
>> CO3SCH050511031_ignite-b58b1935.log
>> <http://apache-ignite-users.70518.x6.nabble.com/file/n7153/
>> CO3SCH050511031_ignite-b58b1935.log>
>> CO3SCH050520537_ignite-5672fb54.log
>> <http://apache-ignite-users.70518.x6.nabble.com/file/n7153/
>> CO3SCH050520537_ignite-5672fb54.log>
>> default-config.xml
>> <http://apache-ignite-users.70518.x6.nabble.com/file/n7153/
>> default-config.xml>
>>
>> thanks,
>> -Jason
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Failed-to-wait-for-initial-partition-map-
>> exchange-tp6252p7153.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Vladislav Pyatkov
>
--
Vladislav Pyatkov
Re: Failed to wait for initial partition map exchange
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Jason,
Please, add system property -DIGNITE_THREAD_DUMP_ON_EXCHANGE_TIMEOUT=true
and attach new logs.
It prints dump on log when message "Failed to wait for partition eviction"
appeared.
On Thu, Aug 18, 2016 at 6:11 PM, Jason <fq...@outlook.com> wrote:
> Thanks for your suggestion, Vladislav.
>
> After use them, the same issue still happens.
> There must be some logic which causes the race contention between eviction
> and cache partition re-balance.
>
> Attached the logs for all the server nodes and all the config files. please
> help take a look, if any further info, don't hesitate to ask for.
>
> Apache.config
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7153/Apache.config>
> cache_config_in_client.cs
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7153/cache_config_in_client.cs>
> CO3SCH010254814_ignite-c90e3d10.log
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7153/CO3SCH010254814_ignite-c90e3d10.log>
> CO3SCH050481640_ignite-ae9e33dc.log
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7153/CO3SCH050481640_ignite-ae9e33dc.log>
> CO3SCH050500219_ignite-da04136c.log
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7153/CO3SCH050500219_ignite-da04136c.log>
> CO3SCH050511031_ignite-b58b1935.log
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7153/CO3SCH050511031_ignite-b58b1935.log>
> CO3SCH050520537_ignite-5672fb54.log
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7153/CO3SCH050520537_ignite-5672fb54.log>
> default-config.xml
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7153/default-config.xml>
>
> thanks,
> -Jason
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Failed-to-wait-for-initial-partition-
> map-exchange-tp6252p7153.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Vladislav Pyatkov
Re: Failed to wait for initial partition map exchange
Posted by Jason <fq...@outlook.com>.
Thanks for your suggestion, Vladislav.
After use them, the same issue still happens.
There must be some logic which causes the race contention between eviction
and cache partition re-balance.
Attached the logs for all the server nodes and all the config files. please
help take a look, if any further info, don't hesitate to ask for.
Apache.config
<http://apache-ignite-users.70518.x6.nabble.com/file/n7153/Apache.config>
cache_config_in_client.cs
<http://apache-ignite-users.70518.x6.nabble.com/file/n7153/cache_config_in_client.cs>
CO3SCH010254814_ignite-c90e3d10.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n7153/CO3SCH010254814_ignite-c90e3d10.log>
CO3SCH050481640_ignite-ae9e33dc.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n7153/CO3SCH050481640_ignite-ae9e33dc.log>
CO3SCH050500219_ignite-da04136c.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n7153/CO3SCH050500219_ignite-da04136c.log>
CO3SCH050511031_ignite-b58b1935.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n7153/CO3SCH050511031_ignite-b58b1935.log>
CO3SCH050520537_ignite-5672fb54.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n7153/CO3SCH050520537_ignite-5672fb54.log>
default-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/n7153/default-config.xml>
thanks,
-Jason
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p7153.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Leave a comment not in the thread
http://apache-ignite-users.70518.x6.nabble.com/Fail-to-join-topology-and-repeat-join-process-tt6987.html#a7148
Duplicate here:
If you think what dedlock there, you can increase
IGNITE_LONG_OPERATIONS_DUMP_TIMEOUT (through jvm system properties) and
networkTimeout (through Ignite configuration xml) to several minutes.
-DIGNITE_LONG_OPERATIONS_DUMP_TIMEOUT=300000
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="networkTimeout" value="150000" />
On Wed, Aug 17, 2016 at 5:48 PM, Jason <fq...@outlook.com> wrote:
> hi Val,
>
> I reduce the server nodes to 5 with big cache in off_heap and can
> definitely
> reproduce this issue when the new node tries to join the topology.
> For the new joining node, it takes hundreds of seconds for syncing the
> cache
> partitions, and it says it has finished with the log "Completed (final)
> rebalancing [cache=cache_raw_gbievent", but still "Failed to wait for
> partition map exchange".
>
> From the log, seems that there're two waiting partition future: one is the
> partition exchange map and the other one is the cache eviction.
>
> I've attached the full logs for 5 server nodes and the config files for
> them.
> Would you like to help take a look at and provide some suggestion? If any
> further info, don't hesitate to ask for and I can easily reproduce it to
> provide.
>
> FYI, CO3SCH050520537 is the new added node and you can use its time as a
> reference.
>
> Any advice or suggestion should be appreciated.
>
> Apache.config
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7135/Apache.config>
> default-config.xml
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n7135/default-config.xml>
> logs.zip
> <http://apache-ignite-users.70518.x6.nabble.com/file/n7135/logs.zip>
>
> Thanks,
> -Jason
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Failed-to-wait-for-initial-partition-
> map-exchange-tp6252p7135.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Vladislav Pyatkov
Re: Failed to wait for initial partition map exchange
Posted by Jason <fq...@outlook.com>.
hi Val,
I reduce the server nodes to 5 with big cache in off_heap and can definitely
reproduce this issue when the new node tries to join the topology.
For the new joining node, it takes hundreds of seconds for syncing the cache
partitions, and it says it has finished with the log "Completed (final)
rebalancing [cache=cache_raw_gbievent", but still "Failed to wait for
partition map exchange".
From the log, seems that there're two waiting partition future: one is the
partition exchange map and the other one is the cache eviction.
I've attached the full logs for 5 server nodes and the config files for
them.
Would you like to help take a look at and provide some suggestion? If any
further info, don't hesitate to ask for and I can easily reproduce it to
provide.
FYI, CO3SCH050520537 is the new added node and you can use its time as a
reference.
Any advice or suggestion should be appreciated.
Apache.config
<http://apache-ignite-users.70518.x6.nabble.com/file/n7135/Apache.config>
default-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/n7135/default-config.xml>
logs.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/n7135/logs.zip>
Thanks,
-Jason
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p7135.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Jason <fq...@outlook.com>.
hi Val,
have tried the 1.7.0, and still the same error.
seems that it can be reproduced definitely when the cache is very big like
over 15G per node in the off-heap.
Any logic can be correlated with this?
Thanks,
-Jason
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p7046.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by vkulichenko <va...@gmail.com>.
Hi Jason,
Can you try with 1.7? There were couple of serious fixes in communication
SPI implementation which could potentially cause these issues.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p7029.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Jason <fq...@outlook.com>.
Thanks Val.
But seems that the failed node cannot be isolated by setting the
FailureDetectionTimeout, say 2000ms.
When the cluster with 22 server nodes and 146 client runs some time, e.g.
half day, and if there's failed server nodes, e.g. network issue, the new
started nodes will be hang with "Failed to wait for initial partition map
exchange" error.
I've done some debugging for the hang node, found that after send the
GridDhtAffinityAssignmentRequest in the requestFromNextNode method, then
it's blocked by GridDhtAssignmentFetchFuture in the "exchange worker" thread
but no response incoming, and this keeps for almost one day, and still
cannot recover.
And i've also checked the log in the Node that's requested in the
requestFromNextNode, no error except the normal metric.
BTW, the attached is the thread_dump for this failed node and the requested
node, please help took a look, and any suggestion will be appreciated.
FYI, we've spent over one month on testing the Ignite so far, but seems that
if this cannot be rooted cause and resolved, we can only give up all our
effort in Ignite now.
thread_dump_for_failed_to_wait_for_initial_partition_map_exchange.txt
<http://apache-ignite-users.70518.x6.nabble.com/file/n6830/thread_dump_for_failed_to_wait_for_initial_partition_map_exchange.txt>
thread_dump_for_requested_node.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n6830/thread_dump_for_requested_node.log>
failed_node.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n6830/failed_node.log>
reqeusted_node.log
<http://apache-ignite-users.70518.x6.nabble.com/file/n6830/reqeusted_node.log>
Thanks,
-Jason
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p6830.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by vkulichenko <va...@gmail.com>.
Hi Jason,
Try the failure detection timeout. This will address vast majority of such
issues.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p6809.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by Jason <fq...@outlook.com>.
Thanks Val.
When will this bug be fixed? seems that it's very risky to run a cluster
stably with this, especially for the data fabric in memory.
BTW, in the Enterprise version, this has been fixed?
BTW again, is there any simple way to mitigate this first before it's really
fixed?
Thanks,
-Jason
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p6730.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Failed to wait for initial partition map exchange
Posted by vkulichenko <va...@gmail.com>.
Jason,
If there is OOM or long GC pause, usually a node can't send any heartbeats
and will be removed after failure detection timeout [1]. However, we have a
ticket [2] for some improvements in this area.
[1]
https://apacheignite.readme.io/v1.6/docs/cluster-config#failure-detection-timeout
[2] https://issues.apache.org/jira/browse/IGNITE-3616
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Failed-to-wait-for-initial-partition-map-exchange-tp6252p6726.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.