You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by yucigou <yu...@gmail.com> on 2016/08/05 11:57:13 UTC
Local node seems to be disconnected from topology (failure
detection timeout is reached)
Hello,
One of my Ignite nodes was stopped and the logs were appended as below. It
seems that grid-timeout-worker checks the health of the cluster every
minute. But then in my case, before the due time 23:34:19, at 23:34:03 it
reported "Local node seems to be disconnected from topology (failure
detection timeout is reached)", and the Ignite node got stopped. In turn,
the web session clustering, and so on, stopped working.
Just wonder what could cause this to happen? There should be no network
issue etc with the host machine then. It is a bit scary to us, as it can
happen to our production servers in the near future.
Thank you for your help.
Yuci
===================Ignite logs======================
[23:31:19,896][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9a069f70, name=null, uptime=10:37:03:793]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=43.17%, avg=12.83%, GC=1.1%]
^-- Heap [used=2115MB, free=61.26%, comm=3955MB]
^-- Non heap [used=138MB, free=-1%, comm=143MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[23:32:19,904][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9a069f70, name=null, uptime=10:38:03:801]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=0.83%, avg=12.87%, GC=0%]
^-- Heap [used=2638MB, free=51.69%, comm=3957MB]
^-- Non heap [used=138MB, free=-1%, comm=143MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[23:33:19,913][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=9a069f70, name=null, uptime=10:39:03:808]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=0.5%, avg=12.86%, GC=0%]
^-- Heap [used=796MB, free=85.41%, comm=3921MB]
^-- Non heap [used=138MB, free=-1%, comm=143MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[23:34:03,752][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Local
node seems to be disconnected from topology (failure detection timeout is
reached) [failureDetectionTimeout=10000, connCheckFreq=3333]
[23:34:03,783][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
is out of topology (probably, due to short-time network problems).
[23:34:03,786][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
Local node SEGMENTED: TcpDiscoveryNode
[id=9a069f70-d49d-472e-9771-7ac2353e751f, addrs=[10.3.0.64, 127.0.0.1],
sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
/127.0.0.1:47500], discPort=47500, order=56, intOrder=29,
lastExchangeTime=1470350043783, loc=true, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[23:34:03,819][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
Stopping local node according to configured segmentation policy.
[23:34:03,825][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=cef7fc5e-b854-4072-8e16-396a87d5d556,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=58, intOrder=30,
lastExchangeTime=1470311808664, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[23:34:03,827][INFO ][disco-event-worker-#44%null%][GridDiscoveryManager]
Topology snapshot [ver=59, servers=1, clients=0, CPUs=2, heap=5.3GB]
[23:34:03,874][INFO ][Thread-32][GridTcpRestProtocol] Command protocol
successfully stopped: TCP binary
[23:34:03,902][INFO ][Thread-32][GridJettyRestProtocol] Command protocol
successfully stopped: Jetty REST
[23:34:04,571][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
session-cache
[23:34:04,572][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
ignite-marshaller-sys-cache
[23:34:04,572][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
ignite-sys-cache
[23:34:04,573][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
ignite-atomics-sys-cache
[23:34:04,583][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
wicket-data-store
[23:34:04,623][INFO ][Thread-32][IgniteKernal]
>>> +---------------------------------------------------------------------------------+
>>> Ignite ver. 1.6.0#20160518-sha1:0b22c45bb9b97692208fd0705ddf8045ff34a031
>>> stopped OK
>>> +---------------------------------------------------------------------------------+
>>> Grid uptime: 10:39:48:518
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
Ever since the web application has not be usable. Whenever I try to open a
web page, I get the following error message in my application log file:
2016-08-10 15:39:43,052 ERROR root/error 495 - Failed to update web session:
null
java.lang.IllegalStateException: Grid is in invalid state to perform this
operation. It either not started yet or has already being or have stopped
[gridName=null, state=STOPPED]
at
org.apache.ignite.internal.GridKernalGatewayImpl.illegalState(GridKernalGatewayImpl.java:190)
at
org.apache.ignite.internal.GridKernalGatewayImpl.readLock(GridKernalGatewayImpl.java:90)
at
org.apache.ignite.internal.IgniteKernal.guard(IgniteKernal.java:3107)
at
org.apache.ignite.internal.IgniteKernal.cache(IgniteKernal.java:2436)
at
org.apache.ignite.cache.websession.WebSessionFilter.initCache(WebSessionFilter.java:333)
at
org.apache.ignite.cache.websession.WebSessionFilter.handleCacheOperationException(WebSessionFilter.java:877)
at
org.apache.ignite.cache.websession.WebSessionFilter.handleLoadSessionException(WebSessionFilter.java:597)
at
org.apache.ignite.cache.websession.WebSessionFilter.doFilterV2(WebSessionFilter.java:523)
at
org.apache.ignite.cache.websession.WebSessionFilter.doFilterDispatch(WebSessionFilter.java:407)
at
org.apache.ignite.cache.websession.WebSessionFilter.doFilter(WebSessionFilter.java:383)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6928.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
Happened again.
(1) Ignite logs:
[08:14:01,990][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7faa417f, name=null, uptime=14:52:13:438]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=16.5%, avg=17.81%, GC=1.6%]
^-- Heap [used=625MB, free=89.82%, comm=2047MB]
^-- Non heap [used=93MB, free=-1%, comm=96MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[08:15:02,008][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7faa417f, name=null, uptime=14:53:13:457]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=18.67%, avg=17.81%, GC=1.63%]
^-- Heap [used=373MB, free=93.92%, comm=2047MB]
^-- Non heap [used=93MB, free=-1%, comm=96MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[08:16:00,861][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Local
node seems to be disconnected from topology (failure detection timeout is
reached) [failureDetectionTimeout=10000, connCheckFreq=3333]
[08:16:00,900][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=0, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768
cap=32768], recovery=GridNioRecoveryDescriptor [acked=17892, resendCnt=0,
rcvCnt=18348, sentCnt=17901, reserved=true, lastAck=18348, nodeLeft=false,
node=TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false], connected=true, connectCnt=0, queueLimit=5120],
super=GridNioSessionImpl [locAddr=/10.3.0.64:47100,
rmtAddr=/10.3.0.65:36219, createTime=1470759707458, closeTime=0,
bytesSent=9992093, bytesRcvd=5477037971, sndSchedTime=1470813320058,
lastSndTime=1470813320341, lastRcvTime=1470813320068, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@276d0d91, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]
[08:16:00,904][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
[08:16:00,919][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
is out of topology (probably, due to short-time network problems).
[08:16:00,926][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Local node SEGMENTED: TcpDiscoveryNode
[id=7faa417f-9760-434d-bc66-6551611ebf42, addrs=[10.3.0.64, 127.0.0.1],
sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1470813360913, loc=true, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[08:16:01,104][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Stopping local node according to configured segmentation policy.
[08:16:01,109][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[08:16:01,113][INFO ][disco-event-worker-#45%null%][GridDiscoveryManager]
Topology snapshot [ver=7, servers=1, clients=0, CPUs=2, heap=6.0GB]
[08:16:01,150][INFO ][Thread-31][GridTcpRestProtocol] Command protocol
successfully stopped: TCP binary
[08:16:02,641][INFO ][Thread-31][GridJettyRestProtocol] Command protocol
successfully stopped: Jetty REST
[08:16:02,685][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
session-cache
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-marshaller-sys-cache
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-sys-cache
[08:16:02,688][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-atomics-sys-cache
(2) GC logs
2016-08-10T08:14:02.391+0100: 53542.801: [GC (Allocation Failure)
2016-08-10T08:14:02.391+0100: 53542.801: [ParNew: 130944K->0K(131008K),
0.0423711 secs] 741795K->633860K(2097088K), 0.0425532 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:02.672+0100: 53543.081: [GC (Allocation Failure)
2016-08-10T08:14:02.672+0100: 53543.081: [ParNew: 130944K->0K(131008K),
0.0390475 secs] 764804K->644675K(2097088K), 0.0392145 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:02.757+0100: 53543.166: [GC (Allocation Failure)
2016-08-10T08:14:02.757+0100: 53543.166: [ParNew: 130944K->0K(131008K),
0.0121756 secs] 775619K->649138K(2097088K), 0.0123266 secs] [Times:
user=0.02 sys=0.00, real=0.02 secs]
2016-08-10T08:14:05.892+0100: 53546.302: [GC (Allocation Failure)
2016-08-10T08:14:05.892+0100: 53546.302: [ParNew: 130944K->0K(131008K),
0.0435628 secs] 780082K->672552K(2097088K), 0.0437536 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:06.261+0100: 53546.670: [GC (Allocation Failure)
2016-08-10T08:14:06.261+0100: 53546.670: [ParNew: 130944K->0K(131008K),
0.0369493 secs] 803496K->682986K(2097088K), 0.0371186 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:06.344+0100: 53546.753: [GC (Allocation Failure)
2016-08-10T08:14:06.344+0100: 53546.754: [ParNew: 130900K->0K(131008K),
0.0129105 secs] 813886K->686672K(2097088K), 0.0130476 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:14:08.915+0100: 53549.325: [GC (Allocation Failure)
2016-08-10T08:14:08.915+0100: 53549.325: [ParNew: 130944K->0K(131008K),
0.0445546 secs] 817616K->708798K(2097088K), 0.0447388 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:14:09.177+0100: 53549.586: [GC (Allocation Failure)
2016-08-10T08:14:09.177+0100: 53549.586: [ParNew: 130944K->0K(131008K),
0.0355325 secs] 839742K->718747K(2097088K), 0.0356986 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:09.276+0100: 53549.685: [GC (Allocation Failure)
2016-08-10T08:14:09.276+0100: 53549.685: [ParNew: 130280K->0K(131008K),
0.0130139 secs] 849028K->725765K(2097088K), 0.0131672 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:11.732+0100: 53552.142: [GC (Allocation Failure)
2016-08-10T08:14:11.732+0100: 53552.142: [ParNew: 130944K->0K(131008K),
0.0438233 secs] 856709K->744928K(2097088K), 0.0439934 secs] [Times:
user=0.09 sys=0.00, real=0.04 secs]
2016-08-10T08:14:12.092+0100: 53552.502: [GC (Allocation Failure)
2016-08-10T08:14:12.092+0100: 53552.502: [ParNew: 130944K->0K(131008K),
0.0356766 secs] 875872K->754500K(2097088K), 0.0358664 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:12.197+0100: 53552.607: [GC (Allocation Failure)
2016-08-10T08:14:12.197+0100: 53552.607: [ParNew: 130141K->0K(131008K),
0.0163417 secs] 884641K->764814K(2097088K), 0.0165273 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-10T08:14:14.733+0100: 53555.143: [GC (Allocation Failure)
2016-08-10T08:14:14.733+0100: 53555.143: [ParNew: 130944K->0K(131008K),
0.0441146 secs] 895758K->780588K(2097088K), 0.0443328 secs] [Times:
user=0.09 sys=0.00, real=0.04 secs]
2016-08-10T08:14:14.992+0100: 53555.401: [GC (Allocation Failure)
2016-08-10T08:14:14.992+0100: 53555.402: [ParNew: 130944K->0K(131008K),
0.0332392 secs] 911532K->789827K(2097088K), 0.0334145 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:15.087+0100: 53555.497: [GC (Allocation Failure)
2016-08-10T08:14:15.087+0100: 53555.497: [ParNew: 129838K->0K(131008K),
0.0170639 secs] 919665K->803490K(2097088K), 0.0172879 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:17.582+0100: 53557.991: [GC (Allocation Failure)
2016-08-10T08:14:17.582+0100: 53557.992: [ParNew: 130944K->0K(131008K),
0.0459274 secs] 934434K->815938K(2097088K), 0.0461121 secs] [Times:
user=0.09 sys=0.00, real=0.04 secs]
2016-08-10T08:14:17.880+0100: 53558.289: [GC (Allocation Failure)
2016-08-10T08:14:17.880+0100: 53558.289: [ParNew: 130944K->0K(131008K),
0.0482629 secs] 946882K->825204K(2097088K), 0.0484556 secs] [Times:
user=0.07 sys=0.00, real=0.05 secs]
2016-08-10T08:14:19.811+0100: 53560.221: [GC (Allocation Failure)
2016-08-10T08:14:19.811+0100: 53560.221: [ParNew: 130944K->0K(131008K),
0.0180128 secs] 956148K->840033K(2097088K), 0.0182454 secs] [Times:
user=0.04 sys=0.00, real=0.01 secs]
2016-08-10T08:14:20.651+0100: 53561.060: [GC (Allocation Failure)
2016-08-10T08:14:20.651+0100: 53561.061: [ParNew: 130944K->0K(131008K),
0.0369033 secs] 970977K->851956K(2097088K), 0.0370763 secs] [Times:
user=0.07 sys=0.00, real=0.03 secs]
2016-08-10T08:14:20.864+0100: 53561.274: [GC (Allocation Failure)
2016-08-10T08:14:20.864+0100: 53561.274: [ParNew: 130944K->0K(131008K),
0.0321971 secs] 982900K->861367K(2097088K), 0.0323569 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:23.123+0100: 53563.533: [GC (Allocation Failure)
2016-08-10T08:14:23.124+0100: 53563.533: [ParNew: 130944K->0K(131008K),
0.0213990 secs] 992311K->875859K(2097088K), 0.0216225 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:23.743+0100: 53564.152: [GC (Allocation Failure)
2016-08-10T08:14:23.743+0100: 53564.152: [ParNew: 130944K->0K(131008K),
0.0520434 secs] 1006803K->887511K(2097088K), 0.0522115 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:23.969+0100: 53564.378: [GC (Allocation Failure)
2016-08-10T08:14:23.969+0100: 53564.378: [ParNew: 130944K->0K(131008K),
0.0327217 secs] 1018455K->896405K(2097088K), 0.0328889 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:26.132+0100: 53566.542: [GC (Allocation Failure)
2016-08-10T08:14:26.132+0100: 53566.542: [ParNew: 130944K->0K(131008K),
0.0273679 secs] 1027349K->912677K(2097088K), 0.0276098 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:26.653+0100: 53567.062: [GC (Allocation Failure)
2016-08-10T08:14:26.653+0100: 53567.062: [ParNew: 130944K->0K(131008K),
0.0434405 secs] 1043621K->923816K(2097088K), 0.0436319 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:26.948+0100: 53567.358: [GC (Allocation Failure)
2016-08-10T08:14:26.948+0100: 53567.358: [ParNew: 130944K->0K(131008K),
0.0286392 secs] 1054760K->931796K(2097088K), 0.0288045 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-10T08:14:29.227+0100: 53569.637: [GC (Allocation Failure)
2016-08-10T08:14:29.227+0100: 53569.637: [ParNew: 130944K->0K(131008K),
0.0247875 secs] 1062740K->948545K(2097088K), 0.0249828 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-10T08:14:29.710+0100: 53570.120: [GC (Allocation Failure)
2016-08-10T08:14:29.710+0100: 53570.120: [ParNew: 130944K->0K(131008K),
0.0421544 secs] 1079489K->960092K(2097088K), 0.0423354 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:14:29.962+0100: 53570.372: [GC (Allocation Failure)
2016-08-10T08:14:29.962+0100: 53570.372: [ParNew: 130944K->0K(131008K),
0.0285003 secs] 1091036K->967147K(2097088K), 0.0286752 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:32.367+0100: 53572.776: [GC (Allocation Failure)
2016-08-10T08:14:32.367+0100: 53572.777: [ParNew: 130944K->0K(131008K),
0.0324680 secs] 1098091K->984433K(2097088K), 0.0327239 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:32.831+0100: 53573.241: [GC (Allocation Failure)
2016-08-10T08:14:32.831+0100: 53573.241: [ParNew: 130944K->0K(131008K),
0.0433405 secs] 1115377K->996268K(2097088K), 0.0435092 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:14:33.015+0100: 53573.424: [GC (Allocation Failure)
2016-08-10T08:14:33.015+0100: 53573.424: [ParNew: 130944K->0K(131008K),
0.0244849 secs] 1127212K->1002321K(2097088K), 0.0246967 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:35.461+0100: 53575.871: [GC (Allocation Failure)
2016-08-10T08:14:35.461+0100: 53575.871: [ParNew: 130944K->0K(131008K),
0.0295759 secs] 1133265K->1019270K(2097088K), 0.0297637 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:35.873+0100: 53576.282: [GC (Allocation Failure)
2016-08-10T08:14:35.873+0100: 53576.282: [ParNew: 130944K->0K(131008K),
0.0435538 secs] 1150214K->1031487K(2097088K), 0.0437339 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:36.044+0100: 53576.453: [GC (Allocation Failure)
2016-08-10T08:14:36.044+0100: 53576.453: [ParNew: 130944K->0K(131008K),
0.0226148 secs] 1162431K->1037326K(2097088K), 0.0228325 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-10T08:14:38.420+0100: 53578.830: [GC (Allocation Failure)
2016-08-10T08:14:38.420+0100: 53578.830: [ParNew: 130944K->0K(131008K),
0.0367163 secs] 1168270K->1055107K(2097088K), 0.0369606 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:38.862+0100: 53579.271: [GC (Allocation Failure)
2016-08-10T08:14:38.862+0100: 53579.272: [ParNew: 130944K->0K(131008K),
0.0468229 secs] 1186051K->1067113K(2097088K), 0.0470378 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:39.072+0100: 53579.482: [GC (Allocation Failure)
2016-08-10T08:14:39.072+0100: 53579.482: [ParNew: 130944K->0K(131008K),
0.0225087 secs] 1198057K->1072718K(2097088K), 0.0226874 secs] [Times:
user=0.04 sys=0.00, real=0.03 secs]
2016-08-10T08:14:41.457+0100: 53581.867: [GC (Allocation Failure)
2016-08-10T08:14:41.458+0100: 53581.867: [ParNew: 130944K->0K(131008K),
0.0363914 secs] 1203662K->1090562K(2097088K), 0.0366144 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:41.949+0100: 53582.359: [GC (Allocation Failure)
2016-08-10T08:14:41.949+0100: 53582.359: [ParNew: 130944K->0K(131008K),
0.0445878 secs] 1221506K->1103285K(2097088K), 0.0447325 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:42.098+0100: 53582.508: [GC (Allocation Failure)
2016-08-10T08:14:42.098+0100: 53582.508: [ParNew: 130944K->0K(131008K),
0.0190605 secs] 1234229K->1108111K(2097088K), 0.0192336 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:44.517+0100: 53584.926: [GC (Allocation Failure)
2016-08-10T08:14:44.517+0100: 53584.926: [ParNew: 130944K->0K(131008K),
0.0407204 secs] 1239055K->1126424K(2097088K), 0.0409601 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:44.973+0100: 53585.382: [GC (Allocation Failure)
2016-08-10T08:14:44.973+0100: 53585.383: [ParNew: 130944K->0K(131008K),
0.0496677 secs] 1257368K->1139637K(2097088K), 0.0498551 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:45.171+0100: 53585.581: [GC (Allocation Failure)
2016-08-10T08:14:45.171+0100: 53585.581: [ParNew: 130944K->0K(131008K),
0.0184316 secs] 1270581K->1143661K(2097088K), 0.0186625 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:47.603+0100: 53588.013: [GC (Allocation Failure)
2016-08-10T08:14:47.603+0100: 53588.013: [ParNew: 130944K->0K(131008K),
0.0389169 secs] 1274605K->1162333K(2097088K), 0.0391510 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:48.020+0100: 53588.430: [GC (Allocation Failure)
2016-08-10T08:14:48.020+0100: 53588.430: [ParNew: 130944K->0K(131008K),
0.0495658 secs] 1293277K->1175922K(2097088K), 0.0497736 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:48.156+0100: 53588.566: [GC (Allocation Failure)
2016-08-10T08:14:48.157+0100: 53588.566: [ParNew: 130944K->0K(131008K),
0.0163738 secs] 1306866K->1179170K(2097088K), 0.0165479 secs] [Times:
user=0.04 sys=0.00, real=0.01 secs]
2016-08-10T08:14:50.392+0100: 53590.801: [GC (Allocation Failure)
2016-08-10T08:14:50.392+0100: 53590.801: [ParNew: 130944K->0K(131008K),
0.0359011 secs] 1310114K->1198502K(2097088K), 0.0361635 secs] [Times:
user=0.06 sys=0.00, real=0.04 secs]
2016-08-10T08:14:50.436+0100: 53590.846: [GC (CMS Initial Mark) [1
CMS-initial-mark: 1198502K(1966080K)] 1201865K(2097088K), 0.0043556 secs]
[Times: user=0.01 sys=0.00, real=0.00 secs]
2016-08-10T08:14:50.440+0100: 53590.850: [CMS-concurrent-mark-start]
2016-08-10T08:14:50.716+0100: 53591.126: [CMS-concurrent-mark: 0.269/0.276
secs] [Times: user=0.54 sys=0.01, real=0.28 secs]
2016-08-10T08:14:50.716+0100: 53591.126: [CMS-concurrent-preclean-start]
2016-08-10T08:14:50.721+0100: 53591.130: [CMS-concurrent-preclean:
0.004/0.005 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2016-08-10T08:14:50.721+0100: 53591.130:
[CMS-concurrent-abortable-preclean-start]
2016-08-10T08:14:50.857+0100: 53591.266: [GC (Allocation Failure)
2016-08-10T08:14:50.857+0100: 53591.266: [ParNew: 130944K->0K(131008K),
0.0559454 secs] 1329446K->1212314K(2097088K), 0.0561386 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:51.049+0100: 53591.459: [GC (Allocation Failure)
2016-08-10T08:14:51.049+0100: 53591.459: [ParNew: 130944K->0K(131008K),
0.0131197 secs] 1343258K->1214690K(2097088K), 0.0132465 secs] [Times:
user=0.03 sys=0.00, real=0.01 secs]
2016-08-10T08:14:51.292+0100: 53591.701: [CMS-concurrent-abortable-preclean:
0.078/0.571 secs] [Times: user=0.51 sys=0.00, real=0.57 secs]
2016-08-10T08:14:51.292+0100: 53591.702: [GC (CMS Final Remark) [YG
occupancy: 72425 K (131008 K)]2016-08-10T08:14:51.292+0100: 53591.702:
[Rescan (parallel) , 0.0131918 secs]2016-08-10T08:14:51.306+0100: 53591.715:
[weak refs processing, 0.0002923 secs]2016-08-10T08:14:51.306+0100:
53591.716: [class unloading, 0.0282178 secs]2016-08-10T08:14:51.334+0100:
53591.744: [scrub symbol table, 0.0086011 secs]2016-08-10T08:14:51.343+0100:
53591.752: [scrub string table, 0.0018392 secs][1 CMS-remark:
1214690K(1966080K)] 1287116K(2097088K), 0.0523555 secs] [Times: user=0.06
sys=0.00, real=0.06 secs]
2016-08-10T08:14:51.345+0100: 53591.755: [CMS-concurrent-sweep-start]
2016-08-10T08:14:52.236+0100: 53592.645: [CMS-concurrent-sweep: 0.882/0.891
secs] [Times: user=0.90 sys=0.01, real=0.89 secs]
2016-08-10T08:14:52.236+0100: 53592.645: [CMS-concurrent-reset-start]
2016-08-10T08:14:52.253+0100: 53592.663: [CMS-concurrent-reset: 0.017/0.017
secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:14:53.407+0100: 53593.817: [GC (Allocation Failure)
2016-08-10T08:14:53.407+0100: 53593.817: [ParNew: 130944K->0K(131008K),
0.0330565 secs] 330252K->219163K(2097088K), 0.0332613 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:53.826+0100: 53594.236: [GC (Allocation Failure)
2016-08-10T08:14:53.826+0100: 53594.236: [ParNew: 130944K->0K(131008K),
0.0509833 secs] 350107K->233352K(2097088K), 0.0511630 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:53.986+0100: 53594.396: [GC (Allocation Failure)
2016-08-10T08:14:53.986+0100: 53594.396: [ParNew: 130944K->0K(131008K),
0.0140883 secs] 364296K->234932K(2097088K), 0.0142692 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:14:56.302+0100: 53596.711: [GC (Allocation Failure)
2016-08-10T08:14:56.302+0100: 53596.711: [ParNew: 130944K->0K(131008K),
0.0335848 secs] 365876K->255257K(2097088K), 0.0340425 secs] [Times:
user=0.06 sys=0.00, real=0.04 secs]
2016-08-10T08:14:56.829+0100: 53597.239: [GC (Allocation Failure)
2016-08-10T08:14:56.829+0100: 53597.239: [ParNew: 130944K->0K(131008K),
0.0519926 secs] 386201K->269242K(2097088K), 0.0521780 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:56.971+0100: 53597.381: [GC (Allocation Failure)
2016-08-10T08:14:56.972+0100: 53597.381: [ParNew: 130944K->0K(131008K),
0.0134896 secs] 400186K->270766K(2097088K), 0.0136625 secs] [Times:
user=0.03 sys=0.00, real=0.01 secs]
2016-08-10T08:14:59.314+0100: 53599.724: [GC (Allocation Failure)
2016-08-10T08:14:59.314+0100: 53599.724: [ParNew: 130944K->0K(131008K),
0.0335347 secs] 401710K->291474K(2097088K), 0.0337116 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:59.682+0100: 53600.091: [GC (Allocation Failure)
2016-08-10T08:14:59.682+0100: 53600.091: [ParNew: 130944K->0K(131008K),
0.0494621 secs] 422418K->304965K(2097088K), 0.0496462 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:59.779+0100: 53600.189: [GC (Allocation Failure)
2016-08-10T08:14:59.779+0100: 53600.189: [ParNew: 130944K->0K(131008K),
0.0111250 secs] 435909K->306471K(2097088K), 0.0112882 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:02.265+0100: 53602.674: [GC (Allocation Failure)
2016-08-10T08:15:02.265+0100: 53602.674: [ParNew: 130944K->0K(131008K),
0.0276427 secs] 437415K->327724K(2097088K), 0.0278357 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-10T08:15:02.615+0100: 53603.025: [GC (Allocation Failure)
2016-08-10T08:15:02.616+0100: 53603.025: [ParNew: 130944K->0K(131008K),
0.0387177 secs] 458668K->340715K(2097088K), 0.0389586 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:02.702+0100: 53603.111: [GC (Allocation Failure)
2016-08-10T08:15:02.702+0100: 53603.111: [ParNew: 130944K->0K(131008K),
0.0119337 secs] 471659K->344790K(2097088K), 0.0121084 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:05.217+0100: 53605.627: [GC (Allocation Failure)
2016-08-10T08:15:05.217+0100: 53605.627: [ParNew: 130944K->0K(131008K),
0.0371829 secs] 475734K->366549K(2097088K), 0.0373651 secs] [Times:
user=0.08 sys=0.00, real=0.03 secs]
2016-08-10T08:15:05.607+0100: 53606.017: [GC (Allocation Failure)
2016-08-10T08:15:05.607+0100: 53606.017: [ParNew: 130944K->0K(131008K),
0.0619285 secs] 497493K->379133K(2097088K), 0.0621140 secs] [Times:
user=0.12 sys=0.00, real=0.06 secs]
2016-08-10T08:15:05.736+0100: 53606.146: [GC (Allocation Failure)
2016-08-10T08:15:05.737+0100: 53606.146: [ParNew: 130944K->0K(131008K),
0.0144612 secs] 510077K->383232K(2097088K), 0.0146605 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:15:08.312+0100: 53608.721: [GC (Allocation Failure)
2016-08-10T08:15:08.312+0100: 53608.721: [ParNew: 130944K->0K(131008K),
0.0372281 secs] 514176K->405043K(2097088K), 0.0373988 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:15:08.642+0100: 53609.051: [GC (Allocation Failure)
2016-08-10T08:15:08.642+0100: 53609.051: [ParNew: 130944K->0K(131008K),
0.0451608 secs] 535987K->417587K(2097088K), 0.0453694 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:08.756+0100: 53609.166: [GC (Allocation Failure)
2016-08-10T08:15:08.756+0100: 53609.166: [ParNew: 130944K->0K(131008K),
0.0131418 secs] 548531K->421687K(2097088K), 0.0133133 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:15:11.397+0100: 53611.806: [GC (Allocation Failure)
2016-08-10T08:15:11.397+0100: 53611.806: [ParNew: 130944K->0K(131008K),
0.0419434 secs] 552631K->444073K(2097088K), 0.0421394 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:15:11.819+0100: 53612.229: [GC (Allocation Failure)
2016-08-10T08:15:11.820+0100: 53612.229: [ParNew: 130944K->0K(131008K),
0.0442636 secs] 575017K->456082K(2097088K), 0.0444590 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:11.932+0100: 53612.341: [GC (Allocation Failure)
2016-08-10T08:15:11.932+0100: 53612.341: [ParNew: 130944K->0K(131008K),
0.0129157 secs] 587026K->460534K(2097088K), 0.0130707 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:14.465+0100: 53614.874: [GC (Allocation Failure)
2016-08-10T08:15:14.465+0100: 53614.874: [ParNew: 130944K->0K(131008K),
0.0399574 secs] 591478K->482586K(2097088K), 0.0401414 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:14.890+0100: 53615.300: [GC (Allocation Failure)
2016-08-10T08:15:14.890+0100: 53615.300: [ParNew: 130944K->0K(131008K),
0.0425668 secs] 613530K->494223K(2097088K), 0.0427569 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:15.002+0100: 53615.411: [GC (Allocation Failure)
2016-08-10T08:15:15.002+0100: 53615.411: [ParNew: 130944K->0K(131008K),
0.0124776 secs] 625167K->498661K(2097088K), 0.0126650 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:17.487+0100: 53617.897: [GC (Allocation Failure)
2016-08-10T08:15:17.488+0100: 53617.897: [ParNew: 130944K->0K(131008K),
0.0397801 secs] 629605K->521272K(2097088K), 0.0399852 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:15:17.776+0100: 53618.186: [GC (Allocation Failure)
2016-08-10T08:15:17.776+0100: 53618.186: [ParNew: 130944K->0K(131008K),
0.0364991 secs] 652216K->532353K(2097088K), 0.0366642 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:15:17.875+0100: 53618.285: [GC (Allocation Failure)
2016-08-10T08:15:17.875+0100: 53618.285: [ParNew: 130944K->0K(131008K),
0.0119357 secs] 663297K->536837K(2097088K), 0.0120945 secs] [Times:
user=0.03 sys=0.00, real=0.01 secs]
2016-08-10T08:16:00.956+0100: 53661.365: [GC (Allocation Failure)
2016-08-10T08:16:00.956+0100: 53661.365: [ParNew: 130931K->0K(131008K),
0.0574650 secs] 667768K->558574K(2097088K), 0.0577257 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
2016-08-10T08:16:02.368+0100: 53662.777: [GC (Allocation Failure)
2016-08-10T08:16:02.368+0100: 53662.777: [ParNew: 130944K->0K(131008K),
0.0588387 secs] 689518K->573412K(2097088K), 0.0589743 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
2016-08-10T08:16:02.577+0100: 53662.987: [GC (Allocation Failure)
2016-08-10T08:16:02.578+0100: 53662.987: [ParNew: 130944K->0K(131008K),
0.0199449 secs] 704356K->576049K(2097088K), 0.0201792 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:24:22.773+0100: 54163.182: [GC (Allocation Failure)
2016-08-10T08:24:22.773+0100: 54163.182: [ParNew: 130938K->0K(131008K),
0.0344864 secs] 706988K->591374K(2097088K), 0.0346710 secs] [Times:
user=0.07 sys=0.00, real=0.03 secs]
2016-08-10T08:58:20.910+0100: 56201.319: [GC (Allocation Failure)
2016-08-10T08:58:20.910+0100: 56201.320: [ParNew: 130944K->0K(131008K),
0.0619050 secs] 722318K->598191K(2097088K), 0.0621083 secs] [Times:
user=0.12 sys=0.00, real=0.07 secs]
2016-08-10T09:32:16.079+0100: 58236.488: [GC (Allocation Failure)
2016-08-10T09:32:16.079+0100: 58236.489: [ParNew: 130944K->0K(131008K),
0.0613423 secs] 729135K->605005K(2097088K), 0.0615247 secs] [Times:
user=0.12 sys=0.00, real=0.06 secs]
(3) JVM arguments
-Djava.awt.headless=true
-Dtomcat.hostname=ves-hx-40
-Dhttp.port=8100
-Dsecure.port=8101
-Dshutdown.port=8105
-DIGNITE_HOME=/nfs/public/rw/webadmin/tomcat/bases/3rd-party/apache-ignite-fabric-1.6.0-bin
-DIGNITE_JETTY_PORT=8574
-Dtomcat.major.version=7
-Dwebadmin.path=/nfs/public/rw/webadmin
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Xms2g
-Xmx6g
-XX:MaxPermSize=128m
-Doracle.net.tns_admin=
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8107
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-DIGNITE_QUIET=false
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+UseTLAB
-XX:NewSize=128m
-XX:MaxNewSize=128m
-XX:MaxTenuringThreshold=0
-XX:SurvivorRatio=1024
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=60
-XX:+DisableExplicitGC
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps
-verbose:gc
-XX:+PrintGCDetails
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=100M
-Xloggc:/nfs/public/rw/webadmin/tomcat/bases/data/ves-hx-40/gc/gc_events.log
-Djava.endorsed.dirs=/nfs/public/rw/webadmin/tomcat/homes/apache-tomcat-7.0.55/endorsed
-Dcatalina.base=/nfs/public/rw/webadmin/tomcat/bases
-Dcatalina.home=/nfs/public/rw/webadmin/tomcat/homes/apache-tomcat-7.0.55
-Djava.io.tmpdir=/nfs/public/rw/webadmin/tomcat/bases/temp
Thanks for your kind help!
Yuci
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6926.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
Happened again.
(1) Ignite logs:
[08:14:01,990][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7faa417f, name=null, uptime=14:52:13:438]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=16.5%, avg=17.81%, GC=1.6%]
^-- Heap [used=625MB, free=89.82%, comm=2047MB]
^-- Non heap [used=93MB, free=-1%, comm=96MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[08:15:02,008][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7faa417f, name=null, uptime=14:53:13:457]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=18.67%, avg=17.81%, GC=1.63%]
^-- Heap [used=373MB, free=93.92%, comm=2047MB]
^-- Non heap [used=93MB, free=-1%, comm=96MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[08:16:00,861][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Local
node seems to be disconnected from topology (failure detection timeout is
reached) [failureDetectionTimeout=10000, connCheckFreq=3333]
[08:16:00,900][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=0, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768
cap=32768], recovery=GridNioRecoveryDescriptor [acked=17892, resendCnt=0,
rcvCnt=18348, sentCnt=17901, reserved=true, lastAck=18348, nodeLeft=false,
node=TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false], connected=true, connectCnt=0, queueLimit=5120],
super=GridNioSessionImpl [locAddr=/10.3.0.64:47100,
rmtAddr=/10.3.0.65:36219, createTime=1470759707458, closeTime=0,
bytesSent=9992093, bytesRcvd=5477037971, sndSchedTime=1470813320058,
lastSndTime=1470813320341, lastRcvTime=1470813320068, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@276d0d91, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]
[08:16:00,904][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
[08:16:00,919][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
is out of topology (probably, due to short-time network problems).
[08:16:00,926][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Local node SEGMENTED: TcpDiscoveryNode
[id=7faa417f-9760-434d-bc66-6551611ebf42, addrs=[10.3.0.64, 127.0.0.1],
sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1470813360913, loc=true, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[08:16:01,104][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Stopping local node according to configured segmentation policy.
[08:16:01,109][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[08:16:01,113][INFO ][disco-event-worker-#45%null%][GridDiscoveryManager]
Topology snapshot [ver=7, servers=1, clients=0, CPUs=2, heap=6.0GB]
[08:16:01,150][INFO ][Thread-31][GridTcpRestProtocol] Command protocol
successfully stopped: TCP binary
[08:16:02,641][INFO ][Thread-31][GridJettyRestProtocol] Command protocol
successfully stopped: Jetty REST
[08:16:02,685][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
session-cache
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-marshaller-sys-cache
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-sys-cache
[08:16:02,688][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-atomics-sys-cache
(2) GC logs
2016-08-10T08:14:02.391+0100: 53542.801: [GC (Allocation Failure)
2016-08-10T08:14:02.391+0100: 53542.801: [ParNew: 130944K->0K(131008K),
0.0423711 secs] 741795K->633860K(2097088K), 0.0425532 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:02.672+0100: 53543.081: [GC (Allocation Failure)
2016-08-10T08:14:02.672+0100: 53543.081: [ParNew: 130944K->0K(131008K),
0.0390475 secs] 764804K->644675K(2097088K), 0.0392145 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:02.757+0100: 53543.166: [GC (Allocation Failure)
2016-08-10T08:14:02.757+0100: 53543.166: [ParNew: 130944K->0K(131008K),
0.0121756 secs] 775619K->649138K(2097088K), 0.0123266 secs] [Times:
user=0.02 sys=0.00, real=0.02 secs]
2016-08-10T08:14:05.892+0100: 53546.302: [GC (Allocation Failure)
2016-08-10T08:14:05.892+0100: 53546.302: [ParNew: 130944K->0K(131008K),
0.0435628 secs] 780082K->672552K(2097088K), 0.0437536 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:06.261+0100: 53546.670: [GC (Allocation Failure)
2016-08-10T08:14:06.261+0100: 53546.670: [ParNew: 130944K->0K(131008K),
0.0369493 secs] 803496K->682986K(2097088K), 0.0371186 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:06.344+0100: 53546.753: [GC (Allocation Failure)
2016-08-10T08:14:06.344+0100: 53546.754: [ParNew: 130900K->0K(131008K),
0.0129105 secs] 813886K->686672K(2097088K), 0.0130476 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:14:08.915+0100: 53549.325: [GC (Allocation Failure)
2016-08-10T08:14:08.915+0100: 53549.325: [ParNew: 130944K->0K(131008K),
0.0445546 secs] 817616K->708798K(2097088K), 0.0447388 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:14:09.177+0100: 53549.586: [GC (Allocation Failure)
2016-08-10T08:14:09.177+0100: 53549.586: [ParNew: 130944K->0K(131008K),
0.0355325 secs] 839742K->718747K(2097088K), 0.0356986 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:09.276+0100: 53549.685: [GC (Allocation Failure)
2016-08-10T08:14:09.276+0100: 53549.685: [ParNew: 130280K->0K(131008K),
0.0130139 secs] 849028K->725765K(2097088K), 0.0131672 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:11.732+0100: 53552.142: [GC (Allocation Failure)
2016-08-10T08:14:11.732+0100: 53552.142: [ParNew: 130944K->0K(131008K),
0.0438233 secs] 856709K->744928K(2097088K), 0.0439934 secs] [Times:
user=0.09 sys=0.00, real=0.04 secs]
2016-08-10T08:14:12.092+0100: 53552.502: [GC (Allocation Failure)
2016-08-10T08:14:12.092+0100: 53552.502: [ParNew: 130944K->0K(131008K),
0.0356766 secs] 875872K->754500K(2097088K), 0.0358664 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:12.197+0100: 53552.607: [GC (Allocation Failure)
2016-08-10T08:14:12.197+0100: 53552.607: [ParNew: 130141K->0K(131008K),
0.0163417 secs] 884641K->764814K(2097088K), 0.0165273 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-10T08:14:14.733+0100: 53555.143: [GC (Allocation Failure)
2016-08-10T08:14:14.733+0100: 53555.143: [ParNew: 130944K->0K(131008K),
0.0441146 secs] 895758K->780588K(2097088K), 0.0443328 secs] [Times:
user=0.09 sys=0.00, real=0.04 secs]
2016-08-10T08:14:14.992+0100: 53555.401: [GC (Allocation Failure)
2016-08-10T08:14:14.992+0100: 53555.402: [ParNew: 130944K->0K(131008K),
0.0332392 secs] 911532K->789827K(2097088K), 0.0334145 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:15.087+0100: 53555.497: [GC (Allocation Failure)
2016-08-10T08:14:15.087+0100: 53555.497: [ParNew: 129838K->0K(131008K),
0.0170639 secs] 919665K->803490K(2097088K), 0.0172879 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:17.582+0100: 53557.991: [GC (Allocation Failure)
2016-08-10T08:14:17.582+0100: 53557.992: [ParNew: 130944K->0K(131008K),
0.0459274 secs] 934434K->815938K(2097088K), 0.0461121 secs] [Times:
user=0.09 sys=0.00, real=0.04 secs]
2016-08-10T08:14:17.880+0100: 53558.289: [GC (Allocation Failure)
2016-08-10T08:14:17.880+0100: 53558.289: [ParNew: 130944K->0K(131008K),
0.0482629 secs] 946882K->825204K(2097088K), 0.0484556 secs] [Times:
user=0.07 sys=0.00, real=0.05 secs]
2016-08-10T08:14:19.811+0100: 53560.221: [GC (Allocation Failure)
2016-08-10T08:14:19.811+0100: 53560.221: [ParNew: 130944K->0K(131008K),
0.0180128 secs] 956148K->840033K(2097088K), 0.0182454 secs] [Times:
user=0.04 sys=0.00, real=0.01 secs]
2016-08-10T08:14:20.651+0100: 53561.060: [GC (Allocation Failure)
2016-08-10T08:14:20.651+0100: 53561.061: [ParNew: 130944K->0K(131008K),
0.0369033 secs] 970977K->851956K(2097088K), 0.0370763 secs] [Times:
user=0.07 sys=0.00, real=0.03 secs]
2016-08-10T08:14:20.864+0100: 53561.274: [GC (Allocation Failure)
2016-08-10T08:14:20.864+0100: 53561.274: [ParNew: 130944K->0K(131008K),
0.0321971 secs] 982900K->861367K(2097088K), 0.0323569 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:23.123+0100: 53563.533: [GC (Allocation Failure)
2016-08-10T08:14:23.124+0100: 53563.533: [ParNew: 130944K->0K(131008K),
0.0213990 secs] 992311K->875859K(2097088K), 0.0216225 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:23.743+0100: 53564.152: [GC (Allocation Failure)
2016-08-10T08:14:23.743+0100: 53564.152: [ParNew: 130944K->0K(131008K),
0.0520434 secs] 1006803K->887511K(2097088K), 0.0522115 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:23.969+0100: 53564.378: [GC (Allocation Failure)
2016-08-10T08:14:23.969+0100: 53564.378: [ParNew: 130944K->0K(131008K),
0.0327217 secs] 1018455K->896405K(2097088K), 0.0328889 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:26.132+0100: 53566.542: [GC (Allocation Failure)
2016-08-10T08:14:26.132+0100: 53566.542: [ParNew: 130944K->0K(131008K),
0.0273679 secs] 1027349K->912677K(2097088K), 0.0276098 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:26.653+0100: 53567.062: [GC (Allocation Failure)
2016-08-10T08:14:26.653+0100: 53567.062: [ParNew: 130944K->0K(131008K),
0.0434405 secs] 1043621K->923816K(2097088K), 0.0436319 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:26.948+0100: 53567.358: [GC (Allocation Failure)
2016-08-10T08:14:26.948+0100: 53567.358: [ParNew: 130944K->0K(131008K),
0.0286392 secs] 1054760K->931796K(2097088K), 0.0288045 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-10T08:14:29.227+0100: 53569.637: [GC (Allocation Failure)
2016-08-10T08:14:29.227+0100: 53569.637: [ParNew: 130944K->0K(131008K),
0.0247875 secs] 1062740K->948545K(2097088K), 0.0249828 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-10T08:14:29.710+0100: 53570.120: [GC (Allocation Failure)
2016-08-10T08:14:29.710+0100: 53570.120: [ParNew: 130944K->0K(131008K),
0.0421544 secs] 1079489K->960092K(2097088K), 0.0423354 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:14:29.962+0100: 53570.372: [GC (Allocation Failure)
2016-08-10T08:14:29.962+0100: 53570.372: [ParNew: 130944K->0K(131008K),
0.0285003 secs] 1091036K->967147K(2097088K), 0.0286752 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:32.367+0100: 53572.776: [GC (Allocation Failure)
2016-08-10T08:14:32.367+0100: 53572.777: [ParNew: 130944K->0K(131008K),
0.0324680 secs] 1098091K->984433K(2097088K), 0.0327239 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:32.831+0100: 53573.241: [GC (Allocation Failure)
2016-08-10T08:14:32.831+0100: 53573.241: [ParNew: 130944K->0K(131008K),
0.0433405 secs] 1115377K->996268K(2097088K), 0.0435092 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:14:33.015+0100: 53573.424: [GC (Allocation Failure)
2016-08-10T08:14:33.015+0100: 53573.424: [ParNew: 130944K->0K(131008K),
0.0244849 secs] 1127212K->1002321K(2097088K), 0.0246967 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-10T08:14:35.461+0100: 53575.871: [GC (Allocation Failure)
2016-08-10T08:14:35.461+0100: 53575.871: [ParNew: 130944K->0K(131008K),
0.0295759 secs] 1133265K->1019270K(2097088K), 0.0297637 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:35.873+0100: 53576.282: [GC (Allocation Failure)
2016-08-10T08:14:35.873+0100: 53576.282: [ParNew: 130944K->0K(131008K),
0.0435538 secs] 1150214K->1031487K(2097088K), 0.0437339 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:36.044+0100: 53576.453: [GC (Allocation Failure)
2016-08-10T08:14:36.044+0100: 53576.453: [ParNew: 130944K->0K(131008K),
0.0226148 secs] 1162431K->1037326K(2097088K), 0.0228325 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-10T08:14:38.420+0100: 53578.830: [GC (Allocation Failure)
2016-08-10T08:14:38.420+0100: 53578.830: [ParNew: 130944K->0K(131008K),
0.0367163 secs] 1168270K->1055107K(2097088K), 0.0369606 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:38.862+0100: 53579.271: [GC (Allocation Failure)
2016-08-10T08:14:38.862+0100: 53579.272: [ParNew: 130944K->0K(131008K),
0.0468229 secs] 1186051K->1067113K(2097088K), 0.0470378 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:39.072+0100: 53579.482: [GC (Allocation Failure)
2016-08-10T08:14:39.072+0100: 53579.482: [ParNew: 130944K->0K(131008K),
0.0225087 secs] 1198057K->1072718K(2097088K), 0.0226874 secs] [Times:
user=0.04 sys=0.00, real=0.03 secs]
2016-08-10T08:14:41.457+0100: 53581.867: [GC (Allocation Failure)
2016-08-10T08:14:41.458+0100: 53581.867: [ParNew: 130944K->0K(131008K),
0.0363914 secs] 1203662K->1090562K(2097088K), 0.0366144 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:41.949+0100: 53582.359: [GC (Allocation Failure)
2016-08-10T08:14:41.949+0100: 53582.359: [ParNew: 130944K->0K(131008K),
0.0445878 secs] 1221506K->1103285K(2097088K), 0.0447325 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:42.098+0100: 53582.508: [GC (Allocation Failure)
2016-08-10T08:14:42.098+0100: 53582.508: [ParNew: 130944K->0K(131008K),
0.0190605 secs] 1234229K->1108111K(2097088K), 0.0192336 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:44.517+0100: 53584.926: [GC (Allocation Failure)
2016-08-10T08:14:44.517+0100: 53584.926: [ParNew: 130944K->0K(131008K),
0.0407204 secs] 1239055K->1126424K(2097088K), 0.0409601 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:14:44.973+0100: 53585.382: [GC (Allocation Failure)
2016-08-10T08:14:44.973+0100: 53585.383: [ParNew: 130944K->0K(131008K),
0.0496677 secs] 1257368K->1139637K(2097088K), 0.0498551 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:45.171+0100: 53585.581: [GC (Allocation Failure)
2016-08-10T08:14:45.171+0100: 53585.581: [ParNew: 130944K->0K(131008K),
0.0184316 secs] 1270581K->1143661K(2097088K), 0.0186625 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:14:47.603+0100: 53588.013: [GC (Allocation Failure)
2016-08-10T08:14:47.603+0100: 53588.013: [ParNew: 130944K->0K(131008K),
0.0389169 secs] 1274605K->1162333K(2097088K), 0.0391510 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:48.020+0100: 53588.430: [GC (Allocation Failure)
2016-08-10T08:14:48.020+0100: 53588.430: [ParNew: 130944K->0K(131008K),
0.0495658 secs] 1293277K->1175922K(2097088K), 0.0497736 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:48.156+0100: 53588.566: [GC (Allocation Failure)
2016-08-10T08:14:48.157+0100: 53588.566: [ParNew: 130944K->0K(131008K),
0.0163738 secs] 1306866K->1179170K(2097088K), 0.0165479 secs] [Times:
user=0.04 sys=0.00, real=0.01 secs]
2016-08-10T08:14:50.392+0100: 53590.801: [GC (Allocation Failure)
2016-08-10T08:14:50.392+0100: 53590.801: [ParNew: 130944K->0K(131008K),
0.0359011 secs] 1310114K->1198502K(2097088K), 0.0361635 secs] [Times:
user=0.06 sys=0.00, real=0.04 secs]
2016-08-10T08:14:50.436+0100: 53590.846: [GC (CMS Initial Mark) [1
CMS-initial-mark: 1198502K(1966080K)] 1201865K(2097088K), 0.0043556 secs]
[Times: user=0.01 sys=0.00, real=0.00 secs]
2016-08-10T08:14:50.440+0100: 53590.850: [CMS-concurrent-mark-start]
2016-08-10T08:14:50.716+0100: 53591.126: [CMS-concurrent-mark: 0.269/0.276
secs] [Times: user=0.54 sys=0.01, real=0.28 secs]
2016-08-10T08:14:50.716+0100: 53591.126: [CMS-concurrent-preclean-start]
2016-08-10T08:14:50.721+0100: 53591.130: [CMS-concurrent-preclean:
0.004/0.005 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2016-08-10T08:14:50.721+0100: 53591.130:
[CMS-concurrent-abortable-preclean-start]
2016-08-10T08:14:50.857+0100: 53591.266: [GC (Allocation Failure)
2016-08-10T08:14:50.857+0100: 53591.266: [ParNew: 130944K->0K(131008K),
0.0559454 secs] 1329446K->1212314K(2097088K), 0.0561386 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:51.049+0100: 53591.459: [GC (Allocation Failure)
2016-08-10T08:14:51.049+0100: 53591.459: [ParNew: 130944K->0K(131008K),
0.0131197 secs] 1343258K->1214690K(2097088K), 0.0132465 secs] [Times:
user=0.03 sys=0.00, real=0.01 secs]
2016-08-10T08:14:51.292+0100: 53591.701: [CMS-concurrent-abortable-preclean:
0.078/0.571 secs] [Times: user=0.51 sys=0.00, real=0.57 secs]
2016-08-10T08:14:51.292+0100: 53591.702: [GC (CMS Final Remark) [YG
occupancy: 72425 K (131008 K)]2016-08-10T08:14:51.292+0100: 53591.702:
[Rescan (parallel) , 0.0131918 secs]2016-08-10T08:14:51.306+0100: 53591.715:
[weak refs processing, 0.0002923 secs]2016-08-10T08:14:51.306+0100:
53591.716: [class unloading, 0.0282178 secs]2016-08-10T08:14:51.334+0100:
53591.744: [scrub symbol table, 0.0086011 secs]2016-08-10T08:14:51.343+0100:
53591.752: [scrub string table, 0.0018392 secs][1 CMS-remark:
1214690K(1966080K)] 1287116K(2097088K), 0.0523555 secs] [Times: user=0.06
sys=0.00, real=0.06 secs]
2016-08-10T08:14:51.345+0100: 53591.755: [CMS-concurrent-sweep-start]
2016-08-10T08:14:52.236+0100: 53592.645: [CMS-concurrent-sweep: 0.882/0.891
secs] [Times: user=0.90 sys=0.01, real=0.89 secs]
2016-08-10T08:14:52.236+0100: 53592.645: [CMS-concurrent-reset-start]
2016-08-10T08:14:52.253+0100: 53592.663: [CMS-concurrent-reset: 0.017/0.017
secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:14:53.407+0100: 53593.817: [GC (Allocation Failure)
2016-08-10T08:14:53.407+0100: 53593.817: [ParNew: 130944K->0K(131008K),
0.0330565 secs] 330252K->219163K(2097088K), 0.0332613 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:14:53.826+0100: 53594.236: [GC (Allocation Failure)
2016-08-10T08:14:53.826+0100: 53594.236: [ParNew: 130944K->0K(131008K),
0.0509833 secs] 350107K->233352K(2097088K), 0.0511630 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:53.986+0100: 53594.396: [GC (Allocation Failure)
2016-08-10T08:14:53.986+0100: 53594.396: [ParNew: 130944K->0K(131008K),
0.0140883 secs] 364296K->234932K(2097088K), 0.0142692 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:14:56.302+0100: 53596.711: [GC (Allocation Failure)
2016-08-10T08:14:56.302+0100: 53596.711: [ParNew: 130944K->0K(131008K),
0.0335848 secs] 365876K->255257K(2097088K), 0.0340425 secs] [Times:
user=0.06 sys=0.00, real=0.04 secs]
2016-08-10T08:14:56.829+0100: 53597.239: [GC (Allocation Failure)
2016-08-10T08:14:56.829+0100: 53597.239: [ParNew: 130944K->0K(131008K),
0.0519926 secs] 386201K->269242K(2097088K), 0.0521780 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:14:56.971+0100: 53597.381: [GC (Allocation Failure)
2016-08-10T08:14:56.972+0100: 53597.381: [ParNew: 130944K->0K(131008K),
0.0134896 secs] 400186K->270766K(2097088K), 0.0136625 secs] [Times:
user=0.03 sys=0.00, real=0.01 secs]
2016-08-10T08:14:59.314+0100: 53599.724: [GC (Allocation Failure)
2016-08-10T08:14:59.314+0100: 53599.724: [ParNew: 130944K->0K(131008K),
0.0335347 secs] 401710K->291474K(2097088K), 0.0337116 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:14:59.682+0100: 53600.091: [GC (Allocation Failure)
2016-08-10T08:14:59.682+0100: 53600.091: [ParNew: 130944K->0K(131008K),
0.0494621 secs] 422418K->304965K(2097088K), 0.0496462 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:14:59.779+0100: 53600.189: [GC (Allocation Failure)
2016-08-10T08:14:59.779+0100: 53600.189: [ParNew: 130944K->0K(131008K),
0.0111250 secs] 435909K->306471K(2097088K), 0.0112882 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:02.265+0100: 53602.674: [GC (Allocation Failure)
2016-08-10T08:15:02.265+0100: 53602.674: [ParNew: 130944K->0K(131008K),
0.0276427 secs] 437415K->327724K(2097088K), 0.0278357 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-10T08:15:02.615+0100: 53603.025: [GC (Allocation Failure)
2016-08-10T08:15:02.616+0100: 53603.025: [ParNew: 130944K->0K(131008K),
0.0387177 secs] 458668K->340715K(2097088K), 0.0389586 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:02.702+0100: 53603.111: [GC (Allocation Failure)
2016-08-10T08:15:02.702+0100: 53603.111: [ParNew: 130944K->0K(131008K),
0.0119337 secs] 471659K->344790K(2097088K), 0.0121084 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:05.217+0100: 53605.627: [GC (Allocation Failure)
2016-08-10T08:15:05.217+0100: 53605.627: [ParNew: 130944K->0K(131008K),
0.0371829 secs] 475734K->366549K(2097088K), 0.0373651 secs] [Times:
user=0.08 sys=0.00, real=0.03 secs]
2016-08-10T08:15:05.607+0100: 53606.017: [GC (Allocation Failure)
2016-08-10T08:15:05.607+0100: 53606.017: [ParNew: 130944K->0K(131008K),
0.0619285 secs] 497493K->379133K(2097088K), 0.0621140 secs] [Times:
user=0.12 sys=0.00, real=0.06 secs]
2016-08-10T08:15:05.736+0100: 53606.146: [GC (Allocation Failure)
2016-08-10T08:15:05.737+0100: 53606.146: [ParNew: 130944K->0K(131008K),
0.0144612 secs] 510077K->383232K(2097088K), 0.0146605 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:15:08.312+0100: 53608.721: [GC (Allocation Failure)
2016-08-10T08:15:08.312+0100: 53608.721: [ParNew: 130944K->0K(131008K),
0.0372281 secs] 514176K->405043K(2097088K), 0.0373988 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:15:08.642+0100: 53609.051: [GC (Allocation Failure)
2016-08-10T08:15:08.642+0100: 53609.051: [ParNew: 130944K->0K(131008K),
0.0451608 secs] 535987K->417587K(2097088K), 0.0453694 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:08.756+0100: 53609.166: [GC (Allocation Failure)
2016-08-10T08:15:08.756+0100: 53609.166: [ParNew: 130944K->0K(131008K),
0.0131418 secs] 548531K->421687K(2097088K), 0.0133133 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:15:11.397+0100: 53611.806: [GC (Allocation Failure)
2016-08-10T08:15:11.397+0100: 53611.806: [ParNew: 130944K->0K(131008K),
0.0419434 secs] 552631K->444073K(2097088K), 0.0421394 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-10T08:15:11.819+0100: 53612.229: [GC (Allocation Failure)
2016-08-10T08:15:11.820+0100: 53612.229: [ParNew: 130944K->0K(131008K),
0.0442636 secs] 575017K->456082K(2097088K), 0.0444590 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:11.932+0100: 53612.341: [GC (Allocation Failure)
2016-08-10T08:15:11.932+0100: 53612.341: [ParNew: 130944K->0K(131008K),
0.0129157 secs] 587026K->460534K(2097088K), 0.0130707 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:14.465+0100: 53614.874: [GC (Allocation Failure)
2016-08-10T08:15:14.465+0100: 53614.874: [ParNew: 130944K->0K(131008K),
0.0399574 secs] 591478K->482586K(2097088K), 0.0401414 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:14.890+0100: 53615.300: [GC (Allocation Failure)
2016-08-10T08:15:14.890+0100: 53615.300: [ParNew: 130944K->0K(131008K),
0.0425668 secs] 613530K->494223K(2097088K), 0.0427569 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:15:15.002+0100: 53615.411: [GC (Allocation Failure)
2016-08-10T08:15:15.002+0100: 53615.411: [ParNew: 130944K->0K(131008K),
0.0124776 secs] 625167K->498661K(2097088K), 0.0126650 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:15:17.487+0100: 53617.897: [GC (Allocation Failure)
2016-08-10T08:15:17.488+0100: 53617.897: [ParNew: 130944K->0K(131008K),
0.0397801 secs] 629605K->521272K(2097088K), 0.0399852 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:15:17.776+0100: 53618.186: [GC (Allocation Failure)
2016-08-10T08:15:17.776+0100: 53618.186: [ParNew: 130944K->0K(131008K),
0.0364991 secs] 652216K->532353K(2097088K), 0.0366642 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:15:17.875+0100: 53618.285: [GC (Allocation Failure)
2016-08-10T08:15:17.875+0100: 53618.285: [ParNew: 130944K->0K(131008K),
0.0119357 secs] 663297K->536837K(2097088K), 0.0120945 secs] [Times:
user=0.03 sys=0.00, real=0.01 secs]
2016-08-10T08:16:00.956+0100: 53661.365: [GC (Allocation Failure)
2016-08-10T08:16:00.956+0100: 53661.365: [ParNew: 130931K->0K(131008K),
0.0574650 secs] 667768K->558574K(2097088K), 0.0577257 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
2016-08-10T08:16:02.368+0100: 53662.777: [GC (Allocation Failure)
2016-08-10T08:16:02.368+0100: 53662.777: [ParNew: 130944K->0K(131008K),
0.0588387 secs] 689518K->573412K(2097088K), 0.0589743 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
2016-08-10T08:16:02.577+0100: 53662.987: [GC (Allocation Failure)
2016-08-10T08:16:02.578+0100: 53662.987: [ParNew: 130944K->0K(131008K),
0.0199449 secs] 704356K->576049K(2097088K), 0.0201792 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:24:22.773+0100: 54163.182: [GC (Allocation Failure)
2016-08-10T08:24:22.773+0100: 54163.182: [ParNew: 130938K->0K(131008K),
0.0344864 secs] 706988K->591374K(2097088K), 0.0346710 secs] [Times:
user=0.07 sys=0.00, real=0.03 secs]
2016-08-10T08:58:20.910+0100: 56201.319: [GC (Allocation Failure)
2016-08-10T08:58:20.910+0100: 56201.320: [ParNew: 130944K->0K(131008K),
0.0619050 secs] 722318K->598191K(2097088K), 0.0621083 secs] [Times:
user=0.12 sys=0.00, real=0.07 secs]
2016-08-10T09:32:16.079+0100: 58236.488: [GC (Allocation Failure)
2016-08-10T09:32:16.079+0100: 58236.489: [ParNew: 130944K->0K(131008K),
0.0613423 secs] 729135K->605005K(2097088K), 0.0615247 secs] [Times:
user=0.12 sys=0.00, real=0.06 secs]
Thanks for your kind help!
Yuci
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6925.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
Happened again.
(1) Ignite logs:
[08:14:01,990][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7faa417f, name=null, uptime=14:52:13:438]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=16.5%, avg=17.81%, GC=1.6%]
^-- Heap [used=625MB, free=89.82%, comm=2047MB]
^-- Non heap [used=93MB, free=-1%, comm=96MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[08:15:02,008][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7faa417f, name=null, uptime=14:53:13:457]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=18.67%, avg=17.81%, GC=1.63%]
^-- Heap [used=373MB, free=93.92%, comm=2047MB]
^-- Non heap [used=93MB, free=-1%, comm=96MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[08:16:00,861][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Local
node seems to be disconnected from topology (failure detection timeout is
reached) [failureDetectionTimeout=10000, connCheckFreq=3333]
[08:16:00,900][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=0, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768
cap=32768], recovery=GridNioRecoveryDescriptor [acked=17892, resendCnt=0,
rcvCnt=18348, sentCnt=17901, reserved=true, lastAck=18348, nodeLeft=false,
node=TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false], connected=true, connectCnt=0, queueLimit=5120],
super=GridNioSessionImpl [locAddr=/10.3.0.64:47100,
rmtAddr=/10.3.0.65:36219, createTime=1470759707458, closeTime=0,
bytesSent=9992093, bytesRcvd=5477037971, sndSchedTime=1470813320058,
lastSndTime=1470813320341, lastRcvTime=1470813320068, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@276d0d91, directMode=true],
GridConnectionBytesVerifyFilter], accepted=true]]
[08:16:00,904][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
[08:16:00,919][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
is out of topology (probably, due to short-time network problems).
[08:16:00,926][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Local node SEGMENTED: TcpDiscoveryNode
[id=7faa417f-9760-434d-bc66-6551611ebf42, addrs=[10.3.0.64, 127.0.0.1],
sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
/127.0.0.1:47500], discPort=47500, order=6, intOrder=4,
lastExchangeTime=1470813360913, loc=true, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[08:16:01,104][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Stopping local node according to configured segmentation policy.
[08:16:01,109][WARN ][disco-event-worker-#45%null%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=358fcad3-1e47-45df-8cd4-875bdcd7011a,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=4, intOrder=3,
lastExchangeTime=1470759707287, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[08:16:01,113][INFO ][disco-event-worker-#45%null%][GridDiscoveryManager]
Topology snapshot [ver=7, servers=1, clients=0, CPUs=2, heap=6.0GB]
[08:16:01,150][INFO ][Thread-31][GridTcpRestProtocol] Command protocol
successfully stopped: TCP binary
[08:16:02,641][INFO ][Thread-31][GridJettyRestProtocol] Command protocol
successfully stopped: Jetty REST
[08:16:02,685][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
session-cache
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-marshaller-sys-cache
[08:16:02,687][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-sys-cache
[08:16:02,688][INFO ][Thread-31][GridCacheProcessor] Stopped cache:
ignite-atomics-sys-cache
(2) GC logs
2016-08-10T08:08:11.335+0100: 53191.745: [GC (Allocation Failure)
2016-08-10T08:08:11.335+0100: 53191.745: [ParNew: 130944K->0K(131008K),
0.0430220 secs] 507259K->395398K(2097088K), 0.0432639 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-10T08:08:11.760+0100: 53192.169: [GC (Allocation Failure)
2016-08-10T08:08:11.760+0100: 53192.169: [ParNew: 130944K->0K(131008K),
0.0496240 secs] 526342K->408993K(2097088K), 0.0498060 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:08:11.964+0100: 53192.374: [GC (Allocation Failure)
2016-08-10T08:08:11.964+0100: 53192.374: [ParNew: 130944K->0K(131008K),
0.0200889 secs] 539937K->411803K(2097088K), 0.0203260 secs] [Times:
user=0.04 sys=0.00, real=0.03 secs]
2016-08-10T08:08:14.370+0100: 53194.780: [GC (Allocation Failure)
2016-08-10T08:08:14.370+0100: 53194.780: [ParNew: 130944K->0K(131008K),
0.0288092 secs] 542747K->431458K(2097088K), 0.0289763 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:08:14.715+0100: 53195.125: [GC (Allocation Failure)
2016-08-10T08:08:14.715+0100: 53195.125: [ParNew: 130944K->0K(131008K),
0.0512209 secs] 562402K->445397K(2097088K), 0.0514020 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-10T08:08:14.889+0100: 53195.299: [GC (Allocation Failure)
2016-08-10T08:08:14.889+0100: 53195.299: [ParNew: 130944K->0K(131008K),
0.0139681 secs] 576341K->447351K(2097088K), 0.0141538 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:08:17.371+0100: 53197.781: [GC (Allocation Failure)
2016-08-10T08:08:17.371+0100: 53197.781: [ParNew: 130944K->0K(131008K),
0.0294319 secs] 578295K->467533K(2097088K), 0.0296133 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:08:17.716+0100: 53198.126: [GC (Allocation Failure)
2016-08-10T08:08:17.717+0100: 53198.126: [ParNew: 130944K->0K(131008K),
0.0559223 secs] 598477K->481546K(2097088K), 0.0561051 secs] [Times:
user=0.11 sys=0.00, real=0.05 secs]
2016-08-10T08:08:17.829+0100: 53198.238: [GC (Allocation Failure)
2016-08-10T08:08:17.829+0100: 53198.239: [ParNew: 130944K->0K(131008K),
0.0105030 secs] 612490K->483049K(2097088K), 0.0106662 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:08:20.374+0100: 53200.783: [GC (Allocation Failure)
2016-08-10T08:08:20.374+0100: 53200.783: [ParNew: 130944K->0K(131008K),
0.0348846 secs] 613993K->503361K(2097088K), 0.0351087 secs] [Times:
user=0.06 sys=0.00, real=0.04 secs]
2016-08-10T08:08:20.830+0100: 53201.239: [GC (Allocation Failure)
2016-08-10T08:08:20.830+0100: 53201.240: [ParNew: 130944K->0K(131008K),
0.0613934 secs] 634305K->517255K(2097088K), 0.0618376 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
2016-08-10T08:08:20.977+0100: 53201.386: [GC (Allocation Failure)
2016-08-10T08:08:20.977+0100: 53201.386: [ParNew: 130944K->0K(131008K),
0.0100182 secs] 648199K->518802K(2097088K), 0.0101398 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
2016-08-10T08:08:23.450+0100: 53203.859: [GC (Allocation Failure)
2016-08-10T08:08:23.450+0100: 53203.859: [ParNew: 130944K->0K(131008K),
0.0337522 secs] 649746K->539495K(2097088K), 0.0339619 secs] [Times:
user=0.06 sys=0.00, real=0.03 secs]
2016-08-10T08:08:23.878+0100: 53204.288: [GC (Allocation Failure)
2016-08-10T08:08:23.878+0100: 53204.288: [ParNew: 130944K->0K(131008K),
0.0497076 secs] 670439K->552950K(2097088K), 0.0498960 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-10T08:08:23.994+0100: 53204.403: [GC (Allocation Failure)
2016-08-10T08:08:23.994+0100: 53204.403: [ParNew: 130944K->0K(131008K),
0.0129457 secs] 683894K->557057K(2097088K), 0.0131245 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-10T08:08:26.403+0100: 53206.813: [GC (Allocation Failure)
2016-08-10T08:08:26.403+0100: 53206.813: [ParNew: 130944K->0K(131008K),
0.0382209 secs] 688001K->578332K(2097088K), 0.0384684 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-10T08:08:26.838+0100: 53207.248: [GC (Allocation Failure)
2016-08-10T08:08:26.838+0100: 53207.248: [ParNew: 130944K->0K(131008K),
0.0450175 secs] 709276K->591347K(2097088K), 0.0452092 secs] [Times:
user=0.09 sys=0.00, real=0.04 secs]
2016-08-10T08:08:26.948+0100: 53207.358: [GC (Allocation Failure)
2016-08-10T08:08:26.948+0100: 53207.358: [ParNew: 130944K->0K(131008K),
0.0114625 secs] 722291K->595446K(2097088K), 0.0115929 secs] [Times:
user=0.02 sys=0.00, real=0.01 secs]
Thanks for your kind help!
Yuci
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6924.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by vkulichenko <va...@gmail.com>.
Yuci,
RESTART_JVM policy restarts the whole JVM (surprisingly :) ), so it works
only for standalone nodes started with ignite.sh script.
In your case you can use NOOP policy, listen to EVT_NODE_SEGMENTED event and
restart the node in a listener.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p10459.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
Hi Val,
Thanks for the explanation for not allowing re-connection of server nodes.
Regarding the RESTART_JVM policy, this policy does not work for our web
application. The reason is that in our web application Ignite is started by
the servlet context listener
org.apache.ignite.startup.servlet.ServletContextListenerStartup.
And according to the Ignite documentation, the RESTART_JVM policy will work
only if Ignite is started with CommandLineStartup via standard
ignite.{sh|bat} shell script.
https://ignite.apache.org/releases/mobile/org/apache/ignite/plugin/segmentation/SegmentationPolicy.html#RESTART_JVM
Wonder if it is possible to make the RESTART_JVM policy also work for Ignite
being started by the servlet context listener?
Thanks,
Yuci
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p10452.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by vkulichenko <va...@gmail.com>.
Hi Yuci,
We do not allow reconnection of server nodes because it's dangerous for data
consistency. Segmented node must rejoin topology as a new one. There is
RESTART_JVM policy which restarts node immediately.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p10401.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
A follow-up question about recovery.
Node ves-hx-40 was frozen for about a minute due to VM backup, and was
considered failed by the cluster.
Then ves-hx-40 woke up after the VM backup, and found itself being
disconnected from topoloyg (see below the logs). It then stopped itself.
[23:34:03,752][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Local
node seems to be disconnected from topology (failure detection timeout is
reached) [failureDetectionTimeout=10000, connCheckFreq=3333]
[23:34:03,783][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
is out of topology (probably, due to short-time network problems).
[23:34:03,786][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
Local node SEGMENTED: TcpDiscoveryNode
[id=9a069f70-d49d-472e-9771-7ac2353e751f, addrs=[10.3.0.64, 127.0.0.1],
sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
/127.0.0.1:47500], discPort=47500, order=56, intOrder=29,
lastExchangeTime=1470350043783, loc=true, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[23:34:03,819][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
Stopping local node according to configured segmentation policy.
I understand that in such situations Apache Ignite would stop the local node
according to the segmentation policy.
My question is, why Apache Ignite does not give an option to try to
reconnect to the cluster, in stead of just stopping the local node (or doing
nothing, or restart JVM)?
I think it is a reasonable policy option, that is, to regard the
disconnected local node as a new potential member of the cluster, clear all
of its local caches and states, and then rejoin the cluster.
Thanks.
Yuci
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p10386.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by vkulichenko <va...@gmail.com>.
Hi Yuci,
It's one or another - you either use failureDetectionTimeout or fine grained
settings that you mentioned. In case you use the latter,
failureDetectionTimeout is ignored.
Having said that I would recommend to try failureDetectionTimeout first and
see how it works. Then try everything else only if needed.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p7417.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
Hi vdpyatkov,
Thanks for the advice. I've just looked at the documentation
(http://apacheignite.gridgain.org/docs/cluster-config), there are actually
four timeouts to set for TcpDiscoverySpi:
1. setNetworkTimeout(long)
2. setSocketTimeout(long)
3. setAckTimeout(long)
4. setJoinTimeout(long)
Wonder if I should set up the other timeouts accordingly. In my case, set
socketTimeout 6000L, ackTimeout 6000L, networkTimeout 15000L with
joinTimeout staying at 0 as default.
What's your opinion?
Thanks,
Yuci
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p7409.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Hello,
I recoment to use failureDetectionTimeout only for test, because this
timeout determines all timeout for all SPI (Communication, Dicovery).
Use specific timeout of DiscoverySPI for prevention of segmentation
(org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi#setSocketTimeout).
The configuration lead to slow change of topology in case failure of node.
On Fri, Aug 26, 2016 at 2:49 PM, yucigou <yu...@gmail.com> wrote:
> The root cause of the problem just found is that the VMs are frozen
> sometimes.
>
> Our service team takes backup of the VMs once per day. During the backup,
> the VMs that our application servers are running on would be frozen for a
> few seconds usually, but sometimes more than 40 seconds! When I say a VM is
> frozen here, I mean it is frozen literally, and nothing is going to run
> during this period of time.
>
> So when one VM is frozen, the other Ignite node will consider it is down,
> and as a result, the node on the frozen VM is disconnected with topology
> segmented, etc.
>
> So the solution seems to be set the failureDetectionTimeout property to 60
> seconds, to tolerate the VM being frozen in its worst cases.
>
> My question is, would there be some side effects to set
> failureDetectionTimeout 60 seconds? Any advice in such a situation? Thank
> you.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Local-node-seems-to-be-disconnected-
> from-topology-failure-detection-timeout-is-reached-tp6797p7347.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Vladislav Pyatkov
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
The root cause of the problem just found is that the VMs are frozen
sometimes.
Our service team takes backup of the VMs once per day. During the backup,
the VMs that our application servers are running on would be frozen for a
few seconds usually, but sometimes more than 40 seconds! When I say a VM is
frozen here, I mean it is frozen literally, and nothing is going to run
during this period of time.
So when one VM is frozen, the other Ignite node will consider it is down,
and as a result, the node on the frozen VM is disconnected with topology
segmented, etc.
So the solution seems to be set the failureDetectionTimeout property to 60
seconds, to tolerate the VM being frozen in its worst cases.
My question is, would there be some side effects to set
failureDetectionTimeout 60 seconds? Any advice in such a situation? Thank
you.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p7347.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
OK, I'll upload all the logs when they are ready.
By the way, I've just upgraded to Ignite 1.7.0, but it does not help. The
problem persists.
Just in case if useful, please find below my Ignite configuration:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="session-cache" />
<property name="cacheMode" value="PARTITIONED"/>
<property name="backups" value="1"/>
<property name="memoryMode" value="OFFHEAP_VALUES"/>
<property name="offHeapMaxMemory" value="#{2 * 1024L * 1024L *
1024L}"/>
<property name="evictionPolicy">
<bean
class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<property name="maxSize" value="10000"/>
</bean>
</property>
</bean>
</list>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localPort" value="48100"/>
<property name="localPortRange" value="10" />
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<value>10.3.0.64:48100..48110</value>
<value>10.3.0.65:48100..48110</value>
<value>172.22.70.110:48100..48110</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="communicationSpi">
<bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="localPort" value="48500" />
<property name="localPortRange" value="10" />
</bean>
</property>
<property name="binaryConfiguration">
<bean class="org.apache.ignite.configuration.BinaryConfiguration">
<property name="compactFooter" value="false"/>
</bean>
</property>
</bean>
</beans>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p7175.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Please, attach all logs as files.
In additional, can you provide system statistics (using dstat -t --top-mem
-m -s -g -d --fs --top-io-adv)?
All logs must be collected at the moment of topology segmentation, hence
dstat must works parallel with the application.
The actions will help us discover system state when grid is segmented.
On Thu, Aug 11, 2016 at 4:37 PM, yucigou <yu...@gmail.com> wrote:
> Another question is: What is the reason that Ignite shall stop the node in
> the case of big GC pause or temporary network issue? (By the way, I'm not
> saying big GC pause or temporary network issue is the case for this topic
> though.)
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Local-node-seems-to-be-disconnected-
> from-topology-failure-detection-timeout-is-reached-tp6797p6983.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Vladislav Pyatkov
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
Another question is: What is the reason that Ignite shall stop the node in
the case of big GC pause or temporary network issue? (By the way, I'm not
saying big GC pause or temporary network issue is the case for this topic
though.)
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6983.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
Happened again, the Ignite node on ves-hx-40 went down. The other node
ves-hx-41 has been running fine, and the network should not be of any issue.
(1) Ignite logs on ves-hx-40:
[01:03:11,571][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7ba70b8b, name=null, uptime=08:29:07:892]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=12.83%, avg=12.96%, GC=0.4%]
^-- Heap [used=1040MB, free=83.06%, comm=2047MB]
^-- Non heap [used=133MB, free=-1%, comm=139MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[01:04:11,587][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=7ba70b8b, name=null, uptime=08:30:07:908]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=14.5%, avg=12.96%, GC=0.4%]
^-- Heap [used=1036MB, free=83.14%, comm=2047MB]
^-- Non heap [used=133MB, free=-1%, comm=139MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[01:05:00,007][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Local
node seems to be disconnected from topology (failure detection timeout is
reached) [failureDetectionTimeout=10000, connCheckFreq=3333]
[01:05:00,031][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Failed to process selector key (will close): GridSelectorNioSessionImpl
[selectorIdx=0, queueSize=0, writeBuf=java.nio.DirectByteBuffer[pos=0
lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768
cap=32768], recovery=GridNioRecoveryDescriptor [acked=2266, resendCnt=0,
rcvCnt=1764, sentCnt=2267, reserved=true, lastAck=1764, nodeLeft=false,
node=TcpDiscoveryNode [id=3bc24639-7d87-44da-a585-59093dd08955,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=12, intOrder=7,
lastExchangeTime=1470843764981, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false], connected=true, connectCnt=1, queueLimit=5120],
super=GridNioSessionImpl [locAddr=/10.3.0.64:54485,
rmtAddr=ves-hx-41.ebi.ac.uk/10.3.0.65:47100, createTime=1470843765245,
closeTime=0, bytesSent=33838638, bytesRcvd=31435858,
sndSchedTime=1470873900018, lastSndTime=1470873900018,
lastRcvTime=1470873854689, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@25c570a4, directMode=true],
GridConnectionBytesVerifyFilter], accepted=false]]
[01:05:00,040][WARN ][grid-nio-worker-0-#36%null%][TcpCommunicationSpi]
Closing NIO session because of unhandled exception [cls=class
o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
[01:05:00,054][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
is out of topology (probably, due to short-time network problems).
[01:05:00,058][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
Local node SEGMENTED: TcpDiscoveryNode
[id=7ba70b8b-44a8-4203-a229-cd2e3f286747, addrs=[10.3.0.64, 127.0.0.1],
sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
/127.0.0.1:47500], discPort=47500, order=8, intOrder=5,
lastExchangeTime=1470873900048, loc=true, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[01:05:00,118][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
Stopping local node according to configured segmentation policy.
[01:05:00,121][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=3bc24639-7d87-44da-a585-59093dd08955,
addrs=[10.3.0.65, 127.0.0.1],
sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
/127.0.0.1:47500], discPort=47500, order=12, intOrder=7,
lastExchangeTime=1470843764981, loc=false, ver=1.6.0#20160518-sha1:0b22c45b,
isClient=false]
[01:05:00,124][INFO ][disco-event-worker-#44%null%][GridDiscoveryManager]
Topology snapshot [ver=13, servers=1, clients=0, CPUs=2, heap=6.0GB]
[01:05:00,153][INFO ][Thread-33][GridTcpRestProtocol] Command protocol
successfully stopped: TCP binary
[01:05:00,196][INFO ][Thread-33][GridJettyRestProtocol] Command protocol
successfully stopped: Jetty REST
[01:05:00,985][INFO ][Thread-33][GridCacheProcessor] Stopped cache:
session-cache
[01:05:00,986][INFO ][Thread-33][GridCacheProcessor] Stopped cache:
ignite-marshaller-sys-cache
[01:05:00,986][INFO ][Thread-33][GridCacheProcessor] Stopped cache:
ignite-sys-cache
[01:05:00,987][INFO ][Thread-33][GridCacheProcessor] Stopped cache:
ignite-atomics-sys-cache
[01:05:01,039][INFO ][Thread-33][IgniteKernal]
>>> +---------------------------------------------------------------------------------+
>>> Ignite ver. 1.6.0#20160518-sha1:0b22c45bb9b97692208fd0705ddf8045ff34a031
>>> stopped OK
>>> +---------------------------------------------------------------------------------+
>>> Grid uptime: 08:30:57:361
(2) Ignite logs on ves-hx-41:
[00:00:49,968][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=3bc24639, name=null, uptime=07:18:03:296]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=1.17%, avg=1.37%, GC=0%]
^-- Heap [used=681MB, free=88.91%, comm=2047MB]
^-- Non heap [used=131MB, free=-1%, comm=134MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
[00:01:50,072][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=3bc24639, name=null, uptime=07:19:03:403]
^-- H/N/C [hosts=2, nodes=2, CPUs=4]
^-- CPU [cur=1.17%, avg=1.37%, GC=0%]
^-- Heap [used=617MB, free=89.94%, comm=2047MB]
^-- Non heap [used=131MB, free=-1%, comm=134MB]
^-- Public thread pool [active=0, idle=16, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=0]
(3) GC logs on ves-hx-40
2016-08-11T01:01:55.964+0100: 30484.918: [GC (Allocation Failure)
2016-08-11T01:01:55.964+0100: 30484.918: [ParNew: 130944K->0K(131008K),
0.0222134 secs] 1099957K->970725K(2097088K), 0.0224306 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:02:01.562+0100: 30490.516: [GC (Allocation Failure)
2016-08-11T01:02:01.562+0100: 30490.516: [ParNew: 130944K->0K(131008K),
0.0189181 secs] 1101669K->971698K(2097088K), 0.0191399 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:02:07.108+0100: 30496.062: [GC (Allocation Failure)
2016-08-11T01:02:07.108+0100: 30496.062: [ParNew: 130944K->0K(131008K),
0.0179867 secs] 1102642K->971963K(2097088K), 0.0182138 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-11T01:02:12.583+0100: 30501.536: [GC (Allocation Failure)
2016-08-11T01:02:12.583+0100: 30501.537: [ParNew: 130944K->0K(131008K),
0.0218757 secs] 1102907K->972258K(2097088K), 0.0221288 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-11T01:02:18.095+0100: 30507.048: [GC (Allocation Failure)
2016-08-11T01:02:18.095+0100: 30507.048: [ParNew: 130944K->0K(131008K),
0.0206840 secs] 1103202K->972384K(2097088K), 0.0209311 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:02:23.669+0100: 30512.623: [GC (Allocation Failure)
2016-08-11T01:02:23.669+0100: 30512.623: [ParNew: 130944K->0K(131008K),
0.0192695 secs] 1103328K->973391K(2097088K), 0.0194975 secs] [Times:
user=0.04 sys=0.01, real=0.02 secs]
2016-08-11T01:02:29.260+0100: 30518.214: [GC (Allocation Failure)
2016-08-11T01:02:29.260+0100: 30518.214: [ParNew: 130944K->0K(131008K),
0.0206649 secs] 1104335K->974242K(2097088K), 0.0209092 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:02:34.821+0100: 30523.775: [GC (Allocation Failure)
2016-08-11T01:02:34.821+0100: 30523.775: [ParNew: 130944K->0K(131008K),
0.0207085 secs] 1105186K->974537K(2097088K), 0.0209536 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:02:40.361+0100: 30529.315: [GC (Allocation Failure)
2016-08-11T01:02:40.361+0100: 30529.315: [ParNew: 130944K->0K(131008K),
0.0203313 secs] 1105481K->975204K(2097088K), 0.0205796 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:02:45.865+0100: 30534.819: [GC (Allocation Failure)
2016-08-11T01:02:45.865+0100: 30534.819: [ParNew: 130944K->0K(131008K),
0.0186987 secs] 1106148K->976465K(2097088K), 0.0188950 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:02:51.460+0100: 30540.414: [GC (Allocation Failure)
2016-08-11T01:02:51.460+0100: 30540.414: [ParNew: 130944K->0K(131008K),
0.0192286 secs] 1107409K->977249K(2097088K), 0.0194516 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:02:56.971+0100: 30545.925: [GC (Allocation Failure)
2016-08-11T01:02:56.971+0100: 30545.925: [ParNew: 130944K->0K(131008K),
0.0240526 secs] 1108193K->978593K(2097088K), 0.0242959 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-11T01:03:02.961+0100: 30551.915: [GC (Allocation Failure)
2016-08-11T01:03:02.961+0100: 30551.915: [ParNew: 130944K->0K(131008K),
0.0207524 secs] 1109537K->979186K(2097088K), 0.0210339 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:03:07.975+0100: 30556.928: [GC (Allocation Failure)
2016-08-11T01:03:07.975+0100: 30556.928: [ParNew: 130944K->0K(131008K),
0.0230154 secs] 1110130K->980869K(2097088K), 0.0232501 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-11T01:03:13.562+0100: 30562.515: [GC (Allocation Failure)
2016-08-11T01:03:13.562+0100: 30562.515: [ParNew: 130944K->0K(131008K),
0.0185308 secs] 1111813K->981706K(2097088K), 0.0187684 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-11T01:03:18.972+0100: 30567.926: [GC (Allocation Failure)
2016-08-11T01:03:18.972+0100: 30567.926: [ParNew: 130944K->0K(131008K),
0.0184323 secs] 1112650K->982939K(2097088K), 0.0186324 secs] [Times:
user=0.03 sys=0.00, real=0.01 secs]
2016-08-11T01:03:24.473+0100: 30573.426: [GC (Allocation Failure)
2016-08-11T01:03:24.473+0100: 30573.426: [ParNew: 130944K->0K(131008K),
0.0231739 secs] 1113883K->983951K(2097088K), 0.0234473 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-11T01:03:29.986+0100: 30578.940: [GC (Allocation Failure)
2016-08-11T01:03:29.986+0100: 30578.940: [ParNew: 130944K->0K(131008K),
0.0173278 secs] 1114895K->984105K(2097088K), 0.0175596 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:03:35.562+0100: 30584.516: [GC (Allocation Failure)
2016-08-11T01:03:35.562+0100: 30584.516: [ParNew: 130944K->0K(131008K),
0.0179701 secs] 1115049K->984790K(2097088K), 0.0182000 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:03:41.063+0100: 30590.017: [GC (Allocation Failure)
2016-08-11T01:03:41.063+0100: 30590.017: [ParNew: 130944K->0K(131008K),
0.0197956 secs] 1115734K->985729K(2097088K), 0.0200094 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:03:46.639+0100: 30595.592: [GC (Allocation Failure)
2016-08-11T01:03:46.639+0100: 30595.592: [ParNew: 130944K->0K(131008K),
0.0165813 secs] 1116673K->986217K(2097088K), 0.0167872 secs] [Times:
user=0.03 sys=0.00, real=0.01 secs]
2016-08-11T01:03:52.068+0100: 30601.022: [GC (Allocation Failure)
2016-08-11T01:03:52.069+0100: 30601.022: [ParNew: 130944K->0K(131008K),
0.0187290 secs] 1117161K->987146K(2097088K), 0.0189981 secs] [Times:
user=0.03 sys=0.00, real=0.02 secs]
2016-08-11T01:03:57.662+0100: 30606.616: [GC (Allocation Failure)
2016-08-11T01:03:57.662+0100: 30606.616: [ParNew: 130944K->0K(131008K),
0.0181821 secs] 1118090K->987894K(2097088K), 0.0184117 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:04:03.164+0100: 30612.117: [GC (Allocation Failure)
2016-08-11T01:04:03.164+0100: 30612.118: [ParNew: 130944K->0K(131008K),
0.0220690 secs] 1118838K->988924K(2097088K), 0.0223413 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:04:08.667+0100: 30617.620: [GC (Allocation Failure)
2016-08-11T01:04:08.667+0100: 30617.621: [ParNew: 130944K->0K(131008K),
0.0239423 secs] 1119868K->990345K(2097088K), 0.0241677 secs] [Times:
user=0.04 sys=0.00, real=0.03 secs]
2016-08-11T01:04:14.172+0100: 30623.126: [GC (Allocation Failure)
2016-08-11T01:04:14.172+0100: 30623.126: [ParNew: 130944K->0K(131008K),
0.0252451 secs] 1121289K->991455K(2097088K), 0.0255631 secs] [Times:
user=0.05 sys=0.00, real=0.02 secs]
2016-08-11T01:05:00.072+0100: 30669.025: [GC (Allocation Failure)
2016-08-11T01:05:00.072+0100: 30669.025: [ParNew: 130926K->0K(131008K),
0.0209069 secs] 1122381K->991927K(2097088K), 0.0212086 secs] [Times:
user=0.04 sys=0.00, real=0.02 secs]
2016-08-11T01:23:07.573+0100: 31756.527: [GC (Allocation Failure)
2016-08-11T01:23:07.573+0100: 31756.527: [ParNew: 130944K->0K(131008K),
0.0808140 secs] 1122871K->999909K(2097088K), 0.0811343 secs] [Times:
user=0.16 sys=0.00, real=0.08 secs]
2016-08-11T01:57:07.602+0100: 33796.556: [GC (Allocation Failure)
2016-08-11T01:57:07.602+0100: 33796.556: [ParNew: 130944K->0K(131008K),
0.0589586 secs] 1130853K->1006728K(2097088K), 0.0591637 secs] [Times:
user=0.11 sys=0.01, real=0.06 secs]
2016-08-11T02:31:07.731+0100: 35836.685: [GC (Allocation Failure)
2016-08-11T02:31:07.731+0100: 35836.685: [ParNew: 130944K->0K(131008K),
0.0632614 secs] 1137672K->1013488K(2097088K), 0.0643351 secs] [Times:
user=0.12 sys=0.00, real=0.07 secs]
2016-08-11T03:05:05.282+0100: 37874.235: [GC (Allocation Failure)
2016-08-11T03:05:05.282+0100: 37874.236: [ParNew: 130944K->0K(131008K),
0.0543631 secs] 1144432K->1020279K(2097088K), 0.0546948 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
2016-08-11T03:39:07.654+0100: 39916.608: [GC (Allocation Failure)
2016-08-11T03:39:07.655+0100: 39916.608: [ParNew: 130944K->0K(131008K),
0.0543531 secs] 1151223K->1027200K(2097088K), 0.0546226 secs] [Times:
user=0.11 sys=0.00, real=0.05 secs]
2016-08-11T04:13:30.740+0100: 41979.693: [GC (Allocation Failure)
2016-08-11T04:13:30.740+0100: 41979.694: [ParNew: 130944K->0K(131008K),
0.0567574 secs] 1158144K->1034040K(2097088K), 0.0570319 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
2016-08-11T04:47:39.905+0100: 44028.859: [GC (Allocation Failure)
2016-08-11T04:47:39.905+0100: 44028.859: [ParNew: 130944K->0K(131008K),
0.0781639 secs] 1164984K->1040869K(2097088K), 0.0783822 secs] [Times:
user=0.15 sys=0.00, real=0.08 secs]
2016-08-11T05:21:41.016+0100: 46069.970: [GC (Allocation Failure)
2016-08-11T05:21:41.016+0100: 46069.970: [ParNew: 130944K->0K(131008K),
0.0582058 secs] 1171813K->1047690K(2097088K), 0.0584024 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
2016-08-11T05:56:06.191+0100: 48135.144: [GC (Allocation Failure)
2016-08-11T05:56:06.191+0100: 48135.145: [ParNew: 130944K->0K(131008K),
0.0631365 secs] 1178634K->1054521K(2097088K), 0.0641970 secs] [Times:
user=0.12 sys=0.00, real=0.07 secs]
2016-08-11T06:30:07.630+0100: 50176.583: [GC (Allocation Failure)
2016-08-11T06:30:07.630+0100: 50176.584: [ParNew: 130944K->0K(131008K),
0.0562791 secs] 1185465K->1061410K(2097088K), 0.0564979 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
(4) GC logs on ves-hx-41
2016-08-11T00:56:05.110+0100: 29607.742: [GC (Allocation Failure)
2016-08-11T00:56:05.110+0100: 29607.742: [ParNew: 130944K->0K(131008K),
0.0266464 secs] 717163K->587053K(2097088K), 0.0268836 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-11T00:58:08.189+0100: 29730.821: [GC (Allocation Failure)
2016-08-11T00:58:08.189+0100: 29730.822: [ParNew: 130944K->0K(131008K),
0.0285321 secs] 717997K->588106K(2097088K), 0.0287730 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-11T01:00:11.219+0100: 29853.851: [GC (Allocation Failure)
2016-08-11T01:00:11.219+0100: 29853.851: [ParNew: 130944K->0K(131008K),
0.0281378 secs] 719050K->588719K(2097088K), 0.0283973 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-11T01:02:13.178+0100: 29975.810: [GC (Allocation Failure)
2016-08-11T01:02:13.178+0100: 29975.810: [ParNew: 130944K->0K(131008K),
0.0265786 secs] 719663K->589281K(2097088K), 0.0268661 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-11T01:04:16.454+0100: 30099.086: [GC (Allocation Failure)
2016-08-11T01:04:16.454+0100: 30099.086: [ParNew: 130944K->0K(131008K),
0.0271111 secs] 720225K->589977K(2097088K), 0.0273313 secs] [Times:
user=0.05 sys=0.00, real=0.03 secs]
2016-08-11T01:05:36.518+0100: 30179.150: [GC (Allocation Failure)
2016-08-11T01:05:36.518+0100: 30179.150: [ParNew: 130944K->0K(131008K),
0.0756634 secs] 720921K->594946K(2097088K), 0.0759647 secs] [Times:
user=0.14 sys=0.00, real=0.08 secs]
2016-08-11T01:15:11.086+0100: 30753.718: [GC (Allocation Failure)
2016-08-11T01:15:11.086+0100: 30753.718: [ParNew: 130944K->0K(131008K),
0.0346118 secs] 725890K->597225K(2097088K), 0.0348076 secs] [Times:
user=0.07 sys=0.00, real=0.04 secs]
2016-08-11T01:35:49.905+0100: 31992.537: [GC (Allocation Failure)
2016-08-11T01:35:49.905+0100: 31992.537: [ParNew: 130944K->0K(131008K),
0.0546067 secs] 728169K->601659K(2097088K), 0.0549078 secs] [Times:
user=0.10 sys=0.00, real=0.06 secs]
2016-08-11T01:56:24.223+0100: 33226.855: [GC (Allocation Failure)
2016-08-11T01:56:24.223+0100: 33226.855: [ParNew: 130944K->0K(131008K),
0.0502786 secs] 732603K->606321K(2097088K), 0.0505826 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-11T02:16:59.495+0100: 34462.127: [GC (Allocation Failure)
2016-08-11T02:16:59.495+0100: 34462.127: [ParNew: 130944K->0K(131008K),
0.0472343 secs] 737265K->610859K(2097088K), 0.0475460 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-11T02:37:43.455+0100: 35706.087: [GC (Allocation Failure)
2016-08-11T02:37:43.455+0100: 35706.088: [ParNew: 130944K->0K(131008K),
0.0489817 secs] 741803K->615325K(2097088K), 0.0494726 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-11T02:58:04.997+0100: 36927.629: [GC (Allocation Failure)
2016-08-11T02:58:04.997+0100: 36927.629: [ParNew: 130944K->0K(131008K),
0.0402853 secs] 746269K->619851K(2097088K), 0.0404998 secs] [Times:
user=0.08 sys=0.00, real=0.04 secs]
2016-08-11T03:18:49.133+0100: 38171.765: [GC (Allocation Failure)
2016-08-11T03:18:49.133+0100: 38171.765: [ParNew: 130944K->0K(131008K),
0.0534768 secs] 750795K->624281K(2097088K), 0.0538719 secs] [Times:
user=0.10 sys=0.00, real=0.06 secs]
2016-08-11T03:39:20.298+0100: 39402.931: [GC (Allocation Failure)
2016-08-11T03:39:20.299+0100: 39402.931: [ParNew: 130944K->0K(131008K),
0.0504364 secs] 755225K->628853K(2097088K), 0.0509379 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-11T03:59:50.008+0100: 40632.640: [GC (Allocation Failure)
2016-08-11T03:59:50.008+0100: 40632.640: [ParNew: 130944K->0K(131008K),
0.0448188 secs] 759797K->633370K(2097088K), 0.0451403 secs] [Times:
user=0.08 sys=0.00, real=0.05 secs]
2016-08-11T04:20:31.408+0100: 41874.040: [GC (Allocation Failure)
2016-08-11T04:20:31.408+0100: 41874.041: [ParNew: 130944K->0K(131008K),
0.0496370 secs] 764314K->637833K(2097088K), 0.0500137 secs] [Times:
user=0.10 sys=0.01, real=0.05 secs]
2016-08-11T04:41:13.371+0100: 43116.003: [GC (Allocation Failure)
2016-08-11T04:41:13.371+0100: 43116.003: [ParNew: 130944K->0K(131008K),
0.0493521 secs] 768777K->642405K(2097088K), 0.0496875 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-11T05:01:49.946+0100: 44352.578: [GC (Allocation Failure)
2016-08-11T05:01:49.946+0100: 44352.579: [ParNew: 130944K->0K(131008K),
0.0473887 secs] 773349K->646845K(2097088K), 0.0476503 secs] [Times:
user=0.09 sys=0.01, real=0.04 secs]
2016-08-11T05:22:20.050+0100: 45582.682: [GC (Allocation Failure)
2016-08-11T05:22:20.050+0100: 45582.683: [ParNew: 130944K->0K(131008K),
0.0476691 secs] 777789K->651393K(2097088K), 0.0480185 secs] [Times:
user=0.09 sys=0.00, real=0.05 secs]
2016-08-11T05:42:50.038+0100: 46812.670: [GC (Allocation Failure)
2016-08-11T05:42:50.038+0100: 46812.670: [ParNew: 130944K->0K(131008K),
0.0503541 secs] 782337K->655936K(2097088K), 0.0505806 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-11T06:03:33.827+0100: 48056.459: [GC (Allocation Failure)
2016-08-11T06:03:33.828+0100: 48056.460: [ParNew: 130944K->0K(131008K),
0.0544074 secs] 786880K->660387K(2097088K), 0.0548633 secs] [Times:
user=0.11 sys=0.00, real=0.06 secs]
2016-08-11T06:24:07.159+0100: 49289.791: [GC (Allocation Failure)
2016-08-11T06:24:07.159+0100: 49289.791: [ParNew: 130944K->0K(131008K),
0.0504913 secs] 791331K->664945K(2097088K), 0.0508051 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
2016-08-11T06:44:49.963+0100: 50532.595: [GC (Allocation Failure)
2016-08-11T06:44:49.963+0100: 50532.595: [ParNew: 130944K->0K(131008K),
0.0538779 secs] 795889K->669395K(2097088K), 0.0543000 secs] [Times:
user=0.10 sys=0.00, real=0.06 secs]
2016-08-11T07:05:30.285+0100: 51772.917: [GC (Allocation Failure)
2016-08-11T07:05:30.285+0100: 51772.917: [ParNew: 130944K->0K(131008K),
0.0509373 secs] 800339K->673953K(2097088K), 0.0512089 secs] [Times:
user=0.10 sys=0.00, real=0.05 secs]
(5) JVM arguments for ves-hx-40 (essentially same as ves-hx-41)
-Djava.awt.headless=true
-Dtomcat.hostname=ves-hx-40
-Dhttp.port=8100
-Dsecure.port=8101
-Dshutdown.port=8105
-DIGNITE_HOME=/nfs/public/rw/webadmin/tomcat/bases/3rd-party/apache-ignite-fabric-1.6.0-bin
-DIGNITE_JETTY_PORT=8574
-Dtomcat.major.version=7
-Dwebadmin.path=/nfs/public/rw/webadmin
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Xms2g
-Xmx6g
-XX:MaxPermSize=128m
-Doracle.net.tns_admin=
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8107
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-DIGNITE_QUIET=false
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+UseTLAB
-XX:NewSize=128m
-XX:MaxNewSize=128m
-XX:MaxTenuringThreshold=0
-XX:SurvivorRatio=1024
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=60
-XX:+DisableExplicitGC
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps
-verbose:gc
-XX:+PrintGCDetails
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=100M
-Xloggc:/nfs/public/rw/webadmin/tomcat/bases/data/ves-hx-40/gc/gc_events.log
-Djava.endorsed.dirs=/nfs/public/rw/webadmin/tomcat/homes/apache-tomcat-7.0.55/endorsed
-Dcatalina.base=/nfs/public/rw/webadmin/tomcat/bases
-Dcatalina.home=/nfs/public/rw/webadmin/tomcat/homes/apache-tomcat-7.0.55
-Djava.io.tmpdir=/nfs/public/rw/webadmin/tomcat/bases/temp
(6) Application logs on ves-hx-40
Ever since the web application has not be usable. Whenever I try to open a
web page, I get the following error message in my application log file:
2016-08-11 09:23:52,129 ERROR root/error 495 - Failed to update web session:
null
java.lang.IllegalStateException: Grid is in invalid state to perform this
operation. It either not started yet or has already being or have stopped
[gridName=null, state=STOPPED]
at
org.apache.ignite.internal.GridKernalGatewayImpl.illegalState(GridKernalGatewayImpl.java:190)
at
org.apache.ignite.internal.GridKernalGatewayImpl.readLock(GridKernalGatewayImpl.java:90)
at org.apache.ignite.internal.IgniteKernal.guard(IgniteKernal.java:3107)
at org.apache.ignite.internal.IgniteKernal.cache(IgniteKernal.java:2436)
at
org.apache.ignite.cache.websession.WebSessionFilter.initCache(WebSessionFilter.java:333)
at
org.apache.ignite.cache.websession.WebSessionFilter.handleCacheOperationException(WebSessionFilter.java:877)
at
org.apache.ignite.cache.websession.WebSessionFilter.handleLoadSessionException(WebSessionFilter.java:597)
at
org.apache.ignite.cache.websession.WebSessionFilter.doFilterV2(WebSessionFilter.java:523)
at
org.apache.ignite.cache.websession.WebSessionFilter.doFilterDispatch(WebSessionFilter.java:407)
at
org.apache.ignite.cache.websession.WebSessionFilter.doFilter(WebSessionFilter.java:383)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Thanks for your kind help!
Yuci
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6967.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by Vladislav Pyatkov <vl...@gmail.com>.
Hi
Unfortunately, this charts do not reflect full picture of GC works. I need
to understand how long continueing pause of garbage collect accurate to the
milliseconds.
Could you please look at the article[1] and provide log file of GC (which
will be by path loggc:/path/to/gc/logs/log.txt)?
[1]:
https://apacheignite.readme.io/docs/jvm-and-system-tuning#section-detailed-garbage-collection-stats
On Fri, Aug 5, 2016 at 5:59 PM, yucigou <yu...@gmail.com> wrote:
> Here is what looked like for the other node:
>
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6804/usedMemory.png>
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6804/cpu.png>
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6804/gc.png>
>
> By the way, there were no HTTP traffic to the clustered servers.
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Local-node-seems-to-be-disconnected-
> from-topology-failure-detection-timeout-is-reached-tp6797p6804.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Vladislav Pyatkov
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
Here is what looked like for the other node:
<http://apache-ignite-users.70518.x6.nabble.com/file/n6804/usedMemory.png>
<http://apache-ignite-users.70518.x6.nabble.com/file/n6804/cpu.png>
<http://apache-ignite-users.70518.x6.nabble.com/file/n6804/gc.png>
By the way, there were no HTTP traffic to the clustered servers.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6804.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by Vladislav Pyatkov <vl...@gmail.com>.
If in topology contains only two nodes, need to view logs from both.
A pause can occur as a single node and the second. Please collect log from
both nodes and attach files to the topic.
[1]
https://apacheignite.readme.io/docs/jvm-and-system-tuning#section-detailed-garbage-collection-stats
On Fri, Aug 5, 2016 at 4:58 PM, yucigou <yu...@gmail.com> wrote:
> Thanks for your reply. Indeed the CPU and memory was a bit intense during
> the
> period of time when I ran the Apache Ignite Web Console Agent, so as to use
> the Apache Ignite Web Console to monitoring my Ignite nodes. But it seemed
> under control (notes: figures below are provided by JavaMelody):
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6802/usedMemory.png>
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6802/cpu.png>
> Also the garbage collector time was minimal:
> <http://apache-ignite-users.70518.x6.nabble.com/file/n6802/gc.png>
> Could there be something wrong caused by the Apache Ignite Web Console
> Agent? Or any other possibilities?
>
> Cheers,
> Yuci
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Local-node-seems-to-be-disconnected-
> from-topology-failure-detection-timeout-is-reached-tp6797p6802.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Vladislav Pyatkov
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
Thanks for your reply. Indeed the CPU and memory was a bit intense during the
period of time when I ran the Apache Ignite Web Console Agent, so as to use
the Apache Ignite Web Console to monitoring my Ignite nodes. But it seemed
under control (notes: figures below are provided by JavaMelody):
<http://apache-ignite-users.70518.x6.nabble.com/file/n6802/usedMemory.png>
<http://apache-ignite-users.70518.x6.nabble.com/file/n6802/cpu.png>
Also the garbage collector time was minimal:
<http://apache-ignite-users.70518.x6.nabble.com/file/n6802/gc.png>
Could there be something wrong caused by the Apache Ignite Web Console
Agent? Or any other possibilities?
Cheers,
Yuci
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6802.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by yucigou <yu...@gmail.com>.
Thanks for your reply. Indeed the CPU and memory was a bit intense during the
period of time when I ran the Apache Ignite Web Console Agent, so as to use
the Apache Ignite Web Console to monitoring my Ignite nodes. But it seemed
under control (notes: figures below are provided by JavaMelody):
<http://apache-ignite-users.70518.x6.nabble.com/file/n6801/cpu.png>
<http://apache-ignite-users.70518.x6.nabble.com/file/n6801/cpu.png>
Also the garbage collector time was minimal.
<http://apache-ignite-users.70518.x6.nabble.com/file/n6801/gc.png>
Could there be something wrong caused by the Apache Ignite Web Console
Agent? Or any other possibilities?
Cheers,
Yuci
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-node-seems-to-be-disconnected-from-topology-failure-detection-timeout-is-reached-tp6797p6801.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by Vladislav Pyatkov <vp...@gridgain.com>.
It look like as segmentation by the reasong pause of garbage collector.
You need attentively look the article [1] and collect GC logs, if GC works
with long pauses (around 10 seconds) try to tune JVM.
If the issue persists, please provide GC log and Ignite log for analyze.
[1] https://apacheignite.readme.io/docs/jvm-and-system-tuning
On Fri, Aug 5, 2016 at 2:57 PM, yucigou <yu...@gmail.com> wrote:
> Hello,
>
> One of my Ignite nodes was stopped and the logs were appended as below. It
> seems that grid-timeout-worker checks the health of the cluster every
> minute. But then in my case, before the due time 23:34:19, at 23:34:03 it
> reported "Local node seems to be disconnected from topology (failure
> detection timeout is reached)", and the Ignite node got stopped. In turn,
> the web session clustering, and so on, stopped working.
>
> Just wonder what could cause this to happen? There should be no network
> issue etc with the host machine then. It is a bit scary to us, as it can
> happen to our production servers in the near future.
>
> Thank you for your help.
>
> Yuci
>
> ===================Ignite logs======================
> [23:31:19,896][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=9a069f70, name=null, uptime=10:37:03:793]
> ^-- H/N/C [hosts=2, nodes=2, CPUs=4]
> ^-- CPU [cur=43.17%, avg=12.83%, GC=1.1%]
> ^-- Heap [used=2115MB, free=61.26%, comm=3955MB]
> ^-- Non heap [used=138MB, free=-1%, comm=143MB]
> ^-- Public thread pool [active=0, idle=16, qSize=0]
> ^-- System thread pool [active=0, idle=16, qSize=0]
> ^-- Outbound messages queue [size=0]
> [23:32:19,904][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=9a069f70, name=null, uptime=10:38:03:801]
> ^-- H/N/C [hosts=2, nodes=2, CPUs=4]
> ^-- CPU [cur=0.83%, avg=12.87%, GC=0%]
> ^-- Heap [used=2638MB, free=51.69%, comm=3957MB]
> ^-- Non heap [used=138MB, free=-1%, comm=143MB]
> ^-- Public thread pool [active=0, idle=16, qSize=0]
> ^-- System thread pool [active=0, idle=16, qSize=0]
> ^-- Outbound messages queue [size=0]
> [23:33:19,913][INFO ][grid-timeout-worker-#33%null%][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=9a069f70, name=null, uptime=10:39:03:808]
> ^-- H/N/C [hosts=2, nodes=2, CPUs=4]
> ^-- CPU [cur=0.5%, avg=12.86%, GC=0%]
> ^-- Heap [used=796MB, free=85.41%, comm=3921MB]
> ^-- Non heap [used=138MB, free=-1%, comm=143MB]
> ^-- Public thread pool [active=0, idle=16, qSize=0]
> ^-- System thread pool [active=0, idle=16, qSize=0]
> ^-- Outbound messages queue [size=0]
> [23:34:03,752][INFO ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi]
> Local
> node seems to be disconnected from topology (failure detection timeout is
> reached) [failureDetectionTimeout=10000, connCheckFreq=3333]
> [23:34:03,783][WARN ][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi] Node
> is out of topology (probably, due to short-time network problems).
> [23:34:03,786][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
> Local node SEGMENTED: TcpDiscoveryNode
> [id=9a069f70-d49d-472e-9771-7ac2353e751f, addrs=[10.3.0.64, 127.0.0.1],
> sockAddrs=[ves-hx-40.ebi.ac.uk/10.3.0.64:47500, /10.3.0.64:47500,
> /127.0.0.1:47500], discPort=47500, order=56, intOrder=29,
> lastExchangeTime=1470350043783, loc=true, ver=1.6.0#20160518-sha1:
> 0b22c45b,
> isClient=false]
> [23:34:03,819][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
> Stopping local node according to configured segmentation policy.
> [23:34:03,825][WARN ][disco-event-worker-#44%null%][GridDiscoveryManager]
> Node FAILED: TcpDiscoveryNode [id=cef7fc5e-b854-4072-8e16-396a87d5d556,
> addrs=[10.3.0.65, 127.0.0.1],
> sockAddrs=[ves-hx-41.ebi.ac.uk/10.3.0.65:47500, /10.3.0.65:47500,
> /127.0.0.1:47500], discPort=47500, order=58, intOrder=30,
> lastExchangeTime=1470311808664, loc=false, ver=1.6.0#20160518-sha1:
> 0b22c45b,
> isClient=false]
> [23:34:03,827][INFO ][disco-event-worker-#44%null%][GridDiscoveryManager]
> Topology snapshot [ver=59, servers=1, clients=0, CPUs=2, heap=5.3GB]
> [23:34:03,874][INFO ][Thread-32][GridTcpRestProtocol] Command protocol
> successfully stopped: TCP binary
> [23:34:03,902][INFO ][Thread-32][GridJettyRestProtocol] Command protocol
> successfully stopped: Jetty REST
> [23:34:04,571][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
> session-cache
> [23:34:04,572][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
> ignite-marshaller-sys-cache
> [23:34:04,572][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
> ignite-sys-cache
> [23:34:04,573][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
> ignite-atomics-sys-cache
> [23:34:04,583][INFO ][Thread-32][GridCacheProcessor] Stopped cache:
> wicket-data-store
> [23:34:04,623][INFO ][Thread-32][IgniteKernal]
>
> >>> +-----------------------------------------------------------
> ----------------------+
> >>> Ignite ver. 1.6.0#20160518-sha1:0b22c45bb9b97692208fd0705ddf80
> 45ff34a031
> >>> stopped OK
> >>> +-----------------------------------------------------------
> ----------------------+
> >>> Grid uptime: 10:39:48:518
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Local-node-seems-to-be-disconnected-
> from-topology-failure-detection-timeout-is-reached-tp6797.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Local node seems to be disconnected from topology (failure
detection timeout is reached)
Posted by luqmanahmad <lu...@gmail.com>.
See [1] for free network segmentation plugin
[1] https://github.com/luqmanahmad/ignite-plugins
<https://github.com/luqmanahmad/ignite-plugins>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/