You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by kvenkatramtreddy <kv...@gmail.com> on 2018/06/30 12:24:51 UTC
Node stopped automatically
Hi,
Node stopped automatically after 48 hours uptime. Please find the logs
below.
I got 2 hosts and 3 Ignite nodes
Host 1, Node 1
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=5582ed60, uptime=48:00:04.981]
^-- H/N/C [hosts=2, nodes=3, CPUs=4]
^-- CPU [cur=15.67%, avg=15.01%, GC=0.43%]
^-- PageMemory [pages=390178]
^-- Heap [used=1387MB, free=11.75%, comm=1572MB]
^-- Non heap [used=369MB, free=-1%, comm=409MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
[6/27/18 8:28:43:098 UTC] 000000c5
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I TCP discovery
accepted incoming connection [rmtAddr=/57.20.44.84, rmtPort=47056]
[6/27/18 8:28:43:098 UTC] 000000c5
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I TCP discovery
spawning a new thread for connection [rmtAddr=/57.20.44.84, rmtPort=4
7056]
[6/27/18 8:28:43:117 UTC] 000d83ee
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I Started
serving remote node connection [rmtAddr=/57.20.44.84:47056, rmtPort=47056]
[6/27/18 8:28:43:128 UTC] 000000c4
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi W Node is out
of topology (probably, due to short-time network problems).
[6/27/18 8:28:43:129 UTC] 000000cb
ache.ignite.internal.managers.discovery.GridDiscoveryManager W Local node
SEGMENTED: TcpDiscoveryNode [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, ad
drs=[57.20.44.85], sockAddrs=[myhost1.org/57.20.44.85:48500],
discPort=48500, order=2, intOrder=2, lastExchangeTime=1530088123126,
loc=true, ver=2.4.0#201803
05-sha1:aa342270, isClient=false]
[6/27/18 8:28:43:135 UTC] 000000c5
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I TCP discovery
accepted incoming connection [rmtAddr=/57.20.44.84, rmtPort=58172]
[6/27/18 8:28:43:137 UTC] 000d83ee
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I Finished
serving remote node connection [rmtAddr=/57.20.44.84:47056, rmtPort=47056
[6/27/18 8:28:43:140 UTC] 000000c5
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I TCP discovery
spawning a new thread for connection [rmtAddr=/57.20.44.84, rmtPort=5
8172]
[6/27/18 8:28:43:140 UTC] 000d83f0
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I Started
serving remote node connection [rmtAddr=/57.20.44.84:58172, rmtPort=58172]
[6/27/18 8:28:43:162 UTC] 000d83f0
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I Finished
serving remote node connection [rmtAddr=/57.20.44.84:58172, rmtPort=58172
[6/27/18 8:28:43:184 UTC] 000000cb
ache.ignite.internal.managers.discovery.GridDiscoveryManager W Stopping
local node according to configured segmentation policy.
Host 2(have 2 ignite nodes) Node1
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=58a3d911, uptime=48:00:05.572]
^-- H/N/C [hosts=2, nodes=3, CPUs=4]
^-- CPU [cur=8.17%, avg=8.27%, GC=0.5%]
^-- PageMemory [pages=70]
^-- Heap [used=1306MB, free=16.92%, comm=1572MB]
^-- Non heap [used=355MB, free=-1%, comm=392MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
[6/27/18 8:28:34:381 UTC] 00000095
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi W Timed out
waiting for message delivery receipt (most probably, the reason is in lon
g GC pauses on remote node; consider tuning GC and increasing 'ackTimeout'
configuration property). Will retry to send message with increased timeout
[currentTimeout=10000, rmtAddr=
myhost2.org/57.20.44.85:48500, rmtPort=48500]
[6/27/18 8:28:34:382 UTC] 00000095
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi W Failed to
send message to next node [msg=TcpDiscoveryMetricsUpdateMessage [super=Tc
pDiscoveryAbstractMessage [sndNodeId=null,
id=27b0fa80461-58a3d911-aa4c-453d-ae5a-0fc3d450a09f,
verifierNodeId=58a3d911-aa4c-453d-ae5a-0fc3d450a09f, topVer=0, pendingIdx=0,
failedNo
des=null, isClient=false]], next=TcpDiscoveryNode
[id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, addrs=[57.20.44.85],
sockAddrs=[myhost2.org/57.20.44.85:48500], d
iscPort=48500, order=2, intOrder=2, lastExchangeTime=1529051134737,
loc=false, ver=2.4.0#20180305-sha1:aa342270, isClient=false], errMsg=Failed
to send message to next node [msg=Tcp
DiscoveryMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=27b0fa80461-58a3d911-aa4c-453d-ae5a-0fc3d450a09f,
verifierNodeId=58a3d911-aa4c-453d-ae5a-0fc3d45
0a09f, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]],
next=ClusterNode [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, order=2,
addr=[57.20.44.85], daemon=false]]]
[6/27/18 8:28:34:480 UTC] 00000095
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi W Local node
has detected failed nodes and started cluster-wide procedure. To speed u
p failure detection please see 'Failure Detection' section under javadoc for
'TcpDiscoverySpi'
[6/27/18 8:28:34:492 UTC] 00000097
ache.ignite.internal.managers.discovery.GridDiscoveryManager W Node FAILED:
TcpDiscoveryNode [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, addrs=[57.2
0.44.85], sockAddrs=[myhost2.org/57.20.44.85:48500], discPort=48500,
order=2, intOrder=2, lastExchangeTime=1529051134737, loc=false,
ver=2.4.0#20180305-sha1:
aa342270, isClient=false]
[6/27/18 8:28:34:492 UTC] 00000097 SystemOut
O [08:28:34] Topology snapshot [ver=20, servers=2, clients=0, CPUs=2,
offheap=1.8GB,
heap=3.5GB]
[6/27/18 8:28:34:492 UTC] 00000097 SystemOut
O [08:28:34] Data Regions Configured:
[6/27/18 8:28:34:492 UTC] 00000097 SystemOut
O [08:28:34] ^-- default [initSize=256.0 MiB, maxSize=1.5 GiB,
persistenceEnabled=t
rue]
[6/27/18 8:28:34:492 UTC] 00000097
ache.ignite.internal.managers.discovery.GridDiscoveryManager I Topology
snapshot [ver=20, servers=2, clients=0, CPUs=2, offheap=1.8GB, heap=3.5GB]
[6/27/18 8:28:34:493 UTC] 00000097
ache.ignite.internal.managers.discovery.GridDiscoveryManager I Data Regions
Configured:
[6/27/18 8:28:34:493 UTC] 00000097
ache.ignite.internal.managers.discovery.GridDiscoveryManager I ^-- default
[initSize=256.0 MiB, maxSize=1.5 GiB, persistenceEnabled=true]
[6/27/18 8:28:34:611 UTC] 0000009a org.apache.ignite.internal.exchange.time
I Started exchange init [topVer=AffinityTopologyVersion [topVer=20,
minorTopVer=0], c
rd=true, evt=NODE_FAILED, evtNode=5582ed60-76f5-42b1-bd19-ed3af12b12f7,
customEvt=null, allowMerge=true]
[6/27/18 8:28:34:612 UTC] 0000009a
he.distributed.dht.preloader.GridDhtPartitionsExchangeFuture I Finished
waiting for partition release future [topVer=AffinityTopologyVersion [topV
er=20, minorTopVer=0], waitTime=0ms, futInfo=NA]
[6/27/18 8:28:34:761 UTC] 0000009a org.apache.ignite.internal.exchange.time
I Finished exchange init [topVer=AffinityTopologyVersion [topVer=20,
minorTopVer=0],
crd=true]
[6/27/18 8:28:34:781 UTC] 000bff06
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi I Accepted
incoming communication connection [locAddr=null, rmtAddr=null]
[6/27/18 8:28:34:783 UTC] 000bff07
he.distributed.dht.preloader.GridDhtPartitionsExchangeFuture I Coordinator
received single message [ver=AffinityTopologyVersion [topVer=20, minorT
opVer=0], node=eafdbb30-ac44-4335-85eb-ca155dd7c61d, allReceived=true]
[6/27/18 8:28:34:783 UTC] 000bff07
he.distributed.dht.preloader.GridDhtPartitionsExchangeFuture I Coordinator
received all messages, try merge [ver=AffinityTopologyVersion [topVer=2
0, minorTopVer=0]]
[6/27/18 8:28:34:783 UTC] 000bff07
.internal.processors.cache.GridCachePartitionExchangeManager I Stop merge,
custom task found: WalStateNodeLeaveExchangeTask [node=TcpDiscoveryNode
[id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, addrs=[57.20.44.85],
sockAddrs=[myhost1.org/57.20.44.85:48500], discPort=48500, order=2,
intOrder=2, lastExchangeT
ime=1529051134737, loc=false, ver=2.4.0#20180305-sha1:aa342270,
isClient=false]]
[6/27/18 8:28:34:784 UTC] 000bff07
he.distributed.dht.preloader.GridDhtPartitionsExchangeFuture I
finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=20,
minorTopVer
=0], resVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]]
[6/27/18 8:28:34:842 UTC] 000bfefd
com.mbi.mft.travel.mat.SecuritySearchScheduler E Exception in security
search scheduler
class org.apache.ignite.IgniteException: Failed to map keys for cache (all
partition nodes left the grid) [topVer=AffinityTopologyVersion [topVer=20,
minorTopVer=0], cache=security-
search]
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.containsKey(GridCacheAdapter.java:633)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.containsKey(IgniteCacheProxyImpl.java:972)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.containsKey(GatewayProtectedCacheProxy.java:816)
at
com.mbi.mft.travel.mat.SecuritySearchScheduler.lambda$run$1(SecuritySearchScheduler.java:59)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at
java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at
java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at
java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
at
java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at
com.mbi.mft.travel.mat.SecuritySearchScheduler.run(SecuritySearchScheduler.java:55)
at
com.ibm.ws.concurrent.internal.ScheduledTask.call(ScheduledTask.java:383)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: class
org.apache.ignite.internal.cluster.ClusterTopologyServerNotFoundException:
Failed to map keys for cache (all partition nodes left the grid)
[topVer=AffinityTopology
Version [topVer=20, minorTopVer=0], cache=security-search]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.serverNotFoundError(GridPartitionedSingleGetFuture.java:711)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.mapKeyToNode(GridPartitionedSingleGetFuture.java:332)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:216)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.access$300(GridPartitionedSingleGetFuture.java:67)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture$4.run(GridPartitionedSingleGetFuture.java:771)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6679)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
... 3 more
[6/27/18 8:28:34:871 UTC] 000bfefe SystemOut
O 2018-06-27 08:28:34 ERROR EventManager:62 - - - - Error in Event update
org.apache.ignite.cache.CacheServerNotFoundException: Failed to map keys for
cache (all partition nodes left the grid) [topVer=AffinityTopologyVersion
[topVer=20, minorTopVer=0], ca
che=flightusers]
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1282)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1673)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:852)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:676)
at
com.mbi.mft.travel.mat.polling.EventManager.sendMessageUpdate(EventManager.java:32)
at com.mbi.CacheResource.storingDataInCache(CacheResource.java:448)
at com.mbi.CacheResource.nodeElementList(CacheResource.java:276)
at com.mbi.CacheResource.helloUserQuery(CacheResource.java:212)
at sun.reflect.GeneratedMethodAccessor366.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:181)
at
org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:97)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:200)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:251)
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:171)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:293)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:212)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:268)
at
com.ibm.mfp.server.java.adapter.shared.JAXRSSandbox$3.doFilter(JAXRSSandbox.java:579)
at
com.ibm.mfp.server.java.adapter.shared.FilterChainImpl.doFilter(FilterChainImpl.java:86)
at
com.ibm.mfp.server.java.adapter.shared.JAXRSSandbox.handleRequest(JAXRSSandbox.java:584)
at
com.ibm.mfp.server.java.adapter.internal.rest.AdaptersEndpoint.adapterServing(AdaptersEndpoint.java:115)
at sun.reflect.GeneratedMethodAccessor357.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)
at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1290)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:778)
at
com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
at
com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1161)
at
com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:82)
at
com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:956)
at
com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:280)
at
com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:967)
at
com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:359)
at
com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:318)
at
com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:471)
at
com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:405)
at
com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:285)
at
com.ibm.ws.http.channel.internal.inbound.HttpICLReadCallback.complete(HttpICLReadCallback.java:66)
at
com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)
at
com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)
at
com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:929)
at
com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1018)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: class
org.apache.ignite.internal.cluster.ClusterTopologyServerNotFoundException:
Failed to map keys for cache (all partition nodes left the grid)
[topVer=AffinityTopology
Version [topVer=20, minorTopVer=0], cache=flightusers]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.serverNotFoundError(GridPartitionedSingleGetFuture.java:711)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.mapKeyToNode(GridPartitionedSingleGetFuture.java:332)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:216)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.access$300(GridPartitionedSingleGetFuture.java:67)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture$4.run(GridPartitionedSingleGetFuture.java:771)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6679)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
... 3 more
[6/27/18 8:28:34:871 UTC] 000bfefe com.mbi.CacheResource
E XMl Processed
[6/27/18 8:28:34:919 UTC] 000bff07
he.distributed.dht.preloader.GridDhtPartitionsExchangeFuture I Finish
exchange future [startVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]
, resVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], err=null]
[6/27/18 8:28:34:972 UTC] 0000009a
.internal.processors.cache.GridCachePartitionExchangeManager I Skipping
rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=20, m
inorTopVer=0], evt=NODE_FAILED, node=5582ed60-76f5-42b1-bd19-ed3af12b12f7]
Host 2 Node 2
[6/27/18 8:28:34:517 UTC] 00000082
ache.ignite.internal.managers.discovery.GridDiscoveryManager W Node FAILED:
TcpDiscoveryNode [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7,
addrs=[57.20.44.85], sockAddrs=[myhost1.org/57.20.44.85:48500],
discPort=48500, order=2, intOrder=2, lastExchangeTime=1529923709805,
loc=false, ver=2.4.0#20180305-sha1:aa342270, isClient=false]
[6/27/18 8:28:34:518 UTC] 00000082 SystemOut
O [08:28:34] Topology snapshot [ver=20, servers=2, clients=0, CPUs=2,
offheap=1.8GB, heap=3.5GB]
[6/27/18 8:28:34:519 UTC] 00000082 SystemOut
O [08:28:34] Data Regions Configured:
[6/27/18 8:28:34:519 UTC] 00000082 SystemOut
O [08:28:34] ^-- default [initSize=256.0 MiB, maxSize=256.0 MiB,
persistenceEnabled=true]
[6/27/18 8:28:34:946 UTC] 0000a166
st.travel.tam.allegro.processes.TurnAroundProcessAdapterImpl E Failed to map
keys for cache (all partition nodes left the grid)
[topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0],
cache=turnAroundProcesses]
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Node stopped automatically
Posted by Denis Mekhanikov <dm...@gmail.com>.
Venkat,
Please don't paste your logs into the body of your message, it makes them
look unreadable.
Use file attachment or provide a link instead.
Node on host 1 got segmented and killed according to the configured
segmentation policy.
It may happen due to a network problem or long GC pause.
Looks like you have persistence enabled and a single node in baseline
topology.
So, all data is actually stored on a single data node, and when it leaves
the grid, all partitions are marked as lost.
Refer to the documentation:
https://apacheignite.readme.io/docs/baseline-topology
You should add all nodes, that you want to store the data on, to the
baseline.
And when a node leaves topology forever, then you should drop it from the
baseline.
Denis
пн, 2 июл. 2018 г. в 12:15, kvenkatramtreddy <kv...@gmail.com>:
> Please could you provide some pointers, so that I can look deeper into the
> issue.
>
> 1) Does PageMemory cause any issues
>
> as per the metrics, one node is contains only 70 and other node is
> PageMemory [pages=390178]
>
> 2) Why whole cluster is going down, when one node is down
>
> 3) Can I restart the ignite programatically if this error occurs.
>
> Thanks & Regards,
> Venkat
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Node stopped automatically
Posted by kvenkatramtreddy <kv...@gmail.com>.
Please could you provide some pointers, so that I can look deeper into the
issue.
1) Does PageMemory cause any issues
as per the metrics, one node is contains only 70 and other node is
PageMemory [pages=390178]
2) Why whole cluster is going down, when one node is down
3) Can I restart the ignite programatically if this error occurs.
Thanks & Regards,
Venkat
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Node stopped automatically
Posted by kvenkatramtreddy <kv...@gmail.com>.
Ignite related only that log,
After that only cache stopped exception in the log.
Even we have enabled -DIGNITE_QUIET=false, but it is printing that much
only.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Node stopped automatically
Posted by Dmitriy Govorukhin <dm...@gmail.com>.
Hi, kvenkatramtreddy!
Could you please provide a full log, not only stack trace with an exception?
On Sat, Jun 30, 2018 at 3:24 PM kvenkatramtreddy <kv...@gmail.com>
wrote:
> Hi,
>
> Node stopped automatically after 48 hours uptime. Please find the logs
> below.
>
> I got 2 hosts and 3 Ignite nodes
>
>
> Host 1, Node 1
>
>
>
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=5582ed60, uptime=48:00:04.981]
> ^-- H/N/C [hosts=2, nodes=3, CPUs=4]
> ^-- CPU [cur=15.67%, avg=15.01%, GC=0.43%]
> ^-- PageMemory [pages=390178]
> ^-- Heap [used=1387MB, free=11.75%, comm=1572MB]
> ^-- Non heap [used=369MB, free=-1%, comm=409MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=6, qSize=0]
>
>
>
> [6/27/18 8:28:43:098 UTC] 000000c5
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I TCP
> discovery
> accepted incoming connection [rmtAddr=/57.20.44.84, rmtPort=47056]
> [6/27/18 8:28:43:098 UTC] 000000c5
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I TCP
> discovery
> spawning a new thread for connection [rmtAddr=/57.20.44.84, rmtPort=4
> 7056]
>
> [6/27/18 8:28:43:117 UTC] 000d83ee
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I Started
> serving remote node connection [rmtAddr=/57.20.44.84:47056, rmtPort=47056]
> [6/27/18 8:28:43:128 UTC] 000000c4
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi W Node is out
> of topology (probably, due to short-time network problems).
> [6/27/18 8:28:43:129 UTC] 000000cb
> ache.ignite.internal.managers.discovery.GridDiscoveryManager W Local node
> SEGMENTED: TcpDiscoveryNode [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, ad
> drs=[57.20.44.85], sockAddrs=[myhost1.org/57.20.44.85:48500],
> discPort=48500, order=2, intOrder=2, lastExchangeTime=1530088123126,
> loc=true, ver=2.4.0#201803
> 05-sha1:aa342270, isClient=false]
> [6/27/18 8:28:43:135 UTC] 000000c5
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I TCP
> discovery
> accepted incoming connection [rmtAddr=/57.20.44.84, rmtPort=58172]
> [6/27/18 8:28:43:137 UTC] 000d83ee
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I Finished
> serving remote node connection [rmtAddr=/57.20.44.84:47056, rmtPort=47056
> [6/27/18 8:28:43:140 UTC] 000000c5
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I TCP
> discovery
> spawning a new thread for connection [rmtAddr=/57.20.44.84, rmtPort=5
> 8172]
> [6/27/18 8:28:43:140 UTC] 000d83f0
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I Started
> serving remote node connection [rmtAddr=/57.20.44.84:58172, rmtPort=58172]
> [6/27/18 8:28:43:162 UTC] 000d83f0
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi I Finished
> serving remote node connection [rmtAddr=/57.20.44.84:58172, rmtPort=58172
> [6/27/18 8:28:43:184 UTC] 000000cb
> ache.ignite.internal.managers.discovery.GridDiscoveryManager W Stopping
> local node according to configured segmentation policy.
>
>
>
>
> Host 2(have 2 ignite nodes) Node1
>
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=58a3d911, uptime=48:00:05.572]
> ^-- H/N/C [hosts=2, nodes=3, CPUs=4]
> ^-- CPU [cur=8.17%, avg=8.27%, GC=0.5%]
> ^-- PageMemory [pages=70]
> ^-- Heap [used=1306MB, free=16.92%, comm=1572MB]
> ^-- Non heap [used=355MB, free=-1%, comm=392MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=6, qSize=0]
>
>
>
> [6/27/18 8:28:34:381 UTC] 00000095
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi W Timed out
> waiting for message delivery receipt (most probably, the reason is in lon
> g GC pauses on remote node; consider tuning GC and increasing 'ackTimeout'
> configuration property). Will retry to send message with increased timeout
> [currentTimeout=10000, rmtAddr=
> myhost2.org/57.20.44.85:48500, rmtPort=48500]
> [6/27/18 8:28:34:382 UTC] 00000095
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi W Failed to
> send message to next node [msg=TcpDiscoveryMetricsUpdateMessage [super=Tc
> pDiscoveryAbstractMessage [sndNodeId=null,
> id=27b0fa80461-58a3d911-aa4c-453d-ae5a-0fc3d450a09f,
> verifierNodeId=58a3d911-aa4c-453d-ae5a-0fc3d450a09f, topVer=0,
> pendingIdx=0,
> failedNo
> des=null, isClient=false]], next=TcpDiscoveryNode
> [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, addrs=[57.20.44.85],
> sockAddrs=[myhost2.org/57.20.44.85:48500], d
> iscPort=48500, order=2, intOrder=2, lastExchangeTime=1529051134737,
> loc=false, ver=2.4.0#20180305-sha1:aa342270, isClient=false], errMsg=Failed
> to send message to next node [msg=Tcp
> DiscoveryMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
> [sndNodeId=null, id=27b0fa80461-58a3d911-aa4c-453d-ae5a-0fc3d450a09f,
> verifierNodeId=58a3d911-aa4c-453d-ae5a-0fc3d45
> 0a09f, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]],
> next=ClusterNode [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, order=2,
> addr=[57.20.44.85], daemon=false]]]
> [6/27/18 8:28:34:480 UTC] 00000095
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi W Local node
> has detected failed nodes and started cluster-wide procedure. To speed u
> p failure detection please see 'Failure Detection' section under javadoc
> for
> 'TcpDiscoverySpi'
> [6/27/18 8:28:34:492 UTC] 00000097
> ache.ignite.internal.managers.discovery.GridDiscoveryManager W Node FAILED:
> TcpDiscoveryNode [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, addrs=[57.2
> 0.44.85], sockAddrs=[myhost2.org/57.20.44.85:48500], discPort=48500,
> order=2, intOrder=2, lastExchangeTime=1529051134737, loc=false,
> ver=2.4.0#20180305-sha1:
> aa342270, isClient=false]
> [6/27/18 8:28:34:492 UTC] 00000097 SystemOut
>
> O [08:28:34] Topology snapshot [ver=20, servers=2, clients=0, CPUs=2,
> offheap=1.8GB,
> heap=3.5GB]
> [6/27/18 8:28:34:492 UTC] 00000097 SystemOut
>
> O [08:28:34] Data Regions Configured:
> [6/27/18 8:28:34:492 UTC] 00000097 SystemOut
>
> O [08:28:34] ^-- default [initSize=256.0 MiB, maxSize=1.5 GiB,
> persistenceEnabled=t
> rue]
> [6/27/18 8:28:34:492 UTC] 00000097
> ache.ignite.internal.managers.discovery.GridDiscoveryManager I Topology
> snapshot [ver=20, servers=2, clients=0, CPUs=2, offheap=1.8GB, heap=3.5GB]
> [6/27/18 8:28:34:493 UTC] 00000097
> ache.ignite.internal.managers.discovery.GridDiscoveryManager I Data Regions
> Configured:
> [6/27/18 8:28:34:493 UTC] 00000097
> ache.ignite.internal.managers.discovery.GridDiscoveryManager I ^--
> default
> [initSize=256.0 MiB, maxSize=1.5 GiB, persistenceEnabled=true]
> [6/27/18 8:28:34:611 UTC] 0000009a
> org.apache.ignite.internal.exchange.time
> I Started exchange init [topVer=AffinityTopologyVersion [topVer=20,
> minorTopVer=0], c
> rd=true, evt=NODE_FAILED, evtNode=5582ed60-76f5-42b1-bd19-ed3af12b12f7,
> customEvt=null, allowMerge=true]
> [6/27/18 8:28:34:612 UTC] 0000009a
> he.distributed.dht.preloader.GridDhtPartitionsExchangeFuture I Finished
> waiting for partition release future [topVer=AffinityTopologyVersion [topV
> er=20, minorTopVer=0], waitTime=0ms, futInfo=NA]
> [6/27/18 8:28:34:761 UTC] 0000009a
> org.apache.ignite.internal.exchange.time
> I Finished exchange init [topVer=AffinityTopologyVersion [topVer=20,
> minorTopVer=0],
> crd=true]
> [6/27/18 8:28:34:781 UTC] 000bff06
> org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi I Accepted
> incoming communication connection [locAddr=null, rmtAddr=null]
> [6/27/18 8:28:34:783 UTC] 000bff07
> he.distributed.dht.preloader.GridDhtPartitionsExchangeFuture I Coordinator
> received single message [ver=AffinityTopologyVersion [topVer=20, minorT
> opVer=0], node=eafdbb30-ac44-4335-85eb-ca155dd7c61d, allReceived=true]
> [6/27/18 8:28:34:783 UTC] 000bff07
> he.distributed.dht.preloader.GridDhtPartitionsExchangeFuture I Coordinator
> received all messages, try merge [ver=AffinityTopologyVersion [topVer=2
> 0, minorTopVer=0]]
> [6/27/18 8:28:34:783 UTC] 000bff07
> .internal.processors.cache.GridCachePartitionExchangeManager I Stop merge,
> custom task found: WalStateNodeLeaveExchangeTask [node=TcpDiscoveryNode
> [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, addrs=[57.20.44.85],
> sockAddrs=[myhost1.org/57.20.44.85:48500], discPort=48500, order=2,
> intOrder=2, lastExchangeT
> ime=1529051134737, loc=false, ver=2.4.0#20180305-sha1:aa342270,
> isClient=false]]
> [6/27/18 8:28:34:784 UTC] 000bff07
> he.distributed.dht.preloader.GridDhtPartitionsExchangeFuture I
> finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=20,
> minorTopVer
> =0], resVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]]
> [6/27/18 8:28:34:842 UTC] 000bfefd
> com.mbi.mft.travel.mat.SecuritySearchScheduler E Exception in
> security
> search scheduler
> class org.apache.ignite.IgniteException: Failed to map keys for cache (all
> partition nodes left the grid) [topVer=AffinityTopologyVersion [topVer=20,
> minorTopVer=0], cache=security-
> search]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.containsKey(GridCacheAdapter.java:633)
> at
>
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.containsKey(IgniteCacheProxyImpl.java:972)
> at
>
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.containsKey(GatewayProtectedCacheProxy.java:816)
> at
>
> com.mbi.mft.travel.mat.SecuritySearchScheduler.lambda$run$1(SecuritySearchScheduler.java:59)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at
>
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
> at
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
> at
>
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
>
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
>
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
>
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
>
> com.mbi.mft.travel.mat.SecuritySearchScheduler.run(SecuritySearchScheduler.java:55)
> at
> com.ibm.ws.concurrent.internal.ScheduledTask.call(ScheduledTask.java:383)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class
> org.apache.ignite.internal.cluster.ClusterTopologyServerNotFoundException:
> Failed to map keys for cache (all partition nodes left the grid)
> [topVer=AffinityTopology
> Version [topVer=20, minorTopVer=0], cache=security-search]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.serverNotFoundError(GridPartitionedSingleGetFuture.java:711)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.mapKeyToNode(GridPartitionedSingleGetFuture.java:332)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:216)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.access$300(GridPartitionedSingleGetFuture.java:67)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture$4.run(GridPartitionedSingleGetFuture.java:771)
> at
>
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6679)
> at
>
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> ... 3 more
>
> [6/27/18 8:28:34:871 UTC] 000bfefe SystemOut
>
> O 2018-06-27 08:28:34 ERROR EventManager:62 - - - - Error in Event
> update
> org.apache.ignite.cache.CacheServerNotFoundException: Failed to map keys
> for
> cache (all partition nodes left the grid) [topVer=AffinityTopologyVersion
> [topVer=20, minorTopVer=0], ca
> che=flightusers]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1282)
> at
>
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1673)
> at
>
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:852)
> at
>
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:676)
> at
>
> com.mbi.mft.travel.mat.polling.EventManager.sendMessageUpdate(EventManager.java:32)
> at com.mbi.CacheResource.storingDataInCache(CacheResource.java:448)
> at com.mbi.CacheResource.nodeElementList(CacheResource.java:276)
> at com.mbi.CacheResource.helloUserQuery(CacheResource.java:212)
> at sun.reflect.GeneratedMethodAccessor366.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
> org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:181)
> at
>
> org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:97)
> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:200)
> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)
> at
>
> org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
> at
>
> org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
> at
>
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
> at
>
> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
> at
>
> org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:251)
> at
>
> org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
> at
>
> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
> at
>
> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
> at
>
> org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:171)
> at
>
> org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:293)
> at
>
> org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:212)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
> at
>
> org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:268)
> at
>
> com.ibm.mfp.server.java.adapter.shared.JAXRSSandbox$3.doFilter(JAXRSSandbox.java:579)
> at
>
> com.ibm.mfp.server.java.adapter.shared.FilterChainImpl.doFilter(FilterChainImpl.java:86)
> at
>
> com.ibm.mfp.server.java.adapter.shared.JAXRSSandbox.handleRequest(JAXRSSandbox.java:584)
> at
>
> com.ibm.mfp.server.java.adapter.internal.rest.AdaptersEndpoint.adapterServing(AdaptersEndpoint.java:115)
> at sun.reflect.GeneratedMethodAccessor357.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
> at
>
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
> at
>
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
> at
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776)
> at
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)
> at
>
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
>
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
> at
>
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
> at
>
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
> at
>
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
> at
>
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
> at
>
> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1290)
> at
>
> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:778)
> at
>
> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:475)
> at
>
> com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1161)
> at
>
> com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:82)
> at
> com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:956)
> at
>
> com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:280)
> at
>
> com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:967)
> at
>
> com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:359)
> at
>
> com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:318)
> at
>
> com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:471)
> at
>
> com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:405)
> at
>
> com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:285)
> at
>
> com.ibm.ws.http.channel.internal.inbound.HttpICLReadCallback.complete(HttpICLReadCallback.java:66)
> at
>
> com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)
> at
>
> com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)
> at
>
> com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:929)
> at
>
> com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1018)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class
> org.apache.ignite.internal.cluster.ClusterTopologyServerNotFoundException:
> Failed to map keys for cache (all partition nodes left the grid)
> [topVer=AffinityTopology
> Version [topVer=20, minorTopVer=0], cache=flightusers]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.serverNotFoundError(GridPartitionedSingleGetFuture.java:711)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.mapKeyToNode(GridPartitionedSingleGetFuture.java:332)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:216)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.access$300(GridPartitionedSingleGetFuture.java:67)
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture$4.run(GridPartitionedSingleGetFuture.java:771)
> at
>
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6679)
> at
>
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> ... 3 more
> [6/27/18 8:28:34:871 UTC] 000bfefe com.mbi.CacheResource
>
> E XMl Processed
> [6/27/18 8:28:34:919 UTC] 000bff07
> he.distributed.dht.preloader.GridDhtPartitionsExchangeFuture I Finish
> exchange future [startVer=AffinityTopologyVersion [topVer=20,
> minorTopVer=0]
> , resVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], err=null]
> [6/27/18 8:28:34:972 UTC] 0000009a
> .internal.processors.cache.GridCachePartitionExchangeManager I Skipping
> rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=20, m
> inorTopVer=0], evt=NODE_FAILED, node=5582ed60-76f5-42b1-bd19-ed3af12b12f7]
>
>
> Host 2 Node 2
>
>
> [6/27/18 8:28:34:517 UTC] 00000082
> ache.ignite.internal.managers.discovery.GridDiscoveryManager W Node FAILED:
> TcpDiscoveryNode [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7,
> addrs=[57.20.44.85], sockAddrs=[myhost1.org/57.20.44.85:48500],
> discPort=48500, order=2, intOrder=2, lastExchangeTime=1529923709805,
> loc=false, ver=2.4.0#20180305-sha1:aa342270, isClient=false]
> [6/27/18 8:28:34:518 UTC] 00000082 SystemOut
>
> O [08:28:34] Topology snapshot [ver=20, servers=2, clients=0, CPUs=2,
> offheap=1.8GB, heap=3.5GB]
> [6/27/18 8:28:34:519 UTC] 00000082 SystemOut
>
> O [08:28:34] Data Regions Configured:
> [6/27/18 8:28:34:519 UTC] 00000082 SystemOut
>
> O [08:28:34] ^-- default [initSize=256.0 MiB, maxSize=256.0 MiB,
> persistenceEnabled=true]
> [6/27/18 8:28:34:946 UTC] 0000a166
> st.travel.tam.allegro.processes.TurnAroundProcessAdapterImpl E Failed to
> map
> keys for cache (all partition nodes left the grid)
> [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0],
> cache=turnAroundProcesses]
>
>
>
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>