You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by sa...@barclays.com on 2021/09/23 04:25:26 UTC
Node left Cluster Topology
Hi Team,
We faced one issue where Ignite Node left topology and closed . We are establishing connection to Ignite cluster using .NET thin client. Is it possible that .NET thin client could not establish connection and throw system.exception which led to crashing the Ignite node? Here is the attached complete log when node was down. Also find the spring configuration at bottom.
[01:44:34,886][SEVERE][grid-nio-worker-client-listener-4-#56][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=4, bytesRcvd=1882888095, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-4, igniteInstanceName=null, finished=false, heartbeatTs=1632271472760, hashCode=941750299, interrupted=false, runner=grid-nio-worker-client-listener-4-#56]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=65320 cap=65536], readBuf=java.nio.DirectByteBuffer[pos=0 lim=65536 cap=65536], inRecovery=null, outRecovery=null, closeSocket=true, outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl [locAddr=/10.49.124.247:10800, rmtAddr=/10.49.124.247:7340, createTime=1632270121965, closeTime=0, bytesSent=14741547, bytesRcvd=1144669, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1632271472450, lastSndTime=1632271472450, lastRcvTime=1632271456534, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerNioMessageParser, directMode=true], SSL filter], accepted=true, markedForClose=false]]]
java.io.IOException: An established connection was aborted by the software in your host machine
at java.base/sun.nio.ch.SocketDispatcher.read0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356)
at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1337)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2494)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2259)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1896)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.base/java.lang.Thread.run(Thread.java:829)
[01:44:34,886][SEVERE][grid-nio-worker-client-listener-0-#52][ClientListenerProcessor] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=4449364592, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-client-listener-0, igniteInstanceName=null, finished=false, heartbeatTs=1632271472760, hashCode=178188300, interrupted=false, runner=grid-nio-worker-client-listener-0-#52]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=65320 cap=65536], readBuf=java.nio.DirectByteBuffer[pos=0 lim=65536 cap=65536], inRecovery=null, outRecovery=null, closeSocket=true, outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl [locAddr=/10.49.124.247:10800, rmtAddr=/10.113.40.200:13689, createTime=1632270206351, closeTime=0, bytesSent=13652314, bytesRcvd=1009832, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1632271457073, lastSndTime=1632271472450, lastRcvTime=1632271456534, readsPaused=false, filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter, GridNioCodecFilter [parser=ClientListenerNioMessageParser, directMode=true], SSL filter], accepted=true, markedForClose=false]]]
java.io.IOException: An established connection was aborted by the software in your host machine
at java.base/sun.nio.ch.SocketDispatcher.read0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356)
at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1337)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2494)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2259)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1896)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.base/java.lang.Thread.run(Thread.java:829)
[01:44:34,902][SEVERE][disco-event-worker-#77][FailureProcessor] Ignite node is in invalid state due to a critical failure.
[01:44:34,886][WARNING][grid-nio-worker-client-listener-4-#56][ClientListenerProcessor] Client disconnected abruptly due to network connection loss or because the connection was left open on application shutdown. [cls=class o.a.i.i.util.nio.GridNioException, msg=An established connection was aborted by the software in your host machine]
[01:44:34,902][WARNING][grid-nio-worker-client-listener-0-#52][ClientListenerProcessor] Client disconnected abruptly due to network connection loss or because the connection was left open on application shutdown. [cls=class o.a.i.i.util.nio.GridNioException, msg=An established connection was aborted by the software in your host machine]
[01:44:34,902][WARNING][disco-event-worker-#77][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=aff824d1-bb29-4998-99ca-9c7e5652a2bb, consistentId=10.49.124.245,127.0.0.1:47500, addrs=ArrayList [10.49.124.245, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500, GBRPSM030001094.INTRANET.BARCAPINT.com/10.49.124.245:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1631951752550, loc=false, ver=2.10.0#20210310-sha1:bc24f6ba, isClient=false]
[01:44:34,902][SEVERE][node-stopper][] Stopping local node on Ignite failure: [failureCtx=FailureContext [type=SEGMENTATION, err=null]]
[01:44:34,916][INFO][disco-event-worker-#77][GridDiscoveryManager] Topology snapshot [ver=11, locNode=27dd1221, servers=3, clients=0, state=ACTIVE, CPUs=48, offheap=38.0GB, heap=30.0GB]
[01:44:34,916][INFO][disco-event-worker-#77][GridDiscoveryManager] Coordinator changed [prev=TcpDiscoveryNode [id=aff824d1-bb29-4998-99ca-9c7e5652a2bb, consistentId=10.49.124.245,127.0.0.1:47500, addrs=ArrayList [10.49.124.245, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500, GBRPSM030001094.INTRANET.BARCAPINT.com/10.49.124.245:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1631951752550, loc=false, ver=2.10.0#20210310-sha1:bc24f6ba, isClient=false], cur=TcpDiscoveryNode [id=615027f1-8204-4dae-bc36-61fbd3c8bebb, consistentId=10.113.42.143,127.0.0.1:47500, addrs=ArrayList [10.113.42.143, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500, GBRPSM030001194.INTRANET.BARCAPINT.com/10.113.42.143:47500], discPort=47500, order=2, intOrder=2, lastExchangeTime=1631951752550, loc=false, ver=2.10.0#20210310-sha1:bc24f6ba, isClient=false]]
[01:44:34,916][INFO][disco-event-worker-#77][GridDiscoveryManager] ^-- Baseline [id=0, size=3, online=1, offline=2]
[01:44:34,916][INFO][disco-event-worker-#77][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], nodeId=615027f1-8204-4dae-bc36-61fbd3c8bebb, ver=1631951752282, local=false, initialized=false]]
[01:44:34,948][INFO][node-stopper][GridTcpRestProtocol] Command protocol successfully stopped: TCP binary
[01:44:34,948][SEVERE][client-connector-#122][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=aff824d1-bb29-4998-99ca-9c7e5652a2bb, consistentId=10.49.124.245,127.0.0.1:47500, addrs=ArrayList [10.49.124.245, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500, GBRPSM030001094.INTRANET.BARCAPINT.com/10.49.124.245:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1631951752550, loc=false, ver=2.10.0#20210310-sha1:bc24f6ba, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest [futId=1632063136249, key=UserKeyCacheObjectImpl [part=575, val=systemconfiguration, hasValBytes=true], flags=1, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], subjId=27dd1221-1887-41b2-b6ae-3b3e08bc6715, taskNameHash=0, createTtl=-1, accessTtl=-1, txLbl=null, mvccSnapshot=null]]]
class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to send message (node left topology): TcpDiscoveryNode [id=aff824d1-bb29-4998-99ca-9c7e5652a2bb, consistentId=10.49.124.245,127.0.0.1:47500, addrs=ArrayList [10.49.124.245, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500, GBRPSM030001094.INTRANET.BARCAPINT.com/10.49.124.245:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1631951752550, loc=false, ver=2.10.0#20210310-sha1:bc24f6ba, isClient=false]
at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:420)
at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:691)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:1255)
at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:689)
at org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.createCommunicationClient(ConnectionClientPool.java:453)
at org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.reserveClient(ConnectionClientPool.java:228)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1186)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1133)
at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2101)
at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2196)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1257)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:381)
at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.init(GridPartitionedSingleGetFuture.java:248)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.getAsync0(GridDhtAtomicCache.java:1459)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1600(GridDhtAtomicCache.java:146)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$16.apply(GridDhtAtomicCache.java:488)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$16.apply(GridDhtAtomicCache.java:486)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:785)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.getAsync(GridDhtAtomicCache.java:486)
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4886)
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.repairableGet(GridCacheAdapter.java:4852)
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1470)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:1107)
at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:676)
at org.apache.ignite.internal.processors.platform.client.cache.ClientCacheGetRequest.process(ClientCacheGetRequest.java:41)
at org.apache.ignite.internal.processors.platform.client.ClientRequestHandler.handle(ClientRequestHandler.java:99)
at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:202)
at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:56)
at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.body(GridNioAsyncNotifyFilter.java:97)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at org.apache.ignite.internal.util.worker.GridWorkerPool$1.run(GridWorkerPool.java:70)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Below is the spring configuration.
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<list>
<!-- Partitioned cache example configuration (Atomic mode). -->
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="default"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="0"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="onheapCacheEnabled" value="true"/>
</bean>
</list>
</property>
<property name="clientConnectorConfiguration">
<bean class="org.apache.ignite.configuration.ClientConnectorConfiguration">
<property name="sslEnabled" value="true"/>
</bean>
</property>
<property name="sslContextFactory">
<bean class="org.apache.ignite.ssl.SslContextFactory">
<property name="keyStoreFilePath" value="C: "/>
<property name="keyStorePassword" value="xxx "/>
<property name="trustManagers">
<bean class="org.apache.ignite.ssl.SslContextFactory" factory-method="getDisabledTrustManager"/>
</property>
</bean>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery that can be used
instead os static IP based discovery. For information on all options refer
to our documentation: http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">-->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value> xx.xxx.xx.xxx:47500</value>
<value> xx.xxx.xx.xxx:47500</value>
<value>xx.xxx.xx.xxx:47500</value>
<value>xx.xxx.xx.xxx:47500</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
Restricted - Internal
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________
�This message is for information purposes only, it is not a recommendation, advice, offer or solicitation to buy or sell a product or service nor an official confirmation of any transaction. It is directed at persons who are professionals and is not intended for retail customer use. Intended for recipient only. This message is subject to the terms at: www.barclays.com/emaildisclaimer.
For important disclosures, please see: www.barclays.com/salesandtradingdisclaimer regarding market commentary from Barclays Sales and/or Trading, who are active market participants; https://www.investmentbank.barclays.com/disclosures/barclays-global-markets-disclosures.html regarding our standard terms for the Investment Bank of Barclays where we trade with you in principal-to-principal wholesale markets transactions; and in respect of Barclays Research, including disclosures relating to specific issuers, please see http://publicresearch.barclays.com.�
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________
If you are incorporated or operating in Australia, please see https://www.home.barclays/disclosures/importantapacdisclosures.html for important disclosure.
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________
How we use personal information see our privacy notice https://www.investmentbank.barclays.com/disclosures/personalinformationuse.html
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________