You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Neeraj Vaidya <ne...@yahoo.co.in> on 2018/04/05 23:33:47 UTC
TcpDiscoverSpi error connection refused
Hi,
I have 2 virtualbox guest OS'es (CentOS7 64-bit) , each having firewalld stopped and disabled.
The nodes are axlrate-node-1 and axlrate-node-2. I have updated default-config.xml in both servers
My default-config.xml looks like below :
---------------------------------------------------------default-config.xml @ axlrate-node-1---------------------------------------------------------
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<!--
Alter configuration below as needed.
-->
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localAddress" value="axlrate-node-1"/>
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<!--
Explicitly specifying address of a local node to let it start and operate normally even if there is no more nodes in the cluster.
You can also optionally specify an individual port or port range.
-->
<value>axlrate-node-1</value>
<value>axlrate-node-2</value>
<!--
IP Address and optional port range of a remote node.
You can also optionally specify an individual port and don't set the port range at all.
-->
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------default-config.xml @ axlrate-node-2---------------------------------------------------------
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<!--
Alter configuration below as needed.
-->
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localAddress" value="axlrate-node-2"/>
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<!--
Explicitly specifying address of a local node to let it start and operate normally even if there is no more nodes in the cluster.
You can also optionally specify an individual port or port range.
-->
<value>axlrate-node-1</value>
<value>axlrate-node-2</value>
<!--
IP Address and optional port range of a remote node.
You can also optionally specify an individual port and don't set the port range at all.
-->
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
When I try to start ignite on axlrate-node-1, I see the following error at startup.
---------------------------------------------------------------------------log on axlrate-node-1---------------------------------------------------------------------------------------
[09:14:41,747][SEVERE][main][TcpDiscoverySpi] Exception on direct send: Connection refused (Connection refused)
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1386)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1349)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1169)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1016)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:860)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:360)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1846)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:882)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1852)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1002)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1909)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1652)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1080)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:998)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:884)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:783)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:653)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:622)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Then, when I start the axlrate-node-2, the nodes discover each other successfully.
However, when I stop axlrate-node-2, I see the following error in console of axlrate-node-1. There are no errors in axlrate-node-2 console/logs.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[09:25:02,451][SEVERE][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Caught exception on message read [sock=Socket[addr=/192.168.56.102,port=35969,localport=47500], locNodeId=cb6e4fc3-315c-4212-9a0e-139add4d328b, rmtNodeId=cf3f084c-662b-4ea2-bc9b-8140a61ed109]
class org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: sun.misc.Launcher$AppClassLoader@764c12b6
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2671)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
... 4 more
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
I can successfully telnet to axlrate-node-2 on port 47100 as well as 47500.
Any clues on what the issue could be ? I am especially worried about the Marshaller errors. There are no custom classes being used here. It is purely using Ignite installed JAR's only.
Regards,
Neeraj
Re: TcpDiscoverSpi error connection refused
Posted by Neeraj Vaidya <ne...@yahoo.co.in>.
Thanks Denis.
Sent from my iPhone
> On 9 Apr 2018, at 18:22, Denis Mekhanikov <dm...@gmail.com> wrote:
>
> There are the following lines in the TcpDiscoverySpi implementation:
>
> if (log.isDebugEnabled())
> log.error("Exception on direct send: " + e.getMessage(), e);
>
> So, these exceptions are shown only when debug logging is enabled, and they are logged as errors.
>
> Neeraj, you probably just didn't enable debug log on the second node, so you don't see these exceptions on it.
>
> Denis
>
> пн, 9 апр. 2018 г. в 7:00, Neeraj Vaidya <ne...@yahoo.co.in>:
>> Hi Denis,
>> But this issue does not occur on node2 even if it is started before node1. I feel even otherwise this message related to Discovery of level SEVERE is a bit misleading. Wouldn’t it be visible when log level is anything higher than DEBUG as well ? I agree this is not a functional issue as it only occurs at startup when performing discovery and gets cleared when a peer is discovered.
>>
>> Regards,
>> Neeraj
>>
>> Sent from my iPhone
>>
>>> On 8 Apr 2018, at 01:58, Amir Akhmedov <am...@gmail.com> wrote:
>>>
>>> Hi Denis,
>>>
>>> I had a similar logs before and I can confirm I got them when enabled "debug" level but the log level "severe" was really confusing.
>>>
>>>> On Fri, Apr 6, 2018, 8:52 AM Denis Mekhanikov <dm...@gmail.com> wrote:
>>>> Neeraj,
>>>>
>>>> You probably enabled debug logging on the following package: org.apache.ignite.spi.discovery.tcp
>>>>
>>>> There is nothing wrong in these exceptions actually.
>>>> The first connection refused exception happens because there are two hosts in the IP finder configuration, but when you start the first node, nobody is bound to the discovery port on the second host. So, you get "Connection refused" exception, when discovery SPI is trying to connect to the second host.
>>>>
>>>> If you see a message like this, then it means, that discovery finished its work successfully:
>>>> Topology snapshot [ver=1, servers=1, ...]
>>>>
>>>> You can disable debug logging not to see these exceptions.
>>>>
>>>> The second exception is also doesn't mean anything bad.
>>>> It only means, that some object was transferred to this node, and connection was suddenly closed.
>>>>
>>>> So, none of these exceptions mean anything bad.
>>>>
>>>> Denis
>>>>
>>>> пт, 6 апр. 2018 г. в 2:34, Neeraj Vaidya <ne...@yahoo.co.in>:
>>>>> Hi,
>>>>> I have 2 virtualbox guest OS'es (CentOS7 64-bit) , each having firewalld stopped and disabled.
>>>>> The nodes are axlrate-node-1 and axlrate-node-2. I have updated default-config.xml in both servers
>>>>> My default-config.xml looks like below :
>>>>> ---------------------------------------------------------default-config.xml @ axlrate-node-1---------------------------------------------------------
>>>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>> xsi:schemaLocation="
>>>>> http://www.springframework.org/schema/beans
>>>>> http://www.springframework.org/schema/beans/spring-beans.xsd">
>>>>> <!--
>>>>> Alter configuration below as needed.
>>>>> -->
>>>>> <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
>>>>> <property name="discoverySpi">
>>>>> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>>>>> <property name="localAddress" value="axlrate-node-1"/>
>>>>> <property name="ipFinder">
>>>>> <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>>>>> <property name="addresses">
>>>>> <list>
>>>>> <!--
>>>>> Explicitly specifying address of a local node to let it start and operate normally even if there is no more nodes in the cluster.
>>>>> You can also optionally specify an individual port or port range.
>>>>> -->
>>>>> <value>axlrate-node-1</value>
>>>>> <value>axlrate-node-2</value>
>>>>> <!--
>>>>> IP Address and optional port range of a remote node.
>>>>> You can also optionally specify an individual port and don't set the port range at all.
>>>>> -->
>>>>> </list>
>>>>> </property>
>>>>> </bean>
>>>>> </property>
>>>>> </bean>
>>>>> </property>
>>>>> </bean>
>>>>> </beans>
>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>
>>>>> ---------------------------------------------------------default-config.xml @ axlrate-node-2---------------------------------------------------------
>>>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>> xsi:schemaLocation="
>>>>> http://www.springframework.org/schema/beans
>>>>> http://www.springframework.org/schema/beans/spring-beans.xsd">
>>>>> <!--
>>>>> Alter configuration below as needed.
>>>>> -->
>>>>> <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
>>>>> <property name="discoverySpi">
>>>>> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>>>>> <property name="localAddress" value="axlrate-node-2"/>
>>>>> <property name="ipFinder">
>>>>> <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>>>>> <property name="addresses">
>>>>> <list>
>>>>> <!--
>>>>> Explicitly specifying address of a local node to let it start and operate normally even if there is no more nodes in the cluster.
>>>>> You can also optionally specify an individual port or port range.
>>>>> -->
>>>>> <value>axlrate-node-1</value>
>>>>> <value>axlrate-node-2</value>
>>>>> <!--
>>>>> IP Address and optional port range of a remote node.
>>>>> You can also optionally specify an individual port and don't set the port range at all.
>>>>> -->
>>>>> </list>
>>>>> </property>
>>>>> </bean>
>>>>> </property>
>>>>> </bean>
>>>>> </property>
>>>>> </bean>
>>>>> </beans>
>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> When I try to start ignite on axlrate-node-1, I see the following error at startup.
>>>>>
>>>>> ---------------------------------------------------------------------------log on axlrate-node-1---------------------------------------------------------------------------------------
>>>>> [09:14:41,747][SEVERE][main][TcpDiscoverySpi] Exception on direct send: Connection refused (Connection refused)
>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>> at java.net.Socket.connect(Socket.java:589)
>>>>> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1386)
>>>>> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1349)
>>>>> at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1169)
>>>>> at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1016)
>>>>> at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:860)
>>>>> at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:360)
>>>>> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1846)
>>>>> at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
>>>>> at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:882)
>>>>> at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1852)
>>>>> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1002)
>>>>> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1909)
>>>>> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1652)
>>>>> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1080)
>>>>> at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:998)
>>>>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:884)
>>>>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:783)
>>>>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:653)
>>>>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:622)
>>>>> at org.apache.ignite.Ignition.start(Ignition.java:347)
>>>>> at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>
>>>>> Then, when I start the axlrate-node-2, the nodes discover each other successfully.
>>>>> However, when I stop axlrate-node-2, I see the following error in console of axlrate-node-1. There are no errors in axlrate-node-2 console/logs.
>>>>>
>>>>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>> [09:25:02,451][SEVERE][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Caught exception on message read [sock=Socket[addr=/192.168.56.102,port=35969,localport=47500], locNodeId=cb6e4fc3-315c-4212-9a0e-139add4d328b, rmtNodeId=cf3f084c-662b-4ea2-bc9b-8140a61ed109]
>>>>> class org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: sun.misc.Launcher$AppClassLoader@764c12b6
>>>>> at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
>>>>> at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>>>> at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
>>>>> at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
>>>>> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>>>>> Caused by: java.io.EOFException
>>>>> at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2671)
>>>>> at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)
>>>>> at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
>>>>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
>>>>> at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
>>>>> at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
>>>>> ... 4 more
>>>>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> I can successfully telnet to axlrate-node-2 on port 47100 as well as 47500.
>>>>>
>>>>> Any clues on what the issue could be ? I am especially worried about the Marshaller errors. There are no custom classes being used here. It is purely using Ignite installed JAR's only.
>>>>>
>>>>> Regards,
>>>>> Neeraj
>>>>>
Re: TcpDiscoverSpi error connection refused
Posted by Denis Mekhanikov <dm...@gmail.com>.
There are the following lines in the TcpDiscoverySpi implementation:
if (log.isDebugEnabled())
log.error("Exception on direct send: " + e.getMessage(), e);
So, these exceptions are shown only when debug logging is enabled, and they
are logged as errors.
Neeraj, you probably just didn't enable debug log on the second node, so
you don't see these exceptions on it.
Denis
пн, 9 апр. 2018 г. в 7:00, Neeraj Vaidya <ne...@yahoo.co.in>:
> Hi Denis,
> But this issue does not occur on node2 even if it is started before node1.
> I feel even otherwise this message related to Discovery of level SEVERE is
> a bit misleading. Wouldn’t it be visible when log level is anything higher
> than DEBUG as well ? I agree this is not a functional issue as it only
> occurs at startup when performing discovery and gets cleared when a peer is
> discovered.
>
> Regards,
> Neeraj
>
> Sent from my iPhone
>
> On 8 Apr 2018, at 01:58, Amir Akhmedov <am...@gmail.com> wrote:
>
> Hi Denis,
>
> I had a similar logs before and I can confirm I got them when enabled
> "debug" level but the log level "severe" was really confusing.
>
> On Fri, Apr 6, 2018, 8:52 AM Denis Mekhanikov <dm...@gmail.com>
> wrote:
>
>> Neeraj,
>>
>> You probably enabled debug logging on the following package:
>> *org.apache.ignite.spi.discovery.tcp*
>>
>> There is nothing wrong in these exceptions actually.
>> The first connection refused exception happens because there are two
>> hosts in the IP finder configuration, but when you start the first node,
>> nobody is bound to the discovery port on the second host. So, you get *"Connection
>> refused" *exception, when discovery SPI is trying to connect to the
>> second host.
>>
>> If you see a message like this, then it means, that discovery finished
>> its work successfully:
>> *Topology snapshot [ver=1, servers=1, ...]*
>>
>> You can disable debug logging not to see these exceptions.
>>
>> The second exception is also doesn't mean anything bad.
>> It only means, that some object was transferred to this node, and
>> connection was suddenly closed.
>>
>> So, none of these exceptions mean anything bad.
>>
>> Denis
>>
>> пт, 6 апр. 2018 г. в 2:34, Neeraj Vaidya <ne...@yahoo.co.in>:
>>
>>> Hi,
>>> I have 2 virtualbox guest OS'es (CentOS7 64-bit) , each having firewalld
>>> stopped and disabled.
>>> The nodes are axlrate-node-1 and axlrate-node-2. I have updated
>>> default-config.xml in both servers
>>> My default-config.xml looks like below :
>>> ---------------------------------------------------------default-config.xml
>>> @ axlrate-node-1---------------------------------------------------------
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="
>>> http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans.xsd">
>>> <!--
>>> Alter configuration below as needed.
>>> -->
>>> <bean id="grid.cfg"
>>> class="org.apache.ignite.configuration.IgniteConfiguration">
>>> <property name="discoverySpi">
>>> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>>> <property name="localAddress" value="axlrate-node-1"/>
>>> <property name="ipFinder">
>>> <bean
>>> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>>> <property name="addresses">
>>> <list>
>>> <!--
>>> Explicitly specifying address of a local node to let it
>>> start and operate normally even if there is no more nodes
>>> in the cluster.
>>> You can also optionally specify an individual port or port
>>> range.
>>> -->
>>> <value>axlrate-node-1</value>
>>> <value>axlrate-node-2</value>
>>> <!--
>>> IP Address and optional port range of a remote node.
>>> You can also optionally specify an individual port and
>>> don't set the port range at all.
>>> -->
>>> </list>
>>> </property>
>>> </bean>
>>> </property>
>>> </bean>
>>> </property>
>>> </bean>
>>> </beans>
>>>
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>> ---------------------------------------------------------default-config.xml
>>> @ axlrate-node-2---------------------------------------------------------
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="
>>> http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans.xsd">
>>> <!--
>>> Alter configuration below as needed.
>>> -->
>>> <bean id="grid.cfg"
>>> class="org.apache.ignite.configuration.IgniteConfiguration">
>>> <property name="discoverySpi">
>>> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>>> <property name="localAddress" value="axlrate-node-2"/>
>>> <property name="ipFinder">
>>> <bean
>>> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>>> <property name="addresses">
>>> <list>
>>> <!--
>>> Explicitly specifying address of a local node to let it
>>> start and operate normally even if there is no more nodes
>>> in the cluster.
>>> You can also optionally specify an individual port or port
>>> range.
>>> -->
>>> <value>axlrate-node-1</value>
>>> <value>axlrate-node-2</value>
>>> <!--
>>> IP Address and optional port range of a remote node.
>>> You can also optionally specify an individual port and
>>> don't set the port range at all.
>>> -->
>>> </list>
>>> </property>
>>> </bean>
>>> </property>
>>> </bean>
>>> </property>
>>> </bean>
>>> </beans>
>>>
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>>
>>> When I try to start ignite on axlrate-node-1, I see the following error
>>> at startup.
>>>
>>> ---------------------------------------------------------------------------log
>>> on
>>> axlrate-node-1---------------------------------------------------------------------------------------
>>> [09:14:41,747][SEVERE][main][TcpDiscoverySpi] Exception on direct send:
>>> Connection refused (Connection refused)
>>> java.net.ConnectException: Connection refused (Connection refused)
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net
>>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>> at java.net
>>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>> at java.net
>>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>> at java.net.Socket.connect(Socket.java:589)
>>> at
>>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1386)
>>> at
>>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1349)
>>> at
>>> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1169)
>>> at
>>> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1016)
>>> at
>>> org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:860)
>>> at
>>> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:360)
>>> at
>>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1846)
>>> at
>>> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
>>> at
>>> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:882)
>>> at
>>> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1852)
>>> at
>>> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1002)
>>> at
>>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1909)
>>> at
>>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1652)
>>> at
>>> org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1080)
>>> at
>>> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:998)
>>> at
>>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:884)
>>> at
>>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:783)
>>> at
>>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:653)
>>> at
>>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:622)
>>> at org.apache.ignite.Ignition.start(Ignition.java:347)
>>> at
>>> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
>>>
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>> Then, when I start the axlrate-node-2, the nodes discover each other
>>> successfully.
>>> However, when I stop axlrate-node-2, I see the following error in
>>> console of axlrate-node-1. There are no errors in axlrate-node-2
>>> console/logs.
>>>
>>>
>>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>> [09:25:02,451][SEVERE][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Caught
>>> exception on message read [sock=Socket[addr=/192.168.56.102,port=35969,localport=47500],
>>> locNodeId=cb6e4fc3-315c-4212-9a0e-139add4d328b,
>>> rmtNodeId=cf3f084c-662b-4ea2-bc9b-8140a61ed109]
>>> class org.apache.ignite.IgniteCheckedException: Failed to deserialize
>>> object with given class loader: sun.misc.Launcher$AppClassLoader@764c12b6
>>> at
>>> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
>>> at
>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>> at
>>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
>>> at
>>> org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
>>> at
>>> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>>> Caused by: java.io.EOFException
>>> at
>>> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2671)
>>> at
>>> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)
>>> at
>>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
>>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
>>> at
>>> org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
>>> at
>>> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
>>> ... 4 more
>>>
>>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>>
>>> I can successfully telnet to axlrate-node-2 on port 47100 as well as
>>> 47500.
>>>
>>> Any clues on what the issue could be ? I am especially worried about the
>>> Marshaller errors. There are no custom classes being used here. It is
>>> purely using Ignite installed JAR's only.
>>>
>>> Regards,
>>> Neeraj
>>>
>>>
Re: TcpDiscoverSpi error connection refused
Posted by Neeraj Vaidya <ne...@yahoo.co.in>.
Hi Denis,
But this issue does not occur on node2 even if it is started before node1. I feel even otherwise this message related to Discovery of level SEVERE is a bit misleading. Wouldn’t it be visible when log level is anything higher than DEBUG as well ? I agree this is not a functional issue as it only occurs at startup when performing discovery and gets cleared when a peer is discovered.
Regards,
Neeraj
Sent from my iPhone
> On 8 Apr 2018, at 01:58, Amir Akhmedov <am...@gmail.com> wrote:
>
> Hi Denis,
>
> I had a similar logs before and I can confirm I got them when enabled "debug" level but the log level "severe" was really confusing.
>
>> On Fri, Apr 6, 2018, 8:52 AM Denis Mekhanikov <dm...@gmail.com> wrote:
>> Neeraj,
>>
>> You probably enabled debug logging on the following package: org.apache.ignite.spi.discovery.tcp
>>
>> There is nothing wrong in these exceptions actually.
>> The first connection refused exception happens because there are two hosts in the IP finder configuration, but when you start the first node, nobody is bound to the discovery port on the second host. So, you get "Connection refused" exception, when discovery SPI is trying to connect to the second host.
>>
>> If you see a message like this, then it means, that discovery finished its work successfully:
>> Topology snapshot [ver=1, servers=1, ...]
>>
>> You can disable debug logging not to see these exceptions.
>>
>> The second exception is also doesn't mean anything bad.
>> It only means, that some object was transferred to this node, and connection was suddenly closed.
>>
>> So, none of these exceptions mean anything bad.
>>
>> Denis
>>
>> пт, 6 апр. 2018 г. в 2:34, Neeraj Vaidya <ne...@yahoo.co.in>:
>>> Hi,
>>> I have 2 virtualbox guest OS'es (CentOS7 64-bit) , each having firewalld stopped and disabled.
>>> The nodes are axlrate-node-1 and axlrate-node-2. I have updated default-config.xml in both servers
>>> My default-config.xml looks like below :
>>> ---------------------------------------------------------default-config.xml @ axlrate-node-1---------------------------------------------------------
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="
>>> http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans.xsd">
>>> <!--
>>> Alter configuration below as needed.
>>> -->
>>> <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
>>> <property name="discoverySpi">
>>> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>>> <property name="localAddress" value="axlrate-node-1"/>
>>> <property name="ipFinder">
>>> <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>>> <property name="addresses">
>>> <list>
>>> <!--
>>> Explicitly specifying address of a local node to let it start and operate normally even if there is no more nodes in the cluster.
>>> You can also optionally specify an individual port or port range.
>>> -->
>>> <value>axlrate-node-1</value>
>>> <value>axlrate-node-2</value>
>>> <!--
>>> IP Address and optional port range of a remote node.
>>> You can also optionally specify an individual port and don't set the port range at all.
>>> -->
>>> </list>
>>> </property>
>>> </bean>
>>> </property>
>>> </bean>
>>> </property>
>>> </bean>
>>> </beans>
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>> ---------------------------------------------------------default-config.xml @ axlrate-node-2---------------------------------------------------------
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="
>>> http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans.xsd">
>>> <!--
>>> Alter configuration below as needed.
>>> -->
>>> <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
>>> <property name="discoverySpi">
>>> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>>> <property name="localAddress" value="axlrate-node-2"/>
>>> <property name="ipFinder">
>>> <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>>> <property name="addresses">
>>> <list>
>>> <!--
>>> Explicitly specifying address of a local node to let it start and operate normally even if there is no more nodes in the cluster.
>>> You can also optionally specify an individual port or port range.
>>> -->
>>> <value>axlrate-node-1</value>
>>> <value>axlrate-node-2</value>
>>> <!--
>>> IP Address and optional port range of a remote node.
>>> You can also optionally specify an individual port and don't set the port range at all.
>>> -->
>>> </list>
>>> </property>
>>> </bean>
>>> </property>
>>> </bean>
>>> </property>
>>> </bean>
>>> </beans>
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>>
>>> When I try to start ignite on axlrate-node-1, I see the following error at startup.
>>>
>>> ---------------------------------------------------------------------------log on axlrate-node-1---------------------------------------------------------------------------------------
>>> [09:14:41,747][SEVERE][main][TcpDiscoverySpi] Exception on direct send: Connection refused (Connection refused)
>>> java.net.ConnectException: Connection refused (Connection refused)
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>> at java.net.Socket.connect(Socket.java:589)
>>> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1386)
>>> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1349)
>>> at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1169)
>>> at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1016)
>>> at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:860)
>>> at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:360)
>>> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1846)
>>> at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
>>> at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:882)
>>> at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1852)
>>> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1002)
>>> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1909)
>>> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1652)
>>> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1080)
>>> at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:998)
>>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:884)
>>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:783)
>>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:653)
>>> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:622)
>>> at org.apache.ignite.Ignition.start(Ignition.java:347)
>>> at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>> Then, when I start the axlrate-node-2, the nodes discover each other successfully.
>>> However, when I stop axlrate-node-2, I see the following error in console of axlrate-node-1. There are no errors in axlrate-node-2 console/logs.
>>>
>>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>> [09:25:02,451][SEVERE][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Caught exception on message read [sock=Socket[addr=/192.168.56.102,port=35969,localport=47500], locNodeId=cb6e4fc3-315c-4212-9a0e-139add4d328b, rmtNodeId=cf3f084c-662b-4ea2-bc9b-8140a61ed109]
>>> class org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: sun.misc.Launcher$AppClassLoader@764c12b6
>>> at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
>>> at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>>> at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
>>> at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
>>> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>>> Caused by: java.io.EOFException
>>> at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2671)
>>> at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)
>>> at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
>>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
>>> at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
>>> at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
>>> ... 4 more
>>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>>
>>> I can successfully telnet to axlrate-node-2 on port 47100 as well as 47500.
>>>
>>> Any clues on what the issue could be ? I am especially worried about the Marshaller errors. There are no custom classes being used here. It is purely using Ignite installed JAR's only.
>>>
>>> Regards,
>>> Neeraj
>>>
Re: TcpDiscoverSpi error connection refused
Posted by Amir Akhmedov <am...@gmail.com>.
Hi Denis,
I had a similar logs before and I can confirm I got them when enabled
"debug" level but the log level "severe" was really confusing.
On Fri, Apr 6, 2018, 8:52 AM Denis Mekhanikov <dm...@gmail.com> wrote:
> Neeraj,
>
> You probably enabled debug logging on the following package:
> *org.apache.ignite.spi.discovery.tcp*
>
> There is nothing wrong in these exceptions actually.
> The first connection refused exception happens because there are two hosts
> in the IP finder configuration, but when you start the first node, nobody
> is bound to the discovery port on the second host. So, you get *"Connection
> refused" *exception, when discovery SPI is trying to connect to the
> second host.
>
> If you see a message like this, then it means, that discovery finished its
> work successfully:
> *Topology snapshot [ver=1, servers=1, ...]*
>
> You can disable debug logging not to see these exceptions.
>
> The second exception is also doesn't mean anything bad.
> It only means, that some object was transferred to this node, and
> connection was suddenly closed.
>
> So, none of these exceptions mean anything bad.
>
> Denis
>
> пт, 6 апр. 2018 г. в 2:34, Neeraj Vaidya <ne...@yahoo.co.in>:
>
>> Hi,
>> I have 2 virtualbox guest OS'es (CentOS7 64-bit) , each having firewalld
>> stopped and disabled.
>> The nodes are axlrate-node-1 and axlrate-node-2. I have updated
>> default-config.xml in both servers
>> My default-config.xml looks like below :
>> ---------------------------------------------------------default-config.xml
>> @ axlrate-node-1---------------------------------------------------------
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="
>> http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans.xsd">
>> <!--
>> Alter configuration below as needed.
>> -->
>> <bean id="grid.cfg"
>> class="org.apache.ignite.configuration.IgniteConfiguration">
>> <property name="discoverySpi">
>> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>> <property name="localAddress" value="axlrate-node-1"/>
>> <property name="ipFinder">
>> <bean
>> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>> <property name="addresses">
>> <list>
>> <!--
>> Explicitly specifying address of a local node to let it
>> start and operate normally even if there is no more nodes
>> in the cluster.
>> You can also optionally specify an individual port or port
>> range.
>> -->
>> <value>axlrate-node-1</value>
>> <value>axlrate-node-2</value>
>> <!--
>> IP Address and optional port range of a remote node.
>> You can also optionally specify an individual port and
>> don't set the port range at all.
>> -->
>> </list>
>> </property>
>> </bean>
>> </property>
>> </bean>
>> </property>
>> </bean>
>> </beans>
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> ---------------------------------------------------------default-config.xml
>> @ axlrate-node-2---------------------------------------------------------
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="
>> http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans.xsd">
>> <!--
>> Alter configuration below as needed.
>> -->
>> <bean id="grid.cfg"
>> class="org.apache.ignite.configuration.IgniteConfiguration">
>> <property name="discoverySpi">
>> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>> <property name="localAddress" value="axlrate-node-2"/>
>> <property name="ipFinder">
>> <bean
>> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>> <property name="addresses">
>> <list>
>> <!--
>> Explicitly specifying address of a local node to let it
>> start and operate normally even if there is no more nodes
>> in the cluster.
>> You can also optionally specify an individual port or port
>> range.
>> -->
>> <value>axlrate-node-1</value>
>> <value>axlrate-node-2</value>
>> <!--
>> IP Address and optional port range of a remote node.
>> You can also optionally specify an individual port and
>> don't set the port range at all.
>> -->
>> </list>
>> </property>
>> </bean>
>> </property>
>> </bean>
>> </property>
>> </bean>
>> </beans>
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>> When I try to start ignite on axlrate-node-1, I see the following error
>> at startup.
>>
>> ---------------------------------------------------------------------------log
>> on
>> axlrate-node-1---------------------------------------------------------------------------------------
>> [09:14:41,747][SEVERE][main][TcpDiscoverySpi] Exception on direct send:
>> Connection refused (Connection refused)
>> java.net.ConnectException: Connection refused (Connection refused)
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net
>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>> at java.net
>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>> at java.net
>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:589)
>> at
>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1386)
>> at
>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1349)
>> at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1169)
>> at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1016)
>> at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:860)
>> at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:360)
>> at
>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1846)
>> at
>> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
>> at
>> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:882)
>> at
>> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1852)
>> at
>> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1002)
>> at
>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1909)
>> at
>> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1652)
>> at
>> org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1080)
>> at
>> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:998)
>> at
>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:884)
>> at
>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:783)
>> at
>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:653)
>> at
>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:622)
>> at org.apache.ignite.Ignition.start(Ignition.java:347)
>> at
>> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> Then, when I start the axlrate-node-2, the nodes discover each other
>> successfully.
>> However, when I stop axlrate-node-2, I see the following error in console
>> of axlrate-node-1. There are no errors in axlrate-node-2 console/logs.
>>
>>
>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>> [09:25:02,451][SEVERE][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Caught
>> exception on message read [sock=Socket[addr=/192.168.56.102,port=35969,localport=47500],
>> locNodeId=cb6e4fc3-315c-4212-9a0e-139add4d328b,
>> rmtNodeId=cf3f084c-662b-4ea2-bc9b-8140a61ed109]
>> class org.apache.ignite.IgniteCheckedException: Failed to deserialize
>> object with given class loader: sun.misc.Launcher$AppClassLoader@764c12b6
>> at
>> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
>> at
>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>> at
>> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
>> at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
>> at
>> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>> Caused by: java.io.EOFException
>> at
>> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2671)
>> at
>> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)
>> at
>> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
>> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
>> at
>> org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
>> at
>> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
>> ... 4 more
>>
>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>> I can successfully telnet to axlrate-node-2 on port 47100 as well as
>> 47500.
>>
>> Any clues on what the issue could be ? I am especially worried about the
>> Marshaller errors. There are no custom classes being used here. It is
>> purely using Ignite installed JAR's only.
>>
>> Regards,
>> Neeraj
>>
>>
Re: TcpDiscoverSpi error connection refused
Posted by Denis Mekhanikov <dm...@gmail.com>.
Neeraj,
You probably enabled debug logging on the following package:
*org.apache.ignite.spi.discovery.tcp*
There is nothing wrong in these exceptions actually.
The first connection refused exception happens because there are two hosts
in the IP finder configuration, but when you start the first node, nobody
is bound to the discovery port on the second host. So, you get *"Connection
refused" *exception, when discovery SPI is trying to connect to the second
host.
If you see a message like this, then it means, that discovery finished its
work successfully:
*Topology snapshot [ver=1, servers=1, ...]*
You can disable debug logging not to see these exceptions.
The second exception is also doesn't mean anything bad.
It only means, that some object was transferred to this node, and
connection was suddenly closed.
So, none of these exceptions mean anything bad.
Denis
пт, 6 апр. 2018 г. в 2:34, Neeraj Vaidya <ne...@yahoo.co.in>:
> Hi,
> I have 2 virtualbox guest OS'es (CentOS7 64-bit) , each having firewalld
> stopped and disabled.
> The nodes are axlrate-node-1 and axlrate-node-2. I have updated
> default-config.xml in both servers
> My default-config.xml looks like below :
> ---------------------------------------------------------default-config.xml
> @ axlrate-node-1---------------------------------------------------------
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd">
> <!--
> Alter configuration below as needed.
> -->
> <bean id="grid.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="localAddress" value="axlrate-node-1"/>
> <property name="ipFinder">
> <bean
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> <property name="addresses">
> <list>
> <!--
> Explicitly specifying address of a local node to let it
> start and operate normally even if there is no more nodes
> in the cluster.
> You can also optionally specify an individual port or port
> range.
> -->
> <value>axlrate-node-1</value>
> <value>axlrate-node-2</value>
> <!--
> IP Address and optional port range of a remote node.
> You can also optionally specify an individual port and don't
> set the port range at all.
> -->
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
> </bean>
> </beans>
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> ---------------------------------------------------------default-config.xml
> @ axlrate-node-2---------------------------------------------------------
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd">
> <!--
> Alter configuration below as needed.
> -->
> <bean id="grid.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="localAddress" value="axlrate-node-2"/>
> <property name="ipFinder">
> <bean
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> <property name="addresses">
> <list>
> <!--
> Explicitly specifying address of a local node to let it
> start and operate normally even if there is no more nodes
> in the cluster.
> You can also optionally specify an individual port or port
> range.
> -->
> <value>axlrate-node-1</value>
> <value>axlrate-node-2</value>
> <!--
> IP Address and optional port range of a remote node.
> You can also optionally specify an individual port and don't
> set the port range at all.
> -->
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
> </bean>
> </beans>
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> When I try to start ignite on axlrate-node-1, I see the following error at
> startup.
>
> ---------------------------------------------------------------------------log
> on
> axlrate-node-1---------------------------------------------------------------------------------------
> [09:14:41,747][SEVERE][main][TcpDiscoverySpi] Exception on direct send:
> Connection refused (Connection refused)
> java.net.ConnectException: Connection refused (Connection refused)
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net
> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at java.net
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at java.net
> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1386)
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1349)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1169)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1016)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:860)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:360)
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1846)
> at
> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:882)
> at
> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1852)
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1002)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1909)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1652)
> at
> org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1080)
> at
> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:998)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:884)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:783)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:653)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:622)
> at org.apache.ignite.Ignition.start(Ignition.java:347)
> at
> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Then, when I start the axlrate-node-2, the nodes discover each other
> successfully.
> However, when I stop axlrate-node-2, I see the following error in console
> of axlrate-node-1. There are no errors in axlrate-node-2 console/logs.
>
>
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> [09:25:02,451][SEVERE][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Caught
> exception on message read [sock=Socket[addr=/192.168.56.102,port=35969,localport=47500],
> locNodeId=cb6e4fc3-315c-4212-9a0e-139add4d328b,
> rmtNodeId=cf3f084c-662b-4ea2-bc9b-8140a61ed109]
> class org.apache.ignite.IgniteCheckedException: Failed to deserialize
> object with given class loader: sun.misc.Launcher$AppClassLoader@764c12b6
> at
> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
> at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
> at
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Caused by: java.io.EOFException
> at
> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2671)
> at
> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3146)
> at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:354)
> at
> org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
> at
> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
> ... 4 more
>
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> I can successfully telnet to axlrate-node-2 on port 47100 as well as 47500.
>
> Any clues on what the issue could be ? I am especially worried about the
> Marshaller errors. There are no custom classes being used here. It is
> purely using Ignite installed JAR's only.
>
> Regards,
> Neeraj
>
>