You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Charlin S <Ch...@hotelhub.com> on 2021/02/15 07:58:21 UTC
Ignite 2.9.1 - An existing connection was forcibly closed by the
remote host
Hi,
i'm running an ASP.Net application with ignite 2.9.1 and seeing below error
details in ignite client log and my web site stopped working. I have faced
the same issue with ignite 2.8.1 and I have upgraded 2.9.1 just before 1
day, since ignite 2.9.0 release notes saying below points:
1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a
node fails to send a message or ping, now it drops the current connection
strictly within this timeout and begins establishing a new connection much
faster.
2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If a
node loses connection, now it strictly obtains a new connection to the ring
of gets segmented within this timeout.
It's back to normal after restarting the application pool.
Please any suggestions? to avoid such issues.
Log files attached here for your reference.
Thanks & Regards,
Charlin
Re: Ignite 2.9.1 - An existing connection was forcibly closed by the
remote host
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Have you gotten around collecting thread dump from the client?
Regards,
--
Ilya Kasnacheev
вт, 16 февр. 2021 г. в 16:59, Charlin S <Ch...@hotelhub.com>:
> Hello,
> Thanks for your help,
> I am not sure where to check the network misconfiguration either in System
> or in Application configuration (ignite ean file).
> I am thinking to reduce interval to 1 min and retry count 5 and currently
> its 2 minutes and 10 times.
> My current configuration is as below and please let me know if any changes
> are required.
>
> <?xml version="1.0" encoding="UTF-8"?>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:util="http://www.springframework.org/schema/util"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
>
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://www.springframework.org/schema/util
>
> http://www.springframework.org/schema/util/spring-util.xsd">
> <bean class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="clientMode" value="true"/>
> <property name="localHost" value="10.212.120.180"/>
> <property name="clientFailureDetectionTimeout" value="60000"/>
> <property name="igniteInstanceName" value="hh_StaticGrid"/>
> <property name="longQueryWarningTimeout" value="4000"/>
> <property name="failureDetectionTimeout" value="60000"/>
> <property name="segmentationPolicy" value="RESTART_JVM"/>
> <property name="segmentCheckFrequency" value="20000"/>
> <property name="segmentationResolveAttempts" value="5"/>
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="localPort" value="51500"/>
> <property name="ipFinder">
> <bean
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> <property name="addresses">
> <list>
> <value>10.212.120.66:51500</value>
> <value>10.212.120.67:51500</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
> <property name="communicationSpi">
> <bean
> class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> <property name="sharedMemoryPort" value="-1"/>
> <property name="localPort" value="51605"/>
> </bean>
> </property>
> </bean>
> </beans>
>
> Thanks & Regards
> Charlin
>
>
>
> On Tue, 16 Feb 2021 at 15:00, Ilya Kasnacheev <il...@gmail.com>
> wrote:
>
>> Hello!
>>
>> It should be JVM thread dump.
>>
>> Packet drops is one thing, network misconfiguration is another.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 15 февр. 2021 г. в 15:12, Charlin S <Ch...@hotelhub.com>:
>>
>>> Hello,
>>> Thanks for your help,
>>> As per our network team, there were no packet drops between client and
>>> server.
>>> Do I need to collect either jvm thread dump or IIS worker process thread
>>> dump ?
>>>
>>> Thanks & Regards,
>>> Charlin
>>>
>>>
>>>
>>> On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <il...@gmail.com>
>>> wrote:
>>>
>>>> Hello!
>>>>
>>>> I think I've just answered the same question for 2.8.1:
>>>>
>>>> https://lists.apache.org/thread.html/r102cf85775f7759e6772ceeed6d3a6843ea736b20647020fb8374833%40%3Cuser.ignite.apache.org%3E
>>>>
>>>> I'm not sure why your node does not recognize itself as SEGMENTED.
>>>> [15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi]
>>>> Failed to send message: null
>>>> java.io.IOException: Failed to get acknowledge for message:
>>>> TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
>>>> [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d,
>>>> verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
>>>> isClient=true]]
>>>> at
>>>> org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
>>>> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
>>>>
>>>> Maybe there's something in your network configuration which prevents
>>>> relaying of "connection closed" packets from client to server?
>>>> Can you collect thread dump from such failing client when it has
>>>> already started to exhibit that behavior?
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> пн, 15 февр. 2021 г. в 11:00, Charlin S <Ch...@hotelhub.com>:
>>>>
>>>>> Hi,
>>>>>
>>>>> i'm running an ASP.Net application with ignite 2.9.1 and seeing below
>>>>> error details in ignite client log and my web site stopped working. I have
>>>>> faced the same issue with ignite 2.8.1 and I have upgraded 2.9.1 just
>>>>> before 1 day, since ignite 2.9.0 release notes saying below points:
>>>>> 1. Fixed processing of failure detection timeout in TcpDiscoverySpi.
>>>>> If a node fails to send a message or ping, now it drops the current
>>>>> connection strictly within this timeout and begins establishing a new
>>>>> connection much faster.
>>>>> 2. Fixed processing of connection recovery timeout in TcpDiscoverySpi.
>>>>> If a node loses connection, now it strictly obtains a new connection to the
>>>>> ring of gets segmented within this timeout.
>>>>>
>>>>> It's back to normal after restarting the application pool.
>>>>> Please any suggestions? to avoid such issues.
>>>>> Log files attached here for your reference.
>>>>>
>>>>> Thanks & Regards,
>>>>> Charlin
>>>>>
>>>>>
Re: Ignite 2.9.1 - An existing connection was forcibly closed by the
remote host
Posted by Charlin S <Ch...@hotelhub.com>.
Hello,
Thanks for your help,
I am not sure where to check the network misconfiguration either in System
or in Application configuration (ignite ean file).
I am thinking to reduce interval to 1 min and retry count 5 and currently
its 2 minutes and 10 times.
My current configuration is as below and please let me know if any changes
are required.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="clientMode" value="true"/>
<property name="localHost" value="10.212.120.180"/>
<property name="clientFailureDetectionTimeout" value="60000"/>
<property name="igniteInstanceName" value="hh_StaticGrid"/>
<property name="longQueryWarningTimeout" value="4000"/>
<property name="failureDetectionTimeout" value="60000"/>
<property name="segmentationPolicy" value="RESTART_JVM"/>
<property name="segmentCheckFrequency" value="20000"/>
<property name="segmentationResolveAttempts" value="5"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localPort" value="51500"/>
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>10.212.120.66:51500</value>
<value>10.212.120.67:51500</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="communicationSpi">
<bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="sharedMemoryPort" value="-1"/>
<property name="localPort" value="51605"/>
</bean>
</property>
</bean>
</beans>
Thanks & Regards
Charlin
On Tue, 16 Feb 2021 at 15:00, Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> It should be JVM thread dump.
>
> Packet drops is one thing, network misconfiguration is another.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 15 февр. 2021 г. в 15:12, Charlin S <Ch...@hotelhub.com>:
>
>> Hello,
>> Thanks for your help,
>> As per our network team, there were no packet drops between client and
>> server.
>> Do I need to collect either jvm thread dump or IIS worker process thread
>> dump ?
>>
>> Thanks & Regards,
>> Charlin
>>
>>
>>
>> On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <il...@gmail.com>
>> wrote:
>>
>>> Hello!
>>>
>>> I think I've just answered the same question for 2.8.1:
>>>
>>> https://lists.apache.org/thread.html/r102cf85775f7759e6772ceeed6d3a6843ea736b20647020fb8374833%40%3Cuser.ignite.apache.org%3E
>>>
>>> I'm not sure why your node does not recognize itself as SEGMENTED.
>>> [15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi]
>>> Failed to send message: null
>>> java.io.IOException: Failed to get acknowledge for message:
>>> TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
>>> [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d,
>>> verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
>>> isClient=true]]
>>> at
>>> org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
>>> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
>>>
>>> Maybe there's something in your network configuration which prevents
>>> relaying of "connection closed" packets from client to server?
>>> Can you collect thread dump from such failing client when it has already
>>> started to exhibit that behavior?
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пн, 15 февр. 2021 г. в 11:00, Charlin S <Ch...@hotelhub.com>:
>>>
>>>> Hi,
>>>>
>>>> i'm running an ASP.Net application with ignite 2.9.1 and seeing below
>>>> error details in ignite client log and my web site stopped working. I have
>>>> faced the same issue with ignite 2.8.1 and I have upgraded 2.9.1 just
>>>> before 1 day, since ignite 2.9.0 release notes saying below points:
>>>> 1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If
>>>> a node fails to send a message or ping, now it drops the current connection
>>>> strictly within this timeout and begins establishing a new connection much
>>>> faster.
>>>> 2. Fixed processing of connection recovery timeout in TcpDiscoverySpi.
>>>> If a node loses connection, now it strictly obtains a new connection to the
>>>> ring of gets segmented within this timeout.
>>>>
>>>> It's back to normal after restarting the application pool.
>>>> Please any suggestions? to avoid such issues.
>>>> Log files attached here for your reference.
>>>>
>>>> Thanks & Regards,
>>>> Charlin
>>>>
>>>>
Re: Ignite 2.9.1 - An existing connection was forcibly closed by the
remote host
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
It should be JVM thread dump.
Packet drops is one thing, network misconfiguration is another.
Regards,
--
Ilya Kasnacheev
пн, 15 февр. 2021 г. в 15:12, Charlin S <Ch...@hotelhub.com>:
> Hello,
> Thanks for your help,
> As per our network team, there were no packet drops between client and
> server.
> Do I need to collect either jvm thread dump or IIS worker process thread
> dump ?
>
> Thanks & Regards,
> Charlin
>
>
>
> On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <il...@gmail.com>
> wrote:
>
>> Hello!
>>
>> I think I've just answered the same question for 2.8.1:
>>
>> https://lists.apache.org/thread.html/r102cf85775f7759e6772ceeed6d3a6843ea736b20647020fb8374833%40%3Cuser.ignite.apache.org%3E
>>
>> I'm not sure why your node does not recognize itself as SEGMENTED.
>> [15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi]
>> Failed to send message: null
>> java.io.IOException: Failed to get acknowledge for message:
>> TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
>> [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d,
>> verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
>> isClient=true]]
>> at
>> org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
>> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
>>
>> Maybe there's something in your network configuration which prevents
>> relaying of "connection closed" packets from client to server?
>> Can you collect thread dump from such failing client when it has already
>> started to exhibit that behavior?
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 15 февр. 2021 г. в 11:00, Charlin S <Ch...@hotelhub.com>:
>>
>>> Hi,
>>>
>>> i'm running an ASP.Net application with ignite 2.9.1 and seeing below
>>> error details in ignite client log and my web site stopped working. I have
>>> faced the same issue with ignite 2.8.1 and I have upgraded 2.9.1 just
>>> before 1 day, since ignite 2.9.0 release notes saying below points:
>>> 1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If
>>> a node fails to send a message or ping, now it drops the current connection
>>> strictly within this timeout and begins establishing a new connection much
>>> faster.
>>> 2. Fixed processing of connection recovery timeout in TcpDiscoverySpi.
>>> If a node loses connection, now it strictly obtains a new connection to the
>>> ring of gets segmented within this timeout.
>>>
>>> It's back to normal after restarting the application pool.
>>> Please any suggestions? to avoid such issues.
>>> Log files attached here for your reference.
>>>
>>> Thanks & Regards,
>>> Charlin
>>>
>>>
Re: Ignite 2.9.1 - An existing connection was forcibly closed by the
remote host
Posted by Charlin S <Ch...@hotelhub.com>.
Hello,
Thanks for your help,
As per our network team, there were no packet drops between client and
server.
Do I need to collect either jvm thread dump or IIS worker process thread
dump ?
Thanks & Regards,
Charlin
On Mon, 15 Feb 2021 at 16:40, Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> I think I've just answered the same question for 2.8.1:
>
> https://lists.apache.org/thread.html/r102cf85775f7759e6772ceeed6d3a6843ea736b20647020fb8374833%40%3Cuser.ignite.apache.org%3E
>
> I'm not sure why your node does not recognize itself as SEGMENTED.
> [15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi]
> Failed to send message: null
> java.io.IOException: Failed to get acknowledge for message:
> TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
> [sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d,
> verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
> isClient=true]]
> at
> org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
>
> Maybe there's something in your network configuration which prevents
> relaying of "connection closed" packets from client to server?
> Can you collect thread dump from such failing client when it has already
> started to exhibit that behavior?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 15 февр. 2021 г. в 11:00, Charlin S <Ch...@hotelhub.com>:
>
>> Hi,
>>
>> i'm running an ASP.Net application with ignite 2.9.1 and seeing below
>> error details in ignite client log and my web site stopped working. I have
>> faced the same issue with ignite 2.8.1 and I have upgraded 2.9.1 just
>> before 1 day, since ignite 2.9.0 release notes saying below points:
>> 1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a
>> node fails to send a message or ping, now it drops the current connection
>> strictly within this timeout and begins establishing a new connection much
>> faster.
>> 2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If
>> a node loses connection, now it strictly obtains a new connection to the
>> ring of gets segmented within this timeout.
>>
>> It's back to normal after restarting the application pool.
>> Please any suggestions? to avoid such issues.
>> Log files attached here for your reference.
>>
>> Thanks & Regards,
>> Charlin
>>
>>
Re: Ignite 2.9.1 - An existing connection was forcibly closed by the
remote host
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I think I've just answered the same question for 2.8.1:
https://lists.apache.org/thread.html/r102cf85775f7759e6772ceeed6d3a6843ea736b20647020fb8374833%40%3Cuser.ignite.apache.org%3E
I'm not sure why your node does not recognize itself as SEGMENTED.
[15:04:29,711][SEVERE][tcp-client-disco-sock-writer-#2%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%-#47%ignite-instance-a5430afb-05b7-4492-9825-460966133ad1%][TcpDiscoverySpi]
Failed to send message: null
java.io.IOException: Failed to get acknowledge for message:
TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=0610eed9771-841c8962-775e-4109-bc74-2c003598a02d,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
isClient=true]]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1471)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
Maybe there's something in your network configuration which prevents
relaying of "connection closed" packets from client to server?
Can you collect thread dump from such failing client when it has already
started to exhibit that behavior?
Regards,
--
Ilya Kasnacheev
пн, 15 февр. 2021 г. в 11:00, Charlin S <Ch...@hotelhub.com>:
> Hi,
>
> i'm running an ASP.Net application with ignite 2.9.1 and seeing below
> error details in ignite client log and my web site stopped working. I have
> faced the same issue with ignite 2.8.1 and I have upgraded 2.9.1 just
> before 1 day, since ignite 2.9.0 release notes saying below points:
> 1. Fixed processing of failure detection timeout in TcpDiscoverySpi. If a
> node fails to send a message or ping, now it drops the current connection
> strictly within this timeout and begins establishing a new connection much
> faster.
> 2. Fixed processing of connection recovery timeout in TcpDiscoverySpi. If
> a node loses connection, now it strictly obtains a new connection to the
> ring of gets segmented within this timeout.
>
> It's back to normal after restarting the application pool.
> Please any suggestions? to avoid such issues.
> Log files attached here for your reference.
>
> Thanks & Regards,
> Charlin
>
>