You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Charlin S <Ch...@hotelhub.com> on 2022/01/28 03:51:31 UTC
Ignite 2.10.0 client "Node_Failed" during application pool recycle
Hi Ignite team,
We are using Ignite 2.10.0 with .Net 5 WebAPI and we have a 16-nodes(2
servers) Ignite cluster
The client nodes have the following property as below
IgniteConfiguration.AutoGenerateIgniteInstanceName = true;
IgniteConfiguration.ConsistentId = Guid.NewGuid().ToString().ToUpper();
Some of the client nodes are failing as "Node_Failed" during application
pool recycle every day, but it was different behaviour with .Net 4.6.2
WebAPI as "Node Left Topology"
I check the logs with the following findings:
Some nodes are marking as "Node Left Topology" and some nodes are marking
as "Node_Failed"
We would like to figure out the root cause of the "Node_Failed" during
application pool recycle.
Regards,
Charlin
Re: Ignite 2.10.0 client "Node_Failed" during application pool recycle
Posted by Charlin S <Ch...@hotelhub.com>.
Hi,
Thanks for your reply.
Already handled applicationLifetime(C# 5) and called
Ignition.Stop(gridname,true) but now changed to Ignition.StopAll(true);
seems to be working fine now.
Regards,
Charlin
On Mon, 31 Jan 2022 at 18:06, Pavel Tupitsyn <pt...@apache.org> wrote:
> It looks like the app pool gets recycled, but the Ignite node is not shut
> down gracefully, causing the abrupt disconnect.
>
> You can use code like this to stop all Ignite nodes gracefully when the
> application exits:
> AppDomain.CurrentDomain.ProcessExit += Ignition.StopAll(true);
>
> On Sat, Jan 29, 2022 at 7:18 AM Charlin S <Ch...@hotelhub.com> wrote:
>
>> Hi,
>>
>> We are using IIS.
>>
>> Regards,
>> Charlin
>>
>>
>> On Fri, 28 Jan 2022 at 16:54, Pavel Tupitsyn <pt...@apache.org>
>> wrote:
>>
>>> ConsistentId is important for persistence, I don't think you should set
>>> it at all.
>>> "application pool" - do you use IIS or something like that?
>>>
>>> On Fri, Jan 28, 2022 at 1:48 PM Charlin S <Ch...@hotelhub.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> 1. Is there a reason to set it manually?
>>>> --We had node disconnection multiple times and were unable to join
>>>> back so we decided to set a new consistentId.
>>>> .Net 4.6.2 and .Net 5 WebAPI's are having the same logic.
>>>> 2. Do you use Ignite persistence?
>>>> -- No, our cache type is in-memory.
>>>>
>>>> Regards,
>>>> Charlin
>>>>
>>>> On Fri, 28 Jan 2022 at 12:57, Pavel Tupitsyn <pt...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Charlin,
>>>>>
>>>>> ConsistentId should not be random [1]
>>>>>
>>>>> 1. Is there a reason to set it manually?
>>>>> 2. Do you use Ignite persistence?
>>>>>
>>>>> [1]
>>>>> https://ignite.apache.org/docs/latest/persistence/native-persistence
>>>>>
>>>>> On Fri, Jan 28, 2022 at 6:53 AM Charlin S <Ch...@hotelhub.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Ignite team,
>>>>>> We are using Ignite 2.10.0 with .Net 5 WebAPI and we have a
>>>>>> 16-nodes(2 servers) Ignite cluster
>>>>>> The client nodes have the following property as below
>>>>>> IgniteConfiguration.AutoGenerateIgniteInstanceName = true;
>>>>>> IgniteConfiguration.ConsistentId =
>>>>>> Guid.NewGuid().ToString().ToUpper();
>>>>>>
>>>>>> Some of the client nodes are failing as "Node_Failed" during
>>>>>> application pool recycle every day, but it was different behaviour with
>>>>>> .Net 4.6.2 WebAPI as "Node Left Topology"
>>>>>>
>>>>>> I check the logs with the following findings:
>>>>>> Some nodes are marking as "Node Left Topology" and some nodes are
>>>>>> marking as "Node_Failed"
>>>>>>
>>>>>> We would like to figure out the root cause of the "Node_Failed"
>>>>>> during application pool recycle.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Charlin
>>>>>>
>>>>>
Re: Ignite 2.10.0 client "Node_Failed" during application pool recycle
Posted by Pavel Tupitsyn <pt...@apache.org>.
It looks like the app pool gets recycled, but the Ignite node is not shut
down gracefully, causing the abrupt disconnect.
You can use code like this to stop all Ignite nodes gracefully when the
application exits:
AppDomain.CurrentDomain.ProcessExit += Ignition.StopAll(true);
On Sat, Jan 29, 2022 at 7:18 AM Charlin S <Ch...@hotelhub.com> wrote:
> Hi,
>
> We are using IIS.
>
> Regards,
> Charlin
>
>
> On Fri, 28 Jan 2022 at 16:54, Pavel Tupitsyn <pt...@apache.org> wrote:
>
>> ConsistentId is important for persistence, I don't think you should set
>> it at all.
>> "application pool" - do you use IIS or something like that?
>>
>> On Fri, Jan 28, 2022 at 1:48 PM Charlin S <Ch...@hotelhub.com> wrote:
>>
>>> Hi,
>>>
>>> 1. Is there a reason to set it manually?
>>> --We had node disconnection multiple times and were unable to join
>>> back so we decided to set a new consistentId.
>>> .Net 4.6.2 and .Net 5 WebAPI's are having the same logic.
>>> 2. Do you use Ignite persistence?
>>> -- No, our cache type is in-memory.
>>>
>>> Regards,
>>> Charlin
>>>
>>> On Fri, 28 Jan 2022 at 12:57, Pavel Tupitsyn <pt...@apache.org>
>>> wrote:
>>>
>>>> Hi Charlin,
>>>>
>>>> ConsistentId should not be random [1]
>>>>
>>>> 1. Is there a reason to set it manually?
>>>> 2. Do you use Ignite persistence?
>>>>
>>>> [1]
>>>> https://ignite.apache.org/docs/latest/persistence/native-persistence
>>>>
>>>> On Fri, Jan 28, 2022 at 6:53 AM Charlin S <Ch...@hotelhub.com>
>>>> wrote:
>>>>
>>>>> Hi Ignite team,
>>>>> We are using Ignite 2.10.0 with .Net 5 WebAPI and we have a 16-nodes(2
>>>>> servers) Ignite cluster
>>>>> The client nodes have the following property as below
>>>>> IgniteConfiguration.AutoGenerateIgniteInstanceName = true;
>>>>> IgniteConfiguration.ConsistentId = Guid.NewGuid().ToString().ToUpper();
>>>>>
>>>>> Some of the client nodes are failing as "Node_Failed" during
>>>>> application pool recycle every day, but it was different behaviour with
>>>>> .Net 4.6.2 WebAPI as "Node Left Topology"
>>>>>
>>>>> I check the logs with the following findings:
>>>>> Some nodes are marking as "Node Left Topology" and some nodes are
>>>>> marking as "Node_Failed"
>>>>>
>>>>> We would like to figure out the root cause of the "Node_Failed" during
>>>>> application pool recycle.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Charlin
>>>>>
>>>>
Re: Ignite 2.10.0 client "Node_Failed" during application pool recycle
Posted by Charlin S <Ch...@hotelhub.com>.
Hi,
We are using IIS.
Regards,
Charlin
On Fri, 28 Jan 2022 at 16:54, Pavel Tupitsyn <pt...@apache.org> wrote:
> ConsistentId is important for persistence, I don't think you should set it
> at all.
> "application pool" - do you use IIS or something like that?
>
> On Fri, Jan 28, 2022 at 1:48 PM Charlin S <Ch...@hotelhub.com> wrote:
>
>> Hi,
>>
>> 1. Is there a reason to set it manually?
>> --We had node disconnection multiple times and were unable to join back
>> so we decided to set a new consistentId.
>> .Net 4.6.2 and .Net 5 WebAPI's are having the same logic.
>> 2. Do you use Ignite persistence?
>> -- No, our cache type is in-memory.
>>
>> Regards,
>> Charlin
>>
>> On Fri, 28 Jan 2022 at 12:57, Pavel Tupitsyn <pt...@apache.org>
>> wrote:
>>
>>> Hi Charlin,
>>>
>>> ConsistentId should not be random [1]
>>>
>>> 1. Is there a reason to set it manually?
>>> 2. Do you use Ignite persistence?
>>>
>>> [1] https://ignite.apache.org/docs/latest/persistence/native-persistence
>>>
>>> On Fri, Jan 28, 2022 at 6:53 AM Charlin S <Ch...@hotelhub.com>
>>> wrote:
>>>
>>>> Hi Ignite team,
>>>> We are using Ignite 2.10.0 with .Net 5 WebAPI and we have a 16-nodes(2
>>>> servers) Ignite cluster
>>>> The client nodes have the following property as below
>>>> IgniteConfiguration.AutoGenerateIgniteInstanceName = true;
>>>> IgniteConfiguration.ConsistentId = Guid.NewGuid().ToString().ToUpper();
>>>>
>>>> Some of the client nodes are failing as "Node_Failed" during
>>>> application pool recycle every day, but it was different behaviour with
>>>> .Net 4.6.2 WebAPI as "Node Left Topology"
>>>>
>>>> I check the logs with the following findings:
>>>> Some nodes are marking as "Node Left Topology" and some nodes are
>>>> marking as "Node_Failed"
>>>>
>>>> We would like to figure out the root cause of the "Node_Failed" during
>>>> application pool recycle.
>>>>
>>>>
>>>> Regards,
>>>> Charlin
>>>>
>>>
Re: Ignite 2.10.0 client "Node_Failed" during application pool recycle
Posted by Pavel Tupitsyn <pt...@apache.org>.
ConsistentId is important for persistence, I don't think you should set it
at all.
"application pool" - do you use IIS or something like that?
On Fri, Jan 28, 2022 at 1:48 PM Charlin S <Ch...@hotelhub.com> wrote:
> Hi,
>
> 1. Is there a reason to set it manually?
> --We had node disconnection multiple times and were unable to join back
> so we decided to set a new consistentId.
> .Net 4.6.2 and .Net 5 WebAPI's are having the same logic.
> 2. Do you use Ignite persistence?
> -- No, our cache type is in-memory.
>
> Regards,
> Charlin
>
> On Fri, 28 Jan 2022 at 12:57, Pavel Tupitsyn <pt...@apache.org> wrote:
>
>> Hi Charlin,
>>
>> ConsistentId should not be random [1]
>>
>> 1. Is there a reason to set it manually?
>> 2. Do you use Ignite persistence?
>>
>> [1] https://ignite.apache.org/docs/latest/persistence/native-persistence
>>
>> On Fri, Jan 28, 2022 at 6:53 AM Charlin S <Ch...@hotelhub.com> wrote:
>>
>>> Hi Ignite team,
>>> We are using Ignite 2.10.0 with .Net 5 WebAPI and we have a 16-nodes(2
>>> servers) Ignite cluster
>>> The client nodes have the following property as below
>>> IgniteConfiguration.AutoGenerateIgniteInstanceName = true;
>>> IgniteConfiguration.ConsistentId = Guid.NewGuid().ToString().ToUpper();
>>>
>>> Some of the client nodes are failing as "Node_Failed" during application
>>> pool recycle every day, but it was different behaviour with .Net 4.6.2
>>> WebAPI as "Node Left Topology"
>>>
>>> I check the logs with the following findings:
>>> Some nodes are marking as "Node Left Topology" and some nodes are
>>> marking as "Node_Failed"
>>>
>>> We would like to figure out the root cause of the "Node_Failed" during
>>> application pool recycle.
>>>
>>>
>>> Regards,
>>> Charlin
>>>
>>
Re: Ignite 2.10.0 client "Node_Failed" during application pool recycle
Posted by Charlin S <Ch...@hotelhub.com>.
Hi,
1. Is there a reason to set it manually?
--We had node disconnection multiple times and were unable to join back
so we decided to set a new consistentId.
.Net 4.6.2 and .Net 5 WebAPI's are having the same logic.
2. Do you use Ignite persistence?
-- No, our cache type is in-memory.
Regards,
Charlin
On Fri, 28 Jan 2022 at 12:57, Pavel Tupitsyn <pt...@apache.org> wrote:
> Hi Charlin,
>
> ConsistentId should not be random [1]
>
> 1. Is there a reason to set it manually?
> 2. Do you use Ignite persistence?
>
> [1] https://ignite.apache.org/docs/latest/persistence/native-persistence
>
> On Fri, Jan 28, 2022 at 6:53 AM Charlin S <Ch...@hotelhub.com> wrote:
>
>> Hi Ignite team,
>> We are using Ignite 2.10.0 with .Net 5 WebAPI and we have a 16-nodes(2
>> servers) Ignite cluster
>> The client nodes have the following property as below
>> IgniteConfiguration.AutoGenerateIgniteInstanceName = true;
>> IgniteConfiguration.ConsistentId = Guid.NewGuid().ToString().ToUpper();
>>
>> Some of the client nodes are failing as "Node_Failed" during application
>> pool recycle every day, but it was different behaviour with .Net 4.6.2
>> WebAPI as "Node Left Topology"
>>
>> I check the logs with the following findings:
>> Some nodes are marking as "Node Left Topology" and some nodes are marking
>> as "Node_Failed"
>>
>> We would like to figure out the root cause of the "Node_Failed" during
>> application pool recycle.
>>
>>
>> Regards,
>> Charlin
>>
>
Re: Ignite 2.10.0 client "Node_Failed" during application pool recycle
Posted by Pavel Tupitsyn <pt...@apache.org>.
Hi Charlin,
ConsistentId should not be random [1]
1. Is there a reason to set it manually?
2. Do you use Ignite persistence?
[1] https://ignite.apache.org/docs/latest/persistence/native-persistence
On Fri, Jan 28, 2022 at 6:53 AM Charlin S <Ch...@hotelhub.com> wrote:
> Hi Ignite team,
> We are using Ignite 2.10.0 with .Net 5 WebAPI and we have a 16-nodes(2
> servers) Ignite cluster
> The client nodes have the following property as below
> IgniteConfiguration.AutoGenerateIgniteInstanceName = true;
> IgniteConfiguration.ConsistentId = Guid.NewGuid().ToString().ToUpper();
>
> Some of the client nodes are failing as "Node_Failed" during application
> pool recycle every day, but it was different behaviour with .Net 4.6.2
> WebAPI as "Node Left Topology"
>
> I check the logs with the following findings:
> Some nodes are marking as "Node Left Topology" and some nodes are marking
> as "Node_Failed"
>
> We would like to figure out the root cause of the "Node_Failed" during
> application pool recycle.
>
>
> Regards,
> Charlin
>