You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Ranjit Sahu <ra...@gmail.com> on 2018/02/01 08:18:49 UTC
Re: Rebalancing mode set to None
Hi Val,
Not always but out of 10, we see at least once the issue. Whats happening
is when one node crashes\stops the new node joins . The loading process
restarts but what ever was happening in few minutes (3-5) goes to 2-3
hours.
Thanks,
Ranjit
On Wed, Jan 31, 2018 at 3:12 AM, vkulichenko <va...@gmail.com>
wrote:
> Ranjit,
>
> Is it a really frequent event for node to crash in the middle of loading
> process? If so, then I think you should fix that instead of working around
> by disabling rebalancing. Such configuration definitely has a lot drawbacks
> and therefore can cause issues.
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Rebalancing mode set to None
Posted by Evgenii Zhuravlev <e....@gmail.com>.
But currently you start the same Ignite nodes which you will start in
standalone mode, but you run them embedded.
If you have 3 backups in the cluster and one of your nodes fails, then part
of your data will have 2 backups. Also, after the returning to the cluster,
without manual rebalancing, some data still will have only 2 backups of
some data and newly added node won't have any data on it. So, if 3 other
nodes will fail in future, you will lose all backups to the certain
partitions(without manual rebalancing, of course)
Evgenii
2018-02-01 12:03 GMT+03:00 Ranjit Sahu <ra...@gmail.com>:
> yes i am using it in embeded mode. Standalone mode we can go for, but
> additional hardware needed for that which will stay idle when we don't use
> it. That was the reason to try this embedded stuff. We build the cache on
> fly , which gets shut down with spark.
>
> My question basically was, when i set rebalancing to NONE and have back up
> count as 3, how it behaves ?
> Will i still have all the data when one node stops cause of back up set to
> 3 ?
>
>
> On Thu, Feb 1, 2018 at 2:26 PM, Evgenii Zhuravlev <
> e.zhuravlev.wk@gmail.com> wrote:
>
>> > When due to some reason a task fails, the ignite node stops and when
>> the task re-starts,
>> Looks like you run in embedded mode. Do avoid too frequently node
>> stopping events, you need to run Ignite in standalone mode, in this case,
>> the node will run even if your task fails.
>>
>> Please let me know if I missed something.
>>
>> Evgenii
>>
>> 2018-02-01 11:44 GMT+03:00 Ranjit Sahu <ra...@gmail.com>:
>>
>>> I read the data which is in avro format using spark sql and load it to
>>> cache from spark program. I build the Ignite key-store inside spark
>>> executors. When due to some reason a task fails, the ignite node stops and
>>> when the task re-starts,
>>> the new node joins back.I see slowness from here onwards. I was thinking
>>> cause of rebalancing this is becoming slow. I can look at tuning the
>>> rebalancing too. Let me know if you have any suggestions.
>>>
>>> On Thu, Feb 1, 2018 at 1:54 PM, Evgenii Zhuravlev <
>>> e.zhuravlev.wk@gmail.com> wrote:
>>>
>>>> Ranjit,
>>>>
>>>> How do you load data to the cache?
>>>>
>>>> Evgenii
>>>>
>>>> 2018-02-01 11:18 GMT+03:00 Ranjit Sahu <ra...@gmail.com>:
>>>>
>>>>> Hi Val,
>>>>>
>>>>> Not always but out of 10, we see at least once the issue. Whats
>>>>> happening is when one node crashes\stops the new node joins . The loading
>>>>> process restarts but what ever was happening in few minutes (3-5) goes to
>>>>> 2-3 hours.
>>>>>
>>>>> Thanks,
>>>>> Ranjit
>>>>>
>>>>> On Wed, Jan 31, 2018 at 3:12 AM, vkulichenko <
>>>>> valentin.kulichenko@gmail.com> wrote:
>>>>>
>>>>>> Ranjit,
>>>>>>
>>>>>> Is it a really frequent event for node to crash in the middle of
>>>>>> loading
>>>>>> process? If so, then I think you should fix that instead of working
>>>>>> around
>>>>>> by disabling rebalancing. Such configuration definitely has a lot
>>>>>> drawbacks
>>>>>> and therefore can cause issues.
>>>>>>
>>>>>> -Val
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Rebalancing mode set to None
Posted by Ranjit Sahu <ra...@gmail.com>.
yes i am using it in embeded mode. Standalone mode we can go for, but
additional hardware needed for that which will stay idle when we don't use
it. That was the reason to try this embedded stuff. We build the cache on
fly , which gets shut down with spark.
My question basically was, when i set rebalancing to NONE and have back up
count as 3, how it behaves ?
Will i still have all the data when one node stops cause of back up set to
3 ?
On Thu, Feb 1, 2018 at 2:26 PM, Evgenii Zhuravlev <e....@gmail.com>
wrote:
> > When due to some reason a task fails, the ignite node stops and when
> the task re-starts,
> Looks like you run in embedded mode. Do avoid too frequently node stopping
> events, you need to run Ignite in standalone mode, in this case, the node
> will run even if your task fails.
>
> Please let me know if I missed something.
>
> Evgenii
>
> 2018-02-01 11:44 GMT+03:00 Ranjit Sahu <ra...@gmail.com>:
>
>> I read the data which is in avro format using spark sql and load it to
>> cache from spark program. I build the Ignite key-store inside spark
>> executors. When due to some reason a task fails, the ignite node stops and
>> when the task re-starts,
>> the new node joins back.I see slowness from here onwards. I was thinking
>> cause of rebalancing this is becoming slow. I can look at tuning the
>> rebalancing too. Let me know if you have any suggestions.
>>
>> On Thu, Feb 1, 2018 at 1:54 PM, Evgenii Zhuravlev <
>> e.zhuravlev.wk@gmail.com> wrote:
>>
>>> Ranjit,
>>>
>>> How do you load data to the cache?
>>>
>>> Evgenii
>>>
>>> 2018-02-01 11:18 GMT+03:00 Ranjit Sahu <ra...@gmail.com>:
>>>
>>>> Hi Val,
>>>>
>>>> Not always but out of 10, we see at least once the issue. Whats
>>>> happening is when one node crashes\stops the new node joins . The loading
>>>> process restarts but what ever was happening in few minutes (3-5) goes to
>>>> 2-3 hours.
>>>>
>>>> Thanks,
>>>> Ranjit
>>>>
>>>> On Wed, Jan 31, 2018 at 3:12 AM, vkulichenko <
>>>> valentin.kulichenko@gmail.com> wrote:
>>>>
>>>>> Ranjit,
>>>>>
>>>>> Is it a really frequent event for node to crash in the middle of
>>>>> loading
>>>>> process? If so, then I think you should fix that instead of working
>>>>> around
>>>>> by disabling rebalancing. Such configuration definitely has a lot
>>>>> drawbacks
>>>>> and therefore can cause issues.
>>>>>
>>>>> -Val
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Rebalancing mode set to None
Posted by Evgenii Zhuravlev <e....@gmail.com>.
> When due to some reason a task fails, the ignite node stops and when the
task re-starts,
Looks like you run in embedded mode. Do avoid too frequently node stopping
events, you need to run Ignite in standalone mode, in this case, the node
will run even if your task fails.
Please let me know if I missed something.
Evgenii
2018-02-01 11:44 GMT+03:00 Ranjit Sahu <ra...@gmail.com>:
> I read the data which is in avro format using spark sql and load it to
> cache from spark program. I build the Ignite key-store inside spark
> executors. When due to some reason a task fails, the ignite node stops and
> when the task re-starts,
> the new node joins back.I see slowness from here onwards. I was thinking
> cause of rebalancing this is becoming slow. I can look at tuning the
> rebalancing too. Let me know if you have any suggestions.
>
> On Thu, Feb 1, 2018 at 1:54 PM, Evgenii Zhuravlev <
> e.zhuravlev.wk@gmail.com> wrote:
>
>> Ranjit,
>>
>> How do you load data to the cache?
>>
>> Evgenii
>>
>> 2018-02-01 11:18 GMT+03:00 Ranjit Sahu <ra...@gmail.com>:
>>
>>> Hi Val,
>>>
>>> Not always but out of 10, we see at least once the issue. Whats
>>> happening is when one node crashes\stops the new node joins . The loading
>>> process restarts but what ever was happening in few minutes (3-5) goes to
>>> 2-3 hours.
>>>
>>> Thanks,
>>> Ranjit
>>>
>>> On Wed, Jan 31, 2018 at 3:12 AM, vkulichenko <
>>> valentin.kulichenko@gmail.com> wrote:
>>>
>>>> Ranjit,
>>>>
>>>> Is it a really frequent event for node to crash in the middle of loading
>>>> process? If so, then I think you should fix that instead of working
>>>> around
>>>> by disabling rebalancing. Such configuration definitely has a lot
>>>> drawbacks
>>>> and therefore can cause issues.
>>>>
>>>> -Val
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>
>>>
>>
>
Re: Rebalancing mode set to None
Posted by Ranjit Sahu <ra...@gmail.com>.
I read the data which is in avro format using spark sql and load it to
cache from spark program. I build the Ignite key-store inside spark
executors. When due to some reason a task fails, the ignite node stops and
when the task re-starts,
the new node joins back.I see slowness from here onwards. I was thinking
cause of rebalancing this is becoming slow. I can look at tuning the
rebalancing too. Let me know if you have any suggestions.
On Thu, Feb 1, 2018 at 1:54 PM, Evgenii Zhuravlev <e....@gmail.com>
wrote:
> Ranjit,
>
> How do you load data to the cache?
>
> Evgenii
>
> 2018-02-01 11:18 GMT+03:00 Ranjit Sahu <ra...@gmail.com>:
>
>> Hi Val,
>>
>> Not always but out of 10, we see at least once the issue. Whats happening
>> is when one node crashes\stops the new node joins . The loading process
>> restarts but what ever was happening in few minutes (3-5) goes to 2-3
>> hours.
>>
>> Thanks,
>> Ranjit
>>
>> On Wed, Jan 31, 2018 at 3:12 AM, vkulichenko <
>> valentin.kulichenko@gmail.com> wrote:
>>
>>> Ranjit,
>>>
>>> Is it a really frequent event for node to crash in the middle of loading
>>> process? If so, then I think you should fix that instead of working
>>> around
>>> by disabling rebalancing. Such configuration definitely has a lot
>>> drawbacks
>>> and therefore can cause issues.
>>>
>>> -Val
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>
>>
>
Re: Rebalancing mode set to None
Posted by Evgenii Zhuravlev <e....@gmail.com>.
Ranjit,
How do you load data to the cache?
Evgenii
2018-02-01 11:18 GMT+03:00 Ranjit Sahu <ra...@gmail.com>:
> Hi Val,
>
> Not always but out of 10, we see at least once the issue. Whats happening
> is when one node crashes\stops the new node joins . The loading process
> restarts but what ever was happening in few minutes (3-5) goes to 2-3
> hours.
>
> Thanks,
> Ranjit
>
> On Wed, Jan 31, 2018 at 3:12 AM, vkulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
>> Ranjit,
>>
>> Is it a really frequent event for node to crash in the middle of loading
>> process? If so, then I think you should fix that instead of working around
>> by disabling rebalancing. Such configuration definitely has a lot
>> drawbacks
>> and therefore can cause issues.
>>
>> -Val
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>