You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@ignite.apache.org by Stephen Darlington <st...@gridgain.com> on 2019/06/27 15:17:57 UTC

Checking for rebalancing?

Hi,

I’m looking to be able to automate a rolling update of Ignite, that is, take nodes down one at a time until the whole cluster has the new configuration.

I have my caches configured with at least one backup. What’s the easiest way of checking that the cluster has finished rebalancing all the caches before moving onto the next node?

I can see the information in CacheMetrics and it seems to be exported to JMX, but it feels like there should be a way to see this in a command line tool. Not figured out how to see it in “control” or “ignitevisor”. Am I missing something?

Thanks in advance.

Regards,
Stephen

Re: Checking for rebalancing?

Posted by Maxim Muzafarov <ma...@gmail.com>.

Stephen,

I doubt that we should be able to have different configuration values
of some cluster aware properties on cluster nodes (all such values
must be the same). This will lead to unstable behaviour of the whole
cluster. In the opposite, build a cluster which contains different
Ignites node versions - is ok.

Coming back to your original question, sure you can check JMX metrics
(like getKeysToRebalanceLeft()) but I think it's not a full guarantee
that the rebalance of the whole cluster finished. Probably, the more
appropriate thing is listening [1] events like
EVT_CACHE_REBALANCE_STOPPED.

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/events/EventType.html#EVT_CACHE_REBALANCE_STOPPED

On Fri, 28 Jun 2019 at 12:20, Stephen Darlington
<st...@gridgain.com> wrote:
>
> Thanks for your response.
>
> Certainly getting a future indicating when the process finishes would be a lot cleaner, but I was looking in CacheMetrics where there are a bunch of methods around rebalancing, and many of those appear to be exported by JMX. Would waiting until getKeysToRebalanceLeft() returns to zero be a good proxy, for example? You’d have to iterate over all the caches.
>
> I was thinking of updating the configuration of the cluster, for example changing thread pool sizes, or JVM memory configurations, even adding a cache if done on the server side. There’s a bunch of things you can’t change (versions of Ignite, load balancing configuration, etc) but it felt like the kind of thing that people must be doing, whether with something like Ansible or Kubernetes.
>
> Regards,
> Stephen
>
> On 27 Jun 2019, at 19:01, Maxim Muzafarov <ma...@gmail.com> wrote:
>
> Stephen,
>
> AFAIK there is no such public option to wait for all the cluster
> caches are fully rebalanced. Please, take a look here [1]. This is an
> internal method which returns the future (will be completed when cache
> group rebalance is finished) and it's not exposed to any Ignites
> public API, so I doubt your goal is achievable out of the box. But I
> think this method can be accessed via some custom Ignite plugin.
>
> Also, I thought that the rolling upgrade is a part of GridGain and not
> supported by pure Apache Ignite, Am I missing something?
> What the automation tool do you use for such purpose?
>
> [1] https://github.com/Mmuzaf/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCachePreloader.java#L128
>
> On Thu, 27 Jun 2019 at 18:18, Stephen Darlington
> <st...@gridgain.com> wrote:
>
>
> Hi,
>
> I’m looking to be able to automate a rolling update of Ignite, that is, take nodes down one at a time until the whole cluster has the new configuration.
>
> I have my caches configured with at least one backup. What’s the easiest way of checking that the cluster has finished rebalancing all the caches before moving onto the next node?
>
> I can see the information in CacheMetrics and it seems to be exported to JMX, but it feels like there should be a way to see this in a command line tool. Not figured out how to see it in “control” or “ignitevisor”. Am I missing something?
>
> Thanks in advance.
>
> Regards,
> Stephen
>
>
>

Re: Checking for rebalancing?

Posted by Stephen Darlington <st...@gridgain.com>.

Thanks for your response.

Certainly getting a future indicating when the process finishes would be a lot cleaner, but I was looking in CacheMetrics where there are a bunch of methods around rebalancing, and many of those appear to be exported by JMX. Would waiting until getKeysToRebalanceLeft() returns to zero be a good proxy, for example? You’d have to iterate over all the caches.

I was thinking of updating the configuration of the cluster, for example changing thread pool sizes, or JVM memory configurations, even adding a cache if done on the server side. There’s a bunch of things you can’t change (versions of Ignite, load balancing configuration, etc) but it felt like the kind of thing that people must be doing, whether with something like Ansible or Kubernetes.

Regards,
Stephen

> On 27 Jun 2019, at 19:01, Maxim Muzafarov <ma...@gmail.com> wrote:
> 
> Stephen,
> 
> AFAIK there is no such public option to wait for all the cluster
> caches are fully rebalanced. Please, take a look here [1]. This is an
> internal method which returns the future (will be completed when cache
> group rebalance is finished) and it's not exposed to any Ignites
> public API, so I doubt your goal is achievable out of the box. But I
> think this method can be accessed via some custom Ignite plugin.
> 
> Also, I thought that the rolling upgrade is a part of GridGain and not
> supported by pure Apache Ignite, Am I missing something?
> What the automation tool do you use for such purpose?
> 
> [1] https://github.com/Mmuzaf/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCachePreloader.java#L128
> 
> On Thu, 27 Jun 2019 at 18:18, Stephen Darlington
> <st...@gridgain.com> wrote:
>> 
>> Hi,
>> 
>> I’m looking to be able to automate a rolling update of Ignite, that is, take nodes down one at a time until the whole cluster has the new configuration.
>> 
>> I have my caches configured with at least one backup. What’s the easiest way of checking that the cluster has finished rebalancing all the caches before moving onto the next node?
>> 
>> I can see the information in CacheMetrics and it seems to be exported to JMX, but it feels like there should be a way to see this in a command line tool. Not figured out how to see it in “control” or “ignitevisor”. Am I missing something?
>> 
>> Thanks in advance.
>> 
>> Regards,
>> Stephen
>>

Re: Checking for rebalancing?

Posted by Maxim Muzafarov <ma...@gmail.com>.

Stephen,

AFAIK there is no such public option to wait for all the cluster
caches are fully rebalanced. Please, take a look here [1]. This is an
internal method which returns the future (will be completed when cache
group rebalance is finished) and it's not exposed to any Ignites
public API, so I doubt your goal is achievable out of the box. But I
think this method can be accessed via some custom Ignite plugin.

Also, I thought that the rolling upgrade is a part of GridGain and not
supported by pure Apache Ignite, Am I missing something?
What the automation tool do you use for such purpose?

[1] https://github.com/Mmuzaf/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCachePreloader.java#L128

On Thu, 27 Jun 2019 at 18:18, Stephen Darlington
<st...@gridgain.com> wrote:
>
> Hi,
>
> I’m looking to be able to automate a rolling update of Ignite, that is, take nodes down one at a time until the whole cluster has the new configuration.
>
> I have my caches configured with at least one backup. What’s the easiest way of checking that the cluster has finished rebalancing all the caches before moving onto the next node?
>
> I can see the information in CacheMetrics and it seems to be exported to JMX, but it feels like there should be a way to see this in a command line tool. Not figured out how to see it in “control” or “ignitevisor”. Am I missing something?
>
> Thanks in advance.
>
> Regards,
> Stephen
>