You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@helix.apache.org by kishore g <g....@gmail.com> on 2014/01/18 20:21:05 UTC

Composite rebalancer

 Currently in each stage we iterate over each resource and compute the
idealstate, the calculation of one idealstate is independent of other
resource. Even though nothing is stopping a rebalancer for reasource A to
read the computed idealstate of another resource B its not implicit.

I think it will be a good idea to have concept of composite rebalancer.

For example the rebalancer interface that computes the idealstate looks
like this currently.
  public ResourceAssignment computeResourceMapping(RebalancerConfig
rebalancerConfig,
      ResourceAssignment prevAssignment, Cluster cluster,
ResourceCurrentState currentState);

we can add a compositerebalancer like
  public Map<ResourceId,ResourceAssignment>
computeResourceMapping(RebalancerConfig rebalancerConfig,
      Map<ResourceId,ResourceAssignment> prevAssignments, Cluster cluster,
Map<ResourceId,ResourceCurrentState> currentStateMap);

In the pipeline, if we see the rebalancer type is compositerebalancer then
we can invoke computeResourceMapping once for all resources that are
configured to use this specific rebalancer.


 Thoughts?

Cheers,
Kishore G

Re: Composite rebalancer

Posted by Zhen Zhang <zz...@linkedin.com>.

Agree. Rebalancer isolation seems necessary for multi-tenancy clusters. We could probably tag rebalancers with resource names, and for example, if the pipeline is triggered by resource-A's ideal-state change, there is no need to invoke rebalancers not related to resource-A?

From: kishore g <g....@gmail.com>>
Reply-To: "user@helix.apache.org<ma...@helix.apache.org>" <us...@helix.apache.org>>
Date: Saturday, January 18, 2014 11:21 AM
To: "dev@helix.apache.org<ma...@helix.apache.org>" <de...@helix.apache.org>>, "user@helix.apache.org<ma...@helix.apache.org>" <us...@helix.apache.org>>
Subject: Composite rebalancer

Currently in each stage we iterate over each resource and compute the idealstate, the calculation of one idealstate is independent of other resource. Even though nothing is stopping a rebalancer for reasource A to read the computed idealstate of another resource B its not implicit.

I think it will be a good idea to have concept of composite rebalancer.

For example the rebalancer interface that computes the idealstate looks like this currently.
  public ResourceAssignment computeResourceMapping(RebalancerConfig rebalancerConfig,
      ResourceAssignment prevAssignment, Cluster cluster, ResourceCurrentState currentState);

we can add a compositerebalancer like
  public Map<ResourceId,ResourceAssignment> computeResourceMapping(RebalancerConfig rebalancerConfig,
      Map<ResourceId,ResourceAssignment> prevAssignments, Cluster cluster, Map<ResourceId,ResourceCurrentState> currentStateMap);

In the pipeline, if we see the rebalancer type is compositerebalancer then we can invoke computeResourceMapping once for all resources that are configured to use this specific rebalancer.

 Thoughts?

Cheers,
Kishore G

Re: Composite rebalancer

Posted by Zhen Zhang <zz...@linkedin.com>.

Agree. Rebalancer isolation seems necessary for multi-tenancy clusters. We could probably tag rebalancers with resource names, and for example, if the pipeline is triggered by resource-A's ideal-state change, there is no need to invoke rebalancers not related to resource-A?

From: kishore g <g....@gmail.com>>
Reply-To: "user@helix.apache.org<ma...@helix.apache.org>" <us...@helix.apache.org>>
Date: Saturday, January 18, 2014 11:21 AM
To: "dev@helix.apache.org<ma...@helix.apache.org>" <de...@helix.apache.org>>, "user@helix.apache.org<ma...@helix.apache.org>" <us...@helix.apache.org>>
Subject: Composite rebalancer

Currently in each stage we iterate over each resource and compute the idealstate, the calculation of one idealstate is independent of other resource. Even though nothing is stopping a rebalancer for reasource A to read the computed idealstate of another resource B its not implicit.

I think it will be a good idea to have concept of composite rebalancer.

For example the rebalancer interface that computes the idealstate looks like this currently.
  public ResourceAssignment computeResourceMapping(RebalancerConfig rebalancerConfig,
      ResourceAssignment prevAssignment, Cluster cluster, ResourceCurrentState currentState);

we can add a compositerebalancer like
  public Map<ResourceId,ResourceAssignment> computeResourceMapping(RebalancerConfig rebalancerConfig,
      Map<ResourceId,ResourceAssignment> prevAssignments, Cluster cluster, Map<ResourceId,ResourceCurrentState> currentStateMap);

In the pipeline, if we see the rebalancer type is compositerebalancer then we can invoke computeResourceMapping once for all resources that are configured to use this specific rebalancer.

 Thoughts?

Cheers,
Kishore G