You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@helix.apache.org by Brent <br...@gmail.com> on 2020/09/21 18:08:33 UTC

Changing the rebalancing strategy to CrushED via CLI or REST

Hello,

I'm interested in using a topology-aware rebalancing strategy.  It's my
understanding that the default AutoRebalanceStrategy is not able to be
topology-aware (based on "Partition Assignment Strategies" in
https://cwiki.apache.org/confluence/display/HELIX/Helix+Topology-aware+Rebalance+Strategy).
Based on the "Simple User Guide" at the bottom of
http://helix.apache.org/1.0.1-docs/design_crushed.html, I'm attempting to
set my resource to use the CrushED REBALANCE_STRATEGY via this command (I'd
like to avoid the Java API if I can):

./helix-admin.sh --zkSvr 1.2.3.4:2181 --setConfig RESOURCE MyCluster,MyDB
"REBALANCE_STRATEGY=org.apache.helix.controller.rebalancer.strategy.CrushEdRebalanceStrategy"

When I pull the data from the idealState or externalView, it doesn't seem
to reflect the change (only idealState shown below):

~/helix-binaries/helix-core-1.0.1/bin$ curl
http://localhost:8100/admin/v2/clusters/MyCluster/resources/MyDB/idealState
{
  "id" : "MyDB",
  "mapFields" : {
    "MyDB_0" : {},
    "MyDB_1" : {},
    "MyDB_2" : {}
  },
  "simpleFields" : {
    "IDEAL_STATE_MODE" : "AUTO_REBALANCE",
    "NUM_PARTITIONS" : "3",
    "REBALANCE_MODE" : "FULL_AUTO",
    "REBALANCE_STRATEGY" : "DEFAULT",
    "REPLICAS" : "1",
    "STATE_MODEL_DEF_REF" : "MasterSlave",
    "STATE_MODEL_FACTORY_NAME" : "DEFAULT"
  },
  "listFields" : {
    "MyDB_0" : [ ],
    "MyDB_1" : [ ],
    "MyDB_2" : [ ]
  }
}

However, when I pull the config for the resource itself, it seems to show
what I'd expect:
~/helix-binaries/helix-core-1.0.1/bin$ curl
http://localhost:8100/admin/v2/clusters/MyCluster/resources/MyDB/configs
{
  "id" : "MyDB",
  "mapFields" : {
  },
  "simpleFields" : {
    "REBALANCE_STRATEGY" :
"org.apache.helix.controller.rebalancer.strategy.CrushEdRebalanceStrategy"
  },
  "listFields" : {
  }
}

Am I doing this correctly?  Has that setting properly taken effect?  I'm a
bit confused by why I see it in the resource config, but not in the
IdealState or externalView and I want to make sure this is the proper way
to actually use an alternate rebalancing strategy.

Thank you!

Re: Changing the rebalancing strategy to CrushED via CLI or REST

Posted by Brent <br...@gmail.com>.
I think I figured out some of my confusion which is that "--setConfig" and
"--addResourceProperty" on the CLI are two different things.

If I do this, then the setting shows up in the Resource configuration:

./helix-admin.sh --zkSvr 1.2.3.4:2181 --setConfig RESOURCE MyCluster,MyDB
"REBALANCE_STRATEGY=org.apache.helix.controller.rebalancer.strategy.CrushEdRebalanceStrategy"

But if I do this, the setting shows up in the Resource IdealState (though
not the Resource ExternalView):

./helix-admin.sh --zkSvr 1.2.3.4:2181 --addResourceProperty MyCluster MyDB
REBALANCE_STRATEGY
"org.apache.helix.controller.rebalancer.strategy.CrushRebalanceStrategy"

How do I know when to set configuration vs. add a property?  There are a
bunch of other settings I'd like to set, but I'm not sure if they're
configuration or properties.

Thank you!

~Brent

On Mon, Sep 21, 2020 at 11:08 AM Brent <br...@gmail.com> wrote:

> Hello,
>
> I'm interested in using a topology-aware rebalancing strategy.  It's my
> understanding that the default AutoRebalanceStrategy is not able to be
> topology-aware (based on "Partition Assignment Strategies" in
> https://cwiki.apache.org/confluence/display/HELIX/Helix+Topology-aware+Rebalance+Strategy).
> Based on the "Simple User Guide" at the bottom of
> http://helix.apache.org/1.0.1-docs/design_crushed.html, I'm attempting to
> set my resource to use the CrushED REBALANCE_STRATEGY via this command (I'd
> like to avoid the Java API if I can):
>
> ./helix-admin.sh --zkSvr 1.2.3.4:2181 --setConfig RESOURCE MyCluster,MyDB
> "REBALANCE_STRATEGY=org.apache.helix.controller.rebalancer.strategy.CrushEdRebalanceStrategy"
>
> When I pull the data from the idealState or externalView, it doesn't seem
> to reflect the change (only idealState shown below):
>
> ~/helix-binaries/helix-core-1.0.1/bin$ curl
> http://localhost:8100/admin/v2/clusters/MyCluster/resources/MyDB/idealState
> {
>   "id" : "MyDB",
>   "mapFields" : {
>     "MyDB_0" : {},
>     "MyDB_1" : {},
>     "MyDB_2" : {}
>   },
>   "simpleFields" : {
>     "IDEAL_STATE_MODE" : "AUTO_REBALANCE",
>     "NUM_PARTITIONS" : "3",
>     "REBALANCE_MODE" : "FULL_AUTO",
>     "REBALANCE_STRATEGY" : "DEFAULT",
>     "REPLICAS" : "1",
>     "STATE_MODEL_DEF_REF" : "MasterSlave",
>     "STATE_MODEL_FACTORY_NAME" : "DEFAULT"
>   },
>   "listFields" : {
>     "MyDB_0" : [ ],
>     "MyDB_1" : [ ],
>     "MyDB_2" : [ ]
>   }
> }
>
> However, when I pull the config for the resource itself, it seems to show
> what I'd expect:
> ~/helix-binaries/helix-core-1.0.1/bin$ curl
> http://localhost:8100/admin/v2/clusters/MyCluster/resources/MyDB/configs
> {
>   "id" : "MyDB",
>   "mapFields" : {
>   },
>   "simpleFields" : {
>     "REBALANCE_STRATEGY" :
> "org.apache.helix.controller.rebalancer.strategy.CrushEdRebalanceStrategy"
>   },
>   "listFields" : {
>   }
> }
>
> Am I doing this correctly?  Has that setting properly taken effect?  I'm a
> bit confused by why I see it in the resource config, but not in the
> IdealState or externalView and I want to make sure this is the proper way
> to actually use an alternate rebalancing strategy.
>
> Thank you!
>