[jira] [Created] (STORM-2286) Strom Rebalance command should support arbitrary component parallelism

["Yuzhao Chen (JIRA)" <ji...@apache.org>]

Yuzhao Chen created STORM-2286:
----------------------------------

             Summary: Strom Rebalance command should support arbitrary component parallelism
                 Key: STORM-2286
                 URL: https://issues.apache.org/jira/browse/STORM-2286
             Project: Apache Storm
          Issue Type: Bug
          Components: storm-core
    Affects Versions: 0.9.3, 0.9.6, 0.10.1, 1.0.1
            Reporter: Yuzhao Chen
             Fix For: 1.x


For legacy reasons, config TOPOLOGY-TASKS is considered first when schedule a topology, for a component, if user don’t specify TOPOLOGY-TASKS, storm just override it to be equal to component parallelism hint, and schedule based on TOPOLOGY-TASKS later on.

This works for the most cases, but not Rebalance command. Now, when do Rebalance, the StormBase :component->executors attribute will be overridden in Zookeeper which is used to partition component tasks into executors, as we said above, the TOPOLOGY-TASKS is considered here as the real tasks number for components, something goes weird here:

If we override a bigger executor numbers for a component when do rebalance, it just don’t work because smaller TOPOLOGY-TASKS [ not changed since first submitted at all ]is partitioned into bigger number of executors which read from ZooKeeper overridden by Rebalance command, but for smaller task, it works fine.

I see that storm support a command like this now: [storm rebalance topology-name [-w wait-time-secs] [-n new-num-workers] [-e component=parallelism]*] which indicate that user can override a component parallelism freely, i think it’s more sensible to support this and it's meaningless to have a restriction like before.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)