[jira] [Created] (STORM-3093) Cache the storm id to executors mapping on master to avoid repeat computation

["Yuzhao Chen (JIRA)" <ji...@apache.org>]

Yuzhao Chen created STORM-3093:
----------------------------------

             Summary: Cache the storm id to executors mapping on master to avoid repeat computation
                 Key: STORM-3093
                 URL: https://issues.apache.org/jira/browse/STORM-3093
             Project: Apache Storm
          Issue Type: Improvement
          Components: storm-server
    Affects Versions: 2.0.0
            Reporter: Yuzhao Chen
            Assignee: Yuzhao Chen
             Fix For: 2.0.0


Now nimbus will collect all the topologies's conf/topology-ser/storm-base to compute in a scheduling round, which is a very heavy work. The scheduling will still take to minutes even we now change to PRC heartbeats and assignment distribution.

So i decide to redesign the scheduler, so we can only schedule the topologies that need to: that have dead workers or not enough number workers.

Here i checkout out the code and found that the id->executors mapping is computed every time for every topology, which is really a heavy computation and totally not that necessary, because this mapping is fixed invariable for a topology unless we rebalance or kill it.

So i refactor the code a little here, and this is more powerful after the scheduler is resigned for delta-scheduling[  which is very lightweight even there are thousands of topologies on one cluster.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)