numWorkers

[Tyson Norris <tn...@adobe.com>]

Hi - 
I am trying to determine if we can support the following use case for dynamically adjusting our cluster at runtime:

* when new node is added, existing tasks can be scheduled 
* when node is removed, existing tasks are rescheduled to remaining nodes

This seems to work already, so if I want to make sure to leverage new nodes it seems like we should:
* set the numWorkers in Config to max number that might be added to cluster
* when nodes are added/removed, perform a rebalance with new numbers for parallelism hint for whichever bolts we want to adjust

I think to prevent a single topology from “hogging” workers, we need to implement a custom scheduler - is that true in this case of setting the numWorkers to usually be larger than the number of available slots?

Ideally we’d like the parallelism to be set during scheduling based on the number of nodes, but I don’t believe this is a feature yet.

Thanks
Tyson