How does Rebalancing happen in Storm

[Yashwant Ganti <ya...@gmail.com>]

Hello All,

We have been running into throughput issues with our topology and to solve
this, we are trying to add more supervisors and at the next step, more
executors for what seem like the bottle neck components.

My initial set up has 8 supervisors, each running 8 worker processes and I
have two identical server sets running two instances of the same topology.
We use one server set as a playground for tuning and testing our code and
backend DB performance, before porting over to the "main" server set.

I added 2 supervisors to both the farms, so each server set now has 10
supervisors machines, which are supposed to run a total of 64 workers.
Before I hit rebalance, the two newly added supervisors were not assigned
workers (on both farms). Next, I hit the 'rebalance' option from the Nimbus
UI.

The outcome was different on the two server sets. One server set got what
seemed like a more event distribution of workers [7,7,7,7,6,6,6,6,6,6]
while the other had [8,8,8,8,7,6,7,6,8,0]. I was just wondering about the
criteria that are taken into account for a rebalance. Both the server sets
are reading from identical queues and have identical settings for any DBs
that they write to.

Thanks,
Yash