Preferred supervison in resource allocation?

[Marc MADAULE <ma...@thales-services.fr>]

 

Hello, 

I have the following cluster : 

Hardware:
Hosts srv01 and
srv03 have 256 Gb of RAM while srv02 and srv04 have 128 Gb of RAM. They
all have 28 cores.
Hosts srv01 and srv02 are in the same chassis while
srv03 and srv04 are in another one. 

Supervisor configuration :
Host
srv01, 1 supervisor with 1 SLOT, cpu capacity 2800 and memory capacity
100000
Host srv02, 1 supervisor with 1 SLOT, cpu capacity 2800 and
memory capacity 100000
Host srv03, 1 supervisor with 1 SLOT, cpu
capacity 2800 and memory capacity 100000
Host srv04, 1 supervisor with 4
SLOTS, cpu capacity 2800 and memory capacity 100000 

I have a test
Trident topology with the following total needs : cpu 150 and memory
1920 

I'm using Storm 1.0.3 (also tested with same results on 1.1.1)
and ResourceAwareScheduler. 

When I submit the topology with
setTopologyWorkerMaxHeapSize(512) I get the following resource
assignment : 

srv01 gets 1 worker
srv02 gets 1 worker
srv03 gets no
worker
srv04 gets 2 workers 

QUESTION 1 - I WAS EXPECTING THAT STORM
WOULD SCHEDULE ALL 4 WORKERS TO SRV04. INSTEAD STORM CHOSE A SOLUTION
THAT SEEMS SUB-OPTIMAL TO ME. COULD YOU PLEASE EXPLAIN WHY ? AND MAYBE
WAYS TO MAKE IT BETTER ? 

Now if I stop the topology, stop the cluster
and change the Supervisors configuration to allocate 4 workers on srv02
instead of srv04, then Storm schedules my topology like that : 

srv01
gets no worker
srv02 gets 4 workers
srv03 gets no worker
srv04 gets no
worker 

QUESTION 2 - I FIND IT STRANGE THAT SCHEDULING DOES NOT HAPPEN
THE SAME WAY. STORM SEEMS TO 'PREFER' SRV02. COULD YOU PLEASE EXPLAIN TO
ME WHY THIS IS HAPPENING OR MAYBE PROPOSE WAYS TO MITIGATE THIS ?


Thank you, 

Marc