deploy bolts to a specific supervisor

[Ohad Edelstein <oh...@mintigo.com>]

Hey,

We are running a storm application using a single type on instance in AWS and a single topology to run our system.

This is causing some resource limitation issues. The way we want to address this is by splitting our IO intense bolts into a cluster of a few dozens/hundred  t1.small machines (for example) and all our CPU intense bolts to a few large machines with lots of cpu & memory.

Basically what i am asking is, is there a way to start all this supervisors and then deploy one topology that include cpu intense bolts on the big machines and to the small machines the deploy IO bolts?

All documentation I see, is relevant to version 0.9.x

Thanks!

Re: deploy bolts to a specific supervisor

[Ambud Sharma <as...@gmail.com>]

Storm workers are suppose to be identical for the most part. You can tune
things a little by setting odd number of executors compared to the worker
count.

To ideally accomplish what you are trying to do you can:
1. make these "cpu intensive" bolts register their task ids to zookeeper or
other KV store
2. query this information and use direct grouping to manually route the
message

You will still have bolt instances for "cpu intensive" workload on your t1s
 however they will be idle.

On Thu, Nov 24, 2016 at 1:56 PM, Ohad Edelstein <oh...@mintigo.com> wrote:

> Hey,
>
> We are running a storm application using a single type on instance in AWS
> and a single topology to run our system.
>
> This is causing some resource limitation issues. The way we want to
> address this is by splitting our IO intense bolts into a cluster of a few
> dozens/hundred  t1.small machines (for example) and all our CPU intense
> bolts to a few large machines with lots of cpu & memory.
>
> Basically what i am asking is, is there a way to start all this
> supervisors and then deploy one topology that include cpu intense bolts on
> the big machines and to the small machines the deploy IO bolts?
>
> All documentation I see, is relevant to version 0.9.x
>
> Thanks!
>