add new tasks to already assigned slots after scheduling for the first time

[AMir Firouzi <fi...@gmail.com>]

I need some migration features for my scheduler in storm. i schedule the
tasks first time based on some logic and then after a while i need to
migrate some tasks to another workers. but after doing so if i assign the
task to an already used slot storm(nimbus) nags about not being able to use
an already used slot. is it impossible to do so? otherwise i have to use
another slots while some old used slots have the capacity to contains more
tasks and also compacting related tasks to a slot reduces intra-worker
traffic.

thanks

Re: add new tasks to already assigned slots after scheduling for the first time

[AMir Firouzi <fi...@gmail.com>]

thanks Bobby for your reply. I thought it might cause problems in the
background and i was looking for an alternative way with lower overhead.
Actually i can remove the old tasks from current slot and occupy a new slot
and put together all those tasks, but i'm worried freeing a slot and taking
another one causes too much overhead and reduces performance. is that right
or i shouldn't be worried too much about it.

thanks again for your time.
appreciate it

Regards

On Thu, Aug 3, 2017 at 7:34 PM Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:

> I am not completely sure why that was a requirement.  I know internally we
> have removed it, not sure if we pushed that change back or not yet.  On
> newer versions of storm 1.0.3+ that have the rewritten supervisor that
> check is invalid and you should be able to remove the check and just update
> the slot.  You might be able to get away with it on older versions too, I
> just don't know.  I think it had something to do with race conditions in
> both the supervisor where if the scheduling changed without clearing it out
> first the supervisor would not know that it changed and not do the right
> thing.
> - Bobby
>
>
> On Tuesday, August 1, 2017, 4:16:10 PM CDT, AMir Firouzi <
> firouziam@gmail.com> wrote:
>
> I need some migration features for my scheduler in storm. i schedule the
> tasks first time based on some logic and then after a while i need to
> migrate some tasks to another workers. but after doing so if i assign the
> task to an already used slot storm(nimbus) nags about not being able to use
> an already used slot. is it impossible to do so? otherwise i have to use
> another slots while some old used slots have the capacity to contains more
> tasks and also compacting related tasks to a slot reduces intra-worker
> traffic.
>
> thanks
>

Re: add new tasks to already assigned slots after scheduling for the first time

[Bobby Evans <ev...@yahoo-inc.com.INVALID>]

I am not completely sure why that was a requirement.  I know internally we have removed it, not sure if we pushed that change back or not yet.  On newer versions of storm 1.0.3+ that have the rewritten supervisor that check is invalid and you should be able to remove the check and just update the slot.  You might be able to get away with it on older versions too, I just don't know.  I think it had something to do with race conditions in both the supervisor where if the scheduling changed without clearing it out first the supervisor would not know that it changed and not do the right thing.
- Bobby


On Tuesday, August 1, 2017, 4:16:10 PM CDT, AMir Firouzi <fi...@gmail.com> wrote:

I need some migration features for my scheduler in storm. i schedule the
tasks first time based on some logic and then after a while i need to
migrate some tasks to another workers. but after doing so if i assign the
task to an already used slot storm(nimbus) nags about not being able to use
an already used slot. is it impossible to do so? otherwise i have to use
another slots while some old used slots have the capacity to contains more
tasks and also compacting related tasks to a slot reduces intra-worker
traffic.

thanks