Scheduler question: Is there a way to limit the number of threads running on a certain worker process?

[Serkan Uzunbaz <uz...@gmail.com>]

Hi all,
I have a heavy spout task that has a high emission rate and many bolt tasks
running accross the cluster. I observe that, the bolt tasks that are
running on the worker process which also runs the heavy spout task have a
higher execution latency than its peers (the other tasks of the same bolt
type). My goal is to have an even execute latency and hence even capacity
among the tasks of the same bolt.

I believe, this is because high production rate of the spout task keeps the
transfer queue of the worker process congested and hence the emit call of
the co-tenant bolt task gets blocked and the execute function waits for an
empty space in the queue (I know that there is a separete transfer queue
for the bolt task's executor but I think it also gets congested since the
transfer queue of the worker process is congested). First of all, is this
inference correct? What else might be the main reason?

I tried throttling the spout with max.spout.pending, and played with the
sizes of the queues but could not find the best setting that alleviates the
problem.
I am trying to solve the problem by writing my own pluggable scheduler
following this post (
http://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/).
As done in the post, I just want to schedule the spout task (or tasks) in
my scheduler and use the EvenScheduler to schedule the rest of the tasks.
However, the question is how can I make sure that EvenScheduler does not
schedule tasks to the same worker process that my scheduler assigned the
spout tasks? Is there a way to mark a worker process as occupied or full,
or limit the number of threads that can be assigned to it?
If not, and if the only way is to write a complete scheduler, is there a
java version of the EvenScheduler that I can reuse?
Thanks and sorry for the long message.
-Serkan

Re: Scheduler question: Is there a way to limit the number of threads running on a certain worker process?

[James Xu <xu...@gmail.com>]

One api my blog post does not cover is `blacklistHost`: https://github.com/nathanmarz/storm/blob/master/storm-core/src/jvm/backtype/storm/scheduler/Cluster.java?source=c#L55 (it is introduced after I wrote that blog), with this you can tell the EvenScheduler that `host-xxx` is dedicated to run my spout, don’t assign anything else to it.

On 2013年12月5日, at 上午10:24, Serkan Uzunbaz <uz...@gmail.com> wrote:

> Hi all,
> I have a heavy spout task that has a high emission rate and many bolt tasks running accross the cluster. I observe that, the bolt tasks that are running on the worker process which also runs the heavy spout task have a higher execution latency than its peers (the other tasks of the same bolt type). My goal is to have an even execute latency and hence even capacity among the tasks of the same bolt.
> 
> I believe, this is because high production rate of the spout task keeps the transfer queue of the worker process congested and hence the emit call of the co-tenant bolt task gets blocked and the execute function waits for an empty space in the queue (I know that there is a separete transfer queue for the bolt task's executor but I think it also gets congested since the transfer queue of the worker process is congested). First of all, is this inference correct? What else might be the main reason?
> 
> I tried throttling the spout with max.spout.pending, and played with the sizes of the queues but could not find the best setting that alleviates the problem.
> I am trying to solve the problem by writing my own pluggable scheduler following this post (http://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/). As done in the post, I just want to schedule the spout task (or tasks) in my scheduler and use the EvenScheduler to schedule the rest of the tasks. However, the question is how can I make sure that EvenScheduler does not schedule tasks to the same worker process that my scheduler assigned the spout tasks? Is there a way to mark a worker process as occupied or full, or limit the number of threads that can be assigned to it?
> If not, and if the only way is to write a complete scheduler, is there a java version of the EvenScheduler that I can reuse?
> Thanks and sorry for the long message.
> -Serkan

Re: Scheduler question: Is there a way to limit the number of threads running on a certain worker process?

[Serkan Uzunbaz <uz...@gmail.com>]

Thanks James, but what I wanted was at the slot (worker process) level not
at the machine level.

Actually, I figured out that calling EvenScheduler's schedule in my
scheduler after I schedule the spout tasks works the way I wanted. If I
assign my spout tasks to some slots, EvenScheduler does not see them as
available and do not assign any other executor to those processes.


On Thu, Dec 5, 2013 at 5:14 PM, James Xu <xu...@gmail.com> wrote:

> (Seems the mailing list have eaten part of my message, resend again)
>
> One api my blog post does not cover is *blacklistHost*:
> https://github.com/nathanmarz/storm/blob/master/storm-core/src/jvm/backtype/storm/scheduler/Cluster.java?source=c#L55 (it
> is introduced after I wrote that blog), with this you can tell the
> EvenScheduler that *host-xxx* is dedicated to run my spout, don’t assign
> anything else to it.
>
> On 2013年12月6日, at 上午9:09, James Xu <xu...@gmail.com> wrote:
>
> One api my blog post does not cover is `blacklist
> On 2013年12月5日, at 上午10:24, Serkan Uzunbaz <uz...@gmail.com> wrote:
>
> Hi all,
> I have a heavy spout task that has a high emission rate and many bolt
> tasks running accross the cluster. I observe that, the bolt tasks that are
> running on the worker process which also runs the heavy spout task have a
> higher execution latency than its peers (the other tasks of the same bolt
> type). My goal is to have an even execute latency and hence even capacity
> among the tasks of the same bolt.
>
> I believe, this is because high production rate of the spout task keeps
> the transfer queue of the worker process congested and hence the emit call
> of the co-tenant bolt task gets blocked and the execute function waits for
> an empty space in the queue (I know that there is a separete transfer queue
> for the bolt task's executor but I think it also gets congested since the
> transfer queue of the worker process is congested). First of all, is this
> inference correct? What else might be the main reason?
>
> I tried throttling the spout with max.spout.pending, and played with the
> sizes of the queues but could not find the best setting that alleviates the
> problem.
> I am trying to solve the problem by writing my own pluggable scheduler
> following this post (
> http://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/).
> As done in the post, I just want to schedule the spout task (or tasks) in
> my scheduler and use the EvenScheduler to schedule the rest of the tasks.
> However, the question is how can I make sure that EvenScheduler does not
> schedule tasks to the same worker process that my scheduler assigned the
> spout tasks? Is there a way to mark a worker process as occupied or full,
> or limit the number of threads that can be assigned to it?
> If not, and if the only way is to write a complete scheduler, is there a
> java version of the EvenScheduler that I can reuse?
> Thanks and sorry for the long message.
> -Serkan
>
>
>
>

Re: Scheduler question: Is there a way to limit the number of threads running on a certain worker process?

[James Xu <xu...@gmail.com>]

(Seems the mailing list have eaten part of my message, resend again)

One api my blog post does not cover is *blacklistHost*: https://github.com/nathanmarz/storm/blob/master/storm-core/src/jvm/backtype/storm/scheduler/Cluster.java?source=c#L55 (it is introduced after I wrote that blog), with this you can tell the EvenScheduler that *host-xxx* is dedicated to run my spout, don’t assign anything else to it.

On 2013年12月6日, at 上午9:09, James Xu <xu...@gmail.com> wrote:

> One api my blog post does not cover is `blacklist
> On 2013年12月5日, at 上午10:24, Serkan Uzunbaz <uz...@gmail.com> wrote:
> 
>> Hi all,
>> I have a heavy spout task that has a high emission rate and many bolt tasks running accross the cluster. I observe that, the bolt tasks that are running on the worker process which also runs the heavy spout task have a higher execution latency than its peers (the other tasks of the same bolt type). My goal is to have an even execute latency and hence even capacity among the tasks of the same bolt.
>> 
>> I believe, this is because high production rate of the spout task keeps the transfer queue of the worker process congested and hence the emit call of the co-tenant bolt task gets blocked and the execute function waits for an empty space in the queue (I know that there is a separete transfer queue for the bolt task's executor but I think it also gets congested since the transfer queue of the worker process is congested). First of all, is this inference correct? What else might be the main reason?
>> 
>> I tried throttling the spout with max.spout.pending, and played with the sizes of the queues but could not find the best setting that alleviates the problem.
>> I am trying to solve the problem by writing my own pluggable scheduler following this post (http://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/). As done in the post, I just want to schedule the spout task (or tasks) in my scheduler and use the EvenScheduler to schedule the rest of the tasks. However, the question is how can I make sure that EvenScheduler does not schedule tasks to the same worker process that my scheduler assigned the spout tasks? Is there a way to mark a worker process as occupied or full, or limit the number of threads that can be assigned to it?
>> If not, and if the only way is to write a complete scheduler, is there a java version of the EvenScheduler that I can reuse?
>> Thanks and sorry for the long message.
>> -Serkan
>