Transfer tuples from executors queue to worker queue

[Anishek Agarwal <an...@gmail.com>]

Hello,

I would like to understand how the transfer of tuples emitted by the
executors move to the workers transfer queue. From what i understand


topology.executor.send.buffer.size :1024
is the number of tuples, once in the executors send queue, will be
transferred to the workers send queue as a batch.

topology.transfer.buffer.size :1024
is the number of messages sent by the worker over the network in batch.

question is: if i have bolts which take a lot of time to execute, then the
number of messages each executor can process is very low per second, say
about 2 tuples per sec, then would the executor outgoing transfer thread
move messages to worker transfer queue after 1024/2 = 512 seconds ? i have
40 executors per worker.

if not when does it decide to move messages from executor to worker
transfer queue ?

storm :0.9.4

Thanks
Anishek

Re: Transfer tuples from executors queue to worker queue

[Abhishek Agarwal <ab...@gmail.com>]

topology.executor.send.buffer.size is the capacity of the send queue.
Messages are moved around when the transfer thread is ready. It doesn't
wait for a specific number of messages to be available.

If the bolts are always slower than the spout, then increasing the send
queue size isn't going to help much.

On Thu, Dec 24, 2015 at 11:31 AM, Anishek Agarwal <an...@gmail.com> wrote:

> Hello,
>
> I would like to understand how the transfer of tuples emitted by the
> executors move to the workers transfer queue. From what i understand
>
>
> topology.executor.send.buffer.size :1024
> is the number of tuples, once in the executors send queue, will be
> transferred to the workers send queue as a batch.
>
> topology.transfer.buffer.size :1024
> is the number of messages sent by the worker over the network in batch.
>
> question is: if i have bolts which take a lot of time to execute, then the
> number of messages each executor can process is very low per second, say
> about 2 tuples per sec, then would the executor outgoing transfer thread
> move messages to worker transfer queue after 1024/2 = 512 seconds ? i have
> 40 executors per worker.
>
> if not when does it decide to move messages from executor to worker
> transfer queue ?
>
> storm :0.9.4
>
> Thanks
> Anishek
>



-- 
Regards,
Abhishek Agarwal