You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Navin Ipe <na...@searchlighthealth.com> on 2017/01/27 09:41:57 UTC
Why aren't storm topology send and receive buffers equal sized?
Hi,
When going through Michael Noll's explanation
<http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/>
on the buffers, I noticed the topology.receive.buffer.size default is 8 and
topology.send.buffer.size default is 1024.
If I have a topology where Spout --> Bolt A ---> BoltB, then if Bolt A
sends a tuple of size 1024, then BoltB won't be able to handle 1024 easily
and will get overwhelmed because Bolt B's receive buffer size is just 8.
Is my above understanding correct? Why are the buffers so different in
sizes?
--
Regards,
Navin
Re: Why aren't storm topology send and receive buffers equal sized?
Posted by Ambud Sharma <as...@gmail.com>.
Arraylist is a resizable data structure, it will resize when more tuples
come in.
More importantly, this is simply a disruptor optimization step to avoid
calling next sequence for each tuple. Rather it will call it once for 8
tuples slots at a time.
On Jan 27, 2017 1:53 AM, "Navin Ipe" <na...@searchlighthealth.com>
wrote:
I found a part of the answer in the article itself "Note that
topology.receiver.buffer.size is in contrast to the other buffer size
related parameters described in this article actually not configuring the
size of an LMAX Disruptor queue. Rather it sets the size of a simple
ArrayList that is used to buffer incoming messages because in this specific
case the data structure does not need to be shared with other threads, i.e.
it is local to the worker’s receive thread."
But the question still remains. If Bolt A sends 1024 tuples, then won't it
overwhelm Bolt B?
On Fri, Jan 27, 2017 at 3:11 PM, Navin Ipe <na...@searchlighthealth.com>
wrote:
> Hi,
>
> When going through Michael Noll's explanation
> <http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/>
> on the buffers, I noticed the topology.receive.buffer.size default is 8 and
> topology.send.buffer.size default is 1024.
>
> If I have a topology where Spout --> Bolt A ---> BoltB, then if Bolt A
> sends a tuple of size 1024, then BoltB won't be able to handle 1024 easily
> and will get overwhelmed because Bolt B's receive buffer size is just 8.
>
> Is my above understanding correct? Why are the buffers so different in
> sizes?
>
> --
> Regards,
> Navin
>
--
Regards,
Navin
Re: Why aren't storm topology send and receive buffers equal sized?
Posted by Navin Ipe <na...@searchlighthealth.com>.
I found a part of the answer in the article itself "Note that
topology.receiver.buffer.size is in contrast to the other buffer size
related parameters described in this article actually not configuring the
size of an LMAX Disruptor queue. Rather it sets the size of a simple
ArrayList that is used to buffer incoming messages because in this specific
case the data structure does not need to be shared with other threads, i.e.
it is local to the worker’s receive thread."
But the question still remains. If Bolt A sends 1024 tuples, then won't it
overwhelm Bolt B?
On Fri, Jan 27, 2017 at 3:11 PM, Navin Ipe <na...@searchlighthealth.com>
wrote:
> Hi,
>
> When going through Michael Noll's explanation
> <http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/>
> on the buffers, I noticed the topology.receive.buffer.size default is 8 and
> topology.send.buffer.size default is 1024.
>
> If I have a topology where Spout --> Bolt A ---> BoltB, then if Bolt A
> sends a tuple of size 1024, then BoltB won't be able to handle 1024 easily
> and will get overwhelmed because Bolt B's receive buffer size is just 8.
>
> Is my above understanding correct? Why are the buffers so different in
> sizes?
>
> --
> Regards,
> Navin
>
--
Regards,
Navin