You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Kashyap Mhaisekar <ka...@gmail.com> on 2015/08/25 05:58:40 UTC
Spout latencies vs Bolt complete latencies
Hi,
How do you interpret if complete latency at spout is high (800ms) but
complete and execute latencies at bolt is very low (4 ms)
My topology has a max pending of 1.
I am emitting from a self emitting spout that emits a message and sleeps
for 100 ms.
Thanks
Kashyap
Re: Spout latencies vs Bolt complete latencies
Posted by Kishore Senji <ks...@gmail.com>.
(1) - Yes it is correct assuming that you can increase parallelism (i.e you
have enough hardware to support it). But once you can increase parallelism
you can increase spout pending as well. So the best way to go about this is
to get the appropriate parallelism for the throughput that you need and
then slowly increase the spout pending from a small number until it does
not affect the throughput any more. (This is actually mentioned in the
FAQ/best practices on Storm website)
(2) - This is true as well. You can figure this out by looking at the
capacity of the bolt. If the capacity is closer to 1, then it means the
bolts are running full speed (i.e a bolt can process A = 1000/X
messages/sec where X is the latency of Bolt in ms. If there are A messages
that the bolt is processing then the capacity will be 1) and if the overall
spout latency is high then it means that there are more messages than what
the bolt can process with the given bolt latency.
You can figure out from the bolt capacity and the overall spout latency
(and your overall topology ie. how many messages are emitted by each
spout/bolt) whether there are messages getting queued up or not.
On Tue, Sep 1, 2015 at 8:52 AM, Kashyap Mhaisekar <ka...@gmail.com>
wrote:
> Kishore,
> Thanks for the effort. Very helpful and much appreciated. Am computing it
> based on your explanation and I find that it sometimes adds up and
> sometimes is way off. two question though -
> 1. If max spout pending is high (or multiple spout instances with low max
> spout pending), then increasing the parallelism should help alleviate
> latency issues
> 2. If the bolt latencies are low but spout latencies are high then the
> main reason would be the waiting of messages for getting processed. In this
> case also, the increase in paralellism should help.
>
> Am I right on the above two points? Also, is there a way I can find out
> the back log of messages at each level?
>
> Thanks
> Kashyap
>
> On Thu, Aug 27, 2015 at 6:07 PM, Kishore Senji <ks...@gmail.com> wrote:
>
>> If the max spout pending is more then the complete latency will increase.
>> You would have to calculate it like the following at every bolt like the
>> following
>>
>> Let us say the max spout pending is 100, the complete latency will
>> increase further. This depends upon the overall throughput you are able to
>> achieve as well out of the topology. With a max spout pending of 100, for
>> the 100 th message to be completely acknowledged, the system has to process
>> the previous 99 messages and then process the 100th message. So 100th
>> message will be waiting in the queue to be processed until all the 99
>> messages are processed. So if you have a good parallelism so that your
>> throughput is high, you can reduce the overall latency, otherwise your
>> throughput will be low and overall higher complete latency.
>>
>> So the calculations with the current parallelism and max spout pending
>> will be (similar to what we had earlier, except that we multiply by 100 and
>> also use the 4 parallel tasks for go1 bolt)
>>
>> = 100 * (3 * (1/4) + (680/40) * (1/8) * 3.853 * (1/2) + (6820/40) *
>> (1/16) * 1.308 * (1/2) + (20600/40) * (1/32) * 0.923 + (137940/40) * (1/64)
>> * 6.017 + (137900/40) * (1/64) * 0.218)
>>
>> = 36251ms
>>
>> I tried to explain what each term means in this gist:
>> https://gist.github.com/ksenji/514d105e069882839316
>>
>> So to reduce the complete latency you have to increase your throughput by
>> increasing parallelism. Sometimes, even though you have good throughput,
>> the complete latency could be high because you have a much bigger spout
>> pending. It all depends on the configuration you choose, so the important
>> metrics are throughput and the spout pending.
>>
>>
>> On Wed, Aug 26, 2015 at 9:43 PM, Kashyap Mhaisekar <ka...@gmail.com>
>> wrote:
>>
>>> Kishore
>>> Just a question on computation. How do the equations change if the max
>>> spout pending is greater than 1?
>>> Also is it fair to say that if the transferred count from parent (one
>>> level higher) is greater than executed count of the bolt in a given period
>>> (10 mins), then the parallelism can be increased? In short, in the topology
>>> I used, go1 parallelism needs to be increased, right?
>>>
>>> Finally, is there a way to know if buffer overflow is occurring by
>>> looking at storm UI?
>>>
>>> Thanks for the time.
>>>
>>> Kashyap
>>>
>>> Thanks
>>> Kashyap
>>> On Aug 26, 2015 17:11, "Kashyap Mhaisekar" <ka...@gmail.com> wrote:
>>>
>>>> Wow Thanks Kishore. Will need to spend some time understanding your
>>>> computation. But what can I do it bring down the latency to two digit
>>>> figures?
>>>>
>>>> Thanks,
>>>> Kashyap
>>>>
>>>> On Wed, Aug 26, 2015 at 4:38 PM, Kishore Senji <ks...@gmail.com>
>>>> wrote:
>>>>
>>>>> Total latency / message ~ go1 + go2 + go3 + split + evaluate + result
>>>>> (with proper normalizations and ignoring go4). This is slightly
>>>>> approximate because you have go3 & split to be in parallel so this
>>>>> calculation below is some approximation.
>>>>>
>>>>> = 3 + (680/40) * (1/8) * 3.853 + (6820/40) * (1/16) * 1.308 +
>>>>> (20600/40) * (1/32) * 0.923 + (137940/40) * (1/64) * 6.017 + (137900/40) *
>>>>> (1/64) * 0.218
>>>>>
>>>>> = 375ms
>>>>>
>>>>> The calculation at a bolt is straight forward, how many messages it is
>>>>> processing relative to 1 message (I normalized it 40 as that is what your
>>>>> go1 bolt executed in last 10m). We also divide by 1/8, 1/16, 1/32 etc as we
>>>>> have that many tasks running in parallel. As you can see the number of
>>>>> messages that a particular section increases the latency would increase
>>>>> because if you have 137940 messages to be processed in the buffer and you
>>>>> are processing them 64 at a time, so there are so many messages queued up
>>>>> to be processed at those stages which adds to the complete latency.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Aug 26, 2015 at 11:38 AM, Kashyap Mhaisekar <
>>>>> kashyap.m@gmail.com> wrote:
>>>>>
>>>>>> Kishore,
>>>>>>
>>>>>> The topology is defined like this --
>>>>>> Spout -> go1 -> go2 -> go3 -> split -> evaluate -> process.
>>>>>> Only thing to note is when one message comes from spout, go1 emits 10
>>>>>> messages and go2 emits 10 more for each message and go 3 emits 10 more each
>>>>>> message it recieves. All this are aggregated in "split" which then sends it
>>>>>> to "evaluate" for computing and finally to "result". All bolts operate with
>>>>>> under 5 ms of process time/execute time. I know it is not a simple sum at
>>>>>> spout level, but somehow 343 ms looks very high.
>>>>>>
>>>>>> UI
>>>>>> [image: Inline image 1]
>>>>>> [image: Inline image 2]
>>>>>> [image: Inline image 3]
>>>>>>
>>>>>> Thanks
>>>>>> kashyap
>>>>>>
>>>>>> On Tue, Aug 25, 2015 at 11:49 PM, Kishore Senji <ks...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> If you max spout pending is 1, then there is no need for the Spout
>>>>>>> to sleep. How many bolts do you have in the topology?
>>>>>>>
>>>>>>> On Tue, Aug 25, 2015 at 2:30 PM, Kashyap Mhaisekar <
>>>>>>> kashyap.m@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi, Any thoughts?
>>>>>>>>
>>>>>>>> On Mon, Aug 24, 2015 at 10:58 PM, Kashyap Mhaisekar <
>>>>>>>> kashyap.m@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> How do you interpret if complete latency at spout is high (800ms)
>>>>>>>>> but complete and execute latencies at bolt is very low (4 ms)
>>>>>>>>> My topology has a max pending of 1.
>>>>>>>>> I am emitting from a self emitting spout that emits a message and
>>>>>>>>> sleeps for 100 ms.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Kashyap
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
Re: Spout latencies vs Bolt complete latencies
Posted by Kashyap Mhaisekar <ka...@gmail.com>.
Kishore,
Thanks for the effort. Very helpful and much appreciated. Am computing it
based on your explanation and I find that it sometimes adds up and
sometimes is way off. two question though -
1. If max spout pending is high (or multiple spout instances with low max
spout pending), then increasing the parallelism should help alleviate
latency issues
2. If the bolt latencies are low but spout latencies are high then the main
reason would be the waiting of messages for getting processed. In this case
also, the increase in paralellism should help.
Am I right on the above two points? Also, is there a way I can find out the
back log of messages at each level?
Thanks
Kashyap
On Thu, Aug 27, 2015 at 6:07 PM, Kishore Senji <ks...@gmail.com> wrote:
> If the max spout pending is more then the complete latency will increase.
> You would have to calculate it like the following at every bolt like the
> following
>
> Let us say the max spout pending is 100, the complete latency will
> increase further. This depends upon the overall throughput you are able to
> achieve as well out of the topology. With a max spout pending of 100, for
> the 100 th message to be completely acknowledged, the system has to process
> the previous 99 messages and then process the 100th message. So 100th
> message will be waiting in the queue to be processed until all the 99
> messages are processed. So if you have a good parallelism so that your
> throughput is high, you can reduce the overall latency, otherwise your
> throughput will be low and overall higher complete latency.
>
> So the calculations with the current parallelism and max spout pending
> will be (similar to what we had earlier, except that we multiply by 100 and
> also use the 4 parallel tasks for go1 bolt)
>
> = 100 * (3 * (1/4) + (680/40) * (1/8) * 3.853 * (1/2) + (6820/40) * (1/16)
> * 1.308 * (1/2) + (20600/40) * (1/32) * 0.923 + (137940/40) * (1/64) *
> 6.017 + (137900/40) * (1/64) * 0.218)
>
> = 36251ms
>
> I tried to explain what each term means in this gist:
> https://gist.github.com/ksenji/514d105e069882839316
>
> So to reduce the complete latency you have to increase your throughput by
> increasing parallelism. Sometimes, even though you have good throughput,
> the complete latency could be high because you have a much bigger spout
> pending. It all depends on the configuration you choose, so the important
> metrics are throughput and the spout pending.
>
>
> On Wed, Aug 26, 2015 at 9:43 PM, Kashyap Mhaisekar <ka...@gmail.com>
> wrote:
>
>> Kishore
>> Just a question on computation. How do the equations change if the max
>> spout pending is greater than 1?
>> Also is it fair to say that if the transferred count from parent (one
>> level higher) is greater than executed count of the bolt in a given period
>> (10 mins), then the parallelism can be increased? In short, in the topology
>> I used, go1 parallelism needs to be increased, right?
>>
>> Finally, is there a way to know if buffer overflow is occurring by
>> looking at storm UI?
>>
>> Thanks for the time.
>>
>> Kashyap
>>
>> Thanks
>> Kashyap
>> On Aug 26, 2015 17:11, "Kashyap Mhaisekar" <ka...@gmail.com> wrote:
>>
>>> Wow Thanks Kishore. Will need to spend some time understanding your
>>> computation. But what can I do it bring down the latency to two digit
>>> figures?
>>>
>>> Thanks,
>>> Kashyap
>>>
>>> On Wed, Aug 26, 2015 at 4:38 PM, Kishore Senji <ks...@gmail.com> wrote:
>>>
>>>> Total latency / message ~ go1 + go2 + go3 + split + evaluate + result
>>>> (with proper normalizations and ignoring go4). This is slightly
>>>> approximate because you have go3 & split to be in parallel so this
>>>> calculation below is some approximation.
>>>>
>>>> = 3 + (680/40) * (1/8) * 3.853 + (6820/40) * (1/16) * 1.308 +
>>>> (20600/40) * (1/32) * 0.923 + (137940/40) * (1/64) * 6.017 + (137900/40) *
>>>> (1/64) * 0.218
>>>>
>>>> = 375ms
>>>>
>>>> The calculation at a bolt is straight forward, how many messages it is
>>>> processing relative to 1 message (I normalized it 40 as that is what your
>>>> go1 bolt executed in last 10m). We also divide by 1/8, 1/16, 1/32 etc as we
>>>> have that many tasks running in parallel. As you can see the number of
>>>> messages that a particular section increases the latency would increase
>>>> because if you have 137940 messages to be processed in the buffer and you
>>>> are processing them 64 at a time, so there are so many messages queued up
>>>> to be processed at those stages which adds to the complete latency.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Aug 26, 2015 at 11:38 AM, Kashyap Mhaisekar <
>>>> kashyap.m@gmail.com> wrote:
>>>>
>>>>> Kishore,
>>>>>
>>>>> The topology is defined like this --
>>>>> Spout -> go1 -> go2 -> go3 -> split -> evaluate -> process.
>>>>> Only thing to note is when one message comes from spout, go1 emits 10
>>>>> messages and go2 emits 10 more for each message and go 3 emits 10 more each
>>>>> message it recieves. All this are aggregated in "split" which then sends it
>>>>> to "evaluate" for computing and finally to "result". All bolts operate with
>>>>> under 5 ms of process time/execute time. I know it is not a simple sum at
>>>>> spout level, but somehow 343 ms looks very high.
>>>>>
>>>>> UI
>>>>> [image: Inline image 1]
>>>>> [image: Inline image 2]
>>>>> [image: Inline image 3]
>>>>>
>>>>> Thanks
>>>>> kashyap
>>>>>
>>>>> On Tue, Aug 25, 2015 at 11:49 PM, Kishore Senji <ks...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> If you max spout pending is 1, then there is no need for the Spout to
>>>>>> sleep. How many bolts do you have in the topology?
>>>>>>
>>>>>> On Tue, Aug 25, 2015 at 2:30 PM, Kashyap Mhaisekar <
>>>>>> kashyap.m@gmail.com> wrote:
>>>>>>
>>>>>>> Hi, Any thoughts?
>>>>>>>
>>>>>>> On Mon, Aug 24, 2015 at 10:58 PM, Kashyap Mhaisekar <
>>>>>>> kashyap.m@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> How do you interpret if complete latency at spout is high (800ms)
>>>>>>>> but complete and execute latencies at bolt is very low (4 ms)
>>>>>>>> My topology has a max pending of 1.
>>>>>>>> I am emitting from a self emitting spout that emits a message and
>>>>>>>> sleeps for 100 ms.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Kashyap
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
Re: Spout latencies vs Bolt complete latencies
Posted by Kishore Senji <ks...@gmail.com>.
If the max spout pending is more then the complete latency will increase.
You would have to calculate it like the following at every bolt like the
following
Let us say the max spout pending is 100, the complete latency will increase
further. This depends upon the overall throughput you are able to achieve
as well out of the topology. With a max spout pending of 100, for the 100
th message to be completely acknowledged, the system has to process the
previous 99 messages and then process the 100th message. So 100th message
will be waiting in the queue to be processed until all the 99 messages are
processed. So if you have a good parallelism so that your throughput is
high, you can reduce the overall latency, otherwise your throughput will be
low and overall higher complete latency.
So the calculations with the current parallelism and max spout pending will
be (similar to what we had earlier, except that we multiply by 100 and also
use the 4 parallel tasks for go1 bolt)
= 100 * (3 * (1/4) + (680/40) * (1/8) * 3.853 * (1/2) + (6820/40) * (1/16)
* 1.308 * (1/2) + (20600/40) * (1/32) * 0.923 + (137940/40) * (1/64) *
6.017 + (137900/40) * (1/64) * 0.218)
= 36251ms
I tried to explain what each term means in this gist:
https://gist.github.com/ksenji/514d105e069882839316
So to reduce the complete latency you have to increase your throughput by
increasing parallelism. Sometimes, even though you have good throughput,
the complete latency could be high because you have a much bigger spout
pending. It all depends on the configuration you choose, so the important
metrics are throughput and the spout pending.
On Wed, Aug 26, 2015 at 9:43 PM, Kashyap Mhaisekar <ka...@gmail.com>
wrote:
> Kishore
> Just a question on computation. How do the equations change if the max
> spout pending is greater than 1?
> Also is it fair to say that if the transferred count from parent (one
> level higher) is greater than executed count of the bolt in a given period
> (10 mins), then the parallelism can be increased? In short, in the topology
> I used, go1 parallelism needs to be increased, right?
>
> Finally, is there a way to know if buffer overflow is occurring by looking
> at storm UI?
>
> Thanks for the time.
>
> Kashyap
>
> Thanks
> Kashyap
> On Aug 26, 2015 17:11, "Kashyap Mhaisekar" <ka...@gmail.com> wrote:
>
>> Wow Thanks Kishore. Will need to spend some time understanding your
>> computation. But what can I do it bring down the latency to two digit
>> figures?
>>
>> Thanks,
>> Kashyap
>>
>> On Wed, Aug 26, 2015 at 4:38 PM, Kishore Senji <ks...@gmail.com> wrote:
>>
>>> Total latency / message ~ go1 + go2 + go3 + split + evaluate + result
>>> (with proper normalizations and ignoring go4). This is slightly
>>> approximate because you have go3 & split to be in parallel so this
>>> calculation below is some approximation.
>>>
>>> = 3 + (680/40) * (1/8) * 3.853 + (6820/40) * (1/16) * 1.308 + (20600/40)
>>> * (1/32) * 0.923 + (137940/40) * (1/64) * 6.017 + (137900/40) * (1/64) *
>>> 0.218
>>>
>>> = 375ms
>>>
>>> The calculation at a bolt is straight forward, how many messages it is
>>> processing relative to 1 message (I normalized it 40 as that is what your
>>> go1 bolt executed in last 10m). We also divide by 1/8, 1/16, 1/32 etc as we
>>> have that many tasks running in parallel. As you can see the number of
>>> messages that a particular section increases the latency would increase
>>> because if you have 137940 messages to be processed in the buffer and you
>>> are processing them 64 at a time, so there are so many messages queued up
>>> to be processed at those stages which adds to the complete latency.
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Aug 26, 2015 at 11:38 AM, Kashyap Mhaisekar <kashyap.m@gmail.com
>>> > wrote:
>>>
>>>> Kishore,
>>>>
>>>> The topology is defined like this --
>>>> Spout -> go1 -> go2 -> go3 -> split -> evaluate -> process.
>>>> Only thing to note is when one message comes from spout, go1 emits 10
>>>> messages and go2 emits 10 more for each message and go 3 emits 10 more each
>>>> message it recieves. All this are aggregated in "split" which then sends it
>>>> to "evaluate" for computing and finally to "result". All bolts operate with
>>>> under 5 ms of process time/execute time. I know it is not a simple sum at
>>>> spout level, but somehow 343 ms looks very high.
>>>>
>>>> UI
>>>> [image: Inline image 1]
>>>> [image: Inline image 2]
>>>> [image: Inline image 3]
>>>>
>>>> Thanks
>>>> kashyap
>>>>
>>>> On Tue, Aug 25, 2015 at 11:49 PM, Kishore Senji <ks...@gmail.com>
>>>> wrote:
>>>>
>>>>> If you max spout pending is 1, then there is no need for the Spout to
>>>>> sleep. How many bolts do you have in the topology?
>>>>>
>>>>> On Tue, Aug 25, 2015 at 2:30 PM, Kashyap Mhaisekar <
>>>>> kashyap.m@gmail.com> wrote:
>>>>>
>>>>>> Hi, Any thoughts?
>>>>>>
>>>>>> On Mon, Aug 24, 2015 at 10:58 PM, Kashyap Mhaisekar <
>>>>>> kashyap.m@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> How do you interpret if complete latency at spout is high (800ms)
>>>>>>> but complete and execute latencies at bolt is very low (4 ms)
>>>>>>> My topology has a max pending of 1.
>>>>>>> I am emitting from a self emitting spout that emits a message and
>>>>>>> sleeps for 100 ms.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Kashyap
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: Spout latencies vs Bolt complete latencies
Posted by Kashyap Mhaisekar <ka...@gmail.com>.
Kishore
Just a question on computation. How do the equations change if the max
spout pending is greater than 1?
Also is it fair to say that if the transferred count from parent (one level
higher) is greater than executed count of the bolt in a given period (10
mins), then the parallelism can be increased? In short, in the topology I
used, go1 parallelism needs to be increased, right?
Finally, is there a way to know if buffer overflow is occurring by looking
at storm UI?
Thanks for the time.
Kashyap
Thanks
Kashyap
On Aug 26, 2015 17:11, "Kashyap Mhaisekar" <ka...@gmail.com> wrote:
> Wow Thanks Kishore. Will need to spend some time understanding your
> computation. But what can I do it bring down the latency to two digit
> figures?
>
> Thanks,
> Kashyap
>
> On Wed, Aug 26, 2015 at 4:38 PM, Kishore Senji <ks...@gmail.com> wrote:
>
>> Total latency / message ~ go1 + go2 + go3 + split + evaluate + result
>> (with proper normalizations and ignoring go4). This is slightly
>> approximate because you have go3 & split to be in parallel so this
>> calculation below is some approximation.
>>
>> = 3 + (680/40) * (1/8) * 3.853 + (6820/40) * (1/16) * 1.308 + (20600/40)
>> * (1/32) * 0.923 + (137940/40) * (1/64) * 6.017 + (137900/40) * (1/64) *
>> 0.218
>>
>> = 375ms
>>
>> The calculation at a bolt is straight forward, how many messages it is
>> processing relative to 1 message (I normalized it 40 as that is what your
>> go1 bolt executed in last 10m). We also divide by 1/8, 1/16, 1/32 etc as we
>> have that many tasks running in parallel. As you can see the number of
>> messages that a particular section increases the latency would increase
>> because if you have 137940 messages to be processed in the buffer and you
>> are processing them 64 at a time, so there are so many messages queued up
>> to be processed at those stages which adds to the complete latency.
>>
>>
>>
>>
>>
>> On Wed, Aug 26, 2015 at 11:38 AM, Kashyap Mhaisekar <ka...@gmail.com>
>> wrote:
>>
>>> Kishore,
>>>
>>> The topology is defined like this --
>>> Spout -> go1 -> go2 -> go3 -> split -> evaluate -> process.
>>> Only thing to note is when one message comes from spout, go1 emits 10
>>> messages and go2 emits 10 more for each message and go 3 emits 10 more each
>>> message it recieves. All this are aggregated in "split" which then sends it
>>> to "evaluate" for computing and finally to "result". All bolts operate with
>>> under 5 ms of process time/execute time. I know it is not a simple sum at
>>> spout level, but somehow 343 ms looks very high.
>>>
>>> UI
>>> [image: Inline image 1]
>>> [image: Inline image 2]
>>> [image: Inline image 3]
>>>
>>> Thanks
>>> kashyap
>>>
>>> On Tue, Aug 25, 2015 at 11:49 PM, Kishore Senji <ks...@gmail.com>
>>> wrote:
>>>
>>>> If you max spout pending is 1, then there is no need for the Spout to
>>>> sleep. How many bolts do you have in the topology?
>>>>
>>>> On Tue, Aug 25, 2015 at 2:30 PM, Kashyap Mhaisekar <kashyap.m@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi, Any thoughts?
>>>>>
>>>>> On Mon, Aug 24, 2015 at 10:58 PM, Kashyap Mhaisekar <
>>>>> kashyap.m@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>> How do you interpret if complete latency at spout is high (800ms) but
>>>>>> complete and execute latencies at bolt is very low (4 ms)
>>>>>> My topology has a max pending of 1.
>>>>>> I am emitting from a self emitting spout that emits a message and
>>>>>> sleeps for 100 ms.
>>>>>>
>>>>>> Thanks
>>>>>> Kashyap
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Spout latencies vs Bolt complete latencies
Posted by Kashyap Mhaisekar <ka...@gmail.com>.
Wow Thanks Kishore. Will need to spend some time understanding your
computation. But what can I do it bring down the latency to two digit
figures?
Thanks,
Kashyap
On Wed, Aug 26, 2015 at 4:38 PM, Kishore Senji <ks...@gmail.com> wrote:
> Total latency / message ~ go1 + go2 + go3 + split + evaluate + result
> (with proper normalizations and ignoring go4). This is slightly
> approximate because you have go3 & split to be in parallel so this
> calculation below is some approximation.
>
> = 3 + (680/40) * (1/8) * 3.853 + (6820/40) * (1/16) * 1.308 + (20600/40) *
> (1/32) * 0.923 + (137940/40) * (1/64) * 6.017 + (137900/40) * (1/64) * 0.218
>
> = 375ms
>
> The calculation at a bolt is straight forward, how many messages it is
> processing relative to 1 message (I normalized it 40 as that is what your
> go1 bolt executed in last 10m). We also divide by 1/8, 1/16, 1/32 etc as we
> have that many tasks running in parallel. As you can see the number of
> messages that a particular section increases the latency would increase
> because if you have 137940 messages to be processed in the buffer and you
> are processing them 64 at a time, so there are so many messages queued up
> to be processed at those stages which adds to the complete latency.
>
>
>
>
>
> On Wed, Aug 26, 2015 at 11:38 AM, Kashyap Mhaisekar <ka...@gmail.com>
> wrote:
>
>> Kishore,
>>
>> The topology is defined like this --
>> Spout -> go1 -> go2 -> go3 -> split -> evaluate -> process.
>> Only thing to note is when one message comes from spout, go1 emits 10
>> messages and go2 emits 10 more for each message and go 3 emits 10 more each
>> message it recieves. All this are aggregated in "split" which then sends it
>> to "evaluate" for computing and finally to "result". All bolts operate with
>> under 5 ms of process time/execute time. I know it is not a simple sum at
>> spout level, but somehow 343 ms looks very high.
>>
>> UI
>> [image: Inline image 1]
>> [image: Inline image 2]
>> [image: Inline image 3]
>>
>> Thanks
>> kashyap
>>
>> On Tue, Aug 25, 2015 at 11:49 PM, Kishore Senji <ks...@gmail.com> wrote:
>>
>>> If you max spout pending is 1, then there is no need for the Spout to
>>> sleep. How many bolts do you have in the topology?
>>>
>>> On Tue, Aug 25, 2015 at 2:30 PM, Kashyap Mhaisekar <ka...@gmail.com>
>>> wrote:
>>>
>>>> Hi, Any thoughts?
>>>>
>>>> On Mon, Aug 24, 2015 at 10:58 PM, Kashyap Mhaisekar <
>>>> kashyap.m@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> How do you interpret if complete latency at spout is high (800ms) but
>>>>> complete and execute latencies at bolt is very low (4 ms)
>>>>> My topology has a max pending of 1.
>>>>> I am emitting from a self emitting spout that emits a message and
>>>>> sleeps for 100 ms.
>>>>>
>>>>> Thanks
>>>>> Kashyap
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Spout latencies vs Bolt complete latencies
Posted by Kishore Senji <ks...@gmail.com>.
Total latency / message ~ go1 + go2 + go3 + split + evaluate + result
(with proper normalizations and ignoring go4). This is slightly
approximate because you have go3 & split to be in parallel so this
calculation below is some approximation.
= 3 + (680/40) * (1/8) * 3.853 + (6820/40) * (1/16) * 1.308 + (20600/40) *
(1/32) * 0.923 + (137940/40) * (1/64) * 6.017 + (137900/40) * (1/64) * 0.218
= 375ms
The calculation at a bolt is straight forward, how many messages it is
processing relative to 1 message (I normalized it 40 as that is what your
go1 bolt executed in last 10m). We also divide by 1/8, 1/16, 1/32 etc as we
have that many tasks running in parallel. As you can see the number of
messages that a particular section increases the latency would increase
because if you have 137940 messages to be processed in the buffer and you
are processing them 64 at a time, so there are so many messages queued up
to be processed at those stages which adds to the complete latency.
On Wed, Aug 26, 2015 at 11:38 AM, Kashyap Mhaisekar <ka...@gmail.com>
wrote:
> Kishore,
>
> The topology is defined like this --
> Spout -> go1 -> go2 -> go3 -> split -> evaluate -> process.
> Only thing to note is when one message comes from spout, go1 emits 10
> messages and go2 emits 10 more for each message and go 3 emits 10 more each
> message it recieves. All this are aggregated in "split" which then sends it
> to "evaluate" for computing and finally to "result". All bolts operate with
> under 5 ms of process time/execute time. I know it is not a simple sum at
> spout level, but somehow 343 ms looks very high.
>
> UI
> [image: Inline image 1]
> [image: Inline image 2]
> [image: Inline image 3]
>
> Thanks
> kashyap
>
> On Tue, Aug 25, 2015 at 11:49 PM, Kishore Senji <ks...@gmail.com> wrote:
>
>> If you max spout pending is 1, then there is no need for the Spout to
>> sleep. How many bolts do you have in the topology?
>>
>> On Tue, Aug 25, 2015 at 2:30 PM, Kashyap Mhaisekar <ka...@gmail.com>
>> wrote:
>>
>>> Hi, Any thoughts?
>>>
>>> On Mon, Aug 24, 2015 at 10:58 PM, Kashyap Mhaisekar <kashyap.m@gmail.com
>>> > wrote:
>>>
>>>> Hi,
>>>> How do you interpret if complete latency at spout is high (800ms) but
>>>> complete and execute latencies at bolt is very low (4 ms)
>>>> My topology has a max pending of 1.
>>>> I am emitting from a self emitting spout that emits a message and
>>>> sleeps for 100 ms.
>>>>
>>>> Thanks
>>>> Kashyap
>>>>
>>>
>>>
>>
>
Re: Spout latencies vs Bolt complete latencies
Posted by Kashyap Mhaisekar <ka...@gmail.com>.
Kishore,
The topology is defined like this --
Spout -> go1 -> go2 -> go3 -> split -> evaluate -> process.
Only thing to note is when one message comes from spout, go1 emits 10
messages and go2 emits 10 more for each message and go 3 emits 10 more each
message it recieves. All this are aggregated in "split" which then sends it
to "evaluate" for computing and finally to "result". All bolts operate with
under 5 ms of process time/execute time. I know it is not a simple sum at
spout level, but somehow 343 ms looks very high.
UI
[image: Inline image 1]
[image: Inline image 2]
[image: Inline image 3]
Thanks
kashyap
On Tue, Aug 25, 2015 at 11:49 PM, Kishore Senji <ks...@gmail.com> wrote:
> If you max spout pending is 1, then there is no need for the Spout to
> sleep. How many bolts do you have in the topology?
>
> On Tue, Aug 25, 2015 at 2:30 PM, Kashyap Mhaisekar <ka...@gmail.com>
> wrote:
>
>> Hi, Any thoughts?
>>
>> On Mon, Aug 24, 2015 at 10:58 PM, Kashyap Mhaisekar <ka...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> How do you interpret if complete latency at spout is high (800ms) but
>>> complete and execute latencies at bolt is very low (4 ms)
>>> My topology has a max pending of 1.
>>> I am emitting from a self emitting spout that emits a message and sleeps
>>> for 100 ms.
>>>
>>> Thanks
>>> Kashyap
>>>
>>
>>
>
Re: Spout latencies vs Bolt complete latencies
Posted by Kishore Senji <ks...@gmail.com>.
If you max spout pending is 1, then there is no need for the Spout to
sleep. How many bolts do you have in the topology?
On Tue, Aug 25, 2015 at 2:30 PM, Kashyap Mhaisekar <ka...@gmail.com>
wrote:
> Hi, Any thoughts?
>
> On Mon, Aug 24, 2015 at 10:58 PM, Kashyap Mhaisekar <ka...@gmail.com>
> wrote:
>
>> Hi,
>> How do you interpret if complete latency at spout is high (800ms) but
>> complete and execute latencies at bolt is very low (4 ms)
>> My topology has a max pending of 1.
>> I am emitting from a self emitting spout that emits a message and sleeps
>> for 100 ms.
>>
>> Thanks
>> Kashyap
>>
>
>
Re: Spout latencies vs Bolt complete latencies
Posted by Kashyap Mhaisekar <ka...@gmail.com>.
Hi, Any thoughts?
On Mon, Aug 24, 2015 at 10:58 PM, Kashyap Mhaisekar <ka...@gmail.com>
wrote:
> Hi,
> How do you interpret if complete latency at spout is high (800ms) but
> complete and execute latencies at bolt is very low (4 ms)
> My topology has a max pending of 1.
> I am emitting from a self emitting spout that emits a message and sleeps
> for 100 ms.
>
> Thanks
> Kashyap
>