[help]storm spout complete latency very high

["daxwang (王文军)" <da...@tencent.com>]

in my storm topology, spouts have very high complete latency 28794.477(ms)，but  bolt capacity(last 10m) and execute latency not high,  i start backpressure with default settings. and  set  topology.max.spout.pending=256, in my topology the size of one tuple is about 1024KB,  so what's wrong?  help please.

in my topology:
storm version: 1.0.2
worker : 10
worker mem: 1G
spout: 40 executor
bolt: 10 executor

[cid:_Foxmail.1@aa476a7a-ca15-22be-2f37-5b2d4d09a585]

Re: [help]storm spout complete latency very high

[Gireesh Ramji <gi...@yahoo.com>]

Have you checked the receive queue population of your bolts? If the executor receive buffer size is 1024, say, and those buffers are full (or even non empty), that will add to the complete latency of the tuples in the buffer without having any impact on the execute latency of the bolt. This gets compounded across the 40 bolt executors that you have so there are potentially 40k tuples for which the complete latency is much higher than you anticipate based on the execute latency.
In reality since you have set max.spout.pending to 256, you probably have a situation where each of your 40 bolt executors have 256 tuples in their buffer, so there are 10k tuples for which the complete latency is amiss and this is contributing the discrepancy you see in the spout complete latency.
Gireesh

      From: daxwang(王文军) <da...@tencent.com>
 To: user <us...@storm.apache.org> 
 Sent: Thursday, February 23, 2017 12:06 PM
 Subject: [help]storm spout complete latency very high
   
 #yiv3716946653 body {line-height:1.5;}#yiv3716946653 blockquote {margin-top:0px;margin-bottom:0px;margin-left:0.5em;}#yiv3716946653 div.yiv3716946653foxdiv20170223120554122386 {}#yiv3716946653 body {font-size:10.5pt;color:rgb(0, 0, 0);line-height:1.5;}


in my storm topology, spouts have very high complete latency 28794.477(ms)，but  bolt capacity(last 10m) and execute latency not high,  i start backpressure with default settings. and  set  topology.max.spout.pending=256, in my topology the size of one tuple is about 1024KB,  so what's wrong?  help please.
in my topology:storm version: 1.0.2worker : 10worker mem: 1Gspout: 40 executorbolt: 10 executor






   

Re: [help]storm spout complete latency very high

[Jungtaek Lim <ka...@gmail.com>]

Would you mind trying out Storm 1.0.3 which contains STORM-1742
<http://issues.apache.org/jira/browse/STORM-1742> and see it helps?

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 2월 23일 (목) 오후 1:06, daxwang(王文军) <da...@tencent.com>님이 작성:

> in my storm topology, spouts have very high complete latency 28794.477(ms)，but
>  bolt capacity(last 10m) and execute latency not high,  i start
> backpressure with default settings. and  set  topology.max.spout.pending=256,
> in my topology the size of one tuple is about 1024KB,  so what's wrong?
>  help please.
>
> in my topology:
> storm version: 1.0.2
> worker : 10
> worker mem: 1G
> spout: 40 executor
> bolt: 10 executor
>
> [image: Catch(02-23-11-1(02-23-12-05-54).jpg]
>
>