You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by 王 纯超 <wa...@outlook.com> on 2017/08/07 02:57:44 UTC
How to Improve Storm Application's Throughput
Hi,
I am now considering improve a Storm application's throughput because I find that the consumption speed of KafkaSpout is slower than the producing speed. And the lag gets larger and larger. Below is the bolt statistics. I tried to bring forward the tuple projection and filtering logic in a custom scheme with intention of reducing network traffic. However, after observation, things go contrary to my wishes. Am I going the wrong way? Are there any principles tuning Storm applications? Or could anyone give some suggestions for this specific case?
[cid:_Foxmail.1@e1f0b071-9726-10be-1fe4-bc642c9456f9]
________________________________
wangchunchao@outlook.com
回复: RE: How to Improve Storm Application's Throughput
Posted by "fanxinpu@travelsky.com" <fa...@travelsky.com>.
Hi LiBo, Jungtaek ：
Yes, storm tunning depends on situations. Thank u for your kindly advice.
The follow is one of my situations, any hints from you will be appreciated.
The topology just has a spout which reads message from kafka and a bolt to parse the message and put it into the hbase.
As you can see from the above picture, the Execute latency of the bolt is small(0.5ms), but the Complete latency is much more larger(4365ms), so as to slow down the throughput of the topology.
Which part will consume so much additional time? the transfer between the spout and the bolt ? or the ack part? I tried to increase parallelism for the component, but it did not work.
Is there a tool to analyze the time consumption in general? It will be a great news to know it.
There is another thing to explain in the above picture. It seems that the Capacity is high as 1.617, but there are 64 bolts, most Capacity of it is low, as picture below shows.
So, another puzzle is both the Execute latency and the Executed is about equal, but the Capacity turns out to be so much different. Any hints?
The follow is another topology.
Maybe the history_Put bolt has both high Capacity and larger Execute latency, this would definitely lead to the Complete latency as 56964ms?
Thank you all for your time.
Joshua
发件人： 李波
发送时间： 2017-08-07 16:56
收件人： user@storm.apache.org
抄送： jiangyc_cui_gd@si-tech.com.cn; 'zhangxl_bds'
主题： RE: How to Improve Storm Application's Throughput
你好！
Storm的性能排查过程需要不断的尝试最后达到一个经验值，我个人的排查过程如下，希望可以有一些帮助：
1、Kafka PartitionNumber and KafkaSpout’s parallelism
首先你要确认是KafkaSpout的接收能力不行导致的延迟，还是由于后续的bolt处理能力有限造成拥堵，导致的上游KafkaSpout也一起拥堵造成的延迟
KafkaSpout接收能力不行的话，需要增加Kakfa的分区数量，同时把KafkaSpout的并行度设置为和分区数量一致，这样逐步提升以达到吞吐要求
2、Bolt’s business logical and Bolt’s parallelism
首先看下是不是硬件资源不行了。。
其次，需要查出拥堵在哪一个bolt造成的拥堵，可以配合Storm的那个动态图来判断越红拥堵越厉害，同时参考Capacity这个值来判断超过1的Bolt或多或少会出现拥堵的情况
根据自己的算法来优化处理逻辑提升效率，或者通过增加Bolt并行度的方式来提升Bolt处理能力（前提是硬件资源没有到上限）
从你的这个业务来看应该是后续的几个Bolt要访问外部的数据存储系统进行最终结果的存储，关注一下存取数据是否延迟较大，目标系统压力比较大
多数情况都是由于Bolt处理能力不够造成的，需要找出压力点优化业务处理逻辑和同时调整Bolt并行度
关注下bolt的逻辑是计算密集型还是以来外部io，计算密集型只能增加worker
3、Config.TOPOLOGY_BACKPRESSURE_ENABLE
是否开启了反压机制Config.TOPOLOGY_BACKPRESSURE_ENABLE (好像是storm1.0.0以后才有的)
4、Netty
另外要想提升Storm在底层传输上的吞吐量，可以通过修改storm.yaml的netty配置，来提升netty的发送批量大
5、Executor‘s throughput params
// Net io set
config.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 1024 * 16); // default is 1024
config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 1024 * 16);// batched; default is 1024
config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 1024 * 16); // individual tuples; default is 1024
config.put(Config.TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS, 200);
李波 13813887096 libo19@asiainfo.com 北京亚信智慧数据科技有限公司
亚信是我家 发展靠大家
From: 王 纯超 [mailto:wangchunchao@outlook.com]
Sent: 2017年8月7日 10:58
To: user <us...@storm.apache.org>
Cc: 姜艳春jiangyc_cui_gd@si-tech.com.cn <ji...@si-tech.com.cn>; zhangxl_bds <zh...@si-tech.com.cn>
Subject: How to Improve Storm Application's Throughput
Hi,
I am now considering improve a Storm application's throughput because I find that the consumption speed of KafkaSpout is slower than the producing speed. And the lag gets larger and larger. Below is the bolt statistics. I tried to bring forward the tuple projection and filtering logic in a custom scheme with intention of reducing network traffic. However, after observation, things go contrary to my wishes. Am I going the wrong way? Are there any principles tuning Storm applications? Or could anyone give some suggestions for this specific case?
wangchunchao@outlook.com
回复: RE: How to Improve Storm Application's Throughput
Posted by "fanxinpu@travelsky.com" <fa...@travelsky.com>.
Hi LiBo, Jungtaek ：
Yes, storm tunning depends on situations. Thank u for your kindly advice.
The follow is one of my situations, any hints from you will be appreciated. The storm version is 1.0.0.
The topology just has a spout which reads message from kafka and a bolt to parse the message and put it into the hbase.
As you can see from the above picture, the Execute latency of the bolt is small(0.5ms), but the Complete latency is much more larger(4365ms), so as to slow down the throughput of the topology.
Which part will consume so much additional time? the transfer between the spout and the bolt ? or the ack part? I tried to increase parallelism for the component, but it did not work.
Is there a tool to analyze the time consumption in general? It will be a great news to know it.
There is another thing to explain in the above picture. It seems that the Capacity is high as 1.617, but there are 64 bolts, most Capacity of it is low, as picture below shows.
So, another puzzle is both the Execute latency and the Executed is about equal, but the Capacity turns out to be so much different. Any hints?
The follow is another topology.
Maybe the history_Put bolt has both high Capacity and larger Execute latency, this would definitely lead to the Complete latency as 56964ms?
Thank you all for your time.
Joshua
发件人： 李波
发送时间： 2017-08-07 16:56
收件人： user@storm.apache.org
抄送： jiangyc_cui_gd@si-tech.com.cn; 'zhangxl_bds'
主题： RE: How to Improve Storm Application's Throughput
你好！
Storm的性能排查过程需要不断的尝试最后达到一个经验值，我个人的排查过程如下，希望可以有一些帮助：
1、Kafka PartitionNumber and KafkaSpout’s parallelism
首先你要确认是KafkaSpout的接收能力不行导致的延迟，还是由于后续的bolt处理能力有限造成拥堵，导致的上游KafkaSpout也一起拥堵造成的延迟
KafkaSpout接收能力不行的话，需要增加Kakfa的分区数量，同时把KafkaSpout的并行度设置为和分区数量一致，这样逐步提升以达到吞吐要求
2、Bolt’s business logical and Bolt’s parallelism
首先看下是不是硬件资源不行了。。
其次，需要查出拥堵在哪一个bolt造成的拥堵，可以配合Storm的那个动态图来判断越红拥堵越厉害，同时参考Capacity这个值来判断超过1的Bolt或多或少会出现拥堵的情况
根据自己的算法来优化处理逻辑提升效率，或者通过增加Bolt并行度的方式来提升Bolt处理能力（前提是硬件资源没有到上限）
从你的这个业务来看应该是后续的几个Bolt要访问外部的数据存储系统进行最终结果的存储，关注一下存取数据是否延迟较大，目标系统压力比较大
多数情况都是由于Bolt处理能力不够造成的，需要找出压力点优化业务处理逻辑和同时调整Bolt并行度
关注下bolt的逻辑是计算密集型还是以来外部io，计算密集型只能增加worker
3、Config.TOPOLOGY_BACKPRESSURE_ENABLE
是否开启了反压机制Config.TOPOLOGY_BACKPRESSURE_ENABLE (好像是storm1.0.0以后才有的)
4、Netty
另外要想提升Storm在底层传输上的吞吐量，可以通过修改storm.yaml的netty配置，来提升netty的发送批量大
5、Executor‘s throughput params
// Net io set
config.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 1024 * 16); // default is 1024
config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 1024 * 16);// batched; default is 1024
config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 1024 * 16); // individual tuples; default is 1024
config.put(Config.TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS, 200);
李波 13813887096 libo19@asiainfo.com 北京亚信智慧数据科技有限公司
亚信是我家 发展靠大家
From: 王 纯超 [mailto:wangchunchao@outlook.com]
Sent: 2017年8月7日 10:58
To: user <us...@storm.apache.org>
Cc: 姜艳春jiangyc_cui_gd@si-tech.com.cn <ji...@si-tech.com.cn>; zhangxl_bds <zh...@si-tech.com.cn>
Subject: How to Improve Storm Application's Throughput
Hi,
I am now considering improve a Storm application's throughput because I find that the consumption speed of KafkaSpout is slower than the producing speed. And the lag gets larger and larger. Below is the bolt statistics. I tried to bring forward the tuple projection and filtering logic in a custom scheme with intention of reducing network traffic. However, after observation, things go contrary to my wishes. Am I going the wrong way? Are there any principles tuning Storm applications? Or could anyone give some suggestions for this specific case?
wangchunchao@outlook.com
RE: How to Improve Storm Application's Throughput
Posted by 李波 <li...@asiainfo.com>.
你好！
Storm的性能排查过程需要不断的尝试最后达到一个经验值，我个人的排查过程如下，
希望可以有一些帮助：
1、Kafka PartitionNumber and KafkaSpout’s parallelism
首先你要确认是KafkaSpout的接收能力不行导致的延迟，还是由于后续的bolt处理能力
有限造成拥堵，导致的上游KafkaSpout也一起拥堵造成的延迟
KafkaSpout接收能力不行的话，需要增加Kakfa的分区数量，同时把KafkaSpout的并行
度设置为和分区数量一致，这样逐步提升以达到吞吐要求
2、Bolt’s business logical and Bolt’s parallelism
首先看下是不是硬件资源不行了。。
其次，需要查出拥堵在哪一个bolt造成的拥堵，可以配合Storm的那个动态图来判断越
红拥堵越厉害，同时参考Capacity这个值来判断超过1的Bolt或多或少会出现拥堵的情
况
根据自己的算法来优化处理逻辑提升效率，或者通过增加Bolt并行度的方式来提升Bolt
处理能力（前提是硬件资源没有到上限）
从你的这个业务来看应该是后续的几个Bolt要访问外部的数据存储系统进行最终结果的
存储，关注一下存取数据是否延迟较大，目标系统压力比较大
多数情况都是由于Bolt处理能力不够造成的，需要找出压力点优化业务处理逻辑和同时
调整Bolt并行度
关注下bolt的逻辑是计算密集型还是以来外部io，计算密集型只能增加worker
3、Config.TOPOLOGY_BACKPRESSURE_ENABLE
是否开启了反压机制Config.TOPOLOGY_BACKPRESSURE_ENABLE (好像是storm1.0.0以后
才有的)
4、Netty
另外要想提升Storm在底层传输上的吞吐量，可以通过修改storm.yaml的netty配置，来
提升netty的发送批量大
5、Executor‘s throughput params
// Net io set
config.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 1024 * 16); // default is
1024
config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 1024 * 16);//
batched; default is 1024
config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 1024 * 16); //
individual tuples; default is 1024
config.put(Config.TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS, 200);
_____
李波 13813887096 <ma...@asiainfo.com> libo19@asiainfo.com 北京亚信
智慧数据科技有限公司
亚信是我家 发展靠大家
From: 王 纯超 [mailto:wangchunchao@outlook.com]
Sent: 2017年8月7日 10:58
To: user <us...@storm.apache.org>
Cc: 姜艳春jiangyc_cui_gd@si-tech.com.cn <ji...@si-tech.com.cn>;
zhangxl_bds <zh...@si-tech.com.cn>
Subject: How to Improve Storm Application's Throughput
Hi,
I am now considering improve a Storm application's throughput because I
find that the consumption speed of KafkaSpout is slower than the producing
speed. And the lag gets larger and larger. Below is the bolt statistics. I
tried to bring forward the tuple projection and filtering logic in a custom
scheme with intention of reducing network traffic. However, after
observation, things go contrary to my wishes. Am I going the wrong way? Are
there any principles tuning Storm applications? Or could anyone give some
suggestions for this specific case?
_____
wangchunchao@outlook.com <ma...@outlook.com>
Re: How to Improve Storm Application's Throughput
Posted by Jungtaek Lim <ka...@gmail.com>.
According to statistics of topology, some of your Bolts are not keeping up
handling tuples. (capacity is close to or even over 1.0 from some bolts)
- increase parallelism for the component, or try to ensure locality between
near components
- look into details on statistic from each task in component and see
whether load is distributed evenly
- try to change shuffle grouping to local or shuffle grouping when
applicable
- you may want to combine some near bolts into one if needed
Hope this helps.
Thanks,
Jungtaek Lim (HeartSaVioR)
On Mon, 7 Aug 2017 at 11:58 AM 王 纯超 <wa...@outlook.com> wrote:
> Hi,
>
> I am now considering improve a Storm application's throughput because
> I find that the consumption speed of KafkaSpout is slower than the
> producing speed. And the lag gets larger and larger. Below is the bolt
> statistics. I tried to bring forward the tuple projection and filtering
> logic in a custom scheme with intention of reducing network traffic.
> However, after observation, things go contrary to my wishes. Am I going the
> wrong way? Are there any principles tuning Storm applications? Or could
> anyone give some suggestions for this specific case?
> ------------------------------
> wangchunchao@outlook.com
>