You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "daxwang (王文军)" <da...@tencent.com> on 2017/02/26 16:24:51 UTC
some spout executor complete latency very high
i have a topology in storm-1.0.3, there is a interesting thing puzzle me, there have total 40 spout executors, executor[28-28] and exeuctor[37-37] complete latency about 10ms, but the other
executor complete latency is very high, help me please!
Id Uptime Host Port Actions Emitted Transferred Complete latency (ms) Acked Failed
[29-29] 1h 10m 22s test-heron-01 6710<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6710%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6710> 37900 37900 120617.570 17720 600
[20-20] 1h 10m 22s test-heron-01 6710<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6710%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6710> 37780 37780 121528.516 17640 620
[38-38] 1h 10m 22s test-heron-01 6710<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6710%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6710> 38160 38160 119780.141 17860 600
[21-21] 1h 7m 51s test-heron-01 6711<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6711%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6711> 46440 46440 106324.648 22600 0
[30-30] 1h 7m 51s test-heron-01 6711<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6711%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6711> 46480 46480 106045.703 22600 0
[39-39] 1h 7m 51s test-heron-01 6711<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6711%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6711> 46440 46440 106304.484 22580 0
[13-13] 1h 10m 23s test-heron-01 6712<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6712%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6712> 37340 37340 122911.156 17420 620
[31-31] 1h 10m 23s test-heron-01 6712<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6712%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6712> 38180 38180 120113.297 17840 600
[22-22] 1h 10m 23s test-heron-01 6712<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6712%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6712> 37700 37700 121553.383 17620 600
[14-14] 1h 10m 19s test-heron-01 6713<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6713%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6713> 37360 37360 122606.969 17440 600
[5-5] 1h 10m 19s test-heron-01 6713<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6713%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6713> 36820 36820 123499.117 17180 600
[33-33] 1h 10m 19s test-heron-01 6713<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6713%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6713> 37460 37460 122159.930 17500 600
[25-25] 1h 10m 20s test-heron-01 6714<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6714%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6714> 37860 37840 120260.672 17700 600
[34-34] 1h 10m 20s test-heron-01 6714<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6714%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6714> 37960 37960 120791.211 17720 640
[16-16] 1h 10m 20s test-heron-01 6714<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6714%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6714> 38520 38520 119519.859 18000 620
[17-17] 1h 9m 22s test-heron-01 6715<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6715%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6715> 38920 38920 116597.844 18280 540
[36-36] 1h 9m 22s test-heron-01 6715<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6715%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6715> 41220 41220 110444.297 19460 500
[8-8] 1h 9m 22s test-heron-01 6715<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6715%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6715> 69160 69160 71475.906 33940 0
[37-37] 1h 8m 30s test-heron-01 6720<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6720%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6720> 361860 361860 9.106 180300 0
[28-28] 1h 8m 30s test-heron-01 6720<http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6720%2Fworker.log> files<http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6720> 560240 560240 10.636 279480 0
Re: some spout executor complete latency very high
Posted by Jungtaek Lim <ka...@gmail.com>.
You're running 20 executors in one machine. Does the machine have enough
cores so that it can run 'really' parallel?
(Where spouts pull the data? If it doesn't involve CPU much it will be OK,
but where bolts are placed affects the issue.)
And how much ackers topology has, and which workers ackers being placed?
Please examine CPU, memory, GC, IO for that machine to ensure there's no
resource issue.
And check your topology physical plan, how spouts/bolts are placed and how
they're connected.
Try profiling to bad workers and see if there's memory leak issue, CPU go
high, GC too much, etc.
Hope this helps.
Thanks,
Jungtaek Lim (HeartSaVioR)
2017년 2월 27일 (월) 오전 1:25, daxwang(王文军) <da...@tencent.com>님이 작성:
> i have a topology in storm-1.0.3, there is a interesting thing puzzle me,
> there have total 40 spout executors, executor[28-28] and exeuctor[37-37]
> complete latency about 10ms, but the other
> executor complete latency is very high, help me please!
>
>
> Id Uptime Host Port Actions Emitted Transferred Complete latency (ms)
> Acked Failed
> [29-29] 1h 10m 22s test-heron-01 6710
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6710%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6710>
> 37900 37900 120617.570 17720 600
> [20-20] 1h 10m 22s test-heron-01 6710
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6710%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6710>
> 37780 37780 121528.516 17640 620
> [38-38] 1h 10m 22s test-heron-01 6710
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6710%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6710>
> 38160 38160 119780.141 17860 600
> [21-21] 1h 7m 51s test-heron-01 6711
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6711%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6711>
> 46440 46440 106324.648 22600 0
> [30-30] 1h 7m 51s test-heron-01 6711
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6711%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6711>
> 46480 46480 106045.703 22600 0
> [39-39] 1h 7m 51s test-heron-01 6711
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6711%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6711>
> 46440 46440 106304.484 22580 0
> [13-13] 1h 10m 23s test-heron-01 6712
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6712%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6712>
> 37340 37340 122911.156 17420 620
> [31-31] 1h 10m 23s test-heron-01 6712
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6712%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6712>
> 38180 38180 120113.297 17840 600
> [22-22] 1h 10m 23s test-heron-01 6712
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6712%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6712>
> 37700 37700 121553.383 17620 600
> [14-14] 1h 10m 19s test-heron-01 6713
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6713%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6713>
> 37360 37360 122606.969 17440 600
> [5-5] 1h 10m 19s test-heron-01 6713
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6713%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6713>
> 36820 36820 123499.117 17180 600
> [33-33] 1h 10m 19s test-heron-01 6713
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6713%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6713>
> 37460 37460 122159.930 17500 600
> [25-25] 1h 10m 20s test-heron-01 6714
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6714%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6714>
> 37860 37840 120260.672 17700 600
> [34-34] 1h 10m 20s test-heron-01 6714
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6714%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6714>
> 37960 37960 120791.211 17720 640
> [16-16] 1h 10m 20s test-heron-01 6714
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6714%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6714>
> 38520 38520 119519.859 18000 620
> [17-17] 1h 9m 22s test-heron-01 6715
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6715%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6715>
> 38920 38920 116597.844 18280 540
> [36-36] 1h 9m 22s test-heron-01 6715
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6715%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6715>
> 41220 41220 110444.297 19460 500
> [8-8] 1h 9m 22s test-heron-01 6715
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6715%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6715>
> 69160 69160 71475.906 33940 0
> [37-37] 1h 8m 30s test-heron-01 6720
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6720%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6720>
> 361860 361860 9.106 180300 0
> [28-28] 1h 8m 30s test-heron-01 6720
> <http://test-heron-01:8000/log?file=topology-38816-3-1488121028%2F6720%2Fworker.log>
> files
> <http://test-heron-01:8000/dumps/topology-38816-3-1488121028/test-heron-01%3A6720>
> 560240 560240 10.636 279480 0
>