You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Roshan Naik (JIRA)" <ji...@apache.org> on 2018/09/14 09:37:00 UTC
[jira] [Commented] (STORM-3205) Optimization in TuplImpl
[ https://issues.apache.org/jira/browse/STORM-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614579#comment-16614579 ]
Roshan Naik commented on STORM-3205:
------------------------------------
*Raw TVL readings:*
+MASTER: reading 1+
{code}
start_time(s) end_time(s) rate(tuple/s) mean(ms) 99%ile(ms) 99.9%ile(ms) cores mem(MB)
0 30 363,105.633 552.793 844.104 853.017 3.804 174.941
30 60 447,122.200 2,142.178 5,695.865 5,804.917 4.550 51.488
60 90 416,901.400 9,418.863 13,019.120 13,086.228 4.414 155.132
90 120 420,867.467 16,801.866 20,099.105 20,149.436 4.352 100.409
120 150 429,340.700 23,451.448 26,575.110 26,742.882 4.412 105.838
150 180 427,491.233 30,118.689 33,386.660 33,436.991 4.417 145.061
180 210 410,376.700 37,208.048 41,003.516 41,070.625 4.390 139.888
210 240 420,193.767 44,735.209 48,083.501 48,184.164 4.392 97.299
240 270 424,774.967 51,567.712 54,962.160 55,029.268 4.412 127.620
{code}
+MASTER: reading2+
{code}
start_time(s) end_time(s) rate(tuple/s) mean(ms) 99%ile(ms) 99.9%ile(ms) cores mem(MB)
0 30 352,283.467 666.673 988.807 996.147 3.833 131.411
30 60 452,429.567 2,457.883 5,981.078 6,106.907 4.825 128.944
60 90 389,890.233 10,536.345 14,738.784 14,856.225 4.246 145.399
90 120 399,513.033 19,105.363 22,917.677 23,068.672 4.138 135.547
120 150 412,216.733 26,820.408 30,450.647 30,601.642 4.195 55.722
150 180 415,783.600 34,411.969 37,882.954 37,950.063 4.235 95.921
180 210 428,794.100 41,234.477 44,493.177 44,560.286 4.383 71.357
210 240 427,405.133 47,840.604 51,204.063 51,271.172 4.349 112.849
{code}
+STORM-3205 : TVL reading1+
{code}
start_time(s) end_time(s) rate(tuple/s) mean(ms) 99%ile(ms) 99.9%ile(ms) cores mem(MB)
0 30 365,140.233 389.399 851.444 865.599 3.733 112.723
30 60 551,828.200 2.096 58.917 109.838 4.593 68.210
60 90 550,123.200 0.658 4.268 6.423 4.433 107.547
90 120 550,244.433 0.567 1.696 6.160 4.146 113.571
120 150 550,271.567 0.547 1.245 3.652 4.170 144.445
150 180 550,279.300 1.077 18.792 25.756 4.229 112.253
180 210 550,234.733 0.529 1.136 1.579 4.069 136.361
210 240 550,241.433 0.530 1.150 1.587 4.087 160.657
{code}
+STORM-3205 : TVL reading2+
{code}
start_time(s) end_time(s) rate(tuple/s) mean(ms) 99%ile(ms) 99.9%ile(ms) cores mem(MB)
0 30 366,992.967 280.818 569.377 575.144 3.693 155.890
30 60 550,076.200 0.556 1.385 5.751 4.023 163.471
60 91 532,387.032 0.555 1.333 5.321 3.958 142.208
91 121 550,137.600 0.545 1.209 4.522 4.024 120.298
121 151 550,156.633 0.569 1.647 5.382 4.210 87.505
151 181 550,222.900 0.574 1.808 5.730 4.176 169.807
181 211 550,226.400 0.535 1.202 1.917 4.133 154.749
211 241 550,156.300 0.530 1.150 1.644 4.052 117.847
{code}
> Optimization in TuplImpl
> ------------------------
>
> Key: STORM-3205
> URL: https://issues.apache.org/jira/browse/STORM-3205
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-client
> Affects Versions: 2.0.0
> Reporter: Roshan Naik
> Assignee: Roshan Naik
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Wrapping {{TuplImpl.values}} with Collections.unmodifiableList() turns out be very expensive. Its intention is obviously to check and prevent accidental tweaking TuplImpl once created. Given the high cost, if needed, we can limit this extra checking mechanism in debug/dev mode. Being in the critical path it means several thousand/million additional allocations per second of the List wrapper object .... proportional to the number of bolt/spout instances.
> *TVL :*
> | |throughput (k/sec)|cores|mem (mb)|
> |*master (#f5a410ba3)*|412 |4.26|103|
> |*storm-3205*|547 (+33%)|4.09|132|
> {{+cmd:+ bin/storm jar topos/storm-loadgen-2.0.0-SNAPSHOT.jar org.apache.storm.loadgen.ThroughputVsLatency *--rate 550000* --spouts 1 --splitters 3 --counters 2 -c topology.acker.executors=0}}
> *+ConstSpoutIdentityBoltNullBolt :+*
> | |throughput|
> |*master*|4.25 mill/sec|
> |*storm-3205*|5.4 mill/sec (+27%)|
> +cmd+: {{bin/storm jar topos/storm-perf-2.0.0-SNAPSHOT.jar org.apache.storm.perf.ConstSpoutIdBoltNullBoltTopo -c topology.acker.executors=0 -c topology.producer.batch.size=1000 400}}
> *Note:* The perf gains are more evident when operating at high thoughputs w/o backpressure occurring (i.e. some bolts have not yet become a bottleneck)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)