You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2019/09/26 08:20:36 UTC
[GitHub] [flink] wsry commented on a change in pull request #9706:
[FLINK-14118][runtime]Reduce the unnecessary flushing when there is no data
available for flush.
wsry commented on a change in pull request #9706: [FLINK-14118][runtime]Reduce the unnecessary flushing when there is no data available for flush.
URL: https://github.com/apache/flink/pull/9706#discussion_r328490790
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferConsumer.java
##########
@@ -163,6 +170,10 @@ public int getCached() {
return PositionMarker.getAbsolute(cachedPosition);
}
+ private int getLatest() {
+ return PositionMarker.getAbsolute(positionMarker.get());
Review comment:
I ran some benchmark cases with 1ms bufferTimeout and no evident performance difference was seen before and after the fix. The followings are the result.
Before the fix:
Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units
NetworkThroughput 1000,1ms thrpt 30 23032.384 ± 871.883 ops/ms
KeyByBenchmarks.arrayKeyBy1MS thrpt 30 1923.863 ± 78.518 ops/ms
KeyByBenchmarks.tupleKeyBy1MS thrpt 30 3377.401 ± 216.982 ops/ms
InputBenchmark.mapRebalanceMapSink1MS thrpt 30 6091.213 ± 92.658 ops/ms
InputBenchmark.mapSinkBufferTimeout1MS thrpt 30 9107.194 ± 211.169 ops/ms
After the fix:
Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units
NetworkThroughput 1000,1ms thrpt 30 23985.588 ± 990.037 ops/ms
KeyByBenchmarks.arrayKeyBy1MS thrpt 30 2011.356 ± 40.347 ops/ms
KeyByBenchmarks.tupleKeyBy1MS thrpt 30 3440.238 ± 211.906 ops/ms
InputBenchmark.mapRebalanceMapSink1MS thrpt 30 6118.888 ± 94.517 ops/ms
InputBenchmark.mapSinkBufferTimeout1MS thrpt 30 9120.144 ± 252.023 ops/ms
As discussed in the mailing list. High data skew between large number of channels (subpartitions) with low bufferTimeout can lead to throughput regression. The results are as follows.
Before the fix:
Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units
NetworkThroughput 1000,1ms thrpt 30 18240.197 ± 1892.419 ops/ms
After the fix:
Benchmark (channelsFlushTimeout) Mode Cnt Score Error Units
NetworkThroughput 1000,1ms thrpt 30 24532.313 ± 1118.312 ops/ms
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services