[jira] [Commented] (FLINK-10762) Do not try to request partitions in SingleInputGate per getRecord

["Piotr Nowojski (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675079#comment-16675079 ] 

Piotr Nowojski commented on FLINK-10762:
----------------------------------------

[~NicoK] on the codespeed there was no [visible|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=networkThroughput.1000,1ms&env=2&revs=200&equid=off&quarts=on&extr=on] change in [benchmark|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=networkThroughput.1000,100ms&env=2&revs=200&equid=off&quarts=on&extr=on] results neither after merging the previous fix nor after reverting it.

Your local results where also not very conclusive. It could be that this improvement is very small and visible only on certain setups, but it also could be that it doesn't really matter and the benchmark results are a bit noisy and some of your runs were just lucky. Maybe before spending more time on this we should validate if it's really worth a trouble?

> Do not try to request partitions in SingleInputGate per getRecord
> -----------------------------------------------------------------
>
>                 Key: FLINK-10762
>                 URL: https://issues.apache.org/jira/browse/FLINK-10762
>             Project: Flink
>          Issue Type: Improvement
>          Components: Network
>    Affects Versions: 1.7.0
>            Reporter: Nico Kruber
>            Priority: Major
>             Fix For: 1.8.0
>
>
> Every call to {{SingleInputGate#getNextBufferOrEvent()}} will also try to request partitions via {{SingleInputGate#requestPartitions()}}. However, in the current design, this must be under the {{requestLock}} which is costly and unnecessary.
> I propose a redesign to change that and make the {{SingleInputGate#requestPartitions()}} a one-time only thing that should be called *before* going into any loop asking for new records. In that case, we could keep the lock and ignore and performance-related improvements.
> Alternatively, if we had a separate {{unrequestedPartitions}} list aside to {{inputChannels}} and only go through these when executing {{SingleInputGate#requestPartitions()}}, we would change {{updateInputChannel}} to always request the newly created partition and will not have the concurrency issue between these two methods anymore. This {{unrequestedPartitions}} list would basically also make {{requestedPartitionsFlag}} obsolete if we, for example, set it to {{Optional.empty}} after finishing the requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)