You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Nico Kruber (JIRA)" <ji...@apache.org> on 2018/12/19 11:09:00 UTC
[jira] [Updated] (FLINK-10762) Do not try to request partitions in
SingleInputGate per getRecord
[ https://issues.apache.org/jira/browse/FLINK-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nico Kruber updated FLINK-10762:
--------------------------------
Description:
Every call to {{SingleInputGate#getNextBufferOrEvent()}} will also try to request partitions via {{SingleInputGate#requestPartitions()}}. However, in the current design, this must be under the {{requestLock}} which is costly and unnecessary.
I propose a redesign to change that and make the {{SingleInputGate#requestPartitions()}} a one-time only thing that should be called *before* going into any loop asking for new records. In that case, we could keep the lock and ignore any performance-related penalties.
Alternatively, if we had a separate {{unrequestedPartitions}} list aside to {{inputChannels}} and only go through these when executing {{SingleInputGate#requestPartitions()}}, we would change {{updateInputChannel}} to always request the newly created partition and will not have the concurrency issue between these two methods anymore. This {{unrequestedPartitions}} list would basically also make {{requestedPartitionsFlag}} obsolete if we, for example, set it to {{Optional.empty}} after finishing the requests.
was:
Every call to {{SingleInputGate#getNextBufferOrEvent()}} will also try to request partitions via {{SingleInputGate#requestPartitions()}}. However, in the current design, this must be under the {{requestLock}} which is costly and unnecessary.
I propose a redesign to change that and make the {{SingleInputGate#requestPartitions()}} a one-time only thing that should be called *before* going into any loop asking for new records. In that case, we could keep the lock and ignore any performance-related pernalties.
Alternatively, if we had a separate {{unrequestedPartitions}} list aside to {{inputChannels}} and only go through these when executing {{SingleInputGate#requestPartitions()}}, we would change {{updateInputChannel}} to always request the newly created partition and will not have the concurrency issue between these two methods anymore. This {{unrequestedPartitions}} list would basically also make {{requestedPartitionsFlag}} obsolete if we, for example, set it to {{Optional.empty}} after finishing the requests.
> Do not try to request partitions in SingleInputGate per getRecord
> -----------------------------------------------------------------
>
> Key: FLINK-10762
> URL: https://issues.apache.org/jira/browse/FLINK-10762
> Project: Flink
> Issue Type: Improvement
> Components: Network
> Affects Versions: 1.7.0
> Reporter: Nico Kruber
> Priority: Major
> Fix For: 1.8.0
>
>
> Every call to {{SingleInputGate#getNextBufferOrEvent()}} will also try to request partitions via {{SingleInputGate#requestPartitions()}}. However, in the current design, this must be under the {{requestLock}} which is costly and unnecessary.
> I propose a redesign to change that and make the {{SingleInputGate#requestPartitions()}} a one-time only thing that should be called *before* going into any loop asking for new records. In that case, we could keep the lock and ignore any performance-related penalties.
> Alternatively, if we had a separate {{unrequestedPartitions}} list aside to {{inputChannels}} and only go through these when executing {{SingleInputGate#requestPartitions()}}, we would change {{updateInputChannel}} to always request the newly created partition and will not have the concurrency issue between these two methods anymore. This {{unrequestedPartitions}} list would basically also make {{requestedPartitionsFlag}} obsolete if we, for example, set it to {{Optional.empty}} after finishing the requests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)