You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "zhijiang (JIRA)" <ji...@apache.org> on 2018/02/02 07:44:00 UTC
[jira] [Updated] (FLINK-8547) Implement CheckpointBarrierHandler
not to spill data for exactly-once based on credit-based flow control
[ https://issues.apache.org/jira/browse/FLINK-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhijiang updated FLINK-8547:
----------------------------
Description:
Currently in exactly-once mode, the {{BarrierBuffer}} would block inputs with barriers until all inputs have received the barrier for a given checkpoint. To avoid back-pressuring the input streams which may cause distributed deadlocks, the {{BarrierBuffer}} has to spill the data in disk files to recycle the buffers for blocked channels.
Based on credit-based flow control, every channel has exclusive buffers, so it is no need to spill data for avoiding deadlock. Then we implement a new {{CheckpointBarrierHandler}} for only buffering the data for blocked channels for better performance.
And this new {{CheckpointBarrierHandler}} can also be configured to use or not in order to rollback the original mode for unexpected risks.
was:
Currently in exactly-once mode, the `BarrierBuffer` would block inputs with barriers until all inputs have received the barrier for a given checkpoint. To avoid back-pressuring the input streams which may cause distributed deadlocks, the `BarrierBuffer` has to spill the data in disk files to recycle the buffers for blocked channels.
Based on credit-based flow control, every channel has exclusive buffers, so it is no need to spill data for avoiding deadlock. Then we implement a new 'CheckpointBarrierHandler` for only buffering the data for blocked channels for better performance.
And this new `CheckpointBarrierHandler` can also be configured to use or not in order to rollback the original mode for unexpected risks.
> Implement CheckpointBarrierHandler not to spill data for exactly-once based on credit-based flow control
> --------------------------------------------------------------------------------------------------------
>
> Key: FLINK-8547
> URL: https://issues.apache.org/jira/browse/FLINK-8547
> Project: Flink
> Issue Type: Sub-task
> Components: Network
> Affects Versions: 1.5.0
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Major
>
> Currently in exactly-once mode, the {{BarrierBuffer}} would block inputs with barriers until all inputs have received the barrier for a given checkpoint. To avoid back-pressuring the input streams which may cause distributed deadlocks, the {{BarrierBuffer}} has to spill the data in disk files to recycle the buffers for blocked channels.
>
> Based on credit-based flow control, every channel has exclusive buffers, so it is no need to spill data for avoiding deadlock. Then we implement a new {{CheckpointBarrierHandler}} for only buffering the data for blocked channels for better performance.
>
> And this new {{CheckpointBarrierHandler}} can also be configured to use or not in order to rollback the original mode for unexpected risks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)