You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "zhijiang (JIRA)" <ji...@apache.org> on 2018/02/02 07:43:00 UTC

[jira] [Created] (FLINK-8547) Implement CheckpointBarrierHandler not to spill data for exactly-once based on credit-based flow control

zhijiang created FLINK-8547:
-------------------------------

             Summary: Implement CheckpointBarrierHandler not to spill data for exactly-once based on credit-based flow control
                 Key: FLINK-8547
                 URL: https://issues.apache.org/jira/browse/FLINK-8547
             Project: Flink
          Issue Type: Sub-task
          Components: Network
    Affects Versions: 1.5.0
            Reporter: zhijiang
            Assignee: zhijiang


Currently in exactly-once mode, the `BarrierBuffer` would block inputs with barriers until all inputs have received the barrier for a given checkpoint. To avoid back-pressuring the input streams which may cause distributed deadlocks, the `BarrierBuffer` has to spill the data in disk files to recycle the buffers for blocked channels.

 

Based on credit-based flow control, every channel has exclusive buffers, so it is no need to spill data for avoiding deadlock. Then we implement a new 'CheckpointBarrierHandler` for only buffering the data for blocked channels for better performance.

 

And this new `CheckpointBarrierHandler` can also be configured to use or not in order to rollback the original mode for unexpected risks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)