[jira] [Commented] (FLINK-8523) Stop assigning floating buffers for blocked input channels in exactly-once mode

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-8523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344611#comment-16344611 ] 

ASF GitHub Bot commented on FLINK-8523:
---------------------------------------

GitHub user zhijiangW opened a pull request:

    https://github.com/apache/flink/pull/5381

    [FLINK-8523][network] Stop assigning floating buffers for blocked input channels in exactly-once mode

    ## What is the purpose of the change
    
    In exactly-once mode, the input channel is set blocked state when reading barrier from it. And the blocked state will be released after barrier alignment or cancelled.
    
    In credit-based network flow control, we should avoid assigning floating buffers for blocked input channels because the buffers after barrier will not be processed by operator until alignment.
    
    To do so, we can fully make use of floating buffers and speed up barrier alignment in some extent.
    
    ## Brief change log
    
      - *Add `blockInputChannel` and `releaseBlockedInputChannels` in `InputGate` interface*
      - *`UnionInputGate` constructs the mapping from channel index to `InputGate`*
      - *`SingleInputGate` constructs the mapping from channel index to `InputChannel`*
      - *`BarrierBuffer` determines the logic of blocking input channel or releasing it*
      - *Avoid assigning floating buffers for blocked input channels*
    
    
    ## Verifying this change
    
    This change added tests and can be verified as follows:
    
      - *Added unit tests that validates the floating buffers are not assigned for blocked input channels*
      - *Added unit test that validates the `BarrierBuffer` blocks or releases the `InputChannel` correctly*
      - *Added unit test that validates the `UnionInputGate` and `SingleInputGate` constructs the mapping correctly*
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (don't know)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
      - The S3 file system connector: (no)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zhijiangW/flink FLINK-8523

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5381.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5381
    
----
commit 8c7ef5a22c348c3a6eedb04708b00093ae666e37
Author: Zhijiang <wa...@...>
Date:   2018-01-30T07:14:20Z

    [FLINK-8523][network] Stop assigning floating buffers for blocked input channels in exactly-once mode

----


> Stop assigning floating buffers for blocked input channels in exactly-once mode
> -------------------------------------------------------------------------------
>
>                 Key: FLINK-8523
>                 URL: https://issues.apache.org/jira/browse/FLINK-8523
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Network
>    Affects Versions: 1.5.0
>            Reporter: zhijiang
>            Assignee: zhijiang
>            Priority: Major
>
> In exactly-once mode, the input channel is set blocked state when reading barrier from it. And the blocked state will be released after barrier alignment or cancelled.
>  
> In credit-based network flow control, we should avoid assigning floating buffers for blocked input channels because the buffers after barrier will not be processed by operator until alignment.
> To do so, we can fully make use of floating buffers and speed up barrier alignment in some extent.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)