[jira] [Commented] (FLINK-16404) Avoid caching buffers for blocked input channels before barrier alignment

["Piotr Nowojski (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164492#comment-17164492 ] 

Piotr Nowojski commented on FLINK-16404:
----------------------------------------

Hey [~zjwang] [~kevin.cyj], I think you have forgotten to remove this metric in at least one remaining place [in the web UI|https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/subtask/job-checkpoints-subtask.component.html#L105].

Potentially we should also drop it from {{TaskCheckpointStatisticsWithSubtaskDetails.CheckpointAlignment#CheckpointAlignment}} - but this might affect our REST API and I'm not sure if the API is {{@Public}} and stable?

> Avoid caching buffers for blocked input channels before barrier alignment
> -------------------------------------------------------------------------
>
>                 Key: FLINK-16404
>                 URL: https://issues.apache.org/jira/browse/FLINK-16404
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Network
>            Reporter: Zhijiang
>            Assignee: Yingjie Cao
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.11.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> One motivation of this issue is for reducing the in-flight data in the case of back pressure to speed up checkpoint. The current default exclusive buffers per channel is 2. If we reduce it to 0 and increase somewhat floating buffers for compensation, it might cause deadlock problem because all the floating buffers might be requested away by some blocked input channels and never recycled until barrier alignment.
> In order to solve above deadlock concern, we can make some logic changes on both sender and receiver sides.
>  * Sender side: It should revoke previous received credit after sending checkpoint barrier, that means it would not send any following buffers until receiving new credits.
>  * Receiver side: The respective channel releases the requested floating buffers if barrier is received from the network. After barrier alignment, it would request floating buffers for the channels with positive backlog, and notify the sender side of available credits. Then the sender can continue transporting the buffers.
> Based on above changes, we can also remove the `BufferStorage` component completely, because the receiver would never reading buffers for blocked channels. Another possible benefit is that the floating buffers might be more properly made use of before barrier alignment.
> The only side effect would bring somehow cold setup after barrier alignment. That means the sender side has to wait for credit feedback to transport data just after alignment, which would impact on delay and network throughput. But considering the checkpoint interval not too short in general, so the above side effect can be ignored in practice. We can further verify it via existing micro-benchmark.
> After this ticket done, we still can not set exclusive buffers to zero ATM, there exists another deadlock issue which would be solved separately in another ticket.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)