You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Piotr Nowojski (Jira)" <ji...@apache.org> on 2021/06/10 18:54:00 UTC

[jira] [Comment Edited] (FLINK-22950) Back-pressure report of final sink operator is not accurate

    [ https://issues.apache.org/jira/browse/FLINK-22950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361170#comment-17361170 ] 

Piotr Nowojski edited comment on FLINK-22950 at 6/10/21, 6:53 PM:
------------------------------------------------------------------

Yes, that's current expected behaviour. I've changed type of this issue to improvement. 

In short, the generic back-pressure mechanism used in Flink relies on the presence/absence of the network `Buffer`s in the Task's buffer pool. If task is not writing something to an external system like a Sink, there is no generic way for Flink to distinguish backpressure from busy. To solve this, we would need to first provide availability API to the Sinks. Next each sink would need to implement it on its own. Many of the sinks would not be able to implement it thought. 

Part of this was being proposed and discussed in FLIP-143 [1][2] and more recently in FLIP-171 

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-143-Unified-Sink-API-tp44602p44872.html
[2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-143-Unified-Sink-API-tp44602p44930.html
[3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink


was (Author: pnowojski):
Yes, that's current expected behaviour. I've changed type of this issue to improvement. 

In short, the generic back-pressure mechanism used in Flink relies on the presence/absence of the network `Buffer`s in the Task's buffer pool. If task is not writing something to an external system like a Sink, there is no generic way for Flink to distinguish backpressure from busy. To solve this, we would need to first provide availability API to the Sinks. Next each sink would need to implement it on its own. Many of the sinks would not be able to implement it thought. 

Part of this was being proposed and discussed in FLIP-143 [1][2] and more recently in FLIP-171 

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-143-Unified-Sink-API-tp44602p44872.html
[2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-143-Unified-Sink-API-tp44602p44930.html
[3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink#FLIP171:AsyncSink-SimplifiedAsyncSinkWriterforKinesisDataStreams

> Back-pressure report of final sink operator is not accurate
> -----------------------------------------------------------
>
>                 Key: FLINK-22950
>                 URL: https://issues.apache.org/jira/browse/FLINK-22950
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Metrics, Runtime / Task
>    Affects Versions: 1.13.1
>            Reporter: Fu Kai
>            Priority: Major
>
> We noticed the back-pressure status of the final sink on the job graph is not accurate. It's apparently the final sink operator should receive back-pressure from the sink storage(ElasitcSearch/Kafka), while it's always showing not back-pressured(with value of 0, but always showing 100% busy though).
> This is confirmed by switching the sink to be more powerful and without changing any other configurations. We noticed the record publish rate increases significantly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)