You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "tartarus (Jira)" <ji...@apache.org> on 2023/03/29 08:44:00 UTC

[jira] [Commented] (FLINK-31655) Adaptive Channel selection for partitioner

    [ https://issues.apache.org/jira/browse/FLINK-31655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706281#comment-17706281 ] 

tartarus commented on FLINK-31655:
----------------------------------

[~gaoyunhaii]  [~fanrui]  Please take a look at this issue, after our company use the Adaptive Rebalance/Rescale, job throughput improved by about 20% on average compared to the community version.

We can discuss how we can contribute this feature to the community!

 

please assign to me, thanks~

> Adaptive Channel selection for partitioner
> ------------------------------------------
>
>                 Key: FLINK-31655
>                 URL: https://issues.apache.org/jira/browse/FLINK-31655
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Task
>            Reporter: tartarus
>            Priority: Major
>
> In Flink, if the upstream and downstream operator parallelism is not the same, then by default the RebalancePartitioner will be used to select the target channel.
> In our company, users often use flink to access redis, hbase or other rpc services, If some of the Operators are slow to return requests (for external service reasons), then because Rebalance/Rescale are Round-Robin the Channel selection policy, so the job is easy to backpressure.
> Because the Rebalance/Rescale policy does not care which subtask the data is sent to downstream, so we expect Rebalance/Rescale to refer to the processing power of the downstream subtask when choosing a Channel.
> Send more data to the free subtask, this ensures the best possible throughput of job!
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)