[jira] [Commented] (FLINK-30193) could flink has dynamic rebalance according to the load of downstream operators

["Gen Luo (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17638336#comment-17638336 ] 

Gen Luo commented on FLINK-30193:
---------------------------------

HI [~landlord],
We did some work like this, which makes rebalance partitioner able to distribute the records according the load of downstream operators, and we are considering that maybe we would contribute the code to the community when we can make the time. Maybe we can discuss about the feature in the mail list if you'd like. The jira should be created after the discuss in the mail list reaches a consensus.

> could flink has dynamic rebalance  according to the load of downstream operators 
> ---------------------------------------------------------------------------------
>
>                 Key: FLINK-30193
>                 URL: https://issues.apache.org/jira/browse/FLINK-30193
>             Project: Flink
>          Issue Type: New Feature
>          Components: API / DataStream
>    Affects Versions: 1.15.2
>            Reporter: Lu
>            Priority: Major
>
> Different records cause different loads to the downstream and different task slot has different compute power. So rebalancing according to the load of downstream operators will greatly reduce data skew and improve resource utilization.
> I thaught the point is how to gain the load of downstream.  I have seen the 
> StreamPartitioner , and it dosen't has any context or method to get the load information of downstream operators. So ordinary developers cannot easily implement such functions.
> finally，Could flink provide such a function?
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)