[jira] [Updated] (HUDI-7574) Auto-scaling for Flink Hudi sink tasks

["Kate Huber (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-7574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kate Huber updated HUDI-7574:
-----------------------------
    Fix Version/s: 1.1.0
                       (was: 1.0.0)

> Auto-scaling for Flink Hudi sink tasks
> --------------------------------------
>
>                 Key: HUDI-7574
>                 URL: https://issues.apache.org/jira/browse/HUDI-7574
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: flink
>            Reporter: Danny Chen
>            Assignee: Danny Chen
>            Priority: Major
>             Fix For: 1.1.0
>
>
> Currently the flink write task parallelism is set up through 
> {code:java}
> write.tasks{code}
> it is kind of a fixed number during the lifecycle of the ingestion pipeline, while for streaming, there are always fluctuation of the workload, it is great if we can tune the parallelism of write tasks based on the job load profile dynamically.
> On K8s, Flink provides a [https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/] which is suitable for the purpose, which deserves a further investigation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)