You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Ethan Guo (Jira)" <ji...@apache.org> on 2023/09/15 05:14:00 UTC

[jira] [Updated] (HUDI-6864) Auto-tune dedup parallelism without affecting write parallelism

     [ https://issues.apache.org/jira/browse/HUDI-6864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ethan Guo updated HUDI-6864:
----------------------------
    Description: 
See HUDI-6863 and https://github.com/apache/hudi/pull/6802#issuecomment-1455802492

I think we need to make sure that the dedup parallelism is only applied to the dedup stage, not affecting subsequent stages, which may require better parallelism control by repartitioning with right parallelism before workload profiling.

> Auto-tune dedup parallelism without affecting write parallelism
> ---------------------------------------------------------------
>
>                 Key: HUDI-6864
>                 URL: https://issues.apache.org/jira/browse/HUDI-6864
>             Project: Apache Hudi
>          Issue Type: Improvement
>            Reporter: Ethan Guo
>            Priority: Major
>             Fix For: 0.14.1
>
>
> See HUDI-6863 and https://github.com/apache/hudi/pull/6802#issuecomment-1455802492
> I think we need to make sure that the dedup parallelism is only applied to the dedup stage, not affecting subsequent stages, which may require better parallelism control by repartitioning with right parallelism before workload profiling.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)