You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Udit Mehrotra (Jira)" <ji...@apache.org> on 2021/08/12 22:39:00 UTC
[jira] [Resolved] (HUDI-2194) Skip the latest N partitions when
creating ClusteringPlan
[ https://issues.apache.org/jira/browse/HUDI-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Udit Mehrotra resolved HUDI-2194.
---------------------------------
Fix Version/s: 0.9.0
Resolution: Fixed
> Skip the latest N partitions when creating ClusteringPlan
> ---------------------------------------------------------
>
> Key: HUDI-2194
> URL: https://issues.apache.org/jira/browse/HUDI-2194
> Project: Apache Hudi
> Issue Type: Task
> Reporter: Yue Zhang
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> As we known, SparkRecentDaysClusteringPlanStrategy is the default clustering strategy to create ClusteringPlan. And it is useful when Hudi table is partitioned by time.
>
> For now, users can set `hoodie.clustering.plan.strategy.daybased.lookback.partitions` to control the number of partitions to list from the latest partition to create ClusteringPlan.
> For example, we have 6 partitions based on date, and users set `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2
> |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)|
> |<----- choose to cluster ---->|
> Sometimes users also what to skip x partitions from latest when make clustering plan because latest partitions contains lots of update data or some reasons else.
>
> This patch will add a new config named `
> hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions
> ` to set the
> number of partitions to skip from latest when choosing partitions to create ClusteringPlan
>
> for example users set `hoodie.clustering.plan.strategy.daybased.lookback.partitions` 2 and
> `
> hoodie.clustering.plan.strategy.daybased.skipfromlatest.partitions
> ` 2
> |20210718|20210719 |20210720 |20210721 |20210722 |20210723(latest)|
> |<----- choose ----->|
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)