You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 04:38:13 UTC
[I] Add support to sync all partitions in hive sync tool [hudi]
hudi-bot opened a new issue, #14969:
URL: https://github.com/apache/hudi/issues/14969
If a user runs hive sync occationally and if archival kicked in and trimmed some commits and if there were partitions added during those commits which was never updated later, hive sync will miss out those partitions.
{code:java}
LOG.info("Last commit time synced is " + lastCommitTimeSynced.get() + ", Getting commits since then");
return TimelineUtils.getPartitionsWritten(metaClient.getActiveTimeline().getCommitsTimeline()
.findInstantsAfter(lastCommitTimeSynced.get(), Integer.MAX_VALUE));
} {code}
bcoz, we for recurrent syncs, we always fetch new commits from timeline after the last synced instant and fetch commit metadata and go on to fetch the partitions added as part of it.
We can add a new config to hive sync tool to override this behavior.
--sync-all-partitions
when this config is set to true, we should ignore last synced instant and should go the below route which is done when syncing for the first time.
{code:java}
if (!lastCommitTimeSynced.isPresent()) {
LOG.info("Last commit time synced is not known, listing all partitions in " + basePath + ",FS :" + fs);
HoodieLocalEngineContext engineContext = new HoodieLocalEngineContext(metaClient.getHadoopConf());
return FSUtils.getAllPartitionPaths(engineContext, basePath, useFileListingFromMetadata, assumeDatePartitioning);
} {code}
Ref issue:
https://github.com/apache/hudi/issues/3890
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3068
- Type: New Feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support to sync all partitions in hive sync tool [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #14969: Add support to sync all partitions in hive sync tool
URL: https://github.com/apache/hudi/issues/14969
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support to sync all partitions in hive sync tool [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14969:
URL: https://github.com/apache/hudi/issues/14969#issuecomment-4795293835
This appears to be addressed in current master by two changes:
1. **HUDI-5816** ([commit 3a953cc5](https://github.com/apache/hudi/commit/3a953cc5746477ce857c682b93f1a326af0e12b2), PR #8388) — Added an **automatic** fallback in `HiveSyncTool.validateAndSyncPartitions`: when `lastCommitTimeSynced` is before the start of the active timeline (i.e., archival trimmed those commits), the sync falls back to `syncAllPartitions` and lists all partitions on storage. This directly handles the scenario described here.
2. **HUDI-5988** ([commit 6f5aaf6](https://github.com/apache/hudi/commit/6f5aaf6586d6296c0b07efa0095e93d0ca301099), PR #8301) — Added the explicit override config `hoodie.meta.sync.incremental` (CLI flag `--sync-incremental=false`), which is essentially the `--sync-all-partitions` flag this issue requested.
Relevant code: `hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java#L317-L357`.
If this matches your need, can this issue (and HUDI-3068) be closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support to sync all partitions in hive sync tool [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #14969:
URL: https://github.com/apache/hudi/issues/14969#issuecomment-3630133423
### Linked PR(s)
- https://github.com/apache/hudi/pull/4405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org