You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 05:03:18 UTC
[I] Add skip archive option when syncing to AWS Glue tables [hudi]
hudi-bot opened a new issue, #15244:
URL: https://github.com/apache/hudi/issues/15244
The issue is each time when Hudi upserts records, it would sync to the catalog and update {{last_commit_time_sync}} for the Glue table. Each time it updates this property, Glue by default would create a new table version and archive old versions. So the problem is if customers update the Hudi table frequently, eventually they would hit the Glue table version limit.
So here inside Hudi, we pass a parameter {{skipGlueArchive}} to the environment context to finally pass it to {{{}AWS Glue metadata service{}}}, so Glue client has an option to decide whether to skip archive or not.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4278
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add skip archive option when syncing to AWS Glue tables [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #15244:
URL: https://github.com/apache/hudi/issues/15244#issuecomment-4965766455
This issue was reviewed as part of the JIRA-migrated backlog triage.
Findings: The linked PR #5900 was closed unmerged, but this capability was subsequently delivered by #8744 ([HUDI-6232], merged 2023-11-30). Hudi now exposes the `hoodie.datasource.hive_sync.glue.skip_table_archive` config (GlueCatalogSyncClientConfig#GLUE_SKIP_TABLE_ARCHIVE); when enabled, AWSGlueCatalogSyncClient passes `skipArchive(true)` on Glue UpdateTable requests (AWSGlueCatalogSyncClient.java:589,1182), so frequent syncs no longer create new Glue table versions and hit the version limit — exactly the ask here.
Closing as done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add skip archive option when syncing to AWS Glue tables [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #15244: Add skip archive option when syncing to AWS Glue tables
URL: https://github.com/apache/hudi/issues/15244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add skip archive option when syncing to AWS Glue tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15244:
URL: https://github.com/apache/hudi/issues/15244#issuecomment-3630143113
### Linked PR(s)
- https://github.com/apache/hudi/pull/5900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org