You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "kbuci (via GitHub)" <gi...@apache.org> on 2026/01/16 02:18:47 UTC
[I] [to be discussed] Support having metadata table compaction plans be safely executed by other writer service [hudi]
kbuci opened a new issue, #17908:
URL: https://github.com/apache/hudi/issues/17908
### Task Description
**What needs to be done:**
- Implement https://issues.apache.org/jira/browse/HUDI-9407 so that users can specify using `OPTIMISTIC_CONCURRENCY_CONTROL` when creating metadata table writer `org.apache.hudi.metadata.HoodieMetadataWriteUtils#createMetadataWriteConfig`
Add a new config that lists the table services supported for inline scheduling but async execution in metadata table. Initially we can choose to support just compaction and logcompaction. If enabled,
- metadata table writer should still schedule all table service plans inline but not execute the specified types of table service plans inline. Note that this means it should also not re-try these plans in `org.apache.hudi.metadata.HoodieBackedTableMetadataWriter#runPendingTableServicesOperationsAndRefreshTimeline`
- Another concurrent writer can initialize a metadata table writer (passing in OCC as concurrency type in the metadata write config) and execute these plans.
**Why this task is needed:**
For datasets with large metadata table partitions (like RECORD_INDEX) we cannot have metadata table compaction be executed during the write, as that will impact runtimes. Rather, we only schedule the plan inline, and have a separate platform that executes these plans. We need the above features to ensure that we can have specific table services on the metadata table be guaranteed to not execute inline during write. And instead, outside concurrent writers can safely execute these plans. This "async" execution will only take the table lock when transitioning the instants, in order to not block the writer job. We can upstream our implementations for above once we reach consensus.
### Task Type
Code improvement/refactoring
### Related Issues
**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [to be discussed] Support having other concurrent writer service being able to directly call `performTableServices` and execute metadata table compaction plans (outside of ingestion) [hudi]
Posted by "kbuci (via GitHub)" <gi...@apache.org>.
kbuci commented on issue #17908:
URL: https://github.com/apache/hudi/issues/17908#issuecomment-3923891773
>
> ```
> In addition, regardless of the above async execution, we should ensure that it is safe for a concurrent writer to directly call performTableServices
> ...
> ```
Actually, let me move this out and put it in a separate issue. For context, I added this since we want a way to have our table service platform perform all MDT table services for a dataset that is no longer receiving ingestion writes to the data table. Since in our org, we can do a backfill for clustering writes on a dataset, and our internal build of HUDI we don't perform MDT table services during clustering. For this case it's fine if we hold the table lock the whole time (for table version 6) while calling `performTableServices` - but we wanted to make sure there weren't any other edge case in doing this, since although hypotetncally it should be safe we've not attempted this in production before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [to be discussed] Support having other concurrent writer service being able to directly call `performTableServices` and execute metadata table compaction plans (outside of ingestion) [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #17908:
URL: https://github.com/apache/hudi/issues/17908#issuecomment-3923436575
thanks for the feature request.
Seems a valid one for high scale users like uber.
Can you clarify this statement though
```
In addition, regardless of the above async execution, we should ensure that it is safe for a concurrent writer to directly call performTableServices
...
```
note sure I get this.
Back to the requirements:
Essentially, you are looking for ways to offload the execution to async job so that we can unblock the ingestion writer asap. And the compaction execution does not hold the table lock for extended period of time, but just when wrapping up the compaction commit. Hence, this improves the throughput of the system in general.
- We have some intricacies or inter dependencies between data table archival, metadata table compaction, metadata archival, and rollbacks. Have we accounted for scenarios w/ your proposed solution.
- What happens if the async execution did not happen at all? Do you have any guard rails to fail ingestion if we reach some max no of log files in mdt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [to be discussed] Support having other concurrent writer service execute metadata table compaction plans (outside of ingestion) [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #17908:
URL: https://github.com/apache/hudi/issues/17908#issuecomment-3923972917
thanks.
NTS: lets think through separate mdt locks for viable option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org