You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "wangyinsheng (via GitHub)" <gi...@apache.org> on 2026/07/21 13:03:31 UTC
[I] [SUPPORT][RFC] Support staged / incremental Metadata Table (MDT) initialization for large tables with long partition history [hudi]
wangyinsheng opened a new issue, #19339:
URL: https://github.com/apache/hudi/issues/19339
### Describe the problem you faced
### Problem：
When enabling the Metadata Table (MDT) on existing Hudi tables that have accumulated a large partition history, MDT bootstrap becomes a hard blocker for the writer. In our production setup we routinely operate tables with the following characteristics:
- Daily partitioning strategy (one partition per day)
- ~2 years of retained partitions (≈ 700+ partitions)
- Hundreds of thousands of files per partition
At this scale, the initial HoodieBackedTableMetadataWriter bootstrap performs a full-table FILES (and optionally COLUMN_STATS / PARTITION_STATS / BLOOM_FILTERS) scan across all historical partition before the writer is allowed to resume normal ingestion. The consequences as follows:
1. The upstream writer (Flink / Spark streaming) has to be paused for a very long window while MDT is being populated.
2. Business SLAs on ingestion latency are broken during the one-time MDT rollout, which in practice discourages teams from turning MDT on at all.
### To Reproduce
1.
2.
3.
4.
### Expected behavior
### Proposal
We would like to open a discussion on a staged / phased MDT initialization flow, roughly:
1. Phase 1 — Foreground bootstrap of hot partitions only.
During the initial write-side enablement, only bootstrap the partition(s) that the current writer is actively producing into (e.g. today's partition, or the last N partitions defined by a config knob). This should complete in seconds/minutes and unblock the writer.
2. Phase 2 — Asynchronous / offline backfill of historical partitions.
Provide a Spark SQL CALL procedure that can be triggered independently — e.g. `CALL run_metadata_backfill(table => '...', partitions => '...', parallelism => ...)`. This job walks historical partitions and appends the corresponding MDT records
incrementally, without blocking the online writer. It should be resumable, idempotent, and safe to run concurrently with the writer (or at least with well-defined isolation via existing lock providers / TxnManager).
3. Phase 3 — Reader activation gate.
MDT should stay invisible / disabled for readers until an explicit "activation" step is performed (after Phase 2 completes). Concretely, readers should either (a) fall back to filesystem listing for partitions that have not yet been backfilled, or (b) treat MDT as unavailable until
a table-level flag / property (e.g. `hoodie.metadata.ready = true`) is set. This avoids the pitfall of readers getting a partially-populated MDT view during backfill and returning incorrect / incomplete results.
### Environment Description
* Hudi version:
* Spark version:
* Flink version:
* Hive version:
* Hadoop version:
* Storage (HDFS/S3/GCS..):
* Running on Docker? (yes/no):
### Additional context
_No response_
### Stacktrace
```shell
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT][RFC] Support staged / incremental Metadata Table (MDT) initialization for large tables with long partition history [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #19339:
URL: https://github.com/apache/hudi/issues/19339#issuecomment-5086958854
the proposal may work if RLI is not enabled, since the write path works with file listing for paths like COW merge. But if RLI is enabled, the write path needs to query the `record key -> file group` mappings, did you have any chance to though about that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org