You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2026/10/23 02:33:55 UTC
Re: [I] Support REGISTER_ONLY bootstrap mode for cold storage partitions [hudi]
nsivabalan commented on issue #18135:
URL: https://github.com/apache/hudi/issues/18135#issuecomment-6356000305
Hey @shangxinli,
Thanks for the write up. before we dive into the approaches, I would like to align on the requirements.
Can you confirm these are the requirements.
```
R1. No reads of cold files at registration/bootstrap. Only listing (names and sizes) is allowed.
R2. One table, all tiers. Hot/warm partitions are native Hudi; cold partitions stay at the old Hive path. One query sees all.
R3. A partition lives in exactly one place: under the Hudi table base path or under the old base path, never both. in other words, if a partition is tagged as cold (external path), it can never move to native hudi layer or new target path. This goes hand in hand w/ R4.
R4. Old partitions are read-only. If one needs updates, the whole partition is first moved under the table base path. But the move as such will not happen seamlessly during a new write. Its an admin operation outside of ingestion.
R5: Any kind of writes to older partitions are rejected (ingestion (insert, upsert, deletes), table services, insert overwrite, delete_partition etc). Only reads are allowed based on partition predicates.
R6. Correct results or a clear failure, never silently missing rows: across readers with and without the metadata table (MDT), atleast for readers who used the patched version. For older readers who may not patch the support we plan to develop, we can't guarantee fail fast.
R7: Supports only single partition column. preferrably only datestr style partitions. So that during bootstrap, users can provide a predicate for list of partitions to be kept in cold storage.
R8: As long as no one issues reads to older partitions, hudi should not resurrect the data files for those cold partitions to hot tier. But if someone were to trigger a read either w/o any partition predicate or by supplying a predicate matching cold partitions, Hudi will serve data for the cold partitions, on which case, data files will be resurrected to hot storage.
R9. No breaking storage-format change unless versioned and gated.
```
Clarification:
```
1: Would any of these be mutable streams? if so, global indexes? and whats the configuration for `update.partition.path`? Irrespective of the config value, if we are looking for global index, our indexing will result in reading record keys from all partitions. How would we guard against cold data files resurrecting to hot tier in this case?
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org