You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "zhangyue19921010 (via GitHub)" <gi...@apache.org> on 2025/01/24 09:19:21 UTC
[I] [Feature Discuss] Partition-Level Bucket Index [hudi]
zhangyue19921010 opened a new issue, #12699:
URL: https://github.com/apache/hudi/issues/12699
Hi Hudis:
As we known, Hudi proposed and introduced Bucket Index in RFC-29. Bucket Index can well unify the indexes of Flink and Spark, that is, Spark and Flink could upsert the same Hudi table using bucket index.
However, Bucket Index has a limit of fixed number of buckets. In order to solve this problem, RFC-42 proposed the ability of consistent hashing achieving bucket resizing by splitting or merging several local buckets dynamically.
But from PRD experience, sometimes we only need to implement the Partition-Level Bucket Index and could do offline bucket rescale without introducing additional efforts (multiple writes, clustering, automatic resizing,etc.). Because the more complex the Architecture, the more error-prone it is and the greater the operation and maintenance pressure.
In this regard, I want to **upgrade the traditional Bucket Index to implement a Partition-Level Bucket Index,** so that users can set a specific number of buckets for different partitions through a rule engine (such as regular expression matching). On the other hand, for a certain existing partitions, an off-line command is provided to reorganized the data using insert overwrite(need to stop the data writing of the current partition).
Some thoughts on this change? Any feedback would be greatly appreciated !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Discuss] Partition-Level Bucket Index [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12699:
URL: https://github.com/apache/hudi/issues/12699#issuecomment-2614743629
Are you saying to support configuring the bucket number in partition-level? Can the bucket number be changed after an explicit configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Discuss] Partition-Level Bucket Index [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on issue #12699:
URL: https://github.com/apache/hudi/issues/12699#issuecomment-2641737498
> Can the bucket number be dynamically inferred with the inputs? Hudi has a partition metadata file under each partition, maybe we can record the bucket number there.
Hi danny, Thanks for your replay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Discuss] Partition-Level Bucket Index [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on issue #12699:
URL: https://github.com/apache/hudi/issues/12699#issuecomment-2612045696
Reference: Bucket Rescale in Paimon https://paimon.apache.org/docs/master/maintenance/rescale-bucket/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Discuss] Partition-Level Bucket Index [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12699:
URL: https://github.com/apache/hudi/issues/12699#issuecomment-2635554315
Can the bucket number be dynamically inferred with the inputs? Hudi has a partition metadata file under each partition, maybe we can record the bucket number there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Discuss] Partition-Level Bucket Index [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on issue #12699:
URL: https://github.com/apache/hudi/issues/12699#issuecomment-2642684431
> [@zhangyue19921010](https://github.com/zhangyue19921010) We have implemented dynamic partition bucketing, which supports regular expressions, similar to your idea. The only difference is that we store bucket information in the ./hoodie/.bucket directory. Since the bucket information is minimal, it's efficient to store it in a single file. This approach simplifies the process of retrieving partition-level bucket counts and performing bucket pruning. At the same time, with the help of Hudi's timeline, we can easily ensure the consistency of bucket information
Hi @xiarixiaoyao Thanks for your replay! It seems that dynamic partition-level bucket index is indeed a common requirement.
`./hoodie/.bucket directory` is a good idea. But how to solve the problem of two jobs concurrently writing? At this time, there may be multiple tasks operating on the partition meta file(Even if wrote to different partitions)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Discuss] Partition-Level Bucket Index [hudi]
Posted by "xiarixiaoyao (via GitHub)" <gi...@apache.org>.
xiarixiaoyao commented on issue #12699:
URL: https://github.com/apache/hudi/issues/12699#issuecomment-2642607291
@zhangyue19921010 We have implemented dynamic partition bucketing, which supports regular expressions, similar to your idea. The only difference is that we store bucket information in the ./hoodie/.bucket directory. Since the bucket information is minimal, it's efficient to store it in a single file. This approach simplifies the process of retrieving partition-level bucket counts and performing bucket pruning. At the same time, with the help of Hudi's timeline, we can easily ensure the consistency of bucket information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Discuss] Partition-Level Bucket Index [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #12699: [Feature Discuss] Partition-Level Bucket Index
URL: https://github.com/apache/hudi/issues/12699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Discuss] Partition-Level Bucket Index [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #12699:
URL: https://github.com/apache/hudi/issues/12699#issuecomment-2646787312
@xiarixiaoyao Can you share the data structure under `./hoodie/.bucket` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Discuss] Partition-Level Bucket Index [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on issue #12699:
URL: https://github.com/apache/hudi/issues/12699#issuecomment-2624434096
> Are you saying to support configuring the bucket number in partition-level? Can the bucket number be changed after an explicit configuration?
Hi @danny0405
Thanks for your attention. Yes, we will provide a new mechanism to specify the number of buckets for different partitions through an expression. For existing partitions, we can change current partition's bucket number through an offline job (Insert Overwrite) with the new bucket number. For new partitions, the initial value of the bucket number is obtained based on the expression. Additionally, updates to the expression are also supported, but the updated expression will only take effect for new partitions.
Function similar to https://paimon.apache.org/docs/master/maintenance/rescale-bucket/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Discuss] Partition-Level Bucket Index [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on issue #12699:
URL: https://github.com/apache/hudi/issues/12699#issuecomment-2614760416
Hi Danny,
Thanks for your attention. Yes, we will provide a new mechanism to specify the number of buckets for different partitions through an expression. For existing partitions, we can change current partition's bucket number through an offline job (Insert Overwrite) with the new bucket number. For new partitions, the initial value of the bucket number is obtained based on the expression. Additionally, updates to the expression are also supported, but the updated expression will only take effect for new partitions.
Function similar to https://paimon.apache.org/docs/master/maintenance/rescale-bucket/
Best regards,
zhangyue19921010
At 2025-01-27 10:31:18, "Danny Chan" ***@***.***> wrote:
Are you saying to support configuring the bucket number in partition-level? Can the bucket number be changed after an explicit configuration?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org