You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2025/04/21 09:41:07 UTC
[PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
TheR1sing3un opened a new pull request, #13189:
URL: https://github.com/apache/hudi/pull/13189
We execute a sql with:
1. `bulk insert` operation
2. `92160 ` buckets per partition
3. `set hoodie.bulkinsert.shuffle.parallelism=92160`
And then there a error occurs:
<img width="894" alt="image" src="https://github.com/user-attachments/assets/5fa5f11e-06fb-467e-8c60-527b486f95e9" />
The reason for this error is that when our partitioner calculates the partition id, it returns a number that is not within the parallelism range.
As for why an illegal id was returned, is it because there is an overflow of int type during the process of calculating the id.
<img width="937" alt="image" src="https://github.com/user-attachments/assets/8084a4f2-0ffe-401a-8ea3-c1cfe3c7fab9" />
### Change Logs
1. avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil`
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
fix the error when we set a large buckets num and large `hoodie.bulkinsert.shuffle.parallelism`
### Risk level (write none, low medium or high below)
low
### Documentation Update
none
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13189:
URL: https://github.com/apache/hudi/pull/13189#issuecomment-2818131657
> https://github.com/apache/hudi/actions/runs/14571270475/job/40868869529?pr=13189 it there a flaky test?
yeah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13189:
URL: https://github.com/apache/hudi/pull/13189#discussion_r2053216245
##########
hudi-common/src/main/java/org/apache/hudi/common/util/hash/BucketIndexUtil.java:
##########
@@ -37,9 +38,12 @@ public class BucketIndexUtil {
*/
public static Functions.Function3<Integer, String, Integer, Integer> getPartitionIndexFunc(int parallelism) {
return (bucketNum, partition, curBucket) -> {
- int partitionIndex = (partition.hashCode() & Integer.MAX_VALUE) % parallelism * bucketNum;
- int globalIndex = partitionIndex + curBucket;
- return globalIndex % parallelism;
+ long partitionIndex = (partition.hashCode() & Integer.MAX_VALUE) % parallelism * (long) bucketNum;
+ long globalIndex = partitionIndex + curBucket;
+ int partitionId = (int) (globalIndex % parallelism);
Review Comment:
Is this long to int causing anything unexpected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13189:
URL: https://github.com/apache/hudi/pull/13189#discussion_r2053224319
##########
hudi-common/src/main/java/org/apache/hudi/common/util/hash/BucketIndexUtil.java:
##########
@@ -37,9 +38,12 @@ public class BucketIndexUtil {
*/
public static Functions.Function3<Integer, String, Integer, Integer> getPartitionIndexFunc(int parallelism) {
return (bucketNum, partition, curBucket) -> {
- int partitionIndex = (partition.hashCode() & Integer.MAX_VALUE) % parallelism * bucketNum;
- int globalIndex = partitionIndex + curBucket;
- return globalIndex % parallelism;
+ long partitionIndex = (partition.hashCode() & Integer.MAX_VALUE) % parallelism * (long) bucketNum;
+ long globalIndex = partitionIndex + curBucket;
+ int partitionId = (int) (globalIndex % parallelism);
Review Comment:
> Is this long to int causing anything unexpected?
I think not. Because according to the calculation logic, the step of `% parallelism ` will ensure that the calculated result is smaller than `parallelism `, and `parallelism ` is a variable of `int` type. So even if we force cast the result from `long` to `int`, no accident will occur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13189:
URL: https://github.com/apache/hudi/pull/13189#issuecomment-2818287564
https://dev.azure.com/apachehudi/hudi-oss-ci/_build/results?buildId=5020&view=results Azure CI passed, Please have a look! @danny0405 @zhangyue19921010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13189:
URL: https://github.com/apache/hudi/pull/13189#discussion_r2052211416
##########
hudi-common/src/main/java/org/apache/hudi/common/util/hash/BucketIndexUtil.java:
##########
@@ -37,9 +38,12 @@ public class BucketIndexUtil {
*/
public static Functions.Function3<Integer, String, Integer, Integer> getPartitionIndexFunc(int parallelism) {
return (bucketNum, partition, curBucket) -> {
Review Comment:
Maybe we can change `Functions.Function3<Integer, String, Integer, Integer>` to `Functions.Function3<Long, String, Integer, Integer>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun merged PR #13189:
URL: https://github.com/apache/hudi/pull/13189
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13189:
URL: https://github.com/apache/hudi/pull/13189#issuecomment-2818054454
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5740eae879a38f3c76fa772dd9e82e6b2baf02c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5740eae879a38f3c76fa772dd9e82e6b2baf02c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5740eae879a38f3c76fa772dd9e82e6b2baf02c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13189:
URL: https://github.com/apache/hudi/pull/13189#issuecomment-2818369943
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5740eae879a38f3c76fa772dd9e82e6b2baf02c9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5020",
"triggerID" : "5740eae879a38f3c76fa772dd9e82e6b2baf02c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5740eae879a38f3c76fa772dd9e82e6b2baf02c9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5020)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13189:
URL: https://github.com/apache/hudi/pull/13189#discussion_r2052214647
##########
hudi-common/src/main/java/org/apache/hudi/common/util/hash/BucketIndexUtil.java:
##########
@@ -37,9 +38,12 @@ public class BucketIndexUtil {
*/
public static Functions.Function3<Integer, String, Integer, Integer> getPartitionIndexFunc(int parallelism) {
return (bucketNum, partition, curBucket) -> {
Review Comment:
> Maybe we can change `Functions.Function3<Integer, String, Integer, Integer>` to `Functions.Function3<Long, String, Integer, Integer>`
Anyway, bucket-num is expressed using the int type in all code paths. I think this change is not necessary. After all, we just want to avoid overflow during the calculation process. The input parameters and the output partition id itself all need to be of int type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13189:
URL: https://github.com/apache/hudi/pull/13189#issuecomment-2818056902
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5740eae879a38f3c76fa772dd9e82e6b2baf02c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5020",
"triggerID" : "5740eae879a38f3c76fa772dd9e82e6b2baf02c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5740eae879a38f3c76fa772dd9e82e6b2baf02c9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5020)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9329] Avoid obtaining an incorrect partition id due to int type overflow in `BucketIndexUtil` [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13189:
URL: https://github.com/apache/hudi/pull/13189#issuecomment-2818111245
https://github.com/apache/hudi/actions/runs/14571270475/job/40868869529?pr=13189 it there a flaky test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org