You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/05 20:07:24 UTC
[GitHub] [hudi] wqwl611 opened a new pull request, #6858: [HUDI-4824]RANGE Bucket index, base logic and test.
wqwl611 opened a new pull request, #6858:
URL: https://github.com/apache/hudi/pull/6858
### Change Logs
The rangeBucket is mainly used in the scenario of sync mysql tables to hudi in near real time, which avoids the disadvantage of the fixed number of buckets in simpleBucket.
Usually, in the mysql table, there is an auto-increment id primary key field. In the mysql cdc synchronization scenario, we can use the database name and table name as the partition field of hudi, and id as the primary key field of the hudi table，This can deal with sub-library and sub-table.
In order to reach better sync performance, we usually use bucket index, but if we use simple bucket index, because the number of buckets is fixed, it is difficult for us to determine a suitable number of buckets, and as the table grows, The previous number of buckets will no longer be appropriate.
So, I propose rangeBucekt, in the simpleBucket index, the bucket number is （hash % bucketNum）, and in rangetBucket, we will use （ id / fixedStep） to determine the bucket number, so that as the id grows，The number of buckets also increases. For example, if step = 10 is set, then, because the id is self-increasing, a bucket will be generated for every 10 pieces of data.
In the actual scenario, I set step=1,000,000, the usual size of each mysql record is similar, then the approximate size of each bucket will be 50M ~ 350M, which avoids the disadvantage of the fixed number of buckets in simpleBucket
### Impact
Introduce a new index RANGE_BUCKET, people can ust it like following:
option(HoodieIndexConfig.INDEX_TYPE.key, IndexType.BUCKET.name()).
option(HoodieIndexConfig.BUCKET_INDEX_ENGINE_TYPE.key, "RANGE_BUCKET").
option(HoodieIndexConfig.RANGE_BUCKET_STEP_SIZE.key, 2).
option(HoodieLayoutConfig.LAYOUT_TYPE.key, "BUCKET").
**Risk level: none | low | medium | high**
low
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wqwl611 commented on pull request #6858: [HUDI-4824]RANGE Bucket index, base logic and test.
Posted by GitBox <gi...@apache.org>.
wqwl611 commented on PR #6858:
URL: https://github.com/apache/hudi/pull/6858#issuecomment-1265709727
I accidentally deleted my previous fork, so I opened a new PR, and modified the code according to the previous review requirements, please help me review it，thanks.
@danny0405 @YuweiXiao @xushiyan @alexeykudinkin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6858: [HUDI-4824]RANGE BUCKET index, base logic and test.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6858:
URL: https://github.com/apache/hudi/pull/6858#issuecomment-1279832831
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "76d14eb325d62a026248cc5c30de0d415e0c92a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11975",
"triggerID" : "76d14eb325d62a026248cc5c30de0d415e0c92a2",
"triggerType" : "PUSH"
}, {
"hash" : "61462e2a87cb320bd3c8b20c6ac6fe17de2501bd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12223",
"triggerID" : "61462e2a87cb320bd3c8b20c6ac6fe17de2501bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 61462e2a87cb320bd3c8b20c6ac6fe17de2501bd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12223)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6858: [HUDI-4824]RANGE BUCKET index, base logic and test.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6858:
URL: https://github.com/apache/hudi/pull/6858#issuecomment-1279805413
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "76d14eb325d62a026248cc5c30de0d415e0c92a2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11975",
"triggerID" : "76d14eb325d62a026248cc5c30de0d415e0c92a2",
"triggerType" : "PUSH"
}, {
"hash" : "61462e2a87cb320bd3c8b20c6ac6fe17de2501bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "61462e2a87cb320bd3c8b20c6ac6fe17de2501bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 76d14eb325d62a026248cc5c30de0d415e0c92a2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11975)
* 61462e2a87cb320bd3c8b20c6ac6fe17de2501bd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4824]RANGE BUCKET index, base logic and test. [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #6858: [HUDI-4824]RANGE BUCKET index, base logic and test.
URL: https://github.com/apache/hudi/pull/6858
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6858: [HUDI-4824]RANGE BUCKET index, base logic and test.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6858:
URL: https://github.com/apache/hudi/pull/6858#issuecomment-1265746656
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "76d14eb325d62a026248cc5c30de0d415e0c92a2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11975",
"triggerID" : "76d14eb325d62a026248cc5c30de0d415e0c92a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 76d14eb325d62a026248cc5c30de0d415e0c92a2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11975)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6858: [HUDI-4824]RANGE BUCKET index, base logic and test.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6858:
URL: https://github.com/apache/hudi/pull/6858#issuecomment-1265741058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "76d14eb325d62a026248cc5c30de0d415e0c92a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "76d14eb325d62a026248cc5c30de0d415e0c92a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 76d14eb325d62a026248cc5c30de0d415e0c92a2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6858: [HUDI-4824]RANGE BUCKET index, base logic and test.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6858:
URL: https://github.com/apache/hudi/pull/6858#issuecomment-1279806465
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "76d14eb325d62a026248cc5c30de0d415e0c92a2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11975",
"triggerID" : "76d14eb325d62a026248cc5c30de0d415e0c92a2",
"triggerType" : "PUSH"
}, {
"hash" : "61462e2a87cb320bd3c8b20c6ac6fe17de2501bd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12223",
"triggerID" : "61462e2a87cb320bd3c8b20c6ac6fe17de2501bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 76d14eb325d62a026248cc5c30de0d415e0c92a2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11975)
* 61462e2a87cb320bd3c8b20c6ac6fe17de2501bd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12223)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-4824]RANGE BUCKET index, base logic and test. [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #6858:
URL: https://github.com/apache/hudi/pull/6858#issuecomment-6699992228
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #6858: [HUDI-4824]RANGE BUCKET index, base logic and test.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #6858:
URL: https://github.com/apache/hudi/pull/6858#issuecomment-1280363263
Thanks for the contribution, i have reviewed and applied a patch:
[4824.patch.zip](https://github.com/apache/hudi/files/9797534/4824.patch.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6858: [HUDI-4824]RANGE BUCKET index, base logic and test.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6858:
URL: https://github.com/apache/hudi/pull/6858#issuecomment-1283618796
interesting index to support. thanks for the contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org