You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2026/08/25 04:11:37 UTC
[PR] feat(spark): support bucket index for LSM tables [hudi]
cshuo opened a new pull request, #19727:
URL: https://github.com/apache/hudi/pull/19727
### Describe the issue this Pull Request addresses
Closes #19439.
Spark bucket-index partitioners own file-group routing and file-id assignment, so LSM tables cannot replace them with the generic LSM bulk-insert partitioners. The existing bucket paths therefore need to preserve their routing behavior while guaranteeing UTF-8 record-key ordering inside every output bucket.
### Summary and Changelog
- Sort records within each LSM bucket by the full record key using UTF-8 ordering while preserving existing simple and consistent-hashing bucket routing and file-id assignment.
- Enable the ordering for both RDD and Dataset Row bulk-insert paths, including the Dataset bucket-rescale path, without changing non-LSM ordering behavior or the Row schema.
- Reject custom bucket sort columns for LSM tables before writes reach the inflight state.
- Add unit and functional coverage for non-ASCII record keys, COW and MOR tables, simple and consistent-hashing bucket indexes, RDD and Row writers, subsequent upserts, and file-id stability.
### Impact
This enables Spark LSM tables to use simple and consistent-hashing bucket indexes. It introduces no new public API or configuration. LSM Dataset Row simple-bucket writes now include the record key in the shuffle sort key, adding the key copy and comparisons required to maintain the LSM physical ordering invariant; other table layouts retain their existing behavior.
### Risk Level
Medium. The change affects Spark bucket-index write partitioning and ordering. The risk is mitigated by 14 passing partitioner tests and 7 passing targeted functional cases covering routing, UTF-8 ordering, schema preservation, custom-sort rejection, file-id stability, and snapshot correctness after upsert. Compilation, the relevant Checkstyle and Scalastyle checks, and `git diff --check` also completed without violations.
### Documentation Update
None.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5433330519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
}, {
"hash" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529",
"triggerID" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"triggerType" : "PUSH"
}, {
"hash" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16605",
"triggerID" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44cf6615b83109445008bb2beffcfd93f9ee1176 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529)
* 365ab3c277f0dbde59402b19642a998bce81fda3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16605)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3871577515
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/RDDBucketIndexPartitioner.java:
##########
@@ -121,7 +122,9 @@ private JavaRDD<HoodieRecord<T>> doPartitionAndSortByRecordKey(JavaRDD<HoodieRec
LOG.warn("Bucket index does not support global sort mode, the sort will only be done within each data partition");
}
- Comparator<HoodieKey> comparator = (Comparator<HoodieKey> & Serializable) (t1, t2) -> t1.getRecordKey().compareTo(t2.getRecordKey());
+ Comparator<HoodieKey> comparator = table.getMetaClient().getTableConfig().isLSMTreeStorageLayout()
Review Comment:
🤖 nit: the two ternary branches differ only in the string-compare call — could you pull the boolean out and build a single comparator (e.g. select a `BiFunction<String,String,Integer>` and wrap it once) to avoid duplicating the lambda?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5408003227
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd62bcb525f367c35ebf37bb6035196c3c4b96e8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3853514284
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/BucketSortBulkInsertPartitioner.java:
##########
@@ -32,13 +34,26 @@ public abstract class BucketSortBulkInsertPartitioner<T> implements BulkInsertPa
public BucketSortBulkInsertPartitioner(HoodieTable table, String sortString) {
this.table = table;
+ validateCustomSortColumns(table, sortString);
if (!StringUtils.isNullOrEmpty(sortString)) {
this.sortColumnNames = sortString.split(",");
} else {
this.sortColumnNames = null;
}
}
+ public static void validateCustomSortColumns(HoodieTable table, String sortString) {
Review Comment:
Fixed in 44cf6615b831. Good point — this config is ignored by bucket-index partitioners, as it was before LSM support, so validating it was redundant. The validation now only checks the actual sortString.
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/commit/DatasetBucketRescaleCommitActionExecutor.java:
##########
@@ -57,7 +57,9 @@ public DatasetBucketRescaleCommitActionExecutor(HoodieWriteConfig config,
*/
@Override
protected BulkInsertPartitioner<Dataset<Row>> getPartitioner(boolean populateMetaFields, boolean isTablePartitioned) {
- return new BucketIndexBulkInsertPartitionerWithRows(writeClient.getConfig(), expression, rule, bucketNumber);
+ return new BucketIndexBulkInsertPartitionerWithRows(
Review Comment:
Addressed in 44cf6615b831. The config validation was removed because BULKINSERT_USER_DEFINED_PARTITIONER_SORT_COLUMNS is ignored by bucket-index partitioners. Therefore the rescale path no longer bypasses a relevant validation or risks failing on this config after persisting the hashing config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3852080703
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/BucketSortBulkInsertPartitioner.java:
##########
@@ -32,13 +34,26 @@ public abstract class BucketSortBulkInsertPartitioner<T> implements BulkInsertPa
public BucketSortBulkInsertPartitioner(HoodieTable table, String sortString) {
this.table = table;
+ validateCustomSortColumns(table, sortString);
if (!StringUtils.isNullOrEmpty(sortString)) {
this.sortColumnNames = sortString.split(",");
} else {
this.sortColumnNames = null;
}
}
+ public static void validateCustomSortColumns(HoodieTable table, String sortString) {
Review Comment:
didn't quite get this, the `BucketSortBulkInsertPartitioner` always utilitize passed in `sortString` but why we are validating the `HoodieWriteConfig.BULKINSERT_USER_DEFINED_PARTITIONER_SORT_COLUMNS` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5412611622
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
}, {
"hash" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529",
"triggerID" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44cf6615b83109445008bb2beffcfd93f9ee1176 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5411304150
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
}, {
"hash" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529",
"triggerID" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd62bcb525f367c35ebf37bb6035196c3c4b96e8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520)
* 44cf6615b83109445008bb2beffcfd93f9ee1176 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5411228586
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
}, {
"hash" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd62bcb525f367c35ebf37bb6035196c3c4b96e8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520)
* 44cf6615b83109445008bb2beffcfd93f9ee1176 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5439019650
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
}, {
"hash" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529",
"triggerID" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"triggerType" : "PUSH"
}, {
"hash" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16605",
"triggerID" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"triggerType" : "PUSH"
}, {
"hash" : "43c8b4619d3dab413d8ca27a4932590d955b07a9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16630",
"triggerID" : "43c8b4619d3dab413d8ca27a4932590d955b07a9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 365ab3c277f0dbde59402b19642a998bce81fda3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16605)
* 43c8b4619d3dab413d8ca27a4932590d955b07a9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16630)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5405227815
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19727?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `7 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 77.87%. Comparing base ([`81fd499`](https://app.codecov.io/gh/apache/hudi/commit/81fd4992b8dc0687b8ccfa6190d0570682e55ee1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d17fb84`](https://app.codecov.io/gh/apache/hudi/commit/d17fb84130673182bdcd180ba679eb7010fbddf0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19727?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...he/hudi/table/BucketSortBulkInsertPartitioner.java](https://app.codecov.io/gh/apache/hudi/pull/19727?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Ftable%2FBucketSortBulkInsertPartitioner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL0J1Y2tldFNvcnRCdWxrSW5zZXJ0UGFydGl0aW9uZXIuamF2YQ==) | 0.00% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19727?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19727 +/- ##
============================================
- Coverage 77.88% 77.87% -0.01%
+ Complexity 33258 33252 -6
============================================
Files 2533 2533
Lines 140328 140325 -3
Branches 16912 16915 +3
============================================
- Hits 109288 109278 -10
- Misses 23403 23410 +7
Partials 7637 7637
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.36% <ø> (ø)` | |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.94% <0.00%> (-0.03%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.56% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.36% <ø> (-0.01%)` | :arrow_down: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.29% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.06% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.08% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.58% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.85% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.33% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19727/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19727/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19727/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.89% <0.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19727?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...sert/BucketIndexBulkInsertPartitionerWithRows.java](https://app.codecov.io/gh/apache/hudi/pull/19727?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fexecution%2Fbulkinsert%2FBucketIndexBulkInsertPartitionerWithRows.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL2J1bGtpbnNlcnQvQnVja2V0SW5kZXhCdWxrSW5zZXJ0UGFydGl0aW9uZXJXaXRoUm93cy5qYXZh) | `91.66% <ø> (-2.09%)` | :arrow_down: |
| [...xecution/bulkinsert/RDDBucketIndexPartitioner.java](https://app.codecov.io/gh/apache/hudi/pull/19727?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fexecution%2Fbulkinsert%2FRDDBucketIndexPartitioner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXhlY3V0aW9uL2J1bGtpbnNlcnQvUkREQnVja2V0SW5kZXhQYXJ0aXRpb25lci5qYXZh) | `92.85% <ø> (-0.25%)` | :arrow_down: |
| [...la/org/apache/spark/sql/BucketPartitionUtils.scala](https://app.codecov.io/gh/apache/hudi/pull/19727?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2FBucketPartitionUtils.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9zcGFyay9zcWwvQnVja2V0UGFydGl0aW9uVXRpbHMuc2NhbGE=) | `86.95% <ø> (-1.51%)` | :arrow_down: |
| [...mit/BaseDatasetBulkInsertCommitActionExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19727?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommit%2FBaseDatasetBulkInsertCommitActionExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbW1pdC9CYXNlRGF0YXNldEJ1bGtJbnNlcnRDb21taXRBY3Rpb25FeGVjdXRvci5qYXZh) | `68.57% <ø> (-0.45%)` | :arrow_down: |
| [...mmit/DatasetBucketRescaleCommitActionExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19727?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommit%2FDatasetBucketRescaleCommitActionExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbW1pdC9EYXRhc2V0QnVja2V0UmVzY2FsZUNvbW1pdEFjdGlvbkV4ZWN1dG9yLmphdmE=) | `100.00% <ø> (ø)` | |
| [...he/hudi/table/BucketSortBulkInsertPartitioner.java](https://app.codecov.io/gh/apache/hudi/pull/19727?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Ftable%2FBucketSortBulkInsertPartitioner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL3RhYmxlL0J1Y2tldFNvcnRCdWxrSW5zZXJ0UGFydGl0aW9uZXIuamF2YQ==) | `43.75% <0.00%> (-34.03%)` | :arrow_down: |
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5406966763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d17fb84130673182bdcd180ba679eb7010fbddf0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517)
* bd62bcb525f367c35ebf37bb6035196c3c4b96e8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3871456388
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/BucketPartitionUtils.scala:
##########
@@ -49,10 +57,35 @@ object BucketPartitionUtils extends SparkAdapterSupport {
val getPartitionKey = getPartitionKeyExtractor()
// use internalRow to avoid extra convert.
- val reRdd = df.queryExecution.toRdd
- .keyBy(row => getPartitionKey(row))
- .repartitionAndSortWithinPartitions(partitioner)
- .values
+ val internalRows = df.queryExecution.toRdd
+ val reRdd = if (sortByRecordKey) {
+ val utf8StringFactory = sparkAdapter.getUTF8StringFactory
+ // Use (bucket route, record key) as the shuffle key. Tuple ordering groups rows by
+ // (partition path, bucket id) first, then orders each bucket by the full record key.
+ // Scala derives the record-key ordering from HoodieUTF8String's Comparable implementation,
+ // which uses binary UTF-8 ordering for both Spark 3 and Spark 4.
+ val keyedRows = internalRows.keyBy(row => {
+ // InternalRow may reuse its mutable backing buffer, so keep an independent key for shuffle.
+ val recordKey = utf8StringFactory.wrapUTF8String(
+ row.getUTF8String(HoodieRecord.RECORD_KEY_META_FIELD_ORD).copy())
Review Comment:
Makes sense. The Kryo shuffle serializes both key and value before advancing the iterator. Removed the redundant key copy and comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #19727:
URL: https://github.com/apache/hudi/pull/19727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5407031685
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d17fb84130673182bdcd180ba679eb7010fbddf0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517)
* bd62bcb525f367c35ebf37bb6035196c3c4b96e8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5440790493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
}, {
"hash" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529",
"triggerID" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"triggerType" : "PUSH"
}, {
"hash" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16605",
"triggerID" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"triggerType" : "PUSH"
}, {
"hash" : "43c8b4619d3dab413d8ca27a4932590d955b07a9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16630",
"triggerID" : "43c8b4619d3dab413d8ca27a4932590d955b07a9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43c8b4619d3dab413d8ca27a4932590d955b07a9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16630)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5406079516
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d17fb84130673182bdcd180ba679eb7010fbddf0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3852956100
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/RDDBucketIndexPartitioner.java:
##########
@@ -121,7 +122,10 @@ private JavaRDD<HoodieRecord<T>> doPartitionAndSortByRecordKey(JavaRDD<HoodieRec
LOG.warn("Bucket index does not support global sort mode, the sort will only be done within each data partition");
}
- Comparator<HoodieKey> comparator = (Comparator<HoodieKey> & Serializable) (t1, t2) -> t1.getRecordKey().compareTo(t2.getRecordKey());
+ boolean lsmTable = table.getMetaClient().getTableConfig().isLSMTreeStorageLayout();
Review Comment:
🤖 nit: since this is a boolean, could you prefix it with `is` — `isLsmTable` — so it reads as a predicate consistently with the rest of the codebase?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/commit/BaseDatasetBulkInsertCommitActionExecutor.java:
##########
@@ -140,8 +141,11 @@ public String getCommitActionType() {
protected BulkInsertPartitioner<Dataset<Row>> getPartitioner(boolean populateMetaFields, boolean isTablePartitioned) {
if (populateMetaFields) {
if (writeConfig.getIndexType() == HoodieIndex.IndexType.BUCKET) {
+ BucketSortBulkInsertPartitioner.validateCustomSortColumns(table, null);
Review Comment:
🤖 nit: passing `null` explicitly here is a bit surprising — the method has a `sortString` parameter, but at this call site there's no sort string at all. Would an overload `validateCustomSortColumns(HoodieTable table)` (delegating to the existing one with `null`) make the intent clearer?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3870192169
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/BucketPartitionUtils.scala:
##########
@@ -49,10 +57,35 @@ object BucketPartitionUtils extends SparkAdapterSupport {
val getPartitionKey = getPartitionKeyExtractor()
// use internalRow to avoid extra convert.
- val reRdd = df.queryExecution.toRdd
- .keyBy(row => getPartitionKey(row))
- .repartitionAndSortWithinPartitions(partitioner)
- .values
+ val internalRows = df.queryExecution.toRdd
+ val reRdd = if (sortByRecordKey) {
+ val utf8StringFactory = sparkAdapter.getUTF8StringFactory
+ // Use (bucket route, record key) as the shuffle key. Tuple ordering groups rows by
+ // (partition path, bucket id) first, then orders each bucket by the full record key.
+ // Scala derives the record-key ordering from HoodieUTF8String's Comparable implementation,
+ // which uses binary UTF-8 ordering for both Spark 3 and Spark 4.
+ val keyedRows = internalRows.keyBy(row => {
+ // InternalRow may reuse its mutable backing buffer, so keep an independent key for shuffle.
+ val recordKey = utf8StringFactory.wrapUTF8String(
+ row.getUTF8String(HoodieRecord.RECORD_KEY_META_FIELD_ORD).copy())
Review Comment:
Could we avoid this per-record copy? Hudi requires Kryo, and Spark’s serialized shuffle serializes both the key and value before advancing the input iterator, so the reduce-side sorter sees independently deserialized objects. Also, the shuffled value is still the original `row`; if this path retained a reused `InternalRow` buffer, copying only the key would preserve ordering but not the row contents. Please remove `.copy()` and the preceding comment. If this utility is intended to support a non-serialized shuffle path, it would need to copy the entire row once and derive the key from that copied row instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5433288927
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
}, {
"hash" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529",
"triggerID" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"triggerType" : "PUSH"
}, {
"hash" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 44cf6615b83109445008bb2beffcfd93f9ee1176 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529)
* 365ab3c277f0dbde59402b19642a998bce81fda3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3849547179
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/commit/BaseDatasetBulkInsertCommitActionExecutor.java:
##########
@@ -140,8 +141,11 @@ public String getCommitActionType() {
protected BulkInsertPartitioner<Dataset<Row>> getPartitioner(boolean populateMetaFields, boolean isTablePartitioned) {
if (populateMetaFields) {
if (writeConfig.getIndexType() == HoodieIndex.IndexType.BUCKET) {
+ BucketSortBulkInsertPartitioner.validateCustomSortColumns(table, null);
Review Comment:
🤖 nit: passing `null` here to mean "no sort string, just check the config" is a bit surprising at the call site. Could you add a no-arg overload `validateCustomSortColumns(HoodieTable table)` that only checks the config path, so callers don't need to know that `null` is a valid sentinel?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5405222354
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d17fb84130673182bdcd180ba679eb7010fbddf0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5433954623
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
}, {
"hash" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529",
"triggerID" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"triggerType" : "PUSH"
}, {
"hash" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16605",
"triggerID" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 365ab3c277f0dbde59402b19642a998bce81fda3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16605)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3868065570
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/BucketIndexBulkInsertPartitionerWithRows.java:
##########
@@ -36,20 +36,32 @@ public class BucketIndexBulkInsertPartitionerWithRows implements BulkInsertParti
private final String indexKeyFields;
private final NumBucketsFunction numBucketsFunction;
private final HoodieWriteConfig writeConfig;
+ private final boolean sortByRecordKey;
private FileSystemViewStorageConfig viewConfig;
- public BucketIndexBulkInsertPartitionerWithRows(String indexKeyFields, HoodieWriteConfig writeConfig) {
- this(writeConfig, NumBucketsFunction.fromWriteConfig(writeConfig), indexKeyFields);
+ public BucketIndexBulkInsertPartitionerWithRows(String indexKeyFields,
+ HoodieWriteConfig writeConfig,
+ boolean sortByRecordKey) {
Review Comment:
Fixed in 365ab3c277f0. Restored both public constructor overloads and delegated them to the boolean-aware constructors.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestLSMDataSource.scala:
##########
@@ -175,6 +176,57 @@ class TestLSMDataSource extends SparkClientFunctionalTestHarness {
"Ａ-row-p2" -> "v1"))
}
+ @ParameterizedTest
+ @MethodSource(Array("bucketBulkInsertParams"))
+ def testBucketIndexBulkInsert(
+ tableType: HoodieTableType,
+ bucketEngineType: HoodieIndex.BucketIndexEngineType,
+ enableRowWriter: Boolean): Unit = {
+ val tablePath = s"${basePath}_${tableType.name.toLowerCase}_${bucketEngineType.name.toLowerCase}_$enableRowWriter"
+ val options = baseOptions(tableType) ++ Map(
+ DataSourceWriteOptions.ENABLE_ROW_WRITER.key -> enableRowWriter.toString,
+ HoodieWriteConfig.BULK_INSERT_SORT_MODE.key -> BulkInsertSortMode.NONE.name,
+ HoodieIndexConfig.INDEX_TYPE.key -> HoodieIndex.IndexType.BUCKET.name,
+ HoodieIndexConfig.BUCKET_INDEX_ENGINE_TYPE.key -> bucketEngineType.name,
+ HoodieIndexConfig.BUCKET_INDEX_HASH_FIELD.key -> "id",
+ HoodieIndexConfig.BUCKET_INDEX_NUM_BUCKETS.key -> "1")
Review Comment:
Fixed in 365ab3c277f0. The test now uses two buckets and verifies record-to-file-group routing for simple and consistent hashing across RDD and Row writers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3852086982
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/RDDBucketIndexPartitioner.java:
##########
@@ -121,7 +122,10 @@ private JavaRDD<HoodieRecord<T>> doPartitionAndSortByRecordKey(JavaRDD<HoodieRec
LOG.warn("Bucket index does not support global sort mode, the sort will only be done within each data partition");
}
- Comparator<HoodieKey> comparator = (Comparator<HoodieKey> & Serializable) (t1, t2) -> t1.getRecordKey().compareTo(t2.getRecordKey());
+ boolean lsmTable = table.getMetaClient().getTableConfig().isLSMTreeStorageLayout();
+ Comparator<HoodieKey> comparator = (Comparator<HoodieKey> & Serializable) (t1, t2) -> lsmTable
Review Comment:
generate different compatator for different storage layout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3857223722
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/RDDBucketIndexPartitioner.java:
##########
@@ -121,7 +122,9 @@ private JavaRDD<HoodieRecord<T>> doPartitionAndSortByRecordKey(JavaRDD<HoodieRec
LOG.warn("Bucket index does not support global sort mode, the sort will only be done within each data partition");
}
- Comparator<HoodieKey> comparator = (Comparator<HoodieKey> & Serializable) (t1, t2) -> t1.getRecordKey().compareTo(t2.getRecordKey());
+ Comparator<HoodieKey> comparator = table.getMetaClient().getTableConfig().isLSMTreeStorageLayout()
Review Comment:
🤖 nit: the two ternary branches only differ in the comparison expression — could you extract a single `Comparator<HoodieKey>` that picks between `StringUtils.compareUtf8Bytes` and `String.compareTo` on the record keys to avoid duplicating the cast/lambda boilerplate?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3861784555
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/BucketIndexBulkInsertPartitionerWithRows.java:
##########
@@ -36,20 +36,32 @@ public class BucketIndexBulkInsertPartitionerWithRows implements BulkInsertParti
private final String indexKeyFields;
private final NumBucketsFunction numBucketsFunction;
private final HoodieWriteConfig writeConfig;
+ private final boolean sortByRecordKey;
private FileSystemViewStorageConfig viewConfig;
- public BucketIndexBulkInsertPartitionerWithRows(String indexKeyFields, HoodieWriteConfig writeConfig) {
- this(writeConfig, NumBucketsFunction.fromWriteConfig(writeConfig), indexKeyFields);
+ public BucketIndexBulkInsertPartitionerWithRows(String indexKeyFields,
+ HoodieWriteConfig writeConfig,
+ boolean sortByRecordKey) {
Review Comment:
[P1] Please retain the two existing public constructor overloads. This change replaces `(String, HoodieWriteConfig)` and `(HoodieWriteConfig, String, String, int)` with new signatures, so downstream code compiled against the current `hudi-spark-client` will fail with `NoSuchMethodError` (and source users will no longer compile), even for non-LSM tables. Delegating overloads can preserve compatibility while the new boolean-aware constructors serve the new call sites.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestLSMDataSource.scala:
##########
@@ -175,6 +176,57 @@ class TestLSMDataSource extends SparkClientFunctionalTestHarness {
"Ａ-row-p2" -> "v1"))
}
+ @ParameterizedTest
+ @MethodSource(Array("bucketBulkInsertParams"))
+ def testBucketIndexBulkInsert(
+ tableType: HoodieTableType,
+ bucketEngineType: HoodieIndex.BucketIndexEngineType,
+ enableRowWriter: Boolean): Unit = {
+ val tablePath = s"${basePath}_${tableType.name.toLowerCase}_${bucketEngineType.name.toLowerCase}_$enableRowWriter"
+ val options = baseOptions(tableType) ++ Map(
+ DataSourceWriteOptions.ENABLE_ROW_WRITER.key -> enableRowWriter.toString,
+ HoodieWriteConfig.BULK_INSERT_SORT_MODE.key -> BulkInsertSortMode.NONE.name,
+ HoodieIndexConfig.INDEX_TYPE.key -> HoodieIndex.IndexType.BUCKET.name,
+ HoodieIndexConfig.BUCKET_INDEX_ENGINE_TYPE.key -> bucketEngineType.name,
+ HoodieIndexConfig.BUCKET_INDEX_HASH_FIELD.key -> "id",
+ HoodieIndexConfig.BUCKET_INDEX_NUM_BUCKETS.key -> "1")
Review Comment:
[P2] Could this coverage use at least two buckets and assert the expected record-to-bucket/file-id mapping? With `NUM_BUCKETS = 1`, every key is forced through the same bucket, so the test only verifies partition-path routing; an implementation that sorts correctly but changes/ignores bucket routing would still pass. The PR’s central invariant is preserving bucket/file-group routing while adding the secondary record-key ordering, so exercising multiple buckets would protect that behavior for both RDD and Row parameters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5405131386
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d17fb84130673182bdcd180ba679eb7010fbddf0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3852123402
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/commit/DatasetBucketRescaleCommitActionExecutor.java:
##########
@@ -57,7 +57,9 @@ public DatasetBucketRescaleCommitActionExecutor(HoodieWriteConfig config,
*/
@Override
protected BulkInsertPartitioner<Dataset<Row>> getPartitioner(boolean populateMetaFields, boolean isTablePartitioned) {
- return new BucketIndexBulkInsertPartitionerWithRows(writeClient.getConfig(), expression, rule, bucketNumber);
+ return new BucketIndexBulkInsertPartitionerWithRows(
Review Comment:
[P2] Validate custom sort columns on the bucket-rescale path
This override bypasses `BaseDatasetBulkInsertCommitActionExecutor#getPartitioner`, which is the only Dataset Row call site for `validateCustomSortColumns`. As a result, an LSM `BUCKET_RESCALE` write with `hoodie.bulkinsert.user.defined.partitioner.sort.columns` set is accepted even though the normal simple/consistent bucket paths reject the same configuration. The call order makes this more consequential: `execute()` invokes this class’s `preExecute()` first, and that method persists the new `PartitionBucketIndexHashingConfig`, before `getPartitioner()` runs. Could this path perform the same validation before saving the hashing config, with a rescale regression test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #19727:
URL: https://github.com/apache/hudi/pull/19727#discussion_r3853514284
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/BucketSortBulkInsertPartitioner.java:
##########
@@ -32,13 +34,26 @@ public abstract class BucketSortBulkInsertPartitioner<T> implements BulkInsertPa
public BucketSortBulkInsertPartitioner(HoodieTable table, String sortString) {
this.table = table;
+ validateCustomSortColumns(table, sortString);
if (!StringUtils.isNullOrEmpty(sortString)) {
this.sortColumnNames = sortString.split(",");
} else {
this.sortColumnNames = null;
}
}
+ public static void validateCustomSortColumns(HoodieTable table, String sortString) {
Review Comment:
Fixed in 44cf6615b831. Good point — this config is ignored by bucket-index partitioners, as it was before LSM support, so validating it was redundant. The validation now only checks the actual .
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/RDDBucketIndexPartitioner.java:
##########
@@ -121,7 +122,10 @@ private JavaRDD<HoodieRecord<T>> doPartitionAndSortByRecordKey(JavaRDD<HoodieRec
LOG.warn("Bucket index does not support global sort mode, the sort will only be done within each data partition");
}
- Comparator<HoodieKey> comparator = (Comparator<HoodieKey> & Serializable) (t1, t2) -> t1.getRecordKey().compareTo(t2.getRecordKey());
+ boolean lsmTable = table.getMetaClient().getTableConfig().isLSMTreeStorageLayout();
+ Comparator<HoodieKey> comparator = (Comparator<HoodieKey> & Serializable) (t1, t2) -> lsmTable
Review Comment:
Fixed in 44cf6615b831. The comparator is now selected once based on the storage layout before sorting.
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/commit/DatasetBucketRescaleCommitActionExecutor.java:
##########
@@ -57,7 +57,9 @@ public DatasetBucketRescaleCommitActionExecutor(HoodieWriteConfig config,
*/
@Override
protected BulkInsertPartitioner<Dataset<Row>> getPartitioner(boolean populateMetaFields, boolean isTablePartitioned) {
- return new BucketIndexBulkInsertPartitionerWithRows(writeClient.getConfig(), expression, rule, bucketNumber);
+ return new BucketIndexBulkInsertPartitionerWithRows(
Review Comment:
Addressed in 44cf6615b831. The config validation was removed because is ignored by bucket-index partitioners. Therefore the rescale path no longer bypasses a relevant validation or risks failing on this config after persisting the hashing config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(spark): support bucket index for LSM tables [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19727:
URL: https://github.com/apache/hudi/pull/19727#issuecomment-5438922133
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16517",
"triggerID" : "d17fb84130673182bdcd180ba679eb7010fbddf0",
"triggerType" : "PUSH"
}, {
"hash" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16520",
"triggerID" : "bd62bcb525f367c35ebf37bb6035196c3c4b96e8",
"triggerType" : "PUSH"
}, {
"hash" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16529",
"triggerID" : "44cf6615b83109445008bb2beffcfd93f9ee1176",
"triggerType" : "PUSH"
}, {
"hash" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16605",
"triggerID" : "365ab3c277f0dbde59402b19642a998bce81fda3",
"triggerType" : "PUSH"
}, {
"hash" : "43c8b4619d3dab413d8ca27a4932590d955b07a9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "43c8b4619d3dab413d8ca27a4932590d955b07a9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 365ab3c277f0dbde59402b19642a998bce81fda3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16605)
* 43c8b4619d3dab413d8ca27a4932590d955b07a9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org