You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 05:52:05 UTC
[I] Add an option leverage RDDCustomColumnsSortPartitioner in bulk-insert [hudi]
hudi-bot opened a new issue, #15437:
URL: https://github.com/apache/hudi/issues/15437
Currently, to sort incoming batch using DeltaStreamer you'll have to resort to using SqlQueryBasedTransformer.
This however should not be necessary provided that RDDCustomColumnsSortPartitioner does already exist and could be leveraged exactly for that.
This come up during discussions w/ [~abmodi]
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4862
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add an option leverage RDDCustomColumnsSortPartitioner in bulk-insert [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15437:
URL: https://github.com/apache/hudi/issues/15437#issuecomment-4795424480
This already works in current master without needing `SqlQueryBasedTransformer`. DeltaStreamer/HoodieStreamer wires through to `RDDCustomColumnsSortPartitioner` via the existing configs:
```properties
hoodie.bulkinsert.user.defined.partitioner.class=org.apache.hudi.execution.bulkinsert.RDDCustomColumnsSortPartitioner
hoodie.bulkinsert.user.defined.partitioner.sort.columns=<col1>,<col2>
```
See `HoodieWriteConfig.BULKINSERT_USER_DEFINED_PARTITIONER_CLASS_NAME` / `..._SORT_COLUMNS` (the doc explicitly mentions `RDDCustomColumnsSortPartitioner` as the built-in option), and `TestHoodieDeltaStreamer` (e.g. `testBulkInsertSkewedSortColumns`) which exercises this exact flow end-to-end. HUDI-7617 (25da2b0b1b08) fixed the StreamSync-side wiring for user-defined bulk-insert partitioners. Closing as already addressed — please reopen with a repro if you're hitting a specific gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add an option leverage RDDCustomColumnsSortPartitioner in bulk-insert [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15437: Add an option leverage RDDCustomColumnsSortPartitioner in bulk-insert
URL: https://github.com/apache/hudi/issues/15437
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org