You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/12/08 20:28:29 UTC
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7411: [HUDI-5351] Handle populateMetaFields when repartitioning in sort partitioner
alexeykudinkin commented on code in PR #7411:
URL: https://github.com/apache/hudi/pull/7411#discussion_r1043794079
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/BulkInsertPartitioner.java:
##########
@@ -38,7 +38,7 @@
* @param outputPartitions Expected number of output partitions
* @return
*/
- I repartitionRecords(I records, int outputPartitions);
+ I repartitionRecords(I records, int outputPartitions, boolean populateMetaFields);
Review Comment:
@codope we should pass this in ctor (we actually should be passing the HoodieWriteConfig to access whole config), rather than t/h the API
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/GlobalSortPartitionerWithRows.java:
##########
@@ -31,11 +31,14 @@
public class GlobalSortPartitionerWithRows implements BulkInsertPartitioner<Dataset<Row>> {
@Override
- public Dataset<Row> repartitionRecords(Dataset<Row> rows, int outputSparkPartitions) {
+ public Dataset<Row> repartitionRecords(Dataset<Row> rows, int outputSparkPartitions, boolean populateMetaFields) {
// Now, sort the records and line them up nicely for loading.
// Let's use "partitionPath + key" as the sort key.
- return rows.sort(functions.col(HoodieRecord.PARTITION_PATH_METADATA_FIELD), functions.col(HoodieRecord.RECORD_KEY_METADATA_FIELD))
- .coalesce(outputSparkPartitions);
+ if (populateMetaFields) {
Review Comment:
A few notes:
- We should actually properly support virtual-keys in here by instantiating the key-gen and applying it here
- Current approach is actually obscuring the failure currently -- instead of failing, we'll let it go t/h but we'll breach the contract of the Partitioner by not sorting the rows actually. Instead, we should just fail outright in case meta-fields are disabled (as a stop-gap solution until we fully support virtual-keys).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org