[GitHub] [hudi] nsivabalan commented on a diff in pull request #7411: [HUDI-5351] Handle populateMetaFields when repartitioning in sort partitioner

[GitBox <gi...@apache.org>]

nsivabalan commented on code in PR #7411:
URL: https://github.com/apache/hudi/pull/7411#discussion_r1044860847


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/GlobalSortPartitionerWithRows.java:
##########
@@ -31,11 +31,14 @@
 public class GlobalSortPartitionerWithRows implements BulkInsertPartitioner<Dataset<Row>> {
 
   @Override
-  public Dataset<Row> repartitionRecords(Dataset<Row> rows, int outputSparkPartitions) {
+  public Dataset<Row> repartitionRecords(Dataset<Row> rows, int outputSparkPartitions, boolean populateMetaFields) {
     // Now, sort the records and line them up nicely for loading.
     // Let's use "partitionPath + key" as the sort key.
-    return rows.sort(functions.col(HoodieRecord.PARTITION_PATH_METADATA_FIELD), functions.col(HoodieRecord.RECORD_KEY_METADATA_FIELD))
-        .coalesce(outputSparkPartitions);
+    if (populateMetaFields) {
+      return rows.sort(functions.col(HoodieRecord.PARTITION_PATH_METADATA_FIELD), functions.col(HoodieRecord.RECORD_KEY_METADATA_FIELD))
+          .coalesce(outputSparkPartitions);
+    }
+    return rows.coalesce(outputSparkPartitions);

Review Comment:
   virtual keys are supported only for 2 key gens. Simple and NonPartitioned Key gen. 
   So, we should be able to support sorting based on original columns instead of meta fields even when meta fields are disabled. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org