Re: [PR] fix(table-services): When single clustering group config is disabled, clustering should not create clustering groups with same number of input/output files [hudi]

["kbuci (via GitHub)" <gi...@apache.org>]

kbuci commented on code in PR #18172:
URL: https://github.com/apache/hudi/pull/18172#discussion_r2875123361


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/strategy/PartitionAwareClusteringPlanStrategy.java:
##########
@@ -102,15 +102,23 @@ protected Pair<Stream<HoodieClusteringGroup>, Boolean> buildClusteringGroupsForP
     }
 
     if (!currentGroup.isEmpty()) {
-      if (currentGroup.size() > 1 || writeConfig.shouldClusteringSingleGroup()) {
-        int numOutputGroups = getNumberOfOutputFileGroups(totalSizeSoFar, writeConfig.getClusteringTargetFileMaxBytes());
-        log.info("Adding final clustering group " + totalSizeSoFar + " max bytes: "
-            + writeConfig.getClusteringMaxBytesInGroup() + " num input slices: " + currentGroup.size() + " output groups: " + numOutputGroups);
-        fileSliceGroups.add(Pair.of(currentGroup, numOutputGroups));
-      }
+      int numOutputGroups = getNumberOfOutputFileGroups(totalSizeSoFar, writeConfig.getClusteringTargetFileMaxBytes());
+      log.info("Adding final clustering group " + totalSizeSoFar + " max bytes: "
+          + writeConfig.getClusteringMaxBytesInGroup() + " num input slices: " + currentGroup.size() + " output groups: " + numOutputGroups);
+      fileSliceGroups.add(Pair.of(currentGroup, numOutputGroups));
     }
 
-    return Pair.of(fileSliceGroups.stream().map(fileSliceGroup ->
+    return Pair.of(fileSliceGroups.stream().filter(fileSliceGroup -> {
+      if (fileSliceGroup.getLeft().size() == 1 && fileSliceGroup.getRight() == 1 && !writeConfig.shouldClusteringSingleGroup()) {
+        FileSlice targetedFileSlice = fileSliceGroup.getLeft().get(0);
+        long size = targetedFileSlice.getBaseFile().isPresent() ? targetedFileSlice.getBaseFile().get().getFileSize() : writeConfig.getParquetMaxFileSize();
+        log.info(String.format("Removing clustering group due to input and output slices both being 1 and single group clustering is disabled."
+                + " Group stats: currentGroupSize= %s, maxBytesPerClusteringGroup= %s",
+            size, writeConfig.getClusteringMaxBytesInGroup()));

Review Comment:
   If there are sort columns then we shouldn't enter this if-statement , since we make sure that shouldClusteringSingleGroup is false. And that would return turn if there were sort columns
   
   
   ```
   
     public boolean shouldClusteringSingleGroup() {
       return isClusteringSortEnabled() || isSingleGroupClusteringEnabled();
     }
   
     public String getClusteringSortColumns() {
       return getString(HoodieClusteringConfig.PLAN_STRATEGY_SORT_COLUMNS);
     }
   
     public boolean isClusteringSortEnabled() {
       return !StringUtils.isNullOrEmpty(getString(HoodieClusteringConfig.PLAN_STRATEGY_SORT_COLUMNS));
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org