You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/10/23 22:28:09 UTC

[GitHub] [hudi] satishkotha opened a new pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

satishkotha opened a new pull request #2202:
URL: https://github.com/apache/hudi/pull/2202


   ## What is the purpose of the pull request
   
   Add FileSystemView APIs to query pending clustering ops
   
   ## Brief change log
   
   * Clustering is tracked using 'REPLACE' action
   * Introduce metadata for replace requested file. OperationType is set to cluster to differentiate between Clustering and other use cases
   * Changes to support IncrementalTimeline pending. HUDI-1353
   * Changes to block updates on files in pending clustering is pending HUDI-1354
   * How clustering interacts with other files such as bootstrapped (non-hudi) files is pending.
   
   ## Verify this pull request
   
   This change added tests and can be verified as follows:
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r518201140



##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieSliceInfo",
+   "fields":[
+      {
+        "name":"dataFilePath",
+        "type":["null","string"],
+        "default": null
+      },
+      {
+        "name":"deltaFilePaths",
+        "type":["null", {
+           "type":"array",
+           "items":"string"
+        }],
+        "default": null
+      },
+      {
+        "name":"fileId",
+        "type":["null","string"]
+      },
+      {
+        "name":"partitionPath",
+        "type":["null","string"],
+        "default": null
+      },
+      {
+        "name":"bootstrapFilePath",

Review comment:
       its okay. I recall some discussion with @bvaradar on why we did that. but blanking out now. let's just go with this as-is




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513680058



##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",

Review comment:
       I see value in keeping these in separate files. It was suggested to me in earlier reviews as well. Please tell me if there is any value in combining these. We can perhaps add this to coding guidelines.Without that, different reviewers are giving conflicting suggestions.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513513829



##########
File path: hudi-common/src/main/avro/HoodieClusteringGroup.avsc
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieClusteringGroup",
+   "type":"record",
+   "fields":[
+      {
+         /* Group of files that needs to merged. All the slices in a group will belong to same partition initially.

Review comment:
       @satishkotha Also, can we not create multiple avsc files for each "Pojo" that you need, you can just define them all under the HoodieClusteringPlan,  take a look at this for example -> https://github.com/apache/hudi/blob/master/hudi-common/src/main/avro/HoodieCleanerPlan.avsc#L78. Once the code-gen creates the POJO from the AVSC, you will have access to them.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513680279



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/ClusteringOpDTO.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline.dto;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.collection.Pair;
+
+/**
+ * The data transfer object of compaction.

Review comment:
       Will fix it. thanks




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r518200329



##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieSliceInfo",
+   "fields":[
+      {
+        "name":"dataFilePath",
+        "type":["null","string"],
+        "default": null
+      },
+      {
+        "name":"deltaFilePaths",
+        "type":["null", {
+           "type":"array",
+           "items":"string"
+        }],
+        "default": null
+      },
+      {
+        "name":"fileId",
+        "type":["null","string"]
+      },
+      {
+        "name":"partitionPath",
+        "type":["null","string"],
+        "default": null
+      },
+      {
+        "name":"bootstrapFilePath",
+        "type":["null", "string"],
+        "default": null
+      },
+      {
+        "name":"version",

Review comment:
       Realize we can achieve this by passing writer/reader schema and also there are places where this is not being done uniformly. So treat this more as a nit




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513682149



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
##########
@@ -710,6 +721,40 @@ private String formatPartitionKey(String partitionStr) {
    */
   abstract void removePendingCompactionOperations(Stream<Pair<String, CompactionOperation>> operations);
 
+  /**
+   * Check if there is an outstanding clustering operation (requested/inflight) scheduled for this file.
+   *
+   * @param fgId File-Group Id
+   * @return true if there is a pending clustering, false otherwise
+   */
+  protected abstract boolean isPendingClusteringScheduledForFileId(HoodieFileGroupId fgId);
+
+  /**
+   *  Get pending clustering instant time for specified file group. Return None if file group is not in pending
+   *  clustering operation.
+   */
+  protected abstract Option<HoodieInstant> getPendingClusteringInstant(final HoodieFileGroupId fileGroupId);
+
+  /**
+   * Fetch all file groups in pending clustering.
+   */
+  protected abstract Stream<Pair<HoodieFileGroupId, HoodieInstant>> fetchFileGroupsInPendingClustering();
+
+  /**
+   * resets the pending clustering operation and overwrite with the new list.
+   */
+  abstract void resetFileGroupsInPendingClustering(Map<HoodieFileGroupId, HoodieInstant> fgIdToInstantMap);
+
+  /**
+   * Add metadata for pending clustering operation to store for given instant.

Review comment:
       I will expand it to explain better




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r517780294



##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieSliceInfo",
+   "fields":[
+      {
+        "name":"dataFilePath",
+        "type":["null","string"],
+        "default": null
+      },
+      {
+        "name":"deltaFilePaths",
+        "type":["null", {
+           "type":"array",
+           "items":"string"
+        }],
+        "default": null
+      },
+      {
+        "name":"fileId",
+        "type":["null","string"]
+      },
+      {
+        "name":"partitionPath",
+        "type":["null","string"],
+        "default": null
+      },
+      {
+        "name":"bootstrapFilePath",

Review comment:
        I tried to keep  sliceInfo similar to compaction. But I can remove it, if you think we can change the approach to lookup from index.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513528066



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.avro.model.HoodieClusteringGroup;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieClusteringStrategy;
+import org.apache.hudi.avro.model.HoodieRequestedReplaceMetadata;
+import org.apache.hudi.avro.model.HoodieSliceInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Helper class to generate clustering plan from metadata.
+ */
+public class ClusteringUtils {
+
+  private static final Logger LOG = LogManager.getLogger(ClusteringUtils.class);
+
+  public static final String TOTAL_IO_READ_MB = "TOTAL_IO_READ_MB";
+  public static final String TOTAL_LOG_FILE_SIZE = "TOTAL_LOG_FILES_SIZE";
+  public static final String TOTAL_LOG_FILES = "TOTAL_LOG_FILES";
+
+  /**
+   * Get all pending clustering plans along with their instants.
+   */
+  public static Stream<Pair<HoodieInstant, HoodieClusteringPlan>> getAllPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    List<HoodieInstant> pendingReplaceInstants =
+        metaClient.getActiveTimeline().filterPendingReplaceTimeline().getInstants().collect(Collectors.toList());
+    return pendingReplaceInstants.stream().map(instant -> getClusteringPlan(metaClient, instant))
+        .filter(Option::isPresent).map(Option::get);
+  }
+
+  public static Option<Pair<HoodieInstant, HoodieClusteringPlan>> getClusteringPlan(HoodieTableMetaClient metaClient, HoodieInstant requestedReplaceInstant) {
+    try {
+      Option<byte[]> content = metaClient.getActiveTimeline().getInstantDetails(requestedReplaceInstant);
+      if (!content.isPresent() || content.get().length == 0) {
+        // few operations create requested file without any content. Assume these are not clustering
+        LOG.warn("No content found in requested file for instant " + requestedReplaceInstant);
+        return Option.empty();
+      }
+      HoodieRequestedReplaceMetadata requestedReplaceMetadata = TimelineMetadataUtils.deserializeRequestedReplaceMetadta(content.get());
+      if (WriteOperationType.CLUSTER.name().equals(requestedReplaceMetadata.getOperationType())) {
+        return Option.of(Pair.of(requestedReplaceInstant, requestedReplaceMetadata.getClusteringPlan()));
+      }
+      return Option.empty();
+    } catch (IOException e) {
+      throw new HoodieIOException("Error reading clustering plan " + requestedReplaceInstant.getTimestamp(), e);
+    }
+  }
+
+  /**
+   * Get filegroups to pending clustering instant mapping for all pending clustering plans.
+   * This includes all clustering operattions in 'requested' and 'inflight' states.
+   */
+  public static Map<HoodieFileGroupId, HoodieInstant> getAllFileGroupsInPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    Stream<Pair<HoodieInstant, HoodieClusteringPlan>> pendingClusteringPlans = getAllPendingClusteringPlans(metaClient);
+    Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> resultStream = pendingClusteringPlans.flatMap(clusteringPlan ->
+        // get all filegroups in the plan
+        getFileGroupEntriesInClusteringPlan(clusteringPlan.getLeft(), clusteringPlan.getRight()));
+
+    Map<HoodieFileGroupId, HoodieInstant> resultMap = resultStream.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+    LOG.info("Found " + resultMap.size() + " files in pending clustering operations");
+    return resultMap;
+  }
+
+  public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupsInPendingClusteringInstant(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    Stream<HoodieFileGroupId> partitionToFileIdLists = clusteringPlan.getInputGroups().stream().flatMap(ClusteringUtils::getFileGroupsFromClusteringGroup);
+    return partitionToFileIdLists.map(e -> Pair.of(e, instant));
+  }
+
+  private static Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesInClusteringPlan(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    return getFileGroupsInPendingClusteringInstant(instant, clusteringPlan).map(entry ->
+        new AbstractMap.SimpleEntry<>(entry.getLeft(), entry.getRight()));
+  }
+
+  private static Stream<HoodieFileGroupId> getFileGroupsFromClusteringGroup(HoodieClusteringGroup group) {
+    return group.getSlices().stream().map(slice -> new HoodieFileGroupId(slice.getPartitionPath(), slice.getFileId()));
+  }
+
+  /**
+   * Create clustering plan from input fileSliceGroups.
+   */
+  public static HoodieClusteringPlan createClusteringPlan(String strategyClassName,
+                                                          Map<String, String> strategyParams,
+                                                          List<FileSlice>[] fileSliceGroups) {
+    List<HoodieClusteringGroup> clusteringGroups = new ArrayList<>();
+    for (int i = 0; i < fileSliceGroups.length; i++) {

Review comment:
       Can we change this loop to java 8 stream style, fileSliceGroups.stream().foreach()..




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r514294873



##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",

Review comment:
       @satishkotha For code reviews, there may be different opinions from different reviewers. We should just treat that as opportunities to weed out finer details and  the idea should be drive consensus on a single solution. We need to take most of these case by case, I think this particular case you want to reuse the building blocks of the different avsc in the future for other plans etc, that makes sense to me.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513520923



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/ClusteringOpDTO.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline.dto;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.collection.Pair;
+
+/**
+ * The data transfer object of compaction.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class ClusteringOpDTO {
+
+  @JsonProperty("id")
+  private String fileId;

Review comment:
       What does a fileID mean here ? Is this the lowest representation of the clustering plan, will the clustering plan contain a bunch of ClusteringOpDTO's ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513515321



##########
File path: hudi-common/src/main/avro/HoodieRequestedReplaceMetadata.avsc
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",

Review comment:
       Same comment, just roll up all the datastructures that you need into the HoodieClusteringPlan, is it possible to merge the clustering plan with the insert-overwrite plan ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513681802



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/ClusteringOpDTO.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline.dto;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.collection.Pair;
+
+/**
+ * The data transfer object of compaction.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class ClusteringOpDTO {
+
+  @JsonProperty("id")
+  private String fileId;

Review comment:
       This is only used for RemoteFileSystemView implementation. RemoteFSV uses DTOs to serialize and deserialize the data over wire. One of the APIs filesystem view provides is to list all fileIds that have pending clustering scheduled. So this file contains all the information we need to send over network. It has nothing to do with clustering plan stored in avro format. Its just there to support getFilesInPendingClusteringOperation API




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r514287031



##########
File path: hudi-common/src/main/avro/HoodieClusteringGroup.avsc
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieClusteringGroup",
+   "type":"record",
+   "fields":[
+      {
+         /* Group of files that needs to merged. All the slices in a group will belong to same partition initially.

Review comment:
       Hmm, I see. If there are use-cases where we plan to reuse them to create different POJO's, I'm fine with this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513529595



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.avro.model.HoodieClusteringGroup;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieClusteringStrategy;
+import org.apache.hudi.avro.model.HoodieRequestedReplaceMetadata;
+import org.apache.hudi.avro.model.HoodieSliceInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Helper class to generate clustering plan from metadata.
+ */
+public class ClusteringUtils {
+
+  private static final Logger LOG = LogManager.getLogger(ClusteringUtils.class);
+
+  public static final String TOTAL_IO_READ_MB = "TOTAL_IO_READ_MB";
+  public static final String TOTAL_LOG_FILE_SIZE = "TOTAL_LOG_FILES_SIZE";
+  public static final String TOTAL_LOG_FILES = "TOTAL_LOG_FILES";
+
+  /**
+   * Get all pending clustering plans along with their instants.
+   */
+  public static Stream<Pair<HoodieInstant, HoodieClusteringPlan>> getAllPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    List<HoodieInstant> pendingReplaceInstants =
+        metaClient.getActiveTimeline().filterPendingReplaceTimeline().getInstants().collect(Collectors.toList());
+    return pendingReplaceInstants.stream().map(instant -> getClusteringPlan(metaClient, instant))
+        .filter(Option::isPresent).map(Option::get);
+  }
+
+  public static Option<Pair<HoodieInstant, HoodieClusteringPlan>> getClusteringPlan(HoodieTableMetaClient metaClient, HoodieInstant requestedReplaceInstant) {
+    try {
+      Option<byte[]> content = metaClient.getActiveTimeline().getInstantDetails(requestedReplaceInstant);
+      if (!content.isPresent() || content.get().length == 0) {
+        // few operations create requested file without any content. Assume these are not clustering
+        LOG.warn("No content found in requested file for instant " + requestedReplaceInstant);
+        return Option.empty();
+      }
+      HoodieRequestedReplaceMetadata requestedReplaceMetadata = TimelineMetadataUtils.deserializeRequestedReplaceMetadta(content.get());
+      if (WriteOperationType.CLUSTER.name().equals(requestedReplaceMetadata.getOperationType())) {
+        return Option.of(Pair.of(requestedReplaceInstant, requestedReplaceMetadata.getClusteringPlan()));
+      }
+      return Option.empty();
+    } catch (IOException e) {
+      throw new HoodieIOException("Error reading clustering plan " + requestedReplaceInstant.getTimestamp(), e);
+    }
+  }
+
+  /**
+   * Get filegroups to pending clustering instant mapping for all pending clustering plans.
+   * This includes all clustering operattions in 'requested' and 'inflight' states.
+   */
+  public static Map<HoodieFileGroupId, HoodieInstant> getAllFileGroupsInPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    Stream<Pair<HoodieInstant, HoodieClusteringPlan>> pendingClusteringPlans = getAllPendingClusteringPlans(metaClient);
+    Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> resultStream = pendingClusteringPlans.flatMap(clusteringPlan ->
+        // get all filegroups in the plan
+        getFileGroupEntriesInClusteringPlan(clusteringPlan.getLeft(), clusteringPlan.getRight()));
+
+    Map<HoodieFileGroupId, HoodieInstant> resultMap = resultStream.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+    LOG.info("Found " + resultMap.size() + " files in pending clustering operations");
+    return resultMap;
+  }
+
+  public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupsInPendingClusteringInstant(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    Stream<HoodieFileGroupId> partitionToFileIdLists = clusteringPlan.getInputGroups().stream().flatMap(ClusteringUtils::getFileGroupsFromClusteringGroup);
+    return partitionToFileIdLists.map(e -> Pair.of(e, instant));
+  }
+
+  private static Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesInClusteringPlan(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    return getFileGroupsInPendingClusteringInstant(instant, clusteringPlan).map(entry ->
+        new AbstractMap.SimpleEntry<>(entry.getLeft(), entry.getRight()));
+  }
+
+  private static Stream<HoodieFileGroupId> getFileGroupsFromClusteringGroup(HoodieClusteringGroup group) {
+    return group.getSlices().stream().map(slice -> new HoodieFileGroupId(slice.getPartitionPath(), slice.getFileId()));
+  }
+
+  /**
+   * Create clustering plan from input fileSliceGroups.
+   */
+  public static HoodieClusteringPlan createClusteringPlan(String strategyClassName,
+                                                          Map<String, String> strategyParams,
+                                                          List<FileSlice>[] fileSliceGroups) {
+    List<HoodieClusteringGroup> clusteringGroups = new ArrayList<>();
+    for (int i = 0; i < fileSliceGroups.length; i++) {
+      List<FileSlice> fileSliceGroup = fileSliceGroups[i];
+      Map<String, Double> groupMetrics = buildMetrics(fileSliceGroup);
+      List<HoodieSliceInfo> sliceInfos = getFileSliceInfo(fileSliceGroup);
+      clusteringGroups.add(HoodieClusteringGroup.newBuilder().setSlices(sliceInfos).setMetrics(groupMetrics).build());
+    }
+
+    HoodieClusteringStrategy strategy = HoodieClusteringStrategy.newBuilder()
+        .setStrategyClassName(strategyClassName).setStrategyParams(strategyParams)
+        .build();
+
+    HoodieClusteringPlan plan = HoodieClusteringPlan.newBuilder()
+        .setInputGroups(clusteringGroups)
+        .setExtraMetadata(new HashMap<>())
+        .setStrategy(strategy)
+        .build();
+
+    return plan;
+  }
+
+  private static List<HoodieSliceInfo> getFileSliceInfo(List<FileSlice> slices) {
+    FileSlice slice1 = slices.get(0);
+    System.out.println(slice1.getLogFiles().count());
+    System.out.println(slice1.getBaseFile().get());
+    System.out.println(slice1.getPartitionPath());
+    return slices.stream().map(slice -> new HoodieSliceInfo().newBuilder()
+        .setPartitionPath(slice.getPartitionPath())
+        .setFileId(slice.getFileId())
+        .setDataFilePath(slice.getBaseFile().map(BaseFile::getPath).orElse(""))

Review comment:
       Should the dataFilePath be empty string or empty Option ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513675911



##########
File path: hudi-common/src/main/avro/HoodieClusteringGroup.avsc
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieClusteringGroup",
+   "type":"record",
+   "fields":[
+      {
+         /* Group of files that needs to merged. All the slices in a group will belong to same partition initially.

Review comment:
       From earlier reviews, creating multiple avsc was preferred because we can reuse pojos for other operations. Please see discussion in #2048 . Let me know if there are any advantages of keeping them together in one file




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513530025



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.avro.model.HoodieClusteringGroup;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieClusteringStrategy;
+import org.apache.hudi.avro.model.HoodieRequestedReplaceMetadata;
+import org.apache.hudi.avro.model.HoodieSliceInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Helper class to generate clustering plan from metadata.
+ */
+public class ClusteringUtils {
+
+  private static final Logger LOG = LogManager.getLogger(ClusteringUtils.class);
+
+  public static final String TOTAL_IO_READ_MB = "TOTAL_IO_READ_MB";
+  public static final String TOTAL_LOG_FILE_SIZE = "TOTAL_LOG_FILES_SIZE";
+  public static final String TOTAL_LOG_FILES = "TOTAL_LOG_FILES";
+
+  /**
+   * Get all pending clustering plans along with their instants.
+   */
+  public static Stream<Pair<HoodieInstant, HoodieClusteringPlan>> getAllPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    List<HoodieInstant> pendingReplaceInstants =
+        metaClient.getActiveTimeline().filterPendingReplaceTimeline().getInstants().collect(Collectors.toList());
+    return pendingReplaceInstants.stream().map(instant -> getClusteringPlan(metaClient, instant))
+        .filter(Option::isPresent).map(Option::get);
+  }
+
+  public static Option<Pair<HoodieInstant, HoodieClusteringPlan>> getClusteringPlan(HoodieTableMetaClient metaClient, HoodieInstant requestedReplaceInstant) {
+    try {
+      Option<byte[]> content = metaClient.getActiveTimeline().getInstantDetails(requestedReplaceInstant);
+      if (!content.isPresent() || content.get().length == 0) {
+        // few operations create requested file without any content. Assume these are not clustering
+        LOG.warn("No content found in requested file for instant " + requestedReplaceInstant);
+        return Option.empty();
+      }
+      HoodieRequestedReplaceMetadata requestedReplaceMetadata = TimelineMetadataUtils.deserializeRequestedReplaceMetadta(content.get());
+      if (WriteOperationType.CLUSTER.name().equals(requestedReplaceMetadata.getOperationType())) {
+        return Option.of(Pair.of(requestedReplaceInstant, requestedReplaceMetadata.getClusteringPlan()));
+      }
+      return Option.empty();
+    } catch (IOException e) {
+      throw new HoodieIOException("Error reading clustering plan " + requestedReplaceInstant.getTimestamp(), e);
+    }
+  }
+
+  /**
+   * Get filegroups to pending clustering instant mapping for all pending clustering plans.
+   * This includes all clustering operattions in 'requested' and 'inflight' states.
+   */
+  public static Map<HoodieFileGroupId, HoodieInstant> getAllFileGroupsInPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    Stream<Pair<HoodieInstant, HoodieClusteringPlan>> pendingClusteringPlans = getAllPendingClusteringPlans(metaClient);
+    Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> resultStream = pendingClusteringPlans.flatMap(clusteringPlan ->
+        // get all filegroups in the plan
+        getFileGroupEntriesInClusteringPlan(clusteringPlan.getLeft(), clusteringPlan.getRight()));
+
+    Map<HoodieFileGroupId, HoodieInstant> resultMap = resultStream.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+    LOG.info("Found " + resultMap.size() + " files in pending clustering operations");
+    return resultMap;
+  }
+
+  public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupsInPendingClusteringInstant(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    Stream<HoodieFileGroupId> partitionToFileIdLists = clusteringPlan.getInputGroups().stream().flatMap(ClusteringUtils::getFileGroupsFromClusteringGroup);
+    return partitionToFileIdLists.map(e -> Pair.of(e, instant));
+  }
+
+  private static Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesInClusteringPlan(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    return getFileGroupsInPendingClusteringInstant(instant, clusteringPlan).map(entry ->
+        new AbstractMap.SimpleEntry<>(entry.getLeft(), entry.getRight()));
+  }
+
+  private static Stream<HoodieFileGroupId> getFileGroupsFromClusteringGroup(HoodieClusteringGroup group) {
+    return group.getSlices().stream().map(slice -> new HoodieFileGroupId(slice.getPartitionPath(), slice.getFileId()));
+  }
+
+  /**
+   * Create clustering plan from input fileSliceGroups.
+   */
+  public static HoodieClusteringPlan createClusteringPlan(String strategyClassName,
+                                                          Map<String, String> strategyParams,
+                                                          List<FileSlice>[] fileSliceGroups) {
+    List<HoodieClusteringGroup> clusteringGroups = new ArrayList<>();
+    for (int i = 0; i < fileSliceGroups.length; i++) {
+      List<FileSlice> fileSliceGroup = fileSliceGroups[i];
+      Map<String, Double> groupMetrics = buildMetrics(fileSliceGroup);
+      List<HoodieSliceInfo> sliceInfos = getFileSliceInfo(fileSliceGroup);
+      clusteringGroups.add(HoodieClusteringGroup.newBuilder().setSlices(sliceInfos).setMetrics(groupMetrics).build());
+    }
+
+    HoodieClusteringStrategy strategy = HoodieClusteringStrategy.newBuilder()
+        .setStrategyClassName(strategyClassName).setStrategyParams(strategyParams)
+        .build();
+
+    HoodieClusteringPlan plan = HoodieClusteringPlan.newBuilder()
+        .setInputGroups(clusteringGroups)
+        .setExtraMetadata(new HashMap<>())
+        .setStrategy(strategy)
+        .build();
+
+    return plan;
+  }
+
+  private static List<HoodieSliceInfo> getFileSliceInfo(List<FileSlice> slices) {
+    FileSlice slice1 = slices.get(0);
+    System.out.println(slice1.getLogFiles().count());
+    System.out.println(slice1.getBaseFile().get());
+    System.out.println(slice1.getPartitionPath());
+    return slices.stream().map(slice -> new HoodieSliceInfo().newBuilder()
+        .setPartitionPath(slice.getPartitionPath())
+        .setFileId(slice.getFileId())
+        .setDataFilePath(slice.getBaseFile().map(BaseFile::getPath).orElse(""))
+        .setDeltaFilePaths(slice.getLogFiles().map(f -> f.getPath().getName()).collect(Collectors.toList()))
+        .setBootstrapFilePath(slice.getBaseFile().map(bf -> bf.getBootstrapBaseFile().map(bbf -> bbf.getPath()).orElse("")).orElse(""))

Review comment:
       Same question here for data file path empty




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#issuecomment-722142071


   > @satishkotha this pull request is so good. Base on this pull request. Can i begin to work on implement Simple ClusteringStrategy and ClusteringActionExecutor? or have other task i can participate about clustering now. cc @n3nash @vinothchandar @leesf
   
   @lw309637554 we can use subtasks here to distribute https://issues.apache.org/jira/browse/HUDI-1042. I have already started ClusteringStrategy and executor based on leesf's PR and description in RFC-19.  You can pick up other subtasks in HUDI-1042? You could also add another strategy after I send PR for first strategy. The strategy I'm adding is very simple and may not be sufficient for all usecases. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513729072



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.avro.model.HoodieClusteringGroup;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieClusteringStrategy;
+import org.apache.hudi.avro.model.HoodieRequestedReplaceMetadata;
+import org.apache.hudi.avro.model.HoodieSliceInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Helper class to generate clustering plan from metadata.
+ */
+public class ClusteringUtils {
+
+  private static final Logger LOG = LogManager.getLogger(ClusteringUtils.class);
+
+  public static final String TOTAL_IO_READ_MB = "TOTAL_IO_READ_MB";
+  public static final String TOTAL_LOG_FILE_SIZE = "TOTAL_LOG_FILES_SIZE";
+  public static final String TOTAL_LOG_FILES = "TOTAL_LOG_FILES";
+
+  /**
+   * Get all pending clustering plans along with their instants.
+   */
+  public static Stream<Pair<HoodieInstant, HoodieClusteringPlan>> getAllPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    List<HoodieInstant> pendingReplaceInstants =
+        metaClient.getActiveTimeline().filterPendingReplaceTimeline().getInstants().collect(Collectors.toList());
+    return pendingReplaceInstants.stream().map(instant -> getClusteringPlan(metaClient, instant))
+        .filter(Option::isPresent).map(Option::get);
+  }
+
+  public static Option<Pair<HoodieInstant, HoodieClusteringPlan>> getClusteringPlan(HoodieTableMetaClient metaClient, HoodieInstant requestedReplaceInstant) {
+    try {
+      Option<byte[]> content = metaClient.getActiveTimeline().getInstantDetails(requestedReplaceInstant);
+      if (!content.isPresent() || content.get().length == 0) {
+        // few operations create requested file without any content. Assume these are not clustering
+        LOG.warn("No content found in requested file for instant " + requestedReplaceInstant);
+        return Option.empty();
+      }
+      HoodieRequestedReplaceMetadata requestedReplaceMetadata = TimelineMetadataUtils.deserializeRequestedReplaceMetadta(content.get());
+      if (WriteOperationType.CLUSTER.name().equals(requestedReplaceMetadata.getOperationType())) {
+        return Option.of(Pair.of(requestedReplaceInstant, requestedReplaceMetadata.getClusteringPlan()));
+      }
+      return Option.empty();
+    } catch (IOException e) {
+      throw new HoodieIOException("Error reading clustering plan " + requestedReplaceInstant.getTimestamp(), e);
+    }
+  }
+
+  /**
+   * Get filegroups to pending clustering instant mapping for all pending clustering plans.
+   * This includes all clustering operattions in 'requested' and 'inflight' states.
+   */
+  public static Map<HoodieFileGroupId, HoodieInstant> getAllFileGroupsInPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    Stream<Pair<HoodieInstant, HoodieClusteringPlan>> pendingClusteringPlans = getAllPendingClusteringPlans(metaClient);
+    Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> resultStream = pendingClusteringPlans.flatMap(clusteringPlan ->
+        // get all filegroups in the plan
+        getFileGroupEntriesInClusteringPlan(clusteringPlan.getLeft(), clusteringPlan.getRight()));
+
+    Map<HoodieFileGroupId, HoodieInstant> resultMap = resultStream.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+    LOG.info("Found " + resultMap.size() + " files in pending clustering operations");
+    return resultMap;
+  }
+
+  public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupsInPendingClusteringInstant(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    Stream<HoodieFileGroupId> partitionToFileIdLists = clusteringPlan.getInputGroups().stream().flatMap(ClusteringUtils::getFileGroupsFromClusteringGroup);
+    return partitionToFileIdLists.map(e -> Pair.of(e, instant));
+  }
+
+  private static Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesInClusteringPlan(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    return getFileGroupsInPendingClusteringInstant(instant, clusteringPlan).map(entry ->
+        new AbstractMap.SimpleEntry<>(entry.getLeft(), entry.getRight()));
+  }
+
+  private static Stream<HoodieFileGroupId> getFileGroupsFromClusteringGroup(HoodieClusteringGroup group) {
+    return group.getSlices().stream().map(slice -> new HoodieFileGroupId(slice.getPartitionPath(), slice.getFileId()));
+  }
+
+  /**
+   * Create clustering plan from input fileSliceGroups.
+   */
+  public static HoodieClusteringPlan createClusteringPlan(String strategyClassName,
+                                                          Map<String, String> strategyParams,
+                                                          List<FileSlice>[] fileSliceGroups) {
+    List<HoodieClusteringGroup> clusteringGroups = new ArrayList<>();
+    for (int i = 0; i < fileSliceGroups.length; i++) {
+      List<FileSlice> fileSliceGroup = fileSliceGroups[i];
+      Map<String, Double> groupMetrics = buildMetrics(fileSliceGroup);
+      List<HoodieSliceInfo> sliceInfos = getFileSliceInfo(fileSliceGroup);
+      clusteringGroups.add(HoodieClusteringGroup.newBuilder().setSlices(sliceInfos).setMetrics(groupMetrics).build());
+    }
+
+    HoodieClusteringStrategy strategy = HoodieClusteringStrategy.newBuilder()
+        .setStrategyClassName(strategyClassName).setStrategyParams(strategyParams)
+        .build();
+
+    HoodieClusteringPlan plan = HoodieClusteringPlan.newBuilder()
+        .setInputGroups(clusteringGroups)
+        .setExtraMetadata(new HashMap<>())
+        .setStrategy(strategy)
+        .build();
+
+    return plan;
+  }
+
+  private static List<HoodieSliceInfo> getFileSliceInfo(List<FileSlice> slices) {
+    FileSlice slice1 = slices.get(0);
+    System.out.println(slice1.getLogFiles().count());
+    System.out.println(slice1.getBaseFile().get());
+    System.out.println(slice1.getPartitionPath());
+    return slices.stream().map(slice -> new HoodieSliceInfo().newBuilder()
+        .setPartitionPath(slice.getPartitionPath())
+        .setFileId(slice.getFileId())
+        .setDataFilePath(slice.getBaseFile().map(BaseFile::getPath).orElse(""))
+        .setDeltaFilePaths(slice.getLogFiles().map(f -> f.getPath().getName()).collect(Collectors.toList()))
+        .setBootstrapFilePath(slice.getBaseFile().map(bf -> bf.getBootstrapBaseFile().map(bbf -> bbf.getPath()).orElse("")).orElse(""))

Review comment:
       Changed it to null to match with compaction.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar merged pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #2202:
URL: https://github.com/apache/hudi/pull/2202


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] codecov-io commented on pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#issuecomment-718281647


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=h1) Report
   > Merging [#2202](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/c5e10d668f9366f29bdf7721f7efe4140782527b?el=desc) will **increase** coverage by `0.34%`.
   > The diff coverage is `69.59%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/2202/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #2202      +/-   ##
   ============================================
   + Coverage     53.63%   53.98%   +0.34%     
   - Complexity     2848     2904      +56     
   ============================================
     Files           359      361       +2     
     Lines         16545    16735     +190     
     Branches       1780     1788       +8     
   ============================================
   + Hits           8874     9034     +160     
   - Misses         6912     6938      +26     
   - Partials        759      763       +4     
   ```
   
   | Flag | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | #hudicli | `38.37% <ø> (ø)` | `193.00 <ø> (ø)` | |
   | #hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | #hudicommon | `55.12% <68.09%> (+0.36%)` | `1843.00 <41.00> (+48.00)` | |
   | #hudihadoopmr | `33.05% <ø> (ø)` | `181.00 <ø> (ø)` | |
   | #hudispark | `65.95% <ø> (+0.46%)` | `304.00 <ø> (ø)` | |
   | #huditimelineservice | `65.30% <100.00%> (+3.01%)` | `56.00 <3.00> (+6.00)` | |
   | #hudiutilities | `70.09% <ø> (+0.01%)` | `327.00 <ø> (+2.00)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | `93.18% <ø> (ø)` | `43.00 <0.00> (ø)` | |
   | [...udi/common/table/timeline/dto/ClusteringOpDTO.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL2R0by9DbHVzdGVyaW5nT3BEVE8uamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...common/table/view/PriorityBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUHJpb3JpdHlCYXNlZEZpbGVTeXN0ZW1WaWV3LmphdmE=) | `97.10% <0.00%> (-1.43%)` | `33.00 <0.00> (ø)` | |
   | [...on/table/view/RemoteHoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUmVtb3RlSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | [...i/common/table/view/HoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `66.66% <28.57%> (-8.61%)` | `34.00 <3.00> (+3.00)` | :arrow_down: |
   | [.../common/table/view/RocksDbBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUm9ja3NEYkJhc2VkRmlsZVN5c3RlbVZpZXcuamF2YQ==) | `80.40% <66.66%> (-0.06%)` | `72.00 <8.00> (+10.00)` | :arrow_down: |
   | [...on/table/view/SpillableMapBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvU3BpbGxhYmxlTWFwQmFzZWRGaWxlU3lzdGVtVmlldy5qYXZh) | `74.19% <77.77%> (+0.60%)` | `13.00 <1.00> (+1.00)` | |
   | [...common/table/view/FileSystemViewStorageConfig.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvRmlsZVN5c3RlbVZpZXdTdG9yYWdlQ29uZmlnLmphdmE=) | `60.00% <83.33%> (+1.57%)` | `11.00 <1.00> (+1.00)` | |
   | [...a/org/apache/hudi/common/util/ClusteringUtils.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQ2x1c3RlcmluZ1V0aWxzLmphdmE=) | `90.47% <90.47%> (ø)` | `18.00 <18.00> (?)` | |
   | [...g/apache/hudi/common/model/WriteOperationType.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL1dyaXRlT3BlcmF0aW9uVHlwZS5qYXZh) | `55.55% <100.00%> (+1.70%)` | `2.00 <0.00> (ø)` | |
   | ... and [28 more](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree-more) | |
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r514294873



##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",

Review comment:
       @satishkotha For code reviews, there may be different opinions from different reviewers. We should just treat that as opportunities to weed out finer details and drive consensus on a single solution going forward. We need to take most of these case by case, I think this particular case you want to reuse the building blocks of the different avsc in the future for other plans etc, that makes sense to me.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r517780720



##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieSliceInfo",

Review comment:
       Fixed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513514484



##########
File path: hudi-common/src/main/avro/HoodieClusteringStrategy.avsc
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",

Review comment:
       At a high level, don't think we need a clustering strategy avsc for this, we can just drive this from Reflection and classnames, just like CompactionPlans ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513676980



##########
File path: hudi-common/src/main/avro/HoodieClusteringStrategy.avsc
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",

Review comment:
       There needs to be coordination between scheduling and executing clustering. For example, when scheduling, we specify parameters such as 'sortColumns', 'targetFileSize' etc. These needs to be passed to async job that clusters data. These parameters are serialized and stored in the strategy/plan. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r514295564



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/ClusteringOpDTO.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline.dto;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.collection.Pair;
+
+/**
+ * The data transfer object of compaction.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class ClusteringOpDTO {
+
+  @JsonProperty("id")
+  private String fileId;

Review comment:
       Thanks for the explanation.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513522510



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
##########
@@ -710,6 +721,40 @@ private String formatPartitionKey(String partitionStr) {
    */
   abstract void removePendingCompactionOperations(Stream<Pair<String, CompactionOperation>> operations);
 
+  /**
+   * Check if there is an outstanding clustering operation (requested/inflight) scheduled for this file.
+   *
+   * @param fgId File-Group Id
+   * @return true if there is a pending clustering, false otherwise
+   */
+  protected abstract boolean isPendingClusteringScheduledForFileId(HoodieFileGroupId fgId);
+
+  /**
+   *  Get pending clustering instant time for specified file group. Return None if file group is not in pending
+   *  clustering operation.
+   */
+  protected abstract Option<HoodieInstant> getPendingClusteringInstant(final HoodieFileGroupId fileGroupId);
+
+  /**
+   * Fetch all file groups in pending clustering.
+   */
+  protected abstract Stream<Pair<HoodieFileGroupId, HoodieInstant>> fetchFileGroupsInPendingClustering();
+
+  /**
+   * resets the pending clustering operation and overwrite with the new list.
+   */
+  abstract void resetFileGroupsInPendingClustering(Map<HoodieFileGroupId, HoodieInstant> fgIdToInstantMap);
+
+  /**
+   * Add metadata for pending clustering operation to store for given instant.

Review comment:
       Is this comment the right description for this API ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513679290



##########
File path: hudi-common/src/main/avro/HoodieRequestedReplaceMetadata.avsc
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",

Review comment:
       I think this model is flexible for future operations. Replace could be used for other operations such as column pruning and many other use cases. Trying to fit metadata for all these special operations into one format may cause problems (InsertOverwrite doesnt really have a plan today.  so we have to generate plan just for sake of creating it). We also dont want to introduce other top level commits because its lot of work, so i want to keep this as generic as possible.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r518197476



##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieSliceInfo",
+   "fields":[
+      {
+        "name":"dataFilePath",
+        "type":["null","string"],
+        "default": null
+      },
+      {
+        "name":"deltaFilePaths",
+        "type":["null", {
+           "type":"array",
+           "items":"string"
+        }],
+        "default": null
+      },
+      {
+        "name":"fileId",
+        "type":["null","string"]
+      },
+      {
+        "name":"partitionPath",
+        "type":["null","string"],
+        "default": null
+      },
+      {
+        "name":"bootstrapFilePath",
+        "type":["null", "string"],
+        "default": null
+      },
+      {
+        "name":"version",

Review comment:
       Can we add this as the first field (so no issues with reordering etc) we should always be able to read this 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r518196610



##########
File path: hudi-common/src/main/avro/HoodieClusteringGroup.avsc
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieClusteringGroup",
+   "type":"record",
+   "fields":[
+      {
+         /* Group of files that needs to merged. All the slices in a group will belong to same partition initially.

Review comment:
       My concern was mostly about metadata being able to model that. So we good here




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r514290232



##########
File path: hudi-common/src/main/avro/HoodieClusteringStrategy.avsc
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",

Review comment:
       The parameters such as 'sortColumns', 'targetFileSize' etc serialized to the clustering job makes sense to me, my question is around the strategy class name itself being written to the file. Is it just for tracking the strategy or will the async clustering job actually make use of this strategy to read our the strategyParams ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513520048



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/ClusteringOpDTO.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline.dto;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.collection.Pair;
+
+/**
+ * The data transfer object of compaction.

Review comment:
       Alter comments accordingly




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513515594



##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",

Review comment:
       Same comment, please roll this up into HoodieClusteringPlan.avsc




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513511044



##########
File path: hudi-common/src/main/avro/HoodieClusteringGroup.avsc
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieClusteringGroup",
+   "type":"record",
+   "fields":[
+      {
+         /* Group of files that needs to merged. All the slices in a group will belong to same partition initially.

Review comment:
       @satishkotha AVRO has a way to add description with a type called "doc", we can either use that or add these comments at the beginning of the avro file.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#issuecomment-718911806


   @satishkotha @n3nash will do 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] lw309637554 commented on pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
lw309637554 commented on pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#issuecomment-721218484


   @satishkotha  this pull request is so good.   Base on this pull request. Can i begin to work on implement Simple ClusteringStrategy and ClusteringActionExecutor? or have other task i can participate about clustering now. cc @n3nash @vinothchandar @leesf  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r516826336



##########
File path: hudi-common/src/main/avro/HoodieClusteringGroup.avsc
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieClusteringGroup",
+   "type":"record",
+   "fields":[
+      {
+         /* Group of files that needs to merged. All the slices in a group will belong to same partition initially.

Review comment:
       I wonder if we should start out with the assumption of having all files in a given partition. Does this simplify anything in the design now? I would love to implement this generically right away, otherwise we will spend cycles thinking about migrations, eetc etc down the line. 

##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieSliceInfo",
+   "fields":[
+      {
+        "name":"dataFilePath",
+        "type":["null","string"],
+        "default": null
+      },
+      {
+        "name":"deltaFilePaths",
+        "type":["null", {
+           "type":"array",
+           "items":"string"
+        }],
+        "default": null
+      },
+      {
+        "name":"fileId",
+        "type":["null","string"]
+      },
+      {
+        "name":"partitionPath",
+        "type":["null","string"],
+        "default": null
+      },
+      {
+        "name":"bootstrapFilePath",

Review comment:
       this can be fetched from the bootstrap index as well right? its immutable anyway?

##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieSliceInfo",

Review comment:
       lets ensure this also contains a version field? in fact, any of these new avsc should have a version at the start preferably, so we can evolve. 

##########
File path: hudi-common/src/main/avro/HoodieClusteringGroup.avsc
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieClusteringGroup",
+   "type":"record",
+   "fields":[
+      {
+         /* Group of files that needs to merged. All the slices in a group will belong to same partition initially.

Review comment:
       or you are just suggesting that the clustreing strategy currently only does it within partitions? (totally reasonable approach!)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513683470



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.avro.model.HoodieClusteringGroup;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieClusteringStrategy;
+import org.apache.hudi.avro.model.HoodieRequestedReplaceMetadata;
+import org.apache.hudi.avro.model.HoodieSliceInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Helper class to generate clustering plan from metadata.
+ */
+public class ClusteringUtils {
+
+  private static final Logger LOG = LogManager.getLogger(ClusteringUtils.class);
+
+  public static final String TOTAL_IO_READ_MB = "TOTAL_IO_READ_MB";
+  public static final String TOTAL_LOG_FILE_SIZE = "TOTAL_LOG_FILES_SIZE";
+  public static final String TOTAL_LOG_FILES = "TOTAL_LOG_FILES";
+
+  /**
+   * Get all pending clustering plans along with their instants.
+   */
+  public static Stream<Pair<HoodieInstant, HoodieClusteringPlan>> getAllPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    List<HoodieInstant> pendingReplaceInstants =
+        metaClient.getActiveTimeline().filterPendingReplaceTimeline().getInstants().collect(Collectors.toList());
+    return pendingReplaceInstants.stream().map(instant -> getClusteringPlan(metaClient, instant))
+        .filter(Option::isPresent).map(Option::get);
+  }
+
+  public static Option<Pair<HoodieInstant, HoodieClusteringPlan>> getClusteringPlan(HoodieTableMetaClient metaClient, HoodieInstant requestedReplaceInstant) {
+    try {
+      Option<byte[]> content = metaClient.getActiveTimeline().getInstantDetails(requestedReplaceInstant);
+      if (!content.isPresent() || content.get().length == 0) {
+        // few operations create requested file without any content. Assume these are not clustering
+        LOG.warn("No content found in requested file for instant " + requestedReplaceInstant);
+        return Option.empty();
+      }
+      HoodieRequestedReplaceMetadata requestedReplaceMetadata = TimelineMetadataUtils.deserializeRequestedReplaceMetadta(content.get());
+      if (WriteOperationType.CLUSTER.name().equals(requestedReplaceMetadata.getOperationType())) {
+        return Option.of(Pair.of(requestedReplaceInstant, requestedReplaceMetadata.getClusteringPlan()));
+      }
+      return Option.empty();
+    } catch (IOException e) {
+      throw new HoodieIOException("Error reading clustering plan " + requestedReplaceInstant.getTimestamp(), e);
+    }
+  }
+
+  /**
+   * Get filegroups to pending clustering instant mapping for all pending clustering plans.
+   * This includes all clustering operattions in 'requested' and 'inflight' states.
+   */
+  public static Map<HoodieFileGroupId, HoodieInstant> getAllFileGroupsInPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    Stream<Pair<HoodieInstant, HoodieClusteringPlan>> pendingClusteringPlans = getAllPendingClusteringPlans(metaClient);
+    Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> resultStream = pendingClusteringPlans.flatMap(clusteringPlan ->
+        // get all filegroups in the plan
+        getFileGroupEntriesInClusteringPlan(clusteringPlan.getLeft(), clusteringPlan.getRight()));
+
+    Map<HoodieFileGroupId, HoodieInstant> resultMap = resultStream.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+    LOG.info("Found " + resultMap.size() + " files in pending clustering operations");
+    return resultMap;
+  }
+
+  public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupsInPendingClusteringInstant(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    Stream<HoodieFileGroupId> partitionToFileIdLists = clusteringPlan.getInputGroups().stream().flatMap(ClusteringUtils::getFileGroupsFromClusteringGroup);
+    return partitionToFileIdLists.map(e -> Pair.of(e, instant));
+  }
+
+  private static Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesInClusteringPlan(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    return getFileGroupsInPendingClusteringInstant(instant, clusteringPlan).map(entry ->
+        new AbstractMap.SimpleEntry<>(entry.getLeft(), entry.getRight()));
+  }
+
+  private static Stream<HoodieFileGroupId> getFileGroupsFromClusteringGroup(HoodieClusteringGroup group) {
+    return group.getSlices().stream().map(slice -> new HoodieFileGroupId(slice.getPartitionPath(), slice.getFileId()));
+  }
+
+  /**
+   * Create clustering plan from input fileSliceGroups.
+   */
+  public static HoodieClusteringPlan createClusteringPlan(String strategyClassName,
+                                                          Map<String, String> strategyParams,
+                                                          List<FileSlice>[] fileSliceGroups) {
+    List<HoodieClusteringGroup> clusteringGroups = new ArrayList<>();
+    for (int i = 0; i < fileSliceGroups.length; i++) {

Review comment:
       Sure. will fix it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513680058



##########
File path: hudi-common/src/main/avro/HoodieSliceInfo.avsc
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",

Review comment:
       I see value in keeping these in separate files. It was suggested to me in earlier reviews as well. Please tell me if there is any value in combining these. We can perhaps add this to coding guidelines.Without that, different reviewers are giving different suggestions.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513728542



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.avro.model.HoodieClusteringGroup;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieClusteringStrategy;
+import org.apache.hudi.avro.model.HoodieRequestedReplaceMetadata;
+import org.apache.hudi.avro.model.HoodieSliceInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Helper class to generate clustering plan from metadata.
+ */
+public class ClusteringUtils {
+
+  private static final Logger LOG = LogManager.getLogger(ClusteringUtils.class);
+
+  public static final String TOTAL_IO_READ_MB = "TOTAL_IO_READ_MB";
+  public static final String TOTAL_LOG_FILE_SIZE = "TOTAL_LOG_FILES_SIZE";
+  public static final String TOTAL_LOG_FILES = "TOTAL_LOG_FILES";
+
+  /**
+   * Get all pending clustering plans along with their instants.
+   */
+  public static Stream<Pair<HoodieInstant, HoodieClusteringPlan>> getAllPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    List<HoodieInstant> pendingReplaceInstants =
+        metaClient.getActiveTimeline().filterPendingReplaceTimeline().getInstants().collect(Collectors.toList());
+    return pendingReplaceInstants.stream().map(instant -> getClusteringPlan(metaClient, instant))
+        .filter(Option::isPresent).map(Option::get);
+  }
+
+  public static Option<Pair<HoodieInstant, HoodieClusteringPlan>> getClusteringPlan(HoodieTableMetaClient metaClient, HoodieInstant requestedReplaceInstant) {
+    try {
+      Option<byte[]> content = metaClient.getActiveTimeline().getInstantDetails(requestedReplaceInstant);
+      if (!content.isPresent() || content.get().length == 0) {
+        // few operations create requested file without any content. Assume these are not clustering
+        LOG.warn("No content found in requested file for instant " + requestedReplaceInstant);
+        return Option.empty();
+      }
+      HoodieRequestedReplaceMetadata requestedReplaceMetadata = TimelineMetadataUtils.deserializeRequestedReplaceMetadta(content.get());
+      if (WriteOperationType.CLUSTER.name().equals(requestedReplaceMetadata.getOperationType())) {
+        return Option.of(Pair.of(requestedReplaceInstant, requestedReplaceMetadata.getClusteringPlan()));
+      }
+      return Option.empty();
+    } catch (IOException e) {
+      throw new HoodieIOException("Error reading clustering plan " + requestedReplaceInstant.getTimestamp(), e);
+    }
+  }
+
+  /**
+   * Get filegroups to pending clustering instant mapping for all pending clustering plans.
+   * This includes all clustering operattions in 'requested' and 'inflight' states.
+   */
+  public static Map<HoodieFileGroupId, HoodieInstant> getAllFileGroupsInPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    Stream<Pair<HoodieInstant, HoodieClusteringPlan>> pendingClusteringPlans = getAllPendingClusteringPlans(metaClient);
+    Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> resultStream = pendingClusteringPlans.flatMap(clusteringPlan ->
+        // get all filegroups in the plan
+        getFileGroupEntriesInClusteringPlan(clusteringPlan.getLeft(), clusteringPlan.getRight()));
+
+    Map<HoodieFileGroupId, HoodieInstant> resultMap = resultStream.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+    LOG.info("Found " + resultMap.size() + " files in pending clustering operations");
+    return resultMap;
+  }
+
+  public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupsInPendingClusteringInstant(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    Stream<HoodieFileGroupId> partitionToFileIdLists = clusteringPlan.getInputGroups().stream().flatMap(ClusteringUtils::getFileGroupsFromClusteringGroup);
+    return partitionToFileIdLists.map(e -> Pair.of(e, instant));
+  }
+
+  private static Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesInClusteringPlan(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    return getFileGroupsInPendingClusteringInstant(instant, clusteringPlan).map(entry ->
+        new AbstractMap.SimpleEntry<>(entry.getLeft(), entry.getRight()));
+  }
+
+  private static Stream<HoodieFileGroupId> getFileGroupsFromClusteringGroup(HoodieClusteringGroup group) {
+    return group.getSlices().stream().map(slice -> new HoodieFileGroupId(slice.getPartitionPath(), slice.getFileId()));
+  }
+
+  /**
+   * Create clustering plan from input fileSliceGroups.
+   */
+  public static HoodieClusteringPlan createClusteringPlan(String strategyClassName,
+                                                          Map<String, String> strategyParams,
+                                                          List<FileSlice>[] fileSliceGroups) {
+    List<HoodieClusteringGroup> clusteringGroups = new ArrayList<>();
+    for (int i = 0; i < fileSliceGroups.length; i++) {
+      List<FileSlice> fileSliceGroup = fileSliceGroups[i];
+      Map<String, Double> groupMetrics = buildMetrics(fileSliceGroup);
+      List<HoodieSliceInfo> sliceInfos = getFileSliceInfo(fileSliceGroup);
+      clusteringGroups.add(HoodieClusteringGroup.newBuilder().setSlices(sliceInfos).setMetrics(groupMetrics).build());
+    }
+
+    HoodieClusteringStrategy strategy = HoodieClusteringStrategy.newBuilder()
+        .setStrategyClassName(strategyClassName).setStrategyParams(strategyParams)
+        .build();
+
+    HoodieClusteringPlan plan = HoodieClusteringPlan.newBuilder()
+        .setInputGroups(clusteringGroups)
+        .setExtraMetadata(new HashMap<>())

Review comment:
       fixed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] codecov-io edited a comment on pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#issuecomment-718281647


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=h1) Report
   > Merging [#2202](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/c5e10d668f9366f29bdf7721f7efe4140782527b?el=desc) will **decrease** coverage by `0.12%`.
   > The diff coverage is `69.59%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/2202/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #2202      +/-   ##
   ============================================
   - Coverage     53.63%   53.50%   -0.13%     
   + Complexity     2848     2767      -81     
   ============================================
     Files           359      348      -11     
     Lines         16545    16098     -447     
     Branches       1780     1642     -138     
   ============================================
   - Hits           8874     8614     -260     
   + Misses         6912     6789     -123     
   + Partials        759      695      -64     
   ```
   
   | Flag | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | hudicli | `38.37% <ø> (ø)` | `193.00 <ø> (ø)` | |
   | hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | hudicommon | `55.26% <68.09%> (+0.51%)` | `1853.00 <41.00> (+58.00)` | |
   | hudihadoopmr | `32.94% <ø> (-0.12%)` | `183.00 <ø> (+2.00)` | :arrow_down: |
   | hudispark | `65.58% <ø> (+0.10%)` | `155.00 <ø> (-149.00)` | :arrow_up: |
   | huditimelineservice | `65.30% <100.00%> (+3.01%)` | `56.00 <3.00> (+6.00)` | |
   | hudiutilities | `70.09% <ø> (+0.01%)` | `327.00 <ø> (+2.00)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | `93.18% <ø> (ø)` | `43.00 <0.00> (ø)` | |
   | [...udi/common/table/timeline/dto/ClusteringOpDTO.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL2R0by9DbHVzdGVyaW5nT3BEVE8uamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...common/table/view/PriorityBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUHJpb3JpdHlCYXNlZEZpbGVTeXN0ZW1WaWV3LmphdmE=) | `97.10% <0.00%> (-1.43%)` | `33.00 <0.00> (ø)` | |
   | [...on/table/view/RemoteHoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUmVtb3RlSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | [...i/common/table/view/HoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `66.66% <28.57%> (-8.61%)` | `34.00 <3.00> (+3.00)` | :arrow_down: |
   | [.../common/table/view/RocksDbBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUm9ja3NEYkJhc2VkRmlsZVN5c3RlbVZpZXcuamF2YQ==) | `80.40% <66.66%> (-0.06%)` | `72.00 <8.00> (+10.00)` | :arrow_down: |
   | [...on/table/view/SpillableMapBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvU3BpbGxhYmxlTWFwQmFzZWRGaWxlU3lzdGVtVmlldy5qYXZh) | `74.19% <77.77%> (+0.60%)` | `13.00 <1.00> (+1.00)` | |
   | [...common/table/view/FileSystemViewStorageConfig.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvRmlsZVN5c3RlbVZpZXdTdG9yYWdlQ29uZmlnLmphdmE=) | `60.00% <83.33%> (+1.57%)` | `11.00 <1.00> (+1.00)` | |
   | [...a/org/apache/hudi/common/util/ClusteringUtils.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQ2x1c3RlcmluZ1V0aWxzLmphdmE=) | `90.47% <90.47%> (ø)` | `18.00 <18.00> (?)` | |
   | [...g/apache/hudi/common/model/WriteOperationType.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL1dyaXRlT3BlcmF0aW9uVHlwZS5qYXZh) | `55.55% <100.00%> (+1.70%)` | `2.00 <0.00> (ø)` | |
   | ... and [45 more](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree-more) | |
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r514292159



##########
File path: hudi-common/src/main/avro/HoodieRequestedReplaceMetadata.avsc
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",

Review comment:
       Keeping the model flexible sounds fair to me, I'm not following how that is connected to introducing other top level commits, could you please expand ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#issuecomment-721332533


   @satishkotha this is good to go, except for some minor comments. Feel free to address or file follow ups (except may be the version field, which is good to do now). 
   
   @n3nash I will let you take it from here. 
   
   @lw309637554 I believe @satishkotha has a umbrella jira with all the issues. Any thoughts on how we can parallelize @satishkotha ? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] codecov-io edited a comment on pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#issuecomment-718281647


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=h1) Report
   > Merging [#2202](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/c5e10d668f9366f29bdf7721f7efe4140782527b?el=desc) will **decrease** coverage by `9.04%`.
   > The diff coverage is `68.09%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/2202/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #2202      +/-   ##
   ============================================
   - Coverage     53.63%   44.58%   -9.05%     
   + Complexity     2848     2277     -571     
   ============================================
     Files           359      316      -43     
     Lines         16545    14311    -2234     
     Branches       1780     1462     -318     
   ============================================
   - Hits           8874     6381    -2493     
   - Misses         6912     7449     +537     
   + Partials        759      481     -278     
   ```
   
   | Flag | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | hudicli | `38.37% <ø> (ø)` | `193.00 <ø> (ø)` | |
   | hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | hudicommon | `55.26% <68.09%> (+0.51%)` | `1853.00 <41.00> (+58.00)` | |
   | hudihadoopmr | `32.94% <ø> (-0.12%)` | `183.00 <ø> (+2.00)` | :arrow_down: |
   | hudispark | `?` | `?` | |
   | huditimelineservice | `?` | `?` | |
   | hudiutilities | `10.39% <ø> (-59.69%)` | `48.00 <ø> (-277.00)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | `93.18% <ø> (ø)` | `43.00 <0.00> (ø)` | |
   | [...udi/common/table/timeline/dto/ClusteringOpDTO.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL2R0by9DbHVzdGVyaW5nT3BEVE8uamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...common/table/view/PriorityBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUHJpb3JpdHlCYXNlZEZpbGVTeXN0ZW1WaWV3LmphdmE=) | `97.10% <0.00%> (-1.43%)` | `33.00 <0.00> (ø)` | |
   | [...on/table/view/RemoteHoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUmVtb3RlSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | [...i/common/table/view/HoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `66.66% <28.57%> (-8.61%)` | `34.00 <3.00> (+3.00)` | :arrow_down: |
   | [.../common/table/view/RocksDbBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUm9ja3NEYkJhc2VkRmlsZVN5c3RlbVZpZXcuamF2YQ==) | `80.40% <66.66%> (-0.06%)` | `72.00 <8.00> (+10.00)` | :arrow_down: |
   | [...on/table/view/SpillableMapBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvU3BpbGxhYmxlTWFwQmFzZWRGaWxlU3lzdGVtVmlldy5qYXZh) | `74.19% <77.77%> (+0.60%)` | `13.00 <1.00> (+1.00)` | |
   | [...common/table/view/FileSystemViewStorageConfig.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvRmlsZVN5c3RlbVZpZXdTdG9yYWdlQ29uZmlnLmphdmE=) | `60.00% <83.33%> (+1.57%)` | `11.00 <1.00> (+1.00)` | |
   | [...a/org/apache/hudi/common/util/ClusteringUtils.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQ2x1c3RlcmluZ1V0aWxzLmphdmE=) | `90.47% <90.47%> (ø)` | `18.00 <18.00> (?)` | |
   | [...g/apache/hudi/common/model/WriteOperationType.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL1dyaXRlT3BlcmF0aW9uVHlwZS5qYXZh) | `55.55% <100.00%> (+1.70%)` | `2.00 <0.00> (ø)` | |
   | ... and [100 more](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree-more) | |
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513728970



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.avro.model.HoodieClusteringGroup;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieClusteringStrategy;
+import org.apache.hudi.avro.model.HoodieRequestedReplaceMetadata;
+import org.apache.hudi.avro.model.HoodieSliceInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Helper class to generate clustering plan from metadata.
+ */
+public class ClusteringUtils {
+
+  private static final Logger LOG = LogManager.getLogger(ClusteringUtils.class);
+
+  public static final String TOTAL_IO_READ_MB = "TOTAL_IO_READ_MB";
+  public static final String TOTAL_LOG_FILE_SIZE = "TOTAL_LOG_FILES_SIZE";
+  public static final String TOTAL_LOG_FILES = "TOTAL_LOG_FILES";
+
+  /**
+   * Get all pending clustering plans along with their instants.
+   */
+  public static Stream<Pair<HoodieInstant, HoodieClusteringPlan>> getAllPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    List<HoodieInstant> pendingReplaceInstants =
+        metaClient.getActiveTimeline().filterPendingReplaceTimeline().getInstants().collect(Collectors.toList());
+    return pendingReplaceInstants.stream().map(instant -> getClusteringPlan(metaClient, instant))
+        .filter(Option::isPresent).map(Option::get);
+  }
+
+  public static Option<Pair<HoodieInstant, HoodieClusteringPlan>> getClusteringPlan(HoodieTableMetaClient metaClient, HoodieInstant requestedReplaceInstant) {
+    try {
+      Option<byte[]> content = metaClient.getActiveTimeline().getInstantDetails(requestedReplaceInstant);
+      if (!content.isPresent() || content.get().length == 0) {
+        // few operations create requested file without any content. Assume these are not clustering
+        LOG.warn("No content found in requested file for instant " + requestedReplaceInstant);
+        return Option.empty();
+      }
+      HoodieRequestedReplaceMetadata requestedReplaceMetadata = TimelineMetadataUtils.deserializeRequestedReplaceMetadta(content.get());
+      if (WriteOperationType.CLUSTER.name().equals(requestedReplaceMetadata.getOperationType())) {
+        return Option.of(Pair.of(requestedReplaceInstant, requestedReplaceMetadata.getClusteringPlan()));
+      }
+      return Option.empty();
+    } catch (IOException e) {
+      throw new HoodieIOException("Error reading clustering plan " + requestedReplaceInstant.getTimestamp(), e);
+    }
+  }
+
+  /**
+   * Get filegroups to pending clustering instant mapping for all pending clustering plans.
+   * This includes all clustering operattions in 'requested' and 'inflight' states.
+   */
+  public static Map<HoodieFileGroupId, HoodieInstant> getAllFileGroupsInPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    Stream<Pair<HoodieInstant, HoodieClusteringPlan>> pendingClusteringPlans = getAllPendingClusteringPlans(metaClient);
+    Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> resultStream = pendingClusteringPlans.flatMap(clusteringPlan ->
+        // get all filegroups in the plan
+        getFileGroupEntriesInClusteringPlan(clusteringPlan.getLeft(), clusteringPlan.getRight()));
+
+    Map<HoodieFileGroupId, HoodieInstant> resultMap = resultStream.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+    LOG.info("Found " + resultMap.size() + " files in pending clustering operations");
+    return resultMap;
+  }
+
+  public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupsInPendingClusteringInstant(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    Stream<HoodieFileGroupId> partitionToFileIdLists = clusteringPlan.getInputGroups().stream().flatMap(ClusteringUtils::getFileGroupsFromClusteringGroup);
+    return partitionToFileIdLists.map(e -> Pair.of(e, instant));
+  }
+
+  private static Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesInClusteringPlan(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    return getFileGroupsInPendingClusteringInstant(instant, clusteringPlan).map(entry ->
+        new AbstractMap.SimpleEntry<>(entry.getLeft(), entry.getRight()));
+  }
+
+  private static Stream<HoodieFileGroupId> getFileGroupsFromClusteringGroup(HoodieClusteringGroup group) {
+    return group.getSlices().stream().map(slice -> new HoodieFileGroupId(slice.getPartitionPath(), slice.getFileId()));
+  }
+
+  /**
+   * Create clustering plan from input fileSliceGroups.
+   */
+  public static HoodieClusteringPlan createClusteringPlan(String strategyClassName,
+                                                          Map<String, String> strategyParams,
+                                                          List<FileSlice>[] fileSliceGroups) {
+    List<HoodieClusteringGroup> clusteringGroups = new ArrayList<>();
+    for (int i = 0; i < fileSliceGroups.length; i++) {
+      List<FileSlice> fileSliceGroup = fileSliceGroups[i];
+      Map<String, Double> groupMetrics = buildMetrics(fileSliceGroup);
+      List<HoodieSliceInfo> sliceInfos = getFileSliceInfo(fileSliceGroup);
+      clusteringGroups.add(HoodieClusteringGroup.newBuilder().setSlices(sliceInfos).setMetrics(groupMetrics).build());
+    }
+
+    HoodieClusteringStrategy strategy = HoodieClusteringStrategy.newBuilder()
+        .setStrategyClassName(strategyClassName).setStrategyParams(strategyParams)
+        .build();
+
+    HoodieClusteringPlan plan = HoodieClusteringPlan.newBuilder()
+        .setInputGroups(clusteringGroups)
+        .setExtraMetadata(new HashMap<>())
+        .setStrategy(strategy)
+        .build();
+
+    return plan;
+  }
+
+  private static List<HoodieSliceInfo> getFileSliceInfo(List<FileSlice> slices) {
+    FileSlice slice1 = slices.get(0);
+    System.out.println(slice1.getLogFiles().count());
+    System.out.println(slice1.getBaseFile().get());
+    System.out.println(slice1.getPartitionPath());
+    return slices.stream().map(slice -> new HoodieSliceInfo().newBuilder()
+        .setPartitionPath(slice.getPartitionPath())
+        .setFileId(slice.getFileId())
+        .setDataFilePath(slice.getBaseFile().map(BaseFile::getPath).orElse(""))

Review comment:
       Because this is avro, I'm not sure if Option would work. Changed it to null to match with compaction.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] codecov-io edited a comment on pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#issuecomment-718281647


   # [Codecov](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=h1) Report
   > Merging [#2202](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/c5e10d668f9366f29bdf7721f7efe4140782527b?el=desc) will **increase** coverage by `0.34%`.
   > The diff coverage is `69.59%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/hudi/pull/2202/graphs/tree.svg?width=650&height=150&src=pr&token=VTTXabwbs2)](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #2202      +/-   ##
   ============================================
   + Coverage     53.63%   53.98%   +0.34%     
   - Complexity     2848     2904      +56     
   ============================================
     Files           359      361       +2     
     Lines         16545    16735     +190     
     Branches       1780     1788       +8     
   ============================================
   + Hits           8874     9034     +160     
   - Misses         6912     6938      +26     
   - Partials        759      763       +4     
   ```
   
   | Flag | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | #hudicli | `38.37% <ø> (ø)` | `193.00 <ø> (ø)` | |
   | #hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
   | #hudicommon | `55.12% <68.09%> (+0.36%)` | `1843.00 <41.00> (+48.00)` | |
   | #hudihadoopmr | `33.05% <ø> (ø)` | `181.00 <ø> (ø)` | |
   | #hudispark | `65.95% <ø> (+0.46%)` | `304.00 <ø> (ø)` | |
   | #huditimelineservice | `65.30% <100.00%> (+3.01%)` | `56.00 <3.00> (+6.00)` | |
   | #hudiutilities | `70.09% <ø> (+0.01%)` | `327.00 <ø> (+2.00)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2202?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...che/hudi/common/table/timeline/HoodieTimeline.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL0hvb2RpZVRpbWVsaW5lLmphdmE=) | `93.18% <ø> (ø)` | `43.00 <0.00> (ø)` | |
   | [...udi/common/table/timeline/dto/ClusteringOpDTO.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3RpbWVsaW5lL2R0by9DbHVzdGVyaW5nT3BEVE8uamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...common/table/view/PriorityBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUHJpb3JpdHlCYXNlZEZpbGVTeXN0ZW1WaWV3LmphdmE=) | `97.10% <0.00%> (-1.43%)` | `33.00 <0.00> (ø)` | |
   | [...on/table/view/RemoteHoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUmVtb3RlSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | |
   | [...i/common/table/view/HoodieTableFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvSG9vZGllVGFibGVGaWxlU3lzdGVtVmlldy5qYXZh) | `66.66% <28.57%> (-8.61%)` | `34.00 <3.00> (+3.00)` | :arrow_down: |
   | [.../common/table/view/RocksDbBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvUm9ja3NEYkJhc2VkRmlsZVN5c3RlbVZpZXcuamF2YQ==) | `80.40% <66.66%> (-0.06%)` | `72.00 <8.00> (+10.00)` | :arrow_down: |
   | [...on/table/view/SpillableMapBasedFileSystemView.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvU3BpbGxhYmxlTWFwQmFzZWRGaWxlU3lzdGVtVmlldy5qYXZh) | `74.19% <77.77%> (+0.60%)` | `13.00 <1.00> (+1.00)` | |
   | [...common/table/view/FileSystemViewStorageConfig.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL3ZpZXcvRmlsZVN5c3RlbVZpZXdTdG9yYWdlQ29uZmlnLmphdmE=) | `60.00% <83.33%> (+1.57%)` | `11.00 <1.00> (+1.00)` | |
   | [...a/org/apache/hudi/common/util/ClusteringUtils.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3V0aWwvQ2x1c3RlcmluZ1V0aWxzLmphdmE=) | `90.47% <90.47%> (ø)` | `18.00 <18.00> (?)` | |
   | [...g/apache/hudi/common/model/WriteOperationType.java](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL21vZGVsL1dyaXRlT3BlcmF0aW9uVHlwZS5qYXZh) | `55.55% <100.00%> (+1.70%)` | `2.00 <0.00> (ø)` | |
   | ... and [28 more](https://codecov.io/gh/apache/hudi/pull/2202/diff?src=pr&el=tree-more) | |
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513683354



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
##########
@@ -710,6 +721,40 @@ private String formatPartitionKey(String partitionStr) {
    */
   abstract void removePendingCompactionOperations(Stream<Pair<String, CompactionOperation>> operations);
 
+  /**
+   * Check if there is an outstanding clustering operation (requested/inflight) scheduled for this file.
+   *
+   * @param fgId File-Group Id
+   * @return true if there is a pending clustering, false otherwise
+   */
+  protected abstract boolean isPendingClusteringScheduledForFileId(HoodieFileGroupId fgId);
+
+  /**
+   *  Get pending clustering instant time for specified file group. Return None if file group is not in pending
+   *  clustering operation.
+   */
+  protected abstract Option<HoodieInstant> getPendingClusteringInstant(final HoodieFileGroupId fileGroupId);
+
+  /**
+   * Fetch all file groups in pending clustering.
+   */
+  protected abstract Stream<Pair<HoodieFileGroupId, HoodieInstant>> fetchFileGroupsInPendingClustering();
+
+  /**
+   * resets the pending clustering operation and overwrite with the new list.
+   */
+  abstract void resetFileGroupsInPendingClustering(Map<HoodieFileGroupId, HoodieInstant> fgIdToInstantMap);

Review comment:
       This is called from FileSystemView#init method to refresh list of files in pending clustering. The list is used to block updates  HUDI-1354 and scheduling further clustering operations.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r517779823



##########
File path: hudi-common/src/main/avro/HoodieClusteringGroup.avsc
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "namespace":"org.apache.hudi.avro.model",
+   "type":"record",
+   "name":"HoodieClusteringGroup",
+   "type":"record",
+   "fields":[
+      {
+         /* Group of files that needs to merged. All the slices in a group will belong to same partition initially.

Review comment:
       Yes, strategy only groups within a partition. But the metadata is flexible in case we need to implement more complex disk layouts in future.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513529174



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ClusteringUtils.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.avro.model.HoodieClusteringGroup;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieClusteringStrategy;
+import org.apache.hudi.avro.model.HoodieRequestedReplaceMetadata;
+import org.apache.hudi.avro.model.HoodieSliceInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BaseFile;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * Helper class to generate clustering plan from metadata.
+ */
+public class ClusteringUtils {
+
+  private static final Logger LOG = LogManager.getLogger(ClusteringUtils.class);
+
+  public static final String TOTAL_IO_READ_MB = "TOTAL_IO_READ_MB";
+  public static final String TOTAL_LOG_FILE_SIZE = "TOTAL_LOG_FILES_SIZE";
+  public static final String TOTAL_LOG_FILES = "TOTAL_LOG_FILES";
+
+  /**
+   * Get all pending clustering plans along with their instants.
+   */
+  public static Stream<Pair<HoodieInstant, HoodieClusteringPlan>> getAllPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    List<HoodieInstant> pendingReplaceInstants =
+        metaClient.getActiveTimeline().filterPendingReplaceTimeline().getInstants().collect(Collectors.toList());
+    return pendingReplaceInstants.stream().map(instant -> getClusteringPlan(metaClient, instant))
+        .filter(Option::isPresent).map(Option::get);
+  }
+
+  public static Option<Pair<HoodieInstant, HoodieClusteringPlan>> getClusteringPlan(HoodieTableMetaClient metaClient, HoodieInstant requestedReplaceInstant) {
+    try {
+      Option<byte[]> content = metaClient.getActiveTimeline().getInstantDetails(requestedReplaceInstant);
+      if (!content.isPresent() || content.get().length == 0) {
+        // few operations create requested file without any content. Assume these are not clustering
+        LOG.warn("No content found in requested file for instant " + requestedReplaceInstant);
+        return Option.empty();
+      }
+      HoodieRequestedReplaceMetadata requestedReplaceMetadata = TimelineMetadataUtils.deserializeRequestedReplaceMetadta(content.get());
+      if (WriteOperationType.CLUSTER.name().equals(requestedReplaceMetadata.getOperationType())) {
+        return Option.of(Pair.of(requestedReplaceInstant, requestedReplaceMetadata.getClusteringPlan()));
+      }
+      return Option.empty();
+    } catch (IOException e) {
+      throw new HoodieIOException("Error reading clustering plan " + requestedReplaceInstant.getTimestamp(), e);
+    }
+  }
+
+  /**
+   * Get filegroups to pending clustering instant mapping for all pending clustering plans.
+   * This includes all clustering operattions in 'requested' and 'inflight' states.
+   */
+  public static Map<HoodieFileGroupId, HoodieInstant> getAllFileGroupsInPendingClusteringPlans(
+      HoodieTableMetaClient metaClient) {
+    Stream<Pair<HoodieInstant, HoodieClusteringPlan>> pendingClusteringPlans = getAllPendingClusteringPlans(metaClient);
+    Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> resultStream = pendingClusteringPlans.flatMap(clusteringPlan ->
+        // get all filegroups in the plan
+        getFileGroupEntriesInClusteringPlan(clusteringPlan.getLeft(), clusteringPlan.getRight()));
+
+    Map<HoodieFileGroupId, HoodieInstant> resultMap = resultStream.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+    LOG.info("Found " + resultMap.size() + " files in pending clustering operations");
+    return resultMap;
+  }
+
+  public static Stream<Pair<HoodieFileGroupId, HoodieInstant>> getFileGroupsInPendingClusteringInstant(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    Stream<HoodieFileGroupId> partitionToFileIdLists = clusteringPlan.getInputGroups().stream().flatMap(ClusteringUtils::getFileGroupsFromClusteringGroup);
+    return partitionToFileIdLists.map(e -> Pair.of(e, instant));
+  }
+
+  private static Stream<Map.Entry<HoodieFileGroupId, HoodieInstant>> getFileGroupEntriesInClusteringPlan(
+      HoodieInstant instant, HoodieClusteringPlan clusteringPlan) {
+    return getFileGroupsInPendingClusteringInstant(instant, clusteringPlan).map(entry ->
+        new AbstractMap.SimpleEntry<>(entry.getLeft(), entry.getRight()));
+  }
+
+  private static Stream<HoodieFileGroupId> getFileGroupsFromClusteringGroup(HoodieClusteringGroup group) {
+    return group.getSlices().stream().map(slice -> new HoodieFileGroupId(slice.getPartitionPath(), slice.getFileId()));
+  }
+
+  /**
+   * Create clustering plan from input fileSliceGroups.
+   */
+  public static HoodieClusteringPlan createClusteringPlan(String strategyClassName,
+                                                          Map<String, String> strategyParams,
+                                                          List<FileSlice>[] fileSliceGroups) {
+    List<HoodieClusteringGroup> clusteringGroups = new ArrayList<>();
+    for (int i = 0; i < fileSliceGroups.length; i++) {
+      List<FileSlice> fileSliceGroup = fileSliceGroups[i];
+      Map<String, Double> groupMetrics = buildMetrics(fileSliceGroup);
+      List<HoodieSliceInfo> sliceInfos = getFileSliceInfo(fileSliceGroup);
+      clusteringGroups.add(HoodieClusteringGroup.newBuilder().setSlices(sliceInfos).setMetrics(groupMetrics).build());
+    }
+
+    HoodieClusteringStrategy strategy = HoodieClusteringStrategy.newBuilder()
+        .setStrategyClassName(strategyClassName).setStrategyParams(strategyParams)
+        .build();
+
+    HoodieClusteringPlan plan = HoodieClusteringPlan.newBuilder()
+        .setInputGroups(clusteringGroups)
+        .setExtraMetadata(new HashMap<>())

Review comment:
       What happens if we don't setExtraMetadata to new Hashmap ? In general, we should only set the params which we need to and rest of the uninitialized params should be handled internally in the HoodieClusteringPlan.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] satishkotha commented on pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
satishkotha commented on pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#issuecomment-718182030


   > @satishkotha left some comments.
   > 
   > High level, few more questions:
   > 
   > 1. Seems like a lot of commonality of code with compaction, are we planning on merging them together ?
   > 2. I see that we are going to allow schedule clustering for file groups that even have log files, in the scheduling process are we checking if a compaction is already scheduled for some file groups for which we cannot schedule clustering ?
   
   1. This PR just includes FileSystemView APIs. I think its better to keep them separate APIs because  the functionality is different (updates are allowed for files in compaction for example).  When actual scheduling and clustering is implemented, its possible to reuse some of core logic such as ActionExecutors. Let me know if you see better way to organize this and share  FileSystemView API as well
   2. scheduling logic is not implemented yet. This is just first PR to cover high level FileSystemView APIs. The next PR will include this logic.  Let me know if you want me to combine everything into one PR or if you have any other feedback.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] n3nash commented on a change in pull request #2202: [HUDI-1352] Add FileSystemView APIs to query pending clustering ops

Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #2202:
URL: https://github.com/apache/hudi/pull/2202#discussion_r513522889



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
##########
@@ -710,6 +721,40 @@ private String formatPartitionKey(String partitionStr) {
    */
   abstract void removePendingCompactionOperations(Stream<Pair<String, CompactionOperation>> operations);
 
+  /**
+   * Check if there is an outstanding clustering operation (requested/inflight) scheduled for this file.
+   *
+   * @param fgId File-Group Id
+   * @return true if there is a pending clustering, false otherwise
+   */
+  protected abstract boolean isPendingClusteringScheduledForFileId(HoodieFileGroupId fgId);
+
+  /**
+   *  Get pending clustering instant time for specified file group. Return None if file group is not in pending
+   *  clustering operation.
+   */
+  protected abstract Option<HoodieInstant> getPendingClusteringInstant(final HoodieFileGroupId fileGroupId);
+
+  /**
+   * Fetch all file groups in pending clustering.
+   */
+  protected abstract Stream<Pair<HoodieFileGroupId, HoodieInstant>> fetchFileGroupsInPendingClustering();
+
+  /**
+   * resets the pending clustering operation and overwrite with the new list.
+   */
+  abstract void resetFileGroupsInPendingClustering(Map<HoodieFileGroupId, HoodieInstant> fgIdToInstantMap);

Review comment:
       How will this API be used ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org