You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2025/07/23 20:39:26 UTC

[PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

jonvex opened a new pull request, #13603:
URL: https://github.com/apache/hudi/pull/13603

   ### Change Logs
   
   Create the HUDI Record Index within the Metadata Table for a partitioned dataset where a pair of partition path and record key is unique across the entire table
   
   ### Impact
   
   RLI support for non-gobally unique record keys
   
   ### Risk level (write none, low medium or high below)
   
   High wrote extensive tests
   
   ### Documentation Update
   
   N/A
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3114301209

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 6f8c2b0230c43f44509d16f4472adca831d66fea Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3119911738

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 204a05a68c5875a42a77e9f24252eec07f45e42e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234562986


##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadata.java:
##########
@@ -232,6 +233,18 @@ Map<Pair<String, String>, List<HoodieMetadataColumnStats>> getColumnStats(List<P
    */
   HoodiePairData<String, HoodieRecordGlobalLocation> readRecordIndex(HoodieData<String> recordKeys);
 
+  /**
+   * Returns the location of record keys which are found in the record index.
+   * Records that are not found are ignored and wont be part of map object that is returned.
+   * @param recordKeys list of recordkeys to look up in the index
+   * @param dataTablePartition option of the data table partition to look up from. This is only applicable to partitioned rli
+   *                           record keys are not globally unique so the record key alone does not sufficiently identify an
+   *                           individual record.
+   * @return map from recordkey to the location of the record that was read from the index
+   *
+   */
+  HoodiePairData<String, HoodieRecordGlobalLocation>  readRecordIndex(HoodieData<String> recordKeys, Option<String> dataTablePartition);

Review Comment:
   The param `Option<String> dataTablePartition` is confusing, if the partitioned RLI always got a partition param there, we should rename the method to `readLocalRecordIndex` or something and make the param `dataTablePartition ` just as `String.`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3113926260

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 7344e77705233f8271c2024830e6389d92419a06 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029) 
   * 6f8c2b0230c43f44509d16f4472adca831d66fea UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2232001499


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndex.java:
##########
@@ -193,8 +193,15 @@ public enum IndexType {
 
     @EnumFieldDescription("Index which saves the record key to location mappings in the "
         + "HUDI Metadata Table. Record index is a global index, enforcing key uniqueness across all "
-        + "partitions in the table. Supports sharding to achieve very high scale.")
-    RECORD_INDEX
+        + "partitions in the table. Supports sharding to achieve very high scale. For a table with "
+        + "keys that are only unique inside each partition, use `PARTITIONED_RECORD_INDEX` instead.")
+    RECORD_INDEX,
+
+    @EnumFieldDescription("Index which saves the record key to location mappings in the "
+        + "HUDI Metadata Table. This is a non global index, keys only need to be unique inside each "
+        + "partition in the table. Supports sharding to achieve very high scale. If a table has keys "

Review Comment:
   minor.
   `Index which saves the record key to location mappings in the HUDI Metadata Table. Supports sharding to achieve very high scale. This is a non global index, where keys can be replicated across partitions, since a pair of partition path and record keys will uniquely map to a location using this index. If users expect record keys to be unique across all partitions, use `RECORD_INDEX` instead.` 
   



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1291,10 +1364,52 @@ protected Pair<List<HoodieFileGroupId>, HoodieData<HoodieRecord>> tagRecordsWith
   @Override
   public void update(HoodieCommitMetadata commitMetadata, String instantTime) {
     mayBeReinitMetadataReader();
+    maybeInitializeNewFileGroupsForPartitionedRLI(commitMetadata, instantTime);
     processAndCommit(instantTime, new BatchMetadataConversionFunction(instantTime, commitMetadata, getMetadataPartitionsToUpdate()));
     closeInternal();
   }
 
+  /**
+   * This method is used to initialize new file groups for partitioned record index during the in-flight commit.
+   * It will initialize new file groups for partitions that are newly added in the inflight commit.
+   *
+   * @param commitMetadata metadata for the inflight commit
+   * @param instantTime    Timestamp for the mdt commit
+   */
+  private void maybeInitializeNewFileGroupsForPartitionedRLI(HoodieCommitMetadata commitMetadata, String instantTime) {
+    if (dataWriteConfig.isPartitionedRecordIndexEnabled()) {
+      Set<String> partitionsTouchedByInflightCommit = commitMetadata.getPartitionToWriteStats().keySet();
+      initializeNewFileGroupsForPartitionedRLIHelper(partitionsTouchedByInflightCommit, instantTime);
+    }
+  }
+
+  private void maybeInitializeNewFileGroupsForPartitionedRLI(HoodieData<WriteStatus> writeStatus, String instantTime) {
+    if (dataWriteConfig.isPartitionedRecordIndexEnabled()) {
+      Set<String> partitionsTouchedByInflightCommit = new HashSet<>(writeStatus.map(WriteStatus::getPartitionPath).collectAsList());
+      initializeNewFileGroupsForPartitionedRLIHelper(partitionsTouchedByInflightCommit, instantTime);
+    }
+  }
+
+  private void initializeNewFileGroupsForPartitionedRLIHelper(Set<String> partitionsTouchedByInflightCommit, String instantTime) {
+    try {
+      Set<String> partitionsFromFilesIndex = new HashSet<>(metadata.getAllPartitionPaths());
+      if (!partitionsFromFilesIndex.containsAll(partitionsTouchedByInflightCommit)) {
+        for (String partitionToWrite : partitionsTouchedByInflightCommit) {
+          if (!partitionsFromFilesIndex.contains(partitionToWrite)) {

Review Comment:
   can we do `partitionsTouchedByInflightCommit.removeAll(partitionsFromFilesIndex)` and then do a for loop from the diff set



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -454,29 +457,26 @@ private boolean initializeFromFilesystem(String dataTableInstantTime, List<Metad
       String instantTimeForPartition = generateUniqueInstantTime(dataTableInstantTime);
       String partitionTypeName = partitionType.name();
       LOG.info("Initializing MDT partition {} at instant {}", partitionTypeName, instantTimeForPartition);
-      String partitionName;
+      String relativePartitionPath;
       Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
-      List<String> columnsToIndex = new ArrayList<>();
       Lazy<Option<Schema>> tableSchema = Lazy.lazily(() -> HoodieTableMetadataUtil.tryResolveSchemaForTable(dataMetaClient));
       try {
         switch (partitionType) {
           case FILES:
             fileGroupCountAndRecordsPair = initializeFilesPartition(partitionIdToAllFilesMap);
-            partitionName = FILES.getPartitionPath();
+            initializeFilegroupsAndCommitToMetadataPartition(partitionType, FILES.getPartitionPath(), fileGroupCountAndRecordsPair, instantTimeForPartition, Collections.emptyList());

Review Comment:
   little too long of a name. 
   can we just keep it as `initializeFilegroupsAndCommit` 
   Anyways, entire class is meant to write to mdt :) 



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -721,30 +700,126 @@ private Lazy<List<Pair<String, FileSlice>>> getLazyLatestMergedPartitionFileSlic
     });
   }
 
+  void initializeFilegroupsAndCommitToMetadataPartition(MetadataPartitionType partitionType,

Review Comment:
   can we add a overloaded method which can avoid taking in the last arg which is only required to be set for col stats partition



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/SparkMetadataTablePartitionedRecordIndex.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.data.HoodieListData;
+import org.apache.hudi.common.data.HoodiePairData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordGlobalLocation;
+import org.apache.hudi.common.util.Either;
+import org.apache.hudi.common.util.HoodieDataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaPairRDD;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.metadata.HoodieTableMetadataUtil;
+import org.apache.hudi.metadata.MetadataPartitionType;
+import org.apache.hudi.metadata.BucketizedMetadataTableFileGroupIndexParser;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.function.PairFlatMapFunction;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import scala.Tuple2;
+
+/**
+ * Index to be used with partitioned RLI. Queries the record index for tables with non-global record keys
+ */
+public class SparkMetadataTablePartitionedRecordIndex extends SparkMetadataTableRecordIndex {
+
+  public SparkMetadataTablePartitionedRecordIndex(HoodieWriteConfig config) {
+    super(config);
+  }
+
+  @Override
+  public boolean isGlobal() {
+    return false;
+  }
+
+  @Override
+  protected HoodieIndex.IndexType getFallbackIndexType() {
+    return IndexType.SIMPLE;
+  }
+
+  @Override
+  protected <R> HoodiePairData<String, HoodieRecordGlobalLocation> lookupRecords(HoodieData<HoodieRecord<R>> records, HoodieEngineContext context,
+                                                                                 HoodieTable hoodieTable, Either<Integer, Map<String, Integer>> fileGroupSize) {
+    Map<String, Integer> fileGroupCountPerDataPartition = fileGroupSize.asRight();
+    int numFileGroups = getTotalFileGroupCount(fileGroupSize);
+    Map<String, Integer> partitionOffsetIndexes = BucketizedMetadataTableFileGroupIndexParser.generatePartitionToBaseIndexOffsets(fileGroupCountPerDataPartition);
+
+    // Partition the record keys to lookup such that each partition looks up one record index shard
+    JavaRDD<Pair<String, String>> partitionedKeyRDD = HoodieJavaRDD.getJavaRDD(records)
+        .filter(record -> partitionOffsetIndexes.containsKey(record.getPartitionPath()))
+        .map(record -> Pair.of(record.getPartitionPath(), record.getRecordKey()))
+        // get offset from partitionOffsetIndexes then add the hash of the key
+        .keyBy(k -> partitionOffsetIndexes.get(k.getLeft()) + HoodieTableMetadataUtil.mapRecordKeyToFileGroupIndex(k.getRight(), fileGroupCountPerDataPartition.get(k.getLeft())))
+        .partitionBy(new PartitionIdPassthrough(numFileGroups))
+        .map(t -> t._2);
+    ValidationUtils.checkState(partitionedKeyRDD.getNumPartitions() <= numFileGroups);
+    // Lookup the keys in the record index
+    return HoodieJavaPairRDD.of(partitionedKeyRDD.mapPartitionsToPair(new PartitionedRecordIndexFileGroupLookupFunction(hoodieTable)));
+  }
+
+  @Override
+  protected Either<Integer, Map<String, Integer>> fetchFileGroupSize(HoodieTable hoodieTable) {
+    Map<String, Integer> partitionSizes = new HashMap<>();
+    Map<String, List<FileSlice>> fileGroups = hoodieTable.getMetadataTable().getBucketizedFileGroupsForPartitionedRLI(MetadataPartitionType.RECORD_INDEX);
+    fileGroups.keySet().forEach(k -> partitionSizes.put(k, fileGroups.get(k).size()));
+    return Either.right(partitionSizes);
+  }
+
+  @Override
+  protected int getTotalFileGroupCount(Either<Integer, Map<String, Integer>> fileGroupSize) {
+    return BucketizedMetadataTableFileGroupIndexParser.calculateNumberOfFileGroups(fileGroupSize.asRight());
+  }
+
+  @Override
+  protected boolean shouldUpdatePartitionPath(HoodieTable hoodieTable) {
+    return false;
+  }
+
+  /**
+   * Function that lookups a list of keys in a single shard of the record index
+   */
+  private static class PartitionedRecordIndexFileGroupLookupFunction implements PairFlatMapFunction<Iterator<Pair<String,String>>, String, HoodieRecordGlobalLocation> {
+    private final HoodieTable hoodieTable;
+
+    public PartitionedRecordIndexFileGroupLookupFunction(HoodieTable hoodieTable) {
+      this.hoodieTable = hoodieTable;
+    }
+
+    @Override
+    public Iterator<Tuple2<String, HoodieRecordGlobalLocation>> call(Iterator<Pair<String, String>> partitionPathRecordKeyIterator) throws Exception {
+      //Needs to be final, so we must use 1 element array to store the value
+      final String[] partitionName = {null};
+      List<String> keysToLookup = new ArrayList<>();
+      partitionPathRecordKeyIterator.forEachRemaining(p -> {
+        keysToLookup.add(p.getRight());
+        if (partitionName[0] == null) {
+          partitionName[0] = p.getLeft();
+        }
+      });
+
+      // recordIndexInfo object only contains records that are present in record_index.
+      assert partitionName[0] != null || keysToLookup.isEmpty();
+      Map<String, HoodieRecordGlobalLocation> recordIndexInfo = HoodieDataUtils.dedupeAndCollectAsMap(
+          hoodieTable.getMetadataTable().readRecordIndex(HoodieListData.eager(keysToLookup), Option.ofNullable(partitionName[0])));

Review Comment:
   May I know why do we need to dedup? 
   



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPartitionedRecordLevelIndex.scala:
##########
@@ -0,0 +1,461 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.client.SparkRDDWriteClient
+import org.apache.hudi.client.common.HoodieSparkEngineContext
+import org.apache.hudi.common.config.{HoodieMetadataConfig, TypedProperties}
+import org.apache.hudi.common.data.HoodieListData
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.model.{HoodieRecordGlobalLocation, HoodieTableType}
+import org.apache.hudi.common.table.TableSchemaResolver
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator
+import org.apache.hudi.common.testutils.RawTripTestPayload.recordsToStrings
+import org.apache.hudi.common.util.{Option => HOption}
+import org.apache.hudi.config.{HoodieCompactionConfig, HoodieIndexConfig, HoodieWriteConfig}
+import org.apache.hudi.functional.TestPartitionedRecordLevelIndex.TestPartitionedRecordLevelIndexTestCase
+import org.apache.hudi.index.HoodieIndex.IndexType.PARTITIONED_RECORD_INDEX
+import org.apache.hudi.metadata.HoodieBackedTableMetadata
+import org.apache.hudi.storage.StoragePath
+import org.apache.hudi.table.action.compact.strategy.UnBoundedCompactionStrategy
+
+import org.apache.spark.sql.{Row, SaveMode}
+import org.apache.spark.sql.functions.lit
+import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertTrue, fail}
+import org.junit.jupiter.api.Tag
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource, ValueSource}
+
+import java.util.stream.Collectors
+
+import scala.collection.JavaConverters
+import scala.collection.JavaConverters._
+
+@Tag("functional")
+class TestPartitionedRecordLevelIndex extends RecordLevelIndexTestBase {
+  private class testPartitionedRecordLevelIndexHolder {
+    var bulkRecordKeys: java.util.List[String] = null
+    var options: Map[String, String] = null
+    var recordKeys: java.util.List[String] = null
+    var newRecordKeys: java.util.List[String] = null
+  }
+
+  def testPartitionedRecordLevelIndex(tableType: HoodieTableType, streamingWriteEnabled: Boolean, holder: testPartitionedRecordLevelIndexHolder): Unit = {
+    val dataGen = new HoodieTestDataGenerator();
+    val inserts = dataGen.generateInserts("001", 5)
+    val latestBatch = recordsToStrings(inserts).asScala.toSeq
+    val latestBatchDf = spark.read.json(spark.sparkContext.parallelize(latestBatch, 1))
+    val insertDf = latestBatchDf.withColumn("data_partition_path", lit("partition1")).union(latestBatchDf.withColumn("data_partition_path", lit("partition2")))
+    val options = Map(HoodieWriteConfig.TBL_NAME.key -> "hoodie_test",
+      DataSourceWriteOptions.TABLE_TYPE.key -> tableType.name(),
+      RECORDKEY_FIELD.key -> "_row_key",
+      PARTITIONPATH_FIELD.key -> "data_partition_path",
+      PRECOMBINE_FIELD.key -> "timestamp",
+      HoodieMetadataConfig.RECORD_INDEX_ENABLE_PROP.key()-> "false",
+      HoodieMetadataConfig.PARTITIONED_RECORD_INDEX_ENABLE_PROP.key() -> "true",
+      HoodieMetadataConfig.STREAMING_WRITE_ENABLED.key() -> streamingWriteEnabled.toString,
+      HoodieCompactionConfig.INLINE_COMPACT.key() -> "false",
+      HoodieIndexConfig.INDEX_TYPE.key() -> PARTITIONED_RECORD_INDEX.name())
+    holder.options = options
+    insertDf.write.format("org.apache.hudi")
+      .options(options)
+      .mode(SaveMode.Overwrite)
+      .save(basePath)
+    assertEquals(10, spark.read.format("hudi").load(basePath).count())
+    val props = TypedProperties.fromMap(JavaConverters.mapAsJavaMapConverter(options).asJava)
+    val writeConfig = HoodieWriteConfig.newBuilder()
+      .withProps(props)
+      .withPath(basePath)
+      .build()
+    var metadata = metadataWriter(writeConfig).getTableMetadata
+    val recordKeys = inserts.asScala.map(i => i.getRecordKey).asJava.stream().collect(Collectors.toList())
+    holder.recordKeys = recordKeys
+    var partition1Locations = readRecordIndex(metadata, recordKeys, HOption.of("partition1"))
+    assertEquals(5, partition1Locations.size)
+    var partition2Locations = readRecordIndex(metadata, recordKeys, HOption.of("partition2"))
+    assertEquals(5, partition2Locations.size)
+    var df = spark.read.format("hudi").load(basePath).collect()
+    validateDFWithLocations(df, partition1Locations, "partition1")
+    validateDFWithLocations(df, partition2Locations, "partition2")
+
+    val newDeletes =  dataGen.generateUpdates("004", 1)
+    val updates =  dataGen.generateUniqueUpdates("002", 3)
+    val nextBatch = recordsToStrings(updates).asScala.toSeq
+    val nextBatchDf = spark.read.json(spark.sparkContext.parallelize(nextBatch, 1))
+    val updateDf = nextBatchDf.withColumn("data_partition_path", lit("partition1"))
+
+    updateDf.write.format("org.apache.hudi")
+      .options(options)
+      .option(DataSourceWriteOptions.OPERATION.key(), UPSERT_OPERATION_OPT_VAL)
+      .mode(SaveMode.Append)
+      .save(basePath)
+    assertEquals(10, spark.read.format("hudi").load(basePath).count())
+    partition1Locations = readRecordIndex(metadata, recordKeys, HOption.of("partition1"))
+    assertEquals(5, partition1Locations.size)
+    partition2Locations = readRecordIndex(metadata, recordKeys, HOption.of("partition2"))
+    assertEquals(5, partition2Locations.size)
+    df = spark.read.format("hudi").load(basePath).collect()
+    validateDFWithLocations(df, partition1Locations, "partition1")
+    validateDFWithLocations(df, partition2Locations, "partition2")
+
+    val newInserts =  dataGen.generateInserts("003", 3)
+    val newInsertBatch = recordsToStrings(newInserts).asScala.toSeq
+    val newInsertBatchDf = spark.read.json(spark.sparkContext.parallelize(newInsertBatch, 1))
+    val newInsertDf = newInsertBatchDf.withColumn("data_partition_path", lit("partition2")).union(newInsertBatchDf.withColumn("data_partition_path", lit("partition3")))
+    newInsertDf.write.format("org.apache.hudi")
+      .options(options)
+      .option(DataSourceWriteOptions.OPERATION.key(), UPSERT_OPERATION_OPT_VAL)
+      .mode(SaveMode.Append)
+      .save(basePath)
+    assertEquals(16, spark.read.format("hudi").load(basePath).count())
+    metadata = metadataWriter(writeConfig).getTableMetadata
+    partition1Locations = readRecordIndex(metadata, recordKeys, HOption.of("partition1"))
+    assertEquals(5, partition1Locations.size)
+    partition2Locations = readRecordIndex(metadata, recordKeys, HOption.of("partition2"))
+    assertEquals(5, partition2Locations.size)
+    df = spark.read.format("hudi").load(basePath).collect()
+    validateDFWithLocations(df, partition1Locations, "partition1")
+    validateDFWithLocations(df, partition2Locations, "partition2")
+
+    val newRecordKeys = newInserts.asScala.map(i => i.getRecordKey).asJava.stream().collect(Collectors.toList())
+    holder.newRecordKeys = newRecordKeys
+    partition1Locations = readRecordIndex(metadata, newRecordKeys, HOption.of("partition1"))
+    assertEquals(0, partition1Locations.size)
+    partition2Locations = readRecordIndex(metadata, newRecordKeys, HOption.of("partition2"))
+    assertEquals(3, partition2Locations.size)
+    var partition3Locations = readRecordIndex(metadata, newRecordKeys, HOption.of("partition3"))
+    assertEquals(3, partition3Locations.size)
+    validateDFWithLocations(df, partition3Locations, "partition3")
+
+    val newDeletesBatch = recordsToStrings(newDeletes).asScala.toSeq
+    val newDeletesBatchDf = spark.read.json(spark.sparkContext.parallelize(newDeletesBatch, 1))
+    val newDeletesDf = newDeletesBatchDf.withColumn("data_partition_path", lit("partition1"))
+    newDeletesDf.write.format("org.apache.hudi")
+      .options(options)
+      .option(DataSourceWriteOptions.OPERATION.key(), DELETE_OPERATION_OPT_VAL)
+      .mode(SaveMode.Append)
+      .save(basePath)
+    assertEquals(15, spark.read.format("hudi").load(basePath).count())
+    metadata = metadataWriter(writeConfig).getTableMetadata
+    partition1Locations = readRecordIndex(metadata, recordKeys, HOption.of("partition1"))
+    assertEquals(4, partition1Locations.size)
+    partition2Locations = readRecordIndex(metadata, recordKeys, HOption.of("partition2"))
+    assertEquals(5, partition2Locations.size)
+    df = spark.read.format("hudi").load(basePath).collect()
+    validateDFWithLocations(df, partition1Locations, "partition1")
+    validateDFWithLocations(df, partition2Locations, "partition2")
+
+    assertFalse(partition1Locations.contains(newDeletes.get(0).getRecordKey))
+    assertTrue(partition2Locations.contains(newDeletes.get(0).getRecordKey))
+    partition1Locations = readRecordIndex(metadata, newRecordKeys, HOption.of("partition1"))
+    assertEquals(0, partition1Locations.size)
+    partition2Locations = readRecordIndex(metadata, newRecordKeys, HOption.of("partition2"))
+    assertEquals(3, partition2Locations.size)
+    partition3Locations = readRecordIndex(metadata, newRecordKeys, HOption.of("partition3"))
+    assertEquals(3, partition3Locations.size)
+    validateDFWithLocations(df, partition2Locations, "partition2")
+    validateDFWithLocations(df, partition3Locations, "partition3")
+
+    val bulkInserts = dataGen.generateInserts("005", 5)
+    val bulkInsertBatch = recordsToStrings(bulkInserts).asScala.toSeq
+    val bulkInsertDf = spark.read.json(spark.sparkContext.parallelize(bulkInsertBatch, 1))
+    val bulkInsertPartitionedDf = bulkInsertDf.withColumn("data_partition_path", lit("partition0"))
+
+    // Use bulk_insert operation explicitly
+    bulkInsertPartitionedDf.write.format("org.apache.hudi")
+      .options(options)
+      .option(DataSourceWriteOptions.OPERATION.key(), DataSourceWriteOptions.BULK_INSERT_OPERATION_OPT_VAL)
+      .mode(SaveMode.Append)
+      .save(basePath)
+
+    val bulkRecordKeys = bulkInserts.asScala.map(_.getRecordKey).asJava
+    holder.bulkRecordKeys = bulkRecordKeys
+    metadata = metadataWriter(writeConfig).getTableMetadata
+    val partition0Locations = readRecordIndex(metadata, bulkRecordKeys, HOption.of("partition0"))
+    assertEquals(5, partition0Locations.size)
+    df = spark.read.format("hudi").load(basePath).collect()
+    validateDFWithLocations(df, partition0Locations, "partition0")
+  }
+
+  @ParameterizedTest
+  @MethodSource(Array("testArgsForPartitionedRecordLevelIndex"))
+  def testPartitionedRecordLevelIndexRollback(testCase: TestPartitionedRecordLevelIndexTestCase): Unit = {
+    val holder = new testPartitionedRecordLevelIndexHolder
+    testPartitionedRecordLevelIndex(testCase.tableType, testCase.streamingWriteEnabled, holder)
+    val writeConfig = getWriteConfig(holder.options)
+    new SparkRDDWriteClient(new HoodieSparkEngineContext(jsc), writeConfig)
+      .rollback(metaClient.getActiveTimeline.lastInstant().get().requestedTime())
+    val metadata = metadataWriter(writeConfig).getTableMetadata
+    try {
+      val partition0Locations = readRecordIndex(metadata, holder.bulkRecordKeys, HOption.of("partition0"))
+      fail("rollback happened, so partition should be deleted")
+    } catch {
+      case t: Throwable => assertTrue(t.isInstanceOf[ArithmeticException])
+    }
+  }
+
+  @ParameterizedTest
+  @ValueSource(booleans = Array(true, false))
+  def testPartitionedRecordLevelIndexCompact(streamingWriteEnabled: Boolean): Unit = {
+    val holder = new testPartitionedRecordLevelIndexHolder
+    testPartitionedRecordLevelIndex(HoodieTableType.MERGE_ON_READ, streamingWriteEnabled, holder)
+    assertEquals("deltacommit", metaClient.getActiveTimeline.lastInstant().get().getAction)
+    val writeConfig = getWriteConfig(holder.options)
+    var metadata = metadataWriter(writeConfig).getTableMetadata
+    doAllAssertions(holder, metadata)
+    val writeClient = new SparkRDDWriteClient(new HoodieSparkEngineContext(jsc), writeConfig)
+    val timeOpt = writeClient.scheduleCompaction(HOption.empty())
+    assertTrue(timeOpt.isPresent)
+    writeClient.compact(timeOpt.get())
+    metaClient.reloadActiveTimeline()
+    assertEquals("compaction", metaClient.getActiveTimeline.lastInstant().get().getAction)
+    metadata = metadataWriter(writeConfig).getTableMetadata
+    doAllAssertions(holder, metadata)
+  }
+
+  @ParameterizedTest
+  @MethodSource(Array("testArgsForPartitionedRecordLevelIndex"))
+  def testPartitionedRecordLevelIndexCluster(testCase: TestPartitionedRecordLevelIndexTestCase): Unit = {
+    val holder = new testPartitionedRecordLevelIndexHolder
+    testPartitionedRecordLevelIndex(testCase.tableType, testCase.streamingWriteEnabled, holder)
+    assertEquals(if (testCase.tableType.equals(HoodieTableType.MERGE_ON_READ)) "deltacommit" else "commit",
+      metaClient.getActiveTimeline.lastInstant().get().getAction)
+    val writeConfig = getWriteConfig(holder.options ++ Map(HoodieWriteConfig.AVRO_SCHEMA_STRING.key() -> HoodieTestDataGenerator.AVRO_SCHEMA.toString))
+    var metadata = metadataWriter(writeConfig).getTableMetadata
+    doAllAssertions(holder, metadata)
+    val writeClient = new SparkRDDWriteClient(new HoodieSparkEngineContext(jsc), writeConfig)
+    val timeOpt = writeClient.scheduleClustering(HOption.empty())
+    assertTrue(timeOpt.isPresent)
+    writeClient.cluster(timeOpt.get())
+    metaClient.reloadActiveTimeline()
+    assertEquals("replacecommit", metaClient.getActiveTimeline.lastInstant().get().getAction)
+    metadata = metadataWriter(writeConfig).getTableMetadata
+    doAllAssertions(holder, metadata)
+  }
+
+  private def validateDFWithLocations(df: Array[Row], locations: Map[String, HoodieRecordGlobalLocation],
+                                       partition: String): Unit = {
+    var count: Int = 0
+    for (row <- df) {
+      val recordKey = row.getString(2)
+      locations.get(recordKey).foreach { loc =>
+        if (partition == row.getString(3)) {
+          count += 1
+          assertEquals(row.getString(3), loc.getPartitionPath)
+          assertEquals(FSUtils.getFileId(row.getString(4)), loc.getFileId)
+        }
+      }
+    }
+    assertEquals(locations.size, count)
+  }
+
+  private def doAllAssertions(holder: testPartitionedRecordLevelIndexHolder, metadata: HoodieBackedTableMetadata): Unit = {
+    val df = spark.read.format("hudi").load(basePath).collect()
+    var partition0Locations = readRecordIndex(metadata, holder.recordKeys, HOption.of("partition0"))
+    assertEquals(0, partition0Locations.size)
+    var partition1Locations = readRecordIndex(metadata, holder.recordKeys, HOption.of("partition1"))
+    assertEquals(4, partition1Locations.size)
+    validateDFWithLocations(df, partition1Locations, "partition1")
+    var partition2Locations = readRecordIndex(metadata, holder.recordKeys, HOption.of("partition2"))
+    assertEquals(5, partition2Locations.size)
+    validateDFWithLocations(df, partition2Locations, "partition2")
+    var partition3Locations = readRecordIndex(metadata, holder.recordKeys, HOption.of("partition3"))
+    assertEquals(0, partition3Locations.size)
+
+    partition0Locations = readRecordIndex(metadata, holder.newRecordKeys, HOption.of("partition0"))
+    assertEquals(0, partition0Locations.size)
+    partition1Locations = readRecordIndex(metadata, holder.newRecordKeys, HOption.of("partition1"))
+    assertEquals(0, partition1Locations.size)
+    partition2Locations = readRecordIndex(metadata, holder.newRecordKeys, HOption.of("partition2"))
+    assertEquals(3, partition2Locations.size)
+    validateDFWithLocations(df, partition2Locations, "partition2")
+    partition3Locations = readRecordIndex(metadata, holder.newRecordKeys, HOption.of("partition3"))
+    assertEquals(3, partition3Locations.size)
+    validateDFWithLocations(df, partition3Locations, "partition3")
+
+    partition0Locations = readRecordIndex(metadata, holder.bulkRecordKeys, HOption.of("partition0"))
+    assertEquals(5, partition0Locations.size)
+    validateDFWithLocations(df, partition0Locations, "partition0")
+    partition1Locations = readRecordIndex(metadata, holder.bulkRecordKeys, HOption.of("partition1"))
+    assertEquals(0, partition1Locations.size)
+    partition2Locations = readRecordIndex(metadata, holder.bulkRecordKeys, HOption.of("partition2"))
+    assertEquals(0, partition2Locations.size)
+    partition3Locations = readRecordIndex(metadata, holder.bulkRecordKeys, HOption.of("partition3"))
+    assertEquals(0, partition3Locations.size)
+  }
+
+  @ParameterizedTest
+  @MethodSource(Array("testArgsForPartitionedRecordLevelIndex"))
+  def testPartitionedRecordLevelIndexInitializationBasic(testCase: TestPartitionedRecordLevelIndexTestCase): Unit = {
+    testPartitionedRecordLevelIndexInitialization(testCase.tableType, testCase.streamingWriteEnabled, failAndDoRollback = false, compact = false, cluster = false)
+  }
+
+  @ParameterizedTest
+  @MethodSource(Array("testArgsForPartitionedRecordLevelIndex"))
+  def testPartitionedRecordLevelIndexInitializationRollback(testCase: TestPartitionedRecordLevelIndexTestCase): Unit = {
+    testPartitionedRecordLevelIndexInitialization(testCase.tableType, testCase.streamingWriteEnabled, failAndDoRollback = true, compact = false, cluster = false)
+  }
+
+  @ParameterizedTest
+  @ValueSource(booleans = Array(true, false))
+  def testPartitionedRecordLevelIndexInitializationCompact(streamingWriteEnabled: Boolean): Unit = {
+    testPartitionedRecordLevelIndexInitialization(HoodieTableType.MERGE_ON_READ, streamingWriteEnabled, failAndDoRollback = false, compact = true, cluster = false)
+  }
+
+  @ParameterizedTest
+  @MethodSource(Array("testArgsForPartitionedRecordLevelIndex"))
+  def testPartitionedRecordLevelIndexInitializationCluster(testCase: TestPartitionedRecordLevelIndexTestCase): Unit = {
+    testPartitionedRecordLevelIndexInitialization(testCase.tableType, testCase.streamingWriteEnabled, failAndDoRollback = false, compact = false, cluster = true)
+  }
+
+  def testPartitionedRecordLevelIndexInitialization(tableType: HoodieTableType,
+                                                    streamingWriteEnabled: Boolean,
+                                                    failAndDoRollback: Boolean,
+                                                    compact: Boolean,
+                                                    cluster: Boolean): Unit = {
+    initMetaClient(tableType)
+    val dataGen = new HoodieTestDataGenerator()
+    val inserts = dataGen.generateInserts("001", 5)
+    val latestBatch = recordsToStrings(inserts).asScala.toSeq
+    val latestBatchDf = spark.read.json(spark.sparkContext.parallelize(latestBatch, 1))
+    val insertDf = latestBatchDf.withColumn("data_partition_path", lit("partition1")).union(latestBatchDf.withColumn("data_partition_path", lit("partition2")))
+    val options = Map(HoodieWriteConfig.TBL_NAME.key -> "hoodie_test",
+      DataSourceWriteOptions.TABLE_TYPE.key -> tableType.name(),
+      RECORDKEY_FIELD.key -> "_row_key",
+      PARTITIONPATH_FIELD.key -> "data_partition_path",
+      PRECOMBINE_FIELD.key -> "timestamp",
+      HoodieMetadataConfig.RECORD_INDEX_ENABLE_PROP.key()-> "false",
+      HoodieMetadataConfig.SECONDARY_INDEX_ENABLE_PROP.key() -> "false",
+      HoodieMetadataConfig.STREAMING_WRITE_ENABLED.key() -> streamingWriteEnabled.toString,
+      HoodieCompactionConfig.INLINE_COMPACT.key() -> "false",
+      HoodieIndexConfig.INDEX_TYPE.key() -> PARTITIONED_RECORD_INDEX.name())
+    insertDf.write.format("org.apache.hudi")
+      .options(options)
+      .mode(SaveMode.Overwrite)
+      .save(basePath)
+
+    assertEquals(10, spark.read.format("hudi").load(basePath).count())
+
+    val updates =  dataGen.generateUniqueUpdates("002", 3)
+    val nextBatch = recordsToStrings(updates).asScala.toSeq
+    val nextBatchDf = spark.read.json(spark.sparkContext.parallelize(nextBatch, 1))
+    val updateDf = nextBatchDf.withColumn("data_partition_path", lit("partition1"))
+    updateDf.write.format("org.apache.hudi")
+      .options(options)
+      .option(DataSourceWriteOptions.OPERATION.key(), UPSERT_OPERATION_OPT_VAL)
+      .mode(SaveMode.Append)
+      .save(basePath)
+    assertEquals(10, spark.read.format("hudi").load(basePath).count())
+    metaClient.reloadActiveTimeline()
+    val tableSchemaResolver = new TableSchemaResolver(metaClient)
+    val latestTableSchemaFromCommitMetadata = tableSchemaResolver.getTableAvroSchemaFromLatestCommit(false)
+
+    if (failAndDoRollback) {
+      val updatesToFail =  dataGen.generateUniqueUpdates("003", 3)
+      val batchToFail = recordsToStrings(updatesToFail).asScala.toSeq
+      val batchToFailDf = spark.read.json(spark.sparkContext.parallelize(batchToFail, 1))
+      val failDf = batchToFailDf.withColumn("data_partition_path", lit("partition1")).union(batchToFailDf.withColumn("data_partition_path", lit("partition3")))
+      failDf.write.format("org.apache.hudi")
+        .options(options)
+        .option(DataSourceWriteOptions.OPERATION.key(), UPSERT_OPERATION_OPT_VAL)
+        .mode(SaveMode.Append)
+        .save(basePath)
+      assertEquals(13, spark.read.format("hudi").load(basePath).count())
+
+      metaClient.reloadActiveTimeline()
+      val lastInstant = metaClient.getActiveTimeline.lastInstant().get()
+      assertTrue(storage.deleteFile(new StoragePath(metaClient.getTimelinePath, metaClient.getInstantFileNameGenerator.getFileName(lastInstant))))
+      assertEquals(10, spark.read.format("hudi").load(basePath).count())
+
+      // rollback
+      val writeConfig = HoodieWriteConfig.newBuilder()
+        .withProps(TypedProperties.fromMap(JavaConverters
+          .mapAsJavaMapConverter(options ++ Map(HoodieWriteConfig.AVRO_SCHEMA_STRING.key() -> latestTableSchemaFromCommitMetadata.get().toString)).asJava))
+        .withPath(basePath)
+        .build()
+      new SparkRDDWriteClient(new HoodieSparkEngineContext(jsc), writeConfig)
+        .rollback(lastInstant.requestedTime())
+    }
+
+    if (compact) {
+      assertEquals("deltacommit", metaClient.getActiveTimeline.lastInstant().get().getAction)
+      val writeConfig = getWriteConfig(options ++
+        Map(HoodieCompactionConfig.COMPACTION_STRATEGY.key() -> classOf[UnBoundedCompactionStrategy].getName,
+          HoodieCompactionConfig.INLINE_COMPACT_NUM_DELTA_COMMITS.key() -> "1"))
+      val writeClient = new SparkRDDWriteClient(new HoodieSparkEngineContext(jsc), writeConfig)
+      val timeOpt = writeClient.scheduleCompaction(HOption.empty())
+      assertTrue(timeOpt.isPresent)
+      writeClient.compact(timeOpt.get())
+      metaClient.reloadActiveTimeline()
+      assertEquals("compaction", metaClient.getActiveTimeline.lastInstant().get().getAction)
+    }
+
+    if (cluster) {
+      assertEquals(if (tableType.equals(HoodieTableType.MERGE_ON_READ)) "deltacommit" else "commit",
+        metaClient.getActiveTimeline.lastInstant().get().getAction)
+      val writeConfig = getWriteConfig(options ++ Map(HoodieWriteConfig.AVRO_SCHEMA_STRING.key() -> HoodieTestDataGenerator.AVRO_SCHEMA.toString))
+      val writeClient = new SparkRDDWriteClient(new HoodieSparkEngineContext(jsc), writeConfig)
+      val timeOpt = writeClient.scheduleClustering(HOption.empty())
+      assertTrue(timeOpt.isPresent)
+      writeClient.cluster(timeOpt.get())
+      metaClient.reloadActiveTimeline()
+      assertEquals("replacecommit", metaClient.getActiveTimeline.lastInstant().get().getAction)
+    }
+
+    //init mdt
+    val updateOptions = options ++ Map(HoodieMetadataConfig.PARTITIONED_RECORD_INDEX_ENABLE_PROP.key() -> "true",
+      HoodieWriteConfig.AVRO_SCHEMA_STRING.key() -> latestTableSchemaFromCommitMetadata.get().toString)
+    val props = TypedProperties.fromMap(JavaConverters.mapAsJavaMapConverter(updateOptions).asJava)
+    val writeConfig = HoodieWriteConfig.newBuilder()
+      .withProps(props)
+      .withPath(basePath)
+      .build()
+    val metadata = metadataWriter(writeConfig).getTableMetadata
+    val recordKeys = inserts.asScala.map(i => i.getRecordKey).asJava.stream().collect(Collectors.toList())
+    val partition1Locations = readRecordIndex(metadata, recordKeys, HOption.of("partition1"))
+    assertEquals(5, partition1Locations.size)
+    val partition2Locations = readRecordIndex(metadata, recordKeys, HOption.of("partition2"))
+    assertEquals(5, partition2Locations.size)
+    val df = spark.read.format("hudi").load(basePath).collect()
+    validateDFWithLocations(df, partition1Locations, "partition1")
+    validateDFWithLocations(df, partition2Locations, "partition2")
+  }
+
+  def readRecordIndex(metadata: HoodieBackedTableMetadata, recordKeys: java.util.List[String], dataTablePartition: HOption[String]): Map[String, HoodieRecordGlobalLocation] = {
+    metadata.readRecordIndex(HoodieListData.eager(recordKeys), dataTablePartition)
+      .collectAsList().asScala.map(p => p.getKey -> p.getValue).toMap
+  }
+}
+
+object TestPartitionedRecordLevelIndex {
+
+  case class TestPartitionedRecordLevelIndexTestCase(tableType: HoodieTableType, streamingWriteEnabled: Boolean)
+
+  def testArgsForPartitionedRecordLevelIndex: java.util.stream.Stream[Arguments] = {
+    java.util.stream.Stream.of(
+      Arguments.arguments(TestPartitionedRecordLevelIndexTestCase(HoodieTableType.COPY_ON_WRITE, streamingWriteEnabled = true)),
+      Arguments.arguments(TestPartitionedRecordLevelIndexTestCase(HoodieTableType.COPY_ON_WRITE, streamingWriteEnabled = false)),
+      Arguments.arguments(TestPartitionedRecordLevelIndexTestCase(HoodieTableType.MERGE_ON_READ, streamingWriteEnabled = true)),
+      Arguments.arguments(TestPartitionedRecordLevelIndexTestCase(HoodieTableType.MERGE_ON_READ, streamingWriteEnabled = false))
+    )

Review Comment:
   I see we end up writing lot of end to end functional tests. 
   
   Can you take a look test classes in here 
   https://github.com/apache/hudi/tree/93e3df5af813290d0b60beb780b2791c40fb80a4/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io 
   
   Here we try to directly use write handles to write and then use metadata writer to commit to mdt. 
   
   If we can use something similar, might save us test run time and we could test more combinations 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3138981838

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     }, {
       "hash" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378",
       "triggerID" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7386",
       "triggerID" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7396",
       "triggerID" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 028e2a1fe0be05d3a045bad459c80d29ef16939b UNKNOWN
   * b9c2250aacb6e071b27aa69e172e2eab3b8b517b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7386) 
   * d855f5c0c907217a2ea6a87fc59167fbbf7ae236 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7396) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3134165041

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 204a05a68c5875a42a77e9f24252eec07f45e42e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134) 
   * 5cd49ce761480de054e3ce0516cec7f9af06f2ed Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3110112788

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 05841e04b843d75afe5c48d9e288991787be1fb0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3138217727

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     }, {
       "hash" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 5cd49ce761480de054e3ce0516cec7f9af06f2ed Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321) 
   * 028e2a1fe0be05d3a045bad459c80d29ef16939b UNKNOWN
   * 539a101dd2109c191d673ddde1761d95eedcfa99 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3138207771

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     }, {
       "hash" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 5cd49ce761480de054e3ce0516cec7f9af06f2ed Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321) 
   * 028e2a1fe0be05d3a045bad459c80d29ef16939b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2240957433


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1291,10 +1364,52 @@ protected Pair<List<HoodieFileGroupId>, HoodieData<HoodieRecord>> tagRecordsWith
   @Override
   public void update(HoodieCommitMetadata commitMetadata, String instantTime) {
     mayBeReinitMetadataReader();
+    maybeInitializeNewFileGroupsForPartitionedRLI(commitMetadata, instantTime);
     processAndCommit(instantTime, new BatchMetadataConversionFunction(instantTime, commitMetadata, getMetadataPartitionsToUpdate()));
     closeInternal();
   }
 
+  /**
+   * This method is used to initialize new file groups for partitioned record index during the in-flight commit.
+   * It will initialize new file groups for partitions that are newly added in the inflight commit.
+   *
+   * @param commitMetadata metadata for the inflight commit
+   * @param instantTime    Timestamp for the mdt commit
+   */
+  private void maybeInitializeNewFileGroupsForPartitionedRLI(HoodieCommitMetadata commitMetadata, String instantTime) {

Review Comment:
   we can't



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2232301605


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/SparkMetadataTablePartitionedRecordIndex.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.data.HoodieListData;
+import org.apache.hudi.common.data.HoodiePairData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordGlobalLocation;
+import org.apache.hudi.common.util.Either;
+import org.apache.hudi.common.util.HoodieDataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaPairRDD;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.metadata.HoodieTableMetadataUtil;
+import org.apache.hudi.metadata.MetadataPartitionType;
+import org.apache.hudi.metadata.BucketizedMetadataTableFileGroupIndexParser;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.function.PairFlatMapFunction;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import scala.Tuple2;
+
+/**
+ * Index to be used with partitioned RLI. Queries the record index for tables with non-global record keys
+ */
+public class SparkMetadataTablePartitionedRecordIndex extends SparkMetadataTableRecordIndex {
+
+  public SparkMetadataTablePartitionedRecordIndex(HoodieWriteConfig config) {
+    super(config);
+  }
+
+  @Override
+  public boolean isGlobal() {
+    return false;
+  }
+
+  @Override
+  protected HoodieIndex.IndexType getFallbackIndexType() {
+    return IndexType.SIMPLE;
+  }
+
+  @Override
+  protected <R> HoodiePairData<String, HoodieRecordGlobalLocation> lookupRecords(HoodieData<HoodieRecord<R>> records, HoodieEngineContext context,
+                                                                                 HoodieTable hoodieTable, Either<Integer, Map<String, Integer>> fileGroupSize) {
+    Map<String, Integer> fileGroupCountPerDataPartition = fileGroupSize.asRight();
+    int numFileGroups = getTotalFileGroupCount(fileGroupSize);
+    Map<String, Integer> partitionOffsetIndexes = BucketizedMetadataTableFileGroupIndexParser.generatePartitionToBaseIndexOffsets(fileGroupCountPerDataPartition);
+
+    // Partition the record keys to lookup such that each partition looks up one record index shard
+    JavaRDD<Pair<String, String>> partitionedKeyRDD = HoodieJavaRDD.getJavaRDD(records)
+        .filter(record -> partitionOffsetIndexes.containsKey(record.getPartitionPath()))
+        .map(record -> Pair.of(record.getPartitionPath(), record.getRecordKey()))
+        // get offset from partitionOffsetIndexes then add the hash of the key
+        .keyBy(k -> partitionOffsetIndexes.get(k.getLeft()) + HoodieTableMetadataUtil.mapRecordKeyToFileGroupIndex(k.getRight(), fileGroupCountPerDataPartition.get(k.getLeft())))
+        .partitionBy(new PartitionIdPassthrough(numFileGroups))
+        .map(t -> t._2);
+    ValidationUtils.checkState(partitionedKeyRDD.getNumPartitions() <= numFileGroups);
+    // Lookup the keys in the record index
+    return HoodieJavaPairRDD.of(partitionedKeyRDD.mapPartitionsToPair(new PartitionedRecordIndexFileGroupLookupFunction(hoodieTable)));
+  }
+
+  @Override
+  protected Either<Integer, Map<String, Integer>> fetchFileGroupSize(HoodieTable hoodieTable) {
+    Map<String, Integer> partitionSizes = new HashMap<>();
+    Map<String, List<FileSlice>> fileGroups = hoodieTable.getMetadataTable().getBucketizedFileGroupsForPartitionedRLI(MetadataPartitionType.RECORD_INDEX);
+    fileGroups.keySet().forEach(k -> partitionSizes.put(k, fileGroups.get(k).size()));
+    return Either.right(partitionSizes);
+  }
+
+  @Override
+  protected int getTotalFileGroupCount(Either<Integer, Map<String, Integer>> fileGroupSize) {
+    return BucketizedMetadataTableFileGroupIndexParser.calculateNumberOfFileGroups(fileGroupSize.asRight());
+  }
+
+  @Override
+  protected boolean shouldUpdatePartitionPath(HoodieTable hoodieTable) {
+    return false;
+  }
+
+  /**
+   * Function that lookups a list of keys in a single shard of the record index
+   */
+  private static class PartitionedRecordIndexFileGroupLookupFunction implements PairFlatMapFunction<Iterator<Pair<String,String>>, String, HoodieRecordGlobalLocation> {
+    private final HoodieTable hoodieTable;
+
+    public PartitionedRecordIndexFileGroupLookupFunction(HoodieTable hoodieTable) {
+      this.hoodieTable = hoodieTable;
+    }
+
+    @Override
+    public Iterator<Tuple2<String, HoodieRecordGlobalLocation>> call(Iterator<Pair<String, String>> partitionPathRecordKeyIterator) throws Exception {
+      //Needs to be final, so we must use 1 element array to store the value
+      final String[] partitionName = {null};
+      List<String> keysToLookup = new ArrayList<>();
+      partitionPathRecordKeyIterator.forEachRemaining(p -> {
+        keysToLookup.add(p.getRight());
+        if (partitionName[0] == null) {
+          partitionName[0] = p.getLeft();
+        }
+      });
+
+      // recordIndexInfo object only contains records that are present in record_index.
+      assert partitionName[0] != null || keysToLookup.isEmpty();
+      Map<String, HoodieRecordGlobalLocation> recordIndexInfo = HoodieDataUtils.dedupeAndCollectAsMap(
+          hoodieTable.getMetadataTable().readRecordIndex(HoodieListData.eager(keysToLookup), Option.ofNullable(partitionName[0])));

Review Comment:
   good catch. will fix



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3110179714

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 05841e04b843d75afe5c48d9e288991787be1fb0 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016) 
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 6091ed96934c72ace34090dbc282aeb5620ae675 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3110900317

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 6091ed96934c72ace34090dbc282aeb5620ae675 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3111598444

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 6091ed96934c72ace34090dbc282aeb5620ae675 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020) 
   * 7344e77705233f8271c2024830e6389d92419a06 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2244060993


##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1083,24 +1084,32 @@ public static Pair<Set<String>, Set<String>> computeRevivedAndDeletedKeys(Set<St
    * @return
    */
   @VisibleForTesting
-  public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord> recordIndexRecords, int parallelism) {
-    return recordIndexRecords.mapToPair(
-            (SerializablePairFunction<HoodieRecord, HoodieKey, HoodieRecord>) t -> Pair.of(t.getKey(), t))
-        .reduceByKey((SerializableBiFunction<HoodieRecord, HoodieRecord, HoodieRecord>) (record1, record2) -> {
-          boolean isRecord1Deleted = record1.getData() instanceof EmptyHoodieRecordPayload;
-          boolean isRecord2Deleted = record2.getData() instanceof EmptyHoodieRecordPayload;
-          if (isRecord1Deleted && !isRecord2Deleted) {
-            return record2;
-          } else if (!isRecord1Deleted && isRecord2Deleted) {
-            return record1;
-          } else if (isRecord1Deleted && isRecord2Deleted) {
-            // let's delete just 1 of them
-            return record1;
-          } else {
-            throw new HoodieIOException("Two HoodieRecord updates to RLI is seen for same record key " + record2.getRecordKey() + ", record 1 : "
-                + record1.getData().toString() + ", record 2 : " + record2.getData().toString());
-          }
-        }, parallelism).values();
+  public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord> recordIndexRecords, int parallelism, boolean isPartitionedRLI) {
+    HoodiePairData<HoodieKey, HoodieRecord> recordIndexRecordsPair;
+    if (isPartitionedRLI) {
+      recordIndexRecordsPair = recordIndexRecords.mapToPair(r -> {
+        String recordPartitionPath = r.getData() instanceof EmptyHoodieRecordPayloadWithPartition

Review Comment:
   Got it, the hashing is customized and the partition path seems useless in the hasing code computation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234839732


##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1083,24 +1084,32 @@ public static Pair<Set<String>, Set<String>> computeRevivedAndDeletedKeys(Set<St
    * @return
    */
   @VisibleForTesting
-  public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord> recordIndexRecords, int parallelism) {
-    return recordIndexRecords.mapToPair(
-            (SerializablePairFunction<HoodieRecord, HoodieKey, HoodieRecord>) t -> Pair.of(t.getKey(), t))
-        .reduceByKey((SerializableBiFunction<HoodieRecord, HoodieRecord, HoodieRecord>) (record1, record2) -> {
-          boolean isRecord1Deleted = record1.getData() instanceof EmptyHoodieRecordPayload;
-          boolean isRecord2Deleted = record2.getData() instanceof EmptyHoodieRecordPayload;
-          if (isRecord1Deleted && !isRecord2Deleted) {
-            return record2;
-          } else if (!isRecord1Deleted && isRecord2Deleted) {
-            return record1;
-          } else if (isRecord1Deleted && isRecord2Deleted) {
-            // let's delete just 1 of them
-            return record1;
-          } else {
-            throw new HoodieIOException("Two HoodieRecord updates to RLI is seen for same record key " + record2.getRecordKey() + ", record 1 : "
-                + record1.getData().toString() + ", record 2 : " + record2.getData().toString());
-          }
-        }, parallelism).values();
+  public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord> recordIndexRecords, int parallelism, boolean isPartitionedRLI) {
+    HoodiePairData<HoodieKey, HoodieRecord> recordIndexRecordsPair;
+    if (isPartitionedRLI) {
+      recordIndexRecordsPair = recordIndexRecords.mapToPair(r -> {
+        String recordPartitionPath = r.getData() instanceof EmptyHoodieRecordPayloadWithPartition
+            ? ((EmptyHoodieRecordPayloadWithPartition) r.getData()).getPartitionPath() : ((HoodieMetadataPayload) r.getData()).getRecordGlobalLocation().getPartitionPath();

Review Comment:
   `getRecordGlobalLocation().getPartitionPath()` is costly because the instant time and fileId parsing are all there, can we just fetch through `recordIndexMetadata..getPartitionName()`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3138339346

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     }, {
       "hash" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378",
       "triggerID" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 028e2a1fe0be05d3a045bad459c80d29ef16939b UNKNOWN
   * 539a101dd2109c191d673ddde1761d95eedcfa99 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2242867516


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1291,10 +1364,52 @@ protected Pair<List<HoodieFileGroupId>, HoodieData<HoodieRecord>> tagRecordsWith
   @Override
   public void update(HoodieCommitMetadata commitMetadata, String instantTime) {
     mayBeReinitMetadataReader();
+    maybeInitializeNewFileGroupsForPartitionedRLI(commitMetadata, instantTime);
     processAndCommit(instantTime, new BatchMetadataConversionFunction(instantTime, commitMetadata, getMetadataPartitionsToUpdate()));
     closeInternal();
   }
 
+  /**
+   * This method is used to initialize new file groups for partitioned record index during the in-flight commit.
+   * It will initialize new file groups for partitions that are newly added in the inflight commit.
+   *
+   * @param commitMetadata metadata for the inflight commit
+   * @param instantTime    Timestamp for the mdt commit
+   */
+  private void maybeInitializeNewFileGroupsForPartitionedRLI(HoodieCommitMetadata commitMetadata, String instantTime) {

Review Comment:
   Because they call HoodieBackedTableMetadataWriter methods inside



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3138397690

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     }, {
       "hash" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378",
       "triggerID" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 028e2a1fe0be05d3a045bad459c80d29ef16939b UNKNOWN
   * 539a101dd2109c191d673ddde1761d95eedcfa99 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378) 
   * b9c2250aacb6e071b27aa69e172e2eab3b8b517b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3140318332

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d855f5c0c907217a2ea6a87fc59167fbbf7ae236 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234855395


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1689,40 +1804,75 @@ protected Pair<HoodieData<HoodieRecord>, List<HoodieFileGroupId>> tagRecordsWith
     try (HoodieTableFileSystemView fsView = HoodieTableMetadataUtil.getFileSystemViewForMetadataTable(metadataMetaClient)) {
       List<HoodieFileGroupId> hoodieFileGroupIdList = new ArrayList<>();
       for (Map.Entry<String, HoodieData<HoodieRecord>> entry : partitionRecordsMap.entrySet()) {
-        final String partitionName = entry.getKey();
+        final String partitionPath = entry.getKey();
         HoodieData<HoodieRecord> records = entry.getValue();
+        boolean isPartitionedRLI = Objects.equals(partitionPath, RECORD_INDEX.getPartitionPath()) && dataWriteConfig.isPartitionedRecordIndexEnabled();
         List<FileSlice> fileSlices =
-            HoodieTableMetadataUtil.getPartitionLatestFileSlices(metadataMetaClient, Option.ofNullable(fsView), partitionName);
-        if (fileSlices.isEmpty()) {
-          // scheduling or initialising of INDEX only initializes the file group and not add commit
-          // so if there are no committed file slices, look for inflight slices
-          ValidationUtils.checkState(isInitializing || dataMetaClient.getTableConfig().getMetadataPartitionsInflight().contains(partitionName),
-              String.format("Partition %s should be part of inflight metadata partitions here %s", partitionName, dataMetaClient.getTableConfig().getMetadataPartitionsInflight()));
-          fileSlices = getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient, Option.ofNullable(fsView), partitionName);
+            HoodieTableMetadataUtil.getPartitionLatestFileSlices(metadataMetaClient, Option.ofNullable(fsView), partitionPath);
+        // scheduling of INDEX only initializes the file group and not add commit
+        // so if there are no committed file slices, look for inflight slices
+        if (isPartitionedRLI || fileSlices.isEmpty()) {
+          // For isPartitionedRLI, new partitions added to the data table will cause new filegroups that are not yet commited
+          // therefore, we always need to look for inflight filegroups
+          ValidationUtils.checkState(isInitializing || dataMetaClient.getTableConfig().getMetadataPartitionsInflight().contains(partitionPath) || isPartitionedRLI,
+              String.format("Partition %s should be part of inflight metadata partitions here %s", partitionPath, dataMetaClient.getTableConfig().getMetadataPartitionsInflight()));
+          fileSlices = getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient, Option.ofNullable(fsView), partitionPath);
         }
-        final int fileGroupCount = fileSlices.size();
-        ValidationUtils.checkArgument(fileGroupCount > 0, String.format("FileGroup count for MDT partition %s should be > 0", partitionName));
-        hoodieFileGroupIdList.addAll(fileSlices.stream().map(fileSlice -> new HoodieFileGroupId(partitionName, fileSlice.getFileId())).collect(Collectors.toList()));
-
-        List<FileSlice> finalFileSlices = fileSlices;
-        HoodieIndexVersion indexVersion = existingIndexVersionOrDefault(partitionName, dataMetaClient);
-
-        // Determine key format once per partition to avoid repeated checks
-        SerializableBiFunction<String, Integer, Integer> mappingFunction = MetadataPartitionType.fromPartitionPath(partitionName).getFileGroupMappingFunction(indexVersion);
-        HoodieData<HoodieRecord> rddSinglePartitionRecords = records.map(r -> {
-          FileSlice slice = finalFileSlices.get(mappingFunction.apply(r.getRecordKey(), fileGroupCount));
-          r.unseal();
-          r.setCurrentLocation(new HoodieRecordLocation(slice.getBaseInstantTime(), slice.getFileId()));
-          r.seal();
-          return r;
-        });
-
-        allPartitionRecords = allPartitionRecords.union(rddSinglePartitionRecords);
+        hoodieFileGroupIdList.addAll(fileSlices.stream().map(fileSlice -> new HoodieFileGroupId(partitionPath, fileSlice.getFileId())).collect(Collectors.toList()));
+        SerializableFunction<HoodieRecord, HoodieRecord> recordTagger = getRecordTagger(partitionPath, fileSlices);
+        allPartitionRecords = allPartitionRecords.union(records.map(recordTagger));
       }
       return Pair.of(allPartitionRecords, hoodieFileGroupIdList);
     }
   }
 
+  private SerializableFunction<HoodieRecord, HoodieRecord> getRecordTagger(String partitionPath, List<FileSlice> fileSlices) {
+    HoodieIndexVersion indexVersion = existingIndexVersionOrDefault(partitionPath, dataMetaClient);
+    MetadataPartitionType partitionType = MetadataPartitionType.fromPartitionPath(partitionPath);
+    // Determine key format once per partition to avoid repeated checks
+    SerializableBiFunction<String, Integer, Integer> mappingFunction = partitionType.getFileGroupMappingFunction(indexVersion);
+    if (partitionType == RECORD_INDEX && dataWriteConfig.isPartitionedRecordIndexEnabled()) {
+      return getRecordTaggerPartitionedRLI(fileSlices, mappingFunction);
+    }
+    final int fileGroupCount = fileSlices.size();
+    ValidationUtils.checkArgument(fileGroupCount > 0, String.format("FileGroup count for MDT partition %s should be > 0", partitionPath));
+    return r -> {
+      FileSlice slice = fileSlices.get(mappingFunction.apply(r.getRecordKey(), fileGroupCount));
+      r.unseal();
+      r.setCurrentLocation(new HoodieRecordLocation(slice.getBaseInstantTime(), slice.getFileId()));
+      r.seal();
+      return r;
+    };
+  }
+
+  /**
+   * Add the location for each incoming record in the partitioned record level index. Hashing needs to be done within the data table partition and
+   * the number of filegroups can also vary by partition. Therefore, we need to lookup and group the filegroups by partition to understand the number
+   * of indexes we can hash into
+   *
+   * @param fileSlices latest fileslices in the mdt rli partition
+   * @return the records with the correct location set
+   */
+  private SerializableFunction<HoodieRecord, HoodieRecord> getRecordTaggerPartitionedRLI(List<FileSlice> fileSlices, SerializableBiFunction<String, Integer, Integer> mappingFunction) {
+    final Map<String, List<HoodieRecordLocation>> fileSlicesPerHudiPartition = new HashMap<>();
+    fileSlices.forEach(s -> fileSlicesPerHudiPartition.computeIfAbsent(HoodieTableMetadataUtil.getDataTablePartitionNameFromFileGroupName(s.getFileId()), x -> new ArrayList<>())
+        .add(new HoodieRecordLocation(s.getBaseInstantTime(), s.getFileId())));
+    return r -> {
+      String partitionPath;
+      if (r.getData() instanceof EmptyHoodieRecordPayloadWithPartition) {
+        partitionPath = ((EmptyHoodieRecordPayloadWithPartition) r.getData()).getPartitionPath();
+      } else {
+        partitionPath = ((HoodieMetadataPayload) r.getData()).getRecordGlobalLocation().getPartitionPath();

Review Comment:
   `.getRecordGlobalLocation().getPartitionPath()` is not efficient, we may need to add a new API on `HoodieMetadataPayload` to fetch the partition path directly from `recordIndexMetadata`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3140325260

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7396",
       "triggerID" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d855f5c0c907217a2ea6a87fc59167fbbf7ae236 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7396) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3110106932

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 05841e04b843d75afe5c48d9e288991787be1fb0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3110326249

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 05841e04b843d75afe5c48d9e288991787be1fb0 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016) 
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 6091ed96934c72ace34090dbc282aeb5620ae675 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234510320


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1689,40 +1804,75 @@ protected Pair<HoodieData<HoodieRecord>, List<HoodieFileGroupId>> tagRecordsWith
     try (HoodieTableFileSystemView fsView = HoodieTableMetadataUtil.getFileSystemViewForMetadataTable(metadataMetaClient)) {
       List<HoodieFileGroupId> hoodieFileGroupIdList = new ArrayList<>();
       for (Map.Entry<String, HoodieData<HoodieRecord>> entry : partitionRecordsMap.entrySet()) {
-        final String partitionName = entry.getKey();
+        final String partitionPath = entry.getKey();
         HoodieData<HoodieRecord> records = entry.getValue();
+        boolean isPartitionedRLI = Objects.equals(partitionPath, RECORD_INDEX.getPartitionPath()) && dataWriteConfig.isPartitionedRecordIndexEnabled();
         List<FileSlice> fileSlices =
-            HoodieTableMetadataUtil.getPartitionLatestFileSlices(metadataMetaClient, Option.ofNullable(fsView), partitionName);
-        if (fileSlices.isEmpty()) {
-          // scheduling or initialising of INDEX only initializes the file group and not add commit
-          // so if there are no committed file slices, look for inflight slices
-          ValidationUtils.checkState(isInitializing || dataMetaClient.getTableConfig().getMetadataPartitionsInflight().contains(partitionName),
-              String.format("Partition %s should be part of inflight metadata partitions here %s", partitionName, dataMetaClient.getTableConfig().getMetadataPartitionsInflight()));
-          fileSlices = getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient, Option.ofNullable(fsView), partitionName);
+            HoodieTableMetadataUtil.getPartitionLatestFileSlices(metadataMetaClient, Option.ofNullable(fsView), partitionPath);
+        // scheduling of INDEX only initializes the file group and not add commit
+        // so if there are no committed file slices, look for inflight slices
+        if (isPartitionedRLI || fileSlices.isEmpty()) {
+          // For isPartitionedRLI, new partitions added to the data table will cause new filegroups that are not yet commited
+          // therefore, we always need to look for inflight filegroups
+          ValidationUtils.checkState(isInitializing || dataMetaClient.getTableConfig().getMetadataPartitionsInflight().contains(partitionPath) || isPartitionedRLI,

Review Comment:
   can we simplify the logic to be more clear:
   
   ```java
   if (fileSlices.isEmpty()) {
     ValidationUtils.checkState(isInitializing || dataMetaClient.getTableConfig() ...;
     fileSlices = getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient, Option.ofNullable(fsView), partitionName);
   } else if (isPartitionedRLI) {
     fileSlices = getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient, Option.ofNullable(fsView), partitionPath);
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234547869


##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1083,24 +1084,32 @@ public static Pair<Set<String>, Set<String>> computeRevivedAndDeletedKeys(Set<St
    * @return
    */
   @VisibleForTesting
-  public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord> recordIndexRecords, int parallelism) {
-    return recordIndexRecords.mapToPair(
-            (SerializablePairFunction<HoodieRecord, HoodieKey, HoodieRecord>) t -> Pair.of(t.getKey(), t))
-        .reduceByKey((SerializableBiFunction<HoodieRecord, HoodieRecord, HoodieRecord>) (record1, record2) -> {
-          boolean isRecord1Deleted = record1.getData() instanceof EmptyHoodieRecordPayload;
-          boolean isRecord2Deleted = record2.getData() instanceof EmptyHoodieRecordPayload;
-          if (isRecord1Deleted && !isRecord2Deleted) {
-            return record2;
-          } else if (!isRecord1Deleted && isRecord2Deleted) {
-            return record1;
-          } else if (isRecord1Deleted && isRecord2Deleted) {
-            // let's delete just 1 of them
-            return record1;
-          } else {
-            throw new HoodieIOException("Two HoodieRecord updates to RLI is seen for same record key " + record2.getRecordKey() + ", record 1 : "
-                + record1.getData().toString() + ", record 2 : " + record2.getData().toString());
-          }
-        }, parallelism).values();
+  public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord> recordIndexRecords, int parallelism, boolean isPartitionedRLI) {
+    HoodiePairData<HoodieKey, HoodieRecord> recordIndexRecordsPair;
+    if (isPartitionedRLI) {
+      recordIndexRecordsPair = recordIndexRecords.mapToPair(r -> {
+        String recordPartitionPath = r.getData() instanceof EmptyHoodieRecordPayloadWithPartition

Review Comment:
   this is so hacky, the record from MDT should have a `HoodieKey` that represents it's actual MDT partition, here it should be `record_index`, I feel like we should use the record key + data table partition path as the composite record keys under partitionedRLI to make the same keys from different DT partition distinct.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3134039383

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 204a05a68c5875a42a77e9f24252eec07f45e42e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134) 
   * 5cd49ce761480de054e3ce0516cec7f9af06f2ed UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3114032008

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 7344e77705233f8271c2024830e6389d92419a06 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029) 
   * 6f8c2b0230c43f44509d16f4472adca831d66fea Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3138417593

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     }, {
       "hash" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378",
       "triggerID" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7386",
       "triggerID" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 028e2a1fe0be05d3a045bad459c80d29ef16939b UNKNOWN
   * 539a101dd2109c191d673ddde1761d95eedcfa99 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378) 
   * b9c2250aacb6e071b27aa69e172e2eab3b8b517b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7386) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3138601844

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     }, {
       "hash" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378",
       "triggerID" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7386",
       "triggerID" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 028e2a1fe0be05d3a045bad459c80d29ef16939b UNKNOWN
   * b9c2250aacb6e071b27aa69e172e2eab3b8b517b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7386) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2240890598


##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1083,24 +1084,32 @@ public static Pair<Set<String>, Set<String>> computeRevivedAndDeletedKeys(Set<St
    * @return
    */
   @VisibleForTesting
-  public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord> recordIndexRecords, int parallelism) {
-    return recordIndexRecords.mapToPair(
-            (SerializablePairFunction<HoodieRecord, HoodieKey, HoodieRecord>) t -> Pair.of(t.getKey(), t))
-        .reduceByKey((SerializableBiFunction<HoodieRecord, HoodieRecord, HoodieRecord>) (record1, record2) -> {
-          boolean isRecord1Deleted = record1.getData() instanceof EmptyHoodieRecordPayload;
-          boolean isRecord2Deleted = record2.getData() instanceof EmptyHoodieRecordPayload;
-          if (isRecord1Deleted && !isRecord2Deleted) {
-            return record2;
-          } else if (!isRecord1Deleted && isRecord2Deleted) {
-            return record1;
-          } else if (isRecord1Deleted && isRecord2Deleted) {
-            // let's delete just 1 of them
-            return record1;
-          } else {
-            throw new HoodieIOException("Two HoodieRecord updates to RLI is seen for same record key " + record2.getRecordKey() + ", record 1 : "
-                + record1.getData().toString() + ", record 2 : " + record2.getData().toString());
-          }
-        }, parallelism).values();
+  public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord> recordIndexRecords, int parallelism, boolean isPartitionedRLI) {
+    HoodiePairData<HoodieKey, HoodieRecord> recordIndexRecordsPair;
+    if (isPartitionedRLI) {
+      recordIndexRecordsPair = recordIndexRecords.mapToPair(r -> {
+        String recordPartitionPath = r.getData() instanceof EmptyHoodieRecordPayloadWithPartition

Review Comment:
   If we do that, then whenever a partition is updated, we will need to update all the filegroups in the table. Because record key + data table partition path can hash to any of the mdt filegroups. So using this way, the number of filegroups updated in the mdt will be proportional to the number of partitions updated in data table. This helps with large scale tables



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3138039130

   @jonvex : can you rebase w/ latest master. 
   once its green we are good to go


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3118797168

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 6f8c2b0230c43f44509d16f4472adca831d66fea Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069) 
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234483340


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1291,10 +1364,52 @@ protected Pair<List<HoodieFileGroupId>, HoodieData<HoodieRecord>> tagRecordsWith
   @Override
   public void update(HoodieCommitMetadata commitMetadata, String instantTime) {
     mayBeReinitMetadataReader();
+    maybeInitializeNewFileGroupsForPartitionedRLI(commitMetadata, instantTime);
     processAndCommit(instantTime, new BatchMetadataConversionFunction(instantTime, commitMetadata, getMetadataPartitionsToUpdate()));
     closeInternal();
   }
 
+  /**
+   * This method is used to initialize new file groups for partitioned record index during the in-flight commit.
+   * It will initialize new file groups for partitions that are newly added in the inflight commit.
+   *
+   * @param commitMetadata metadata for the inflight commit
+   * @param instantTime    Timestamp for the mdt commit
+   */
+  private void maybeInitializeNewFileGroupsForPartitionedRLI(HoodieCommitMetadata commitMetadata, String instantTime) {

Review Comment:
   It's great if we can move these utilities to `HoodieTableMetadataUtil`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3110139978

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 05841e04b843d75afe5c48d9e288991787be1fb0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016) 
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3111573847

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 6091ed96934c72ace34090dbc282aeb5620ae675 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020) 
   * 7344e77705233f8271c2024830e6389d92419a06 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2244003475


##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1083,24 +1084,32 @@ public static Pair<Set<String>, Set<String>> computeRevivedAndDeletedKeys(Set<St
    * @return
    */
   @VisibleForTesting
-  public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord> recordIndexRecords, int parallelism) {
-    return recordIndexRecords.mapToPair(
-            (SerializablePairFunction<HoodieRecord, HoodieKey, HoodieRecord>) t -> Pair.of(t.getKey(), t))
-        .reduceByKey((SerializableBiFunction<HoodieRecord, HoodieRecord, HoodieRecord>) (record1, record2) -> {
-          boolean isRecord1Deleted = record1.getData() instanceof EmptyHoodieRecordPayload;
-          boolean isRecord2Deleted = record2.getData() instanceof EmptyHoodieRecordPayload;
-          if (isRecord1Deleted && !isRecord2Deleted) {
-            return record2;
-          } else if (!isRecord1Deleted && isRecord2Deleted) {
-            return record1;
-          } else if (isRecord1Deleted && isRecord2Deleted) {
-            // let's delete just 1 of them
-            return record1;
-          } else {
-            throw new HoodieIOException("Two HoodieRecord updates to RLI is seen for same record key " + record2.getRecordKey() + ", record 1 : "
-                + record1.getData().toString() + ", record 2 : " + record2.getData().toString());
-          }
-        }, parallelism).values();
+  public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord> recordIndexRecords, int parallelism, boolean isPartitionedRLI) {
+    HoodiePairData<HoodieKey, HoodieRecord> recordIndexRecordsPair;
+    if (isPartitionedRLI) {
+      recordIndexRecordsPair = recordIndexRecords.mapToPair(r -> {
+        String recordPartitionPath = r.getData() instanceof EmptyHoodieRecordPayloadWithPartition

Review Comment:
   yes. one high level goal we had while designing the partitioned RLI is: 
   the lookup we do in MDT rli file groups should be proportional to the data table hudi partition. 
   
   for eg, 
   if p1 is 1GB and p2 is 100GB. 
   the number of file groups we lookup in MDT RLI should be relative. 
   if we take the approach you are suggesting, then even for p1, we might be polling 100s of RLI file groups. 
   
   then we can't really show benefits from RLI in all cases where different data table partitions have diff sizes. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3139271405

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     }, {
       "hash" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378",
       "triggerID" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7386",
       "triggerID" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7396",
       "triggerID" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 028e2a1fe0be05d3a045bad459c80d29ef16939b UNKNOWN
   * d855f5c0c907217a2ea6a87fc59167fbbf7ae236 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7396) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3138262197

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     }, {
       "hash" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378",
       "triggerID" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 5cd49ce761480de054e3ce0516cec7f9af06f2ed Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321) 
   * 028e2a1fe0be05d3a045bad459c80d29ef16939b UNKNOWN
   * 539a101dd2109c191d673ddde1761d95eedcfa99 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3140378547

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7396",
       "triggerID" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d855f5c0c907217a2ea6a87fc59167fbbf7ae236 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7396) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3134366797

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 5cd49ce761480de054e3ce0516cec7f9af06f2ed Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234445588


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -454,29 +457,26 @@ private boolean initializeFromFilesystem(String dataTableInstantTime, List<Metad
       String instantTimeForPartition = generateUniqueInstantTime(dataTableInstantTime);
       String partitionTypeName = partitionType.name();
       LOG.info("Initializing MDT partition {} at instant {}", partitionTypeName, instantTimeForPartition);
-      String partitionName;
+      String relativePartitionPath;
       Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
-      List<String> columnsToIndex = new ArrayList<>();
       Lazy<Option<Schema>> tableSchema = Lazy.lazily(() -> HoodieTableMetadataUtil.tryResolveSchemaForTable(dataMetaClient));
       try {
         switch (partitionType) {
           case FILES:
             fileGroupCountAndRecordsPair = initializeFilesPartition(partitionIdToAllFilesMap);
-            partitionName = FILES.getPartitionPath();
+            initializeFilegroupsAndCommitToMetadataPartition(partitionType, FILES.getPartitionPath(), fileGroupCountAndRecordsPair, instantTimeForPartition, Collections.emptyList());

Review Comment:
   +1, the name is too long, or `initMetadataPartition` if you like.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3111698840

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 7344e77705233f8271c2024830e6389d92419a06 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3110144213

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 05841e04b843d75afe5c48d9e288991787be1fb0 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016) 
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3119231338

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 6f8c2b0230c43f44509d16f4472adca831d66fea Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069) 
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 204a05a68c5875a42a77e9f24252eec07f45e42e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3119047012

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * 6f8c2b0230c43f44509d16f4472adca831d66fea Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069) 
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 204a05a68c5875a42a77e9f24252eec07f45e42e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234504672


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1689,40 +1804,75 @@ protected Pair<HoodieData<HoodieRecord>, List<HoodieFileGroupId>> tagRecordsWith
     try (HoodieTableFileSystemView fsView = HoodieTableMetadataUtil.getFileSystemViewForMetadataTable(metadataMetaClient)) {
       List<HoodieFileGroupId> hoodieFileGroupIdList = new ArrayList<>();
       for (Map.Entry<String, HoodieData<HoodieRecord>> entry : partitionRecordsMap.entrySet()) {
-        final String partitionName = entry.getKey();
+        final String partitionPath = entry.getKey();
         HoodieData<HoodieRecord> records = entry.getValue();
+        boolean isPartitionedRLI = Objects.equals(partitionPath, RECORD_INDEX.getPartitionPath()) && dataWriteConfig.isPartitionedRecordIndexEnabled();
         List<FileSlice> fileSlices =
-            HoodieTableMetadataUtil.getPartitionLatestFileSlices(metadataMetaClient, Option.ofNullable(fsView), partitionName);
-        if (fileSlices.isEmpty()) {
-          // scheduling or initialising of INDEX only initializes the file group and not add commit
-          // so if there are no committed file slices, look for inflight slices
-          ValidationUtils.checkState(isInitializing || dataMetaClient.getTableConfig().getMetadataPartitionsInflight().contains(partitionName),
-              String.format("Partition %s should be part of inflight metadata partitions here %s", partitionName, dataMetaClient.getTableConfig().getMetadataPartitionsInflight()));
-          fileSlices = getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient, Option.ofNullable(fsView), partitionName);
+            HoodieTableMetadataUtil.getPartitionLatestFileSlices(metadataMetaClient, Option.ofNullable(fsView), partitionPath);
+        // scheduling of INDEX only initializes the file group and not add commit
+        // so if there are no committed file slices, look for inflight slices
+        if (isPartitionedRLI || fileSlices.isEmpty()) {
+          // For isPartitionedRLI, new partitions added to the data table will cause new filegroups that are not yet commited
+          // therefore, we always need to look for inflight filegroups
+          ValidationUtils.checkState(isInitializing || dataMetaClient.getTableConfig().getMetadataPartitionsInflight().contains(partitionPath) || isPartitionedRLI,

Review Comment:
   If there are no new partitions added in `partitionedRLI`, should we get the inflight file slices here, is there any inconsistency issues?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234467577


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -957,22 +1032,28 @@ private List<DirectoryInfo> listAllPartitionsFromMDT(String initializationTime,
    * Let's say we configure 10 file groups for record level index partition, and prefix as "record-index-bucket-"
    * File groups will be named as :
    * record-index-bucket-0000, .... -> ..., record-index-bucket-0009
+   *
+   * @param tryErasePartition  During initialization, we want to delete any existing files in the partition. However, for partitioned rli we call this method
+   *                           multiple times. We guard the deletion code with tryErasePartition and only set it to true the first time this method is called.
+   *                           Otherwise, each repeated call would erase the files created by the previous call
    */
   private void initializeFileGroups(HoodieTableMetaClient dataMetaClient, MetadataPartitionType metadataPartition, String instantTime,
-                                    int fileGroupCount, String partitionName) throws IOException {
-    // Remove all existing file groups or leftover files in the partition
-    final StoragePath partitionPath = new StoragePath(metadataWriteConfig.getBasePath(), partitionName);
-    HoodieStorage storage = metadataMetaClient.getStorage();
-    try {
-      final List<StoragePathInfo> existingFiles = storage.listDirectEntries(partitionPath);
-      if (existingFiles.size() > 0) {
-        LOG.warn("Deleting all existing files found in MDT partition {}", partitionName);
-        storage.deleteDirectory(partitionPath);
-        ValidationUtils.checkState(!storage.exists(partitionPath),
-            "Failed to delete MDT partition " + partitionName);
+                                    int fileGroupCount, String relativePartitionPath, Option<String> dataPartitionName, boolean tryErasePartition) throws IOException {
+    if (tryErasePartition) {

Review Comment:
   Can we move the cleaning logic out as a separate utility method, so that we can get rid of this confusing flag and control the cleaning more flexibly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13603:
URL: https://github.com/apache/hudi/pull/13603#issuecomment-3138885115

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7016",
       "triggerID" : "05841e04b843d75afe5c48d9e288991787be1fb0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c59d0556bb02359bb39f3fad88486813a2b3f0b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7020",
       "triggerID" : "6091ed96934c72ace34090dbc282aeb5620ae675",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7344e77705233f8271c2024830e6389d92419a06",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7029",
       "triggerID" : "7344e77705233f8271c2024830e6389d92419a06",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7069",
       "triggerID" : "6f8c2b0230c43f44509d16f4472adca831d66fea",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7134",
       "triggerID" : "204a05a68c5875a42a77e9f24252eec07f45e42e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7321",
       "triggerID" : "5cd49ce761480de054e3ce0516cec7f9af06f2ed",
       "triggerType" : "PUSH"
     }, {
       "hash" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "028e2a1fe0be05d3a045bad459c80d29ef16939b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7378",
       "triggerID" : "539a101dd2109c191d673ddde1761d95eedcfa99",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7386",
       "triggerID" : "b9c2250aacb6e071b27aa69e172e2eab3b8b517b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d855f5c0c907217a2ea6a87fc59167fbbf7ae236",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c59d0556bb02359bb39f3fad88486813a2b3f0b5 UNKNOWN
   * d74e0aaa29233df959fd1c0d5d2cfc93d6142c8d UNKNOWN
   * 028e2a1fe0be05d3a045bad459c80d29ef16939b UNKNOWN
   * b9c2250aacb6e071b27aa69e172e2eab3b8b517b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7386) 
   * d855f5c0c907217a2ea6a87fc59167fbbf7ae236 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9630] Support Partitioned RLI in the Metadata Table [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #13603:
URL: https://github.com/apache/hudi/pull/13603


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org