You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2025/10/29 22:35:23 UTC

[PR] fix: Fixing record index related configs and enums [hudi]

nsivabalan opened a new pull request, #14180:
URL: https://github.com/apache/hudi/pull/14180

   ### Describe the issue this Pull Request addresses
   
   Standardize config names and enums related to record level index. 
   
   Index type: 
   Prior to this patch: 
   RECORD_INDEX - > refers to global version of RLI
   PARTITIONED_RECORD_INDEX -> refers to non global version of RLI 
   
   Change with this PR: 
   GLOBAL_RECORD_LEVEL_INDEX -> refers to global version of RLI
   RECORD_LEVEL_INDEX -> refers to non global version of RLI 
   
   Writer property to enable for mdt writes: 
   Prior to this patch: 
   hoodie.metadata.record.index.enable  -> enables global version of RLI with mdt writes.
   hoodie.metadata.partitioned.record.index.enable -> enables non global version of RLI with mdt writes.
   
   Change with this PR:
   hoodie.metadata.global.record.level.index.enable -> enables global version of RLI with mdt writes.
   hoodie.metadata.record.level.index.enable -> enables non global version of RLI with mdt writes.
   
   ### Summary and Changelog
   
   Switching index type of global version of RLI from RECORD_INDEX to GLOBAL_RECORD_LEVEL_INDEX. (Added b/w compatibilty support as well). 
   Switching index type of non-global version of RLI from PARTITIONED_RECORD_INDEX to RECORD_LEVEL_INDEX.
   
   Switching writer property to enable global RL with mdt writes from `hoodie.metadata.record.index.enable` to `hoodie.metadata.global.record.level.index.enable`. Added b/w compatibility support as well. 
   Switching writer property to enable non global RL with mdt writes from `hoodie.metadata.partitioned.record.index.enable` to  `hoodie.metadata.record.level.index.enable`.
   
   ### Impact
   
   Standard naming conventions in line with other indexes. 
   
   ### Risk Level
   
   low.
   
   ### Documentation Update
   
   <!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
   
   - The config description must be updated if new configs are added or the default value of the configs are changed.
   - Any new feature or user-facing change requires updating the Hudi website. Please follow the 
     [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Enough context is provided in the sections above
   - [ ] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14180:
URL: https://github.com/apache/hudi/pull/14180#discussion_r2487785647


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestGlobalRecordLevelIndex.scala:
##########
@@ -0,0 +1,775 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.client.SparkRDDWriteClient
+import org.apache.hudi.client.common.HoodieSparkEngineContext
+import org.apache.hudi.client.transaction.PreferWriterConflictResolutionStrategy
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.model._
+import org.apache.hudi.common.table.{HoodieTableConfig, HoodieTableMetaClient}
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+import org.apache.hudi.common.testutils.{HoodieTestDataGenerator, InProcessTimeGenerator}
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator.recordsToStrings
+import org.apache.hudi.config._
+import org.apache.hudi.exception.HoodieWriteConflictException
+import org.apache.hudi.index.HoodieIndex
+import org.apache.hudi.index.record.HoodieRecordIndex
+import org.apache.hudi.metadata.{HoodieBackedTableMetadata, HoodieTableMetadataUtil, MetadataPartitionType}
+import org.apache.hudi.util.JavaConversions
+
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
+import org.apache.spark.sql.functions.lit
+import org.apache.spark.sql.types.{StringType, StructField, StructType}
+import org.junit.jupiter.api._
+import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertTrue}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, CsvSource, EnumSource, MethodSource}
+import org.junit.jupiter.params.provider.Arguments.arguments
+
+import java.util
+import java.util.{Collections, Properties}
+import java.util.concurrent.Executors
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{Await, ExecutionContext, Future}
+import scala.concurrent.duration.Duration
+import scala.language.postfixOps
+import scala.util.Using
+
+@Tag("functional-b")
+class TestGlobalRecordLevelIndex extends RecordLevelIndexTestBase {
+
+  @ParameterizedTest
+  @EnumSource(classOf[HoodieTableType])
+  def testRLIInitialization(tableType: HoodieTableType): Unit = {
+    val hudiOpts = commonOpts + (DataSourceWriteOptions.TABLE_TYPE.key -> tableType.name())
+    doWriteAndValidateDataAndRecordIndex(hudiOpts,
+      operation = DataSourceWriteOptions.INSERT_OPERATION_OPT_VAL,
+      saveMode = SaveMode.Overwrite)
+    metaClient = HoodieTableMetaClient.reload(metaClient)
+    assertFalse(HoodieRecordIndex.isPartitioned(metaClient.getIndexMetadata.get().getIndexDefinitions.get(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX)))
+  }
+
+  @Test
+  def testRLIInitializationForMorGlobalIndex(): Unit = {
+    val tableType = HoodieTableType.MERGE_ON_READ
+    val hudiOpts = commonOpts + (DataSourceWriteOptions.TABLE_TYPE.key -> tableType.name()) +
+      (HoodieMetadataConfig.RECORD_INDEX_MIN_FILE_GROUP_COUNT_PROP.key -> "1") +
+      (HoodieMetadataConfig.RECORD_INDEX_MAX_FILE_GROUP_COUNT_PROP.key -> "1") +
+      (HoodieIndexConfig.INDEX_TYPE.key -> HoodieIndex.IndexType.GLOBAL_RECORD_LEVEL_INDEX.name()) +
+      (HoodieIndexConfig.RECORD_INDEX_UPDATE_PARTITION_PATH_ENABLE.key -> "true") -
+      HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key
+
+    val dataGen1 = HoodieTestDataGenerator.createTestGeneratorFirstPartition()
+    val dataGen2 = HoodieTestDataGenerator.createTestGeneratorSecondPartition()
+
+    // batch1 inserts (5 records)
+    val instantTime1 = getNewInstantTime()
+    val latestBatch = recordsToStrings(dataGen1.generateInserts(instantTime1, 5)).asScala.toSeq
+    var operation = INSERT_OPERATION_OPT_VAL
+    val latestBatchDf = spark.read.json(spark.sparkContext.parallelize(latestBatch, 1))
+    latestBatchDf.cache()
+    latestBatchDf.write.format("hudi")
+      .options(hudiOpts)
+      .mode(SaveMode.Overwrite)
+      .save(basePath)
+    val deletedDf1 = calculateMergedDf(latestBatchDf, operation, true)
+    deletedDf1.cache()
+
+    // batch2. upsert. update few records to 2nd partition from partition1 and insert a few to partition2.
+    val instantTime2 = getNewInstantTime()
+
+    val latestBatch2_1 = recordsToStrings(dataGen1.generateUniqueUpdates(instantTime2, 3)).asScala.toSeq
+    val latestBatchDf2_1 = spark.read.json(spark.sparkContext.parallelize(latestBatch2_1, 1))
+    val latestBatchDf2_2 = latestBatchDf2_1.withColumn("partition", lit(HoodieTestDataGenerator.DEFAULT_SECOND_PARTITION_PATH))
+      .withColumn("partition_path", lit(HoodieTestDataGenerator.DEFAULT_SECOND_PARTITION_PATH))
+    val latestBatch2_3 = recordsToStrings(dataGen2.generateInserts(instantTime2, 2)).asScala.toSeq
+    val latestBatchDf2_3 = spark.read.json(spark.sparkContext.parallelize(latestBatch2_3, 1))
+    val latestBatchDf2Final = latestBatchDf2_3.union(latestBatchDf2_2)
+    latestBatchDf2Final.cache()
+    latestBatchDf2Final.write.format("hudi")
+      .options(hudiOpts)
+      .mode(SaveMode.Append)
+      .save(basePath)
+    operation = UPSERT_OPERATION_OPT_VAL
+    val deletedDf2 = calculateMergedDf(latestBatchDf2Final, operation, true)
+    deletedDf2.cache()
+
+    val hudiOpts2 = commonOpts + (DataSourceWriteOptions.TABLE_TYPE.key -> tableType.name()) +
+      (HoodieMetadataConfig.RECORD_INDEX_MIN_FILE_GROUP_COUNT_PROP.key -> "1") +
+      (HoodieMetadataConfig.RECORD_INDEX_MAX_FILE_GROUP_COUNT_PROP.key -> "1") +
+      (HoodieIndexConfig.INDEX_TYPE.key -> HoodieIndex.IndexType.GLOBAL_RECORD_LEVEL_INDEX.name()) +
+      (HoodieIndexConfig.RECORD_INDEX_UPDATE_PARTITION_PATH_ENABLE.key -> "true") +
+      (HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "true")
+
+    val instantTime3 = getNewInstantTime()
+    // batch3. update 2 records from newly inserted records from commit 2 to partition2
+    val latestBatch3 = recordsToStrings(dataGen2.generateUniqueUpdates(instantTime3, 2)).asScala.toSeq
+    val latestBatchDf3 = spark.read.json(spark.sparkContext.parallelize(latestBatch3, 1))
+    latestBatchDf3.cache()
+    latestBatchDf3.write.format("hudi")
+      .options(hudiOpts2)
+      .mode(SaveMode.Append)
+      .save(basePath)
+    val deletedDf3 = calculateMergedDf(latestBatchDf3, operation, true)
+    deletedDf3.cache()
+    metaClient = HoodieTableMetaClient.builder().setBasePath(basePath).setConf(storageConf).build()
+    validateDataAndRecordIndices(hudiOpts, deletedDf3)
+    deletedDf2.unpersist()
+    deletedDf3.unpersist()
+  }
+
+  private def getNewInstantTime(): String = {
+    InProcessTimeGenerator.createNewInstantTime();
+  }
+
+  @ParameterizedTest
+  @EnumSource(classOf[HoodieTableType])
+  def testRLIUpsertTableTypes(tableType: HoodieTableType): Unit = {
+    testRLIUpsert(tableType, "GLOBAL_RECORD_LEVEL_INDEX")
+  }
+
+  @ParameterizedTest
+  @CsvSource(value = Array(
+    "RECORD_INDEX",
+    "GLOBAL_RECORD_LEVEL_INDEX")
+  )
+  // tests b/w compatibility of configs for global RLI
+  def testRLIUpsertIndexTypes(indexType: String): Unit = {
+    testRLIUpsert(HoodieTableType.COPY_ON_WRITE, indexType)
+  }
+
+  @ParameterizedTest
+  @CsvSource(value = Array(
+    "hoodie.metadata.record.index.enable",
+    "hoodie.metadata.global.record.level.index.enable")
+  )
+  // tests b/w compatibility of configs for global RLI

Review Comment:
   tests added for b/w compatibility



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3483526308

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414",
       "triggerID" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9454",
       "triggerID" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1250c4ecb565aaec51274eedc2618f3ec87be1f1",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9456",
       "triggerID" : "1250c4ecb565aaec51274eedc2618f3ec87be1f1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6352a9ba580665f876daec4350b5b3fc9ee2c5ce UNKNOWN
   * 1250c4ecb565aaec51274eedc2618f3ec87be1f1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9456) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14180:
URL: https://github.com/apache/hudi/pull/14180#discussion_r2487784802


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestGlobalRecordLevelIndex.scala:
##########
@@ -0,0 +1,775 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.client.SparkRDDWriteClient
+import org.apache.hudi.client.common.HoodieSparkEngineContext
+import org.apache.hudi.client.transaction.PreferWriterConflictResolutionStrategy
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.model._
+import org.apache.hudi.common.table.{HoodieTableConfig, HoodieTableMetaClient}
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+import org.apache.hudi.common.testutils.{HoodieTestDataGenerator, InProcessTimeGenerator}
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator.recordsToStrings
+import org.apache.hudi.config._
+import org.apache.hudi.exception.HoodieWriteConflictException
+import org.apache.hudi.index.HoodieIndex
+import org.apache.hudi.index.record.HoodieRecordIndex
+import org.apache.hudi.metadata.{HoodieBackedTableMetadata, HoodieTableMetadataUtil, MetadataPartitionType}
+import org.apache.hudi.util.JavaConversions
+
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
+import org.apache.spark.sql.functions.lit
+import org.apache.spark.sql.types.{StringType, StructField, StructType}
+import org.junit.jupiter.api._
+import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertTrue}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, CsvSource, EnumSource, MethodSource}
+import org.junit.jupiter.params.provider.Arguments.arguments
+
+import java.util
+import java.util.{Collections, Properties}
+import java.util.concurrent.Executors
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{Await, ExecutionContext, Future}
+import scala.concurrent.duration.Duration
+import scala.language.postfixOps
+import scala.util.Using
+
+@Tag("functional-b")
+class TestGlobalRecordLevelIndex extends RecordLevelIndexTestBase {
+
+  @ParameterizedTest
+  @EnumSource(classOf[HoodieTableType])
+  def testRLIInitialization(tableType: HoodieTableType): Unit = {
+    val hudiOpts = commonOpts + (DataSourceWriteOptions.TABLE_TYPE.key -> tableType.name())
+    doWriteAndValidateDataAndRecordIndex(hudiOpts,
+      operation = DataSourceWriteOptions.INSERT_OPERATION_OPT_VAL,
+      saveMode = SaveMode.Overwrite)
+    metaClient = HoodieTableMetaClient.reload(metaClient)
+    assertFalse(HoodieRecordIndex.isPartitioned(metaClient.getIndexMetadata.get().getIndexDefinitions.get(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX)))
+  }
+
+  @Test
+  def testRLIInitializationForMorGlobalIndex(): Unit = {
+    val tableType = HoodieTableType.MERGE_ON_READ
+    val hudiOpts = commonOpts + (DataSourceWriteOptions.TABLE_TYPE.key -> tableType.name()) +
+      (HoodieMetadataConfig.RECORD_INDEX_MIN_FILE_GROUP_COUNT_PROP.key -> "1") +
+      (HoodieMetadataConfig.RECORD_INDEX_MAX_FILE_GROUP_COUNT_PROP.key -> "1") +
+      (HoodieIndexConfig.INDEX_TYPE.key -> HoodieIndex.IndexType.GLOBAL_RECORD_LEVEL_INDEX.name()) +
+      (HoodieIndexConfig.RECORD_INDEX_UPDATE_PARTITION_PATH_ENABLE.key -> "true") -
+      HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key
+
+    val dataGen1 = HoodieTestDataGenerator.createTestGeneratorFirstPartition()
+    val dataGen2 = HoodieTestDataGenerator.createTestGeneratorSecondPartition()
+
+    // batch1 inserts (5 records)
+    val instantTime1 = getNewInstantTime()
+    val latestBatch = recordsToStrings(dataGen1.generateInserts(instantTime1, 5)).asScala.toSeq
+    var operation = INSERT_OPERATION_OPT_VAL
+    val latestBatchDf = spark.read.json(spark.sparkContext.parallelize(latestBatch, 1))
+    latestBatchDf.cache()
+    latestBatchDf.write.format("hudi")
+      .options(hudiOpts)
+      .mode(SaveMode.Overwrite)
+      .save(basePath)
+    val deletedDf1 = calculateMergedDf(latestBatchDf, operation, true)
+    deletedDf1.cache()
+
+    // batch2. upsert. update few records to 2nd partition from partition1 and insert a few to partition2.
+    val instantTime2 = getNewInstantTime()
+
+    val latestBatch2_1 = recordsToStrings(dataGen1.generateUniqueUpdates(instantTime2, 3)).asScala.toSeq
+    val latestBatchDf2_1 = spark.read.json(spark.sparkContext.parallelize(latestBatch2_1, 1))
+    val latestBatchDf2_2 = latestBatchDf2_1.withColumn("partition", lit(HoodieTestDataGenerator.DEFAULT_SECOND_PARTITION_PATH))
+      .withColumn("partition_path", lit(HoodieTestDataGenerator.DEFAULT_SECOND_PARTITION_PATH))
+    val latestBatch2_3 = recordsToStrings(dataGen2.generateInserts(instantTime2, 2)).asScala.toSeq
+    val latestBatchDf2_3 = spark.read.json(spark.sparkContext.parallelize(latestBatch2_3, 1))
+    val latestBatchDf2Final = latestBatchDf2_3.union(latestBatchDf2_2)
+    latestBatchDf2Final.cache()
+    latestBatchDf2Final.write.format("hudi")
+      .options(hudiOpts)
+      .mode(SaveMode.Append)
+      .save(basePath)
+    operation = UPSERT_OPERATION_OPT_VAL
+    val deletedDf2 = calculateMergedDf(latestBatchDf2Final, operation, true)
+    deletedDf2.cache()
+
+    val hudiOpts2 = commonOpts + (DataSourceWriteOptions.TABLE_TYPE.key -> tableType.name()) +
+      (HoodieMetadataConfig.RECORD_INDEX_MIN_FILE_GROUP_COUNT_PROP.key -> "1") +
+      (HoodieMetadataConfig.RECORD_INDEX_MAX_FILE_GROUP_COUNT_PROP.key -> "1") +
+      (HoodieIndexConfig.INDEX_TYPE.key -> HoodieIndex.IndexType.GLOBAL_RECORD_LEVEL_INDEX.name()) +
+      (HoodieIndexConfig.RECORD_INDEX_UPDATE_PARTITION_PATH_ENABLE.key -> "true") +
+      (HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "true")
+
+    val instantTime3 = getNewInstantTime()
+    // batch3. update 2 records from newly inserted records from commit 2 to partition2
+    val latestBatch3 = recordsToStrings(dataGen2.generateUniqueUpdates(instantTime3, 2)).asScala.toSeq
+    val latestBatchDf3 = spark.read.json(spark.sparkContext.parallelize(latestBatch3, 1))
+    latestBatchDf3.cache()
+    latestBatchDf3.write.format("hudi")
+      .options(hudiOpts2)
+      .mode(SaveMode.Append)
+      .save(basePath)
+    val deletedDf3 = calculateMergedDf(latestBatchDf3, operation, true)
+    deletedDf3.cache()
+    metaClient = HoodieTableMetaClient.builder().setBasePath(basePath).setConf(storageConf).build()
+    validateDataAndRecordIndices(hudiOpts, deletedDf3)
+    deletedDf2.unpersist()
+    deletedDf3.unpersist()
+  }
+
+  private def getNewInstantTime(): String = {
+    InProcessTimeGenerator.createNewInstantTime();
+  }
+
+  @ParameterizedTest
+  @EnumSource(classOf[HoodieTableType])
+  def testRLIUpsertTableTypes(tableType: HoodieTableType): Unit = {
+    testRLIUpsert(tableType, "GLOBAL_RECORD_LEVEL_INDEX")
+  }
+
+  @ParameterizedTest
+  @CsvSource(value = Array(
+    "RECORD_INDEX",
+    "GLOBAL_RECORD_LEVEL_INDEX")
+  )
+  // tests b/w compatibility of configs for global RLI

Review Comment:
   tests added for b/w compatibility



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3465067828

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412) 
   * 6352a9ba580665f876daec4350b5b3fc9ee2c5ce UNKNOWN
   * c69189bac38397112f4886f6e9e1b209e8885d15 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3482851923

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414",
       "triggerID" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9454",
       "triggerID" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6352a9ba580665f876daec4350b5b3fc9ee2c5ce UNKNOWN
   * ba483f02ae2c722f9f7eee7d9375466650203f1b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9454) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14180:
URL: https://github.com/apache/hudi/pull/14180#discussion_r2482347471


##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -274,15 +274,16 @@ public final class HoodieMetadataConfig extends HoodieConfig {
       .withDocumentation("When there is a pending instant in data table, this config limits the allowed number of deltacommits in metadata table to "
           + "prevent the metadata table's timeline from growing unboundedly as compaction won't be triggered due to the pending data table instant.");
 
-  public static final ConfigProperty<Boolean> RECORD_INDEX_ENABLE_PROP = ConfigProperty
-      .key(METADATA_PREFIX + ".record.index.enable")
+  public static final ConfigProperty<Boolean> GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP = ConfigProperty
+      .key(METADATA_PREFIX + ".global.record.level.index.enable")

Review Comment:
   yes, I added tests for this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3482303005

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414",
       "triggerID" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6352a9ba580665f876daec4350b5b3fc9ee2c5ce UNKNOWN
   * c69189bac38397112f4886f6e9e1b209e8885d15 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414) 
   * ba483f02ae2c722f9f7eee7d9375466650203f1b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3464614273

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412) 
   * 6352a9ba580665f876daec4350b5b3fc9ee2c5ce UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14180:
URL: https://github.com/apache/hudi/pull/14180#discussion_r2487786432


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestGlobalRecordLevelIndexWithSQL.scala:
##########
@@ -52,7 +52,7 @@ class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
     val hudiOpts = commonOpts ++ Map(
       DataSourceWriteOptions.TABLE_TYPE.key -> HoodieTableType.MERGE_ON_READ.name(),
       "hoodie.metadata.index.column.stats.enable" -> "false",
-      HoodieMetadataConfig.RECORD_INDEX_ENABLE_PROP.key -> "false")
+      HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "false")

Review Comment:
   https://github.com/apache/hudi/pull/14180/files/#r2487785647



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3482557527

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414",
       "triggerID" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9454",
       "triggerID" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6352a9ba580665f876daec4350b5b3fc9ee2c5ce UNKNOWN
   * c69189bac38397112f4886f6e9e1b209e8885d15 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414) 
   * ba483f02ae2c722f9f7eee7d9375466650203f1b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9454) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3464592337

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3483074876

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414",
       "triggerID" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9454",
       "triggerID" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1250c4ecb565aaec51274eedc2618f3ec87be1f1",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1250c4ecb565aaec51274eedc2618f3ec87be1f1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6352a9ba580665f876daec4350b5b3fc9ee2c5ce UNKNOWN
   * ba483f02ae2c722f9f7eee7d9375466650203f1b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9454) 
   * 1250c4ecb565aaec51274eedc2618f3ec87be1f1 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14180:
URL: https://github.com/apache/hudi/pull/14180#discussion_r2482348411


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestGlobalRecordLevelIndexWithSQL.scala:
##########
@@ -52,7 +52,7 @@ class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
     val hudiOpts = commonOpts ++ Map(
       DataSourceWriteOptions.TABLE_TYPE.key -> HoodieTableType.MERGE_ON_READ.name(),
       "hoodie.metadata.index.column.stats.enable" -> "false",
-      HoodieMetadataConfig.RECORD_INDEX_ENABLE_PROP.key -> "false")
+      HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "false")

Review Comment:
   I fixed one of the test to make it parametrized for this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3465453100

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414",
       "triggerID" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412) 
   * 6352a9ba580665f876daec4350b5b3fc9ee2c5ce UNKNOWN
   * c69189bac38397112f4886f6e9e1b209e8885d15 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3464582291

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3465732688

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414",
       "triggerID" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6352a9ba580665f876daec4350b5b3fc9ee2c5ce UNKNOWN
   * c69189bac38397112f4886f6e9e1b209e8885d15 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3465134719

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414",
       "triggerID" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412) 
   * 6352a9ba580665f876daec4350b5b3fc9ee2c5ce UNKNOWN
   * c69189bac38397112f4886f6e9e1b209e8885d15 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14180:
URL: https://github.com/apache/hudi/pull/14180#discussion_r2482161895


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndex.java:
##########
@@ -191,19 +191,28 @@ public enum IndexType {
     @EnumFieldDescription("Internal Config for indexing based on Flink state.")
     FLINK_STATE,
 
+    @Deprecated
     @EnumFieldDescription("Index which saves the record key to location mappings in the "
         + "HUDI Metadata Table. Record index is a global index, enforcing key uniqueness across all "
         + "partitions in the table. Supports sharding to achieve very high scale. For a table with "
-        + "keys that are only unique inside each partition, use `PARTITIONED_RECORD_INDEX` instead.")
+        + "keys that are only unique inside each partition, use `PARTITIONED_RECORD_INDEX` instead. "

Review Comment:
   ```suggestion
           + "keys that are only unique inside each partition, use `RECORD_LEVEL_INDEX` instead. "
   ```



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestGlobalRecordLevelIndexWithSQL.scala:
##########
@@ -52,7 +52,7 @@ class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
     val hudiOpts = commonOpts ++ Map(
       DataSourceWriteOptions.TABLE_TYPE.key -> HoodieTableType.MERGE_ON_READ.name(),
       "hoodie.metadata.index.column.stats.enable" -> "false",
-      HoodieMetadataConfig.RECORD_INDEX_ENABLE_PROP.key -> "false")
+      HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "false")

Review Comment:
   Could we also add a test with the old config naming to validate it works and prevents any breaking change in the future?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndex.java:
##########
@@ -191,19 +191,28 @@ public enum IndexType {
     @EnumFieldDescription("Internal Config for indexing based on Flink state.")
     FLINK_STATE,
 
+    @Deprecated
     @EnumFieldDescription("Index which saves the record key to location mappings in the "
         + "HUDI Metadata Table. Record index is a global index, enforcing key uniqueness across all "
         + "partitions in the table. Supports sharding to achieve very high scale. For a table with "
-        + "keys that are only unique inside each partition, use `PARTITIONED_RECORD_INDEX` instead.")
+        + "keys that are only unique inside each partition, use `PARTITIONED_RECORD_INDEX` instead. "
+        + "This enum is deprecated. Use GLOBAL_RECORD_LEVEL_INDEX or RECORD_LEVEL_INDEX "
+        + "based on global uniqueness constraints for indexing.")

Review Comment:
   ```suggestion
           + "This enum is deprecated. Use GLOBAL_RECORD_LEVEL_INDEX for global uniqueness of record keys or RECORD_LEVEL_INDEX for partition-level uniqueness of record keys.")
   ```



##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -274,15 +274,16 @@ public final class HoodieMetadataConfig extends HoodieConfig {
       .withDocumentation("When there is a pending instant in data table, this config limits the allowed number of deltacommits in metadata table to "
           + "prevent the metadata table's timeline from growing unboundedly as compaction won't be triggered due to the pending data table instant.");
 
-  public static final ConfigProperty<Boolean> RECORD_INDEX_ENABLE_PROP = ConfigProperty
-      .key(METADATA_PREFIX + ".record.index.enable")
+  public static final ConfigProperty<Boolean> GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP = ConfigProperty
+      .key(METADATA_PREFIX + ".global.record.level.index.enable")

Review Comment:
   Have you audited the code that when fetching this config the alternative config naming is always considered (i.e., no direct usage of `GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key()`)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14180:
URL: https://github.com/apache/hudi/pull/14180#issuecomment-3483244819

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9412",
       "triggerID" : "1aa75f5eddd30020e48d0fa6f7d41c80bfaeb00d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6352a9ba580665f876daec4350b5b3fc9ee2c5ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9414",
       "triggerID" : "c69189bac38397112f4886f6e9e1b209e8885d15",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9454",
       "triggerID" : "ba483f02ae2c722f9f7eee7d9375466650203f1b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1250c4ecb565aaec51274eedc2618f3ec87be1f1",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9456",
       "triggerID" : "1250c4ecb565aaec51274eedc2618f3ec87be1f1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6352a9ba580665f876daec4350b5b3fc9ee2c5ce UNKNOWN
   * ba483f02ae2c722f9f7eee7d9375466650203f1b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9454) 
   * 1250c4ecb565aaec51274eedc2618f3ec87be1f1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9456) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: Fixing record index related configs and enums [hudi]

Posted by "bhasudha (via GitHub)" <gi...@apache.org>.
bhasudha merged PR #14180:
URL: https://github.com/apache/hudi/pull/14180


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org