You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2024/10/24 17:55:00 UTC
[PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
codope opened a new pull request, #12160:
URL: https://github.com/apache/hudi/pull/12160
### Change Logs
Fix data skipping with RLI if record key is composite. When there are multiple record key fields, we need to form the record keys sequence correctly. Added a test to show skipping with more than one record key fields.
### Impact
Support data skipping for complex primary keys
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12160:
URL: https://github.com/apache/hudi/pull/12160#issuecomment-2436157635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398",
"triggerID" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbeb41bdd47f5cc0064d95002885ad03b788718e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12160:
URL: https://github.com/apache/hudi/pull/12160#issuecomment-2437625001
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398",
"triggerID" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"triggerType" : "PUSH"
}, {
"hash" : "2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1416",
"triggerID" : "2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbeb41bdd47f5cc0064d95002885ad03b788718e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398)
* 2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1416)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12160:
URL: https://github.com/apache/hudi/pull/12160#discussion_r1817566167
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -122,47 +123,78 @@ object RecordLevelIndexSupport {
val INDEX_NAME = "RECORD_LEVEL"
/**
- * If the input query is an EqualTo or IN query on simple record key columns, the function returns a tuple of
+ * If the input query is an EqualTo or IN query on record key columns, the function returns a tuple of
* list of the query and list of record key literals present in the query otherwise returns an empty option.
*
* @param queryFilter The query that need to be filtered.
* @return Tuple of filtered query and list of record key literals that need to be matched
*/
- def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Option[String]): Option[(Expression, List[String])] = {
+ def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Array[String]): Option[(Expression, List[String])] = {
+ val isComplexRecordKey = recordKeyOpt.length > 1
queryFilter match {
+ // Handle EqualTo expressions
case equalToQuery: EqualTo =>
val attributeLiteralTuple = getAttributeLiteralTuple(equalToQuery.left, equalToQuery.right).orNull
if (attributeLiteralTuple != null) {
val attribute = attributeLiteralTuple._1
val literal = attributeLiteralTuple._2
- if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
- Option.apply(equalToQuery, List.apply(literal.value.toString))
+ if (attribute != null && attribute.name != null && recordKeyOpt.contains(attribute.name)) {
+ val recordKeyLiteral = getRecordKeyLiteral(attribute, literal, isComplexRecordKey)
Review Comment:
> For now, we can just see if all record key fields are present or not. If not, then don't use RLI. What do you think?
I kind of think we should also restrain all the expression type to `EQUALS`. For e.g, we have record key `col_0,col_1` and expression `col_0 >0 && col_1 <100`, you can not utilitize RLI in sucn use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12160:
URL: https://github.com/apache/hudi/pull/12160#discussion_r1853295109
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala:
##########
@@ -240,4 +251,61 @@ class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
val fileNames = rliIndexSupport.computeCandidateFileNames(fileIndex, Seq(dataFilter), null, prunedPaths, false)
assertEquals(if (includeLogFiles) 2 else 1, fileNames.get.size)
}
+
+ @Test
+ def testRLIWithMultipleRecordKeyFields(): Unit = {
+ val tableName = "dummy_table_multi_pk"
+ val dummyTablePath = tempDir.resolve("dummy_table_multi_pk").toAbsolutePath.toString
+ val hudiOpts = Map(
+ "hoodie.insert.shuffle.parallelism" -> "4",
+ "hoodie.upsert.shuffle.parallelism" -> "4",
+ HoodieWriteConfig.TBL_NAME.key -> tableName,
+ DataSourceWriteOptions.RECORDKEY_FIELD.key -> "record_key_col,name",
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "partition_key_col",
+ DataSourceReadOptions.ENABLE_DATA_SKIPPING.key -> "true"
+ ) ++ metadataOpts
+
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | record_key_col string,
+ | name string,
+ | not_record_key_col string,
+ | partition_key_col string
+ |) using hudi
+ | options (
+ | primaryKey ='record_key_col,name',
+ | hoodie.metadata.enable = 'true',
+ | hoodie.metadata.record.index.enable = 'true',
+ | hoodie.datasource.write.recordkey.field = 'record_key_col,name',
+ | hoodie.enable.data.skipping = 'true'
+ | )
+ | partitioned by(partition_key_col)
+ | location '$dummyTablePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values('row1', 'name1', 'a', 'p1')")
+ spark.sql(s"insert into $tableName values('row2', 'name2', 'b', 'p2')")
+ spark.sql(s"insert into $tableName values('row3', 'name3', 'c', 'p3')")
+
+ val latestSnapshotDf = spark.read.format("hudi").options(hudiOpts).load(dummyTablePath)
+ this.metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(dummyTablePath)
+ .setConf(HoodieTestUtils.getDefaultStorageConf)
+ .build()
+ verifyEqualToQuery(hudiOpts, Seq("record_key_col", "name"), tableName, latestSnapshotDf, HoodieTableMetaClient.reload(metaClient))
+ }
+
+ def verifyEqualToQuery(hudiOpts: Map[String, String], colNames: Seq[String], tableName: String, latestSnapshotDf: DataFrame, metaClient: HoodieTableMetaClient): Unit = {
Review Comment:
also valite `IN` and `AND`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12160:
URL: https://github.com/apache/hudi/pull/12160#discussion_r1853209132
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -122,47 +123,78 @@ object RecordLevelIndexSupport {
val INDEX_NAME = "RECORD_LEVEL"
/**
- * If the input query is an EqualTo or IN query on simple record key columns, the function returns a tuple of
+ * If the input query is an EqualTo or IN query on record key columns, the function returns a tuple of
* list of the query and list of record key literals present in the query otherwise returns an empty option.
*
* @param queryFilter The query that need to be filtered.
* @return Tuple of filtered query and list of record key literals that need to be matched
*/
- def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Option[String]): Option[(Expression, List[String])] = {
+ def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Array[String]): Option[(Expression, List[String])] = {
+ val isComplexRecordKey = recordKeyOpt.length > 1
queryFilter match {
+ // Handle EqualTo expressions
case equalToQuery: EqualTo =>
val attributeLiteralTuple = getAttributeLiteralTuple(equalToQuery.left, equalToQuery.right).orNull
if (attributeLiteralTuple != null) {
val attribute = attributeLiteralTuple._1
val literal = attributeLiteralTuple._2
- if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
- Option.apply(equalToQuery, List.apply(literal.value.toString))
+ if (attribute != null && attribute.name != null && recordKeyOpt.contains(attribute.name)) {
+ val recordKeyLiteral = getRecordKeyLiteral(attribute, literal, isComplexRecordKey)
Review Comment:
yes.
- lets do RLI look up only when all fields in complex key gen are part of the predicates.
- If I am not wrong, we already only support "IN" and "EQUAL"s. So, not sure what Danny is bringing up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12160:
URL: https://github.com/apache/hudi/pull/12160#discussion_r1853294309
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -122,47 +123,78 @@ object RecordLevelIndexSupport {
val INDEX_NAME = "RECORD_LEVEL"
/**
- * If the input query is an EqualTo or IN query on simple record key columns, the function returns a tuple of
+ * If the input query is an EqualTo or IN query on record key columns, the function returns a tuple of
* list of the query and list of record key literals present in the query otherwise returns an empty option.
*
* @param queryFilter The query that need to be filtered.
* @return Tuple of filtered query and list of record key literals that need to be matched
*/
- def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Option[String]): Option[(Expression, List[String])] = {
+ def filterQueryWithRecordKey(queryFilter: Expression, recordKeyFields: Array[String]): Option[(Expression, List[String])] = {
+ val isComplexRecordKey = recordKeyFields.length > 1
queryFilter match {
+ // Handle EqualTo expressions
case equalToQuery: EqualTo =>
val attributeLiteralTuple = getAttributeLiteralTuple(equalToQuery.left, equalToQuery.right).orNull
if (attributeLiteralTuple != null) {
val attribute = attributeLiteralTuple._1
val literal = attributeLiteralTuple._2
- if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
- Option.apply(equalToQuery, List.apply(literal.value.toString))
+ if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyFields)) {
+ val recordKeyLiteral = getRecordKeyLiteral(attribute, literal, isComplexRecordKey)
+ Option.apply(equalToQuery, List.apply(recordKeyLiteral))
} else {
Option.empty
}
} else {
Option.empty
}
+ // Handle In expressions
case inQuery: In =>
var validINQuery = true
inQuery.value match {
case attribute: AttributeReference =>
- if (!attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
+ if (!attributeMatchesRecordKey(attribute.name, recordKeyFields)) {
validINQuery = false
}
case _ => validINQuery = false
}
var literals: List[String] = List.empty
inQuery.list.foreach {
- case literal: Literal => literals = literals :+ literal.value.toString
+ case literal: Literal => literals = literals :+ getRecordKeyLiteral(inQuery.value.asInstanceOf[AttributeReference], literal, isComplexRecordKey)
Review Comment:
We need to check all the record key fields are got involved.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -122,47 +123,78 @@ object RecordLevelIndexSupport {
val INDEX_NAME = "RECORD_LEVEL"
/**
- * If the input query is an EqualTo or IN query on simple record key columns, the function returns a tuple of
+ * If the input query is an EqualTo or IN query on record key columns, the function returns a tuple of
* list of the query and list of record key literals present in the query otherwise returns an empty option.
*
* @param queryFilter The query that need to be filtered.
* @return Tuple of filtered query and list of record key literals that need to be matched
*/
- def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Option[String]): Option[(Expression, List[String])] = {
+ def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Array[String]): Option[(Expression, List[String])] = {
+ val isComplexRecordKey = recordKeyOpt.length > 1
queryFilter match {
+ // Handle EqualTo expressions
case equalToQuery: EqualTo =>
val attributeLiteralTuple = getAttributeLiteralTuple(equalToQuery.left, equalToQuery.right).orNull
if (attributeLiteralTuple != null) {
val attribute = attributeLiteralTuple._1
val literal = attributeLiteralTuple._2
- if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
- Option.apply(equalToQuery, List.apply(literal.value.toString))
+ if (attribute != null && attribute.name != null && recordKeyOpt.contains(attribute.name)) {
+ val recordKeyLiteral = getRecordKeyLiteral(attribute, literal, isComplexRecordKey)
Review Comment:
We need to check all the record key fields are got involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed pull request #12160: [HUDI-8432] Fix data skipping with RLI if record key is composite
URL: https://github.com/apache/hudi/pull/12160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12160:
URL: https://github.com/apache/hudi/pull/12160#discussion_r1853294752
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkBaseIndexSupport.scala:
##########
@@ -164,29 +164,34 @@ abstract class SparkBaseIndexSupport(spark: SparkSession,
var recordKeys: List[String] = List.empty
for (query <- queryFilters) {
val recordKeyOpt = getRecordKeyConfig
- RecordLevelIndexSupport.filterQueryWithRecordKey(query, recordKeyOpt).foreach({
- case (exp: Expression, recKeys: List[String]) =>
- recordKeys = recordKeys ++ recKeys
- recordKeyQueries = recordKeyQueries :+ exp
- })
+ recordKeyOpt.foreach { recordKeysArray =>
+ // Handle composite record keys
+ RecordLevelIndexSupport.filterQueryWithRecordKey(query, recordKeysArray).foreach {
+ case (exp: Expression, recKeys: List[String]) =>
+ exp match {
+ // For IN, add each element individually to recordKeys
+ case _: In =>
+ recordKeys = recordKeys ++ recKeys
+
+ // For other cases, basically EqualTo, concatenate recKeys with the default separator
+ case _ =>
+ recordKeys = recordKeys :+ recKeys.mkString(DEFAULT_RECORD_KEY_PARTS_SEPARATOR)
+ }
+ recordKeyQueries = recordKeyQueries :+ exp
+ }
+ }
}
-
- Tuple2.apply(recordKeyQueries, recordKeys)
+ (recordKeyQueries, recordKeys)
}
}
/**
- * Returns the configured record key for the table if it is a simple record key else returns empty option.
+ * Returns the configured record key for the table.
*/
- private def getRecordKeyConfig: Option[String] = {
+ private def getRecordKeyConfig: Option[Array[String]] = {
val recordKeysOpt: org.apache.hudi.common.util.Option[Array[String]] = metaClient.getTableConfig.getRecordKeyFields
- val recordKeyOpt = recordKeysOpt.map[String](JFunction.toJavaFunction[Array[String], String](arr =>
- if (arr.length == 1) {
- arr(0)
- } else {
- null
- }))
- Option.apply(recordKeyOpt.orElse(null))
+ // Convert the Hudi Option to Scala Option and return if present
+ Option(recordKeysOpt.orElse(null)).filter(_.nonEmpty)
Review Comment:
The `nonEmpty` does now work when the record key is null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12160:
URL: https://github.com/apache/hudi/pull/12160#discussion_r1816572972
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -122,47 +123,78 @@ object RecordLevelIndexSupport {
val INDEX_NAME = "RECORD_LEVEL"
/**
- * If the input query is an EqualTo or IN query on simple record key columns, the function returns a tuple of
+ * If the input query is an EqualTo or IN query on record key columns, the function returns a tuple of
* list of the query and list of record key literals present in the query otherwise returns an empty option.
*
* @param queryFilter The query that need to be filtered.
* @return Tuple of filtered query and list of record key literals that need to be matched
*/
- def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Option[String]): Option[(Expression, List[String])] = {
+ def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Array[String]): Option[(Expression, List[String])] = {
+ val isComplexRecordKey = recordKeyOpt.length > 1
queryFilter match {
+ // Handle EqualTo expressions
case equalToQuery: EqualTo =>
val attributeLiteralTuple = getAttributeLiteralTuple(equalToQuery.left, equalToQuery.right).orNull
if (attributeLiteralTuple != null) {
val attribute = attributeLiteralTuple._1
val literal = attributeLiteralTuple._2
- if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
- Option.apply(equalToQuery, List.apply(literal.value.toString))
+ if (attribute != null && attribute.name != null && recordKeyOpt.contains(attribute.name)) {
+ val recordKeyLiteral = getRecordKeyLiteral(attribute, literal, isComplexRecordKey)
Review Comment:
good point.. we should check if the predicate contains all of record key fields. Technically, we can support subset of record key fields as well but that involves pulling all records in RLI and checking for subset match. Something to followup. For now, we can just see if all record key fields are present or not. If not, then don't use RLI. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12160:
URL: https://github.com/apache/hudi/pull/12160#discussion_r1853213011
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -122,47 +123,78 @@ object RecordLevelIndexSupport {
val INDEX_NAME = "RECORD_LEVEL"
/**
- * If the input query is an EqualTo or IN query on simple record key columns, the function returns a tuple of
+ * If the input query is an EqualTo or IN query on record key columns, the function returns a tuple of
* list of the query and list of record key literals present in the query otherwise returns an empty option.
*
* @param queryFilter The query that need to be filtered.
* @return Tuple of filtered query and list of record key literals that need to be matched
*/
- def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Option[String]): Option[(Expression, List[String])] = {
+ def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Array[String]): Option[(Expression, List[String])] = {
+ val isComplexRecordKey = recordKeyOpt.length > 1
queryFilter match {
+ // Handle EqualTo expressions
case equalToQuery: EqualTo =>
val attributeLiteralTuple = getAttributeLiteralTuple(equalToQuery.left, equalToQuery.right).orNull
if (attributeLiteralTuple != null) {
val attribute = attributeLiteralTuple._1
val literal = attributeLiteralTuple._2
- if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
- Option.apply(equalToQuery, List.apply(literal.value.toString))
+ if (attribute != null && attribute.name != null && recordKeyOpt.contains(attribute.name)) {
+ val recordKeyLiteral = getRecordKeyLiteral(attribute, literal, isComplexRecordKey)
Review Comment:
We don't support range queries with RLI, so it will fallback to no data skipping (exxcept for partition pruning which is done before we read RLI).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #12160:
URL: https://github.com/apache/hudi/pull/12160#issuecomment-2499851704
@nsivabalan @danny0405 I have pushed a new PR with comments addressed https://github.com/apache/hudi/pull/12336. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #12160:
URL: https://github.com/apache/hudi/pull/12160#discussion_r1857938763
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkBaseIndexSupport.scala:
##########
@@ -164,29 +164,34 @@ abstract class SparkBaseIndexSupport(spark: SparkSession,
var recordKeys: List[String] = List.empty
for (query <- queryFilters) {
val recordKeyOpt = getRecordKeyConfig
- RecordLevelIndexSupport.filterQueryWithRecordKey(query, recordKeyOpt).foreach({
- case (exp: Expression, recKeys: List[String]) =>
- recordKeys = recordKeys ++ recKeys
- recordKeyQueries = recordKeyQueries :+ exp
- })
+ recordKeyOpt.foreach { recordKeysArray =>
+ // Handle composite record keys
+ RecordLevelIndexSupport.filterQueryWithRecordKey(query, recordKeysArray).foreach {
+ case (exp: Expression, recKeys: List[String]) =>
+ exp match {
+ // For IN, add each element individually to recordKeys
+ case _: In =>
+ recordKeys = recordKeys ++ recKeys
+
+ // For other cases, basically EqualTo, concatenate recKeys with the default separator
+ case _ =>
+ recordKeys = recordKeys :+ recKeys.mkString(DEFAULT_RECORD_KEY_PARTS_SEPARATOR)
+ }
+ recordKeyQueries = recordKeyQueries :+ exp
+ }
+ }
}
-
- Tuple2.apply(recordKeyQueries, recordKeys)
+ (recordKeyQueries, recordKeys)
}
}
/**
- * Returns the configured record key for the table if it is a simple record key else returns empty option.
+ * Returns the configured record key for the table.
*/
- private def getRecordKeyConfig: Option[String] = {
+ private def getRecordKeyConfig: Option[Array[String]] = {
val recordKeysOpt: org.apache.hudi.common.util.Option[Array[String]] = metaClient.getTableConfig.getRecordKeyFields
- val recordKeyOpt = recordKeysOpt.map[String](JFunction.toJavaFunction[Array[String], String](arr =>
- if (arr.length == 1) {
- arr(0)
- } else {
- null
- }))
- Option.apply(recordKeyOpt.orElse(null))
+ // Convert the Hudi Option to Scala Option and return if present
+ Option(recordKeysOpt.orElse(null)).filter(_.nonEmpty)
Review Comment:
It does work. Filter would only work on Some option type.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala:
##########
@@ -240,4 +251,61 @@ class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
val fileNames = rliIndexSupport.computeCandidateFileNames(fileIndex, Seq(dataFilter), null, prunedPaths, false)
assertEquals(if (includeLogFiles) 2 else 1, fileNames.get.size)
}
+
+ @Test
+ def testRLIWithMultipleRecordKeyFields(): Unit = {
+ val tableName = "dummy_table_multi_pk"
+ val dummyTablePath = tempDir.resolve("dummy_table_multi_pk").toAbsolutePath.toString
+ val hudiOpts = Map(
+ "hoodie.insert.shuffle.parallelism" -> "4",
+ "hoodie.upsert.shuffle.parallelism" -> "4",
+ HoodieWriteConfig.TBL_NAME.key -> tableName,
+ DataSourceWriteOptions.RECORDKEY_FIELD.key -> "record_key_col,name",
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "partition_key_col",
+ DataSourceReadOptions.ENABLE_DATA_SKIPPING.key -> "true"
+ ) ++ metadataOpts
+
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | record_key_col string,
+ | name string,
+ | not_record_key_col string,
+ | partition_key_col string
+ |) using hudi
+ | options (
+ | primaryKey ='record_key_col,name',
+ | hoodie.metadata.enable = 'true',
+ | hoodie.metadata.record.index.enable = 'true',
+ | hoodie.datasource.write.recordkey.field = 'record_key_col,name',
+ | hoodie.enable.data.skipping = 'true'
+ | )
+ | partitioned by(partition_key_col)
+ | location '$dummyTablePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values('row1', 'name1', 'a', 'p1')")
+ spark.sql(s"insert into $tableName values('row2', 'name2', 'b', 'p2')")
+ spark.sql(s"insert into $tableName values('row3', 'name3', 'c', 'p3')")
+
+ val latestSnapshotDf = spark.read.format("hudi").options(hudiOpts).load(dummyTablePath)
+ this.metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(dummyTablePath)
+ .setConf(HoodieTestUtils.getDefaultStorageConf)
+ .build()
+ verifyEqualToQuery(hudiOpts, Seq("record_key_col", "name"), tableName, latestSnapshotDf, HoodieTableMetaClient.reload(metaClient))
+ }
+
+ def verifyEqualToQuery(hudiOpts: Map[String, String], colNames: Seq[String], tableName: String, latestSnapshotDf: DataFrame, metaClient: HoodieTableMetaClient): Unit = {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12160:
URL: https://github.com/apache/hudi/pull/12160#issuecomment-2437752888
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398",
"triggerID" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"triggerType" : "PUSH"
}, {
"hash" : "2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1416",
"triggerID" : "2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b",
"triggerType" : "PUSH"
}, {
"hash" : "b20398a1a5e45de9eadf8ef73d667e3469479d86",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1417",
"triggerID" : "b20398a1a5e45de9eadf8ef73d667e3469479d86",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1416)
* b20398a1a5e45de9eadf8ef73d667e3469479d86 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12160:
URL: https://github.com/apache/hudi/pull/12160#discussion_r1815867538
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -122,47 +123,78 @@ object RecordLevelIndexSupport {
val INDEX_NAME = "RECORD_LEVEL"
/**
- * If the input query is an EqualTo or IN query on simple record key columns, the function returns a tuple of
+ * If the input query is an EqualTo or IN query on record key columns, the function returns a tuple of
* list of the query and list of record key literals present in the query otherwise returns an empty option.
*
* @param queryFilter The query that need to be filtered.
* @return Tuple of filtered query and list of record key literals that need to be matched
*/
- def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Option[String]): Option[(Expression, List[String])] = {
+ def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Array[String]): Option[(Expression, List[String])] = {
+ val isComplexRecordKey = recordKeyOpt.length > 1
queryFilter match {
+ // Handle EqualTo expressions
case equalToQuery: EqualTo =>
val attributeLiteralTuple = getAttributeLiteralTuple(equalToQuery.left, equalToQuery.right).orNull
if (attributeLiteralTuple != null) {
val attribute = attributeLiteralTuple._1
val literal = attributeLiteralTuple._2
- if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
- Option.apply(equalToQuery, List.apply(literal.value.toString))
+ if (attribute != null && attribute.name != null && recordKeyOpt.contains(attribute.name)) {
+ val recordKeyLiteral = getRecordKeyLiteral(attribute, literal, isComplexRecordKey)
Review Comment:
How can we ensure all the consitituent record key values got involved in the expression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12160:
URL: https://github.com/apache/hudi/pull/12160#discussion_r1853293898
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -122,47 +123,78 @@ object RecordLevelIndexSupport {
val INDEX_NAME = "RECORD_LEVEL"
/**
- * If the input query is an EqualTo or IN query on simple record key columns, the function returns a tuple of
+ * If the input query is an EqualTo or IN query on record key columns, the function returns a tuple of
* list of the query and list of record key literals present in the query otherwise returns an empty option.
*
* @param queryFilter The query that need to be filtered.
* @return Tuple of filtered query and list of record key literals that need to be matched
*/
- def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Option[String]): Option[(Expression, List[String])] = {
+ def filterQueryWithRecordKey(queryFilter: Expression, recordKeyFields: Array[String]): Option[(Expression, List[String])] = {
+ val isComplexRecordKey = recordKeyFields.length > 1
queryFilter match {
+ // Handle EqualTo expressions
case equalToQuery: EqualTo =>
val attributeLiteralTuple = getAttributeLiteralTuple(equalToQuery.left, equalToQuery.right).orNull
if (attributeLiteralTuple != null) {
val attribute = attributeLiteralTuple._1
val literal = attributeLiteralTuple._2
- if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
- Option.apply(equalToQuery, List.apply(literal.value.toString))
+ if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyFields)) {
+ val recordKeyLiteral = getRecordKeyLiteral(attribute, literal, isComplexRecordKey)
+ Option.apply(equalToQuery, List.apply(recordKeyLiteral))
} else {
Option.empty
}
} else {
Option.empty
}
+ // Handle In expressions
case inQuery: In =>
var validINQuery = true
inQuery.value match {
case attribute: AttributeReference =>
- if (!attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
+ if (!attributeMatchesRecordKey(attribute.name, recordKeyFields)) {
validINQuery = false
}
case _ => validINQuery = false
}
var literals: List[String] = List.empty
inQuery.list.foreach {
- case literal: Literal => literals = literals :+ literal.value.toString
+ case literal: Literal => literals = literals :+ getRecordKeyLiteral(inQuery.value.asInstanceOf[AttributeReference], literal, isComplexRecordKey)
case _ => validINQuery = false
}
if (validINQuery) {
Option.apply(inQuery, literals)
} else {
Option.empty
}
+
+ // Handle And expression (composite filter)
+ case andQuery: And =>
+ val leftResult = filterQueryWithRecordKey(andQuery.left, recordKeyFields)
+ val rightResult = filterQueryWithRecordKey(andQuery.right, recordKeyFields)
+
+ // If both left and right filters are valid, concatenate their results
+ (leftResult, rightResult) match {
+ case (Some((leftExp, leftKeys)), Some((rightExp, rightKeys))) =>
+ // Return concatenated expressions and record keys
+ Option.apply(And(leftExp, rightExp), leftKeys ++ rightKeys)
+ case _ => Option.empty
+ }
+
+ // Handle Or expression (for completeness, though usually not used for record key filtering)
+ case orQuery: Or =>
Review Comment:
I guess the `OR` does not work as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12160:
URL: https://github.com/apache/hudi/pull/12160#issuecomment-2436007861
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398",
"triggerID" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbeb41bdd47f5cc0064d95002885ad03b788718e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12160:
URL: https://github.com/apache/hudi/pull/12160#issuecomment-2436003548
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbeb41bdd47f5cc0064d95002885ad03b788718e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #12160:
URL: https://github.com/apache/hudi/pull/12160#issuecomment-2500209664
Closing this PR in favor of #12336
The update PR has all the comments addressed from this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12160:
URL: https://github.com/apache/hudi/pull/12160#issuecomment-2438006743
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398",
"triggerID" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"triggerType" : "PUSH"
}, {
"hash" : "2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1416",
"triggerID" : "2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b",
"triggerType" : "PUSH"
}, {
"hash" : "b20398a1a5e45de9eadf8ef73d667e3469479d86",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1417",
"triggerID" : "b20398a1a5e45de9eadf8ef73d667e3469479d86",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b20398a1a5e45de9eadf8ef73d667e3469479d86 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #12160:
URL: https://github.com/apache/hudi/pull/12160#discussion_r1857938535
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -122,47 +123,78 @@ object RecordLevelIndexSupport {
val INDEX_NAME = "RECORD_LEVEL"
/**
- * If the input query is an EqualTo or IN query on simple record key columns, the function returns a tuple of
+ * If the input query is an EqualTo or IN query on record key columns, the function returns a tuple of
* list of the query and list of record key literals present in the query otherwise returns an empty option.
*
* @param queryFilter The query that need to be filtered.
* @return Tuple of filtered query and list of record key literals that need to be matched
*/
- def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Option[String]): Option[(Expression, List[String])] = {
+ def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Array[String]): Option[(Expression, List[String])] = {
+ val isComplexRecordKey = recordKeyOpt.length > 1
queryFilter match {
+ // Handle EqualTo expressions
case equalToQuery: EqualTo =>
val attributeLiteralTuple = getAttributeLiteralTuple(equalToQuery.left, equalToQuery.right).orNull
if (attributeLiteralTuple != null) {
val attribute = attributeLiteralTuple._1
val literal = attributeLiteralTuple._2
- if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
- Option.apply(equalToQuery, List.apply(literal.value.toString))
+ if (attribute != null && attribute.name != null && recordKeyOpt.contains(attribute.name)) {
+ val recordKeyLiteral = getRecordKeyLiteral(attribute, literal, isComplexRecordKey)
Review Comment:
Addressed
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -122,47 +123,78 @@ object RecordLevelIndexSupport {
val INDEX_NAME = "RECORD_LEVEL"
/**
- * If the input query is an EqualTo or IN query on simple record key columns, the function returns a tuple of
+ * If the input query is an EqualTo or IN query on record key columns, the function returns a tuple of
* list of the query and list of record key literals present in the query otherwise returns an empty option.
*
* @param queryFilter The query that need to be filtered.
* @return Tuple of filtered query and list of record key literals that need to be matched
*/
- def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Option[String]): Option[(Expression, List[String])] = {
+ def filterQueryWithRecordKey(queryFilter: Expression, recordKeyFields: Array[String]): Option[(Expression, List[String])] = {
+ val isComplexRecordKey = recordKeyFields.length > 1
queryFilter match {
+ // Handle EqualTo expressions
case equalToQuery: EqualTo =>
val attributeLiteralTuple = getAttributeLiteralTuple(equalToQuery.left, equalToQuery.right).orNull
if (attributeLiteralTuple != null) {
val attribute = attributeLiteralTuple._1
val literal = attributeLiteralTuple._2
- if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
- Option.apply(equalToQuery, List.apply(literal.value.toString))
+ if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyFields)) {
+ val recordKeyLiteral = getRecordKeyLiteral(attribute, literal, isComplexRecordKey)
+ Option.apply(equalToQuery, List.apply(recordKeyLiteral))
} else {
Option.empty
}
} else {
Option.empty
}
+ // Handle In expressions
case inQuery: In =>
var validINQuery = true
inQuery.value match {
case attribute: AttributeReference =>
- if (!attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
+ if (!attributeMatchesRecordKey(attribute.name, recordKeyFields)) {
validINQuery = false
}
case _ => validINQuery = false
}
var literals: List[String] = List.empty
inQuery.list.foreach {
- case literal: Literal => literals = literals :+ literal.value.toString
+ case literal: Literal => literals = literals :+ getRecordKeyLiteral(inQuery.value.asInstanceOf[AttributeReference], literal, isComplexRecordKey)
case _ => validINQuery = false
}
if (validINQuery) {
Option.apply(inQuery, literals)
} else {
Option.empty
}
+
+ // Handle And expression (composite filter)
+ case andQuery: And =>
+ val leftResult = filterQueryWithRecordKey(andQuery.left, recordKeyFields)
+ val rightResult = filterQueryWithRecordKey(andQuery.right, recordKeyFields)
+
+ // If both left and right filters are valid, concatenate their results
+ (leftResult, rightResult) match {
+ case (Some((leftExp, leftKeys)), Some((rightExp, rightKeys))) =>
+ // Return concatenated expressions and record keys
+ Option.apply(And(leftExp, rightExp), leftKeys ++ rightKeys)
+ case _ => Option.empty
+ }
+
+ // Handle Or expression (for completeness, though usually not used for record key filtering)
+ case orQuery: Or =>
Review Comment:
Yeah..removed the OR handling for now. We can add it in a separate jira.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -122,47 +123,78 @@ object RecordLevelIndexSupport {
val INDEX_NAME = "RECORD_LEVEL"
/**
- * If the input query is an EqualTo or IN query on simple record key columns, the function returns a tuple of
+ * If the input query is an EqualTo or IN query on record key columns, the function returns a tuple of
* list of the query and list of record key literals present in the query otherwise returns an empty option.
*
* @param queryFilter The query that need to be filtered.
* @return Tuple of filtered query and list of record key literals that need to be matched
*/
- def filterQueryWithRecordKey(queryFilter: Expression, recordKeyOpt: Option[String]): Option[(Expression, List[String])] = {
+ def filterQueryWithRecordKey(queryFilter: Expression, recordKeyFields: Array[String]): Option[(Expression, List[String])] = {
+ val isComplexRecordKey = recordKeyFields.length > 1
queryFilter match {
+ // Handle EqualTo expressions
case equalToQuery: EqualTo =>
val attributeLiteralTuple = getAttributeLiteralTuple(equalToQuery.left, equalToQuery.right).orNull
if (attributeLiteralTuple != null) {
val attribute = attributeLiteralTuple._1
val literal = attributeLiteralTuple._2
- if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
- Option.apply(equalToQuery, List.apply(literal.value.toString))
+ if (attribute != null && attribute.name != null && attributeMatchesRecordKey(attribute.name, recordKeyFields)) {
+ val recordKeyLiteral = getRecordKeyLiteral(attribute, literal, isComplexRecordKey)
+ Option.apply(equalToQuery, List.apply(recordKeyLiteral))
} else {
Option.empty
}
} else {
Option.empty
}
+ // Handle In expressions
case inQuery: In =>
var validINQuery = true
inQuery.value match {
case attribute: AttributeReference =>
- if (!attributeMatchesRecordKey(attribute.name, recordKeyOpt)) {
+ if (!attributeMatchesRecordKey(attribute.name, recordKeyFields)) {
validINQuery = false
}
case _ => validINQuery = false
}
var literals: List[String] = List.empty
inQuery.list.foreach {
- case literal: Literal => literals = literals :+ literal.value.toString
+ case literal: Literal => literals = literals :+ getRecordKeyLiteral(inQuery.value.asInstanceOf[AttributeReference], literal, isComplexRecordKey)
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12160:
URL: https://github.com/apache/hudi/pull/12160#issuecomment-2437622149
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398",
"triggerID" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"triggerType" : "PUSH"
}, {
"hash" : "2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbeb41bdd47f5cc0064d95002885ad03b788718e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398)
* 2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12160:
URL: https://github.com/apache/hudi/pull/12160#issuecomment-2437749413
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398",
"triggerID" : "cbeb41bdd47f5cc0064d95002885ad03b788718e",
"triggerType" : "PUSH"
}, {
"hash" : "2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1416",
"triggerID" : "2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b",
"triggerType" : "PUSH"
}, {
"hash" : "b20398a1a5e45de9eadf8ef73d667e3469479d86",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b20398a1a5e45de9eadf8ef73d667e3469479d86",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbeb41bdd47f5cc0064d95002885ad03b788718e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1398)
* 2ecc9e7fa8e65aad7b919b2ae0a4f6b9deaf129b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1416)
* b20398a1a5e45de9eadf8ef73d667e3469479d86 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org