You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2024/11/26 07:19:17 UTC
[PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
lokeshj1703 opened a new pull request, #12336:
URL: https://github.com/apache/hudi/pull/12336
### Change Logs
Fix data skipping with RLI if record key is composite. When there are multiple record key fields, we need to form the record keys sequence correctly. Added a test to show skipping with more than one record key fields.
### Impact
Support data skipping for complex primary keys
### Risk level (write none, low medium or high below)
low
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #12336:
URL: https://github.com/apache/hudi/pull/12336#discussion_r1858971353
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -162,18 +164,49 @@ object RecordLevelIndexSupport {
}
var literals: List[String] = List.empty
inQuery.list.foreach {
- case literal: Literal => literals = literals :+ literal.value.toString
+ case literal: Literal =>
+ val recordKeyLiteral = getRecordKeyLiteral(inQuery.value.asInstanceOf[AttributeReference], literal, isComplexRecordKey)
+ literals = literals :+ recordKeyLiteral
case _ => validINQuery = false
}
if (validINQuery) {
Option.apply(inQuery, literals)
} else {
Option.empty
}
+
+ // Handle And expression (composite filter)
+ case andQuery: And =>
+ val leftResult = filterQueryWithRecordKey(andQuery.left, recordKeyOpt, isComplexRecordKey, attributeFetcher)
+ val rightResult = filterQueryWithRecordKey(andQuery.right, recordKeyOpt, isComplexRecordKey, attributeFetcher)
+
+ // If both left and right filters are valid, concatenate their results
+ (leftResult, rightResult) match {
+ case (Some((leftExp, leftKeys)), Some((rightExp, rightKeys))) =>
+ // Return concatenated expressions and record keys
+ Option.apply(And(leftExp, rightExp), leftKeys ++ rightKeys)
Review Comment:
This case is already handled. Test cases are added for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12336:
URL: https://github.com/apache/hudi/pull/12336#issuecomment-2499983965
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885e80032a81b10256e258dca03f4a10b6580858",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1965",
"triggerID" : "885e80032a81b10256e258dca03f4a10b6580858",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 885e80032a81b10256e258dca03f4a10b6580858 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1965)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12336:
URL: https://github.com/apache/hudi/pull/12336#discussion_r1858225267
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/FunctionalIndexSupport.scala:
##########
@@ -92,7 +92,7 @@ class FunctionalIndexSupport(spark: SparkSession,
def filterQueriesWithFunctionalFilterKey(queryFilters: Seq[Expression], sourceFieldOpt: Option[String]): List[Tuple2[Expression, List[String]]] = {
var functionalIndexQueries: List[Tuple2[Expression, List[String]]] = List.empty
for (query <- queryFilters) {
- filterQueryWithRecordKey(query, sourceFieldOpt, (expr: Expression) => {
+ filterQueryWithRecordKey(query, sourceFieldOpt, false, (expr: Expression) => {
Review Comment:
why hard code as false?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #12336:
URL: https://github.com/apache/hudi/pull/12336#discussion_r1858969836
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala:
##########
@@ -240,4 +269,63 @@ class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
val fileNames = rliIndexSupport.computeCandidateFileNames(fileIndex, Seq(dataFilter), null, prunedPaths, false)
assertEquals(if (includeLogFiles) 2 else 1, fileNames.get.size)
}
+
+ @Test
+ def testRLIWithMultipleRecordKeyFields(): Unit = {
+ val tableName = "dummy_table_multi_pk"
+ val dummyTablePath = tempDir.resolve("dummy_table_multi_pk").toAbsolutePath.toString
+ val hudiOpts = Map(
+ "hoodie.insert.shuffle.parallelism" -> "4",
+ "hoodie.upsert.shuffle.parallelism" -> "4",
+ HoodieWriteConfig.TBL_NAME.key -> tableName,
+ DataSourceWriteOptions.RECORDKEY_FIELD.key -> "record_key_col,name",
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "partition_key_col",
+ DataSourceReadOptions.ENABLE_DATA_SKIPPING.key -> "true"
+ ) ++ metadataOpts
+
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | record_key_col string,
+ | name string,
+ | not_record_key_col string,
+ | partition_key_col string
+ |) using hudi
+ | options (
+ | primaryKey ='record_key_col,name',
+ | hoodie.metadata.enable = 'true',
+ | hoodie.metadata.record.index.enable = 'true',
+ | hoodie.datasource.write.recordkey.field = 'record_key_col,name',
+ | hoodie.enable.data.skipping = 'true'
+ | )
+ | partitioned by(partition_key_col)
+ | location '$dummyTablePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values('row1', 'name1', 'a', 'p1')")
+ spark.sql(s"insert into $tableName values('row2', 'name2', 'b', 'p2')")
+ spark.sql(s"insert into $tableName values('row3', 'name3', 'c', 'p3')")
+
+ val latestSnapshotDf = spark.read.format("hudi").options(hudiOpts).load(dummyTablePath)
+ this.metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(dummyTablePath)
+ .setConf(HoodieTestUtils.getDefaultStorageConf)
+ .build()
+ verifyEqualToQuery(hudiOpts, Seq("record_key_col", "name"), tableName, latestSnapshotDf, HoodieTableMetaClient.reload(metaClient))
+ verifyInQuery(hudiOpts, Array("record_key_col", "name"), latestSnapshotDf, tableName, "partition_key_col", Seq("p2", "p3"), true)
Review Comment:
Added test cases for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12336:
URL: https://github.com/apache/hudi/pull/12336#discussion_r1858236015
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkBaseIndexSupport.scala:
##########
@@ -155,32 +158,59 @@ abstract class SparkBaseIndexSupport(spark: SparkSession,
(List.empty, List.empty)
} else {
var recordKeyQueries: List[Expression] = List.empty
- var recordKeys: List[String] = List.empty
- for (query <- queryFilters) {
- val recordKeyOpt = getRecordKeyConfig
- RecordLevelIndexSupport.filterQueryWithRecordKey(query, recordKeyOpt).foreach({
- case (exp: Expression, recKeys: List[String]) =>
- recordKeys = recordKeys ++ recKeys
- recordKeyQueries = recordKeyQueries :+ exp
- })
+ var compositeRecordKeys: List[String] = List.empty
+ val recordKeyOpt = getRecordKeyConfig
+ val isComplexRecordKey = recordKeyOpt.map(recordKeys => recordKeys.length).getOrElse(0) > 1
+ recordKeyOpt.foreach { recordKeysArray =>
+ // Handle composite record keys
+ breakable {
+ for (recordKey <- recordKeysArray) {
+ var recordKeys: List[String] = List.empty
+ for (query <- queryFilters) {
+ {
+ RecordLevelIndexSupport.filterQueryWithRecordKey(query, Option.apply(recordKey), isComplexRecordKey).foreach {
+ case (exp: Expression, recKeys: List[String]) =>
+ exp match {
+ // For IN, add each element individually to recordKeys
+ case _: In => recordKeys = recordKeys ++ recKeys
+
+ // For other cases, basically EqualTo, concatenate recKeys with the default separator
+ case _ => recordKeys = recordKeys ++ recKeys
+ }
+ recordKeyQueries = recordKeyQueries :+ exp
+ }
+ }
+ }
+
+ if (recordKeys.isEmpty) {
+ recordKeyQueries = List.empty
+ compositeRecordKeys = List.empty
+ break()
+ } else if (!isComplexRecordKey || compositeRecordKeys.isEmpty) {
+ compositeRecordKeys = recordKeys
+ } else {
+ var tempCompositeRecordKeys: List[String] = List.empty
+ for (compRecKey <- compositeRecordKeys) {
Review Comment:
Still we need to check all the fields of the composition records keys are involved in the query predicates, for e.g. if we have `col_0,col_1,col_2` as the record key fields, then the query predicate `col_0 = val_0 and col_1 = val_1` does not work? Only `col_0 = val_0 and col_1 = val_1 and col_2 = val_2` should work if we do not matches the MDT RLI with prefix search of the keys. (need to add UT to confirm this part.)
Also we should be very careful with the field sequence in the returned `compositeRecordKeys`, if `col_0,col_1,col_2` are the record key fields, then literal `col_0:val_0,col_2:val_2,col_1:val_1` does not work either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #12336:
URL: https://github.com/apache/hudi/pull/12336#discussion_r1858966527
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkBaseIndexSupport.scala:
##########
@@ -155,32 +158,59 @@ abstract class SparkBaseIndexSupport(spark: SparkSession,
(List.empty, List.empty)
} else {
var recordKeyQueries: List[Expression] = List.empty
- var recordKeys: List[String] = List.empty
- for (query <- queryFilters) {
- val recordKeyOpt = getRecordKeyConfig
- RecordLevelIndexSupport.filterQueryWithRecordKey(query, recordKeyOpt).foreach({
- case (exp: Expression, recKeys: List[String]) =>
- recordKeys = recordKeys ++ recKeys
- recordKeyQueries = recordKeyQueries :+ exp
- })
+ var compositeRecordKeys: List[String] = List.empty
+ val recordKeyOpt = getRecordKeyConfig
+ val isComplexRecordKey = recordKeyOpt.map(recordKeys => recordKeys.length).getOrElse(0) > 1
+ recordKeyOpt.foreach { recordKeysArray =>
+ // Handle composite record keys
+ breakable {
+ for (recordKey <- recordKeysArray) {
+ var recordKeys: List[String] = List.empty
+ for (query <- queryFilters) {
+ {
+ RecordLevelIndexSupport.filterQueryWithRecordKey(query, Option.apply(recordKey), isComplexRecordKey).foreach {
+ case (exp: Expression, recKeys: List[String]) =>
+ exp match {
+ // For IN, add each element individually to recordKeys
+ case _: In => recordKeys = recordKeys ++ recKeys
+
+ // For other cases, basically EqualTo, concatenate recKeys with the default separator
+ case _ => recordKeys = recordKeys ++ recKeys
+ }
+ recordKeyQueries = recordKeyQueries :+ exp
+ }
+ }
+ }
+
+ if (recordKeys.isEmpty) {
+ recordKeyQueries = List.empty
+ compositeRecordKeys = List.empty
+ break()
+ } else if (!isComplexRecordKey || compositeRecordKeys.isEmpty) {
+ compositeRecordKeys = recordKeys
+ } else {
+ var tempCompositeRecordKeys: List[String] = List.empty
+ for (compRecKey <- compositeRecordKeys) {
Review Comment:
Added test cases. These cases are handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12336:
URL: https://github.com/apache/hudi/pull/12336#issuecomment-2501516506
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885e80032a81b10256e258dca03f4a10b6580858",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1965",
"triggerID" : "885e80032a81b10256e258dca03f4a10b6580858",
"triggerType" : "PUSH"
}, {
"hash" : "647f20e3c4f377425c1cce7988491c3aef1b3746",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "647f20e3c4f377425c1cce7988491c3aef1b3746",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 885e80032a81b10256e258dca03f4a10b6580858 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1965)
* 647f20e3c4f377425c1cce7988491c3aef1b3746 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #12336:
URL: https://github.com/apache/hudi/pull/12336#discussion_r1858966026
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/FunctionalIndexSupport.scala:
##########
@@ -92,7 +92,7 @@ class FunctionalIndexSupport(spark: SparkSession,
def filterQueriesWithFunctionalFilterKey(queryFilters: Seq[Expression], sourceFieldOpt: Option[String]): List[Tuple2[Expression, List[String]]] = {
var functionalIndexQueries: List[Tuple2[Expression, List[String]]] = List.empty
for (query <- queryFilters) {
- filterQueryWithRecordKey(query, sourceFieldOpt, (expr: Expression) => {
+ filterQueryWithRecordKey(query, sourceFieldOpt, false, (expr: Expression) => {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12336:
URL: https://github.com/apache/hudi/pull/12336#discussion_r1858236015
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkBaseIndexSupport.scala:
##########
@@ -155,32 +158,59 @@ abstract class SparkBaseIndexSupport(spark: SparkSession,
(List.empty, List.empty)
} else {
var recordKeyQueries: List[Expression] = List.empty
- var recordKeys: List[String] = List.empty
- for (query <- queryFilters) {
- val recordKeyOpt = getRecordKeyConfig
- RecordLevelIndexSupport.filterQueryWithRecordKey(query, recordKeyOpt).foreach({
- case (exp: Expression, recKeys: List[String]) =>
- recordKeys = recordKeys ++ recKeys
- recordKeyQueries = recordKeyQueries :+ exp
- })
+ var compositeRecordKeys: List[String] = List.empty
+ val recordKeyOpt = getRecordKeyConfig
+ val isComplexRecordKey = recordKeyOpt.map(recordKeys => recordKeys.length).getOrElse(0) > 1
+ recordKeyOpt.foreach { recordKeysArray =>
+ // Handle composite record keys
+ breakable {
+ for (recordKey <- recordKeysArray) {
+ var recordKeys: List[String] = List.empty
+ for (query <- queryFilters) {
+ {
+ RecordLevelIndexSupport.filterQueryWithRecordKey(query, Option.apply(recordKey), isComplexRecordKey).foreach {
+ case (exp: Expression, recKeys: List[String]) =>
+ exp match {
+ // For IN, add each element individually to recordKeys
+ case _: In => recordKeys = recordKeys ++ recKeys
+
+ // For other cases, basically EqualTo, concatenate recKeys with the default separator
+ case _ => recordKeys = recordKeys ++ recKeys
+ }
+ recordKeyQueries = recordKeyQueries :+ exp
+ }
+ }
+ }
+
+ if (recordKeys.isEmpty) {
+ recordKeyQueries = List.empty
+ compositeRecordKeys = List.empty
+ break()
+ } else if (!isComplexRecordKey || compositeRecordKeys.isEmpty) {
+ compositeRecordKeys = recordKeys
+ } else {
+ var tempCompositeRecordKeys: List[String] = List.empty
+ for (compRecKey <- compositeRecordKeys) {
Review Comment:
Still we need to check all the fields of the composition records keys are involved in the query predicates, for e.g. if we have `col_0,col_1,col_2` as the record key fields, then the query predicate `col_0 = val_0 and col_1 = val_1` does not work? Only `col_0 = val_0 and col_1 = val_1 and col_2 = val_2` should work if we do not matches the MDT RLI with prefix search of the keys.
Also we should be very careful with the field sequence in the returned `compositeRecordKeys`, if `col_0,col_1,col_2` are the record key fields, then literal `col_0:val_0,col_2:val_2,col_1:val_1` does not work either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12336:
URL: https://github.com/apache/hudi/pull/12336#issuecomment-2499850725
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885e80032a81b10256e258dca03f4a10b6580858",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "885e80032a81b10256e258dca03f4a10b6580858",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 885e80032a81b10256e258dca03f4a10b6580858 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12336:
URL: https://github.com/apache/hudi/pull/12336#issuecomment-2499853822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885e80032a81b10256e258dca03f4a10b6580858",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1965",
"triggerID" : "885e80032a81b10256e258dca03f4a10b6580858",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 885e80032a81b10256e258dca03f4a10b6580858 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1965)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12336:
URL: https://github.com/apache/hudi/pull/12336#discussion_r1858333353
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -162,18 +164,49 @@ object RecordLevelIndexSupport {
}
var literals: List[String] = List.empty
inQuery.list.foreach {
- case literal: Literal => literals = literals :+ literal.value.toString
+ case literal: Literal =>
+ val recordKeyLiteral = getRecordKeyLiteral(inQuery.value.asInstanceOf[AttributeReference], literal, isComplexRecordKey)
+ literals = literals :+ recordKeyLiteral
case _ => validINQuery = false
}
if (validINQuery) {
Option.apply(inQuery, literals)
} else {
Option.empty
}
+
+ // Handle And expression (composite filter)
+ case andQuery: And =>
+ val leftResult = filterQueryWithRecordKey(andQuery.left, recordKeyOpt, isComplexRecordKey, attributeFetcher)
+ val rightResult = filterQueryWithRecordKey(andQuery.right, recordKeyOpt, isComplexRecordKey, attributeFetcher)
+
+ // If both left and right filters are valid, concatenate their results
+ (leftResult, rightResult) match {
+ case (Some((leftExp, leftKeys)), Some((rightExp, rightKeys))) =>
+ // Return concatenated expressions and record keys
+ Option.apply(And(leftExp, rightExp), leftKeys ++ rightKeys)
Review Comment:
How are we checking the order of the keys? For examples, table config has keys in the following order : `recordKeyField1, recordKeyField2` while the query filter is `recordKeyField2='key2' AND re ordKeyField1='key1`. In that case, do we lookup RLI with key as `key2key1` or `key1key2`. Note that RLI will have keys in same order as keygen which has same order as fields in table config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12336:
URL: https://github.com/apache/hudi/pull/12336#discussion_r1858227779
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkBaseIndexSupport.scala:
##########
@@ -155,32 +158,59 @@ abstract class SparkBaseIndexSupport(spark: SparkSession,
(List.empty, List.empty)
} else {
var recordKeyQueries: List[Expression] = List.empty
- var recordKeys: List[String] = List.empty
- for (query <- queryFilters) {
- val recordKeyOpt = getRecordKeyConfig
- RecordLevelIndexSupport.filterQueryWithRecordKey(query, recordKeyOpt).foreach({
- case (exp: Expression, recKeys: List[String]) =>
- recordKeys = recordKeys ++ recKeys
- recordKeyQueries = recordKeyQueries :+ exp
- })
+ var compositeRecordKeys: List[String] = List.empty
+ val recordKeyOpt = getRecordKeyConfig
+ val isComplexRecordKey = recordKeyOpt.map(recordKeys => recordKeys.length).getOrElse(0) > 1
+ recordKeyOpt.foreach { recordKeysArray =>
+ // Handle composite record keys
+ breakable {
+ for (recordKey <- recordKeysArray) {
+ var recordKeys: List[String] = List.empty
+ for (query <- queryFilters) {
+ {
+ RecordLevelIndexSupport.filterQueryWithRecordKey(query, Option.apply(recordKey), isComplexRecordKey).foreach {
+ case (exp: Expression, recKeys: List[String]) =>
+ exp match {
+ // For IN, add each element individually to recordKeys
+ case _: In => recordKeys = recordKeys ++ recKeys
+
+ // For other cases, basically EqualTo, concatenate recKeys with the default separator
+ case _ => recordKeys = recordKeys ++ recKeys
+ }
+ recordKeyQueries = recordKeyQueries :+ exp
+ }
+ }
+ }
+
+ if (recordKeys.isEmpty) {
+ recordKeyQueries = List.empty
+ compositeRecordKeys = List.empty
+ break()
+ } else if (!isComplexRecordKey || compositeRecordKeys.isEmpty) {
+ compositeRecordKeys = recordKeys
+ } else {
+ var tempCompositeRecordKeys: List[String] = List.empty
+ for (compRecKey <- compositeRecordKeys) {
Review Comment:
This is for "Cartesian product" for m * n pattern: where m and n are enumerations of the possible values for a record key field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12336:
URL: https://github.com/apache/hudi/pull/12336#discussion_r1858243803
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala:
##########
@@ -240,4 +269,63 @@ class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
val fileNames = rliIndexSupport.computeCandidateFileNames(fileIndex, Seq(dataFilter), null, prunedPaths, false)
assertEquals(if (includeLogFiles) 2 else 1, fileNames.get.size)
}
+
+ @Test
+ def testRLIWithMultipleRecordKeyFields(): Unit = {
+ val tableName = "dummy_table_multi_pk"
+ val dummyTablePath = tempDir.resolve("dummy_table_multi_pk").toAbsolutePath.toString
+ val hudiOpts = Map(
+ "hoodie.insert.shuffle.parallelism" -> "4",
+ "hoodie.upsert.shuffle.parallelism" -> "4",
+ HoodieWriteConfig.TBL_NAME.key -> tableName,
+ DataSourceWriteOptions.RECORDKEY_FIELD.key -> "record_key_col,name",
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "partition_key_col",
+ DataSourceReadOptions.ENABLE_DATA_SKIPPING.key -> "true"
+ ) ++ metadataOpts
+
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | record_key_col string,
+ | name string,
+ | not_record_key_col string,
+ | partition_key_col string
+ |) using hudi
+ | options (
+ | primaryKey ='record_key_col,name',
+ | hoodie.metadata.enable = 'true',
+ | hoodie.metadata.record.index.enable = 'true',
+ | hoodie.datasource.write.recordkey.field = 'record_key_col,name',
+ | hoodie.enable.data.skipping = 'true'
+ | )
+ | partitioned by(partition_key_col)
+ | location '$dummyTablePath'
+ """.stripMargin)
+ spark.sql(s"insert into $tableName values('row1', 'name1', 'a', 'p1')")
+ spark.sql(s"insert into $tableName values('row2', 'name2', 'b', 'p2')")
+ spark.sql(s"insert into $tableName values('row3', 'name3', 'c', 'p3')")
+
+ val latestSnapshotDf = spark.read.format("hudi").options(hudiOpts).load(dummyTablePath)
+ this.metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(dummyTablePath)
+ .setConf(HoodieTestUtils.getDefaultStorageConf)
+ .build()
+ verifyEqualToQuery(hudiOpts, Seq("record_key_col", "name"), tableName, latestSnapshotDf, HoodieTableMetaClient.reload(metaClient))
+ verifyInQuery(hudiOpts, Array("record_key_col", "name"), latestSnapshotDf, tableName, "partition_key_col", Seq("p2", "p3"), true)
Review Comment:
Need to validate the `AND` with different predicates sequence:
- `record_key_col = k1 and name = n1`
- `name = n1 and record_key_col = k1`.
Need to validate the `IN` with different predicates sequence:
- `record_key_col in (k1, k2) and name in (n1, n2)`
- `name in (n1, n2) and record_key_col in (k1, k2)` .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #12336:
URL: https://github.com/apache/hudi/pull/12336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12336:
URL: https://github.com/apache/hudi/pull/12336#issuecomment-2501520714
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885e80032a81b10256e258dca03f4a10b6580858",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1965",
"triggerID" : "885e80032a81b10256e258dca03f4a10b6580858",
"triggerType" : "PUSH"
}, {
"hash" : "647f20e3c4f377425c1cce7988491c3aef1b3746",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1977",
"triggerID" : "647f20e3c4f377425c1cce7988491c3aef1b3746",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 885e80032a81b10256e258dca03f4a10b6580858 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1965)
* 647f20e3c4f377425c1cce7988491c3aef1b3746 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1977)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8432] Fix data skipping with RLI if record key is composite [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12336:
URL: https://github.com/apache/hudi/pull/12336#issuecomment-2501659498
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "885e80032a81b10256e258dca03f4a10b6580858",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1965",
"triggerID" : "885e80032a81b10256e258dca03f4a10b6580858",
"triggerType" : "PUSH"
}, {
"hash" : "647f20e3c4f377425c1cce7988491c3aef1b3746",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1977",
"triggerID" : "647f20e3c4f377425c1cce7988491c3aef1b3746",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 647f20e3c4f377425c1cce7988491c3aef1b3746 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1977)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org