You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bhat-vinay (via GitHub)" <gi...@apache.org> on 2024/04/02 17:15:30 UTC
[PR] [HUDI-7569] [RLI] Fix wrong result generated by query [hudi]
bhat-vinay opened a new pull request, #10955:
URL: https://github.com/apache/hudi/pull/10955
Record level index speeds up queries (when appropriate config properties are enabled) by pruning files based on metadata's RLI partition entries. The current implementation can prune files only when the query predicate has a EqalTo (i.e '=') OR In filters/expressions on the record-key column.
However, the logic to detect if a 'In' query predicate references a record-key column is buggy. This can result in wrong results when the query predicate is a `In` expression ona column other than record-key column. This seems like a serious bug that needs to be fixed.
This PR fixes this problem and adds a unit test for the same.
### Change Logs
However, the logic to detect if a 'In' query predicate references a record-key column is buggy. This can result in wrong results when the query predicate is a `In` expression ona column other than record-key column. This seems like a serious bug that needs to be fixed. This PR fixes this problem and adds a unit test for the same.
The changes introduced are:
RecordLevelIndexSupport.scala: The method `filterQueryWithRecordKey(...)` is where the bug exists. The switch-case for `In` expression was not filtering out expressions that are based on non-record-key columns. Fixed it correctly.
TestRecordLevelIndexWithSQL.scala: Adds a unit test that clearly shows the problem with wrong results and the changes in this PR that fixes the wrong result issue.
### Impact
Bug fix
### Risk level (write none, low medium or high below)
None
### Documentation Update
None. No new configs are user visible options/changes added
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7569] [RLI] Fix wrong result generated by query [hudi]
Posted by "bhat-vinay (via GitHub)" <gi...@apache.org>.
bhat-vinay commented on PR #10955:
URL: https://github.com/apache/hudi/pull/10955#issuecomment-2032614653
@codope tagging you for a review. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7569] [RLI] Fix wrong result generated by query [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #10955:
URL: https://github.com/apache/hudi/pull/10955#discussion_r1548866685
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala:
##########
@@ -155,4 +156,36 @@ class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
val readDf = spark.read.format("hudi").options(hudiOpts).load(basePath)
readDf.registerTempTable(sqlTempTable)
}
+
+ @Test
+ def testInFilterOnNonRecordKey(): Unit = {
Review Comment:
@bhat-vinay : Can you confirm if this unit-test fails without your change in RecordLevelIndexSupport.scala
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -162,7 +162,10 @@ class RecordLevelIndexSupport(spark: SparkSession,
case inQuery: In =>
var validINQuery = true
inQuery.value match {
- case _: AttributeReference =>
+ case attribute: AttributeReference =>
Review Comment:
Good catch !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7569] [RLI] Fix wrong result generated by query [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10955:
URL: https://github.com/apache/hudi/pull/10955#issuecomment-2032724837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0587be7239c626084fc8657ae55f3818519c568b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23081",
"triggerID" : "0587be7239c626084fc8657ae55f3818519c568b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0587be7239c626084fc8657ae55f3818519c568b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23081)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7569] [RLI] Fix wrong result generated by query [hudi]
Posted by "bhat-vinay (via GitHub)" <gi...@apache.org>.
bhat-vinay commented on code in PR #10955:
URL: https://github.com/apache/hudi/pull/10955#discussion_r1548899945
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala:
##########
@@ -155,4 +156,36 @@ class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
val readDf = spark.read.format("hudi").options(hudiOpts).load(basePath)
readDf.registerTempTable(sqlTempTable)
}
+
+ @Test
+ def testInFilterOnNonRecordKey(): Unit = {
Review Comment:
Yes. Without the fix, the test query return 0 rows - as the file pruning happens based on record-key (even when the query predicate references non-record-key col), thus incorrectly returning candidate files from partition 'p1'.
```
org.opentest4j.AssertionFailedError:
Expected :2
Actual :0
<Click to see difference>
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:166)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:161)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:611)
at org.apache.hudi.functional.TestRecordLevelIndexWithSQL.testInFilterOnNonRecordKey(TestRecordLevelIndexWithSQL.scala:189)
.......
.......
.......
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7569] [RLI] Fix wrong result generated by query [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10955:
URL: https://github.com/apache/hudi/pull/10955#issuecomment-2032709320
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0587be7239c626084fc8657ae55f3818519c568b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0587be7239c626084fc8657ae55f3818519c568b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0587be7239c626084fc8657ae55f3818519c568b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7569] [RLI] Fix wrong result generated by query [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar merged PR #10955:
URL: https://github.com/apache/hudi/pull/10955
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7569] [RLI] Fix wrong result generated by query [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10955:
URL: https://github.com/apache/hudi/pull/10955#issuecomment-2033006127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0587be7239c626084fc8657ae55f3818519c568b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23081",
"triggerID" : "0587be7239c626084fc8657ae55f3818519c568b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0587be7239c626084fc8657ae55f3818519c568b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23081)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7569] [RLI] Fix wrong result generated by query [hudi]
Posted by "bhat-vinay (via GitHub)" <gi...@apache.org>.
bhat-vinay commented on code in PR #10955:
URL: https://github.com/apache/hudi/pull/10955#discussion_r1548899945
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRecordLevelIndexWithSQL.scala:
##########
@@ -155,4 +156,36 @@ class TestRecordLevelIndexWithSQL extends RecordLevelIndexTestBase {
val readDf = spark.read.format("hudi").options(hudiOpts).load(basePath)
readDf.registerTempTable(sqlTempTable)
}
+
+ @Test
+ def testInFilterOnNonRecordKey(): Unit = {
Review Comment:
Ye. Without the fix, the test query return 0 rows - as the file pruning happens based on record-key (even when the query predicate references non-record-key col), thus incorrectly returning candidate files from partition 'p1'.
```
org.opentest4j.AssertionFailedError:
Expected :2
Actual :0
<Click to see difference>
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:166)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:161)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:611)
at org.apache.hudi.functional.TestRecordLevelIndexWithSQL.testInFilterOnNonRecordKey(TestRecordLevelIndexWithSQL.scala:189)
.......
.......
.......
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org